User Analytics

TestingMarcus Merrell, RetailMeNot, inc

@mmerrell

What We’re Talking about

Overview of Web/User Analytics

Explanation of A/B Testing

Why this matters to you

Examples

How we test this stuff

User Analytics – The Basics

Hits

Sessions

Users

User Analytics - Services

Web Analytics - Advanced

Conduct Experiments

Tell stories from disparate points of data

Incremental learning

“If you are not paying for it… you are the

product being sold”

--Andrew Lewis (blue_beetle)

A/B Testing Explained

Basics

Don’t change everything at once

The Highball Incident

A Newer Example

Our 404 page had nothing on it

People landed there a lot by mis-typing store names

Should we put coupons on it?

A/B Test:

A: Control – No coupons on 404 page

B: Test – Coupons on 404 page

Keep it simple: top 15 coupons site-wide

Slice in 10% of traffic

RetailMeNot

User Analytics – Telling a Story

OK, so people did some clicking How many?

How many resulted in a transaction?

The big question: The amount of money we expect to make from

coupons on the 404 page: Is it worth the bandwidth? The load on the

servers/database?

Is it worth the potential future maintenance of this page?

Drawing a Conclusion Results after 2 weeks of testing tell us that the

B variation won!

Enough people used coupons, so it justified the relatively low expense

Ergo: Continue to put coupons on the page

Promote “B” test to “A”

New A/B Test: should we indicate coupon popularity on the 404 page coupons?

A: Control – No popularity indication

B: Test – Indicate coupon popularity

Real-world Examples

Shopping cart—shipping & tax calculation

Suggesting products and content based

on cookie, not login

Why You Should Care

What if the beacons they’re sending contain the wrong information?

But furthermore…

This is everywhere

It is only growing

Companies are becoming smart

(Really really smart)

You do not want to miss this opportunity to provide value

Why You Should Really Care

As a tester: There is a team of people working on this

It gets worked into features as they are developed

It is rarely called out separately in a scheduled task

It rarely receives QA outside of the PM and BI people who really care about it

*This is anecdotal, but I have yet to be told I’m wrong

Fortunately, It’s Easy

Usually one extra HTTP request, made

during a navigation event

Intercept this request, then verify the data

within it

Examples

Wells Fargo (s.gif)

Amazon (a.gif)

Netflix ([….]?trkId=xxx, beacon?s=xxx)

The New York Times (pixel.gif, dcs.gif)

OpenTable (/b/ss/otcom)

(and RetailMeNot – can you find it?)

Classic Approach

Marketing asks the BI team to figure out our ROI on TV ads during a period of time

BI requests PM to create a series of analytics

PM gives Dev the particulars

Dev assigns the code task to the newest person on the team

If anyone tests it, it’s also the newest person on the team

Classic Approach

Manual testing of web analytics is about

as exciting as reconciling a large column

of data with another large column of

data

…what if it’s wrong?

…what if it changes?

…why not let the software do it?

What We Do

Test Cycle

@Test

Launch Browser

Navigate to

Position

Start Proxy

Perform Main

Action

Stop Proxy

Clean-up Test

Execution

CI Maven TestNG

Our Tech Stack

Reporting

Report to a dashboard

Indicates “PASS”, “FAIL”, and “Staleness”

Conclusion

User Analytics are your CEO’s favorite

subject!

Deliver real value—million-dollar decisions

are made with this data

Can be implemented with just as many

bugs as any other kind of software

Questions?

@mmerrell

mmerrell@rmn.com