Making Decisions With Data: Moving Beyond Basic A/B Testing

Anthony RindoneProduct Manager – WayfairTwitter: @AnthonyRindone

https://www.linkedin.com/in/arindone

Housekeeping❖ Connect to the "Cambridge" wireless network❖ Open an Internet browser❖ You'll be redirected to a logon page

❖When prompted, use the code: pc0409

❖ Follow us @PCampBoston

❖ Official Hashtag – include in tweets: #PCampBoston

Session Evaluationswww.lanyrd.com/2016/productcamp-boston-sessions/schedule/

OR download Lanyrd App

The Fun Stuff

We’re Discussing Beyond The Basics…

1-1 Personalization & Automation

A/B & Multivariate Testing

“On-Demand” or Ad-Hoc Analytics

Reporting & KPI Scorecards

Raw Data Collected

Analytics Themes – Increasing in Maturity:

Focus for Today

Making Decisions with Data: A-B Testing Secrets They Don’t Want You To Know!

1.

2.

3.

4.

5.

CEOs hate him!

Making Decisions with Data: Advanced A/B Testing Principles

1.

2.

3.

4.

5.

❑ What set of actionable guidelines can help us make better Product/Marketing/Operations decisions?

❑ What does it mean to make “better” decisions?

Making Decisions with Data: Advanced A-B Testing Principles

1.

2.

3.

4.

5.

Principle 1: Averages are your enemy - focus on time series

Principle 1: Averages are your enemy - focus on time series

16% increase in conversion! We win!...Right???

Anti-Pattern: Results are outputs as summary tables in Excel with averages leading the way. All days and all users are considered equal.

Better Solution:

Principle 1: Averages are your enemy - focus on time series

Better Solution:

Expose hidden trends via cohort views and time series

Look for “steady-state” or equilibrium trends

Principle 2: Beware of perverse incentives - focus on the bigger picture

Principle 2: Beware of perverse incentives - focus on the bigger picture

Hypothetical Product Manager Goal: Grow app downloads by 10%

Is there a better way?

Principle 2: Beware of perverse incentives - focus on the bigger picture

Principle 2: Beware of perverse incentives - focus on the bigger picture

Principle 2: Beware of perverse incentives - focus on the bigger picture

Anti-Pattern: Grow app downloads by 10% (or some specific metric in a vacuum)

Better Solution: A single metric that determines the success and failure of the test that will determine what’s best for your site as a whole.

- Everything else are drivers or “secondary” KPI

Examples: 7-day revenue-per-user (7D$RPU) post-exposure, daily active users (DAU)

Principle 3: Everything is a bet - focus on adaptability

Principle 3: Everything is a bet - focus on adaptability

Anti-Pattern: How long do we need to run this test for the data to be significant?● What’s current “conversion rate?● What’s daily traffic look like?● (Assumption: “significant” = 95% confidence with binomial distribution.)● Analyst recommends 3 months or 2 days (!!)

Better Solution: Balance risk appropriately; separate “significance” from “confidence in the data.” (Especially pertinent for lower-traffic areas / sites.)

Principle 4: Beware of post-hoc storytelling - focus on a priori hypothesizing

Principle 4: Beware of post-hoc storytelling - focus on a priori hypothesizing

Anti-Pattern: Cherry-picking data that proves your story - and tossing out everything that doesn’t

Better Solution: Focus on your model, your Primary KPI, and your drivers that you are testing - everything else can be new hypothesis to test later

Principle 5: Avoid Product-Centric mentality - focus on User-Centric approaches

Principle 5: Avoid Product-Centric mentality - focus on User-Centric approaches

Peter Fader - Customer Centricity: Focus on the Right Customers for Strategic Advantage:

“Not all customers deserve your best efforts: in the world of customer centricity, there are good customers…and then there is pretty much everybody else.”

Anti-Pattern Hypotheses:● “We’ll launch this feature as long as long as it doesn’t tank the site.” -- (Smoke

Tests)○ Better: Not always bad, but keep minimal in your testing portfolio.○ “We’ll launch this feature as long as users respond at least neutrally according

to X KPI.”● “We will know we are successful when conversion increases x% with this test.”

○ Better: “Same-session conversion should increase x% with this test for this segment of users for this variation. For other segments, another variation will drive y% increase in same-session conversion.”

● “We should improve the user experience for everyone.”○ Better: “We should optimize towards our targeted demographic - our most

valuable users.”

Conclusion: Print Out This Slide!

• Averages are your enemy - focus on time series

• Beware of perverse incentives - focus on the bigger picture

• Everything is a bet - focus on adaptability

• Beware of post-hoc storytelling - focus on a priori hypothesizing

• Avoid Product-centric mentality - focus on User-centric approaches