[#500distro] measuring for impact: knowing when, what & how to a/b test

of 25 /25
Measuring For Impact: Knowing What, and How to A/B Test @mike_greenfield CEO/Co-Founder, Laserlike 2014-08-07 @mike_greenfield

Author: 500-startups

Post on 23-Aug-2014




8 download

Embed Size (px)


[#500Distro] Measuring for Impact: Knowing When, What & How to A/B Test


  • Measuring For Impact: Knowing What, and How to A/B Test @mike_greenfield CEO/Co-Founder, Laserlike 2014-08-07 @mike_greenfield
  • You know you should A/B test. @mike_greenfield
  • You also know you should exercise more eat less sugar spend less on coffee wear sunscreen etc., etc. @mike_greenfield
  • (Dont worry, Im not going to say anything else about sugar or sunscreen.) @mike_greenfield
  • So, how do you create a culture in which people will constructively A/B test? Do six things. @mike_greenfield
  • 1. Embrace I dont know We have 2+ ideas. I dont know which one will be more effective. @mike_greenfield
  • @mike_greenfield
  • 2. Have Data, Choose Metrics To test, you need: People using your product (Approximate) agreement on the metrics that matter @mike_greenfield
  • Not Many Users? Dont A/B test! Laserlike, has ~60 users and has never run an A/B test We will run many, many tests when we have enough users A test should have at least a few hundred instances (and a lot more if effect sizes are likely to be small) Test iff you can have business significance @mike_greenfield
  • Know What You Want to Optimize If its important, you should be running tests to improve it If its not important, spend time on other things Most tests should be aimed at improving 1-2 specific variables @mike_greenfield
  • 3. Have Clear Process, Tech for Testing @mike_greenfield
  • A/B Testing Process New feature: if possible, roll out to a small test subset first (10s or 100s of thousands) Version change: always test things that could (cumulatively) have business impact Everyone on the product team should be running and resolving tests @mike_greenfield
  • A/B Testing Tech Using a third party testing service is akin to building your site on Wordpress: great at some scales/competency levels No matter how youre testing, a new test should be at most a few lines of code It should be easy to see how each side of a test compares across many variables @mike_greenfield
  • 4. Understand the Math of What to Test @mike_greenfield
  • Process: Same vs. New Tweak Whats the probability your tweak will have a positive effect? What kind of effect might that have, and how might that effect change the companys prospects? Will you be able to measure the change? Optimize on one variable, but look at others @mike_greenfield
  • Process: Same vs. Big Change Whats the probability that your change will have a negative impact? How big an impact might there be? Will you be able to measure the change? Holistic approach @mike_greenfield
  • A/B Test for Quality Circle of Moms: test warning users when questions seemed short, low quality Resulting questions were graded for quality, without grader knowing test bucket End result: warning yielded ~5% fewer questions, but much higher quality @mike_greenfield
  • 5. Understand the Math of Picking Winners @mike_greenfield
  • Resolving Too Soon vs. Resolving Too Late How big is the potential audience for this test? Example 1: end of year most popular baby names email that will never be sent again Example 2: Facebook signup flow @mike_greenfield
  • Longitudinal Tests vs. Immediate Tests Longitudinal: change home page, email frequency, product framing Need to examine effect over a long period Immediate: change button color, email subject Likely that long-term effects will be minimal @mike_greenfield
  • Automatically Resolve Tests? Longitudinal tests should not be automatically resolved Example: new home page design Immediate tests can be automatically resolved when speed is important and there is one clear objective function Example: Circle of Moms email subject optimization @mike_greenfield
  • Choose robust statistics Bad: # of page views Good: % of users viewing at least [5, 25, 100] pages Potentially bad: # of sales (when small) Potentially good: # of people getting through the second step of a sales funnel @mike_greenfield
  • 6. Celebrate A/B Testing Successes @mike_greenfield
  • @mike_greenfield
  • Thanks. [email protected] @mike_greenfield @mike_greenfield