4 steps toward scientific a/b testing
DESCRIPTION
To build a successful A/B testing strategy, you'll need more than just ideas of what to test, you'll need a plan that builds data into a repeatable strategy for producing winning experiments.TRANSCRIPT
![Page 1: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/1.jpg)
![Page 2: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/2.jpg)
What is A/B Testing?
![Page 3: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/3.jpg)
MYTH BUSTING
![Page 4: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/4.jpg)
#ScienceOfTesting
A/B testing is not...
Validation of guesswork
Consumer psychology gimmicks “Meek Tweaking”
Images: Hubspot, Conversion Rate Experts
![Page 5: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/5.jpg)
#ScienceOfTesting
It’s also not...
A waste of time Impossible to
get right Beyond the scope of
your job
![Page 6: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/6.jpg)
A/B Testing: Defined
Conducting experiments to optimize your customer experience.
What is A/B testing?
OR
![Page 7: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/7.jpg)
4 Steps of Scientific A/B Testing
![Page 8: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/8.jpg)
#ScienceOfTesting
The 4 Steps of A/B Testing
Step 1
Analyze data
Step 2
Form a hypothesis
Step 3
Construct an experiment
Step 4
Interpret results
![Page 9: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/9.jpg)
STEP 1 | ANALYZE DATA
![Page 10: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/10.jpg)
Asking the right questions is hard. Arm yourself with data. #ScienceOfTesting
![Page 11: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/11.jpg)
#ScienceOfTesting
Use quantitative & qualitative data
Quantitative data tells you
where to test
Qualitative data gives you an idea of
what should be tested
![Page 12: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/12.jpg)
#ScienceOfTesting
Quantitative datasets
• Web traffic • Email marketing
• Order history • CRM interactions • Support tickets
…and more!
![Page 13: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/13.jpg)
#ScienceOfTesting
Run high-impact tests
![Page 14: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/14.jpg)
Don’t choose tests randomly
Access this spreadsheet in this blog post: http://blog.optimizely.com/2014/07/02/how-to-use-data-to-choose-your-next-ab-test/
![Page 15: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/15.jpg)
#ScienceOfTesting
Qualitative data
• User testing • Survey data • Heat mapping • Your sales & account teams
![Page 16: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/16.jpg)
STEP 2 | FORM A HYPOTHESIS
![Page 17: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/17.jpg)
#ScienceOfTesting
Parts of a hypothesis
“If [Variable], then [Result], because [Rationale].”
• The element that is modified • Isolate one variable for an A/B test • Call to action, visual media, forms
![Page 18: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/18.jpg)
#ScienceOfTesting
Parts of a hypothesis
“If [Variable], then [Result], because [Rationale].”
• The predicted outcome • Use data to determine the size of effect • More email sign-ups, clicks on a CTA
![Page 19: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/19.jpg)
#ScienceOfTesting
Parts of a hypothesis
“If [Variable], then [Result], because [Rationale].”
• Demonstrate your customer knowledge • What assumption will be proven wrong if
the experiment is a draw or loses?
![Page 20: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/20.jpg)
#ScienceOfTesting
All hypotheses are not created equal
Weak Hypothesis
If the call-to-action is shorter, the conversion rate will increase.
Strong Hypothesis
If the call-to-action text is changed to “Complete My Order,” the conversion rates in the checkout will increase, because the copy is more specific and personalized.
![Page 21: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/21.jpg)
#ScienceOfTesting
All hypotheses are not created equal
Weak Hypothesis
If the checkout funnel is shortened to fewer pages, the checkout completionrate will increase.
Strong Hypothesis
If the navigation is removed from checkout pages, the conversion rate on each step will increase because our website analytics shows portions of our traffic drop out of the funnel by clicking on these links.
![Page 22: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/22.jpg)
STEP 3 | CONSTRUCT AN EXPERIMENT
![Page 23: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/23.jpg)
A/B Testing: Defined Every test
has 3 parts
DESIGN TECH
CONTENT
![Page 24: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/24.jpg)
#ScienceOfTesting
Content: What are you saying?
VS.
![Page 25: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/25.jpg)
#ScienceOfTesting
Design: How does it look?
VS.
![Page 26: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/26.jpg)
#ScienceOfTesting
Tech: How does it work?
VS.
![Page 27: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/27.jpg)
The most effective tests often combine all 3 elements: content, design, tech
#ScienceOfTesting
![Page 28: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/28.jpg)
STEP 4 | EVALUATE RESULTS
![Page 29: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/29.jpg)
#ScienceOfTesting
What are we looking for?
• How confident am I that the observed difference from my experiment was not due to chance?
• 95% Statistical Significance = 5% probability that the observed difference was due to chance.
![Page 30: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/30.jpg)
#ScienceOfTesting
Confidence intervals
High statistical confidence
Lower risk of implementing a test that won by chance
![Page 31: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/31.jpg)
#ScienceOfTesting
Sample size calculator
http://optimize.ly/StatCalculator
![Page 32: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/32.jpg)
#ScienceOfTesting
Once you reach significance:
• Variation wins: Launch the variation or update your website.
• Original wins: Learn why hypothesis was incorrect.
• In either case: Think about what to test next.
![Page 33: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/33.jpg)
Examples!
![Page 34: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/34.jpg)
A/B Testing: Defined A simple test
![Page 35: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/35.jpg)
A/B Testing: Defined
Iterative testing on a core hypothesis
A solid test
![Page 36: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/36.jpg)
A/B Testing: Defined
Cohort analysis + website changes + biz process changes
A more complicated test
A
B
![Page 37: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/37.jpg)
#ScienceOfTesting
Step 1: Data collection
![Page 38: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/38.jpg)
#ScienceOfTesting
Step 2: Hypothesis
“If [Variable], then [Result], because [Rationale].”
If prospects’ access to a free trial is gated by a conversation with a sales rep, we’ll be able to increase prospect to trial conversion rate. Talking to sales will ensure all their questions get answered, improving their overall experience and increasing willingness to take the next step with RJMetrics.
![Page 39: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/39.jpg)
#ScienceOfTesting
Step 3: Experiment
• Changes to heading text • Custom fields in Salesforce.com • Business process changes for
sales reps • Custom analysis in RJMetrics
based on offline conversion event
![Page 40: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/40.jpg)
#ScienceOfTesting
Step 4: Results
TBD A
B
![Page 41: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/41.jpg)
Arm Your Organization
![Page 42: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/42.jpg)
Marketing
Increase the impact of your tests by bringing more team members into theprocess #ScienceOfTesting
Product
Sales
Engineering
![Page 43: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/43.jpg)
Document your test results in a central repository. #ScienceOfTesting
Heat maps
Optimizely results
Hypothesis
What we learned
Variations
![Page 44: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/44.jpg)
#ScienceOfTesting
Other tried and true tactics
• Build excitement by sharing your wins with the company • Hold a competition for the biggest winning variation • Votes on variations to see who has the highest accuracy
of predicting winners
![Page 45: 4 Steps Toward Scientific A/B Testing](https://reader034.vdocument.in/reader034/viewer/2022051322/5409cd8b8d7f72e54b8b459e/html5/thumbnails/45.jpg)
Thanks!