optimizing website conversions through a/b split testing
Post on 19-Oct-2014
588 views
DESCRIPTION
In this presentation Javid Jamae CTO/Co-founder of Camperoo (www.camperoo.com) will teach you how to define, run, and measure A/B tests and other experiments in your Lean Startup. This was presented at the Houston Lean Startup Circle on June 18, 2013.TRANSCRIPT
![Page 1: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/1.jpg)
OPTIMIZINGCONVERSIONSTHROUGHA/B SPLIT TESTING
Javid JamaeCTO / Co-founder CamperooPresented at the Houston Lean Startup CircleJune 18, 2013
Special thanks to - Emiliano Villareal
![Page 2: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/2.jpg)
‘‘The goal of a Lean Startup is to inform our riskiest business model assumptions through
empirical testing with customers – not rhetorical reasoning on a white board.
~ ASH MAURYA
![Page 3: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/3.jpg)
![Page 4: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/4.jpg)
AirBNB
Zappos
![Page 5: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/5.jpg)
![Page 6: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/6.jpg)
Users coming in through a Zappos-style homepage will be 10% more likely
to go from the homepage into a product search results page
EXPERIMENTDESIGN
FALSIFIABLE HYPOTHESIS
EXPERIMENT SCOPE40 man-hours to design and code750 unique visitors per variant
![Page 7: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/7.jpg)
UNIQUE VISITORS RANDOMIZER ANALYTICS
CONTROL
VARIANT
EXPERIMENTIMPLEMENTATION
50%
50%
![Page 8: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/8.jpg)
THEWINNERWAS?
5.86% ??
Percentage of people going from the homepage to a product search results page
Control Variant
![Page 9: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/9.jpg)
THEWINNERWAS?
5.86% 6.84%
16.89% increase!Percentage of people going from the homepage
to a product search results page
Control Variant
![Page 10: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/10.jpg)
BUTWHY?
Was it the:- Filters- Category browsing- Different slider- The combination
?
![Page 11: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/11.jpg)
In Lean Startups, success is defined by learning, not by test results
X
A/B TOOL √
ValidatedLearning
Loop
√Variant
![Page 12: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/12.jpg)
+
![Page 13: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/13.jpg)
Users coming into an AirBNB-style homepage with categories will be 5%
more likely to go into a product search results page than without categories
EXPERIMENTDESIGN
FALSIFIABLE HYPOTHESIS
EXPERIMENT SCOPE1 man-hours to design and code750 unique visitors per variant
![Page 14: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/14.jpg)
THEWINNERWAS?
5.90% ??
Percentage of people going from the homepage to a product search results page
Control Variant
![Page 15: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/15.jpg)
THEWINNERWAS?
5.90% 5.98%
Percentage of people going from the homepage to a product search results page
Control Variant
1.35% increase
![Page 16: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/16.jpg)
THELEANPROCESS
IDEA
HYPOTHESIS
EXPERIMENT DESIGN
EXPERIMENTATION
MEASUREMENT
Scientific= Process
![Page 17: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/17.jpg)
IDEA
HYPOTHESIS
EXPERIMENT DESIGN
EXPERIMENTATION
MEASUREMENT
Prioritize
Define
Design
Implement
Track
How do I...
![Page 18: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/18.jpg)
PRIORITIZINGEXPERIMENTS
![Page 19: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/19.jpg)
‘‘If you measure something and it’s not attached to a goal, in turn changing your
behavior, you’re wasting your time.
ALISTAIR CROLL & BENJAMIN YOSKOVITZ BOOK: LEAN ANALYTICS
![Page 20: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/20.jpg)
GOALORIENTEDPLANNING
Landing PageChanges
SEO Rankings
Goal: IncreaseAcquisition
InboundChannels
![Page 21: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/21.jpg)
AcquisitionActivationReferralRevenueRetention
PIRATEMETRICS
![Page 22: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/22.jpg)
OneMetricThat
Matters
![Page 23: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/23.jpg)
Lean Analytics (Lean Series). O’Reilly Media.
![Page 24: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/24.jpg)
Stage
RISKSACROSSSTAGESPr
oduc
tC
usto
mer
Mar
ket
Problem/Solution
Identify
Competition/Positioning
Early / Inception
MVP
Narrow/Outbound
Pricing
Product
Scale
Inbound
Cost Structure
Scale
RisksQualitative,
Activation / RetentionQuantitative,
Revenue / Growth
![Page 25: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/25.jpg)
Favor experiments that
make sense for your current stage & goals
![Page 26: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/26.jpg)
Favor experiments that
are innovative
![Page 27: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/27.jpg)
Favor experiments that
have high risk
![Page 28: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/28.jpg)
Favor experiments that
you can measure
![Page 29: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/29.jpg)
Favor experiments that
engaged customers seem to need or are asking for
![Page 30: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/30.jpg)
Theory of Constraints
A chain is no stronger than its weakest link.
The Goal - Eliyahu M. Goldratt
![Page 31: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/31.jpg)
DEFININGHYPOTHESES
![Page 32: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/32.jpg)
Is Manchester United causingglobal warming?
![Page 33: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/33.jpg)
http://theconversation.com/how-david-beckham-caused-global-warming-the-man-u-climate-model-4548
![Page 34: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/34.jpg)
‘‘If you torture the data long enough, it will confess to anything.
~UNKNOWN
![Page 35: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/35.jpg)
Don’t take Leaps of Faith
![Page 36: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/36.jpg)
Leap of faith: A Zappos-style front page will increase browsability on our website
Falsifiable hypothesis: Users coming in through a Zappos-style homepage will be 10% more likely to go from the homepage into a product search results page
(vs. the AirBNB-style homepage)
![Page 37: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/37.jpg)
Formula:
[Repeatable Action] will [Expected Result]
Writing a blog post about Houston activities will result in 25 new accounts
Users entering Variation B of the store page will be 2% more likely to purchase
Putting an item on sale will increase the chances that it will be added to a cart by 10%
![Page 38: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/38.jpg)
Draw a linein the sand...
...before you start gathering data.
![Page 39: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/39.jpg)
DESIGNINGEXPERIMENTS
![Page 40: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/40.jpg)
Are you measuringimpact on a
Goal or a Metric
![Page 41: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/41.jpg)
LEARNINGFROMDATA
A/B Testing
Cohort AnalysisAnalysis
Testing
Longitudinal Cross-Sectional
Multivariate Testing
Segmentation
(Conditional)Feature Rollout
![Page 42: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/42.jpg)
Decide on a exit criteria up front
![Page 43: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/43.jpg)
Cost It will take 5-10 hours to implement, if
it takes longer, we will abandon.
ScopeWe need 1,000 unique visitors per variant
before the data has any meaning to us.
TimeboxIf the test doesn’t end by July 2, then we
will abandon.
Follow upIf the test fails, then try experiment C. If
the test passes, then experiment B.
![Page 44: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/44.jpg)
Pick the type of analysis
![Page 45: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/45.jpg)
TYPESOFANALYSIS
JanuaryFebruary
MarchApril
May
segmentation(cross-sectional)
cohortanalysis
(longitudinal)
![Page 46: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/46.jpg)
Pick the type of test
![Page 47: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/47.jpg)
UNIQUE VISITORS RANDOMIZER
CONTROL
VARIANT
Version A
Version B
A/B
![Page 48: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/48.jpg)
UNIQUE VISITORS RANDOMIZER
CONTROL
VARIANT 1
A/B/N
VARIANT 2
VARIANT 3
![Page 49: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/49.jpg)
MULTIVARIATETESTING
Enroll Sign Up Join Now
Color:
Text:
Size:
27 Combinations
![Page 50: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/50.jpg)
MULTIVARIATETESTINGEnroll Sign Up Join Now
Enroll Sign Up Join Now
Enroll Sign Up Join Now
Enroll Sign Up Join Now
Enroll Sign Up Join Now
Enroll Sign Up Join Now
Enroll
Enroll
Enroll
Sign Up
Sign Up
Sign Up
Join Now
Join Now
Join Now
![Page 51: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/51.jpg)
IMPLEMENTINGEXPERIMENTS
![Page 52: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/52.jpg)
Amazon A/B Testing Service (beta)
FREETOOLS
Google Analytics - Experiments APIGoogle Analytics - Content Experiments
![Page 53: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/53.jpg)
WYSIWYGTOOLS
![Page 54: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/54.jpg)
ENTERPRISETOOLS
![Page 55: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/55.jpg)
http://www.roidna.com/tools/ab-testing-tool/
![Page 56: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/56.jpg)
THECAMPEROOWAY?
github.com/andrew/split
SPLIT + MIXPANELOpen Source, Ruby
Freemium,SAAS
mixpanel.com
![Page 57: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/57.jpg)
TRACKINGEXPERIMENTS
![Page 58: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/58.jpg)
Goal Build Measure Learn
IncreaseAcquisition
ZapposRedesign
AirBNBwith category
navigation
Search engine display of
customer ratings
IncreaseActivation
Draconian modal dialog
box
KANBANBOARD
![Page 59: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/59.jpg)
VALIDATEDLEARNINGBOARD
http://www.ashmaurya.com/2012/06/the-lean-stack/
![Page 60: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/60.jpg)
EXPERIMENTREPORTING
http://spark59.s3.amazonaws.com/presentations/All%20About%20Experiments.pdf
![Page 61: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/61.jpg)
WEEKLYPLANNING
Weekly meeting:
Review experiment results
Challenge interpretation
Kill or restart overdue experiments
Decide next experiments
![Page 62: Optimizing website conversions through A/B split testing](https://reader033.vdocument.in/reader033/viewer/2022051411/5444af71afaf9fa0098b49f6/html5/thumbnails/62.jpg)
THEEND
...to be continued?