testing beyond the button color - the definitive guide to a/b testing
TRANSCRIPT
TESTING BEYOND THE BUTTON COLOR
The definitive guide to A/B testing
Ron Schneidermannhttps://angel.co/ron-schneidermann
https://www.linkedin.com/in/schneidermann
The Easy StuffPart 1
What is A/B testing?
A/B testing (also known as split testing or bucket testing) is a method of
comparing two versions of a webpage or app against each other to determine
which one performs better.
optimizely.com/ab-testing/
Alright, that’s easy enough.
But tell me again why I care about this?
But the real secret…
Data > ego
Ego in a startup???
The truth is you don’t know shit.
And I don’t know shit.
But your users know what’s up
Three Pre Testing Questions
Part 2
c/o http://www.coelevate.com/essays/5-steps-to-choose-your-customer-acquisition-channel
Question #1:Are we at the right phase in our
lifecycle for testing?
Question #2:What are we optimizing for?
Different phases of your company lifecycle call for optimizing different things:
Different phases of your company lifecycle call for optimizing different things:
Learning
Different phases of your company lifecycle call for optimizing different things:
LearningProduct channel fit?
Different phases of your company lifecycle call for optimizing different things:
LearningProduct channel fit?Audience segmentation?
Different phases of your company lifecycle call for optimizing different things:
LearningProduct channel fit?Audience segmentation?Engagement?
Different phases of your company lifecycle call for optimizing different things:
LearningProduct channel fit?Audience segmentation?Engagement?
Monetization?
Different phases of your company lifecycle call for optimizing different things:
LearningProduct channel fit?Audience segmentation?Engagement?
Monetization?Conversion rate?
Different phases of your company lifecycle call for optimizing different things:
LearningProduct channel fit?Audience segmentation?Engagement?
Monetization?Conversion rate?Messaging?
Different phases of your company lifecycle call for optimizing different things:
Learning
Volume
Different phases of your company lifecycle call for optimizing different things:
Learning
VolumeHow far can I crank the dial on a given channel?
Different phases of your company lifecycle call for optimizing different things:
Learning
Volume
Profitability
Different phases of your company lifecycle call for optimizing different things:
Learning
Volume
ProfitabilityCan I drive down CPAs, reduce churn, increase AOV?
Question #3:What are our constraints?
Time
Money
Target Audience
Legal
Part 3
How to Find Good Test
Hypotheses
Analytics
User testingWhat you think your users do
What your usersactually do
Heat Map Tracking
Qualitative Feedback
Competitor Research
Ideas to TestPart 4
1. Landing page ideas to test:
Messaging
1. Landing page ideas to test:
Messaging
Form fields
1. Landing page ideas to test:
Messaging
Layout (long scroll vs. short, # of columns)
Form fields
1. Landing page ideas to test:
Messaging
Layout (long scroll vs. short, # of columns)Plan background vs. rich imagery
Form fields
1. Landing page ideas to test:
Messaging
Layout (long scroll vs. short, # of columns)Plan background vs. rich imagery# of CTAs
Form fields
1. Landing page ideas to test:
Messaging
Layout (long scroll vs. short, # of columns)Plan background vs. rich imagery# of CTAs
CTA copy
Form fields
1. Landing page ideas to test:
Messaging
Layout (long scroll vs. short, # of columns)Plan background vs. rich imagery# of CTAs
CTA copy
Form fields
Social proof (testimonials/endorsements/as-seen-in)
1. Landing page ideas to test:
Messaging
Layout (long scroll vs. short, # of columns)Plan background vs. rich imagery# of CTAs
CTA copy
Video vs. none
Form fields
Social proof (testimonials/endorsements/as-seen-in)
1. Landing page ideas to test:
Messaging
Layout (long scroll vs. short, # of columns)Plan background vs. rich imagery# of CTAs
CTA copy
Access to site wide navigation vs. locked down navigation
Video vs. none
Form fields
Social proof (testimonials/endorsements/as-seen-in)
1. Landing page ideas to test:
II. Home page ideas to test:
Messaging
II. Home page ideas to test:
Messaging
Imagery
II. Home page ideas to test:
Messaging
Layout (long scroll vs. short, # of columns)
Imagery
II. Home page ideas to test:
Messaging
Layout (long scroll vs. short, # of columns)Primary navigation (search vs. browse)
Imagery
II. Home page ideas to test:
Messaging
Layout (long scroll vs. short, # of columns)Primary navigation (search vs. browse)# of CTAs
Imagery
II. Home page ideas to test:
Messaging
Layout (long scroll vs. short, # of columns)Primary navigation (search vs. browse)# of CTAs
CTA copy
Imagery
II. Home page ideas to test:
Messaging
Layout (long scroll vs. short, # of columns)Primary navigation (search vs. browse)# of CTAs
CTA copy
Social proof (testimonials/endorsements/as-seen-in)
Imagery
II. Home page ideas to test:
Messaging
Layout (long scroll vs. short, # of columns)Primary navigation (search vs. browse)# of CTAs
CTA copy
Testimonials/endorsements/as-seen-inVideo vs. none
Imagery
II. Home page ideas to test:
III. Checkout flow ideas to test:
Form fields and required steps
III. Checkout flow ideas to test:
Form fields and required steps
Fewer long pages vs. more short pages
III. Checkout flow ideas to test:
Form fields and required steps
Access to site wide navigation vs. locked down navigation
Fewer long pages vs. more short pages
III. Checkout flow ideas to test:
Form fields and required steps
Access to site wide navigation vs. locked down navigationBreadcrumbs
Fewer long pages vs. more short pages
III. Checkout flow ideas to test:
Form fields and required steps
Access to site wide navigation vs. locked down navigationBreadcrumbs
Cross-selling of other products
Fewer long pages vs. more short pages
III. Checkout flow ideas to test:
Form fields and required steps
Access to site wide navigation vs. locked down navigationBreadcrumbs
Cross-selling of other products
Phone number/live chat for assisted bookings vs. self serve
Fewer long pages vs. more short pages
III. Checkout flow ideas to test:
Form fields and required steps
Access to site wide navigation vs. locked down navigationBreadcrumbs
Cross-selling of other products
Phone number/live chat for assisted bookings vs. self servePost-conversion marketing
Fewer long pages vs. more short pages
III. Checkout flow ideas to test:
IV. Product/pricing ideas to test:
Free trial
IV. Product/pricing ideas to test:
Free trial
Money back guarantee
IV. Product/pricing ideas to test:
Free trial
Money back guarantee
Annual subscription vs. monthly
IV. Product/pricing ideas to test:
V. Email ideas to test:
Subject line
V. Email ideas to test:
Subject line
Content (short vs. long, imagery vs. plain text, etc.)
V. Email ideas to test:
Subject line
# of CTAs
Content (short vs. long, imagery vs. plain text, etc.)
V. Email ideas to test:
Subject line
# of CTAs
From name
Content (short vs. long, imagery vs. plain text, etc.)
V. Email ideas to test:
Subject line
# of CTAs
From name
Send time
Content (short vs. long, imagery vs. plain text, etc.)
V. Email ideas to test:
Subject line
# of CTAs
From name
Send time
Cart abandonment
Content (short vs. long, imagery vs. plain text, etc.)
V. Email ideas to test:
Subject line
# of CTAs
From name
Send time
Cart abandonment
Content (short vs. long, imagery vs. plain text, etc.)
Post purchase offer
V. Email ideas to test:
How to Set Up Your Tests
Part 5
Pick a platform that works for you
https://conversionxl.com/conversion-optimization-tools/
http://mashable.com/2015/01/30/ab-testing-tools/
http://conversionsciences.com/blog/ab-testing-tools/
http://www.conversion-rate-experts.com/split-testing-software/
Generate Hypotheses
Prioritize
Create Variations
Run Experiment
Analyze Results
Watch out for the gotchas!
Watch out for the gotchas!Don’t test too many variations concurrently!
Watch out for the gotchas!Don’t test too many variations concurrently!Don’t end your test too early!
Watch out for the gotchas!Don’t test too many variations concurrently!
Don’t ignore external factors!
Don’t end your test too early!
Watch out for the gotchas!Don’t test too many variations concurrently!
Don’t ignore external factors!
Don’t give up after the first test!
Don’t end your test too early!
Additional Reading
Appendix
http://www.coelevate.com/essays/5-steps-to-choose-your-customer-acquisition-channelhttp://www.coelevate.com/essays/traction-vs-growthhttp://www.coelevate.com/essays/customer-acquisition
https://conversionxl.com/12-ab-split-testing-mistakes-i-see-businesses-make-all-the-time/https://conversionxl.com/ab-testing-guide/
http://www.conversion-rate-experts.com/understanding-your-visitors/http://www.conversion-rate-experts.com/website-usability/
https://moz.com/blog/most-entertaining-guide-to-landing-page-optimization
ANY QUESTION
S?
GOOD LUCK!
Ron Schneidermannhttps://angel.co/ron-schneidermann
https://www.linkedin.com/in/schneidermann