the best a/b test idea you haven’t thought of
TRANSCRIPT
The best A/B test idea you haven’t thought of
PHIL SHARP
@NEILPATEL@Kissmetrics
#KissWebinar
@ThueLMadsen
Thue is the Kissmetrics Webinar Wizard and Marketing Ops Manager. Before joining forces with
Kissmetrics, he was a Lyft driver in SF, which is also how he ended up as a KISSmetrics marketer.
Whenever Thue is not trying to automate everything around him, you can find him hiking in the Sierras.
THUE MADSEN Marketing Operations Manager, Kissmetrics
@ThueLMadsen
Phil leads the UserTesting marketing team in helping companies discover the importance of user
experience and usability testing.
PHIL SHARP VP of Marketing, UserTesting
@iamphilsharp
@NEILPATEL@Kissmetrics
#KissWebinar
@IAmPhilSharp
WATCH WEBINAR RECORDING NOW
The problem
Not so much
Winner!
Not so much
Not so much
Not so much
Not so much
Not so much
Not so much
Not so much
Not so much
The real problem
One month
Winner!
Two months
Three months
Four months
Five months
Six months
Seven months
Eight months
Nine months
1 winner per year
QUOTE FROM ELON MUSK BIOGRAPHY
“ He would place this urgency that he expected the revenue in ten years to be ten million dollars a day and that every day we were slower to achieve our goals was a day of missing out onthat money.
$10,000,000 days
We’re missing out on this
But…why?
We’re smart
And hard working
And nice :)
And know our products
Why?
TOO MANY IDEAS!
170 ideas in 3 minutes
Try new CTA Try new CTA location
Test multiple CTAs Test different button colors
Add an “About” section Try a different headline
Add an image to the homepage Shorten my form
Auto play the video Try modals for forms
Test a free trial Try annual billing
Try a high anchor price
Try new CTA Try new CTA location
Test multiple CTAs Test different button colors
Add an “About” section Try a different headline
Add an image to the homepage Shorten my form
Auto play the video Try modals for forms
Test a free trial Try annual billing
Try a high anchor price
Try new CTA Try new CTA location
Test multiple CTAs Test different button colors
Add an “About” section Try a different headline
Add an image to the homepage Shorten my form
Auto play the video Try modals for forms
Test a free trial Try annual billing
Try a high anchor price
Er.
We need a process
Process = Sleepy time
Process = More revenue
Process = More experiments
The core of our process…
Find out what’s stopping folks from converting…
…fix it.
Thanks
I’ll be here all night
Three Steps
Step 1: Take these ideas…
Try new CTA Try new CTA location
Test multiple CTAs Test different button colors
Add an “About” section Try a different headline
Add an image to the homepage Shorten my form
Auto play the video Try modals for forms
Test a free trial Try annual billing
Try a high anchor price
…and throw them away.
Step 2: Listen to your customers and prospects
Find out what’s stopping folks from converting…
…fix it.
Step 2: Listen to your customers and prospects
Try new CTA Try new CTA location
Test multiple CTAs Test different button colors
Add an “About” section Try a different headline
Add an image to the homepage Shorten my form
Auto play the video Try modals for forms
Test a free trial Try annual billing
Try a high anchor price
These ideas are good. They just might not be for you.
Gut
Your gut is good. It just knows too much.
How to listen
QUALAROO
QUALAROO - USERTESTING EXAMPLE
QUALAROO
Is anything preventing you from signing up?
QUALAROO
“Trying to work out a password that meets your criteria — it’s taking me a while.”
QUALAROO
“A 10 character minimum for a password! Seriously? That’s insane.”
QUALAROO
“10 characters for the password? That’s way too many. This isn’t a bank site or something…”
CHAT TRANSCRIPTS
SALES PEOPLE AND SUPPORT
USER TESTING
Send 3 users through your funnel
USER TESTING
USER TESTING
Send 3 users to your competition
Send 3 users to a HUGE company in your space
PEEK
Step 3: Use a framework to make decisions
Test summary Test objective Hypothesis
Opportunity size Time to test
Likely scenarios Next steps
What worked? What didn’t work?
HTTP://BIT.LY/ABTEST-FRAMEWORK
Test ObjectiveIn 1-2 sentences, why are we running this test?
HypothesisIf X, then Y because Z
Opportunity SizeIf this test works, what’s the total impact we could
expect to see over a year?
Time to TestHow long will it take to run this test?
Likely ScenariosWrite out your assumptions and 3 of the most
likely scenarios.
Next StepsNow that you know the results, what will you do? If the campaign worked, how can you expand it?
What can you test that is similar?
What worked? What didn’t?
What did you learn?
The problem
Not so much
Winner!
Not so much
Not so much
Not so much
Not so much
Not so much
Not so much
Not so much
Not so much
The problem opportunity
Not so much
Winner!
Not so much
Not so much
Not so much
Not so much
Not so much
Not so much
Not so much
Not so much
Not so much
Winner!
Not so much
Not so much
Not so much Winner!
Not so much
Not so much
Not so much
Not so much
Not so much
Not so much
Not so much Winner! Not so
muchNot so much
Winner! Not so much
Not so much Winner!
LOG IN WITH GOOGLE
Start Your Free KISSmetrics Trial
PHIL SHARP VP of Marketing, UserTesting
@iamphilsharp
THUE MADSEN Marketing Operations Manager, Kissmetrics
@ThueLMadsen
Questions?