the best a/b test idea you haven’t thought of

The best A/B test idea you haven’t thought of

PHIL SHARP

@NEILPATEL@Kissmetrics

#KissWebinar

@ThueLMadsen

Thue is the Kissmetrics Webinar Wizard and Marketing Ops Manager. Before joining forces with

Kissmetrics, he was a Lyft driver in SF, which is also how he ended up as a KISSmetrics marketer.

Whenever Thue is not trying to automate everything around him, you can find him hiking in the Sierras.

THUE MADSEN Marketing Operations Manager, Kissmetrics

@ThueLMadsen

Phil leads the UserTesting marketing team in helping companies discover the importance of user

experience and usability testing.

PHIL SHARP VP of Marketing, UserTesting

@iamphilsharp

@NEILPATEL@Kissmetrics

#KissWebinar

@IAmPhilSharp

WATCH WEBINAR RECORDING NOW

The problem

Not so much

Winner!

Not so much

Not so much

Not so much

Not so much

Not so much

Not so much

Not so much

Not so much

The real problem

One month

Winner!

Two months

Three months

Four months

Five months

Six months

Seven months

Eight months

Nine months

1 winner per year

QUOTE FROM ELON MUSK BIOGRAPHY

“ He would place this urgency that he expected the revenue in ten years to be ten million dollars a day and that every day we were slower to achieve our goals was a day of missing out onthat money.

$10,000,000 days

We’re missing out on this

But…why?

We’re smart

And hard working

And nice :)

And know our products

Why?

TOO MANY IDEAS!

170 ideas in 3 minutes

Try new CTA Try new CTA location

Test multiple CTAs Test different button colors

Add an “About” section Try a different headline

Add an image to the homepage Shorten my form

Auto play the video Try modals for forms

Test a free trial Try annual billing

Try a high anchor price

Try new CTA Try new CTA location

Test multiple CTAs Test different button colors

Add an “About” section Try a different headline

Add an image to the homepage Shorten my form

Auto play the video Try modals for forms

Test a free trial Try annual billing

Try a high anchor price

Try new CTA Try new CTA location

Test multiple CTAs Test different button colors

Add an “About” section Try a different headline

Add an image to the homepage Shorten my form

Auto play the video Try modals for forms

Test a free trial Try annual billing

Try a high anchor price

Er.

We need a process

Process = Sleepy time

Process = More revenue

Process = More experiments

The core of our process…

Find out what’s stopping folks from converting…

…fix it.

Thanks

I’ll be here all night

Three Steps

Step 1: Take these ideas…

Try new CTA Try new CTA location

Test multiple CTAs Test different button colors

Add an “About” section Try a different headline

Add an image to the homepage Shorten my form

Auto play the video Try modals for forms

Test a free trial Try annual billing

Try a high anchor price

…and throw them away.

Step 2: Listen to your customers and prospects

Find out what’s stopping folks from converting…

…fix it.

Step 2: Listen to your customers and prospects

Try new CTA Try new CTA location

Test multiple CTAs Test different button colors

Add an “About” section Try a different headline

Add an image to the homepage Shorten my form

Auto play the video Try modals for forms

Test a free trial Try annual billing

Try a high anchor price

These ideas are good. They just might not be for you.

Gut

Your gut is good. It just knows too much.

Curse of knowledge

HTTPS://HBR.ORG/2006/12/THE-CURSE-OF-KNOWLEDGE

How to listen

QUALAROO

QUALAROO - USERTESTING EXAMPLE

QUALAROO

Is anything preventing you from signing up?

QUALAROO

“Trying to work out a password that meets your criteria — it’s taking me a while.”

QUALAROO

“A 10 character minimum for a password! Seriously? That’s insane.”

QUALAROO

“10 characters for the password? That’s way too many. This isn’t a bank site or something…”

CHAT TRANSCRIPTS

SALES PEOPLE AND SUPPORT

USER TESTING

Send 3 users through your funnel

USER TESTING

USER TESTING

Send 3 users to your competition

Send 3 users to a HUGE company in your space

PEEK

Step 3: Use a framework to make decisions

Test summary Test objective Hypothesis

Opportunity size Time to test

Likely scenarios Next steps

What worked? What didn’t work?

HTTP://BIT.LY/ABTEST-FRAMEWORK

Test ObjectiveIn 1-2 sentences, why are we running this test?

HypothesisIf X, then Y because Z

Opportunity SizeIf this test works, what’s the total impact we could

expect to see over a year?

Time to TestHow long will it take to run this test?

Likely ScenariosWrite out your assumptions and 3 of the most

likely scenarios.

Next StepsNow that you know the results, what will you do? If the campaign worked, how can you expand it?

What can you test that is similar?

What worked? What didn’t?

What did you learn?

The problem

Not so much

Winner!

Not so much

Not so much

Not so much

Not so much

Not so much

Not so much

Not so much

Not so much

The problem opportunity

Not so much

Winner!

Not so much

Not so much

Not so much

Not so much

Not so much

Not so much

Not so much

Not so much

Not so much

Winner!

Not so much

Not so much

Not so much Winner!

Not so much

Not so much

Not so much

Not so much

Not so much

Not so much

Not so much Winner! Not so

muchNot so much

Winner! Not so much

Not so much Winner!

LOG IN WITH GOOGLE

Start Your Free KISSmetrics Trial

PHIL SHARP VP of Marketing, UserTesting

@iamphilsharp

phil@usertesting.com

THUE MADSEN Marketing Operations Manager, Kissmetrics

@ThueLMadsen

tmadsen@kissmetrics.com

Questions?