IS THIS NORMAL?Debunking myths & getting real about the challenges optimization

programs face.

Ryan Garner: Co-Founder, Clearhead

HI. I’M RYAN GARNER.This is the part where I tell you about myself.

(@ryantgarner)

YOU’VE PROBABLY SEEN LOTS OF SLIDES THAT LOOK LIKE THIS. RIGHT?

+125% lift!

control variation

OR THIS.

control variation

+5,500% lift!

AND YOU’VE PROBABLE PLAYED“PICK THE WINNER”

Jocelyn Michael

good for you!

Jocelyn Michael

NOW LET’S PICK WHICH IS THE MOST AWESOME.

A B

Everything is awesome!

THIS STUFF IS EASY, RIGHT?

LET’S GET REAL.

AND WE ARE GOING TO DO THIS BY ASKING 1 SIMPLE QUESTION ALONG THE WAY...

IS THIS NORMAL?

WE RAN OUR FIRST SET OF TESTS. MADE BUTTONS BIGGER, CHANGED COPY, TESTED DIFFERENT COLORS. AND ZIP, ZERO…

IS THIS NORMAL?

DEFINITELY.

• It’s okay to crawl before you walk, but set expectations.-You may (probably will) NOT materially improve your bottom line.

• Risk (Disruption) & Reward (Lift) are correlated. -Small tests often provide small returns…especially for smaller businesses!

WE DON’T KNOW WHERE TO START TESTING OR WHICH IDEAS ARE GOOD AND WHICH ARE BAD OR HOW TO KEEP MOMENTUM.

IS THIS NORMAL?

VERY.

YOU’VE NO DOUBT HEARD…

• “Always start at the bottom of the funnel.”

• “Test the whole funnel simultaneously.”

• “NO, Find the bottlenecks IN the funnel.”

• “Navigation is always a good place to test.”

• “Forms, baby, forms.”

WE SAY, TO SIMPLY START WITH THE THE BEST IDEAS.

Look Inward Look Outward

Your product roadmap User tests!

Your customer service team! Competitors

Ideas from around the company Vendors trying to sell you stuff you

Your previous test results Companies you admire

OMFG! WE HAVE 1,000+ TESTING IDEAS. LOTS AND LOTS OF THEM SEEM GOOD. NOW WHERE DO WE START?

IS THIS NORMAL?

NORMAL? NO.AWESOME? HELL, YES!

• It’s time to prioritize your “testing roadmap”.

• Create a simple scoring system and get the cream to the top.- It should need no real explanation.- No new acronyms!- Why? No hurt feelings & better ideas!!

• Then use your own judgment.

• Repeat regularly.

WE LAUNCHED A TEST THAT BROKE OUR SITE! I THOUGHT BUILDING TESTS WAS EASY.

IS THIS NORMAL?

NO COMMENT.

• Let’s be real here. When you launch a test, you are pushing new code onto your production website.- Should just anyone be doing this?

• Do you want a high velocity, high returning optimization program? - Of course you do! Otherwise it’s just your normal dev process.

• So mitigate the risks by..- Using your best developers and designers. Not your marketers- Monitor, Monitor, Monitor.

WE CONTINUE TO BUILD AND RELEASE NEW FEATURES AND DESIGNS THAT WILL NEVER BE TESTED.

IS THIS NORMAL?

INDEED IT IS.

THEY HAVE MORE PEOPLE THAN YOU DO.

The Optimization Team Marketing, Engineering, Design, Product, Content,

Everyone requesting features…

• Be patient. Be persistent.

• Look for opportunities to test the roadmap…even when you are not asked.

• Stress the possibilities of minimum viable testing.

• Show value and then ask for (dedicated) resources.

WE TEST EVERYTHING! EVERY LITTLE TWEAK AND CHANGE, BIG & SMALL ARE INDIVIDUALLY BROKEN OUT AND TESTED.

IS THIS NORMAL?

NOT REALLY.

QUESTION: WHY ARE WE TESTING?

• To collect pristine data in the pursuit of science and knowledge no matter how long it takes.

(or)

• To grow the business…and the sooner the better.

• Knowing it all is a luxury few have.

• Even when you have all the resources you need, do you have the traffic and the time to test everything?

• And is that strategy really the quickest way to the end goal of growing the business?

• Better is better even if you don’t always know exactly why.

• Like all resources testing cycles are limited, so be pragmatic.

LEADERSHIP WANTS A GROWTH ESTIMATE FROM ALL OUR WINNING TESTS, BUT WHEN WE ADD IT UP, IT SEEMS TOO GOOD TO BE TRUE.

IS THIS NORMAL?

ABSOLUTELY.

• There is no perfect way to do this.

• Stress the difference between statistical significance and observed lift.

• Trust that there is improvement and move on to new opportunity.- The only accurate way to confirm lift a year from now is to re-

test.- Do you really want to do that?

• And if you must show something…

***

SO, WHAT DID WE LEARN…

YOU ARE NORMAL IF…

• It’s not as easy as you thought it was going to be.

• Your initial efforts don’t double your business.

• You struggle to consistently find great testing ideas.

• You break stuff.

• You are not sure how to accurately measure your return.

AND REMEMBER - TESTING IS AWESOME!

THANK YOU.QUESTIONS?

Contact Info:ryan@clearhead.mewww.clearhead.me