the nuts and bolts: how one company implements an entire testing methodology every day

How one company implements an entire testing methodology every day

The Nuts and Bolts

Ryan HutchingsDirector of MarketingVacationRoost

Daniel BursteinDirector of Editorial ContentMECLABS

Session speaker

2

Ryan Hutchings

Director of Marketing

VacationRoost

Hutchings has more than 10 years of online marketing experience. He leads the marketing team for VacationRoost, a $120 million private company in the travel industry. From advertising to analytics, he organizes a comprehensive and robust online marketing strategy focused on tracking and optimization. His specialties include SEO, PPC, testing, conversion rate optimization, advanced data modeling, analytics and every technical aspect related to online marketing.

Hutchings is a Google AdWords certified professional, SEOmoz Pro member, nationally recognized industry speaker and participant of top online marketing communities. He also help found the SLCSEM organization, which is Utah’s largest

nonprofit group of online marketing professionals.

@HutchSEO

About VacationRoost

About VacationRoost

• 25 websites • Always running 2 to 3 tests • 100+ tests run across landing pages, homepages,

lead forms and shopping carts• Goal of 10% incremental lift in total revenue

generated by testing

Fundamental Question:

How can marketers systematically run tests to achieve consistent lifts?

and hundreds of other tests….

An overall testing process …

Build

… to support an iterative optimization cycle

Collecting data

Planning experiments

Preparing treatments

Conducting tests

Analyzing results

The dual testing track

A/B split test• Internal system with server-side

splitting, custom variables• Full manual reporting –

spreadsheets and database reports

Larg

e te

st

Search results

Cart

Products

Test

ing

Landing page optimization• Third-party software• Internal analysis determines

what to testSmal

l tes

t PPC and SEO landing pages

Headlines/CTAs

Images

The dual testing track

A/B split test• Internal system with server-side

splitting, custom variables• Full manual reporting –

spreadsheets and database reports

Larg

e te

st

Search results

Cart

Products

Test

ing

Landing page optimization• Third-party software• Internal analysis determines

what to testSmal

l tes

t PPC and SEO landing pages

Headlines/CTAs

Images

How to test small

Decide Identify Hypothesize Design Analyze

Decide what to test

90%Replace “our recommendations” section with content snippet for value proposition banner usage

Restyle “most popular destinations” change hierarchy 40%

How to test small

Decide Identify Hypothesize Design Analyze

Identify the target conversion goal

What’s a good goal for bounce rates?

> 50%

40%-50% 30%-40%

< 30%

What’s a good goal for CTR to the next page or step?

> 60%

50%-60%30%-50%

< 30%

How to test small

Decide Identify Hypothesize Design Analyze

Create a hypothesis

Small Test: PPC landing pages

17

Small Test: PPC landing pages

18

Small Test: PPC landing pages

19

How can we reduce friction and

anxiety on this page?

How to test small

Decide Identify Hypothesize Design Analyze

Wireframe, design and launch treatment

Small Test: PPC landing pages

21

Small Test: PPC landing pages

22

Small Test: PPC landing pages

23

Small Test: PPC landing pages

24

How to test small

Decide Identify Hypothesize Design Analyze

Analyze results and calculate ROI

Small Test: PPC landing pages

26

427% increase in CTR

Small test validation tool

95.2%

ROI template

Let’s take a closer look…

ROI templateAverage ticket

JanFebMar.AprMayJunJul.AugSeptOctNov.Dec

Close rate

JanFebMar.AprMayJunJul.AugSeptOctNov.Dec

Lead conversion

JanFebMar.AprMayJunJul.AugSeptOctNov.Dec

Contribution% of sales

ROI template

Incremental IncreasesNew Bounce RateLeads Sales $ Contribution $

53$ 40,307$ 3,225

Incremental IncreasesNew Bounce RateLeads Sales $ Contribution $

20%7

5,246 $420 $

At 10%

At 30%

158$ 119,256$ 9,541

25 %2 d

1,749 $140 $

How do you get the resources to run large tests?

The dual testing track

A/B split test• Internal system with server-side

splitting, custom variables• Full manual reporting –

spreadsheets and database reports

Larg

e te

st

Search results

Cart

Products

Test

ing

Landing page optimization• Third-party software• Internal analysis determines

what to testSmal

l tes

t PPC and SEO landing pages

Headlines/CTAs

Images

How to test big

Decide Collaborate Identify Create Design Analyze

Decide on an element to optimize

Looking at the same elements as the small

tests.

The basic funnel

Landing Pages

Search Results

Property

Cart

Billing

Purchase

3.5%

1.7%

“Rising tide”A/B

(large test)

Single page LPO(small test)

40%

1.7%

Purchase

Determining a “good” conversion rate

Source: MarketingSherpa.com/ecommerce

How to test big

Decide Collaborate Identify Create Design Analyze

Work with IT to define variables or events

Large Testing: Variables are the key

38

ConditionsSegment your users and/or their visits to single or multi-visit conditions.

inlineLeadForm1mrSearchPage1NotestsimpleModalLeadFormsimpleModalLeadForm4

controltest

How to test big

Decide Collaborate Identify Create Design Analyze

Identify target conversion goal for control

The basic funnel

Landing Pages

Search Results

Property

Cart

Billing

Book now, Web lead, interactions/events

Checkout, add “cross sell”

Book, fill out info

Bounce rates, CTR

Purchase

“Rising tide”A/B

(large test)

Single page LPO(small test)

The basic funnel

Landing Pages

Search Results

Property

Cart

Billing

Book Now, web lead, interactions/events

Checkout, add “cross sell”

Book, fill out info

Bounce rates, CTR

Purchase

Book Now, web lead, interactions/events

“Rising tide”A/B

(large test)

Single page LPO(small test)

10% - 30% minimum targeted gains

Conversion goal results

How to test big

Decide Collaborate Identify Create Design Analyze

Create hypothesis and set up validation sheet

Manual testing database

Number of Days

40

Control Treatment

Large Test: Lead forms 1

Security seals

Control

Fail – no lift.

Large Test: Lead forms 1

Treatment

Control Treatment

Large Test: Lead forms 2

Shorter form

Control

Fail – no lift.

Large Test: Lead forms 2

Treatment

Control Treatment

Large Test: Lead forms 3

Option to call

Control Treatment

Fail – no lift.

Large Test: Lead forms 3

How to test big

Decide Collaborate Identify Create Design Analyze

Wireframe, design and launch treatment

Large Test: Lead forms

Large Test: Lead forms

Large Test: Lead forms

19% Lift

How to test big

Decide Collaborate Identify Create Design Analyze

Analyze results and calculate ROI

Manual testing database

Testing roadmap – it’s really:

Search Results

Property Details

Cart

Billing info

Booking

ROI template

Same as a small test.

Setting up your own programmatic testing

1

2

3

4

5

Custom variable tracking system, spreadsheet templates are vital building blocks

Dedicate a person and tie performance to testing

“Top of the funnel” to the “bottom conversion” may have too much noise in

between – keep it simple and isolated

Build a reporting system that ties front end and back end together

Get buy-in from the management team

Questions?

Thank you

Ryan Hutchings

Marketing Director

VacationRoost

@HutchSEO

Daniel Burstein

Director of Editorial Content

MECLABS

@DanielBurstein

Black & White Headshot

Black & White Headshot