OCTOBER 2012

TEST AND LEARN

How experimenting with content improves the user experience and drives business value

MARKETING SCIENCES ANALYTICS � MONETIZATION � OPTIMIZATION

OUR METHODOLOGY IS DESIGNED TO ANSWER QUESTIONS RATHER THAN BUILD REPORTS Where should I invest my marketing dollars? How does my digital media relate to offline purchase? What is the value of my customers? How can I increase conversion? Who should I target? Who are my most valuable customers? What is the impact of my social media strategy?

METHODOLOGY

: Goals and KPI definition

: Technical and organizational support for measurement

: Holistic view of what is working and what is not

: Create and use an ROI-driven system to get smarter

ANALYTICS

ASSIGNING MONETARY VALUES TO DIFFERENT SITE BEHAVIORS TO UNDERSTAND AND QUANTIFY THE OVERALL VALUE OF THE WEB CHANNEL.

BENEFITS

1. Measure total website value and track value over time

2. Understand how website actions relate to future profit

3. Evaluate return on marketing spend

4. Create a common language within the organization by using a common value for stakeholders to report on

5. Project prioritization

MONETIZATION

COMMON TRAITS OF OUR MOST SUCCESSFUL CLIENTS

: Digital Channel Goals are imperative

: Analyze performance based on overall business goals

: Understand data holistically, not in silos and not just digital

: Focus on opportunities and recommendations

: Monetize all key digital behaviors

: Prioritize opportunities

: Measure & quantify relationships between brands and people

: Take action with an optimization program

: Avoid average experiences

: �$+-�($34'2;

Website Optimization delivers relevancy to customers, ROI to clients Improve site performance (business objectives)

: Engage more users : Increase sales and profits : Improve lead generation and opt-ins

Validate assumptions Discover what content resonates with target audience(s)

Bounce Rate

KPIs

OPTIMIZATION

op�ti�mize [op-tuh-mahyz] Verb

1. To make as effective, or useful as possible.

Right Message

Right Audience

Right Time

Optimisation

Copy Imagery Call-to-Action �4%;

First time visitors Return visitors Paid Search traffic �4%;

Evening Weekend End of Month �4%;

Performance improvement

� �����������#��������;

A

B

Visitors randomly split into groups

Each group is shown a unique version

Results are analyzed & winner determined

Days

conversion s

� �����������#��������;

TESTING DIFFERENT DESIGN AND CONTENT VERSIONS TO DETERMINE WHICH IS MOST EFFECTIVE

EXISTING CONTENT

ALTERNATE CONTENT

Single element testing

Multiple element testing

Serve content by visitor segment

People who did this are likely to do that

Serve content unique to visitor

A/B/n Testing

Multivariate Testing (MVT)

Behavioral Targeting

Predictive Modeling

Personalization

Multi Channel Optimise messaging across fixed web, mobile, tablet, call centre, etc

Audience size Com

plex

ity

TECHNIQUES

Guessing, Before and After Testing

A/B/n and Multivariate Testing

Segmentation, Personalization, etc.

Behavioral Targeting

Multi-Channel

EVOLUTIONARY PROCESS

THE PERFORMANCE OF THE GENERAL POPULATION LIKELY DOES NOT REPRESENT THE BEHAVIOUR OF KEY ONLINE BUSINESS SEGMENTS : Making single answer decisions based on one bucket of visitors may miss

opportunities for large incremental benefits : Optimising content for key segments transforms a good website to a

great online experience

ONE SIZE DOES NOT FIT ALL

: We use data to understand customer journeys, motivations, and what messages are working and what are not

: We translate this data into insights to create personalised and targeted conversations

across touch points : We can demonstrate how establishing a culture of measurement and testing has a

revolutionary impact on marketing strategies and performance

SUCCESS COMES FROM UNDERSTANDING AND MEETING THE NEEDS OF THE AUDIENCE

WE BELIEVE

Environment Variables

Site Behavior Variables

Referrer Variables

Temporal Variables

Offline Variables

Environment Variables

IP address + Country of origin +

Time zone + Device type +

Operating system + Browser type +

Screen resolution +

Site Behavior Variables

Customer / Prospect + New/return visitor +

Previous visit patterns + Previous Product interests +

Searches + Previous online purchases +

Previous Campaign exposure + Previous Campaign responses +

Temporal Variables

Time of day + Day of week +

Recency + Frequency +

Enhance visitor engagement and

traffic flow

Improve messaging

Increase Form Completion

Provide relevant / personalized messages

HOW DOES HOMEPAGE CONTENT IMPACT BOUNCE RATE AND SITE ENGAGEMENT?

WHAT PAGE CONTENT ENGAGES VISITORS BEST? ���?����������������������������"����

HOW MANY MORE LEADS COULD WE GENERATE WITH SIMPLER FORMS?

HOW CAN WE REDUCE BOUNCE AND INCREASE VALUE OF PAID TRAFFIC? HOW CAN WE BETTER ENGAGE VISITORS FROM SOCIAL NETWORKS?

WHEN AND WHERE SHOULD CROSS-SELLS BE INTRODUCED? TO WHOM AND WHEN SHOULD DISCOUNTS AND INCENTIVES BE OFFERED?

Drive purchasing

ANSWERING BUSINESS QUESTIONS THROUGH TESTING

WEB OPTIMIZATION METHODOLOGY

Deepen engagement

Boost conversions

Increase value of media spend

Teaches what resonates/performs best with target audience

TEST RESULTS

TEST DESIGN AND VARIANT CREATION

ESTABLISH SITE GOALS AND SUCCESS METRICS (KEY PERFORMANCE INDICATORS)

IDENTIFY AND PRIORITIZE TESTING OPPORTUNITIES

FOUNDATION

1. ASSESSMENT : Define Objectives : Primary goals of site : Goals of individual sections of site 2. TEST STRATEGY : Develop hypothesis : Define Success metrics : Determine test methodology : Create metrics plan : Calculate test duration 3. EXECUTION : Create alternates : Tool set-up : Launch tests 4. RESULTS ANALYSIS : Analyze performance by variant : Analyze performance by segment : Recommendations/Next steps : Plan follow up tests

Assessment Test Strategy

Analysis Execution

WEB OPTIMISATION PROCESS

RESULTS

1 2

3 4

TESTING AND TARGETING CASE STUDIES

OBJECTIVE Increase click thru rate

ACTION TAKEN : Removed GOOD, BETTER, BEST design

elements : Reduced copy to call out only 3 features : Changed background from blue to white RESULTS : 25% CTR lift : 61% increase in revenue

Control

Winner

A/B TEST EXAMPLE: DELL iNSPIRON NOTEBOOK LANDING PAGE

Reformatted fields to make form appear shorter/consider shading Reduced the prominence of the reset button Added relevant image

Control < Existing Page Alternate

Results: 129% form completion increase

A/B TEST EXAMPLE: SIMPLIFIED TEST DRIVE FORM

1

2

3

A/B TEST EXAMPLE: FORD OF RUSSIA HOMEPAGE ROTATION

RESULTS: : 8% reduction in bounce rate : 13% increase in buying activities

OBJECTIVE: Increase engagement and buying activities

Winner Control

OBJECTIVE: Improve merchandising of accessories to increase conversion rate and RPV

Product Page Cart Page

RESULTS: : Removal of Accessories on Brand product pages delivered 60% lift in RPV : Adding Accessories to Cart page increased purchase conversion rate by 5%

TEST EXAMPLE: NOKIA ACCESSORIES CROSS SELL

1 2

1

4

3

Winner

Control

MULTIVARIATE TEST EXAMPLE: HELIO OFFER PAGE

OBJECTIVE : Reduce landing page

: abandon rate ACTION TAKEN : Redesigned call to action to

better inform customers

: 480 total variants RESULTS : 57% higher CTR

: 130% sales conversion rate

increase

A (Control)

B (Big Sexy Image)

C (Lead with Offer)

D (Name/Prices on phones)

E (Fa-La-La-La-La!)

RIGHT PANEL OPTIONS

A (Control) B (More text + price) C (more text, no price)

D (no right image or button)

RIGHT PANEL OPTIONS

A (Control)

B (Choose Device)

C (Green Choose Device)

D (Bold Choose Device)

E (Green Buy Now)

F (Bold Buy Now)

A (Control)

B (Red Button)

C (Learn More)

D (No Right Button)

��8���8��8��;����04$-��$2+$/43

LEFT PANEL RIGHT PANEL

BUTTON OPTIONS

OBJECTIVE: Increase purchase conversion rate and RPV ACTION TAKEN Targeted visitors who previously self-identified themselves as laptop shoppers with laptop content AREAS TESTED: : Content & layout in center of page : Products promoted : A/B/n test (9 variants)

RESULTS: : 24% lift in revenue per visit : 19% lift in purchase conversion rate : Significant annual monetized lift

Winner

Control

TARGETING EXAMPLE: DELL SMB RETARGETING

Popular Games Discount Games

35

OFFER TEST: Discounted games vs Popular games RESULTS: The popular games version significantly outperformed the discounted titles:

: 18% higher open rate : 91% higher click through rate : 107% more orders : 300% higher revenue

EMAIL OPTIMISATION: YAHOO GAMES

CONTENT CHANGES ARE JUST THE BEGINNING

THE QUESTION IS NOT CAN WE SERVE DIFFERENT CONTENT

Copy? Imagery?

Layout? CTA?

Offer?

New Visitors? Paid Search Traffic?

Repeat Purchasers?

PROGRAM NOT PROJECT

WEB OPTIMIZATION TEST VALUATION/PRIORITIZATION

In order to prioritize we need an estimated impact of the test. �(�7'�2'$--9�,/'7�4*'�+.1$%4�0(�4*'�4'34��7'�705-&/?4�/''&�40�4'34�+4�

WHAT WILL I GET IF I RUN THIS TEST?

What is the existing conversion rate? How epic is the scale of the test variables? How consistent is the lift relative to other tests? Would you spend the time on the test at 5%? Would you think 20% unrealistic?

5%? 10%? 20%?

HOW TO CHOOSE?

TWO GOALS: Show opportunity Show range

Upside SummaryProject Check out page redesignGoal Increase percent conversion rate on final page of purchase funnelMonthly Visits 45,000Current Conversion 5.00%Project Cost $45,000

Conversion rate

Change in Coversion

Total Units Sold

Incremental Units Sold

Monthly Incremental Profit

12-Month Incremental Profit 12-Month ROI

5.00% 2,250 - - -5.05% 0.05% 2,273 23 $2,813 $33,750 (0.25) 5.10% 0.10% 2,295 45 $5,625 $67,500 0.50 5.15% 0.15% 2,318 68 $8,438 $101,250 1.25 5.20% 0.20% 2,340 90 $11,250 $135,000 2.00 5.25% 0.25% 2,363 113 $14,063 $168,750 2.75 5.30% 0.30% 2,385 135 $16,875 $202,500 3.50 5.35% 0.35% 2,408 158 $19,688 $236,250 4.25 5.40% 0.40% 2,430 180 $22,500 $270,000 5.00 5.45% 0.45% 2,453 203 $25,313 $303,750 5.75 5.50% 0.50% 2,475 225 $28,125 $337,500 6.50 5.55% 0.55% 2,498 248 $30,937 $371,250 7.25 5.60% 0.60% 2,520 270 $33,750 $405,000 8.00 5.65% 0.65% 2,543 293 $36,562 $438,750 8.75 5.70% 0.70% 2,565 315 $39,375 $472,500 9.50 5.75% 0.75% 2,588 338 $42,188 $506,250 10.25 5.80% 0.80% 2,610 360 $45,000 $540,000 11.00 5.85% 0.85% 2,633 383 $47,813 $573,750 11.75 5.90% 0.90% 2,655 405 $50,625 $607,500 12.50

Expected Range of Values Expected Case

WHAT WILL I GET IF I RUN THIS TEST?

Follow the goals Follow the revenue

Test Plan/Roadmap

PLANNING & PRIORITIZATION

= *9�3*05-&�7'�25/�4*+3�4'34�(+234�>

In addition to ROI/lift Consider implementation risks: :Ease of technical solution : Time/production costs :Measurability of results

CONSIDER

PLANNING & PRIORITIZATION

EXECUTE

AVOID CONSIDER

RE

TUR

N O

N IN

VE

STM

EN

T (R

OI)

EASE OF IMPLEMENTATION

WEB OPTIMIZATION TEST OR REDESIGN

Test Plan Analyze

Test Plan Analyze

Test Plan Analyze

Test Plan Analyze

Test Plan Analyze

Test Plan Analyze

Test Plan Analyze

Test Plan Analyze

����������#��?����� ���� � ���������������+.1-+(+'&�

Fix Replace

OPTIMIZATION OR REDESIGN

THE THEORY OF THE LOCAL MAXIMUM:

�*'�-0%$-�.$8+.5.�+3�$�10+/4�+/�7*+%*�905?6'�*+4�4*'�-+.+4�0(�4*'�%522'/4�&'3+)/ It is as effective as its ever going to be on its current structural foundation. Even if you make 100 tweaks you can only get so much improvement.

OPTIMIZATION OR REDESIGN

������ "���������� �������?�� ASSUMING A REDESIGN FIRST IS EXPENSIVE LOOK FOR STAGNANT OR DECLINING MEASUREMENT OVER TIME. USER RESEARCH RADICAL MODULAR OPTIMIZATION FAILS TO MOVE THE NEEDLE. CIRCUMSTANTIAL EVIDENCE FROM COMPETITORS.

�� ������� ���� �?�����������������!�

WEB OPTIMIZATION MOVING BEYOND A ONE-SIZE FITS ALL EXPERIENCE PROVIDES OPPORUNITIES FOR CREATIVE

ARE YOUR BEST IDEAS BEING DISCARDED?

�������!����������=����">������

EXISTING CONTENT

ALTERNATE CONTENT

90% 10%

IS IT REALISTIC TO ANTICIPATE THE PREFERENCES OF HUNDREDS OF THOUSANDS OF INDIVIDUALS?

START BY ANALYZING RESULTS BY SEGMENT

: GOALS FIRST, TESTS SECOND

: FOSTER AN OBSESSION WITH OUTCOMES

: PREPARE TO BE SURPRISED WHEN YOUR CUSTOMERS CHOOSE THEIR FAVOURITES

: EXPERIMENTATION HAS A REVOLUTIONARY IMPACT ON MARKETING STRATEGIES

: SEGMENTED TESTS ARE SMARTER TESTS

: MAKE IT A PROGRAM, NOT A PROJECT

: DEMONSTRATE HOW INSIGHTS ARE BEING INCORPORATED THROUGHOUT ������?�������������������

BUILDING A CULTURE OF TESTING AND MEASUREMENT

WEB OPTIMIZATION OPEN QUESTION/DISCUSSIONS?

PETER ERNST Associate Director of Optimization Possible peter.ernst@possible.com

THANK YOU