The Best Test: Optimizing for Success

#EMPOWERBL16

Presenter

Optimization Practice Lead, Stratigent

TIM WALKER

#EMPOWERBL16

This session will cover…

Why You Need to Test

To Test or Not to Test3

1

Testing Basics2

Bridging Testing to Personalization4

#EMPOWERBL16

Why You Need To Test

#EMPOWERBL16

Goal of testing

Finding the optimal customer experience…

Across medium (Web, app)

Across the spectrum of personas

#EMPOWERBL16

Testing can help…

Perfect website design and flow

Maximize conversions

Provide insight into user behavior

Feedback on new features

#EMPOWERBL16

Testing stats

Source: Forrester’s Q3 2015 Global Online Testing Platform Customer Online Survey

#EMPOWERBL16

Pre-testing considerations

People

How much of an effect will this test have on the

users?

Process

Do you have the right technology and skills in place to develop and

execute the test?

Technology

Do you have the right technology and skills in place to develop and

execute the test?

#EMPOWERBL16

Testing Basics

#EMPOWERBL16

Testing Process

1.Ideation & prioritizatio

n

3.Execution

2.Implementation

4.Analysis

5.Iterate

#EMPOWERBL16

Types of Tests

A/B test

Multivariate test

Multi-page tests

split up site traffic in a balanced way, and then

show one t of users version A and the second version B

setup to test many different individual changes on a

page all at once

These tests involve changing the experience

across multiple pages

CHANNELSlow high

COMP L E X I T Y

high

#EMPOWERBL16

Client Example |A/B Testing

Client was concerned that not having a clearly defined 'guest' sign in option for the checkout flow was driving away customers from converting

#EMPOWERBL16

The new design led to a higher conversion level

Client Example |A/B Testing

Tested different versions comparing the default with a new design that had guest sign-in front and center

#EMPOWERBL16

Client Example |Multivariate Testing

#EMPOWERBL16

Client Example |Multivariate Testing

#EMPOWERBL16

To Test or Not to Test

#EMPOWERBL16Define what metrics to determine test success

Translate a test idea into a formal hypothesis

Define what metrics to determine test success

Grade each hypothesis

#EMPOWERBL16

Three dimensional grading

Reach

Impact

Technical difficulty

How many users will see this test?

How much of an effect will this test have on the users?

Do you have the right technology and skills in place to develop and execute the test?

#EMPOWERBL16

Metric selectionMetric selection is key to gauging test results, ensure metrics that align with test goals & can detect anomalies are selected.

Conversions | the primary metric of a test - measures what you are trying to accomplish (e.g. orders, leads, sign-ups)

Conversion

Secondary Interactions

Tactical Engagement

Secondary Interactions | measure intermediate steps towards conversion or post conversion (e.g. cart additions, lead quality)

Tactical Engagement | these metrics monitor conversion related values (e.g. units per order, average order value, revenue, profit)

#EMPOWERBL16

When not to test.

Your site design isn’t stable

Don’t know what they are trying to achieve

Don’t have the resources aligned in order to run a testing program

Aren’t mature enough to execute a complex test

#EMPOWERBL16

Bridging Testing to Personalization

#EMPOWERBL16

Testing as a precursor to personalization

We don’t have to treat all users the same.

If the retailor could predict the price the customers were willing to pay, then lower friction and high conversion value. An optimization program could have produced $75 in revenue versus $50

User Example: Coupons for everyone

Same shirt, but two customers are willing to pay different prices…

A retailer generally releases coupons for 50% off retail price

Customer B: $25

Customer A: $50 (retail price)

Customer B pays $25

Customer A pays $25

#EMPOWERBL16

Personalization modeling

Rules based personalization

Build out individual personas

Personalization based on visit history

Personalization based on integration of online/offline data

Customized Personalization

Predictive Personalization

Using 1st hit attributes

Triggers can change experience on the flyCustomers fit into audiences

Customers fit into behavioral segments

Advanced targeting

Look-a-like Models

Segments fit onto customers

Machine learning

#EMPOWERBL16

Client Example |A/B Testing

Client needed to test the size of tiles when page viewed on mobile phones

Saw a 36% increase in click-thru rate with wider tile format

A test was run on both layouts

#EMPOWERBL16

Q & A

THANK YOULinkedIn.com/Ebiquity

@ebiquityglobal

www.ebiquity.com