adma video testing and measurement

Australian Direct Marketing Association

Testing


What is testing?

Testing is a marketing experiment in a live environment:

•  Observing what customers actually do rather than what they say they are going to do

•  Monitoring how results vary as various components of a campaign are altered.

•  Using the results of testing to:

–  predict the outcome of future activity.

–  Improve our marketing performance



3 approaches

?

1.  Test on a small scale first then roll-out

3. Same approach to all then analyse results among different segments to spot variations

2. Do testing on a portion of a live campaign

Test

Test

Roll-out

Live campaign

Live campaign


Summary of testing options

Message components •  Product •  Proposition •  Offer •  Creative •  Call to action

Delivery components •  Targeting & segmentation •  Communication channels •  Format •  Timing


Test the important things…..


Testing case study

•  Two simple objectives –  Improve response rates –  Increase amount donated

•  Understanding donor segments –  Relationship to disease –  Value

With permission from Salmat


•  Relationship to disease –  Have the disease –  Parent of someone with the disease –  Relative / friend of someone with the disease –  No relationship to the disease


Test different propositions for each segment


•  Value –  Variable donations boxes based on last donation, increased in

increments of 20%


Personalisation case study



Testing case study


Example: Landing Page Test Options

Page components 1. Headline 2. Secondary Headline 3 Body Copy 4 Social Proof 5/6 Call to Action 7  Navigation 8  Image 9  Above/below the Fold



Guidelines for accurate testing

• Ensure you always have a “control” – So you have a benchmark for comparison

• Have a big enough sample to test with – To produce reliable results

• Ensure your test sample is not skewed at all (compared to control) – Use a randomly-selected sample

• Decide how you are going to measure performance: –The measurement chosen must prove/disprove your hypothesis



The importance of a control


May July June

Response rate

Standard offer

New offer A

New offer B

Here there is no control: - A separate offer has been run in each month - Offer A appears to have out- performed the current offer - Offer B appears to have performed worse = Offer A appears to win

Offer test


The importance of a control


May July June

Response rate

Standard offer

New offer A

New offer B

Introduction of control: - The current offer has been run in each month as a benchmark - Offer A did not perform as well as the current offer - Offer B performed better than the current offer = Offer B is the real winner

Offer test


The “significance” of a test result

Control New Offer A

Response rate

1.7%

1.4%

The test out-performed the control by 0.3% Should you adopt New Offer A – or could it be just a chance result? We need to know whether these results are likely to be repeated if we ran the test again


Having a Big Enough Sample

• In direct marketing, your sample is typically a subset of your customer base or contact list

•  In digital, your sample is typically the volume of unique visitors or unique impressions accumulated over a period of time

•  You need a quantity of sample that reflects your desire to detect even small differences between your test and control groups


50%$ 60%$ 70%$ 80%$ 90%$ 95%$ 99%$

Req

uire

d S

ampl

e Q

uant

ity

Ability to Detect Statistically Significant Differences


Not Just Statistical Significance

Do a sense-check when interpreting results: • What was the competition doing when this test was running? • Just because this worked in one location does it mean it will work in

another? • The offer was successful in Summer – would it still work in Winter? •  Were there any other abnormal factors in the marketplace which

might have affected the response?



Not Just Statistical Significance

Did we make money?

When we balance the costs of a campaign against the incremental volume of sales – was it worth it?

What should we test next? e.g. offer A performed better than offer B, what will beat offer A?


Measurement


Tracking the customer journey

Awareness Interest Action Desire

A buyer typically goes through several stages before they buy

…which means we should have measures to track performance for each stage

Reach Influence Closure


Example: eCommerce

Awareness

Interest

Action

Desire

Daily Unique Visitors

Product Page Views

Completes a Registra:on Form

Purchase


Tracking issues

How do you match response to source when:

–  Your campaign runs in several channels simultaneously

–  Your product is sold through a third party sales channel

–  Your product has a long consideration period


Common tracking options

Unique campaign and comms channel specific phone numbers

Outbound Channel Response Channel

Offline Broadcast

Mail

Display Media

Call-Centre

In-Store

Website

Unique redemption codes

Specific Search Call to Action


Indirect sales – a missing link

Clicks/visits

Sales

Calls

Known

Known

Unknown

Browse

Enquire

Buy


Long sales cycles

Sales

Time

What does this response curve look like for your product?

•  Track enquiries & sales over a long period

•  Establish a pattern

•  Create a rule of thumb


Rule of Thumb Approach

•  Can be used for indirect sales as well as an ‘early read’ for long campaign cycles

•  Typical approach:

1.  Establish a ratio for website visits or calls to reseller enquiries/sales

2.  Establish a pre-campaign baseline for calls and website visits

3.  Measure the uplift in calls/visits during and following the promotion

4.  Extrapolate to sales using typical ratio

adma video testing and measurement

Technology

offer b

marketing experiment

separate offer

test sample

control offer testhere

results of testing

quantity of sample

selected sample