The Art of Testing

August 2012

Agenda This is a Test. This is only a

Test. Testing in Direct Mail Show Me the Numbers Oooops! Pop… TEST…. Valid or

Invalid – you be the Judge!

What Is a test?

A procedure intended to establish the quality, performance, or reliability of something, esp. before it is taken into widespread use.

A/B testing, split testing or bucket testing compares a control sample to other samples in order to discover how to improve response or conversion rates

What Is NOT a Test?

NO, REALLY - A TEST Is NOT…

…..Comparing results of two different campaigns mailed during two different times of the year.

A TEST Is NOT…

… Comparing results of two mailings at two different organizations.

A TEST Is NOT…

… Mailing two completely different packages, with different offers, and different asks , with different inserts.

Testing in Direct Mail

Offer 40%

List40%

Creative20%

Direct Mail Success

Testing in Direct Mail

Component Test

• Postage• Ask• List• Offer• Who signs the

letter?• Color

Creative Test

• Package• 2 page vs. 4 page

letter.

Low Risk/ Lower Reward

Higher Risk/Higher Reward

Testing in Direct Mail: Knowing When & What to Test

When – Quantity is RIGHT Timing is RIGHT There is Rollout potential

What – Tests should be designed to meet both

station goals and budget. All tests should start with your

hypothesis!

Splitting the File

RANDOM- Tips on pulling a random file:

Team Approach has a “Random”” field! EQUAL:

The Test Bucket and the Control bucket need an equal # of records (which may mean a third bucket (“fall out”) if it is not a 50/50 split!

CODED: Each segment needs a unique code for

tracking purposes.

Don’t forget those pesky follow-ups!

Set yourself up for Statistical Success…

Final Objective: Statistic validity

What makes a test statistically valid? Significance Sample Size

What does it mean to be statistically significant? Results are repeatable, not by random chance

Significance

Significance measures the likelihood of

repeatable results. Determined by mathematical formulas and expressed as a

percentage (the confidence level) 95% Confidence Level is best practice Significance tests differ for Response Rates & Average Gifts A winning test does not necessarily mean you have a

winning package Free significance calculators are available online

Significance Calculators

Sample Size

Why is sample size important? Measuring significance Minimizing risk & controlling costs

How do we determine sample size? In general: There is no “right” quantity to test

The larger the sample size, the better Rule of 100 Statistical Calculations

Sample Size: Rule of 100

“100 responses will make this test statistically significant.”

Some say that this rule-of-thumb has never proved them wrong.

Some believe this statement underestimates the actual sample size that is needed.

Sample Size Calculator:Response Rate

Sample Size: Statistical Calculations

Response Rate:

N = 31.38 * p * (1-p)/(significant effect)²

Average Gift:

N = 31.38 * σ² /(significant effect)²

Have no fear! Many free calculators are available online

Rule of 100 vs. CalculationsHead to head test… Scenario 1 Control RR: 2% Expected Lift: 25%

Calculations show that… Responses needed: 308 Control/Test Quantity:

12,302

Scenario 2 Control RR: 2% Expected Lift: 47%

Calculations show that… Responses needed: 102 Control/Test Quantity:

3,481

Results: In scenario 1, the Rule of 100 underestimates the quantity needed.In scenario 2, the Rule of 100 holds true.

Quick Anatomy Takeaways

The goal is to set yourself up for statistically valid results from the start

Make sure that you have an adequate sample size To determine the right size you need to know the expected

response rate, effect, and confidence level.

Run the test… One variable at a time; multivariate testing increases sample

size.

Results are In! Check for significance. Use the calculators to see if your test was significant at the 95%

confidence level.

OOOPS!

Mistakes that

Muddle

Mistakes that Muddle

1. Knowing when something “just isn’t right.”

2. Uncovering what that something is.

YOU BE THE JUDGE!

Questions?

Thank You!!!