multivariable testing in direct mail
DESCRIPTION
A presentation on multivariable testing in Direct Marketing and Direct Mail. Marketing Day 2010.TRANSCRIPT
![Page 1: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/1.jpg)
Multivariable Testing in Direct MailEfficient tests with limited resources
Gordon H. Bell
President
LucidView
![Page 2: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/2.jpg)
Analytics & Testing: 2 sources of insight
Testing– Front-end planning– Prove cause-and-effect– Ask new questions – Design the test to give the specific
data you want
Analytics– Back-end analysis– Look for correlations– Answer questions, but cannot ask any– General data
![Page 3: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/3.jpg)
Testing has always been important
• “Almost any question can be answered, cheaply, quickly and finally, by a test campaign.”
– Claude Hopkins, Scientific Advertising, 1923
• “Testing is still the best way to find true breakthroughs.”– Stone & Jacobs, Successful Direct Marketing Methods, 2008
![Page 4: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/4.jpg)
Testing has always been important
• “Almost any question can be answered, cheaply, quickly and finally, by a test campaign.”
Many questions
^multivariable
is now more efficient
![Page 5: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/5.jpg)
What is Multivariable Testing?
• Many different test designs and statistical techniques developed to test variables more efficiently
• Change many variables at once – in an organized way – so you can separate out the impact of each– Number of test “recipes” (test versions)– Each with a unique combination of all elements– Analyzing all recipes together to quantify effects
• Test design = collection of related test recipes
![Page 6: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/6.jpg)
4 Key Benefits
Benefit Scientific Multivariable Testing Split-run Testing
Efficiency 2 - 35 variables in one test 1 variable = 1 test
Speed and Sample size
Constant sample size (no matter how many variables)
Increasing sample size
Depth of insights
Accurate, robust, and comparative main effects and interactions
One main effect (conditional, with no interactions)
Flexibility Wide range of test designs One choice
![Page 7: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/7.jpg)
Getting Started
Important steps include how to:
1. Define the scope
2. Select a good design
3. Determine the right sample size
4. Create multivariable test recipes
5. Analyze and interpret results
![Page 8: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/8.jpg)
#1 Define the Scope
Where is multivariable testing valuable and manageable?
• High potential return-on-investment• Clear metrics• Fast results• Flexibility to create many versions of the
mailing• Insights can be applied long-term
![Page 9: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/9.jpg)
Financial Times: Price Test(from the DMA07 Conference)
• Focus on price testing• Measure response rate and revenue• Test in one direct mail campaign• All variables lasered in black ink
(easy and inexpensive to change)
Test Elements (-) Control (+) New Idea
A Price $49 for 26 weeks $99 for 12 monthsB Longer term (+25%) No Yes (32 weeks / 15 months)C Shipping & handling fee No Yes, $4.95D Streamline price box
(3 lines vs 5)No Yes, redesign price box and
reply slip
![Page 10: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/10.jpg)
FT “control” mail package
![Page 11: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/11.jpg)
#2 Select a Good Test Design
Many choices of multivariable test designs• Which “handful” of combinations should be tested to
clearly see results?• Want to have
– Few test recipes (versions of the mailing), but…– Lots of data for better insights– Clear, actionable results– A test design that the marketing team can
understand and believe
![Page 12: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/12.jpg)
Pric
e
Lon
ger
term
(+
25%
)
S&
H (
deliv
ery)
fee
Str
eam
line
pric
e bo
x
Recipe A B C D AB AC BC Mailed
1 – – – – + + + 25,000
2 + – – + – – + 25,000
3 – + – + – + – 25,000
4 + + – – + – – 25,000
5 – – + + + – – 25,000
6 + – + – – + – 25,000
7 – + + – – – + 25,000
8 + + + + + + + 25,000
Financial Times:test design and recipes
Pric
e
Lon
ger
term
(+
25%
)
S&
H (
del
iver
y) fe
e
Str
eam
line
pric
e bo
xRecipe A B C D Mailed
1 – – – – 25,000
2 + – – + 25,000
3 – + – + 25,000
4 + + – – 25,000
5 – – + + 25,000
6 + – + – 25,000
7 – + + – 25,000
8 + + + + 25,000
![Page 13: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/13.jpg)
Financial Times:test design and recipes
A: PriceB: Longer term
(+25%)C: S&H fee
D: Streamline price box
Recipe 1(control)
$49 No
($49 for 26 weeks)No No
Recipe 2 $99 No
($99 for 12 months)No
Yes, redesign price box
Recipe 3 $49 Yes
($49 for 32 weeks)No
Yes, redesign price box
Recipe 4 $99 Yes
($99 for 15 months)No No
Recipe 5 $49 No
($49 for 26 weeks)Yes, $4.95
Yes, redesign price box
Recipe 6 $99 No
($99 for 12 months)Yes, $4.95 No
Recipe 7 $49 Yes
($49 for 32 weeks)Yes, $4.95 No
Recipe 8 $99 Yes
($99 for 15 months)Yes, $4.95
Yes, redesign price box
Pric
e
Lon
ger
term
(+
25%
)
S&
H (
del
iver
y) fe
e
Str
eam
line
pric
e bo
xRecipe A B C D Mailed
1 – – – – 25,000
2 + – – + 25,000
3 – + – + 25,000
4 + + – – 25,000
5 – – + + 25,000
6 + – + – 25,000
7 – + + – 25,000
8 + + + + 25,000
![Page 14: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/14.jpg)
#3 Find the Right Sample Size
The best sample size depends upon– Response rate (or variation in sales)– How small a change you want to see
Lower response rate, or the need to see a smaller change, requires a larger sample size.
![Page 15: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/15.jpg)
Sample Size (N) example
• For Financial Times…– Say we usually have a 1.5% response rate (R)– And we want to see if any test elements affect
response by 10% or more (0.15%)– Then…
31.38 ● R ● (1-R)
(smallest change)2N =
31.38 ● 1.50% ● 98.5%
(0.15%)2N = = 206,062
![Page 16: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/16.jpg)
#4 Create the multivariable recipes
• The science determines which “handful” of combinations should be tested
• The art of testing is making sure…– Elements remain bold
(in the difference between “–” and “+” levels)
– Execution is clear and consistent
![Page 17: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/17.jpg)
Creatives: control (recipe 1) and “all plus” recipe 8
A+
B+
C+
D+
![Page 18: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/18.jpg)
#5 Analyze and Interpret Results
• A well-designed test is relatively easy to analyze– Analysis of variance, regression, or…
pencil and calculator
(with some more in-depth analysis)
• Three main steps in the analysis are:1. Analyze main effects (each column in the design)
2. Analyze interactions
3. Consider alternative solutions and additional metrics
![Page 19: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/19.jpg)
FT price test: main effects & interactions
FT Price Test: Gross response rateoptimal = 32.7% lift
+
-
-
-
-
+
+
0.000% 0.010% 0.020% 0.030% 0.040% 0.050% 0.060% 0.070%
AB interaction
AC interaction
A: Price
D: Streamline price box
BC interaction
C: Shipping &handling fee (no)
B: Longer term(yes, +25%)
Significant effects(beyond line)
18.9% increase in response versus the control
17.2% drop
13.8% further increase in response if...
BC Interaction
B-: Current(26 wk / 12 mo)
B+: Longer term (+25%)
Subscription Term
C-: No S&H fee C+: Add $4.95 S&H fee
Term and S&H do notaffect response rate…
… unless both are setat the best combination
![Page 20: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/20.jpg)
Financial Times: summary
Multivariable testing offered a way to• Test easy, high-impact changes• Learn more with a small sample size• Increase insights
– More accurate main effects– Clear interactions
• In contrast, with split-run testing– We would have needed 3x the sample for equal confidence– At this sample size, not one effect would be significant– Interactions would be impossible to see
![Page 21: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/21.jpg)
Hearst Launch of theFood Network Magazine(presented at the DMA09 conference)
New magazine for a new market =
Need for speed • Learn fast ramp-up quickly• Test a lot of ideas to see what works• Increase confidence in test results for a
rapid roll-out• Optimize price/offer along with creatives
![Page 22: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/22.jpg)
Food Network Magazine Creative Test(13 Creative Elements)
A Window on OE H Reply By dateB Change logo on OE and form J Add chef photos on formC Return address on OE K Change descriptionD Form layout L BrochureE Change "Charter Discount" M ChitF Audience Development copy N Lift noteG Savings presentation
Changes to the outer envelope
Additional inserts
![Page 23: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/23.jpg)
Elements on the Envelope
B: Logo C: Return Address A: Window
![Page 24: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/24.jpg)
B: Change logoE: Change “Charter Discount”F: Audience Development copyG: Savings presentation
H: Reply-By date
J: Add chef photos
K: Change description
D: form layout
![Page 25: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/25.jpg)
Win
dow
on
OE
Cha
nge
logo
on
OE
and
form
Ret
urn
addr
ess
on O
E
For
m la
yout
Cha
nge
"Cha
rter
Dis
coun
t"
Aud
ienc
e D
evel
opm
ent c
opy
Sav
ings
pre
sent
atio
n
Rep
ly B
y da
te
Add
che
f pho
tos
on fo
rm
Cha
nge
desc
riptio
n
Bro
chur
e
Chi
t
Lift
not
e
Recipe A B C D E F G H J K L M N MailedNet
response1 – – – – – – – – – – – – – 25,000 0%2 + + – – – – + + – – + + + 25,000 -21%3 – – – + – + + + + – – + – 25,000 -8%4 + + + – + – – – + – – + – 25,000 -6%5 + + + + + + + + – – – – – 25,000 -11%6 – + – + + – + – – + – + + 25,000 -25%7 – – + – + – + + + – + – + 25,000 -20%8 + – + – – + – + – + – + + 25,000 -10%9 + – + + – – + – + + – – + 25,000 -11%
10 + – – + + – – + – + + – – 25,000 -42%11 – + – – + + – + + + – – + 25,000 -20%12 + + – + – + – – + – + – + 25,000 -10%13 – + + + – – – + + + + + – 25,000 -31%14 – – + + + + – – – – + + + 25,000 -26%15 + – – – + + + – + + + + – 25,000 -37%16 – + + – – + + – – + + – – 25,000 -38%
400,000
16-recipe Creative Test
Design
![Page 26: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/26.jpg)
Test Elements (on the 1-page form)
Control (recipe #1) Recipe #11
B+
E+
F+
J+
K+
H+
![Page 27: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/27.jpg)
FNM: Net Response RateNet Response Rate
optimal = 9.75% lift
0.00% 0.25% 0.50% 0.75% 1.00% 1.25%
F-: Audience Development copy
B-: Change logo on OE and form
C+: Return address on OE
M-: Chit
H-: Reply By date
D-: Form layout
A+: No window on OE (closed-faced)
G-: Savings presentation (control)
N+: Lift note (yes)
J+: Add chef photos on form (yes)
E-: Change Charter Discount (no)
K-: Change description (control)
L-: Brochure (yes) Response drops 16.8% when brochure is removed
Test element (optimal setting)
"Research Discount" hurts 7.3%
New description reduces response 14.3%
3.8% lift
+2.5%
-3.1%
3.5% lift
Significant effectsbeyond line
![Page 28: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/28.jpg)
Multivariable "refining" tests(July 09 test results)
8.8% increase in NRR17.3% increase in Profit
Multivariable "screening" tests(Mar09 tests - Jul09 confirmation)
8.4% increase in NRR17.6% increase in Profit
Original control(March 2009)
Food Network Magazine Results
35% jump in profitover 2 campaigns
(versus 2 years with A/B splits)
(sample size of 600,000 vs. 4.7 MM for split-run testing)
![Page 29: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/29.jpg)
Whirlpool Contact Strategy Test(presented at DM Days – New York, 2007)
Test Elements
A TimingB Early mail dropC Expiration mail dropD Late mail dropE Outer envelopeF Lift noteG Phone Call in Week 1
Focus = Annual renewal of the extended service contract
![Page 30: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/30.jpg)
Contact Strategy Test
Week: 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 1 2 3 4 5 6 7
Control
A+
B+
B+
C+
C+
D+
D+
E+E+
E+E+
G+
X
X
Expiration Date
![Page 31: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/31.jpg)
Tim
ing
Ear
ly m
ail d
rop
Exp
irat
ion
mai
l dro
p
Lat
e m
ail d
rop
Out
er
enve
lope
Lift
not
e
Pho
ne
Recipe A B C D E F G Response
1 - - - - - - -2 + - - - + - +3 - + - - + + -4 + + - - - + +5 - - + - + + +6 + - + - - + -7 - + + - - - +8 + + + - + - -9 - - - + - + +10 + - - + + + -11 - + - + + - +12 + + - + - - -13 - - + + + - -14 + - + + - - +15 - + + + - + -16 + + + + + + +
WhirlpoolTest Design
Week 1customers
Week 2customers
Week 3customers
Week 4customer
![Page 32: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/32.jpg)
Test ResultsEffects: Total Response Rate
-4.67%
+3.00%
-2.68%
-2.43%
+2.22%
+1.92%
+1.61%
+1.41%
-0.83%
+0.79%
+0.63%
+0.61%
+0.27%
-0.11%
+0.02%
0.0% 0.5% 1.0% 1.5% 2.0% 2.5%
AEACAF
E: Outer envelopeC: Expiration mail drop
ADAG
F: Lift noteB: Early mail drop (yes)
ABDBD
AB (add early drop)G: Phone call (yes)
D: Late mail drop (yes)A: Timing (spread out)
(effects in percentage points on bottom; as % of control on end)
Significant effects(beyond line)
![Page 33: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/33.jpg)
Interaction Effect
AB Interaction (total response rate)
45%
46%
47%
48%
49%
A-: control (9,4,1,-4) A+: condensed (6,3,1,-3)
A: Timing of contacts
B-: No early mail drop B+: Yes (+14 or +9)
On average,the early mail drop(B) has no impact
But it does help ifcontacts are keptspread out
![Page 34: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/34.jpg)
Whirlpool Test: summary
• Response rate increased 7.81% (7.3% predicted)
• 8.3% increase in revenue (worth $ millions)• 6.76% increase in annual profit
• With split-run tests, would have…– Needed 2½ years of testing (vs. 4 months)– Completely missed the interaction– Seen nothing significant in 4 months
(due to 3x greater error)
![Page 35: Multivariable testing in Direct Mail](https://reader034.vdocument.in/reader034/viewer/2022051412/54c37cab4a79590e0b8b45ff/html5/thumbnails/35.jpg)
“Testing is still the best wayto find true breakthroughs.”
– Bob Stone and Ron Jacobs, Successful Direct Marketing Methods, 2008
“ Like trading in your bicycle for a helicopter, multivariable testing gives you the speed, power, and flexibility to get farther, faster.”
– Gordon Bell, Marketing Day, 23 March 2010