understanding multivariate testing techniques and how to ... · understanding multivariate testing...

12
Multivariate testing Understanding multivariate testing techniques and how to apply them to your email marketing strategies An Experian Marketing Services white paper

Upload: others

Post on 06-Apr-2020

15 views

Category:

Documents


0 download

TRANSCRIPT

Multivariate testing

Understanding multivariate testing techniques and how to apply them to your email marketing strategies

An Experian Marketing Services white paper

Multivariate testing

Page 1 | Understanding multivariate testing techniques and how to apply them to your email marketing strategies

Testing should be standard operating procedureWhether it is in search, display, direct mail, email, mobile or social, one would be hard-pressed to find a marketer today who isn’t aware of the concepts of A/B or multivariate testing (MVT) and the value they offer. With the amount of data that is being captured, stored, processed and analyzed in today’s world, combined with the speed at which results boomerang back to the tester, we operate in a time and place where there is no excuse not to continually optimize your marketing campaigns via some sort of testing procedure.

Even the president testsTesting has indeed become mainstream. Highlighting this fact is a recent article in Businessweek, which details the use of statistical testing by the Obama reelection campaign when trying to solicit donations from their email subscribers. The article describes how the campaign staff would initially experiment with a handful of different subject lines for their emails. After sending these seed emails out to a limited number of subscribers, the team would wait for the results (amount of money donated) to trickle in before it became clear which email subject lines performed best. Now equipped with the knowledge of what emails would perform significantly better than others, the team would send the winning emails out to the rest of the millions of email subscribers on the president’s list, arguably increasing the donations haul from email marketing by millions of dollars.

What’s being tested?Testing ranges in complexity. A/B and A/B/C testing are the most common forms of testing used in marketing today. These types of tests are fairly straightforward to perform and interpret, as they focus solely on measuring the effects of changing values from only one factor. As shown in the graph on the following page from Experian Marketing Services’ December 2012 Email Market Study, where email marketers across eight verticals were surveyed about their email marketing initiatives, most companies today are leveraging A/B and A/B/C testing to optimize different facets of their email campaigns (subject line, frequency, time of day, day of week, call-to-action, etc.), but very few (11 percent) are leveraging the more complicated, yet more powerful, tool of multivariate testing to optimize their email marketing efforts.

Multivariate testing Multivariate testing

An Experian Marketing Services white paper | Page 2

What type of testing do you perform on email campaigns?

0%

20%

40%

60%

80%

100%

MultivariateFriendlyfrom

Numberof

products

Productplacement

Call-to-action

HTMLversus

text

Dayof week

Timeof day

FrequencyCreativeSubjectline

97%

81%

39%

50%

36%

11%

43%

21%

28%

11% 11%

Source: Experian Marketing Services’ Email Market Study, Acquisition and engagement tactics, December 2012

A common A/B testing mistakeA common mistake when implementing A/B testing is performing sequential A/B tests in an effort to arrive at optimal levels for multiple factors. These experiments often start with the standard or status quo settings of the key factors to be tested. The levels of the factor that is believed to be the most responsible for performance are tested first, while the other factor levels remain constant. After the responses have been gathered and the optimal level for the first factor is determined, the factor regarded as the second most influential is tested next, with the “optimal” first level factor remaining fixed for the rest of the experiment. This process continues to repeat itself until each factor level has been individually tested.

To better illustrate this flawed process, consider the following example with two factors, each with just two levels (the simplest case possible). Suppose an organization wants to determine the best image and ad copy to put in an email with the click-to-open ratio being the metric to maximize. We will denote the different images as I1 and I2 and the different ad copy as C1 and C2. In their first email blast to 20,000 customers, the company decides to send half of these customers an email with the combination (I1, C1) and the other half with the combination of (I1, C2).

Multivariate testing

Page 3 | Understanding multivariate testing techniques and how to apply them to your email marketing strategies

The results are as follows:

Click to open percent:(I1, C1) = 7.5 percent

(I1, C2) = 8.5 percent

Based on these results, the company believes that Copy 2 is the preferred ad copy and fixes the next email blast to 10,000 more customers at this level so that the next test in the sequence will only vary the image level not already tested.

Click to open percent:(I2, C2) = 9.5 percent

Seeing these results, the company decides that (I2, C2) is the optimal combination in terms of being able to generate the highest click-to-open ratio.

The problem is that the company may have missed out on finding the global

optimum by not testing the fourth combination of (I2, C1), which may have

yielded a click-to-open ratio even greater than 9.5 percent.

A/B tests assess one level of one factor versus the control group, but cannot measure the interaction effect across factors.

The reason this method of sequentially testing one factor at a time fails to find the optimal factor levels is that an interaction effect exists between factors 1 and 2. That is, the factor effects are not additive, but rather the combinations among different factors and their levels produce an additional effect (interaction) when used simultaneously. By not being able to capture interaction effects, this sequential approach may miss the optimum altogether.

In situations such as these, it is more appropriate to perform a multivariate test (factorial test to be exact), where all factors are changed together and all combinations are accounted for. There is a wide array of different kinds of multivariate tests available, but when the number of factors and the number of factor levels to be tested are limited, the full factorial approach is the best option, as it retains the most amount of information about the factors.

Multivariate testing Multivariate testing

An Experian Marketing Services white paper | Page 4

Multivariate testing: a brief historyMultivariate testing has its roots in the statistical discipline of experimental design. Experimental design is the process of planning a study to meet specified objectives. Planning an experiment properly is important in order to ensure that the right data and sufficient sample sizes are available to answer the research question of interest, as clearly and efficiently as possible. One of the first recorded applications of experimental design comes from the early 20th century and a statistician named R.A. Fisher. Fisher was hired by an agricultural research center to determine the effects that various factors (soil type, sun exposure, rainfall, fertilizer, etc.) had on plant growth. By designing proper, randomized controlled experiments before any seeds had been planted into the ground, Fisher was able to harvest results that allowed him to determine the optimal levels of each factor that would maximize plant growth and crop output. His work proved seminal, and others soon found good use of experimental design in a wide variety of industrial and manufacturing problems. Only recently have brands discovered the power of experimental design (now termed multivariate testing) in optimizing marketing campaigns.

Advantages of MVT over A/B testingMultivariate testing has several advantages over that of A/B or A/B/C testing. In A/B or A/B/C testing, it is easy to determine which particular creative performed best at generating the most amount of conversions, but this type of testing can’t tell you why one creative performed better than the next. Was the increase in conversions due to the change in background color, the call to action, the main image, etc.? MVT allows you to measure the size of the effect that each of these factors has in generating conversions and provides guidance on where to focus your next round of testing efforts.

Multivariate testing

Page 5 | Understanding multivariate testing techniques and how to apply them to your email marketing strategies

The graph below is an example of what may be found through MVT testing:

Factor influence on click rates

0% 10% 20% 30% 40% 50%

Color

Size

Call to action

Main image

Layout 42.0%

Fact

ors

Percent effect

39.0%

10.8%

5.9%

2.3%

A/B

MVT

MVT also saves the tester both time and money, as it tests multiple factor-level settings simultaneously. Instead of being limited to varying the values of only one factor, MVT, as its name implies, allows you to test multiple factors at the same time, thereby eliminating the need to run multiple A/B tests. This increased efficiency gives the tester the ability to find the optimal factor-level settings more quickly, and the quicker

the optimal settings are found, the sooner results improve and return on investment increases.

Lastly, as noted in the earlier example, MVT provides the decision maker with better information as potential factor-level interaction effects can be captured. It is most likely that the different factors that make up your email campaigns don’t work in isolation, meaning synergistic effects do exist among the different images, colors, calls to action, sizes, etc. that make up your email messages. The phrase “The whole is greater than the sum of its parts” rings true in these situations. For example, you may have a particular image that performs OK on its own, and a particular background color that also performs OK on its own, but when used together, these particular settings create an email that performs amazingly well. This type of interaction information cannot be captured via A/B or A/B/C testing because only one factor is being analyzed at a time. By setting up tests that more

Multivariate testing Multivariate testing

An Experian Marketing Services white paper | Page 6

accurately account for the true multidimensional nature of your data, you will be able to gather deeper analytic insights that just aren’t possible to collect via A/B or A/B/C testing.

Applications of multivariate testingEmail creative testing

There are many different dimensions of an email creative that could possibly affect subscriber engagement: layout, main image, call to action, color, size, etc. Depending on the number of factors you are interested in testing and the amount of variations you have to test for each factor, the overall number of potential creative combinations can quickly become rather daunting. For example, if you are interested in assessing the effects of two different layouts, two different main images, five different calls to action, two different background colors and two different sizes, you are left with 80 different email combinations to test. However, through the science of experimental design, there exists ways to reduce this number of combinations to test down to a much more reasonable figure. In fact, a properly designed multivariate test can reduce the number of combinations to test from 80 down to as few as 16, a much more manageable situation. Another benefit to this approach is that the tester loses very little in terms of back end, actionable insights when reducing the number of creative combinations to be tested in an intelligent manner. In other words, the ability to identify which factors have a significant effect (and conversely, which factors show little to no effect) on user engagement is still able to be captured, along with the identification of what the optimal settings of the email factors should be in order to maximize subscriber response/engagement.

X X XX

X XX XX

X

X

X

X

X

X

Buy now

Act now

Get moreinfo

Single factor A/B

Full factorial design

Fractional factorial design

Easy to manage, but slow and narrow

Broadest learning, but expensive and often infeasible

Complex to design, but most efficient and effective

Layout

Call to action

Call to action

Layout

Call to action

Imag

e

Imag

e

Source: Experian Marketing Services

Multivariate testing

Page 7 | Understanding multivariate testing techniques and how to apply them to your email marketing strategies

Time of day/Day of week testing

The following graphs show at an aggregate level what was historically observed as the best days and times for an email deployment in Q4 2012. Charts with the best days and times by major industry can be found in the appendix section of this report.

Day of week performance Q4 2012 — all industry

Days

Unique opens

Monday

ThuesdayWednesday

Thursday

Friday

Saturday

Sunday

17% 16.6% 2.7% $.17 $173

16.8% 2.7% $.16 $189

16.7% 2.6% $.13 $188

16.5% 2.5% $.14 $195

16.4% 2.6% $.16 $179

17.8% 2.9% $.20 $17617.8%

0.0% 20.0%Unique clicks0.0% 4.0%

Trans. rate0.10% .20%

Rev. per email$.10 $.20

Average order$0 $100 $200

2.9% $.19

.12%

.12%

.11%

.11%

.13%

.16%.14% $189

15%16%

15%

17%

9%

10%

Volume %

Time of day performance Q4 2012 – all industry

Range name% of

Transactions

Uniqueopens

12:00a.m.–3:59a.m.4:00a.m.–7:59a.m.8:00a.m.–11:59a.m.12:00p.m.–3:59p.m.

4:00p.m.–7:59p.m.

8:00p.m.–11:59p.m.

12%20%40%15%

10%

2%

14%22%42%12%

9%

2%

17.6% 3.2% $.22 $135

0.0% 30% 0.0% 6.0% 0.0% .40% $0 $.60 $0 $300Uniqueclicks

Trans.rate

Rev. peremail

Averageorder

.20%16.2% 2.5% $.18 $164.15%

16.1% 2.4% $.17 $174.13%17.6% 2.8% $.15 $188.13%

18.2% 2.9% $.16 $182.15%21.7% 4.2% $.48 $246.34%

% ofVolume

Source: Experian Marketing Services

Multivariate testing Multivariate testing

An Experian Marketing Services white paper | Page 8

Looking at these results, we see that Saturday and Sunday had higher open, click and transaction rates, but much lower volume. For weekdays, Monday had the highest revenue per mail, but Friday had the better click rate. Volume also plays a role in time of day, as 8 p.m. to midnight and midnight to 4 a.m. had higher responses than the more popular 4 a.m. to noon times. However, as every brand’s customers behave differently, it is most likely the case that the optimal day of week and time of day for your email deployments will show a different pattern of performance. It is important to remember that the data from the aggregate graph is retrospective, not a controlled study. When designing a time of day/day of week study for your subscribers, it is imperative that the various day/time groups are defined ahead of time and in a manner that incorporates random sampling techniques.

A properly designed MVT will give you clarity into what matters most in terms of deployment decisions and which day/time combinations have proven themselves to be statistically significant in their ability to provide improved metrics for your business.

Frequency and recency testingThe question of “How often to mail?” cannot be answered without a comprehensive testing plan. Trying to assess the effects of frequency or recency on retrospective data has a much higher risk of confounding or misallocating performance due to other subscriber attributes like engagement or tenure. To help remove this noise from the process, MVT can be leveraged to create proper test groups beforehand, as well as identify what sample size requirements need to be met in order for results to be statistically significant. The use of forward-looking multivariate test designs versus backward-looking retrospective analyses will provide you with the assurance that the conclusions drawn from your study are accurate and statistically meaningful determinations of the optimal delivery settings for your subscribers.

When testing for optimal frequencies, it is also important to remember that not all engagement segments should be exposed to the same amount of experimentation. Your engaged audience is already demonstrating desirable levels of performance, and you want this behavior to remain. You don’t want to run the risk of disrupting this routine with too many frequency tests. At the other end of the spectrum, you do not have much to lose when testing against your inactive audience; there is only upside to be gained. Take advantage of this situation, and test against a wider range of subscribers to see if any different patterns of frequency tend to spur added levels of engagement.

Multivariate testing

Page 9 | Understanding multivariate testing techniques and how to apply them to your email marketing strategies

Email cadence

Testconservatively

Testliberally

Highlyengaged

Moderately engaged

Inactive

Sample learnings

SegmentCurrent frequency/ recency

Optimal frequency/ recency

Highly engaged12x per month/ evenly staggered

15x per month/ evenly staggered

Moderately engaged12x per month/ evenly staggered

10x per month/ evenly staggered

Inactive12x per month/ evenly staggered

2x per month/ clustered

For more information on how to re-engage inactives, download Experian Marketing Services’ reactivation white paper.

Multivariate testing Multivariate testing

An Experian Marketing Services white paper | Page 10

ConclusionRemember, testing all the various aspects of your email program for optimal performance is an important tactic for any successful email marketing campaign. The testing process is also iterative, with no prescribed end. As the business environment and goals of your brand change alongside the needs and wants of your customers, it is important to continually test and fine-tune user experiences on a regular basis to ensure your marketing messages are primed for maximum effect.

For more information on how to design and execute multivariate testing with your email marketing programs, please visit our website at www.experian.com/marketingservices or call us at 1 866 626 6479.

Experian Marketing Services955 American LaneSchaumburg, IL 601731 866 626 6479 www.experian.com/marketingservices

© 2013 Experian Information Solutions, Inc. • All rights reserved Experian and the Experian marks used herein are service marks or registered trademarks of Experian Information Solutions, Inc.

Other product and company names mentioned herein are the property of their respective owners.

03/2013