marketingsherpa email testing

65
Join Daniel Burstein as he interviews Amelia Showalter, Director of Digital Analytics, Obama for America as they discuss the important role of Email testing during the 2012 Presidential Campaign. Live Event June 19, 2013 at 3:30 EDT! Access our other webinars

Upload: act-on-software

Post on 08-May-2015

213 views

Category:

Technology


2 download

TRANSCRIPT

Page 1: MarketingSherpa Email Testing

SPONSOR LOGO

Join Daniel Burstein as he interviews Amelia Showalter, Director of Digital Analytics, Obama for America as they discuss the important role of Email testing during the 2012 Presidential Campaign.

Live Event June 19, 2013 at 3:30 EDT!

Access our other webinars

Page 2: MarketingSherpa Email Testing

Join the conversation

#SherpaWebinar

Page 3: MarketingSherpa Email Testing

Email Optimization How A/B testing generated $500 million in 

donations

Page 4: MarketingSherpa Email Testing

Today’s presenters

Daniel BursteinDirector of Editorial ContentMECLABS@DanielBurstein @ameliashowalter

Amelia Showalter Director of Digital AnalyticsObama for America

Presen

ters

Page 5: MarketingSherpa Email Testing

Join the conversation

#SherpaWebinar

Page 6: MarketingSherpa Email Testing

Optimization Summit 2013 Wrap‐up: Top 5 takeaways for testing websites, pay‐per‐click ads and email– http://www.marketingsherpa.com/article/wrap‐up/top‐5‐takeaways‐op‐summit‐2013

Email Testing: How the Obama campaign generated approximately $500 million in donations from email marketing – http://www.marketingsherpa.com/article/case‐study/obama‐email‐campaign‐testing

Email Marketing: 77% of marketers use website registration pages to build email lists –http://sherpablog.marketingsherpa.com/email‐marketing/77‐percent‐use‐website‐registration/

Marketing Strategy: How to find answers to the most common marketing questions –http://sherpablog.marketingsherpa.com/marketing/most‐common‐marketing‐questions/

Resources

Further Resources

Page 7: MarketingSherpa Email Testing

You’ve heard this story…

Page 8: MarketingSherpa Email Testing

• The Ground Game

• Thousands of paid staffers organized 2 million volunteers to get voters registered and to the polls.

Fact check: TRUE

Source: TheMonkeyCage.Org

0

200

400

600

800

1000

Obama Romney

Field Offices

786

284

Ohio

Page 9: MarketingSherpa Email Testing

You’ve heard this story…

Page 10: MarketingSherpa Email Testing

Fact check: TRUE

Sophisticated technology and targeting• Giant data  microtargeting operation

• Fully integrated databases

• Smarter new methods of targeting TV ads

• New technology developed in‐house for social sharing, polling place lookup, phone banking, volunteer mobilization, vote tracking, election day rapid response

Page 11: MarketingSherpa Email Testing

What you didn’t hear

All of that costs money

Page 12: MarketingSherpa Email Testing

What you didn’t hear

All of that costs money

Page 13: MarketingSherpa Email Testing

• In 2008, Obama campaign raised $750 million• Would not be enough in 2012

The fundraising challenge

Not impressed

$750 million?

Page 14: MarketingSherpa Email Testing

The fundraising challenge

Fundraising was proving more difficult in 2012 than in 2008

• President was less available for fundraising events• Early in the campaign, we saw average online donations were half of what it had been in 2008

• People were giving less, and less often

We had to be smarter, and more tenacious

Page 15: MarketingSherpa Email Testing

The fundraising challenge

The real game changer? It’s something most of you already do every day.

Page 16: MarketingSherpa Email Testing

First, the results:

more than $500,000,000 

in online donations

4.5 million donors 

$53 average gift

Page 17: MarketingSherpa Email Testing

How Did We Do It?

Being Pretty isn’t 

Everything

Don’t Trust Your Gut

Content Matters

Foster a Culture of Testing

Invest in Your Team

Incentives Work

Page 18: MarketingSherpa Email Testing

SPONSOR LOGO

Winning with A/B Testing

Page 19: MarketingSherpa Email Testing

What impact can testing have?

Page 20: MarketingSherpa Email Testing

Testing = constant improvement• Little improvements add up

• Improving 1% here and 2% there isn’t a lot at first, but over time, it adds up

Page 21: MarketingSherpa Email Testing

Test every element

Question: What footer language should we use to reduce unsubscribes?

Variation: Recipients UnsubscribesUnsubscribe/Recipient

Significant difference in unsubscribe/recipient

578,994 195 0.018% None

578,814 79 0.014% Smaller than D4

578,620 86 0.015% Smaller than D4

580,507 115 0.020% Larger than 3D and 4D

Page 22: MarketingSherpa Email Testing

Tests upon tests upon tests

• Every piece of communication is an opportunity to test

• A single email can have many tests attached

Subject and draft tests Full‐list tests

Background personalization 

tests

Page 23: MarketingSherpa Email Testing

No, really. Test every element.

Running tests in the background via personalized content

Page 24: MarketingSherpa Email Testing

Longitudinal tests: 

Experiment: gave sample audience higher volume of email for an extended time, and compared to control

Results: more email = more donations• People may say they get too much email• But mild annoyance proved to be the worst result• Unsubscribes accrued linearly• Donations did, too

Implementing a “more email” policy probably led to $20 to $30 million in additional revenue for the campaign.

How much email should we send?

*

Page 25: MarketingSherpa Email Testing

So, we made shirts.

Page 26: MarketingSherpa Email Testing

Lessons

Page 27: MarketingSherpa Email Testing

SPONSOR LOGO

Drafts and Subject Lines Matter

Page 28: MarketingSherpa Email Testing

Example: Subject lines

version Subject line v1s1 Heyv1s2 Two things:v1s3 Your turnv2s1 Heyv2s2 My opponentv2s3 You decidev3s1 Heyv3s2 Last nightv3s3 Stand with me todayv4s1 Heyv4s2 This is my last campaignv4s3 [NAME]v5s1 Hey

v5s2There won't be many more of these deadlines

v5s3 What you saw this weekv6s1 Heyv6s2 Let's win.v6s3 Midnight deadline

Each draft was tested with three subject lines

One subject line would usually be common across all drafts to help make comparisons across messages

Test sends

Page 29: MarketingSherpa Email Testing

Example: Draft language – Version 1

Page 30: MarketingSherpa Email Testing

Example: Draft language – Version 2

Page 31: MarketingSherpa Email Testing

Example: Draft language – Version 3

Page 32: MarketingSherpa Email Testing

Example: Draft language – Version 4

Page 33: MarketingSherpa Email Testing

Example: Draft language – Version 5

Page 34: MarketingSherpa Email Testing

Example: Draft language – Version 6

Page 36: MarketingSherpa Email Testing
Page 37: MarketingSherpa Email Testing

Example: Best vs. Worst Versions 

version Subject line  donors moneyv1s1 Hey 263 $17,646v1s2 Two things: 268 $18,830v1s3 Your turn 276 $22,380v2s1 Hey 300 $17,644v2s2 My opponent 246 $13,795v2s3 You decide 222 $27,185v3s1 Hey 370 $29,976v3s2 Last night 307 $16,945v3s3 Stand with me today 381 $25,881v4s1 Hey 444 $25,643v4s2 This is my last campaign 369 $24,759v4s3 [NAME] 514 $34,308v5s1 Hey 353 $22,190

v5s2There won't be many more of these deadlines 273 $22,405

v5s3 What you saw this week 263 $21,014v6s1 Hey 363 $25,689v6s2 Let's win. 237 $17,154v6s3 Midnight deadline 352 $23,244

$0

$1

$2

$3

$4

ACTUAL ($3.7m)

IF SENDING AVG

IF SENDING WORST

Full send (in millions)

$2.2 million additional revenue from sending best draft vs. worst, or $1.5 million additional from sending best vs. average

Test sends

Page 38: MarketingSherpa Email Testing

The Best Performer:

Amelia

Barack Obama <[email protected]>From:

Subject:

Name as subject

Version 4

Page 39: MarketingSherpa Email Testing

Some of the best subject lines:

Page 40: MarketingSherpa Email Testing

Our research team at Meclabs wanted to know….

Is This Transferable? 

Page 41: MarketingSherpa Email Testing

… So we tested.Twice.

Is This Transferable? 

http://3.bp.blogspot.com/‐T66ipEp81J8/UGQ7sbe‐ViI/AAAAAAAAARY/JNgjYQVVI5g/s1600/Doc+Brown.jpg

Our research team at Meclabs wanted to know….

Page 42: MarketingSherpa Email Testing

Control

Subject: “[Live Webinar] How A/B testing 

generated $500 million in donations”

First Line: “Come see an inside look into the email and digital 

analytics efforts that fueled President Barack 

Obama’s 2012 presidential campaign.”

Page 43: MarketingSherpa Email Testing

Treatment

Hey

Subject: “Hey”

First Line: “Tune in this Wednesday to see why we used the “Hey” 

subject line.”

Page 44: MarketingSherpa Email Testing

44

Results:

Increase in Opens 51.5%Increase in Clickthrough214.6%

*Results are at a 99% Level of Confidence

Increase in Opens 33%Increase in Clickthrough112%

Experiment 1

Experiment 1 Experiment 2

Experiment 2

Open Rate CTR Open Rate CTR

Control 19.48% 1.006% 15.27% 1.065%Treatment 29.53% 3.164% 20.43% 2.257%RelativeDifference 51.5% 214.6% 33% 112%

Page 45: MarketingSherpa Email Testing

SPONSOR LOGO

Don’t Trust Your Gut

Page 46: MarketingSherpa Email Testing

We don’t have all the answers• Conventional wisdom is often wrong• Long‐held best practices are often wrong• Going with things that previously tested well were often wrong

Testing = data‐driven decisions

There was this thing called the Email Derby…

E‐mail

Page 47: MarketingSherpa Email Testing

SPONSOR LOGO

The Prettiest aren’t Always the Best

Page 48: MarketingSherpa Email Testing

Experiments: Ugly vs. Pretty

• We tested sleek and pretty• That failed, so we asked: what about ugly?

• Ugly yellow highlighting got us better results• But at some point, it lost its novelty and stopped working

• always important to re‐test!

Ready to fight? Please donate $19 or more today, ahead of the FEC deadlineBecause you’ve saved your payment information, your donation will go through immediately.

QUICK DONATE‐ $19

‐ QUICK DONATE $35‐ QUICK DONATE $50‐ QUICK DONATE $100

‐ QUICK DONATE $250

Or donate another amount

https://donate.barackobama.com/june‐deadline 

Ready to fight? Please donate $19 or more today, ahead of the FEC deadlineBecause you’ve saved your payment information, your donation will go through immediately.

‐ QUICK DONATE $35‐ QUICK DONATE $50‐ QUICK DONATE $100

‐ QUICK DONATE $250

Or donate another amount

https://donate.barackobama.com/june‐deadline 

QUICK DONATE ‐ $19

Page 49: MarketingSherpa Email Testing

SPONSOR LOGO

Incentives Matter

Page 50: MarketingSherpa Email Testing

People respond to incentives

Giving away bumper stickers and car magnets and then daisy chaining to a donate page yielded enough donations to pay for the freebies immediately.

Our Quick Donate program, in turn, raised donation rates by 50% or more

Offering a free bumper sticker for enrolling in our Quick Donate program increased conversions by 30%

Page 51: MarketingSherpa Email Testing

SPONSOR LOGO

Invest in Your Team

Page 52: MarketingSherpa Email Testing

Email, Social Media, Mobile, Blog

OFA Digital Department

OutboundDigital Analysis

Front‐End Development

ProjectManagement

DesignVideo

2012Small Team

2011

Department of 200+

2012

Page 53: MarketingSherpa Email Testing

More voices, more talents

• Outbound Team•18 email writers•4 social media writers and bloggers

Digital Analytics Team•15 analysts with overlapping skills

•Database management •Data analysis •Web analytics 

Page 54: MarketingSherpa Email Testing

The human element and our voice

Honesty

Authenticity

Honesty-and-

Authenticity

Page 55: MarketingSherpa Email Testing

Real people, real characters

Rufus GiffordAn emotional roller coaster

Ann Marie HabershawTough love

Page 56: MarketingSherpa Email Testing

SPONSOR LOGO

Foster a Culture of Testing

Page 57: MarketingSherpa Email Testing

The culture of testing

• Check your ego at the door• Use every opportunity to test something• Compare against yourself, not against your competitors or “the industry”

Are you doing better this month than last month?

Are you doing better than you would have otherwise?

Page 58: MarketingSherpa Email Testing

Keep a testing calendar

• We had short‐term and long‐term calendars for national emails

• We added a “tests” column to plan out which tests would be attached to which emails

• If we saw blank spaces, it would remind us to think of more tests to run

Important to do frequent brainstorming sessions

Page 59: MarketingSherpa Email Testing

Circulate your test results internally

Internal list serve for 

circulating test results

Helped get  buy‐in and increased 

familiarity with testing process

Prompted     discussion and generated new testing 

ideas

Page 60: MarketingSherpa Email Testing

The Big Picture

Page 61: MarketingSherpa Email Testing

Testing wins.

• This mentality was applied across the board:

• Did we mention raising half‐a‐billion dollars?• Testing resulted in about $200,000,000 in additional revenue 

• Helped recruit 2,000,000 volunteers• Helped build for thousands of phone banks, rallies and events• Got information and “the message” into the hands of our best messengers

…and that’s a conservative estimate

Page 62: MarketingSherpa Email Testing

Big data ≠ big brother

Testing allows you to listen to your user baseLet them tell you what they like

Optimization gives them a better experience

Usually, the interactions that are the most human are the ones that win.

Page 63: MarketingSherpa Email Testing

Experiments: PersonalizationAdding “drop‐in sentences” that reference past behavior can increase conversion rates

…it's going to take a lot more of us to match them. Will you donate $25 or more today?

…it's going to take a lot more of us to match them.You stepped up recently to help out ‐‐ thank you. We all need to dig a little deeper if we're going to win, so I'm asking you to pitch in again. Will you donate $25 or more today?

Example: asking recent donors for more money

Added sentence significantly raised donation rateConfirmed in several similar experiments

Version 1

Version 2

Page 64: MarketingSherpa Email Testing

Final Conclusions

• Big groups of smart people working together can accomplish a lot, even in 18 months

• But, you don’t need a staff of hundreds to have a good testing program

Train existing staffers, hire more when you canFoster a culture of testing: Every piece of communication     is an opportunity to test something

Even a small list can be split in two – do what you can

Page 65: MarketingSherpa Email Testing