best practice webinar: effective a/b tests [oct. 18, 2012]
Post on 29-Jan-2015
106 Views
Preview:
DESCRIPTION
TRANSCRIPT
Testing for SuccessHow to Effectively Utilize A/B Split Testing
Fishbowl Best Practices Webinar
Hosted by:
Drew HebbleSenior Marketing Specialist
AGENDA
• The Basics
• Getting Started
• Improving Open Rates
• Improving ClickThrough Rates
• After the A/B Split Test
• Q & A
THE BASICS: WHAT IS AN A/B SPLIT TEST & HOW DOES IT WORK
What is an A/B Split Test?
A simple method for comparing the performance of two different versions of an e-mail campaign
How Does an A/B Test Split Work?
•A small subset of the total member list is selected as a test group and divided into two equal parts.
•Each half of the test group receives a different version of the same message and a winner is identified
The higher-performing message is blasted to the remainder of the list.
How Will A/B Split Testing Help my eClub?
Understanding the behavior of your e-club members will help you construct a messaging strategy that better engages them
THE BETTER YOUR PROGRAM PERFORMS AGAINST THESE KEY METRICS, THE GREATER THE SALES IMPACT!
Engaged members drive:
– Higher open rates
– Higher click-through rates
– Lower unsubscribe rates
GETTING STARTED: CHOOSING THE APPROPRIATE A/B SPLIT TEST
Getting Started: What Should You Test?
Common test variables include:
Subject lines
Creative design
Call to action language
Call to action placement
Call to action style
Send times (time of day or week)
From lines
Actions based on findings
Each A/B split test should isolate a specific message component that has the potential to provide actionable results
Getting Started: What Should I Measure?
Open Rate
• Key Test Variables– Subject line
– Time of day & day of week
Click-through Rate (CTR)
• Key Test Variables
– Call to action
• Language,
• Style
• Placement
Open CTR
What’s Most Common
Getting Started: Guidelines for a Clean & Effective Test
1. Restrict each test to 1 variable
2. Make sure that your results can be measured
3. Construct a fair comparison
A comparison of two subject lines sent at different times of day makes it difficult to directly correlate results with to one of the variables
Comparing two non-clickable calls to action would not provide an action to measure
“Free appetizer today!” vs. “Your October Restaurant Newsletter” are too different to provide a fair comparison.
IMPROVING OPEN RATES: A/B SPLIT TESTING OF SUBJECT LINES
Subject Line Testing: Good Examples
• Offer “teaser” vs. Specific offer
Enjoy this Special Birthday Gift!
Enjoy your Free Entrée Birthday Gift!
• Personalization
Drew, enjoy a free dessert on us!
Enjoy a free dessert on us!
• Subject line length
We’ve got it all!
We’ve got it all! Good food, music, giveaways and more!
AB
AB
AB
Subject Line Testing: Good Examples
• Symbols
☂ Rainy day special! ☂
Rainy day special!
• Placement of offer copy
Free Appetizer for you and a Friend!
You and a Friend can enjoy a Free Appetizer!
Adjust testing types, based on your branding, assumptions and historical data
AB
AB
Subject Line Testing: Is It a “TRUE” Test?
Have a Free Appetizer on us! vs. Drew, we’d like to offer you a Free Appetizer – today only!
♥ You’re going to love this great deal! ♥ vs. A Free Entrée for you and a friend!
CAUTION… UNFAIR
COMPARISONAHEAD
Subject Line Testing: “I’ve sent my test, now what?”
1. Review key performance metrics for each version of the message (opens, click-throughs, and unsubscribes)
2. Send the top performing subject line to remaining members
3. Monitor winning subject line performance?
4. Analyze the results and use them to improve on next campaign?
The results of each test should add one more piece to the puzzle
What Does Winning Performance Look Like…..
Open Rate ClickThrough Rate Unsubscribe Rate
Subject Line A: Enjoy this Special Birthday Gift!
0.19 0.012 0.0012
Subject Line B: Enjoy your Free Entrée Birthday Gift!
0.23 0.014 0.0013
3%
8%
13%
18%
23%
IMPORTANT! Consider ALL messaging
metrics, NOT just open rate
IMPROVING CLICKTHROUGH RATES: A/B TESTING OF CALL TO ACTION
• Question vs. statement
• Click Here vs. (an implied clickthrough icon)
Call to Action Testing: Language
Want a Free Appetizer?
Click Here for your Free Appetizer
• Left vs. right
• Top vs. bottom
• Inside an image vs. stand alone
Happy Birthday on Us!
Click Here for a free
brownie during the week of
your birthday.
Happy Birthday on Us!
Click Here for a free brownie
during the week of your birthday.
Call to Action Testing: Placement
Call to Action Testing: Visual Style
To help you through this rainy weather, Click here to enjoy a free appetizer with any entrée purchase.
To help you through this rainy weather
Click Here for your free appetizer with any entrée purchase.
• Clickable body copy vs. button
• Animation vs. static image
• Different color copy/button
• Different images on each version
• One has a clickable call to action, the other has no measurable variable
Call to Action Testing: Is It A “TRUE” Test
Click Here for your free appetizer with any entrée purchase.
AVOID UNFAIR COMPARISONS
Improving CTR: “I’ve sent my test, now what?”
Want a Free Appetizer?
Click Here for your Free Appetizer
Version A (2.8% CTR) Version B (1.7% CTR)
1. Review click-through performance metrics for each version of the message
2. Send top performing call to action to remaining members3. Monitor winning call to action performance across the remaining audience?4. Analyze the results and use them to improve on next campaign?
AFTER THE A/B TEST:ANALYZING RESULTS & APPLYING LEARNING’S
There Was No Clear Winner…What Should I Do?
• Both versions performed below your average?
– Negative results can be as useful as positive ones. Learn from what your guests don’t respond to, and try a different strategy on your next campaign.
• Click-through results don’t lean one way or the other?
– Review your testing method to ensure that there the difference between the two message versions is significant.
A “Click here to RSVP” vs. “Click below to RSVP” call to action may
not yield a different result
There Was No Clear Winner…What Should I Do?
One version had a much higher open rate, while the other version had a far superior clickthrough rate?
Subject line test results
A) 23% open rate & 1.1% CTR
B) 21% open rate & 3.3% CTR
It’s important to consider the amount of variation between metrics.
Depending on the focus of your message, it’s sometimes better to sacrifice one slightly lower metric for another that is much greater.
Acting on Your Results
Make Assumptions based on learning and test on future campaigns
• Shorter subject line outperformed a longer subject line?
• Red call to action yielded a higher click-through than a blue one?
If YES – You’re on the right track to better understanding your audience. If NO –
Was there a difference on this test that may have swayed results?
Refine your methodology and continue to test until results are consistent.
Are you able to replicate the results
Three Key A/B Testing Takeaways
1. ALWAYS test variable at a time
2. ALWAYS review the performance of your entire mailing, NOT just the A/B test
3. ALWAYS apply your learning's to future campaigns
4. NEVER stop testing and refining
Questions?
Thanks for your time!
Upcoming Fishbowl Webinars
October 18th Optimizing your website to attract new e-club members
November 15th Messaging Best Practices
Resources
For more information on Fishbowl’s Creative and strategic Professional Services offerings:
Enterprise Clients: Please contact your Account Director
Local Clients: Please contact your Sales Representative
Technical questions on how to do A/B subject line tests?Call our Support line 800.883.1984 or
email clientsupport@fishbowl.comSign up for a webinar: http://
www.fishbowl.com/feature/webinar_enterprise_series
top related