![Page 1: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/1.jpg)
Testing for SuccessHow to Effectively Utilize A/B Split Testing
![Page 2: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/2.jpg)
Fishbowl Best Practices Webinar
Hosted by:
Drew HebbleSenior Marketing Specialist
![Page 3: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/3.jpg)
AGENDA
• The Basics
• Getting Started
• Improving Open Rates
• Improving ClickThrough Rates
• After the A/B Split Test
• Q & A
![Page 4: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/4.jpg)
THE BASICS: WHAT IS AN A/B SPLIT TEST & HOW DOES IT WORK
![Page 5: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/5.jpg)
What is an A/B Split Test?
A simple method for comparing the performance of two different versions of an e-mail campaign
![Page 6: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/6.jpg)
How Does an A/B Test Split Work?
•A small subset of the total member list is selected as a test group and divided into two equal parts.
•Each half of the test group receives a different version of the same message and a winner is identified
The higher-performing message is blasted to the remainder of the list.
![Page 7: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/7.jpg)
How Will A/B Split Testing Help my eClub?
Understanding the behavior of your e-club members will help you construct a messaging strategy that better engages them
THE BETTER YOUR PROGRAM PERFORMS AGAINST THESE KEY METRICS, THE GREATER THE SALES IMPACT!
Engaged members drive:
– Higher open rates
– Higher click-through rates
– Lower unsubscribe rates
![Page 8: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/8.jpg)
GETTING STARTED: CHOOSING THE APPROPRIATE A/B SPLIT TEST
![Page 9: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/9.jpg)
Getting Started: What Should You Test?
Common test variables include:
Subject lines
Creative design
Call to action language
Call to action placement
Call to action style
Send times (time of day or week)
From lines
Actions based on findings
Each A/B split test should isolate a specific message component that has the potential to provide actionable results
![Page 10: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/10.jpg)
Getting Started: What Should I Measure?
Open Rate
• Key Test Variables– Subject line
– Time of day & day of week
Click-through Rate (CTR)
• Key Test Variables
– Call to action
• Language,
• Style
• Placement
Open CTR
What’s Most Common
![Page 11: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/11.jpg)
Getting Started: Guidelines for a Clean & Effective Test
1. Restrict each test to 1 variable
2. Make sure that your results can be measured
3. Construct a fair comparison
A comparison of two subject lines sent at different times of day makes it difficult to directly correlate results with to one of the variables
Comparing two non-clickable calls to action would not provide an action to measure
“Free appetizer today!” vs. “Your October Restaurant Newsletter” are too different to provide a fair comparison.
![Page 12: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/12.jpg)
IMPROVING OPEN RATES: A/B SPLIT TESTING OF SUBJECT LINES
![Page 13: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/13.jpg)
Subject Line Testing: Good Examples
• Offer “teaser” vs. Specific offer
Enjoy this Special Birthday Gift!
Enjoy your Free Entrée Birthday Gift!
• Personalization
Drew, enjoy a free dessert on us!
Enjoy a free dessert on us!
• Subject line length
We’ve got it all!
We’ve got it all! Good food, music, giveaways and more!
AB
AB
AB
![Page 14: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/14.jpg)
Subject Line Testing: Good Examples
• Symbols
☂ Rainy day special! ☂
Rainy day special!
• Placement of offer copy
Free Appetizer for you and a Friend!
You and a Friend can enjoy a Free Appetizer!
Adjust testing types, based on your branding, assumptions and historical data
AB
AB
![Page 15: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/15.jpg)
Subject Line Testing: Is It a “TRUE” Test?
Have a Free Appetizer on us! vs. Drew, we’d like to offer you a Free Appetizer – today only!
♥ You’re going to love this great deal! ♥ vs. A Free Entrée for you and a friend!
CAUTION… UNFAIR
COMPARISONAHEAD
![Page 16: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/16.jpg)
Subject Line Testing: “I’ve sent my test, now what?”
1. Review key performance metrics for each version of the message (opens, click-throughs, and unsubscribes)
2. Send the top performing subject line to remaining members
3. Monitor winning subject line performance?
4. Analyze the results and use them to improve on next campaign?
The results of each test should add one more piece to the puzzle
![Page 17: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/17.jpg)
What Does Winning Performance Look Like…..
Open Rate ClickThrough Rate Unsubscribe Rate
Subject Line A: Enjoy this Special Birthday Gift!
0.19 0.012 0.0012
Subject Line B: Enjoy your Free Entrée Birthday Gift!
0.23 0.014 0.0013
3%
8%
13%
18%
23%
IMPORTANT! Consider ALL messaging
metrics, NOT just open rate
![Page 18: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/18.jpg)
IMPROVING CLICKTHROUGH RATES: A/B TESTING OF CALL TO ACTION
![Page 19: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/19.jpg)
• Question vs. statement
• Click Here vs. (an implied clickthrough icon)
Call to Action Testing: Language
Want a Free Appetizer?
Click Here for your Free Appetizer
![Page 20: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/20.jpg)
• Left vs. right
• Top vs. bottom
• Inside an image vs. stand alone
Happy Birthday on Us!
Click Here for a free
brownie during the week of
your birthday.
Happy Birthday on Us!
Click Here for a free brownie
during the week of your birthday.
Call to Action Testing: Placement
![Page 21: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/21.jpg)
Call to Action Testing: Visual Style
To help you through this rainy weather, Click here to enjoy a free appetizer with any entrée purchase.
To help you through this rainy weather
Click Here for your free appetizer with any entrée purchase.
• Clickable body copy vs. button
• Animation vs. static image
• Different color copy/button
![Page 22: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/22.jpg)
• Different images on each version
• One has a clickable call to action, the other has no measurable variable
Call to Action Testing: Is It A “TRUE” Test
Click Here for your free appetizer with any entrée purchase.
AVOID UNFAIR COMPARISONS
![Page 23: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/23.jpg)
Improving CTR: “I’ve sent my test, now what?”
Want a Free Appetizer?
Click Here for your Free Appetizer
Version A (2.8% CTR) Version B (1.7% CTR)
1. Review click-through performance metrics for each version of the message
2. Send top performing call to action to remaining members3. Monitor winning call to action performance across the remaining audience?4. Analyze the results and use them to improve on next campaign?
![Page 24: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/24.jpg)
AFTER THE A/B TEST:ANALYZING RESULTS & APPLYING LEARNING’S
![Page 25: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/25.jpg)
There Was No Clear Winner…What Should I Do?
• Both versions performed below your average?
– Negative results can be as useful as positive ones. Learn from what your guests don’t respond to, and try a different strategy on your next campaign.
• Click-through results don’t lean one way or the other?
– Review your testing method to ensure that there the difference between the two message versions is significant.
A “Click here to RSVP” vs. “Click below to RSVP” call to action may
not yield a different result
![Page 26: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/26.jpg)
There Was No Clear Winner…What Should I Do?
One version had a much higher open rate, while the other version had a far superior clickthrough rate?
Subject line test results
A) 23% open rate & 1.1% CTR
B) 21% open rate & 3.3% CTR
It’s important to consider the amount of variation between metrics.
Depending on the focus of your message, it’s sometimes better to sacrifice one slightly lower metric for another that is much greater.
![Page 27: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/27.jpg)
Acting on Your Results
Make Assumptions based on learning and test on future campaigns
• Shorter subject line outperformed a longer subject line?
• Red call to action yielded a higher click-through than a blue one?
If YES – You’re on the right track to better understanding your audience. If NO –
Was there a difference on this test that may have swayed results?
Refine your methodology and continue to test until results are consistent.
Are you able to replicate the results
![Page 28: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/28.jpg)
Three Key A/B Testing Takeaways
1. ALWAYS test variable at a time
2. ALWAYS review the performance of your entire mailing, NOT just the A/B test
3. ALWAYS apply your learning's to future campaigns
4. NEVER stop testing and refining
![Page 29: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/29.jpg)
Questions?
Thanks for your time!
![Page 30: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/30.jpg)
Upcoming Fishbowl Webinars
October 18th Optimizing your website to attract new e-club members
November 15th Messaging Best Practices
![Page 31: Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]](https://reader034.vdocument.in/reader034/viewer/2022051819/54c95dda4a795902588b459c/html5/thumbnails/31.jpg)
Resources
For more information on Fishbowl’s Creative and strategic Professional Services offerings:
Enterprise Clients: Please contact your Account Director
Local Clients: Please contact your Sales Representative
Technical questions on how to do A/B subject line tests?Call our Support line 800.883.1984 or
email [email protected] up for a webinar: http://
www.fishbowl.com/feature/webinar_enterprise_series