![Page 1: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/1.jpg)
TheU: Testing…Testing…123
![Page 2: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/2.jpg)
www.highroadsolution.com
Your Presenter:Jenny Lassi has been affiliated with HighRoad Solution for 8 years first as a client and then as an employee. With experience handling deployments and deliverability for a wide range of clients with optimized recipient engagement top-of-mind, Jenny routinely trouble shoots deliverability issues from ISP level filtering to email client level filtering (email Forensics as she refers to them). She also consults with clients on authentication, content, content strategy, email preference center management & other deliverability optimization tactics.
![Page 3: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/3.jpg)
www.highroadsolution.com
Today’s Content
• What is AB Testing?• What is Multi-Variat Testing?• The 5 C’s & How They Apply to Testing• Starting Simply• Testing the Complex• Best practices• Resources takeaways
![Page 4: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/4.jpg)
www.highroadsolution.com
What is AB Testing?
![Page 5: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/5.jpg)
www.highroadsolution.com
AB Testing Defined
A/B testing is a term commonly used in web development, online marketing, and other forms of advertising to describe simple randomized experiments with two variants, A and B, which are the control and treatment in the controlled experiment.
The formal or scientific name used for this process and other related processes is hypothesis testing. Other names include randomized controlled experiments, online controlled experiments, and split testing.
![Page 6: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/6.jpg)
www.highroadsolution.com
What is Multivariate Testing?
![Page 7: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/7.jpg)
www.highroadsolution.com
Multivariate Testing Defined
In internet marketing, multivariate testing is a process by which more than one component of a website may be tested in a live environment. It can be thought of in simple terms as numerous A/B tests performed on one page at the same time. A/B tests are usually performed to determine the better of two content variations; multivariate testing can theoretically test the effectiveness of limitless combinations.
Note: Be cautious of testing too many variables as it canbe difficult to understand what is behind the lift in response.
![Page 8: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/8.jpg)
www.highroadsolution.com
eMail Strategy
![Page 9: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/9.jpg)
www.highroadsolution.com
The 5 C’s of Email Marketing Strategy
•Configuration (Clean House!)•Content•Create•Criticize•Continue
![Page 10: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/10.jpg)
www.highroadsolution.com
The 5 C’s: Configuration
• IP Address (Dedicated or Shared)• DKIM (Domain Keys)• Sender ID (SPF
Record)
![Page 11: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/11.jpg)
www.highroadsolution.com
The HighRoad Solution Deliverability Cops – Dedicated IP
• Sender Rejected• Message Filtered• Unknown Users• Infrastructure• Volume• Spam Traps• Blacklists• Complaints• FBL Compaints (Routed to
your Opt-Outs)
![Page 12: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/12.jpg)
www.highroadsolution.com
The HighRoad Solution Deliverability Cops – Shared IP
• Spam Traps• Blacklists• Complaints
![Page 13: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/13.jpg)
www.highroadsolution.com
AuthenticationHigh Level View: DKIM/Sender ID (it’s not a scary term or complicated at all… I promise)• You tell us what domain(s) you send from • We supply you with 4 snippets of text that you add to your domain’s DNS and instructions• You add the txt records to your DNS and tell us when you’re done• We test/validate the keys and enable on your account
![Page 14: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/14.jpg)
www.highroadsolution.com
Why not just blast?
Invalid email Addresses +
Excessive email Bounces
+Emails Filtered to Junk/SPAM Folders
=Wasted $$
![Page 15: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/15.jpg)
www.highroadsolution.com
eMail Filtering
Email Filtering Layers
![Page 16: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/16.jpg)
www.highroadsolution.com
Do you know if your eMails are reaching an inbox?
When a SureSend Domain Delivery Report is enabled when sending a Live message, you will see the Domain Delivery tab on the email stats
![Page 17: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/17.jpg)
www.highroadsolution.com
Do you know if your eMails are reaching an inbox?
![Page 18: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/18.jpg)
www.highroadsolution.com
Cleaning HouseDo whatever is in your control to reach an inbox!
• Data Hygiene (Invalids/Bounces)
• DKIM/Sender ID Authentication
• Address any issues with filtering that you can
THEN… and only then should you AB Test
![Page 19: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/19.jpg)
www.highroadsolution.com
The 5 C’s: Content
Common Content Tests:• Subject Lines• Lead Story• TOC• More text vs. read more
![Page 20: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/20.jpg)
www.highroadsolution.com
The 5 C’s: Create
• Test standard html vs. responsive template• Call-To-Action Button Color & Placement • Layout• Images
![Page 21: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/21.jpg)
www.highroadsolution.com
Case Study Workshop
Can you trust your gut?
![Page 22: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/22.jpg)
www.highroadsolution.com
Case Study Workshop
Can being more verbose get a higher
response?
![Page 23: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/23.jpg)
www.highroadsolution.com
Version A
![Page 24: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/24.jpg)
www.highroadsolution.com
Version B
![Page 25: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/25.jpg)
www.highroadsolution.com
Case Study Workshop
What do you think?Version A?Version B?
![Page 26: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/26.jpg)
www.highroadsolution.com
Case Study Workshop
Version B won an astounding 247%
more clicks
![Page 27: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/27.jpg)
www.highroadsolution.com
Case Study Workshop
Does a larger lead story get people to
click?
![Page 28: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/28.jpg)
www.highroadsolution.com
Version A
![Page 29: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/29.jpg)
www.highroadsolution.com
Version B
![Page 30: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/30.jpg)
www.highroadsolution.com
Case Study Workshop
What do you think?Version A?Version B?
![Page 31: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/31.jpg)
www.highroadsolution.com
Case Study Workshop
Version B, the one without bigger
featured content, increased clicks by
75% and online registrations by a stunning 400%
![Page 32: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/32.jpg)
www.highroadsolution.com
Case Study Workshop
Does more or less white space increase
engagement?
![Page 33: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/33.jpg)
www.highroadsolution.com
Version A
![Page 34: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/34.jpg)
www.highroadsolution.com
Version B
![Page 35: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/35.jpg)
www.highroadsolution.com
Case Study Workshop
What do you think?Version A?Version B?
![Page 36: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/36.jpg)
www.highroadsolution.com
Case Study Workshop
Version A Increased clickthroughs by
5.6%, and heatmaps showed that there
was a higher engagement as a whole across the
entire email.
![Page 37: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/37.jpg)
www.highroadsolution.com
Case Study Workshop
Subject line straight shooter or a bit
playful?
![Page 38: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/38.jpg)
www.highroadsolution.com
Version A
![Page 39: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/39.jpg)
www.highroadsolution.com
Version B
![Page 40: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/40.jpg)
www.highroadsolution.com
Case Study Workshop
What do you think?Version A?Version B?
![Page 41: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/41.jpg)
www.highroadsolution.com
Case Study Workshop
Version B – the playful version – increased opens for French-
speaking Canadians. However, Version A – the
plainspoken version – increased opens for English-speaking
Canadians.
![Page 42: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/42.jpg)
www.highroadsolution.com
Case Study Workshop
Which local time works better for B2B email offers 9am or
1pm?
![Page 43: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/43.jpg)
www.highroadsolution.com
Version A – 9am
![Page 44: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/44.jpg)
www.highroadsolution.com
Version B – 1pm
![Page 45: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/45.jpg)
www.highroadsolution.com
Case Study Workshop
What do you think?Version A?Version B?
![Page 46: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/46.jpg)
www.highroadsolution.com
Case Study Workshop
Version A, the email sent at 9 am local time at four U.S. time zones increased lifts in all time
zones – here are the specifics:
• EST – 12.5% lift in opens• CST – 100% lift in opens• MDT – 35% lift in opens• PST – 4,090% STUNNING lift in opens
![Page 47: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/47.jpg)
www.highroadsolution.com
The 5 C’s: Criticize
• Analyze delivery metrics (bounces, opens & clicks)
• Use bounce logs to address deliverability issues 24-48 hours post deployment (not a month later)
• Isolate subscribers who are not engaged with your emails (no opens/click throughs)
• Create an AB Test Win-Back campaign for non-engaged subscribers with an incentive to respond
• Analyze how many subscribers opt-down or unsubscribe from particular messages (maybe the perceived value of content is low)
![Page 48: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/48.jpg)
www.highroadsolution.com
The 5 C’s: Continue
Use what you have learned and apply it
to future deployments
![Page 49: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/49.jpg)
www.highroadsolution.com
Best Practices Takeaways• Be more strategic with your testing with the 5-C’s.• You can learn from other AB Tests, but testing against
your own audience is the only way to see what increases engagement.
• Send to known engaged subscribers when testing or your results may vary based on inbox placement.
![Page 50: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/50.jpg)
www.highroadsolution.com
Questions?
![Page 51: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/51.jpg)
www.highroadsolution.com
ShortcutsControl what you CAN control about deliverability:
• DKIM/Sender ID Authentication • List Hygiene • Having an Email Preference Center• Deliver content members or registrants want, when they
want it, how frequently they want it and looks good where they read it
Analyze and learn from AB Split testing to adjust how you send future emails:
• Use A/B split testing to learn about what has the best engagement
• Review creative/content that had the best engagement and do more of that
• Review bounce logs daily to see if any domains may be blocking you and reach out to postmaster@ and abuse@ those domains to ask them to whitelist you
![Page 52: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/52.jpg)
www.highroadsolution.com
Thank you!Other questions for Jenny?
eMail = [email protected]
Twitter = @highroadjenny
![Page 53: Why Guess When You Can Test? Testing Techniques from the Experts](https://reader035.vdocument.in/reader035/viewer/2022062418/554ad570b4c90542708b56ca/html5/thumbnails/53.jpg)
www.highroadsolution.com
References & Resourceshttps://en.wikipedia.org/wiki/Ab_testinghttp://en.wikipedia.org/wiki/Multivariate_testinghttp://whichtestwon.com/