ntc 2016 step right up testing presentation 16x9
TRANSCRIPT
Step Right Up! Everyone’s a Winner!
Steve Kerhli, PETA & Steve Daigneault, M+R
Ask String on Main Donation Form
Test Control
Vote: Control Test Tie
Winner: Test
Renewal Language to Prospects
Test Control
Vote: Control Test Tie
Winner: Test
Higher Ask String for Prospects
Test Control
Vote: Control Test Tie
Winner: Tie
SMS Deadline Reminder
Test Control
Vote: Control Test Tie
Winner: Test
Nudge to Prospects
Test Control
Vote: Control Test Tie
Winner: Test
Font Type and Size
Test Control
Vote: Control Test Tie
Winner: Test
$5 Ask to Prospects
Test Control
Vote: Control Test Tie
Winner: Test
Make your tax-deductible gift
of $50 or whatever you can
afford now. Help us meet our
urgent Year-End Challenge goal
of $125,000.
Make your tax-deductible gift
now. Pitch in $5 or more to help
us meet our urgent Year-End
Challenge goal of $125,000.
Adding PayPal as Payment Option
Test Control
Vote: Control Test Tie
Winner: Test
“Renew” Language to Prospects
Test Control
Vote: Control Test Tie
Winner: Test
Another Nudge
Test Control
Vote: Control Test Tie
Winner: Tie
Cents Per Day
Test Control
Vote: Control Test Tie
Winner: Control
Yes, Another Nudge Test
Test Control
Vote: Control Test Tie
Winner: Test
Multi-Button vs. Single Button
Test Control
Vote: Control Test Tie
Winner: Tie
Quick Donate Based on HPC
Test Control
Vote: Control Test Tie
Winner: Tie
EOY Lightbox
Test Control
Vote: Control Test Tie
Winner: Tie
$10 Ask to Lapsed and Deeply Lapsed Donors
Test Control
Vote: Control Test Tie
Winner: Test
“Read More” Button
Test Control
Vote: Control Test Tie
Winner: Tie
Nudge to Prospects + HPC<$125 Donors
Test
Control
Vote: Control Test Tie
Winner: Test
Video on Donation Form
Test Control
Vote: Control Test Tie
Winner: Tie
Reverse Ask String
Test Control
Vote: Control Test Tie
Winner: Test
Button Colour
Test Control
Vote: Control Test Tie
Winner: Test
Email Recruitment Lightbox
Version 2 Version 1
Vote: Version 1 Version 2 Version 3
Winner: Version 2
Version 3
Donate vs. Membership Lightbox at EOY
Test Control
Vote: Control Test Tie
Winner: Test
Donate vs. Membership Lightbox at EOY
Test Control
Vote: Control Test Tie
Winner: Tie
Streamlining Donation Form
Test Control
Vote: Control Test Tie
Winner: Control
OMG, Another Nudge Test
Test Control
Vote: Control Test Tie
Winner: Test
Recommendations for Nudge Tests
• Tested variations of nudge tests to 10 organizations.
• 50% saw improvement in at least one key metric. No organization saw nudges depress response or revenue.
• Recommended language: • “Most people are donating at least $X right
now.”
Upper Case vs. Sentence Case Lightbox
Test Control
Vote: Control Test Tie
Winner: Test
Multistep vs. Single Column Form
Test Control
Vote: Control Test Tie
Winner: Tie
Another PayPal Payment Option
Test Control
Vote: Control Test Tie
Winner: Test
Ask String vs. Open Field
Test
Control
Vote: Control Test Tie
Winner: Test
Right Call-Out vs. Centered
Test Control
Vote: Control Test Tie
Winner: Test
Review vs. Donate Button
Test Control
Vote: Control Test Tie
Winner: Tie
Paragraph Spacing
Test Control
Vote: Control Test Tie
Winner: Control
One Column vs. Two Column
Test Control
Vote: Control Test Tie
Winner: Tie
PETA Tests
$5 Ask to Non-Donors
Test
Control
Vote: Control Test Tie
Winner: Test
$5 Ask vs $10 Ask to Non-Donors
Test Control
Vote: Control Test Tie
Winner: Control
Copy Length on Donation Form
Test Control
Vote: Control Test Tie
Winner: Test
Single Image vs. Multi-Image
Test Control
Vote: Control Test Tie
Winner: Test
Larger Link Size
Test Control
Vote: Control Test Tie
Winner: Test
Animal Specific Donation Forms
Test
Control
Vote: Control Test Tie
Winner: Test
Renewal Language to Current + Lapsed Donors
Test Control
Vote: Control Test Tie
Winner: Test
Membership Buttons on Homepage
Test Control
Vote: Control Test Tie
Winner: Test
Tools for Running a Strong Testing Program
✳ The purpose of hypotheses is to make sure we're thinking critically about what we want to test and why and to make sure we learn from the test.
✳ To help create a hypothesis, fill in the blanks:
Changing (control) to (variation) will improve (test objective) because (reason you think the change will help) .
✳ If you craft a good hypothesis, you will learn something from every test, regardless of the outcome.
Crafting the hypothesis
✳ Before running a test, figure out the sample size needed for the test.
✳ To do that, you’ll need to understand how much of a change you’ll be able to detect.
✳ If you’re evaluating specific audiences or pages (e.g. mobile traffic), do this for each page or segment that you will evaluate
Determine Sample Size needed
Baseline Conversion Rate - the current average for the metric you are trying to improve (e.g. response rate, action rate)
Minimum Detectable Effect (MDE) - the amount of change in the conversion rate that you want to be able to detect. We recommend 25% or lower.
Elements of the sample size calculator
Minimum Audience Size - the total number of people you need to include in the experiment, per variation.
Sample Testing Roadmap
Sample Test Plan
Sample Results Tracker
All In One Place