don’t leave it to the perceiver - kobie marketing...- unique member transaction history for earn...
TRANSCRIPT
1
DISPELLING MISPERCEPTIONS ABOUT YOUR PROGRAM’S ROI
DON’T LEAVE IT TO THE PERCEIVER
Steve Flaming, VP Analytics & ReportingHugo St. John III, Manager, Financial Analysis
YOUR ROI STORY SHOULD NOT BE AN AFTERTHOUGHT.
3
How many of you
thought about
measurement when
you implemented
your program or did
a redesign?
80%
50% 50%
30%
THE GOLD STANDARD.
Primary Parts:
1. Pre-period of time prior to launch
2. Post or measurement period
3. Representative test group
4. Matched control group
5
UP TO SNOW GOOD REWARDS – ROLLING OUT A PROGRAM.
OTHER GOOD MARKETSPortland vs SeattleCinci vs ColumbusCleveland vs DetroitNashville vs Louisville
AREAS TO AVOIDHighly volatile economies & weather patterns (e.g., Las Vegas, SF/LA, most of Florida, Chicago after October, etc.)
TEST (PILOT) MARKET – Houston
CONTROL MARKET – Dallas
FULL ADVERTISED ROLLOT FOR 12 MONTHS (MIN. OF 6)
ANALYTICS TO CONTROL FOR ANY REMAINING BIASTotal populations: 2.27M vs 1.30MPercent of families with children: 53.9% vs 55.2%Median household income: $49,399 vs $47,285Percent with a bachelor’s degree: 31.7% vs. 31.6%Unemployment rate: 6.6% vs 5.9%
Aggregate the data over the next 12 months and calculate program uplift feeling confident that we have a true test / control experiment here
CALCULATE THE IMPACT OF UP TO SNOW GOOD REWARDS.
7
EVALUATING LOYALTY PROGRAMS.
THE DIRTY UNDERBELLY OF MEASUREMENT.
Self-selection bias…and why it is all your fault!
Auto-enrollment tracks dirt in.
Waiting to measure…a dangerous game to play.
SELF-SELECTION BIAS IS NOT THE END OF THE WORLD.
“Self-selection bias points to errors due to systematic differences in relevant characteristics between those who are included in study and those who are not; specifically the personal choices of individuals.”
MACHINE LEARNING HELPS TO CONTROL FOR IT.
Non Member Member Non Member Member
Spen
d ($)
POST-MODEL COMPARISONPRE-MODEL COMPARISON
Tier 1 Tier 2 Tier 3 Tier 4
Non Member Member
Tier 1 Tier 2 Tier 3 Tier 4
Non Member Member
House
holds
(%)
Non Member Member Non Member Member
Tenu
re (Y
rs)
11
THE DIRTY SECRET ABOUT AUTO-ENROLLING MEMBERS.
Introduces bias over time via the channel of purchase:
1. Reduces size and similarity of your non-member or control group
2. Pollutes the test group
12
MEASURE EARLY; MEASURE OFTEN.
Time is not on your side when it comes to measuring the effect of your loyalty program.
The longer you wait, the harder it gets to measure loyalty—it’s not impossible, it’s simply case-dependent.
So, be sure to plan for the future!
LET’S CONSIDER OTHER ALTERNATIVES.
RETROFITTING THE GOLD STANDARD.MAKING IT RAIN
$50
$60
$70
$80
$90
$100
$110
$120
-5 -4 -3 -2 -1 0 1 2 3 4 5
Non Rewards Rewards
Requirements• Member data
• Start date/1st transaction/Conversion• Non Member data
• Start date/1st transaction• Pre data
• What did everyone do before the program?
• Post data• What did everyone do after the
program?
Pre-period? We don’t need no stinking pre-period.
Requirement and Considerations:
Key: Take advantage of the buildup to build a pseudo pre-period
Data available• Member data (post program only)
• Start date/1st transaction/Conversion• Non-member data (post program only)
• Start date/1st transaction
$50
$60
$70
$80
$90
$100
$110
$120
0 1 2 3 4 5
Non Rewards Rewards
OPTION 1 –NO DATA PRIOR TO PROGRAM START.
OPTION 2 – NOT TRACKING NON-LOYALTY DATA.Ugh… I hate time series models. They are the worst.
Requirements and considerations.
Key: Utilizing a widely-accepted approach called time series analysis which projects out an expected pre-period.
Data available• Only need total sales
$50
$60
$70
$80
$90
$100
$110
$120
-5 -4 -3 -2 -1 0 1 2 3 4 5
Option 2 Pre/Post Only
Rewards
Uplift
Expected sales if no loyalty program
OPTION 3 -POST-PERIOD DATA ONLY FOR LOYALTY MEMBERS.Hold on, I have a Ouija Board in my closet somewhere.
Requirements and considerations:
Key: Very little you can do to analytically assign a value to incremental sales at this point.
Alternate KPIs become the measure of success:• NPS• OSAT• Emotional Loyalty• Active Customer• Churn Reduction• Lifetime Value
50
60
70
80
90
100
110
120
-5 -4 -3 -2 -1 0 1 2 3 4 5
Rewards
UNIVERSAL CONTROL
GROUPS
YOU CAN’T MEASURE YOUR PROGRAM… SO, NOW WHAT?
CAMPAIGN CONTROL
GROUPS
NPS/OSAT
EMOTIONAL LOYALTY
SCORING
GETTING FROM GOOD TO GREAT.
PROGRAM REDESIGN.
During program redesign, teams should use what they already know to forecast the better future they want to step into.
WHAT YOU NEED BEFORE YOU START.
Complete data from the current program, which should include:- Unique member transaction history for earn and burn- Campaign-specific data as they relate to changes in member behavior- All costs incurred that would not exist, save for having the program (e.g.,
rewards costs, vendor management fees, etc.) in order to focus on profitability
These data and results plug directly into the financial loyalty model!
INCORPORATE UNCERTAINTY.
When you redesign your program, you need to model and simulate:- Changes to expected enrollment- Shifts in tier migration- Increases in transaction behavior- New redemption patterns (e.g., due to higher earning potential
across members, new redemption options, etc.)
WHAT THIS MIGHT LOOK LIKE.
INCREMENTAL SALES
PROGRAM YEAR
$7.2 million increase in cumulative incremental sales by year 3 of program
$5.7 million decrease in cumulative incremental sales by year 3 of program
COST OF BENEFITS
PROGRAM YEAR
$6.0 million increase in total cost of benefits by year 3 of program
$5.2 million decrease in total cost of benefits by year 3 of program
PROGRAM YEAR
THE OFT NEGLECTED PART OF ROI.
We often focus on the numerator of ROI – incremental sales.
The denominator is important and can be used as evidence of program value.
Cost savings can include things like:- Increased app usage- Reduction in call center hours- Paperless communications- Auto-billing
These all represent changes to the program value proposition that will drive higher ROI—you should model and simulate these changes, too!
AIM FOR AN ELEGANT SOLUTION.
MEASUREMENTREVISITED.
Any time a loyalty program’s value proposition changes significantly, teams have an opportunity to use the Gold Standard for measuring the new program’s performance compared to the old program.
Yes, you can go home again.
27
CLOSING THOUGHTS.
28
PROGRAM MEASUREMENT CHECKLIST.
• Can you build a pre-period?• Representative non-member group?• Machine learning to lessen self-selection bias?• Long-term UCG?• Campaign-level control groups?• Non-market valuation (e.g., OSAT/NPS/ELS)?
29
PROACTIVELY USE YOUR DATA.
• Meticulously “test and learn” potential program changes • Focus on how key member segments react • Remember program changes include cost-savings• Model out how a full program redesign, based on these learnings, affects the
program’s ROI• Design the new rollout with the Gold Standard in mind
30