[cxl live 16] beyond test-by-test results: cro metrics for performance & insight by claire vo
TRANSCRIPT
![Page 1: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/1.jpg)
EXPERIMENT ENGINEEEClaire Vo @clairevo
CRO Metrics for Performance & Insight
Beyond test-by-test results: measuring testing
program effectiveness
![Page 2: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/2.jpg)
Things I know to be true:
EXPERIMENT ENGINEEEClaire Vo @clairevo
![Page 3: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/3.jpg)
Most CRO teams focus onand measure the outputs of testing.
EXPERIMENT ENGINEEEClaire Vo @clairevo
(wins make us feel warm & fuzzy)
![Page 4: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/4.jpg)
Claire Vo @clairevo
![Page 5: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/5.jpg)
Claire Vo @clairevo
![Page 6: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/6.jpg)
Claire Vo @clairevo
Justin Rondeau Busted! DigitalMarketer Calls Bull$hit on 4 Conversion Rate Optimization (CRO) Case Studieshttp://www.digitalmarketer.com/conversion-rate-optimization-case-studies/
![Page 7: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/7.jpg)
But more than anything, the success of a testing program
is driven by the inputs.
EXPERIMENT ENGINEEEClaire Vo @clairevo
![Page 8: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/8.jpg)
How to do testing, better:
EXPERIMENT ENGINEEEClaire Vo @clairevo
1. Increase Tests Run (Quantity)
2. Increase Tests Won (Quality)
3. Profit
![Page 9: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/9.jpg)
Claire Vo @clairevo
![Page 10: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/10.jpg)
How to do testing, better—really:
EXPERIMENT ENGINEEEClaire Vo @clairevo
1. Set Goals
2. Measure Performance Regularly
3. Adjust & Iterate
![Page 11: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/11.jpg)
We need a set of CRO performance metrics
to track and improve CRO quantity and quality.
EXPERIMENT ENGINEEEClaire Vo @clairevo
![Page 12: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/12.jpg)
Quantity
EXPERIMENT ENGINEEEClaire Vo @clairevo
![Page 13: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/13.jpg)
EXPERIMENT ENGINEEEClaire Vo @clairevo
“We run [X] tests per week.”
![Page 14: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/14.jpg)
EXPERIMENT ENGINEEEClaire Vo @clairevo
“We run [X] tests per week.”…usually.”
![Page 15: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/15.jpg)
Claire Vo @clairevo
Instead you should know:
What to know Metric to Measure
How many tests can I run Testing Capacity
How many am I running? Testing Velocity & Coverage
Am I getting any better? Trends Over Time
When I’m not running tests, why? Annotations
![Page 16: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/16.jpg)
Claire Vo @clairevo
Testing CapacityCRO PERFORMANCE METRICS
How many tests can you run per year
52
Test Duration (in weeks)x # of simultaneously
testable pages/funnels
![Page 17: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/17.jpg)
Claire Vo @clairevo
Testing VelocityCRO PERFORMANCE METRICS
How many experiments are run per [time period]
0
2.5
5
7.5
10
5/23/2015 6/20/2015 7/18/2015 8/15/2015 9/12/2015 10/10/2015 11/7/2015 12/5/2015
• Weekly for high traffic sites
• Monthly for low traffic sitesGoal 8 tests / week
![Page 18: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/18.jpg)
Claire Vo @clairevo
Testing VelocityCRO PERFORMANCE METRICS
How many experiments are run per [time period]
-0.4
2.2
4.8
7.4
10
5/23/2015 6/20/2015 7/18/2015 8/15/2015 9/12/2015 10/10/2015 11/7/2015 12/5/2015
• Weekly for high traffic sites
• Monthly for low traffic sitesGoal 8 / week
Tracking trend towards goal
BONUS
![Page 19: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/19.jpg)
EXPERIMENT ENGINEEEClaire Vo @clairevo
Make sure you’re not wasting traffic.
(use it or lose it.)
![Page 20: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/20.jpg)
Claire Vo @clairevo
Łukasz Twardowski A/B Testing and the Infinite Monkey Theoryhttp://www.slideshare.net/useitbetter/ab-testing-and-the-infinite-monkey-theory
![Page 21: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/21.jpg)
13%
87%
Claire Vo @clairevo
Testing CoverageCRO PERFORMANCE METRICS
What % of testable days are you running a test?
July
1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30 31
August
1 2 3
4 5 6 7 8 9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30 31
83%
![Page 22: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/22.jpg)
Claire Vo @clairevo
Time Since Last Zero Test Day
CRO PERFORMANCE METRICS
It has been
100DAYS SINCE WE HAD
NO TESTS LIVE
![Page 23: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/23.jpg)
Quality
EXPERIMENT ENGINEEEClaire Vo @clairevo
![Page 24: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/24.jpg)
EXPERIMENT ENGINEEEClaire Vo @clairevo
Stop evaluating the quality of your program on a test-by-test basis.
(it makes you look good, until you look bad)
![Page 25: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/25.jpg)
Claire Vo @clairevo
You should also track:
What to know Metric to Measure
Am I running effective tests? Win Rate, Lift Amount, Expected Value
Am I running tests effectively? ROI
Am I getting any better? Trends Over Time
![Page 26: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/26.jpg)
53%
29%
19%
Claire Vo @clairevo
Win RateCRO PERFORMANCE METRICS
What % of tests run win / lose / are inconclusive?
13%
![Page 27: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/27.jpg)
Claire Vo @clairevo
Rolling Win RateCRO PERFORMANCE METRICS
What % of tests run win / lose / are inconclusive over time?
0
4
8
12
16
April May June July August September October November
Goal 15% Win Rate
Tracking by Page
BONUS
Once you hit a steady testing velocity, ideally your win rate stays level or increases.
![Page 28: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/28.jpg)
Claire Vo @clairevo
Expected ValueCRO PERFORMANCE METRICS
$1 per roll of the die Every time you roll a 3 I pay you $5
Win Rate = 1/6 = 16% Value of Win = $5
Expected Value of Roll = 83 cents
DO NOT PLAY THIS GAME!
![Page 29: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/29.jpg)
Claire Vo @clairevo
Expected ValueCRO PERFORMANCE METRICS
On average, what is the expected value of any test run?
Win Rate x
Average Lift of Winning Test x
Revenue Value of Test
![Page 30: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/30.jpg)
Claire Vo @clairevo
Expected ValueCRO PERFORMANCE METRICS
$1500 every time you run a test Every time you win, you get 10% increase on your revenue, which is $1,000,000 and you win 10% of the time
Win Rate = 10% Value of Win = 10% x $1,000,000 = $100,000
Expected Value of Test = $10,000RUN THIS TEST FOR SURE!
![Page 31: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/31.jpg)
Claire Vo @clairevo
Annual Expected Value of ProgramCRO PERFORMANCE METRICS
Expected Value of Test x
Annual Testing Capacity
What is the estimated potential of my testing program?
![Page 32: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/32.jpg)
Claire Vo @clairevo
Annual Expected Value of ProgramCRO PERFORMANCE METRICS
What is the estimated potential of my testing program?
High Complexity Low Velocity
Low ComplexityHigh Velocity
Type of Tests Take longer, bigger changes Easier, smaller changes
Win Rate 20% 10%
Avg Lift 30% 10%
Expected Value of Test $30,000 $10,000
Tests / Year 25 100
Annual Expected Value of Testing Program $1.5M $1M
![Page 33: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/33.jpg)
EXPERIMENT ENGINEEEClaire Vo @clairevo
Yes, you should trend this over time.
(and set a goal!)
![Page 34: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/34.jpg)
Putting it all together
EXPERIMENT ENGINEEEClaire Vo @clairevo
![Page 35: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/35.jpg)
CRO Metrics for Performance:
EXPERIMENT ENGINEEEClaire Vo @clairevo
1. Track quantity by measuring velocity & coverage
2. Track quality by win rate, avg. lift, and expected value
3. Set goals & measure everything over time
![Page 36: [CXL Live 16] Beyond Test-by-Test Results: CRO Metrics for Performance & Insight by Claire Vo](https://reader034.vdocument.in/reader034/viewer/2022052116/587017221a28ab7f428b5b95/html5/thumbnails/36.jpg)
EXPERIMENT ENGINEEEClaire Vo @clairevo
Questions?
experimentengine.com/cxl-live-2016