[cxl live 16] what to test next - prioritizing your tests by pauline marol
TRANSCRIPT
Whattotestnext?PrioritizingyourtestsPaulineMarol
SeniorProductManager
JosephineFoucher
OptimizationProductManager
ConversionXLLive2016
Agenda • Wheretostart
• DefiningPrioritizationRules
• TheHotwireframework
• MonitoringtheHealthofyourOptimizationProgram
2
Wheretostart?
Whyyouneedaprioritizationprocess
• Managing stakeholders• Bringtransparencytooptimization• Buildtrustfromthecompany• Empoweremployees
4
Makeyourlifeeasier
Knowyourdestination
• Testingprofile• Productstrategy• Optimizationgoals
5
Knowwhatyouwant
Knowyourself
• Getyouranalyticsright• Listentoyourcustomers• Lookatyourcompetitors
6
Identifyyouropportunities
Structureyourideas
• Translateideasintohypotheses• Centralize• Organizeandtag
7
Buildyourbacklogofhypotheses
Definingprioritizationrules
BryanEisenbergrules MonetatemodelPIEframework Pointsmodel
Therearemanyprioritizationframeworks
ItallcomesdowntoValueandEffort
Ifyougetvalueandeffortright,therestiseasy
10
High
Low
High Low
Value
Effort
ValidateHypothesis
Doit!
Forgetit Ifyouhavetheresources
Reach Lift Strategicfit
IdentifythedriversofValueandEffort
11
Value Effort
Howmanypeoplewillbeimpacted?
Reasonstoexpectastrongreaction
Alignedwithstrategy
Alltheframeworksbreakdownvalueand/oreffortintosub-elements,herearethemostcommon.
Creative Development Coordination
InthePIEframework
12
Value
Reach Lift Strategicfit
Effort
Creative Development Coordination
Potential Importance Ease
BreakingdownValueintoPotentialandImportance
IntheMonetatemodel
13
Value
Reach Lift Strategicfit
Effort
Creative Development Coordination
Potential CreativeLOE TechnicalLOE
BreakingdownEffortintoCreativeandDevelopment
IntheBryanEisenbergrules
14
Value
Reach Lift Strategicfit
Effort
Creative Development Coordination
Impact Resources
Time
BreakingdownEffortintoResourcesandTime
Definebinaryvaluerulesthat:• Areobjective (yes/no)• Discriminate ideas• Leverageyourknowledge
Takingittothenextlevel:ThePointsmodel
15
Value
Reach Lift Strategicfit
Effort
Creative Development Coordination
Defineeffortrulesthat:• Representyourbottlenecks• Arealigned withexistingsizing
Breakdowneachsub-element intospecificrules,toremovepersonalbias
HotwireCarFramework
1. Primarymetricisconversion
2. Addingorremovinginfo
3. Research/analytics4. Benchmarking
InthePointsmodel
17
Value
1. Above thefold2. Allcustomers
willseeit
1. $$product2. Cancellation3. Choice4. Multiplethemes
Effort
Creative Development Coordination
UX/UIpointsCopywritingneeds
Engineeringpoints
Reach Lift Strategicfit
HotwireCarprioritization
If weaddaFreecancellationflagonthecheckoutpageThen ourconversionratewillincreaseBecause customerswillfeelmoreconfidentaboutreservingacarknowingthattheycanchangetheirmind
18
HotwireCarprioritization
19
Fold
Test 1
HotwireCarprioritizationFree cancellationflag
20
Fold
100%
Test 1 0
Fold
100%
Mainmetric
Test 1 0 1
Fold
100%
Mainmetric
Adding
or
remov
ing
Test 1 0 1 1
Fold
100%
Mainmetric
Adding
or
remov
ing
Analytics
or
research
Test 1 0 1 1 0
Fold
100%
Mainmetric
Adding
or
remov
ing
Analytics
or
research
Benchm
arkin
g
Test 1 0 1 1 0 1
Fold
100%
Mainmetric
Adding
or
remov
ing
Analytics
or
research
Benchm
arkin
g
$$produ
ct
Test 1 0 1 1 0 1 0
Fold
100%
Mainmetric
Adding
or
remov
ing
Analytics
or
research
Benchm
arkin
g
$$produ
ct
Cancellatio
n
Test 1 0 1 1 0 1 0 0
Fold
100%
Mainmetric
Adding
or
remov
ing
Analytics
or
research
Benchm
arkin
g
$$produ
ct
Cancellatio
n
Choice
Test 1 0 1 1 0 1 0 0 0
Fold
100%
Mainmetric
Adding
or
remov
ing
Analytics
or
research
Benchm
arkin
g
$$produ
ct
Cancellatio
n
Choice
2them
es
Test 1 0 1 1 0 1 0 0 0 1
Fold
100%
Mainmetric
Adding
or
remov
ing
Analytics
or
research
Benchm
arkin
g
$$produ
ct
Cancellatio
n
Choice
2them
es
Total
Test 1 0 1 1 0 1 0 0 0 1 5
Fold
100%
Mainmetric
Adding
or
remov
ing
Analytics
or
research
Benchm
arkin
g
$$produ
ct
Cancellatio
n
Choice
2them
es
Total UX Dev
Test 1 0 1 1 0 1 0 0 0 1 5 S S
HotwireCarprioritization
21
Fold
100%
Mainmetric
Adding
or
remov
ing
Analytics
or
research
Benchm
arkin
g
$$produ
ct
Cancellatio
n
Choice
2them
es
Total UX Dev
Test1 1 0 1 1 0 1 0 0 0 1 5 S S
Test2 1 1 1 1 1 0 1 0 0 0 6 M L
Test150 0 0 1 1 1 1 1 1 1 1 8 L S
MonitoringtheHealthofyourOptimizationProgram
UseKPIstomonitorandcommunicateperformance
Choosemetricsbasedonyourriskprofileandtestingstrategy.
Werecommend:• Velocity• WinRate• Conversionlift
IfyourKPIsgodown,revisityourrules.
Benchmarkyourprioritizationrulestostayrelevant
Validatethecorrelationbetweenscoreandreality• Valuescorevs.Winrate
Ifyourrulesefficiencyflattens,revisityourrules.
0%
25%
50%
75%
100%
0 2 4 6 8 10
WinRate
ValueScore
BenchmarkingValuePoints
ShareandEducate
• Shareyourprioritization rulesandbetransparent• Getfeedbacktomaketherulesbetter• Educateandscaleoutsideyouroptimizationteam
Whenyouaggregatefeedback,revisityourrules.
Buildknowledge
• Aggregateresults• TranslateCRliftintolearning andcustomer
insights• Adjustyourtestingprofile,youroptimization
goals
Asyouaccumulatelearning,revisityourrules.
Yourturn!
27
1 2 3
Doyourresearch Buildprioritizationrules Keepimprovingthem