going beyond ab testing - unbounce · going beyond ab testing . how to connect customers to...
TRANSCRIPT
![Page 1: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/1.jpg)
Going Beyond AB Testing
![Page 2: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/2.jpg)
How to Connect Customers to
Experiences
1. Decision Rules as a Framework
2. Picking Rules -> Optimization
3. Multi-Armed Bandit
twitter: @mgershoff
Blog: www.conductrics.com/blog
Overview
![Page 3: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/3.jpg)
twitter: @mgershoff
Blog: www.conductrics.com/blog
Bandity Tools
![Page 4: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/4.jpg)
twitter: @mgershoff
Blog: www.conductrics.com/blog
Bandity Tools
![Page 5: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/5.jpg)
twitter: @mgershoff
Blog: www.conductrics.com/blog
Bandity Tools
![Page 6: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/6.jpg)
twitter: @mgershoff
Blog: www.conductrics.com/blog
Bandity Tools
![Page 7: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/7.jpg)
Decision Rules
![Page 8: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/8.jpg)
Decision Rule
IF [THIS]
![Page 9: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/9.jpg)
Decision Rule
IF [THIS]
THEN [THAT]
![Page 10: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/10.jpg)
A Logic that links
Observations to Actions
Why Decision Rules?
![Page 11: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/11.jpg)
Decision Rule: General Form
Conductrics twitter: @mgershoff Blog: www.conductrics.com/blog
If [Observed Data]
![Page 12: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/12.jpg)
Conductrics twitter: @mgershoff Blog: www.conductrics.com/blog
If [Observed Data]
Then [Causal Action]
Decision Rule: General Form
![Page 13: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/13.jpg)
Conductrics twitter: @mgershoff Blog: www.conductrics.com/blog
If [Anyone]
Decision Rule: Simple Test
![Page 14: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/14.jpg)
Conductrics twitter: @mgershoff Blog: www.conductrics.com/blog
If [Anyone]
Then [A or B]
Decision Rule: Simple Test
![Page 15: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/15.jpg)
Conductrics twitter: @mgershoff Blog: www.conductrics.com/blog
If [Unbounce User]
Decision Rule: Targeted Test
![Page 16: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/16.jpg)
Conductrics twitter: @mgershoff Blog: www.conductrics.com/blog
If [Unbounce User]
Then [ or ]
Decision Rule: Targeted Test
![Page 17: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/17.jpg)
Conductrics twitter: @mgershoff Blog: www.conductrics.com/blog
Two Types of Data
• Day Part
• Weekend/Weekday
• Mobile/DeskTop
• Browser Type
• User Age
• Geo/Census
• Weather
• Tenure/RFM Score
Observed Data
• The Button – Ha!
• Price
• Sales Offers
• Shipping Type
• Layout/UX
• Which Products
• Suggested
Quantities
Causal Action
![Page 18: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/18.jpg)
Learning Efficiently
+
Apply ‘Best’ Rules
Optimization is Balance
![Page 19: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/19.jpg)
How do we pick the rules?
But …
If THIS Then THAT
If THIS Then THAT
If THIS Then THAT If THIS Then THAT
If THIS Then THAT
If THIS Then THAT
If THIS Then THAT
If THIS Then THAT
If THIS Then THAT If THIS Then THAT
If THIS Then THAT
If THIS Then THAT
![Page 20: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/20.jpg)
Learn From Domain Experts …
* Image Source http://en.wikipedia.org/wiki/Anna_Wintour
![Page 21: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/21.jpg)
…and from Experiments
* Image Source http://en.wikipedia.org/wiki/Long_Ashton_Research_Station
![Page 23: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/23.jpg)
Which is Better to Experiment?
Vs
![Page 24: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/24.jpg)
"DevonLarratt-TarasIvakin" by Above99 - Own work. Licensed under CC BY 3.0 via Wikimedia Commons - http://commons.wikimedia.org/wiki/File:DevonLarratt-TarasIvakin.jpg#/media/File:DevonLarratt-TarasIvakin.jpg
Source: http://www.startrek.com/article/ten-for-ward-10-favorite-kirk-fight-scenes
![Page 25: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/25.jpg)
Experimentation Review
Conductrics twitter: @mgershoff Blog: www.conductrics.com/blog
The A/B
TEST
![Page 26: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/26.jpg)
Anyone
A
B
Simple AB: Data Collection
![Page 27: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/27.jpg)
Anyone
A
B
Simple AB: Evaluation Phase
![Page 28: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/28.jpg)
1) Calculate Average/Means of A & B
2) Calculate Spread/Variance in A & B
+
Mean(B) – Mean(A) t=
4) Use evidence to make a selection
3)
28 [email protected]; www.conductrics.com
(Or favorite Test)
Simple AB: Evaluation Phase
B A
![Page 29: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/29.jpg)
29
Simple AB: Apply Rule
If [Anyone]
Offer B Then
![Page 32: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/32.jpg)
A measure of Uncertainty
Mean - 2*Std +2*Std
32 [email protected]; www.conductrics.com
Recall Confidence Interval
![Page 33: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/33.jpg)
Mean +Bonus
Score each option using the upper
portion of the interval as a Bonus
33 [email protected]; www.conductrics.com
Upper Confidence as Bonus
![Page 34: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/34.jpg)
0
Conversion Rate
5% 10%
A
B
C
1) Use upper portion of CI as ‘Bonus’
Select A
2) Select based on highest ‘Score’
34 [email protected]; www.conductrics.com
Upper Confidence as Bonus
![Page 35: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/35.jpg)
0 5% 10%
A
B
C
1) Selecting Action ‘A’ reduces uncertainty
bonus (because more data)
Select C
2) Action ‘C’ now has highest score
35 [email protected]; www.conductrics.com
Upper Confidence as Bonus
Conversion Rate
![Page 36: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/36.jpg)
When MAB?
![Page 37: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/37.jpg)
• Headlines
• Flash Sales
Adaptive/Bandit AB Test
37 [email protected]; www.conductrics.com
Is Learning Perishable?
Yes! Not Really
• Site Redesign
![Page 38: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/38.jpg)
• Many Options
• Many Tests
• Personalized
• Top ‘k of M
38 [email protected]; www.conductrics.com
Complex Problem?
Yes! Not Really
• Few Options
• Few Tests
• No/Simple Targeting
• Single Option
![Page 39: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/39.jpg)
Pfizer Use Case The ‘IF‘ Part
• Three Different User Segments
• Wyeth Pharma Search Segment
• King Pharma Search Segment
• All others Segment
![Page 40: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/40.jpg)
Conductrics Confidential
Pfizer Use Case
The ‘Then’ Part
• Best Content Topic for each Segment
• Best Navigation UI
The ‘IF‘ Part
• Three Different User Segments
• Wyeth Pharam Search Segment
• King Pharma Search Segment
• All others Segment
![Page 41: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/41.jpg)
Conductrics Confidential
Pfizer Use Case
The ‘Then’ Part
• Best Content Topic for each Segment
• Best Navigation UI
The ‘IF‘ Part
• Three Different User Segments
• Wyeth Pharam Search Segment
• King Pharma Search Segment
• All others Segment
The ‘How’ Part
• Test Bandit Approach
![Page 42: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/42.jpg)
Conductrics Confidential
Pfizer Use Case
![Page 43: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/43.jpg)
Conductrics Confidential
Topics
1)Popular: Default 2) Wyeth Related 3) King Related
![Page 44: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/44.jpg)
Conductrics Confidential
Investor Navigation
1) NavA: Default
2) NavB: Box
3) NavC: Tab
![Page 45: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/45.jpg)
Create Agent
![Page 46: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/46.jpg)
Create Segments
Wyeth Interested
King Interested
![Page 47: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/47.jpg)
Top Option 8 times more often than the default
47 [email protected]; www.conductrics.com
Results: Wyeth Segment
![Page 48: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/48.jpg)
0%
2%
4%
6%
8%
10%
12%
Tab Box Tab Box Tab Box Tab Box
Wyeth King
Bandit Random
Bandit Random
48 [email protected]; www.conductrics.com
Confidence Intervals MAB & AB C
on
vers
ion
Rate
![Page 49: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/49.jpg)
0%
2%
4%
6%
8%
10%
12%
Tab Box Tab Box Tab Box Tab Box
Wyeth King
Bandit Random
Bandit Random
49 [email protected]; www.conductrics.com
Co
nve
rsio
n R
ate
High Value Option-> Smaller CIs
![Page 50: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/50.jpg)
0%
2%
4%
6%
8%
10%
12%
Tab Box Tab Box Tab Box Tab Box
Wyeth King
Bandit Random
Bandit Random
50 [email protected]; www.conductrics.com
Low Value -> Higher Uncertainty C
on
vers
ion
Rate
![Page 51: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/51.jpg)
0%
2%
4%
6%
8%
10%
12%
Tab Box Tab Box Tab Box Tab Box
Wyeth King
Bandit Random
Bandit Random
51 [email protected]; www.conductrics.com
Low Value -> Higher Uncertainty C
on
vers
ion
Rate
![Page 52: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/52.jpg)
All Others
Wyeth Searchers
King Searchers
Content Options
Navigation Options
• For the segments - the content selection had a greater impact
• For ‘all others’ – the Navigation selection had the most impact
Conductrics Confidential
Nav or Topics: Impact
![Page 53: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/53.jpg)
1. Automation
2. Changing World/Perishable
3. Complexity / Scale
53 [email protected]; www.conductrics.com
When to Use Bandits
![Page 54: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/54.jpg)
See Russell & Norvig ‘s Artificial Intelligence: A Modern Approach
Optimization
Just 5 Parts
![Page 55: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/55.jpg)
Goal
Measureable Goals
![Page 56: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/56.jpg)
Ability to ‘Sense’ the Environment
![Page 57: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/57.jpg)
Ability to Act/Execute
A B
![Page 58: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/58.jpg)
Observe Outcomes
![Page 59: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/59.jpg)
Play ‘Better’ Options More Often
![Page 60: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/60.jpg)
If Then Receive
Learn Decision Logic
![Page 62: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/62.jpg)
![Page 63: Going Beyond AB Testing - Unbounce · Going Beyond AB Testing . How to Connect Customers to Experiences 1. Decision Rules as a Framework 2. Picking Rules -> Optimization 3. Multi-Armed](https://reader036.vdocument.in/reader036/viewer/2022070810/5f08f5587e708231d4248cc4/html5/thumbnails/63.jpg)
Image source: http://www.washingtonpost.com/blogs/london-2012-olympics/wp/tag/soccer/ (Andrew Medichini — Associated Press)