multi-armed bandits: intro, examples and tricks
TRANSCRIPT
Multi-Armed Bandits:Intro, examples and tricks
Dr Ilias Flaounas Senior Data Scientist at Atlassian
Data Science Sydney meetup 22 March 2016
Motivation
Increase awareness of some very useful but less known techniques
Demo some current work at Atlassian
Connect it with some research from my past
Hopefully, there will be something useful for everybody — apologies for the few equations and loose notation
http://www.nancydixonblog.com/2012/05/-why-knowledge-management-didnt-save-general-motors-addressing-complex-issues-by-convening-conversat.html
( rA,1 )
( rC,2 )
rB,3
+ rA,4 + rA,5
+ rC,6
+ rA,7 / nA
+ rC,8
/ nB
/ nc
µA =
µB =
µC =
1. e-greedy: the best arm is selected for a proportion of 1-e of the trials and a random arm in e trials.
2. e-greedy with variable e
3. Pure exploration first, then pure exploitation.
4. …
5. Thompson sampling (Draw from the estimated beta-distrom
6. Upper Confidence Bound (UCB)
Many solutions…
Disadvantages
Reaching significance for non-winning arms takes longer
Unclear stopping criteria
Hard to order non-winning arms and assess reliably their impact
Advantages
Reaching significance for the winning arm is faster
Best arm can change over time
There are no false positives in the long term
Optimizely recently introduced MAB rebranded as: “Traffic auto-allocation”
Let’s add some context
What happens if we want to assess 100 variations?
How about 1,000 or 10,000 variations?
Contextual Multi-Armed Bandits
rA, t = f(xA,1, xA,2, xA,3…)A -> {xA,1, xA,2, xA,3…}
rB,t = f(xB,1, xB,2, xB,3…)
rC,t = f(xC,1, xC,2, xC,3…)
Experiment parameters, e.g., price, #users, product, bundles, colour of UI elements…
B -> {xB,1, xB,2, xB,3…}
C -> {xC,1, xC,2, xC,3…}
We introduce a notion of proximity or similarity
between arms
A -> {xA,1, xA,2, xA,3…}B -> {xB,1, xB,2, xB,3…}
Contextual Multi-Armed Bandits
LinUCB
L. Li, W. Chu, J. Langford, R. E. Schapire, “A Contextual-Bandit Approach to Personalized News Article Recommendation”, WWW, 2010.
The UCB is some expectation plus some confidence level:
µ↵(t) + �↵(t)
We assume there is some unknown vector θ∗, the same for each arm, for which:
E[ra,t|xa,t] = x
Ta,t✓
⇤
✓̂t := C�1t XT
t yt
Xt := {xa(1),1, xa(2),2, . . . , xa(t),t}T
yt := {ra(1),1, ra(2),2, . . . , ra(t),t}T
Ct := XTt Xt
Using least squares:
µ̂a(t) := x
Ta,t✓̂t
E[ra,t|xa,t] = x
Ta,t✓
⇤ µ↵(t) + �↵(t)
µ̂a := x
Ta,tC
�1t X
Tt yt
The upper confidence bound is some expectation plus some confidence level:
µ↵(t) + �↵(t)
�̂(t) :=qx
Ta,tC
�1t xa,tµ̂a := x
Ta,tC
�1t X
Tt yt
L. Li, W. Chu, J. Langford, R. E. Schapire, A Contextual-Bandit Approach to Personalized News Article Recommendation, WWW, 2010.
Product onboarding…
Which arm would you pull?
• How can we locate the city of Bristol from tweets?
• 10K candidate locations organised in a 100x100 grid
• At every step we get tweets from one location and count mentions of “Bristol”
• Challenge: find the target in sub-linear time complexity!
Linear methods fail on this problem.
How can we go non-linear?
John-Shawe Taylor & Nello Cristianini, “Kernel Methods for Pattern Analysis”, Cambridge University press, 2004.
The Kernel trick! —no, it’s not just for SVMs
µ̂a(t) := x
Ta,t✓̂t µ̂
a
(t) = kTx,t
K�1t
yt
�̂a
(t) =q
tkTx,t
K�2t
kx,t
�̂(t) :=qx
Ta,tC
�1t xa,t
Ct := XTt Xt Kt = XtX
Tt
LinUCB:
M. Valko, N. Korda, R. Munos, I. Flaounas, N. Cristianini, “Finite-Time Analysis of Kernelised Contextual Bandits”, UAI, 2013.
KernelUCB:
• The last few steps of the algorithm before it locates Bristol.
• KernelUCB with RBF kernel converges after ~300 iterations (instead of >>10K).
Target is the red dot. We locate it using KernelUCB with RBF kernel.
KernelUCB code: http://www.complacs.org/pmwiki.php/CompLACS/KernelUCB
What if we have a high-dimensional space?
Hashing trick
Implementation in Vowpal Wabbit, by J. Langford, et al.
ReferencesM. Valko, N. Korda, R. Munos, I. Flaounas, N. Cristianini, “Finite-Time Analysis of Kernelised Contextual Bandits”, UAI, 2013.
L. Li, W. Chu, J. Langford, R. E. Schapire, “A Contextual-Bandit Approach to Personalized News Article Recommendation”, WWW, 2010.
John-Shawe Taylor & Nello Cristianini, “Kernel Methods for Pattern Analysis”, Cambridge University press, 2004.
Implementation of KernelUCB in Complacs toolkit:http://www.complacs.org/pmwiki.php/CompLACS/KernelUCB
https://en.wikipedia.org/wiki/Multi-armed_bandit
https://github.com/JohnLangford/vowpal_wabbit/wiki/Contextual-Bandit-Example
Thank you - We are hiring!
Dr Ilias Flaounas Senior Data Scientist <first>.<last>@atlassian.com