Gamification of Pure Exploration for Linear Bandits

Remy Degenne * 1 Pierre Menard * 2 Xuedong Shang 3 Michal Valko 4

AbstractWe investigate an active pure-exploration setting,that includes best-arm identification, in the con-text of linear stochastic bandits. While asymptot-ically optimal algorithms exist for standard multi-arm bandits, the existence of such algorithms forthe best-arm identification in linear bandits hasbeen elusive despite several attempts to addressit. First, we provide a thorough comparison andnew insight over different notions of optimality inthe linear case, including G-optimality, transduc-tive optimality from optimal experimental designand asymptotic optimality. Second, we design thefirst asymptotically optimal algorithm for fixed-confidence pure exploration in linear bandits. Asa consequence, our algorithm naturally bypassesthe pitfall caused by a simple but difficult instance,that most prior algorithms had to be engineeredto deal with explicitly. Finally, we avoid the needto fully solve an optimal design problem by pro-viding an approach that entails an efficient imple-mentation.

1. IntroductionMulti-armed bandits (MAB) probe fundamental exploration-exploitation trade-offs in sequential decision learning. Westudy the pure exploration framework, from among differ-ent MAB models, which is subject to the maximization ofinformation gain after an exploration phase. We are par-ticularly interested in the case where noisy linear payoffsdepending on some regression parameter θ are assumed.Inspired by Degenne et al. (2019), we treat the problemas a two-player zero-sum game between the agent and thenature (in a sense described in Section 2), and we searchfor algorithms that are able to output a correct answer with

*Equal contribution 1INRIA - DIENS - PSL Research Uni-versity, Paris, France 2INRIA 3INRIA - Universite de Lille4DeepMind Paris. Correspondence to: Remy Degenne <remyde-genne@gmail.com>.

Proceedings of the 37 th International Conference on MachineLearning, Online, PMLR 119, 2020. Copyright 2020 by the au-thor(s).

high confidence to a given query using as few samples aspossible.

Since the early work of Robbins (1952), a great amount ofliterature explores MAB in their standard stochastic settingwith its numerous extensions and variants. Even-Dar et al.(2002) and Bubeck et al. (2009) are among the first to studythe pure exploration setting for stochastic bandits. A non-exhaustive list of pure exploration game includes best-armidentification (BAI), top-m identification (Kalyanakrishnan& Stone, 2010), threshold bandits (Locatelli et al., 2016),minimum threshold (Kaufmann et al., 2018), signed ban-dits (Menard, 2019), pure exploration combinatorial ban-dits (Chen et al., 2014), Monte-Carlo tree search (Teraokaet al., 2014), etc.

In this work, we consider a general pure-exploration set-ting (see Appendix D for details). Nevertheless, forthe sake of simplicity, in the main text we primarily fo-cus on BAI. For stochastic bandits, BAI has been stud-ied within two major theoretical frameworks. The firstone, fixed-budget BAI, aims at minimizing the probabil-ity of misidentifying the optimal arm within a given num-ber of pulls (Audibert & Bubeck, 2010). In this work,we consider another setting, fixed-confidence BAI, intro-duced by Even-dar et al. (2003). Its goal is to ensure thatthe algorithm returns a wrong arm with probability lessthan a given risk level, while using a small total numberof samples before making the decision. Existing fixed-confidence algorithms are either elimination-based suchas SuccessiveElimination (Karnin et al., 2013), rely onconfidence intervals such as UGapE (Gabillon et al., 2012),or follow plug-in estimates of the optimal pulling propor-tions by a lower bound such as Track-and-Stop (Garivier& Kaufmann, 2016). We pay particular attention to thefirst two since they have been extended to the linear setting,which is the focus of this paper. In particular, a naturalextension of pure exploration to linear bandits. Linear ban-dits were first investigated by Auer (2002) in the stochasticsetting for regret minimization and later considered for fixed-confidence BAI problems by Soare et al. (2014).

Linear bandits. We consider a finite-arm linear banditproblem, where the collection of armsA ⊂ Rd is given with|A| = A, and spans Rd. We assume that ∀a ∈ A, ‖a‖ ≤ L,where ‖a‖ denotes the Euclidean norm of the vector a. The

Gamification of Pure Exploration for Linear Bandits

learning protocol goes as follows: for each round 1 ≤ t ≤T, the agent chooses an arm at ∈ A and observes a noisysample

Yt = 〈θ, at〉+ ηt ,

where ηt ∼ N (0, σ2) is conditionally independent from thepast and θ is some unknown regression parameter. For thesake of simplicity, we use σ2 = 1 in the rest of this paper.

Pure exploration for linear bandits. We assume that θbelongs to some set M ⊂ Rd known to the agent. Foreach parameter a unique correct answer is given by thefunction i? :M→ I among the I = |I| possible ones (theextension of pure exploration to multiple correct answers isstudied by Degenne & Koolen 2019). Given a parameter θ,the agent then aims to find the correct answer i?(θ) byinteracting with the finite-armed linear bandit environmentparameterized by θ.

In particular, we detail the setting of BAI for which theobjective is to identify the arm with the largest mean.That is, the correct answer is given by i?(θ) = a?(θ) :=argmaxa∈A〈θ, a〉 for θ ∈ M = Rd and the set of pos-sible correct answers is I = A. We provide other pure-exploration examples in Appendix D.

Algorithm. Let Ft = σ(a1, Y1, . . . , at, Yt) be the infor-mation available to the agent after t round. A deterministicpure-exploration algorithm under the fixed-confidence set-ting is given by three components: (1) a sampling rule(at)t≥1, where at ∈ A is Ft−1-measurable, (2) a stop-ping rule τδ , a stopping time for the filtration (Ft)t≥1, and(3) a decision rule ı ∈ I which is Fτδ -measurable. Non-deterministic algorithms could also be considered by allow-ing the rules to depend on additional internal randomization.The algorithms we present are deterministic.

δ-correctness and fixed-confidence objective. An algo-rithm is δ-correct if it predicts the correct answer with prob-ability at least 1 − δ, precisely if Pθ

(ı 6= i?(θ)

)≤ δ and

τδ < +∞ almost surely for all θ ∈M. Our goal is to finda δ-correct algorithm that minimizes the sample complexity,that is, Eθ[τδ] the expected number of sample needed topredict an answer.

Pure exploration (in particular BAI) for linear bandits hasbeen previously studied by Soare et al. (2014); Tao et al.(2018); Xu et al. (2018); Zaki et al. (2019); Fiez et al. (2019);Kazerouni & Wein (2019). They all consider the fixed-confidence setting. To the best of our knowledge, only Hoff-man et al. (2014) study the problem with a fixed-budget.

Beside studying fixed-confidence sample complexity, Gariv-ier & Kaufmann (2016) and some subsequent works (Qinet al., 2017; Shang et al., 2020) investigate a general crite-rion of judging the optimality of a BAI sampling rule: Algo-

rithms that achieve the minimal sample complexity when δtends to zero are called asymptotically optimal. Menard(2019) and Degenne et al. (2019) further study the problemin a game theoretical point of view, and extend the asymp-totic optimality to the general pure exploration for structuredbandits. Note that a naive adaptation of the algorithm pro-posed by Degenne et al. (2019) may not work smoothly inour setting. In this paper we use some different confidenceintervals that benefit better from the linear structure.

Contributions. 1) We provide new insights on the com-plexity of linear pure exploration bandits. In particular, werelate the asymptotic complexity of the BAI problem andother measures of complexity inspired by optimal designtheory, which were used in prior work. 2) We develop asaddle-point approach to the lower bound optimization prob-lem, which also guides the design of our algorithms. In par-ticular we highlight a new insight on a convex formulationof that problem. It leads to an algorithm with a more directanalysis than previous lower-bound inspired methods. 3)We obtain two algorithms for linear pure exploration banditsin the fixed-confidence regime. Their sample complexity isasymptotically optimal and their empirical performance iscompetitive with the best existing algorithms.

2. Asymptotic OptimalityIn this section we extend the lower bound of Garivier &Kaufmann (2016), to hold for pure exploration in finite-armed linear bandit problems.

Alternative. For any answer i ∈ I we define the alter-native to i, denoted by ¬i the set of parameters where theanswer i is not correct, i.e. ¬i := θ ∈M : i 6= i?(θ) .

We also define, for any w ∈ (R+)A, the design matrix

Vw :=∑a∈A

waaaᵀ .

Further, we define ‖x‖V :=√xᵀV x for x ∈ Rd and a

symmetric positive matrix V ∈ Rd×d. Note that it is a normonly if V is positive definite. We also denote by ΣK theprobability simplex of dimension K − 1 for all K ≥ 2.

Lower bound. We have the following non-asymptoticlower bound, proved in Appendix C, on the sample com-plexity of any δ-correct algorithm. This bound was alreadyproved by Soare et al. (2014) for the BAI example.

Theorem 1. For all δ-correct algorithms, for all θ ∈M,

lim infδ→0

Eθ[τδ]log(1/δ)

≥ T ?(θ) ,

Gamification of Pure Exploration for Linear Bandits

where the characteristic time T ?(θ) is defined by

T ?(θ)−1 := maxw∈ΣA

infλ∈¬i?(θ)

1

2‖θ − λ‖2Vw .

In particular, we say that a δ-correct algorithm is asymptoti-cally optimal if for all θ ∈M,

lim supδ→0

Eθ[τδ]log(1/δ)

≤ T ?(θ) .

As noted in the seminal work of Chernoff (1959), the com-plexity T ?(θ)−1 is the value of a fictitious zero-sum gamebetween the agent choosing an optimal proportion of alloca-tion of pulls w and a second player, the nature, that tries tofool the agent by choosing the most confusing alternative λleading to an incorrect answer.

Minimax theorems. Using Sion’s minimax theorem wecan invert the order of the players if we allow nature to playmixed strategies

T ?(θ)−1 = maxw∈ΣA

infλ∈¬i?(θ)

1

2‖θ − λ‖2Vw (1)

= infq∈P(¬i?(θ))

maxa∈A

1

2Eλ∼q‖θ − λ‖2aaᵀ ,

where P(X ) denotes the set of probability distributions overthe set X . The annoying part in this formulation of the char-acteristic time is that the set ¬i?(θ) where the nature plays isa priori unknown (as the parameter is unknown to the agent).Indeed, to find an asymptotically optimal algorithm oneshould somehow solve this minimax game. But it is easyto remove this dependency noting that infλ∈¬i‖θ − λ‖ = 0for all i 6= i?(θ),

T ?(θ)−1 = maxi∈I

maxw∈ΣA

infλ∈¬i

1

2‖θ − λ‖2Vw .

Now we can see the characteristic time T ?(θ)−1 as thevalue of an other game where the agent plays a proportionof allocation of pulls w and an answer i. The agent couldalso use mixed strategies for the answer which leads to

T ?(θ)−1 = maxρ∈ΣI

maxw∈ΣA

1

2

∑i∈I

infλi∈¬i

ρi‖θ − λi‖2Vw

= maxρ∈ΣI

maxw∈ΣA

infλ∈

∏i(¬i)

1

2

∑i∈I

ρi‖θ − λi‖2Vw ,

where∏i∈I(¬i) denotes the Cartesian product of the alter-

native sets ¬i. But the function that appears in the value ofthe new game is not anymore convex in (w, ρ) and Sion’sminimax theorem does not apply anymore. We can how-ever convexify the problem by letting the agent to play adistribution w ∈ ΣAI over the arm-answer pairs (a, i), seeLemma 1 below proved in Appendix C.

Lemma 1. For all θ ∈M,

T ?(θ)−1 = maxw∈ΣAI

infλ∈

∏i(¬i)

1

2

∑(a,i)∈A×I

wa,i‖θ − λi‖2aaᵀ

= infq∈

∏i P(¬i)

1

2max

(a,i)∈A×BEλi∼qi‖θ − λ

i‖2aaᵀ .

Thus in this formulation the characteristic time is the valueof a fictitious zero-sum game where the agent plays a distri-bution w ∈ ΣAI over the arm-answer pairs (a, i) ∈ A× Iand nature chooses an alternative λi ∈ ¬i for all the answersi ∈ I. The algorithm LinGame-C that we propose in Sec-tion 3 is based on this formulation of the characteristic timewhereas algorithm LinGame is based on the formulation ofTheorem 1.

Best-arm identification complexity. The inverse of thecharacteristic time of Theorem 1 specializes to

T ?(θ)−1 = maxw∈ΣA

mina6=a?(θ)

⟨θ, a?(θ)− a

⟩22‖a?(θ)− a‖2

V −1w

for BAI (see Appendix D.1 for a proof). It is also possibleto explicit the characteristic time

T ?(θ) = minw∈ΣA

maxa6=a?(θ)

2‖a?(θ)− a‖2V −1w⟨

θ, a?(θ)− a⟩2 .

Since the characteristic time involves many problem de-pendent quantities that are unknown to the agent, previouspapers target loose problem-independent upper bounds onthe characteristic time. Soare et al. (2014) (see also Taoet al. 2018, Fiez et al. 2019) introduce the G-complexity (de-noted by AA) which coincides with the G-optimal designof experimental design theory (see Pukelsheim 2006) andthe ABdir-complexity1 (denoted by ABdir) inspired by thetransductive experimental design theory (Yu et al., 2006),

AA = minw∈ΣA

maxa∈A‖a‖2

V −1w

,

ABdir = minw∈ΣA

maxb∈Bdir

‖b‖2V −1w

,

where Bdir := a − a′ : (a, a′) ∈ A × A. For the G-optimal complexity we seek for a proportion of pulls w thatexplores uniformly the means of the arms, since the statis-tical uncertainty for estimating 〈θ, a〉 scales roughly with‖a‖V −1

w. In theAB-complexity we try to estimate uniformly

all the directions a−a′. On the contrary in this paper we tryto maximize directly the characteristic times, that is try to es-timate all the directions a?(θ)−a scaled by the squared gaps〈θ, a?(θ)− a〉. Note that the characteristic time can also beseen as a particular optimal transductive design. Indeed for

1This complexity is denoted as XY by Soare et al. (2014).

Gamification of Pure Exploration for Linear Bandits

B? :=

(a?(θ)− a)/∣∣⟨θ, a?(θ)− a⟩∣∣ : a ∈ A/

a?(θ)

,

it holds

T ?(θ) = 2AB?(θ) := 2 minw∈ΣA

maxb∈B?(θ)

‖b‖2V −1w

.

We have the following ordering on the complexities

T ?(θ) ≤ 2ABdir

∆min(θ)2≤ 8

AA∆min(θ)2

=8d

∆min(θ)2, (2)

where ∆min = mina 6=a?(θ)〈θ, a?(θ)− a〉 and the last equal-ity follows from the Kiefer-Wolfowitz equivalence theo-rem (Kiefer & Wolfowitz, 1959). Conversely the AA-complexity and theABdir-complexity are linked to an otherpure exploration problem, the thresholding bandits (see Ap-pendix D.2).

Remark 1. In order to compute all these complexities, it issufficient to solve the following generic optimal transductivedesign problem: for B a finite set of elements in Rd,

AB = minw∈ΣK

maxb∈B‖b‖2

V −1w

.

When B = A we can use an algorithm inspired by Frank-Wolfe (Frank & Wolfe, 1956) which possesses convergenceguarantees (Atwood, 1969; Ahipasaoglu et al., 2008). But inthe general case, up to our knowledge, there is no algorithmwith the same kind of guarantees. Previous works usedan heuristic based on a straightforward adaptation of theaforementioned algorithm for general sets B but it seemsto not converge on particular instances, see Appendix I. Weinstead propose in the same appendix an algorithm based onSaddle point Frank-Wolfe algorithm that seems to convergeon the different instances we tested.

3. AlgorithmWe present two asymptotically optimal algorithms for thegeneral pure-exploration problem. We also make the addi-tional assumption that the set of parameter is bounded, thatis we know M > 0 such that for all θ ∈ M, ‖θ‖ ≤ M .This assumption is shared by most of the works on linearbandits (e.g. Abbasi-Yadkori et al. 2011; Soare et al. 2014).

We describe primarily LinGame-C, detailed in Algorithm 2.The principle behind LinGame, detailed in Algorithm 1, issimilar and significant differences will be highlighted.

3.1. Notations

Counts. At each round t the algorithms will play an armat and choose (fictitiously) an answer it. We denote byNa,it :=

∑ts=1 1(at,it)=(a,i) the number of times the pair

(a, i) is chosen up to and including time t, and by Nat =∑

i∈I Na,is and N i

t =∑a∈I N

a,is the partial sums. The

vectors of counts at time t is denoted by Nt := (Nat )a∈A

Algorithm 1 LinGameInput: Agent learners for each answers (Liw)i∈I , thresh-old β(·, δ)for t = 1 . . . do

// Stopping ruleif maxi∈I infλ∈¬i

12‖θt−1−λ‖2VNt−1

≥β(t− 1, δ) thenstop and return ı = i?(θt−1)

end if// Best answerit = i?(θt−1)// Agent plays firstGet wt from Litw and update Wt = Wt−1 + wt// Best response for the natureλt ∈ argminλ∈¬it‖θt−1 − λ‖2Vwt// Feed optimistic gainsFeed learner Litw with gt(w) =

∑a∈A w

aUat /2// Track the weightsPull at ∈ argmina∈AN

at−1 −W a

t

end for

and when it is clear from the context we will also denoteby Na

t = (Na,it )i∈I and N i

t = (Na,it )i∈I the vectors of

partial counts.

Regularized least square estimator. We fix a regulariza-tion parameter η > 0. The regularized least square estimatorfor the parameter θ ∈M at time t is

θt = (VNt + ηId)−1

t∑s=1

Ysas ,

where Id is the identity matrix. By convention θ0 = 0.

3.2. AlgorithmsStopping rule. Our algorithms share the same stoppingrule. Following Garivier & Kaufmann (2016), our algo-rithms stop if a generalized likelihood ratio exceeds a thresh-old. It stops if

maxi∈I

infλi∈¬i

1

2‖θt − λi‖2VNt > β(t, δ) , (3)

and return i?t ∈argmaxi∈I

infλi∈¬i

‖θt − λi‖2VNt /2. This stopping

and decision rules ensures that the algorithms LinGame andLinGame-C are δ-correct regardless of the sampling ruleused, see lemma below2 proved in Appendix F.Lemma 2. Regardless of the sampling rule, the stoppingrule (3) with the threshold

β(t, δ) =

(√log

(1

δ

)+d

2log

(1 +

tL2

ηd

)+

√η

2M

)2

, (4)

2The fact that τδ < +∞ is a consequence of our analysis, seeAppendix E.

Gamification of Pure Exploration for Linear Bandits

Algorithm 2 LinGame-CInput: Agent learner Lw, threshold β(·, δ)for t = 1 . . . do

// Stopping ruleif maxi∈I infλ∈¬i

12‖θt−1−λ‖2VNt−1

≥ β(t−1, δ) thenstop and return ı = i?(θt−1).

end if// Agent plays firstGet wt from Lw and update Wt = Wt−1 + wt// Best response for the natureFor all i ∈ I, λit ∈ argminλ∈¬i‖θt−1 − λ‖2V

wit

// Feed optimistic gainsFeed learner Lw with gt(w) =

∑(a,i)∈A×I w

a,iUa,it /2

// Track the weightsPull (at, it) ∈ argmin(a,i)∈A×I N

a,it−1 − W

a,it

end for

satisfy Pθ(τδ <∞∧ i?τδ 6= i?(θ)

)≤ δ.

Our contribution is a sampling rule that minimizes the sam-ple complexity when combined with these stopping anddecision rules. We now explain our sampling strategy toensure that the stopping threshold is reached as soon aspossible.

Saddle point computation. Suppose in this paragraph,for simplicity, that the parameter vector θ is known. By thedefinition of the stopping rule and the generalized likelihoodratio, as long as the algorithm does not stop,

β(t, δ) ≥ infλ∈¬i?(θ)

∑a∈A

Nat ‖θ − λ‖2aaᵀ/2 .

If we manage to have Nt ≈ tw?(θ) (the optimal pullingproportions at θ), then this leads to β(t, δ) ≥ tT ?(θ)−1 and,solving that equation, we have asymptotic optimality.

Since there is only one correct answer, the parameter θbelongs to all sets ¬i for i 6= i?(θ). Hence

infλ∈¬i?(θ)

1

2

∑a∈A

Nat ‖θ − λ‖2aaᵀ

≥ infλt∈

∏i(¬i)

1

2

∑(a,i)∈A×I

Na,it ‖θ − λi‖2aaᵀ .

Introducing the sum removes the dependence in the un-known i∗(θ). LinGame-C then uses an agent playingweights w in ΣAI . LinGame does not use that sum overanswers, but uses a guess for i?(θ). Its analysis involvesproving that the guess is wrong only finitely many times inexpectation.

Our sampling rule implements the lower bound game be-tween an agent, playing at each stage s a weight vector ws

in the probability simplex ΣA×I , and nature, who computesat each stage a point λis ∈ ¬i for all i ∈ I. We additionallyensure that Na,i

t ≈∑ts=1 w

a,is . Suppose that the sampling

rule is such that at stage t, a εt-approximate saddle point isreached for the lower bound game, see Lemma 1. That is,

infλ∈

∏i(¬i)

t∑s=1

∑(a,i)∈A×I

wi,as ‖θ − λi‖2aaᵀ/2 + εt

≥t∑

s=1

∑(a,i)∈A×I

wi,as ‖θ − λis‖2aaᵀ/2

≥ max(a,i)∈A×I

t∑s=1

‖θ − λis‖2aaᵀ/2− εt .

Then if the algorithm did not stop, it verifies, usingLemma 1,

β(t, δ) ≥ t max(a,i)∈A×I

1

t

t∑s=1

‖θ − λis‖2aaᵀ/2− 2εt

≥ t infq∈

∏i∈IP(¬i)

max(a,i)∈A×I

Eλi∼qi‖θ−λi‖2aaᵀ/2−2εt

= tT ?(θ)−1 − 2εt .

Solving that equation, we get asymptotically the wantedt . T ?(θ) log(1/δ).

We implement the saddle point algorithm by using Ada-Hedge for the agent (a regret minimizing algorithm of theexponential weights family), and using best-response forthe nature, which plays after the agent. Precisely the learnerLw for LinGame-C is AdaHedge on ΣAI with the gains

gθt (w) =1

2

∑(a,i)∈A×I

wa,i‖θ − λis‖2aaᵀ .

Whereas LinGame uses I learners Liw, one for each possibleguess of i?(θ) with the gains. For i ∈ I, the learner Liw isalso AdaHedge but only on ΣA with the gains (when theguess is i)

gθt (w) =1

2

∑a∈A

wa‖θ − λis‖2aaᵀ .

εt is then the sum of the regrets of the two players. Best-response has regret 0, while the regret of AdaHedge isO(√t) for bounded gains, as seen in the following lemma,

taken from de Rooij et al. (2014).

Lemma 3. On the online learning problem with K armsand gains gs(w) =

∑k∈[K] w

kUks for s ∈ [t], AdaHedge,

Gamification of Pure Exploration for Linear Bandits

predicting (ws)s∈[t], has regret

Rt := maxw∈ΣK

t∑s=1

gs(w)− gs(ws)

≤ 2σ√t log(K) + 16σ(2 + log(K)/3) ,

where σ = maxs≤t

( maxk∈[K]

Uks − mink∈[K]

Uks ) .

Other combinations of algorithms are possible, as long asthe sum of their regrets is sufficiently small. At each staget ∈ N, both algorithms advance only by one iteration and astime progresses, the quality of the saddle point approxima-tion improves. This is in contrast with Track-and-Stop(Garivier & Kaufmann, 2016), in which an exact saddlepoint is computed at each stage, at a potentially much greatercomputational cost.

Optimism. The above saddle point argument would becorrect for a known game, while our algorithm is confrontedto a game depending on the unknown parameter θ. Follow-ing a long tradition of stochastic bandit algorithms, we usethe principle of Optimism in Face of Uncertainty. Givenan estimate θt−1, we compute upper bounds for the gain ofthe agent at θ, and feed these optimistic gains to the learner.Precisely, given the best response λit ∈ ¬i we define,

Ua,it =

maxξ min

(‖ξ − λit‖2aaᵀ , 4L2M2

)s.t. ‖θt−1 − ξ‖2VNt−1

+ηId≤ 2h(t)

,

where h(t) = β(t, 1/t3) is some exploration function. Weclipped the values, using that M and A are bounded toensure bounded gains for the learners. Under the event thatthe true parameter verifies ‖θt−1 − θ‖2 ≤ 2h(t), this isindeed an optimistic estimate of ‖θ − λit‖2aaᵀ . Note thatUa,it has a closed form expression, see Appendix E. Theoptimistic gain is then, for LinGame-C (see Algorithm 1 forthe one of LinGame),

gt(w) =1

2

∑(a,i)∈A×I

wa,iUa,it .

Tracking. In both Algorithm 1 and 2, the agent playsweight vectors in a simplex. Since the bandit procedureallows only to pull one arm at each stage, our algorithmneeds a procedure to transcribe weights into pulls. Thisis what we call tracking, following Garivier & Kaufmann(2016). The choice of arm (or arm and answer) is

at+1 ∈ argmina∈A

Nat −W a

t+1 for Algorithm 1,

(at+1, it+1) ∈ argmin(a,i)∈A×I

Na,it − W

a,it+1 for Algorithm 2.

This procedure guarantees that for all t ∈ N, u ∈ U , withU = A (resp. U = I × A) for Algo. 1 (resp. Algo. 2),− log(|U|) ≤ Nu

t −Wut ≤ 1. That result is due to Degenne

et al. (2020) and its proof is reproduced in Appendix G.

Theorem 2. For a regularization parameter3 η ≥ 2(1 +log(A))AL2 + M2, for the threshold β(t, δ) given by (4),for an exploration function h(t) = β(t, 1/t3), LinGameand LinGame-C are δ-correct and asymptotically optimal.That is, they verify for all θ ∈M,

lim supδ→0

Eθ[τδ]log 1/δ

≤ T ?(θ) .

The main ideas used in the proof are explained above. Thefull proof is in appendix E with finite δ upper bounds.

3.3. Bounded parameters

We provide a bounded version of different examples (e.g.BAI) in Appendix D where we add the assumption that theparameter setM is bounded. In particular we show howit affects the lower bound of Theorem 1: the characteristictime T ?(θ) is reduced (or equivalently T ?(θ)−1 increases).This is not surprising since we add a new constraint in theoptimization problem. This means that the algorithm shouldstop earlier. The counterpart of this improvement is that itis often difficult to compute the best response for nature.Indeed, for example, in BAI, there is an explicit expressionof the best response, see Appendix D.1. When the constraint‖λ‖ ≤M is added there is no explicit expression anymoreand one needs to solve an uni-dimensional optimizationproblem, see Lemma 5. To devise an asymptotically optimalalgorithm without the boundedness assumption remains anopen problem.

Note that in the proof of Theorem 2 we only use two timesthe boundedness assumption, first in the definition of thethreshold β(t, δ) (see Theorem 3) to handle the bias inducedby the regularization. Second, since the regret of AdaHedgeis proportional to the maximum of the upper confidencebounds U i,as , we need to ensure that they are bounded.

4. Related WorkWe survey previous work on linear BAI. The major focusis put on sampling rules in this section. We stress thatall the stopping rules employed in the linear BAI litera-ture are equivalent up to the choice of their explorationrate (More discussion given in Appendix H). As afore-mentioned, existing sampling rules are either based onSuccessiveElimination or UGapE. Elimination-basedsampling rules usually operate in phases and progressively

3This condition is a simple technical trick to simplify the anal-ysis. An η independent of A,L,M will lead to the same results upto minor adaptations of the proof.

Gamification of Pure Exploration for Linear Bandits

discard sub-optimal directions. Gap-based sampling rulesalways play the most informative arm that reduces the un-certainty of the gaps between the empirical best arm and theothers.

XY-Static and XY-Adaptive. Soare et al. (2014)first propose a static allocation design XY-Static thataims at reducing the uncertainty of the gaps of allarms. More precisely, it requires to either solve theABdir-complexity or use a greedy version that pulls thearm argmina∈Amaxb∈Bdir

‖b‖2V −1w

at the cost of havingno guarantees. An elimination-like alternative calledXY-Adaptive is proposed then to overcome that issue.We say elimination-like since XY-Adaptive does not dis-card arms once and for all, but reset the active arm setat each phase. XY-Adaptive and XY-Static are thefirst algorithms being linked to AA-optimality, but are notasymptotically optimal.

ALBA. ALBA is also an eliminations-based algorithmdesigned by Tao et al. (2018) that improves overXY-Adaptive by a factor of d in the sample complexityusing a tighter elimination criterion.

RAGE. Fiez et al. (2019) extend XY-Static andXY-Adaptive to a more general transductive bandits set-ting. RAGE is also elimination-based and requires the com-putation of ABdir-complexity at each phase.

LinGapE and variants. LinGapE (Xu et al., 2018) is thefirst gap-based sampling rule for linear BAI. LinGapE isinspired by UGapE (Gabillon et al., 2012). It is, how-ever, not clear whether LinGapE is asymptotically opti-mal or not. Similar to XY-Static, LinGapE either re-quires to solve a time-consuming optimization problemat each step, or can use a greedy version that pulls armargmina∈A‖ait−ajt‖2(Vw+aaᵀ)−1 instead, again at the cost

of losing guarantees. Here it = i?(θt) and ajt is the mostambiguous arm w.r.t. ait , i.e. argmaxj 6=it〈θt, aj−a

?(θt)〉+‖a?(θt)−ajt‖V −1

Nt

√2β(t, δ). On the other hand, Zaki et al.

(2019) propose a new algorithm based on LUCB. With a care-ful examination, we note that the sampling rule of GLUCB isequivalent to that of the greedy LinGapE using the Sherman-Morrison formula. Later, Kazerouni & Wein (2019) providea natural extension of LinGapE to the generalized linearbandits setting, where the rewards depend on a strictly in-creasing inverse link function. GLGapE reduces to LinGapEwhen the inverse link function is the identity function.

Note that all the sampling rules presented here depend onδ (except XY-Static), while our sampling rules have a δ-free property which is appealing for applications as arguedby Jun & Nowak (2016). Also all the guarantees in theliterature are of the form C log(δ) + O

(log(1/δ)

)for a

constant C that is strictly larger than T ?(θ)−1.

5. ExperimentsBesides our algorithms, we implement the following algo-rithms, all using the same stopping rule (more discussiongiven in Appendix H): uniform sampling, the greedy ver-sion of XY-Static (including AA-allocation and ABdir-allocation), XY-Adaptive, and the greedy version ofLinGapE. We skip GLUCB/GLGapE since they are more orless equivalent to LinGapE in the scope of this paper.

The usual hard instance. Usual sampling rules for classi-cal BAI may not work for the linear case. This can be under-stood on a well-studied instance already discussed by Soareet al. (2014); Xu et al. (2018), which encapsulates the diffi-culty of BAI in a linear bandit, and thus is the first instanceon which we test our algorithms. In this instance, arms arethe canonical basis a1 = e1, a2 = e2, ad = ed, plus an ad-ditional disturbing arm ad+1 = (cos(α), sin(α), 0, . . . , 0)ᵀ,and a true regression parameter θ equal to e1. In this prob-lem, the best arm is always a1, but when the angle α is small,the disturbing arm ad+1 is hard to discriminate from a1. Asalready argued by Soare et al. (2014), an efficient samplingrule for this problem instance would rather pull a2 in orderto reduce the uncertainty in the direction a1 − ad+1. Naiveadaptation of classical BAI algorithms cannot deal with thatsituation naturally. We further use a simple set of experi-ments to justify that intuition. We run our two algorithmsand the one of Degenne et al. (2019) that we call DKM overthe problem instance whence d = 2, δ = 0.01 and α = 0.1.We show the number of pulls for each arm averaged over100 replications of experiments in Table 1. We see that,indeed, DKM pulls too much a3, while our algorithms focusmostly on a2.

LinGame LinGame-C DKMa1 1912 1959 1943a2 5119 4818 4987a3 104 77 1775

Total 7135 6854 8705

Table 1. Average number of pulls of each arm.

Comparison of different complexities. We use the pre-vious setting to illustrate various complexities differ in prac-tice. In Table 2 we compare the different complexities men-tioned in this paper: the characteristic time T ?(θ) and itsassociated optimal weights w?AB?(θ), the ABdir-complexityand its associated optimal design w?ABdir

, the G-optimalcomplexityAA and its associated optimal design w?AA. Foreach weight vector w ∈ w?AB?(θ), wABdir , wAA, we alsoprovide the lower bound Tw given by Therorem 1, i.e.

Tw = maxa 6=a?(θ)

⟨θ, a?(θ)− a

⟩22‖a?(θ)− a‖2

V −1w

log(1/δ).

Gamification of Pure Exploration for Linear Bandits

CG-C Lk-C RR fix GS XYS XYA LG0

200

400

600

800

1000

1200

CG-C Lk-C RR fix GS XYS XYA LG0

500

1000

1500

CG-C Lk-C RR fix GS XYS XYA LG0

1000

2000

3000

Figure 1. Sample complexity over the usual counter-example with δ = 0.1, 0.01, 0.0001 respectively. CG = LinGame-C, Lk = LinGame,RR = uniform sampling, fix = tracking the fixed weights, GS = XY-Static with AA-allocation, XYS = XY-Static with ABdir-allocation, LG = LinGapE. The mean stopping time is represented by a black cross.

CG-C Lk-C RR fix GS XYS XYA LG

2000

4000

6000

8000

CG-C Lk-C RR fix GS XYS XYA LG

500

1000

1500

2000

CG-C Lk-C RR fix GS XYS XYA LG

1000

2000

3000

4000

CG-C Lk-C RR fix GS XYS XYA LG

500

1000

1500

2000

Figure 2. Sample complexity over random unit sphere vectors with d = 6, 8, 10, 12 from left to right.

In particular we notice that targeting the proportions of pullswABdir , wAA leads to a much larger lower bound than theone obtained with the optimal weights.

w?AB? w?ABdirw?AA

a1 0.047599 0.499983 0.499983a2 0.952354 0.499983 0.499983a3 0.000047 0.000033 0.000033Tw 369 2882 2882

T ?(θ) 2ABdir/∆2min 8AA/∆2

minComplexity 0.124607 32.0469 64.0939

Table 2. Optimal w for various complexities (∆min = 0.0049958).

Comparison with other algorithms. Finally we bench-mark our two sampling rules against others from the liter-ature. We test over two synthetic problem instances, withthe first being the previous counter-example. We set d = 2,α = π/6. Fig. 1 shows the empirical stopping time of eachalgorithms averaged over 100 runs, with a confidence levelδ = 0.1, 0.01, 0.0001 from left to right. Our two algorithmsshow competitive performance (the two leftmost boxes oneach plot), and are only slightly worse than LinGapE.

For the second instance, we consider 20 arms randomly gen-erated from the unit sphere Sd−1 := a ∈ Rd; ‖a‖2 =1. We choose the two closest arms a, a′ and we setθ = a + 0.01(a′ − a) so that a is the best arm. Thissetting has already been considered by Tao et al. (2018).We report the same box plots over 100 replications as be-fore with increasing dimension in Fig. 2. More precisely,

we set d = 6, 8, 10, 12 respectively, and always keep asame δ = 0.01. Our algorithms consistently show strongperformances compared to other algorithms apart fromLinGapE. Moreover, we can see that in these random exam-ples, LinGame-C works better than the non-confexified one,and is even competitive compared to LinGapE.

We stress that although the main focus of this paper is the-oretical, with algorithms that are asymptotically optimal,our methods are also competitive with earlier algorithmsexperimentally.

6. ConclusionIn this paper, we designed the first practically usable asymp-totically optimal sampling rules for the pure explorationgame for finite-arm linear bandits. Should the boundednessassumption be necessary to have optimal algorithm remainsan open question.

Another concern about the current sampling rules could betheir computational complexity. In BAI, the one step com-plexity of LinGame-C (or LinGame) is dominated by thecomputation of the best response for nature, which requiresa full matrix inversion. Alternatives that involve rank-1updates should be considered.

More generally, however, the part of fixed-confidence pureexploration algorithms that needs an improvement the mostis the stopping rule. While the one we used guarantees δ-correctness, it is very conservative. Indeed, the experimentalerror rates of algorithms using that stopping rule are ordersof magnitude below δ. This means that the concentration

Gamification of Pure Exploration for Linear Bandits

inequality does not reflect the query we seek to answer. Itquantifies deviations of the d-dimensional estimate in alldirections (morally, along 2d directions). However, for theusual BAI setting with d arms in an orthogonal basis, itwould be sufficient to control the deviation of that estimatorin d− 1 directions to make sure that i∗(θ) = i∗(θt).

Finally, the good performance of LinGapE raises the natu-ral question of whether it could be proven to have similarasymptotic optimality.

AcknowledgementsThe research presented was supported by European CHIST-ERA project DELTA, French Ministry of Higher Educationand Research, Nord-Pas-de-Calais Regional Council, andby French National Research Agency as part of the “In-vestissements d’avenir” program, reference ANR19-P3IA-0001 (PRAIRIE 3IA Institute) and the project BOLD, refer-ence ANR19-CE23-0026-04.

ReferencesAbbasi-Yadkori, Y., Pal, D., and Szepesvari, C. Im-

proved algorithms for linear stochastic bandits. InAdvances in Neural Information Processing Sys-tems 24 (NIPS), 2011. ISBN 9781618395993.URL https://sites.ualberta.ca/˜szepesva/papers/linear-bandits-NIPS2011.pdf.

Ahipasaoglu, S. D., Sun, P., and Todd, M. J. Linearconvergence of a modified Frank-Wolfe algorithmfor computing minimum-volume enclosing ellipsoids.Optimization Methods and Software, 23(1):5–19, 2008.ISSN 10556788. doi: 10.1080/10556780701589669.URL https://www.tandfonline.com/doi/full/10.1080/10556780701589669.

Atwood, C. L. Optimal and efficient designs of experiments.The Annals of Mathematical Statistics, 40(5):1570–1602,1969. ISSN 0003-4851. doi: 10.1214/aoms/1177697374.

Audibert, J.-Y. and Bubeck, S. Best arm identification inmulti-armed bandits. In Proceedings of the 23rd AnnualConference on Learning Theory (CoLT), 2010. ISBN9780982252925. URL https://hal-enpc.archives-ouvertes.fr/hal-00654404/document.

Auer, P. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine LearningResearch, 3:397–422, 2002. URL https://www.jmlr.org/papers/volume3/auer02a/auer02a.pdf.

Bubeck, S., Munos, R., and Stoltz, G. Pure explorationin multi-armed bandits problems. In Proceedings of the20th International Conference on Algorithmic Learning

Theory (ALT), pp. 23–37, 2009. ISBN 3642044131. doi:10.1007/978-3-642-04414-4 7. URL https://arxiv.org/pdf/0802.2655.pdf.

Chen, S., Lin, T., King, I., Lyu, M. R., and Chen,W. Combinatorial pure exploration of multi-armedbandits. In Advances in Neural Information Pro-cessing Systems 27 (NIPS), pp. 379–387, 2014.URL https://papers.nips.cc/paper/5433-combinatorial-pure-exploration-of-multi-armed-bandits.pdf.

Chernoff, H. Sequential design of experiments. TheAnnals of Mathematical Statistics, 30(3):755–770,1959. ISSN 0003-4851. doi: 10.1214/aoms/1177706205. URL https://projecteuclid.org/download/pdf_1/euclid.aoms/1177706205.

de Rooij, S., Van Erven, T., Grunwald, P. D., and Koolen,W. M. Follow the leader if you can, hedge if you must.Journal of Machine Learning Research, 15:1281–1316,2014. ISSN 15337928. URL https://arxiv.org/pdf/1301.0534.pdf.

Degenne, R. and Koolen, W. M. Pure exploration withmultiple correct answers. In Advances in Neural Infor-mation Processing Systems 32 (NeurIPS), 2019. URLhttps://arxiv.org/pdf/1902.03475.pdf.

Degenne, R., Koolen, W., and Menard, P. Non-asymptoticpure exploration by solving games. In Advances in Neu-ral Information Processing Systems 32 (NeurIPS), 2019.URL http://arxiv.org/pdf/1906.10431.pdf.

Degenne, R., Shao, H., and Koolen, W. M. Structure Adap-tive Algorithms for Stochastic Bandits. InternationalConference on Machine Learning, 2020.

Even-Dar, E., Mannor, S., and Mansour, Y. PAC bounds forMulti-armed Bandit and Markov Decision Processes. InProceedings of the15th Annual Conference on LearningTheory (CoLT), volume 2375, pp. 255–270, 2002. ISBN354043836X. doi: 10.1007/3-540-45435-7 18.

Even-dar, E., Mannor, S., and Mansour, Y. Action elimina-tion and stopping conditions for reinforcement learning.In Proceedings of the 20th International Conference onMachine Learning (ICML), pp. 162–169, 2003. ISBN1577351894. URL https://www.aaai.org/Papers/ICML/2003/ICML03-024.pdf.

Fiez, T., Jain, L., Jamieson, K., and Ratliff, L. Sequentialexperimental design for transductive linear bandits. InAdvances in Neural Information Processing Systems 32(NeurIPS), 2019. URL http://arxiv.org/pdf/1906.08399.pdf.

Gamification of Pure Exploration for Linear Bandits

Frank, M. and Wolfe, P. An algorithm for quadratic pro-gramming. Naval Research Logistics Quarterly, 3(1-2):95–110, 1956. URL https://onlinelibrary.wiley.com/doi/epdf/10.1002/nav.3800030109.

Gabillon, V., Ghavamzadeh, M., and Lazaric, A. Bestarm identification: A unified approach to fixedbudget and fixed confidence. In Advances in Neu-ral Information Processing Systems 25 (NIPS), pp.3212–3220, 2012. ISBN 9781627480031. URLhttp://papers.nips.cc/paper/4640-best-arm-identification-a-unified-approach-to-fixed-budget-and-fixed-confidence.pdf.

Garivier, A. and Kaufmann, E. Optimal best arm identifi-cation with fixed confidence. In Proceedings of the 29thAnnual Conference on Learning Theory (CoLT), 2016.URL http://arxiv.org/pdf/1602.04589.pdf.

Garivier, A., Menard, P., and Stoltz, G. Explore first,exploit next: The true shape of regret in bandit prob-lems. Mathematics of Operations Research, 44(2):377–399, 2018. ISSN 15265471. doi: 10.1287/moor.2017.0928. URL https://pubsonline.informs.org/doi/pdf/10.1287/moor.2017.0928.

Hoffman, M. W., Shahriari, B., and de Freitas, N. On cor-relation and budget constraints in model-based banditoptimization with application to automatic machine learn-ing. In Proceedings of the 17th International Conferenceon Artificial Intelligence and Statistics (AIStats), pp. 365–374, 2014. URL http://proceedings.mlr.press/v33/hoffman14.pdf.

Jun, K.-S. and Nowak, R. Anytime exploration for multi-armed bandits using confidence information. In Pro-ceedings of the 33rd International Conference on Ma-chine Learning (ICML), pp. 974–982, 2016. ISBN9781510829008. URL http://proceedings.mlr.press/v48/jun16.pdf.

Kalyanakrishnan, S. and Stone, P. Efficient selectionof multiple bandit arms: Theory and practice. InProceedings of the 27th International Conferenceon Machine Learning (ICML), pp. 511–518, 2010.ISBN 9781605589077. URL https://www.cs.utexas.edu/users/pstone/Papers/bib2html-links/ICML10-kalyanakrishnan.pdf.

Karnin, Z., Koren, T., and Somekh, O. Almost opti-mal exploration in multi-armed bandits. In Proceed-ings of the 30th International Conference on MachineLearning (ICML), pp. 1238–1246, 2013. URL http://proceedings.mlr.press/v28/karnin13.pdf.

Kaufmann, E., Koolen, W. M., and Garivier, A. Se-quential test for the lowest mean: From Thompson to

Murphy sampling. In Advances in Neural InformationProcessing Systems 31 (NeurIPS), pp. 6332–6342,2018. URL https://papers.nips.cc/paper/7870-sequential-test-for-the-lowest-mean-from-thompson-to-murphy-sampling.pdf.

Kazerouni, A. and Wein, L. M. Best arm identifi-cation in generalized linear bandits. arXiv preprintarXiv:1905.08224, 2019. URL http://arxiv.org/pdf/1905.08224.pdf.

Kiefer, J. and Wolfowitz, J. Optimum designs in regres-sion problems. The Annals of Mathematical Statistics,30(2):271–294, 1959. doi: 10.1007/978-1-4615-6660-1 2. URL https://projecteuclid.org/download/pdf_1/euclid.aoms/1177706252.

Lattimore, T. and Szepesvari, C. Bandit Algorithms.Cambridge University Press, 2018. URL http://downloads.tor-lattimore.com/book.pdf.

Locatelli, A., Gutzeit, M., and Carpentier, A. An op-timal algorithm for the thresholding bandit problem.In Proceedings of the 33rd International Conferenceon Machine Learning (ICML), pp. 2539–2554, 2016.ISBN 9781510829008. URL http://proceedings.mlr.press/v48/locatelli16.pdf.

Lu, H., Freund, R. M., and Nesterov, Y. Relatively-smoothconvex optimization by first-order methods , and appli-cations. SIAM Journal of Optimization, 28(1):333–354,2018. URL https://epubs.siam.org/doi/pdf/10.1137/16M1099546.

Menard, P. Gradient ascent for active exploration in banditproblems. arXiv preprint arXiv:1905.08165, 2019. URLhttp://arxiv.org/pdf/1905.08165.pdf.

Pukelsheim, F. Optimal Design of Experiments. So-ciety for Industrial and Applied Mathematics, 2006.doi: 10.1002/0471667196.ess3056.pub2. URLhttps://epubs.siam.org/doi/pdf/10.1137/1.9780898719109.bm.

Qin, C., Klabjan, D., and Russo, D. Improving the expectedimprovement algorithm. In Advances in Neural Informa-tion Processing Systems 30 (NIPS), pp. 5381–5391, 2017.URL http://arxiv.org/pdf/1705.10033.pdf.

Robbins, H. Some aspects of the sequential de-sign of experiments. Bulletin of the AmericanMathematics Society, 58(5):527–535, 1952. doi:10.1090/S0002-9904-1952-09620-8. URL http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.335.3232&rep=rep1&type=pdf.

Gamification of Pure Exploration for Linear Bandits

Shang, X., de Heide, R., Kaufmann, E., Menard, P., andValko, M. Fixed-confidence guarantees for Bayesian best-arm identification. In Proceedings of the 23rd Interna-tional Conference on Artificial Intelligence and Statistics(AIStats), 2020. URL http://arxiv.org/pdf/1910.10945.pdf.

Soare, M., Lazaric, A., and Munos, R. Best-arm identifica-tion in linear bandits. In Advances in Neural InformationProcessing Systems 26 (NIPS), pp. 828–836, 2014. URLhttps://arxiv.org/pdf/1409.6110.pdf.

Tao, C., Blanco, S. A., and Zhou, Y. Best arm identi-fication in linear bandits with linear dimension depen-dency. In Proceedings of the 35th International Confer-ence on Machine Learning (ICML), pp. 7773–7786, 2018.ISBN 9781510867963. URL http://proceedings.mlr.press/v80/tao18a/tao18a.pdf.

Teraoka, K., Hatano, K., and Takimoto, E. Effi-cient sampling method for monte carlo tree searchproblem. IEICE Transactions on Informationand Systems, E97-D(3):392–398, 2014. ISSN17451361. doi: 10.1587/transinf.E97.D.392. URLhttps://pdfs.semanticscholar.org/30c9/85d7eb0deb9e40fa63ec5a7df818efc85952.pdf.

Xu, L., Honda, J., and Sugiyama, M. A fully adaptivealgorithm for pure exploration in linear bandits. In Pro-ceedings of the 21st International Conference on Arti-ficial Intelligence and Statistics (AIStats), pp. 843–851,2018. URL http://proceedings.mlr.press/v84/xu18d/xu18d.pdf.

Yu, K., Bi, J., and Tresp, V. Active learning viatransductive experimental design. In Proceedingsof the 23rd International Conference on MachineLearning (ICML), pp. 1081–1088, 2006. ISBN1595933832. URL https://dl.acm.org/doi/pdf/10.1145/1143844.1143980?download=true.

Zaki, M., Mohan, A., and Gopalan, A. Towards optimaland efficient best arm identification in linear bandits. InWorkshop on Machine Learning at Neural InformationProcessing Systems (NeurIPS-CausalML), 2019. URLhttps://arxiv.org/pdf/1911.01695.pdf.