adversarial linear contextual bandits with graph

33
Adversarial Linear Contextual Bandits with Graph-Structured Side Observations Lingda Wang Department of Electrical and Computer Engineering University of Illinois at Urbana-Champaign 35th AAAI Conference on Artificial Intelligence (AAAI 2021) February 2 nd -9 th , 2021 Joint work with Bingcong Li (UMN), Huozhi Zhou (UIUC), Georgios B. Giannakis (UMN), Lav R. Varshney (UIUC), and Zhizhen Zhao (UIUC) Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 1 / 17

Upload: others

Post on 11-Apr-2022

8 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Adversarial Linear Contextual Bandits with Graph

Adversarial Linear Contextual Bandits withGraph-Structured Side Observations

Lingda Wang

Department of Electrical and Computer EngineeringUniversity of Illinois at Urbana-Champaign

35th AAAI Conference on Artificial Intelligence (AAAI 2021)

February 2nd - 9th, 2021

Joint work with Bingcong Li (UMN), Huozhi Zhou (UIUC), Georgios B. Giannakis (UMN),Lav R. Varshney (UIUC), and Zhizhen Zhao (UIUC)

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 1 / 17

Page 2: Adversarial Linear Contextual Bandits with Graph

Motivation

Agent

Item

Viral marketing over a social network:

The Agent offers a survey with a promotion to a user;

The user finishes the survey and share it in the social network;

The agent gets side observations from followers.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 2 / 17

Page 3: Adversarial Linear Contextual Bandits with Graph

Linear Contextual Bandits with Side Observations

In each time step t = 1, 2, . . . ,T :

Adversary picks the feedback graph Gt , and loss vectors {θi,t ∈ Rd}Ki=1;

Environment reveals the context Xt ∈ Rd , Xt ∼ D;

The learning agent chooses action It ∈ [K ];

The agent incurs and observes loss `t(Xt , It) = 〈Xt , θIt ,t〉 ∈ [−1, 1];

The agent also observes losses of It ’s neighbors in Gt ;

The adversary discloses Gt to the agent.

Goal: Minimize the regret:

RT = maxπT∈Π

E

[T∑t=1

⟨Xt , θIt ,t − θπT (Xt),t

⟩],

against the best policy πT ∈ Π := {πT∣∣all policies πT : Rd 7→ V }.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 3 / 17

Page 4: Adversarial Linear Contextual Bandits with Graph

Linear Contextual Bandits with Side Observations

In each time step t = 1, 2, . . . ,T :

Adversary picks the feedback graph Gt , and loss vectors {θi,t ∈ Rd}Ki=1;

Environment reveals the context Xt ∈ Rd , Xt ∼ D;

The learning agent chooses action It ∈ [K ];

The agent incurs and observes loss `t(Xt , It) = 〈Xt , θIt ,t〉 ∈ [−1, 1];

The agent also observes losses of It ’s neighbors in Gt ;

The adversary discloses Gt to the agent.

Goal: Minimize the regret:

RT = maxπT∈Π

E

[T∑t=1

⟨Xt , θIt ,t − θπT (Xt),t

⟩],

against the best policy πT ∈ Π := {πT∣∣all policies πT : Rd 7→ V }.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 3 / 17

Page 5: Adversarial Linear Contextual Bandits with Graph

Existing Results on Contextual Bandits

Regreti.i.d. context adversarial context

i.i.d. loss1 O(√dKT ) O(

√dKT )

adversarial loss see below Θ(T )

For adversarial loss and i.i.d context:

BISTRO++2: O(T 2/3(K logN)1/2), oracle-efficient;

RobustLinEXP33: O(T 2/3(Kd logK )1/3);

RealLinEXP33: O(√dKT logK logT ).

Question: Is it possible to do better with side observations?

Answer: Yes!

1Abbasi-Yadkori, Y., Pal, D., and Szepesvari, C. (2011). Improved algorithms for linear stochastic bandits. In Advances in

Neural Information Processing Systems, pages 2312-2320.2

Rakhlin, A. and Sridharan, K. (2016). BISTRO: An efficient relaxation-based method for contextual bandits. InInternational Conference on Machine Learning, pages 1977-1985.

3Neu, G. and Olkhovskaya, J. (2020). Efficient and robust algorithms for adversarial linear contextual bandits. In

Conference on Learning Theory, pages 1-20.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 4 / 17

Page 6: Adversarial Linear Contextual Bandits with Graph

Existing Results on Contextual Bandits

Regreti.i.d. context adversarial context

i.i.d. loss1 O(√dKT ) O(

√dKT )

adversarial loss see below Θ(T )

Question: Is it possible to do better with side observations?

Answer: Yes!

1Abbasi-Yadkori, Y., Pal, D., and Szepesvari, C. (2011). Improved algorithms for linear stochastic bandits. In Advances in

Neural Information Processing Systems, pages 2312-2320.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 4 / 17

Page 7: Adversarial Linear Contextual Bandits with Graph

Existing Results on Contextual Bandits

Regreti.i.d. context adversarial context

i.i.d. loss1 O(√dKT ) O(

√dKT )

adversarial loss see below Θ(T )

Question: Is it possible to do better with side observations?

Answer: Yes!

1Abbasi-Yadkori, Y., Pal, D., and Szepesvari, C. (2011). Improved algorithms for linear stochastic bandits. In Advances in

Neural Information Processing Systems, pages 2312-2320.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 4 / 17

Page 8: Adversarial Linear Contextual Bandits with Graph

Contributions

A new bandit model: jointly leverages contexts and side observations.

Introducing two new algorithms, EXP3-LGC-U and EXP3-LGC-IX,with:

Novel loss vector estimates design;

Better dependence on K (# of actions).

Promising numerical performance.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 5 / 17

Page 9: Adversarial Linear Contextual Bandits with Graph

Assumptions

I.i.d contexts: The context Xt ∈ Rd is drawn from a distribution Di.i.d., where D is known by the agent in advance .

Extra observation oracle: At each time step t, there exists an oraclethat draws a context Xt ∈ Rd from D, and discloses Xt together withthe losses lt(Xt , i) to the agent.

Nonoblivious adversary: The adversary can be nonoblivious, who isallowed to choose Gt and θi ,t ,∀i ∈ V at time t according to arbitraryfunctions of the interaction history Ft−1 before time step t.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 6 / 17

Page 10: Adversarial Linear Contextual Bandits with Graph

Assumptions

I.i.d contexts: The context Xt ∈ Rd is drawn from a distribution Di.i.d., where D is known by the agent in advance .

Extra observation oracle: At each time step t, there exists an oraclethat draws a context Xt ∈ Rd from D, and discloses Xt together withthe losses lt(Xt , i) to the agent.

Nonoblivious adversary: The adversary can be nonoblivious, who isallowed to choose Gt and θi ,t ,∀i ∈ V at time t according to arbitraryfunctions of the interaction history Ft−1 before time step t.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 6 / 17

Page 11: Adversarial Linear Contextual Bandits with Graph

Assumptions

I.i.d contexts: The context Xt ∈ Rd is drawn from a distribution Di.i.d., where D is known by the agent in advance .

Extra observation oracle: At each time step t, there exists an oraclethat draws a context Xt ∈ Rd from D, and discloses Xt together withthe losses lt(Xt , i) to the agent.

Nonoblivious adversary: The adversary can be nonoblivious, who isallowed to choose Gt and θi ,t ,∀i ∈ V at time t according to arbitraryfunctions of the interaction history Ft−1 before time step t.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 6 / 17

Page 12: Adversarial Linear Contextual Bandits with Graph

Algorithm: EXP3-LGC-U

Main steps in each time step t:

Compute the exponential weight:

wt(Xt , i) = exp

(−η

t−1∑s=1

⟨Xt , θi,s

⟩);

Draw the action It ∈ [K ] with probability:

πat (i∣∣Xt) = (1− γ)

wt(Xt , i)∑j∈V wt(Xt , j)

K;

Estimate the loss vector θi,t :

θi,t =I{i ∈ SIt ,t}qt(i

∣∣Xt)Σ−1Xt

˜t(Xt , i),

where qt(i∣∣Xt) = πa

t (i∣∣Xt) +

∑j :j

t−→iπat (j∣∣Xt).

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 7 / 17

Page 13: Adversarial Linear Contextual Bandits with Graph

Algorithm: EXP3-LGC-U

Main steps in each time step t:

Compute the exponential weight:

wt(Xt , i) = exp

(−η

t−1∑s=1

⟨Xt , θi,s

⟩);

Draw the action It ∈ [K ] with probability:

πat (i∣∣Xt) = (1− γ)

wt(Xt , i)∑j∈V wt(Xt , j)

K;

Estimate the loss vector θi,t :

θi,t =I{i ∈ SIt ,t}qt(i

∣∣Xt)Σ−1Xt

˜t(Xt , i),

where qt(i∣∣Xt) = πa

t (i∣∣Xt) +

∑j :j

t−→iπat (j∣∣Xt).

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 7 / 17

Page 14: Adversarial Linear Contextual Bandits with Graph

Algorithm: EXP3-LGC-U

Main steps in each time step t:

Compute the exponential weight:

wt(Xt , i) = exp

(−η

t−1∑s=1

⟨Xt , θi,s

⟩);

Draw the action It ∈ [K ] with probability:

πat (i∣∣Xt) = (1− γ)

wt(Xt , i)∑j∈V wt(Xt , j)

K;

Estimate the loss vector θi,t :

θi,t =I{i ∈ SIt ,t}qt(i

∣∣Xt)Σ−1Xt

˜t(Xt , i),

where qt(i∣∣Xt) = πa

t (i∣∣Xt) +

∑j :j

t−→iπat (j∣∣Xt).

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 7 / 17

Page 15: Adversarial Linear Contextual Bandits with Graph

Algorithm: EXP3-LGC-U

Main steps in each time step t:

Compute the exponential weight:

wt(Xt , i) = exp

(−η

t−1∑s=1

⟨Xt , θi,s

⟩);

Draw the action It ∈ [K ] with probability:

πat (i∣∣Xt) = (1− γ)

wt(Xt , i)∑j∈V wt(Xt , j)

K;

Estimate the loss vector θi,t :

θi,t =I{i ∈ SIt ,t}qt(i

∣∣Xt)Σ−1Xt

˜t(Xt , i),

where qt(i∣∣Xt) = πa

t (i∣∣Xt) +

∑j :j

t−→iπat (j∣∣Xt).

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 7 / 17

Page 16: Adversarial Linear Contextual Bandits with Graph

Properties of θi ,t

Advantages and properties of θi ,t :

The estimate is unbiased w.r.t. Ft−1 and Xt :

E[θi,t |Xt ,Ft−1

]= θi,t ;

Regret can be evaluated using θi,t directly:

RT = maxπT∈Π

E

[T∑t=1

∑i∈V

(πat (i |Xt)− πT (i |Xt))

⟨Xt , θi,t

⟩].

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 8 / 17

Page 17: Adversarial Linear Contextual Bandits with Graph

Properties of θi ,t

Advantages and properties of θi ,t :

The estimate is unbiased w.r.t. Ft−1 and Xt :

E[θi,t |Xt ,Ft−1

]= θi,t ;

Regret can be evaluated using θi,t directly:

RT = maxπT∈Π

E

[T∑t=1

∑i∈V

(πat (i |Xt)− πT (i |Xt))

⟨Xt , θi,t

⟩].

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 8 / 17

Page 18: Adversarial Linear Contextual Bandits with Graph

Main Results I

Theorem

For any positive η ∈ (0, 1), choosing γ = ηKσ2/λmin, the expectedcumulative regret of EXP3-LGC-U satisfies:

Rt ≤logK

η+

2ηKσ2

λminT + ηd

T∑t=1

E [Qt ] ,

where Qt = α(Gt) if Gt is undirected, andQt = 4α(Gt) log(4K 2/(α(Gt)γ)) if Gt is directed.

Corollary

The regret of EXP3-LGC-U is O(√

(K + α(G )d)T logK ) in the undirectedgraph settting, and O(

√(K + α(G )d)T log(KdT )) in the directed graph

setting, where α(G ) ≤ K is the averaged independence number of {Gt}.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 9 / 17

Page 19: Adversarial Linear Contextual Bandits with Graph

Proof Sketch of EXP3-LGC-U

Analyze log Wt+1(Xt)Wt(Xt)

, where Wt(x) =∑

i∈V wt(x , i):

T∑t=1

E

∑i∈V

πat (i |Xt)

1− γ

(−η⟨Xt , θi,t

⟩+ η2

⟨Xt , θi,t

⟩2)

+ηγ

K(1− γ)

∑i∈V

⟨Xt , θi,t

⟩≤ E

[T∑t=1

logWt+1(Xt)

Wt(Xt)

]≤ E

−η T∑t=1

∑i∈V

πT (i |Xt)⟨Xt , θi,t

⟩− log K

.

Reorder the terms on both sides and apply some straightforward facts:

RT ≤log K

η+ 2γT + η E

T∑t=1

∑i∈V

πat (i |Xt)

⟨Xt , θi,t

⟩2

︸ ︷︷ ︸

A

.

Use graph-theoretic results form Alon et al. and properties of θi ,t tobound term A.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 10 / 17

Page 20: Adversarial Linear Contextual Bandits with Graph

Proof Sketch of EXP3-LGC-U

Analyze log Wt+1(Xt)Wt(Xt)

, where Wt(x) =∑

i∈V wt(x , i):

T∑t=1

E

∑i∈V

πat (i |Xt)

1− γ

(−η⟨Xt , θi,t

⟩+ η2

⟨Xt , θi,t

⟩2)

+ηγ

K(1− γ)

∑i∈V

⟨Xt , θi,t

⟩≤ E

[T∑t=1

logWt+1(Xt)

Wt(Xt)

]≤ E

−η T∑t=1

∑i∈V

πT (i |Xt)⟨Xt , θi,t

⟩− log K

.

Reorder the terms on both sides and apply some straightforward facts:

RT ≤log K

η+ 2γT + η E

T∑t=1

∑i∈V

πat (i |Xt)

⟨Xt , θi,t

⟩2

︸ ︷︷ ︸

A

.

Use graph-theoretic results form Alon et al. and properties of θi ,t tobound term A.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 10 / 17

Page 21: Adversarial Linear Contextual Bandits with Graph

Proof Sketch of EXP3-LGC-U

Analyze log Wt+1(Xt)Wt(Xt)

, where Wt(x) =∑

i∈V wt(x , i):

T∑t=1

E

∑i∈V

πat (i |Xt)

1− γ

(−η⟨Xt , θi,t

⟩+ η2

⟨Xt , θi,t

⟩2)

+ηγ

K(1− γ)

∑i∈V

⟨Xt , θi,t

⟩≤ E

[T∑t=1

logWt+1(Xt)

Wt(Xt)

]≤ E

−η T∑t=1

∑i∈V

πT (i |Xt)⟨Xt , θi,t

⟩− log K

.

Reorder the terms on both sides and apply some straightforward facts:

RT ≤log K

η+ 2γT + η E

T∑t=1

∑i∈V

πat (i |Xt)

⟨Xt , θi,t

⟩2

︸ ︷︷ ︸

A

.

Use graph-theoretic results form Alon et al. and properties of θi ,t tobound term A.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 10 / 17

Page 22: Adversarial Linear Contextual Bandits with Graph

Algorithm: EXP3-LGC-IX

Additional assumption: The support of θi ,t and Xt is non-negative,and elements of Xt are independent.

Main steps in each time step t:

Draw the action It ∈ [K ] proportional to the exponential weight:

πat (i∣∣Xt) ∝ wt(Xt , i) =

1

Kexp

(−ηt

t−1∑s=1

⟨Xt , θi,s

⟩);

Estimate the loss vector θi,t :

θi,t =I{i ∈ SIt ,t}qt(i

∣∣Xt) + βtΣ−1Xt

˜t(Xt , i).

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 11 / 17

Page 23: Adversarial Linear Contextual Bandits with Graph

Algorithm: EXP3-LGC-IX

Additional assumption: The support of θi ,t and Xt is non-negative,and elements of Xt are independent.

Main steps in each time step t:

Draw the action It ∈ [K ] proportional to the exponential weight:

πat (i∣∣Xt) ∝ wt(Xt , i) =

1

Kexp

(−ηt

t−1∑s=1

⟨Xt , θi,s

⟩);

Estimate the loss vector θi,t :

θi,t =I{i ∈ SIt ,t}qt(i

∣∣Xt) + βtΣ−1Xt

˜t(Xt , i).

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 11 / 17

Page 24: Adversarial Linear Contextual Bandits with Graph

Algorithm: EXP3-LGC-IX

Additional assumption: The support of θi ,t and Xt is non-negative,and elements of Xt are independent.

Main steps in each time step t:

Draw the action It ∈ [K ] proportional to the exponential weight:

πat (i∣∣Xt) ∝ wt(Xt , i) =

1

Kexp

(−ηt

t−1∑s=1

⟨Xt , θi,s

⟩);

Estimate the loss vector θi,t :

θi,t =I{i ∈ SIt ,t}qt(i

∣∣Xt) + βtΣ−1Xt

˜t(Xt , i).

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 11 / 17

Page 25: Adversarial Linear Contextual Bandits with Graph

Algorithm: EXP3-LGC-IX

Additional assumption: The support of θi ,t and Xt is non-negative,and elements of Xt are independent.

Main steps in each time step t:

Draw the action It ∈ [K ] proportional to the exponential weight:

πat (i∣∣Xt) ∝ wt(Xt , i) =

1

Kexp

(−ηt

t−1∑s=1

⟨Xt , θi,s

⟩);

Estimate the loss vector θi,t :

θi,t =I{i ∈ SIt ,t}qt(i

∣∣Xt) + βtΣ−1Xt

˜t(Xt , i).

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 11 / 17

Page 26: Adversarial Linear Contextual Bandits with Graph

Property of θi ,t

Claim

The loss vector estimate θi ,t in EXP3-LGC-IX is optimistic:

Et

[∑i∈V

πat (i |Xt)⟨Xt , θi ,t

⟩∣∣∣∣∣Xt

]

=∑i∈V

πat (i |Xt) 〈Xt , θi ,t〉 − βt∑i∈V

πat (i |Xt)

qt(i |Xt) + βt〈Xt , θi ,t〉,

where

0 ≤∑i∈V

πat (i |Xt)

qt(i |Xt) + βt〈Xt , θi ,t〉 ≤

∑i∈V

πat (i |Xt)

qt(i |Xt) + βt.

Control the variance of loss vector estimates.

Regret can be evaluated using θi ,t directly.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 12 / 17

Page 27: Adversarial Linear Contextual Bandits with Graph

Main Results II

Theorem

Setting βt =√

logK/(K +∑t−1

s=1 Qs) and

ηt =√

logK/(dK + d∑t−1

s=1 Qs), the expected regret of EXP3-LGC-IX

satisfies:

RT ≤ 2(1 +√d)E

√√√√(K +

T∑t=1

Qt

)logK

,for both directed and undirected graph settings, where

Qt = 2α(Gt) log(

1 + dK2/βte+Kα(Gt)

)+ 2.

Corollary

The regret of EXP3-LGC-IX is O(√α(G )dT logK log(KT )) for both

directed and undirected graph settings.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 13 / 17

Page 28: Adversarial Linear Contextual Bandits with Graph

Proof Sketch of EXP3-LGC-IX

Analyze logW ′

t+1(Xt)

Wt(Xt), where Wt(x) =

∑i∈V wt(x , i) and

W ′t (x) =

∑i∈V

1K exp(−ηt−1

∑t−1s=1〈x , θi ,s〉):

− E[

log K

ηT+1

]− E

[T∑t=1

⟨Xt , θπT (Xt ),t

⟩]≤ E

[T∑t=1

(log Wt+1(Xt)

ηt+1−

log Wt(Xt)

ηt

)]

≤ E

[T∑t=1

1

ηtlog

W ′t+1(Xt)

Wt(Xt)

]≤ E

T∑t=1

∑i∈V

πat (i |Xt)

(−⟨Xt , θi,t

⟩+

1

2ηt⟨Xt , θi,t

⟩2)

︸ ︷︷ ︸B

.

Bound term B using properties of θi ,t and graph-theoretic result fromKocak et al..Reorder the terms on both sides:

RT ≤ 2(1 +√d)E

√√√√(K +

T∑t=1

Qt

)log K

.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 14 / 17

Page 29: Adversarial Linear Contextual Bandits with Graph

Proof Sketch of EXP3-LGC-IX

Analyze logW ′

t+1(Xt)

Wt(Xt), where Wt(x) =

∑i∈V wt(x , i) and

W ′t (x) =

∑i∈V

1K exp(−ηt−1

∑t−1s=1〈x , θi ,s〉):

− E[

log K

ηT+1

]− E

[T∑t=1

⟨Xt , θπT (Xt ),t

⟩]≤ E

[T∑t=1

(log Wt+1(Xt)

ηt+1−

log Wt(Xt)

ηt

)]

≤ E

[T∑t=1

1

ηtlog

W ′t+1(Xt)

Wt(Xt)

]≤ E

T∑t=1

∑i∈V

πat (i |Xt)

(−⟨Xt , θi,t

⟩+

1

2ηt⟨Xt , θi,t

⟩2)

︸ ︷︷ ︸B

.

Bound term B using properties of θi ,t and graph-theoretic result fromKocak et al..

Reorder the terms on both sides:

RT ≤ 2(1 +√d)E

√√√√(K +

T∑t=1

Qt

)log K

.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 14 / 17

Page 30: Adversarial Linear Contextual Bandits with Graph

Proof Sketch of EXP3-LGC-IX

Analyze logW ′

t+1(Xt)

Wt(Xt), where Wt(x) =

∑i∈V wt(x , i) and

W ′t (x) =

∑i∈V

1K exp(−ηt−1

∑t−1s=1〈x , θi ,s〉):

− E[

log K

ηT+1

]− E

[T∑t=1

⟨Xt , θπT (Xt ),t

⟩]≤ E

[T∑t=1

(log Wt+1(Xt)

ηt+1−

log Wt(Xt)

ηt

)]

≤ E

[T∑t=1

1

ηtlog

W ′t+1(Xt)

Wt(Xt)

]≤ E

T∑t=1

∑i∈V

πat (i |Xt)

(−⟨Xt , θi,t

⟩+

1

2ηt⟨Xt , θi,t

⟩2)

︸ ︷︷ ︸B

.

Bound term B using properties of θi ,t and graph-theoretic result fromKocak et al..Reorder the terms on both sides:

RT ≤ 2(1 +√d)E

√√√√(K +

T∑t=1

Qt

)log K

.Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 14 / 17

Page 31: Adversarial Linear Contextual Bandits with Graph

Summary

RegretUnsdirected Setting Directed Setting

EXP3-LGC-U O(√

(K + α(G)d)T log K) O(√

(K + α(G)d)T log(KdT ))

EXP3-LGC-IX O(√α(G )dT logK log(KT ))

RobustLinEXP32 O(T 2/3(Kd logK )1/3)

RealLinEXP32 O(√dKT logK logT )

2Neu, G. and Olkhovskaya, J. (2020). Efficient and robust algorithms for adversarial linear contextual bandits. In

Conference on Learning Theory, pages 1 - 20.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 15 / 17

Page 32: Adversarial Linear Contextual Bandits with Graph

Numerical Results

1 2

3 4

65

7

89

10

(a) Graph structure.

0 1 2 3 4 5 6 7 8 9 10

Time ×104

0

1000

2000

3000

4000

5000

Re

gre

t*

EXP3_LGC_U

EXP3_LGC_U*

EXP3_LGC_IX

EXP3_LGC_IX*

RobustLinEXP3

(b) Regret comparison.

Experiment settings:

K = 10, d = 10, and T = 105;

Xt(j) ∼ Bern(0.5)/sqrt(d);

θi,t(j) = 0.1i | cos t|/(√ddt/50000e).

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 16 / 17

Page 33: Adversarial Linear Contextual Bandits with Graph

Thank You!

Full version of Adversarial Linear Contextual Bandits withGraph-Structured Side Observations is available online at:

https://arxiv.org/abs/2012.05756.

Lingda Wang (UIUC) Adversarial Graphical Contextual Bandits 17 / 17