combining game theory and bayesian optimization to solve many … · 2018-04-16 · nice link to...
Post on 11-Aug-2020
0 Views
Preview:
TRANSCRIPT
Combining game theory and Bayesian optimization tosolve many-objective problems
Victor Picheny (INRA), Mickael Binois (Argonne national lab.),Abderrahmane Habbal (Universite Cote d’Azur)
Mascot-Num, Nantes
March 23rd, 2018
Introduction
Bayesian optimization context
“Black-box” model, multiple outputs
x ∈ X ⊂ Rd Black-box
y1...
yp
Working hypotheses: expensive to compute, complex yi ’s
non-convex
no derivatives available
possibly observed in noise
2 ≤ p ≤ 20
X: typically a box of dimension 2 ≤ d ≤ 50
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 2 / 37
Introduction
Multi-objective optimization
minx∈X y1 (x)
...minx∈X yp (x)
Ensemble of non-dominatedsolutions
Pareto front (objective space) andPareto set (design space)
0.2 0.4 0.6 0.8
0.0
0.2
0.4
0.6
0.8
1.0
Y(1)
Y(2
)Classical BO algorithm objective
Obtain a good discrete approximation of the Pareto set = non-dominatedsolutions along the entire Pareto front
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 3 / 37
Introduction
Challenging situation: many objectives / restricted budget
Accurate Pareto set may not be attainable
Many objectives ⇒ very large Pareto set
Not enough budget to cover all the solutions
Accurate Pareto set may not be desirable
Too many solutions to choose from
Our proposition: use game theory to seek compromise solutions
Game theory literature
Extensive for small discrete problems or convex objectives, scarce forexpensive black-boxes⇒ Explore BO alternatives for solving game equilibrium problems
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 4 / 37
Games and equilibria
Outline
1 Introduction
2 Games and equilibria
3 Solving games with Bayesian Optimization
4 Application to model calibration
5 Conclusion
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 5 / 37
Games and equilibria
Game Theory and equilibrium problems
Game theory is the study of mathematical models of conflict andcooperation between intelligent rational decision-makers
Initially from economics
Now common in artificial intelligence, engineering or machine learning
Game
Multiple decision makers (players) with antagonistic goals
Acceptable compromise = game equilibrium
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 6 / 37
Games and equilibria
Nash equilibrium problems (NEP)
NEP: canonical antagonist game
Each player tries to solve his optimization problem:
(Pi ) minxi∈Xi
yi (x), 1 ≤ i ≤ p
with x = [x1, . . . , xp] ∈ X. We assume here some territory splitting:
X = X1 × . . .× Xp
Definition: writing x = (xi , x−i ) (no actual permutation), x∗ is a NE if:
∀i , 1 ≤ i ≤ p, x∗i = arg minxi∈Xi
yi (xi , x∗−i )
⇒ When everyone plays a NE, then no player has incentive to move from it
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 7 / 37
Games and equilibria
An engineering example: multidisciplinary optimization
Each team makes a decision with antagonistic goals
Natural territory splitting: each team acts on its own set of variables
J.A. Desideri (2012), Cooperation and competition in multidisciplinary optimization,Comp. Optim. and Applications
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 8 / 37
Games and equilibria
A PDE example: data completion
Steady-state heat equation
∇.(k∇u) = 0 in Ω
u = f on Γc (Dirichlet)k∇u.ν = ψ on Γc (Neumann)
Boundary partially unobservable
∂Ω = Γc + Γi
Data to recover: u|Γiand k∇u.ν|Γi
Nash game: Dirichlet vs. Neumann
A. Habbal and M. Kallel (2013), Neumann-Dirichlet Nash strategies for the solution ofelliptic Cauchy problems, SIAM J. Control Optim
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 9 / 37
Games and equilibria
The Kalai-Smorodinsky solution
Disadvantages of Nash equilibria
Assumes some territory splitting
In general not efficient (in the Pareto sense)
The Kalai-Smorodinsky (KS) idea
Players start negociation from a disagreement point
Progress towards an efficient solution while ensuring equity ofmarginal gains
KS solution
Intersection of the disagreement point - ideal point (= utopia or shadow)straight line with the Pareto front
Kalai, Smorodinsky (1975). Other solutions to Nash’s bargaining problem, Econometrica
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 10 / 37
Games and equilibria
Illustration: 2 objectives, 2 variables
y1
x y
z
y2
x y
z
0 50 100 150 200 250 300
−35
−30
−25
−20
−15
−10
−5
Objective space
y1
y 2
Here: disagreement point = Nadir
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 11 / 37
Games and equilibria
Some properties
For 2 players case, KS is proved to be the unique bargaining solutionwhich fulfills the following
Axioms
Pareto optimality
Symmetry
Invariance w.r.t. affine transformations
Restricted Monotonicity
For N players
Efficiency, Symmetry, Affine invariance
Effective selection device even when combined with refinementconcepts based on stability with respect to perturbations [1].
[1] De Marco G. and Morgan J. (2010). Kalai-Smorodinsky bargaining solution equilibria. JOTA, 145(3), 429-449.
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 12 / 37
Games and equilibria
Going further: robust KS using copula
Motivation: unlike Nash, KS is sensitive to non-linear rescaling
0 50 100 200 300
−35
−25
−15
−5
Original space
Y1
Y2
−1 0 1 2 3 4 5
−12
00−
800
−40
00
Monotonic transformation
log(Y1)
Y2^
2
Why copulas?
Nice link to Pareto optimality (see next slide)
Insensitive to non-linear transformations on the marginals
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 13 / 37
Games and equilibria
Copulas and Pareto optimality
Pareto optimality and multivariate statistics
Take: Y = (y1(X ), . . . , yp(X )) , X ∼ U(X)
Pareto front = (part of the) zero level-line of themultivariate CDF FY
Multivariate CDF = marginal densities + copulas
FY (y) = C (F1(y1), . . . ,Fp(yp)) −2 0 2 4
−3
−2
−1
01
23
x
y
with C : Rp → R copula, Fi = P(Yi ≤ yi ) marginal densities
M. Binois, D. Rulliere, D., O. Roustant,On the estimation of Pareto fronts from the point of view of copula theoryInformation Sciences (2015) 324: 270-285.
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 14 / 37
Games and equilibria
Our proposition: KS in the copula space (CKS)
Assuming a distribution for X (e.g. X ∼ U(X))
We look for the KS of F1(y1(X )), . . . ,Fp(yp(X ))
Original space Copula space
−30
−20
−10
0 100 200 300
Y1
Y2
0.00
0.25
0.50
0.75
1.00
0.00 0.25 0.50 0.75 1.00
F1F
2
Bonus of copulas: since Fi [min yi (X )] = 0, Fi [max yi (X )] = 1
Utopia point is (0 . . . 0)
Worst solution for all objectives is (1 . . . 1) ⇒ new disagreement point
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 15 / 37
Games and equilibria
KS in the copula space (CKS)
0 50 100 200 300
−35
−25
−15
−5
Original space
Y1
Y2
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Copula space
F1
F2
On discrete spaces: rank space
CKS = Pareto optimal solution with closests ranks among objectives
Invariant by any (strictly) monotonic transformation of marginals
... but depends on X ’s distribution
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 16 / 37
BO and games
Outline
1 Introduction
2 Games and equilibria
3 Solving games with Bayesian Optimization
4 Application to model calibration
5 Conclusion
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 17 / 37
BO and games
Bayesian optimization: two ingredients
Quick-to-evaluate surrogate of the objective: Gaussian processregression
Sequential sampling via maximizing acquisition functions
Initial set of
observations
Objectives evaluation
Surrogate building (nested loop)
Acq. func. maximization (nested loop)
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 18 / 37
BO and games
Gaussian process regression
Prior π : y realization of a GP Y
Entirely defined by its mean Eπ(Y (x)) and covariance covπ(Y (x),Y (x′))
Regression model (y) = law of the GP conditioned on observations
0.0 0.2 0.4 0.6 0.8 1.0
−5
05
1015
x
y(x)
(Parametric form assumed for mean and covariance, estimation by maximum likelihood)
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 19 / 37
BO and games
Sequential sampling (EGO algorithm)
The acquisition function balances between exploration and exploitation
0.0 0.2 0.4 0.6 0.8 1.0
−15
−10
−5
05
1015
x
f(x)
01
23
J(x)
0.0 0.2 0.4 0.6 0.8 1.0−
15−
10−
50
510
15
x
f(x)
0.0
0.1
0.2
0.3
0.4
0.5
0.6
J(x)
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 20 / 37
BO and games
BO and equilibria
Regression: one GP model for each obj yi (correlation is neglected)
GP is used to “denoise” the objectives (if needed)
Multivariate regressor: Y(.) ∼ GP (µ(.),Σ (., .)) with Σ diagonal.
Sequential sampling boils down to finding a “good” acquisition function
⇒ Based on Y(.), which xnew should I visit next?
Canonical acquisitions: Expected Improvement, Upper Confidence Bound
Not usable here due to the complex learning task (measure of progress?)
Nash: first-order stationarity
Kalai-Smorodinsky: part of the Pareto front + Nadir + Shadow
Copula-Kalai-Smorodinsky: marginals + copula
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 21 / 37
BO and games
What we will use: stepwise uncertainty reduction
Villemonteix, Vazquez, Walter (2009) An informational approach to the globaloptimization of expensive-to-evaluate functions, J. of Glob. Opt.
Bect, Ginsbourger, Li, Picheny, Vazquez, (2012) Sequential design of computerexperiments for the estimation of a probability of failure, Statistics and Computing.
Hernandez-Lobato, Hoffman, Ghahramani, (2014) Predictive entropy search for efficientglobal optimization of black-box functions, NIPS
...
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 22 / 37
BO and games
What do we know about the equilibrium?
Computing equilibria on discrete sets is easy
The search for Nash, KS, or CKS can be done by (smart) exhaustivesearch on a grid.
What we can do
Discretize X (grid, LHS...)
Draw GP samplesY1, . . . ,YM of Y(Xdisc)
Get (quickly) equilibriumof each Y i
y1
x y
z
y2
x y
z
x y
z
x y
z
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 23 / 37
BO and games
What do we know about the equilibrium?
GP sample ⇒ cloud of potential solutions (+)
x y
z
x y
z
0 50 100 150 200 250 300
−35
−30
−25
−20
−15
−10
−5
Iteration 1
f1
f 2
Actual EqSimulated EqCurrent obsNew obs
⇒ We want to choose xnew so that the cloud shrinks!
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 24 / 37
BO and games
Formalizing: Stepwise uncertainty reduction (SUR)
Random equilibrium
y(x∗) = Ψ(y) (Nash, KS or CKS) equilibrium for cost function y
Y random (GP) ⇒ Ψ(Y) random ( → the +’s)
Uncertainty measure of equilibrium ≡ “volume” of Ψ(Y):
Γ(Y) = det [cov (Ψ(Y))]
⇒ Γk if Y conditioned on observations (x1, y1), . . . , (xk , yk )
Greedy (stepwise) uncertainty reduction
Optimal choice: xk+1 = arg min Γk+1
Out of reach without y(xk+1)!
Acquisition function: expected reduction
J(xk+1) = EFk+1(Γ [Y|Fk+1 = Y(xk+1)])
with Y(xk+1) ∼ GP (µ(xk+1),Σ (xk+1, xk+1))
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 25 / 37
BO and games
Computing the acquisition function
Acquisition function
J(xnew ) = EFnew (Γ [Y|Fnew = Y(xnew )])with Y(xnew ) Gaussian
Two-layer Monte Carlo
Y(new)1 , . . . ,Y(new)
K observation drawings at xnew
We condition the paths by these observations:
Ym|Y(new)k
J(xnew ) ≈
1
K
K∑k=1
det[cov
(Y1|Y(new)
k , . . . ,YM |Y(new)k
)]
Y1, …, YK
Y1 Y1new, …, YK Y1
new
Y1
new
Y1 Y2new, …, YK Y2
new
Y2
new
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 26 / 37
BO and games
Illustration: iteration 1, 3 and 7 (KS search)
0 50 100 150 200 250 300
−35
−30
−25
−20
−15
−10
−5
Iteration 1
f1
f 2
Actual EqSimulated EqCurrent obsNew obs
0 50 100 150 200 250 300
−35
−30
−25
−20
−15
−10
−5
Iteration 3
f1
f 2
Actual EqSimulated EqCurrent obsNew obs
0 50 100 150 200 250 300
−35
−30
−25
−20
−15
−10
−5
Iteration 7
f1
f 2
Actual EqSimulated EqCurrent obsNew obs
With only 13 points: KS identified, very small residual uncertainty
Trade-off between learning the center and the extremities of thePareto front
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 27 / 37
BO and games
In practice: Many numerical tricks are necessary...
Efficient data structure and discrete games solver
Use of parallel computationFast algorithm for drawing and updating simulated GP paths
Chevalier, Emery, Ginsbourger (2015) Fast update of conditional simulationensembles, Mathematical Geosciences
Small discretized space by choosing only useful points(⇒ shorter paths, smaller games)
Optimization of the acquisition function only over the discretizedspace
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 28 / 37
Application to model calibration
Outline
1 Introduction
2 Games and equilibria
3 Solving games with Bayesian Optimization
4 Application to model calibration
5 Conclusion
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 29 / 37
Application to model calibration
li-BIM: an agent-based model for designing buildings
Simulated behavior of occupants in a building
Numerical modeling of the building: thermal, air quality, lighting, etc.
Evolved occupational cognitive model (Belief-Desire-Intention)
F. Taillandier, A. Micolier, P. Taillandier (2017). Li-BIM (Version 1.0.0), CoMSESComputational Model Library
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 30 / 37
Application to model calibration
The li-BIM calibration problem
Many parameters to tune
Behavioral: sensitivity to hot, cold, air quality, hunger, tidiness...
Appliances characteristics: electrical power, efficiency...
Model should match real data (records or surveys)
Electrical consumption
Average temperature, air quality
Time spent on various activities
A challenging optimization problem
11 parameters, 10 objectives
Expensive, stochastic model (30min / simulation)
Outputs of different nature → how to define an acceptable compromise?
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 31 / 37
Application to model calibration
Experimental setup
BO configuration
100-point LHS + 100 infills
Space discretization: 1000 points (renewed at each iteration)
Choosing the next point ≈ 2 min
Objectives: square errors w.r.t. targets (averaged over 8 repetitions)
KS + CKS (with empirical copula)
Preliminary result
Out of the 300 points, only 5 were dominated
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 32 / 37
Application to model calibration
Results - objective space (% error)
0
100
200
300
Conso_chauffage_tot
0
50
100
Conso_ECS_tot
0
20
40
60
Conso_cuisson_tot
0
30
60
90
Conso_autres_tot
5
10
Temp_int_moy_hiver
2
4
6
Temp_int_moy_ete
50
100
150
Nb_h_moy_cuisiner
0
20
40
60
80Nb_h_moy_relax
0
10
20
30
40
Nb_h_moy_dormir
0
20
40
60
Nb_h_moy_sortir
DOECKSKS
Apparent trade-off between exploration and exploitationCKS more exploratory and sensitive to local mass on marginals
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 33 / 37
Application to model calibration
Results - objective space (2D)
Conso_chauffage_tot
Conso_ECS_tot
Conso_cuisson_tot
Conso_autres_tot
Temp_int_moy_hiver
Temp_int_moy_ete
Nb_h_moy_cuisiner
Nb_h_moy_relax
Nb_h_moy_dormir
Nb_h_moy_sortir
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 34 / 37
Conclusion
Outline
1 Introduction
2 Games and equilibria
3 Solving games with Bayesian Optimization
4 Application to model calibration
5 Conclusion
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 35 / 37
Conclusion
Summary and future steps
Ingredients
Game theory concepts to define compromise solutions
GPs + Stepwise Uncertainty Reduction to find them
⇒ Parsimonious algorithm to tackle automatically many black-boxobjectives
SUR is very accomodating, so why not also...
Generalized Nash problems (i.e. with constraints)
KS with smart disagreement points: e.g. Nash-Kalai-Smorodinsky
Batch-sequential strategies
Smart use of repetitions (for stochastic simulators)
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 36 / 37
Conclusion
References
Want to give it a try? R package GPGame
https://CRAN.R-project.org/package=GPGame
Implements Nash, KS, CKS + Nash-KS (experimental)
Want to know more?
V. Picheny, M. Binois, A. HabbalA Bayesian optimization approach to find Nash equilibria (2017+)preprint: https://arxiv.org/abs/1611.02440
M. Binois, V. Picheny, A. HabbalThe Kalai-Smorodinsky solution for many-objective Bayesian optimization (2017+)NIPS BayesOpt workshop, https://bayesopt.github.io/papers/2017/28.pdf
V. Picheny, M. Binois, A. Habbal Bayesian optimization and game theory March 23rd, 2018 37 / 37
top related