optimal control of the fokker-planck equation · optimal control of the fokker-planck equation ......

Johann Radon Institute for Computational and Applied Mathematics

Optimal Controlof the Fokker-Planck equation

Roberto Guglielmi

Johann Radon Institute for Computational and Applied Mathematics (RICAM)

Austrian Academy of Sciences (OAW)

Linz, Austria

INdAM Meeting OCERTO 2016

Optimal Control for Evolutionary PDEs and Related Topics

Cortona, Italy

June 20-24, 2016

roberto.guglielmi@ricam.oeaw.ac.at R. Guglielmi (RICAM), Optimal Control of the FP equation

Motivation

Consider an Optimal Control Problem (OCP)

J(X , u)

constrained to a Ito Stochastic Differential Equation (SDE)

dXt = b(Xt , t; u)dt + σ(Xt , t)dWt , X (0) = x0

t ∈ [0,TE ] for a fixed terminal time TE > 0 and

Xt is a random variable representing the state of the SDE

The control u acts through the drift term b

Xt random ⇒ The cost functional J(X , u) is a random variable

Motivation

J(X , u)

Motivation

J(X , u)

Choice of the cost functional

Standard approach: consider the averaged objective

E[J(X , u)] = minu

E[∫ TE

0L (t,Xt , u(t)) dt + ψ(XTE

Alternative approach: express the objective in terms of theProbability Density Function (PDF) associated with the state Xt ,which characterize the shape of its statistical distribution.

Related works: deterministic objectives defined by

- the Kullback-Leibler distance (G. Jumarie 1992, M. Karny 1996) or

- the square distance (M.G. Forbes, M. Guay, J.F. Forbes 2004,

Wang 1999)

between the state PDF and a desired one.

However, stochastic models are needed to obtain the PDFby averaging or by interpolation

E[J(X , u)] = minu

E[∫ TE

]Alternative approach: express the objective in terms of theProbability Density Function (PDF) associated with the state Xt ,which characterize the shape of its statistical distribution.

Wang 1999)

E[J(X , u)] = minu

E[∫ TE

]Alternative approach: express the objective in terms of theProbability Density Function (PDF) associated with the state Xt ,which characterize the shape of its statistical distribution.

Wang 1999)

The Fokker-Planck Equation /1

New approach pursued by Annunziato and Borzı (2010, 2013):Reformulate the objective using the underlying PDF

y(x , t) :=

∫Ωy(x , t; z , 0)ρ(z , 0)dz

t > 0, ρ(z , 0) given initial density probability,y transition density probability distribution function

y(x , t; z , s) := PX (t) ∈ (x , x + dx) : X (s) = z , t > s

and control the PDF directly.

The next essential step:

the PDF evolves according to the Fokker-Planck Equation(or forward Kolmogorov Equation)

New approach pursued by Annunziato and Borzı (2010, 2013):Reformulate the objective using the underlying PDF

y(x , t) :=

∫Ωy(x , t; z , 0)ρ(z , 0)dz

t > 0, ρ(z , 0) given initial density probability,y transition density probability distribution function

y(x , t; z , s) := PX (t) ∈ (x , x + dx) : X (s) = z , t > s

and control the PDF directly.

The next essential step:

the PDF evolves according to the Fokker-Planck Equation(or forward Kolmogorov Equation)

Fokker-Planck Equation

∂ty(x , t)− 1

2∂2xx

(σ(x , t)2y(x , t)

)+ ∂x

(b(x , t; u)y(x , t)

y(·, 0) = y0

where y : R× [0,∞[→ R≥0 is the PDF

constrained to∫R y(x , t)dx = 1 ∀t > 0,

y0 : R→ R≥0 is the initial PDF(∫

R y0(x)dx = 1),

σ : R× [0,∞[→ R and b : R× [0,∞[×R→ Rare given by the SDE

Fokker-Planck Equation

∂ty(x , t)− 1

2∂2xx

(σ(x , t)2y(x , t)

)+ ∂x

(b(x , t; u)y(x , t)

y(·, 0) = y0

where y : R× [0,∞[→ R≥0 is the PDF

constrained to∫R y(x , t)dx = 1 ∀t > 0,

y0 : R→ R≥0 is the initial PDF(∫

R y0(x)dx = 1),

σ : R× [0,∞[→ R and b : R× [0,∞[×R→ Rare given by the SDE

A Fokker-Planck control framework

Deterministic PDE-constrained Optimal Control Problem

E[J(X , u)] minu

J(y , u)

s.t. Ito SDE s.t. Fokker-Planck PDE

dXt = b(u)dt + σdWt , yt − 12 (σ2y)xx + (b(u)y)x = 0

1) the class of objectives described by minu

J(y , u) is larger than

that expressed by minu

E[J(X , u)], indeed

TE∫0

L(t,Xt , u(t)) + ψ(XTE)

TE∫0

L(t, x , u)y(t, x)+

ψ(x)y(TE , x) .

2) Bilinear control through the coefficient of the divergence term

E[J(X , u)] minu

J(y , u)

E[J(X , u)], indeed

TE∫0

L(t, x , u)y(t, x)+

ψ(x)y(TE , x) .

E[J(X , u)] minu

J(y , u)

E[J(X , u)], indeed

TE∫0

L(t, x , u)y(t, x)+

ψ(x)y(TE , x) .

MPC for the Fokker-Planck equation

In order to build a real-time sub-optimal control feedback law weapply a Model Predictive Control scheme, also known as RecedingHorizon scheme, to the Fokker-Planck equation.

Basic idea of MPC:

OCP on a long Several iterative OCPs(possibly infinite) on (shorter) finite

time horizon time horizons

Related References

Stochastic optimal control with cost functional expressed bythe PDF: Jumarie 1992, Karny 1996, Wang 1999,Forbes, Forbes, Guay 2004

FP-MPC approach: Annunziato and Borzı 2010, 2013, . . .

Existence of optimal controls for bilinear controls:Addou and Benbrik 2002, for a control u = u(t)

Analysis of the FPE with general coefficients: Aronson 1968,Risken 1989, Figalli 2008, Le Bris and Lions 2008,Porretta 2015

Controllability of the FPE: Blaquiere 1992, Porretta 2014

Connection with mean field type control:Bensoussan, Frehse, Yam 2013

Existing Work by Annunziato and Borzı, 2010, 2013

Track a desired PDF over a given time interval.

Optimal Control Problem

Ω ⊂ R open, ua, ub ∈ R with ua < ub, yd ∈ L2(Ω) and λ > 0.Consider the following OCP on [0,T ]:

J(y , u) :=1

2‖y(·,T )− yd(·,T )‖2

L2(Ω) +λ

s.t.∂ty − 1

2∂2xx

)+ ∂x (b(u)y) = 0 in Ω× (0,T )

y(·, 0) = yn in Ωy = 0 in ∂Ω× (0,T )

u ∈ Uad := u ∈ R |ua ≤ u ≤ ubb(x ; u) := γ(x) + u with γ ∈ C 1(Ω)

Existing Work by Annunziato and Borzı, 2010, 2013

Track a desired PDF over a given time interval.

Optimal Control Problem

Ω ⊂ R open, ua, ub ∈ R with ua < ub, yd ∈ L2(Ω) and λ > 0.Consider the following OCP on [0,T ]:

J(y , u) :=1

2‖y(·,T )− yd(·,T )‖2

L2(Ω) +λ

s.t.∂ty − 1

2∂2xx

)+ ∂x (b(u)y) = 0 in Ω× (0,T )

y(·, 0) = yn in Ωy = 0 in ∂Ω× (0,T )

u ∈ Uad := u ∈ R |ua ≤ u ≤ ubb(x ; u) := γ(x) + u with γ ∈ C 1(Ω)

Space-Time Dependent Controls

Collaboration with Arthur Fleig and Lars Grune, Univ. BayreuthConsider a control function u(x , t)

Proposition (Aronson)

Let the following assumptions hold:

σ ∈ C 1(Ω) such that

σ(x , t) ≥ θ ∀(x , t) ∈ Q , for some constant θ > 0

b ∈ Lq(0,TE ; Lp(Ω)) with 2 < p, q ≤ ∞ and d2p + 1

q <12 .

y0 ∈ L2(Ω) is nonnegative and bounded.

Then there exists a unique nonnegative weak solution y to theFokker-Planck IBVP (1)

Space-Time Dependent Controls

Collaboration with Arthur Fleig and Lars Grune, Univ. BayreuthConsider a control function u(x , t)

Proposition (Aronson)

Let the following assumptions hold:

σ ∈ C 1(Ω) such that

σ(x , t) ≥ θ ∀(x , t) ∈ Q , for some constant θ > 0

b ∈ Lq(0,TE ; Lp(Ω)) with 2 < p, q ≤ ∞ and d2p + 1

q <12 .

y0 ∈ L2(Ω) is nonnegative and bounded.

Then there exists a unique nonnegative weak solution y to theFokker-Planck IBVP (1)

Towards existence of Optimal Solutions

Let Ω ⊂ Rd , d ≥ 1, H := L2(Ω),V := H10 (Ω) and V ′ dual of V .

The Fokker-Planck equation E(y0, u, f ) can be rewritten asy(t) + Ay(t) + div(b(t, u(t))y(t)) = f (t) in V ′ , t ∈ (0,T )y(0) = y0 ,

where y0 ∈ H, A : V → V ′ linear and continuous, f ∈ L2(0,T ;V ′),b : Rd+1 × U → Rd , (x , t; u) 7→ b(x , t; u(x , t)) satisfies

d∑i=1

|bi (x , t; u)|2 ≤ M(1 + |u(x , t)|2) ∀x ∈ Ω , ∀t ∈ [0,T ] ,

and ∀u ∈ U := Lq(0,T ; L∞(Ω;Rd)), for some q > 2,

Rmk: optimization problem in a Banach space

Towards existence of Optimal Solutions

Let Ω ⊂ Rd , d ≥ 1, H := L2(Ω),V := H10 (Ω) and V ′ dual of V .

The Fokker-Planck equation E(y0, u, f ) can be rewritten asy(t) + Ay(t) + div(b(t, u(t))y(t)) = f (t) in V ′ , t ∈ (0,T )y(0) = y0 ,

where y0 ∈ H, A : V → V ′ linear and continuous, f ∈ L2(0,T ;V ′),b : Rd+1 × U → Rd , (x , t; u) 7→ b(x , t; u(x , t)) satisfies

d∑i=1

|bi (x , t; u)|2 ≤ M(1 + |u(x , t)|2) ∀x ∈ Ω , ∀t ∈ [0,T ] ,

and ∀u ∈ U := Lq(0,T ; L∞(Ω;Rd)), for some q > 2,

Rmk: optimization problem in a Banach space

A-priori estimates

Let y0 ∈ H, f ∈ L2(0,T ;V ′) and u ∈ U . Then a solution y of theFokker-Planck equation satisfies the estimates

|y |2L∞(0,T ;H) ≤ Cec|u|2U

(|y(0)|2H + |f |2L2(0,T ;V ′)

|y |2L2(0,T ;V ) ≤ C (1 + |u|2Uec|u|2U )(|y(0)|2H + |f |2L2(0,T ;V ′)

|y |2L2(0,T ;V ′) ≤ C (1 + |u|2Uec|u|2U )(|y(0)|2H + |f |2L2(0,T ;V ′)

for some positive constants c , C .

Existence of Optimal Controls

Theorem (Fleig - G.)

Let y0 ∈ V , yd ∈ H , ua , ub ∈ L∞(0,T ; L∞(Ω;Rd)) ,

consider minu∈Uad

J(y , u), where y is the unique solution to

y(t) + Ay(t) + div(b(t, u(t)), y(t)) = 0 in V ′ , t ∈ (0,T )y(0) = y0 ,

Uad := u ∈ U : ua(x , t) ≤ u(x , t) ≤ ub(x , t) for a.e. (x , t) ∈ Q.Then there exists a pair

(y , u) ∈ C ([0,T ],H)× Uad

such that y is a solution of E(y0, u, 0) and u minimizes J in Uad .

Necessary Optimality Conditions

Let yd ∈ L2(0,T ;H), yΩ ∈ H, α, β, λ ≥ 0 with maxα, β > 0.

J(u) :=α

2‖y − yd‖2

L2(0,T ;H) +β

2‖y(T )− yΩ‖2

2‖u‖2

L2(0,T ;H) .

We derive the first-order necessary optimality system for u(x , t)

∂t y −d∑

i ,j=1∂2ij(aij y) +

d∑i=1

∂i(ui y)

= 0 , in Q ,

−∂t p −d∑

i ,j=1aij∂

2ij p −

d∑i=1

ui∂i p = α[y − yd ] , in Q ,

y = p = 0 on Σ ,

y(0) = y0 , p(T ) = β[y(T )− yΩ] , in Ω ,∫∫Q

[y∂i p + λui ] (ui − ui ) dxdt ≥ 0 ∀u ∈ Uad , i = 1, . . . , d .

Numerical Example

Consider the Ornstein-Uhlenbeck process with

σ(x , t) ≡ σ = 0.8, b(x , t, u) := u − x

on Ω :=]− 5, 5[ with ua = −10, ub = 10, λ = 0.001, and TE = 5.

The target and initial PDF are given by

yd(x , t) :=exp

(− [x−2 sin(πt/5)]2

2·0.22

2π · 0.22

y0(x) := yd(x , 0) =exp

(− x2

2·0.22

2π · 0.22,

respectively.

Numerical Example

Consider the Ornstein-Uhlenbeck process with

σ(x , t) ≡ σ = 0.8, b(x , t, u) := u − x

on Ω :=]− 5, 5[ with ua = −10, ub = 10, λ = 0.001, and TE = 5.

The target and initial PDF are given by

yd(x , t) :=exp

(− [x−2 sin(πt/5)]2

2·0.22

2π · 0.22

y0(x) := yd(x , 0) =exp

(− x2

2·0.22

2π · 0.22,

respectively.

Numerical Examples (2)

With space-dependent control, larger class of objectives possible:

region avoidance, without prescribing the shape of the PDF,e.g. try to force the state PDF into [0, 0.5].

Try to track non-smooth targets, e.g.

yd(x , t) :=

0.5 if x ∈ [−1 + 0.15t, 1 + 0.15t]

0 otherwise.

Numerical Example (3)

Consider the two-dimensional stochastic process, modeling thedispersion of substance in shallow water [Heemink, 1990]

σ(x , t) :=

(√2D(x) 0

with D(x) := − 164

((x1 − 4)2 + (x2 − 4)2

5 > 0 in Ω and

b(x , t; u) :=

(u1 + ∂D

∂x1− 1

u2 + ∂D∂x2− 1

(u1 − x1

32 + 140

u2 − x232 + 1

on Q := Ω× [0, 5] with Ω :=]0, 8[×]0, 8[ .

Initial distribution y0: (smoothed) delta-Dirac located at (4, 4).

Target PDF is given by

yd(x1, x2) = m

σ2log(x1)

)− 2

σ2(x1 − 1)

σ2log(x2)

)− 2

σ2(x2 − 1)

]with C1 = 2.625,C2 = 2.125, σ = 0.5,m ≈ 0.00004591595108.(Equilibrium PDF of a stochastic Lotka-Volterra two-speciesprey-predator model [Yeung and Stewart, 2007])

Other parameters: sampling time T = 0.5, regularization parameterλ = 0.001, and control bounds ua = −10, ub = 10.

yd(x1, x2) = m

σ2log(x1)

)− 2

σ2(x1 − 1)

σ2log(x2)

)− 2

σ2(x2 − 1)

yd(x1, x2) = m

σ2log(x1)

)− 2

σ2(x1 − 1)

σ2log(x2)

)− 2

σ2(x2 − 1)

t = 0.00

u ∈ R2:

t = 0.50

u ∈ R2:

t = 1.0

u ∈ R2:

t = 1.5

u ∈ R2:

t = 2.0

u ∈ R2:

t = 2.5

u ∈ R2:

t = 3.0

u ∈ R2:

t = 3.5

u ∈ R2:

t = 4.0

u ∈ R2:

t = 4.5

u ∈ R2:

t = 5.0

u ∈ R2:

t = 0.00

t = 0.00 1.4

t = 0.00

t = 0.50

t = 0.50 1.4

t = 0.50

t = 1.0

t = 1.0 1.4

t = 1.0

t = 1.5

t = 1.5 1.4

t = 1.5

t = 2.0

t = 2.0 1.4

t = 2.0

t = 2.5

t = 2.5 1.4

t = 2.5

t = 3.0

t = 3.0 1.4

t = 3.0

t = 3.5

t = 3.5 1.4

t = 3.5

t = 4.0

t = 4.0 1.4

t = 4.0

t = 4.5

t = 4.5 1.4

t = 4.5

t = 5.0

Some remarks

– The computed optimal control of the FPE is then applied tothe stochastic process

– Right boundary conditions of Robin type(already embedded in the numerical scheme)

– Annunziato, Borzı et al. have applied the same Fokker-PlanckOptimal Control framework to

- the class of piecewise deterministic processes- optimal control of open quantum systems- subdiffusion processes

– estimates for the minimal horizon N that guarantees stabilityof the MPC closed-loop system by Fleig & Grune, 2016

Some remarks

Thank you for your attention!

optimal control of the fokker-planck equation · optimal control of the fokker-planck equation ......

Documents

statistical estimation of fokker-planck equation at fixed...

existence of global weak solutions to...

fokker-planck approximation of the master equation … ·...

variational methods for the kinetic fokker...

research article new approaches for solving fokker planck...

progress in vlasov-fokker- planck simulations of laser...

study on synchronization of coupled oscillators using the...

a numerical scheme for the quantum fokker-planck-landau...

the fokker-planck...

degenerate fokker–planck equations: bismut formula...

fokker-planck analysis of transverse collective ... ·...

fokker–planck equations for a free energy functional or...

applications of the fokker-planck equation in circuit...

on the approximation of the fokker–planck equation...

optimal control and mpc for the fokker-planck equation ·...

beyond the fokker{planck equation: pathwise control of...

a split-step solution of the fokker–planck equation for...

onsager equations, nonlinear fokker-planck...

master, fokker-planck and langevin equations

discrete-time ratchets, the fokker–planck equation and...