model discrepancy and physical parameters in calibration and ......ignore model discrepancy at your...

Model discrepancy and physical parameters incalibration and prediction of computer models

Jenný Brynjarsdóttir

Joint work with Anthony O’Hagan, University of Sheffield, UK, andJon Hobbs and Amy Braverman, Jet Propulsion Laboratory

Bayesian Inference in Subatomic PhysicsA Marcus Wallenberg Symposium, 17-20 September 2019

Jenný Brynjarsdóttir (CWRU) Model discrepancy September 17, 2019 1 / 42

Introduction

Outline

IntroductionConfounding between Model Discrepancy and parameters

Example 1: Simple machine showing the effect of modeldiscrepancy on

Estimating physical parametersInterpolation - Predicting within the scope of the dataExtrapolation

Example 2: Model discrepancy in remote sensing of CO2


Introduction

Uncertainty Quantification (UQ)for deterministic science-based models

Computer Model

η(x,θ)

6=Model

Discrepancy(MD)

Reality

ζ(x)

x: controllable inputsθ: unknown inputs

physical parameters withtrue value θ∗

tuning parameters

Obs: Zi = ζ(xi) + εii = 1, . . . ,nεi i.i.d. N(0, σ2

ε )


Introduction


Computer Model

η(x,θ)

6=Model

Discrepancy(MD)

Reality

ζ(x)

UQ objectives include:Calibration: estimate θ∗

Predict ζ(x)

interpolation orextrapolation

Obs: Zi = ζ(xi) + εii = 1, . . . ,nεi i.i.d. N(0, σ2

ε )


Introduction


Computer Model

η(x,θ)

6=Model

Discrepancy(MD)

Reality

ζ(x)

Calibration:

Zi = η(xi ,θ) + εi

εi i.i.d. N(0, σ2ε )

Ignores Model discrepancy

Kennedy & O’Hagan (2001):

Zi = η(xi ,θ) + δ(xi) + εi

Model δ(x) as a zero-meanGaussian Process


Introduction Some Statistical Preliminaries

Gaussian Process

KOH-approach:Z = η(x , θ) + δ(x) + ε

where δ(x) is a Gaussian Process

If δ(x) ∼ GP(m(x), c(x , x ′)) then for any collection x1, . . . xnδ(x1)...

δ(xn)

∼ MVN

m(x1)

...m(xn)

,c(x1, x1) · · · c(x1, xn)

.... . .

...c(xn, x1) · · · c(xn, xn)

Can get flexible (non-parametric) sample paths but still enjoy themathematical convenience of the multivariate normal distribution



Sample paths from from a Gaussian process

0 1 2 3 4

−2

01

2

ψ = 0.3

x

Rea

lizat

ions

0 1 2 3 4

−2

01

2

ψ = 1

x

Rea

lizat

ions

m(x) = 0 ∀x and c(x1, x2|ψ) = exp

(−(

x1 − x2

ψ

)2)



Estimating Parameters (θ) the Bayesian way

Observations: z1, z2, . . . , zn

Model: Zi = η(xi ,θ) + δ(xi) + εi −→ Determines p(z | θ)

Put a prior distribution on unknown parameters: p(θ)

Find posterior distribution via Bayes Theorem

p(θ | z) =p(z | θ)p(θ)∫p(z | θ)p(θ)dθ

Posterior = Conditional distribution of parameter given the dataDetermined by the likelihood and the prior

Posterior: summarizes our uncertainty about parameter values,given all the information we have


Confounding between Model Discrepancy and parameters

Confounding between θ and δ(·)

KOH-approach:Z = η(x , θ) + δ(x) + ε

For any θ there is a δ(x) that gives us ζ(x):

ζ(x) = η(x , θ) + δ(x)

⇒ θ and δ(x) are not identifiable.Even if we learn ζ(x) well (large sample size), calibration onlygives a joint posterior distribution for θ and δ(·) over the manifold

Mζ = {(θ, δ(·)) : ζ(x) = η(x , θ) + δ(x); x ∈ Xobs}

Need more information⇒ need to think carefully about priors on θand/or δ(x)


Confounding between Model Discrepancy and parameters


KOH-approach:

ζ(x) = η(θ, x) + δ(x)

Pick any θ = t ,then set

δ(x) = ζ(x)− η(t , x)

for all xModel discrepancy (δ)

Par

amet

er (

θ)

Prior

Posterior


Example 1: Simple Machine

The Simple Machine

Produces work according to the amount of effort put into it

Computer model:

η(x , θ) = θx

Does not accountfor losses due tofriction, etc.

θ has physicalmeaning:

efficiency in africtionless worldgradient at zerotrue value:θ∗ = 0.65

0 1 2 3 4 5 6

01

23

4

x (effort)

y (w

ork)

●

Simple Machine η(x, 0.65)True process ζ(x)Observations

●

●

●

●

●

●

●

●●

●

●



Parameter estimation for the Simple Machine

Without Model discrepancy

Z | θ, σ2ε ∼ MVN(θx, σ2

ε I)

p(θ, σ2ε ) ∝ 1/σ2

With Model discrepancy (KOH)

Z | θ, σ2ε , σ

2, ψ ∼ MVN(θx, σ2ε I + σ2R(ψ))

p(θ) ∝ 1

σ2ε ∼ IG(aε,bε)

σ2 ∼ IG(a,b)

ψ ∼ G(0,4)(aψ,bψ)

Posterior

p(θ, σ2ε | z)

Posterior

p(θ, σ2ε , σ

2, ψ | z)



Parameter estimation

0.45 0.55 0.65 0.75

050

100

150

No MD

θ

Pos

terio

r de

nsity

θ*11 obs.31 obs.61 obs.

0.45 0.55 0.65 0.75

010

2030

40

GP prior on MD

θP

oste

rior

dens

ity


0.45 0.55 0.65 0.75

010

2030

40

Constr. GP prior on MD

β

Pos

terio

r de

nsity

β*11 obs.31 obs.61 obs.

a) b) c)

Posterior densities do not cover the true value of θ.Posterior mean: 0.562. True value: θ∗ = 0.65

Ignoring model discrepancy: More observations only make usmore sure about the wrong value!



What do we know about Model Discrepancy for ourSimple Machine?

Model does not account for frictionWhat does that mean for the δ(x) function?

1 The function δ(x) is very smooth2 δ(0) = 03 δ(x) < 0 for all x > 04 δ(x) is decreasing,δ′(x) < 0 for all x > 0

5 δ′(0) = 0We want a prior for δ that reflects this

0 1 2 3 4 5 6

01

23

4

x (effort)y

(wor

k)

●

Simple Machine η(x, 0.65)True process ζ(x)Observations

●

●

●

●

●

●

●

●●

●

●



Priors for the Model Discrepancy (δ)

0 1 2 3 4

−2

01

2

ψ = 0.3

x

Rea

lizat

ions

0 1 2 3 4

−2

01

2

ψ = 1

x

Rea

lizat

ions

0 1 2 3 4

−3

−1

12

3

ψ = 0.3

x

Rea

lizat

ions

0 1 2 3 4

−3

−1

12

3

ψ = 1

x

Rea

lizat

ions

Constrained prior:δ(0) = 0 and δ′(0) = 0,δ′(0.5) < 0 and δ′(1.5) < 0



Parameter estimation

0.45 0.55 0.65 0.75

050

100

150

No MD

θ

Pos

terio

r de

nsity


0.45 0.55 0.65 0.75

010

2030

40

GP prior on MD

θ

Pos

terio

r de

nsity


0.45 0.55 0.65 0.75

010

2030

40


θ

Pos

terio

r de

nsity


a) b) c)

Constrained GP prior on the model discrepancy (MD):Posterior mean is slightly biased.Unlike before, posterior densities cover the true value of θ.


Example 1: Simple Machine Prediction

Prediction - InterpolationPosterior of ζ(x0) = θ ∗ x0 + δ(x0) at x0 = 1.5

0.80 0.85 0.90 0.95

050

100

150

No MD

ζ(1.5)

Pos

terio

r de

nsity

ζ*(1.5)

11 obs.31 obs.61 obs.

0.80 0.85 0.90 0.95

050

100

150

GP prior on MD

ζ(1.5)

Pos

terio

r de

nsity

ζ*(1.5)


0.80 0.85 0.90 0.95

050

100

150


ζ(1.5)

Pos

terio

r de

nsity

ζ*(1.5)


a) b) c)

The interpolations get better with more dataIgnoring model discrepancy: More observations only make usmore sure about the wrong value!



Prediction - ExtrapolationPosterior of ζ(x0) = θ ∗ x0 + δ(x0) at x0 = 6

2.5 3.0 3.5 4.0 4.5

05

1015

2025

30

No MD

ζ(6)

Pos

terio

r de

nsity

ζ*(6)


2.5 3.0 3.5 4.0 4.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

GP prior on MD

ζ(6)

Pos

terio

r de

nsity

ζ*(6)


2.5 3.0 3.5 4.0 4.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0


ζ(6)

Pos

terio

r de

nsity

ζ*(6)


a) b) c)

Constrained GP prior on the model discrepancy (MD):Posterior densities do not cover the true value of ζ(6).Does worse than the go-to method we used before

Extrapolation is always dangerous!



Prediction - ExtrapolationPosterior of ζ(x0) = θ ∗ x0 + δ(x0) at x0 = 8

3 4 5 6 7

05

1015

20

No MD

ζ(8)

Pos

terio

r de

nsity


3 4 5 6 7

0.0

0.5

1.0

1.5

GP prior on MD

ζ(8)

Pos

terio

r de

nsity


3 4 5 6 7

0.0

0.5

1.0

1.5


ζ(8)

Pos

terio

r de

nsity


a) b) c)

Constrained GP prior on the model discrepancy (MD):Posterior densities do not cover the true value of ζ(8).Does worse than the go-to method we used before

Extrapolation is always dangerous!



Why was the extrapolation worse for Constrained GP?

0 1 2 3 4 5 6

01

23

4

x (effort)

y (w

ork)

●

Simple Machine η(x, 0.65)True process ζ(x)ObservationsFitted line

●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●



Lessons from the Simple Machine

Ignore model discrepancy at your own peril!

Using non-informative priors on MDPrediction within range of data should work fineCan get very wrong estimates of parametersExtrapolate at your own peril!

Using informative priors on MDMore likely to get correct parameter estimates (if prior information iscorrect)Extrapolate at your own peril.


Example 2: Orbiting Carbon Observatory - 2




CO2 measurements from space

●

●●●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●●

●

●

●●●●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●●

●

●

●●●●

●

●

●

●●●

●●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●●

●●

●

●

●

●

●●

●

●

●

●

●

●●

●●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●●

●

●●

●

●

●

●

●●

●

●

●

●

●●

●●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●●●●●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●

0.760 0.765 0.770

0.00

0.01

0.02

0.03

Oxygen band

Rad

ianc

e

●●●●●

●

●

●

●●●●●●●●●●●●

●●●●●●●●●

●●●●●●●●●●●●●

●●●●●●●

●

●●●●●●●●●●●●●●●

●

●●●●

●●●●●●●

●●●●●●●

●

●●●●

●

●●●●●

●●

●●●

●●●

●

●●●●●

●●●●●●●●●●●●

●

●●●

●●

●

●●●●●●●●

●●●

●

●●●●●

●

●

●●●●●

●●●●●

●

●●●

●●

●

●●●●

●●

●●●●

●

●●

●●●●

●

●●●●●●●

●

●

●

●●●●●●

●

●●●●●

●●●

●

●●●

●

●

●

●●

●●●●●●●

●

●●●●●●

●

●

●

●

●●●●●

●

●●●●●●

●●●

●●

●

●

●

●●

●●●●●●

●

●●●●●●●●

●

●

●

●●●

●●

●●●●●●●●●●●

●

●

●

●

●●●●●●

●●

●●●●●●●

●

●

●

●●

●●●●●●

●

●●●●●

●

●

●

●

●

●●●

●●●●●●

●

●●●

●

●

●

●

●

●●

●●

●

●●●●●●

●●

●

●

●

●

●

●●●

●●●

●

●●●

●

●●

●

●●

●

●●

●

●●●

●

●●●●●

●

●

●

●

●

●

●

●●●●●●

●

●

●●

●

●

●

●

●

●●●●

●

●●

●

●●

●

●

●

●

●

●

●●

●

●●●●●●●

●

●

●

●

●

●

●●●●●

●

●

●●●

●

●

●

●

●

●

●

●●

●●●

●●

●

●

●

●

●

●

●●

●●●

●

●

●●●

●

●

●

●

●

●●

●

●●●●●

●

●

●

●

●

●

●

●●

●●●●●

●●

●

●

●

●

●

●●●●●●●

●

●

●

●

●

●

●●

●●●●●●●

●

●

●

●

●

●●●●●●●

●●

●

●

●

●

●●●

●●●●●●●●●●●

●●

●

●

●●●●

●●●●

●●

●

●

●

●●●●●●●

●

●

●

●

●

●

●●●●●

●●

●

●

●

●

●

●●●

●●●

●

●

●

●

●

●

●●●●●

●

●

●

●

●

●

●●●●●

●

●

●

●

●

●

●●●●●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●●

●●●

●

●

●

●

●

●●●●

●

●

●

●

●

●●

●●●

●

●

●

●

●

●●●●

●

●

●

●

●●●●

●

●

●

●

●

●●●●

●

●

●

●●

●●●

●

●

●

●

●●

●●

●

●

●

●

●●●

●

●

●

●

●●●●

●

●

●

●

●●●

●

●

●

●●

●●

●

●

●

●●●●●

●

●

●●●●

●●

●

●●●

●●

●●●●

●●

●●

●●

●

●

●●●●

●●●●

●●●●●●●●●●●●●●●●

●●●●

●●

●●●●

●●●●

●●●●●●●●●

●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●

●●●●●●●●●●●●●

●●●●●●●●●●●●

●●●●●●●●●●●●

●●●●●●●●●●

●●●●●●●●●

●●●●●●●●●

●●●●●●●●●

●●

●●●●

●●●●●●●

●●●●●●●●●

●●

●●●●●●●●●●●●●●●●●●

1.590 1.595 1.600 1.605 1.610 1.615 1.620

0.00

00.

015

0.03

0

Strong CO2 band

Rad

ianc

e

●●●●●●●●●●●●●●●●●●●●

●

●●

●●●●●●●

●

●●●●●●●●●●●●●●●●

●

●●●

●●●

●

●

●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●

●●●●●●●●●●●

●

●●●●●

●●

●

●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●

●

●●●●●●●●

●

●

●●

●●●●●●

●

●

●●●

●●●●●●●●

●●●

●●●●●●●●●●●●

●●●●●●●

●

●●●●●●●●●

●

●

●

●●

●●●

●

●

●●●●

●●●

●

●●●●●●●●

●

●●●●●

●

●●●●●●●●●●

●●●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●●

●●●●

●

●●●●●●●

●

●●●

●●

●

●●●●●●

●●●●●

●●●●

●

●

●

●

●●●

●

●

●

●

●

●●●

●●●●●●●●

●

●●●●●●●●●●●●●

●

●●●●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●●●●●●

●●

●●●●●●●●●●●●

●

●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●

●●●●●●●●●●●●●●●●●●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●●

●

●●●●●

●●●●●

●

●●●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●●

●●●

●

●●●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●●●●●

●●

●●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●●

●●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●●

●●●●●●●●●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●●

●

●●●

●●

●

●

●

●

●

●

●

●

●●

●●

●●

●

●

●

●

●

●●●

●●

●

●

●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

2.05 2.06 2.07 2.08

0.00

00.

010

0.02

0

Weak CO2 band

wavelength (micron)

Rad

ianc

e

Y = observed radiances at 3048 wavelengths



State Vector and forward model

X =

X1...

X20

X21...

X50

CO2 mole fraction indifferent layers ofthe atmosphere

Other variables, such as:surface pressure, aerosolswater vapor, temperature offsetalbedo, chlorophyll fluorescence

Y: noisy obs. of natures transformation of atmosphere

Y = F(ν) + measurement error, ν = wavelengths

Don’t know F exactly, use a “full physics” model instead:

Y = F(ν,x) + ε

But F 6= FQOI: Column averaged CO2 XCO2 = h>X1:20


Forward model

The forward model F FP

Computationally feasibleSimplification of a dream model F D

Solves radiative transfer equations (integro-differential equations)

Spectrally-dependent surface and atmosphere optical propertiesCloud and aerosol single scattering optical propertiesGas absorption and scattering cross-sectionsSurface reflectance (albedo)

Spectrum effectsSolar model, Fluorescence, Instrument Doppler shift

Instrument modelSpectral dispersion, Instrument line shape (ILS) function,Polarization response


CO2 retrieval

Optimal EstimationRodgers, 2000

Find the state vector x that minimizes

c =(

y− FFP(x,B))T

S−1e

(y− FFP(x,B)

)+ (x−µa)T S−1

a (x−µa)

using Levenberg-Marquardt algorithm

and provide an estimate of uncertainty:

S = (K T S−1e K + S−1

a )−1 where K =δFFP(x,B)

δx

∣∣∣∣x=x

Use N(x, S) as an estimate of the true state X

xCO2 = hT x1:20 and Var(XCO2) = hT SCO2h


CO2 retrieval

Optimal estimation ≈ Bayesian Retrieval

Bayesian model:

Y|X ∼ N(FFP(X,B),Se)

X ∼ N(µa,Sa)

Posterior density: p(x|y) ∝ exp{−1

2c}

where

c =(

y− FFP(x,B))T

S−1e

(y− FFP(x,B)

)+ (x−µa)T S−1

a (x−µa)

assuming that B, Se, µa and Sa are known.

x = posterior mode

S ≈ posterior covariance matrix


CO2 retrieval

Model discrepancy for CO2 Retrieval

X is a physical parameter in this model:

Yi = F FPi (X,B) + εi , i = 1,2, . . . ,3048

whereεi

ind∼ N(0, σ2i ), X ∼ N(µa,Σa)

We know there is model discrepancyF FP 6= FF FP 6= F D

The problem with estimating physical parameters:If model discrepancy is not accounted for, parameter estimation isbiased and over confident


CO2 retrieval

Current approach to model error in the OCO-2 mission

1 Calculate 3 EOFs from residuals and fit

Y (νi) = F FPi (z,B) + Uz + εi , εi iid. N(0, σ2

i )

cost function for OE procedure becomes

c = (y−F(x,b)−Uz)>Σ−1ε (y−F(x,b)−Uz)+(x′−µ′a)>Σ′−1

a (x′−µ′a)

State vector now includes EOF amplitudes, x′ = (x>, z>)>.Akin to the Kennedy & O’Hagan approach

Y (νi ) = F FPi (X,B) + δ(νi ) + εi , εi iid. N(0, σ2

i )

2 Apply bias correction for XCO2 after the retrieval using groundmeasurements of XCO2


CO2 retrieval

Bias correction for OCO-2

TCCON stations that measure XCO2 from the groundOCO-2 in target mode over TCCON stations


CO2 retrieval

Model discrepancy for CO2 Retrieval

Kennedy and O’Hagan approach for OCO-2:

Y (νi) = F FPi (X,B) + δ(νi) + εi , i = 1,2, . . . ,3048

We need a model discrepancy term that describes the actualmodel discrepancy (not residuals)

StrategyBorrow information between spatial locations

Use ground measurements to learn δ at that locationTransfer that δ to nearby locations

Row rank modeling of δ

Explore the form of model discrepancy with simulation studiesF D, F FP , F SURR , F CS


CO2 retrieval

Strategy

*o Elf

→ ← a.⇐ * ¥TsTt

Ts

i-

-

-

e.e. s bwee

Og

Q%

7-

-,

←+

++

c'

→A

os

ss

⇐** *

G.

-

+W

.+

+

adore

of→

nn

a.

OO

q,

=°

u +

of

g-1-

a§


CO2 retrieval

Preliminary studyModel error in the clear-sky model due to parameter misspecification

What does the model error look like?

Simplified "clear sky" forward model, FCS

21-dim state vector, fewer parameters

Generate two soundings (without measurement error)

ytrue = FCS(Xtrue,b)

yWM = FCS(Xtrue,bWM)

The difference between ytrue and yWM gives the modeldiscrepancy that is due to that particular misspecification of b.


CO2 retrieval

Model discrepancy

2.05 2.06 2.07 2.08

0.00

00.

002

0.00

40.

006

Strong CO2 band

Wavelength

Diff

eren

ce

1.590 1.600 1.610 1.620

0.00

00.

002

0.00

40.

006

0.00

80.

010

Weak CO2 band

Wavelength

Diff

eren

ce

0.760 0.765 0.770

0.00

00.

002

0.00

40.

006

0.00

8

Oxygen A band

Wavelength

Diff

eren

ce

Model discrepancy is very spiky⇒ Gaussian Process model for δ is not appropriate

But: The spikes in δ are at the same positions as absorption linesSuggests basis vector model for δ:

δ = Uz


CO2 retrieval

Basis vectors

2.05 2.06 2.07 2.08

0.00

0.05

0.10

0.15

Strong CO2 band

Wavelength1.590 1.595 1.600 1.605 1.610 1.615 1.620

0.00

0.05

0.10

0.15

Weak CO2 band

Wavelength0.760 0.765 0.770

0.00

0.05

0.10

0.15

Oxygen A band

Wavelength

Chose ≈ 50 absorption lines for each bandUsed the Laplace pdf to create basis vectors (U):

f (ν) =1

2βexp

{−|ν − µ|

β

}, ν, µ ∈ R and β > 0

µ = absorption line, β controls spread, truncated at ± 4 stdev.


CO2 retrieval

Preliminary study

Generated 3 true state vectors xtrue from N(µa,Σa)

Yi = FCS(xtruei ,btrue) + ε i = 1,2,3

Location 1: Validation siteHave independent observation of XCO2Use them to define an informative prior on XUse a vague prior on ZGet the posterior distribution p(Z | Y1)

Locations 2 and 3:Vague prior on XPrior on Z: posterior from location 1

Working model: tweaked b by 1%


CO2 retrieval

Retrieval

Without model discrepancy (MD) :

Y|X ∼ N(FCS(X,bWM),Σε)

X ∼ N(µa,Σa)

With model discrepancy:

Y|X,Z ∼ N(FCS(X,bWM) + UZ,Σε)

X ∼ N(µa,Σa)

Z ∼ N(µZ ,ΣZ )

Vague prior: Large variances

Informative prior: Small variances


CO2 retrieval

Results for validation site

386 388 390 392 394

01

23

4

Sounding 1

Xco2

Den

sity

Informative prior on XNon−inform. MD priorNo MDPrior on Xco2True value

Vague prior for Z: µZ = 0 and ΣZ = I.Jenný Brynjarsdóttir (CWRU) Model discrepancy September 17, 2019 37 / 42

CO2 retrieval

Model discrepancy at validation site

0 50 100 150

−0.

40.

00.

20.

4

Element of Z

● ●

●

●

●

●

●

●

●

●

●

● ●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●●

●

●●

●

●

● ●

●

●

●

● ●

●

●

●

●

● ● ●

●

●

●

● ●

●● ●

●

●

●

●

●

●●

●●

● ●●

●

●

●

●●

● ●

●●

●

●

● ●

● ● ●●

●●

● ● ●●

●

●

●

●

●

●

●

●

●

●

● ●

● ●

●

● ●

●

●

●

●●

●

●

●

●

●

●

● ●

●

●

●●

●

● ● ●●

● ●

●

●

●●

●●

Strong CO2 band Weak CO2 band A band

90% posterior credible intervals of Z1 from this retrievalprior 90% credible interval: (−1.645,1.645)


CO2 retrieval

Results for locations 2 and 3

386 388 390 392 394

01

23

4

Sounding 1

Xco2

Den

sity

Informative prior on XNon−inform. MD priorNo MDPrior on Xco2True value

386 388 390 392 394

0.0

0.2

0.4

0.6

0.8

1.0

Sounding 2

Xco2

Den

sity

Informative MD priorNon−inform. MD priorNo MDPrior on Xco2True value

386 388 390 392 394

0.0

0.2

0.4

0.6

0.8

1.0

Sounding 3

Xco2

Den

sity

Informative MD priorNon−inform. MD priorNo MDPrior on Xco2True value

Locations 2 and 3: the posterior recovers the true value of XCO2


Conclusions

Discussion

Learning about model discrepancy at one location has thepotential to greatly improve retrievals in other locations.

Lots of details have to be figured outHow well does model discrepancy extrapolate across space?Computational efficiency (choice of basis vectors)

UQ-test bedRealistic surrogate model, FSURR

Templates of simulated (X, Y) pairs that reflect the range of physicalconditions around the globe


References

References

Jenný Brynjarsdóttir, Jonathan Hobbs, Amy Braverman, and LukasMandrake.Optimal Estimation Versus MCMC for CO2 Retrievals.Journal of Agricultural, Biological, and Environmental Statistics,23(2):297–316, 2018.

Jenný Brynjarsdóttir and Anthony O’Hagan.Learning about physical parameters: The importance of modeldiscrepancy.Inverse Problems, 30, 2014.

M. C. Kennedy and A. O’Hagan.Bayesian calibration of computer models.Journal of the Royal Statistical Society B, 63(Part 3):425–464, 2001.


Bayes In Space

Thanks!


Extra

Analysis without accounting for model discrepancy

To estimate the parameter we fit

zi = xiθ + εi , εi iid. N(0, σ2ε ).

Prior: π(θ, σ2ε ) ∝ 1/σ2

ε

Posterior: tn−1 distributionDon’t need any MCMC

Posterior distribution does notcover the true value of θ.

Posterior mean: 0.562True value: 0.65

More observations only makeus more sure about the wrongvalue!

0.45 0.55 0.65 0.75

050

100

150

No MD

θ

Pos

terio

r de

nsity


0.45 0.55 0.65 0.75

010

2030

40

GP prior on MD

θ

Pos

terio

r de

nsity


0.45 0.55 0.65 0.75

010

2030

40


β

Pos

terio

r de

nsity

β*11 obs.31 obs.61 obs.

b) c)


Extra

Analysis with Model Discrepancy (MD)

As in Kennedy & O’Hagan (2001), we model δ(x) as a Gaussianprocess:

δ(x) ∼ GP(0, σ2c(·, ·|ψ)) where c(x1, x2|ψ) = exp

(−(

x1 − x2

ψ

)2)

Bayesian model:

Z | θ, δ, σ2ε ∼ N

(Xθ + δ, σ2

ε I)

δ | σ2, ψ ∼ N(0, σ2Λ(ψ)

)θ, σ2

ε , σ2, ψ ∼ π(θ, σ2

ε , σ2, ψ)

We are interested in both θ and δ so we want to sample the posteriordistributions of both. Problem:

Full conditional of δ can have a numerically singular covariancematrix


Extra

Complete Bayesian model

Our solution: Sample δ(xi) at few locations, and set the rest equalto their conditional meanxo = (0.2, 0.96, 1.72, 2.48, 3.24, 4.00)ᵀ and δo = δ(xo)Set δr = E (δr | δ(xo))

A complete formulation of the model:

Z|δo,θ ∼ N(Xθ + H(ψ)δo, σ2ε In)

δo ∼ N(

0, σ2Λ(ψ)o,o

)[θ]∝ 1, σ2

ε ∼ IG(aε,bε), σ2 ∼ IG(a,b),

ψ ∼ trGamma(0,4)(aψ,bψ)

where

H(ψ) =

[I6

Λ(ψ)r ,oΛ(ψ)−1o,o

]Jenný Brynjarsdóttir (CWRU) Model discrepancy September 17, 2019 45 / 42

Extra

The prior distributions

p(θ) ∝ 1

σ2 ∼ InvGamma(mode = 0.22,mean = 0.32

)ψ ∼ TrGamma[0,4] (mean = 1, var = 0.2)

σ2ε ∼ InvGamma

(mode = 0.0092,mean = 0.012

)

MCMC: Gibbs sampler with aMetropolis-Hastings step for ψ

0 1 2 3 4 5 6

01

23

4

x (effort)

y (w

ork)

●

Simple Machine η(x, 0.65)True process ζ(x)ObservationsFitted line

●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●


model discrepancy and physical parameters in calibration and ......ignore model discrepancy at your...

Documents