calibration and uncertainty analysis of models with mcmc … · lake model code . mylake • model...

Calibration and uncertainty analysis of models with MCMC-simulation

Tuomo Saloranta Section for glaciers, snow and ice

Hydrology department

Norwegian Water Resources and Energy Directorate (NVE)

Norwegian Water Resources and Energy Directorate

How to increase the quality and benefits from models? ■ Apply techniques for model analysis

■ better transparency and understanding ■ optimized model performance ■ uncertainties analyzed and quantified

■ Automatic calibration and uncertainty analysis (MCMC)

■ more ”honest” results ■ probabilities (risk = probability x consequence) ■ scanning for all plausible parameter-alternatives

2

03.03.2012

Norwegian Water Resources and Energy Directorate

Background & different types of uncertainty

Norwegian Water Resources and Energy Directorate

Why uncertainty analysis?

■ Traditionally uncertainties are unwanted and often ”suppressed”.

■ However, sometimes they are an inherent part of the system, and cannot be reduced.

■ Uncertainty management especially important in: ■ policy-relevant science (many stakeholders) ■ when stakes are high

4

03.03.2012

Norwegian Water Resources and Energy Directorate

What is uncertainty analysis?

■ Point values are replaced by probability distributions. ■ What we know, and what we don’t know, is reflected in our

answers.

”It is better to be roughly right than precisely wrong” (Keynes)

5

03.03.2012

Norwegian Water Resources and Energy Directorate

Distributions, means, percentiles

2.5% percentil = 1

50% percentil = 2

97.5% percentil = 3

Unif (0.5, 2.5) Norm (2, 0.5)

Logn (ln2, 0.5)

Norwegian Water Resources and Energy Directorate

A typology of uncertainty

by Funtowicz & Ravetz (1990)

■ Technical uncertainty (inexactness)

■ Methodological uncertainty (unreliability)

■ Epistemological uncertainty (border with ignorance)

7

03.03.2012

Norwegian Water Resources and Energy Directorate

Model uncertainty – a simple example

8

03.03.2012

20

25

30

35

40

Y

0 2 4 6 8 10X

Norwegian Water Resources and Energy Directorate

Monte Carlo simulation

9

03.03.2012

Say, based on empirical data, we have estimed uncertainty of PEC and PNEC.

How about the uncertainty of the ratio PEC/PNEC?

Norwegian Water Resources and Energy Directorate

Monte Carlo simulation

10

03.03.2012

Sample parameters (p1, p2)

Save results from simulation round n

n=n+1 (until n=N)

Run model

Analyse the N saved results statistically

Start here

Norwegian Water Resources and Energy Directorate

Model uncertainty – a simple example

11

03.03.2012

Norwegian Water Resources and Energy Directorate

Model uncertainty – a simple example

12

03.03.2012

In Monte Carlo simulation the PEC/PNEC ratio was >1 in 21 of 1000 samples. Thus, by 97.9 % probability we are on the ”safe side”.

Norwegian Water Resources and Energy Directorate

What is MCMC simulation?

■ Markov chain Monte Carlo simulation ■ Based on Bayesian inference and Monte Carlo simulation ■ Automatic model calibration against observations

(statistical fit) ■ Reveals all possible parameter combinations that give a

proper model fit with observations ■ Previous knowledge can be utilized in estimating

parameters (prior distributions)

13

03.03.2012

Norwegian Water Resources and Energy Directorate

Monte Carlo simulation

14

03.03.2012

Sample parameters (p1, p2)

Save results from simulation round n

n=n+1 (until n=N)

Run model

Analyse the N saved results statistically

Start here

Norwegian Water Resources and Energy Directorate

Markov chain Monte Carlo simulation

15

03.03.2012

Sample parameters (p1, p2)

n=n+1 (until n=N)

Analyse the N saved results statistically

Start here

Save p1, p2

Reject/accept p1, p2 based on a comparison of simulations and observations

Run model

Norwegian Water Resources and Energy Directorate

16

03.03.2012

Monte Carlo simulation:

Parameter uncertainty estimates based on data or experts

• some simulation results from the uncertainty analysis may not correspond well to observations.

Markov Chain Monte Carlo simulation:

Parameter uncertainty estimates fitted to observations

• gives both a model calibration and an uncertainty analysis consistent with observations.

Simulations

Obser-vations +

Pr (accept)

recect

Norwegian Water Resources and Energy Directorate

Steps in MCMC simulation

p1

p2 o

tid

C x x

x

If better ”match” than at previous round, accept p

p1

p2 o

tid

C x x

x

If worse ”match” than at previous round, discard p with probability α o

o

p1

p2 o

o

p1

p2 o

o

o o

Norwegian Water Resources and Energy Directorate

Example #1

Norwegian Water Resources and Energy Directorate

Saloranta, T. M et al. 2009: Hydrology Research 40, 234-248.

AIMS:

• To simulate the impacts of the projected future climate on “lake water climate” between control (1961-1990) and A2/B2 (2071-2100) scenarios

• To take properly into account model parameter-related uncertainties in the simulation results

LAKE SITES IN FINLAND:

• Small and shallow Valkea-Kotinen and larger and deeper Pääjärvi

Lake modelling & uncertainty analysis

Norwegian Water Resources and Energy Directorate

METHODS:

• Lake model code MyLake

• Model calibration and uncertainty estimation by Markov chain Monte Carlo (MCMC) simulation method

Parameter uncertainties estimated in the calibration period (2000-2002), are used to express model uncertainties in the climate scenarios.

Lake modelling & uncertainty analysis

Norwegian Water Resources and Energy Directorate

MCMC simulation

Melting ice albedo Melting ice albedo

Prior distribution

3 separate chains

Norwegian Water Resources and Energy Directorate

Conclusions - future lake climate

Valkea-Kotinen

Norwegian Water Resources and Energy Directorate

Example #2

Norwegian Water Resources and Energy Directorate

Saloranta et al. 2008, Environmental Science and Technology 42, 200-206.

Updating of parameter distributions

posterior

prior

Fjord POP modelling & uncertainty analysis

Norwegian Water Resources and Energy Directorate

Saloranta et al. 2008, Environmental Science and Technology 42, 200-206.

Fjord POP modelling & uncertainty analysis

Norwegian Water Resources and Energy Directorate

Thanks!