uncertainty quantification and calibration of computer modelshuang251/xian_0212.pdf · uncertainty...
TRANSCRIPT
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Uncertainty quantification and calibration ofcomputer models
Xian He
February 12, 2014
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Outline
1 Uncertainty QuantificationWhat is uncertainty ?
2 EmulatorWhat and why ?
3 Bayesian CalibrationModel componentsComputer model emulaltorDiscrepancy modelComplete model
4 Discussion
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Introduction
Physical model is defined by a set of differential equations.
Such mathematical models are implemented as computercodes
To make predictions using the computer codes, somenecessary parameters need to be estimated
Calibration : with observations and computer output, we learnabout the parameters.
Bayesian calibration method allows for all sources ofuncertainty and attempts to correct for the inadequacy of themodel.
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
What is uncertainty ?
Inaccurate inputs
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
What is uncertainty ?
Initial/boundary condition uncertainty
From :www.easterbrook.ca
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
What is uncertainty ?
Inherent randomness of the system
TEM Terrestrial Ecosystem model(TEM)Xiao et al. [1998]
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
What is uncertainty ?
Parameters uncertainty
In TEM, the 6 key parameters unknown Chen et al. [2011]
Cmax Maximium rate of photosynthesis C, ranges[50, 1500]gm−2mon−1
KI Half saturation constant for PAR used by plants, ranges[20, 600]Jcm−2d−1
KC Half saturation constant for C02 − C uptake by plants, ranges[20, 600]µLL−1
Tmax Maximum temperature for GPP, range [25, 35]◦C
Tmin Minimum temperature for GPP, range [−12,−1]◦C
Topt Maximum optimum temperature for GPP, range [15, 25]◦C
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
What is uncertainty ?
He and Zhang [2012] has reviewed the methods of uncertaintyquantification. Here are three popular approaches
Monte Carlo simulation and variants, Doubilet et al. [1984]
Polynomial Chaos methods, Xiu and Em Karniadakis [2002]
Statistical emulation, OHagan [2006] and Conti and O’Hagan[2010]
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Outline
1 Uncertainty QuantificationWhat is uncertainty ?
2 EmulatorWhat and why ?
3 Bayesian CalibrationModel componentsComputer model emulaltorDiscrepancy modelComplete model
4 Discussion
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
What and why ?
What’s an emulator ?
Definition : A surrogate model and a statistical representationof the computer model.
Examples :
Gaussian process models Sacks et al. [1989] and Conti andO’Hagan [2010]Polynomial Chaos expansions
Implementations :
Climate modeling, Rougier et al. [2009]Environmental science, Challenor et al. [2006]Medical science, Dev et al. [2002]
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
What and why ?
What’s an emulator ?
Explanation in flow-chart
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
What and why ?
Why emulator ?
Resembles the behaviour of the real system.
An approximation of complex computer model.
Simpler/easier surrogate of the computer model.
much faster to run than the computer model.
Makes fully probabilistic predictions of what the simulatorwould produce
By giving a probability distribution of the model output, wecould quantify uncertainties.
predicts output at training data points without uncertainty
predict outputs at another point with non-zero variance
Given enough training data, it would predict simulator outputat any desired accuracy
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
What and why ?
Why emulator ?
Prediction
Approximate the behavior of computer models
Given any input x, we could make predictions
Calibration
By comparing observation with emulator prediction, we canestimate the parameters that minimizes the discrepancies.
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
What and why ?
When to use emulator ?
Prediction
The simulator output vary smoothly in response to inputs
The simulator is computer expensive
The dimension of inputs is not high
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
What and why ?
Reference papers
The development of emulator
Sacks et al. [1989] first introduced the emulator for computermodels
Kennedy and O’Hagan [2001] developed the framework ofBayesian calibration of computer models which accounts forall the possible uncertainties
Conti and O’Hagan [2010] extended the emulator tomulti-output
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Outline
1 Uncertainty QuantificationWhat is uncertainty ?
2 EmulatorWhat and why ?
3 Bayesian CalibrationModel componentsComputer model emulaltorDiscrepancy modelComplete model
4 Discussion
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
What’s calibration ?
Definition : Model calibration is the process to identifyparameter values θ for which model outputs agrees withobservation data d .
Approaches :
Traditional method which is ad hoc fitting,see Chen andZhuang [2012]Bayesian calibration which provides parameter estimates anduncertainty of parameters,see Kennedy and O’Hagan [2001]
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Advantages of Bayesian calibration
Allows for all sources of uncertainty
Attempt to correct for any inadequacy of the model, revealedby discrepancy between data and computer model.
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Model components
The Bayesian calibration framework in Kennedy and O’Hagan[2001] :
z(x) = y(x,θ) + δ(x) + ǫ(x) (1)
θ :the unknown parameters, to be calibrated
x ∈ X : the variables inputs, such as temperature,precipitation.
z(x) : the real observations
y(x,θ) : the emulator of the computer model
δ(x) : discrepancy between z(x) and y(x,θ) besidesmeasurement error ǫ(x)
Measurement error ǫ(x) : observational error which is iidN(0, σ2), where σ2 is estimated by experts.
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Model components
Model notations
n : number of observations (z)
N : number of computer output ( y)
t1, t2, ..., tN : set of calibration parameters, given by samplingfrom the prior
x∗1, x∗
2, ..., x∗
N : set of variable inputs at which we get thecomputer output y(x∗, t).
x1, x2, ..., xn : set of variable inputs at which we observe theoutput z.
y(x∗i , tk) : set of computer outputs given x∗1, x∗
2, ..., x∗
N andt1, t2, ..., tN
z(xj) : the observations at x1, x2, ..., xn
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Computer model emulaltor
Here is how to construct an emulator :
Suppose we can afford to obtain output y(si ) at inputx ∈ S = {s1, ..., sn}, n is training sample size.
Next, we establish some statistical relationship betweenoutput function y(x) and the obtained y(si ).
Quantify the uncertainty of y(x) from the statisticalrelationship of y(x) and y(si ).
A Gaussian process emulator
probability distribution for an unknown function
conditional on some unknown parameters θ
estimated from training data
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Computer model emulaltor
Gaussian process emulator : y(x, θ)
The computer emulator with unknown parameters θ is representedby y(x,θ). It is modelled by Gaussian process as follows :
y = η(x,θ) ∼ N(m1(x,θ), c1((x,θ), (x′,θ)))
The mean function m1(x,θ) is models as regression component.For example, hi(x,θ) are features, then
m1(x,θ) = β0 + β1(h1(x,θ)) + · · ·+ βp(hp(x,θ))
where βi are estimated from training set.
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Computer model emulaltor
Gaussian process emulator y(x, θ)
The model of covariance function c1((x,θ), (x′,θ)) is the key step.
It should satisfy
smoothness of y
correction between x and x′
should be close to 1 if x and x′
are sufficiently close
Borrowed from Kennedy and O’Hagan [2001], a proportional modelis chosen when assumed stationary
c1((x,θ), (x′,θ)) = σ2R(x− x
′
)
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Discrepancy model : δ(x)
δ(x) is normal distribution :
δ ∼ N(m2(x), c2(x, x′))
The measurement error ǫ ∼ N(0, λ0), λ0 is elicited from expert.
m1(x , ) = h1(x, )Tβ
1
m2(x) = h2(x)Tβ2, β = (βT
1 ,βT2 )
T
c1(., .) and c2(., .) have hyperparameters φ1 and φ2
prior information about θ is independent of the others.
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Posterior Distribution
We would never know the true θ. The purpose of calibration is forfurther using of the computer models.Posterior distributions
D1 = (x∗1, t1), ..., (x∗
N , tN) : set of points where y are available
D2 = (x1,θ), ..., (xn,θ) : set of points where z are observed
H1(D1) : matrix of h1(x∗
1, t1), ..., h1(x∗
N , tN)
the expectation of y is H1(D1)β1
the expectation of z is H1(D2)β1 + H2(D2)β2
V1(D1) is the variance matrix of y, N × N matrix.
V2(D2) is the variance matrix of z, n × n matrix.
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Posterior distributions
The full set of data dT = (yT , zT ) is normal with the followingmean and covariance
E (d|θ,β,φ) = md (θ) = H(θ)β
var(d|θ,β,φ) = Vd (θ)
Where
H(θ) =
(
H1(D1) 0H1(D2(θ)) H2(D2)
)
and
Vd(θ) =
(
V1(D1) C1D1,D2(θ)T
C1D1,D2(θ)T V1D2(θ) + V2(D2)
)
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Posterior distributions
With specification of prior, we could get the full posteriordistribution
p(θ,β,φ|d) ∝ p(θ)p(φ)f (d ;md (θ),Vd (θ) (2)
We could integrate β out from (2). Then, we need to estimate thehyperparameters φ from z and y. Finally, we could get theposterior p(θ|d).
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Outline
1 Uncertainty QuantificationWhat is uncertainty ?
2 EmulatorWhat and why ?
3 Bayesian CalibrationModel componentsComputer model emulaltorDiscrepancy modelComplete model
4 Discussion
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
The shorting comings of Bayesian calibration in Kennedy andO’Hagan [2001]
Inefficient when the output is multivariate
When the output is time varying time series, the dimension ofy would be very high. Estimation of covariance function wouldbe difficult.
Kennedy and O’Hagan [2001] applies well when n << N.Now, data arrives sequentially that n >> N.
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
How to overcome ?
One way to overcome the time series output is to consider dynamicemulator for the computer model. There are some reference papers
Liu and West [2009] describes a dynamic emulator fordynamic computer model.
Williamson and Blaker [2014] develops Bayesian dynamiclinear model Gaussian processes for emulation of time seriesoutput.
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Peter G Challenor, Robin KS Hankin, and Robert Marsh. Towardsthe probability of rapid climate change. 2006.
M. Chen, Q. Zhuang, D. R. Cook, R. Coulter, M. Pekour, R. L.Scott, J. W. Munger, and K. Bible. Quantification of terrestrialecosystem carbon dynamics in the conterminous united statescombining a process-based biogeochemical model and MODISand AmeriFlux data. Biogeosciences, 8(9) :2665–2688,September 2011. ISSN 1726-4189.
Min Chen and Qianlai Zhuang. Spatially explicit parameterizationof a terrestrial ecosystem model and its application to thequantification of carbon dynamics of forest ecosystems in theconterminous united states. Earth Interactions, 16(5) :1–22,April 2012. ISSN 1087-3562.
Stefano Conti and Anthony O’Hagan. Bayesian emulation ofcomplex multi-output and dynamic computer models. Journal of
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Statistical Planning and Inference, 140(3) :640–651, March2010. ISSN 0378-3758.
Parvati Dev, Dale Harris, David Gutierrez, Amol Shah, andS Senger. End-to-end performance measurement of internetbased medical applications. In Proceedings of the AMIASymposium, page 205. American Medical InformaticsAssociation, 2002.
Peter Doubilet, Colin B Begg, Milton C Weinstein, Peter Braun,and Barbara J McNeil. Probabilistic sensitivity analysis usingmonte carlo simulation. a practical approach. Medical decisionmaking : an international journal of the Society for MedicalDecision Making, 5(2) :157–177, 1984.
Xian He and Hao Zhang. Uncertainty quantification, physicalmodels. In Encyclopedia of Environmetrics. John Wiley & Sons,Ltd, 2012.
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Marc C. Kennedy and Anthony O’Hagan. Bayesian calibration ofcomputer models. Journal of the Royal Statistical Society :Series B (Statistical Methodology), 63(3) :425–464, 2001. ISSN1467-9868.
Fei Liu and Mike West. A dynamic modelling strategy for bayesiancomputer model emulation. Bayesian Analysis, 4(2) :393–411,June 2009. ISSN 1936-0975.
Anthony OHagan. Bayesian analysis of computer code outputs : atutorial. Reliability Engineering & System Safety, 91(10) :1290–1300, 2006.
Jonathan Rougier, David MH Sexton, James M Murphy, andDavid Stainforth. Analyzing the climate sensitivity of thehadsm3 climate model using ensembles from different butrelated experiments. Journal of Climate, 22(13), 2009.
Xian He
Uncertainty quantification and calibration of computer models
Uncertainty Quantification Emulator Bayesian Calibration Discussion References
Jerome Sacks, William J. Welch, Toby J. Mitchell, and Henry P.Wynn. Design and analysis of computer experiments. Statist.Sci., 4(4) :409–423, November 1989. ISSN 0883-4237.
Daniel Williamson and Adam Blaker. Evolving bayesian emulatorsfor structured chaotic time series, with application to largeclimate models. SIAM Journal of Uncertainty Quantification, 2(1) :1–28, January 2014.
X. Xiao, J. M. Melillo, D. W. Kicklighter, A. D. McGuire, R. G.Prinn, C. Wang, P. H. Stone, and A. Sokolov. Transient climatechange and net ecosystem production of the terrestrialbiosphere. Global Biogeochemical Cycles, 12(2) :345–360, 1998.ISSN 1944-9224.
Dongbin Xiu and George Em Karniadakis. Modeling uncertainty insteady state diffusion problems via generalized polynomial chaos.
Xian He
Uncertainty quantification and calibration of computer models