designing ensembles for climate prediction peter challenor national oceanography centre
TRANSCRIPT
![Page 1: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/1.jpg)
Designing Ensembles for
Climate PredictionPeter Challenor
National Oceanography Centre
![Page 2: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/2.jpg)
Why Ensembles for Climate Prediction?
•Not just a point estimate
•Uncertainty estimates as well
•Calibration of models against data
•Sensitivity analysis
![Page 3: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/3.jpg)
![Page 4: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/4.jpg)
Overview•What is experimental design?
•Why should we be interested?
•Perturbed physics ensembles
•Space filling designs
•Some recent results
•Multimodel ensembles
•Conclusions
![Page 5: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/5.jpg)
What is experimental
design?•Developed from agricultural experiments in the 1920’s
•How should you apply treatments to experimental plots in a field?
![Page 6: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/6.jpg)
R.A. Fisher
•Greatest Statistician of the 20th Century
•Randomisation
•Block designs
•Latin squares
•Split plots …
![Page 7: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/7.jpg)
Clinical Trials
•Randomisation
•Blind and Double Blind Trials
•Sequential Designs
![Page 8: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/8.jpg)
Why should we worry about
designing our experiments?•Would you take medication that
hadn’t been through a properly designed clinical trial?
•Would you set climate policy without a properly designed climate model experiment?
![Page 9: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/9.jpg)
Computer Experiments(Climate model
ensembles)•Computer experiments are very different from either clinical trials or field experiments.
•In general we are using them to explore the properties of some computer simulator (model). This is usually the numerical solution of a system of PDE’s or ODE’s
![Page 10: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/10.jpg)
Computer Experiments
•Mathematically we can write our computer simulator as
•where y is the output, x is the input and η(.) is the unknown mathematical function represented by the simulator
•y and x are often very high dimension
![Page 11: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/11.jpg)
Computer Experiments
•Normally the purpose of our computer experiment is to make some inference about the model
•Estimate what the model does at inputs we haven’t run it at
•Optimise the model parameters w.r.to some data
•Make predictions
![Page 12: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/12.jpg)
Types of ensemble
1. Perturbed physics ensembles
Change inputs (parameters, initial conditions, …) to a single model
2. Multimodel ensembles
Look at multiple models
![Page 13: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/13.jpg)
What is a good experimental
design?
Make our inferences to the highest accuracy with the minimum cost
![Page 14: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/14.jpg)
Optimal Design
•Fisher Information ≃ inverse of variance
•Maximise the information = minimising the variance
•D-optimal designs
•Minimise the determinant of the variance matrix
•A-optimal designs
•Minimise the trace of the variance matrix
•There are others (I, V, G, E optimality)
![Page 15: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/15.jpg)
Bayesian Design
•Set up a D-optimal design by maximising the utility
is the variance matrix of θ in design S
![Page 16: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/16.jpg)
An Possible Design•‘Star’ design
![Page 17: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/17.jpg)
•No aphorism is more frequently repeated in connection with field trials experiments, than that we must ask Nature few questions or, ideally, one question, at a time. The writer is convinced that this view is wholly mistaken. Nature, he suggests, will best respond to a logical and carefully thought out questionnaire; indeed, if we ask her a single question, she will often refuse to answer until some other topic has been discussed.
•R.A. Fisher, 1926
![Page 18: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/18.jpg)
What we need from the design of a climate model ensemble
We want to
1.Span the whole input space
2.Observe interactions
3.Minimise the number of simulator evaluations
![Page 19: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/19.jpg)
Space Filling Designs
•Factorial Designs
•Latin Hypercubes
•Pseudo random sequences
•Sobol sequence
![Page 20: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/20.jpg)
The Full Factorial
•We set each input (factor) at a set number of levels
•All combinations are included in the design
•n levels of m factors needs nm points
•This gets large quickly
![Page 21: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/21.jpg)
An Example•52 factorial
![Page 22: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/22.jpg)
Fractional Factorial
•Full factorials are expensive
•For large number of factors only 2 or 3 levels
•Can use fractional factorials (2 levels)
![Page 23: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/23.jpg)
The Latin Hypercube
•Decide how many simulator runs you can afford
•Divide each input range into that number of intervals
•Allocate a point to each interval
•Randomly permute across each input
![Page 24: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/24.jpg)
The Latin Hypercube
![Page 25: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/25.jpg)
The Latin Hypercube
•We don’t have an algorithm for the optimal Latin hypercube
•What is a good Latin hypercube?
‣ Maximin
‣ Orthogonal designs
‣ Pragmatic designs
![Page 26: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/26.jpg)
A Latin Hypercube
![Page 27: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/27.jpg)
A maximin LHC
![Page 28: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/28.jpg)
Are Factorials better than Latin
Hypercubes
![Page 29: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/29.jpg)
Low Discrepancy Sequences
•Alternative to Latin hypercubes
•Designed for multi-dimensional integrals
•Examples include Halton sequences, Niederreiter nets and Sobol sequences
![Page 30: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/30.jpg)
Sobol Sequences
•A low discrepancy sequence
•A 2n-1 Sobol sequence is a Latin hypercube
•Some projections of multi-dimensional Sobol sequences are not ‘good’
![Page 31: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/31.jpg)
Sobol Sequences
![Page 32: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/32.jpg)
Sobol Sequences
![Page 33: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/33.jpg)
Sequential Designs
•So far our designs have been one off
•We make a design and that dictates how we run the simulator
•We do not learn from the early runs
•An idea from clinical trials is to learn as we carry out the experiment
![Page 34: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/34.jpg)
Sequential Design for Computer Experiments
•Perform an initial experiment (usually space filling)
•Add additional points to satisfy some criteria
•We might add additional points where our predictions of simulator output are most uncertain
•We might add additional points for optimisation
![Page 35: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/35.jpg)
A D-optimal design for smoothness
•I’m fitting an emulator to a computer experiment
•Can we design an experiment to estimate the ‘smoothness’ parameters of the emulator optimally?
![Page 36: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/36.jpg)
Emulators
•δ is a zero mean Gaussian process
•This is defined in terms of a variance (σ2and a correlation function (C(x1,x2))
![Page 37: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/37.jpg)
An Example of an Emulator
![Page 38: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/38.jpg)
Zhu and Stein (2004)
•In the geostatistical context Zhu and Stein show that the Fisher information is approximately given by
![Page 39: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/39.jpg)
Bayesian Design
•Approximate the inverse of the covariance matrix by the Fisher information matrix
•Set up a D-optimal design by maximising the utility
![Page 40: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/40.jpg)
5-point Sobol
![Page 41: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/41.jpg)
10-point Sobol
![Page 42: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/42.jpg)
![Page 43: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/43.jpg)
One at time (5) Five at a time
Sobol 10 +5
![Page 44: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/44.jpg)
![Page 45: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/45.jpg)
Designing for Multiple Climate
Models•So far we have considered designs
for single simulators
•How might we design for multiple models?
•The IPCC problem
•‘Ensemble of opportunity’?
![Page 46: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/46.jpg)
So What’s the Problem
•Common outputs between simulators
•Not common inputs
•An important area for research
![Page 47: Designing Ensembles for Climate Prediction Peter Challenor National Oceanography Centre](https://reader035.vdocument.in/reader035/viewer/2022062712/56649c915503460f9494b9d6/html5/thumbnails/47.jpg)
Conclusions•Designing model ensembles can
- make them more efficient
- make the experimenter think about the problem
•There are a variety of designs around
•Consult a statistician before you design the experiment
•Design of computer experiments is an active area of research (not only in climate/environmental sciences)