Bayesian Processor of Ensemble Members:combining the Bayesian Processor of Output with Bayesian ModelAveraging for reliable ensemble forecasting

R. Marty1,2 V. Fortin2 H. Kuswanto3

A.-C. Favre4 E. Parent5

1Universite Laval, Quebec, Canada2Environnement Canada, Dorval, Canada3Institute Technology of Sepuluh Nopember, Indonesia4Grenoble INP/LTHE, Grenoble, France5AgroParisTech/INRA, Paris, France

12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

Outline

1. IntroductionEnsemble ForecastsObservation Dataset and VerificationWhy Calibrate Ensemble Predictions ?

2. Bayesian Processor of Ensemble MembersNotations and HypothesesBMA ComponentBPO ComponentCombining BPO with BMA

3. ResultsIdentification of the Optimum Training LengthVerification of Calibrated Ensemble Forecasts

4. Conclusions

2 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

Ensemble Forecasts

. . . a mean to assess the uncertainty in meteorological forecasts

North American Ensemble Forecasting System

• 21 members from GEM(CMC)

• 21 members from GFS(NCEP)

• Site-located air temperaturedownscaled from 1-degreegrid

• Runs 00Z

• Lead times from +24h to+384h by 24h

• Summer 2008observations vs ensemble forecast (Quebec City)

3 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

Observation Dataset and Verification

Canadian Daily Climate Data

• Temperature observation from meteorological station in JeanLesage Intl. Airport (YQB) in Quebec City over the period1978-2007

• Temperature observed at 00Z

• http://www.climat.meteo.gc.ca

Continuous Ranked Probability Score

Reliability

4 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

Observation Dataset and Verification

Canadian Daily Climate Data

Continuous Ranked Probability Score

• Comparison of cumulativedistribution functions fromensemble forecasts F, withobservation y through theHeaviside function

• Negatively oriented

å smaller is better

CRPS =

∫ ∞−∞{F (u)− H(u − y)}2 du

Reliability

4 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

Observation Dataset and Verification

Canadian Daily Climate Data

Continuous Ranked Probability Score

Reliability

• Statistical consistency between a priori predicted probabilitiesand a posteriori observed frequencies of the occurrence

• Reliability measured by the Reliability Component of theCRPS decomposition (Hersbach, 2000)

4 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

Why Calibrate Ensemble Predictions ?

Verification of raw ensemble forecasts on Summer 2008

• GFS: lowest skill for all leadtimes

• GEM+GFS: skill closed toGEM

å No additional benefit inforecasting by using 42members

• Ensemble predictions lessskillful than climatology

• Ensemble predictions are notreliable

GEM vs GFS vs GEM+GFS vs Clim

Skill (CRPS)

Reliability (CRPS Rel)

+ (y) log scale

5 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

Why Calibrate Ensemble Predictions ?

Verification of raw ensemble forecasts on Summer 2008

• Unfortunately uncertainty underestimated by current ensembleprediction systems (EPS)

• Unfortunately ensemble often provided at unsuitablespatial/temporal scales (e.g. for hydrological predictions)

5 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

Why Calibrate Ensemble Predictions ?

Verification of raw ensemble forecasts on Summer 2008

• Unfortunately uncertainty underestimated by current ensembleprediction systems (EPS)

• Unfortunately ensemble often provided at unsuitablespatial/temporal scales (e.g. for hydrological predictions)

å Statistical post-processing required to obtain reliable ensembleforecasts at appropriate scales

å Here: calibration of ensemble predictions from GEM

5 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

Notations and Hypotheses

Notationst valid dateS ensemble sizeh forecast’s lead timeyt predictand (quantity to predict)

X(h)t ensemble forecasts

X(h)t = {X (h)

t,s , s = 1, 2, ...,S}temporal independance: h is omitted

Assumptions

6 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

Notations and Hypotheses

Notations

Assumptions

• Ensemble members generated in the same way

å exchangeability

• Numerical model well suited for predicting an unobserved(latent) variable ξt (e.g. gridded temperature)

• Latent variable ξt exchangeable with all ensemble membersXt,s

• ξt contains all the information required to predict ytå Xt and yt are conditionally independant

6 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

BMA Component (Raftery et al., 2005)

• Law of total probability

p(yt |Xt) =

∫p(yt |ξt ,Xt)p(ξt |Xt)dξt

• As Xt and yt are conditionally independant

p(yt |Xt) =

∫p(yt |ξt)p(ξt |Xt)dξt

• Since the latent variable is exchangeable with ensemblemembers

p(ξt |Xt) ≈1

S

S∑s=1

δ(ξt − Xt,s)

å BMA framework: Non-parametric approximation of thepredictive distribution

p(yt |Xt) ≈1

S

S∑s=1

p(yt |ξt = Xt,s)

7 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

BMA Component (Raftery et al., 2005)

• Law of total probability

p(yt |Xt) =

∫p(yt |ξt ,Xt)p(ξt |Xt)dξt

• As Xt and yt are conditionally independant

p(yt |Xt) =

∫p(yt |ξt)p(ξt |Xt)dξt

• Since the latent variable is exchangeable with ensemblemembers

p(ξt |Xt) ≈1

S

S∑s=1

δ(ξt − Xt,s)

å BMA framework: Non-parametric approximation of thepredictive distribution

p(yt |Xt) ≈1

S

S∑s=1

p(yt |ξt = Xt,s)

7 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

BPO Component (Krzysztofowicz, 2004)

Bayes’ rulep(yt |ξt)︸ ︷︷ ︸posterior

∝ p(ξt |yt)︸ ︷︷ ︸likelihood

p(yt)︸ ︷︷ ︸prior

Prior distribution of the predictand = climatology

• Temperature: Gaussian distribution

• Estimated from the past 30 years (1978–2007) with a movingwindow of 5 days around the valid date

p(yt) = N(yt ;µ, σ

2)

Likelihood

8 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

BPO Component (Krzysztofowicz, 2004)

Likelihood function

• Assuming linear model with Gaussian residuals between thepredictand and the latent variable

ξt = α + βyt + εt εt ∼ N (0, σ2ε|y l1)

• But ξt not observable

Xt = α + βyt + ηt ηt ∼ N (0, σ2X |y l1)

• Bayesian specification of the linear regression with informativeconjugate priors

• Less information in Xt than in ξt : σ2ε|y ∈

[0;σ2

X |y

]å Optimal variance σ2

ε|y estimated by minimizing the CRPS

8 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

BPO Component (Krzysztofowicz, 2004)

Bayes’ rule p(yt |ξt)︸ ︷︷ ︸posterior

∝ p(ξt |yt)︸ ︷︷ ︸likelihood

p(yt)︸ ︷︷ ︸prior

Prior p(yt) = N(yt ;µ, σ

2)

Likelihood p(ξt |yt) = N(ξt ;α + βyt , σ

2ε|y

)

8 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

BPO Component (Krzysztofowicz, 2004)

Bayes’ rule p(yt |ξt)︸ ︷︷ ︸posterior

∝ p(ξt |yt)︸ ︷︷ ︸likelihood

p(yt)︸ ︷︷ ︸prior

Prior p(yt) = N(yt ;µ, σ

2)

Likelihood p(ξt |yt) = N(ξt ;α + βyt , σ

2ε|y

)å BPO framework:

p(yt |ξt) = N (yt ;µy |ξ, σ2y |ξ)

µy |ξ =σ2β2( ξt−αβ ) + σ2

ε|yµ

σ2β2 + σ2ε|y

σ2y |ξ =

σ2σ2ε|y

σ2β2 + σ2ε|y

8 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

Bayesian Processor of Ensemble Members

BMA

p(yt |Xt) ≈1

S

S∑s=1

p(yt |ξt = Xt,s)

BPOp(yt |ξt) = N (yt ;µy |ξ, σ

2y |ξ)

BPEM

p(yt |Xt) ≈1

S

S∑s=1

N (yt ;µy |Xt,s, σ2

y |Xt,s)

µy |Xt,s=σ2β2(

Xt,s−αβ ) + σ2

ε|yµ

σ2β2 + σ2ε|y

σ2y |Xt,s

=σ2σ2

ε|y

σ2β2 + σ2ε|y

9 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

Bayesian Processor of Ensemble Members

BPEM

p(yt |Xt) ≈1

S

S∑s=1

N (yt ;µy |Xt,s, σ2

y |Xt,s)

µy |Xt,s=σ2β2(

Xt,s−αβ ) + σ2

ε|yµ

σ2β2 + σ2ε|y

σ2y |Xt,s

=σ2σ2

ε|y

σ2β2 + σ2ε|y

• µy |Xt,s: weighted average of bias-corrected member

Xt,s−αβ

and of the prior mean from climatology µ

• σ2y |Xt,s

: weighted mixture of residuals variance of the linear

model σ2ε|y and of the climatological variance σ2

9 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

Bayesian Processor of Ensemble Members

• The predictive distributions’s shape depends on the empiricaldistribution of the ensemble members

å the predictive distribution is not necessary Gaussian

◦ • GEM / calibrated members Predictive dist.Observation Constituent dist.

9 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

Identification of the Optimum Training Length

Summer 2008

• training period:10-days to 50-daysjoint samples

• No significant gain inforecasts’s skill beyond15 days

+ (y) log scale

• Calibrated forecasts less skillful than climatology-basedforecasts for longer-range lead times

• Optimal training length: 15 days

• Short training length: limit effects of seasonality and frequentchanges to operational forecasting systems

10 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

Verification of Calibrated Ensemble Forecasts

Summer 2008 (Quebec City only)

• Optimal training length foreach calibration method

• Forecasts’ skill improved bycalibration (up to +192h)with similar pattern for eachcalibration method

• Significant improvment offorecasts’ reliability

+ (y) log scale

11 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

Verification of Calibrated Ensemble Forecasts

Summer 2008 (8 sites in Quebec)+24h +96h

GEM Clim. BPEM BPO ensBMA

11 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

Conclusions

Bayesian Processor of Ensemble Members. . .

. . . a new approach to calibrated ensemble forecasts

• Based on BMA and BPO frameworks

• Capable to generate reliable forecasts

• Outperforms slightly both the BMA and BPO approaches aswell as a climatology

• Short optimal training length: avoid negative impacts ofseasonality and of frequent changes to operational forecastingsystems

• Successfully applied to 7 other stations accross the Quebec

12 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France

Introduction Bayesian Processor of Ensemble Members Results Conclusions

Thanks for your attention

13 / 13 International Conference on Ensemble Methods in Geophysical Sciences 12-16 November 2012, Toulouse, France