an evaluation of the probabilistic information in multi-model ensembles

An evaluation of the probabilistic information in multi-model

ensembles

David Unger

Huug van den Dool , Malaquias Pena, Peitao Peng, and Suranjana Saha

30th Annual Climate Prediction and Diagnostics Workshop

October 25, 2005

Objective

• Produce a Probability Distribution Function (PDF) from the ensembles.

Challenge• Calibration• Account for skill• Retain information from ensembles (Or not if no skill)

Schematic illustration

Temperature

σ

Schematic example

Temperature (F)

σz

Kernel vs. Mean

Regression

ZF F

F

( )_ _ _ _

Z RZ^

Step 1. Standardization

Step 3. Make the forecast

Step 2. Skill Adjustment

F Z CC

^ ^

^ ^

^

σe

σe= σc√ 1-Rm2

______

Analysis of Ensemble Variance

V = Variance

Total = Explained + Unexplained

Variance Variance Variance

σc2 = Ve + Vu

σc2 = σFm

2 + σe

2

σc2 = σFm

2 + (E2 + σz

2)

^

^

Analysis of Ensemble Variance (Continued)

With help of some relationships commonly used in linear regression:

<E2 > = (Rm2 - Ri

2) σc2

σz2 = (1-2Rm

2 + Ri2) σc

2

Rz2 = 2Rm

2 - Ri2 Rz≤ 1.

^

Ensemble Calibration

ZF F

ii

F

( )

_ _ _ _

Step 1. Standardization

Step 3. Skill Adjustment

Zi = Rz Zi ’

Step 2. Ensemble Spread Adjustment

Zi’ = K(Zi - Zm) + Zm

F Z Ci C i

^ ^

^ ^

^

Step 4. Make the Forecast

Schematic example

Temperature (F)

σz

Rz=.97, Rfm=.94, Ri=.90

Rz=.93, Rfm=.87, Rf=.30

Rz=.85, Rfm=.67, Ri=.41

Rz=.62, Rfm=.46, Rf=.20

Weighting

Wgts: 50% Ens. 1, 17%Ens 2, 3, 4

Real time system

• Time series estimates of Statistics.– Exponential filter

FT+1 = (1-α)FT + α fT+1

- Initial guess provided from 1956-1981 CA statistics

Continuous Ranked Probability Score

Some Results

• Nino 3.4 SSTs

Operational system 15 CFS ,12 CA, 1 CCA , 1 MKV

• Demeter Data9 CFS, 12 CA, 1 CCA, 1 MKV

9 UKM, 9 MFR, 9 MPI,

9 ECM, 9 ING, 9 LOD, 9 CER,

Nino 3.4 SSTs

Nino 3.4 SST 5-month lead by initial time 1982-2001

-0.1

0

0.1

0.2

0.3

0.4

0.5

CR

PS

S

Feb May Aug Nov All

Con-Op

Con-86

Ens-86b

Ens86

0

0.1

0.2

0.3

0.4

0.5

0.6

CR

PS

S

1 2 3 4 5

Con-Op

Con-86

Ens-86b

CRPSS – Nino 3.4 SSTs

All Initial times 1982-2001

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

CR

PS

S

1 2 3 4 5

Con-86

Ens-86b

CRPSS – Nino 3.4 SSTs

All Initial times 1990-2001 (Independent)

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

CR

PS

S

c09 c21 c30 c39 c48 c57 c66 c75 c84 c86 t27 b27

Con-86

Ens-86b

CRPSS – Nino 3.4 SSTs 5-month Lead, All Initial times 1990-2001 (Independent data)

Reliability Nino 3.4 SST (1990-2001)

U.S. Temperature and Precipitation Consolidation

• 15 CFS

• 1 CCA

• 1 SMLR

Trends are removed from models

Statistics and distribution are computed

Trend added to end result.

Trend Problem

• Should a skill mask be applied? How much?

- This technique requires a quantitative estimate of the trend.

• Component models sometimes “learn” trends, making bias correction difficult. – Doubles trends.

• Errors in estimating high frequency model forecasts

U.S. T and P consolidation

Skill Mask on Trends No Skill Mask on Trends

.060

.091

.053 .063

.101 .160

.074 .105

.228 .372

.173 .259

.010 .067

.000 .000

Ensm

CRPS RPS

Cons

CRPS and RPS-3 (BNA) Skill Scores: Temperature

.020 .040

.010 -.025

.026 .047

.051 .045

.057 .070

.039 .045

.016 -.007

.052 .029

190 .253

.138 .190

.033 .048

.030 .037

-.001 -.013

.039 .019

.084 .121

.043 .074

.086 .110

.044 .050

High

Moderate

Low

None

Skill

.15

.07

.02

1-Month Lead, All initial times

Reliability U. S. Temperatures(1995-2003)

.018

.018

.017 .018

.005 .002

.005 .002

.078 .059

.076 .059

.055 .064

.055 .064

Ensm

CRPS RPS

Cons

CRPS and RPS-3 (BNA) Skill Scores: Precipitation

.001 .002

.002 .002

.005 .005

.005 .007

-.004 -.009

-.004 -.009

-.006 -.004

-.006 .004

.006 .011

.006 .009

.030 .037

.030 .037

.013 .007

.012 .007

.016 .031

.016 .029

.014 .020

.013 .020

High

Moderate

Low

None

Skill

.10

.05

.01

1-Month Lead, All initial times

Reliability U.S. Precipitation(1995-2003)

Conclusions

• Calibrated ensemble and ensemble means score very closely (by CRPS)

Calibrated ensembles seem to have a slight edge.

• No penalty for including many ensembles (but not much benefit either)

• Considerable penalty for including less skillful ensembles – Weighting is critical.

• Probabilistic predictions are reliable (when looked at in terms of a continuous PDF)

Conclusions (Continued)

• Calibrated ensembles tend to be slightly overconfident

• Trends are a major problem – and are outside the realm of consolidation (but they are critically important for seasonal temperature forecasting).

6-10 day Forecasts (based on Analogs)