eumetcal nwp-course 2007: the concept of ensemble forecasting renate hagedorn european centre for...
TRANSCRIPT
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Renate Hagedorn European Centre for Medium-Range Weather Forecasts
The General Concept of
Ensemble Forecasting in Theory and Practice
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Goals
• Students to learn:
Why & how are probabilistic forecasts produced & used?
• Teacher to learn:
What are your greatest needs & expectations from an EPS?
• Achieve together:
What is the best way forward to integrate uncertainty information
as an integral component into public weather forecasts?
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Outline
• Why do we need Ensemble Prediction Systems?
Chaos theory and its consequences for weather prediction
• How are probabilistic forecasts made in practice?
How do we represent uncertainties?
From ensemble members to PDF’s and CDF’s
• Good ensembles – bad ensembles?
How to verify probabilistic forecasts
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
The Philosophical Point of View…
To know what you know, and to know what you do not know,that is real knowledge
ConfuciusThe Analects
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
The Practical Point of View…
orPredicting predictability is as important as predicting rainfall
No forecast is complete without a forecast of forecast skill
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
The Practical Point of View…
orPredicting predictability is as important as predicting rainfall
• Weather Forecasts have errors (are uncertain)
• Ultimate goal of weather forecasting is to improve user decisions, i.e. decision-making based on forecast information should be superior to decision-making without forecast information
• Decision-making can be improved when uncertainty information is available
WHY?
No forecast is complete without a forecast of forecast skill
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Weather Forecasting: How does it work?
Numerical modelto describe the processes
in the earth system
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Weather Forecasting: How does it work?
Observationsto start the forecast
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Weather Forecasting: How does it work?
Computer
Observations
Model
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Deterministic Forecasting
Forecast time
Tem
pera
ture
Initial condition Forecast
Is this forecast “correct”?
Initial Uncertainty
Model Error
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
The Lorenz Attractor
“… one flap of a sea-gull’s wing
may forever change the
future course of the weather”
(Lorenz, 1963)
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
The Lorenz Attractor…
bZXYZ
YrXXZY
YXX
...is the visualization of the
time-evolution of a three-
dimensional non-linear
dynamical system described
by the ‘Lorenz-63‘ equations
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Scientific basis for Ensemble Predictions
In a non-linear dynamical system, the growth of uncertainties in initial conditions is flow dependant
bZXYZ
YrXXZY
YXX
IC IC IC
cold warmcold warm cold warm
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Ensemble Forecasting
Forecast time
Tem
pera
ture
Complete description of weather prediction in terms of aProbability Density Function (PDF)
Initial condition Forecast
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Flow dependence of forecast errors
If the forecasts are coherent (small spread) the atmosphere is in a more
predictable state than if the forecasts diverge (large spread)
aa
34
30
28
26
24
22
20
18
16
14
12
10
0 1 2 3 4 5 6 7 8 9 10Forecast day
UK
Control Analysis Ensemble
ECMWF ensemble forecast - Air temperatureDate: 26/06/1994 London Lat: 51.5 Long: 0
30
28
26
24
22
20
18
16
14
12
10
8
0 1 2 3 4 5 6 7 8 9 10Forecast day
UK
Control Analysis Ensemble
ECMWF ensemble forecast - Air temperatureDate: 26/06/1995 London Lat: 51.5 Long: 0
26th June 1995 26th June 1994
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Goal of Ensemble Prediction
• Represent/predict uncertainty of prediction
• Move from deterministic to probabilistic forecast
• Ensemble Spread should
capture “truth” (spread ~ RMS error)
indicate range of uncertainty
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Outline
• Why do we need Ensemble Prediction Systems?
Chaos theory and its consequences for weather prediction
• How are probabilistic forecasts made in practice?
How do we represent uncertainties?
From ensemble members to PDF’s and CDF’s
• Good ensembles – bad ensembles?
How to verify probabilistic forecasts
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Sources of uncertainties
• Initial conditions: limited accuracy of observations and data assimilation
Run ensemble of forecasts from slightly different conditions. Initial perturbations generated via singular vector technique, breeding vectors, ETKF, etc…
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Initial Perturbations
• Problem: How to construct “good” initial perturbations, given that only a
number of limited integrations can be carried out?
• Solution: Find initial perturbations with maximum amplification rate
• Singular vector approach: Perturbations with the fastest growth over a finite time intervall
(SV) can be identified in solving an eigenvalue problem of the product of the tangent forward and adjoint model propagator
22/101
**2/10 PMEEPME
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Example of initial perturbations
150OW 100 OW 50OW 0O 50OE 100OE 150OE
50.0N
1000800600500400300
200
Cross section of temp 20060321 00 step 0 Expver 0001
60°S60°S
30°S30°S
0°0°
30°N30°N
60°N60°N
120°W
120°W
60°W
60°W
0°
0°
60°E
60°E
120°E
120°E21/03/2006 00UTC, Temperature (every 0.2K) @~700hPa
@ 50°N
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Sources of uncertainties
• Initial conditions: limited accuracy of observations and data assimilation
Run ensemble of forecasts from slightly different conditions. Initial perturbations generated via singular vector technique, breeding vectors, ETKF, etc…
• Model error parameterisations: how to represent unresolved processes
o stochastic physics approach physical parameter values: inaccurate knowledge of parameter space
o perturbed parameter approach model structure: how to represent physical processes in models
o multi-model approach
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Stochastic Physics
• Background:
Assume that parameterization is tuned to give correct ensemble mean Account for statistical fluctuations using random numbers
• ECMWF implementation: assign random numbers [0.5,1.5] to 10º lat/lon boxes multiply model parameterization tendencies by these random numbers assign new random numbers every 6 hours
Skill measure: area under ROC curve Event: precipitation > 40 mm/day
Top curves: winter performance
Bottom curves: summer performance
Buizza et al, 1999
SP: yes
SP: no
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
RMS error
Spread
No SPBS
SPBS
New Stochastic Physics (SPBS)
Courtesy: Judith Berner
under-dispersion reduced for all forecast ranges
RMS error reduced
u-component 850hPa, Tropics
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Sources of uncertainties
• Initial conditions: limited accuracy of observations and data assimilation
Run ensemble of forecasts from slightly different conditions. Initial perturbations generated via singular vector technique, breeding vectors, ETKF, etc…
• Model error parameterisations: how to represent unresolved processes
o stochastic physics approach physical parameter values: inaccurate knowledge of parameter space
o perturbed parameter approach model structure: how to represent physical processes in models
o multi-model approach
• Boundary conditions: SST, soil moisture, sea ice, etc.
Unknown changes in boundary conditions are source of uncertainty, however, known (or well modelled) external forcing can be source of predictability for extended range forecasts
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Outline
• Why do we need Ensemble Prediction Systems?
Chaos theory and its consequences for weather prediction
• How are probabilistic forecasts made in practice?
How do we represent uncertainties?
From ensemble members to PDF’s and CDF’s
• Good ensembles – bad ensembles?
How to verify probabilistic forecasts
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Ensemble Prediction System
• 1 control run + 50 perturbed runs (TL399 L62)
added dimension of ensemble members
f(x,y,z,t,e)
• How do we deal with added dimension when
interpreting, verifying and using EPS output?
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
From ensembles to PDF’s
6 8 10 12 14Temperature [Degree Celsius]
0.0
0.5
1.0
1.5
2.0
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Count members per bin
6 8 10 12 14Temperature [Degree Celsius]
0
5
10
15
20
25
num
ber
of m
embe
rs in
1 d
eg b
in
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Discrete probability distribution
6 8 10 12 14Temperature [Degree Celsius]
0.0
0.1
0.2
0.3
0.4
0.5
prob
abili
ty o
f T
emp
in 1
deg
bin
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Continuous probability density function
6 8 10 12 14Temperature [Degree Celsius]
0.0
0.1
0.2
0.3
0.4
0.5
prob
abili
ty d
ensi
ty f
unct
ion
(PD
F)
f (x)1
2exp
x 2 2
2
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Continuous PDF
6 8 10 12 14Temperature [Degree Celsius]
0.0
0.1
0.2
0.3
0.4
0.5
prob
abili
ty d
ensi
ty f
unct
ion
(PD
F)
f (x)1
2exp
x 2 2
2
f (x) is the “probability density” function,
or PDF
is the “mean”
is the “standard deviation”
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
-5 0 5 10 15 20 25Temperature [Degree Celsius]
0.0
0.2
0.4
0.6
0.8
1.0
prob
abili
ty d
ensi
ty f
unct
ion
(PD
F)
Continuous PDF’s
f (x)1
2exp
x 2 2
2
Which forecast has a mean of 10 degree celsius?
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
-5 0 5 10 15 20 25Temperature [Degree Celsius]
0.0
0.2
0.4
0.6
0.8
1.0
prob
abili
ty d
ensi
ty f
unct
ion
(PD
F)
Continuous PDF’s
f (x)1
2exp
x 2 2
2
Which forecast has the highest σ (standard deviation)?
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Continuous PDF’s
-5 0 5 10 15 20 25Temperature [Degree Celsius]
0.0
0.2
0.4
0.6
0.8
1.0
prob
abili
ty d
ensi
ty f
unct
ion
(PD
F)
f (x)1
2exp
x 2 2
2
= 0.0, = 0.5
= 10.0, = 1.0
= 15.0, = 2.0
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
What information can we get from PDF’s?
• PDF’s describe the probability density of a continuous spectrum of possible outcomes
• Probability density describes relative likelihood to be near a particular value
• We have to distinguish between: Continuous events: unlimited number of possible outcomes
(temperature, windspeed, …) Discrete events: limited number of possible outcomes
(rain/no-rain, temperature below/above freezing,…)
• Probabilities are only meaningful for discrete events P(9≤T≤11) can be determined from PDF (or CDF) P(T=10°C) = 0
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Probabilities related to event
6 8 10 12 14Temperature [Degree Celsius]
0.0
0.1
0.2
0.3
0.4
0.5
prob
abili
ty o
f T
emp
in 1
deg
bin
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Probabilities related to event
6 8 10 12 14Temperature [Degree Celsius]
0.0
0.1
0.2
0.3
0.4
0.5
prob
abili
ty o
f T
emp
in 0
.2 d
eg b
in
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
6 8 10 12 14Temperature [Degree Celsius]
0.0
0.1
0.2
0.3
0.4
0.5
prob
abili
ty d
ensi
ty f
unct
ion
(PD
F)
Probability density function
f (x)1
2exp
x 2 2
2
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
6 8 10 12 14Temperature [Degree Celsius]
0.0
0.1
0.2
0.3
0.4
0.5
prob
abili
ty d
ensi
ty f
unct
ion
(PD
F)
From PDF’s to probabilities
a
f(x)dxa)P(X
f (x)1
2exp
x 2 2
2
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
From PDF’s to CDF’s
6 8 10 12 14Temperature [Degree Celsius]
0.0
0.2
0.4
0.6
0.8
1.0
PD
F &
CD
F
X
f(x)dxF(X)
f (x)1
2exp
x 2 2
2
a
f(x)dxa)P(X
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Probabilities from a CDF
6 8 10 12 14Temperature [Degree Celsius]
0.0
0.2
0.4
0.6
0.8
1.0
CD
F
What is the probability that the temperature will be 10°C?
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Probabilities from a CDF
6 8 10 12 14Temperature [Degree Celsius]
0.0
0.2
0.4
0.6
0.8
1.0
CD
F
What is the probability that the temperature will be 10°C?
P(X=10) = 0
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Probabilities from a CDF
6 8 10 12 14Temperature [Degree Celsius]
0.0
0.2
0.4
0.6
0.8
1.0
CD
F
What is the probability that the temperature will be ≤10°C?
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Probabilities from a CDF
6 8 10 12 14Temperature [Degree Celsius]
0.0
0.2
0.4
0.6
0.8
1.0
CD
F
What is the probability that the temperature will be ≤10°C?
P(X≤10) = 0.5
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Probabilities from a CDF
6 8 10 12 14Temperature [Degree Celsius]
0.0
0.2
0.4
0.6
0.8
1.0
CD
F
What is the probability that the temperature will be >11°C?
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Probabilities from a CDF
6 8 10 12 14Temperature [Degree Celsius]
0.0
0.2
0.4
0.6
0.8
1.0
CD
F
What is the probability that the temperature will be >11°C?
P(X>11) = P(X≤∞) – P(X≤11) = 1. - 0.85 = 0.15
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Probabilities from a CDF
6 8 10 12 14Temperature [Degree Celsius]
0.0
0.2
0.4
0.6
0.8
1.0
CD
F
What is the probability that the temperature will be between 9-11°C?
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Probabilities from a CDF
6 8 10 12 14Temperature [Degree Celsius]
0.0
0.2
0.4
0.6
0.8
1.0
CD
F
What is the probability that the temperature will be between 9-11°C?
P(9≤X≤11) = P(X≤11) – P(X<9) = 0.85 – 0.15 = 0.70
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
From PDF’s to CDF’s
-5 0 5 10 15 20 25Temperature [Degree Celsius]
0.0
0.2
0.4
0.6
0.8
1.0P
DF
-5 0 5 10 15 20 25Temperature [Degree Celsius]
0.0
0.2
0.4
0.6
0.8
1.0
CD
F
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Ensemble Prediction System
• 1 control run + 50 perturbed runs (TL399 L62)
added dimension of ensemble members
f(x,y,z,t,e)
• How do we deal with added dimension when
interpreting, verifying and using EPS output?
• Transition from forecasting local events (22°C) to categorical events (>20°)
deterministic (yes/no) to probabilistic (x%)
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Outline
• Why do we need Ensemble Prediction Systems?
Chaos theory and its consequences for weather prediction
• How are probabilistic forecasts made in practice?
How do we represent uncertainties?
From ensemble members to PDF’s and CDF’s
• Good ensembles – bad ensembles?
How to verify probabilistic forecasts
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Objective of diagnostic/verification tools
Assessing the goodness of a forecast system involvesdetermining skill and value of forecasts
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Objective of diagnostic/verification tools
Assessing the goodness of a forecast system involvesdetermining skill and value of forecasts
A forecast has skill if it predicts the observed conditions well according to some objective or subjective criteria.
A forecast has value if it helps the user to make better decisions than without knowledge of the forecast.
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Objective of diagnostic/verification tools
Assessing the goodness of a forecast system involvesdetermining skill and value of forecasts
A forecast has skill if it predicts the observed conditions well according to some objective or subjective criteria.
A forecast has value if it helps the user to make better decisions than without knowledge of the forecast.
• Forecasts with poor skill can be valuable (e.g. location mismatch)
• Forecasts with high skill can be of little value (e.g. blue sky desert)
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Assessing the quality of a forecast
• The forecast indicated 10% probability for rain
• It did rain on the day
• Was it a good forecast?
□ Yes
□ No
□ I don’t know
• Single probabilistic forecasts are never completely wrong or right (unless they give 0% or 100% probabilities)
• To evaluate a forecast system we need to look at a (large) number of forecast–observation pairs
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Assessing the quality of a forecast system
• Characteristics of a forecast system:
Consistency*: Do the observations statistically belong to the distributions of the forecast ensembles? (consistent degree of ensemble dispersion)
Reliability: Can I trust the probabilities to mean what they say?
Sharpness: How much do the forecasts differ from the climatological mean probabilities of the event?
Resolution: How much do the forecasts differ from the climatological mean probabilities of the event, and the systems gets it right?
Skill: Are the forecasts better than my reference system (chance, climatology, persistence,…)?
* Note that terms like consistency, reliability etc. are not always well defined in verification theory and can be used with different meanings in other contexts
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Rank Histogram
• Rank Histograms asses whether the ensemble spread is consistent with the assumption that the observations are statistically just another member of the forecast distribution
Check whether observations are equally distributed amongst predicted ensemble
Sort ensemble members in increasing order and determine where the observation lies with respect to the ensemble members
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Rank Histogram
• Rank Histograms asses whether the ensemble spread is consistent with the assumption that the observations are statistically just another member of the forecast distribution
Check whether observations are equally distributed amongst predicted ensemble
Sort ensemble members in increasing order and determine where the observation lies with respect to the ensemble members
Temperature ->
Rank 1 case
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Rank Histogram
• Rank Histograms asses whether the ensemble spread is consistent with the assumption that the observations are statistically just another member of the forecast distribution
Check whether observations are equally distributed amongst predicted ensemble
Sort ensemble members in increasing order and determine where the observation lies with respect to the ensemble members
Temperature ->
Rank 1 case Rank 4 case
Temperature ->
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Rank Histograms
A uniform rank histogram is a necessary but not sufficient criterion for determining that the ensemble is reliable (see also: T. Hamill, 2001, MWR)
OBS is indistinguishable from any other ensemble member
OBS is too often below the ensemble members (biased forecast)
OBS is too often outside the ensemble spread
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Reliability
• A forecast system is reliable if: statistically the predicted probabilities agree with the observed
frequencies, i.e. taking all cases in which the event is predicted to occur with a
probability of x%, that event should occur exactly in x% of these cases; not more and not less.
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Reliability
• A forecast system is reliable if: statistically the predicted probabilities agree with the observed
frequencies, i.e. taking all cases in which the event is predicted to occur with a
probability of x%, that event should occur exactly in x% of these cases; not more and not less.
• A reliability diagram displays whether a forecast system is reliable (unbiased) or produces over-confident / under-confident probability forecasts
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Reliability
• A forecast system is reliable if: statistically the predicted probabilities agree with the observed
frequencies, i.e. taking all cases in which the event is predicted to occur with a
probability of x%, that event should occur exactly in x% of these cases; not more and not less.
• A reliability diagram displays whether a forecast system is reliable (unbiased) or produces over-confident / under-confident probability forecasts
• A reliability diagram also gives information on the resolution (and sharpness) of a forecast system
Forecast PDFClimatological PDF
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Reliability Diagram
Take a sample of probabilistic forecasts: e.g. 30 days x 2200 GP = 66000 forecasts
How often was event (T > 25) forecasted with X probability?
FC Prob. # FC “perfect FC”OBS-Freq.
“real” OBS-Freq.
100% 8000 8000 (100%) 7200 (90%)
90% 5000 4500 ( 90%) 4000 (80%)
80% 4500 3600 ( 80%) 3000 (66%)
…. …. …. ….
…. …. …. ….
…. …. …. ….
10% 5500 550 ( 10%) 800 (15%)
0% 7000 0 ( 0%) 700 (10%)
25
25
25
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Reliability Diagram
Take a sample of probabilistic forecasts: e.g. 30 days x 2200 GP = 66000 forecasts
How often was event (T > 25) forecasted with X probability?
FC Prob. # FC “perfect FC”OBS-Freq.
“real” OBS-Freq.
100% 8000 8000 (100%) 7200 (90%)
90% 5000 4500 ( 90%) 4000 (80%)
80% 4500 3600 ( 80%) 3000 (66%)
…. …. …. ….
…. …. …. ….
…. …. …. ….
10% 5500 550 ( 10%) 800 (15%)
0% 7000 0 ( 0%) 700 (10%)
OB
S-F
req
uency
0 100
100
••
•
••FC-Probability
0
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Reliability Diagram
over-confident model perfect model
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Reliability Diagram
under-confident model perfect model
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Reliability diagram
Reliability score (the smaller, the better)
imperfect model perfect model
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Components of the Brier Score
2
1
)(1
ii
I
ii ofn
NREL
N = total number of casesI = number of probability binsni = number of cases in probability bin i
fi = forecast probability in probability bin I
oi = frequency of event being observed when forecasted with fi
Reliability: forecast probability vs. observed relative frequencies
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Sharpness
Reliability Diagram
0.0 0.2 0.4 0.6 0.8 1.0Forecast Probability
0.0
0.2
0.4
0.6
0.8
1.0
Obs
erve
d F
requ
ency
Event: 500 hPa Geopotential anom. > 0.00 sigma
Area: West Africa (land only)
Model: DEMETER I
Start dates: Feb / 1980-2001
Avg. over FC period: 2-4 months (MAM)
Brier (Skill) Score: 0.203 ( 0.189)
B(S)S_Reliability: 0.026 ( 0.897)
B(S)S_Resolution: 0.073 ( 0.292)
Uncertainty: 0.250
0.0 0.2 0.4 0.6 0.8 1.00.0rel FC distribution
0.000.02
0.040.060.080.10
0.12
Reliability Diagram
0.0 0.2 0.4 0.6 0.8 1.0Forecast Probability
0.0
0.2
0.4
0.6
0.8
1.0
Obs
erve
d F
requ
ency
Event: 500 hPa Geopotential anom. > 0.00 sigma
Area: Northern Extratropics (land+sea)
Model: DEMETER I
Start dates: Feb / 1980-2001
Avg. over FC period: 2-4 months (MAM)
Brier (Skill) Score: 0.247 ( 0.011)
B(S)S_Reliability: 0.006 ( 0.977)
B(S)S_Resolution: 0.008 ( 0.034)
Uncertainty: 0.250
0.0 0.2 0.4 0.6 0.8 1.00.0rel FC distribution
0.00
0.01
0.02
0.03
0.04
Diagrams show the distribution of issued forecast probabilities
FC Probability FC Probability
Rel.
Fre
qu
en
cy
Rel.
Fre
qu
en
cy
Sample A Sample B
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Sharpness
Reliability Diagram
0.0 0.2 0.4 0.6 0.8 1.0Forecast Probability
0.0
0.2
0.4
0.6
0.8
1.0
Obs
erve
d F
requ
ency
Event: 500 hPa Geopotential anom. > 0.00 sigma
Area: West Africa (land only)
Model: DEMETER I
Start dates: Feb / 1980-2001
Avg. over FC period: 2-4 months (MAM)
Brier (Skill) Score: 0.203 ( 0.189)
B(S)S_Reliability: 0.026 ( 0.897)
B(S)S_Resolution: 0.073 ( 0.292)
Uncertainty: 0.250
0.0 0.2 0.4 0.6 0.8 1.00.0rel FC distribution
0.000.02
0.040.060.080.10
0.12
Reliability Diagram
0.0 0.2 0.4 0.6 0.8 1.0Forecast Probability
0.0
0.2
0.4
0.6
0.8
1.0
Obs
erve
d F
requ
ency
Event: 500 hPa Geopotential anom. > 0.00 sigma
Area: Northern Extratropics (land+sea)
Model: DEMETER I
Start dates: Feb / 1980-2001
Avg. over FC period: 2-4 months (MAM)
Brier (Skill) Score: 0.247 ( 0.011)
B(S)S_Reliability: 0.006 ( 0.977)
B(S)S_Resolution: 0.008 ( 0.034)
Uncertainty: 0.250
0.0 0.2 0.4 0.6 0.8 1.00.0rel FC distribution
0.00
0.01
0.02
0.03
0.04
Diagrams show the distribution of issued forecast probabilities
FC Probability FC Probability
Rel.
Fre
qu
en
cy
Rel.
Fre
qu
en
cy
Sample A Sample B
Which sample contains the sharper probability forecasts?
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Reliability diagram
Poor resolution Good resolution
Reliability score (the smaller, the better)
Resolution score (the bigger, the better)
c c
Size of red bullets represents number of forecasts in probability category (sharpness)
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Components of the Brier Score
2
1
)(1
ii
I
ii ofn
NREL
N = total number of casesI = number of probability binsni = number of cases in probability bin i
fi = forecast probability in probability bin I
oi = frequency of event being observed when forecasted with fi
c = frequency of event being observed in whole sample
Reliability: forecast probability vs. observed relative frequencies
Resolution: ability to issue reliable forecasts close to 0% or 100%
2
1
)(1
conN
RES i
I
ii
Uncertainty: variance of observations frequency in sample
)1( ccUNC
Brier Score = Reliability – Resolution + Uncertainty
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Brier Score
• The Brier score is a measure of the accuracy of probability forecasts
N
nnnN
BS op1
2
)(1
with p: forecast probability (fraction of members predicting event) o: observed outcome (1 if event occurs; 0 if event does not occur)
• BS varies from 0 (perfect deterministic forecasts) to 1 (perfectly wrong!)
• Considering N forecast – observation pairs the BS is defined as:
• BS corresponds to RMS error for deterministic forecasts
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Brier Skill Score
• Skill scores are used to compare the performance of forecasts with that
of a reference forecast such as climatology or persistence
cBS
BSBSS 1
• positive (negative) BSS better (worse) than reference
• Constructed so that perfect FC takes value 1 and reference FC = 0
Skill score = score of current FC – score for ref FC
score for perfect FC – score for ref FC
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Assessing the quality of a forecast system
• Characteristics of a forecast system:
Consistency: Do the observations statistically belong to the distributions of the forecast ensembles? (consistent degree of ensemble dispersion)
Reliability: Can I trust the probabilities to mean what they say?
Sharpness: How much do the forecasts differ from the climatological mean probabilities of the event?
Resolution: How much do the forecasts differ from the climatological mean probabilities of the even, and the systems gets it right?
Skill: Are the forecasts better than my reference system (chance, climatology, persistence,…)?
Relia
bili
ty D
iag
ram
Rank Histogram
Brier Skill Score
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Brier Score -> Ranked Probability Score
5 10 15 20 25
f(y)
• Brier Score used for two category (yes/no) situations (e.g. T > 15oC)
5 10 15 20 25
• RPS takes into account ordered nature of variable (“extreme errors”)
F(y)1
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Ranked Probability Score
category
f(y)
category
F(y)1
PD
F
CD
F
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Ranked Probability Score
K
kkOBSkFC CDFCDF
KRPS
1
2,, )(
1
1
category
f(y)
category
F(y)1
PD
F
CD
F
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Ranked Probability Score
category
f(y)
PD
F
RPS=0.01sharp & accurate
category
f(y)
PD
F
RPS=0.15sharp, but biased
category
f(y)
PD
F
RPS=0.05not very sharp, slightly biased
category
f(y)
PD
F
RPS=0.08accurate, but not sharp
climatology
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Ranked Probability Score
• Measures the quadratic distance between forecast and verification probabilities for several probability categories k
K
kkBS
KRPS
11
1• It is the average Brier score across the range of the variable
• Ranked Probability Skill Score (RPSS) is a measure for skill relative to a reference forecast
cRPS
RPSRPSS 1
• Emphasizes accuracy by penalizing large errors more than “near misses”• Rewards sharp forecast if it is accurate
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Example of RPSS for ECMWF’s EPS
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Goals
• Students to learn:
Why & how are probabilistic forecasts produced & used?
• Teacher to learn:
What are your greatest needs & expectations from an EPS?
• Achieve together:
What is the best way forward to integrate uncertainty information
as an integral component into public weather forecasts?
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Assignment-I
• What are your greatest needs and/or expectations from the probabilistic products of an EPS?
Are there any areas in your day-to-day work which benefit from probabilistic forecasts (now or in the future)?
Which aspect of the output of an EPS is most valuable for you?
Can you think of any information valuable for you which is currently not available as EPS product?
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Assignment-II
• How would you present a probabilistic weather forecast to the general public?
• Prepare one or more examples of a weather forecast containing probabilistic information for:
TV Radio Newspaper Internet Governmental agency (weather warning) Commercial company …
• Some more hints next week, but you might think already this week about your general concept
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
References and further reading
• Palmer, T. and R. Hagedorn (editors), 2006: Predictability of weather and climate. Cambridge University Press, pp.702
• Jolliffe, I.T. and D.B. Stephenson, 2003: Forecast Verification. A Practitioner’s Guide in Atmospheric Science. Wiley, pp. 240
• Wilks, D. S., 2006: Statistical methods in the atmospheric sciences. 2nd ed. Academic Press, pp.627
• ECMWF newsletter for updates on EPS performance
• Hamill, T., 2001: Interpretation of Rank Histograms for Verifying Ensemble Forecasts. Monthly Weather Review, 129, 550-560
• Buizza, R., Bidlot, J.-R., Wedi, N., Fuentes, M., Hamrud, M., Holt, G., and Vitart, F., 2007: The new ECMWF VAREPS (Variable Resolution Ensemble Prediction System). Q. J. Roy. Meteorol. Soc., 133, 681-695
• Leutbecher, M. and T.N. Palmer, 2007: Ensemble forecasting. J. Comp. Phys., in press
EUMETCAL NWP-course 2007: The Concept of Ensemble Forecasting
Web links:
• ECMWF products and training:
http://www.ecmwf.int/products/forecasts/d/charts
http://www.ecmwf.int/newsevents/training/meteorological_presentations/MET_PR.html
• NCEP ensemble training:
http://www.emc.ncep.noaa.gov/gmb/ens/training.html
http://www.hpc.ncep.noaa.gov/ensembletraining/
• Interactive learning on probabilities:
http://www.shodor.org/interactivate/activities/