Probabilistic Prediction

Uncertainty in Forecasting

• All of the model forecasts I have talked about reflect a deterministic approach.

• This means that we do the best job we can for a single forecast and do not consider uncertainties in the model, initial conditions, or the very nature of the atmosphere. These uncertainties are often very significant.

• Traditionally, this has been the way forecasting has been done, but that is changing now.

A More Fundamental Issue• The work of Lorenz (1963, 965, 1968)

demonstrated that the atmosphere is a chaotic system, in which small differences in the initialization…well within observational error… can have large impacts on the forecasts, particularly for longer forecasts.

• Similarly, uncertainty in model physics can result in large forecast differences..and errors.

• Not unlike a pinball game….• Often referred to as the “butterfly

effect”

Probabilistic-Ensemble NWP

• One approach would be to add uncertainty terms to all terms in the primitive equations. Not practical.

• Another: Instead of running one forecast, run a collection (ensemble) of forecasts, each starting from a different initial state or with different physics. Became practical in the late 1980s as computer power increased.

Ensemble Prediction

•The variations in the resulting forecasts could be used to estimate the uncertainty of the prediction. Can use ensembles to provide a new generation of products that give the probabilities that some weather feature will occur.

•Can predict forecast skill or forecast reliability!•It appears that when forecasts are similar, forecast skill is higher.•When forecasts differ greatly, forecast skill is less.

•The ensemble mean is usually more accurate on average than any individual ensemble member.

Probabilistic Prediction

• A critical issue will be the development of mesoscale ensemble systems that provide probabilistic guidance that is both reliable and sharp.

Elements of a Good Probability Forecast

• Reliability (a.k.a. calibration) – A probability forecast p, ought to verify with relative

frequency p.

– Forecasts from climatology are reliable (by definition), so calibration alone is not enough.

Elements of a Good Probability Forecast

• Sharpness (a.k.a. resolution) – The variance, or confidence interval,of the

predicted distribution should be as small as possible.

Probability Density Function (PDF)for some forecast quantity

Sharp

LessSharp

Early Forecasting Started Probabilistically

• Early forecasters, faced with large gaps in their nascent science, understood the uncertain nature of the weather prediction process and were comfortable with a probabilistic approach to forecasting.

• Cleveland Abbe, who organized the first forecast group in the United States as part of the U.S. Signal Corp, did not use the term “forecast” for his first prediction in 1871, but rather used the term “probabilities,” resulting in him being known as “Old Probabilities” or “Old Probs” to the public.

• A few years later, the term ‘‘indications’’ was substituted for probabilities and by 1889 the term ‘‘forecasts’’ received official sanction (Murphy 1997).

“Ol Probs”

Professor Cleveland Abbe, who issued the first public“Weather Synopsis and Probabilities” on February 19, 1871

•Cleveland Abbe (“Ol’ Probabilities”), who led the establishment of a weather forecasting division within the U.S. Army Signal Corps,

•Produced the first known communication of a weather probability to users and the public.

History of Probabilistic Prediction

• The first operational probabilistic forecasts in the United States were produced in 1965. These forecasts, for the probability of precipitation, were produced by human weather forecasters and thus were subjective predictions. The first objective probabilistic forecasts were produced as part of the Model Output Statistics (MOS) system that began in 1969.

Ensemble Prediction

• Ensemble prediction began an NCEP in the early 1990s. ECMWF rapidly joined the club.

• During the past decades the size and sophistication of the NCEP and ECMWF ensemble systems have grown considerably, with the medium-range, global ensemble system becoming an integral tool for many forecasters.

• Also during this period, NCEP has constructed a higher resolution, short-range ensemble system (SREF) that uses breeding to create initial condition variations.

NCEP Global Ensemble System• Begun in 1993 with the MRF (now GFS)• First tried “lagged” ensembles as basis…using runs of various

initializations verifying at the same time.• For the last ten years have used the “breeding” method to find

perturbations to the initial conditions of each ensemble members.• Breeding adds random perturbations to an initial state, let them

grow, then reduce amplitude down to a small level, lets them grow again, etc.

• Give an idea of what type of perturbations are growing rapidly in the period BEFORE the forecast.

• Does not include physics uncertainty.• Coarse spatial resolution..only for synoptic features.

NCEP Global EnsembleAt 00Z:• T254L64 high resolution control) out to 7 days, after which this run

gets truncated and is run out to 16 days at a T170L42 resolution• T62 control that is started with a truncated T170 analysis • 10 perturbed forecasts each run at T62 horizontal resolution. The

perturbations are from five independent breeding cycle.

At 12Z:• T254L64 control out to 3 days that gets truncated and run at

T170L42 resolution out to 16 days• Two pairs of perturbed forecasts based on two independent breeding

cycles (four perturbed integrations out to 16 days.

The Thanksgiving Forecast 200142h forecast (valid Thu 10AM)

13: avn*

11: ngps*

12: cmcg*

10: tcwb*

9: ukmo*

8: eta*

Verification

1: cent

7: avn

5: ngps

6: cmcg

4: tcwb

3: ukmo

2: eta

- Reveals high uncertainty in storm track and intensity- Indicates low probability of Puget Sound wind event

SLP and winds

NCEP Short-Range Ensembles (SREF)

• Resolution of 32 km• Out to 87 h twice a day (09 and 21 UTC

initialization)• Uses both initial condition uncertainty

(breeding) and physics uncertainty.• Uses the Eta and Regional Spectral Models

and recently the WRF model (21 total members)

SREF Current System

Model Res (km) Levels Members Cloud Physics ConvectionRSM-SAS 45 28 Ctl,n,p GFS physics Simple Arak-SchubertRSM-RAS 45 28 n,p GFS physics Relaxed Arak-Schubert

Eta-BMJ 32 60 Ctl,n,p Op Ferrier Betts-Miller-JanjicEta-SAT 32 60 n,p Op Ferrier BMJ-moist prof

Eta-KF 32 60 Ctl,n,p Op Ferrier Kain-FritschEta-KFD 32 60 n,p Op Ferrier Kain-Fritsch

with enhanced detrainment

PLUS

* NMM-WRF control and 1 pert. Pair* ARW-WRF control and 1 pert. pair

There is a whole theory on using probabilistic information for

economic savings

C= cost of protection

L= loss if bad event event occurs

Decision theory says you should protect if the probability of

occurrence is greater than C/L

Critical Event: sfc winds > 50kt

Cost (of protecting): $150K

Loss (if damage ): $1M

Hit

FalseAlarm

Miss

CorrectRejection

YES NO

YES

NO

Forecast?

Obs

erve

d?

Decision Theory Example

Deterministic Observation ProbabilisticCase Forecast (kt) (kt) Cost ($K) Forecast 0% 20% 40% 60% 80% 100%

1 65 54 150 42% 150 150 150 1000 1000 10002 58 63 150 71% 150 150 150 150 1000 10003 73 57 150 95% 150 150 150 150 150 10004 55 37 150 13% 150 0 0 0 0 05 39 31 0 3% 150 0 0 0 0 06 31 55 1000 36% 150 150 1000 1000 1000 10007 62 71 150 85% 150 150 150 150 150 10008 53 42 150 22% 150 150 0 0 0 09 21 27 0 51% 150 150 150 0 0 0

10 52 39 150 77% 150 150 150 150 0 0Total Cost: 2,050$ 1,500$ 1,200$ 1,900$ 2,600$ 3,300$ 5,000$

$150K $1000K

$150K $0K

Deterministic Observation ProbabilisticCase Forecast (kt) (kt) Cost ($K) Forecast 0% 20% 40% 60% 80% 100%

1 65 54 150 42% 150 150 150 1000 1000 10002 58 63 150 71% 150 150 150 150 1000 10003 73 57 150 95% 150 150 150 150 150 10004 55 37 150 13% 150 0 0 0 0 05 39 31 0 3% 150 0 0 0 0 06 31 55 1000 36% 150 150 1000 1000 1000 10007 62 71 150 85% 150 150 150 150 150 10008 53 42 150 22% 150 150 0 0 0 09 21 27 0 51% 150 150 150 0 0 0

10 52 39 150 77% 150 150 150 150 0 0Total Cost: 2,050$ 1,500$ 1,200$ 1,900$ 2,600$ 3,300$ 5,000$

Cost ($K) by Threshold for Protective Action

Optimal Threshold = 15%

The Most Difficult Part: Communication of Uncertainty

Deterministic Nature?

• People seem to prefer deterministic products: “tell me exactly what is going to happen”

• People complain they find probabilistic information confusing. Many don’t understand POP.

• Media and internet not moving forward very quickly on this.

Icons are not effective in providing probabilities

Even worse…they use the same icons for likely rain and rain as they do for chance

rain. Also, they used “likely rain” for 70% on this page and “chance rain” for

70% in the example on the previous page

And a “slight” chance of freezing drizzle reminds one of a trip to

Antarctica

Commercial sector

is no better

A great deal of research and development is required to

develop effective approaches for communicating probabilistic

forecasts which will not overwhelm people and allow them to get value out of them.