Leonard A. Smith & Mark S. RoulstonLeonard A. Smith & Mark S. RoulstonCentre for the Analysis of Time SeriesCentre for the Analysis of Time Series

London School of EconomicsLondon School of Economics&&

Pembroke College, OxfordPembroke College, Oxfordwww.maths.ox.ac.uk/~roulstonwww.maths.ox.ac.uk/~roulstonwww.lse.ac.uk/collections/catswww.lse.ac.uk/collections/cats

Embracing Probability Forecasts on All Scales:Embracing Probability Forecasts on All Scales:Formulation, Communication, Value & Evaluation (End-to-End Forecasting)

Forecast Based Decision Making:Forecast Based Decision Making:

• Is there any information in the forecast?

• How can I best extract that information?

• Will anyone listen?– How to best communicate with rich, numerate users?– How to best communicate with the general public?– What to communicate the numerate managers of the

public?

• What (exactly) is the decision I am trying to make?(a) Time scale (short, medium, seasonal, climate).(b) Number of expected events given duration of interest .(c) Who is the user? (evacuation? insurance? or building?)

Main Points:Main Points:• The most useful flood forecasts will be probability forecasts.The most useful flood forecasts will be probability forecasts.

– Especially on longer time scales; where is there evidence of skill? Especially on longer time scales; where is there evidence of skill?

• Useful probability forecasts require end-to-end forecasting.Useful probability forecasts require end-to-end forecasting.• What is the baseline for current societal and economic usage?What is the baseline for current societal and economic usage?• Is there Is there (flood relevant)(flood relevant) value in current value in current (ensemble)(ensemble) NWP forecasts? NWP forecasts?

– Standard targets need not apply (heave provides an example).Standard targets need not apply (heave provides an example).– Standard methods may not be optimal (precip forecasts in a product space).Standard methods may not be optimal (precip forecasts in a product space).

• How can we follow uncertainty and inadequacy in compound models (end-How can we follow uncertainty and inadequacy in compound models (end-to-end)?to-end)?

Risk

Reaction• Economic value: Cost/Loss ratios apply to rich, rational, focused Economic value: Cost/Loss ratios apply to rich, rational, focused

users.users.

• To have To have societal valuesocietal value requires a response as well as a warning. requires a response as well as a warning.

• Realistic response models are required, providing new morals for:Realistic response models are required, providing new morals for:• The boy who cried wolfThe boy who cried wolf

• And NoahAnd NoahWhat is the goal of an Operational Warning System?

Cost Loss Evaluation: Binary Cost Loss Evaluation: Binary EventsEvents

• Assume a rich numerate user subject to:– A cost C to protect against event E;– A loss L if no protection is taken and event E occurs;– Zero loss if protection is taken.

• It follows that action should be taken if the objective probability of E is greater than C/L, assuming:– The user is interested in the long run (rich);– The user faces a binary choice (focused);– The forecasts are accurate PDFs;– The cost of the forecasts is negligible.

• Note an ensemble of size N is neither necessary nor sufficient for PDF resolution 1/N.

Beyond Cost/LossBeyond Cost/Loss

a)There may be no natural binary a)There may be no natural binary alternative.alternative.

b) If the cost is near the cost of ruin, it b) If the cost is near the cost of ruin, it can be rational to ignore the forecast.can be rational to ignore the forecast.

c) In a societal application (an c) In a societal application (an evacuation), the probability of action evacuation), the probability of action will show hysteresis.will show hysteresis.

We will touch each of these in turn.

Weather Roulette: A Simple ExampleWeather Roulette: A Simple Example

Each day you gamble your entire net worth on the temperature at Heathrow.

The amount you place on each outcome is proportional to your predicted probability of that outcome (Kelly Betting).

How would the ECMWF ensemble (EPS) fare against a house that set odds:

- using climatology?

- using Best Forecast Guide (BFG) from the ECMWF hi-resolution forecast?This provides a good analogue for statistical decision making.

Ideally, this calculation is done under the user’s utility function

Many economic users already Many economic users already effectively play weather roulette:effectively play weather roulette:

• They would be happy with probability forecasts.They would be happy with probability forecasts.– Can we deliver accountable PDFs?Can we deliver accountable PDFs?– Can we value operational PDFs?Can we value operational PDFs?

• Flood forecasting cuts across different models:Flood forecasting cuts across different models:– How can we track uncertainty end-to-end How can we track uncertainty end-to-end

across modelling communities?across modelling communities?

No, but assume good PDFs.

Yes, via Ignorance.

A forecast like this one is of great value, even of we cannot interpret it as a PDF.How do we interpret these scenarios?

or pass on the information in them?

Interpreting Simulations

New methods of ensemble interpretation may extract existing information.

Skill at day 8.

RMS skill scores are simply not relevant to extreme events (or events with integrated triggers)

Green > 80%80> Blue > 30%30> Red > 0%

Bonga

FloatingProductionStorage andOffloading vessel

Some users already value probability forecasts; their decision is then one of skill/cost between forecasts.

Postage stamp forecasts can be provided in the users variables: from significant wave height near a buoy to heave at the FPSO.

EPS dynamical ensemble is blue, dressed ensemble is red, verification (buoy) is green

The simulation(s) become a forecast when “dressed” to form a PDF.

The relevant storms have occurred before the forecasts are made.

In this case, the ensemble forecast has little marginal value given the BFG.

Draugen

At other locations, end to end forecasts extract information from ensembles members which is unobtainable from any single BFG.

EPS dynamical ensemble is blue, dressed ensemble is red, verification (buoy) is green

At Draugen local variations have impact and the dressed EPS reflect options which the BFG misses.

Dressed EPS bounds truth.

BFG dynamical ensemble is blue, dressed hi-res forecast is red, verification (buoy) green

Large unexpected waves.

Dressed BFG has a higher ignorance score.

Realistic Societal ResponseRealistic Societal Response

1) The boy who cried wolf:1) The boy who cried wolf:– 6 villagers for 1 hour at $10/hour6 villagers for 1 hour at $10/hour– 3 sheep at $200/sheep3 sheep at $200/sheep

-> C/L = 0.10-> C/L = 0.10

Yet the Villagers were unprepared to Yet the Villagers were unprepared to accept a 67% accept a 67% false alarmfalse alarm rate! rate!

Moral of the Story: If societal benefit is the aim, one must consider imperfect compliance when events are rare.

Realistic Response To Rare Realistic Response To Rare EventsEvents2) The case of Noah:2) The case of Noah:

– Unusual event forecast from trusted source.Unusual event forecast from trusted source.– Huge cost C.Huge cost C.– Unbounded Loss L.Unbounded Loss L.– One off gamble (this user will never face this One off gamble (this user will never face this

event twice, esp if no protection is taken),event twice, esp if no protection is taken),

-> C/L = ????-> C/L = ????

Yet the taking action proved worthwhile.Yet the taking action proved worthwhile.

Moral of the Story: The maths become irrelevant to rational action if the forecasts are not believed (or paid for), the stakes too large, or the costs too high.

Open QuestionsOpen QuestionsHow can we increase/identify forecast value? - Better communication of end-to-end uncertainty in compound models. - Better forecast archives for every operational model. - Active (adaptive) model/ensemble response to the previous forecast.

Alternatives to single model IC ensembles: - multi-model ensembles, - multi-parameterisation models, - product space interpretations, - novel approaches (especially for longer range forecasts).

Different users have different needs/horizons.

How to deal with model inadequacy in the climate change scenario?How to deal with model inadequacy in the climate change scenario?

If model inadequacy kills an accountable probability forecast strategy in the same way that uncertainty killed the single hi-resolution forecast strategy, then how should we evaluate our models?

Discussion Questions:Discussion Questions:

• What is the goal of an Operational Warning System?

• How to propagate uncertainty across families of models?– And between families of researchers?

• How to quantify the value society currently derives?

• How much information do current forecasts contain?

• How to transfer information to industry with maximum value?

• How to transfer information to society with maximum value?For the maths, see: www.maths.ox.ac.uk/~roulston