on the use of data assimilation in the assessment of the...

On the use of data assimilation in the

assessment of the cost/benefit of

meterological observations and observing

systems.

Lars Peter Riishojgaard

Director, Joint Center for Satellite Data Assimilation

Chair, OPAG-IOS, WMO Commission for Basic Systems

6th WMO Data Assimilation Symposium

Overview

• Money and weather forecasting – Enabling capabilities

• Data and weather forecasting – Data requirements

– Cost of meteorological observations

• Data impacts and the WMO RRR

• Can we put a monetary value on individual observing systems and/or individual observations? (and should we?)

– Role of Data Assimilation community and WMO RRR

• Final remarks

WMO Data Assimilation Symposium,

College Park Oct 7-11 2013 2

WMO Data Assimilation Symposium, College

Park Oct 7-11 2013

Weather Prediction and the US

Economy; A Macroscopic View

• Department of Commerce: “20% of overall US economy is weather sensitive”: ~$3 trillion/year

– Impact to air and surface transportation, agriculture, construction, energy production and distribution, etc.

• Assume that half of this is “forecast sensitive”: $1.5 trillion/year

• Assume that the potential savings due to weather forecasting amount to 5% of the “forecast sensitive total”: ~$75B/year

3

(discussed during CBS TECO in Windhoek,

2010)


Park Oct 7-11 2013

… a Macroscopic View … (II)

• “Perfect forecast” is an NWP run with useful skill at two

weeks!

• 0 h useful forecast range => $0 in savings

• 336 h useful forecast (two weeks maximum predictability)

range => $75B in savings

• Assume now that the savings are distributed linearly over the

achieved forecast range for the global NWP system: – $75B/336h ~ $223B/hr

• This implies that the value to the United States economy of

weather observations, dissemination, forecast products and

services is >$220M per hour of forecast range per year !

4

(discussed during TECO in Windhoek, 2010)

The global picture

• The amount of $75B/year is one estimate of the

magnitude of the total potential socioeconomic benefit

of weather prediction activities to the US economy

• Scaling exercise, using World Bank (2011) numbers:

• Annual GDP of United States: ~$15T

• Annual GDP of all nations combined: ~$70T

– Assuming on average (i) equal sensitivity to weather, and (ii)

equal potential benefits from ability to predict across all

nations, we get an estimated

$75B *($15T/$70T) = $350B as the total global potential

benefit of weather prediction activities (indicating a likely

range of $100B to $1T)



Weather Prediction Enabling Capabilities

1. Observing Systems (GOS0

2. Dissemination Systems

3. Numerical Weather Prediction

– Science (modeling, data assimilation)

– High-end computing



• 1, 2 and 3 are of a foundational nature

• Among the foundational capabilities, 1 represents

the single largest expenditure


Park Oct 7-11 2013

NWP requirements for upper-

air data coverage

7

Hence the need for a global

observing system, irrespective

of target location of forecast!

Estimating the total cost of running

all components of the GOS

• Informal exercise launched by the WMO Commission

for Basic Systems in 2012

• Approach was (perhaps too?) simple:

– Survey a small, but representative number of WMO member

states about their total annual investment in running,

maintaining and updating the GOS

– Use Cost-to-GDP ratios to extrapolate that to the rest of the

WMO members

– Add estimates for external (non-NMHS owned) contributions:

• Satellite systems

• Aircraft observations

• Third party networks

• Marine observations

• …



NMHS input to survey (anonymous)

Country GOS cost

(K USD, 2011)

Expenditure

on GOS, GDP

fraction

Comment

1

8330 1.3 x 10-5 Excludes radar data

2 39096 1.1 x 10-5

3 7793 1.6 x 10-3 (LDC)

4 14400 2.3 x 10-4 (Amortization

unclear)

… … … …



What do we think we know about the global

cost of acquiring the observations?

• Ratios are too disparate to be used for scaling, but

the running costs of the conventional parts of the

GOS appear to be in the single-digit $B-range

• “Non-NMHS”-provided data (satellite, third-party

networks, marine, aircraft observations) add an

estimated $5B total

• Total costs of all meteorological observations

add up to an estimated $5-10B per year

• That is a lot of money, and decision-makers are

on the hunt for justification and objective metrics!



What do NWP and data assimilation have to do with this?

Objective, quantitative metrics:

NWP poses a well-defined prediction problem with a “right” answer

(and an infinity of wrong ones)

Well-defined measures for quality of output

Well-established methodologies for assigning merit (or blame) to individual observing systems

WMO Data Assimilation Symposium, College Park Oct 7-11 2013 11

WMO Commission for Basic System, Rolling Review of Requirements

Under its Commission for Basic Systems WMO maintains a standing Expert Team (ET-EGOS) which is responsible for documenting

1. Observational data requirements for all 12 WMO application areas (database)

2. Capability of all relevant observing system (databases)

3. Statements of Guidance, or gap analyses, matching 1. against 2 (a brief narrative for each application area)

4. …

ET-EGOS is helped in its work by numerous other ET’s and

Rapporteurs and by the WMO NWP Impact Workshops taking place once every four years

WMO Data Assimilation Symposium, College Park Oct 7-11 2013 12

WMO Workshops on the Impact of Various

Observing Systems on NWP

Five Workshops so far (the last two in close collaboration with THORPEX):

• 1st - Geneva, 1997

• 2nd – Toulouse, 2000

• 3rd – Alpbach, 2004

• 4th – Geneva, 2008

- Workshop Report available on

http://www.wmo.int/pages/prog/www/OSY/Reports/NWP-4_Geneva2008_index.html

• 5th – Sedona AZ, May 2012

http://www.wmo.int/pages/prog/www/OSY/Meetings/NWP5_Sedona2012/Final_Report.pdf

Workshops aim to bring together scientists from all major NWP centers to

assess the contribution to forecast skill of various elements of the global

observing system; guidance to participants provided well in advance of

Workshop itself.

13 WMO Data Assimilation Symposium,

College Park Oct 7-11 2013




http://www.wmo.int/pages/prog/www/OSY/Meetings/NWP5_Sedona2012/Final_Report.pdf

OSE and FSO (I)

• OSEs (Observing System Experiments) are based on data denial (or addition)

• Impact focuses on the medium to long range

• Results show the impact of withdrawing (or adding) certain data

• OSE results are absolute; e.g. “observing system X extends the useful forecast range by N hours in the NH”

WMO Data Assimilation Symposium, College Park Oct 7-11 2013

14

Jung et al., WMO Impact Workshop in Sedona, May 2012

OSE and FSO (II)

• FSO (Forecast Sensitivity to Observations) are based on the adjoint of the model/analysis system or an ensemble approach

• Approach focuses exclusively on the short (quasi-linear) range

• Results show the impact of observations in the presence of all other observations

• FSO measures of impact are relative (e.g. often expressed in percentages that add up 100, even for poor forecasts or poor system performance)

WMO Data Assimilation Symposium, College Park Oct 7-11 2013

15

Gelaro et al, Fifth WMO Impact Workshop, Sedona 2012

4th WMO Impact Workshop, Alpbach 2008; impact

summary slide

Overall impact

(“marginal skill”) on

global NWP

16 WMO Data Assimilation Symposium,

College Park Oct 7-11 2013

Some Preliminary Conclusions from the Fifth

WMO Impact Workshop in Sedona, May 2012

• Modern, 4-dimensional data assimilation methods (4D-VAR,

ENKF) have led to greatly improved use of data, especially of

– Asynoptic data (e.g. aircraft, satellite observations)

– Observations with complex relationships between measured

and model variables (satellite radiances, GPSRO, radar,…)

• Broad consensus about highest-ranking contributors to forecast

skill, but not necessarily about their ranking order:

– AMSU-A (microwave temperature sounder)

– AIRS/IASI (hyper-spectral infrared sounders)

– Radiosondes

– Aircraft observations

– Atmospheric motion vectors (feature tracking satellite winds)



Some Preliminary Conclusions (II)

• Radio occultation data (GPSRO) also have substantial impact

but data volumes are currently declining as COSMIC is

approaching the end of its lifetime

• There is now no single, dominating satellite sensor; several

sensors contribute to forecast skill in roughly equal measure

• The relative impacts of specific observation types depends on

which other observations are used and how

– If certain data are withheld, other datatypes can in some

contexts compensate for the lost skill

• However, the continued value of in situ data, and in particular of

wind measurements, was clearly demonstrated

• Regional data assimilation systems making progress in the use

of radar and satellite observations

– Radiance assimilation still problematic



Forecast impact experiment from Dec. 2010 to Jan. 2011

Impact Impact / Obs. number

WMO Workshop on the Impact of Various Observing Systems on NWP Sedona – 22-25 May 2012

Could we use this type of FSO information

to rank observing systems by impact per

dollar?

1

9

Of course we can! Simply divide the

impact by the cost of running the

system and come up with a third

“impact per dollar” bar chart!

Obs impact and value, versus cost

• Armed with the following three pieces of information

– Cost of acquiring the observations (difficult but not impossible to

acquire this information)

– Overall economic benefits of products derived from

observations (can be WAG’ed or properly analyzed by trained

economists)

– Individual contribution of observing systems to NWP skill (done

through RRR, previous slide),

we could in fact take the reasoning on the previous

slide one step further and provide cost/benefit

numbers for individual components of the GOS; it is

not an easy exercise, but probably doable



Should we use DA methodology for

Cost Benefit analysis for observations?

• Caveat!

– The respective contributions of the components of the GOS vary

from center to center, even at comparable levels of skill

– The contributions vary depending on which other GOS

components are used in the experiments

– OSE results are expensive to acquire, and often inconclusive

– FSO impacts are relative; even if the overall forecast

performance is poor, some observing systems may stand out

due to their large share in a modest improvement

– The sum of the contributions always add up to 100, down until

the very last GOS component standing

• When are we done cutting?



Final remarks

• The economic impact of weather is relatively well understood

• In contrast, the economic impact of weather prediction is generally not well studied and documented

• The cost/benefit of meteorological observations are a subject of intense interest among program managers and decision makers

– The costs are incurred (and known) mostly at the regional levels, but the benefits are realized and assessed globally

– Tendency to focus too much on NWP diagnostics due to their compelling nature



Final remarks (II)

• The NWP and data assimilation community is being dragged into this whether we like it or not

– Victim of our own success

– Does our community need to collectively (e.g. through WMO/CAS, THORPEX) adopt a position on this issue?

• We need to remind people that while NWP is important and can provide very compelling metrics, it is only one of many application areas supported by the GOS

• New, widely adopted community standard metrics needed?

– Not much room for further improvement of the 500 hPa AC scores



on the use of data assimilation in the assessment of the...

Documents