CTB Science Plan

For

Multi Model Ensembles (MME)

Suru Saha

Environmental Modeling Centre

NCEP/NWS/NOAA

Vision A multi model ensemble forecast system that

leverages the best national and international models for improved predictions on intraseasonal, interannual

and decadal time scales

Background• Studies have shown that the forecast skill of a multi-

model system is higher than that of most of the individual models

• To maximize the amount of model information for NOAA’s seasonal forecasts, a multi-model ensemble strategy will be pursued by the CTB

Current Status of IMME• There has been progress on a possible International

MME (IMME) with the CFS and the EUROSIP models (ECMWF, UKMO and MeteoFrance)

• Preliminary discussions have taken place between NCEP and ECMWF on protocols for the transfer of the hindcast data, as well as real time forecasts.

• Each center will make available new hindcasts, if and when, they upgrade their seasonal forecast system.

Current Status of NMME• There has been little progress on a possible National

MME (NMME) with the CFS and the NCAR, NASA and GFDL models

• Results from a previous version of the GFDL model and data assimilation system did not show significant additional skill to the CFS, for the basic metrics of US surface temperature and precipitation.

Current Status of NMME (contd)• GFDL has made some new hindcasts.

Unfortunately, the CTB has no resources to evaluate these hindcasts.

• Regarding the NCAR model, Ben Kirtman has been funded to make the hindcasts using the same strategy as the CFSRR

• Regarding the NASA model, there has been no update on the completion of their hindcasts.

Gaps• There are three major gaps in executing a NMME

strategy• Firstly, skill levels for all models are barely above

climatological probabilities for US surface temps and precipitation, at monthly and seasonal time scales.

• Unless there is a significant improvement in these metrics, the value of a NMME may not be sufficient to justify the major cost of making hindcasts to support an operational NMME.

Gaps (contd)• Secondly, the computational and human costs of

preparing hindcasts for evaluation of skill and calibration is significant, and not presently funded by NOAA for models other than the CFS.

• Thirdly, the cost and logistics of running a NMME for all models, in operations, are cosiderable and has not yet been supported by NOAA.

Future Strategy of the NMME (Option 1)

• With less than a 100% investment, including modest computer resources and a staff of 1 full time and 2-3 part time personnel, the CTB could support the current strategy of:

• All participating centers make hindcasts of 2 initial months (for the mid-May and mid-November release of CPC’s official monthly and seasonal forecast) over the period 1979-present (or more years, if possible).

Future Strategy of the NMME (Option 1)• A dataset of prescribed output variables from these hindcasts

(full fields) will be made available to the CTB consolidation team. They will make an objective evaluation of these monthly and seasonal hindcasts, to determine if skill is added to the CFS for the basic metrics of US T & P.

• Systematic error correction will be done for each tool separately, and the regression coefficients will be determined under a three-years-out Cross-Validation approach.

• If the weight assigned to a model is near zero, then the consolidation will be redone without that model and the originators of the model will be notified.

Future Strategy of the NMME (Option 1) contd• If there is additional skill to the CFS, the participating center

will work with EMC in porting the frozen model and data assimilation systems to NCEP. Random hindcasts will be made using this system to ensure that there is reproducibility of the results on the NCEP operational/development computers.

• This is important for the application of calibration and weights for consolidation (which are derived from the retrospective forecasts) to real time forecasts which will be made on the NCEP computer.

• This process will involve computer and human resources, which will need to be funded.

Future Strategy of the NMME (Option 1) contd• If the results are reproducible, then the participating

center will complete the rest of the 10 calendar months with the exact same frozen system.

• They will make the same datasets available to the CTB consolidation team, for further evaluation to determine if skill is being added to the CFS for these months.

• If there is additional skill to the CFS, then EMC will work with NCO (computer operations) to implement the NMME into operations at NCEP.

Future Strategy of the NMME (Option 2)• With 100% investment, including a dedicated NOAA

mainframe computer, with a huge number of nodes, disk space, tapes, and a support staff on the order of 20 to 25 full time personnel, CTB could support porting all participating systems (including models and assimilation systems) to the CTB computer.

• CTB would then proceed with making the hindcasts as described above. This could be done with individual models or with combinations of model physics and numerics within the Earth System Modeling Framework (ESMF) to form a true NMME.

Future Strategy of the NMME (Option 2) contd

The seven year cycle• Start CFS Reanalysis at same time as porting all the other

models (GFDL, NCAR, NASA) to the NOAA NMME computer.

• Make sure Reanalysis (CFS) initial conditions can be used in all these models (GFDL, NCAR, NASA), or make Reanalysis for each model system.

• This implies that improvements that have been made to these models and data assimilation systems every 7 years will be utilized in the NMME, in a continuous fashion, hopefully leading to an increase in skill of the NMME every time.

Future Strategy of the NMME (Option 2) contdThe seven year cycle

• Make hindcasts (2 months, mid-May and mid-November release) for each model system separately or in an optimal combination of different physics parameterizations and/or different dynamic cores

• Evaluate to see if NMME has additional skill to the CFS (CTB Consolidation team)

• If additonal skill exists, make hindcasts for remaining 10 calendar months

• Redo evaluation for additional skill to the CFS for these months

• If additional skill exists, then transition to NCEP operations.

Why the Need for Long Hindcast Datafor S/I prediction ?

Huug van den Dool (CPC)

and Suranjana Saha (EMC)National Centers for Environmental Prediction,

NWS/NOAA/DOC

“Statistical correction of today’s numerical forecasts using a long set

of reforecasts (hindcasts) dramatically

improves its forecast skill”

(Hamill, Whitaker and Mullen 2006, BAMS)

(Single model, multi-membered ensemble, NWP)

Data Used for this study• DEMETER : 7 European Coupled GCMs• CFS• 1981-2001 (21 years of hindcasts)• 4 initial months• Monthly Data

Focus:

Skill in Tmp2m over US lead 3

February forecasts from November initial conditions

Explained Variance (%) for US monthly T2m

Feb 1981-2001; lead 3 (Nov starts)Verification : Climate Division data)

SE

corr

CFS ECM PLA MET FRA

UKM INGV LOD CER

0 years

2.1 1.2 0.0 0.0 0.0 0.4 0.2 0.0

8years

4.3 7.1 1.4 1.4 7.5 1.4 0.4 2.2

21years

11.2(0.33 corr)

8.0 0.4 0.4 8.6 0.6 0.1 0.5

The 3 best models are operational models

Skill Estimates for 8-years are not stable

Best skill obtained for all 21 years

SE

corrCFS ECM PLA MET

FRAUKMO

INGV LODYC

CERF

MME (EW)

0 years

2.1 1.2 0.0 0.0 0.0 0.4 0.2 0.0 0.2

8years

4.3 7.1 1.4 1.4 7.5 1.4 0.4 2.2 3.8

21years

(all)

11.2 8.0 0.4 0.4 8.6 0.6 0.1 0.5 2.0

Explained Variance (%) for US monthly T2m

Feb 1981-2001; lead 3 (Nov starts)Verification : Climate Division data)

Skill of MME with equal weights for ALL models is far less than the skill of the 3 best models

Explained Variance (%) for US monthly T2m

Feb 1981-2001; lead 3 (Nov starts)Verification : Climate Division data)

SE

corr

CFS ECM PLA MET FRA

UKM ING LOD CER MME (EW)

MME3

0 years

2.1%

1.2 0.0 0.0 0.0 0.4 0.2 0.0 0.2 0.9

8

years

4.3 7.1 1.4 1.4 7.5 1.4 0.4 2.2 3.8 8.6

21

years

11.2 8.0 0.4 0.4 8.6 0.6 0.1 0.5 2.0 17.0

(corr=

51%)

Skill of MME with equal weights for 3 best models improves with 21 years

Need more years to determine the SE where/when

interannual stand dev is large

8-year 21-year

8yr minus 21yr

CFS US TMP2M LEAD 3 FORECAST VERIFYING IN FEB

8-year 21-year

MME, ALL MODELS, EQUAL WEIGHT

US TMP2M LEAD 3 FORECAST VERIFYING IN FEB

8yr minus 21yr

Skill of MME is very much less

than CFS when all models are

included

MME, 3 BEST MODELS, EQUAL WEIGHT

US TMP2M LEAD 3 FORECAST VERIFYING IN FEB

8-year 21-year

8yr minus 21yr8yr minus 21-year

Skill of MME for 3 best models is the most for 21 years

Explained Variance (%) for US monthly PrecipFeb 1981-2001; lead 3 (Nov starts)Verification : Climate Division data)

SE

corrCFS ECM PLA MET

FRAUKM INGV LOD CER

0 years

0.7 0.4 1.1 1.1 NA 1.1 0.1 0.9

8years

5.1 0.6 1.9 1.6 NA 1.8 0.1 1.5

21years

7.5 1.0 3.0 2.2 NA 3.3 0.1 1.9

Not much Skill in any of the models for US precipitation

Skill in SST Anomaly Prediction for Nino-3.4[DJF 97/98 to AMJ 04]

5-member CFS reforecasts

50

60

70

80

90

100

1 2 3 4 5 6

Forecast Lead [Month]

An

om

aly

Co

rrel

atio

n [

%]

CFS

CMP

CCA

CA

MAR

CON

Skill in SST Anomaly Prediction for Nino-3.4 [DJF 97/98 to AMJ 04]

50

60

70

80

90

100

1 2 3 4 5 6

Forecast Lead [Month]

An

om

aly

Co

rre

lati

on

[%

]

CFSCMPCCACAMRKCON

15-member CFS reforecasts

Skill in SST Anomaly Prediction for Nino-3.4[DJF 81/82 to AMJ 04]

5-member CFS reforecasts

50

60

70

80

90

100

1 2 3 4 5 6

Forecast Lead [Month]

An

om

aly

Co

rrel

atio

n [

%]

CFS

CMP

CCA

CA

MAR

Skill in SST Anomaly Prediction for Nino-3.4 [DJF 81/82 to AMJ 04]

50

60

70

80

90

100

1 2 3 4 5 6

Forecast Lead [Month]

An

om

aly

Co

rre

lati

on

[%

]CFSCMPCCACAMRK

15-member CFS reforecasts

“ONLY 1 BIG ENSO EVENT”

ALL ENSO EVENTS

CONCLUSIONS• Without SEC (systematic error correction) there is no skill

by any method (for presumably the best month: Feb)

• With SEC (1st moment only), there is skill by only a few models (5 out of 8 are still useless)

• MME not good when quality of models varies too much

• MME3 works well, when using just three good models

CONCLUSIONS (contd)

• CFS improves the most from extensive hindcasts (21 years noticeably better than 8) and has the most skill. Other models have far less skill with all years included.

• Cross validation (CV) is problematic (leave 3 years out when doing 8 year based SEC?)

• Need more years to determine the SEC where/when the inter annual standard deviation is large