Bulk Statistics on Ensemble Model Forecasts for MDSS

Demo 2003

Paul Schultz

NOAA Forecast Systems Laboratory

June 17, 2003

The MDSS ensemble modeling component

• What is it?– Several computer model forecasts to supplement the NWS model

forecast services

• Why are we doing this?– Better forecasts. Just seeing if you’re paying attention.

• How does it work?– By combining multiple (imperfect) forecasts of the (imperfectly

observed) atmosphere, we can make a single ensemble forecast that is better than any of the forecasts that went into it.

2

Ensemble modeling

• Did it work during the 2003 Demo?– Not as well as it can. It shows promise. It can be improved.

3

The ensemble for Demo 2003

• Three models, two LBC source models, total of six ensemble members– models: MM5, RAMS, WRF– LBC sources (from NCEP): AVN, Eta– 6-hour cycle– 27-hour forecasts– 12-km grid

5

Bulk statisticsState variables, 12-hr forecasts

Feb 1 – Apr 8, 2003

6

Temperature (K) Wind speed (m/s) Dewpoint (K)

MM5-AVN 3.1 -0.7 2.5 +0.8 5.6 +1.5

MM5-Eta 3.0 -0.5 2.5 +0.8 5.5 +1.6

RAMS-AVN 5.8 -1.1 2.6 +1.6 6.5 -0.9

RAMS-Eta 5.9 -1.1 2.6 +1.7 6.9 -1.0

WRF-AVN 3.1 -0.4 2.4 +1.1 5.7 +1.4

WRF-Eta 3.1 -0.4 2.4 +1.0 5.7 +1.3

Precipitation verification

MDSS 0-3 h QPF Equitable Skill Score 133 runs from 1 Feb - 26 Mar 2003

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.01 0.1 0.25 0.5

Precip Threshold (in)

Eq

uit

able

Ski

ll S

core

MM5

RAMS

WRF

7

MDSS 0-3 h QPF Bias Score 133 runs from 1 Feb - 26 Mar 2003

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

0.01 0.1 0.25 0.5

Precip Threshold (in)

Fre

qu

ency

Bia

s

MM5

RAMS

WRF

A closer look

9 pm model runs, verifying only Iowa stations, entire expt

Improving the ensemble

• Remove unhelpful members– If we can’t fix RAMS problems, it’s gone

– Different LBC models don’t seem to help (?????)

8

Unhelpful members

MM5+Avn

WRF+Avn

MM5+Eta

WRF+Eta

The LBC models don’t add enough

diversity

Improving the ensemble

• Add good models– FSL/RUC a very good candidate for Demo 2004

• Change model configurations– WRF cloud/precip physics– Model cycle frequency, lead times, etc.

• Optimize use of available computing resources• Take advantage of what regional models do best

• Improve post-processing– Better PoP (probability of precip) estimates -- FSL– Better tuning procedures -- NCAR– Hope for “better” weather during tuning period

Reliability9

Percentage of expected FSL model runs

0

10

20

30

40

50

60

70

80

90

100

2/3 2/10 2/17 2/24 3/3 3/10 3/17 3/24 3/31 4/7

NCEP data problems Giant

snowstorm in Boulder

Planned power

outage at FSL

Reliability

• MM5 shows good reliability

• Others will improve with better scripting

10

Percentage of expected FSL model runs

0

10

20

30

40

50

60

70

80

90

100

MM5-AVN MM5-ETA RAMS-AVN RAMS-ETA WRF-AVN WRF-ETA

Photos from MDSS field trip

Bob Stradley and Ron Simmons

Downward-pointed radiometer mounted on rear-view mirror of

Jim Van Sickle’s truck

RWIS tower, I-35 south of Ames