national hurricane center 2008 forecast verification james l. franklin branch chief, hurricane...

National Hurricane Center 2008 Forecast Verification

James L. FranklinBranch Chief, Hurricane Specialists Unit

National Hurricane Center

2009 Interdepartmental Hurricane Conference

1

Verification Rules Verification rules unchanged for 2008. Results

presented here in both basins are final.

System must be a tropical or subtropical cyclone at both forecast initial time and verification time. All verifications include depression stage except for GPRA goal verification.

Special advisories ignored (original advisory is verified.

Skill baselines are recomputed after the season from operational compute data. Decay-SHIFOR5 is the intensity skill benchmark.

2008 Atlantic Verification

VT NT TRACK INT(h) (n mi) (kt)============================ 000 373 5.7 1.8012 346 27.7 7.1 024 318 48.3 10.4036 288 68.6 12.1048 261 88.2 13.6072 221 126.9 14.6096 177 159.8 13.8120 149 191.8 17.2

Values in green exceed all-time records.

* 48 h track error for TS and H only (GPRA goal) was 87.5 n mi, just off last year’s record of 86.2.

Atlantic Track Errors vs. 5-Year Mean

Official forecast was better than the 5-year mean, even though the season’s storms were “harder” than normal.

Atlantic Track Error Trends

Errors have been cut in half over the past 15 years. 2008 was best year ever.

Atlantic Track Skill Trends

2008 was the most skillful year on record at all time periods.

Atlantic 5-Year Mean Track Errors

Track errors increase by about 50-55 n mi per day. 48 hr mean error below 100 n mi for the first time.

Intensity errors level off because intensity is a much more bounded problem. New 5-yr means slightly larger than last year’s.

OFCL Error Distributions and Cone Radii

Only modest reductions in the cone radii.

2008 Track Guidance

Official forecast performance was very close to the consensus models.

Best model was ECMWF, which was so good that it as good or better than the consensus.

BAMD was similar to the poorest of the 3-D models (UKMET).

AEMI excluded due to insufficient availability (less than 67% of the time at 48 or 120 h).

2008 Track Guidance

Examine major dynamical models to increase sample size.

ECMWF best at all time periods (as opposed to last year, when it was mediocre). GFDL also better than last year (and better than HWRF). As we’ve seen before, GFDL skill declines relatively sharply at days 4-5.

NOGAPS and GFNI again performed relatively poorly. GFNI upgrades were delayed.

GFDL-HWRF Comparison

Much larger sample than last year shows that the HWRF is competitive with, but has not quite caught up to the GFDL yet. Consensus of the two (mostly) better than either alone.

Guidance Trends

Return to more “traditional” relationships among the models after the very limited sample of 2007.

Guidance Trends

Relative performance at 120 h is more variable, although GFSI has been strong every year except 2005. GFDL is not a good performer at the longer ranges.

Consensus Models

Best consensus model was TVCN, the variable member consensus that includes EMXI. It does not appear that the “correction” process was beneficial.

Consensus Models

Third year in a row AEMI trailed the control run. Multi-model ensembles remain far more effective for TC forecasting. ECMWF ensemble mean is also not as good as the control run (EEMN v EMX).

Atlantic Intensity Errors vs. 5-Year Mean

OFCL errors in 2008 were at or below the 5-yr means, but the 2008 Decay-SHIFOR errors were also at or below their 5-yr means, so not much change in skill.

Atlantic Intensity Error Trends

No progress with intensity.

Atlantic Intensity Skill Trends

Little net change in skill over the past several years.

2008 Intensity Guidance

Split decision between the dynamical vs statistical models. New ICON consensus, introduced this year, was very successful, beating OFCL except at 12 h.

OFCL adds most value over guidance at shorter ranges. Modest high bias in 2008 (2007 was a low bias).

2008 Intensity Guidance

HWRF competitive through 3 days, with issues at the longer times. Although the sample was smaller, there was a hint of this last year as well. Cannot shut GFDL off yet!

2008 Intensity Guidance

When the complication of timing landfall/track dependence is removed, OFCL performs better relative to the guidance. Dynamical models are relatively poor performers.

2008 East Pacific Verification

VT NT TRACK INT(h) (n mi) (kt)============================000 311 10.7 1.4012 276 30.9 6.0024 240 47.5 9.8036 206 63.7 11.9048 176 78.0 12.9072 124 107.6 15.7096 84 138.8 17.6120 52 161.4 18.0

Values in green tied or exceeded all-time lows.

EPAC Track Error Trends

Since 1990, track errors have decreased by 30%-50%.

EPAC Track Skill Trends

Skill continues to improve.

2008 Track Guidance

EMXI, EGRI, AEMI, FSSE, GUNA, TCON excluded due to insufficient availability.

Official forecast beat the TVCN consensus at later periods; beat each individual model. OFCL far superior to model guidance at longer time periods (also beat consensus at 4-5 days last year).

2008 Track Guidance

Relax selection criteria to see all major dynamical models. ECMWF best overall. OFCL clearly doing something right.

EPAC Intensity Error Trends

Perhaps just a hint of improvement?

EPAC Intensity Skill Trends

Skill does seem to be inching upward…

2008 Intensity Guidance

OFCL mostly beat the individual models and even the consensus at some time periods. OFCL wind biases turn sharply negative at 96-120 h, which was also true in 2007.

Statistical models outperformed dynamical models. This year, DSHP beat LGEM (flip from 2007).

2007-08 Genesis Forecast Verification

2007-08 Genesis Forecast VerificationLead-Time Analysis

for Disturbances that became Tropical Cyclones

Time wrt/Genesi

s-48 h -42 h -36 h -30 h -24 h -18 h -12 h -6 h

Avg. % 31% 31% 34% 37% 41% 45% 53% 61%

Time wrt/Genesi

s-48 h -42 h -36 h -30 h -24 h -18 h -12 h -6 h

Avg. % 29% 30% 28% 30% 31% 35% 42% 52%

Atlantic

Eastern North Pacific

Genesis Bins for 2009ATLANTIC

Range (%) % Expected % Verified # Forecasts

0-20 (Low) 9 6 916

30-50 (Med) 37 34 246

60-100 (High) 70 62 108

EASTERN NORTH PACIFIC

Range (%) % Expected % Verified # Forecasts

0-20 (Low) 11 19 540

30-50 (Med) 38 52 166

60-100 (High) 69 79 63

NHC will issue operational public quantitative/categorical genesis forecasts in 2009 and include categorical forecasts in the text Tropical Weather Outlook.

Summary: Atlantic Track OFCL track errors set records for accuracy at all time

periods. Errors continue their downward trends, skill was also up.

OFCL track forecast skill was very close to that of the consensus models, was beaten by EMXI.

EMXI and GFDL provided best dynamical track guidance. UKMET, which performed well in 2007, did not do so in 2008. NOGAPS lagged again.

HWRF has not quite attained the skill of the GFDL, but is competitive. A combination of the two is better than either alone.

Best consensus model was TVCN (variable consensus with EMXI). Multi-model consensus – good. Single model consensus – not so good. Not a good year for the “corrected consensus” models.

Summary: Atlantic Intensity OFCL errors in 2008 were below the 5-yr

means, but the 2008 Decay-SHIFOR errors were also lower than its 5-yr mean, so no real change in skill.

Still no progress with intensity errors; OFCL errors have remained unchanged over the last 20 years. Skill has been relatively flat over the past 5-6 years.

Split decision between the statistical and dynamical guidance. Simple four-model consensus (DSHP/LGEM/HWRF/GHMI) beat everything else, including the corrected consensus model FSSE.

Summary: East Pacific Track OFCL track errors set records at 24-72 h.

OFCL beat individual dynamical models, and also beat the consensus at 4 and 5 days.

GFDL, HWRF, and ECMWF were strong performers, although ECMWF had trouble holding on to systems through 5 days.

There continues to be a much larger difference between the dynamical models and the consensus in the eastern North Pacific than there is in the Atlantic, which is suggestive of different error mechanisms in the two basins.

Summary: East Pacific Intensity

OFCL mostly beat the individual models and even the consensus at 12 and 36 h. OFCL wind biases turned sharply negative at 96-120 h, which was also true in 2007.

Best model at most time periods was a statistical model. DSHP provided most skillful guidance overall. HWRF continued to have trouble in this basin. Four-model intensity consensus performed very well.