Continued Development of Tropical Cyclone Wind Probability Products

John A. Knaff – PresentingCIRA/Colorado State University

and Mark DeMariaNOAA/NESDIS

Presentation at 60th Interdepartmental Hurricane Conference

Mobile, AL22 March 2006

60th IHC 22 March 2006 - Mobile, AL 2

Outline

• Training Activities• Verification Activities• Results/insights from examination of

hurricane warning break points

60th IHC 22 March 2006 - Mobile, AL 3

Training Activities

• Mark DeMaria coordinated with Rick Knabb to provide feedback on a TPC/NHC training session.

• Several cases rerun for 2004 and 2005– For Pablo Santos, Miami WFO for an experimental

algorithm that uses the probabilities.– Web page with examples and a product description

http://rammb.cira.colostate.edu/projects/tc_wind_prob

60th IHC 22 March 2006 - Mobile, AL 4

60th IHC 22 March 2006 - Mobile, AL 5

Verification: Current Status

• Developed:– Input data handling (GRIB, ATCF….)– Statistical Methods

• Scalar measures of skill, accuracy, confidence• Conditional measures

– Methods to assess deterministic forecasts• Remaining

– Treatment of the OFCL forecast & wind radii through 5 days.

– Integrating the pieces.– How to Interpret the statistics and optimize use.

60th IHC 22 March 2006 - Mobile, AL 6

Statistical methods: Probability Bias

• Mean Forecast Probabilities (Fi) minus the Mean Observed Frequencies (Ei) = 1 or 0

Determines if the probabilities over/under forecast the outcome.

N

kk

N

kk

E

FBias

1

1

60th IHC 22 March 2006 - Mobile, AL 7

Statistical Methods: Brier Score

• Mean of square of the Forecast Probabilities (Fi) minus the Observed Frequencies (Ei) = 1 or 0

Measures the Mean Square Errors (Accuracy) associated with a probabilistic forecast

n

kkk EF

NBS

1

21

60th IHC 22 March 2006 - Mobile, AL 8

Statistical Methods: Brier Skill Score

• A scalar skill score comparing a given Brier Score with the Brier Score of a reference forecasts (OFCL, CLIPER etc.).

Assess relative accuracy (skill) of a probabilistic forecast with respect to a reference forecast.

refBSBSBSS 1

60th IHC 22 March 2006 - Mobile, AL 9

Statistical methods: Discrimination Distance

• Distance between the mean forecast probability (F) for all event (E) and all non-events (E’).

Measures the ability of a forecast scheme to discriminate events.

'EFEFd

60th IHC 22 March 2006 - Mobile, AL 10

Statistical Methods: Conditional Distributions

Obs

erve

d fr

eque

ncy

of e

vent

s

Bins of Forecast Probabilities

60th IHC 22 March 2006 - Mobile, AL 11

Statistical Methods: Relative Operating Characteristics

Forecasts (contingent on the forecast probability)Observation Warning (W) No Warning (W’) TotalEvent (E) h m E

Nonevent (E’) f c E’

Total w w’ N

A series of 2x2 contingency tables, which are conditional on a range of forecast probabilities are constructed.

For instance a warning would be issued if the probability exceeded 1,2,3,4,…100 %

The results of the contingency tables can be quantified in terms of hit rate (hr) = h/(h+m) and false-alarm rate (far)=f/(f+c)

A plot of far vs. hr can be created and a skill score created from the area under the curve.

60th IHC 22 March 2006 - Mobile, AL 12

Statistical Methods: ROC diagram & Skill Score

12 ASSROC,where A is the area under the curve

Mason and Graham (1999)

skill

No skill

ROC Skill Score

60th IHC 22 March 2006 - Mobile, AL 13

Verification Procedure: Test Dataset

• Dataset– 5-day cumulative 64-kt

wind probabilities were generated for 342 coastal break points (195 official breakpoints + 147 additional points)

– These were analyzed when warnings were issued for the 14 storms to the right

– N=128250 points

Storm Name Year

Alex 2004

Charley 2004

Frances 2004

Gaston 2004

Ivan 2004

Jeanne 2004

Arlene 2005

Cindy 2005

Dennis 2005

Emily 2005

Katrina 2005

Ophelia 2005

Rita 2005

Wilma 2005

60th IHC 22 March 2006 - Mobile, AL 14

60th IHC 22 March 2006 - Mobile, AL 15

Verification Procedure: Summary Skill Measures

Bias = 0.893 (under forecasts)BS = 0.0248 BSOFCL= 0.0346BSzero = 0.0392

ROC

Relative Operating Characteristics5-d Cummulative Probabilities for Landfalling Atlantic TC 2004-2005

BSSOFCL = 28.30%

BSSzero = 36.75%

60th IHC 22 March 2006 - Mobile, AL 16

Verification Procedure: Conditional Distribution of Break Point Probabilities

Slight under forecast of probabilities for this dataset – due to Wilma

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

0 to

55

to 1

010

t o 1

515

t o 2

020

t o 2

525

to 3

030

t o 3

535

t o 4

040

t o 4

545

to 5

050

to 5

555

to 6

060

t o 6

565

to 7

070

t o 7

575

t o 8

080

to 8

585

to 9

090

to 9

595

t o 1

00Forecast Probability Bins

Obs

erve

d Fr

eque

ncy

of E

vent

s

Forecast Probabilities

Observed Frequencies

Linear (ObservedFrequencies)

60th IHC 22 March 2006 - Mobile, AL 17

Verification Procedure: Summary

• The probabilities are skillful– Brier Skill Score 28% more accurate than the

OFCL deterministic forecast• Note 50% of the OFCL forecasts verified

– ROC Skill Score 88%– The discrimination distance d=26% is large– Probabilities slightly under forecast and are

well calibrated for this limited dataset

60th IHC 22 March 2006 - Mobile, AL 18

Can the wind speed probabilities be used do decrease the area warned

or increase lead time?

60th IHC 22 March 2006 - Mobile, AL 19

Probability Model for NHC Hurricane Warnings

0

200

400

600

800

1000

1200

1400

1600

1/1/

1963

1/1/

1967

1/1/

1971

1/1/

1975

1/1/

1979

1/1/

1983

1/1/

1987

1/1/

1991

1/1/

1995

1/1/

1999

1/1/

2003

Date

War

ning

Len

gth

(nm

i)

0

12

24

36

48

60

72

84

1/1/

1963

1/1/

1966

1/1/

1969

1/1/

1972

1/1/

1975

1/1/

1978

1/1/

1981

1/1/

1984

1/1/

1987

1/1/

1990

1/1/

1993

1/1/

1996

1/1/

1999

1/1/

2002

1/1/

2005

Date

War

ning

Lea

d Ti

me

(hr)

NHC storm total hurricanewarning lengths 1963-2005

NHC storm average hurricanewarning lead times 1963-2005

Since 2000 warning areas have decreased and lead times increased.

60th IHC 22 March 2006 - Mobile, AL 20

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

0

0 to

5

5 to

10

10 to

15

15 to

20

20 to

25

25 to

30

30 to

35

35 to

40

40 to

45

45 to

50

50 to

55

55 to

60

60 to

65

65 to

70

70 to

75

75 to

80

80 to

85

85 to

90

90 to

95

95 to

100

Bins (%)

Perc

ent O

ccur

ence

Avg. Probabilities at Warned Break Points (Warnings in Place)

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

0

0 to

5

5 to

10

10 to

15

15 to

20

20 to

25

25 to

30

30 to

35

35 to

40

40 to

45

45 to

50

50 to

55

55 to

60

60 to

65

65 to

70

70 to

75

75 to

80

80 to

85

85 to

90

90 to

95

95 to

100

Bins (%)

Perc

ent O

ccur

ence

Avg. Probabilities at Unwarned Break Points (Warnings in Place)

Warned Break Points(Warnings in Place)

Unwarned Break Points(Warnings in Place)

%2.29%,8.27 EFEF %4.3%,4.1 '' EFEF

%4.26d

Discrimination Distance

N=5025 N=123225

60th IHC 22 March 2006 - Mobile, AL 21

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

0

0 to

5

5 to

10

10 to

15

15 to

20

20 to

25

25 to

30

30 to

35

35 to

40

40 to

45

45 to

50

50 to

55

55 to

60

60 to

65

65 to

70

70 to

75

75 to

80

80 to

85

85 to

90

90 to

95

95 to

100

Bins (%)

Perc

ent O

ccur

ence

Avg. Probabilities at Warned Break Points (Warnings in Place)

%2.29%,8.27 EFEF

?

Why are there so many low probabilities at the break points?

60th IHC 22 March 2006 - Mobile, AL 22

Distribution of Probabilities at the Ending Break Points

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

0

0 to

5

5 to

10

10 to

15

15 to

20

20 to

25

25 to

30

30 to

35

35 to

40

40 to

45

45 to

50

50 to

55

55 to

60

60 to

65

65 to

70

70 to

75

75 to

80

80 to

85

85 to

90

90 to

95

95 to

100

Bins (%)

Perc

ent O

ccur

ence

Probabilities at the ending break points

9.14

8.8

EF

EF

end

end

?

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

MX01

TX/MEX B

ORDER

BROWNSVILL

E

PORT ISABEL

STATE PARK P

ADRE ISLA

NDTX01

PORT MANSFIE

LDTX02

TX03TX04

BAFFIN B

AYTX05

TX06

CORPUS CHRIS

TI

PORT ARANSAS

TX07

ROCKPORTTX08

TX09

PORT OCONNOR

TX10

PALACIO

S

MATAGORDATX11

SARGENTTX12

FREEPORT

SAN LUIS

PASS

TX13

GALVESTON

BOLIVAR P

ENINSULA

TX14

HIGH IS

LANDTX15

PORT ARTHUR

SABINE P

ASS

SABINE R

IVERLA

01

CAMERON

LAKE C

HARLESLA

02LA

03LA

04LA

05LA

06

INTRACOASTAL C

ITY

VERMILION B

AYLA

07LA

08

MORGAN CITY

LA19LA

09LA

10LA

11LA

12

GRAND ISLELA

20LA

13

MISSIS

SIPPI R

IVERLA

21LA

14LA

15LA

16LA

17

NEW O

RLEANS

LA22LA

18

PEARL RIV

ER

BAY SAIN

T LOUIS

GULFPORT

Break Points

Prob

abili

ty/W

arni

ngs

(1 o

r 0)

-43.5 hr Prob-43.5 hr Warn-31.5 hr Prob-31.5 hr Warn-19.5 hr Prob- 19.5 hr Warn-7.5 hr Prob -7.5 hr WarnVerification

Example: Hurricane Rita

Warnings are brought down too slowly in this case.

00UTC 24 Sep.

12 UTC 23 Sep. 00 UTC

23 Sep.

12 UTC 22 Sep

t=0 at landfall

60th IHC 22 March 2006 - Mobile, AL 24

Summary of Warning Break Points

• Probabilities are useful in the watch/warning process.– Objectively assign of the warnings at fixed

lead time?– Average at warnings = 28%– Average at end points of the Warnings = 9%

• It appears that warnings can be dropped sooner, thus decreasing the area warned area.

60th IHC 22 March 2006 - Mobile, AL 25

Future Plans

• Seasonal Verification Code• See what we can learn from the

verification.• Report to this audience

Questions?