1 what’s new in verification? a verification briefing for the saws iv workshop october 26, 2011...

1

What’s New in Verification?A Verification Briefing

for the SAWS IV Workshop

October 26, 2011Chuck Kluepfel

National Weather Service HeadquartersSilver Spring, Maryland

301-713-0090 x132

[email protected]

http://www.metacafe.com/watch/1686664/fedex_planes_dodging_storm/

Starter Videos

http://www.youtube.com/watch?v=6E_Z_Ve-ayA

2

Part 1

Lead Time

Measurements

3

History

• NOAA (GSD) was contracted to measure the lead time with which NWS TAFs predict the onset and cessation of IFR conditions (2-category, IFR vs. VFR).

• This software package has been available thru the NWS Performance Management Web site for 2 years.

• Next 6 months: Joint NWS/FAA TRWG (Traffic Flow Management Weather Requirements Working Group) is directing the NWS Performance Branch to measure TAF performance, using more specific Decision Support requirements.

4

GSD Software

• Allows for tight and loose accuracy standards.

• Verifies IFR onsets similar to the way NWS verifies warnings.

• LT = IFR onset time - TAF issuance time.

• Uses the TAF with the longest LT.

5

Example of an IFR Onset at 1800

9

TRWG Decision Support Software

• Requirements have just been defined.

• Code writing should begin soon.

• Very specific requirements set by TRWG.

• Measure NWS skill in forecasting the Onset and Cessation of events critical to operations.

10


• Measure the Onset and Cessation of

• IFR (Phase 1)• VFR (Phase 1)• MVFR (Phase 1)

11



• IFR (Phase 1)• VFR (Phase 1)• MVFR (Phase 1)• Wind shifts (Phase 2)• Wind speed changes (Phase 2)

12



• IFR (Phase 1)• VFR (Phase 1)• MVFR (Phase 1)• Wind shifts (Phase 2)• Wind speed changes (Phase 2)• Thunderstorms (Phase 3)• Freezing precipitation (Phase 3)

13


The Onset of these conditions:

• IFR (Onset - always deterioration)• VFR (Onset - always improvement or clearing)

14


• Measure the Onset and Cessation of the Following Events

• IFR • VFR• MVFR (from IFR – Onset represents clearing)• MVFR (from VFR – Onset represents deterioration)

15


16


17


18


19


20

TRWG Decision Support SoftwareMethodology

• For each hit, determine whether the forecast timing error met the TRWG requirements for scoring a POD hit:

• For 1-hour LT: ± 15 minutes• For 2-hour LT: ± 15 minutes• For 4-hour LT: ± 30 minutes • For 6-hour LT: ± 45 minutes • For 8-hour LT: ± 60 minutes

21


22


23


24

TRWG Decision Support

Case

Study

7 Days (5.5 active) at one terminal

25


26


27

Part 2

CWSU

Verification

28

CWSU Verification

29

Part 3

Traditional

Stats on Demand Results

30

Traditional Stats forModified Southwest United States:

ColoradoNew Mexico

UtahArizonaNevada

California

Minus these WFOs: San Diego, Los Angeles, San Francisco

32

Modified SW US – Fiscal Year 2011Scheduled 3-6 hr IFR and Below

GFS LAMP

(569K - 264K) K ÷ (511K – 482K) ~ 10.5The 3-6 hr GFS LAMP false alarmed over 10 times for every additional hit it got over the forecasters!

POD 0.56

FAR 0.53

CSI 0.35

ForecastYes No

O

b

s

Yes 511 K 402 K

No 569 K 13.1 Million

Prevailing

POD 0.53

FAR 0.35

CSI 0.41

ForecastYes No

O

b

s

Yes 482 K 430 K


33

WFOs MTR, LOX, SGX FY 2011Scheduled 3-6 hr IFR and Below

GFS LAMP

POD 0.54

FAR 0.43

CSI 0.39

ForecastYes No

O

b

s

Yes 213 K 182 K


Prevailing

POD 0.64

FAR 0.31

CSI 0.50

ForecastYes No

O

b

s

Yes 251 K 144 K


34

WFOs PSR, TWC, EPZ FY 2011Scheduled 3-6 hr IFR and Below

GFS LAMP

POD 0.18

FAR 0.73

CSI 0.12

ForecastYes No

O

b

s

Yes 1068 4907

No 2871 1.2 Million

Prevailing

POD 0.18

FAR 0.48

CSI 0.15

ForecastYes No

O

b

s

Yes 1046 4929

No 951 1.2 Million

1 what’s new in verification? a verification briefing for the saws iv workshop october 26, 2011...

Documents