Optimizing WFO Aviation Service & Forecast Performance

Dan ShoemakerAviation CurmudgeonWFO FWD

Optimizing WFO Aviation Service & Forecast Performance

• Background—NWS performance.• Improving Service. – My / FWD’s “TAF Philosophy.”– False Alarm Rate.– Individual Feedback.–Aviation Statistics/Climatology.

NWS TAF Performance: Are we any good?

• Last year I put together a “report card” for a visit with Southwest Airlines.– Airlines have little/no access to TAF stats.

• 2000/3 % improvement over LAMP; 0-6 hours; for every SWA/AirTran city.– We have to be better than an automatic product.– Shows which offices are “paying attention”.

TUSSAN

ONTLA

XSFO SM

FRNO

SEABOI

DENM

SPM

DWCM

H CLE BUFBOS

BDLLG

ABW

IORF

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.021085.2 57.161.5 67.2

CSI % Impv over LAMP 0-6hr 2000/3 FY2010

TAF PerformanceWestern Region SWA Cities

Eastern Region

OMA MSP MKEMDW IND CMH SDF CLE PIT BUF MHT BOS PVD BDL ISP LGA PHL BWI IAD ORF0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

CSI % Impv over LAMP 0-6hr 2000/3 FY2010

Southern Region

-20

-15

-10

-5

0

5

10

15

20 21.9 21.0

-0.7

17.2

-0.8

-16.5

-38.5

CSI % Impv over LAMP 0-6hr 2000/3 FY2010

AirTran Cities

-20.0

-15.0

-10.0

-5.0

0.0

5.0

10.0

15.0

20.0

-0.3-1.4

21.4

-26.3

-1.9

15.2

CSI % Impv over LAMP 0-6hr 2000/3 FY2010

FWD 1000/3 Performance • From Stats on Demand:

“… A review of previous years forecast performance showed that the percent frequency of occurrence of IFR conditions is a reasonable first estimate predictor for overall forecast performance for a given time period, or region...”

% Occurrence of IFR & BLO National POD/FAR Results

10 63 / 39 8 58-62/42-45 6 57-60/42-45 4 56-59/44-46

FWD: 5 70 / 32

Optimizing WFO Aviation Service & Forecast Performance

The Problem?• Office culture can be “stuck” in bad habits.• Forecasters don’t see effects of their TAFs.– AFP must become “lobbyist” for Customers.– Long term advocate for improvement…there are

no quick fixes. • Forecasters must take ownership of their TAFs.– Individual feedback!

My Forecast Philosophy• Optimism beats pessimism; or “when in doubt,

leave it out.” (more later)• On “cautious” TAFs:– If TAFs could crash airplanes, the country would be

littered with aluminum (copper theft would decrease).

• On too much detail:– TAFs are not a portrait of the atmosphere —

they’re a stick figure. As long as they’re anatomically correct, they’re doing their job.– (this is PPTAF philosophy as well).

More Philosophy

• 2000/3 is the most important TAF category!!– TAF is mandated by FAA for airlines, not GA. – 1000/3 is a GA category, not an airline category.– Costs the airlines money.– Occurs more often than lower categories.

• This is where AFPs should place greatest emphasis.– 2000/3 focus will not degrade 1000/3 performance.

False Alarm Rate (FAR) (cig/vis)• 4 possible outcomes for a TAF (consider 2000/3):– Hit: fuel added, good forecast.– Correct Negative: no extra fuel, plane lands at

destination.– Missed event: no extra fuel, plane lands at

destination anyway (vast majority of cig/vis cases).– FA: fuel always added, not needed.

• 100% of FA’s penalize airlines – FA’s always add cost – much worse than missed event.

Optimism beats pessimism (FAR)• If you are “on the fence” about an occurrence, and you

bet on the lower category, you’re basically betting 50/50 on a low probability event. By the way, Las Vegas wants you to stop by on your way home.

• DFW/DAL annual % occurrence: (AVNFPS Climo tools) (SOD)

– VFR 82% Above 2000/3 90%– MVFR 12% Below 2000/3 10%– IFR 5%– LIFR & Below <1%

• Caution/pessimism affects FAR much more than POD.

FWD TAF Rules:

• Forecast probable weather, not possible weather (use AFD for possibilities).

• Don’t cross thresholds unless conditions will probably occur.– 2000/3 is most important category, then 600/2

(400/1), 1000/3 then others. • No TEMPOs in the Outlook Period. If you have

confidence, write a new line.

More FWD TAF Rules:

• No ¼ SM in the main body. Period.– Not our job to shut down operations. We forecast

weather, & let airlines decide whether to fly. • No PROB30s. (more later)• Forecast TSRA during probable times. Use VCTS &

CB for when less likely.• Use the AVNFPS cig/vis trend tool.

• The TAF police will always find you!

Individual Feedback

• Individual feedback is key to lowering FAR & improving performance. – Many think the problem lies elsewhere.

• You’re going to hear excuses…facts will trump them.– Shoe’s 4 stages of feedback — whining, crying,

grumbling, & finally acceptance.• Many forms of feedback — choose what

works best for you.

Individual Feedback

• Run individual TAF stats—at least a year’s worth to get a good sample size.– Annotate/circle/comment/point out good & bad.– The biases are a good place to find where individual

improvements can be made (pessimistic forecasters).

• Show collated statistics so each individual can see where they stand in the group.– Note: LAMP performance can vary considerably

person by person.

Individual Feedback

A

B

C

D

E

F

G

H

I

J

K

L

M FWD

-20 -10 0 10 20 30 40 50 60 70 80

4.0

-4.7

-1.3

-0.8

19.9

18.6

9.8

-3.0

4.2

0.8

2.2

7.9

4.0

22.1

-8.8

70.5

20.7

10.1

-5.3

19.0

-3.5

28.3

4.2

0.0

38.0

12.5

1000&32000&3

A

B

C

D

E

F

G

H

I

J

L

M

AVG

0 50 100 150 200 250 300 350 400 450

Hours of TEMPO Forecast

% Improvement vs LAMP 2009 Fcstr “D” had no clue they wrote that many TEMPOs

Individual Feedback—Forecaster “D”Did not beat MOS his/her first year at FWD

Issued a 24 hr TAF with 18 hours of TSRANote how low FAR improves scores & POD actually increased.

Forecaster “D” LAMP

Uh Oh!

TAF KDAL 082333Z 0900/0924 11014G24KT P6SM BKN050 FM090200 11014KT P6SM BKN050 FM090500 07015KT P6SM OVC025 FM090900 05012KT P6SM -RA OVC015 FM091100 36018G28KT 1/2SM SN OVC006 FM091800 34021KT 3SM -SN OVC010 FM092200 35015KT P6SM OVC025

PROB30 (SR)May – Apr

PROB30 (FWD) May – Apr

Warm Season Afternoon ThunderstormsAfternoon precip occurs @ KDFW 4% of time (SOD).

When precip occurs (AVNFPS climo distribution):

17Z 18Z 19Z 20Z 21Z 22Z 23Z 00Z 01Z 02Z 03Z0

5

10

15

20

25

30

35

40

VFRMVFRIFRLIFRVLIFR

Thunderstormstwo TAFs (warm season):

TAF KDFW 061800/072400 18012KT P6SM SCT060CB TEMPO 0620/0623 3SM TSRA BKN015CB FM070100….

TAF KDFW 061800/072400 18012KT P6SM SCT060CB TEMPO 0620/0623 6SM TSRA BKN050CB FM070100….

What’s the difference? #1 takes 3 hour hit on category.

Warm Season Afternoon Thunderstorms

We’re not ignoring the flight category, we’re forecasting the probable category.

Part of MFL problem?

17Z18Z

19Z20Z

21Z22Z

23Z00Z

01Z02Z

03Z05

10152025303540

VFRMVFRIFRLIFRVLIFR

Feedback Works

FY 2011MFL CSI

15.4 % Improvement

over LAMP

FY 2010MFL CSI

18.8% Worse

than LAMP

MFL LAMP

Miscellaneous

D2D Map backgroundAll ASOS/AWOS voice lines. Get current conditions (no

specials from AWOS). SVR WX wind gusts as well.

Summary• Stress optimistic forecasting (improve FAR) &

PPTAF principles. Focus on 2000/3, but don’t forget other important categories. Individual feedback works.

• Dig into aviation climatology to find areas ripe for improvement (AVNFPS climo tools & SOD).

• Every office has bad forecasts…learn from them. • Improvements will take sustained effort,

especially in the beginning. The office culture will change, & service/scores will improve.

Summary• Constant abrasion produces the pearl; it’s a disease of the oyster.

Lenny Bruce

• Questions/Comments?