1WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

Operational QPF Verificationof End Products and NWP

Pertti NurmiFinnish Meteorological Institute

WMO / WWRP QPF Verification Workshop, Prague, 14-16 May 2001

2WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

“State-of-the-Art” Operational Verification

Provide instant feedback to forecasters

Enable comparison of individual forecast(er)s vs. observations

Produce comprehensive statistics of the quality of End Products

Produce comprehensive statistics of guidance forecasts (e.g.

NWP)

Produce statistics of individual forecasters - personal biases etc.

Utilize a comprehensive set of tailored verification measures

Provide simplified verification statistics for the general public,

the customers, the decision makers

3WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

Operational “online” Verification, Features

• Web user-interface for forecast input + visualization of results

• Running on a UNIX platform, interaction via Perl scripts

• Operational in real-time in the CFO and the 3 RO’s

• Covers all predicted End Product weather parameters:

temperature, precipitation, cloudiness, wind

• Covers historical data of CFO End Products since 1979

• NWP results (HIRLAM, ECMWF) included for comparison

• Input consistent with operational public / customer forecasts

• Used for official monitoring of the forecast quality of FMI

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 4

Presentverificationstations

5WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

Upcomingverificationstations

6WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

NWP guidance

End Product

QPF production

Verification

7WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

The integrated verification system

"B lack -lis ting "o f m iss ing fo recas ts

T im eseries g raphso f fo recas ts v s . obs

Ins tan t feedbacko f fo recas ts v s . obs

R e liab ility d iag ram s

... e tc .

M E , M A ES k ill S co res

E rro r d is tribu tions

T im e-av e raged s ta tis tics

W W Wapplication

A nnua l, qua rte rly ,m on th ly s ta tis tics

o f se lec ted sco res

"A ll-k now n"v e rifica tion m easures

R O fo recas tss ince 1997

20 yea rso f C F O fo recas ts

ReportingTool

Somewhatoutdated !

Under re-construction !

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 8

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 9

PoP inputShort-range

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 10

QPF inputShort-medium-range

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 11

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 12

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 13

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 14

Note:Too little datato be applicable !

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 15

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 16

Official PoPverification:ROC score

“hit calendar”

time-series

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 17

Official PoPverification:

“daily hitcalendar”

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 18

Official PoPverification:

ROC time-series

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 19

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 20

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 21

22WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

The integrated verification system

"B lack -lis ting "o f m iss ing fo recas ts

T im eseries g raphso f fo recas ts v s . obs

Ins tan t feedbacko f fo recas ts v s . obs

R e liab ility d iag ram s

... e tc .

M E , M A ES k ill S co res

E rro r d is tribu tions

T im e-av e raged s ta tis tics

W W Wapplication

A nnua l, qua rte rly ,m on th ly s ta tis tics

o f se lec ted sco res

"A ll-k now n"v e rifica tion m easures

R O fo recas tss ince 1997

20 yea rso f C F O fo recas ts

ReportingTool

Under somemodification !

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 23

Application Development

Pertti Ñurmi, Mika Heiskanen, Mikael Frisk

Verification

reports are

produced:

– Annually

– Quarterly

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 24

Utilizing

a comprehensive

set of

verification

measures

and statistics

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 25

Annual Report (a)Page 1 of handout

Detailed examplesof page information

will follow, first- P o P forecasts

PS:… there are

~ 100 pages inthe annual reports

26WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

Annual P o P ; Ex. 1

or, rather,attributesdiagram

27WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

Annual P o P ; Ex. 2

28WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

Annual P o P ; Ex. 3

29WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

Annual P o P ; Ex. 4

2 * 2

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 30

Annual Report (b)Page 2 of handout

Detailed examplesof page information

will follow, ...- Q P F

(daily rainfall)

31WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

Annual Q P F ; Ex. 1 (Error distribution, various lead times)

32WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

Annual Q P F ; Ex. 2 (Contingency table, 3 *3)

33WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

Annual Q P F ; Ex. 3 (Stratification vs. frequency)

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 34

Annual Report (c)Page 3 of handout

Detailed examplesof page information

will follow, ...- Q P F (cont’d)

(5-day accumulations)

35WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

Annual Q P F ; Ex. 4 (Stratification vs. temperature zone)

36WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

Annual Q P F ; Ex. 5 (Obs vs. fc time series)

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 37

Summer Report (a)Page 4 of handout

Detailed examplesof page information

will follow, first- P o P

PS:… there are

~ 50 pages inthe quarterly reports

38WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

Summer P o P ; Ex. 1

39WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

Summer P o P ; Ex. 2

P. Ñurmi / 15-05-2001 WWRP QPF Verification - Prague 40

Summer Report (b)Page 5 of handout

Detailed examplesof page information

will follow, ...- Q P F

41WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

Summer Q P F ; Ex.

42WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

Operational QPF Verificationof End Products and NWP

The end of the story by Pertti Nurmi

43WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

Hypothesis: The quality of End Products is expected to increase gradually after implementation of an

Operational “online” Verification System… unless you’re

re-organizingthe Weather Service

- like FMI, 1996

“Percent correct” forecasts

44WWRP QPF Verification - PragueP. Ñurmi / 15-05-2001

Hypothesis: The quality of End Products is expected to increase gradually after implementation of an

Operational “online” Verification System… returning back

to normal life withthe best score ever

“Percent correct” forecasts