weather type dependant fuzzy verification of precipitation

Eidgenössisches Departement des Innern EDIBundesamt für Meteorologie und Klimatologie MeteoSchweiz

Weather type dependant fuzzy verification of precipitation

COSMO General Meeting, Offenbach, 07.-11.09.2009

Tanja Weusthoff

2 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff

Fuzzy Verification

• „multi-scale, multi-intensity approach“

• „Fuzzy verification toolbox“ of B. Ebert

• Two methods• Upscaling (UP)• Fraction Skill Score (FSS)

• present output scale dependent

• standard setting: 3h accumulations, Jun-Nov 2007, vs Swiss radar dataCOSMO-2 (2.2km): leadtimes 03-06

COSMO-7 (6.6km): leadtimes 03-06,06-09,09-12,12-15

fcst obs

incr

easi

ng

bo

x s

ize

increasing threshold


… Upscaling (UP)

random

random

hitsalarmsfalsemisseshits

hitshitsETS

1. Principle:

Define box around region of interest and calculate the average of observation and forecast data within this box.

3. Equitable Threat Score (ETS)

Rave

total

alarmsfalsehitsmisseshitshits random

) )((

Event if Rave ≥ thresholdNo-Event if Rave < threshold

yes no

yes HitFalse Alarm

no MissCorrect negative

observation

fore

cast

2. Contingency Table

Q: Which fraction of observed yes - events was correctly forecast?

(Atger, 2001)


… Fraction Skill Score (FSS) (Roberts and Lean, 2005)

X XX X

X Xx XXX

x

1. Principle:

Define box around region of interest and determine the fraction pj and oj of grid points with rain rates above a given threshold.

3. Skill Score for Probabilities

2. Probabilities

Q: On which spatial scales does the forecast resemble the observation?

N

j

N

jjj op

FBS

1 1

22

N1

1FSS

N

jjj op

1

2)(N

1 FBS

FBS worst

no colocation of non-zero fractions

0 < pj < 1 = fraction of fcst grid points > threshold0 < oj < 1 = fraction of obs grid points > threshold


… Fraction Skill Score (FSS) (Roberts and Lean, 2005)

4. Useful Scales

useful scales are marked in bold in the graphics


goodbadCOSMO-7 better COSMO-2 better

COSMO-2 COSMO-7 Difference

- =

- =

Upscaling and Fractions Skill ScoreF

ract

ion

s sk

ill s

core

Up

scal

ing


COSMO-2 vs. COSMO-7, bootstrapping

DIFFERENCES

COSMO-7 better

COSMO-2 better

abs(median) / 0.5(q95-q05)

[ 10 , Inf ]

[ 2 , 5 [

[ 5 , 10 [

[ 1 , 2 [

[ 0 , 1 [

Values = Score of COSMO-2

Size of numbers = abs(Median) / 0.5(q95-q05)

measure for significance of differences

¦ COSMO-2 – COSMO-7 ¦

q(50%)

q(95%)

q(5%)

5% 95%


SensitivitiesOnly 00 and 12 UTC model runsCOSMO-2 & COSMO-7: leadtimes 3-6,6-9,9-12,12-15

absolute values of COSMO-2 slightly lower, but still the same pattern of differences

DIFFERENCES

COSMO-7 better

COSMO-2 better


Weather type verification: COSMO-2

4 7 5 5 7 45 31 23 15 34 7# cases

Threshold [mm/3h]

Upscaling

Fractions Skill Score

Window size: 3gp, 6.6 km

Window size: 27gp, 60 km

Window size: 3gp, 6.6 km

Window size: 27gp, 60 km

dx 0.1 0.2 0.5 1.0 2.0 5.0 10 20

NE 45 45 45 / / / / /

N 45 45 / / / / / /

NW 3 3 9 9 27 27 / /

SE 1 1 1 1 9 / / /

S 1 1 1 1 1 9 / /

SW 1 1 3 9 9 45 / /

E 9 15 27 / / / / 45

W 1 1 3 9 15 / / /

F 15 15 27 27 45 / / /

H 27 27 27 45 / / / /

L 3 9 15 27 45 / / /

dx 0.1 0.2 0.5 1.0 2.0 5.0 10 20

NE 15 / / / / / / /

N / / / / / / / /

NW 3 3 5 5 9 15 / /

SE 1 1 1 3 9 / / /

S 1 1 1 3 3 9 / /

SW 3 3 5 9 9 / / /

E 15 / / / / / / /

W 3 3 3 5 9 / / /

F 9 9 15 15 / / / /

H / 15 / / / / / /

L 3 5 9 9 15 / / /

COSMO-2, gridpoints* 2.2 km COSMO-7, gridpoints* 6.6 km

Smallest spatial scale [gridpoints] where the forecast has been useful regarding to FSS „useful scales“ definition.

+

-+

-

7

4

23

5

7

45

5

31

15

34

7

# cases

% obs gridpts >= thresh (whole period)

16 14 10 7 5 2 0.5 <0.1 16 14 10 7 5 2 0.4 <0.1


Frequency of Weather Classes, June – November 2007

47

5 57 7

45

31

23

15

34


Northerly Winds (NE,N) 11 days

COSMO-2 vs.COSMO-7COSMO-2 (wc) vs.COSMO-2 (all)

COSMO-2 in northerly wind situations clearly worse than over whole period.

DIFFERENCES

COSMO-7 better

COSMO-2 better

DIFFERENCES

COSMO-2 (all) better

COSMO-2 (wc) better


Southerly Winds (SE,S) 12 days


COSMO-2 in southerly wind situations clearly better than over whole period.

DIFFERENCES

COSMO-7 better

COSMO-2 better

DIFFERENCES


COSMO-2 (wc) better


Northwesterly winds (NW) 23 days


COSMO-2 in nothwesterly wind situations at large thresholds clearly better than over whole period.

DIFFERENCES

COSMO-7 better

COSMO-2 betterD

IFFERENCES


COSMO-2 (wc) better


Flat (F) 15 days


DIFFERENCES

COSMO-7 better

COSMO-2 betterD

IFFERENCES


COSMO-2 (wc) better

COSMO-2 in flat pressure situations clearly worse than over whole period.


Summary & Outlook

• Fuzzy verification of the 6 months period in 2007 has shown that COSMO-2 generally performed better than COSMO-7 on nearly all scales

• part of this superiority is caused by the higher update frequency, but using same model runs still shows the same pattern of differences

• the results for different weather types show large variations • best results were found for southerly winds and winds from Northwest

and West, • Northeasterly winds as well as Flat pressure situations lead to worse

perfomance of both models

• operational Fuzzy verification is about to start, including Upscaling and Fraction Skill Score

weather type dependant fuzzy verification of precipitation

Documents

fss useful scales definition

obs gridpts