weather type dependant fuzzy verification of precipitation
Post on 12-Jan-2016
41 Views
Preview:
DESCRIPTION
TRANSCRIPT
Eidgenössisches Departement des Innern EDIBundesamt für Meteorologie und Klimatologie MeteoSchweiz
Weather type dependant fuzzy verification of precipitation
COSMO General Meeting, Offenbach, 07.-11.09.2009
Tanja Weusthoff
2 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
Fuzzy Verification
• „multi-scale, multi-intensity approach“
• „Fuzzy verification toolbox“ of B. Ebert
• Two methods• Upscaling (UP)• Fraction Skill Score (FSS)
• present output scale dependent
• standard setting: 3h accumulationsCOSMO-2 (2.2km): leadtimes 03-06
COSMO-7 (6.6km): leadtimes 03-06,06-09,09-12,12-15
fcst obs
incr
easi
ng
bo
x s
ize
increasing threshold
3 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
… Upscaling (UP)
random
random
hitsalarmsfalsemisseshits
hitshitsETS
1. Principle:
Define box around region of interest and calculate the average of observation and forecast data within this box.
3. Equitable Threat Score (ETS)
Rave
total
alarmsfalsehitsmisseshitshits random
) )((
Event if Rave ≥ thresholdNo-Event if Rave < threshold
yes no
yes HitFalse Alarm
no MissCorrect negative
observation
fore
cast
2. Contingency Table
Q: Which fraction of observed yes - events was correctly forecast?
(Atger, 2001)
4 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
… Fraction Skill Score (FSS) (Roberts and Lean, 2005)
X XX X
X Xx XXX
x
1. Principle:
Define box around region of interest and determine the fraction pj and oj of grid points with rain rates above a given threshold.
3. Skill Score for Probabilities
2. Probabilities
Q: On which spatial scales does the forecast resemble the observation?
N
j
N
jjj op
FBS
1 1
22
N1
1FSS
N
jjj op
1
2)(N
1 FBS
FBS worst
no colocation of non-zero fractions
0 < pj < 1 = fraction of fcst grid points > threshold0 < oj < 1 = fraction of obs grid points > threshold
5 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
… Fraction Skill Score (FSS) (Roberts and Lean, 2005)
4. Useful Scales
useful scales are marked in bold in the graphics
7 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
goodbadCOSMO-7 better COSMO-2 better
COSMO-2 COSMO-7 Difference
- =
- =
Upscaling and Fractions Skill ScoreJun – Nov 2007
Fra
ctio
ns
skill
sco
reU
psc
alin
g
8 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
COSMO-2 vs. COSMO-7
DIFFERENCES
COSMO-7 better
COSMO-2 better
abs(median) / 0.5(q95-q05)[ 10 , Inf ]
[ 2 , 5 [
[ 5 , 10 [
[ 1 , 2 [
[ 0 , 1 [
Values = Score of COSMO-2
Size of numbers = abs(Median) / 0.5(q95-q05)
measure for significance of differences
¦ COSMO-2 – COSMO-7 ¦
q(50%)
q(95%)
q(5%)
5% 95%
10 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
SensitivitiesOnly 00 and 12 UTC model runsCOSMO-2 & COSMO-7: leadtimes 3-6,6-9,9-12,12-15
absolute values of COSMO-2 slightly lower, but still the same pattern of differences
DIFFERENCES
COSMO-7 better
COSMO-2 better
11 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
Weather type verification: COSMO-2
4 7 5 5 7 45 31 23 15 34 7# cases
Threshold [mm/3h]
Upscaling
Fractions Skill Score
Window size: 3gp, 6.6 km
Window size: 27gp, 60 km
Window size: 3gp, 6.6 km
Window size: 27gp, 60 km
dx 0.1 0.2 0.5 1.0 2.0 5.0 10 20
NE 45 45 45 / / / / /
N 45 45 / / / / / /
NW 3 3 9 9 27 27 / /
SE 1 1 1 1 9 / / /
S 1 1 1 1 1 9 / /
SW 1 1 3 9 9 45 / /
E 9 15 27 / / / / 45
W 1 1 3 9 15 / / /
F 15 15 27 27 45 / / /
H 27 27 27 45 / / / /
L 3 9 15 27 45 / / /
dx 0.1 0.2 0.5 1.0 2.0 5.0 10 20
NE 15 / / / / / / /
N / / / / / / / /
NW 3 3 5 5 9 15 / /
SE 1 1 1 3 9 / / /
S 1 1 1 3 3 9 / /
SW 3 3 5 9 9 / / /
E 15 / / / / / / /
W 3 3 3 5 9 / / /
F 9 9 15 15 / / / /
H / 15 / / / / / /
L 3 5 9 9 15 / / /
COSMO-2, gridpoints* 2.2 km COSMO-7, gridpoints* 6.6 km
Smallest spatial scale [gridpoints] where the forecast has been useful regarding to FSS „useful scales“ definition.
+
-+
-
7
4
23
5
7
45
5
31
15
34
7
# cases
% obs gridpts >= thresh (whole period)
16 14 10 7 5 2 0.5 <0.1 16 14 10 7 5 2 0.4 <0.1
13 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
% 0.1 0.2 0.5 1.0 2.0 5.0 10 20
NE 7 6 4 2 1 0.3 <0.1 <0.1
N 2 1 0.8 0.4 0.1 <0.1 <0.1 0
NW 13 11 8 5 2 0.7 0.1 <0.1
SE 17 15 12 9 6 2 0.4 <0.1
S 21 19 15 11 8 3 0.7 0.1
SW 25 22 18 13 9 3 1 0.2
E 3 2 1 0.7 0.2 <0.1 <0.1 <0.1
W 18 15 11 8 4 1 0.2 <0.1
F 17 15 11 8 5 2 0.8 0.2
H 2 2 0.8 0.5 0.2 0.1 <0.1 <0.1
L 42 36 27 19 11 3 0.8 0.1
ALL 16 14 10 7 5 2 0.5 <0.1
Fraction of observation gridpoints >= threshold climatology
14 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
Frequency of Weather Classes, June – November 2007
47
5 57 7
45
31
23
15
34
15 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
Northerly Winds (NE,N) 11 days
COSMO-2 vs.COSMO-7COSMO-2 (wc) vs.COSMO-2 (all)
COSMO-2 in northerly wind situations clearly worse than over whole period.
DIFFERENCES
COSMO-7 better
COSMO-2 better
DIFFERENCES
COSMO-2 (all) better
COSMO-2 (wc) better
16 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
Southerly Winds (SE,S) 12 days
COSMO-2 vs.COSMO-7COSMO-2 (wc) vs.COSMO-2 (all)
COSMO-2 in southerly wind situations clearly better than over whole period.
DIFFERENCES
COSMO-7 better
COSMO-2 better
DIFFERENCES
COSMO-2 (all) better
COSMO-2 (wc) better
17 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
Northwesterly winds (NW) 23 days
COSMO-2 vs.COSMO-7COSMO-2 (wc) vs.COSMO-2 (all)
COSMO-2 in nothwesterly wind situations at large thresholds clearly better than over whole period.
DIFFERENCES
COSMO-7 better
COSMO-2 betterD
IFFERENCES
COSMO-2 (all) better
COSMO-2 (wc) better
18 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
Flat (F) 15 days
COSMO-2 vs.COSMO-7COSMO-2 (wc) vs.COSMO-2 (all)
DIFFERENCES
COSMO-7 better
COSMO-2 betterD
IFFERENCES
COSMO-2 (all) better
COSMO-2 (wc) better
COSMO-2 in flat pressure situations clearly worse than over whole period.
19 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
Summary / Conlusions
• Fuzzy verification of the D-PHASE operations period in 2007 has shown that COSMO-2 generally performed better than COSMO-7 on nearly all scales
• part of this superiority is caused by the higher update frequency, but using same model runs still shows the same pattern of differences
• the results for different weather types show large variations • best results were found for southerly winds and winds
from Northwest and West, • Northeasterly winds as well as Flat pressure situations
lead to worse perfomance of both models
20 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
UP with other scoresPOFD = FA/(CN+FA)Probability of false detection (Perfect score: 0)
FAR = FA/(H+FA)False Alarm Ratio (Perfect score: 0)
POD = H/(H+M)Probability of Detection (Perfect score: 1)
BIAS = (H+FA) / (H+M)Frequency Bias (perfect score: 1)
HK =POD – FARTrue Skill Statistik (perfect score: 1)
OR = (H*CN) / (M*FA)Odds Ratio (perfect score: infinity)
21 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
dx 0.1 0.2 0.5 1.0 2.0 5.0 10 20
NE
N
NW
SE
S
SW
E
W
F
H
L
dx 0.1 0.2 0.5 1.0 2.0 5.0 10 20
NE
N
NW
SE
S
SW
E
W
F
H
L
COSMO-2, gridpoints* 2.2 km COSMO-7, gridpoints* 6.6 km
UP „useful scales“?
How to define usefuls scales for ETS?
22 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
Fuzzy Verifikation Intensity Scale (IS)
(Casati et al. 2004)• Transformation of Fcst and Obs into binary images on a rain/no-rain basis for the rainfall
rate thresholds. • Difference between forecast and observation = binary error image. • Decomposition into the sum of components at different spatial scales by performing a
two dimensional discrete wavelet decomposition.
Binary forecast Binary observation
Score: Mean squared error (MSE) and MSE skill score (SS) for each spatial scale component of the binary error image.
What is the relative improvement of the forecast over some reference forecast?
23 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
Intensity Scale
COSMO-2 COSMO-7
24 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
Intensity Scale
COSMO-2
25 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
Outlook
• operational Fuzzy verification is about to start, including Upscaling and Fraction Skill Score• season, year• weather-type dependant
• Intensity scale results will further be investigated (new developments of B. Casati)
26 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
All Weatherclasses
• Overview over all weatherclasses – differences COSMO-2 minus COSMO-7 and absolute values of COSMO-2, here without bootstrapping
27 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
Northerly Winds (NE,N,NW)N NWNE
23 days4 days7 days
DIFFERENCES
COSMO-7 better
COSMO-2 better
28 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
Southerly Winds (SE,S,SW)SSE SW
7 days 45 days5 days
DIFFERENCES
COSMO-7 better
COSMO-2 better
29 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
East and West (E,W)E W
31 days5 days
DIFFERENCES
COSMO-7 better
COSMO-2 better
30 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
Flat, High, Low (F,H,L)
F H L
34 days 7 days15 days
DIFFERENCES
COSMO-7 better
COSMO-2 better
31 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
Summary
JJA SON DOP
Useful scales of COSMO-2 are the same for all three time periods considered:
<= 0.2 mm/3h scales larger or equal 2.2 km (= 1* gridscale )
0.5 mm/3h scales larger or equal 6.6 km (= 3* gridscale )
1.0 mm/3h scales larger or equal 19.8 km (= 9* gridscale )
2.0 mm/3h scales larger or equal 33.0 km (= 15* gridscale )
5.0 mm/3h scales larger or equal 99.0 km (= 45* gridscale )
Most prominent advantage for COSMO-2! Differences are significant.
COSMO-2 better but on many scales only slightly. Advantages on small scales/all thresholds and large thresholds/large scales. Differences are significant.
COSMO-2 better than COSMO-7 on all scales. Differences are significant.
Most frequent weatherclasses: SW (32 days) and W (21 days).
Most frequent weatherclasses: H (24 days) and NW (17 days).
Most frequent weatherclasses: SW (45 days) and H (34 days).
32 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
JJA SON DOP
NE -Relatively low values for UP-FSS: COSMO-2 better for thresholds <= 2 mm/3h, useful scales only for largest window size (99 km)
- COSMO-2 better on nearly all scales, especially for thresholds of 2-5 mm/3h. Large threshold gave no result in UP only small - scale events (averaging damps values out). Apparently, COSMO-2 could simulate those events quite well.
-Low absolute values (FSS < 0.6, UP < 0.3), no useful scales-COSMO-2 clearly better for thresholds <= 2 mm/3h (FSS)- UP: the averaged precipitation was equally well represented in both models. Only 2 mm/3h threshold better captured in COSMO-2.
N Only few days – no clear superiority of one model. Absolute values relatively low. No useful scales.
NW - COSMO-2 performed better than COSMO-7, especiall for large thresholds (> 1 mm/3h, FSS and UP).
-Many days with that weather type-Over most scales rarely different behaviour-Thresholds 5-10 mm/3h better in COSMO-2 as well as spatial scale of 6.6 km
- COSMO-2 performed better than COSMO-7, especiall for large thresholds (>= 5 mm/3h, FSS and UP).
Summary Weatherclasses I
33 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
JJA SON DOP
SE -Only few days; similar behaviour in all time periods: COSMO-2 better, especially for thresholds around 1 mm/3h and small spatial scales (FSS) and large spatial scales (UP)-Large absolute values (e.g. FSS up to 0.9, UP up to 0.66 for DOP)
S - Only few days; COSMO-2 better on most scales, most clearly for low thresholds concerning averaging (UP) and large thresholds and small spatial scales regarding probabilities (FSS), in JJA COSMO-2 clearly better also for smallest eindow size (6.6 km) for all thresholds
SW - Clearly better performance of COSMO-2, especially on smaller spatial scales
- Similar performance of both models, but COSMO-2 better for moderate thresholds (1-2 mm/3h)
- Clearly better performance of COSMO-2, especially on smaller spatial scales
Summary Weatherclasses II
34 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009Tanja Weusthoff
JJA SON DOP
E -Only few cases (5 for DOP), COSMO-2 clearly better for low thresholds (<= 1 mm), while for a threshold of 2 mm/3h COSMO-7 better on largest spatial scales-Rather low absolute values
W - COSMO-2 clearly better on all scales, especially for thresholds <= 2 mm and smaller spatial scales
- COSMO-2 slightly better only on small spatial scales
- COSMO-2 better for thresholds <= 2 mm and especially smaller spatial scales
F -Comparable low absolute values, but COSMO-2 clearly better than COSMO-7, especially for medium and large precipitation thresholds.
-Comparable low absolute values, rarely difference in the performance of the two models. -COSMO-2 slightly better for small thresholds
- Comparable low absolute values, but COSMO-2 clearly better than COSMO-7, especially for medium and large precipitation thresholds.
H -very low absolute values (especially in SON), but COSMO-2 clearly better than COSMO-7 for precipitation thresholds <= 2 mm/3h
L - Only few cases (7 for DOP), COSMO-2 better for small spatial scales and thresholds <= 2 mm/3h
Summary Weatherclasses III
top related