ECMWF forecast products users meeting – Reading, June 2005 Slide 1

Verification of weather parameters

Anna Ghelli, ECMWF

ECMWF forecast products users meeting – Reading, June 2005 Slide 2

Overview

Deterministic forecast performance for different weather parameters

Precipitation forecast: scores and their confidence

SYNOP on the GTSPrecipitation analysis

Ensemble prediction System: its performance relative to precipitation

ECMWF forecast products users meeting – Reading, June 2005 Slide 3

-0.1

0

0.1

0.2

0.3

0.4

0.5

deg

C-0.1

0

0.1

0.2

0.3

0.4

0.5

J

1998A J O J

1999A J O J

2000A J O J

2001A J O J

2002A J O J

2003A J O J

2004A J O J

2005

fc error of indate findate wp step ntot BIAS STDV RMSE SKILL MAE SKILL CORR M-OB M-FC

skil 48h skil 60h skil 72h skil 84h

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

de

g C

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

J

1998A J O J

1999A J O J

2000A J O J

2001A J O J

2002A J O J

2003A J O J

2004A J O J

2005

fc error of indate findate wp step ntot BIAS STDV RMSE SKILL MAE SKILL CORR M-OB M-FC

skil 48h skil 60h skil 72h skil 84h

North America

Europe

2m TemperatureSkill (rmse) for different forecast ranges

Top panel: North America

Bottom panel: Europe

Higher skill in winter. The positive trend of the timeseries for both winter and summer periods indicates continuous forecast improvements.

Higher skill in winter, interrupted in 2005 by a period of strong inversion at low levels in Central Europe which has not been represented properly by the model..

ECMWF forecast products users meeting – Reading, June 2005 Slide 4

-0.1

0

0.1

0.2

0.3

0.4

0.5

g /

kg

-0.1

0

0.1

0.2

0.3

0.4

0.5

J

1998A J O J

1999A J O J

2000A J O J

2001A J O J

2002A J O J

2003A J O J

2004A J O J

2005

fc error of indate findate wp step ntot BIAS STDV RMSE SKILL MAE SKILL CORR M-OB M-FC

skil 48h skil 60h skil 72h skil 84h

Europe

Europe

Skill for different forecast ranges

Top panel: Specific humidity

Bottom panel: 10m wind speed

Higher skill (MAE) in winter. Last four winters have consistently kept higher level of

performance

Skill (RMSE): Changes in the forecasting model have not greatly improved the performance of the model in forecasting wind speed.

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

m /

s

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

J

1998A J O J

1999A J O J

2000A J O J

2001A J O J

2002A J O J

2003A J O J

2004A J O J

2005

fc error of indate findate wp step ntot BIAS STDV RMSE SKILL MAE SKILL CORR M-OB M-FC

skil 48h skil 60h skil 72h skil 84h

ECMWF forecast products users meeting – Reading, June 2005 Slide 5

The skill has been averaged over a year

1998

2001

Skill (rmse) plotted vs forecast day for different parameters.

Total Cloud Cover: black2m temperature: blue

ECMWF forecast products users meeting – Reading, June 2005 Slide 6

a b

c d

Observed yes Observed no

Forecast

yes

Forecast

no

1. FREQUENCY BIAS INDEX

caba

FBI

2. TRUE SKILL SCORE

3. HIT RATE

3. FALSE ALARM RATE

db

b

ca

aTSS

db

bF

ECMWF forecast products users meeting – Reading, June 2005 Slide 7

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

TS

S

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

S1993

D M1994

J S D M1995

J S D M1996

J S D M1997

J S D M1998

J S D M1999

J S D M2000

J S D M2001

J S D M2002

J S D M2003

J S D M2004

J S D M2005

precipitation exceeding 2.0 mm/24h

t + 42 t + 66

0

0.2

0.4

0.6

0.8

TS

S

0

0.2

0.4

0.6

0.8

S1993

D M1994

J S D M1995

J S D M1996

J S D M1997

J S D M1998

J S D M1999

J S D M2000

J S D M2001

J S D M2002

J S D M2003

J S D M2004

J S D M2005

precipitation exceeding 25.0 mm/24h

t + 42 t + 66

2mm/24h

25mm/24h

Europe

24 hour accumulated precipitation verified against SYNOP on GTS

The forecast is reduced to a yes/no event by selecting thresholds. Confidence intervals have been plotted for each TSS value.

High thresholds have large confidence intervals, important to remember when assessing performance of the system

t+42

t+66

ECMWF forecast products users meeting – Reading, June 2005 Slide 8

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1.4

1.5

1.6

1.7

FB

I

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1.4

1.5

1.6

1.7

S1993

D M1994

J S D M1995

J S D M1996

J S D M1997

J S D M1998

J S D M1999

J S D M2000

J S D M2001

J S D M2002

J S D M2003

J S D M2004

J S D M2005

precipitation exceeding 1.0 mm/24h

t + 42 t + 66

Europe

24 hour accumulated precipitation verified against SYNOP on GTS

The forecasting system over-estimate the number of events for thresholds of 1mm/24h. A decrease of FBI was observed when in the autumn 1999, when vertical resolution was increased and a new convection scheme was implemented.

FBI measures the ratio between the frequency of the forecast events and the frequency of the observed events. FBI>1 over-estimateFBI<1 under-estimate

t+42: solid shadingt+66: dotted shading

ECMWF forecast products users meeting – Reading, June 2005 Slide 9

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1.4

1.5

1.6

1.7

FB

I

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1.4

1.5

1.6

1.7

S1993

D M1994

J S D M1995

J S D M1996

J S D M1997

J S D M1998

J S D M1999

J S D M2000

J S D M2001

J S D M2002

J S D M2003

J S D M2004

J S D M2005

precipitation exceeding 5.0 mm/24h

t + 42 t + 66

FBI decreases to values closer to 1 as we increase the threshold, but higher thresholds have larger confidence intervals!

24 hour accumulated precipitation verified against SYNOP on GTS

Europe 5mm/24h t+42: solid shadingt+66: dotted shading

ECMWF forecast products users meeting – Reading, June 2005 Slide 10

40°N

50°N

60°N

70°N

40°W

40°W

20°W

20°W

0°

0°

20°E

20°E 40°E

40°E

1

2

5

10

15

20

25

40

40°N

50°N

60°N

70°N40°W

40°W

20°W

20°W

0°

0°

20°E

20°E 40°E

40°E

60°E

60°E

1

2

5

10

15

20

25

40

Precipitation analysis for Europe

•High density networks in Europe (Member and Co-operating states)•Upscaling (simple box averaging to obtain a areal precipitation value)

ECMWF forecast products users meeting – Reading, June 2005 Slide 11

5

10

1515

20

50°N

60°N

0°

0°

1

2

5

10

15

20

25

40

Each grid box will contain a certain number of stations. The number of stations will not be constant every day.

The number of stations per grid box indicates how representative the analysis is for the specific grid point.

ECMWF forecast products users meeting – Reading, June 2005 Slide 12

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1.4

1.5

1.6

1.7

FB

I

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1.4

1.5

1.6

1.7

J2002

S D M2003

J S D M2004

precipitation exceeding 0.25 mm/24h

t + 42 t + 42 t + 42 t + 42

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1.4

1.5

1.6

1.7

FB

I

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1.4

1.5

1.6

1.7

J2002

S D M2003

J S D M2004

precipitation exceeding 10.0 mm/24h

t + 42 t + 42 t + 42 t + 42

Europe

FBI plotted for two thresholds (0.25mm/24h, and 10mm/24h)

•Verification against precipitation analysis (yellow shading), •Verification against SYNOP on GTS (blue dotted)

FBI values are higher (lower) in the verification against SYNOP on the GTS (analysis) for lower (higher) thresholds.

0.25mm/24h

10mm/24h

Forecast range t+42

ECMWF forecast products users meeting – Reading, June 2005 Slide 13

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

TS

S

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

J2002

S D M2003

J S D M2004

precipitation exceeding 0.25 mm/24h

t + 42 t + 42 t + 42 t + 42

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

TS

S

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

J2002

S D M2003

J S D M2004

precipitation exceeding 0.25 mm/24h

t + 90 t + 90 t + 90 t + 90

Europe

TSS (threshold 0.25mm/24h) plotted for two forecast ranges: t+42 (top) and t+90 (bottom)

•Verification against precipitation analysis (yellow shading), •Verification against SYNOP on GTS (blue dotted)

TSS values decrease as we increase forecast range. In January 2003 there was a model change: improved cloud scheme numerics, revised cloud scheme and convection

0.25mm/24h

t+42

t+90

ECMWF forecast products users meeting – Reading, June 2005 Slide 14

0.1

0.2

0.3

0.4

0.5

0.6

TS

S

0.1

0.2

0.3

0.4

0.5

0.6

J2002

S D M2003

J S D M2004

precipitation exceeding 5.0 mm/24h

t + 90 t + 90 t + 90 t + 90

0.1

0.2

0.3

0.4

0.5

0.6

TS

S

0.1

0.2

0.3

0.4

0.5

0.6

J2002

S D M2003

J S D M2004

precipitation exceeding 15.0 mm/24h

t + 90 t + 90 t + 90 t + 90

Europe

TSS plotted for two thresholds (5mm/24h, and 15mm/24h)

•Verification against precipitation analysis (yellow shading), •Verification against SYNOP on GTS (blue dotted)

TSS values are higher for winter months. Confidence intervals become larger as threshold increases. Forecast range t+90

5mm/24h

15mm/24h

ECMWF forecast products users meeting – Reading, June 2005 Slide 15

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1994A

1995A D A

1996A D A

1997A D A

1998A D A

1999A D A

2000A D A

2001A D A

2002A D A

2003A D A

2004A D

2005

Brier skill score (sample clim) fc step 96 24h-precipitation exceedingProbability forecast verification against obs ( 3-M. moving sample)

1 mm 5 mm 10 mm 20 mm

Timeseries of Brier Skill Score for Europe

The BSS is written as 1- BS/BSref Sample climate is the reference system

BS measures the mean squared difference between forecast and observation in probability space. Equivalent to MSE for deterministic forecast

Forecast vs. observations

Improvements back in Autumn 1999 – High thresholds performance down at the beginning of 2005 linked to drier conditions over Europe?

Increased resolution

ECMWF forecast products users meeting – Reading, June 2005 Slide 16

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1994A

1995A D A

1996A D A

1997A D A

1998A D A

1999A D A

2000A D A

2001A D A

2002A D A

2003A D A

2004A D

2005

Brier skill score (long term clim) fc step 96 24h-precipitation exceedingProbability forecast verification against an ( 3-M. moving sample)

1 mm 5 mm 10 mm 20 mm

Timeseries of Brier Skill Score for Europe

The BSS is written as 1- BS/BSref Sample climate is the reference system

BS measures the mean squared difference between forecast and observation in probability space. Equivalent to MSE for deterministic forecast C

Forecast vs proxy

Increased resolution

ECMWF forecast products users meeting – Reading, June 2005 Slide 17

0.2

0.2

0.4

0.4

0.6

0.6

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

ob

s fr

equ

ency

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

forecast probability

55666350

3923

3203

2773

2585

2668

2582

2653

2719

4815

sample clim = 0.28 BS = 0.153 SSBS = 0.24 bias(ctr) = 0.3020031001-20040430 STEP 96 24-hour precipitation gt 1.0

0

0.1

0 0.2 0.4 0.6 0.8 1

rel FC distribution

sample clim

0.2

0.2

0.4

0.4

0.6

0.6

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

ob

s f

req

ue

nc

y

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

forecast probability

1484010250

4351

2948

2076

1548

1236

962

741

513372

sample clim = 0.13 BS = 0.082 SSBS = 0.26 bias(ctr) = 0.3020031001-20040430 STEP 96 24-hour precipitation gt 5.0

0

0.1

0.2

0.3

0.4

0 0.2 0.4 0.6 0.8 1

rel FC distribution

sample clim

0.2

0.2

0.4

0.4

0.6

0.6

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

ob

s fr

equ

ency

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

forecast probability

67515918

3400

2853

2555

2301

2269

2322

2443

2777

5345

sample clim = 0.26 BS = 0.157 SSBS = 0.19 bias(ctr) = 0.2820041001-20050430 STEP 96 24-hour precipitation gt 1.0

0

0.1

0 0.2 0.4 0.6 0.8 1

rel FC distribution

sample clim

0.2

0.2

0.4

0.4

0.6

0.6

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

ob

s f

req

ue

nc

y

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

forecast probability

167569578

3778

2385

1665

1263

1047

841

643

519

459

sample clim = 0.11 BS = 0.079 SSBS = 0.21 bias(ctr) = 0.2820041001-20050430 STEP 96 24-hour precipitation gt 5.0

0

0.1

0.2

0.3

0.4

0.5

0 0.2 0.4 0.6 0.8 1

rel FC distribution

sample clim

EuropeRainy season: October to AprilForecast range: t+96Verification against SYNOP on GTS

2003-20041mm/24h BS=0.153

2004-20051mm/24h BS=0.157

Consistent picture for the two seasons

ECMWF forecast products users meeting – Reading, June 2005 Slide 18

0.2

0.2

0.4

0.4

0.6

0.6

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

ob

s fr

equ

ency

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

forecast probability

2295610255

2901

1446

891

579

325 234

131

63

56

sample clim = 0.06 BS = 0.045 SSBS = 0.20 bias(ctr) = 0.3020031001-20040430 STEP 96 24-hour precipitation gt 10.0

0

0.1

0.2

0.3

0.4

0.5

0.6

0 0.2 0.4 0.6 0.8 1

rel FC distribution

sample clim

0.2

0.2

0.4

0.4

0.6

0.6

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

ob

s f

req

ue

nc

y

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

forecast probability

329045612

735

278

13780

48

17

14

10

2

sample clim = 0.02 BS = 0.016 SSBS = 0.10 bias(ctr) = 0.3020031001-20040430 STEP 96 24-hour precipitation gt 20.0

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

rel FC distribution

sample clim

0.2

0.2

0.4

0.4

0.6

0.6

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

ob

s fr

equ

ency

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

forecast probability

259208057

2096

1070

649410

268

183137

7965

sample clim = 0.05 BS = 0.040 SSBS = 0.16 bias(ctr) = 0.2820041001-20050430 STEP 96 24-hour precipitation gt 10.0

0

0.2

0.4

0.6

0.8

0 0.2 0.4 0.6 0.8 1

rel FC distribution

sample clim

0.2

0.2

0.4

0.4

0.6

0.6

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

ob

s f

req

ue

nc

y

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

forecast probability

346083414

477199

95

60

33

2220

4

2

sample clim = 0.01 BS = 0.012 SSBS = 0.12 bias(ctr) = 0.2820041001-20050430 STEP 96 24-hour precipitation gt 20.0

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

rel FC distribution

sample clim

EuropeRainy season: October to AprilForecast range: t+96Verification against SYNOP on GTS

2004-200510 mm/24h BS=0.04

2003-200410 mm/24h BS=0.045

Consistent picture for the two seasons. Higher thresholds better reliability

ECMWF forecast products users meeting – Reading, June 2005 Slide 19

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

hit

rat

e

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

false alarm rate

20031001-20040430 24-hour precipitation gt 1.0

0

2000

4000

6000

8000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

STEP 96

0

2000

4000

6000

8000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

STEP 120

t+ 96 A=0.862

t+120 A=0.828

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

hit

ra

te

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

false alarm rate

20031001-20040430 24-hour precipitation gt 5.0

0

5000

0.100 105

0.150 105

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

STEP 96

0

5000

0.100 105

0.150 105

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

STEP 120

t+ 96 A=0.873

t+120 A=0.838

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

hit

rat

e

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

false alarm rate

20041001-20050430 24-hour precipitation gt 1.0

0

2000

4000

6000

8000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

STEP 96

0

2000

4000

6000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

STEP 120

t+ 96 A=0.864

t+120 A=0.830

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

hit

ra

te

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

false alarm rate

20041001-20050430 24-hour precipitation gt 5.0

0

5000

0.100 105

0.150 105

0.200 105

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

STEP 96

0

5000

0.100 105

0.150 105

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

STEP 120

t+ 96 A=0.864

t+120 A=0.832

EuropeRainy season: October to AprilForecast range: t+96Verification against SYNOP on GTS

2004-20055 mm/24h

2003-20045 mm/24h

Consistent picture for the two seasons.

Full symbol: T511Shape: T255

ECMWF forecast products users meeting – Reading, June 2005 Slide 20

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

RO

CA

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

J1996

S D M1997

J S D M1998

J S D M1999

J S D M2000

J S D M2001

J S D M2002

J S D M2003

J S D M2004

J S D M2005

24 hour total precipitation verified against observations t+ 96

threshold = 1 threshold = 5 threshold = 10 threshold = 20

Increased resolution

EuropeROC Area Verification against SYNOP on GTS for t+96

Drier conditions over Europe?

ECMWF forecast products users meeting – Reading, June 2005 Slide 21

Conclusion 2m Temperature: positive trends

show increased skills for Europe

and North America. Strong

inversion in the winter was not

properly forecast by the T511.

Specific humidity shows increased

skills. Winters more skilful than

summers

Wind: The changes in the model

have not brought large

improvements in the wind speed

forecast

TCC: small improvements in

forecast skill. New cloud scheme

was introduced in April 2005

Importance of confidence intervals

Precipitation forecast

improvements are slow, but evident.

FBI indicates over-estimation of

small threshold events

verification against precipitation

analysis shows a better picture.

Precipitation analysis can be used

for verification in a delayed mode.

The number of station per grid box

gives and indication on how

representative is the analysis at any

grid point

ECMWF forecast products users meeting – Reading, June 2005 Slide 22

Brier skill score and ROC area:

increase in resolution has improved

the system. In recent year the

system has maintained its good

performance. The drier conditions

of the recent winter show up in the

timeseries small sample size

effects?

Reliability diagrams for the last two

rainy seasons show a skilful system

Conclusion