verifying the accuracy of seasonal climate outlooks · the thirteen graphics following depict the...

Verifying the Accuracy ofSeasonal Climate Outlooks

Harvey Stern & Jonathan Pollock, Victorian Regional Climate Services Centre, Bureau of Meteorology, Melbourne

Background

Seasonal rainfall outlooks have been issued by the Bureau of Meteorology since the late 1980s, whilst seasonal minimum and maximum temperature outlooks have been issued from early 2000.

The purpose of this paper is to present preliminary results from an analysis of how accurate these outlooks have been.

For almost the entire period, the outlooks have been represented by a map of Australia with the probability of the parameter predicted (total rainfall, mean minimum and mean maximum temperature) exceeding the median.

2

Methodology

For each state and season, the forecast rainfall, maximum temperature, and minimum temperature, is regarded as equivalent to +1 should there be a region with >60% probability of the element exceeding the median and no region with <40% probability of the element exceeding the median.

For each state and season, the forecast element is regarded as equivalent to -1 should there be a region with <40% probability of the element exceeding the median and no region with >60% of the element exceeding the median.

For each state and season, all other forecasts are regarded as equivalent to 0.

For each state and season, the observed element is regarded as being equivalent to the element’s anomaly in mm (in the case of rainfall) or deg C (in the case of temperature).

3

Summary Graphics

The summary graphics shown on the next few slides depict the percent of the observed variance explained by the forecasts since January 2000 (FMA 2000).

The data are depicted on a ‘rolling’ season by season basis, with each season represented by the performance achieved by all forecasts including any month of that season, and also on a state by state basis.

To illustrate, the spring (SON) graphic refers to data associated with all five three-month seasons including a spring month, i.e. JAS, ASO,SON, OND, NDJ.

4

Rainfall

5

Seasonal Climate Outlooks 2000-2011 (% Variance Explained)

-10%

-5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

1.JF

M

2.FM

A

3.M

AM

4.A

MJ

5.M

JJ

6.JJ

A

7.JA

S

8.A

SO

9.S

ON

10.O

ND

11.N

DJ

12.D

JF

ALL

Rain-NT Rain-QLD Rain-NSW Rain-VIC Rain-TAS Rain-SA Rain-WA

Rainfall

The graphic shows that rainfall outlooks have displayed accuracy predominantly during the spring half of the year.

The State with the strongest performance is the Northern Territory, with more than 25% of the variance explained in ASO, SON and OND.

This is closely followed by Queensland and New South Wales, with a much weaker but still positive performance evident in the case of rainfall outlooks for Western Australia and Victoria.

Little skill is evident in rainfall outlooks for South Australia and Tasmania.

6

Minimum Temperature

7

Seasonal Climate Outlook 2000-2011 (% Variance Explained)

-10%

-5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

1.JF

M

2.FM

A

3.M

AM

4.AM

J

5.M

JJ

6.JJ

A

7.JA

S

8.AS

O

9.SO

N

10.O

ND

11.N

DJ

12.D

JF

ALL

Min-NT Min-QLD Min-NSW Min-VIC Min-TAS Min-SA Min-WA

Minimum Temperature

The graphic shows that minimum temperature outlooks have displayed accuracy predominantly during the autumn half of the year.

As is the case with the rainfall outlooks, the State with the strongest performance is the Northern Territory, with more than 25% of the variance explained in FMA, MAM and AMJ.

This is closely followed by Western Australia.

A much weaker but still positive performance is evident in the case of minimum temperature outlooks for New South Wales, Queensland and Western Australia.

Little skill is evident in minimum temperature outlooks for Victoria and Tasmania.

8

Maximum Temperature

9

Seasonal Climate Outlook 2000-2011 (% Variance Explained)

-10%

-5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

1.JF

M

2.FM

A

3.M

AM

4.AM

J

5.M

JJ

6.JJ

A

7.JA

S

8.A

SO

9.S

ON

10.O

ND

11.N

DJ

12.D

JF

ALL

Max-NT Max-QLD Max-NSW Max-VIC Max-TAS Max-SA Max-WA

Maximum Temperature

The graphic shows that maximum temperature outlooks have displayed some skill at most times of the year, the exception being late winter/early spring .

The States with the strongest performances are Queensland, Northern Territory, Western Australia and New South Wales.

Little skill is displayed by the maximum temperature outlooks for the other states –Victoria, Tasmania and South Australia.

10

Annual & Monthly

The thirteen graphics following depict the percent of the observed variance explained by the forecasts since January 2000 (FMA 2000), annual plus on a ‘rolling’ season by season basis, and also on a state by state basis.

Each season is represented by the performance achieved by all forecasts including any month of that season.

To illustrate, the spring (SON) graphic refers to data associated with all five three-month seasons including a spring month, i.e. JAS, ASO,SON, OND, NDJ.

11

Annual

12

Seasonal Climate Outlooks 2000-2011 (% variance explained)

-10%

-5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

NT QLD NSW VIC TAS SA WA ALL

State

% v

aria

nce

ALL.MAX-T ALL.MIN-T ALL.RAIN

Annual

The graphic shows that the outlooks have displayed some overall skill across the year in the case of predictions for most states.

The best rainfall outlooks are for Northern Territory, Queensland and New South Wales, the best minimum temperature outlooks are for Northern Territory and Western Australia whilst the best maximum temperature outlooks are for Queensland, Northern Territory, Western Australia and New South Wales.

13

JFM/FMA

14


-10%

-5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%


State

% v

aria

nce

01.MAX-T-JFM 01.MIN-T-JFM 01.RAIN-JFM


-10%

-5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%


State

% v

aria

nce

02.MAX-T-FMA 02.MIN-T-FMA 02.RAIN-FMA

The graphic shows that during the early part of the year, whilst the rainfall outlook have limited skill, notable skill is observed in the case of some of the temperature outlooks, especially the minimum temperature outlooks for the Northern Territory, and the maximum temperature outlooks for the Northern Territory, Queensland, New South Wales and Western Australia.

MAM/AMJ

15

As one moves through the autumn months, the skill of the rainfall outlooks all but disappears. However, the skill at predicting temperature increases, especially for the Northern Territory and Western Australia.


-10%

-5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%


State

% v

aria

nce

03.MAX-T-MAM 03.MIN-T-MAM 03.RAIN-MAM


-10%

-5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%


State

% v

aria

nce

04.MAX-T-AMJ 04.MIN-T-AMJ 04.RAIN-AMJ

MJJ/JJA

16

Through autumn and into the early part of winter, the skill of the rainfall outlooks continues to be negligible with the exception of the predictions for New South Wales. However, the skill at predicting temperature continues to be good, especially for the prediction of minimum and maximum temperature in the Northern Territory, minimum temperature in Western Australia and maximum temperature in Queensland.


-10%

-5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%


State

% v

aria

nce

05.MAX-T-MJJ 05.MIN-T-MJJ 05.RAIN-MJJ


-10%

-5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%


State

% v

aria

nce

06.MAX-T-JJA 06.MIN-T-JJA 06.RAIN-JJA

JAS/ASO

17

As we move into spring, the skill at forecasting seasonal rainfall dramatically increases in many states, especially for the Northern Territory, New South Wales, Queensland, Western Australia and, to a lesser extent, for Victoria.


-10%

-5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%


State

% v

aria

nce

07.MAX-T-JAS 07.MIN-T-JAS 07.RAIN-JAS


-10%

-5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%


State

% v

aria

nce

08.MAX-T-ASO 08.MIN-T-ASO 08.RAIN-ASO

SON/OND

18

Through spring and early summer, the skill at forecasting seasonal rainfall remains high in many states, especially for the Northern Territory, New South Wales, Queensland, Victoria and Western Australia.


-10%

-5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%


State

% v

aria

nce

09.MAX-T-SON 09.MIN-T-SON 09.RAIN-SON


-10%

-5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%


State

% v

aria

nce

10.MAX-T-OND 10.MIN-T-OND 10.RAIN-OND

NDJ/DJF

19

Into summer, and the skill at forecasting seasonal rainfall begins to decline, but maximum temperature forecasts remain skillful in Queensland , Northern Territory and Western Australia.


-10%

-5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%


State

% v

aria

nce

11.MAX-T-NDJ 11.MIN-T-NDJ 11.RAIN-NDJ


-10%

-5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%


State

% v

aria

nce

12.MAX-T-DJF 12.MIN-T-DJF 12.RAIN-DJF

Trends

20

Trends in the accuracy of the seasonal forecasts appear to be vary in concert with corresponding long-term variations in the strength of the relationship between predictors and the predictands.

The following graphics illustrate this feature for rainfall outlooks.

Medium term trends

21

Medium term (20+ years) trend in the potential predictive skill(over the preceding 5 years) of the IOD and the SOI (pink, LHS scale),

and the skill actually displayed over the by the official forecasts (blue, RHS scale)

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7M

ay-9

4

May

-95

May

-96

May

-97

May

-98

May

-99

May

-00

May

-01

May

-02

May

-03

May

-04

May

-05

May

-06

May

-07

May

-08

May

-09

May

-10

May

-11

5-year period ended

Aver

age

(NT,

QLD

, NS

W, V

IC, T

AS

, SA

, W

A) m

ultip

le c

orre

latio

n co

effic

ient

be

twee

n su

bseq

uent

rai

nfal

l and

a

com

bina

tion

of p

redi

ctor

s: IO

D, S

OI,

seas

on

-9%

-6%

-3%

0%

3%

6%

9%

12%

15%

18%

21%

Aver

age

(NT,

QLD

, NS

W, V

IC, T

AS

, SA

, W

A) %

var

ianc

e in

rai

nfal

l exp

lain

ed b

y fo

reca

sts

Medium term trends

22

The graphic shows that the skill of the rainfall outlooks during the early 1990s was close to zero, but this period also corresponded to a period when the relationship between seasonal rainfall and its predictors was weak (multiple correlation coefficients between predictors and observed rainfall also being close to zero).

The graphic shows that there was a peak in the skill displayed during the early 2000s (% variance explained ~ 10%), this period corresponding to a period when the relationship between seasonal rainfall and its predictors being strong (multiple correlation coefficients between predictors and observed rainfall ~ +0.3).

The strongest performance was registered during recent years (% variance explained being nearly 20%), when the strength of the relationship also peaked ((multiple correlation coefficients between predictors and observed rainfall ~ +0.5).

Long term trends

23

Long term (100+ years) trend in the potential predictive skill(over the preceding 5 years) of the IOD and the SOI (pink, LHS scale),

and the skill actually displayed over the by the official forecasts (blue, RHS scale)

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7De

c-04

Dec-

09

Dec-

14

Dec-

19

Dec-

24

Dec-

29

Dec-

34

Dec-

39

Dec-

44

Dec-

49

Dec-

54

Dec-

59

Dec-

64

Dec-

69

Dec-

74

Dec-

79

Dec-

84

Dec-

89

Dec-

94

Dec-

99

Dec-

04

Dec-

09

5-year period ended

Aver

age

(NT,

QLD

, NSW

, VIC

, TA

S, S

A,

WA

) mul

tiple

cor

rela

tion

coef

ficie

nt

betw

een

subs

eque

nt r

ainf

all a

nd a

co

mbi

natio

n of

pre

dict

ors:

IOD,

SO

I, se

ason

-9%

-6%

-3%

0%

3%

6%

9%

12%

15%

18%

21%

Aver

age

(NT,

QLD

, NSW

, VIC

, TA

S, S

A,

WA

) % v

aria

nce

in r

ainf

all e

xpla

ined

by

fore

cast

s

Long term trends

24

Looking back over more than 100 years, it may be seen that there have been two periods, in addition to the most recent one, when the relationship between seasonal rainfall and its predictors was strong (multiple correlation coefficients between predictors and observed rainfall being close to +0.5).

It may be suggested that it is no coincidence that the first of these periods (circa1920) precipitated the early work into seasonal forecasting by Quayle and others.

It also may be suggested that the reason interest in this work subsequently declined was the weakness in the relationship between seasonal rainfall and its predictors during the decades following.

The revitalisation of interest by McBride, Nicholls and others during the 1980s follows the peak in the skill displayed by the relationship during the 1970s.

Any Questions?

Thank You

verifying the accuracy of seasonal climate outlooks · the thirteen graphics following depict the...

Documents