csa symposium 2016 - shanice bedward day 1 session 2

“Assessing the Skill of Seasonal Rainfall Forecast for Jamaica and Exploring Methods to Improve Existing

Skill”

Author: Sheldon GrantPresenter: Shanice Bedward

Content

IntroductionMethodology

ResultsThe way forward

LimitationsConclusion

Objectives

1. To use seasonal forecast verification as an approach to identifying strengths and weakness in the current forecast methodologies

2. To examine the influence of local, regional and global scale climate phenomena on Jamaica’s rainfall and to identify those that may be useful towards improving the skill of current forecast models

3. To create new predictive models for select seasons

Introduction

Methodology

Results

Way Forward

Limitations

Purpose of Study

Improving Forecasting

Skill

Inform Public

Early Warning

Planning

Capacity Building

Introduction

Methodology

Results

Way Forward

Limitations

Seasonal Forecast

http://jamaicaclimate.net

JFM FMA MAM AMJ MJJ JJA JAS SON OND NDJ DJF

January-February-March

Introduction

Methodology

Results

Way Forward

Limitations

Conclusion

Climate Predictability Tool (CPT)

Statistical tool currently providing the seasonal rainfall forecasts for Jamaica - Climate Predictability Tool (CPT).CPT offers the option of selecting an area over a predictor data set which correlates well with seasonal rainfall over the target region.

Introduction

Methodology

Results

Way Forward

Limitations

Conclusion

Is this “SKILL”?Introduction

Methodology

Results

Way Forward

Limitations

Validation

• What is a Skill Score?– Heidke Skill Score (HSS)– Ranked Probability Skill Score (RPSS)– Relative Operating Characteristic (ROC)

• Four Seasons Selected– JFM– AMJ– JAS– SON

Introduction

Methodology

Results

Way Forward

Limitations

Heidke Skill Score (HSS)

HSS = (H-E) / (N-E) ____Eq 1H is the number or categorically correct forecastsN is the number of forecasts issuedE is the number of forecast expected to occur by chance

NB:• Negative values indicate that the chance forecast is better;• 0 means no skill;• A perfect forecast obtains a HSS of 1.

Introduction

Methodology

Results

Way Forward

Limitations

Preliminary Results 1

• Skill of forecasts derived using CPT varies by season and by year.• JJA and OND show reasonable skill for 2013-2015.• MJJ shows poor skill. MJJ represents our early rainfall season.

Introduction

Methodology

Results

Way Forward

Limitations

1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

-0.30-0.20-0.100.000.100.200.300.400.500.600.70

Heidke Skill Score - AMJ

Years

Scor

es

1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

-0.40

-0.20

0.00

0.20

0.40

0.60

0.80

Heidke Skill Score - JAS

Years

Scor

es

Ranked Probability Skill Score (RPSS)

RPS = ∑ (PF(cum) – PO(cum))2 ___Eq 2

RPS is the ranked probability score for the forecastPF(cum) is the cumulative forecast probability

PO(cum) is the cumulative observation probability

RPSS = 1 – RPSfct/RPScli _______ Eq 3

RPSfct is the RPS for the forecast RPScli is the RPS climatology forecasts

Introduction

Methodology

Results

Way Forward

Limitations

Ranked Probability Skill Score (RPSS)

RPS = ∑ (PF(cum) – PO(cum))2 ___Eq 2

RPS is the ranked probability score for the forecastPF(cum) is the cumulative forecast probability

PO(cum) is the cumulative observation probability

RPSS = 1 – RPSfct/RPScli _______ Eq 3

RPSfct is the RPS for the forecast RPScli is the RPS climatology forecasts

NB:• Negative values indicate that the forecast is less accurate than the

standard forecast.• A perfect forecast obtains a RPSS of 1

Introduction

Methodology

Results

Way Forward

Limitations

JFM AMJ JAS SON

-1.2-1

-0.8-0.6-0.4-0.2

00.20.40.60.8

Manley - RPSS

2012201320142015

Seasons

Scor

es

JFM AMJ JAS SON

-0.60-0.40-0.200.000.200.400.600.801.001.20

Serge Island - RPSS

201320142015

Seasons

Scor

es JFM AMJ JAS SON

-1.5

-1

-0.5

0

0.5

1

Savanna-la-mar - RPSS

2012201320142015

Seasons

Scor

es

JFM AMJ JAS SON

-2

-1.5

-1

-0.5

0

0.5

1

Sangster - RPSS

2012201320142015

Seasons

Scor

es

Relative Operating Characteristic (ROC)

• It is a plot of the true positive rate (Hit rate) against the false positive rate (False alarm rate)

NB:An area of 1 represents a perfect test; an area of 0.5 represents a worthless test

Introduction

Methodology

Results

Way Forward

Limitations

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

ROC JFM

ROC Below.ROC ABOVE

False Alarm

Hit R

ate

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

ROC JAS

ROC Below.ROC ABOVE

False Alarm

Hit R

ate

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

ROC SON

ROC Below.ROC ABOVE

False Alarm

Hit R

ate

What Next?• Based on preliminary results, skill of models created in

CPT vary by year, season and location.

• Questions arising:– Are different seasons and different areas of the island

influenced by different sections of the ocean?– What are the other significant drivers on all scales? (Global,

Regional, Local)– Do local factors such as topography, influence rainfall on a

large scale?

Introduction

Methodology

Results

Way Forward

Limitations

Methodology and Datasets 2

• Identify the local, regional and global scale drivers that influence Jamaica’s Rainfall.– Analysis will include correlations,

canonical correlation analysis, composite maps, EOF’s

• Create and validate statistical models based motivated by drivers identified.– Analysis will include the use of backward

regression and cross-validation analyses

Rainfall data from the (Met Office).

SST’s and other identified parameters from various global institutions

Atmospheric Temperatures from atmospheric soundings done locally (Met Office)

Introduction

Methodology

Results

Way Forward

Limitations

Limitations

• Missing Data

• Relatively new for Jamaica

– Limited research on drivers other than ENSO especially small scale and local drivers for Jamaica

Introduction

Methodology

Results

Way Forward

Limitations

The End