Local Polynomial Method for Local Polynomial Method for Ensemble Forecast of Time Ensemble Forecast of Time

SeriesSeries

Satish Kumar Regonda, Balaji Rajagopalan,Satish Kumar Regonda, Balaji Rajagopalan,

Upmanu Lall, Martyn Clark, and Young-II MoonUpmanu Lall, Martyn Clark, and Young-II Moon

Hydrology Days 2005Hydrology Days 2005

Colorado State University, Fort Collins, COColorado State University, Fort Collins, CO

• Stochastic models (AR, ARMA,………)Stochastic models (AR, ARMA,………)• Presume time series of a response variable as a realization Presume time series of a response variable as a realization

of a of a random processrandom process

xxtt==ff(x(xt-1t-1,x,xt-2t-2,….x,….xt-kt-k) + e) + ett• Noise, finite data length, high temporal and spatial Noise, finite data length, high temporal and spatial

variation of the data influences estimation of “k” variation of the data influences estimation of “k” • RandomnessRandomness in the system limits the predictability in the system limits the predictability

which could bewhich could be• A result of many independent and irreducible degrees of A result of many independent and irreducible degrees of

freedomfreedom• Due to Due to deterministic chaosdeterministic chaos

Time series modelingTime series modeling

What is deterministic What is deterministic Chaos?Chaos?

• Three coupled non-linear Three coupled non-linear differential equationsdifferential equations

• System apparently seems System apparently seems erratic, complex, and erratic, complex, and almost random (and that almost random (and that are very sensitive to are very sensitive to initial conditions), infact, initial conditions), infact, the system is the system is deterministic.deterministic.

Lorenz Attractor

Chaotic Chaotic systemssystems

Logistic Equation:Logistic Equation:

Xn+1 =A* Xn*(1-Xn)Xn+1 =A* Xn*(1-Xn)

‘‘A’ is constantA’ is constant

‘‘XXnn’ is Current Value’ is Current Value

‘‘XXn+1n+1’ is Future Value’ is Future Value

How these will be predicted?How these will be predicted?

- “nonlinear dynamical - “nonlinear dynamical based time series analysis”based time series analysis”

-0.1

0.1

0.3

0.5

0.7

0.9

1.1

1.3

1.5

100 150 200 250 300 350 400

No. of Iterations

Xn

Constant = 3.9 and Lambda= 0.7095

Logistic Equation

0

0.2

0.4

0.6

0.8

1

0 5 10 15 20 25

No. of iterations

Xn

X0 = 0.4000

X0 = 0.4010

X0 = 0.4050

Nonlinear Dynamics Based Forecasting Nonlinear Dynamics Based Forecasting procedureprocedure

1.1. xxtt== xx11,x,x22,x,x33,…,x,…,xnn2.2. State space reconstruction State space reconstruction

(or (or dynamics recoverydynamics recovery) ) using ‘m’ and ‘using ‘m’ and ‘’’

3.3. Forecast for T time steps Forecast for T time steps into future i.e.,into future i.e.,

xxt+Tt+T = = ff ( (XXtt) + ) + tt

• XXtt is a feature vector is a feature vector• ff is a linear or nonlinear is a linear or nonlinear

function function

nnn

nnn

nnn

n

n

xxx

xxx

xxx

xxx

xxx

xxx

x

x

x

x

x

x

,,

,,

,,

...............

...............

,,

,,

,,

.

.

24

135

246

753

642

531

?

1

8

7

6

],..,,[ )1(1)1( mtttmt xxxx

m = 3 and = 2

Local Map Local Map ff

• Forecast for ‘T’ time step into the futureForecast for ‘T’ time step into the future

xxt+Tt+T = = ff ( (XXtt) + ) + tt

• Typically, Typically, f(.)f(.) estimated locally within neighborhood of estimated locally within neighborhood of the feature vectorthe feature vector

• ff (.) approximated using locally weighted polynomials (.) approximated using locally weighted polynomials defined as LOCFITdefined as LOCFIT• Polynomial order ‘Polynomial order ‘pp’’• Number of neighbors Number of neighbors KK ( = ( = **nn, , is fraction between (0,1] ) is fraction between (0,1] )

• M is estimated using M is estimated using correlation dimension correlation dimension (Grassberger and Procaccio..xx)(Grassberger and Procaccio..xx)

or or False Neighbors False Neighbors (Kennel…xx)(Kennel…xx)

• Tau is estimated via Tau is estimated via Mutual Information Mutual Information (Sweeney, xx; Moon et al., 19xx)(Sweeney, xx; Moon et al., 19xx)

Estimation of m and Estimation of m and tau..tau..

• In real data, due to Noise (sampling and In real data, due to Noise (sampling and dynamical) the phase space parameters (i.e., dynamical) the phase space parameters (i.e., Embedding dimension and Delay time) are not Embedding dimension and Delay time) are not uniquely estimated.uniquely estimated.

• Hence, a suite of plausible parameters of the Hence, a suite of plausible parameters of the state space i.e. state space i.e. D, D, , , , p., p.

Need for Need for Ensembles..Ensembles..

Cont’Cont’dd

• Forecast for ‘T’ time step into the futureForecast for ‘T’ time step into the future

xxt+Tt+T = = ff ( (XXtt) + ) + tt

• Typically, Typically, f(.)f(.) estimated locally within neighborhood of estimated locally within neighborhood of the feature vectorthe feature vector

• ff (.) approximated using locally weighted polynomials (.) approximated using locally weighted polynomials defined as LOCFITdefined as LOCFIT• Polynomial order ‘Polynomial order ‘pp’’• Number of neighbors Number of neighbors KK ( = ( = **nn, , is fraction between (0,1] ) is fraction between (0,1] )

Cont’Cont’dd

• General Cross Validation (GCV) is used to General Cross Validation (GCV) is used to select the optimal parameters (select the optimal parameters ( and and pp))

• Optimal parameter is the one that produces Optimal parameter is the one that produces minimum GCVminimum GCV

21

2

1

),(

nm

ne

pGCV

n

i

i

ei is the errorn-number of data pointsm-number of parameters

Forecast Forecast algorithmalgorithm

1.1. Compute D and Compute D and using the standard methods and using the standard methods and choose a broad range of D and choose a broad range of D and values. values.

2.2. Reconstruct phase space for selected parametersReconstruct phase space for selected parameters3.3. Calculate GCV for the reconstructed phase space by Calculate GCV for the reconstructed phase space by

varying smoothening parameters of LOCFITvarying smoothening parameters of LOCFIT4.4. Repeat steps 2 and 3 for all combinations of Repeat steps 2 and 3 for all combinations of D, D, , , , p, p5.5. Select a suite of “best” parameter combinations that are Select a suite of “best” parameter combinations that are

within 5 percent of the lowest GCVwithin 5 percent of the lowest GCV6.6. Each selected best combination is then used to generate Each selected best combination is then used to generate

a forecast.a forecast.

ApplicationApplicationss

• Synthetic dataSynthetic data• The Henon systemThe Henon system• The Lorenz systemThe Lorenz system

• Geophysical dataGeophysical data• The Great Salt Lake (GSL)The Great Salt Lake (GSL)• NINO3NINO3

Time seriesTime series

Henon X- ordinate

GSL

Lorenz X- ordinate

NINO3

Synthetic Synthetic DataData

• The Lorenz systemThe Lorenz system• A time series of 6000 observations generatedA time series of 6000 observations generated• Embedding dimension: 2.06 and 3.0Embedding dimension: 2.06 and 3.0• Training period: 5500 observationsTraining period: 5500 observations• Selected parameters: D = Selected parameters: D = 2 & 3, 2 & 3, = 1 & 2, and, = 1 & 2, and, pp

= 2 with various neighbor sizes (= 2 with various neighbor sizes ().). • Forecasted 100 time steps into the future.Forecasted 100 time steps into the future.• Predictability less than the Henon system ( large Predictability less than the Henon system ( large

lyapunov exponent)lyapunov exponent)

Blind Blind PredictionPredictionIndex 5368 Index 5371

Unstable Region Stable Region~ 3 to 5 time steps ~ 35 points

The Great Salt Lake of Utah (GSL) The Great Salt Lake of Utah (GSL)

• It is the fourth largest, perennial, closed basin, saline It is the fourth largest, perennial, closed basin, saline lake in the world.lake in the world.

• Biweekly observations 1847-2002.Biweekly observations 1847-2002.• Superposition of strong and recurrent climate patterns Superposition of strong and recurrent climate patterns

at different timescales created a tough job of at different timescales created a tough job of prediction for classical time series models.prediction for classical time series models.

• Closed basin-----integrates hydrologic response and Closed basin-----integrates hydrologic response and filters out high-frequency phenomenon and results filters out high-frequency phenomenon and results into low dimensional phenomena.into low dimensional phenomena.

• GSL – is a low dimensional chaotic system GSL – is a low dimensional chaotic system (Sangoyomi et al. 1996)(Sangoyomi et al. 1996)

GSL AttractorGSL Attractor

• Annual cycle is Annual cycle is approximately motion approximately motion around the smaller around the smaller radius of the ‘spool’radius of the ‘spool’

• Longer term motion Longer term motion which has larger which has larger amplitude moves the amplitude moves the orbits along the longer orbits along the longer axis of the ‘spool’axis of the ‘spool’

ResultResultss

• Embedding dimension: 4 ; delay time: 14Embedding dimension: 4 ; delay time: 14• GCV values computed over D = 2 to 6 and GCV values computed over D = 2 to 6 and = =

10 to 20, and p = 1 to 2, with various neighbor 10 to 20, and p = 1 to 2, with various neighbor sizessizes

• Fall of the lake volumeFall of the lake volume• D = 4 & 5, D = 4 & 5, = 10,14, &15, p = 1 &2, = 10,14, &15, p = 1 &2, = 0.1-0.5 = 0.1-0.5

• Rise of the lake volumeRise of the lake volume• D = 4 & 5, D = 4 & 5, = 10 &15, p = 2, = 10 &15, p = 2, = 0.1-0.4 = 0.1-0.4

Fall of the lake volume

Blind Blind PredictionPrediction

Rise of the lake volume

Blind PredictionBlind Prediction

NINONINO33

• Time series of averaged monthly SST anomalies in Time series of averaged monthly SST anomalies in the tropical Pacific covering the domain of 4the tropical Pacific covering the domain of 4ooN-4N-4ooS S and 90and 90oo-150-150ooWW

• Monthly observations from 1856 onwardsMonthly observations from 1856 onwards• ENSO characteristics (e.g. onset, termination, ENSO characteristics (e.g. onset, termination,

cyclic nature, partial locking to seasonal cycle, and cyclic nature, partial locking to seasonal cycle, and irregularity) explained presuming system as low irregularity) explained presuming system as low order chaotic system order chaotic system (embedding dimension 3.5; Tziperman et al. (embedding dimension 3.5; Tziperman et al. 1994 and 1995)1994 and 1995)

ResultResultss

• El Nino Events (1982 and 1997):El Nino Events (1982 and 1997):• 1982-83: D = 4 and 1982-83: D = 4 and = 16 = 16• 1997-98: D = 5 and 1997-98: D = 5 and = 13 = 13• Selected parameters range: D = 2 to 5, Selected parameters range: D = 2 to 5, = 11 to 21 = 11 to 21

( 8 to 16), p = 1& 2, ( 8 to 16), p = 1& 2, = 0.1 – 1.0 = 0.1 – 1.0• Forecasted issued in different months of the eventForecasted issued in different months of the event• Ensemble prediction did a slightly better job Ensemble prediction did a slightly better job

compared to best AR-modelcompared to best AR-model

1997-98 El Nino

Blind PredictionBlind Prediction

ResultResultss

• La Nina events (1984 and 1989 )La Nina events (1984 and 1989 )• Both events yielded a dimension and delay time of 5 Both events yielded a dimension and delay time of 5

and 17 respectively.and 17 respectively.• Selected parameters range: D = 2 to 5, Selected parameters range: D = 2 to 5, = 12 to 22, = 12 to 22,

p = 1& 2, p = 1& 2, = 0.1 – 1.0 = 0.1 – 1.0• Forecasted issued in different monthsForecasted issued in different months• Both, ensemble and AR, methods performed Both, ensemble and AR, methods performed

similarly with increasing skill of the predictions similarly with increasing skill of the predictions when issued closer to the negative peak of the eventswhen issued closer to the negative peak of the events

1999-2000 La Nina

Blind PredictionBlind Prediction

Recent predictionRecent predictionMay 1, 2002 July, 2004(GSL) (NINO3)

SummarSummaryy

• A new algorithm proposed which selects a suite of A new algorithm proposed which selects a suite of ‘best’ parameters that captures effectively dynamics ‘best’ parameters that captures effectively dynamics of the systemof the system

• Ensemble forecasts provideEnsemble forecasts provide• A natural estimate of the forecast uncertaintyA natural estimate of the forecast uncertainty• The pdf of the response variable and consequently The pdf of the response variable and consequently

threshold exceedance probabilitiesthreshold exceedance probabilities

• Decision makers will be benefited as forecast issued Decision makers will be benefited as forecast issued with a good lead-timewith a good lead-time

• Performs better than the best AR-modelPerforms better than the best AR-model• It could be improved in several waysIt could be improved in several ways

AcknowledgementsAcknowledgements

• Thanks to CADSWES at the Univ. of Thanks to CADSWES at the Univ. of Colorado at Boulder for letting use of its’ Colorado at Boulder for letting use of its’ computational facilities.computational facilities.

• Support from NOAA grant NA17RJ1229 and Support from NOAA grant NA17RJ1229 and NSF grant EAR 9973125 are thankfully NSF grant EAR 9973125 are thankfully acknowledged.acknowledged.

Publication:Publication:-- Regonda, S., B. Rajagopalan, U. Lall, M. Clark and Y. Moon, Local Regonda, S., B. Rajagopalan, U. Lall, M. Clark and Y. Moon, Local

polynomial method for ensemble forecast of time series, (in press)polynomial method for ensemble forecast of time series, (in press) Nonlinear Processes in Geophysics,Nonlinear Processes in Geophysics, Special issue on "Nonlinear Special issue on "Nonlinear Deterministic Dynamics in Hydrologic Systems: Present Activities and Deterministic Dynamics in Hydrologic Systems: Present Activities and

Future Challenges", 2005.Future Challenges", 2005.

Thank You

dx / dt = a (y - x) dy / dt = x (b - z) - y

dz / dt = xy - c z

Lorenz Equations:

Lorenz attractor derived from a simplified model of convection to see the effect of initial conditions. The system is most commonly expressed as 3 coupled non-linear differential equations, which are known as Lorenz Equations.

a, b, and c are the constants

Chua Attractor (electronic circuit) Duffing (nonlinear Oscillator)

State space reconstructionState space reconstruction

• Time series of observationsTime series of observations

xxtt== xx11,x,x22,x,x33,………,x,………,xnn• Embedding time series into ‘m’ dimensional phase Embedding time series into ‘m’ dimensional phase

space i.e., recovering dynamics of the systemspace i.e., recovering dynamics of the system

XXtt = {x = {xtt, x, xt+t+, x, xt+ 2t+ 2,…., x,…., xt+(m-1) t+(m-1) } } (Takens, 1981)(Takens, 1981)

m – embedding dimensionm – embedding dimension - Delay time - Delay time

(Lall et al. 1996)

Henon

LorenzD=3

LorenzD=2, Tau=1

LorenzD=3, Tau=1

Does GSL show Chaotic nature?Does GSL show Chaotic nature?

• Long term persistence of the climate variations i.e.,Long term persistence of the climate variations i.e.,• because of superposition of strong, recurrent patterns at because of superposition of strong, recurrent patterns at

different scales makes difficult to justify classical, time different scales makes difficult to justify classical, time series modelsseries models

• Closed basin-----integrates hydrologic response and Closed basin-----integrates hydrologic response and filters out high-frequency phenomenon and results filters out high-frequency phenomenon and results into low dimensional phenomena.into low dimensional phenomena.

• GSL – is a low dimensional chaotic system GSL – is a low dimensional chaotic system (Sangoyomi et al. 1996)(Sangoyomi et al. 1996)

1982-83 El Nino

Blind PredictionBlind Prediction

1984-85 La Nina

Blind PredictionBlind Prediction

Nonlinear dynamics Based Time SeriesNonlinear dynamics Based Time Series

1.1. State space reconstruction (Takens’ State space reconstruction (Takens’ embedding theorem)embedding theorem)

Time series of a variable Time series of a variable

xxtt== xx11,x,x22,x,x33,………,x,………,xnn

XXtt = {x = {xtt, x, xt+t+, x, xt+ 2t+ 2,…., x,…., xt+(m-1) t+(m-1) } }

m – embedding dimensionm – embedding dimension - Delay - Delay timetime

2. Fit a function that maps different 2. Fit a function that maps different statesstates in the in the phase spacephase space

U. Lalll, 1994

Cont’Cont’dd

Importance of ‘m’ and ‘’

Chaotic behavior in geophysical Chaotic behavior in geophysical processesprocesses

• Lower order chaotic behavior observed in various Lower order chaotic behavior observed in various geophysical variables (e.g., rainfall, runoff, lake geophysical variables (e.g., rainfall, runoff, lake volume) on different scalesvolume) on different scales

• Diagnostic toolsDiagnostic tools• Grassberger-Procaccia algorithmGrassberger-Procaccia algorithm• False Nearest NeighborsFalse Nearest Neighbors• Lyapunov ExponentLyapunov Exponent

ResultResultss

• The Henon systemThe Henon system• 4000 observations of x-ordinate4000 observations of x-ordinate• Embedding dimension 2 and delay time 1Embedding dimension 2 and delay time 1• Training period: 3700 observations and searched Training period: 3700 observations and searched

over D = 1 to 5 and over D = 1 to 5 and = 1 to 10. = 1 to 10.• Selected combinations resulted 15 combinations Selected combinations resulted 15 combinations

and parameter values D = 2, and parameter values D = 2, = 1, = 1, pp=2 and with =2 and with various neighborhood sizes (i.e. various neighborhood sizes (i.e. ))

• Forecasted 100 time steps into futureForecasted 100 time steps into future

Blind Blind predictionpredictionIndex 3701 Index 3711

Ensemble forecast (5th&95th; 25th & 75th percentiles); Real observations; the best AR forecast