![Page 1: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/1.jpg)
TIME SERIES ANALYSISTime Domain Models: Red Noise; AR
and ARMA models
LECTURE 7
Supplementary Readings:
Wilks, chapters 8
![Page 2: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/2.jpg)
Recall:
Statistical Model[Observed Data] = [Signal] + [Noise]
“noise” has to satisfy certain properties! If not, we must iterate on this process...
We will seek a general method of specifying just such a model…
![Page 3: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/3.jpg)
Statistics of the time series don’t change with time
We conventionally assume Stationarity
•Weak Stationarity (Covariance Stationarity)
statistics are a function of lag k but not absolute time t
•Strict Stationarity
•Cyclostationarity
statistics are a periodic function of lag k
![Page 4: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/4.jpg)
Time Series ModelingAll linear time series models can be written in the form:
1111
11
)(
mt
M
mtkt
K
kkt
mxx
We assume that are Gaussian distributed.
Autoregressive Moving-Average Model(“ARMA”)
Box and Jenkins (1976)
ARMA(K,M) MODEL
![Page 5: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/5.jpg)
111
1 )(
tkt
K
kkt xx
Consider Special case of simple Autoregressive AR(k) model (m=0)
Suppose k=1, and define 1
11 )(
ttt xx
This should look familiar!
Special case of a Markov Process (a process for which the state of system depends on previous states)
Time Series ModelingAll linear time series models can be written in the form:
1111
11
)(
mt
M
mtkt
K
kkt
mxx
![Page 6: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/6.jpg)
Suppose k=1, and define 1
11 )(
ttt xx
Time Series Modeling
Assume process is zero mean, then
11
tttxx
Lag-one correlated process or “AR(1)” process…
111
1 )(
tkt
K
kkt xx
Consider Special case of simple Autoregressive AR(k) model (m=0)
All linear time series models can be written in the form:
1111
11
)(
mt
M
mtkt
K
kkt
mxx
![Page 7: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/7.jpg)
11
kky
ky
ky
kky
ky
ky
12
1
ky
kky
ky
ky
n
k
n
k
n
k 12
11
1
1
1
1
1
21
1
1
1
1
ky
ky
ky
n
k
n
k
For simplicity, we assume zero mean
2
1
y
yy
nlii
l
AR(1) Process
![Page 8: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/8.jpg)
-4 -3 -2 -1 0 1 2 3 40
20
40
60
80
100
120
140
Let us take this series as a random forcing
0 100 200 300 400 500 600 700 800 900 1000-4
-3
-2
-1
0
1
2
3
4
85.02
181120 N
AR(1) Process
![Page 9: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/9.jpg)
0 100 200 300 400 500 600 700 800 900 1000-4
-3
-2
-1
0
1
2
3
4
0 10 20 30 40 50 60 70 80 90 100-3
-2
-1
0
1
2
3
4
AR(1) Process
Blue: =0.4Red: =0.7
Let us take this series as a random forcing
![Page 10: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/10.jpg)
What is the standard deviation of an AR(1) process?
2222 yy
2)1
(21
kk
yk
y
11
kky
ky
222
112
kk
yk
y
22
1
2
y
AR(1) Process
![Page 11: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/11.jpg)
0 10 20 30 40 50 60 70 80 90 100-6
-4
-2
0
2
4
6
8
10
Blue: =0.4Red: =0.7Green: =0.9
22
1
2
y
AR(1) Process
![Page 12: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/12.jpg)
010020030040050060070080090010000
5
10
15
20
25
30
35
40
45
50
Suppose =1
Random Walk (“Brownian Motion”)
Not stationary
How might we try to turn this into a stationary time
series?
AR(1) Process
22
1
2
y
11
kky
ky
Variance is infinite!
11
kky
ky
![Page 13: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/13.jpg)
Let us define the lag-k autocorrelation:
We can approximate:
2)(1
xxvar i
n
i
i
n
i
xx 1
varkn
xxxx
krkii
kn
i
)(
))((1
AR(1) ProcessAutocorrelation Function
i
kn
i
xx
1
i
n
ki
xx
1
2)(1
xxvar
i
kn
i
2)(1
xxvar i
n
ki
varvarkn
xxxx
krkii
kn
i
)(
))((1
Let us assume the series x has been de-meaned…
![Page 14: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/14.jpg)
Let us define the lag-k autocorrelation:
2
1i
n
ixvar
varkn
xx
krkii
kn
i
)(1
AR(1) ProcessAutocorrelation Function
Let us define the lag-k autocorrelation:
Then:
i
kn
i
xx
1
i
n
ki
xx
1
2)(1
xxvar
i
kn
i
2)(1
xxvar i
n
ki
varvarkn
xxxx
krkii
kn
i
)(
))((1
![Page 15: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/15.jpg)
0 50 100 150 200 250-0.4
-0.2
0
0.2
0.4
0.6
0.8
1Serial correlation function
Autocorrelation Function
1976 1978 1980 1982 1984 1986 1988 1990 1992 1994 1996336
338
340
342
344
346
348
350
352
354
356
CO2 since 1976
varkn
xx
krkii
kn
i
)(1
![Page 16: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/16.jpg)
11
kky
ky
Recursively, we thus have for an AR(1) process,k
kr
AR(1) ProcessAutocorrelation Function
kky
ky
11
]1
[
kkk
y
)1
(1
2
kkky
kkyky '112
)/exp()lnexp( kkk
rk
(Theoretical)
![Page 17: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/17.jpg)
varkn
xx
krkii
kn
i
)(1
0 5 10 15 20 25-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
“tcf.m” scf.m
0 10 20 30 40 50 60 70 80 90 100-2
-1
0
1
2
3
4
=0.5 N=100
“rednoise.m”
)/exp()lnexp( kkk
rk
AR(1) ProcessAutocorrelation Function
![Page 18: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/18.jpg)
varkn
xx
krkii
kn
i
)(1
0 5 10 15 20 25-0.2
0
0.2
0.4
0.6
0.8
1
1.2
=0.5 N=100
“rednoise.m”
)/exp()lnexp( kkk
rk
AR(1) ProcessAutocorrelation Function
0 50 100 150 200 250 300 350 400 450 500-4
-3
-2
-1
0
1
2
3
4
5
=0.5 N=500
“rednoise.m”
![Page 19: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/19.jpg)
varkn
xx
krkii
kn
i
)(1
0 5 10 15 20 25-0.2
0
0.2
0.4
0.6
0.8
1
1.2
0 100 200 300 400 500 600 700 800 900 1000-4
-3
-2
-1
0
1
2
3
4
)/exp()lnexp( kkk
rk
AR(1) ProcessAutocorrelation Function
Glacial Varves
=0.23 N=1000
![Page 20: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/20.jpg)
varkn
xx
krkii
kn
i
)(1
0 5 10 15 20 250
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 50 100 150-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
)/exp()lnexp( kkk
rk
AR(1) ProcessAutocorrelation Function
Northern Hem Temp
=0.75 N=144
![Page 21: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/21.jpg)
varkn
xx
krkii
kn
i
)(1
0 5 10 15 20 25-0.2
0
0.2
0.4
0.6
0.8
1
1.2
0 50 100 150-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
)/exp()lnexp( kkk
rk
AR(1) ProcessAutocorrelation Function
Northern Hem Temp (linearly detrended)
=0.54 N=144
![Page 22: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/22.jpg)
varkn
xx
krkii
kn
i
)(1
1 2 3 4 5 6 7 8 9 10 11-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
)/exp()lnexp( kkk
rk
AR(1) ProcessAutocorrelation Function
0 50 100 150-2
-1
0
1
2
3
4
Dec-Mar Nino3
![Page 23: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/23.jpg)
AR(1) Process
The sampling distribution for is given by the sampling distribution for the slope parameter in linear regression!
n/)1( 22
11
kky
ky
2/1
)(1
2
xxesbi
22
1
2
y
![Page 24: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/24.jpg)
AR(1) Process
The sampling distribution for is given by the sampling distribution for the slope parameter in linear regression!
n/)1( 22
How do we determine if is significantly non-zero?
nt
/21
0
This is just the t test!
11
kky
ky
![Page 25: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/25.jpg)
When Serial Correlation is Present, the variance of the mean must be adjusted,
nxVx
2)var(
121k k
rV
k
k Recall for AR(1) series,
AR(1) Process
Variance inflation factor
This effects the significance of regression/correlation as we saw previously…
Vnn /'
111
121
1121V
11
211
121
V?V
![Page 26: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/26.jpg)
This effects the significance of
regression/correlation as we saw previously…
111
121
1121V
AR(1) Process
12 2
)/exp()lnexp( kkkk
ln/1
Suppose 1 1ln 1/1
Vnn /'
![Page 27: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/27.jpg)
Multiply this equation by xt-k’ and sum,
111
1 )(
tkt
K
kkt xx
Now consider an AR(K) Process
1'11
'1'
tktkt
K
kkkttkt xxxxx
For simplicity, we assume zero mean
111
1
tkt
K
kktxx
11
'1'
kt
K
kkkttkt xxxx
![Page 28: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/28.jpg)
kKKKK
kk
kk
kk
rrrr
rrrrrrrrrrrr
ˆ...ˆˆˆ...
ˆ...ˆˆˆˆ...ˆˆˆˆ...ˆˆˆ
332211
3312213
2132112
1231211
Yule-Walker Equations
Use
varkn
xx
krkii
kn
i
)(1
11
'1'
kt
K
kkkttkt xxxx
AR(K) Process
![Page 29: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/29.jpg)
kk
x
rK
k
11
22
Several results obtained for the AR(1) model generalize readily to the AR(K) model:
kmk
K
km rr
1 nxVx
2)var(
121k k
rV
AR(K) Process
kKKKK
kk
kk
kk
rrrr
rrrrrrrrrrrr
ˆ...ˆˆˆ...
ˆ...ˆˆˆˆ...ˆˆˆˆ...ˆˆˆ
332211
3312213
2132112
1231211
Yule-Walker Equations
![Page 30: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/30.jpg)
AR(K) Process
kKKKK
kk
kk
kk
rrrr
rrrrrrrrrrrr
ˆ...ˆˆˆ...
ˆ...ˆˆˆˆ...ˆˆˆˆ...ˆˆˆ
332211
3312213
2132112
1231211
Yule-Walker Equations
is particularly important because of the range of behavior it can describe w/ a parsimonious number of parameters
11211
ttttxxx The AR(2) model
The Yule-Walker equations give:
2112
1211
ˆˆˆˆ
rr
rrkk
x
rK
k
11
22
![Page 31: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/31.jpg)
AR(2) Process
is particularly important because of the range of behavior it can describe w/ a parsimonious number of parameters
11211
ttttxxx The AR(2) model
The Yule-Walker equations give:
2112
1211
ˆˆˆˆ
rr
rr
Which readily gives:
2
1
2
122
2
1
211
1ˆ
1)1(ˆ
rrr
rrr
)1)(1(
2
2
1
2
2
2
rx
)1/(
)1/(
22
122
211
rr
kk
x
rK
k
11
22
kmk
K
km rr
1
2
![Page 32: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/32.jpg)
AR(2) ProcessWhich readily gives:
2
1
2
122
2
1
211
1ˆ
1)1(ˆ
rrr
rrr
)1)(1(
2
2
1
2
2
2
rx
)1/(
)1/(
22
122
211
rr
0 1 2-1-2
+1
-1 1
2
1ˆˆ1ˆˆ
1ˆ
12
21
2
For stationarity, we must have:
kmk
K
km rr
1
2
![Page 33: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/33.jpg)
AR(2) ProcessWhich readily gives:
)1/(
)1/(
22
122
211
rr
0 1 2-1-2
+1
-1 1
2
1ˆˆ1ˆˆ
1ˆ
12
21
2
For stationarity, we must have:
Note that this model allows for independent lag-1 and lag-2
correlation, so that both positive correlation and negative correlation
are possible...kmk
K
km rr
1
2
![Page 34: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/34.jpg)
AR(2) ProcessWhich readily gives:
)1/(
)1/(
22
122
211
rr
1ˆˆ1ˆˆ
1ˆ
12
21
2
For stationarity, we must have:
Note that this model allows for independent lag-1 and lag-2
correlation, so that both positive correlation and negative correlation
are possible...
0 2 4 6 8 10 12 14 16 18 200
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
4.0ˆ3.0ˆ
2
1
“artwo.m”
kmk
K
km rr
1
2
![Page 35: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/35.jpg)
Selection Rules
AR(K) Process
nmmsmnnnmBIC )ln1()(
1ln)( 2
Bayesian Information Criterion
)1(2)(1
ln)( 2
mmsmnnnmAIC
Akaike Information Criterion
The minima in AIC or BIC represent an ‘optimal’ tradeoff between degrees of freedom and variance explained
![Page 36: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/36.jpg)
ENSO
Multivariate ENSO Index
(“MEI”)
AR(K) Process
![Page 37: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/37.jpg)
varkn
xx
krkii
kn
i
)(1
1 2 3 4 5 6 7 8 9 10 11-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
AR(1) FitAutocorrelation Function
0 50 100 150-2
-1
0
1
2
3
4
Dec-Mar Nino3
)/exp()lnexp( kkk
rk
![Page 38: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/38.jpg)
varkn
xx
krkii
kn
i
)(1
1 2 3 4 5 6 7 8 9 10 11-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
AR(2) FitAutocorrelation Function
0 50 100 150-2
-1
0
1
2
3
4
Dec-Mar Nino3
)1/()1/(
22
122
211
rr kmk
K
km rr
1
(m>2)
![Page 39: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/39.jpg)
varkn
xx
krkii
kn
i
)(1
1 2 3 4 5 6 7 8 9 10 11-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
AR(3) FitAutocorrelation Function
0 50 100 150-2
-1
0
1
2
3
4
Dec-Mar Nino3
Theoretical AR(3) Fit
![Page 40: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/40.jpg)
varkn
xx
krkii
kn
i
)(1
1 2 3 4 5 6 7 8 9 10 11-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
AR(K) FitAutocorrelation Function
0 50 100 150-2
-1
0
1
2
3
4
Dec-Mar Nino3
Favors AR(K) Fit for K=?
Minimum in BIC?
Minimum in AIC?
![Page 41: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/41.jpg)
MA model
1111
11
)(
mt
M
mtkt
K
kkt
mxx
Now, consider the case k=0
1111
mt
M
mttm
x
Pure Moving Average (MA) model, represents a running mean of the past M values.
Consider case where M=1…
![Page 42: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/42.jpg)
MA(1) model
1111
11
)(
mt
M
mtkt
K
kkt
mxx
Now, consider the case k=0
1111
mt
M
mttm
x tt
11
)1( 2
122 x
)1/( 2
111 r
Consider case where M=1…
![Page 43: TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models](https://reader035.vdocument.in/reader035/viewer/2022081502/56815acd550346895dc8a0e6/html5/thumbnails/43.jpg)
ARMA(1,1) model
tttt xx 1111 )(
2
1
1112
1
)221(2
x
11
2
1
11111 21
)()1(
r