introduction to time series analysis handout 2: stationary...
TRANSCRIPT
Introduction to Time Series Analysis
Handout 2: Stationary Processes. Wold Decomposition and ARMA processes
Laura Mayoral
IAE and BGSEIDEA, Winter 2019
• This lecture introduces the basic linear models for stationary processes.
• Most economic variables are non-stationary.
• However, stationary linear models are used as building blocks in more complicated nonlinear and/or non-stationary models.
Roadmap
§ The Wold decomposition
§ From the Wold decomposition to theARMA representation
§ MA processes and invertibility
§ AR processes, stationarity and causality
§ ARMA, invertibility and causality.
Wold theorem in words:
Any stationary process {Zt} can be expressed as a sum of two components:
- a stochastic component: a linearcombination of lags of a white noise process.
- a deterministic component, uncorrelated with the latter stochastic component.
The Wold Decomposition
If {Zt} is a nondeterministic stationary time series, then:
€
Zt = ψ jj=0
∞
∑ at−j +Vt = Ψ(L )at +Vt ,
where
1. ψ0 = 1 and ψ j2
j=0
∞
∑ <∞,
2. at = Zt - P(Zt | Zt -1,Zt -2 ,...), where P(. | .) denotes linear projection.3. {at} is WN(0,σ 2 ), with σ 2 > 0,3. Cov(as , Vt ) = 0 for all s and t,4. The ψ i 's and the a 's are unique.5. {Vt} is deterministic.
The Wold Decomposition
• This theorem implies that any stationary process can be written as a linear combination of a lagged values of a white noise process (this is the MA(∞) representation).
• By inverting the corresponding polynomial, we can obtain a representation of Zt that depends on past values of the variable and the contemporaneous value of the white noise.
• This is the AR(∞) representation of Zt.
• We will see that the AR representation can be estimated using standard methods: OLS!
• Problem: we might need to estimate a lot of parameters.
• ARMA models: they are an approximation to former representations that tries to be more parsimonious (=less parameters)
Importance of the Wold Decomposition
)L(p
)L(q)L(F
Q»Y
Under general conditions the infinite lag polynomial of the WoldDecomposition can be approximated by the ratio of two finite lag polynomials:
Therefore
€
Zt = Ψ(L)at ≈Θq (L)Φp (L)
at ,
Φp (L)Zt =Θq (L)at
(1−φ1L − ...−φpLp )Zt = (1+ θ1L + ...+ θqL
q )at
Zt −φ1Zt−1 − ...−φpZt− p = at + θ1at−1 + ...+ θqat−q
AR(p) MA(q)
Birth of ARMA(p,q) models
Let
€
at{ } a zero-mean white noise process ),0( 2ata σ→
€
Zt = µ + at +θat−1 →MA(1)
- Expectation
- Variance
Autocovariance€
E(Zt ) = µ + E(at ) +θE(at−1) = µ
€
Var(Zt ) = E(Zt − µ)2 = E(at +θat−1)2 =
= E(at2 +θ 2at−1
2 + 2θatat−1) =σ a2(1+θ 2)
€
1st. orderE(Zt − µ)(Zt−1 − µ) = E(at +θat−1)(at−1 +θat−2) =
= E(atat−1 +θat−12 +θatat−2 +θ 2at−1at−2) = θσ a
2
Moving Average of order 1, MA(1)
-Autocovariance of higher order
- Autocorrelation
€
E(Zt − µ)(Zt− j − µ) = E(at +θat−1)(at− j +θat− j−1) =
= E(atat− j +θat−1at− j +θatat− j−1 +θ 2at−1at− j−1) = 0 j >1
€
ρ1 =γ1γ 0
=θσ 2
(1+ θ 2)σ 2 =θ
1+ θ 2
ρ j = 0 j >1
€
ρ1 =γ1γ 0
=θσ 2
(1+ θ 2)σ 2 =θ
1+ θ 2
ρ j = 0 j >1
StationarityMA(1) process is always covariance-stationary because
22 )1()()( σθµ +== tt ZVarZE
ErgodicityMA(1) process is ergodic for first and second moments because
€
γ j =σ 2(1+θ 2) + θσ 2
j=0
∞
∑ < ∞
If were Gaussian, then would be ergodic for all moments
€
at
€
Zt
MA(1): Stationarity and Ergodicity
MA(q) processes
qtqtttt aaaaZ −−− +++++= θθθµ 2211
A process is MA(q) if it can be written as a linear combination of q lags of a white noise process.
First and Second moments of a MA(q)
€
E(Zt ) = µ
γ 0 = var(Zt ) = (1+ θ12 + θ2
2 ++ θq2)σ a
2
γ j = E(at + θ1at−1 ++ θqat−q )(at− j + θ1at− j−1 ++ θqat− j−q )
γ j =(θ j + θ j+1θ1 + θ j+2θ2 ++ θqθq− j )σ
2 for j ≤ q0 for j > q
' ( )
ρ j =γ j
γ 0
=θ j + θ j+1θ1 + θ j+2θ2 ++ θqθq− j
θi2
i=1
q
∑
Example MA(2)
ρ1 =θ1 + θ1θ2
1+ θ12 + θ2
2 ρ2 =θ2
1+ θ12 + θ2
2 ρ3 = ρ4 = = ρk = 0
- AMA(q) process is said to be invertible if there exists a sequence of constants and
- In other words, Zt is invertible if it admits and autoregressive representation.
€
{π j} such that |π jj= 0
∞
∑ |<∞
€
at = π jj=0
∞
∑ Zt− j, t=0,±1,...
Invertibility: definition
Theorem:Let {Zt} be a MA(q). Then {Zt} is invertible if and only if
The coefficients {pj} are determined by the relation:€
θ(x) ≠ 0 for all x∈C such that | x |≤1.
€
π (x) = π jj=0
∞
∑ x j =1
θ (x), | x |≤1.
Necessary and Sufficient Conditions for Invertibility
MA processes are not uniquely identified
€
1) ρ1 =θ
1+θ 2
2) ρ *1 =1/θ
1+ (1/θ)2 =θ
1+θ 2
Consider the autocorrelation function of these two MA(1) processes:
€
Zt = µ + at +θat−1
€
Z *t = µ + a*t +(1/θ)a*t−1
The autocorrelation functions are:
MA processes are not uniquely identified, II
§ Then, these two processes show identical correlation pattern: The MA coefficient is not uniquely identified.
§ In other words: any MA(1) process has two representations (one with MA parameter larger than 1, and the other, with MA parameter smaller than 1).
MA processes are not uniquely identified, III
•This means that each MA(1) has two representations: one that is invertible, another one that is not.
•We prefer representations that are invertible sowe will choose the representation with q<1.
Z
MA processes are not uniquely identified, IV
•The same problem is present for MA(q) processes.
•In this case, one needs to look at the roots of the MA(q) polynomial: roots smaller than 1 imply non-invertibity.
•There is always an invertible representation, obtained by inverting the root that is smaller than 1.
Z
Exercise
Consider a MA(1) process with a MA coefficient equal to 1.3
1) Is it stationary? Is it ergodic?
2) is it invertible? If it is not, suggest an alternative representation that has identical autocorrelation structure and is invertible
Zt = µ + ψ jat− j ψ0 =1j=0
∞
∑
MA(infinite)
This is the most general MA process.
It contains and infinite number of lags of a white noise process.
€
E(Zt ) = µ, Var(Zt ) =σ a2 ψ i
2
i= 0
∞
∑
γ j = E (Zt −µ)(Zt− j −µ)[ ] =σ 2 ψ iψ i+ ji= 0
∞
∑
ρ j =
ψiψ i+ ji= 0
∞
∑
ψ i2
i= 0
∞
∑
MA(infinite): moments
Notice that in order to define the second order moments we need
The process is covariance-stationaryprovided the former condition holds.€
ψi2
i= 0
∞
∑ <∞
MA(infinite): stationarity condition
Some interesting results
Proposition 1.
Proposition 2.
∞<⇒∞< ∑∑∞
=
∞
= 0
2
0 ii
ii ψψ
(absolutelysummable)
€
ψ ii= 0
∞
∑ <∞⇒ γ ii= 0
∞
∑ <∞
(squaresummable)
Ergodic for second moments
Proof 1. ¥<Þ¥< åå¥
=
¥
= 0
2
0 ii
ii yy
€
If ψ ii= 0
∞
∑ <∞⇒ ∃ N <∞ such that ψ i <1 ∀i ≥ N
ψi2 < ψ i ∀i ≥ N⇒ ψi
2
i=N
∞
∑ < ψ ii=N
∞
∑
Now,
ψi2
i= 0
∞
∑ = ψ i2
i= 0
N−1
∑ + ψ i2
i=N
∞
∑ < ψ i2
i= 0
N−1
∑ + |ψi |i=N
∞
∑(1) (2)
(1) It is finite because N is finite(2) It is finite because is absolutely summableThe picture can't be displayed.
then
Proof 2.
M
MM
jji
i ii
jjii
jji
ii
j j ijiij
ijii
ijiij
ijiij
<
∞<<<=
==≤
≤=
=
∑
∑ ∑∑
∑∑∑ ∑∑
∑∑
∑
∞
=+
∞
=
∞
=
∞
=+
∞
=+
∞
=
∞
=
∞
=
∞
=+
∞
=+
∞
=+
∞
=+
0
22
0 0
2
0
2
0 00 0
2
0
2
0
2
0
2
0
2
assumptionby because ψ
σψσψψσ
ψψσψψσγ
ψψσψψσγ
ψψσγ
€
Zt = c + φZt−1 + at
AR(1) process
An autoregressive process Z is a function of its own past and a contemporaneous value of a white noice sequence
€
Zt = c + φc + φ 2Zt−2 + φat−1 + at =
= c(1+ φ + φ 2 +) + at + φat−1 + φ 2at−2 +geometric progression
)( ∞MA
1 if 1
1)2(
sequence bounded 1
11)1(
1 if
22
00
2
2
<¥<-
==
-=+++
Þ<
åå¥
=
¥
=
ff
fy
fff
f
j
jjj
!
€
ψ 2j
j=0
∞
∑ < ∞ is a sufficient condition for stationarity
AR(1): Stationarity
AR(1) process is stationary if 1<φ
Mean of a stationary AR(1)
€
Zt =c
1−φ+ at + φat−1 + φ 2at−2 +
µ = E(Zt ) =c
1−φ
Variance of a stationary AR(1)
€
γ 0 = 1+ φ 2 + φ 4 +( )σ 2 =1
1−φ 2σ a
2
AR(1): First and second order moments
Autocovariance of a stationary AR(1)
- You need to solve a system of equations:
€
γ j = E Zt −µ( ) Zt− j −µ( )[ ] = E φ Zt−1 −µ( ) + at( ) Zt− j −µ( )[ ] =
= φE Zt−1 −µ( ) Zt− j −µ( ) + at Zt− j −µ( )[ ] = φγ j−1
11 ³= - jjj fgg
Autocorrelation of a stationary AR(1)
€
ρ j =γ j
γ o= φ
γ j−1
γ 0= φρ j−1 j ≥1
ρ j = φ 2ρ j−2 = φ 3ρ j−3 = = φ jρ0 = φ j
PACF: from Yule-Walker equations
€
φ11 = ρ 1̀ = φ
φ22 =
1 ρ1ρ1 ρ21 ρ1ρ1 1
=ρ2 − ρ1
2
1− ρ12 =
φ 2 −φ 2
1− ρ12 = 0
φkk = 0 k ≥ 2
AR(1): Partial autocorrelation function
A stationary AR(1) process is ergodic for first and second moments.
Show this as an exercise.
AR(1): Ergodicity
ttt aZZ += -11f
€
Iterating we obtain
Zt = at + φ1at + ...+ φ1kat -k + φ1Zt -k -1.
If φ1 <1 we showed that
Zt = φ1jat− j
j=0
∞
∑
Consider the AR(1) process,
€
This cannot be done if φ1 ≥1, (no mean - square convergence) However, in this case one could write
Zt = φ1−1Zt+1 −φ1
−1at+1
Then, Zt = − φ1− jat+ j
j=0
∞
∑
and this is a stationary representation of Zt .
Causality and Stationarity
11 <f 11 <f
11 <f
However, this stationary representation depends on future values of
It is customary to restrict attention to AR(1) processes with
Such processes are called stationary but also CAUSAL, or future-indepent AR representations.
11 <f€
at
Causality and Stationarity, II
Definition: An AR(p) process defined by the equation
is said to be causal, or a causal function of {at}, if there exists a sequence of constants
and- A necessary and sufficient condition for causality is
tatZ)L(p =f
€
{ψ j} such that |ψ jj=0
∞
∑ |< ∞
€
Zt = ψ jj=0
∞
∑ at− j, t=0,±1,...
€
φ(x) ≠ 0 for all x∈C such that | x |≤1.
Causality and Stationarity, III
€
Zt = c + φ1Zt−1 + φ2Zt−2 + at
Stationarity Study of the roots of the characteristic equation
€
Let 1/α1 and 1/α2 be the roots of the AR polynomial
such that 1- φ1L −φ2L2 = 1−α1L( ) 1−α2L( ).
Then, φ1 = α1 +α2 and φ2 = −(α1α2)
Zt is stationary iff : α i <1,i = {1,2}.
AR(2)
Mean of AR(2)
€
E(Zt ) = c + φ1E(Zt ) + φ2E(Zt )⇒
E(Zt ) = µ =c
1−φ1 −φ2
Variance
€
γ 0 = E(Zt −µ)2 = φ1E(Zt−1 −µ)(Zt −µ)+ φ2E(Zt−2 −µ) Zt −µ( ) + E(Zt −µ)atγ 0 = φ1γ−1 + φ2γ−2 +σ 2
a
γ 0 = φ1ρ1γ 0 + φ2ρ2γ 0 +σ 2a
γ 0 =σ 2
a
1−φ1ρ1 −φ2ρ2
First and Second order moments
Autocorrelation function
1))(( 2211 ≥+=−−= −−− jZZE jjjttj γφγφµµγ
€
ρ j = φ1ρ j−1 + φ2ρ j−2 j ≥1
Difference equation:
€
ρ j = φ1ρ j−1 + φ2ρ j−2 j ≥1
12213
22
21
2
2
11
02112
12011
31
121
ρφρφρ
φφ
φρ
φφ
ρ
ρφρφρ
ρφρφρ
+==
##$
##%
&
+−
=
−=
→$%&
+==
+==
j
jj
Partial autocorrelations: from Yule-Walker equations
€
φ11 = ρ1 =φ1
1−φ2; φ22 =
ρ2 − ρ12
1− ρ12 ; φ33 = 0
Partial Autocorrelations
tptpttt aZZZcZ ++++= −−− φφφ .......2211
Causality
All p roots of the characteristic equation outside of the unit circle
AR(p) process
Autocorrelation Function
€
ρk = φ1ρk−1 + φ2ρk−2 + ......φpρk− p
ρ1 = φ1ρ0 + φ2ρ1 + ......φpρp−1
ρ2 = φ1ρ11 + φ2ρ0 + ......φpρp−2
ρp = φ1ρp−1 + φ2ρp−2 + ......φpρ0
%
&
' '
(
' '
System of equations.The first pautocorrelations:p unknowns and p equations
ACF decays as mixture of exponentials and/or damped sine waves, Depending on real/complex roots
PACF
€
φkk = 0 for k > p
Second order moments
Relationship between AR(p) and MA(q)
Stationary AR(p)
€
Φp (L)Zt = at Φp (L) = (1−φ1L −φ2L2 − ....φpL
p )1
Φp (L)=Ψ(L)⇒Φp (L)Ψ(L) =1
Zt =1
Φp (L)at =Ψ(L)at Ψ(L) = (1+ψ1L +ψ2L
2 + ....)
€ €
Relationship between AR(p) and MA(q), II
€
€
Zt =Θq (L)at Θq (L) = (1−θ1L −θ2L2 − ....θqL
q )1
Θq (L)=Π(L)⇒Θq (L)Π(L) =1
Π(L)Zt =1
Θq (L)Zt = at Π(L) = (1+π1L +π 2L
2 + ....)
Invertible MA(q)
€
ttp
q
ttq
pt
q
tqtp
aLaLL
aZLL
ZL
xx
xx
aLZL
)()()(
ZtionrepresentaMA Pure
)()(
)( tion representa AR Pure
10)( of roots ty Stationari
10)( of roots ity Invertibil
)()(
t
p
Y=F
Q=®
=Q
F=P®
>=F®
>=Q®
Q=F
ARMA(p,q)
ARMA(1,1)
1)((L)form MApure
1)((L)Zform AR pure
1ityinvertibil
1 ty stationari
)1()1(
1
1t
≥−=Ψ=→
≥−==Π→
<→
<→
−=−
−
−
jaZ
ja
aLZL
jjtt
jjt
tt
φθφψ
θθφπ
θ
φ
θφ
ACF of ARMA(1,1)
€
ZtZt−k = φZt−1Zt−k + atZt−k −θat−1Zt−k
taking expectations
€
γ k = φγ k−1 + E(atZt−k ) −θE(at−1Zt−k )
€
k = 0 E(atZt ) =σ2a E(at−1Zt ) = (φ −θ )σa2
γ 0 = φγ1 +σa2 −θ (φ −θ )σa
2
k =1 γ1 = φγ 0 −θσa2
k ≥ 2 γ k = φγ k−1!"#
10 and for solveunknowns 2 and equations 2 of system
γγ
you get this system of equations
€
ρk =
1 k = 0(φ −θ) 1−φθ( )1+θ 2 − 2φθ
k =1
φρk−1 k ≥ 2
'
( ) )
* ) )
PACF
decay lexponentia)1,1()1( ARMAMA ⊂
ACF