Markov Chain

• Stochastic process (discrete time):

{X1, X2, ..., }

• Markov chain

– Consider a discrete time stochastic process withdiscrete space.Xn ∈ {0, 1, 2, ...}.

– Markovian property

P{Xn+1 = j | Xn = i,Xn−1 = in−1, ..., X0 = i0}= P{Xn+1 = j | Xn = i} = Pi,j

– Pi,j is thetransition probability: the probability ofmaking a transition fromi to j.

– Transition probability matrix

P =

P0,0 P0,1 P0,2 · · ·P1,0 P1,1 P1,2 · · ·

... ... ... ...Pi,0 Pi,1 Pi,2 · · ·

... ... ... ...

1

Example

• Suppose whether it will rain tomorrow depends onpast weather condition ony through whether it rainstoday. Consider the stochastic process{Xn, n = 1, 2, ...}

Xn =

{

0 rain on dayn1 not rain on dayn

P (Xn+1|Xn, Xn−1, ..., X1) = P (Xn+1 | Xn)

• State space{0, 1}.

• Transition matrix:(

P0,0 P0,1

P1,0 P1,1

)

• P0,0 = P (tomorrow rain|today rain) = α. ThenP0,1 =1 − α.

• P1,0 = P (tomorrow rain|today not rain) = β. ThenP1,1 = 1 − β.

• Transition matrix:(

α 1 − αβ 1 − β

)

2

Transforming into a Markov Chain

• Suppose whether it will rain tomorrow depends onwhether it rained today and yesterday.

• P (Xn+1|Xn, Xn−1, ..., X1) = P (Xn+1|Xn, Xn−1). Theprocess is not a first order Markov chain.

• DefineYn:

Yn =

0 Xn = 0, Xn−1 = 0 RR1 Xn = 0, Xn−1 = 1 NR2 Xn = 1, Xn−1 = 0 RN3 Xn = 1, Xn−1 = 1 NN

•P (Yn+1|Yn, Yn−1, ...)

= P (Xn+1, Xn|Xn, Xn−1, ...)

= P (Xn+1, Xn|Xn, Xn−1)

= P (Yn+1|Yn)

• {Yn, n = 1, 2, ...} is a Markov chain.

P0,0 P0,1 P0,2 P0,3

P1,0 P1,1 P1,2 P1,3

P2,0 P2,1 P2,2 P2,3

P3,0 P3,1 P3,2 P3,3

3

• P0,1 = P (Yn+1 = 1|Yn = 0) = P (Xn+1 = 0, Xn =1|Xn = 0, Xn−1 = 0) = 0.

• P0,3 = P (Yn+1 = 3|Yn = 0) = P (Xn+1 = 1, Xn =1|Xn = 0, Xn−1 = 0) = 0.

• Similarly, P1,1 = P1,3 = 0, P2,0 = P2,2 = 0, P3,0 =P3,2 = 0.

• Transition matrix

P0,0 0 P0,2 0P1,0 0 P1,2 00 P2,1 0 P2,3

0 P3,1 0 P3,3

=

P0,0 0 1 − P0,0 0P1,0 0 1 − P1,0 00 P2,1 0 1 − P2,1

0 P3,1 0 1 − P3,1

• The Markov chain is specified byP0,0, P1,0, P2,1, P3,1.

1. P0,0 = P (tomorrow will rain|today rain, yesterday rain).

2. P1,0 = P (tomorrow will rain|today rain, yesterday not rain).

3. P2,1 = P (tomorrow will rain|today not rain, yesterday rain).

4. P3,1 = P (tomorrow will rain|today not rain, yesterday not rain).

4

Chapman-Kolmogorov Equations

• Transition afternth steps:P n

i,j = P (Xn+m = j | Xm = i).

• Chapman-Kolmogorov Equations:

P n+mi,j =

∞∑

k=0

P ni,kP

mk,j, n, m ≥ 0 for all i, j.

• Proof (by Total probability formula):

P n+mi,j = P (Xn+m = j|X0 = i)

=∞

∑

k=0

P (Xn+m = j,Xn = k|X0 = i)

=∞

∑

k=0

P (Xn = k|X0 = i) ·

P (Xn+m = j|Xn = k,X0 = i)

=∞

∑

k=0

P ni,kP

mk,j

5

• n-step transition matrix:

P(n) =

P n0,0 P n

0,1 P n0,2 · · ·

P n1,0 P n

1,1 P n1,2 · · ·

... ... ... ...P n

i,0 P ni,1 P n

i,2 · · ·... ... ... ...

• Chapman-Kolmogorov Equations:

P(n+m) = P

(n) · P(m), P(n) = P

n.

• Weather example:

P =

(

α 1 − αβ 1 − β

)

=

(

0.7 0.30.4 0.6

)

FindP (rain on Tuesday| rain on Sunday) andP (rain on Tuesday and rain on Wednesday| rain on Sunday).

Solution:

P (rain on Tuesday| rain on Sunday) = P 20,0

P(2) = P · P =

(

0.7 0.30.4 0.6

)

×(

0.7 0.30.4 0.6

)

=

(

0.61 0.390.52 0.48

)

P (rain on Tuesday| rain on Sunday) = 0.61

6

P (rain on Tuesday and rain on Wednesday| rain on Sunday)

= P (Xn = 0, Xn+1 = 0 | Xn−2 = 0)

= P (Xn = 0|Xn−2 = 0)P (Xn+1 = 0|Xn = 0, Xn−2 = 0)

= P (Xn = 0|Xn−2 = 0)P (Xn+1 = 0|Xn = 0)

= P 20,0P0,0

= 0.61 × 0.7 = 0.427

7

Classification of States

• Accessible: Statej is accessible from statei if P ni,j >

0 for somen ≥ 0.

– i → j.

– Equivalent to:P (ever enterj|start ini) > 0.

P (ever enterj|start ini)

= P (∪∞n=0{Xn = j}|X0 = i)

≤∞

∑

n=0

P (Xn = j | X0 = i)

=∞

∑

n=0

P ni,j

Hence ifP ni,j = 0 for all n, P (ever enterj|start ini) =

0. On the other hand,

P (ever enterj|start ini)

= P (∪∞n=0{Xn = j}|X0 = i)

≥ P ({Xn = j}|X0 = i) for anyn

= P ni,j .

If P ni,j > 0 for somen, P (ever enterj|start ini) ≥

P ni,j > 0.

– Examples

8

• Communicate: Statei andj communicate if they areaccessible from each other.

– i ↔ j.

– Properties:

1. P 0i,i = P (X0 = i|X0 = i) = 1. Any statei

communicates with itself.2. If i ↔ j, thenj ↔ i.3. If i ↔ j andj ↔ k, theni ↔ k.

Proof:

i ↔ j =⇒ P ni,j > 0 andP n′

j,i > 0

j ↔ k =⇒ Pmj,k > 0 andPm′

k,j > 0

P n+mi,k =

∞∑

l=0

P ni,lP

ml,k Chapman-Kolmogorov Eq.

> P ni,j · Pm

j,k

> 0

Similarly, we can showP n′+m′k,i > 0. Hencei ↔

k.

9

• Class: Two states that communciate are said to bein the same class. A class is a subset of states thatcommunicate with each other.

– Different classes do NOT overlap.

– Classes form a partition of states.

• Irreducible: A Markov chain is irreducible if there isonly one class.

– Consider the Markov chain with transition proba-bility matrix:

P =

12

12 0

12

14

14

0 13

23

The MC is irreducible.

– MC with transition probability matrix:

P =

12

12

0 012

12 0 0

14

14

14

14

0 0 0 1

Three classes:{0, 1}, {2}, {3}.

10

Recurrent and Transient States

• fi: probability that starting in statei, the MC will everreenter statei.

• Recurrent: If fi = 1, statei is recurrent.

– A recurrent states will be visited infinitely manytimes by the process starting fromi.

• Transient: If fi < 1, statei is transient.

– Starting fromi, the MC will be in statei for ex-actlyn times (including the starting state) is

fn−1i (1 − fi) , n = 1, 2, ...

This is a geometric distribution with parameter1−fi. The expected number of times spent in stateiis 1/(1 − fi).

11

• A state is recurrentif and only if the expected numberof time periods that the process is in statei, startingfrom statei, is infinite.

Recurrent⇐⇒ E(number of visits toi|X0 = i) = ∞Transient⇐⇒ E(number of visits toi|X0 = i) < ∞

• ComputeE(number of visits toi|X0 = i). Define

In =

{

1 if Xn = i0 if Xn 6= i

Then the number of visits toi is∑∞

n=0 In.

E(number of visits toi|X0 = i)

=∞

∑

n=0

E(In|X0 = i)

=

∞∑

n=0

P (In = 1|X0 = i)

=∞

∑

n=0

P (Xn = i|X0 = i)

=∞

∑

n=0

P ni,i

12

• Proposition 4.1: Statei is recurrent if∑∞

n=0 P ni,i =

∞, and transient if∑∞

n=0 P ni,i < ∞.

• Corollary 4.2: If statei is recurrent and statei com-municates with statej, then statej is recurrent.

• Corollary 4.3: A finite state Markov chain cannothave all transient states.

– For any irreducible and finite-state Markov chain,all states are recurrent.

13

• Consider a MC with

P =

0 0 12

12

1 0 0 00 1 0 00 1 0 0

The MC is irreducible and finite state, hence all statesare recurrent.

• Consider a MC with

P =

12

12 0 0 0

12

12

0 0 00 0 1

212 0

0 0 12

12

014

14 0 0 1

2

Three classes:{0, 1}, {2, 3}, {4}. State0, 1, 2, 3 arerecurrent and state4 is transient.

14

Random Walk

• A Markov chain with state spacei = 0,±1,±2, ....

• Transition probability:Pi,i+1 = p = 1 − Pi,i−1.

– At every step, move either 1 step forward or 1 stepbackward.

• Example: a gambler either wins a dollar or loses adollar at every game.Xn is the number of dollars hehas when starting thenth game.

• For anyi < j, P j−ii,j = pj−i > 0, P j−i

j,i = (1 − p)j−i >0. The MC is irreducible.

• Hence, either all the states are transient or all thestates are recurrent.

15

• Under which condition are the states transient or re-current?

– Consider State0.∞

∑

n=1

P n0,0 =

{

∞ recurrentfinite transient

– Only for evenm, Pm0,0 > 0.

P 2n0,0 =

(

2nn

)

pn(1 − p)n =(2n)!

n!n!(p(1 − p))n

n = 1, 2, 3, ...

– By Stirling’s approximation

n! ∼ nn+1/2e−n√

2π

– P 2n0,0 ∼ (4p(1−p))n√

πn.

– Whenp = 1/2, 4p(1 − p) = 1.∞

∑

n=0

(4p(1 − p))n√πn

=∞

∑

n=0

1√πn

,

The summation diverges. Hence, all the states arerecurrent.

– Whenp 6= 1/2, 4p(1 − p) < 1.∑∞

n=0(4p(1−p))n√

πn

converges. All the states are transient.

16

Limiting Probabilities

• Weather example

P =

(

0.7 0.30.4 0.6

)

•P

(4) = P4 =

(

0.5749 0.42510.5668 0.4332

)

P(8) = P

(4)P

(4) =

(

0.572 0.4280.570 0.430

)

• P(4) andP

(8) are close. The rows inP(8) are close.

• Limiting probabilities?

17

• Period d: For statei, if P ni,i = 0 whenevern is not

divisible by d and d is the largest integer with thisproperty,d is the period of statei.

– Periodd is the greatest common divisor of all them such thatPm

i,i > 0.

• Aperiodic: Statei is aperiodic if its period is1.

• Positive recurrent: If a statei is recurrent and the ex-pected time until the process returns to statei is finite.

– If i ↔ j andi is positive recurrent, thenj is posi-tive recurrent.

– For a finite-state MC, a recurrent state is also pos-itive recurrent.

– A finite-state irreducible MC contains all positiverecurrent states.

• Ergodic: A positive recurrent and aperiodic state isan ergodic state.

• A Markov chain is ergodic if all its states are ergodic.

18

• Theorem 4.1: For an irreducible ergodic Markov chain,limn→∞ P n

i,j exists and is independent ofi. Let πj =limn→∞ P n

i,j, j ≥ 0, thenπj is the unique nonnegativesolution of

πj =∞

∑

i=0

πiPi,j j ≥ 0

∞∑

j=0

πj = 1 .

• The Weather Example:

P =

(

α 1 − αβ 1 − β

)

The MC is irreducible and ergodic.

π0 + π1 = 1

π0 = π0P0,0 + π1P1,0 = π0α + π1β

π1 = π0P0,1 + π1P1,1 = π0(1 − α) + π1(1 − β)

Solve the linear equations,π0 = β1+β−α

, π1 = 1−α1+β−α

.

19

Gambler’s Ruin Problem

• At each play, a gambler either wins a unit with prob-ability p or loses a unite with probabilityq = 1 − p.Suppose the gambler starts withi units, what is theprobability that the gambler’s fortune will reachNbefore reaching0 (broke)?

• Solution:

– Let Xt be the number of units the gambler has attime t. {Xt; t = 0, 1, 2, ...} is a Markov chain.

– Transition probabilities:

P0,0 = PN,N = 1

Pi,i+1 = p, Pi,i−1 = 1 − p = q .

– Denote the probability that starting fromi units,the gambler’s fortune will reachN before reaching0 by Pi, i = 0, 1, ..., N .

– Condition on the result of the first game and applythe total probability formula:

Pi = pPi+1 + qPi−1 .

20

– Changing forms:

(p + q)Pi = pPi+1 + qPi−1

q(Pi − Pi−1) = p(Pi+1 − Pi)

Pi+1 − Pi =q

p(Pi − Pi−1)

– Recursion:

P0 = 0

P2 − P1 =q

p(P1 − P0) =

q

pP1

P3 − P2 =q

p(P2 − P1) =

(

q

p

)2

P1

...

Pi − Pi−1 =q

p(Pi−1 − Pi−2) =

(

q

p

)i−1

P1

– Add up the equations:

Pi = P1[1 +q

p+ · · · + (

q

p)i−1]

=

P1 ·1−(q

p)i

1−qp

if qp 6= 1

iP1 if qp = 1

– We knowPN = 1

PN =

P1 ·1−(q

p)N

1−qp

if qp 6= 1

NP1 if qp

= 121

– Hence,

P1 =

{

1−qp

1−(qp)N

if p 6= 1/2

1/N if p = 1/2

– In summary,

Pi =

1−(qp)i

1−(qp)N

if p 6= 1/2

i/N if p = 1/2

• Note, if N → ∞,

Pi =

{

1 − (qp)

i if p > 1/2

0 if p ≤ 1/2

• Whenp > 1/2, there is a positive probability that thegambler will win infinitely many units.

• When p ≤ 1/2, the gambler will surely go broke(with probability 1) if not stop at a finite fortune (as-suming the opponent is infinitely rich).

22