1

Markov Chains

2

Description

Sometimes we are interested in how a random variable changes over time.

The study of how a random variable evolves over time includes Stochastic Processes.

3

What is a Stochastic Process?• Suppose we observe some characteristic of a system

at discrete points in time.

• Let Xt be the value of the system characteristic at time t. In most situations, Xt is not known with certainty before time t and may be viewed as a random variable.

• A discrete-time stochastic process is simply a description of the relation between the random variables X0, X1, X2 …..

4

• A continuous –time stochastic process is simply the stochastic process in which the state of the system can be viewed at any time, not just at discrete instants in time.

• For example, the number of people in a supermarket t minutes after the store opens for business may be viewed as a continuous-time stochastic process.

5

The Gambler’s Ruin Problem

At time 0, I have Rs. 2. At times 1, 2, …, I play a game in which I bet Rs. 1, with probabilities p, I win the game, and with probability 1 – p, I lose the game. My goal is to increase my capital to Rs. 4, and as soon as I do, the game is over. The game is also over if my capital is reduced to 0.

–Let Xt represent my capital position after the time t game (if any) is played–X0, X1, X2, …. May be viewed as a discrete-time stochastic process

6

What is a Markov Chain?

• One special type of discrete-time stochastic process is called a Markov Chain.

• Definition: A discrete-time stochastic process is a Markov chain if, for t = 0,1,2… and all statesP(Xt+1 = it+1|Xt = it, Xt-1=it-1,…,X1=i1, X0=i0)

=P(Xt+1=it+1|Xt = it)

• Essentially this says that the probability distribution of the state at time t+1 depends on the state at time t(it) and does not depend on the states the chain passed through on the way to it at time t.

7

• In our study of Markov chains, we make further

assumption that for all states i and j and all t,

P(Xt+1 = j|Xt = i) is independent of t.

• This assumption allows us to write P(Xt+1 = j|Xt = i) = pij where pij is the probability that given

the system is in state i at time t, it will be in a state j at time t+1.

• If the system moves from state i during one period to state j during the next period, we that a transition from i to j has occurred.

8

• The pij’s are often referred to as the transition

probabilities for the Markov chain.• This equation implies that the probability law relating

the next period’s state to the current state does not change over time.

• It is often called the Stationary Assumption and any Markov chain that satisfies it is called a stationary Markov chain.

• We also must define qi to be the probability that the chain is in state i at the time 0; in other words, P(X0=i) = qi.

9

• We call the vector q= [q1, q2,…qs] the initial probability distribution for the Markov chain.

• In most applications, the transition probabilities are displayed as an s x s transition probability matrix P. The transition probability matrix P may be written as

ssss

s

ppp

ppp

ppp

P

21

111121

11211

10

TPM: The Gambler’s Ruin Problem

Rs. 0 Rs. 1 Rs. 2 Rs. 3 Rs. 4

P =

10000

0100

0010

0001

00001

pp

pp

pp

11

• For each i

• We also know that each entry in the P matrix must be nonnegative.

• Hence, all entries in the transition probability matrix are nonnegative, and the entries in each row must sum to 1.

sj

jijp

1

1

12

Question

A company has two machines. During any day, each machine that is working at the beginning of the day has a 1/3 chance of breaking down. If a machine breaks down during the day it is sent to a repair facility and will be working two days after it breaks down. (Thus, if a machine breaks down during day-3, it will be working at the beginning of day 5)

Letting the state of the system to be the number of machines working at the beginning of the day, formulate a transition probability matrix for this situation.

13

n-Step Transition Probabilities

• A question of interest when studying a Markov chain is: If a Markov chain is in a state i at time m, what is the probability that n periods later than the Markov chain will be in state j?

• This probability will be independent of m, so we may write

P(Xm+n =j|Xm = i) = P(Xn =j|X0 = i) = Pij(n)where Pij(n) is called the n-step probability of a transition from state i to state j.

• For n > 1, Pij(n) = ijth element of Pn

14

The Cola Example

• Suppose the entire cola industry produces only two colas.

• Given that a person last purchased cola 1, there is a 90% chance that her next purchase will be cola 1.

• Given that a person last purchased cola 2, there is an 80% chance that her next purchase will be cola 2.

1. If a person is currently a cola 2 purchaser, what is the probability that she will purchase cola 1 two purchases from now?

2. If a person is currently a cola 1 purchaser, what is the probability that she will purchase cola 1 three purchases from now?

15

The Cola Example• We view each person’s purchases as a Markov chain

with the state at any given time being the type of cola the person last purchased.

• Hence, each person’s cola purchases may be represented by a two-state Markov chain, where– State 1 = person has last purchased cola 1

– State 2 = person has last purchased cola 2

• If we define Xn to be the type of cola purchased by a person on her nth future cola purchase, then X0, X1, … may be described as the Markov chain with the following transition matrix:

16

The Cola Example

We can now answer questions 1 and 2.

1. We seek P(X2 = 1|X0 = 2) = P21(2) = element 21 of P2:

1 2

1 0.9 0.1

2 0.2 0.8

cola cola

colaP

cola

66.34.

17.83.

80.20.

10.90.

80.20.

10.90.2P

17

The Cola Example

– Hence, P21(2) =.34. This means that the probability is .34 that two purchases in the future a cola 2 drinker will purchase cola 1.

– By using basic probability theory, we may obtain this answer in a different way.

2. We seek P11(3) = element 11 of P3:

Therefore, P11(3) = .781

562.438.

219.781.

66.34.

17.83.

80.20.

10.90.)( 23 PPP

18

• Many times we do not know the state of the Markov chain at time 0. Then we can determine the probability that the system is in state i at time n by using the reasoning.

• Probability of being in state j at time n

where q=[q1, q2, … q3].

(This is an unconditional probability).

)(1

nPq ij

si

ii

19

Limiting probabilities

• To illustrate the behavior of the n-step transition probabilities for large values of n, we have computed several of the n-step transition probabilities for the Cola example.

• This means that for large n, no matter what the initial state, there is a .67 chance that a person will be a cola 1 purchaser.

• We can easily multiply matrices on a spreadsheet using the MMULT command.

20

Question

A B

A .7 .3

B .5 .5

Find the equilibrium market shares of two firms whose probability transition matrix is as follows

21

Example• Suppose we have a Markov transition matrix:

• 1 2 3

1 0 1/ 2 1/ 2

2 1/ 2 1/ 2 0

3 0 1 0

P

12 3

1/2

1/2

1/2

1/2

1

22

All states communicate with each other

Starting from 1, the MC can return to 1 in three steps via two possible routes:

Route 1: 1 to 3 to 2 to 1 with probability .5×1 ×.5 = 1/4Route 2: 1 to 2 to 2 to 1 with probability .5 ×.5×.5 = 1/8 Hence the required probability is =1/4+1/8 = 3/8 .

23

Steady-State Probabilities

• Steady-state probabilities are used to describe the long-run behavior of a Markov chain.

• Theorem 1: Let P be the transition matrix for an s-state ergodic chain. Then there exists a vector π = [π1 π2 … πs] such that

s

s

s

n

nP

21

21

21

lim

24

• Theorem 1 tells us that for any initial state i,

• The vector π = [π1 π2 … πs] is often called the steady-state distribution, or equilibrium distribution, for the Markov chain.

jijn

nP

)(lim

25

An Example• A supermarket stocks 3 brands of coffee, A, B, and

C, and it has been observed that customers switch from brand to brand according to the following transition matrix:

1 2 3

1 3/4 1/4 0

2 0 2/3 1/3

3 1/4 1/4 1/2

P

In the long

In the long run, what fraction of the customers purchase the respective brands?

26

Solution

• Since the chain is ergodic (all states are communicating, each state is recurrent and aperiodic), the steady-state distribution exists.

• Solving = P gives

1=(3/4) 1+(1/4) 3

2=(1/4) 1+(2/3) 2+(1/4) 3

3=(1/3) 2+(1/2) 3

Subject to 1 + 2+ 3=1. Solving the equations gives

1=(2/7), 2=(3/7), 3=2/7.

27

Inventory Example• A camera store stocks a particular model camera that can be ordered

weekly. Let D1, D2, … represent the demand for this camera (the number of units that would be sold if the inventory is not depleted) during the first week, second week, …, respectively. It is assumed that the Di’s are independent and identically distributed random variables having a Poisson distribution with a mean of 1. Let X0 represent the number of cameras on hand at the outset, X1 the number of cameras on hand at the end of week 1, X2 the number of cameras on hand at the end of week 2, and so on. – Assume that X0 = 3. – On Saturday night the store places an order that is delivered in time for the

next opening of the store on Monday. – The store using the following order policy: If there are no cameras in stock,

3 cameras are ordered. Otherwise, no order is placed. – Sales are lost when demand exceeds the inventory on hand

28

Inventory Example• Xt is the number of Cameras in stock at the end of week t (as defined

earlier), where Xt represents the state of the system at time t• Given that Xt = i, Xt+1 depends only on Dt+1 and Xt (Markovian property)• Dt has a Poisson distribution with mean equal to one. This means that

P(Dt+1 = n) = e-11n/n! for n = 0, 1, …• P(Dt = 0 ) = e-1 = 0.368• P(Dt = 1 ) = e-1 = 0.368• P(Dt = 2 ) = (1/2)e-1 = 0.184• P(Dt 3 ) = 1 – P(Dt 2) = 1 – (.368 + .368 + .184) = 0.08• Xt+1 = max(3-Dt+1, 0) if Xt = 0 and Xt+1 = max(Xt – Dt+1, 0) if Xt 1, for t = 0, 1, 2, ….

29

Inventory Example: (One-Step) Transition Matrix• P03 = P(Dt+1 = 0) = 0.368

• P02 = P(Dt+1 = 1) = 0.368

• P01 = P(Dt+1 = 2) = 0.184

• P00 = P(Dt+1 3) = 0.080

44434241

34333231

23221211

04030201

3

2

1

0

3210

pppp

pppp

pppp

pppp

30

Inventory Example: Transition Diagram

0 1

2 3

31

Inventory Example: (One-Step) Transition Matrix

368.368.184.080.3

0368.368.264.2

00368.632.1

368.368.184.080.0

3210

32

Transition Matrix: Two-Step

• P(2) = PP

165.300.286.249.3

097.233.319.351.2

233.233.252.283.1

165.300.286.249.0

3210

)2( P

33

Transition Matrix: Four-Step

• P(4) = P(2)P(2)

164.261.286.289.3

171.263.283.284.2

166.268.285.282.1

164.261.286.289.0

3210

)4( P

34

Transition Matrix: Eight-Step

• P(8) = P(4)P(4)

166.264.285.286.3

166.264.285.286.2

166.264.285.286.1

166.264.285.286.0

3210

)8( P

35

Steady-State Probabilities

• The steady-state probabilities uniquely satisfy the following steady-state equations

0 = 0p00 + 1p10 + 2p20 + 3p30

1 = 0p01 + 1p11 + 2p21 + 3p31

2 = 0p02 + 1p12 + 2p22 + 3p32

3 = 0p03 + 1p13 + 2p23 + 3p33

• 1 = 0 + 1 + 2 + 3

1

......s 2, 1, 0, jfor

0

0

s

jj

s

iijij p

36

Steady-State Probabilities: Inventory Example

0 = .0800 + .6321 + .2642+ .0803

1 = .1840 + .3681 + .3682 + .1843

2 = .3680 + .3682 + .3683

3 = .3680 + .3683

• 1 = 0 + 1 + 2 + 3

0 = .286, 1 = .285, 2 = .263, 3 = .166• The numbers in each row of matrix P(8) match the

corresponding steady-state probability

37

Mean First Passage Times

• For an ergodic chain, let mij = expected number of transitions before we first reach state j, given that we are currently in state i; mij is called the mean first passage time from state i to state j.

• In the example, we assume we are currently in state i. Then with probability pij, it will take one transition to go from state i to state j. For k ≠ j, we next go with probability pik to state k. In this case, it will take an average of 1 + mkj transitions to go from i and j.

38

• This reasoning implies

• By solving the linear equations of the equation above, we find all the mean first passage times. It can be shown that

jk

kjikij mpm 1

iiim

1

39

• For the cola example, π1=2/3 and π2 = 1/3

• Hence, m11 = 1.5 and m22 = 3

• m12 = 1 + p11m12 = 1 + .9m12

m21 = 1 + p22m21 = 1 + .8m21

Solving these two equations yields,

• m12 = 10 and m21 = 5