Markov Chains - 1

Markov Chains

Chapter 16

Why Study Markov Chains? •  Decision Analysis focuses on decision making in the face of uncertainty

about one future event. However, many decisions need to consider uncertainty about a sequence of future events. –  Uncertain demand for GM SUVs each month over the next year –  Uncertain weather in Napa valley every week over the grape season –  Uncertain daily evolution of stock prices

•  We need probability models for systems that evolve over time in a probabilistic manner – stochastic processes

•  Markov chains are special stochastic processes: –  Probabilities indicating how the process will evolve in the future

depending only on the present state of the process –  They provide the conceptual foundation for Markov Decision

Processes, perhaps the most widely used probabilistic decision models

Decision Analysis-2

Markov Chains - 3

Overview

•  Stochastic process •  Markov chains •  Chapman-Kolmogorov equations •  State classification •  First passage time •  Long-run properties •  Absorption states

Markov Chains - 4

Event vs. Random Variable

•  What is a random variable? –  Recall: a sample space is the set of all possible outcomes of

an experiment –  A random variable takes numerical values depending on the

outcome of the experiment •  Examples of random variables:

–  X = number on a die (integer values) –  X = number of customers purchasing an item (integer values) –  X = inches of rain (could be integer or real-valued) –  X = time until a customer gets served (real-valued)

Markov Chains - 5

Stochastic Processes

•  Suppose now we take a series of observations of that random variable, X0, X1, X2,…

•  A stochastic process is an indexed collection of random variables {Xt}, where t is the index from a given set T. (The index t often denotes time.)

•  Examples: –  Roll a die 10 times, Xi = number on die on ith roll, i=1,2,…,10. Note

that Xi takes integer values from 1 to 6. The stochastic process { Xt } = {X1 ,X2 ,…..} denotes the sequence of rolls.

–  Sales of an item, Xt = number of items sold on day t, t=1,2,…Then the stochastic process { Xt } = {X0 , X1 ,X2 ,…..} provides a mathematical representation of how the sales evolve starting today

Markov Chains - 6

Gambler’s Ruin Example

•  Consider a gambling game where you win $1 with probability p, and lose $1 with probability 1-p on each turn. The game ends when you either accumulate $3 or go broke. You start with $1.

•  Let Xt denote your fortune after t turns of the game. Then the stochastic process {Xt}= {X1, X2,… } describes how your gambling fortune evolves.

•  Questions you might want to answer: –  Should you play? –  Will the game eventually end? –  What is the probability you win $3 or go broke? –  How does everything change with p?

Markov Chains - 7

Space of a Stochastic Process •  The value of Xt is the characteristic of interest •  Xt may be continuous or discrete, but we’ll focus on discrete •  Example: Xt = number of defective items on day t •  This graph is a realization of a stochastic process

•  Here, X4= 2 we say “the state of our stochastic process at

time t=4 is 2”.

Xt 3

2

1

1 0 2 3 4 5 t

Markov Chains - 8

States of a Stochastic Process

•  Note that a stochastic process denotes how the ‘state’ of a system evolves over discrete time points

•  Here, we have discrete states AND discrete time points •  In fact, we’ll consider a finite number of possible states.

We label the states (or values) 0, 1, 2, …, M •  These states will be mutually exclusive and exhaustive

What do those mean? –  Mutually exclusive: States have no intersection – cannot be in

two different states at the same time –  Exhaustive: All possible outcomes are included in the states

Types of Stochastic Processes •  There are several types of stochastic processes depending on how

future values probabilistically depend on present and past values. –  In general, future values may depend on the present value as well as all

the past values (for example, stock prices may depend on past values) –  On the other hand, future values may be completely independent of

present and past values (as in fair coin tossing or fair die rolling). –  In some cases, future values may be independent of past values

and depend only on the present value (as in the gambling example).

•  In INDE 411, we will focus on this last category of stochastic processes (called Markov chains).

•  Hence, our stochastic processes {Xt} are called discrete time finite state Markov chains

Markov Chains - 9

Markov Chains - 10

Weather Example

•  Let Xt be a random variable that takes value 0 if the weather is dry on day t and value 1 if the weather is rainy on day t.

•  Then the stochastic process { Xt }={X0 , X1 ,X2 ,…..} provides a mathematical representation of how the weather evolves starting today (t=0), and the ‘state’ of the system is dry or rainy.

•  Suppose the probability that tomorrow is dry is 0.8 if today is dry, but is 0.6 if it rains today. We write: P(dry tomorrow | dry today) = 0.8 = P(X1=0 | X0=0) P(dry tomorrow | rainy today) = 0.6 = P(X1=0 | X0=1)

•  Or, for any day t, we write: P(Xt+1=0 | Xt=0) = 0.8 and P(Xt+1=0 | Xt=1) = 0.6

Markov Chains - 11

Weather Example, continued

•  Suppose we are given the states of weather on days 0,1,2,3. That is, suppose we know that X0=0 X1=0 X2=1, X3=0 (dry, dry, rainy, dry).

•  What is the probability that X4=0? –  Mathematically, what is P(X4=0 | X3=0, X2=1, X1=0, X0=0) ? –  We have P(X4=0 | X3=0) = 0.8, and, in writing this number we did not care

about the values of X2 X1 X0

–  This observation is true for any values of X3 X2 X1 X0 and in fact for any t.

•  Intuitively, given today’s weather and the weather in the past, the conditional probability of tomorrow’s weather is independent of weather in the past and depends only on today’s weather (this is called the Markovian property).

Markov Chains - 12

Markovian Property

A stochastic process {Xt} satisfies the Markovian property if P(Xt+1=j | X0=k0, X1=k1, … , Xt-1=kt-1, Xt=i) = P(Xt+1=j | Xt=i)

for all t = 0, 1, 2, … and for every possible state, i,j What does this mean?

Future depends only on the present, not on the past

Or, given the current state and the past states, the conditional probability of the next state is independent of past states and depends only on the current state.

Markov Chains - 13

Markov Chain Definition

•  A stochastic process {Xt} for t = 0, 1, 2,… is a Markov chain if it satisfies the Markovian property.

Markov Chains - 14

One-Step Transition Probabilities

•  The conditional probabilities P(Xt+1=j | Xt=i) are called the one-step transition probabilities

•  One-step transition probabilities are stationary if for all t P(Xt+1=j | Xt=i) = P(X1=j | X0=i) = pij

•  Interpretation: the conditional probabilities don’t change over time, they are the same for all t

Xt

j

i

1 0 5 4 t+1 t … …

One-step Transition Probabilities for the Weather Markov Chain

•  The weather chain –  p00 =P(Xt+1 = 0| Xt = 0) = –  p10 =P(Xt+1 = 0| Xt =1) = –  p01 =P(Xt+1 = 1| Xt = 0) = 1-P(Xt+1 = 0| Xt=0) = –  p11 =P(Xt+1 = 1| Xt =1) = 1-P(Xt+1 = 0| Xt =1) =

•  One-step transition matrix: arrange the four one-step transition probabilities in a one-step transition matrix P whose rows and columns correspond to states and entries are pij =P(Xt+1 = j| Xt = i)

Markov Chains - 15

State 0 1 0

1

One-step Transition Probabilities for the Weather Markov Chain

•  The weather chain –  p00 =P(Xt+1 = 0| Xt = 0) = 0.8 –  p10 =P(Xt+1 = 0| Xt =1) = 0.6 –  p01 =P(Xt+1 = 1| Xt = 0) = 1-P(Xt+1 = 0| Xt=0) = 0.2 –  p11 =P(Xt+1 = 1| Xt =1) = 1-P(Xt+1 = 0| Xt =1) = 0.4

•  One-step transition matrix: arrange the four one-step transition probabilities in a one-step transition matrix P whose rows and columns correspond to states and entries are pij =P(Xt+1 = j| Xt = i)

Markov Chains - 16

State 0 1 0 p00 = 0.8 p01 = 0.2

1 p10 = 0.6

p11 = 0.4

Markov Chains - 17

Transition Matrix •  Stationary one-step transition probabilities can be

represented using a one-step transition matrix P, pij = P(Xt+1=j | Xt=i) for i, j ∈ {0,1,…,M}

P =

p00 p01 ... p0Mp10 p11 ... p1M! ! p(M!1)M

pM 0 pM1 ... pMM

"

#

$$$$$

%

&

'''''

Markov Chains - 18

Markov Chain State Transition Diagram

•  A Markov chain with its stationary transition probabilities can also be illustrated using a state transition diagram

•  Weather example:

Weather

Dry 0

Rain

1

0.8 0.4 0.2

0.6

0 1

Dry 0Rain 1

0.8 0.20.6 0.4

!

"#

$

%&

Markov Chains - 19

Weather Example with Variable Probabilities

•  State Transition Diagram

•  Probability Transition Matrix

Dry 0

Rain

1

p 1-q 1-p

q

0 1

Dry 0Rain 1

p 1! pq 1! q

"

#$$

%

&''

Markov Chains - 20

Gambler’s Ruin Stochastic Process

•  Consider again the gambling game with probability p=0.4 of winning on any turn, and you start with $1, stop when you go broke or have $3

•  What are the random variables of interest, Xt? Xt =$fortune on turn t

•  What are the possible values (states) of the random variables? {0,1,2,3}

•  What is the index t? turn of the game

Markov Chains - 21

Gambler’s Ruin as a Markov Chain •  Does the Gambler’s Ruin stochastic process satisfy the

Markovian property? Yes, intuitively, given your current gambling fortune and all past gambling fortunes, the conditional probability of your gambling fortune after one more gamble is independent of your past gambling fortunes and depends only on your current gambling fortune. More formally, P(X5=0 | X4=1, X3=2, X2=1, X1=2, X0=1) = 0.6. In writing this number, you did not care about values of X3 X2 X1 X0

•  Is the Gambler’s Ruin stochastic process stationary?

Yes, intuitively, the probability of winning is the same for all turns of the game. More formally, P(Xt+1=0 | Xt=1) = 0.6 for all t.

Markov Chains - 22

Gambler’s Ruin Markov Chain

•  Suppose the probability of winning on any turn is p=0.4 •  State transition diagram:

•  One-step transition matrix P:

0

1 1

2

3

0.4 0.4

0.6 0.6

1

!

0 1 2 30123

1 0 0 00.6 0 0.4 00 0.6 0 0.40 0 0 1

"

#

$ $ $ $

%

&

' ' ' '

Markov Chains - 23

Gambler’s Ruin Example with Variable Probability

•  Probability p of winning on any turn •  State Transition Diagram

•  Probability Transition Matrix

!

0 1 2 30123

1 0 0 01" p 0 p 0

0 1" p 0 p0 0 0 1

#

$

% % % %

&

'

( ( ( (

0

1 1

2

3

p p

1-p 1-p

1

Markov Chains - 24

Inventory Example

•  A camera store stocks a particular model camera •  Orders may be placed on Saturday night and the

cameras will be delivered first thing Monday morning •  The store uses an (s, S) policy:

–  If the number in inventory is less than s, order enough to bring the supply up to S

–  If the number of cameras in inventory is greater than or equal to s, do not order any cameras

•  The store set the policy with s = 1 and S = 3 –  If zero cameras on hand on Saturday night, order 3 cameras –  If one or more cameras on hand on Saturday night, do not order

any cameras

Markov Chains - 25

Inventory Example

•  What are the random variables of interest, Xt? Xt = number of cameras in inventory on Saturday

night of week t •  What are the possible values (states) of these random

variables? {0,1,2,3}

•  What is the index, t? weeks

Markov Chains - 26

Inventory Example

•  Graph one possible realization of the stochastic process

t

Xt 3

2

1

1 0 2 3 Sat

night X0 = 3

Sat night

X1 = 2

Sat night

X2 = 0

Sat night

X3 = 1

Markov Chains - 27

Inventory Example

•  Describe Xt+1 as a function of Xt, the number of cameras on hand at the end of the tth week, under the (s=1, S=3) inventory policy

•  X0 represents the initial number of cameras on hand •  Let Di represent the demand for cameras during week i •  Assume Di ’s are independent and identically

distributed (iid) random variables Max {3 - Dt+1, 0} if Xt = 0 (Order)

Max {Xt - Dt+1, 0} if Xt ≥ 1 (Don’t order) Xt+1 =

Markov Chains - 28

Inventory Example •  State Transition Diagram

•  Probability Transition Matrix

0

1

2

3

!

P(D " 3)!

P(D " 3)

!

P(D "1)

!

P(D " 2)!

P(D = 2)

!

P(D = 2)

!

P(D = 0)

!

P(D = 0)

!

P(D = 0)

!

P(D = 0)

!

P(D =1)

!

P(D =1)

!

P(D =1)

!

0 1 2 3

0123

P(D " 3) P(D = 2) P(D =1) P(D = 0)P(D "1) P(D = 0) 0 0P(D " 2) P(D =1) P(D = 0) 0P(D " 3) P(D = 2) P(D =1) P(D = 0)

#

$

% % % %

&

'

( ( ( (

Markov Chains - 29

Demand Probabilities with Poisson Distribution

•  Assume Dt ~ Poisson(λ=1) for all t •  Recall, the Poisson pmf is

!

P D = 0( ) =10 e"1

0!= e"1 = 0.368

P D =1( ) =11 e"1

1!= e"1 = 0.368

P D = 2( ) =12 e"1

2!=e"1

2= 0.184

P D #1( ) =1" P D = 0( ) = 0.632

P D # 2( ) =1" P D = 0( )" P D =1( ) = 0.264

P D # 3( ) =1" P D = 0( )" P D =1( )" P D = 2( ) = 0.080

!)(

nenXP n

!

!"

== n = 1, 2,…

Markov Chains - 30

Inventory Example Transition Probabilities

•  Write P, the one-step transition matrix

P

!

0 1 2 3

=

0123

P(D " 3) P(D = 2) P(D = 1) P(D = 0)P(D "1) P(D = 0) 0 0P(D " 2) P(D = 1) P(D = 0) 0P(D " 3) P(D = 2) P(D = 1) P(D = 0)

#

$

% % % %

&

'

( ( ( (

=

0.080 0.184 0.368 0.3680.632 0.368 0 00.264 0.368 0.368 00.080 0.184 0.368 0.368

#

$

% % % %

&

'

( ( ( (