slides prepared by john s. loucks st. edward’s university

1 1© 2003 Thomson© 2003 Thomson/South-Western/South-Western Slide Slide

Slides Prepared bySlides Prepared by

JOHN S. LOUCKSJOHN S. LOUCKS

St. Edward’s UniversitySt. Edward’s University


Chapter 17 Chapter 17 Markov ProcessMarkov Process

Transition ProbabilitiesTransition Probabilities Steady-State ProbabilitiesSteady-State Probabilities Absorbing StatesAbsorbing States Transition Matrix with SubmatricesTransition Matrix with Submatrices Fundamental MatrixFundamental Matrix


Markov ProcessesMarkov Processes

Markov process modelsMarkov process models are useful in studying are useful in studying the evolution of systems over repeated trials the evolution of systems over repeated trials or sequential time periods or stages.or sequential time periods or stages.

They have been used to describe the They have been used to describe the probability that:probability that:

•a machine that is functioning in one period a machine that is functioning in one period will continue to function or break down in will continue to function or break down in the next period.the next period.

•A consumer purchasing brand A in one A consumer purchasing brand A in one period will purchase brand B in the next period will purchase brand B in the next period.period.


Transition ProbabilitiesTransition Probabilities

Transition probabilitiesTransition probabilities govern the manner in govern the manner in which the state of the system changes from which the state of the system changes from one stage to the next. These are often one stage to the next. These are often represented in a represented in a transition matrixtransition matrix. .


Transition ProbabilitiesTransition Probabilities

A system has a A system has a finite Markov chainfinite Markov chain with with stationary transition probabilitiesstationary transition probabilities if: if:

•there are a finite number of states,there are a finite number of states,

•the transition probabilities remain constant the transition probabilities remain constant from stage to stage, andfrom stage to stage, and

•the probability of the process being in a the probability of the process being in a particular state at stage particular state at stage n+n+1 is completely 1 is completely determined by the state of the process at determined by the state of the process at stage stage nn (and not the state at stage (and not the state at stage n-n-1). This 1). This is referred to as the is referred to as the memory-less propertymemory-less property..


Steady-State ProbabilitiesSteady-State Probabilities

The The state probabilitiesstate probabilities at any stage of the at any stage of the process can be recursively calculated by process can be recursively calculated by multiplying the initial state probabilities by the multiplying the initial state probabilities by the state of the process at stage state of the process at stage nn..

The probability of the system being in a The probability of the system being in a particular state after a large number of stages particular state after a large number of stages is called a is called a steady-state probabilitysteady-state probability. .



Steady state probabilitiesSteady state probabilities can be found by can be found by solving the system of equations solving the system of equations PP = = together together with the condition for probabilities that with the condition for probabilities that ii = 1. = 1.

•Matrix Matrix PP is the transition probability matrix is the transition probability matrix

•Vector Vector is the vector of steady state is the vector of steady state probabilities.probabilities.


Absorbing StatesAbsorbing States

An An absorbing stateabsorbing state is one in which the is one in which the probability that the process remains in that state probability that the process remains in that state once it enters the state is 1.once it enters the state is 1.

If there is more than one absorbing state, then a If there is more than one absorbing state, then a steady-state condition independent of initial steady-state condition independent of initial state conditions does not exist.state conditions does not exist.


Transition Matrix with SubmatricesTransition Matrix with Submatrices

If a Markov chain has both absorbing and If a Markov chain has both absorbing and nonabsorbing states, the states may be nonabsorbing states, the states may be rearranged so that the transition matrix can be rearranged so that the transition matrix can be written as the following composition of four written as the following composition of four submatrices: submatrices: II,, 0 0, , RR, and , and QQ: :

II 00

RR QQ


Transition Matrix with SubmatricesTransition Matrix with Submatrices

II = an identity matrix indicating one always = an identity matrix indicating one always remains remains in an absorbing state once it is in an absorbing state once it is reachedreached

00 = a zero matrix representing 0 probability of = a zero matrix representing 0 probability of

transitioning from the absorbing states to transitioning from the absorbing states to the the

nonabsorbing statesnonabsorbing states

RR = the transition probabilities from the = the transition probabilities from the nonabsorbing states to the nonabsorbing states to the

absorbing statesabsorbing states

QQ = the transition probabilities between the = the transition probabilities between the nonabsorbing states nonabsorbing states


Fundamental MatrixFundamental Matrix

The The fundamental matrixfundamental matrix, , NN, is the inverse of , is the inverse of the difference between the identity matrix and the difference between the identity matrix and the the QQ matrix: matrix:

NN = ( = (II - - Q Q ))-1-1


NR MatrixNR Matrix

The The NRNR matrix matrix is the product of the fundamental is the product of the fundamental ((NN) matrix and the ) matrix and the R R matrix. matrix.

It gives the probabilities of eventually moving It gives the probabilities of eventually moving from each nonabsorbing state to each absorbing from each nonabsorbing state to each absorbing state. state.

Multiplying any vector of initial nonabsorbing Multiplying any vector of initial nonabsorbing state probabilities by state probabilities by NRNR gives the vector of gives the vector of probabilities for the process eventually reaching probabilities for the process eventually reaching each of the absorbing states. Such each of the absorbing states. Such computations enable economic analyses of computations enable economic analyses of systems and policies.systems and policies.


Example: North’s HardwareExample: North’s Hardware

Henry, a persistent salesman, calls Henry, a persistent salesman, calls North's Hardware Store once a week hoping to North's Hardware Store once a week hoping to speak with the store's buying agent, Shirley. If speak with the store's buying agent, Shirley. If Shirley does not accept Henry's call this week, Shirley does not accept Henry's call this week, the probability she will do the same next week the probability she will do the same next week is .35. On the other hand, if she accepts is .35. On the other hand, if she accepts Henry's call this week, the probability she Henry's call this week, the probability she will not do so next week is .20. will not do so next week is .20.



Transition MatrixTransition Matrix

Next Week's Next Week's CallCall

Refuses AcceptsRefuses Accepts

ThisThis Refuses .35 Refuses .35 .65 .65 Week'sWeek's CallCall Accepts .20 Accepts .20 .80 .80




QuestionQuestion

How many times per year can Henry expect How many times per year can Henry expect to talk to Shirley?to talk to Shirley?

AnswerAnswer

To find the expected number of accepted calls To find the expected number of accepted calls per year, find the long-run proportion per year, find the long-run proportion (probability) of a call being accepted and (probability) of a call being accepted and multiply it by 52 weeks.multiply it by 52 weeks.

continued . . .continued . . .




AnswerAnswer (continued)(continued)

Let Let 11 = long run proportion of refused calls = long run proportion of refused calls

22 = long run proportion of accepted calls = long run proportion of accepted calls

Then, Then,

.35 .65 .35 .65

[[ ] = [] = [ ]]

.20 .80 .20 .80





Answer (continued)Answer (continued)

+ + = = (1) (1)

+ + = = (2) (2)

+ + = 1 (3) = 1 (3)

Solve for Solve for and and





Answer (continued)Answer (continued)

Solving using equations (2) and (3). Solving using equations (2) and (3). (Equation 1 is redundant.) Substitute (Equation 1 is redundant.) Substitute = 1 - = 1 - into (2) to give:into (2) to give:

.65(1 - .65(1 - 22) + ) + = = 22

This gives This gives = .76471. Substituting back = .76471. Substituting back into equation (3) gives into equation (3) gives = .23529. = .23529.

Thus the expected number of accepted Thus the expected number of accepted calls per year is:calls per year is:

(.76471)(52) = 39.76 or about 40(.76471)(52) = 39.76 or about 40



State ProbabilityState Probability

QuestionQuestion

What is the probability Shirley will What is the probability Shirley will accept Henry's next two calls if she does not accept Henry's next two calls if she does not accept his call this week?accept his call this week?




AnswerAnswer

P = .35(.35) = .1225P = .35(.35) = .1225

P = .35(.65) = .2275P = .35(.65) = .2275

P = .65(.20) = .1300P = .65(.20) = .1300REFUSESREFUSES

REFUSESREFUSES

REFUSESREFUSES

REFUSESREFUSES

ACCEPTSACCEPTS

ACCEPTSACCEPTS

ACCEPTSACCEPTS

.35.35

.35.35

.65.65

.20.20

.80.80.65P = .65(.80) = .5200P = .65(.80) = .5200P = .65(.80) = .5200P = .65(.80) = .5200




QuestionQuestion

What is the probability of Shirley accepting What is the probability of Shirley accepting exactly exactly one of Henry's next two calls if she one of Henry's next two calls if she accepts his call accepts his call this week?this week?




AnswerAnswer

The probability of exactly one of the next The probability of exactly one of the next two calls being accepted if this week's call is two calls being accepted if this week's call is accepted can be found by adding the accepted can be found by adding the probabilities of (accept next week and refuse the probabilities of (accept next week and refuse the following week) and (refuse next week and following week) and (refuse next week and accept the following week) = accept the following week) =

.13 + .16 = .29.13 + .16 = .29


Example: Jetair AerospaceExample: Jetair Aerospace

The vice president of personnel at Jetair Aerospace The vice president of personnel at Jetair Aerospace has noticed that yearly shifts in personnel can be has noticed that yearly shifts in personnel can be modeled by a Markov process. The transition matrix is:modeled by a Markov process. The transition matrix is:

Next YearNext Year Same Pos. Promotion Retire Quit FiredSame Pos. Promotion Retire Quit Fired

Current YearCurrent Year Same Same

Position .55 .10 .05 .20 .10Position .55 .10 .05 .20 .10 Promotion .70 .20 0 .10 Promotion .70 .20 0 .10

00 Retire Retire 0 0 1 0 0 0 1 0

00 Quit Quit 0 0 0 1 0 0 0 1

00 Fired Fired 0 0 0 0 0 0 0 0

1 1



Transition MatrixTransition Matrix

Next YearNext Year Retire Quit Fired Same Retire Quit Fired Same

PromotionPromotion Current YearCurrent Year Retire Retire 1 0 0 0 01 0 0 0 0 Quit Quit 0 1 0 0 0 1 0 0

0 0 Fired Fired 0 0 1 0 0 0 1 0

0 0

Same Same .05 .20 .10 .55 .05 .20 .10 .55 .10 .10

Promotion 0 .10 Promotion 0 .10 0 .70 .200 .70 .20




-1 -1 -1 -1

1 0 .55 .10 1 0 .55 .10 .45 -.10 .45 -.10

N N = ( = (II - - Q Q ) ) -1 -1 = = = =

0 1 .70 .20 0 1 .70 .20 -.70 -.70 .80 .80




The determinant, The determinant, dd = = aaaa - - aaaa

= (.45)(.80) - (-.70)(-.10) = (.45)(.80) - (-.70)(-.10) = .29 = .29

Thus, Thus,

.80/.29 .10/.29 2.76 .34 .80/.29 .10/.29 2.76 .34

NN = = = =

.70/.29 .45/.29 2.41 1.55.70/.29 .45/.29 2.41 1.55



NR NR MatrixMatrix

The probabilities of eventually moving to The probabilities of eventually moving to the absorbing states from the nonabsorbing the absorbing states from the nonabsorbing states are given by:states are given by:

2.76 .34 2.76 .34 .05 .20 .10 .05 .20 .10

NRNR = = x x

2.41 1.55 0 .10 2.41 1.55 0 .10 0 0



NR NR Matrix (continued)Matrix (continued)

Retire Quit FiredRetire Quit Fired

Same .14 .59 .28 Same .14 .59 .28 NRNR = = Promotion .12 .64 .24Promotion .12 .64 .24



Absorbing StatesAbsorbing States

QuestionQuestion

What is the probability of someone who What is the probability of someone who was just was just promoted eventually retiring? . . . promoted eventually retiring? . . . quitting? . . . quitting? . . . being fired? being fired?



Absorbing States (continued)Absorbing States (continued)

AnswerAnswer

The answers are given by the bottom row The answers are given by the bottom row of the of the NRNR matrix. The answers are therefore:matrix. The answers are therefore:

Eventually Retiring = .12Eventually Retiring = .12

Eventually Quitting = .64Eventually Quitting = .64

Eventually Being Fired = .24Eventually Being Fired = .24


End of Chapter 17End of Chapter 17

slides prepared by john s. loucks st. edward’s university

Documents