markov process
TRANSCRIPT
![Page 1: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/1.jpg)
Markov Processes-III
Presented by:
![Page 2: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/2.jpg)
Outline
• Review of steady-state behavior• Probability of blocked phone calls• Calculating absorption probabilities• Calculating expected time to absorption
![Page 3: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/3.jpg)
Preview
• Assume a single class of recurrent states, a-periodic:• Plus transient states, then
• Where Does not depend on the initial conditions
jijnn r )()lim(
j
jnijn xx )|()lim(
0
![Page 4: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/4.jpg)
Preview
• Can be found as the unique solution to the balance equations
• Together with
m1
mjk
kjkj P 1,
j
j1
![Page 5: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/5.jpg)
Example
1 2
7/5,7/221
5.0 5.0 8.0
2.0
![Page 6: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/6.jpg)
Example
• Assume that process starts as state 1
)99()1,1(11111001 rPxx andP
prxx andP1211101100
)100()21(
![Page 7: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/7.jpg)
The Phone Company Problem
• Calls originate as a poison process, rate Each call duration is exponentially distributed(parameter B lines available• Discrete time intervals of( small) length
![Page 8: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/8.jpg)
The Phone Company Problem
ii
i
EquationsBalance
1
:
B
i
iiii
iii
000
!//1!/
i
![Page 9: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/9.jpg)
Steady State Probability: Example#1
• Consider a Markov chain with given transition probabilities and with a single recurrent class which is a-periodic
• Assume that for the n-step transition probabilities are very close to steady state probabilities
• Find
500n
lJKJjkij PPrXXXX ilkJP
)1000(
)|,,(0200010011000
![Page 10: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/10.jpg)
Steady State Probability: Example#1
• B) • Solution:• By using Bay’s Rule:
?/(10011000
jiP XX
jijiP
XXXXX jPjiPjiP
/
)(/),()/(10011001100010011000
![Page 11: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/11.jpg)
Steady State Probability: Example#2
• An absent minded professor has two umbrellas that she uses when coming from home to office and back. If it rains and umbrella is available in her location, she takes it. If it is not raining, she always forget to take an umbrella. Suppose it rains with probability ‘P’ each time comes, independently of other times, what is the steady state probability that she wet during a rain?
![Page 12: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/12.jpg)
Steady State Probability: Example#2
• Markov mode with following states:• State ‘i’ where i=0,1,2• ‘i’ umbrellas are available in current location• Transition probability matrix:
01
10
100
pp
pp
![Page 13: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/13.jpg)
Steady State Probability: Example#2
• The chain has single recurrent class which is a-periodic
• So, steady state convergence theorem applies.
0 2 1 p1
p1 p
p1
![Page 14: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/14.jpg)
Steady State Probability: Example#2
• Balance Equations are as below:
1
)1(
)1(
021
102
211
20
p
pp
p
![Page 15: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/15.jpg)
Steady State Probability Example#2
• After Solving:
• So, the steady state probability that she gets wet is times the probability of rain=
)3(/1
)3(/1
)3(/)1(
1
1
0
p
p
pp
0p 0
![Page 16: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/16.jpg)
Calculating Absorption Probabilities
• What is the probability that: process eventually settles in state 4, given that the initial state is i?
ai
3
2
5
4
1
1 12.0
2.0
3.0
4.0
5.0
6.0
8.0
![Page 17: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/17.jpg)
Calculating Absorption Probabilities
SolutionuniqueiotherallFor
iFor
iFor
apa
aa
jj
iji
i
i
0,5
1,4
![Page 18: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/18.jpg)
Expected time to absorption
• Find expected number of transitions until reaching the Absorbing state, given that the initial state is i?
3
2
4
1
1
2.0
5.0
4.0
5.0
6.0
8.0
i
![Page 19: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/19.jpg)
Expected Time to Absorption
SolutionuniqueiotherallFor
ifor
jjiji
i
p
1:
40
![Page 20: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/20.jpg)
Absorption ProbabilitiesExample#1
• Consider the Markov Chain
![Page 21: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/21.jpg)
Mean First Passage and Recurrence Times
• Chain with one recurrent class;• Fix s recurrent• Mean first passage time from s to i
jjiji
s
m
ni
siallfor
tosolutionuniquetheare
isthatsuchnE
tptt
tttXXt
1
0
,.
]|}0[min{
21
0
![Page 22: Markov process](https://reader035.vdocument.in/reader035/viewer/2022062308/55d23fbdbb61eb8b768b478a/html5/thumbnails/22.jpg)
Mean First Passage and Recurrence Times
• Mean recurrence time of s:
p
ssthatsuchnE
ajj js
ns
ttXXt
1
]|}1[min{0