1 introduction to stochastic models gslm 54100. 2 outline limiting distribution connectivity ...
TRANSCRIPT
![Page 1: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/1.jpg)
1
Introduction to Stochastic ModelsIntroduction to Stochastic ModelsGSLM 54100GSLM 54100
![Page 2: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/2.jpg)
2
OutlineOutline
limiting distribution connectivity
types of states and of irreducible DTMCs transient, recurrent, positive recurrent, null
recurrent
periodicity
limiting behavior of irreducible chains
![Page 3: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/3.jpg)
3
ConnectivityConnectivity
![Page 4: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/4.jpg)
4
Connectivity of a Connectivity of a DTMCDTMC
connectivity: one factor that determines the limiting behavior of a DTMC
1 2
1
0.010.99
A
0.99 0.01
0 1
P
1 2
0.99
0.010.99
B
0.01
0.99 0.01
0.01 0.99
P
![Page 5: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/5.jpg)
5
Connectivity of a Connectivity of a DTMCDTMC
1 2
1
0.010.99
A
0.99 0.01
0 1
P
(4) 0.960596 0.03940399
0 1
P
(16) 0.8514576 0.1485422
0 1
P
(64) 0.5255965 0.4744035
0 1
P
(256) 0.07631498 0.923685
0 1
P
(1024) 0.0000339187 0.9999661
0 1
P
( ) 0 1
0 1
P
![Page 6: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/6.jpg)
6
Connectivity of a Connectivity of a DTMCDTMC
1 2
0.99
0.010.99
B
0.01
0.99 0.01
0.01 0.99
P
(4) 0.96118408 0.03881592
0.03881592 0.96118408
P
(16) 0.8618989 0.1381011
0.1381011 0.8618989
P
(64) 0.6372268 0.3627732
0.3627732 0.6372268
P
(256) 0.5028369 0.4971631
0.4971631 0.5028369
P
(1024) 0.5 0.5
0.5 0.5
P
( ) 0.5 0.5
0.5 0.5
P
![Page 7: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/7.jpg)
7
Connectivity of a Connectivity of a DTMCDTMC
1 2
0.99
0.010.99
B
0.01
0.99 0.01
0.01 0.99
P
( ) 0.5 0.5
0.5 0.5
P
1 2
1
0.010.99
A
0.99 0.01
0 1
P
( ) 0 1
0 1
P
![Page 8: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/8.jpg)
8
Connectivity of a Connectivity of a DTMCDTMC
1 2 0.6
0.10.9
C
1
30.4
0.9 0.1 0
0 0.6 0.4
1 0 0
P
( )
0.7407407 0.1851852 0.07407407
0.7407407 0.1851852 0.07407407
0.7407407 0.1851852 0.07407407
P
![Page 9: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/9.jpg)
9
Connectivity of a Connectivity of a DTMCDTMC
1 2
1
0.010.99
A
0.99 0.01
0 1
P
( ) 0 1
0 1
P
1 2
0.8
0.10.9
D
0.20.9 0.1
0.2 0.8
P
( ) 0.666667 0.333333
0.666667 0.333333
P
( )11lim 0n
np
( )
12lim 1n
np
( )21lim 0n
np
( )
22lim 1n
np
( )11lim 2 / 3n
np
( )
12lim 1/ 3n
np
( )21lim 2 / 3n
np
( )
22lim 1/ 3n
np
![Page 10: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/10.jpg)
10
Connectivity of a Connectivity of a DTMCDTMC
rows of P() may not be the same as in the previous examples
1 2
1
0.5
3 4
0.5
0.5
0.5
0.5
5
0.51
1 0 0 0 0
0.5 0 0.5 0 0
0 0.5 0 0.5 0
0 0 0.5 0 0.5
0 0 0 0 1
P
( )
1 0 0 0 0
0.75 0 0 0 025
0.5 0 0 0 0.5
0.25 0 0 0 0.75
0 0 0 0 1
P
![Page 11: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/11.jpg)
11
Connectivity of a Connectivity of a DTMCDTMC
1 2
1
F
1
0 1
1 0
P
(even) 1 0
0 1
P
(odd) 0 1
1 0
P
limit of P(m) may not exist
![Page 12: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/12.jpg)
12
Connectivity of a Connectivity of a DTMCDTMC0 1 0
0.75 0 0.25
0 1 0
P
even0.75 0 0.25
0 1 0
0.75 0 0.25
P 0 1 0
0.75 0 0.25
0 1 0
odd
P
1 2 3
0.25
1
1
0.75
1 2 3
0.25
0.99
1
0.75
0.01
0 1 0
0.75 0 0.25
0.01 0.99 0
P
0.3757803 0.4993758 0.1248439
0.3757803 0.4993758 0.1248439
0.3757803 0.4993758 0.1248439
P
![Page 13: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/13.jpg)
13
Types of States and of Irreducible DTMCs
![Page 14: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/14.jpg)
14
Limiting Results for a Limiting Results for a DTMCDTMC
depending on the type of states and the chain
type: transient, positive recurrent, null recurrent
connectivity and periodicity
![Page 15: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/15.jpg)
15
Transient State Transient State
1 2
1
0.010.99
A
1 2
1
0.5
3 4
0.5
0.5
0.5
0.5
5
0.51
( ) 0 1
0 1
P( )11lim 0
np
( )
1 0 0 0 0
0.75 0 0 0 025
0.5 0 0 0 0.5
0.25 0 0 0 0.75
0 0 0 0 1
P
( )22lim 0
np
( )33lim 0
np
( )44lim 0
np
![Page 16: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/16.jpg)
16
Recurrent State Recurrent State
state i is recurrent if P(return to i|X0 = i) = 1
( )
0
nii
np
![Page 17: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/17.jpg)
17
Recurrent State Recurrent State
two types of recurrent states positive recurrent:
E(# of transitions to return to i|X0 = i) <
null recurrent:
E(# of transitions to return to i|X0 = i) =
( )
0lim
N nii
n
N
p
N
( )
0lim
N nii
n
N
p
N
![Page 18: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/18.jpg)
18
Periodicity
![Page 19: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/19.jpg)
19
PeriodicityPeriodicity
state i is of period d if Xn can return to state i in multiples of d
states 1, 2, 3 are of period 2
state i of period d
1 2 3
0.25
1
1
0.75
( ) 0 for 1nd kiip k d
![Page 20: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/20.jpg)
20
PeriodicityPeriodicity
period of states 1, 2, 3, and 4 = ?
state 4 of period 2 state 4 can return to itself in 2 steps
1 2 3
0.25
0.99
1
0.754
0.01
1
![Page 21: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/21.jpg)
21
Communicating States Communicating States
communicating states are of the same type transient, positive recurrent, null recurrent at the
same time of the same period
states in an irreducible chain are of the same type transient, positive recurrent, null recurrent at the
same time of the same period
![Page 22: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/22.jpg)
22
Limiting Behavior of Irreducible Chains
![Page 23: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/23.jpg)
23
Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain
j = fraction of time at state j
N: a very large positive integer
# of periods at state j j N
balance of flow j N i (i N)pij j = i ipij
[ ]ijpP
![Page 24: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/24.jpg)
24
Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain
j = fraction of time at state j
j = i ipij
1 = 0.91 + 0.22
2 = 0.11 + 0.82
linearly dependent
normalization equation: 1 + 2 = 1
solving: 1 = 2/3, 2 = 1/3
1 2
0.8
0.10.9
C
0.2
( ) 0.666667 0.333333
0.666667 0.333333
P
![Page 25: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/25.jpg)
25
Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain
1 = 0.752 + 0.013
3 = 0.252
1 + 2 + 3 = 1
1 = 301/801, 2 = 400/801, 3 = 100/801
1 2 3
0.25
0.99
1
0.75
0.01
0 1 0
0.75 0 0.25
0.01 0.99 0
P
0.3757803 0.4993758 0.1248439
0.3757803 0.4993758 0.1248439
0.3757803 0.4993758 0.1248439
P
![Page 26: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/26.jpg)
26
Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain
an irreducible DTMC {Xn} is positive there exists a unique nonnegative solution to
j: stationary (steady-state) distribution of {Xn}
0
0
1 (normalization eqt)
, for all , (balance eqts)
jj
j i iji
p j
![Page 27: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/27.jpg)
27
Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain
j = fraction of time at state j
j = fraction of expected time at state j
average cost cj for each visit at state j
random i.i.d. Cj for each visit at state j
for aperiodic chain:
1
0lim
k
n
Xk
j jn j
E cc
n
1
0lim ( )
k
n
Xk
j jn j
E CE C
n
0lim ( | )n jn
P X j X
![Page 28: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/28.jpg)
28
Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain
1 = 301/801, 2 = 400/801, 3 = 100/801
profit per state: c1 = 4, c2 = 8, c3 = -2
average profit
1 2 3
0.25
0.99
1
0.75
0.01
0 1 0
0.75 0 0.25
0.01 0.99 0
P 301 400 100801 801 801
4201801
(4) (8) (2)
![Page 29: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/29.jpg)
29
Limiting Behavior Limiting Behavior of a Positive Irreducible Chainof a Positive Irreducible Chain
1 = 301/801, 2 = 400/801, 3 = 100/801
C1 ~ unif[0, 8], C2 ~ Geo(1/8), C3 = -4 w.p. 0.5; and = 0 w.p. 0.5 E(C1) = 4, E(C2) = 8, E(C3) = -2
average profit
1 2 3
0.25
0.99
1
0.75
0.01
0 1 0
0.75 0 0.25
0.01 0.99 0
P
301 400 100801 801 801
4201801
(4) (8) (2)
![Page 30: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/30.jpg)
30
Different Interpretations of Different Interpretations of
balance equations: balance of rates
total rate into a group of states = total rate out of a group of states
{0}: 0 = q1
{0, 1}: p1 = q2
{0, 1, …, n}: pn = qn+1 , n 1
0 1
q
2 3
p
q
p
q
p1
…q
0j i ij
ip
![Page 31: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/31.jpg)
31
Example: Condition for the Example: Condition for the Following Chain to be PositiveFollowing Chain to be Positive
{0}: 0 = q1
{0, 1}: p1 = q2 1
{0, 1, …, n}: pn = qn+1 , n 11
positive {j} exists
0 + 1 + 2 + … = 1 has solution
.
p < q 0 1
q
2 3
p
q
p
q
p1
…q
11 0q
12 1 0
p pq q q
11
0
npn q q
21 1 1
0 1 ... 1p pq q q q q
![Page 32: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/32.jpg)
32
Example 4.24 of RossExample 4.24 of Ross four-state production process states {1, 2, 3, 4} up states {3, 4}, down states {1, 2} find E(up time) & E(down time)
time
1
state2
3
4
time
downstate
up
![Page 33: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/33.jpg)
33
Example 4.24 of RossExample 4.24 of Ross
1 = 3/16, 2 = 1/4, 3 = 7/24, 4 = 13/48
how to find E(up time) E(down time)
0.25 0.25 0.5 0
0 0.25 0.5 0.25
0.25 0.25 0.25 0.25
0.25 0.25 0 0.5
P
![Page 34: 1 Introduction to Stochastic Models GSLM 54100. 2 Outline limiting distribution connectivity types of states and of irreducible DTMCs transient,](https://reader036.vdocument.in/reader036/viewer/2022062322/56649ea95503460f94badb94/html5/thumbnails/34.jpg)
34
Example 4.24 of RossExample 4.24 of Ross
fraction of up time = 3 + 4 =
rate of turning from up to down
= (p31+p32)3 + (p41+p42) 4
= rate of turning from down to up
= p131 + (p23+p24) 2
=
0.25 0.25 0.5 0
0 0.25 0.5 0.25
0.25 0.25 0.25 0.25
0.25 0.25 0 0.5
P
(up time)
(up time) (up time)
E
E E
1
(up time) (up time)E E