ctmc. applications in bioinformaticswebdiis.unizar.es/asignaturas/spn/material/ctmc.pdf · • time...
TRANSCRIPT
![Page 1: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/1.jpg)
Continuous Time Markov Chains.Applications in Bioinformatics
Javier CamposUniversidad de Zaragoza
http://webdiis.unizar.es/~jcampos/
![Page 2: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/2.jpg)
2
Outline• Definitions• Steady-state distribution• Examples• Alternative presentation of CTMC:
embedded DTMC of a CTMC• Applications in Bioinformatics
![Page 3: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/3.jpg)
3
Definitions• Remember DTMC
– pij is the transition probability from i to j over one time slot
– The time spent in a state is geometrically distributed
• Result of the Markov (memoryless) property
– When there is a jump from state i, it goes to state j with probability
![Page 4: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/4.jpg)
4
Definitions• Continuous time version
– qij is the transition rate from state i to state j
ij
k
qij
qik
![Page 5: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/5.jpg)
5
Definitions• Formally:
– A CTMC is a stochastic process {X(t) | t ≥0, t lR} s.t. for all t0,...,tn-1,tn,t lR, 0≤t0<…<tn-1<tn<t , for all n lN
– Alternative (equivalent) definition: {X(t) | t ≥0, t lR} s.t. for all t,s ≥ 0
))(|)(())(,,)(,)(|)(( 0011
nn
nnnn
xtXxtXPxtXxtXxtXxtXP
))(|)(()0),(,)(|)((
t
t
xtXxstXPtuuXxtXxstXP
![Page 6: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/6.jpg)
6
Definitions• Homogeneity
– We are considering discrete state (sample) space, then wedenote
pij(t,s) = P(X(t+s)=j | X(t)=i), for s > 0.
– A CTMC is called (time-)homogeneous if
pij(t,s) = pij(s) for all t ≥ 0
![Page 7: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/7.jpg)
7
Definitions• Time spent in a state (sojourn time):
– Markov property and time homogeneity imply that if at time t the process is in state j, the time remaining in state j isindependent of the time already spent in state j : memoryless property.Let S be the random variable “time spent in state j”.
time spent in state j is exponentially distributed.
Sojourn times of a CTMC are exponentially distributed.
![Page 8: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/8.jpg)
8
Definitions• Transition rates:
– In time-homogeneous CTMC, pij(s) is the probability ofjumping from i to j during an interval time of duration s.
– Therefore, we define the instantaneous transition rate fromstate i to state j as:
– And the exit rate fromstate i as – qii
– Q = [qij] is called infinitesimal generator matrix(or transition rate matrix or Q matrix)
ttp
q ij
tij
)(lim
0
ttpqq ii
tijijii
1)(lim
0
ij
k
qij
qik
![Page 9: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/9.jpg)
9
Outline• Definitions• Steady-state distribution• Examples• Alternative presentation of CTMC:
embedded DTMC of a CTMC• Applications in Bioinformatics
![Page 10: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/10.jpg)
10
Steady-state distribution• Kolmogorov differential equation:
Denote the distribution at instant t: i(t) = P(X(t)=i)And denote in matrix form: P(t) = [pij(t)]
Then (t) = (u)P(t-u) , for u < t (we omit vector transposition to simplify notation)
Substituting u = t–t and substracting (t–t):
(t) – (t–t) = (t–t) [P(t) – I], with I the identity matrix
Dividing by t and taking the limit
Then, by definition of Q = [qij], we obtain theKolmogorov differential equation
tItPtt
dtd
t
)(lim)()(0
Qttdtd )()(
![Page 11: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/11.jpg)
11
Steady-state distribution• Since also (t)e = 1, with e = (1,1,…,1)
If the following limit exists
then taking the limit of Kolmogorov differentialequation we get the equations for the steady-stateprobabilities:
Q = 0 (balance equations)
e = 1 (normalizing equation)
)(lim tt
![Page 12: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/12.jpg)
12
Outline• Definitions• Steady-state distribution• Examples• Alternative presentation of CTMC:
embedded DTMC of a CTMC• Applications in Bioinformatics
![Page 13: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/13.jpg)
13
Examples• Example 1: A 2-state CTMC
– Consider a simple two-state CTMC
– The correspondingQ matrix is given by
– The Kolmogorov differentialequation yields:
– Given that 1(0) = 1, we get the transient solution:
Q
t
t
et
et
)(0
)(1
)(
)(
1)()(
)()()(
)()()(
10
101
100
tt
tttdtd
tttdtd
![Page 14: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/14.jpg)
14
Examples– And the steady-state
solution comes fromQ = 0; e = 1:
– We get:
• Which can also be obtained by taking the limits as t ofthe equations for 1(t) and 0(t).
100
10
10
01
10 ;
t
tt
t
tt
et
et
)(00
)(11
lim)(lim
lim)(lim
balance eq.
![Page 15: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/15.jpg)
15
Examples• Example 2:
A simple open system with loss
– Works enter to the system with exponentially distributed(parameter interarrival time (Poisson process)
– The service time in both processing stations is exponentiallydistributed with rate
– If a work ends in station 1 when station 2 is busy, station 1 isblocked
– If station 1 is busy or blocked when a work arrives, arriving work islost
– Questions: • proportion of lost works? • mean number of working stations? • mean number of works in the system?
1 2
may be lost
(Poisson)
![Page 16: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/16.jpg)
16
Examples– The set of states of the system:
S={(0,0), (1,0), (0,1), (1,1), (b,1)}0 empty station1 working stationb blocked station
State transition diagram: Infinitesimal generator matrix:
1 2
may be lost
(Poisson)
0,0 1,0 0,1
1,1 b,1
Q
00100111b1
( )
![Page 17: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/17.jpg)
17
Examples– Steady-state solution:
243 and where,,2,2,2 2222
10
eQ
1
2)(
111011000
111
1101
01110
101100
0100
b
b
b
0,0 1,0 0,1
1,1 b,1
balance eq.
![Page 18: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/18.jpg)
18
Examples– Proportion of lost works?
• Is the probability of the event “when a new work arrives, the firststation is non-empty”, i.e.:
– Mean number of working stations?• In state (0,0) there is no working station and in state (1,1) there are
two; in the rest of states there is only one, thus
– Mean number of works in the system?• In state (0,0) there is no one; in states (1,1) and (b,1) there are two
and in the rest there is only one, thus
10 11b1 32 2
32 42
B0110 b1 211 42 4
32 42
L 01 10 2 b1 2 11 52 4
3 2 4 2
![Page 19: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/19.jpg)
19
Examples• Example 3: I/O buffer with limited capacity
– Records arrive according to a Poisson process (rate )– Buffer capacity: M records– Buffer cleared at times spaced by intervals which are
exponentially distributed (parameter ) and independent ofarrivals
![Page 20: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/20.jpg)
20
Examples– Steady-state solution:
Thus, for example, the mean number of records in the buffer in steady-state:
1
11 ,)()(
0
1
1
10
M
MM
ii
M
Mi
M
M
i
i Mi
10 ,
1
0
where,M
i
iM iMB
![Page 21: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/21.jpg)
21
Outline• Definitions• Steady-state distribution• Examples• Alternative presentation of CTMC:
embedded DTMC of a CTMC• Applications in Bioinformatics
![Page 22: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/22.jpg)
22
Semi-Markov processes• Also called Markov renewal processes
– Generalization of CTMC with arbitrary distributedsojourn times
– As in the case of CTMC, they can be used for obtaining both steady-state distribution and transient distribution
![Page 23: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/23.jpg)
23
Semi-Markov processes• (Time homogeneous) Semi-Markov process:
Transition probability from i to j:
Sojourn time distribution in state i when the next state is j:
Transition prob. matrix of the embedded Discrete Time Markov Chain:
Sojourn time distribution in state i regardless of the next state:
i j
k
Qij(t)
Qik(t)
Distribution, from i to j in time t:
ij
k
pij
pik
![Page 24: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/24.jpg)
24
Semi-Markov processes• Steady-state analysis
– Build transition probability matrix, P,and mean residence (sojourn) time vector, D.
– From them, compute steady-state distribution:
Steady-state of embedded DTMC
Mean residence time vector
Steady-state distribution of semi-Markov process
![Page 25: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/25.jpg)
25
Embedded DTMC of a CTMC• CTMC can be seen as a particular case of
semi-Markov process when sojourn time distributions are exponential
• In other words, if we consider a DTMC withnull diagonal elements and we addexponentially distributed soujourn times in the states, we get a CTMC.
this is called theEmbedded DTMC
of the CTMC
CTMC = DTMC (where to move) + + exponential holding times (when to move)
![Page 26: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/26.jpg)
26
Outline• Definitions• Steady-state distribution• Examples• Alternative presentation of CTMC:
embedded DTMC of a CTMC• Applications in Bioinformatics
![Page 27: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/27.jpg)
27
Simple population size model• We model the size of the population (n = 0,1,2…)
n is the birth rate in a population of size n; n is the death rate 0 is an absorbing state
A case of relevance to population dynamics is the choice:n = n and n = nμ, for constants and μ.
We can ask:What is the extinction probability starting from state k?
(i.e., probability of being absorbed into state 0 (ever))If extinction is certain, what is the mean time to extinction?
![Page 28: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/28.jpg)
28
Extinction probability• Let uk be the probab. of absorption at 0, starting from state k.• Then, since the first step away from k is either to k+1, with
probability k / (k + μk) or to k−1, with probability μk / (k + μk),
and u0 = 1.• Let 0 = 1 and j = (μj / j) j–1 for j ≥ 1.
– If j j = then uk = 1 for all k (extinction is certain).
– If j j is finite then
![Page 29: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/29.jpg)
29
Mean time to extinction• Let wk be the mean time to absorption at 0, starting from state
k. By first-step analysis
and w0 = 0.
–
–
![Page 30: CTMC. Applications in Bioinformaticswebdiis.unizar.es/asignaturas/SPN/material/CTMC.pdf · • Time spent in a state (sojourn time): – Markov property and time homogeneity imply](https://reader033.vdocument.in/reader033/viewer/2022042810/5f9f1d492e27ac0dfd05323f/html5/thumbnails/30.jpg)
30
Many other applications…• As embedded analytical model for more
abstract modelling paradigms:– birth-death processes and population dynamics– queueing processes and queueing networks– stochastic Petri nets– stochastic process algebras…