symbolic dynamics of m arkov chains

Symbolic dynamics of Markov chains

P S ThiagarajanSchool of Computing

National University of Singapore

Joint work with:Manindra Agrawal, S Akshay, Blaise Genest

Acknowledgements

• Samarjit and Javier

• IAS

• TÜV SÜD Foundation

Agenda

• Study complex dynamical systems– verification of temporal logic specification

• High dimensional• Continuous time; continuous value domains.

– Differential equations– Hybrid automata

• Networks of such dynamical systems

Strategy• Approximate the dynamics.

– Discretize the time and value domains• Today’s first talk will have this flavor.

– Sample the dynamics and do a statistical analysis.• Today’s second talk

• To handle large networks: Deterministic synchronizations • Tomorrow’s talk

Probabilistic dynamics

• Illustrate these ideas in the setting of probabilistic dynamical systems.– Yield useful approximations in the presence intractability and

lack of knowledge.– Scalability can be achieved through sampling (simulations)

based statistical analysis techniques.• Statistical model checking

• Key application: (Networks) of biochemical networks.

Study I: Symbolic dynamics of Markov chains

• Well-known probabilistic dynamical systems.• Rich theory• Widely applied.• Our focus (finite state, discrete time):

– Symbolic dynamics• Discretize the probability value space [0, 1].


• Analyze the symbolic dynamics via:– classical (linear) temporal logic specifications.

• Probabilities are sneaked in through the atomic propositions.– Model checking methods

• Details in:– LICS’2012– Journal paper under review.

Two views of Markov chains

• View I– a finite state probabilistic transition system.

• View II – a linear transform of probability distributions over the states.

The transition system view

23

1 4

1

1

1

2/5

3/5


23

1 4

1

1

1

2/5

3/5

1 0

00


23

1 4

1

1

1

2/5

3/5

1 3

4

1

1 2

2/5 3/5

3 3

4 42/5 3/52/5


• Outcomes: Infinite paths• Basic cylinder: The set of infinite paths that have a

common finite prefix. • Path space: -algebra generated by the basic cylinders. • Probabilities are assigned to basic cylinders in a natural

way.• Extends canonically to a probability measure over the

path space.


23

1 4

1

1

1

2/5

3/5

1 3

4

1

1 2

2/5 3/5

3 3

11

4 4

1 1

B

Pr(B) = 1 2/5 1 1 = 2/5

B – The set of all paths that have the prefix 3 4 1 3 4

Measurable properties

• The state 3 will be visited infinitely often.

PCTL, CTL, ….Baier, C. and Katoen, J.-P. Principles of Model Checking

VIEW II

• The Markov chain transforms (linearly) a given distribution over its nodes to a new one.

• The graph of the chain is represented as a stochastic matrix – (all row sums are 1)

• The Perron-Frobinius theory (for non-negative matrices) applies.

VIEW II

• Leads to natural sub-classes:– Irreducible and aperiodic,– periodic, …

• transient states, recurrent states…. • Stationary distributions….

VIEW II

23

1 4

1

1

1

2/5

3/5

The second view

23

1 4

1

1

1

2/5

3/5

1

0

0

0

Trajectory of distributions• Markov chain as (linear)transformer of probability

distributions

Transition matrix M

23

1 4

1

1

1

2/5

3/5

1

0

0

0

Trajectory of distributions

23

1 4

1

1

1

2/5

3/5

Transition matrix M

• Each initial distribution generates a trajectory of distributions:

1

00

0


• Dynamics:– The set of trajectories generated from an initial set of

distributions– This set can be infinite

• We can ask whether all (none) of the trajectories satisfy some desired dynamical property

• What we can ask in this setting is incomparable with View I specifiable properties


• Eventually the probability of being in 1 is always greater than being in 3.

• There is a future time at which 90% of the probability mass lies on {1, 4}.

Symbolic dynamics

• A trajectory: – an infinite sequence of probability distributions (over a finite set

of nodes).– The alphabet of probability distributions -over a finite set of

nodes- is (uncountably) infinite.• Hence a trajectory is an infinite sequence over an infinite

alphabet,• But exactly tracking probability distributions may be

neither necessary nor possible.– Bio-chemical networks – Sensor networks…

Symbolic dynamics

• We propose to reason about sequences of distributions in a symbolic way.– Using a finite alphabet of discretized distributions.

Symbolic dynamics

• A classical tradition dynamical systems theory.– Jacques Hadamard(1898), Morse

and Hedlund (1921), ….–Shannon, Smale…–Significant applications in data

storage and transmission (via coding theory), linear algebra.

Symbolic dynamics

• We know how to do this for:– Timed automata– In some restricted settings for hybrid automata.

• For finite state Markov chains this has been done under very restricted setting and unnatural restrictions (Chada et.al 2011, Korthikanti et.all 2010.)

• We shall instead give up on bisimulation-based equivalence classes:

• Instead, fix a discretization of [0, 1]– With no restrictions handle all finite state Markov chains .

Symbolic dynamics

x

F(x)F2(x)

F3(x)a

b

cd

e

Finite alphabet!

Symbolic dynamics

• Each block is a letter.• Each trajectory is now recorded as a sequence

of letters taken from a finite alphabet.• If each block bisimualtion equivalence class it

is called a Markov partition!• Study the system dynamics in terms of these

sequences.–Sophic shift sequences–Shift sequences of finite type.

Symbolic dynamics

• We know how to get “Markov partitions”) do this for:– Timed automata– For hybrid automata, in a few settings.

• For finite state Markov chains this has been done under very restricted setting and unnatural restrictions (Chada et.al 2011, Korthikanti et.all 2010.)

• We do not look for bisimulations of finite index.:• Instead, we fix a discretization of [0, 1]

– Handle all finite state Markov chains ; no restrictions.


• Discretization– We partition [0,1] into a finite set of intervals,

a b c


• Each (probability) value is mapped to (identified with) the interval it falls in.

a b c

Symbolic dynamics of Markov chains• Each distribution 𝜇 maps to D, a unique n-tuple of intervals.

• 𝜇 ∈D means Γ(𝜇) = D

= D


• Each distribution 𝜇 maps to D, a unique n-tuple of intervals.• Several distributions can map to the same D,

• in fact Γ-1(D) can be infinite

= D


• Discretization– We partition [0,1] into a finite set of intervals,

• Γ-1(D) can be empty

= D?


• Discretized distribution– A tuple of intervals D is discretized distribution ( 𝒟-

distribution for short ) if Γ-1(D) ≠ ø = D

ø = D ×


• The set of discretized distributions 𝒟 is finite

Symbolic dynamics of Markov chains• A Trajectory of M starting from a distribution 𝜇 :• Induces a symbolic trajectory

– a word over 𝒟ω

• ξ 𝜇 = (Γ(𝜇) Γ(𝜇1) Γ(𝜇2) ….)


23

1 4

1

1

1

2/5

3/5

0.080.35

0.120.45

. .


• Fix a set of initial distributions IN. • We use a 𝒟-distribution Din to specify IN. That is IN = Din

This can be an infinite set .

– IN = = , ………..

• We can fix the set of initial distributions in many other ways.


• The symbolic dynamics of (M , Din ) is:

• We wish to reason about this -language.


• The discretization need not be uniform• Can be different for each dimension (node).• {[0, 1]} can be used to mask out “don’t care” nodes.

– Dimension-reduction.

A temporal logic to reason about the symbolic dynamics

• I, a discretization of [0, 1] :– < i , d > is an atomic proposition – node i interval d

– “In the current distribution 𝜇, 𝜇(i) falls in the interval d ”.– In the current discretized distribution D, D(i) = d

• Probabilistic linear-time temporal logic LTL I

The verification problem

• Given,– a Markov chain M,– a discretization I,– an initial set of distributions represented as Din ,

– and a specification φ as an LTL I formula,

• Determine if M, Din ⊨ φ. In other words, – (0) ⊨ for every in LM – Does every symbolic trajectory of M satisfy φ, i.e., – is it the case LM ⊆ L φ?

Example formulas

• Whenever probability of node i is “high”, the probability of node j is “low” :

• The 𝒟 -distribution (d1, d2, . . . , dn) appears infinitely often:

Example formulas

• Extending with FO theory of reals, we can express much more:– e.g., Infinitely often, the probability of node i is at least twice the sum

of probabilities of all other nodes.

• Logics based on path spaces (PCTL etc.) and logics based sequences of probability distributions are incomparable

Examples

• We have modeled a simple version of the Google pageranking algorithms.

• We have also modeled a small pharmacokinetics system for drug delivery.


• Given, M, I, Din , φ, determine if M, Din ⊨ φ , i.e., LM ⊆ L φ– If LM is ω-regular and effectively computable then we can use standard

model checking techniques.

– But LM is not always ω-regular !


LM is not always ω-regular.

𝜖-approximations

• Fix a small 𝜖 > 0 (fraction of length of an interval of I )..𝜇 𝜇’

𝜖-approximations

• Fix a small 𝜖 >0 (fraction of length of an interval of I ). 𝜖𝜇

D..𝜇 𝜇’

𝜖-approximations

• Fix a small 𝜖 >0(fraction of length of an interval of I )

• The 𝜖-nbhd of a concrete distribution 𝜇 is the set of all 𝒟-distributions which are atmost 𝜖-away from it.

. 𝜖𝜇D

..𝜇 𝜇’

𝜖-approximations

• The 𝜖-nbhd of a concrete distribution 𝜇 is the set of all 𝒟-distributions which are atmost 𝜖-away from it. – Nbr()

• 𝓕 is an -neighborhood iff there exists such that Nbr() = 𝓕

𝜖-approximations

• Suppose D, D’ belong to a final class. Then there exist in D and ’ in D’ such that (, ’) 2𝜖

• Suppose D, D’ belong to a final class. Then for every in D and ’ in D’ , (, ’) 2𝜖 + where depends only on the discretization.

𝓕DD

DDD

D’

’2

A basic property of the symbolic dynamics

• There exists a computable constant K𝜖 and a computable ordered collection of 𝜖-nbhds { 𝓕0, 𝓕1, . . . , 𝓕𝜃} called final classes such that:– After K𝜖 steps, ξ 𝜇 will forever cycle through the ordered final

classes. In other words, ξ 𝜇(k) ∈ 𝓕k mod 𝜃 for k > K𝜖


• There exists a computable constant K𝜖 and an ordered collection of 𝜖-nbhds { 𝓕0, 𝓕1, . . . , 𝓕𝜃} called final classes such that:– After K𝜖 steps, ξ 𝜇 will forever cycle through the ordered final


𝜇1 𝜇2𝜇 M M K𝜖 steps . 𝜖𝓕0

𝓕𝜃-1 𝓕2

𝓕1

M

MM


• There exists a computable constant K𝜖 and an ordered collection of 𝜖-nbhds { 𝓕0, 𝓕1, . . . , 𝓕𝜃} called final classes such that:– After K𝜖 steps, ξ 𝜇 will forever cycle through the ordered final


𝜇1 𝜇2𝜇 M M K𝜖 steps .𝓕0

𝓕𝜃-1 𝓕2

𝓕1

M

MM

M

𝜖-approximations of trajectories

• A symbolic trajectory ξ’ is an 𝜖- approximation of ξ 𝜇 (the symbolic trajectory gen. by 𝜇) iff: – ξ 𝜇(k)= ξ’ 𝜇(k) for all 0≤ k ≤ K𝜖 ,– For all k> K𝜖 , ξ’ (k) and ξ (k) belong to the same final class.

Two approximate verification problems

• Determine if– M, Din ⊨𝜖 φ , i.e.,

for every 𝜇 ∈ Din , there exists an 𝜖-approximation of ξ 𝜇 in L φ.– M, Din ⊨𝜖 φ , i.e.,

for every 𝜇 ∈ Din , every 𝜖-approximation of ξ 𝜇 is in L φ.


• Proposition– M, Din ⊨𝜖 φ implies LM ⊆ L φ– M, Din ⊭𝜖 φ implies LM ⊈ L φ


• If M, Din ⊨𝜖 φ and M, Din ⊭𝜖 φ then M 𝜖- approximately meets φ.

• If we want to do better we can set 𝜖’ = 𝜖/2 and iterate (incremental overhead).

The main result

• Theorem:Checking M, Din ⊨𝜖 φ and M, Din ⊨𝜖 φ are both effectively solvable problems.

The main result

• The proof first assumes a single initial distributions and proceeds to establish the theorem for:– Irreducible aperiodic chains– Irreducible periodic chains.– General chains

• This is then extended to a set of initial distributions D in.• The main step is the transient + steady state

characterization of the symbolic dynamics.

The main result

Additional results

• Convex hull of a finite set of distributions can also be handled.

• The atomic propositions can be constraints expressed in the first order theory of reals.

• Restrictions on the eigenvalues can yield -regular behaviors (egs.: distinct and real)

Summary

• Discretizing the values space [0, 1] leads to an interesting and useful symbolic dynamics of finite state Markov chains.

• Extensions:– Optimizations– Implementations– Case studies– Interval Markov chains

symbolic dynamics of m arkov chains

Documents

symbolic dynamicsdiscretize

probability value space

stationary distributions

views of markov chainsview

stochastic matrix

probability measure

path space

focus finite state