Lecture 6

Calculating Pn – how do we raise a matrix to the nth power?

Ergodicity in Markov Chains. When does a chain have equilibrium probabilities?

Balance Equations Calculating equilibrium probabilities without the fuss.

The leaky bucket queue Finally an example which is to do with networks.

For more information: Norris: Markov Chains (Chapter 1) Bertsekas: Appendix A and Section 6.3

How to calculate Pn

If P is diagonalisable (3x3) then we can find some invertible matrix such that:

P=U [ λ 1 0 0

0 λ 2 0

0 0 λ 3]U−1

Pn=U [ λ 1n 0 0

0 λ 2n 0

0 0 λ 3n]U−1

where i are the eigenvalues

Therefore pij(n)=A1

n+B2n+ C3

n

assuming the eigenvalues are distinct

1. For an M state chain. Compute the eigenvalues 1,2,.. M

2. If the eigenvalues are distinct then pij(n) has the general

form:3. If an eigenvalue is repeated once then the general

form includes a term (an+b)n

4. As roots of a polynomial with real coefficients, complex eigenvalues come in conjugate pairs and can be written as sin and cosine pairs.

5. The coefficients of the general form can be found by calculating pij

(n) by hand for n= 0...M-1 and solving.

General Procedure

pij n =a1 λ

1n+aM λM n

Example of Pn

P=[ 0 1 00 1/2 1/2

1/2 0 1/2 ] (where states are no’s 1, 2 and 3)

0=det P− xI =x x −12

2−14

=14

x−1 4x 21

Eigenvalues are 1, i/2, -i/2. Therefore p11(n) has the form:

p11 n =a+b i

2 n

+c − i2

n

=α + 12

n

[β cosnπ

2+γ sin

nπ2 ]

where the substitution can be made since p11(n) must be real

we can calculate that p11(0)=1, p11

(1)=0 and p11(2)=0

where I is the identity matrix

Example of Pn (2)

We now have three simultaneous equations in , and .

Solving we get =1/5, =4/5 and =-2/5.

1=α +β

0=α +12

γ

0=α −14

β

p11 n =

15 1

2 n

[ 45

cosnπ

2−

25

sinnπ2 ]

Equilibrium Probabilities

Recall the distribution vector of equilibrium probabilities. If n is the distribution vector after n steps is given by:

This is also the distribution which solves: When does this limit exist? When is there a unique

solution to the equation? This is when the chain is ergodic:

Irreducible Recurrent non-null (also called positive recurrent) Aperiodic

λ n ¿π =π P

Irreducible

A chain is irreducible if any state can be reached from any other.

More formally for all i and j:

∃ n:p ij n 0

1

0

2

1- 1-

1For what values of and is this chain irreducible?

Aperiodic chains

A state i is periodic if it is returned to after a time period > 1.

Formally, it is periodic if there exists an integer k > 1 where, for all j:

Equivalently, a state is aperiodic if there is always a sufficiently large n that for all m > n:

piin { 0 n=kj

=0 otherwise}

pii m 0

A useful aperiodicity lemma

If P is irreducible and has one aperiodic state i then all states are aperiodic. Proof:

By irreducibility there exists r, s 0 with

pji(r),pik

(s) > 0Therefore there is an n such that for all m > n:

And therefore all the states are aperiodic (consider j=k in the above equation).

p jk r+m+s ≥ p ji

r pii m pik

s 0

Return (Recurrence) Time

If a chain is in state i when will it next return to state i? This is known as “return time”. First we must define the probability that the first return

to state i is after n steps: fi(n)

The probability that we ever return is: A state where fi = 1 is recurrent fi < 1 is called transient.

The expectation of this is the “mean recurrence time” or “mean return time”.

Mi= recurrent null Mi< recurrent non-null

M i=∑n=1

∞

nf i n

f i=∑n=1

∞

f i n

Return (Recurrence) Time

A finite irreducible chain is always recurrent non null. In an irreducible aperiodic Markov Chain the limiting

probabilities

always exist and are independent of the starting distribution. Either:

1. All states are transient or recurrent null in which case j=0 for all states and no stationary distribution exists.

2. All states are recurrent non null and a unique stationary distribution exists with: π j=

1M j

λ n ¿

Ergodicity (summary)

A chain which is irreducible, aperiodic and recurrent non-null is ergodic.

If a chain is ergodic, then there is a unique invariant distribution which is equivalent to the limit:

In Markov Chain theory, the phrases invariant, equilibrium and stationary are often used interchangeably.

λ n ¿

Invariant Density in Periodic Chains

It is worth noting that an irreducible, recurrent non null chain which is periodic, has a solution to the invariant density equation but the limit distribution does not exist. Consider:

However, it should be clear that does not exist in general though it may for specific starting distributions

0 11

1

=( ½ , ½ ) solves =P

P=[ 0 11 0 ]

λ n ¿

Balance Equations

Sometimes it is not practical to calculate the equilibrium probabilities using the limit.

If a distribution is invariant then at every iteration, the inputs to a state must add up to its starting probability.

The inputs to a state i are the probabilities of each state j (j) which leads into it multiplied by the probability pji

Balance Equations (2)

More formally if i is the probability of state i :

And to ensure it is a distribution:

Which, for an n state chain gives us n+1 equations for n unknowns.

∑i=0

∞

π i=1

π i=∑j= 0

∞

p ji π j

Queuing Analysis of the Leaky Bucket

A “leaky bucket” is a mechanism for managing buffers to smooth the downstream flow.

What is described here is what is sometimes called a “token bucket”.

A queue holds a stock of “permits” which arrive at a rate r (one every 1/r seconds) up to W permits may be held.

A packet cannot leave the queue if there is no permit stored.

The idea is that the scheme limits downstream flow but can deal with bursts of traffic.

Modelling the Leaky Bucket

Let us assume that the arrival process is a Poisson process with a rate

Consider how many packets arrive in 1/r seconds. The prob ak that k packets arrive is:

ak=e−λ /r λ /r k

k!

Queue of packets (Poisson)

Queue ofpermits (arriveat 1/r seconds)

Exit queue for packets with permits

Exit ofbuffer

A Markov Model

Model this as a Markov Chain which changes state every 1/r seconds.

States 0iW represent no packets waiting and W-i permits available. States W+i (where i > 1) represent 0 permits and i packets waiting.

Transition probabilities:

0 1 2 W W+1a2 a2 a2. . . . . .a0 a0

a0

a1 a1 a1 a1a0+a1

pij={a0 +a1 i=j= 0

a j−i+1 i≤ j+ 1

0 otherwise}

Solving the Markov Model

By solving the balance equations we get:

π 0=a0 π 1a0+a1π 0 π i=∑j= 0

i+ 1

ai− j+ 1 π j i≥1

π 1=a2 π 0+a1π 1+a0 π 2π 1=1−a0−a1 π 0 /a0

π 2=π 0

a0 1−a0−a1 1−a1

a0

−a2Similarly, we can get expressions for 3 in termsof 2 ,1 and 0. And so on...

Solving the Markov Model (2)

Normally we would solve this using the remaining balance equation:

This is difficult analytically in this case. Instead we note that permits are generated every step

except when we are in state 0 and no packets arrive (W permits none used).

This means permits are generated at a rate (1-0a0)r

This must be equal to since each packet gets a permit (assume none dropped while waiting).

∑i=0

∞

π i=1

π 0=r− λ

ra0

And Finally

The average delay for a packet to get a permit is given by:

Of course this is not a closed form expression. To complete this analysis, look at Bertsekas P515

T=1r

∑j=W+1

∞

π j j−W

Time taken for eachiteration of chain

Amount of timespent in given state

No of iterationstaken to get outof queue fromstate j

For those states with queue