8/12/2019 RW_Intro

1/2

Random walks

April 18, 2014

1 Markov chainsExercise 1.1. Describe the Poisson process as a Markov chain in continuoustime.

2 Walks on graphs

A graph is a pair G= (V,E). The symmetry assumption is usually phrasedby saying that the graph is undirected or that its edges are unoriented.

There is a well-known and easily established correspondence between elec-

trical networks and random walks that holds for all graphs. Namely, given afinite connected graph G with conductances assigned to the edges, we con-sider the random walk that can go from a vertex only to an adjacent vertexand whose transition probabilities from a vertex are proportional to the con-ductances along the edges to be taken. That is, ifxis a vertex with neighborsy1, . . . , yd, and the conductance of the edge (x, yi) is ci, then

p(x, yj) = cjd

i=1 ci.

In fact, we are interested only in reversible Markov chains, where we call

a Markov chain reversible if there is a positive function x (x) on thestate space such that the transition probabilities satisfy pi(x)pxy =pi(y)pyxfor all pairs of states x, y. (Such a function() will then provide a stationarymeasure. Note that () is not generally a probability measure.)

In this case, make a graph G by taking the states of the Markov chainfor vertices and joining two vertices x, y by an edge when pxy > 0. Assign

1

8/12/2019 RW_Intro

2/2

weight

c(x, y) :=(x)pxy

to that edge; note that the condition of reversibility ensures that this weightis the same no matter in what order we take the endpoints of the edge. Withthis network in hand, the Markov chain may be described as a random walkonG: when the walk is at a vertex x, it chooses randomly among the verticesadjacent tox with transition probabilities proportional to the weights of theedges. Conversely, every connected graph with weights on the edges suchthat the sum of the weights incident to every vertex is finite gives rise to arandom walk with transition probabilities proportional to the weights. Sucha random walk is an irreducible reversible Markov chain: define (x) to be

the sum of the weights incident to x.

Exercise 2.1. Show that random walk on a connected weighted graph G ispositive recurrent (i.e., has a stationary probability distribution) if and onlyif

x,yc(x, y) < 1, in which case the stationary probability distribution isproportional to(). Show that if the random walk is not positive recurrent,then() is a stationary infinite measure.

3 Queues

Exercise 3.1. Describe a simple Markov queue as a random walk.

Recall that the queue receives a Poisson inflow of rate and the servicetime is exponentially distributed with service rate . The queue behavesdifferently in cases > , = , and < .

Note that in all these cases the chain is called reversible, though for = and > there is no equilibrium probability distribution! Still there existsan equilibrium measure on the state space {0, 1, . . .}.

Exercise 3.2. Find this measure.

2