virtualizing time

GOSSIP PROTOCOLSBIMODAL MULTICAST AND T-MANKen Birman

Gossip: A valuable tool… So-called gossip protocols can be robust

even when everything else is malfunctioning

Idea is to build a distributed protocol a bit like gossip among humans “Did you hear that Sally and John are going

out?” Gossip spreads like lightening…

Gossip: basic idea First rigorous analysis was by Alan

Demers! Node A encounters “randomly selected”

node B (might not be totally random) Gossip push (“rumor mongering”):

A tells B something B doesn’t know Gossip pull (“anti-entropy”)

A asks B for something it is trying to “find” Push-pull gossip

Combines both mechanisms

Definition: A gossip protocol… Uses [mostly] random pairwise state

merge Runs at a steady rate (much slower than

network) [mostly] Uses bounded-size messages Does not depend on messages getting

through reliably

Gossip benefits… and limitations Information flows around disruptions Scales very well Typically reacts to new events in log(N) Can be made self-repairing

Rather slow Very redundant Guarantees are at best probabilistic Depends heavily on the randomness of

the peer selection

For example We could use gossip to track the health

of system components We can use gossip to report when

something important happens In the remainder of today’s talk we’ll

focus on event notifications. Next week we’ll look at some of these other uses

Typical push-pull protocol Nodes have some form of database of

participating machines Could have a hacked bootstrap, then use

gossip to keep this up to date! Set a timer and when it goes off, select a

peer within the database Send it some form of “state digest” It responds with data you need and its own

state digest You respond with data it needs

Where did the “state” come from? The data eligible for gossip is usually

kept in some sort of table accessible to the gossip protocol

This way a separate thread can run the gossip protocol

It does upcalls to the application when incoming information is received

Gossip often runs over UDP Recall that UDP is an “unreliable”

datagram protocol supported in internet Unlike for TCP, data can be lost Also packets have a maximum size, usually 4k

or 8k bytes (you can control this) Larger packets are more likely to get lost!

What if a packet would get too large? Gossip layer needs to pick the most valuable

stuff to include, and leave out the rest!

Algorithms that use gossip Gossip can be used to…

Notify applications about some event Track the status of applications in a system Organize the nodes in some way (like into a

tree, or even sorted by some index) Find “things” (like files)

Let’s look closely at an example

Bimodal multicast This is Cornell work from about 15 years

ago

Goal is to combine gossip with UDP multicast to make a very robust multicast protocol

Stock Exchange Problem: Sometimes, someone is slow…

Most members are healthy….

… but one is slow

i.e. something is contending with the red process,delaying its handling of incoming messages…

Start by using unreliable UDP multicast to rapidly distribute the message. But some messages may not get through, and some processes may be faulty. So initial state involves partial distribution of multicast(s)

Remedy? Use gossip

Periodically (e.g. every 100ms) each process sends a digest describing its state to some randomly selected group member. The digest identifies messages. It doesn’t include them.

Remedy? Use gossip

Recipient checks the gossip digest against its own history and solicits a copy of any missing message from the process that sent the gossip

Remedy? Use gossip

Processes respond to solicitations received during a round of gossip by retransmitting the requested message. The round lasts much longer than a typical RPC time.

Remedy? Use gossip

Delivery? Garbage Collection? Deliver a message when it is in FIFO order

Report an unrecoverable loss if a gap persists for so long that recovery is deemed “impractical”

Garbage collect a message when you believe that no “healthy” process could still need a copy (we used to wait 10 rounds, but now are using gossip to detect this condition)

Match parameters to intended environment

Need to bound costs Worries:

Someone could fall behind and never catch up, endlessly loading everyone else

What if some process has lots of stuff others want and they bombard him with requests?

What about scalability in buffering and in list of members of the system, or costs of updating that list?

Optimizations Request retransmissions most recent

multicast first Idea is to “catch up quickly” leaving at

most one gap in the retrieved sequence

Optimizations Participants bound the amount of data

they will retransmit during any given round of gossip. If too much is solicited they ignore the excess requests

Optimizations Label each gossip message with senders

gossip round number Ignore solicitations that have expired

round number, reasoning that they arrived very late hence are probably no longer correct

Optimizations Don’t retransmit same message twice in

a row to any given destination (the copy may still be in transit hence request may be redundant)

Optimizations Use UDP multicast when retransmitting a

message if several processes lack a copy For example, if solicited twice Also, if a retransmission is received from “far

away” Tradeoff: excess messages versus low latency

Use regional TTL to restrict multicast scope

Scalability Protocol is scalable except for its use of

the membership of the full process group Updates could be costly Size of list could be costly In large groups, would also prefer not to

gossip over long high-latency links

Router overload problem Random gossip can overload a central

router Yet information flowing through this

router is of diminishing quality as rate of gossip rises

Insight: constant rate of gossip is achievable and adequate

0 10 20 30 40 50 60 70 80 90 1000

20

40

60

80

100

120

140

160

180

200

sec

nmsg

s/se

c

UniformHierarchical

0 5 10 15 20 25 300

50

100

150

group size

nmsg

s/se

c

UniformHierarchical

Sender

Receiver

High noise rate

Sender

Receiver

Limited bandwidth (1500Kbits)

Bandwidth of other links:10Mbits

Load

on

WA

N li

nk (m

sgs/

sec)

Late

ncy

to d

eliv

ery

(ms)

Hierarchical Gossip Weight gossip so that probability of

gossip to a remote cluster is smaller Can adjust weight to have constant load

on router Now propagation delays rise… but just

increase rate of gossip to compensate

0 5 10 15 20 25 300

500

1000

1500

group size

prop

agat

ion

time(

ms)

UniformHierarchicalFast hierarchical

0 5 10 15 20 25 300

50

100

150

group size

nmsg

s/se

c

UniformHierarchicalFast hierarchical

Idea behind analysis Can use the mathematics of epidemic

theory to predict reliability of the protocol

Assume an initial state Now look at result of running B rounds of

gossip: converges exponentially quickly towards atomic delivery

Pbcast bimodal delivery distribution

1.E-30

1.E-25

1.E-20

1.E-15

1.E-10

1.E-05

1.E+00

0 5 10 15 20 25 30 35 40 45 50

number of processes to deliver pbcast

p{#p

roce

sses

=k}

Either sender fails…

… or data gets through w.h.p.

Failure analysis Suppose someone tells me what they

hope to “avoid” Model as a predicate on final system

state Can compute the probability that pbcast

would terminate in that state, again from the model

Two predicates Predicate I: A faulty outcome is one

where more than 10% but less than 90% of the processes get the multicast

… Think of a probabilistic Byzantine General’s problem: a disaster if many but not most troops attack

Two predicates Predicate II: A faulty outcome is one where

roughly half get the multicast and failures might “conceal” true outcome

… this would make sense if using pbcast to distribute quorum-style updates to replicated data. The costly hence undesired outcome is the one where we need to rollback because outcome is “uncertain”

Two predicates Predicate I: More than 10% but less than

90% of the processes get the multicast Predicate II: Roughly half get the multicast

but crash failures might “conceal” outcome

Easy to add your own predicate. Our methodology supports any predicate over final system state

Scalability of Pbcast reliability

1.E-351.E-301.E-251.E-201.E-151.E-101.E-05

10 15 20 25 30 35 40 45 50 55 60

#processes in system

P{f

ailu

re}

Predicate I Predicate II

Ef fects of fanout on reliability

1.E-161.E-141.E-121.E-101.E-081.E-061.E-041.E-021.E+00

1 2 3 4 5 6 7 8 9 1 0

fanout

P{f

ailu

re}


Fanout required for a specif ied reliability

44.555.566.577.588.59

20 25 30 35 40 45 50


fano

ut

Predicate I f or 1E-8 reliability

Predicate II for 1E-12 reliability

Pbcast bimodal delivery distribution

1.E-30

1.E-25

1.E-20

1.E-15

1.E-10

1.E-05

1.E+00

0 5 10 15 20 25 30 35 40 45 50

number of processes to deliver pbcast

p{#p

roce

sses

=k}

Scalability of Pbcast reliability

1.E-351.E-301.E-251.E-201.E-151.E-101.E-05

10 15 20 25 30 35 40 45 50 55 60


P{f

ailu

re}


Ef fects of fanout on reliability

1.E-161.E-141.E-121.E-101.E-081.E-061.E-041.E-021.E+00

1 2 3 4 5 6 7 8 9 1 0

fanout

P{f

ailu

re}


Fanout required for a specif ied reliability

44.555.566.57

7.588.59

20 25 30 35 40 45 50

#processes in systemfa

nout

Predicate I f or 1E-8 reliability

Predicate II for 1E-12 reliability

Figure 5: Graphs of analytical results

Example of a question Create a group of 8 members Perturb one member in style of Figure 1 Now look at “stability” of throughput

Measure rate of received messages during periods of 100ms each

Plot histogram over life of experiment

Histogram of throughput for Ensemble's FIFO Virtual Synchrony Protocol

0

0.2

0.4

0.6

0.8

0.005

0.015

0.025

0.035

0.045

0.055

0.065

Inter-arrival spacing (ms)

Prob

abili

ty o

f occ

uren

ce

Traditional Protocolw ith .05 sleepprobability

Traditional Protocolw ith .45 sleepprobability

Histogram of throughput for pbcast

0

0.2

0.4

0.6

0.8

1

Inte r -arr ival spacing (m s)Pr

oba

bili

ty o

f oc

cure

nce

Pbcast w ith .05sleep probability

Pbcast w ith .45sleep probability

Source to dest latency distributions

Notice that in practice, bimodal multicast is fast!

Now revisit Figure 1 in detail Take 8 machines Perturb 1 Pump data in at varying rates, look at

rate of received messages

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90

20

40

60

80

100

120

140

160

180

200Low bandwidth comparison of pbcast performance at faulty and correct hosts

perturb rate

aver

age

thro

ughp

ut

traditional w/1 perturbed pbcast w/1 perturbed throughput for traditional, measured at perturbed hostthroughput for pbcast measured at perturbed host

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90

20

40

60

80

100

120

140

160

180

200High bandwidth comparison of pbcast performance at faulty and correct hosts

perturb rate

aver

age

thro

ughp

ut

traditional: at unperturbed hostpbcast: at unperturbed host traditional: at perturbed host pbcast: at perturbed host

Revisit our original scenario with perturbations (32 processes)

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5180

185

190

195

200

205

210

215

220mean and standard deviation of pbcast throughput: 16-member group

perturb rate

thro

ughp

ut (m

sgs/

sec)

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5180

185

190

195

200

205

210

215


perturb rate

thro

ughp

ut (m

sgs/

sec)

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5180

185

190

195

200

205

210

215


perturb rate

thro

ughp

ut (m

sgs/

sec)

0 50 100 1500

50

100

150standard deviation of pbcast throughput

process group size

stan

dard

dev

iatio

n

Throughput variation as a function of scale

Impact of packet loss on reliability and retransmission rate

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

10

20

30

40

50

60

70

80

90

100Pbcast background overhead: perturbed process percentage (25%)

perturb rate

retra

nsm

itted

mes

sage

s (%

)

8 nodes 16 nodes 64 nodes 128 nodes

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.20

20

40

60

80

100

120

140

160

180

200

Pbcast with system-wide message loss: high and low bandwidth

system-wide drop rate

aver

age

thro

ughp

ut o

f rec

eive

rs

hbw:8hbw:32hbw:64hbw:96lbw:8lbw:32lbw:64lbw:96

Notice that when network becomes overloaded, healthy processes experience packet loss!

What about growth of overhead? Look at messages other than original

data distribution multicast Measure worst case scenario: costs at

main generator of multicasts Side remark: all of these graphs look

identical with multiple senders or if overhead is measured elsewhere….

64 nodes - 16 perturbed processes

020406080

100

0.1 0.2 0.3 0.4 0.5

perturb rate

aver

age

over

head


020406080

100

0.1 0.2 0.3 0.4 0.5

perturb rate

aver

age

over

head


020406080

100

0.1 0.2 0.3 0.4 0.5

perturb rate

aver

age

over

head

(m

sgs/

sec)


020406080

100

0.1 0.2 0.3 0.4 0.5

perturb rate

aver

age

over

head

Growth of Overhead? Clearly, overhead does grow We know it will be bounded except for

probabilistic phenomena At peak, load is still fairly low

Pbcast versus SRM, 0.1% packet loss rate on all links

0 10 20 30 40 50 60 70 80 90 1000

5

10

15PBCAST and SRM with system wide constant noise, tree topology

group size

requ

ests

/sec

rec

eive

d

SRM

Pbcast

adaptive SRM

Pbcast-IPMC

0 10 20 30 40 50 60 70 80 90 1000

5

10


group size

repa

irs/s

ec re

ceive

d

SRM

Pbcast

adaptive SRM

Pbcast-IPMC

0 10 20 30 40 50 60 70 80 90 1000

5

10

15PBCAST and SRM with system wide constant noise, star topology

group size

requ

ests

/sec

rece

ived SRM

Pbcast

adaptive SRM

Pbcast-IPMC

0 10 20 30 40 50 60 70 80 90 1000

10

20

30

40

50

60PBCAST and SRM with system wide constant noise, star topology

group size

repa

irs/s

ec re

ceive

d

SRM

Pbcast

adaptive SRM

Pbcast-IPMC

Tree networks

Star networks

Pbcast versus SRM: link utilization

0 10 20 30 40 50 60 70 80 90 1000

5

10

15

20

25

30

35

40

45


group size

link

utili

zatio

n on

an

outg

oing

link

from

sen

der

PbcastPbcast-IPMCSRMAdaptive SRM

0 10 20 30 40 50 60 70 80 90 1000

2

4

6

8

10

12

14

16

18


group size

link

utili

zatio

n on

an

inco

min

g lin

k to

sen

der

PbcastPbcast-IPMCSRMAdaptive SRM

Pbcast versus SRM: 300 members on a 1000-node tree, 0.1% packet loss rate

0 20 40 60 80 100 1200

5

10

15

20

25

30Pbcast and SRM with 0.1% system wide constant noise, 1000-node tree topology

group size

requ

ests

/sec

rece

ived

SRM

Pbcast

adaptive SRM

Pbcast-IPMC

0 20 40 60 80 100 1200

5

10

15

20

25

30Pbcast and SRM with 0.1% system wide constant noise, 1000-node tree topology

group size

repa

irs/s

ec re

ceive

d SRM

Pbcast

adaptive SRM

Pbcast-IPMC

Pbcast Versus SRM: Interarrival Spacing

Pbcast versus SRM: Interarrival spacing (500 nodes, 300 members, 1.0% packet loss)

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

10

20

30

40

50

60

70

80

90

1001% system wide noise, 500 nodes, 300 members, pbcast-grb

latency at node level (second)

perc

enta

ge o

f occ

urre

nces

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

10

20

30

40

50

60

70

80

90

1001% system wide noise, 500 nodes, 300 members, pbcast-grb

latency after FIFO ordering (second)

perc

enta

ge o

f occ

urre

nces

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

10

20

30

40

50

60

70

80

90

1001% system wide noise, 500 nodes, 300 members, srm


perc

enta

ge o

f occ

urre

nces

Real Data: Bimodal Multicast on a 10Mbit ethernet (35 Ultrasparc’s)

0 10 20 30 40 50 60 70 80 90 1000

20

40

60

80

100

120

140

160

180

200

sec0 10 20 30 40 50 60 70 80 90 100

50

60

70

80

90

100

110

120

130

140

150

sec

#msg

s

Injected noise, retransmission limit disabled

Injected noise, retransmission limit re-enabled

Networks structured as clusters

Sender

Receiver

High noise rate

Sender

Receiver

Limited bandwidth (1500Kbits)

Bandwidth of other links:10Mbits

Delivery latency in a 2-cluster LAN, 50% noise between clusters, 1% elsewhere

0 1 2 3 4 5 60

5

10

15

20

25

30

35

40

45partitioned nw, 50% noise between clusters, 1% system wide noise, n=80, pbcast-grb


perc

enta

ge o

f occ

urre

nces

0 1 2 3 4 5 60

5

10

15

20

25

30

35

40

45partitioned nw, 50% noise between clusters, 1% system wide noise, n=80, pbcast-grb


perc

enta

ge o

f occ

urre

nces

0 1 2 3 4 5 60

5

10

15

20

25

30

35

40

45srm


perc

enta

ge o

f occ

urre

nces

0 1 2 3 4 5 60

5

10

15

20

25

30

35

40

45srm adaptive


perc

enta

ge o

f occ

urre

nces

Requests/repairs and latencies with bounded router bandwidth

0 20 40 60 80 100 1200

20

40

60

80

100

120

140

160

180

200limited bw on router, 1% system wide constant noise

group size

requ

ests

/sec

rece

ived

pbcast-grbsrm

0 20 40 60 80 100 1200

20

40

60

80

100

120limited bw on router, 1% system wide constant noise

group size

repa

irs/s

ec re

ceive

d

pbcast-grbsrm

0 2 4 6 8 10 12 14 16 18 200

5

10

15limited bw on router, noise=1%, n=100, pbcast-grb


perc

enta

ge o

f occ

urre

nces

0 2 4 6 8 10 12 14 16 18 200

5

10

15limited bw on router, noise=1%, n=100, pbcast-grb


perc

enta

ge o

f occ

urre

nces

0 2 4 6 8 10 12 14 16 18 200

5

10

15limited bw on router, noise=1%, n=100, srm


perc

enta

ge o

f occ

urre

nces

Ozalp Babaoglu

Gossipping in Bologna

58

Background• 2003: Márk Jelasity brings the gossipping

gospel to Bologna from Amsterdam• 2003-2006: We get good milage from

gossipping in the context of Project BISON

• 2005-present: Continue to get milage in the context of Project DELIS

59

What have we done?• We have used gossipping to obtain fast,

robust, decentralized solutions for• Aggregation• Overlay topology management• Heartbeat synchronization• Cooperation in selfish environments

60

Collaborators

• Márk Jelasity• Alberto Montresor• Gianpaolo Jesi• Toni Binci• David Hales• Stefano Arteconi

Proactive gossip framework// active threaddo forever wait(T time units) q = SelectPeer() push S to q pull Sq from q S = Update(S,Sq)

// passive threaddo forever (p,Sp) = pull * from * push S to p S = Update(S,Sp)

62

Proactive gossip framework• To instantiate the framework, need to

define• Local state S• Method SelectPeer()• Style of interaction

push-pullpushpull

• Method Update()

#1Aggregation

64

Gossip framework instantiation• Style of interaction: push-pull• Local state S: Current estimate of global

aggregate• Method SelectPeer(): Single random

neighbor• Method Update(): Numerical function

defined according to desired global aggregate (arithmetic/geometric mean, min, max, etc.)

65

Exponential convergence of averaging

66

Properties of gossip-based aggregation

• In gossip-based averaging, if the selected peer is a globally random sample, then the variance of the set of estimates decreases exponentially

• Convergence factor:

E( i1

2 )E( i

2)

12 e

0.303

67

Robustness of network size estimation

1000 nodes crash at the beginning of each cycle

68

Robustness of network size estimation

20% of messages are lost

#2Topology Management

70

Gossip framework instantiation• Style of interaction: push-pull• Local state S: Current neighbor set• Method SelectPeer(): Single random

neighbor• Method Update(): Ranking function

defined according to desired topology (ring, mesh, torus, DHT, etc.)

71

Mesh Example

Mesh.mov

72

Sorting example

Line.mov

73

Exponential convergence - time

Exponential convergence - network size

#3Heartbeat Synchronization

76

Synchrony in nature• Nature displays astonishing cases of

synchrony among independent actors• Heart pacemaker cells• Chirping crickets• Menstrual cycle of women living together• Flashing of fireflies

• Actors may belong to the same organism or they may be parts of different organisms

77

Coupled oscillators• The “Coupled oscillator” model can be

used to explain the phenomenon of “self-synchronization”

• Each actor is an independent “oscillator”, like a pendulum

• Oscillators coupled through their environment • Mechanical vibrations• Air pressure• Visual clues• Olfactory signals

• They influence each other, causing minor local adjustments that result in global synchrony

78

Fireflies• Certain species of (male) fireflies (e.g.,

luciola pupilla) are known to synchronize their flashes despite:• Small connectivity (each firefly has a small

number of “neighbors”)• Communication not instantaneous• Independent local “clocks” with random

initial periods

79

Gossip framework instantiation• Style of interaction: push• Local state S: Current phase of local

oscillator• Method SelectPeer(): (small) set of

random neighbors• Method Update(): Function to reset the

local oscillator based on the phase of arriving flash

80

Experimental results

fireflies.mov

81

Exponential convergence

#4Cooperation in Selfish

Environments

83

Outline• P2P networks are usually open systems

• Possibility to free-ride• High levels of free-riding can seriously

degrade global performance• A gossip-based algorithm can be used to

sustain high levels of cooperation despite selfish nodes

• Based on simple “copy” and “rewire” operations

84

Gossip framework instantiation• Style of interaction: pull• Local state S: Current utility, strategy

and neighborhood within an interaction network

• Method SelectPeer(): Single random sample

• Method Update(): Copy strategy and neighborhood if the peer is achieving better utility

85

A

“Copy” strategy

SLAC Algorithm: “Copy and Rewire”

B

CA

DE F

H

JK

G

Compare utilities

“Rewire”

86

A

“Mutate” strategy

A

SLAC Algorithm: “Mutate”

B

C

DE F

H

JK

G

Drop current links

Link to random node

87

Prisoner’s Dilemma• Prisoner’s Dilemma in SLAC

• Nodes play PD with neighbors chosen randomly in the interaction network

• Only pure strategies (always C or always D)• Strategy mutation: flip current strategy• Utility: average payoff achieved

88

Cycle 180: Small defective clusters

89

Cycle 220: Cooperation emerges

90

Cycle 230:Cooperating cluster starts to break apart

91

Cycle 300: Defective nodes isolated, small cooperative clusters formed

92

Phase transition of cooperation

% o

f coo

pera

ting

node

s

93

Broadcast Application• How to communicate a piece of

information from a single node to all other nodes

• While:• Minimizing the number of messages sent

(MC)• Maximizing the percentage of nodes that

receive the message (NR)• Minimizing the elapsed time (TR)

94

Broadcast Application• Given a network with N nodes and L links

• A spanning tree has MC = N• A flood-fill algorithm has MC = L

• For fixed networks containing reliable nodes, it is possible to use an initial flood-fill to build a spanning tree from any node

• Practical if broadcasting initiated by a few nodes only

• In P2P applications this is not practical due to network dynamicity and the fact that all nodes may need to broadcast

95

The broadcast game• Node initiates a broadcast by sending a

message to each neighbor• Two different node behaviors determine

what happens when they receive a message for the first time:• Pass: Forward the message to all neighbors• Drop: Do nothing

• Utilities are updated as follows:• Nodes that receive the message gain a

benefit β• Nodes that pass the message incur a cost γ• Assume β > γ > 0, indicating nodes have

an incentive to receive messages but also an incentive to not forward them

96

1000-node static random network

97

1000-node high churn network

98

Fixed random networkAverage over 500 broadcasts x 10 runs

99

High churn networkAverage over 500 broadcasts x 10 runs

100

Some food for thought• What is it that makes a protocol “gossip

based”?• Cyclic execution structure (whether

proactive or reactive)• Bounded information exchange per peer,

per cycle• Bounded number of peers per cycle• Random selection of peer(s)

101

Some food for thought• Bounded information exchange per peer,

per round implies• Information condensation — aggregation

• Is aggregation the mother of all gossip protocols?

102

Some food for thought• Is exponential convergence a universal

characterization of all gossip protocols?• No, depends on the properties of the

peer selection step• What are the minimum properties for

peer selection that are necessary to guarantee exponential convergence?

Gossip versus evolutionary computing • What is the relationship between gossip

and evolutionary computing?• Is one more powerful than the other?

Are they equal?

virtualizing time

Documents