1

Modeling and Analyzing Fault-Tolerant, Real-Time Communication Protocols

Nancy LynchTheory of Distributed SystemsMIT

Second MURI WorkshopBerkeley, CaliforniaJune 4, 2001

2

MIT Participants

• Leaders: Nancy Lynch, Idit Keidar• Students: Carl Livadas, Roger Khazan, Ziv

Bar-Joseph• Collaborators: Paul Attie, Alex Shvartsman,

Roberto Segala, Frits Vaandrager

3

At Last Year’s Workshop…

4

General Models and Proof Methods

• I/O automaton models [Lynch, Tuttle 87]– Nondeterministic, infinite-state machines– Input/output/internal actions, traces – Modularity: Composition, levels of abstraction

• Mathematical, language-independent• Used to model distributed algorithms,

communication protocols• Validation, code generation,

upper and lower bounds

5

Timing, Hybrid Considerations

• Timing: TIOAs [Lynch, Vaandrager] – Timeout-based algorithms.– Local clocks, clock synchronization– Performance analysis

• Hybrid: HIOAs [L, Segala, V, Weinberg 96]– Real world + computer components– Continuous flows of data

6

Other Embellishments

• Probabilities: PIOA, PTIOA [Segala 95]– Probabilistic and nondeterministic behavior.– Randomized distributed algorithms– Systems with probabilistic assumptions

• Dynamic systems: DIOA [Attie, Lynch 99]– Run-time process creation and destruction,

mobility.– Agent systems

7

Communication Protocol Modeling and Analysis.

• At-most-once (AMO) Message Delivery • TCP, T/TCP• Reliable channels from unreliable channels • Self-stabilizing communication protocols • Network clock synchronization• Group communication systems

8

Group Communication Service

• Communication middleware• Manages group membership, current view• Handles joins, leaves, failures, partitions, merges• Multicast communication among members

– Multicasts respect views– Ordering, reliability constraints for message delivery,

e.g., FIFO, causal within each view.• Isis, Transis, Totem,…

9

VStoTO

VS

VStoTO VStoTO

TObcast

brcv

gpsnd

gprcvnewview

10

Conditional Performance Analysis• Assume VS satisfies:

– If a network component C stabilizes, then soon thereafter, views become consistent within C, and messages sent in the final view are delivered everywhere in C, within bounded time.

• And VStoTO satisfies:– Simple timing, fault-tolerance assumptions.

• Then TO satisfies:– If C stabilizes, then soon thereafter, any message

sent or delivered anywhere in C is delivered everywhere in C, within bounded time.

11

Conditional Performance Analysis• Give conditional claims about system performance

under particular assumptions about behavior of environment and of network substrate, e.g.:– Stabilization of underlying network.– Limited rate of change.– Bounds on message delay.– Limited amount of failure (number, density).– Limited input arrivals (number, density).

• Assumptions => Guarantees.• Get probabilistic statements as corollaries.• Composable

12

What we proposed:1. Model, analyze communication protocols. 2. Develop conditional performance analysis

techniques.3. Extend I/O automata theory to accommodate

performance, reliability, hybrid, probability, dynamic considerations.

4. Relate, integrate I/O automata with other frameworks.

13

Progress this year1. Communication protocol design/analysis

– Scalable Group Communication– Totally Ordered Multicast with QoS– Scalable Reliable Multicast

2. Conditional performance analysis methods– Evolving…

3. I/O automaton models– Hybrid I/O Automata– Dynamic I/O Automata– IOA language support

4. Comparing, integrating with other models– A start…

14

1. Protocol Modeling/Analysis

15

Scalable Group Communication[Keidar, Khazan 00]

16

Group Communication Service

• Manages group membership, current view.• Multicast communication among group members,

with ordering, reliability guarantees. • Virtual Synchrony [Birman, Joseph 87]

– Integrates group membership and group communication.– Processes that move together from one view to another deliver

the same messages in the first view.– Useful for replicated data management.– Before announcing new view, processes must synchronize,

exchange messages.

17

Example: Virtual Synchrony

3: i,j,k 3: i,j,k 3: i,j,k

4: i, j 4: i, j

mcast(m)

rcv(m)rcv(m)

VS algorithm supplies missing m

i j k

18

Group Communication in WANs

• Difficulties: – High message latency, message exchanges are expensive– Frequent connectivity changes

• New, scalable GC algorithm:– Uses scalable GM service of [Keidar, Sussman, et al. 00],

implemented on special membership servers.– GC (with virtual synchrony) implemented on clients.

VS

Net GM

VSGC

19

Group Communication in WANs• Try to minimize time from when network stabilizes until

GC delivers new views to clients.• After stabilization: GM forms view, VSGC algorithm

synchronizes.

• Existing systems (LANs):– GM, VSGC uses several message exchange rounds– Continue in spite of new network events

• Inappropriate for WANs

GM AlgorithmVSGC AlgorithmNet event

view(v)

20

New Algorithm• VSGC uses one message exchange round, in parallel with

GM’s agreement on views.• GM usually delivers views in one message exchange.

• Responds to new network events during reconfiguration:– GM produces new membership sets – VSGC responds to membership changes

• Distributed implementation [Tarashchanskiy 00]

GM AlgorithmVSGC AlgorithmNet event

view(v)

21

Correctness Proofs

• Models, proofs (safety and liveness)• Developed new incremental modeling, proof methods

[Keidar, Khazan, Lynch, Shvartsman 00]– Proof Extension Theorem:

• Used new methods for the safety proofs.

S S’

A A’

22

Performance Analysis

• Analyze time from when network stabilizes until GC delivers new views to clients.

• Compare with other strategies.

• System is a composition: – Network and GM services, plus– VSGC processes

• Use composition in the analysis.

23

Performance Analysis

1. Analyze the VSGC algorithm alone, in terms of its inputs and timing assumptions.

2. State reasonable performance guarantees for GM and Network.

3. Combine to get conditional performance properties for the system as a whole.

24

1. Analysis of VSGC algorithm

• Assume component C stabilizes:– GM delivers same views– Net provides reliable communication with latency .

• Let – T[start], T[view] be times of last GM events for C be upper bound on local step time.

• Then VSGC outputs new views by time max (T[start] + + x, T[view]) +

25

Analysis of VSGC Algorithm

GM algorithm

VS AlgorithmNet Event

+ x

T[start] T[view]

view(v)

view(v)startstart

26

2. Assumed Bounds for GM

• Bounds for “Fast Path” of [Keidar, et al. 00], observed empirically in almost all cases.

GM

T[start] T[view]

start start view(v)

27

VSGC + x

GM

T[start] T[view]start start view(v)

3. Combining VSGC and GM Bounds

• Bounds for system, conditional on GM bounds.

view(v)

28

Totally Ordered Multicast with QoS [Bar-Joseph, Keidar, Anker, Lynch 00]

29

Totally Ordered Multicast with QoS

• Multicast to dynamic group, subject to joins, leaves, and failures.

• Global total ordering of messages• QoS: Message delivery latency • Built on reliable network with latency guarantees• Add ordering guarantees, preserve latency bounds.• Target applications

– State machine replication– Military command and control– Distributed games– Shared editing

30

Two Algorithms• Algorithm 1: Basic Totally Ordered Multicast

– Sends, receives consistent with total ordering of messages.– Non-failing processes agree on messages from non-failing processes.– Latency: Constant, even with joins, leaves, failures.

• Algorithm 2: Atomic Multicast– Non-failing processes agree on all messages.– Latency:

• Joins, leaves only: Constant • With failures: Linear in f

Net

TOMfail_i fail_j

31

FrontEnd_i Memb_i

Sniffer_i

Net

Ord_i

rcv(m) joinleave

mcast(m)

join leave mcast(join)

mcast(leave)

rcv(join)rcv(leave)

mcast(m)

rcv(m)

progress(s,j)

joiners(s,J), leavers(s,J)end-slot(s)

members(s,J)

Local Node Process

32

Local Algorithm Operation

• FrontEnd divides time into slots, tags messages with slots.• Ord delivers messages by slot, in order of process indices.• Memb determines slot membership.

– Join, leave messages– Failures:

• Algorithm 1 uses local failure detector.• Algorithm 2 uses consensus on failures.

– Requires new dynamic version of consensus.

• Timing-dependent

33

Architecture for Algorithm 2

Net GM

TO-QoS

34

Performance Analysis (Planned)

1. Latency of TO-QoS in terms of GM2. GM latency bounds3. Combine

35

Using Caching to Improve Reliable Multicast Algorithms

[Livadas]

36

SRM [Floyd, et al.]

• Reliable multicast to dynamic group.• Built over IP multicast• Based on requests (NACKs) and retransmissions• Limits duplicate requests/replies using:

– Deterministic suppression: Ancestors suppress descendants, by scheduling requests/replies based on distance to source.

– Probabilistic suppression: Siblings suppress each other, by spreading out requests/replies.

37

SRM Architecture

IPMcast

SRM

38

New Protocol

• Tries to improve SRM by using loss history information.– Useful if future losses occur on same link.

• Uses deterministic suppression for siblings also• Determines, caches best requestor, best replier

– Chooses requestor closest to source.– Chooses replier closest to requestor.– Break ties with processor ids.

• Defaults to SRMRequestor

Replier

39

Performance• Metrics:

– Loss recovery latency: Time from detection of packet loss to receipt of first retransmission

– Loss recovery overhead: Number of messages multicast to recover from a message loss

• Protocol performance benefits:– Removes delays caused by probabilistic suppression– Following election of requestor and replier:

• Reduces latency by using best requestor and replier.• Reduces overhead by using single requestor and replier.

• Latency analysis (Planned)

40

3. I/O Automaton Models

41

Hybrid I/O Automata [Lynch, Segala, Vaandrager, HSCC 01]

• New, simpler version of HIOA model of [LSVW96]• Supports decomposing hybrid system descriptions:

– External behavior: Discrete actions and continuous flows– Composition: Synchronizes external actions and flows,

respects external behavior– Abstraction: Implementation and simulation relation

notions, respect external behavior.• Separate mechanisms:

– External actions for discrete communication.– External variables for continuous flow.

42

Example: Delay Buffer Del(d)

• Accepts discrete and continuous input, produces isomorphic output, with delay d.

• Compose in sequence, in cycle:

• Composition implements Del(d1 + d2):

Del(d1) Del(d2)Del(d1) Del(d2)

Del(d1) Del(d2)

Del(d1 + d2)

43

Example: Vehicle and Controller

• Keep vehicle speed in [v1, v2]. • Sensor senses velocity, reports to

Controller every time d.• Controller suggests acceleration.• Vehicle follows suggested acceleration,

with uncertainty ε. • Compose: Discrete, continuous

interactions• Prove invariant: velocity in [v1,v2].• Use auxiliary invariants, including timing.

Vehicle

Sensor Actuator

Controller

report(v)

vel-out

suggest(a)

acc-in

44

HOIA definition• U, X, Y: Input, output, internal (state) variables• Θ: Initial states• I, O, H: Input, output, internal actions• D, discrete transitions• T, trajectories

– Mappings from time intervals to valuations of variables• Closure properties• Input-enabling for actions, flows

• Execution: τ0, a1, τ1, a2, τ2, …• Trace: Restrict to external variables and actions

45

Composition and Abstraction

• Abstraction:– A implements B if comparable and traces(A) subset of traces(B).– Simulation relation: Start, step, trajectory conditions– Theorem: Simulation relation implies implementation

• Composition:– Synchronize external actions and variables– Theorems: Projection, pasting, substitutivity

• Receptiveness: – Doesn’t cooperative in producing Zeno behavior– Theorem: Closed under composition (with technical assumption).

46

Dynamic I/O Automata [Attie, Lynch, Concur 01]

• Dynamic version of I/O automata, including:– Automaton creation and destruction– Signature change

• Two-level model: Automata, configurations.• Mobility modeled using signature change

47

IOA Language and Tools

• Language for describing I/O automata: [Garland, Lynch]

• Front end: [Garland] – Translates to Java objects– Completely rewritten this year.– Needs support for composition.

• Theorem-prover connection: [Garland, Bogdanov]– Connection with LP– Seeking connections: SAL, Isabelle, STeP, NuPRL

IO A

48

IOA Language and Tools

• Simulator: [Chefter, Ramirez, Dean]– Has support for paired simulation.– Needs additions.– Being instrumented for invariant discovery using

Daikon [Ernst]• Code generator: Tauber, Tsai

– Local code-gen (translation to Java) running.– Needs composition, communication service calls,

correctness proof.• Challenge examples

49

Plans

50

Plans

1. Protocol modeling/verification– Finish analysis of Scalable GC, Totally Ordered

Multicast with QoS, SRM– Other protocols from this project.

2. Conditional analysis methods– Develop general methods– Compare with other methods (Trivedi)

51

Plans

3. I/O automaton models– Timed models:

• Composition theorems for timing properties• Specially structured TIOAs for CP analysis

– Hybrid models:• Finish basic model• Integrate control theory methods

– Probabilistic models:• Compositional analysis methods• Combine hybrid and probabilistic models

52

Plans

I/O automaton models, cont’d– Dynamic models: External behavior notion,

composition results– Combine extensions– Language constructs to support extensions– IOA tools: Finish, make portions available

4. Integration with other models/methods– Shared variable models

53

Thank you!

54

Best Requestor and Replier

S

Requestor

Replier

55

Latency Analysis (Planned)

1. Bound recovery latency assuming cache hits.2. Bound latency for cache misses.3. Combine4. Compare with SRM

56

1. Cache Hit Performance

• Recovery latency bounded by c tSA* + tAB + d tAB* + tBA, where:

– S = source, – A = requestor, B = retransmitter– c,d : delay parameters of deterministic suppression– tSA*, tAB*

: inter-host delay estimates– tAB ,, tBA : actual inter-host delays

57

2. Cache Miss Performance

• Depends on:– Loss location – Locations of requestor, retransmitter, …– Timing

• Needs analysis• Hope: Similar to SRM