synchronization in message passing systems

Ph. D. Thesis Defense1 May 7, 2004

Synchronization in Message Passing Systems

by Ye Su

Advisor: Dr. Gurdip Singh

Department of Computing and Information Sciences

Kansas State University

May 7, 2004 Ph.D. Thesis Defense2

Outline

1. Introduction to region synchronization problem.2. Brief review of the aspect oriented-based methodology.3. Correctness criteria for the region synchronization algorithm

in distributed systems.4. Algorithm to map the coarse-grained solution to fine-grained

solution in the point-to-point networks.5. Algorithm Optimizations in the point-to-point networks.6. Algorithm to map the coarse-grained solution to fine-grained

solution in the CAN based Systems.7. Integrating our solutions to SyncGen toolset8. solving a complicated example by using the approach we

proposed.9. Summary and future work


1. Introduction

- Regions

- Synchronization

• An aspect oriented approach for developing synchronization for shared memory systems is proposed by Mizuno, Singh and Neilsen.

• Similar as their approach, we focus on the technique to derive algorithms for synchronization in message passing systems.

P1 P3P2- Processes


2. Overview of the aspect oriented-based methodology


Overview of the aspect oriented-based methodology

Identifying synchronization regions

void reader() { void writer(){

while (true) { while (true) {

...other computation.... …other computation

/*** Region-Enter: Reader ***/ /*** Region-Enter: Writer ***/

...read shared variables.... …write shared variables…

/*** Region-Exit: Reader ***/ /*** Region-Exit: Writer ***/

...other computation... ...other computation...

} }

} }



Global invariant specification– A global invariant I is a predicate defined using in and out

counters with arithmetic inequalities, arithmetic operators and

boolean connectives.

RR RW

Reader Writer

In[R] In[W]

out[R] out[W]

((in[R]=out[R])(in[W]=out[W]))(in[W]-out[W]≤1)



Generation of coarse-grained solution– Two types of synchronization constructs, <S> and <await B

S>, are used in a coarse-grained solution.

Reader region:Entry: < await (in[W] = out[W]) in[R]++ >Exit: < out[R]++ >

Writer region:Entry: < await ((in[R] = out[R]) /\ (in[W] = out[W])) in[W]++ > Exit: < out[W]++ >



Translation to synchronization code– fine-grained synchronization code in a target programming

language or platform is obtained from the coarse-grained solution.

– Techniques to map coarse-grained solutions to multi-threaded programs based on monitors [And91] and Java synchronized blocks [Miz99] have been proposed.

– In this thesis, we will focus on how to map a coarse-grained solution to fine-grained solutions in message passing based systems.



Weaving the code– The final step in the methodology is to weave the

synchronization code to functional code. – For example, in the active monitor approach, the

monitor code and the code for the proxies are generated automatically. Furthermore, appropriate method calls are inserted at appropriate points in the functional code.


3. Correctness Criteria

In a distributed program, we define a synchronization statement Syni, associated with entry as well as exit for each region.

Syni is one of the forms:– < Ci++ >

– < await (Bi) Ci++ >Where Ci is the in[x] or out[x] for some region Rx and Bi is composed of local variables.


Correctness Criteria

A simple centralized solution

P1 P2 Pn

Central Site (Pc)

……request

reply



A distributed solution

P2P1 Pn

Mon1 Mon2 Monn

Message passing system

……request

reply



A counter example P1 P2 P3

reqa

reqb

reqc

Sync

Syna

Synb

Sync

Syna

Synb

Real Time

t

inconsistent

Sync

Syna

Synb

Sync

Synb

Syna



• P’: A virtual process executes every process’ synchronization statement in real time.

• Definition: An algorithm A solves the region synchronization problem for invariant I if I' is an invariant of A.

P’ Pi Pj

In[R]’++

out[R]’++

In[W]’++

In[R]++

out[R]++

In[W]++

I((in[R]=out[R])(in[W]=out[W]))(in[W]-out[w]≤1)

I’((in[R]’=out[R]’)(in[W]’=out[W]’))(in[W]’-out[w]’≤1)

Auxiliary shared variable

Local counter variable


4. The algorithm for a point-to-point network

Happened Before () Ea Eb if– Ea and Eb are events in the same process, and Ea occurred

before Eb.– Ea is the event of sending a message in a process, Pi, and

Eb is the event of receiving the same message in another process Pj.

– Ea Ec, and Ec Eb. Total ordering of events.

– Ea tm Eb, if and only if (TMEa < TMEb) or (TMEa = TMEb) /\ i < j) where TMEa is the timestamp for event Ea.

– Ea t Eb, if Ea occurs before Eb in real time


Definitions

Seq is a sequence of statements, each of which increments a counter.

Seq1 || ... || Seqn denotes the concurrent execution of the sequences.

{P}Seq{Q} holds if, whenever the execution of Seq begins in a state satisfying P and the execution of Seq terminates the resulting state satisfies Q.

……Seq…….

QP


Definitions

The weakest precondition, wp(Seq,Q), is a predicate defining the largest set of states such that the execution of Seq in any state satisfying wp(Seq,Q) results in a state satisfying Q.

The strongest postcondition, sp(P,Seq), is a predicate defining the smallest set of states such that the execution of Seq with precondition P results in a state satisfying sp(P,Seq).


Definitions

Let Seqi and Seqj be two sequences in P and I be a global

invariant of P. – If there exists Pi such that Pi wp(Seqi, I) is true but {Pi} Seqi ||

Seqj {I} does not hold then we say Seqj conflicts with Seqi with respect to I and we denote it by Seqj cf Seqi.

– If there exists Pi such that Pi wp(Seqi, I) is false but Pi wp(Seqj;Seqi,I) is true then we say Seqj enables Seqi with respect to I and we denote it by Seqj en Seqi.

……Seqi……

Pi I=true

……Seqj;Seqi……

Pi I=true?

……Seqi……

Pi I≠true

……Seqj;Seqi……

Pi I=true


Notations

reqi and reqj are requests to execute Syni and Synj respectively.

ex_reqj,x denotes the event of Px executing reqi, and ex_reqi denotes the event of local execution of reqi.

reqireqk

ex_reqk,xex_reqk

ex_reqi,y

ex_reqiex_reqi

ex_reqk

P’ Px Py


The algorithm for a point-to-point network

Let reqi be a request issued by Pk. We now define a set of rules that a process may follow to execute this request.

– R1: reqj, where j ≠ i, if reqj cf reqi (reqj en reqi ), then ex_reqj t ex_reqi ex_reqj,k t ex_reqi.

– R2: reqj, where j ≠ i, if reqj en reqi (reqj cf reqi) then ex_reqj,k t ex_reqi ex_reqj t ex_reqi.

– R3: reqj, where j ≠ i, if reqj cf reqi reqj en reqi, then ex_reqj,k t ex_reqi ex_reqj t ex_reqi.



R1: reqj, where j ≠ i, if reqj cf reqi (reqj en reqi ), then ex_reqj t ex_reqi ex_reqj,k t ex_reqi.

P’ Pk Pl

ex_reqj ex_reqj

ex_reqiex_reqi

Pk

ex_reqi

ex_reqj,k



R2: reqj, where j ≠ i, if reqj en reqi (reqj cf reqi) then ex_reqj,k t ex_reqi ex_reqj t ex_reqi.

Pk

ex_reqi

ex_reqj,k

P’ Pk Pl

ex_reqj

ex_reqiex_reqi

ex_reqj



R3: reqj, where j ≠ i, if reqj cf reqi reqj en reqi, then ex_reqj,k t ex_reqi ex_reqj t ex_reqi.

Pk

ex_reqi

ex_reqj,k

P’ Pk Pl

ex_reqj

ex_reqiex_reqi

ex_reqj



Theorem: If all execution sequences of P satisfy R1, R2 and R3, then P is consistent.

P’ Pk

ex_reqi ex_reqi

I=trueI’=true ?

1. I is true after ex_reqi at Pk.

2. Pk satisfies R1, R2 and R3.

Question: Is I’ true after ex_reqi at P’ ?



Let {Pi}Seqi{Qi} holds. If Seqi' is any sequence obtained by reordering statements in Seqi and {Pi}Seqi'{Qi} also holds, then we say that the triple {Pi}Seqi{Qi} is order-free.– For example, the triple, {x=3 y=1} x=7; y=y+1

{x=7 y=2}, is order-free.



Lemma 1: If {Pk}Seqk{Pi}Sti{I} holds and Stj in Seqk where (StjcfSti) (StjenSti), then {Pk}Seqk'{Pi}Sti{I} still holds where Seqk' is obtained by removing Stj from Seqk.

Stk

…

Stj-1

Stj

Stj+1

...

Sti

Pk

Pi

I

Stk

…

Stj-1

Stj+1

...

Sti

Pk

Pi

I

(StjcfSti) (StjenSti)

conflict

SeqkSeq’k

To proof lemma 1, we need use the definitions of order-free, conflict and enable



Lemma 2: If {Pk}Seqk{Pi}Sti{I} holds and Stj where (StjenSti) (StjcfSti), then {Pk}Seqk'{Pi}Sti{I} still holds where Seqk' is obtained by adding Stj in Seqk.

Stk

…

Stj-1

Stj+1

...

Sti

Pk

Pi

I

Stk

…

Stj-1

Stj

Stj+1

...

Sti

Pk

Pi

I

(StjenSti) (StjcfSti)

enable

To proof lemma 2, we will need the definitions of order-free, conflict and enable

Seqk Seq’k



The proof of theorem– R1+R2+R3+Lemma1+Lemma2 Theorem



reqj

reqi (Bi=false)

ex_reqk,j

enablel (Bi=false)

Pk

reql ex_reqk,l

req?

• It is possible that an algorithm satisfying R1, R2 and R3 may not be starvation free.

R4: reqj, where j ≠ i, if reqj cf reqi (reqj en reqi ), then reqj tm reqi ex_reqj t ex_reqi.



The general idea of the algorithm– If a process wants to execute a conflicting

statement, it sends a request to all processes and waits the ack messages.

– If a process wants to execute an enabling statements, a request should be sent out.

– If a process receives a request from other process, the request should be executed.



The algorithm (part 1): reqj cf reqi (reqj en reqi )

and we assume that reqj tm reqi. Px Py

reqi

reqjconflict

This part implements R1 and R4

ex_reqi ex_reqi

ex_reqj ex_reqj

reqi

reqj

ack

ack

Px PyReal Time

ex_reqj,x

ex_reqi,y



The algorithm (part 2): reqj en reqi (reqj cf reqi)

reqj

ex_reqi

ex_reqj

reqiex_reqx,j

reqj

reqi

ex_reqj

ex_reqi

Px Py PyPxReal time

enable

This part implements R2



The algorithm (part 3): reqj cf reqi reqj en reqi

– reqj can be handled as conflicting request and reqj will not be sent out as enable request after the execution of reqj.

reqj

ex_reqi

ex_reqj

reqiex_reqx,j

reqj

reqi

ex_reqj

ex_reqi

Px Py PyPxReal time

conflict & enable

This part implements R3



Our algorithm satisfies the rules R1-R4, so it is consistent.

The complexity of messages less than 3XN where the N is the number of processes.– Message passing takes place only between those

processes that need to synchronize. For example, in readers/writers problem, readers only send requests for entering Reader region to the writers instead of all the processes.



Fault Tolerance– We consider only node failures (or process

failure), and no link failures. To make things simple, we also assume that the node does not crash while sending a message.

– The status of a process (whether it is in a region or not) can be identified by its last request. Each process Pk has a new variable, LASTk,j, to record the last request received from each process Pj.



Node failure

reqi

SomebodyDied(Pz,fakeREQ)

Px Py Pz

-Py captured the crashed node Pz

-Py checks Lasty,z, if reqi is request for entering some region, then fakeREQ is the request to exit the same region. Otherwise, it is empty.

-Py sends SomebodyDied (Pz,fakeREQ) to tell Px

and Py itself that Pz is died.

-Px and Py treats the fakeREQ as the request from Pz then remove the Pz from list.

-However, the algorithm has some limitation. For example, consider the case of Pj entering R1 followed by R2 in a nested manner. If Pj fails after exiting R1 and before entering R2, then other processes wanting to enter R1 may be blocked for ever since Pj will never exit R2.



Node recovery– When a process, Pj recovers from failure, it sends Join(Pj)

message to all other processes and waits for Agree messages from them.

– When a process, Pi, receives the Join(Pj) message, it adds Pj to the list. Pi then sends Agree message along with the latest request that it executed.

– When Pj receives the Agree message from Pi, it checks the latest request Pi executed. If the latest request is for entering Rx, Pj increases the entry counter for Rx, In[x], by one, otherwise, it does nothing. Then, Pj can executes the requests from Pi after it gets the Agree message. After Pj gets all the Agree messages from other processes it can sends its own request to execute.


5. Algorithm Optimizations

The algorithm that we have proposed is for the general problem of region synchronization.

We would like our general algorithm to match the message complexity of algorithms that have been designed for specific synchronization problems.

– For example, our algorithm uses the same number of messages for the distributed mutual exclusion problem as the algorithm proposed by Lamport (3 X (N-1) messages). However, Ricart-Agrawala algorithm only requires 2 X (N-1) messages.

– Chandy gives an algorithm requiring 0-2d messages for dining philosophers problem while our algorithm needs 3d messages where d is the number of neighbors of a philosopher.


Algorithm Optimizations

Optimization to handle remote requests– incl,k,y keeps track of the number of times Pl has executed

Syncy until a request for Syncz from Pk arrives.

Syncz

Syncz

Pl Pk

Syncx

Syncy

conflict

enable

(a)

Pl PkPl Pk

Syncx

Syncy

Syncx

Syncz

Syncyreqz

ack(incl,k,y)

reqz

ack(incl,k,y)

(b) (c)



Optimization to handle local requests

Syncz

Pl Pk

Syncx

Syncy

Syncz

conflict

conflict

Pl Pk

Syncx

Syncy

reqx

reqz

ack

ack(incl,k,y)

(a) (b)



Using Application Structure to optimize performancePl Pk

Syncx

Syncy

Synczconflict

Pl Pk

Syncx

Syncy

Syncz

reqy

ack

reqz

ack(incl,k,x)

(a) (b)



Summary:– We have proposed a general algorithm for

synchronization in point-to-point system. – We also show that our algorithm, by the

optimizing, has performance comparable to known algorithms for specific synchronization problems .


6. The algorithm for CAN based System

Introduction to CAN network– Control Area Network (CAN) is well designed as a serial data

communications bus that supports distributed control systems by sending and receiving short real-time control messages.

– CAN is a broadcast bus. – The message is identified by message identifier. The identifier

not only filters upon reception but also sets the priority of the messages.

– CAN bus behaves like a large AND-gate for all bit sent at the same time.


The algorithm for CAN based System

Review the correctness criteria P1 P2 P3

reqa

reqb

reqc

Sync

Syna

Synb

Sync

Syna

Synb

Real Time

t

inconsistent

Sync

Syna

Synb

Sync

Synb

Syna



When a process (node) wants to execute a synchronization statement, Syni, it must satisfy the following rules:

– C1: If ( j, Syni cf Synj Synj cf Syni) and ( k, (Syni en Synk)), then a request is sent out. Ci' and Ci are incremented when Bi is true locally.

– C2: If j, Syni en Synj and k, (Synk cf Syni Syni cf Synk), then a request is sent out after Bi is true locally. Thus, Ci' and Ci are incremented before the request is sent.

– C3: If j, Syni cf Synj Synj cf Syni and k, Syni en Synk, then the request is sent first. Subsequently, Ci and Ci' are incremented when Bi is true locally, and then a notify message is sent out to all other processes. Other processes increment Ci locally only on receiving this notification.



Implementation on distributed approach

Application

Order Control P1 P1 Pn……

CAN BUS



Implementation on active monitor approach

Application

Proxy P1 Pn M……

CAN BUS

Monitor

Proxy



The example used in our implementation– Sleeping barber problem

A shop has M barbers, one chair for each barber and a waiting room with K chairs. If all barbers are busy when a customer, says A, arrives, then A waits in the waiting room (provided there is an empty chair). If a barber, says B, is free, then A sits in B's chair. After B is done cutting the hair, A leaves the shop. Subsequently, B waits for another customer to sit on its chair.

We have implemented solutions to the sleeping barber problem using the active monitor approach and the distributed approach.

– The system consists of six 167CR boards connected via a 250Kb/s speed CAN network, two barber nodes, three customer nodes and a noise node that is added to the system to adjust the system load. Three customers come to the shop in a random time, between 0 to 25 ms. The barber serves every customer 25ms.



• The performance analysis of two approaches

0

100

200

300

400

500

600

700

800

900

0 200 400 600 800 1000 1200

Frequence of noise messages

Th

e n

um

ber

of

cu

sto

mers

be s

erv

ed

Central Monitor Approach Replicated Monitor Approach



Frequency (# of noise/s)

1000 500 333 250 167 100 50 0

Customer be served (Distributed Monitor)

618 748 779 795 807 815 823 831

Customer be served (Active Monitor)

397 504 538 556 569 582 590 601

Replicated monitoroutperforms active monitorby %

55.7 48.4 44.8 43.0 41.8 40.0 39.5 38.3

•The performance analysis of two approaches


7. Integration with SyncGen

• Introduction to SyncGen tool


Integration with SyncGen

The SyncGen has two components: – one is coarse-grain solution generator, a translation from

the global invariant specification to the coarse-grain representation with await and atomic constructs and notification information (front end);

– the other includes multiple fine-grain solution back-ends. There will be one back-end for each language supported (Java, C, C++, etc.).

Our purpose is to integrate our solutions for point-to-point systems as well as CAN based systems to SyncGen toolset as back-ends.



Back-end for point-to-point systems– Since our solution is for a point-to-point network and every

node runs as a separate process on a machine, we need a configuration file to configure the network, specify which process go through which regions and the functional code.

– The relationship, conflict and enable, can be obtained automatically from the guard expression of the enter and exit requests in the coarse-grained solution.

– A package, “edu.ksu.cis.saves.MessagePassing.P2P”, handles communication and implements the algorithm we proposed.



Back-end for CAN based systems– To test and develop our solution for CAN system, we have

developed a CAN simulator, which utilizes the broadcasting feature of the CAN bus.

– Having the same configuration file as point-to-point backend.

– The package, “edu.ksu.cis.saves.MessagePassing.CAN”, handles communication and implements the algorithm we proposed.

– SyncGen generates a replicated monitor class for every process. Similar to the back-end for shared memory solution, the translation of the coarse-grain solution to the fine grain solution is to implement await and atomic structure and notification information.


8. An example

PartCONVEYOR

Robot Buffer

CuttingMachine

CuttingMachine

In

Out In-path

out-path

• A simple assembly process from [Suraj, Ramaswarm and Barber 1997]


An example

R1

C1

B1P1

Robot Cutter Part Buffer

P2

R2

C3 P3

P4

cuttingout path

in path

C2

R3 B2

counter increment

counter decrement

• The regions synchronization diagram


An example

The output of assembly example with one robot, two cutters and two parts arriving in loop.


An example

Consider the scenario where the robot is moving the part to one of the cutting machines, and two cutters and the out buffer are busy. In this case, a deadlock happens. [SRB97] solves this problem by using exit-safe state.

on conveyor

suspendready forpick up

c7c6

e0

e0 Sensor Signal

e1 Ready for pickup from conveyor

c6 !e0/\(in(Buffer:full) U in(Buffer:Empty) / !e1

c7 in(Buffer:Empty)

safe exit state

The system is desinged for “out-path” process when part is in suspend state


An example

In our design, we use two relays, Relay(B1,P1) and Relay(P2,R1), to solve such kind of synchronization problem.

However, both solutions are not optimal. If the buffer is full, but one of two cutters is free, it is safe for the robot to move the part from the conveyor to cutter. We can solve it by rewriting I of cluster P1B1C2:

– out[P1] ≤ in[B1]+2 out[C2] ≤ in[B1] in[C2]-out[C2] ≤ 1 In addition, our design is suitable for multiple robots,

but [SRB97] has to reconsider the synchronization among robots.


An example

Another typical synchronization problem in assembly example is that two cutters try to put the part in the buffer at the same time.

In [SRB97], the authors do not consider this problem. To solve this problem, they would need to handle the change of status between two cutters.

In our approach, we simply add a bound to region C2, so that the cutters should not be in region C2 at the same time.


9. Summary and further work

The contribution– We described a methodology for synthesizing

synchronization code from high-level specifications in a distributed system.

– We developed algorithms to execute the coarse-grained solution in point-to-point and CAN based message passing systems.

– We have presented optimizations that exploit the synchronization structure of the problem as well the structure of the application to improve performance.

– We have integrated our solutions for message passing systems to the SyncGen tool.


Summary and further work

The future work– We give a preliminary algorithm for fault tolerance

that has several limitations including the one for nested regions. We need to improve our algorithm so that it can handle the node failure and node recovery for all cases.

– We will extend our work to solve more complex synchronization problems where variables other than the in and out counters are allowed in the invariants.


Thanks

Any questions?

synchronization in message passing systems

Documents

based systems

region synchronization

reader region

aspect oriented approach

grained synchronization

region rx

outr writer region

coarsegrained solutions