Operating Systems, 132

Practical Session 13Distributed synchronization

Motivation

• Interest in distributed computation models is rapidly growing (Grids, Cloud computation, internet relay, etc)

• A good model for computation is difficult to come up with:1. Concurrent computation2. No global time and no global state3. Hard to capture effects of possible failures

The model

• Each instance is executed on a different processor

• Assume no shared memory, communication is handled with messages of the following format: <destination, action; parameters>

• Sending is non-blocking and reliable• A processor waits for events (messages) – a

timeout mechanism is possible but we will not discuss it here

Global states and causality

• It is impossible to determine the global state of a distributed system:1. Noninstantaneous communication (delays, lost

messages, etc)2. Can’t synchronize with a timer mechanisms (drift,

initial synchronization)3. Local interruptions (can’t trust simultaneous

reactions)• Thus, we must find global system properties which

we can depend on – causal order of events

Happened before

Same processor event

Send – receive event

Transitivity of <H

• We would like to define some order over system events – a “happened before” relation (denoted <H):1. If (e1 <p e2) then e1 <H e2

2. If (e1 <m e2) then e1 <H e2

3. If (e1 <H e2 && e2 <H e3) then e1 <H e3

• Defines a partial order• Can be defined as a DAG

• Defines a global (and total) order on events

• Order is consistent with <H

• Created on the fly• Will assume that each event

has a timestamp attached to it• An ID is appended to the

timestamp and allows for tie breaking

• Lamport’s algorithm: If e1<He2 then e1.TS < e2.TS

Global time – Lamport’s timestamps

1 Initially my_TS=0

2 Upon event e,3 if e is the receipt of message m4 my_TS=max(m.TS, my_TS)5 my_TS++6 e.TS=my_TS7 If e is the sending of message m8 m.TS=my_TS

Causality violation and vector timestamps

• Lamport’s algorithm does not guarantee that if e.TS < e’.TS then e <H e’

• This make it difficult to detect causality violation• A causality violation occurs if a message m is sent

to a remote processor p before another message m’ is sent, but p receives m’ before m Written as: m<cm’ and r(m’)<pr(m)

• We will use a vector of timestamps to overcome this problem

Global time – vector timestamps1 Initially my_VT=[0,…,0]

2 Upon event e,3 if e is the receipt of message m4 for i=1 to M5 my_VT[i] =

max(m.VT[i],my_VT[i])6 My_VT[self]++7 e.VT=my_VT8 if e is the sending of message m9 m.VT=my_VT

• Vector timestamp:– e.VT ≤v e’.VT iff

e.VT[i] ≤ e’.VT[i], 1 ≤ i ≤ M

– e.VT <v e’.VT iffe.VT ≤v e’.VT and e.VT≠e’.VT

• Can be used to detect causality violations

• VT algorithm: e1<He2 iff e1 <VT e2

Question 1

Consider the following interaction between four processors:TimeP1 P2 P3 P4

e1

e2

e3

e4

e5

e6

e7

e8

e9

e10

e11

e12

e13

e14

e15

e16

e17

e18

e19

e20

e21

e22

e23

e24

Question 1

1. What is the largest Lamport’s timestamp value? (hint: you can answer without calculating all time stamps)

2. List the Lamport timestamp of each event.3. List the vector timestamp of each event.4. Is there a potential causality violation? What

can indicate this violation?

Question 1

The Lamport timestamp mechanism calculates the longest chain of event occurring within a system. Thus, the largest timestamp value would be the number of vertices included in the longest path of events of the underlying DAG.In this case the answer is 12.

P1 P2 P3 P4

e1

e2

e3

e4

e5

e6

e7

e8

e9

e10

e11

e12

e13

e14

e15

e16

e17

e18

e19

e20

e21

e22

e23

e24

Question 1

TimeP1 P2 P3 P4

e1

e2

e3

e4

e5

e6

e7

e8

e9

e10

e11

e12

e13

e14

e15

e16

e17

e18

e19

e20

e21

e22

e23

e24

Event TSe1 1e2 2e3 3e4 4e5 6e6 10e7 11e8 1e9 2e10 3e11 4e12 5e13 8e14 1e15 4e16 7e17 8e18 9e19 12e20 2e21 3e22 5e23 6e24 7

Question 1

TimeP1 P2 P3 P4

e1

e2

e3

e4

e5

e6

e7

e8

e9

e10

e11

e12

e13

e14

e15

e16

e17

e18

e19

e20

e21

e22

e23

e24

Event TSe1 (1,0,0,0)e2 (2,1,0,0)e3 (3,1,0,0)e4 (4,1,0,0)e5 (5,5,1,2)e6 (6,5,5,4)e7 (7,5,5,4)e8 (0,1,0,0)e9 (0,2,1,0)e10 (0,3,1,0)e11 (1,4,1,2)e12 (1,5,1,2)e13 (4,6,1,5)e14 (0,0,1,0)e15 (0,3,2,0)e16 (4,3,3,4)e17 (4,3,4,4)e18 (4,3,5,4)e19 (7,5,6,4)e20 (1,0,0,1)e21 (1,0,0,2)e22 (4,1,0,3)e23 (4,1,0,4)e24 (4,1,0,5)

Question 1

A possible causality violation exist. Note that the send event e3 (a message from p1 to p3) happens before e23 (a message from p4 to p3) but is received afterward. When using VT, e3.VT=(3,1,0,0) but right before the reception of this message (e17) the clock’s VT is e16.VT=(4,3,3,4). Thus, when P3 receives this message it knows that e23 arrived “too soon”.

P1 P2 P3 P4

e1

e2

e3

e4

e5

e6

e7

e8

e9

e10

e11

e12

e13

e14

e15

e16

e17

e18

e19

e20

e21

e22

e23

e24

The Ricart - Agrawala algorithm

• An algorithm for handling distributed mutual exclusion

• Uses Lamport’s timestamps• Each process only uses the following set of

variables / data structures: Timestamp current_time Timestamp my_timestamp integer reply_count boolean isRequesting boolean reply_deferred[M]

The Ricart - Agrawala algorithm

The following code is used to enter the critical section:

Request_CS:1 my_timstamp := current_time2 isRequesting := TRUE3 Reply_count := M-14 for every processor j≠i5 send(REMOTE_REQUEST; my_timestamp)6 wait until reply_count = 0

The Ricart - Agrawala algorithmA listener thread is used so that the node responds to requests from others:

CS_monitoring: Wait until a REMOTE_REUQUEST or REPLY message is received

REMOTE_REQUEST(sender; request_time)1. Let j be the sender of the REMOTE_REQUEST message2. if (not is_requesting or my_timestamp > request_time)3. send(j, REPLY)4. else5. reply_deferred[j] = TRUE

REPLY6. reply_count := reply_count-1

Ties are broken with processor IDs

The Ricart - Agrawala algorithm

Releasing the CS:

Release_CS_monitoring:1. is_requesting := false2. For j=1 through M (other than this processor's ID)3. if (reply_deferred[i]=TRUE)4. send(j, REPLY)5. reply_deferred[j]=FALSE

Question 2

Assume that N processors are handling mutual exclusion with the aid of the Ricart-Agrawala’s mutual exclusion algorithm.1. How many messages will be passed in the

system whenever a processor wishes to enter the critical section? Are there scenarios where this number is lower/greater?

2. Why is this algorithm deadlock free?3. What can happen if a single message is lost?

Question 2

1.To enter the CS a processor must request permission from all other processors – i.e. N-1 messages are sent.Only after the processor received a REPLY response from all other processors may it enter the CS. Note, that these may be deferred for a while…That is, entering the CS will require a total of 2(N-1) messages passed.One means to reduce this network load is by keeping several requests deferred for a while. This will allow agents to enter the CS more than once without having to send messages to all N-1 nodes in the system [Roucairol & Carvalho].

Question 2

2. The algorithm relies on the fact that each timestamp is unique (based on the Lamport’s time and processor ID). Thus, a total order over request can be easily deduced, and CS access is handled through this order.3. The algorithm assumes that the system is failure free and its correctness heavily relies on this condition. It is easy to see that if a single message is lost a deadlock can easily occur.

Raymond’s algorithm

• Solve the mutual exclusion problem via a token (only the token holder may enter CS)

• Communication is based on an underlying tree structure of all nodesThe tree is always oriented towards token holder

• Uses a FIFO queue to prevent starvation• Good performance (number of messages per

CS entry decreases as the load increases!)Uses O(log n) messages only

Raymond’s algorithm

• Each process only uses the following set of variables / data structures: Boolean token_holder Boolean inCS Processor current_dir Queue requests_queue

Raymond’s algorithmRequest_CS:1 If not token_holder2 if requests_queue.isEmpty( )3 send(current_dir, REQUEST)4 requests_queue.enqueue(self)5 wait until token_holder is true6 inCS := true

Release_CS:7. inCS := false8. If not requests_queue.isEmpty( )9. current_dir := requests_queue.dequeue( )10. send(current_dir, TOKEN)11. token_holder := false12. if not requests_queue.isEmpty( )13. send(current_dir, REQUEST)

Raymond’s algorithmMonitor_CS:1 while (true)2 wait for a REQUEST or a TOKEN message REQUEST3. if token_holder4. if inCS5. requests_queue.enqueue(sender)6. else7. current_dir := sender8. send(current_dir, TOKEN)9. token_holder := false10. else11. if requests_queue.isEmpty()12. send(current_dir,REQUEST)13. requests_queue.enqueue(sender)

Raymond’s algorithm (cont.)TOKEN

14. current_dir := requests_queue.dequeue( )15. if current_dir = self16. token_holder := true17. else18. send(current_dir, TOKEN)19. if not requests_queue.isEmpty( )20. send(current_dir, REQUEST)

Question 3, Moed B 2006

The following 8 processor network is using Raymond’s algorithm to solve the mutual exclusion problem. In the initial state, the token is with processor A at the root of the tree (and wants to enter the critical section), and no requests for the CS are recorded.

Question 3, Moed B 2006

A

B F

C D E G H

Directed edges correspond to the current_direction var

Question 3, Moed B 2006

To allow for a convenient representation we define agent steps as the invocation of a procedure or an action. Use the sketch above to describe the result of applying Raymond’s algorithm if nodes C,D,F and G request the token. Provide a detailed description of all concurrent steps (in which a single step is taken by all relevant nodes) by sketching the system’s state after each one and up until three of the four agents receive the token.Note: assume that ties are broken based on ID.

Question 3, Moed B 2006

A

B F

C D E G HC D G

F

REQUEST REQUEST REQUEST

REQUEST

Question 3, Moed B 2006

A

B F

C D E G H

F

CD

C D G

FG

REQUEST

Question 3, Moed B 2006

A

B F

C D E G H

FB

CD

C D G

FG

Question 3, Moed B 2006

A

B F

C D E G H

B

CD

C D G

FG

REQUEST

Question 3, Moed B 2006

A

B F

C D E G H

B

CD

C D G

GA

Question 3, Moed B 2006

A

B F

C D E G H

B

CD

C D G

A

REQUEST

Question 3, Moed B 2006

A

B F

C D E G H

B

CD

C D F

A

Question 3, Moed B 2006

A

B F

C D E G H

B

CD

C D

A

Question 3, Moed B 2006

A

B F

C D E G H

B

CD

C D

A

Question 3, Moed B 2006

A

B F

C D E G H

B

CD

C D

Question 3, Moed B 2006

A

B F

C D E G H

B

CD

C D

Question 3, Moed B 2006

A

B F

C D E G H

CD

C D

Question 3, Moed B 2006

A

B F

C D E G H

CD

C D

Question 3, Moed B 2006

A

B F

C D E G H

D

C D

REQUEST

Question 3, Moed B 2006

A

B F

C D E G H

D

B D