programming -- levels of difficulty sequential –termination –determinism concurrent...

Programming -- Levels of Difficulty

• Sequential– Termination– Determinism

• Concurrent– Non-Termination– Non-Determinism

• Distributed– Multiple Computers– Partial Failure

• Byzantine– Failure at the worst time, in the worst way

Sequential Programs

• Deterministic< x == 3 >

x = x + 1;

< x == 4>

• Terminateexit(0)

Current State

x = x + 1;

Next State

Concurrent Programs

• Non-terminating

• Non-deterministic (exhibit interference)

Concurrent Programs

• Synchronize -- perform actions in a desired order

• Communicate -- transfer value(s) from one thread/process to another

• (NIST) POSIX® (Portable Operating System Interface), FIPS 151-2, ISO/IEC 9945-1: 2003 (IEEE Std. 1003.1: 2001)

• www.opengroup.org/austin

• Option Groupshttp://people.redhat.com/~drepper/posix-option-groups.html

_POSIX_SPIN_LOCKSThe implementation supports the Spin Locks option. If this #define has a value other than -1 or 0, it shall have the value 200112L

POSIX

#include <unistd.h>#include <pthread.h>void *p(void *arg) { int i; for (i=0; i<5; i++) { printf("X\n"); sleep(1); }

pthread_exit((void *)99);} int main() { pthread_t x; void *r; int i;

assert(pthread_create(&x, NULL, p, (void *)34) == 0); for (i=0; i<5; i++) { printf("Y\n"); sleep(1); }

assert(pthread_join(x, &r) == 0); return 0;}OUTPUT

Y X Y X X Y X Y X Y

Parallel DAG (directed acyclic graph) andHappens-before Edges

x=0

x=1

WriteLine(x)

x=2

Practical Parallel and Concurrent Programming DRAFT: comments to msrpcpcp@microsoft.com 76/16/2010

Schedule, Informally

A topological sort (serialization) of the nodes

in a parallel DAG

-

A sequential ordering of the nodes that

respects the happens-before edges

Practical Parallel and Concurrent Programming DRAFT: comments to msrpcpcp@microsoft.com 86/16/2010

Different schedules, different outputs

x=0

x=1

WriteLine(x)

x=2

x=0

x=2

WriteLine(x)

x=1

x = 2 x = 1

Practical Parallel and Concurrent Programming DRAFT: comments to msrpcpcp@microsoft.com 96/16/2010

Determinism

• For the same initial state,

observe the same final state,

regardless of the schedule

• Determinism desirable for most data-parallel problems

Practical Parallel and Concurrent Programming DRAFT: comments to msrpcpcp@microsoft.com 106/16/2010

Another Example - 2 robots in a roomData Structures

Practical Parallel and Concurrent Programming DRAFT: comments to msrpcpcp@microsoft.com 11

struct RoomPoint { public int X; public int Y;}

class Robot { public RoomPoint Location;}

List<Robot> _robots;Robot[][] _roomCells;

r1r2

_roomCells;

(0,0)

6/16/2010

MoveOneStep(Robot r1)

• Find new empty cell for r1• Move r1 to new cell, if not already

occupied

Practical Parallel and Concurrent Programming DRAFT: comments to msrpcpcp@microsoft.com 12

r1r2

r1

r2

6/16/2010

Practical Parallel and Concurrent Programming DRAFT: comments to msrpcpcp@microsoft.com 13

0 1 2 3 4

0 r r r r1 r r1 r2 r2 r r r r r

6/16/2010

0 1 2 3 4

0 r r r1 r r1 r r2 r2 r r r r r

0 1 2 3 4

0 r r r2 r r1 r r1 r2 r r r r r

PerformSimulationStep PerformSimulationStep

Practical Parallel and Concurrent Programming DRAFT: comments to msrpcpcp@microsoft.com 14

0 1 2 3 4

0 r r r r1 r r1 r2 r2 r r r r r

6/16/2010

0 1 2 3 4

0 r r r1r2 r r

1 r r2 r r r r r

Different schedules, different outputs

Is 0,2 empty?

Is 0,2 empty?

Move to 0,2

Move to 0,2

Is 0,2 empty?

Move to 0,2

Move to 0,2

Is 0,2 empty?

error correct

Practical Parallel and Concurrent Programming DRAFT: comments to msrpcpcp@microsoft.com 156/16/2010

REENTRANT

• A procedure with no external communication or synchronization is reentrant

Identifying Communication Points or Data Races

1. Is a variable that is Read by one thread Written by any other thread?

2. Is a variable that is Written by one thread Read by any other thread?

3. Do two threads Write the same variable?

Interference point -- bad communication point An Interference Point for A Singly-Linked List

p = q.head; /* remove the list element */ A.q.head = q.head -> pCB; /* update the list */ B. return p; C.

Possible Execution SequencesAi denotes that thread i executes statement A. (B1B2) denotes that statement B is executed simultaneously by Threads 1 and 2.

A1B1A2B2 If T1 executes statements A and B then T2 executes A and B, both threads are returned a different context block and no error occurs. If threads execute the code independently, it is referred to as serial execution.

1. A1A2(B1B2) If P1 executes A then P2 executes A and then both simultaneously execute B, both threads are returned the same context block, which is an error. Note that (B1B2) and (B2B1) are identical.

2. 2. (A1A2)B1B2 If both threads execute A simultaneously and then execute B sequentially, both threads are returned the same context block yet two blocks are deleted from the free list.

3. 3. The other possible execution sequences are A2B2A1B1, A1A2B1B2, A1A2B2B1, A2A1B1B2, A2A1B2B1, A2A1(B1B2), (A1A2)B2B1, (A1A2)(B1B2), A1(A2B1)B2, A2(A1B2)B1.

Interference Points referred to as CRITICAL SECTIONS

Critical Section Solution

1. The procedures in a critical section are executed indivisibly with respect to the shared variables or resources that are accessed.

2. A thread must not halt inside a critical section.

3. A thread outside a critical section cannot block another thread from entering the critical section.

4. 3. (Optional, fairness and finite progress) A thread trying to enter a critical section will eventually do so.

Important Points!

All Concurrent programs must be certified interference free!!

AndThe property cannot be validated by testing!!

HoweverProper use of Pthreads can guarantee it!!

POSIX FUNCTIONS

int pthread_join(pthread_t thread, void **out);

waits for termination, retrieves return value

pthread_t pthread_self(void);

know thyself! or who am I really?

int pthread_equal(pthread_t t1, pthread_t t2);

are we or are they twins?

int pthread_detach(pthread_t thread);

no one can join on this thread

int pthread_cancel(pthread_t thread);

terminate the thread

void sched_yield(void);

give up CPU

THREAD STATE DIAGRAM

A Little History

• In 1967, E.W. Dijkstra submitted a paper[6] on the "THE Multiprogramming System" to an operating systems conference. It contained the following quotation: 

• ‘Therefore, we have arranged the whole system as a society of sequential processes, progressing with undefined speed ratios. .... Their harmonious cooperation is regulated by means of explicit mutual synchronization statements. On the one hand, this explicit mutual synchronization is necessary, as we do not make any assumption about speed ratios; on the other hand, this mutual synchronization is possible because "delaying the progress of a process temporarily" can never be harmful to the interior logic of the process delayed.’

SEMAPHORE USE DETERMINED BY INITIAL VALUE

• 1 -- CRITICAL SECTION P <CS> V

• >1 -- RESOURCE COUNTERP(RC) //allocate

P(MUTEX) <critical section to allocate resource>V(MUTEX)-------------------------------P(MUTEX) deallocate V(MUTEX) V(RC)

• 0 -- USED TO IMPLEMENT PROCESS

SYNCHRONIZATION GRAPHS

Semaphores to Solve the C.S. Problem

typedef struct {long count =

1;Queue q =

{EMPTY};} Semaphore;

Semaphore s;P(s); if (--s.count<0) SleepOn(s.q);< CRITICAL SECTION>V(s); if (++s.count<=0) Wakeup(s.q);

PROCESS SYNCHRONIZATION GRAPH

Semaphore_t ac=0,bc=0,cd=0,ce=0;

PA PB PC PD PE P(ac) P(cd) P(ce) P(bc)

…. …. ….

V(ac) V(bc) V(cd) V(ce)

D

B

CA

E

Mutex – semaphore only used for C.S.#include <pthread.h>pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER;

void * func(void * arg) { Note: void * arg and return value!!assert(pthread_mutex_lock(&m) == 0);

/****CRITICAL SECTION ***/

assert(pthread_mutex_unlock(&m) == 0);

return 0; }int main() { pthread_t tA, tB;

pthread_lib_init("");assert(pthread_create(&tA, NULL, func, NULL) == 0);assert(pthread_create(&tB, NULL, func, NULL) == 0);assert(pthread_join(&tA, NULL) == 0);assert(pthread_join(&tB, NULL) == 0);

return 0; }