Introduction to Parallel Processing3.1 Basic concepts3.2 Types and levels of parallelism3.3 Classification of parallel architecture3.4 Basic parallel techniques3.5 Relationships between languages and parallel architectureCH03TECH Computer Science

3.1 Basic concepts3.1.1 The concept of programordered set of instructions (programmers view)executable file (operating systems view)

The concept of processOS view, process relates to executionProcess creationsetting up the process descriptionallocating an address spaceloading the program into the allocated address space, andpassing the process description to the schedulerprocess statesready to runrunningwait

Process spawning (independent processes)Figure 3.3 Process spawning.ACBDE

3.1.3 The concept of threadsmaller chunks of code (lightweight)threads are created within and belong to processfor parallel thread processing, scheduling is performed on a per-thread basisfiner-grain, less overhead on switching from thread to thread

Single-thread process or multi-thread (dependent)Figure 3.5 Thread treeProcessThreads

Three basic methods for creating and terminating threads1. unsynchronized creation and unsynchronized terminationcalling library functions: CREATE_THREAD, START_THREAD2. unsynchronized creation and synchronized terminationFORK and JOIN3. synchronized creation and synchronized terminationCOBEGIN and COEND

3.1.4 Processes and threads in languagesBlack box view: T: threadT2 T0 T1 T1 T2 T0 . . . TnFORKJOINFORKJOIN(a)COBEGINCOEND. . . (b)

3.1.5 The concepts of concurrent execution (N-client 1-server) Non pre-emptivePre-emptiveTime-sharedPriotizedPrioritySeverSeverSeverClientClientClient

Parallel executionN-client N-server modelSynchronous or AsynchronousSeverClient

Concurrent and parallel programming languagesClassificationTable 3.1Classification of programming languages.

Languages

1_client

1-server model

N_client

1-server mode

1_client

N-server model

N_client

N-server model

sequential

+

-

-

-

concurrent

-

+

-

-

Data-parallel

-

-

+

-

Parallel

-

+

-

+

3.2 Types and levels of parallelism3.2.1 Available and utilized parallelismavailable: in program or in the problem solutionsutilized: during execution3.2.2 Types of available parallelismfunctionalarises from the logic of a problem solutiondataarises from data structures

Figure 3.11 Available and utilized levels of functional parallelismAvailable levelsUtilized levelsUser (program) levelUser levelProcedure levelProcess levelLoop levelThread levelInstruction levelInstruction level211.Exploited by architectures2.Exploited by means of operating systems

3.2.4 Utilization of functional parallelismAvailable parallelism can be utilized by architecture,instruction-level parallel architectures compilersparallel optimizing compileroperating systemmultitasking

3.2.5 Concurrent execution modelsUser level --- Multiprogramming, time sharingProcess level --- MultitaskingThread level --- Multi-threading

3.2.6 Utilization of data parallelismby using data-parallel architecture

3.3 Classification of parallel architectures3.3.1 Flynns classificationSISDSIMDMISD (Multiple Instruction Single Date)MIMD

Parallel architectures //

Parallel architecturesPAsPart IIIPart IIPart IVData-parallel architecturesFunction-parallel architecturesInstruction-levelPAsThread-levelPAsProcess-levelPAsILPSMIMDsVectorarchitectureAssociativearchitecturearchitectureand neuralSIMDsSystolicPipelinedprocessorsProcessors)processorsVLIWsSuperscalarDitributedmemory(multi-computer)Sharedmemory(multi-MIMDDPs

3.4 Basic parallel technique3.4.1 Pipelining (time)a number of functional units are employed in sequence to perform a single computationa number of steps for each computation3.4.2 Replication (space)a number of functional units perform multiply computation simultaneouslymore processorsmore memorymore I/Omore computers

3.5 Relationships between languages and parallel architectureSPMD (Single Procedure Multiple data)Loop: split into N threads that works on different invocations of the same loopthreads can execute the same code at different speedssynchronize the parallel threads at the end of the loopbarrier synchronizationuse MIMD Data-parallel languagesDAP FortranC = A + Buse SIMD

Synchronization mechanismsFigure 3.17 Progress of language constructs used for synchronizationTest_and_setSemaphoreConditionalCritical regionMonitorRendezvousSend/receivemessageBroadcastShiftnet_receive(processor form)Remote procedure callsNet_send

Using binary Semaphore p1 or queue semaphore p2P1CriticalregionV(S)SSemaphoreShareddatastructureV(S)P(S)P2CriticalregionP(S)Figure 3.18 Using a semaphore to solve the mutual execution problemBusy Wait

Parallel distributed computingAda used rendezvous concepts which combines feature of RPC and monitorsPVM (Parallel Virtual Machine)to support workstation clustersMPI (Message-Passing Interface)programming interface for parallel computersCOBRA ?Windows NT ?

Summary of forms of parallelismSee Table 3.3