Course Policies and outcomes

Chapter 1Chapter 1

Introduction to Parallel Introduction to Parallel Computing Computing

Dr. Muhammad Hanif Durad

Department of Computer and Information Sciences

Pakistan Institute Engineering and Applied Sciences

hanif@pieas.edu.pk

Dr. Hanif Durad 3

Lecture Outline

Motivating Parallelism Scope of Parallel Computing Applications Organization and Contents of the Course Texts and References

Parallel computing

Parallel Processing is a term used to denote a large class of techniques that are used to provide simultaneous data processing tasks for the purpose of Save time and/or money Solve larger problems

Parallel computing is the simultaneous use of multiple compute resources to solve a computational problem 4

parallelcomputing-120214042541-phpapp02-130408023815-phpapp01.ppt

The Universe is Parallel

5

Galaxy formation Planetary movement Weather and ocean patterns Tectonic plate drift Rush hour traffic Automobile assembly line Building a jet Ordering a hamburger

at the drive through.

Areas of Parallel Computing

Physics – applied, nuclear, particle, condensed matter, high pressure, fusion, photonics

Bioscience, Biotechnology, Genetics Chemistry, Molecular Sciences Geology, Seismology Mechanical Engineering - from prosthetics to spacecraft Electrical Engineering, Circuit Design, Microelectronics Computer Science, Mathematics

Dr. Hanif Durad 6

Why Use Parallel Computing? Save time and/or money: In theory, throwing more resources

at a task will shorten its time to completion, with potential cost savings. Parallel computers can be built from cheap, commodity components.

Solve larger problems: Many problems are so large and/or complex that it is impractical or impossible to solve them on a single computer, especially given limited computer memory.

Better response times:  As the computing tasks are engaged by a group of processors, the tasks are completed in a smaller amount of time

7

Dr. Hanif Durad 8

Hardware Models

02Distributed_Computing.pptx

Ways to Classify Parallel Computers

One of the more widely used classifications, in use since 1966, is called Flynn's Taxonomy

The 4 possible classifications according to Flynn’s are :

 Single Instruction, Single Data (SISD)  Single Instruction, Multiple Data (SIMD)  Multiple Instruction, Single Data (MISD)  Multiple Instruction, Multiple Data (MIMD)

Dr. Hanif Durad 9

Dr. Hanif Durad 10

Recent Trends in Computing

Technology Trends: Moore’s Law

Dr. Hanif Durad 11

Slide from Maurice Herlihy

Clock speed

flattening sharply

Transistor count still

rising

6963_L1.ppt

Techology Trends: Power Issues

Dr. Hanif Durad 12

From www.electronics-cooling.com/.../jan00_a2f2.jpg 6963_L1.ppt

Power Perspective

Dr. Hanif Durad 13

Gig

aFlo

p/s

Meg

aWat

ts

0.001

0.01

0.1

1

10

100

1000

10000

100000

1000000

10000000

100000000

1000000000

1960 1970 1980 1990 2000 2010 2020

Performance (Gflops)

Power

Slide source: Bob Lucas

6963_L1.ppt

Increase speed processor

Greater no. of transistors Operation can be done in fewer clock cycles

Increased clock speed More operations per unit time

Example 8088/8086 : 5 Mhz, 29000 transistors E6700 Core 2 Duo: 2.66 GHz, 291 million

transistor

lecture1-100131132656-phpapp02.pptx

The Multi-Core Paradigm Shift

Key ideas: Movement away from increasingly complex

processor design and faster clocks Replicated functionality (i.e., parallel) is simpler to

design Resources more efficiently utilized Huge power management advantages

Dr. Hanif Durad 15

6963_L1.ppt

Multicore

A multi-core processor is one processor that contains two or more complete functional units. Such chips are now the focus of Intel and AMD. A multi-core chip is a form of SMP

lecture1-100131132656-phpapp02.pptx

parallelprogramming-130823023925-phpapp01.pptx

Concept of GPGPU (General-Purpose Computing on GPUs)

Idea: Potential for very high performance at low cost Architecture well suited for certain kinds of parallel applications (data

parallel) Demonstrations of 30-100X speedup over CPU

Early challenges: Architectures very customized to graphics problems (e.g., vertex and

fragment processors) Programmed using graphics-specific programming models or libraries

Recent trends: Some convergence between commodity and GPUs and their associated

parallel programming modelsDr. Hanif Durad 17

6963_L1.ppt

Parallel Programming Model Parallel programming models in common use:

Shared Memory (without threads) Threads Distributed Memory / Message Passing Data Parallel Hybrid Single Program Multiple Data (SPMD) Multiple Program Multiple Data (MPMD)

These models are NOT specific to a particular type of machine or memory architecture

Any of these models can be implemented on any underlying hardware18

parallelprogramming-130823023925-phpapp01.pptx, p-60/114

Classes of Parallel Computers

Cluster computing A cluster is a group of loosely coupled computers that

work together closely, so that in some respects they can be regarded as a single computer

Massive parallel processing A massively parallel processor (MPP) is a single

computer with many networked processors. MPPs have many of the same characteristics as

clusters, but MPPs have specialized interconnect networks  Dr. Hanif Durad 19

parallelprogramming-130823023925-phpapp01.pptx, p-99/114

Classes of Parallel Computers

Grid computing Compared to clusters, grids tend to be more loosely

coupled, heterogeneous, and geographically dispersed

Dr. Hanif Durad 20

Cloud Computing A cloud provider has 100s of thousands of nodes (aka

servers). Cloud computing is massively-parallel computing with

multi-processors (i.e. many multi-core processors) In principle, your application may run on one, two, …

thousands of servers (i.e. processors) For your application to run on one, two, … thousands of

servers, your application code or data must be parallelized. i.e. Split up into independent or relatively independent parts.

Dr. Hanif Durad 21

02Distributed_Computing.pptx

Dr. Hanif Durad 22

Organization and Contents of this Course

1. Fundamentals: This part of the class covers basic parallel platforms, principles of algorithm design, group communication primitives, and analytical modeling techniques.

2. Parallel Programming: This part of the class deals with programming using message passing libraries and threads.

3. Parallel Algorithms: This part of the class covers basic algorithms for matrix computations, graphs, sorting, discrete optimization, and dynamic programming.

1+2=CIS-546 PC3=CIS-645 PA

Summary Introduction to Parallel Computing Technology Trends

Multicore GPGPU (General-Purpose Computing on GPUs) Parallel Programming Model

Classes of Parallel Computers Cluster computing Massive parallel processing (MPP) Grid computing Cloud Computing

Dr. Hanif Durad 23

Dr. Hanif Durad 24

References Grama A., Gupta A., Karypis G., and Kumar V., Introduction

to Parallel Computing, 2nd ed., Addison-Wesley, 2003. (Main Text Book)

Thomas Rauber, Gudula Rünger, Parallel Programming: For Multicore and Cluster Systems, Springer, 2010.

Peter Pacheco, An Introduction to Parallel Programming, Elsevier, 2011.

Michael J. Quinn , Parallel Programming in C with MPI and OpenMP, McGraw-Hill, 2004.

References Mostafa Abd-El-Barr and Hesham El-Rewini, Advanced

Computer Architecture and Parallel Processing, John Wiley and Sons, 2005.

Culler D. E., Singh J. P., and Gupta A., Parallel Computer Architecture: A Hardware/Software Approach, Morgan Kaufmann Publisher, 1999 .

Foster I., Designing and Building Parallel Programs, Addision-Wesley, 1995.

Dr. Hanif Durad 25