introduction - paginas.fe.up.ptapm/cpar/docs/cpar01.pdf2010@feup introduction 15 •low-level...
TRANSCRIPT
![Page 1: Introduction - paginas.fe.up.ptapm/CPAR/docs/cpar01.pdf2010@FEUP Introduction 15 •Low-level approach is most popular •Augment existing language with low-level parallel constructs](https://reader030.vdocument.in/reader030/viewer/2022041010/5eb6f092cd0f214b68030e0d/html5/thumbnails/1.jpg)
Parallel Computing
Introduction
![Page 2: Introduction - paginas.fe.up.ptapm/CPAR/docs/cpar01.pdf2010@FEUP Introduction 15 •Low-level approach is most popular •Augment existing language with low-level parallel constructs](https://reader030.vdocument.in/reader030/viewer/2022041010/5eb6f092cd0f214b68030e0d/html5/thumbnails/2.jpg)
2010@FEUP Introduction 2
What is Parallel and Distributed computing?
• Solving a single problem faster using multiple
CPUs
• E.g. Matrix Multiplication C = A x B
• Parallel = Shared Memory among all CPUs
• Distributed = Local Memory/CPU
• Common Issues: Partition, Synchronization,
Dependencies, load balancing
![Page 3: Introduction - paginas.fe.up.ptapm/CPAR/docs/cpar01.pdf2010@FEUP Introduction 15 •Low-level approach is most popular •Augment existing language with low-level parallel constructs](https://reader030.vdocument.in/reader030/viewer/2022041010/5eb6f092cd0f214b68030e0d/html5/thumbnails/3.jpg)
2010@FEUP Introduction 3
Why Parallel and Distributed Computing?
• Grand Challenge Problems
• Weather forecasting; climate change
• Drug discovery
• Physical limitations of circuits
• Heat
• Light velocity limitation
• Cost
• Multiple average CPUs cost less than a high
end one
![Page 4: Introduction - paginas.fe.up.ptapm/CPAR/docs/cpar01.pdf2010@FEUP Introduction 15 •Low-level approach is most popular •Augment existing language with low-level parallel constructs](https://reader030.vdocument.in/reader030/viewer/2022041010/5eb6f092cd0f214b68030e0d/html5/thumbnails/4.jpg)
2010@FEUP Introduction 4
Microprocessor Revolution
Micros
Minis
Mainframes
Speed (log scale)
Time
Supercomputers
![Page 5: Introduction - paginas.fe.up.ptapm/CPAR/docs/cpar01.pdf2010@FEUP Introduction 15 •Low-level approach is most popular •Augment existing language with low-level parallel constructs](https://reader030.vdocument.in/reader030/viewer/2022041010/5eb6f092cd0f214b68030e0d/html5/thumbnails/5.jpg)
2010@FEUP Introduction 5
Shared Memory Programming
• Easier conceptual environment
• Programmers typically familiar with concurrent
threads and processes sharing address space
• CPUs within multi-core chips share memory
• OpenMP an application programming interface
(API) for shared-memory systems
• Supports higher performance parallel programming of
symmetrical multiprocessors
![Page 6: Introduction - paginas.fe.up.ptapm/CPAR/docs/cpar01.pdf2010@FEUP Introduction 15 •Low-level approach is most popular •Augment existing language with low-level parallel constructs](https://reader030.vdocument.in/reader030/viewer/2022041010/5eb6f092cd0f214b68030e0d/html5/thumbnails/6.jpg)
2010@FEUP Introduction 6
MPI / PVM
• MPI = “Message Passing Interface”
• PVM = “Parallel Virtual Machine”
• Standard specification for message-
passing libraries
• Libraries available on virtually all parallel
computers
• Free libraries also available for networks
of workstations, commodity clusters,
Linux, Unix, and Windows platforms
• Can program in C, C++, and Fortran
![Page 7: Introduction - paginas.fe.up.ptapm/CPAR/docs/cpar01.pdf2010@FEUP Introduction 15 •Low-level approach is most popular •Augment existing language with low-level parallel constructs](https://reader030.vdocument.in/reader030/viewer/2022041010/5eb6f092cd0f214b68030e0d/html5/thumbnails/7.jpg)
2010@FEUP Introduction 7
Classical versus Modern Science
Nature
Observation
TheoryPhysical
Experimentation
Numerical
Simulation
![Page 8: Introduction - paginas.fe.up.ptapm/CPAR/docs/cpar01.pdf2010@FEUP Introduction 15 •Low-level approach is most popular •Augment existing language with low-level parallel constructs](https://reader030.vdocument.in/reader030/viewer/2022041010/5eb6f092cd0f214b68030e0d/html5/thumbnails/8.jpg)
2010@FEUP Introduction 8
60 Years of Speed Increases
ENIAC
350 flops
1946
Today - 2009
1 Peta flops
Roadrunner -
Los Alamos NL
116,640 cores 12K IBM cell
![Page 9: Introduction - paginas.fe.up.ptapm/CPAR/docs/cpar01.pdf2010@FEUP Introduction 15 •Low-level approach is most popular •Augment existing language with low-level parallel constructs](https://reader030.vdocument.in/reader030/viewer/2022041010/5eb6f092cd0f214b68030e0d/html5/thumbnails/9.jpg)
2010@FEUP Introduction 9
Seeking Concurrency
• Data dependence graphs
• Data parallelism
• Functional parallelism
• Pipelining
![Page 10: Introduction - paginas.fe.up.ptapm/CPAR/docs/cpar01.pdf2010@FEUP Introduction 15 •Low-level approach is most popular •Augment existing language with low-level parallel constructs](https://reader030.vdocument.in/reader030/viewer/2022041010/5eb6f092cd0f214b68030e0d/html5/thumbnails/10.jpg)
2010@FEUP Introduction 10
Data Dependence Graph
• Directed graph
• Vertices = tasks
• Edges = dependencies
Taking care of a garden in a residence
![Page 11: Introduction - paginas.fe.up.ptapm/CPAR/docs/cpar01.pdf2010@FEUP Introduction 15 •Low-level approach is most popular •Augment existing language with low-level parallel constructs](https://reader030.vdocument.in/reader030/viewer/2022041010/5eb6f092cd0f214b68030e0d/html5/thumbnails/11.jpg)
2010@FEUP Introduction 11
Data Parallelism
• Independent tasks apply same operation
to different elements of a data set
• Okay to perform operations concurrently
• Speedup: potentially p-fold, p=#processors
for i 0 to 99 do
a[i] b[i] + c[i]
endfor
![Page 12: Introduction - paginas.fe.up.ptapm/CPAR/docs/cpar01.pdf2010@FEUP Introduction 15 •Low-level approach is most popular •Augment existing language with low-level parallel constructs](https://reader030.vdocument.in/reader030/viewer/2022041010/5eb6f092cd0f214b68030e0d/html5/thumbnails/12.jpg)
2010@FEUP Introduction 12
Functional Parallelism
• Independent tasks apply different
operations to different data elements
• First and second statements
• Third and fourth statements
• Speedup: Limited by amount of concurrent
sub-tasks
a 2
b 3
m (a + b) / 2
s (a2
+ b2) / 2
v s - m2
![Page 13: Introduction - paginas.fe.up.ptapm/CPAR/docs/cpar01.pdf2010@FEUP Introduction 15 •Low-level approach is most popular •Augment existing language with low-level parallel constructs](https://reader030.vdocument.in/reader030/viewer/2022041010/5eb6f092cd0f214b68030e0d/html5/thumbnails/13.jpg)
2010@FEUP Introduction 13
Pipelining
• Divide a process into stages
• Produce several items simultaneously
• Speedup: Limited by amount of concurrent
sub-tasks = #of stages in the pipeline
![Page 14: Introduction - paginas.fe.up.ptapm/CPAR/docs/cpar01.pdf2010@FEUP Introduction 15 •Low-level approach is most popular •Augment existing language with low-level parallel constructs](https://reader030.vdocument.in/reader030/viewer/2022041010/5eb6f092cd0f214b68030e0d/html5/thumbnails/14.jpg)
2010@FEUP Introduction 14
Programming Parallel Computers
• Extend compilers: translate sequential
programs into parallel programs
• Extend languages: add parallel operations
• Add parallel language layer on top of
sequential language
• Define totally new parallel language and
compiler system
![Page 15: Introduction - paginas.fe.up.ptapm/CPAR/docs/cpar01.pdf2010@FEUP Introduction 15 •Low-level approach is most popular •Augment existing language with low-level parallel constructs](https://reader030.vdocument.in/reader030/viewer/2022041010/5eb6f092cd0f214b68030e0d/html5/thumbnails/15.jpg)
2010@FEUP Introduction 15
• Low-level approach is most popular
• Augment existing language with low-level
parallel constructs
• MPI, PVM, threads/process-based concurrency
and OpenMP are examples
• Advantages of low-level approach
• Efficiency
• Portability
• Disadvantage: More difficult to program
and debug
Current Status