FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 1
FIT5174 Distributed & Parallel Systems
Lecture 5
Message Passing and MPI
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 2
Acknowledgement
These slides are based on slides and material by:
C Evangelinos,
Scott Baden,
Zhiliang Xu
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 3
Shared Memory Approach
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 4
Message Passing Model
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 5
Distributed Computing using Message Passing
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 6
Single Program / Multiple Program - Multiple Data
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 7
Programming with message passing
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 8
Message passing concept
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 9
Send and Receive concept
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 10
Message completion
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 11
Buffering concept
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 12
Causality concept
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 13
Asynchronous versus Synchronous
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 14
Overlapping operations
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 15
What is MPI?
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 16
MPI websites
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 17
MPI: Message Passing Interface
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 18
MPI history
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 19
MPI in context
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 20
MPI fundamentals
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 21
Minimal MPI subset
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 22
Initializing MPI
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 23
Getting started with MPI
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 24
MPI program details
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 25
MPI program more details
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 26
Simple MPI C Program
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 27
MPI Communicators and handles
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 28
MPI Communicator size
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 29
MPI Communicator size
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 30
MPI Process rank
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 31
MPI Process rank
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 32
MPI exiting
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 33
MPI communications
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 34
MPI basic datatypes for C
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 35
MPI C data types
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 36
MPI messages
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 37
MPI point-to-point communications
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 38
MPI point-to-point messages
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 39
MPI point-to-point messages requirements
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 40
Synchronous versus Asynchronous communications
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 41
MPI blocking standard Send
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 42
Send and Receive code fragment
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 43
Other MPI Send options
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 44
Considerations for MPI Send
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 45
Performance of MPI Send
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 46
MPI blocking Receive
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 47
MPI Receive considerations
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 48
MPI message passing restrictions
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 49
Simple MPI ‘ping-pong’
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 50
MPI deadlock scenario
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 51
MPI correctness and fairness
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 52
MPI bidirectional communication
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 53
MPI non-blocking communication
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 54
MPI non-blocking standard send
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 55
MPI non-blocking receive
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 56
Avoiding Deadlocks
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 57
MPI non-blocking sends
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 58
MPI non-blocking Synchronous Send
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 59
MPI testing instead of waiting
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 60
MPI with many outstanding calls
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 61
Other MPI communications calls
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 62
MPI persistent communication
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 63
MPI persistent communications
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 64
Useful MPI wildcards and constants
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 65
MPI collective communications
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 66
MPI synchronization
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 67
MPI broadcast
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 68
MPI gather
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 69
MPI scatter
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 70
MPI gather to all
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 71
Using vectors
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 72
Using vectors non-contiguously
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 73
Using binary trees
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 74
Using other kinds of trees
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 75
All-to-All personalized communication
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 76
MPI program structure summary
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 77
MPI program summary
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 78
MPI Simple C program
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 79
MPI Safe C program
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 80
MPI Deadlocking C program
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 81
MPI buffering dependent C program
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 82
Summarizing Distributed versus Shared Memory
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 83
Hybrid programming
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture 5 - 2013 84
Why use hybrid programming?
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.