programming using mpi - aau
TRANSCRIPT
![Page 2: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/2.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 2
Topic Overview n Principles of Message-Passing Programming n MPI: the Message Passing Interface n Topologies and Embedding n Overlapping Communication with
Computation n Collective Communication and Computation
Operations n Groups and Communicators
![Page 3: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/3.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 3
Why MPI? n One of the oldest libraries (supercomputing
1992). n Wide-spread adoption, portable. n Minimal requirements on hardware. n Explicit parallelization.
n Intellectually demanding. n High performance. n Scales to large number of processors. n Private address space → no mutex.
?
![Page 4: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/4.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 4
MPI Difficulties n Many low-level details left to the
programmer. n Specify both sending & receiving messages. n Sending & receiving must match (order,
length, types). n Efficient non-blocking communication more
difficult to use because of many assumptions. n Deadlocks & livelocks. n Distributed algorithms with more complex
communication protocols.
![Page 5: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/5.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 5
MPI: The Message Passing Interface n Standard library to develop portable
message-passing programs using either C or Fortran.
n The API defines the syntax and the semantics of a core set of library routines. n Vendor implementations of MPI are available on
almost all commercial parallel computers. n It is possible to write fully-functional
message-passing programs by using only the six routines.
![Page 6: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/6.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 6
MPI Basics n Global view “language” (library in fact). n Only way to communicate is by sending
messages. n Private address space. n Number of processes is fixed. n Process scheduling external to MPI. n Run the same program on different
machines.
![Page 7: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/7.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 7
MPI – The Big Picture
host declaration
start the environment: lamboot/- run the application: mpirun … end the environment: wipe/-
![Page 8: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/8.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 8
MPI Features n Communicator information (com. domain). n Point to point communication. n Collective communication. n Topology support. n Error handling.
send(const void *sendbuf, int nelem, int dest) receive(void *recvbuf, int nelem, int src)
![Page 9: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/9.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 9
Six Golden MPI Functions n Total ~125 functions. n 6 most used function.
MPI_Init Initializes MPI. MPI_Finalize Terminates MPI. MPI_Comm_size Determines the number of processes. MPI_Comm_rank Determines the label of the calling process. MPI_Send Sends a message. MPI_Recv Receives a message.
![Page 10: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/10.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 10
MPI Functions: Initialization n Must be called once by all processes. n MPI_SUCCESS (if successful). n #include <mpi.h>
int MPI_Init(int *argc, char ***argv) int MPI_Finalize()
![Page 11: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/11.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 11
MPI Functions: Communicator n Concept of communication domain. n MPI_COMM_WORLD default for all
processes involved. n If there is a single process per processor,
MPI_Comm_size(MPI_COMM_WORLD, &size) returns the number of processors.
int MPI_Comm_size(MPI_Comm comm, int *size) int MPI_Comm_rank(MPI_Comm comm, int *rank)
![Page 12: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/12.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 12
Hello World!
#include <mpi.h> int main(int argc, char *argv[]) {
int npes, myrank; MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, &npes); MPI_Comm_rank(MPI_COMM_WORLD, &myrank); printf(“From process %d out of %d, Hello world!\n”, myrank, npes); MPI_Finalize(); return 0;
}
![Page 13: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/13.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 13
MPI Functions: Send, Recv n Wildcard for source: MPI_ANY_SOURCE.
int MPI_Send(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) int MPI_Recv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Status *status)
typedef struct MPI_Status { int MPI_SOURCE; int MPI_TAG; int MPI_ERROR;
};
![Page 14: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/14.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 14
Length of Received Message n Not directly accessible. n Reminder: The returned int says if the call
was successful or not.
int MPI_Get_count(MPI_Status *status, MPI_Datatype datatype, int *count)
![Page 15: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/15.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 15
Coun
t3s
![Page 16: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/16.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 16
Coun
t3s
![Page 17: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/17.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 17
Improvement with MPI_Scatter: collective MPI communication primitive.
![Page 18: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/18.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 18
MPI Functions: Data Types n MPI_Datatype. n Correspondence MPI ↔ C data types. n MPI_BYTE and MPI_PACKED MPI specifics. n MPI_
{CHAR,SHORT,INT,LONG,FLOAT,DOUBLE…} see mpi.h.
![Page 19: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/19.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 19
Principles of Message-Passing Programming
n 2 key attributes: n partitioned address space & n only explicit parallelization.
n Logical view: p processes, each with its own exclusive address space. n Each piece of data must belong to a partition,
i.e., explicit partitioned & placed. n All interactions require cooperation of two
processes. Point to point communication.
Expensive but costs are explicit.
Minimize interactions. Local accesses.
![Page 20: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/20.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 20
MPI Programming Structure n Asynchronous.
n Hard to reason about. n Non-deterministic.
n Loosely synchronous. n Synchronize to perform interactions. n Asynchronous in-between. n Easier to reason about.
n Single Program Multiple Data.
![Page 21: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/21.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 21
Send/Recv Example
n Expected: what P1 receives is the value of ‘a’ when it was sent.
n But depending on the implementation… n Design carefully the protocol.
P0 P1 a=100; receive(&a, 1, 0); send(&a, 1, 1); printf(“%d\n”,a); a=0;
![Page 22: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/22.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 22
Latency
![Page 23: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/23.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 23
Blocking Non-Buffered Communication
![Page 24: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/24.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 24
What Happens There?
P0 P1 send(&a, 1, 1); send(&a, 1, 0); recv(&b, 1, 1); recv(&b, 1, 0);
Simple exchange of ‘a’? ?
![Page 25: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/25.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 25
Blocking Buffered Communication
With special hardware.
MPI_Bsend/MPI_Brecv
![Page 26: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/26.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 26
Examples
P0 P1 for(i = 0;i < 1000; i++) { for(i = 0; i < 1000; i++) {
produce_data(&a); receive(&a, 1, 0); send(&a, 1, 1); consume_data(&a);
} }
P0 P1 receive(&a, 1, 1); receive(&a, 1, 0); send(&b, 1, 1); send(&b, 1, 0);
OK?
Deadlock
![Page 27: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/27.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 27
Non-Blocking Non-Buffered Communication
MPI_Ibsend/MPI_Ibrecv
![Page 28: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/28.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 28
Unsafe Program
int a[10], b[10], myrank; MPI_Status status; ... MPI_Comm_rank(MPI_COMM_WORLD, &myrank); if (myrank == 0) { MPI_Send(a, 10, MPI_INT, 1, 1, MPI_COMM_WORLD); MPI_Send(b, 10, MPI_INT, 1, 2, MPI_COMM_WORLD); } else if (myrank == 1) { MPI_Recv(b, 10, MPI_INT, 0, 2, MPI_COMM_WORLD); MPI_Recv(a, 10, MPI_INT, 0, 1, MPI_COMM_WORLD); }
Match the order in which the send and the receive operations are issued.
Programmer’s responsibility.
![Page 29: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/29.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 29
Circular Dependency – Unsafe Program
int a[10], b[10], npes, myrank; MPI_Status status; ... MPI_Comm_size(MPI_COMM_WORLD, &npes); MPI_Comm_rank(MPI_COMM_WORLD, &myrank); MPI_Send(a, 10, MPI_INT, (myrank+1)%npes, 1,
MPI_COMM_WORLD); MPI_Recv(b, 10, MPI_INT, (myrank-1+npes)%npes, 1,
MPI_COMM_WORLD);
![Page 30: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/30.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 30
Circular Send – Safe Program
int a[10], b[10], npes, myrank; MPI_Status status; ... MPI_Comm_size(MPI_COMM_WORLD, &npes); MPI_Comm_rank(MPI_COMM_WORLD, &myrank); if (myrank%2 == 1) {
MPI_Send(a, 10, MPI_INT, (myrank+1)%npes, 1, MPI_COMM_WORLD); MPI_Recv(b, 10, MPI_INT, (myrank-1+npes)%npes, 1, MPI_COMM_WORLD);
} else { MPI_Recv(b, 10, MPI_INT, (myrank-1+npes)%npes, 1, MPI_COMM_WORLD); MPI_Send(a, 10, MPI_INT, (myrank+1)%npes, 1, MPI_COMM_WORLD);
}
![Page 31: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/31.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 31
Sending and Receiving Messages Simultaneously
n No circular deadlock problem.
int MPI_Sendrecv(void *sendbuf, int sendcount, MPI_Datatype senddatatype, int dest, int sendtag, void *recvbuf, int recvcount,MPI_Datatype recvdatatype, int source, int recvtag, MPI_Comm comm, MPI_Status *status)
int MPI_Sendrecv_replace(void *buf, int count, MPI_Datatype datatype, int dest, int sendtag, int source, int recvtag, MPI_Comm comm, MPI_Status *status)
Or with replace:
![Page 32: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/32.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 32
Creating Groups n Create sub-groups by selecting processes
from the current group.
int MPI_Group_incl(MPI_Group group, int size, int *ranks, MPI_Group *newGroup)
matrix of ranks, #rows = #of processes in this group #cols = size
Notes: All processes get a new rank in the new group, and we need to convert the group to a communicator.
![Page 33: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/33.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 33
![Page 34: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/34.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 34
Topologies and Embedding n MPI allows a programmer to organize processors
into logical k-D meshes. n The processor IDs in MPI_COMM_WORLD can be
mapped to other communicators (corresponding to higher-dimensional meshes) in many ways.
n The goodness of any such mapping is determined by the interaction pattern of the underlying program and the topology of the machine.
n MPI does not provide the programmer any control over these mappings… but it finds good mapping automatically.
![Page 35: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/35.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 35
Creating and Using Cartesian Topologies
n Create a new communicator. n All processes in comm_old must call this. n Embed a virtual topology onto the parallel
architecture.
int MPI_Cart_create(MPI_Comm comm_old, int ndims, int *dims, int *periods, int reorder, MPI_Comm *comm_cart)
? More processes before/after?
![Page 36: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/36.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 36
Rank-Coordinates Conversion n Dimensions must match. n Shift processes on the topology.
int MPI_Cart_coord(MPI_Comm comm_cart, int rank, int maxdims, int *coords)
int MPI_Cart_rank(MPI_Comm comm_cart, int *coords, int *rank)
int MPI_Cart_shift(MPI_Comm comm_cart, int dir, int s_step, int *rank_source, int *rank_dest)
![Page 37: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/37.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 37
Overlapping Communication with Computation
n Transmit messages without interrupting the CPU.
n Recall how blocking send/receive operations work.
n Sometimes desirable to have non-blocking.
![Page 38: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/38.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 38
Overlapping Communication with Computation
n Functions return before the operations are completed.
int MPI_Isend(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request *request)
int MPI_Irecv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Request *request)
! Allocate a request object. MPI_Request is in fact a reference (pointer) to it. Leaks…
![Page 39: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/39.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 39
Testing Completion n Sender: before overriding the data. n Receiver: before reading the data. n Test or wait completion. n De-allocate request handler.
int MPI_Test(MPI_Request *request, int *flag, MPI_Status *status)
int MPI_Wait(MPI_Request *request, MPI_Status *status)
![Page 40: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/40.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 40
Previous Example: Safe Program
int a[10], b[10], myrank; MPI_Status status; ... MPI_Comm_rank(MPI_COMM_WORLD, &myrank); if (myrank == 0) { MPI_Isend(a, 10, MPI_INT, 1, 1, …); MPI_Isend(b, 10, MPI_INT, 1, 2, …); } else if (myrank == 1) { MPI_Irecv(b, 10, MPI_INT, 0, 2, …); MPI_Irecv(a, 10, MPI_INT, 0, 1, …); }
One unblocking call is enough since it can be matched by a blocking call.
![Page 41: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/41.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 41
Collective Operation – Later
n One-to-all broadcast – MPI_Bcast. n All-to-one reduction – MPI_Reduce. n All-to-all broadcast – MPI_Allgather. n All-to-all reduction – MPI_Reduce_scatter. n All-reduce and prefix sum – MPI_Allreduce. n Scatter – MPI_Scatter. n Gather – MPI_Gather. n All-to-all personalized – MPI_Alltoall.
![Page 42: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/42.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 42
Collective Communication and Computation Operations
n Common collective operations supported. n Over a group or processes corresponding to a
communicator. n All processes in the communicator must call
these functions.
n These operations act like a virtual synchronization step.
![Page 43: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/43.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 43
Barrier n Communicator: Group of processes that are
synchronized. n The function returns after all processes in
the group have called the function.
int MPI_Barrier(MPI_Comm comm)
![Page 44: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/44.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 44
Barrier
![Page 45: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/45.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 45
One-to-All Broadcast n All the processes must call this function,
even the receivers.
int MPI_Bcast(void *buf, int count, MPI_Datatype datatype,
int source, MPI_Comm comm)
P0
P1
P2
P3
P1
P2
P3
P0 Broadcast
Reduce
![Page 46: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/46.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 46
All-to-One Reduction n Combine elements in sendbuf (of each process in
the group) using the operation op and return in recvbuf of target.
n Predefined operations: min, max, sum,… associative.
int MPI_Reduce(void *sendbuf, void *recvbuf, int count,MPI_Datatype datatype, MPI_Op op, int target, MPI_Comm comm)
?
![Page 47: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/47.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 47
Special Operations
n MPI_MAXLOC and MPI_MINLOC work on pairs (vi, li).
n Compare with vi, use li to break ties, and return (l,v).
n Additional MPI data-pair types defined.
Value for comparison = key.
Payload.
![Page 48: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/48.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 48
Example
Value 15 17 11 12 17 11
Process 0 1 2 3 4 5
MinLoc? MaxLoc?
![Page 49: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/49.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 49
All-Reduce n No target argument since all processes
receive the result.
int MPI_Allreduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm)
P0
P1
P2
P3
P1
P2
P3
P0
All-reduce
![Page 50: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/50.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 50
Prefix-Operations n Not only sums. n Process j has prefix sj as expected.
int MPI_Scan(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm)
a
b
c
d
P0
P1
P2
P3
P1
P2
P3
P0
Prefix-Scan
a
ab
abc
abcd
![Page 51: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/51.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 51
Scatter and Gather
P0
P1
P2
P3
P1
P2
P3
P0 Scatter
Gather
![Page 52: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/52.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 52
All-Gather n Variant of gather.
P0
P1
P2
P3
P1
P2
P3
P0
All-Gather
![Page 53: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/53.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 53
All-to-All Personalized
P0
P1
P2
P3
P1
P2
P3
P0 All-to-All
Personalized
![Page 54: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/54.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 54
Example Matrix*Vector
Partition on rows.
Allgather (All-to-all broadcast)
Multiply
![Page 55: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/55.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 55
Groups and Communicators n How to partition a group of processes into
sub-groups? n Group by color (different communicators). n Sort by key (new ranks in the sub-groups).
int MPI_Comm_split(MPI_Comm comm, int color, int key, MPI_Comm *newcomm)
![Page 56: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/56.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 56
Split Example
P0: MPI_Comm_split(oldc, 0, 1, ...) P0 P1: MPI_Comm_split(oldc, 0, 1, ...) P1 P2: MPI_Comm_split(oldc, 0, 1, ...) P2 P3: MPI_Comm_split(oldc, 1, 1, ...) P0 P4: MPI_Comm_split(oldc, 1, 1, ...) P1 P5: MPI_Comm_split(oldc, 1, 1, ...) P2 P6: MPI_Comm_split(oldc, 1, 1, ...) P3 P7: MPI_Comm_split(oldc, 2, 1, ...) P0
colo
r ke
y
new
gro
ups
![Page 57: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/57.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 57
Splitting Cartesian Topologies n Split Cartesian topology into lower
dimensional grids.
int MPI_Cart_sub(MPI_Comm comm_cart, int *keep_dims, MPI_Comm *comm_subcart)
Original group.
New group. Tell which dimensions to keep, e.g, 2x4x7 and {1,0,1} → 4* sub (2x7)
![Page 58: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/58.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 58
Example
4
2
7 original 2x4x7
(1,0,1) -> 4* (2x7)
2
7
(0,0,1) -> 2*4* (7)
7
![Page 59: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/59.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 59
Example Matrix*Vector
Partition.
Row sub-topology. Colum sub-topology.
Distribute vector.
Local multiplication.
Sum reduce on rows.
![Page 60: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/60.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 60
Performance Evaluation n Elapsed time.
double t1, t2; t1=MPI_Wtime(); … t2=MPI_Wtime(); printf(“Elapsed time is %f sec\n”, t2-t1);
![Page 61: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/61.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 61
Example: Successive Over-Relaxation
![Page 62: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/62.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 62
Spot the problem
![Page 63: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/63.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 63
![Page 64: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/64.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 64
![Page 65: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/65.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 65
![Page 66: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/66.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 66
Performance Issues n Large overhead for every message sent.
n → reduce the number of messages. Minimize interactions. Combine small messages into larger ones.
n MPI programs are static in nature. n Large overhead in communication + fixed
number of processes. n Dynamic work distribution possible but expensive.
n Use non-blocking communication.
![Page 67: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/67.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 67
Dynamic Work Redistribution
![Page 68: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/68.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 68
MPI_Isend
MPI_Irecv
SOR with non-blocking communication
![Page 69: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/69.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 69
MPI_Irecv
Compute inner values.
MPI_Waitall
Compute top + bottom.
![Page 70: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/70.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 70
Compute left+right.
MPI_Reduce
![Page 71: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/71.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 71
Howto
n Compile a hello.c MPI program: n mpicc –Wall –O2 –o hello hello.c
n Start Lam: n lamboot
n Run: n mpirun –np 4 ./hello
n Clean-up before logging off: n wipe
![Page 72: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/72.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 72
MPI In Practice
n Write a configure file hosts with n homer.cs.aau.dk cpu=4
marge.cs.aau.dk cpu=4 bart.cs.aau.dk cpu=4 lisa.cs.aau.dk cpu=4
n Start/stop lam: n export LAMRSH=‘ssh -x’ n lamboot/wipe –b hosts
n Run MPI: n mpirun –np 8 <path>/hello
Which computers to use. They all have the same MPI installation.
![Page 73: Programming Using MPI - AAU](https://reader033.vdocument.in/reader033/viewer/2022052004/62859f8b8ddde21918521e27/html5/thumbnails/73.jpg)
21/3-11/4-2011 MVP'11 - Aalborg University 73
Alternatives to MPI n Partitioned global address space languages.
n Abstract communication. n Use local & non-local memories. n Conceptually close to Peril-L. n Examples:
n Co-Array Fortran n Unified Parallel C n Titanium
n Older distributed shared memory systems n Linda: distributed tuple space.