introduzione al message passing interface (mpi)enrico/ingegneriadel... · processes identification...
TRANSCRIPT
![Page 1: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/1.jpg)
Introduzione al Message Passing
Interface (MPI)Interface (MPI)
Andrea Clematis
IMATI – CNR
![Page 2: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/2.jpg)
Ack. & riferimenti
• An Introduction to MPI Parallel Programming with the
Message Passing InterfaceWilliam Gropp ,Ewing Lusk Argonne National Laboratory
• Online examples available at• Online examples available at
http://www.mcs.anl.gov/mpi/tutorials/perf
• ftp://ftp.mcs.anl.gov/mpi/mpiexmpl.tar.gz contains source code and
run scripts that allows you to evaluate your own MPI implementation
![Page 3: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/3.jpg)
Sommario
�Comunicazione tra due processi
�Aspetti generali di MPI �Aspetti generali di MPI
�Specificazione della comunicazione in MPI
![Page 4: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/4.jpg)
The Message-Passing Model
• A process is (traditionally) a program counter and address
space.
• Processes may have multiple threads (program counters
and associated stacks) sharing a single address space. MPI and associated stacks) sharing a single address space. MPI
is for communication among processes, which have
separate address spaces.
• Interprocess communication consists of
– Synchronization
– Movement of data from one process’s address space to
another’s.
![Page 5: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/5.jpg)
Cooperative Operations for Communication
• The message-passing approach makes the exchange of data
cooperative.
• Data is explicitly sent by one process and received by
another.
• An advantage is that any change in the receiving process’s
memory is made with the receiver’s explicit participation.
• Communication and synchronization are combined.
Process 0 Process 1
Send(data)
Receive(data)
![Page 6: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/6.jpg)
One-Sided Operations for Communication
• One-sided operations between processes include remote memory reads
and writes
• Only one process needs to explicitly participate.
• An advantage is that communication and synchronization are
decoupleddecoupled
• One-sided operations are part of MPI-2.
Process 0 Process 1
Put(data)
(memory)
(memory)
Get(data)
![Page 7: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/7.jpg)
What is message passing?
• Data transfer plus synchronization
DataProcess 0 May I Send? DataData
DataData
• Requires cooperation of sender and receiver
• Cooperation not always apparent in code
Process 1 Yes
Data
Data
Data
Time
![Page 8: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/8.jpg)
Sommario
�Comunicazione tra due processi
�Aspetti generali di MPI �Aspetti generali di MPI
�Specificazione della comunicazione in MPI
![Page 9: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/9.jpg)
What is MPI?
• A message-passing library specification
– extended message-passing model
– not a language or compiler specification
– not a specific implementation or product– not a specific implementation or product
• For parallel computers, clusters, and heterogeneous networks
• Full-featured
• Designed to provide access to advanced parallel hardware for
– end users
– library writers
– tool developers
![Page 10: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/10.jpg)
MPI Sources
• The Standard itself:
– at http://www.mpi-forum.org
– All MPI official releases, in both postscript and HTML
• Books:
– Using MPI: Portable Parallel Programming with the Message-Passing Interface, by Gropp, Lusk, and Skjellum, MIT Press, 1994.
– MPI: The Complete Reference, by Snir, Otto, Huss-Lederman, Walker, and – MPI: The Complete Reference, by Snir, Otto, Huss-Lederman, Walker, and Dongarra, MIT Press, 1996.
– Designing and Building Parallel Programs, by Ian Foster, Addison-Wesley, 1995.
– Parallel Programming with MPI, by Peter Pacheco, Morgan-Kaufmann, 1997.
– MPI: The Complete Reference Vol 1 and 2,MIT Press, 1998(Fall).
• Other information on Web:
– at http://www.mcs.anl.gov/mpi
– pointers to lots of stuff, including other talks and tutorials, a FAQ, other MPI pages
![Page 11: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/11.jpg)
Why Use MPI?
• MPI provides a powerful, efficient, and portable
way to express parallel programs
• MPI was explicitly designed to enable libraries…
• … which may eliminate the need for many users
to learn (much of) MPI
![Page 12: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/12.jpg)
A Minimal MPI Program (C)
#include "mpi.h"
#include <stdio.h>
int main( int argc, char *argv[] )int main( int argc, char *argv[] )
{
MPI_Init( &argc, &argv );
printf( "Hello, world!\n" );
MPI_Finalize();
return 0;
}
![Page 13: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/13.jpg)
A Minimal MPI Program (Fortran)
program main
use MPI
integer ierr
call MPI_INIT( ierr )
print *, 'Hello, world!'
call MPI_FINALIZE( ierr )
end
![Page 14: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/14.jpg)
• #include "mpi.h" provides basic MPI
definitions and types
• MPI_Init starts MPI• MPI_Init starts MPI
• MPI_Finalize exits MPI
• Note that all non MPI routines are local
thus the printf run on each process the actual
behaviour depends on implementation
![Page 15: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/15.jpg)
Notes on C and Fortran
• C and Fortran bindings correspond closely
• In C:
– mpi.h must be #included
– MPI functions return error codes or MPI_SUCCESS
• In Fortran:
– mpif.h must be included, or use MPI module (MPI-2)
– All MPI calls are to subroutines, with a place for the
return code in the last argument.
• C++ bindings, and Fortran-90 issues, are part of MPI-2.
![Page 16: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/16.jpg)
Error Handling
• By default, an error causes all processes to abort.
• The user can cause routines to return (with an error code)
instead.
– In C++, exceptions are thrown (MPI-2)– In C++, exceptions are thrown (MPI-2)
• A user can also write and install custom error handlers.
• Libraries might want to handle errors differently from
applications.
![Page 17: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/17.jpg)
Running MPI Programs
• The MPI-1 Standard does not specify how to run an MPI program, just
as the Fortran standard does not specify how to run a Fortran program.
• In general, starting an MPI program is dependent on the
implementation of MPI you are using, and might require various implementation of MPI you are using, and might require various
scripts, program arguments, and/or environment variables.
• mpiexec <args> is part of MPI-2, as a recommendation, but not
a requirement
![Page 18: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/18.jpg)
Finding Out About the Environment
• Two important questions that arise early in a parallel program
are:
– How many processes are participating in this computation?
– Which one am I?
• MPI provides functions to answer these questions:
– MPI_Comm_size reports the number of processes.
– MPI_Comm_rank reports the rank, a number between 0
and size-1, identifying the calling process
![Page 19: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/19.jpg)
Better Hello (C)
#include "mpi.h"
#include <stdio.h>
int main( int argc, char *argv[] )
{{
int rank, size;
MPI_Init( &argc, &argv );
MPI_Comm_rank( MPI_COMM_WORLD, &rank );
MPI_Comm_size( MPI_COMM_WORLD, &size );
printf( "I am %d of %d\n", rank, size );
MPI_Finalize();
return 0;
}
![Page 20: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/20.jpg)
Sommario
�Comunicazione tra due processi
�Aspetti generali di MPI �Aspetti generali di MPI
�Specificazione della comunicazione in MPI
![Page 21: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/21.jpg)
MPI Basic Send/Receive
• We need to fill in the details in
Process 0 Process 1
Send(data)
Receive(data)
• Things that need specifying:
– How will “data” be described?
– How will processes be identified?
– How will the receiver recognize/screen messages?
– What will it mean for these operations to complete?
Receive(data)
![Page 22: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/22.jpg)
Message passing before MPI
A typical send looks like:
send (dest, type, address, length);
where:
dest : is an integer identifier representing the process to
receive the message;
type: is a non negative integer that the destination may
use to screen the messages;
address, length: describe a contiguosu area in memory
containing the message to be sent
![Page 23: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/23.jpg)
Message passing before MPI
A typical global operation looks like:
broadcast (type, address, length);
All of these specifications are a good match to hardware,
ease to understand but too unflexible
![Page 24: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/24.jpg)
Message passing before MPI – the buffer
Sending and receiving only a contiguous array of bytes:
• hides the real data structure from hardware which might be
able to handle it directly;
•requires pre-packing dispersed data
•rows of a matrix stored columnwise;
•general collections of structures
•prevents communication between machines with different
representations (even lengths) for same data types
![Page 25: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/25.jpg)
Message passing in MPI – generalizing the buffer
Specified in MPI by starting address, datatype and count, where datatype is:
• elementary (all C and Fortran datatypes)
•contiguous array of datatypes;
•strided blocks of datatypes;
•indexed array of blocks of datatypes;
•general structures•general structures
Datatypes are constructed recursively;
Specification of elementary datatypes allow heterogeneous communications;
Elimination of length in favour of count is clearer;
Specifying application oriented layout of data allows maximal use of special
hardware
![Page 26: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/26.jpg)
Why Datatypes?
• Since all data is labeled by type, an MPI implementation can support
communication between processes on machines with very different
memory representations and lengths of elementary datatypes
(heterogeneous communication).
• Specifying application-oriented layout of data in memory
– reduces memory-to-memory copies in the implementation
– allows the use of special hardware (scatter/gather) when available
![Page 27: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/27.jpg)
MPI Datatypes
• The data in a message to sent or received is described by a triple
(address, count, datatype), where
• An MPI datatype is recursively defined as:
– predefined, corresponding to a data type from the language (e.g.,
MPI_INT, MPI_DOUBLE_PRECISION)MPI_INT, MPI_DOUBLE_PRECISION)
– a contiguous array of MPI datatypes
– a strided block of datatypes
– an indexed array of blocks of datatypes
– an arbitrary structure of datatypes
• There are MPI functions to construct custom datatypes, such an array
of (int, float) pairs, or a row of a matrix stored columnwise.
![Page 28: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/28.jpg)
The type of messages and MPI Tag
• A separate communication context for each family of messages used for queueing and matching
• This has often been simulated in the past by
overloading the tag fieldoverloading the tag field
• No wild cards allowed for security
• Allocated by the system for security
• Types tags in MPI retained for normal use wildcards OK
![Page 29: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/29.jpg)
MPI Tags
• Messages are sent with an accompanying user-defined
integer tag, to assist the receiving process in identifying
the message.
• Messages can be screened at the receiving end by • Messages can be screened at the receiving end by
specifying a specific tag, or not screened by specifying
MPI_ANY_TAG as the tag in a receive.
• Some non-MPI message-passing systems have called tags
“message types”. MPI calls them tags to avoid confusion
with datatypes.
![Page 30: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/30.jpg)
Processes Identification: the library problem
Sub1 and Sub2 are from different libraries
The code:
Sub1();
The example is due to Marc Snir
Sub2();
is executed in parallel by Procees0, Process1, and
Process2;
![Page 31: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/31.jpg)
The library problem
Correct execution of Correct execution of communication
![Page 32: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/32.jpg)
The library problem
Incorrect execution of Incorrect execution of communication
![Page 33: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/33.jpg)
Processes Identification: the library problem
Sub1a and Sub1b are from the same library
The code:
Sub1a();
Sub2();
The example is due to Marc Snir
Sub2();
Sub1b();
is executed in parallel by Procees0, Process1, and
Process2 as shown in next slide;
![Page 34: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/34.jpg)
The library problem
Correct execution of Correct execution of communication (one pending)
![Page 35: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/35.jpg)
The library problem
Incorrect execution of Incorrect execution of communication (one pending)
![Page 36: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/36.jpg)
Processes identification in MPI
• Processes can be collected into groups.
• Each message is sent in a context, and must be received in
the same context.
• A group and context together form a communicator.• A group and context together form a communicator.
• A process is identified by its rank in the group associated
with a communicator.
• There is a default communicator whose group contains all
initial processes, called MPI_COMM_WORLD.
![Page 37: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/37.jpg)
Tags and Contexts
• Separation of messages used to be accomplished by use of tags, but
– this requires libraries to be aware of tags used by other libraries.
– this can be defeated by use of “wild card” tags.
• Contexts are different from tags
– no wild cards allowed
– allocated dynamically by the system when a library sets up a
communicator for its own use.
• User-defined tags still provided in MPI for user convenience in
organizing application
• Use MPI_Comm_split to create new communicators
![Page 38: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/38.jpg)
MPI Basic (Blocking) Send
MPI_SEND (start, count, datatype, dest, tag, comm)
• The message buffer is described by (start, count, • The message buffer is described by (start, count, datatype).
• The target process is specified by dest, which is the rank of
the target process in the communicator specified by comm.
• When this function returns, the data has been delivered to the
system and the buffer can be reused. The message may not
have been received by the target process.
![Page 39: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/39.jpg)
MPI Basic (Blocking) Receive
MPI_RECV(start, count, datatype, source, tag, comm, status)
• Waits until a matching (on source and tag) message is received• Waits until a matching (on source and tag) message is received
from the system, and the buffer can be used.
• source is rank in communicator specified by comm, or
MPI_ANY_SOURCE.
• status contains further information
• Receiving fewer than count occurrences of datatype is OK,
but receiving more is an error.
![Page 40: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/40.jpg)
Retrieving Further Information
• Status is a data structure allocated in the user’s program.
• In C:
int recvd_tag, recvd_from, recvd_count;
MPI_Status status;
MPI_Recv(..., MPI_ANY_SOURCE, MPI_ANY_TAG, ..., &status )
recvd_tag = status.MPI_TAG;recvd_tag = status.MPI_TAG;
recvd_from = status.MPI_SOURCE;
MPI_Get_count( &status, datatype, &recvd_count );
• In Fortran:
integer recvd_tag, recvd_from, recvd_count
integer status(MPI_STATUS_SIZE)
call MPI_RECV(..., MPI_ANY_SOURCE, MPI_ANY_TAG, .. status, ierr)
tag_recvd = status(MPI_TAG)
recvd_from = status(MPI_SOURCE)
call MPI_GET_COUNT(status, datatype, recvd_count, ierr)
![Page 41: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/41.jpg)
Simple Fortran Example - 1
program main
use MPI
integer rank, size, to, from, tag, count, i, ierr
integer src, dest
integer st_source, st_tag, st_count
integer status(MPI_STATUS_SIZE)
double precision data(10)
call MPI_INIT( ierr )
call MPI_COMM_RANK( MPI_COMM_WORLD, rank, ierr )
call MPI_COMM_SIZE( MPI_COMM_WORLD, size, ierr )
print *, 'Process ', rank, ' of ', size, ' is alive'
dest = size - 1
src = 0
![Page 42: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/42.jpg)
Simple Fortran Example - 2
if (rank .eq. 0) then
do 10, i=1, 10
data(i) = i
10 continue
call MPI_SEND( data, 10, MPI_DOUBLE_PRECISION,call MPI_SEND( data, 10, MPI_DOUBLE_PRECISION,
+ dest, 2001, MPI_COMM_WORLD, ierr)
else if (rank .eq. dest) then
tag = MPI_ANY_TAG
source = MPI_ANY_SOURCE
call MPI_RECV( data, 10, MPI_DOUBLE_PRECISION,
+ source, tag, MPI_COMM_WORLD,
+ status, ierr)
![Page 43: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/43.jpg)
Simple Fortran Example - 3
call MPI_GET_COUNT( status,MPI_DOUBLE_PRECISION,
st_count, ierr )
st_source = status( MPI_SOURCE )
st_tag = status( MPI_TAG )st_tag = status( MPI_TAG )
print *, 'status info: source = ', st_source,
+ ' tag = ', st_tag, 'count = ', st_count
endif
call MPI_FINALIZE( ierr )
end
![Page 44: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/44.jpg)
MPI is Simple
• Many parallel programs can be written using just these six functions, only two of which are non-trivial:
– MPI_INIT
– MPI_FINALIZE– MPI_FINALIZE
– MPI_COMM_SIZE
– MPI_COMM_RANK
– MPI_SEND
– MPI_RECV
• Point-to-point (send/recv) isn’t the only way...
![Page 45: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/45.jpg)
main(int argc, char **argv)
{
int me, size;
int SOME_TAG = 0;
...
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &me); /* local */
MPI_Comm_size(MPI_COMM_WORLD, &size); /* local */
if ((me % 2) == 0)
{ /* send unless highest-numbered process */
if ((me + 1) < size)
MPI_Send(..., me + 1, SOME_TAG, MPI_COMM_WORLD);
}else
MPI_Recv(..., me - 1, SOME_TAG, MPI_COMM_WORLD);
...
MPI_Finalize();
}
![Page 46: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/46.jpg)
main(int argc, char **argv)
{
int me, size;
int SOME_TAG = 0;
...
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &me); /* local */
MPI_Comm_size(MPI_COMM_WORLD, &size); /* local */
if ((me % 2) == 1) if ((me % 2) == 1)
{ /* send unless highest-numbered process */
MPI_Send(..., me - 1, SOME_TAG, MPI_COMM_WORLD);
}else
if ((me + 1) < size)
MPI_Recv(..., me + 1, SOME_TAG, MPI_COMM_WORLD);
...
MPI_Finalize();
}
![Page 47: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/47.jpg)
main(int argc, char **argv)
{
int me, size;
int SOME_TAG = 0;
...
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &me); /* local */
MPI_Comm_size(MPI_COMM_WORLD, &size); /* local */
if ((me % 2) == 0)
{ /* send unless highest-numbered process */
if ((me + 1) < size)
MPI_Send(..., me + 1, SOME_TAG, MPI_COMM_WORLD); MPI_Send(..., me + 1, SOME_TAG, MPI_COMM_WORLD);
if (me != 0)
MPI_Recv(..., me - 1, SOME_TAG, MPI_COMM_WORLD);
}else
{ MPI_Recv(..., me - 1, SOME_TAG, MPI_COMM_WORLD);
if ((me + 1) < size)
MPI_Send(..., me + 1, SOME_TAG, MPI_COMM_WORLD);
}
...
MPI_Finalize();
}
![Page 48: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/48.jpg)
Types of Point-to-Point Operations:
• There are different types of send and receive routines used for different purposes. For example: – Synchronous send
– Blocking send / blocking receive
– Non-blocking send / non-blocking receive – Non-blocking send / non-blocking receive
– Buffered send
– Combined send/receive
– "Ready" send
• Any type of send routine can be paired with any type of receive routine.
• MPI also provides several routines associated with send - receive operations, such as those used to wait for a message's arrival or probe to find out if a message has arrived.
![Page 49: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/49.jpg)
Data Movement in point to point communication
![Page 50: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/50.jpg)
Data Movement in point to point communication
![Page 51: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/51.jpg)
Buffering
• In a perfect world, every send operation would be perfectly synchronized with its matching receive. This is rarely the case. Somehow or other, the MPI implementation must be able to deal with storing data when the two tasks are out of sync.
• Consider the following two cases:
• A send operation occurs 5 seconds before the receive is ready - where is the message while the receive is pending?
• Multiple sends arrive at the same receiving task which can only accept one send at a time - what • Multiple sends arrive at the same receiving task which can only accept one send at a time - what happens to the messages that are "backing up"?
• The MPI implementation (not the MPI standard) decides what happens to data in these types of cases. Typically, a system buffer area is reserved to hold data in transit. For example:
![Page 52: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/52.jpg)
Buffering
• System buffer space is:
– Opaque to the programmer and managed entirely by the MPI
library
– A finite resource that can be easy to exhaust – A finite resource that can be easy to exhaust
– Often mysterious and not well documented
– Able to exist on the sending side, the receiving side, or both
– Something that may improve program performance because it
allows send - receive operations to be asynchronous.
• User managed address space (i.e. your program variables) is called the
application buffer. MPI also provides for a user managed send buffer.
![Page 53: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/53.jpg)
Blocking vs. Non-blocking
• Most of the MPI point-to-point routines can be used in either blocking or non-blocking mode.
• Blocking: – A blocking send routine will only "return" after it is safe to modify the
application buffer (your send data) for reuse. Safe means that modifications will not affect the data intended for the receive task. Safe modifications will not affect the data intended for the receive task. Safe does not imply that the data was actually received - it may very well be sitting in a system buffer.
– A blocking send can be synchronous which means there is handshaking occurring with the receive task to confirm a safe send.
– A blocking send can be asynchronous if a system buffer is used to hold the data for eventual delivery to the receive.
– A blocking receive only "returns" after the data has arrived and is ready for use by the program.
![Page 54: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/54.jpg)
Blocking vs. Non-blocking
• Non-blocking: – Non-blocking send and receive routines behave similarly - they will return
almost immediately. They do not wait for any communication events to complete, such as message copying from user memory to system buffer space or the actual arrival of message.
– Non-blocking operations simply "request" the MPI library to perform the – Non-blocking operations simply "request" the MPI library to perform the operation when it is able. The user can not predict when that will happen.
– It is unsafe to modify the application buffer (your variable space) until you know for a fact the requested non-blocking operation was actually performed by the library. There are "wait" routines used to do this.
– Non-blocking communications are primarily used to overlap computation with communication and exploit possible performance gains.
![Page 55: Introduzione al Message Passing Interface (MPI)enrico/IngegneriaDel... · Processes identification in MPI • Processes can be collected into groups. • Each message is sent in a](https://reader034.vdocument.in/reader034/viewer/2022050215/5f60bed5a6780c00b755ecc2/html5/thumbnails/55.jpg)
Order and Fairness
• Order: – MPI guarantees that messages will not overtake each other.
– If a sender sends two messages (Message 1 and Message 2) in succession to the same destination, and both match the same receive, the receive operation will receive Message 1 before Message 2.
– If a receiver posts two receives (Receive 1 and Receive 2), in succession, and both are looking for the same message, Receive 1 will receive the message before Receive 2.
– Order rules do not apply if there are multiple threads participating in the communication operations.
– Order rules do not apply if there are multiple threads participating in the communication operations.
• Fairness: – MPI does not guarantee fairness - it's up to the programmer to prevent "operation starvation".
– Example: task 0 sends a message to task 2. However, task 1 sends a competing message that matches task 2's receive. Only one of the sends will complete.