isbi mpi tutorial
Post on 11-May-2015
2.111 Views
Preview:
TRANSCRIPT
MPI & Distributed Computing
Eric Borisch, M.S.Mayo Clinic
Topics
Motivation for distributed computingWhat MPI is Intro to MPI programmingThinking in parallelWrap up
Shared vs. Distributed Memory Shared Memory: all memory within a system is
directly addressable (ignoring access restrictions) by each process [or thread] Single- and multi-CPU desktops & laptops Multi-threaded apps GPGPU * MPI *
Distributed Memory: memory available a given node within a system is unique and distinct from its peers MPI Google MapReduce / Hadoop
Why bother?
1 2 3 4 5 6 7 80
0.5
1
1.5
2
2.5
Centos 5.2; Dual Quad-Core 3GHz P4 [E5472]; DDR2 800MHz
CopyScaleAddTriad
# of processes
Rela
tive p
erf
orm
an
ce
http://www.cs.virginia.edu/stream/
But what about Nehalem?
0 4 8 12 160%
50%100%150%200%250%300%350%400%
STREAM benchmark OpenMP per-formance
Add:Copy:Scale:Triad:
Threads (8 Physical cores + HT)
Rela
tive p
erf
orm
an
ce
http://www.cs.virginia.edu/stream/2x X5570 (2.93GHz; Quad-core; 6.4GT/s QPI); 12x4G 1033 DDR3
Memory Limitations
Bandwidth (FSB, HT, Nehalem, CUDA, …) Frequently run into with high-level languages (MATLAB)
Capacity – cost & availability High-density chips are $$$ (if even available) Memory limits on individual systems
Distributed computing addresses both bandwidth and capacity with multiple systems
MPI is the glue used to connect multiple distributed processes together
Memory Requirements [Example]
Custom iterative SENSE reconstruction 3 x 8 coils x 400 x 320 x 176 x 8 [complex
float] Profile data (img space) Estimate (img <-> k space) Acquired data (k space) > 4GB data touched during each iteration
16, 32 channel data here or on the way…
Trzasko, Josh ISBI 2009 #1349, “Practical Nonconvex Compressive Sensing Reconstruction of Highly-Accelerated 3D Parallel MR Angiograms”M01.R4: MRI Reconstruction Algorithms, Monday @ 1045 in Stanbro
FTx
Real-time SENSE unfoldingDATA
Place view into correct x-Ky-Kz space (AP & LP)
“Traditional” 2D SENSE Unfold (AP & LP)
Homodyne Correction
GW Correction (Y, Z)
GW Correction (X)
MIP
Store / DICOM
FTyz (AP & LP)CAL
RESULT
Root node
Worker nodes
Real-time data
Pre-loaded data
MPI Communication
Root Node
3.6GHz P4
3.6GHz P4
16GB RAM
80GB HDD
1Gb Eth
2x8Gb IB
Worker Node (x7)3.6GHz P4
3.6GHz P4
16GB RAM
80GB HDD2x8Gb IB
24-Port Infiniband Switch
16-Port Gigabit Ethernet Switch
1Gb Eth
MRI System
Site Intranet
x7x2 MPI interconnects16Gb/s bandwidth per node
x7 File system connections
KeyCluster Hardware
External Hardware
2x8Gig Infiniband connection
1Gig Ethernet connection
500GB HDD
1Gb Eth
1Gb Eth
8Gb/s Connection
MRI Reconstruction Cluster
Many Approaches to “Distributed”
Loosely coupled SETI / BOINC “Grid computing”
BIOS-level abstraction ScaleMP
Tightly coupled MPI “Cluster computing”
Hybrid Folding@Home gpugrid.net
http://en.wikipedia.org/wiki/File:After_Dark_Flying_Toasters.pnghttp://en.wikipedia.org/wiki/File:Setiathomeversion3point08.png
Grid vs. Cluster
Master
Worker
Worker
Worker
Worker
WorkerHead Node
WorkerNode
Interconnect
Shared vs. Distributed
HostOS
Process A
Process B
Thread 1
Thread 2
Thread N
Host IOS I
Process A
Host IIOS II
Process B
Host NOS N
Process CMemory Transfers
Network Transfers
Shared vs. Distributed
HostOS
Process A
Process B
Thread 1
Thread 2
Thread N
Host IOS I
Process A
Host IIOS II
Process B
Host NOS N
Process CMemory Transfers
Network Transfers
Process D
Process E
Process F
Topics
Motivation for distributed computingWhat MPI is Intro to MPI programmingThinking in parallelWrap up
MPI
Message Passing Interface is… “a library specification for message-
passing” 1
Available in many implementations on multiple platforms *
A set of functions for moving messages between different processes without a shared memory environment
Low-level*; no concept of overall computing tasks to be performed
[1] http://www.mcs.anl.gov/research/projects/mpi/
MPI history
MPI-1 Version 1.0 draft standard 1994 Version 1.1 in 1995 Version 1.2 in 1997 Version 1.3 in 2008
MPI-2 Added:▪ 1-sided communication▪ Dynamic “world” sizes; spawn / join
Version 2.0 in 1997 Version 2.1 in 2008
MPI-3 In process Enhanced fault handling
Forward compatibility preserved
MPI Status
MPI is the de-facto standard for distributed computing Freely available Open source implementations exist Portable Mature
From a discussion of why MPI is dominant [1]: […] 100s of languages have come and gone. Good stuff must have been created [… yet] it is broadly accepted in the
field that they’re not used. MPI has a lock. OpenMP is accepted, but a distant second. There are substantial barriers to the introduction of new languages and
language constructs. Economic, ecosystem related, psychological, a catch-22 of widespread
use, etc. Any parallel language proposal must come equipped with reasons why it
will overcome those barriers.[1] http://www.ieeetcsc.org/newsletters/2006-01/why_all_mpi_discussion.html
MPI Distributions
MPI itself is just a specification. We want an implementation MPICH, MPICH2
Widely portable MVAPICH, MVAPICH2
Infiniband-centric; MPICH/MPICH2 based OpenMPI
Plug-in architecture; many run-time options And more:
IntelMPI HP-MPI MPI for IBM Blue Gene MPI for Cray Microsoft MPI MPI for SiCortex MPI for Myrinet Express (MX) MPICH2 over SCTP
Implementing a distributed system
Without MPI: Start all of the processes across bank of
machines (shell scripting + ssh) socket(), bind(), listen(), accept() or
connect() each link send(), read() on individual links Raw byte interfaces; no discrete
messages
Implementing a distributed system
With MPI mpiexec –np <n> app MPI_Init() MPI_Send() MPI_Recv() MPI_Finalize()
MPI: Manages the connections Packages messages Provides launching mechanism
MPI (the document)1
Provides definitions for: Communication functions
MPI_Send() MPI_Recv() MPI_Bcast() etc.
Datatype mangement functions MPI_Type_create_hvector()
C, C++, and Fortran bindings Also recommends process startup
mpiexec –np <nproc> <program> <args>[1] http://www.mpi-forum.org/docs/mpi-20-html/mpi2-report.html
MPI FunctionsMPI_AbortMPI_AccumulateMPI_Add_error_classMPI_Add_error_codeMPI_Add_error_stringMPI_AddressMPI_AllgatherMPI_AllgathervMPI_Alloc_memMPI_AllreduceMPI_AlltoallMPI_AlltoallvMPI_AlltoallwMPI_Attr_deleteMPI_Attr_getMPI_Attr_putMPI_BarrierMPI_BcastMPI_BsendMPI_Bsend_initMPI_Buffer_attachMPI_Buffer_detachMPI_CancelMPI_Cart_coordsMPI_Cart_createMPI_Cart_getMPI_Cart_mapMPI_Cart_rankMPI_Cart_shiftMPI_Cart_subMPI_Cartdim_getMPI_Close_portMPI_Comm_acceptMPI_Comm_call_errhandlerMPI_Comm_compareMPI_Comm_connectMPI_Comm_createMPI_Comm_create_errhandlerMPI_Comm_create_keyvalMPI_Comm_delete_attrMPI_Comm_disconnectMPI_Comm_dupMPI_Comm_freeMPI_Comm_free_keyvalMPI_Comm_get_attrMPI_Comm_get_errhandlerMPI_Comm_get_nameMPI_Comm_get_parentMPI_Comm_groupMPI_Comm_joinMPI_Comm_rankMPI_Comm_remote_groupMPI_Comm_remote_sizeMPI_Comm_set_attrMPI_Comm_set_errhandlerMPI_Comm_set_nameMPI_Comm_sizeMPI_Comm_spawnMPI_Comm_spawn_multipleMPI_Comm_splitMPI_Comm_test_interMPI_Dims_createMPI_Errhandler_createMPI_Errhandler_freeMPI_Errhandler_getMPI_Errhandler_setMPI_Error_classMPI_Error_stringMPI_ExscanMPI_File_c2fMPI_File_call_errhandlerMPI_File_closeMPI_File_create_errhandlerMPI_File_deleteMPI_File_f2cMPI_File_get_amodeMPI_File_get_atomicityMPI_File_get_byte_offsetMPI_File_get_errhandlerMPI_File_get_groupMPI_File_get_infoMPI_File_get_positionMPI_File_get_position_sharedMPI_File_get_sizeMPI_File_get_type_extent
MPI_File_get_viewMPI_File_ireadMPI_File_iread_atMPI_File_iread_sharedMPI_File_iwriteMPI_File_iwrite_atMPI_File_iwrite_sharedMPI_File_openMPI_File_preallocateMPI_File_readMPI_File_read_allMPI_File_read_all_beginMPI_File_read_all_endMPI_File_read_atMPI_File_read_at_allMPI_File_read_at_all_beginMPI_File_read_at_all_endMPI_File_read_orderedMPI_File_read_ordered_beginMPI_File_read_ordered_endMPI_File_read_sharedMPI_File_seekMPI_File_seek_sharedMPI_File_set_atomicityMPI_File_set_errhandlerMPI_File_set_infoMPI_File_set_sizeMPI_File_set_viewMPI_File_syncMPI_File_writeMPI_File_write_allMPI_File_write_all_beginMPI_File_write_all_endMPI_File_write_atMPI_File_write_at_allMPI_File_write_at_all_beginMPI_File_write_at_all_endMPI_File_write_orderedMPI_File_write_ordered_beginMPI_File_write_ordered_endMPI_File_write_sharedMPI_FinalizeMPI_FinalizedMPI_Free_memMPI_GatherMPI_GathervMPI_GetMPI_Get_addressMPI_Get_countMPI_Get_elementsMPI_Get_processor_nameMPI_Get_versionMPI_Graph_createMPI_Graph_getMPI_Graph_mapMPI_Graph_neighborsMPI_Graph_neighbors_countMPI_Graphdims_getMPI_Grequest_completeMPI_Grequest_startMPI_Group_compareMPI_Group_differenceMPI_Group_exclMPI_Group_freeMPI_Group_inclMPI_Group_intersectionMPI_Group_range_exclMPI_Group_range_incl
MPI_Group_rankMPI_Group_sizeMPI_Group_translate_ranksMPI_Group_unionMPI_IbsendMPI_Info_createMPI_Info_deleteMPI_Info_dupMPI_Info_freeMPI_Info_getMPI_Info_get_nkeysMPI_Info_get_nthkeyMPI_Info_get_valuelenMPI_Info_setMPI_InitMPI_Init_threadMPI_InitializedMPI_Intercomm_createMPI_Intercomm_mergeMPI_IprobeMPI_IrecvMPI_IrsendMPI_Is_thread_mainMPI_IsendMPI_IssendMPI_Keyval_createMPI_Keyval_freeMPI_Lookup_nameMPI_Op_createMPI_Op_freeMPI_Open_portMPI_PackMPI_Pack_externalMPI_Pack_external_sizeMPI_Pack_sizeMPI_PcontrolMPI_ProbeMPI_Publish_nameMPI_PutMPI_Query_threadMPI_RecvMPI_Recv_initMPI_ReduceMPI_Reduce_scatterMPI_Register_datarepMPI_Request_freeMPI_Request_get_statusMPI_RsendMPI_Rsend_initMPI_ScanMPI_ScatterMPI_ScattervMPI_SendMPI_Send_initMPI_SendrecvMPI_Sendrecv_replaceMPI_SsendMPI_Ssend_initMPI_StartMPI_StartallMPI_Status_set_cancelledMPI_Status_set_elementsMPI_TestMPI_Test_cancelledMPI_TestallMPI_TestanyMPI_TestsomeMPI_Topo_testMPI_Type_commitMPI_Type_contiguousMPI_Type_create_darrayMPI_Type_create_hindexedMPI_Type_create_hvectorMPI_Type_create_indexed_block
MPI_Type_create_keyvalMPI_Type_create_resizedMPI_Type_create_structMPI_Type_create_subarrayMPI_Type_delete_attrMPI_Type_dupMPI_Type_extentMPI_Type_freeMPI_Type_free_keyvalMPI_Type_get_attrMPI_Type_get_contentsMPI_Type_get_envelopeMPI_Type_get_extentMPI_Type_get_nameMPI_Type_get_true_extentMPI_Type_hindexedMPI_Type_hvectorMPI_Type_indexedMPI_Type_lbMPI_Type_match_sizeMPI_Type_set_attrMPI_Type_set_nameMPI_Type_sizeMPI_Type_structMPI_Type_ubMPI_Type_vectorMPI_UnpackMPI_Unpack_externalMPI_Unpublish_nameMPI_WaitMPI_WaitallMPI_WaitanyMPI_WaitsomeMPI_Win_call_errhandlerMPI_Win_completeMPI_Win_createMPI_Win_create_errhandlerMPI_Win_create_keyvalMPI_Win_delete_attrMPI_Win_fenceMPI_Win_freeMPI_Win_free_keyvalMPI_Win_get_attrMPI_Win_get_errhandlerMPI_Win_get_groupMPI_Win_get_nameMPI_Win_lockMPI_Win_postMPI_Win_set_attrMPI_Win_set_errhandlerMPI_Win_set_nameMPI_Win_startMPI_Win_testMPI_Win_unlockMPI_Win_waitMPI_WtickMPI_Wtime
The message passing mindset Each process owns their data – there is no “our”
Makes many things simpler; no mutexes, condition variables, semaphores, etc; memory access order race conditions go away
Every message is an explicit copy I have the memory I sent from, you have the memory
you used to received into Even when running in a “shared memory” environment
Synchronization comes along for free I won’t get your message (or data) until you choose to
send it Programming to MPI first can make it easier to
scale-out later
Topics
Motivation for distributed computingWhat MPI is Intro to MPI programmingThinking in parallelWrap up
Getting started with MPI
Download / decompress MPICH source: http://www.mcs.anl.gov/research/projects/mpich2/ Suports: c / c++ / Fortran Requires Python >= 2.2
./configure make install
installs into /usr/local by default, or use --prefix=<chosen path>
Make sure <prefix>/bin is in PATH Make sure <prefix>/share/man is in
MANPATH
MPI Installation
c compiler wrapper c++ compiler wrapper
MPI job launcher
MPD launcher
MPD launch
Set up passwordless ssh to workers Start the daemons with mpdboot -n <N>
Requires ~/.mpd.conf to exist on each host▪ Contains: (same on each host)▪ MPD_SECRETWORD=<some gibberish string>
▪ permissions set to 600 (r/w access for owner only) Requires ./mpd.hosts to list other host names▪ Unless run as mpdboot -n 1 (run on current host
only)▪ Will not accept current host in list (implicit)
Check for running daemons with mpdtrace
For details: http://www.mcs.anl.gov/research/projects/mpich2/documentation/files/mpich2-1.0.8-installguide.pdf
MPD launch
MPI Compile & launch
Use mpicc / mpicxx for c/c++ compiler Wrapper script around c/c++ compilers
detected during install▪ $ mpicc --showgcc -march=core2 -I/usr/local/builds/mpich2-1.0.8/include -L/usr/local/builds/mpich2-1.0.8/lib -lmpich -lhdf5_cpp -lhdf5 -lpthread -luuid -lpthread –lrt
$ mpicc -o hello hello.c Use mpiexec -np <nproc> <app>
<args> to launch $ mpiexec -np 4 ./hello
Hello, Hello, Hello, world world world
/* hello.c */#include <stdio.h>#include <mpi.h>
int main (int argc, char * argv[]){
int i, rank, nodes;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &nodes);MPI_Comm_rank(MPI_COMM_WORLD, &rank);
for (i=0; i < nodes; i++){
MPI_Barrier(MPI_COMM_WORLD);if (i == rank) printf("Hello from %i of %i!\n", rank, nodes);
}MPI_Finalize();return 0;
}
$ mpicc -o hello hello.c$ mpiexec -np 4 ./hello
Hello, from 0 of 4!Hello, from 2 of 4!Hello, from 1 of 4!Hello, from 3 of 4!
Threads vs. MPI startup./
threaded_app
main()
pthread_create( func() ) func()
pthread_exit()pthread_join()
Memory
Do work Do work
exit()
Thread within threaded_app process
mpi_app [rank 3]
mpi_app [rank 0]
mpi_app [rank 1]
Threads vs. MPI startupmpiexec –np 4
./mpi_app
main() main() main()
MPI comm.MPI_Init() MPI_Init() MPI_Init()
MPI comm.MPI_Bcast() MPI_Bcast() MIP_Bcast()
Do Work on local mem
Do Work on local mem
Do Work on local mem
mpd launches jobs
MPI comm.MPI_Allreduce() MPI_Allreduce() MPI_Allreduce()
MPI comm.MPI_Finalize() MPI_Finalize() MPI_Finalize()
exit() exit() exit()
Hello, world: unique to ranks
/* hello.c */#include <stdio.h>#include <mpi.h>
intmain (int argc, char * argv[]){
int i;int rank;int nodes;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &nodes);MPI_Comm_rank(MPI_COMM_WORLD, &rank);
for (i=0; i < nodes; i++){MPI_Barrier(MPI_COMM_WORLD);if (i == rank) printf("Hello from %i of %i!\n", rank, nodes);}MPI_Finalize();return 0;
}
MPE (Multi-Process Environment)
MPICH2 comes with mpe by default (unless disabled during configure)
Multiple tracing / logging options to track MPI traffic
Enabled through –mpe=<option> at compile timeMacPro:code$ mpicc -mpe=mpilog -o hello hello.cMacPro:code$ mpiexec -np 4 ./helloHello from 0 of 4!Hello from 2 of 4!Hello from 1 of 4!Hello from 3 of 4!Writing logfile....Enabling the Default clock synchronization...Finished writing logfile ./hello.clog2.
jumpshot view of log
Output with -mpe=mpitraceMacPro:code$ mpicc -mpe=mpitrace -o hello hello.cMacPro:code$ mpiexec -np 2 ./hello > trace
MacPro:code$ grep 0 trace [0] Ending MPI_Init[0] Starting MPI_Comm_size...[0] Ending MPI_Comm_size[0] Starting MPI_Comm_rank...[0] Ending MPI_Comm_rank[0] Starting MPI_Barrier...[0] Ending MPI_BarrierHello from 0 of 2![0] Starting MPI_Barrier...[0] Ending MPI_Barrier[0] Starting MPI_Finalize...[0] Ending MPI_Finalize
MacPro:code$ grep 1 trace [1] Ending MPI_Init[1] Starting MPI_Comm_size...[1] Ending MPI_Comm_size[1] Starting MPI_Comm_rank...[1] Ending MPI_Comm_rank[1] Starting MPI_Barrier...[1] Ending MPI_Barrier[1] Starting MPI_Barrier...[1] Ending MPI_BarrierHello from 1 of 2![1] Starting MPI_Finalize...[1] Ending MPI_Finalize
A more interesting log…
3D-sinc interpolation
MPI_Send (Blocking)
int MPI_Send( void *buf,
memory location to send from
int count,number of elements (of type datatype) at buf
MPI_Datatype datatype, MPI_INT, MPI_FLOAT, etc…Or custom datatypes; strided vectors; structures, etc
int dest, rank (within the communicator comm) of destination for this message
int tag, used to distinguish this message from other messages
MPI_Comm comm )communicator for this transferoften MPI_COMM_WORLD
MPI_Recv (Blocking)
int MPI_Recv(void *buf,
memory location to receive data into
int count, number of elements (of type datatype) available to receive into at buf
MPI_Datatype datatype,MPI_INT, MPI_FLOAT, etc…Or custom datatypes; strided vectors; structures, etc.Typically matches sending datatype, but doesn’t have to…
int source, rank (within the communicator comm) of source for this messagecan also be MPI_ANY_SOURCE
int tag, used to distinguish this message from other messagescan also be MPI_ANY_TAG
MPI_Comm comm, communicator for this transferoften MPI_COMM_WORLD
MPI_Status *status )Structure describing the received message, including:
actual count (can be smaller than passed count)source (useful if used with source = MPI_ANY_SOURCE)tag (useful if used with tag = MPI_ANY_TAG)
Another example
/* sr.c */#include <stdio.h>#include <mpi.h>#ifndef SENDSIZE#define SENDSIZE 1#endif
intmain (int argc, char * argv[] ){
int i, rank, nodes, myData[SENDSIZE], theirData[SENDSIZE];MPI_Status sendStatus;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &nodes);MPI_Comm_rank(MPI_COMM_WORLD, &rank);
myData[0] = rank;MPI_Send(myData, SENDSIZE, MPI_INT, (rank + 1) % nodes, 0, MPI_COMM_WORLD);MPI_Recv(theirData, SENDSIZE, MPI_INT, (rank + nodes -1 ) % nodes, 0, MPI_COMM_WORLD, &sendStatus);
printf("%i sent %i; received %i\n", rank, myData[0], theirData[0]);
MPI_Finalize();return 0;
}
Does it run?
$ mpicc -o sr sr.c$ mpiexec -np 2 ./sr0 sent 0; received 11 sent 1; received 0
Log output (-np 4)
May != Will
$ mpicc -o sr sr.c$ mpiexec -np 2 ./sr0 sent 0; received 11 sent 1; received 0
$ mpicc -o sr sr.c -DSENDSIZE="0x1<<13”$ mpiexec -np 2 ./sr0 sent 0; received 11 sent 1; received 0
$ mpicc -o sr sr.c -DSENDSIZE="0x1<<14”$ mpiexec -np 2 ./sr^C
$ mpicc -o sr sr.c -DSENDSIZE="0x1<<14 - 1”$ mpiexec -np 2 ./sr0 sent 0; received 11 sent 1; received 0
What the standard has to say…3.4 Communication Modes
The send call described in Section Blocking send is blocking: it does not return until the message data and envelope have been safely stored away so that the sender is free to access and overwrite the send buffer. The message might be copied directly into the matching receive buffer, or it might be copied into a temporary system buffer.
Message buffering decouples the send and receive operations. A blocking send can complete as soon as the message was buffered, even if no matching receive has been executed by the receiver. On the other hand, message buffering can be expensive, as it entails additional memory-to-memory copying, and it requires the allocation of memory for buffering. MPI offers the choice of several communication modes that allow one to control the choice of the communication protocol.
The send call described in Section Blocking send used the standard communication mode. In this mode, it is up to MPI to decide whether outgoing messages will be buffered. MPI may buffer outgoing messages. In such a case, the send call may complete before a matching receive is invoked. On the other hand, buffer space may be unavailable, or MPI may choose not to buffer outgoing messages, for performance reasons. In this case, the send call will not complete until a matching receive has been posted, and the data has been moved to the receiver.
Thus, a send in standard mode can be started whether or not a matching receive has been posted. It may complete before a matching receive is posted. The standard mode send is non-local: successful completion of the send operation may depend on the occurrence of a matching receive.
http://www.mpi-forum.org/docs/mpi-11-html/node40.html#Node40
Rendezvous vs. eager (simplified)
Process 1 Process 2Send
“small” message &
returnEager send
Eager recv
Request & receive small
message
Send “large”
messageRndv. req.
Request large message
Receive Rndv. req.
Match Rndv. req.
Rndv. send
Receive Rndv. data
Receive large message
Blocks until completion.
User activityMPI activity
MPI communication modes MPI_Bsend (Buffered) (MPI_Ibsend, MPI_Bsend_init)
Sends are “local” – they return independent of any remote activity Message buffer can be touched immediately after call returns Requires a user-provided buffer, provided via MPI_Buffer_attach() Forces an “eager”-like message transfer from sender’s perspective User can wait for completion by calling MPI_Buffer_detach()
MPI_Ssend (Syncronous) (MPI_Issend, MPI_Ssend_init) Won’t return until matching receive is posted Forces a “rendezvous”-like message transfer Can be used to guarantee synchronization without additional MPI_Barrier() calls
MPI_Rsend (Ready) (MPI_Irsend, MPI_Rsend_init) Erroneous if matching receive has not been posted Performance tweak (on some systems) when user can guarantee matching receive is
posted MPI_Isend, MPI_Irecv (Immediate) (MPI_Send_init, MPI_Recv_init)
Non-blocking, immediate return once send/receive request is posted Requires MPI_[Test|Wait][|all|any|some] call to guarantee completion Send/receive buffers should not be touched until completed MPI_Request * argument used for eventual completion
The basic* receive modes are MPI_Recv() and MPI_Irecv(); either can be used to receive any send mode.
Fixing the code/* sr2.c */#include <stdio.h>#include <mpi.h>#ifndef SENDSIZE#define SENDSIZE 1#endif
intmain (int argc, char * argv[] ){
int i, rank, nodes, myData[SENDSIZE], theirData[SENDSIZE];MPI_Status xferStatus[2];MPI_Request xferRequest[2];
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &nodes);MPI_Comm_rank(MPI_COMM_WORLD, &rank);
myData[0] = rank;MPI_Isend(myData, SENDSIZE, MPI_INT, (rank + 1) % nodes, 0, MPI_COMM_WORLD, &xferRequest[0]);MPI_Irecv(theirData, SENDSIZE, MPI_INT, (rank + nodes -1) % nodes, 0, MPI_COMM_WORLD, &xferRequest[1]);
MPI_Waitall(2,xferRequest,xferStatus);
printf("%i sent %i; received %i\n", rank, myData[0], theirData[0]);
MPI_Finalize();return 0;
}
Fixed with MPI_I[send|recv]()
$ mpicc -o sr2 sr2.c -DSENDSIZE="0x1<<14”$ mpiexec -np 4 ./sr20 sent 0; received 32 sent 2; received 11 sent 1; received 03 sent 3; received 2
Topics
Motivation for distributed computingWhat MPI is Intro to MPI programmingThinking in parallelWrap up
Types of parallelism [1/2] Task parallelism
Each process handles a unique kind of task▪ Example: multi-image uploader (with resize/recompress)▪ Thread 1: GUI / user interaction▪ Thread 2: file reader & decompression▪ Thread 3: resize & recompression▪ Thread 3: network communication
Can be used in a grid with a pipeline of separable tasks to be performed on each data set▪ Resample / warp volume▪ Segment volume▪ Calculate metrics on segmented volume
Types of parallelism [2/2]
Data parallelism Each process handles a portion of the
entire data Often used with large data sets▪ [task 0… | … task 1 … | … | … task n]
Frequently used in MPI programming Each process is “doing the same thing,”
just on a different subset of the whole
Data layout
xy
z
Node 0Node 1Node 2Node 3Node 4Node 5Node 6Node 7
Layout is crucial in high-performance computing BW efficiency; cache
efficiency Even more important in
distributed Poor layout extra
communication Shown is an example of
“block” data distribution x is contiguous dimension z is slowest dimension Each node has contiguous
portion of z
FTx
Real-time SENSE unfoldingDATA
Place view into correct x-Ky-Kz space (AP & LP)
“Traditional” 2D SENSE Unfold (AP & LP)
Homodyne Correction
GW Correction (Y, Z)
GW Correction (X)
MIP
Display / DICOM
FTyz (AP & LP)CAL
RESULT
Root node
Worker nodes
Real-time data
Pre-loaded data
MPI Communication
Separability
Completely separable problems: Add 1 to everyone Multiply each a[i] * b[i]
Inseparable problems: [?] Max of a vector Sort a vector MIP of a volume 1D FFT of a volume 2d FFT of a volume 3d FFT of a volume
[Parallel sort] Pacheo, Peter S., Parallel Programming with MPI
3D-sinc interpolation
Next steps
Dynamic datatypes MPI_Type_vector() Enables communication of sub-sets without packing Combined with DMA, permits zero-copy transposes, etc.
Other collectives MPI_Reduce MPI_Scatter MPI_Gather
MPI-2 (MPICH2, MVAPICH2) One-sided (DMA) communication▪ MPI_Put()▪ MPI_Get()
Dynamic world size▪ Ability to spawn new processes during run
Topics
Motivation for distributed computingWhat MPI is Intro to MPI programmingThinking in parallelWrap up
Optimizing MPI code
Take time on the algorithm & data layout Minimize traffic between nodes / separate problem▪ FTx into xKyKz in SENSE example
Cache-friendly (linear, efficient) access patternsOverlap processing and communication
MPI_Isend() / MPI_Irecv() with multiple work buffers While actively transferring one, process the other Larger messages will hit a higher BW (in general)
Other MPI / performance thoughts
Profile Vtune (Intel; Linux / Windows) Shark (Mac) MPI profiling with -mpe=mpilog
Avoid “premature optimization” (Knuth) Implementation time & effort vs. runtime
performance Use derived datatypes rather than packing Using a debugger with MPI is hard
Build in your own debugging messages from go
Conclusion
If you might need MPI, build to MPI. Works well in shared memory environments▪ It’s getting better all the time
Encourages memory locality in NUMA architectures▪ Nehalem, AMD
Portable, reusable, open-source Can be used in conjunction with threads /
OpenMP / TBB / CUDA / OpenCL “Hybrid model of parallel programming”
Messaging paradigm can create “less obfuscated” code than threads / OpenMP
Building a cluster
Homogeneous nodes Private network
Shared filesystem; ssh communication Password-less SSH High-bandwidth private interconnect
MPI communication exclusively GbE, 10GbE Infiniband
Consider using Rocks CentOS / RHEL based Built for building clusters Rapid network boot based install/reinstall of nodes http://www.rocksclusters.org/
References
MPI documents http://www.mpi-forum.org/docs/
MPICH2 http://www.mcs.anl.gov/research/projects/mpich2 http://lists.mcs.anl.gov/pipermail/mpich-discuss/
OpenMPI http://www.open-mpi.org/ http://www.open-mpi.org/community/lists/ompi.php
MVAPICH[1|2] (Infiniband-tuned distribution) http://mvapich.cse.ohio-state.edu/ http://mail.cse.ohio-state.edu/pipermail/mvapich-discuss/
Rocks http://www.rocksclusters.org/ https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/
Books: Pacheo, Peter S., Parallel Programming with MPI Karniadakis, George E., Parallel Scientific Computing in C++ and MPI Gropp, W., Using MPI-2
Questions?
SIMD Example(Transparency painting)
Transparency painting
This is the painting operation for one RGBA pixel (in) onto another (out)
We can do red and blue together, as we know they won’t collide, and we can mask out the unwanted results.
Post-multiply masks are applied in the shifted position to minimize the number of shift operations
Note: we’re using pre-multiplied colors & painting onto an opaque background
#define RB 0x00FF00FFu#define RB_8OFF 0xFF00FF00u#define RGB 0x00FFFFFFu#define G 0x0000FF00u#define G_8OFF 0x00FF0000u#define A 0xFF000000u
inline void blendPreToStatic(const uint32_t & in, uint32_t & out){ uint32_t alpha = in >> 24; if (alpha & 0x00000080u) ++alpha; out = A | RGB & (in + ( ( (alpha * (out & RB) & RB_8OFF) | (alpha * (out & G) & G_8OFF) ) >> 8 ) );}
€
Operation in 0 [00][ ][00][ ]x BB RR 0 [00][00][ ][00]x GG out
Load 0 7 00 80 80x F 0 40 50 60xFF
Mask 0 00 40 00 60x 0 00 00 50 00x
Multiply 0 1 0 2 0x F C F A 0 00 27 0 00x B
Mask 0 1 00 2 00x F F 0 00 27 00 00x
OR 0 1 27 2 00x F F
SHIFT 0 00 1 27 2x F F
ADD 0 7 1 7 x F F A AF
Mask 0 00 1 7 x F A AF
OR 0 1 7 xFF F A AF
Code Detail
€
[R×A][G×A][B×A][1−A]Cout= ′ C2 +C1 ′ α 2
OUT = A | RGB & (IN + ( ( (ALPHA * (OUT & RB) & RB_8OFF) | (ALPHA * (OUT & G) & G_8OFF) ) >> 8 ) );
Vectorizing
For cases where there is no overlap between the four output pixels for four input pixels, we can use vectorized (SSE2) code
128-bit wide registers; load four 32-bit RGBA values, use the same approach as previously (R|B and G) in two registers to perform four paints at once
Vectorizing
inlinevoidblend4PreToStatic(uint32_t ** in, uint32_t * out) // Paints in (quad-word) onto out{ __m128i rb, g, a, a_, o, mask_reg; // Registers rb = _mm_loadu_si128((__m128i *) out); // Load destination (unaligned -- may not be on a 128-bit boundary) a_ = _mm_load_si128((__m128i *) *in); // We make sure the input is on a 128-bit boundary before this call *in += 4; _mm_prefetch((char*) (*in + 28),_MM_HINT_T0); // Fetch the two-cache-line-out memory mask_reg = _mm_set1_epi32(0x0000FF00); // Set green mask (x4) g = _mm_and_si128(rb,mask_reg); // Mask to greens (x4) mask_reg = _mm_set1_epi32(0x00FF00FF); // Set red and blue mask (x4) rb = _mm_and_si128(rb,mask_reg); // Mask to red and blue rb =_mm_slli_epi32(rb,8); // << 8 ; g is already biased by 256 in 16-bit spacing a = _mm_srli_epi32(a_,24); // >> 24 ; These are the four alpha values, shifted to lower 8 bits of each word mask_reg = _mm_slli_epi32(a,16); // << 16 ; A copy of the four alpha values, shifted to bits [16-23] of each word a = _mm_or_si128(a,mask_reg); // We now have the alpha value at both bits [0-7] and [16-23] of each word // These steps add one to transparancy values >= 80 o = _mm_srli_epi16(a,7); // Now the high bit is the low bit a = _mm_add_epi16(a,o);
// We now have 8 16-bit alpha values, and 8 rb or 4 g values. The values are biased by 256, and we want // to muptiply by alpha and then divide by 65536; we achieve this by multiplying the 16-bit values and // storing the upper 16 of the 32 bit result. (This is the operation that is available, so that's why we're // doing it in this fashion!) rb = _mm_mulhi_epu16(rb,a); g = _mm_mulhi_epu16(g,a); g =_mm_slli_epi32(g,8); // Move green into the correct location. // R and B, both the lower 8 bits of their 16 bits, don't need to be shifted o = _mm_set1_epi32(0xFF000000); // Opaque alpha value o = _mm_or_si128(o,g); o = _mm_or_si128(o,rb); // o now has the the background's contribution to the output color
mask_reg = _mm_set1_epi32(0x00FFFFFF); g = _mm_and_si128(mask_reg,a_); // Removes alpha from foreground color o = _mm_add_epi32(o,g); // Add foreground and background contributions together _mm_storeu_si128((__m128i *) out,o); // Unaligned store}
Vectorizing
Vectorizing this code achieves 3-4x speedup on cluster 8x 2x(3.4|3.2GHz) Xeon, 800MHz FSB Render 512x512x409 (400MB) volume in▪ ~22ms (45fps) (SIMD code) ▪ ~92ms (11fps) (Non-vectorized)
~18GB/s memory throughput ~11 cycles / voxel vs. ~45 cycles non-
vectorized
Results
MAN PAGES
MPI_Init() MPI_Init(3) MPI MPI_Init(3)
NAME MPI_Init - Initialize the MPI execution environment
SYNOPSIS int MPI_Init( int *argc, char ***argv )
INPUT PARAMETERS argc - Pointer to the number of arguments argv - Pointer to the argument vector
THREAD AND SIGNAL SAFETY This routine must be called by one thread only. That thread is called the main thread and must be the thread that calls MPI_Finalize .
NOTES The MPI standard does not say what a program can do before an MPI_INIT or after an MPI_FINALIZE . In the MPICH implementation, you should do as little as possible. In particular, avoid anything that changes the external state of the program, such as opening files, reading standard input or writing to standard output.
MPI_Barrier()
MPI_Barrier(3) MPI MPI_Barrier(3)
NAME MPI_Barrier - Blocks until all processes in the communicator have reached this routine.
SYNOPSIS int MPI_Barrier( MPI_Comm comm )
INPUT PARAMETER comm - communicator (handle)
NOTES Blocks the caller until all processes in the communicator have called it; that is, the call returns at any process only after all members of the communicator have entered the call.
MPI_Finalize()
MPI_Finalize(3) MPI MPI_Finalize(3)
NAME MPI_Finalize - Terminates MPI execution environment
SYNOPSIS int MPI_Finalize( void )
NOTES All processes must call this routine before exiting. The number of processes running after this routine is called is undefined; it is best not to perform much more than a return rc after calling MPI_Finalize .
MPI_Comm_size()
MPI_Comm_size(3) MPI MPI_Comm_size(3)
NAME MPI_Comm_size - Determines the size of the group associated with a communicator
SYNOPSIS int MPI_Comm_size( MPI_Comm comm, int *size )
INPUT PARAMETER comm - communicator (handle)
OUTPUT PARAMETER size - number of processes in the group of comm (integer)
MPI_Comm_rank()
MPI_Comm_rank(3) MPI MPI_Comm_rank(3)
NAME MPI_Comm_rank - Determines the rank of the calling process in the com- municator
SYNOPSIS int MPI_Comm_rank( MPI_Comm comm, int *rank )
INPUT ARGUMENT comm - communicator (handle)
OUTPUT ARGUMENT rank - rank of the calling process in the group of comm (integer)
MPI_Send()
MPI_Send(3) MPI MPI_Send(3)
NAME MPI_Send - Performs a blocking send
SYNOPSIS int MPI_Send(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)
INPUT PARAMETERS buf - initial address of send buffer (choice) count - number of elements in send buffer (nonnegative integer) datatype - datatype of each send buffer element (handle) dest - rank of destination (integer) tag - message tag (integer) comm - communicator (handle)
NOTES This routine may block until the message is received by the destination process.
MPI_Recv() MPI_Recv(3) MPI MPI_Recv(3)
NAME MPI_Recv - Blocking receive for a message
SYNOPSIS int MPI_Recv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Status *status)
OUTPUT PARAMETERS buf - initial address of receive buffer (choice) status - status object (Status)
INPUT PARAMETERS count - maximum number of elements in receive buffer (integer) datatype - datatype of each receive buffer element (handle) source - rank of source (integer) tag - message tag (integer) comm - communicator (handle)
NOTES The count argument indicates the maximum length of a message; the actual length of the message can be determined with MPI_Get_count .
MPI_Isend()
MPI_Isend(3) MPI MPI_Isend(3)
NAME MPI_Isend - Begins a nonblocking send
SYNOPSIS int MPI_Isend(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request *request)
INPUT PARAMETERS buf - initial address of send buffer (choice) count - number of elements in send buffer (integer) datatype - datatype of each send buffer element (handle) dest - rank of destination (integer) tag - message tag (integer) comm - communicator (handle)
OUTPUT PARAMETER request - communication request (handle)
MPI_Irecv()
MPI_Irecv(3) MPI MPI_Irecv(3)
NAME MPI_Irecv - Begins a nonblocking receive
SYNOPSIS int MPI_Irecv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Request *request)
INPUT PARAMETERS buf - initial address of receive buffer (choice) count - number of elements in receive buffer (integer) datatype - datatype of each receive buffer element (handle) source - rank of source (integer) tag - message tag (integer) comm - communicator (handle)
OUTPUT PARAMETER request - communication request (handle)
MPI_Bcast()
MPI_Bcast(3) MPI MPI_Bcast(3)
NAME MPI_Bcast - Broadcasts a message from the process with rank "root" to all other processes of the communicator
SYNOPSIS int MPI_Bcast( void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm )
INPUT/OUTPUT PARAMETER buffer - starting address of buffer (choice)
INPUT PARAMETERS count - number of entries in buffer (integer) datatype - data type of buffer (handle) root - rank of broadcast root (integer) comm - communicator (handle)
MPI_Allreduce()
MPI_Allreduce(3) MPI MPI_Allreduce(3)
NAME MPI_Allreduce - Combines values from all processes and distributes the result back to all processes
SYNOPSIS int MPI_Allreduce ( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm )
INPUT PARAMETERS sendbuf - starting address of send buffer (choice) count - number of elements in send buffer (integer) datatype - data type of elements of send buffer (handle) op - operation (handle) comm - communicator (handle)
OUTPUT PARAMETER recvbuf - starting address of receive buffer (choice)
MPI_Type_create_hvector()
MPI_Type_create_hvector(3) MPI MPI_Type_create_hvector(3)
NAME MPI_Type_create_hvector - Create a datatype with a constant stride given in bytes
SYNOPSIS int MPI_Type_create_hvector(int count, int blocklength, MPI_Aint stride, MPI_Datatype oldtype, MPI_Datatype *newtype)
INPUT PARAMETERS count - number of blocks (nonnegative integer) blocklength - number of elements in each block (nonnegative integer) stride - number of bytes between start of each block (address integer) oldtype - old datatype (handle)
OUTPUT PARAMETER newtype - new datatype (handle)
mpicc mpicc(1) MPI mpicc(1)
NAME mpicc - Compiles and links MPI programs written in C
DESCRIPTION This command can be used to compile and link MPI programs written in C. It provides the options and any special libraries that are needed to compile and link MPI programs.
It is important to use this command, particularly when linking pro- grams, as it provides the necessary libraries.
COMMAND LINE ARGUMENTS -show - Show the commands that would be used without runnning them -help - Give short help -cc=name - Use compiler name instead of the default choice. Use this only if the compiler is compatible with the MPICH library (see below) -config=name - Load a configuration file for a particular compiler. This allows a single mpicc command to be used with multiple compil- ers.
[…]
mpiexec mpiexec(1) MPI mpiexec(1)
NAME mpiexec - Run an MPI program
SYNOPSIS mpiexec args executable pgmargs [ : args executable pgmargs ... ]
where args are command line arguments for mpiexec (see below), exe- cutable is the name of an executable MPI program, and pgmargs are com- mand line arguments for the executable. Multiple executables can be specified by using the colon notation (for MPMD - Multiple Program Mul- tiple Data applications). For example, the following command will run the MPI program a.out on 4 processes: mpiexec -n 4 a.out
The MPI standard specifies the following arguments and their meanings:
-n <np> - Specify the number of processes to use -host <hostname> - Name of host on which to run processes -arch <architecture name> - Pick hosts with this architecture type
[…]
top related