the message passing interface (mpi) in layman's terms

KYOSS presentation, January 2011 1

Open MPI

KYOSS presentation12 Jan, 2011

Jeff Squyres


What is the Message Passing Interface (MPI)?

The Bookof MPI

A standards document

www.mpi-forum.org

http://www.mpi-forum.org/


Using MPI

Hardware and software implementthe interface in the MPI standard (book)


MPI implementations

There are many implementationsof the MPI standard

Some areclosed source

Others areopen source


Open MPI

Open MPI is a free, open sourceimplementation of the MPI standard

www.open-mpi.org

http://www.open-mpi.org/


So what is MPI for?

Message Passing Interface

Let’s break it down…


1. Message passing

Process A Process B

Message


1. Message passing

Process A Process B

Pass it


1. Message passing

Process A Process B

Message has been passed


1. Message passing

Process

Thread A Thread B

…as opposed to data that is shared


2. Interface

C programming function calls

Fortran too!

MPI_Send

(buf, co

unt, typ

e, dest,

tag, co

mm)

MPI_Recv(buf, count, type, src, tag, comm, status)

MPI_Init(argv, argc)

MPI_Finalize(void)MPI_Type_size(dtype, s

ize)

MPI_Wait(req, status)

MPI_Test(req, flag, status)MPI_Comm_dup(in, out)


Fortran? Really?

What most modern developers associate with “Fortran”


Yes, really

Some oftoday’s most

advancedsimulationcodes arewritten inFortran


Yes, really

Yes,that Intel

Optimizedfor Nehalem,Westmere,

and beyond!


Fortran is great for what it is

A simple language for mathematical

expressions and computations

Targeted at scientists and engineers

…not computer scientists or web

developers or database developers or …


Back to defining “MPI”…


Putting it back together

Message Passing Interface

“An interface for passing messages”

“C functions for passing messages”

Fortran too!


C/Fortran functions for message passing

Process A Process B

MPI_Send(…)


C/Fortran functions for message passing

Process A Process B

MPI_Recv(…)


Really? Is that all MPI is?

“Can’t I just do that with sockets?”

Yes!(…and no)


Comparison

(TCP) Sockets

• Connections based on IP addresses and ports

• Point-to-point communication

• Stream-oriented

• Raw data (bytes / octets)

• Network-independent

• “Slow”

MPI

• Based on peer integer “rank” (e.g., 8)

• Point-to-point and collective and one-sided and …

• Message oriented

• Typed messages

• Network independent

• Blazing fast


Comparison

MPI




• Typed messages


• Blazing fast

Whoa!What are these?


Peer integer “rank”

MPI




• Typed messages


• Blazing fast

0 1 2

3 4 5

6 7 8

9 10 11


“Collective”: broadcast

MPI




• Typed messages


• Blazing fast

0 1 2

3 4 5

6 7 8

9 10 11


“Collective”: scatter

MPI




• Typed messages


• Blazing fast

0 1 2

9 10 11

3 4 5

6 7 8


“Collective”: gather

MPI




• Typed messages


• Blazing fast

0 1 2

9 10 11

3 4 5

6 7 8


“Collective”: reduce

MPI




• Typed messages


• Blazing fast

0 1 2

9 10 11

3 4 5

6 7 8

4

5

3

2

6

6

2

4

5

3

4

4


“Collective”: reduce

MPI




• Typed messages


• Blazing fast

0 1 2

9 10 11

3 4 5

6 7 8

42


“Collective”: …and others

MPI




• Typed messages


• Blazing fast

0 1 2

9 10 11

3 4 5

6 7 8


Messages, not bytes

MPI




• Typed messages


• Blazing fast

Entire messageis sent andreceived

Not astream of

individual bytes


Messages, not bytes

MPI




• Typed messages


• Blazing fast

Contents:

Not abunch of

bytes!

17 integers23 doubles98 structs

…or whatever


Network independent

MPI




• Typed messages


• Blazing fast

Underlying network

MPI_Send(…)

MPI_Recv(…)

Ethernet Myrinet

InfiniBand Shared memory

TCP iWARP RoCE


Network independent

MPI




• Typed messages


• Blazing fast

Underlying network

MPI_Send(…)

MPI_Recv(…)

Ethernet Myrinet

InfiniBand Shared memory

TCP iWARP RoCE

Regardless of underlyingnetwork or transport

protocol, the applicationcode stays the same


Blazing fast

MPI




• Typed messages


• Blazing fast

Onemicrosecond

(!)

…more on performance later


What is MPI?

MPI isprobably

somewherearound here


What is MPI?

MPI ishides all

the layersunderneath


What is MPI?

A high-level network programming abstraction

IPaddresses

bytestreams

rawbytes


What is MPI?

A high-level network programming abstraction

IPaddresses

bytestreams

rawbytesNothing to see here

Please move along


So what?

What’s all this message passing stuffgot to do with supercomputers?


So what?

Let’s define “supercomputers”


Supercomputers


Supercomputers

“Nebulae”

NationalSupercomputing

Centre,Shenzen,

China


Supercomputers

“Mare Nostrum”(Our Sea)

BarcelonaSupercomputer

Center,Spain

Used to be a church


Supercomputers

Notice anything?


Supercomputers

They’re justracks ofservers!


Generally speaking…

Supercomputer=

Lots ofprocessors +

Lots ofRAM +

Lots ofdisk


Generally speaking…

Supercomputer=

(Many) Racks of (commodity)high-end servers

(this is one definition; there are others)


So if that’s a supercomputer…

Rack of36 1U

servers


How is it different from my web farm?

Rack of36 1U

servers


Just a bunch of servers?

The difference betweensupercomputers and web farms

and database farms (and …)

All the servers act together tosolve a single computational problem


Acting together

Computational problem

Input Output

Take your computational problem…


Acting together


Input Output

…and split it up!


Acting together


Input Output

Distribute the input dataacross a bunch of servers


Acting together

Input Output

Use the network between serversto communicate / coordinate


Acting together

Input Output

MPI is used for this communication


Why go to so much trouble?


One processor

hour

1 processor = …a long time…


Why go to so much trouble?


One processor

hour

One processor

hour

One processor

hour

21 processors = ~1 hour (!)Disclaimer: scaling is rarely perfect

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour

One processor

hour


High Performance Computing

Using supercomputers to solvereal world problems that are

TOO BIGfor laptops, desktops, or individuals servers

HPC=


Why does HPC MPI?

Network abstractionAre thesecores?


Why does HPC MPI?

Network abstraction…orservers?


Why does HPC MPI?

Message semantics

Array of10,000

integers


Why does HPC MPI?

Ultra-low network latency

1 microsecond

(depending on your network type!)


1 microsecond = 0.000001 second

Fromhere

Tohere


Holy smokes!

That’s fast


Let’s get into some details…


MPI Basics

• “6 function MPI” MPI_Init(): startup MPI_Comm_size(): how many peers? MPI_Comm_rank(): my unique (ordered) ID MPI_Send(): send a message MPI_Recv(): receive a message MPI_Finalize(): shutdown

• Can implement a huge number of parallel applications with just these 6 functions


Let’s see “Hello, World” in MPI


MPI Hello, World

#include <stdio.h>#include <mpi.h>

int main(int argc, char **argv) { int rank, size; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); printf("Hello, world! I am %d of %d\n", rank, size); MPI_Finalize(); return 0;}

Initialize MPI

Who am I?

Num. peers?

Shut down MPI


Compile it with Open MPI

shell$ mpicc hello.c -o helloshell$

Hey – what’s that? Where’s gcc?

Open MPI comes standard in many Linux and BSD distributions

(and OS X)


“Wrapper” compiler

shell$ mpicc hello.c -o hello –showmegcc hello.c -o hello -I/opt/openmpi/include -pthread -L/open/openmpi/lib -lmpi -ldl -Wl,--export-dynamic -lnsl -lutil -lm -ldlshell$

mpicc simply fills in a bunch of compiler command line options for you


Now let’s run it

shell$ mpirun –np 4 hello

Hey – what’s that? Why don’t I just

run “./hello”?


mpirun launcher


mpirun launches N copies of yourprogram and “wires them up”

“-np” = “number of processes”

This command launches a 4 process parallel job


mpirun launcher


hello hello

hellohello

Four copiesof “hello”

are launched

Then theyare “wired up”on the network


Now let’s run it

shell$ mpirun –np 4 helloHello, world! I am 0 of 4Hello, world! I am 1 of 4Hello, world! I am 2 of 4Hello, world! I am 3 of 4shell$

By default, all copiesrun on the local host


Run on multiple servers!

shell$ cat my_hostfilehost1.example.comhost2.example.comhost3.example.comhost4.example.comshell$


Run on multiple servers!

shell$ cat my_hostfilehost1.example.comhost2.example.comhost3.example.comhost4.example.comshell$ mpirun –hostfile my_hostfile –np 4 helloHello, world! I am 0 of 4Hello, world! I am 1 of 4Hello, world! I am 2 of 4Hello, world! I am 3 of 4shell$

Ran on host1 Ran on host2 Ran on host3 Ran on host4


Run it again

shell$ mpirun –hostfile my_hostfile –np 4 helloHello, world! I am 2 of 4Hello, world! I am 3 of 4Hello, world! I am 0 of 4Hello, world! I am 1 of 4shell$

Hey – why are the numbers out of

order?

2301


Standard output re-routing

shell$ mpirun –hostfile my_hostfile –np 4 hello

hello0

hello1

hello3

hello2

mpirun Each “hello” program’sstandard outputis intercepted

and sent across thenetwork to mpirun

Hello, world! I am 0 of 4 Hello, world! I am 1 of 4

Hello, world! I am 2 of 4 Hello, world! I am 3 of 4


Standard output re-routing

shell$ mpirun –hostfile my_hostfile –np 4 hello

hello0

hello1

hello3

hello2

mpirun But the exactordering of

received printf’sis non-deterministic

Hello, world! I am 0 of 4





Printf debugging = Bad

If you can’t rely on output ordering,printf debugging is pretty lousy (!)


Parallel debuggers

Fortunately, there are paralleldebuggers and other tools

Paralleldebugger

Attaches to allprocesses inthe MPI job

hello0

hello1

hello3

hello2

mpirun


Now let’s send a simple MPI message


Send a simple message

int rank;double buffer[SIZE];

MPI_Comm_rank(MPI_COMM_WORLD, &rank);if (0 == rank) { /* …initialize buffer[]… */ MPI_Send(buffer, SIZE, MPI_DOUBLE, 1, 123, MPI_COMM_WORLD);} else if (1 == rank) { MPI_Recv(buffer, SIZE, MPI_DOUBLE, 0, 123, MPI_COMM_WORLD, MPI_STATUS_IGNORE);}

If I’m number 0, send the buffer[] array to number 1

If I’m number 1, receive thebuffer[] array from number 0


That’s enough MPI for now…


Open MPI

PACX-MPI

LAM/MPI

LA-MPI

FT-MPI

Sun CT 6

Project founded in 2003 after

intensediscussions

between multiple open source MPI implementations


Open_MPI_Init()

shell$ svn log –r 1 https://svn.open-mpi.org/svn/ompi------------------------------------------------------------------------r1 | jsquyres | 2003-11-22 11:36:58 -0500 (Sat, 22 Nov 2003) | 2 lines

First commit------------------------------------------------------------------------shell$


Open_MPI_Current_status()

shell$ svn log –r HEAD https://svn.open-mpi.org/svn/ompi------------------------------------------------------------------------r24226 | rhc | 2011-01-11 20:57:47 -0500 (Tue, 11 Jan 2011) | 25 lines

Fixes #2683: Move ORTE DPM compiler warning squash to v1.4------------------------------------------------------------------------shell$


Open MPI 2011 Membership

15 members, 11 contributors, 2 partners


Fun stats

• ohloh.net says: 517,400 lines of code 30 developers (over

time) “Well-commented

source code”

• I rank in top-25 ohloh stats for: C Automake Shell script Fortran (ouch!)


Open MPI has grown

It’s amazing (to me) that the Open MPIproject works so well

New features, new releases, new members

Long live Open MPI!


Recap

• Defined Message Passing Interface (MPI)

• Defined “supercomputers”

• Defined High Performance Computing (HPC)

• Showed what MPI is

• Showed some trivial MPI codes

• Discussed Open MPI


Additional Resources

• MPI Forum web site The only site for the official MPI standards http://www.mpi-forum.org/

• NCSA MPI basic and intermediate tutorials Requires a free account http://ci-tutor.ncsa.uiuc.edu/login.php

• “MPI Mechanic” magazine columns http://cw.squyres.com/

http://www.mpi-forum.org/

http://ci-tutor.ncsa.uiuc.edu/login.php

http://cw.squyres.com/


Additional Resources

• Research, Computing, and Engineering (RCE) podcast http://www.rce-cast.com/

• My blog: MPI_BCAST http://blogs.cisco.com/category/performance/

http://www.rce-cast.com/

http://blogs.cisco.com/category/performance/

http://blogs.cisco.com/category/performance/


Questions?

the message passing interface (mpi) in layman's terms

Technology

peer integer rankmpibased

message passing interface

mpi standardwww

mpi implementationsthere

open mpiopen mpi

mpi standard book4

cfortran functions

open sourceimplementation