the message passing interface (mpi) in layman's terms

KYOSS presentation, January 2011 1

Open MPI

KYOSS presentation12 Jan, 2011

Jeff Squyres

What is the Message Passing Interface (MPI)?

The Bookof MPI

A standards document

www.mpi-forum.org

Using MPI

Hardware and software implementthe interface in the MPI standard (book)

MPI implementations

There are many implementationsof the MPI standard

Some areclosed source

Others areopen source

Open MPI

Open MPI is a free, open sourceimplementation of the MPI standard

www.open-mpi.org

So what is MPI for?

Message Passing Interface

Let’s break it down…

1. Message passing

Process A Process B

Message

1. Message passing

Process A Process B

Pass it

1. Message passing

Process A Process B

Message has been passed

1. Message passing

Process

Thread A Thread B

…as opposed to data that is shared

2. Interface

C programming function calls

Fortran too!

MPI_Send

(buf, co

unt, typ

e, dest,

tag, co

MPI_Recv(buf, count, type, src, tag, comm, status)

MPI_Init(argv, argc)

MPI_Finalize(void)MPI_Type_size(dtype, s

MPI_Wait(req, status)

MPI_Test(req, flag, status)MPI_Comm_dup(in, out)

Fortran? Really?

What most modern developers associate with “Fortran”

Yes, really

Some oftoday’s most

advancedsimulationcodes arewritten inFortran

Yes, really

Yes,that Intel

Optimizedfor Nehalem,Westmere,

and beyond!

Fortran is great for what it is

A simple language for mathematical

expressions and computations

Targeted at scientists and engineers

…not computer scientists or web

developers or database developers or …

Back to defining “MPI”…

Putting it back together

Message Passing Interface

“An interface for passing messages”

“C functions for passing messages”

Fortran too!

C/Fortran functions for message passing

Process A Process B

MPI_Send(…)

C/Fortran functions for message passing

Process A Process B

MPI_Recv(…)

Really? Is that all MPI is?

“Can’t I just do that with sockets?”

Yes!(…and no)

Comparison

(TCP) Sockets

• Connections based on IP addresses and ports

• Point-to-point communication

• Stream-oriented

• Raw data (bytes / octets)

• Network-independent

• “Slow”

• Based on peer integer “rank” (e.g., 8)

• Point-to-point and collective and one-sided and …

• Message oriented

• Typed messages

• Network independent

• Blazing fast

Comparison

• Typed messages

• Blazing fast

Whoa!What are these?

Peer integer “rank”

• Typed messages

• Blazing fast

9 10 11

Peer integer “rank”

• Typed messages

• Blazing fast

9 10 11

“Collective”: broadcast

• Typed messages

• Blazing fast

9 10 11

“Collective”: broadcast

• Typed messages

• Blazing fast

9 10 11

“Collective”: scatter

• Typed messages

• Blazing fast

9 10 11

“Collective”: scatter

• Typed messages

• Blazing fast

9 10 11

“Collective”: gather

• Typed messages

• Blazing fast

9 10 11

“Collective”: gather

• Typed messages

• Blazing fast

9 10 11

“Collective”: reduce

• Typed messages

• Blazing fast

9 10 11

“Collective”: reduce

• Typed messages

• Blazing fast

9 10 11

“Collective”: …and others

• Typed messages

• Blazing fast

9 10 11

Messages, not bytes

• Typed messages

• Blazing fast

Entire messageis sent andreceived

Not astream of

individual bytes

Messages, not bytes

• Typed messages

• Blazing fast

Contents:

Not abunch of

bytes!

17 integers23 doubles98 structs

…or whatever

Network independent

• Typed messages

• Blazing fast

Underlying network

MPI_Send(…)

MPI_Recv(…)

Ethernet Myrinet

InfiniBand Shared memory

TCP iWARP RoCE

Network independent

• Typed messages

• Blazing fast

Underlying network

MPI_Send(…)

MPI_Recv(…)

Ethernet Myrinet

InfiniBand Shared memory

TCP iWARP RoCE

Regardless of underlyingnetwork or transport

protocol, the applicationcode stays the same

Blazing fast

• Typed messages

• Blazing fast

Onemicrosecond

…more on performance later

What is MPI?

MPI isprobably

somewherearound here

What is MPI?

MPI ishides all

the layersunderneath

What is MPI?

A high-level network programming abstraction

IPaddresses

bytestreams

rawbytes

What is MPI?

A high-level network programming abstraction

IPaddresses

bytestreams

rawbytesNothing to see here

Please move along

So what?

What’s all this message passing stuffgot to do with supercomputers?

So what?

Let’s define “supercomputers”

Supercomputers

“Nebulae”

NationalSupercomputing

Centre,Shenzen,

Supercomputers

“Mare Nostrum”(Our Sea)

BarcelonaSupercomputer

Center,Spain

Used to be a church

Supercomputers

Notice anything?

Supercomputers

They’re justracks ofservers!

Generally speaking…

Supercomputer=

Lots ofprocessors +

Lots ofRAM +

Lots ofdisk

Generally speaking…

Supercomputer=

(Many) Racks of (commodity)high-end servers

(this is one definition; there are others)

So if that’s a supercomputer…

Rack of36 1U

servers

How is it different from my web farm?

Rack of36 1U

servers

Just a bunch of servers?

The difference betweensupercomputers and web farms

and database farms (and …)

All the servers act together tosolve a single computational problem

Acting together

Computational problem

Input Output

Take your computational problem…

Acting together

Input Output

…and split it up!

Acting together

Input Output

Distribute the input dataacross a bunch of servers

Acting together

Input Output

Use the network between serversto communicate / coordinate

Acting together

Input Output

Use the network between serversto communicate / coordinate

Acting together

Input Output

MPI is used for this communication

Why go to so much trouble?

One processor

1 processor = …a long time…

Why go to so much trouble?

One processor

21 processors = ~1 hour (!)Disclaimer: scaling is rarely perfect

One processor

High Performance Computing

Using supercomputers to solvereal world problems that are

TOO BIGfor laptops, desktops, or individuals servers

Why does HPC MPI?

Network abstractionAre thesecores?

Why does HPC MPI?

Network abstraction…orservers?

Why does HPC MPI?

Message semantics

Array of10,000

integers

Why does HPC MPI?

Message semantics

Array of10,000

integers

Why does HPC MPI?

Ultra-low network latency

1 microsecond

(depending on your network type!)

1 microsecond = 0.000001 second

Fromhere

Tohere

1 microsecond = 0.000001 second

Fromhere

Tohere

Holy smokes!

That’s fast

Let’s get into some details…

MPI Basics

• “6 function MPI” MPI_Init(): startup MPI_Comm_size(): how many peers? MPI_Comm_rank(): my unique (ordered) ID MPI_Send(): send a message MPI_Recv(): receive a message MPI_Finalize(): shutdown

• Can implement a huge number of parallel applications with just these 6 functions

Let’s see “Hello, World” in MPI

MPI Hello, World

#include <stdio.h>#include <mpi.h>

int main(int argc, char **argv) { int rank, size; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); printf("Hello, world! I am %d of %d\n", rank, size); MPI_Finalize(); return 0;}

Initialize MPI

Who am I?

Num. peers?

Shut down MPI

Compile it with Open MPI

shell$ mpicc hello.c -o helloshell$

Hey – what’s that? Where’s gcc?

Open MPI comes standard in many Linux and BSD distributions

(and OS X)

“Wrapper” compiler

shell$ mpicc hello.c -o hello –showmegcc hello.c -o hello -I/opt/openmpi/include -pthread -L/open/openmpi/lib -lmpi -ldl -Wl,--export-dynamic -lnsl -lutil -lm -ldlshell$

mpicc simply fills in a bunch of compiler command line options for you

Now let’s run it

shell$ mpirun –np 4 hello

Hey – what’s that? Why don’t I just

run “./hello”?

mpirun launcher

mpirun launches N copies of yourprogram and “wires them up”

“-np” = “number of processes”

This command launches a 4 process parallel job

mpirun launcher

hello hello

hellohello

Four copiesof “hello”

are launched

Then theyare “wired up”on the network

Now let’s run it

shell$ mpirun –np 4 helloHello, world! I am 0 of 4Hello, world! I am 1 of 4Hello, world! I am 2 of 4Hello, world! I am 3 of 4shell$

By default, all copiesrun on the local host

Run on multiple servers!

shell$ cat my_hostfilehost1.example.comhost2.example.comhost3.example.comhost4.example.comshell$

Run on multiple servers!

shell$ cat my_hostfilehost1.example.comhost2.example.comhost3.example.comhost4.example.comshell$ mpirun –hostfile my_hostfile –np 4 helloHello, world! I am 0 of 4Hello, world! I am 1 of 4Hello, world! I am 2 of 4Hello, world! I am 3 of 4shell$

Ran on host1 Ran on host2 Ran on host3 Ran on host4

Run it again

shell$ mpirun –hostfile my_hostfile –np 4 helloHello, world! I am 2 of 4Hello, world! I am 3 of 4Hello, world! I am 0 of 4Hello, world! I am 1 of 4shell$

Hey – why are the numbers out of

order?

Standard output re-routing

shell$ mpirun –hostfile my_hostfile –np 4 hello

hello0

hello1

hello3

hello2

mpirun Each “hello” program’sstandard outputis intercepted

and sent across thenetwork to mpirun

Hello, world! I am 0 of 4 Hello, world! I am 1 of 4

Hello, world! I am 2 of 4 Hello, world! I am 3 of 4

Standard output re-routing

shell$ mpirun –hostfile my_hostfile –np 4 hello

hello0

hello1

hello3

hello2

mpirun But the exactordering of

received printf’sis non-deterministic

Hello, world! I am 0 of 4

Printf debugging = Bad

If you can’t rely on output ordering,printf debugging is pretty lousy (!)

Parallel debuggers

Fortunately, there are paralleldebuggers and other tools

Paralleldebugger

Attaches to allprocesses inthe MPI job

hello0

hello1

hello3

hello2

mpirun

Now let’s send a simple MPI message

Send a simple message

int rank;double buffer[SIZE];

MPI_Comm_rank(MPI_COMM_WORLD, &rank);if (0 == rank) { /* …initialize buffer[]… */ MPI_Send(buffer, SIZE, MPI_DOUBLE, 1, 123, MPI_COMM_WORLD);} else if (1 == rank) { MPI_Recv(buffer, SIZE, MPI_DOUBLE, 0, 123, MPI_COMM_WORLD, MPI_STATUS_IGNORE);}

If I’m number 0, send the buffer[] array to number 1

If I’m number 1, receive thebuffer[] array from number 0

That’s enough MPI for now…

Open MPI

PACX-MPI

LAM/MPI

LA-MPI

FT-MPI

Sun CT 6

Project founded in 2003 after

intensediscussions

between multiple open source MPI implementations

Open_MPI_Init()

shell$ svn log –r 1 https://svn.open-mpi.org/svn/ompi------------------------------------------------------------------------r1 | jsquyres | 2003-11-22 11:36:58 -0500 (Sat, 22 Nov 2003) | 2 lines

First commit------------------------------------------------------------------------shell$

Open_MPI_Current_status()

shell$ svn log –r HEAD https://svn.open-mpi.org/svn/ompi------------------------------------------------------------------------r24226 | rhc | 2011-01-11 20:57:47 -0500 (Tue, 11 Jan 2011) | 25 lines

Fixes #2683: Move ORTE DPM compiler warning squash to v1.4------------------------------------------------------------------------shell$

Open MPI 2011 Membership

15 members, 11 contributors, 2 partners

Fun stats

• ohloh.net says: 517,400 lines of code 30 developers (over

time) “Well-commented

source code”

• I rank in top-25 ohloh stats for: C Automake Shell script Fortran (ouch!)

Open MPI has grown

It’s amazing (to me) that the Open MPIproject works so well

New features, new releases, new members

Long live Open MPI!

• Defined Message Passing Interface (MPI)

• Defined “supercomputers”

• Defined High Performance Computing (HPC)

• Showed what MPI is

• Showed some trivial MPI codes

• Discussed Open MPI

Additional Resources

• MPI Forum web site The only site for the official MPI standards http://www.mpi-forum.org/

• NCSA MPI basic and intermediate tutorials Requires a free account http://ci-tutor.ncsa.uiuc.edu/login.php

• “MPI Mechanic” magazine columns http://cw.squyres.com/

Additional Resources

• Research, Computing, and Engineering (RCE) podcast http://www.rce-cast.com/

• My blog: MPI_BCAST http://blogs.cisco.com/category/performance/

Questions?

the message passing interface (mpi) in layman's terms

peer integer rankmpibased

message passing interface

mpi standardwww

mpi implementationsthere

open mpiopen mpi

mpi standard book4

cfortran functions

open sourceimplementation

Technology

the message passing interface (mpi). outline introduction to...

message passing and mpi

message passing programming with mpi what is...

mpi the message passing interface

chapter 4 message-passing programming. 2 outline...

message passing programming (mpi)

advanced message passing in mpi - using mpi datatypes with

multiprocesorski sistemi mpi (message passing interface)

introduction to the message passing interface (mpi) ·...

5 mpi : the message passing interface

message passing interface (mpi)...

introduction to message passing interface (mpi)

Стандарт mpi (message passing interface)

mpi: message-passing interface chapter 2

tm message passing mpi on origin systems. tm mpi programming...

mpi (message passing interface)

introduction to mpi (message-passing...

mpi: a message-passing interface standard · this document...

message passing interface (mpi-1) (psc appendix c, §c.1--c...

message passing interface (mpi) 3