![Page 1: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/1.jpg)
MPI Cluster Programming with
Python and Amazon EC2
Pete Skomoroch
datawrangling.comjuiceanalytics.com/writing
![Page 2: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/2.jpg)
Outline
Netflix Prize & Amazon EC2
Python Parallel Programming Options
ElasticWulf
MPI basics in Python
Demo: ElasticWulf, mpi4py, Ipython1
ElasticWulf Performance
EC2 pointers + Q&A
![Page 3: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/3.jpg)
The Netflix Prize
17K movies500K users100M ratings2 GB raw data
A “big” computation?
10% improvement of RMSE wins $1 Million
![Page 4: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/4.jpg)
Crunching the data at home...
Use the Pyflix library to squeeze dataset into 600 MB
http://pyflix.python-hosting.com
Simon Funk iterative SVD algorithm runs on your laptop overnight
“Timely Development” code gets us to 4% improvement over Cinematch
Use Numpy/Scipy, call C when needed
![Page 5: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/5.jpg)
Basic algorithm ties up your machine for 13 hours
Some algorithms take weeks...
Need runs over many sets of parameters
Need to run cross validation over large datasets
Bugs in long jobs suck
The best approach so far ( Bell Labs ) merged the successful results of over 107 different algorithms
I needed more CPUs
![Page 6: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/6.jpg)
But this was my only machine...
![Page 7: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/7.jpg)
How can I get a Beowulf cluster?
Cluster of nearly identical commodity machines Employ message passing model of parallel computationOften have shared filesystem
![Page 8: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/8.jpg)
Amazon EC2
__init__( )
http://www.flickr.com/photos/steverosebush/2241578490/
Launch instances of Amazon Machine Images (AMIs) from your Python code
Pay only for what you use (wall clock time)
![Page 9: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/9.jpg)
EC2 Instance Types
Small 1x Large 4x ExtraLarge 8x
RAM 1.7GB 7.5GB 15GB
Disk 160GB 850GB 1.6TB
CPU (1.7Ghz) 1 32bit 4 64bit 8 64bit
I/O Performance Moderate High High
Price(per instance-hour)
$0.10 $0.40 $0.80
![Page 10: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/10.jpg)
Driving EC2 using Python
yum -y install python-boto
Blog post: “Amazon EC2 Basics for Python Programmers” - James Gardner
PyCon talk: “Like Switching on the Light: Managing an Elastic Compute Cluster with Python” George Belotsky; Heath Johns
http://del.icio.us/pskomoroch/ec2
![Page 11: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/11.jpg)
Parallel Programming in Python
MapReduce (Hadoop + Python)Python MPI options (mpi4py, pyMPI, ...)Wrap existing C++/Fortran libs (ScaLAPACK,PETSc, ...)PyroTwistedIPython1
Which one you use depends on your particular situation
These are not mutually exclusive or exhaustive...
![Page 12: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/12.jpg)
Introducing ElasticWulf
![Page 13: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/13.jpg)
ElasticWulf - batteries included
OpenMPIMPICHMPICH2
LAM
BLAS/LAPACKNumpy/Scipy
ipythonmatplotlib
XwindowsNFS
GangliaC3 tools
TwistedIpython1mpi4py
boto
Master and worker beowulf node Amazon Machine ImagesPython command line scripts to launch/configure clusterIncludes Ipython1 and other good stuff...
The AMIs are publicly visibleUpdated python scripts + docs will be on Google Code
![Page 14: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/14.jpg)
What is MPI?
High performance message passing interface (MPI)
Standard protocol implemented in multiple languages
Point to Point - Collective Operations
Very flexible, complex...
We will just look at a use case involving a master process broadcasting data and receiving results from slave processes via simple primitives (bcast(), scatter(), gather(), reduce())
![Page 15: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/15.jpg)
MPI Basics in Python
Lets look at some code...
size attribute: number of cooperating processes
rank attribute: unique identifier of a process
We will cover some PyMPI examples, but I would recommend using mpi4py (moved over to scipy, used by Ipython1)
http://mpi4py.scipy.org/
The program structure is nearly identical in either implementation.
![Page 16: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/16.jpg)
MPI Broadcast
mpi.bcast()
broadcasts a value from the root process to all other processes
![Page 17: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/17.jpg)
MPI Scatter
mpi.scatter()
scatters an array roughly evenly across processes
![Page 18: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/18.jpg)
MPI Gather
mpi.gather()
Collects results from all processes back to the master process
![Page 19: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/19.jpg)
MPI Reduce
mpi.reduce()
Apply a summary function across all nodes. Like Python reduce (at least until Guido kills it)
![Page 20: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/20.jpg)
A simple example
Calculating Pi.
Throw random darts at a dartboard.
![Page 21: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/21.jpg)
Monte Carlo Pi
Measure the distance from the origin, count the number of “darts” that are within 1 unit of the origin, and the total number of darts...
![Page 22: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/22.jpg)
Throwing more darts
42 inside the circle
8 outside the circle
estimated pi is 3.360 ( 42*4 / (42+8) )
![Page 23: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/23.jpg)
Serial Pi Calculation
![Page 24: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/24.jpg)
Throwing darts in parallel
![Page 25: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/25.jpg)
Parallel Pi Calculation
![Page 26: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/26.jpg)
Starting up your ElasticWulf Cluster
1) Sign up for Amazon Web Services2) Get your keys/certificates + define environment variables4) Download ElasticWulf Python scripts4) Add your Amazon Id to the ElasticWulf config file5) ./start-cluster.py -n 3 -s ‘xlarge’6) ./ec2-mpi-config.py7) ssh in to the master node...
![Page 27: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/27.jpg)
Demo: Start the cluster
$ ./ec2-start-cluster.py
![Page 28: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/28.jpg)
Demo: check progress
$ ./ec2-check-instances.py
![Page 29: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/29.jpg)
Demo: configure the nodes$ ./ec2-mpi-config.py
![Page 30: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/30.jpg)
Demo: login to the master node
![Page 31: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/31.jpg)
Demo: start the MPI daemons
![Page 32: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/32.jpg)
Demo: mpi4py code
![Page 33: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/33.jpg)
Demo: run mpi4py code
![Page 34: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/34.jpg)
Running Ganglia
![Page 35: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/35.jpg)
Demo: start ipython1 controller
![Page 36: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/36.jpg)
Demo: start engines with mpirun
![Page 37: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/37.jpg)
Demo: parallel ipython1
![Page 38: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/38.jpg)
Demo: parallel ipython1
![Page 39: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/39.jpg)
Scaling KNN on ElasticWulf
Run pearson correlation between every pair of movies...Use Numpy & mpi4py to scatter the movie ratings vectorsSpend $2.40 for 24 cpu hours, trade $ for time
![Page 40: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/40.jpg)
Performance
![Page 41: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/41.jpg)
Netpipe Benchmark
Latency: 0.000492 (microseconds)
![Page 42: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/42.jpg)
![Page 43: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/43.jpg)
EC2 “gotchas”
EC2 is still in Beta: instances die, freeze, be prepared
Relatively high latency
Dynamic subnet
ec2-upload-bundle --retry
/etc/fstab overwritten by default in bundle-vol
more: http://del.icio.us/pskomoroch/ec2+gotchas
![Page 44: MPI Cluster Programming with Python and Amazon EC2 · MPICH2 LAM BLAS/LAPACK Numpy/Scipy ipython matplotlib Xwindows NFS Ganglia C3 tools Twisted Ipython1 mpi4py boto Master and worker](https://reader031.vdocument.in/reader031/viewer/2022013008/5cdf2dc688c993a0058b7cb5/html5/thumbnails/44.jpg)
Questions?