hpc on aws
DESCRIPTION
Overview of HPC on Amazon Web ServicesTRANSCRIPT
![Page 1: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/1.jpg)
HPC on Amazon Web Services
Deepak SinghAmazon Web Services
Dec 17, 2010
![Page 2: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/2.jpg)
Image: Simon Cockell under CC-BY
![Page 3: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/3.jpg)
the new reality
![Page 4: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/4.jpg)
lots and lots and lots and lots and lots of data
![Page 5: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/5.jpg)
lots and lots and lots and lots and lots of
compute
![Page 6: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/6.jpg)
lots and lots and lots and lots and lots of
people
![Page 7: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/7.jpg)
lots and lots and lots and lots and lots of
places
![Page 8: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/8.jpg)
constant change
![Page 9: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/9.jpg)
goal
![Page 10: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/10.jpg)
innovate
![Page 11: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/11.jpg)
innovate in a new reality
![Page 12: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/12.jpg)
optimize the most valuable resource
![Page 13: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/13.jpg)
compute, storage, workflows, memory,
transmission, algorithms, cost, …
![Page 14: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/14.jpg)
people drive innovation
Credit: Pieter Musterd a CC-BY-NC-ND license
![Page 15: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/15.jpg)
make people productive
Credit: Pieter Musterd a CC-BY-NC-ND license
![Page 16: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/16.jpg)
challenges
![Page 17: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/17.jpg)
Your Idea SuccessfulProduct
![Page 18: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/18.jpg)
Your Idea SuccessfulProduct
![Page 19: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/19.jpg)
Your Idea SuccessfulProduct
Great Idea Not Prioritized
![Page 20: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/20.jpg)
Your Idea SuccessfulProduct
Great Idea Not Prioritized
![Page 21: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/21.jpg)
Resource Contention
Tight Budgets
Shared Resources
![Page 22: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/22.jpg)
enter the cloud
![Page 23: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/23.jpg)
infrastructure services
![Page 24: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/24.jpg)
building blocks
![Page 25: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/25.jpg)
![Page 26: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/26.jpg)
Undifferentiated Heavy Lifting
![Page 27: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/27.jpg)
pay as you go
![Page 28: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/28.jpg)
pay for what you use
![Page 29: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/29.jpg)
on demand
![Page 30: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/30.jpg)
programmable
![Page 31: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/31.jpg)
import botoimport boto.emrfrom boto.emr.step import StreamingStepfrom boto.emr.bootstrap_action import BootstrapActionimport time
# set your aws keys and S3 bucket, e.g. from environment or .botoAWSKEY= SECRETKEY= S3_BUCKET=NUM_INSTANCES = 1
conn = boto.connect_emr(AWSKEY,SECRETKEY)
bootstrap_step = BootstrapAction("download.tst", "s3://elasticmapreduce/bootstrap-actions/download.sh",None)
step = StreamingStep(name='Wordcount', mapper='s3n://elasticmapreduce/samples/wordcount/wordSplitter.py', cache_files = ["s3n://" + S3_BUCKET + "/boto.mod#boto.mod"], reducer='aggregate', input='s3n://elasticmapreduce/samples/wordcount/input', output='s3n://' + S3_BUCKET + '/output/wordcount_output')
jobid = conn.run_jobflow( name="testbootstrap", log_uri="s3://" + S3_BUCKET + "/logs", steps = [step], bootstrap_actions=[bootstrap_step], num_instances=NUM_INSTANCES)
print "finished spawning job (note: starting still takes time)"
state = conn.describe_jobflow(jobid).stateprint "job state = ", stateprint "job id = ", jobidwhile state != u'COMPLETED': print time.localtime() time.sleep(30) state = conn.describe_jobflow(jobid).state print "job state = ", state print "job id = ", jobid
print "final output can be found in s3://" + S3_BUCKET + "/output" + TIMESTAMPprint "try: $ s3cmd sync s3://" + S3_BUCKET + "/output" + TIMESTAMP + " ."
Connect to Elastic MapReduce
Install packages
Set up mappers &reduces
job state
![Page 32: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/32.jpg)
elastic
![Page 33: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/33.jpg)
Capacity
Time
Realdemand
Elasticcapacity
![Page 34: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/34.jpg)
![Page 35: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/35.jpg)
On demand Faster to market
Pay as you go Maintain focus
Pay to play Efficiency
Elastic resources Capacity planning
![Page 36: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/36.jpg)
Computing with Amazon EC2
![Page 37: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/37.jpg)
Credit: Angel Pizzaro, U. Penn
![Page 38: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/38.jpg)
Credit: Tom Fifield: U. Melbourne
![Page 39: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/39.jpg)
![Page 40: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/40.jpg)
standard “m1”high cpu “c1”
high memory “m2”
http://aws.amazon.com/ec2/instance-types/
EC2
inst
ance
type
s
![Page 41: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/41.jpg)
listening to customers
![Page 42: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/42.jpg)
new EC2 instance type
![Page 43: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/43.jpg)
text
cluster compute instances
http://aws.amazon.com/ec2/instance-types/
![Page 44: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/44.jpg)
2 * Xeon 5570 (“Intel Nehalem”)23 GB RAM
10 gbps Ethernet
1690 TB local disk
HVM-based virtualization
$1.60 / hr
![Page 45: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/45.jpg)
10gbps
![Page 46: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/46.jpg)
PlacementGroup
![Page 47: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/47.jpg)
full bisection bandwidth
![Page 48: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/48.jpg)
HPC on EC2 =
EC2 instance+
high bandwidth, low latency networking
http://aws.amazon.com/ec2/hpc-applications/
![Page 49: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/49.jpg)
Linpack benchmark
880-instance CC1 clusterPerformance: 41.82 TFlops*
*#231 in the most recent Top 500 rankings
![Page 50: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/50.jpg)
![Page 51: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/51.jpg)
CFDMolecular ModelingSequence AnalysisEngineering Design
Energy Trading…
high I/O applications
![Page 52: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/52.jpg)
standard “m1”high cpu “c1”
high memory “m2”
http://aws.amazon.com/ec2/instance-types/
cluster compute “cc1”
EC2
inst
ance
type
s
![Page 53: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/53.jpg)
HPC is evolving
![Page 54: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/54.jpg)
![Page 55: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/55.jpg)
cluster GPU instances
http://aws.amazon.com/ec2/instance-types/
![Page 56: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/56.jpg)
HPC on EC2 =
EC2 instance+
high bandwidth, low latency networking
+GPU
http://aws.amazon.com/ec2/hpc-applications/
![Page 57: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/57.jpg)
2 * Xeon 5570 (“Intel Nehalem”)
22 GB RAM
10 gbps Ethernet
1690 TB local disk
HVM-based virtualization
$2.10 / hr
2 * Tesla M2050 GPU
![Page 58: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/58.jpg)
standard “m1”high cpu “c1”
high memory “m2”
http://aws.amazon.com/ec2/instance-types/
cluster compute “cc1”
EC2
inst
ance
type
s
cluster GPU “cg1”
![Page 59: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/59.jpg)
CFDMolecular DynamicsFinancial Modeling
RenderingVideo Processing
…What is your interest?
![Page 60: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/60.jpg)
“90 percent scaling efficiency on clusters of up to 128 GPUs”
-- Mental Images iRay
![Page 61: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/61.jpg)
Getting Started
![Page 62: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/62.jpg)
http://aws.amazon.com/hpc-applications
![Page 63: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/63.jpg)
![Page 64: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/64.jpg)
4 steps
![Page 65: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/65.jpg)
15 minutes
![Page 67: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/67.jpg)
![Page 68: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/68.jpg)
![Page 69: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/69.jpg)
![Page 70: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/70.jpg)
![Page 71: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/71.jpg)
![Page 72: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/72.jpg)
ecosystem
![Page 73: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/73.jpg)
![Page 74: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/74.jpg)
ISV ecosystem
![Page 75: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/75.jpg)
Mathworksmental imagesRevup Render
Elemental Technologies...
![Page 76: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/76.jpg)
HPC with AWS
![Page 77: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/77.jpg)
E2 instance+
high bandwidth, low latency networking
+Tesla GPU*
*optional
![Page 78: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/78.jpg)
On demand Faster to market
Pay as you go Maintain focus
Pay to play Efficiency
Elastic resources Capacity planning
![Page 79: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/79.jpg)
make people productive
Credit: Pieter Musterd a CC-BY-NC-ND license
![Page 80: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/80.jpg)
Your Idea SuccessfulProduct
Great Idea Not Prioritized
![Page 81: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/81.jpg)
Your Idea SuccessfulProduct
![Page 82: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/82.jpg)
http://aws.amazon.com/hpc-applications
![Page 83: HPC on AWS](https://reader033.vdocument.in/reader033/viewer/2022052506/556d14b3d8b42ad34f8b544f/html5/thumbnails/83.jpg)
[email protected] Twitter:@mndoci
http://slideshare.net/mndocihttp://mndoci.com
Inspiration and ideas from Matt Wood, James Hamilton
& Larry Lessig
Credit” Oberazzi under a CC-BY-NC-SA license