elastic hpc extending the cluster into the cloud ruth lynch, research it service 13 th november 2009
TRANSCRIPT
![Page 1: Elastic HPC Extending the Cluster into the Cloud Ruth Lynch, Research IT Service 13 th November 2009](https://reader036.vdocument.in/reader036/viewer/2022062417/551a0f9a5503462e378b4e1b/html5/thumbnails/1.jpg)
Elastic HPCExtending the Cluster into the Cloud
Ruth Lynch, Research IT Service
13th November 2009
![Page 2: Elastic HPC Extending the Cluster into the Cloud Ruth Lynch, Research IT Service 13 th November 2009](https://reader036.vdocument.in/reader036/viewer/2022062417/551a0f9a5503462e378b4e1b/html5/thumbnails/2.jpg)
HPC in UCD
Research IT Service 2005
HPC is Core Service Phaeton Cluster -
Community & Shared Services
11% ICHEC Stokes Cluster 2008
![Page 3: Elastic HPC Extending the Cluster into the Cloud Ruth Lynch, Research IT Service 13 th November 2009](https://reader036.vdocument.in/reader036/viewer/2022062417/551a0f9a5503462e378b4e1b/html5/thumbnails/3.jpg)
What's the problem?
Physical HPC Cluster is inelastic Users sometimes need more resources than are
available Adding resources can be a slow process Capital intensive process
![Page 4: Elastic HPC Extending the Cluster into the Cloud Ruth Lynch, Research IT Service 13 th November 2009](https://reader036.vdocument.in/reader036/viewer/2022062417/551a0f9a5503462e378b4e1b/html5/thumbnails/4.jpg)
Possible Solutions
"Pre-order" hardware Buy compute time on other
facilities Investigate "The Cloud"
![Page 5: Elastic HPC Extending the Cluster into the Cloud Ruth Lynch, Research IT Service 13 th November 2009](https://reader036.vdocument.in/reader036/viewer/2022062417/551a0f9a5503462e378b4e1b/html5/thumbnails/5.jpg)
Which Cloud?
Cloud options Locations Different definitions
"a style of computing where massively scalable IT enabled capabilities are delivered 'as a service' to external customers using Internet technologies" - Gartner Inc.
![Page 6: Elastic HPC Extending the Cluster into the Cloud Ruth Lynch, Research IT Service 13 th November 2009](https://reader036.vdocument.in/reader036/viewer/2022062417/551a0f9a5503462e378b4e1b/html5/thumbnails/6.jpg)
Getting on the Cloud
Amazon Web Services No technical issues How to pay for the services?
Credit Card? Prepaid credit card? Prepay for Amazon credit? Get billed by Amazon? .......
![Page 7: Elastic HPC Extending the Cluster into the Cloud Ruth Lynch, Research IT Service 13 th November 2009](https://reader036.vdocument.in/reader036/viewer/2022062417/551a0f9a5503462e378b4e1b/html5/thumbnails/7.jpg)
“All Aboard” the Cloud
Contacted Amazon Applied for a research grant with Amazon Got a small grant
Started building...
![Page 8: Elastic HPC Extending the Cluster into the Cloud Ruth Lynch, Research IT Service 13 th November 2009](https://reader036.vdocument.in/reader036/viewer/2022062417/551a0f9a5503462e378b4e1b/html5/thumbnails/8.jpg)
Technical Specifications
Compute Nodes Up to 20 x Large Instances AMD Opteron 2218 HE 4 vCPU Cores 7GB of RAM CentOS 5.2 On Demand
![Page 9: Elastic HPC Extending the Cluster into the Cloud Ruth Lynch, Research IT Service 13 th November 2009](https://reader036.vdocument.in/reader036/viewer/2022062417/551a0f9a5503462e378b4e1b/html5/thumbnails/9.jpg)
Security
Users authenticate through Phaeton Login Node LDAP Authentication
Encrypted Communications 128 bit AES encryption for the data link 160 bit SHA1 checksum for both the data and
control channels
![Page 10: Elastic HPC Extending the Cluster into the Cloud Ruth Lynch, Research IT Service 13 th November 2009](https://reader036.vdocument.in/reader036/viewer/2022062417/551a0f9a5503462e378b4e1b/html5/thumbnails/10.jpg)
Management
Cloud Nodes are managed by Phaeton Head Node MOAB from Adaptive Computing / Torque Specific queue for cloud based resources
![Page 11: Elastic HPC Extending the Cluster into the Cloud Ruth Lynch, Research IT Service 13 th November 2009](https://reader036.vdocument.in/reader036/viewer/2022062417/551a0f9a5503462e378b4e1b/html5/thumbnails/11.jpg)
Data Concerns
Data Protection Laws Data Transfer Rules Intellectual Property Laws Data Privacy
![Page 12: Elastic HPC Extending the Cluster into the Cloud Ruth Lynch, Research IT Service 13 th November 2009](https://reader036.vdocument.in/reader036/viewer/2022062417/551a0f9a5503462e378b4e1b/html5/thumbnails/12.jpg)
Data Concerns
Amazon gives choice of location EU-WEST-1A & EU-WEST-1B
Data not "stored" in the cloud Computation only in the cloud
UCD CampusAWS
![Page 13: Elastic HPC Extending the Cluster into the Cloud Ruth Lynch, Research IT Service 13 th November 2009](https://reader036.vdocument.in/reader036/viewer/2022062417/551a0f9a5503462e378b4e1b/html5/thumbnails/13.jpg)
Computational Results
Real users with real jobs Results dependent on job and user type
0 1 2 3 4 5 6 7 8 90
5
10
15
20
25
30
35
Average Time per Step
Amazon Cloud
Phaeton
Job Number
Tim
e (
se
c)
![Page 14: Elastic HPC Extending the Cluster into the Cloud Ruth Lynch, Research IT Service 13 th November 2009](https://reader036.vdocument.in/reader036/viewer/2022062417/551a0f9a5503462e378b4e1b/html5/thumbnails/14.jpg)
What else?
AMD vs INTEL Licensing Tuning
Cost comparison Currently appears more expensive than local
resources Approx. €9,500 per month for AWS Costs have recently fallen
![Page 15: Elastic HPC Extending the Cluster into the Cloud Ruth Lynch, Research IT Service 13 th November 2009](https://reader036.vdocument.in/reader036/viewer/2022062417/551a0f9a5503462e378b4e1b/html5/thumbnails/15.jpg)
The Benefits
Immediately available resources "Once off" usage charge for resources Relatively quick to set up Easy to maintain Viable option for occasional large scale single
jobs
![Page 16: Elastic HPC Extending the Cluster into the Cloud Ruth Lynch, Research IT Service 13 th November 2009](https://reader036.vdocument.in/reader036/viewer/2022062417/551a0f9a5503462e378b4e1b/html5/thumbnails/16.jpg)
Future Plans
UCD “Cloud” Incorporating existing local resources Access to multiple public cloud resources Seamless service provision Automated “cloud bursting”
![Page 17: Elastic HPC Extending the Cluster into the Cloud Ruth Lynch, Research IT Service 13 th November 2009](https://reader036.vdocument.in/reader036/viewer/2022062417/551a0f9a5503462e378b4e1b/html5/thumbnails/17.jpg)
Thanks to ...
Amazon Kurt Messersmith Iain Galvin
UCD Research IT Team Research Community Lukasz Szmit