hpc in aws - technical workshop
TRANSCRIPT
![Page 1: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/1.jpg)
Alex Coqueiro Solutions Architect, Amazon Web Services
![Page 2: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/2.jpg)
![Page 3: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/3.jpg)
![Page 4: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/4.jpg)
Consumer Business
Milhões de clientes ativos
Operações globais em diversos paises ao redor do mundo
Seller"Business
Vendas nos sites da Amazon
Tecnologia baseada na sua própria rede de varejo
Alavancagem de centros integrados de fulfillment
Cloud Business
Infraestrutura de nuvem para host de aplicações corporativass
Centenas de milhares de clientes em mais de 190 paises
![Page 5: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/5.jpg)
Amplo conjunto de recursos computacionais que permitem as empresas moverem mais rapidamente
CLOUD
![Page 6: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/6.jpg)
![Page 7: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/7.jpg)
Why do researchers love using AWS?
Time to Science Access research
infrastructure in minutes
Globally Accessible Easily Collaborate with
researchers around the world Low Cost
Pay-as-you-go pricing
Secure A collection of tools to
protect data and privacy Elastic
Easily add or remove capacity
Scalable Access to effectively
limitless capacity
![Page 8: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/8.jpg)
Popular HPC workloads on AWS
Genome processing
Modeling and Simulation
Government and Educational Research
Monte Carlo Simulations
Transcoding and Encoding
Computational Chemistry
![Page 9: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/9.jpg)
![Page 10: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/10.jpg)
A marketplace for software in the Cloud
Over 1,900 listings across 23 categories Customers run over 70M hours of software per month
![Page 11: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/11.jpg)
AWS Marketplace – HPC category
aws.amazon.com/marketplace
![Page 12: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/12.jpg)
AWS Public Data Sets
aws.amazon.com/marketplace Free for everyone
![Page 13: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/13.jpg)
AWS Curriculum
http://aws.amazon.com/certification/
![Page 14: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/14.jpg)
![Page 15: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/15.jpg)
Over 1 million active customers across 190 countries 800+ government agencies 3,000+ educational institutions 11 regions 28 availability zones 52 edge locations
Everyday, AWS adds enough new server capacity to support Amazon.com when it was a $7 billion global enterprise.
![Page 16: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/16.jpg)
Availability Zone A
Availability Zone B
Availability Zone C
Region
Customer Decides Where Applications and Data Reside Note: Conceptual drawing only. The number of Availability Zones may vary.
![Page 17: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/17.jpg)
Enterprise Applications
Virtual Desktops Collaboration and Sharing
Platform Services
Databases
Caching
Relational
No SQL
Analytics
Hadoop
Real-time
Data Workflows
Data Warehouse
App Services
Queuing
Orchestration
App Streaming
Transcoding
Search
Deployment & Management
Containers
Dev/ops Tools
Resource Templates
Usage Tracking
Monitoring and Logs
Mobile Services
Identity
Sync
Mobile Analytics
Notifications
Foundation Services
Compute (VMs, Auto-scaling and Load Balancing)
Storage (Object, Block and Archive)
Security & Access Control Networking
Infrastructure Regions CDN and Points of Presence Availability Zones
![Page 18: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/18.jpg)
Compute Analytics Databases Storage
Imaging data
Phenotypes & comparative analysis
Upstream analysis Data mining
![Page 19: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/19.jpg)
![Page 20: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/20.jpg)
Enterprise Applications
Virtual Desktops Collaboration and Sharing
Platform Services
Databases
Caching
Relational
No SQL
Analytics
Hadoop
Real-time
Data Workflows
Data Warehouse
App Services
Queuing
Orchestration
App Streaming
Transcoding
Search
Deployment & Management
Containers
Dev/ops Tools
Resource Templates
Usage Tracking
Monitoring and Logs
Mobile Services
Identity
Sync
Mobile Analytics
Notifications
Foundation Services
Compute (VMs, Auto-scaling and Load Balancing)
Storage (Object, Block and Archive)
Security & Access Control Networking
Infrastructure Regions CDN and Points of Presence Availability Zones
![Page 21: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/21.jpg)
Amazon EC2
![Page 22: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/22.jpg)
![Page 23: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/23.jpg)
• Resizable compute capacity in >25 instance types • Reduces the time required to obtain and boot new server
instances to minutes or seconds • Scale capacity as your computing requirements change • Pay only for capacity that you actually use • Choose Linux or Windows • Deploy across Regions and Availability Zones for reliability • Support for virtual network interfaces that can be attached to
EC2 instances in your VPC
![Page 24: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/24.jpg)
General Purpose
(Burstable or Fixed Performance)
Compute Optimized
Memory Optimized
GPU Instances
Storage Optimized
![Page 25: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/25.jpg)
Compute Optimized
Name vCPU Memory (GiB) Network
c4.large 2 3.75 Moderate
c4.xlarge 4 7.5 Moderate
c4.2xlarge 8 15 High
c4.4xlarge 16 30 High
c4.8xlarge 36 60 10 Gbps
![Page 26: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/26.jpg)
Storage Optimized
Name vCPU Memory (GiB) Network HDD
d2.xlarge 4 30.5 Moderate 3 x 2000
d2.2xlarge 8 61 High 6 x 2000
d2.4xlarge 16 122 High 12 x 2000
d2.8xlarge 36 244 10Gb 24 x 2000
![Page 27: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/27.jpg)
Intel Xeon E5-2670 (Sandy Bridge)
15GB or 60GB RAM
1 NVIDIA Grid k520 GPU 1,536 Cores 4GB Mem
GPU Optimized
Name GPU vCPU Memory (GiB) Network SSD
g2.2xlarge 1 8 15 High 1 x 60
g2.8xlarge 4 32 60 10 Gb 2 x 120
![Page 28: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/28.jpg)
Demo
![Page 29: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/29.jpg)
![Page 30: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/30.jpg)
Time:+00h
Scale using Elastic Capacity
<10 cores
![Page 31: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/31.jpg)
Time: +24h
Scale using Elastic Capacity
>1500 cores
![Page 32: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/32.jpg)
Time:+72h
Scale using Elastic Capacity
<10 cores
![Page 33: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/33.jpg)
Time: +120h
Scale using Elastic Capacity
>600 cores
![Page 34: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/34.jpg)
Demo
![Page 35: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/35.jpg)
![Page 36: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/36.jpg)
Reserved Make a low, one-time payment and receive a significant discount on the hourly charge For committed utilization
Free Tier Get Started on AWS with free usage & no commitment For POCs and getting started
On-Demand Pay for compute capacity by the hour with no long-term commitments For spiky workloads, or to define needs
Spot Bid for unused capacity, charged at a Spot Price which fluctuates based on supply and demand For time-insensitive or transient workloads
Dedicated Launch instances within Amazon VPC that run on hardware dedicated to a single customer For highly sensitive or compliance related workloads
![Page 37: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/37.jpg)
![Page 38: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/38.jpg)
On
On-demand
Reserved capacity
100%
Capacity Over Time
AWS Spot MarketAchieving economies of scale
Spot
0%
![Page 39: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/39.jpg)
![Page 40: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/40.jpg)
![Page 41: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/41.jpg)
![Page 42: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/42.jpg)
* Prices on April 17, 2015
![Page 43: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/43.jpg)
* Prices on April 17, 2015
![Page 44: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/44.jpg)
* Prices on April 17, 2015
![Page 45: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/45.jpg)
* Prices on April 17, 2015
![Page 46: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/46.jpg)
![Page 47: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/47.jpg)
aws autoscale create-launch-configuration --launch-configuration-name spotlc-5cents --image-id ami-e565ba8c --instance-type d2.2xlarge --spot-price “0.25”
aws autoscale create-auto-scaling-group --auto-scaling-group-name spotasg --launch-configuration spotlc-5cents --availability-zones “us-east-1a,us-east-1b” --max-size 16 --min-size 1 --desiredcapacity 3
http://aws.amazon.com/cli/
![Page 48: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/48.jpg)
Demo
![Page 49: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/49.jpg)
![Page 50: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/50.jpg)
Compute Analytics Databases Storage
Imaging data
Phenotypes & comparative analysis
Upstream analysis Data mining
![Page 51: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/51.jpg)
![Page 52: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/52.jpg)
Enterprise Applications
Virtual Desktops Collaboration and Sharing
Platform Services
Databases
Caching
Relational
No SQL
Analytics
Hadoop
Real-time
Data Workflows
Data Warehouse
App Services
Queuing
Orchestration
App Streaming
Transcoding
Search
Deployment & Management
Containers
Dev/ops Tools
Resource Templates
Usage Tracking
Monitoring and Logs
Mobile Services
Identity
Sync
Mobile Analytics
Notifications
Foundation Services
Compute (VMs, Auto-scaling and Load Balancing)
Storage (Object, Block and Archive)
Security & Access Control Networking
Infrastructure Regions CDN and Points of Presence Availability Zones
![Page 53: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/53.jpg)
![Page 54: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/54.jpg)
![Page 55: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/55.jpg)
![Page 56: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/56.jpg)
AWS region
AZ - B
VPC 10.0.0.0/16
SN 10.0.1.0/24
M E
E
E
VPC Endpoint
AZ - A
Internet GW Service
SN 10.0.2.0/24
E
E
E
![Page 57: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/57.jpg)
M E
E
E
S S S
![Page 58: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/58.jpg)
Demo
![Page 59: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/59.jpg)
Enterprise Applications
Virtual Desktops Collaboration and Sharing
Platform Services
Databases
Caching
Relational
No SQL
Analytics
Hadoop
Real-time
Data Workflows
Data Warehouse
App Services
Queuing
Orchestration
App Streaming
Transcoding
Search
Deployment & Management
Containers
Dev/ops Tools
Resource Templates
Usage Tracking
Monitoring and Logs
Mobile Services
Identity
Sync
Mobile Analytics
Notifications
Foundation Services
Compute (VMs, Auto-scaling and Load Balancing)
Storage (Object, Block and Archive)
Security & Access Control Networking
Infrastructure Regions CDN and Points of Presence Availability Zones
![Page 60: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/60.jpg)
![Page 61: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/61.jpg)
Enterprise Applications
Virtual Desktops Collaboration and Sharing
Platform Services
Databases
Caching
Relational
No SQL
Analytics
Hadoop
Real-time
Data Workflows
Data Warehouse
App Services
Queuing
Orchestration
App Streaming
Transcoding
Search
Deployment & Management
Containers
Dev/ops Tools
Resource Templates
Usage Tracking
Monitoring and Logs
Mobile Services
Identity
Sync
Mobile Analytics
Notifications
Foundation Services
Compute (VMs, Auto-scaling and Load Balancing)
Storage (Object, Block and Archive)
Security & Access Control Networking
Infrastructure Regions CDN and Points of Presence Availability Zones
![Page 62: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/62.jpg)
![Page 63: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/63.jpg)
Demo
![Page 64: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/64.jpg)
Demo
![Page 65: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/65.jpg)
Storage Database Compute
![Page 66: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/66.jpg)
![Page 67: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/67.jpg)
Cloud automation allows for security agility “Programmable infrastructure” allows you to automate every aspect your environment. Security properties are “baked in,” constantly checked via logging and auditing, and deviations / alarms are actionable via code Change and speed of change become an asset, not a liability
![Page 68: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/68.jpg)
aws ec2 create-vpc --cidr-block 10.0.0.0/16
aws ec2 replace-route --route-table-id $ROUTE_TABLE_ID
--destination-cidr-block 0.0.0.0/0
--instance-id $INSTANCE_ID
aws ec2 attach-network-interface --network-interface-id $ENI
--instance-id $INSTANCE_ID
--device-index 1
aws ec2 assign-private-ip-addresses --network-interface-id $ENI
--private-ip-addresses 10.0.0.100
• AWS CLI
![Page 69: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/69.jpg)
#!/bin/sh export AWS_DEFAULT_REGION="us-east-1"
VPC_ID=`aws ec2 create-vpc --cidr-block 10.0.0.0/16 --output text | awk '{print $6;}'`
SUBNET_ID=`aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 10.0.1.0/24 --output text | awk '{print $6;}'`
echo "Created $VPC_ID & $SUBNET_ID"
#Clean up
aws ec2 delete-subnet --subnet-id $SUBNET_ID
aws ec2 delete-vpc --vpc-id $VPC_ID
![Page 70: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/70.jpg)
#!/usr/bin/python import boto.vpc
Region=“us-east-1”
conn = boto.vpc.VPCConnection(Region)
vpc = conn.create_vpc(‘10.0.0.0/16’)
subnet = conn.create_subnet(vpc.id ‘10.0.1.0/24’)
Print "Created “+vpc.id+” & “+subnet.id
#Clean up
conn.delete_subnet(subnet.id)
conn.delete_vpc(vpc.id)
• Amazon SDK
![Page 71: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/71.jpg)
"Resources" : { "VPC" : {
"Type" : "AWS::EC2::VPC",
"Properties" : {
"CidrBlock" : “10.0.0.0/16”,
"Tags" : [ { "Key" : “Name", "Value" : “VPCName“ } ]
}
}, "PublicSubnet" : {
"Type" : "AWS::EC2::Subnet",
"Properties" : {
"VpcId" : { "Ref" : "VPC" },
"CidrBlock" : “10.0.1.0/24”,
"Tags" : [ { "Key" : "Network", "Value" : "Public" } ] }
}
• AWS CloudFormation
![Page 72: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/72.jpg)
Demo
![Page 73: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/73.jpg)
Try out our HPC CloudFormation-based demo
CfnCluster (“CloudFormation cluster”)
Command Line Interface Tool Deploy and demo an HPC cluster
For more info:
https://aws.amazon.com/hpc/cfncluster
![Page 74: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/74.jpg)
![Page 75: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/75.jpg)
Facilities
Physical security
Compute infrastructure
Storage infrastructure
Network infrastructure
Virtualization layer (EC2)
Hardened service endpoints
Rich IAM capabilities
Network configuration
Security groups
OS firewalls
Operating systems
Applications
Proper service configuration
Auth & acct management
Authorization policies
+ =
• Re-focus your security professionals on a subset of the problem • Take advantage of high levels of uniformity and automation
Customer/Partner Audited
![Page 76: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/76.jpg)
Web Tier
Application Tier
Database Tier Porta 80 e 443
Time de Engenharia com ssh
Todos os demais acessos bloqueados
Acesso analítico de dados Amazon EC2 Security Group Firewall
![Page 77: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/77.jpg)
Rich control with AWS’s powerful Identity & Access Management capabilities
Authentication: • Multiple options including rich SAML
federation capabilities, MFA, web identities
• Clean separation of identity from proof of identity
• Roles are powerful and flexible pseudo-principals that can be assumed by other identities • Federation scenarios • Cross-account access
![Page 78: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/78.jpg)
Network isolation with Virtual Private Cloud Define your own address space as extension of private network Connect to private network with VPN tunnel or Direct Connect Configure Security Groups (virtual firewalls) for all EC2 instances; update fleet firewall rules with a single API call Configure Network Access Control Lists for subnet level isolation and control
![Page 79: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/79.jpg)
Enhanced isolation and control with encryption Automatic encryption with managed keys (Key Management Service) Dedicated hardware security modules (Cloud HSM) Bring and use your own keys
![Page 80: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/80.jpg)
Encrypt your data prior to sending to AWS
Your applications in your data center
Your applications in Amazon EC2 Encrypted
Data
AWS Services
S3 Glacier Redshift EBS
![Page 81: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/81.jpg)
Encryption Primer
Plaintext PHI
Hardware/ Software
Encrypted PHI
Symmetric Data Key
Encrypted Data Key
Master Key Symmetric Data Key
?
Encrypted Data in Storage
Key Hierarchy
?
![Page 82: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/82.jpg)
S3 Client-Side Encryption Amazon S3 Encryption Client with AWS SDKs
Your key management infrastructure
Your applications in your data
center
Your key management
infrastructure in EC2
Your Encrypted Data in Amazon S3
Your application in Amazon EC2
AWS SDK with S3 Encryption Client
![Page 83: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/83.jpg)
S3 SSE with Customer Provided Keys Works
Plaintext PHI
Encrypted Data
Customer Provided Key S3 Web Server
HTTPS Customer
PHI
S3 Storage Fleet
• Key is used at S3 server, then deleted • Customer must provide same key when
downloading to allow S3 to decrypt data
Customer Provided Key
![Page 84: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/84.jpg)
S3 SSE with AWS fully managed keys
Plaintext PHI
Encrypted PHI
Symmetric Data Key S3 Web Server
HTTPS Customer
PHI
Encrypted Data Key
Master Key Symmetric Data Key
S3 Storage Fleet
A master key managed by the S3 service and protected by systems internal to AWS in a
distinct system
![Page 85: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/85.jpg)
Amazon EBS
Amazon S3
• HTTPS • AES-256 server-side encryption • AWS or customer provided or customer managed keys • Each object gets its own key
• End-to-end secure network traffic • Whole volume encryption • AWS or customer managed keys • Encrypted incremental snapshots • Minimal performance overhead (utilizes Intel AES-NI)
![Page 86: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/86.jpg)
![Page 87: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/87.jpg)
![Page 88: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/88.jpg)
Integrated with AWS IAM Console
![Page 89: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/89.jpg)
Integrated with Amazon EBS
![Page 90: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/90.jpg)
How AWS Services Integrate with KMS • 2-tiered key hierarchy using envelope
encryption
• Data keys encrypt customer data
• KMS customer master keys encrypt data keys
• Benefits: • Limits blast radius of compromised
resources and their keys • Better performance • Easier to manage a small number of master
keys than billions of resource keys
Master Key(s)
Data Key 1
S3 Object EBS Volume
RDS Instance
Redshift Cluster
Data Key 2 Data Key 3 Data Key 4 Data Key 5
Your Application
Keys encrypted
Data encrypted
KMS
![Page 91: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/91.jpg)
bit.ly/aws-dbgap
![Page 92: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/92.jpg)
aws.amazon.com/hpc
![Page 93: HPC in AWS - Technical Workshop](https://reader031.vdocument.in/reader031/viewer/2022032002/55b53fb1bb61eb03558b47b7/html5/thumbnails/93.jpg)
http://bit.ly/aws-dbgap
Architecting for Genomic Data Security and Compliance in AWS