(arc309) building and scaling amazon cloud drive to millions of users | aws re:invent 2014
DESCRIPTION
Learn from the Amazon Cloud Drive team how Amazon Cloud Drive services are built on top of AWS core services using Amazon S3, Amazon DynamoDB, Amazon EC2, Amazon SQS, Amazon Kinesis, and Amazon CloudSearch. This session will cover design and implementation aspects of large-scale data uploads, metadata storage and query, and consistent and fault-tolerant services on top of the AWS stack. The session will provide guidance and best practices about how and when to leverage and integrate AWS infrastructure and managed services for scalable solutions. This session will also cover how Cloud Drive services teams innovated to attain high throughputs.TRANSCRIPT
![Page 1: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/1.jpg)
![Page 2: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/2.jpg)
![Page 4: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/4.jpg)
![Page 5: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/5.jpg)
Amazon S3 Amazon CloudFront
Amazon DynamoDB Amazon CloudSearch
Amazon SQS Elastic Transcoder
ElastiCache
Amazon Route 53
Amazon Kinesis Amazon EC2
Amazon EMR AWS Data PipelineAmazon RedshiftAmazon
CloudWatch
Elastic Load
Balancing
Content Storage, Delivery
& Sharing
Metadata Storage,
Indexing & Querying
Content Ingestion &
Transformation
Analytics, Monitoring &
ManagementAWS CloudFormation
![Page 6: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/6.jpg)
Amazon Cloud Drive Services
Indexing &
Search
AnalyticsAsynchronous
PipelineMetadata
Store
Content
Repository
Apps Route 53Users
ELBCloudFront
Elastic
Cache Kinesis
StreamMessage
Queue EMR RedshiftCloud
Search
EC2
Management
Data
Pipelines
Cloud
WatchCloud
Formation
S3
![Page 7: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/7.jpg)
![Page 8: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/8.jpg)
λ
![Page 9: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/9.jpg)
– Amazon S3
– DynamoDB
– Amazon CloudSearch
– Amazon Redshift
– Amazon Kinesis
– Amazon SQS
![Page 10: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/10.jpg)
Users
Cloud Drive Service
Amazon Kinesis
DynamoDB
CloudSearch
publish
(pk, v)
validate
(pk, v)
Resilient workflow SQS
write
(pk, v)
Merge
recent updates
(cid, q)
query results
(cid, q)
CloudSearch
update batch
batch
Backup,
Recovery
Notifications
Per-customer
processing
![Page 11: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/11.jpg)
HashKey GSI fields Additional Attributes
CustomerID:
NodeID
CID lastModified kind parents revision name status content
A08072:8TLrd A08072 1385438973435 FILE [Cgppb,…] 6 p1.jpg AVAILABLE size,MD5…
A08072:Cgppb A08072 1385449492345 ALBUM [NULL] 2 Rainier AVAILABLE -
![Page 12: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/12.jpg)
![Page 13: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/13.jpg)
![Page 14: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/14.jpg)
![Page 15: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/15.jpg)
![Page 16: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/16.jpg)
![Page 17: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/17.jpg)
![Page 18: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/18.jpg)
developer.amazon.com/public/apis/experience/cloud-drive
![Page 19: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/19.jpg)
amazon.com/clouddrive
amazon.com/clouddrive/primephotos
developer.amazon.com/public/apis/experience/cloud-drive
![Page 20: (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users | AWS re:Invent 2014](https://reader033.vdocument.in/reader033/viewer/2022052908/559444d91a28ab0c308b47f0/html5/thumbnails/20.jpg)
Please give us your feedback on this session.
Complete session evaluations and earn re:Invent swag.
http://bit.ly/awsevals