Download - Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013
![Page 1: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/1.jpg)
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
DAT304 - Mastering NoSQL: Advanced Amazon
DynamoDB Design Patterns for Ultra-High
Performance Apps
David Yanacek, Amazon DynamoDB
David Tuttle, Devicescape
Greg Nelson, Dropcam
November 15, 2013
![Page 2: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/2.jpg)
Plan
• Social Network (hash + range schemas)
• Image Tagging (secondary indexes)
• Game State (conditional writes)
• User Data (fine-grained access control)
![Page 3: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/3.jpg)
Social Network hash + range schemas
![Page 4: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/4.jpg)
Social Network
• Store info about users
• Query for a user’s friends
![Page 5: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/5.jpg)
Social Network
Users Table Friends Table
![Page 6: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/6.jpg)
User Nicknames
Bob [ Rob, Bobby ]
Alice [ Allie ]
Carol [ Caroline ]
Dan [ Daniel, Danny ]
Social Network
Users Table
![Page 7: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/7.jpg)
User Nicknames
Bob [ Rob, Bobby ]
Alice [ Allie ]
Carol [ Caroline ]
Dan [ Daniel, Danny ]
Social Network
Users Table
Item
![Page 8: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/8.jpg)
User Nicknames
Bob [ Rob, Bobby ]
Alice [ Allie ]
Carol [ Caroline ]
Dan [ Daniel, Danny ]
Social Network
Users Table
Attribute
(string, number, binary, set)
![Page 9: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/9.jpg)
User Nicknames
Bob [ Rob, Bobby ]
Alice [ Allie ]
Carol [ Caroline ]
Dan [ Daniel, Danny ]
Social Network
Users Table
Primary Key
(Hash)
![Page 10: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/10.jpg)
User Nicknames
Bob [ Rob, Bobby ]
Alice [ Allie ]
Carol [ Caroline ]
Dan [ Daniel, Danny ]
Social Network
Friends Table
User Friend
Bob Alice
Alice Bob
Alice Carol
Alice Dan
Users Table
![Page 11: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/11.jpg)
User Nicknames
Bob [ Rob, Bobby ]
Alice [ Allie ]
Carol [ Caroline ]
Dan [ Daniel, Danny ]
Social Network
Friends Table
User Friend
Bob Alice
Alice Bob
Alice Carol
Alice Dan
Users Table
Hash-and-Range
Primary Key Schema
![Page 12: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/12.jpg)
User Nicknames
Bob [ Rob, Bobby ]
Alice [ Allie ]
Carol [ Caroline ]
Dan [ Daniel, Danny ]
Social Network
Friends Table
User Friend
Bob Alice
Alice Bob
Alice Carol
Alice Dan
Users Table
Query for Alice’s friends
![Page 13: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/13.jpg)
Recap: Social Network
• Hash schema for key-value lookups
• Hash and Range schema for Query
![Page 14: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/14.jpg)
Image Tagging secondary indexes
![Page 15: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/15.jpg)
Image Tagging
• Query a user’s images
• Query a user’s images by date
• Tag other users in images
• Query images a user is tagged in
![Page 16: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/16.jpg)
Image Tagging
Images Table
![Page 17: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/17.jpg)
Image Tagging
Images Table
User Image Date Link
Bob aed4c 2013-10-01 s3://…
Bob cf2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
![Page 18: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/18.jpg)
Image Tagging
Images Table
User Image Date Link
Bob aed4c 2013-10-01 s3://…
Bob cf2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
Hash and Range
Primary Key Schema
![Page 19: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/19.jpg)
Image Tagging
Images Table
User Image Date Link
Bob aed4c 2013-10-01 s3://…
Bob cf2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
Bob
Query for Bob’s Images
![Page 20: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/20.jpg)
Image Tagging
• Query a user’s images
• Query a user’s images by date
• Tag other users in images
• Query images a user is tagged in
![Page 21: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/21.jpg)
Local Secondary Indexes
• Alternate Range Key for your table
• More flexible Query patterns
• Local to the Hash Key
![Page 22: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/22.jpg)
Image Tagging
Images Table
User Image Date Link
Bob aed4c 2013-10-01 s3://…
Bob cf2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
Local to a Hash Key value
Table
![Page 23: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/23.jpg)
Image Tagging
Images Table
User Image Date Link
Bob aed4c 2013-10-01 s3://…
Bob cf2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
User Date Image
Bob 2013-09-05 cf2e2
Bob 2013-10-01 aed4c
Bob 2013-10-08 f93bae
Alice 2013-09-12 ca61a Table ByDate Local Secondary Index
Local Secondary Index on Date
![Page 24: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/24.jpg)
Image Tagging
Images Table
User Image Date Link
Bob aed4c 2013-10-01 s3://…
Bob cf2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
User Date Image
Bob 2013-09-05 cf2e2
Bob 2013-10-01 aed4c
Bob 2013-10-08 f93bae
Alice 2013-09-12 ca61a Table ByDate Local Secondary Index
Query for Bob’s two most recent images
![Page 25: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/25.jpg)
Image Tagging
• Query a user’s images
• Query a user’s images by date
• Tag other users in images
• Query images a user is tagged in
![Page 26: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/26.jpg)
Image Tagging
Images
Table
ImageTags
Table
![Page 27: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/27.jpg)
Image Tagging
ImageTags Table
Image User
aed4c Alice
aed4c Bob
f93bae Bob
![Page 28: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/28.jpg)
Image Tagging
ImageTags Table
Image User
aed4c Alice
aed4c Bob
f93bae Bob
Hash and Range Primary Key Schema
![Page 29: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/29.jpg)
Image User
aed4c Alice
aed4c Bob
f93bae Bob
Image Tagging
ImageTags Table
Bob
Query Users
tagged in Image aed4c
![Page 30: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/30.jpg)
Image Tagging
ImageTags Table
Image User
aed4c Alice
aed4c Bob
f93bae Alice
f93bae Bob
Bob
Tag Alice in Image f93bae
![Page 31: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/31.jpg)
Image Tagging
• Query a user’s images
• Query a user’s images by date
• Tag other users in images
• Query images a user is tagged in
![Page 32: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/32.jpg)
Global Secondary Indexes
• Alternate Hash and/or Range Key for your table
• Even more flexible Query patterns
![Page 33: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/33.jpg)
Image Tagging
ImageTags Table
Image User
aed4c Alice
aed4c Bob
f93bae Alice
f93bae Bob Table
![Page 34: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/34.jpg)
Image Tagging
ImageTags Table
Global Secondary Index on User, Image
User Image
Bob aed4c
Bob f93bae
Alice aed4c
Alice f93bae ByUser Global Secondary Index
Image User
aed4c Alice
aed4c Bob
f93bae Alice
f93bae Bob Table
![Page 35: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/35.jpg)
Image Tagging
ImageTags Table
User Image
Bob aed4c
Bob f93bae
Alice aed4c
Alice f93bae ByUser Global Secondary Index
Image User
aed4c Alice
aed4c Bob
f93bae Alice
f93bae Bob Table
Alternate Hash and Range Keys
![Page 36: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/36.jpg)
Image Tagging
ImageTags Table
Alice Query for images tagged Alice
User Image
Bob aed4c
Bob f93bae
Alice aed4c
Alice f93bae ByUser Global Secondary Index
Image User
aed4c Alice
aed4c Bob
f93bae Alice
f93bae Bob Table
![Page 37: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/37.jpg)
Recap: Image Tagging
• Local Secondary Indexes support flexible queries
• Global Secondary Indexes unlock even more
flexible queries
![Page 38: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/38.jpg)
Game State conditional writes
![Page 39: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/39.jpg)
Tic Tac Toe
![Page 40: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/40.jpg)
Tic Tac Toe
{
Id : abecd,
Players : [ Alice, Bob ],
State : STARTED,
Turn : Bob,
Top-Right : O
}
Game Item
![Page 41: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/41.jpg)
Tic Tac Toe
Amazon
DynamoDB
Alice Bob
![Page 42: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/42.jpg)
Tic Tac Toe
Amazon
DynamoDB
Alice Bob
Update:
Top-Right : O
Turn : Bob
![Page 43: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/43.jpg)
Tic Tac Toe
Amazon
DynamoDB
Alice Bob
Update:
Top-Left : X
Turn : Alice
![Page 44: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/44.jpg)
Gaming the System
Alice Bob (1)
Amazon
DynamoDB
Bob (2) Bob (3)
![Page 45: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/45.jpg)
Gaming the System
Alice Bob (1)
Amazon
DynamoDB
Bob (2) Bob (3)
![Page 46: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/46.jpg)
Gaming the System
Alice Bob (1)
Amazon
DynamoDB
Bob (2) Bob (3)
![Page 47: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/47.jpg)
Gaming the System
Bob (1)
Amazon
DynamoDB
Bob (2) Bob (3)
State : STARTED,
Turn : Bob,
Top-Right : O
![Page 48: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/48.jpg)
Gaming the System
Bob (1)
Amazon
DynamoDB
Bob (2) Bob (3)
Update:
Turn : Alice
Top-Left : X
Update:
Turn : Alice
Mid : X
State : STARTED,
Turn : Bob,
Top-Right : O
Update:
Turn : Alice
Low-Right : X
![Page 49: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/49.jpg)
Gaming the System
Bob (1)
Amazon
DynamoDB
Bob (2) Bob (3)
Update:
Turn : Alice
Top-Left : X
Update:
Turn : Alice
Mid : X
State : STARTED,
Turn : Alice,
Top-Right : O,
Top-Left : X,
Mid: X,
Low-Right: X
Update:
Turn : Alice
Low-Right : X
![Page 50: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/50.jpg)
Conditional Writes
• Accept a write only if values are as expected
• Otherwise reject the write
• Performance like normal writes
• Single item only (no transactions)
![Page 51: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/51.jpg)
Tic Tac Toe (Fixed)
Bob (1)
Amazon
DynamoDB
Bob (2) Bob (3)
Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null
State : STARTED,
Turn : Bob,
Top-Right : O
Update:
Turn : Alice
Mid : X
Expect:
Turn : Bob
Mid : null
Update:
Turn : Alice
Low-Right : X
Expect:
Turn : Bob
Low-Right : null
![Page 52: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/52.jpg)
Tic Tac Toe (Fixed)
Bob (1)
Amazon
DynamoDB
Bob (2) Bob (3)
State : STARTED,
Turn : Bob,
Top-Right : O
Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null
Update:
Turn : Alice
Low-Right : X
Expect:
Turn : Bob
Low-Right : null
Update:
Turn : Alice
Mid : X
Expect:
Turn : Bob
Mid : null
![Page 53: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/53.jpg)
Tic Tac Toe (Fixed)
Bob (1)
Amazon
DynamoDB
Bob (2) Bob (3)
State : STARTED,
Turn : Alice,
Top-Right : O,
Top-Left : X
Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null
Update:
Turn : Alice
Mid : X
Expect:
Turn : Bob
Mid : null
Update:
Turn : Alice
Low-Right : X
Expect:
Turn : Bob
Low-Right : null
![Page 54: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/54.jpg)
Recap: Game State
• Conditional writes synchronize state transitions
• Multi-item transactions require application-level
coordination
![Page 55: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/55.jpg)
User Data fine-grained access control
![Page 56: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/56.jpg)
User Data
Your App
Users
Amazon
DynamoDB
![Page 57: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/57.jpg)
User Data
Users (Cost, Ops, Latency)
Amazon
DynamoDB
Your App
![Page 58: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/58.jpg)
User Data
Users
Amazon
DynamoDB
![Page 59: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/59.jpg)
User Data
Users
(Access control?)
Amazon
DynamoDB
![Page 60: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/60.jpg)
Web Identity Federation
Users
AWS IAM
Web identity federation
Amazon
DynamoDB
![Page 61: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/61.jpg)
Web Identity Federation
Users
AWS IAM
Web identity federation
(Fine-grained access control)
Amazon
DynamoDB
![Page 62: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/62.jpg)
Fine-Grained Access Control
• Limit access to particular hash key values
• Limit access to specific attributes
• Use policy substitution variables to write the
policy once
![Page 63: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/63.jpg)
Fine-Grained Access Control
Images Table
User Image Date Link
Bob aed4c 2013-10-01 s3://…
Bob 5f2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
“Allow all authenticated
Facebook users to Query the
Images table, but only on items
where their Facebook ID is the
hash key”
![Page 64: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/64.jpg)
Fine-Grained Access Control
Images Table
User Image Date Link
Bob aed4c 2013-10-01 s3://…
Bob 5f2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
Bob
AWS
IAM
Bob “logs in” using
web identity federation
![Page 65: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/65.jpg)
Fine-Grained Access Control
Images Table
User Image Date Link
Bob aed4c 2013-10-01 s3://…
Bob 5f2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
Bob
Bob can Query for Images
where User=“Bob”
![Page 66: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/66.jpg)
Fine-Grained Access Control
Images Table
User Image Date Link
Bob aed4c 2013-10-01 s3://…
Bob 5f2e2 2013-09-05 s3://…
Bob f93bae 2013-10-08 s3://…
Alice ca61a 2013-09-12 s3://…
Bob
Bob cannot Query for Images
where User=“Alice”
![Page 67: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/67.jpg)
Two-tier Architecture Tradeoffs
• Pros: – Lower latency
– Lower cost
– Lower operational complexity
• Cons: – Less visibility into application behavior
– More difficult to make changes to persistence layer
– Requires “scoping” items to a given user
Amazon
DynamoDB
Users
![Page 68: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/68.jpg)
Recap: User Data
• Web identity federation makes it easy for end-
users to call AWS directly
• Fine-grained access control supports a secure
two-tier architecture on Amazon DynamoDB
![Page 69: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/69.jpg)
Recap (Thanks!)
• Social Network (hash + range schemas)
• Image Tagging (secondary indexes)
• Game State (conditional writes)
• User Data (fine-grained access control)
![Page 70: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/70.jpg)
An Amazon DynamoDB Adoption Story:
Before, After, and Beyond David Tuttle, Engineering Manager, Platform Team
![Page 71: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/71.jpg)
Devicescape Software
• Headquartered in Silicon Valley
• Founded in 2005
• Curate Amenity Wi-Fi into a carrier grade service
• Critical services with tight uptime requirements
![Page 72: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/72.jpg)
New York Amenity WiFi CVN Connections
(24 hours later)
![Page 73: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/73.jpg)
An Amazon DynamoDB Adoption Story
• Before: Pre-DynamoDB world
• During: The move to DynamoDB
• After: DynamoDB has improved our lives
• Beyond: Future direction
![Page 74: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/74.jpg)
Before: Challenges in a Pre-DynamoDB World
• Complex manual scaling – Scaling up was not a fast operation
– DB/DB slave maintenance was painful
– Handcrafted table sharding required to scale DB with application
• Slow queries on large tables – Developed processes to rotate tables before they got too large
– Hindered by unpredictable query time
– Developer effort spent optimizing SQL queries
• Schema changes are difficult on large active tables – Forced to code a workaround when we would have preferred altering a table
• Replication delay
table_shardBit0_shardBit1 {
shardKey string, primary key;
}
![Page 75: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/75.jpg)
Move to DynamoDB
• DynamoDB forced a different way of thinking about data – Data organized into “items” around a primary key (hash key, [range key])
– Configured level of read + write throughput in DynamoDB is achievable if and only if certain conditions are met
• Avoid sparse hash keys
• Workload evenly distributed across hash keys
• Throttled scans and queries – A scan or a query without a limit risked consuming all of the throughput for an
internal partition of a DynamoDB table
– Parallel scans required for high-speed jobs
• Porting the application backend to use DynamoDB – Wealth of provided AWS SDKs: Java, PHP, Python boto, etc.
– DynamoDB web service facilitates development in any programming language
![Page 76: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/76.jpg)
Move to DynamoDB – Data Migration • Migration performed in a single 4-hour window
– I/O overhead requires careful balancing to achieve desired processing speeds
– Provisioned throughput was orders of magnitude higher than required for
post-migration load
– Excessive internal partitioning and non-uniform workload lead to
configured/observed throughput mismatch
• Better: dual-writes with multi-phased deployment
Configured
Ideal Actual
![Page 77: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/77.jpg)
Alive – Client Network Health Checks
• Alive is a mission critical Devicescape service – Use event data to both bill customers and grow our WiFi database
– 250 million events per day (and growing!)
• Reliable high performance is needed – Alive is written to DynamoDB within the bounds of a device’s HTTP request
– It must be read from DynamoDB to support real-time processing
• Data is organized as a time series – Each table represents a particular day in UTC
– The next day’s table is created via a Python script
– We keep up to 60 days worth of data and expire old data based on a retention policy
![Page 78: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/78.jpg)
tsUuidShard uuid
1382969157_e2 0e645c9c-08b9-48b4-a5b0-c310957451e2
1382969157_1e c349262e-75a3-444b-9716-20f09823411e
1382969157_1e c349262e-75a3-444b-9716-20f09823411e
Alive – Schema • Hash key: Timestamp + UUID shard
– Indexed by time
– Writes distributed over 256 hash keys/second
tsUuidShard uuidIndex index
1382969157_e2 0e645c9c-08b9-48b4-a5b0-c310957451e2_0
1382969157_1e c349262e-75a3-444b-9716-20f09823411e_0 1
1382969157_1e c349262e-75a3-444b-9716-20f09823411e_1
alive_<YYYYMMDD> {
tsUuidShard, string hash key;
uuidIndex string, range key;
index number;
. . .
}
• Collisions are infrequent, but must be handled
– Append index to range key
– Atomic increment index in first item
![Page 79: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/79.jpg)
Alive – Plan for Failure • Distribute HTTP front ends over 3 Availability Zones
– Amazon DynamoDB and Elastic Load Balancing inherently multi-AZ
• One “Patient uploader” per instance
– HTTPD tries once to DynamoDB ands write to local file on failure
– Patient uploader monitors local file and retries events with backoff
– Operations team alerted if patient uploader is > 1 hour behind
Elastic Load Balancing
Amazon DynamoDB
HTTP AZ-A
HTTP AZ-C
HTTP AZ-B
![Page 80: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/80.jpg)
Alive – Data Processing • System usage and customer billing
– Alive data is a key component of our connection detail record (CDR)
– We have a processor running 2 hours behind current time that queries each second’s event data
• Multiple threads access 1 shard each
• Near real-time WiFi data improvement – Alive helps us quickly discover high-quality access points
– Processor running 5 minutes behind current time
• Batch processing
– Daily extraction to Amazon S3 allows us to perform counting and
aggregation of the alive data in Amazon EMR with HBase and Hive
– We can perform broader trend analysis on the data that is not feasible
in real-time
![Page 81: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/81.jpg)
After: How Devicescape uses DynamoDB • Critical ‘online’ services principally on DynamoDB
• Infrastructure cost savings of 30%
• Greatly reduced DB maintenance
• New developments use DynamoDB
Amazon
DynamoDB
HTTP
AZ-A
HTTP
AZ-B
CloudWatch
Amazon
DynamoDB
Application Amazon S3
Amazon EMR ElastiCache
![Page 82: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/82.jpg)
Beyond: Future Plans
• Move more into DynamoDB! – Some other data sources that we thought could remain in MySQL
(e.g. event logs) have already begun to hit pain points.
• Investigate new Geospatial Library – We have worried about how our Geohash + Memcache + MySQL
solution for WiFi geolocation data would translate into the DynamoDB
and NoSQL worlds.
– Eventually the costs of scaling theses datasources in our handcrafted
MySQL sharding solution will become prohibitive.
• Use DynamoDB Local – Incorporate private, local DynamoDB version into developer sandboxes
![Page 83: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/83.jpg)
Thanks!
We are hiring! www.devicescape.com/company/careers
@devicescape
David Tuttle [email protected]
![Page 84: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/84.jpg)
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
+ Amazon DynamoDB
Greg Nelson
November 15, 2013
![Page 85: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/85.jpg)
What is Dropcam?
• Software company in SF
• Wi-Fi enabled camera
• Intelligent activity recognition
• Apps (iOS, Android, Web)
• Cloud recording
![Page 86: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/86.jpg)
Dropcam Uses
Home security
Toyota dealer saw I-5 bridge collapse The Baconery - NYC Woodland Park Zoo – Seattle, WA
Kyra – N. Virginia Dropcam employee Burglar caught – Bellevue, WA
Family Baby
Pets Small business Just because
![Page 87: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/87.jpg)
Cloud Recording (CVR)
• Continuous recording to the cloud
• Accessible instantly from anywhere
• Activity recognition – Motion detection
– Machine learning
$9.95 / mo.
$29.95 / mo.
![Page 88: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/88.jpg)
Scalability Challenges
2009 2010 2011 2012 2013
Move to AWS
More incoming video than YouTube
Suddenly petabytes
cameras
![Page 89: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/89.jpg)
Evolution to the Cloud
Amazon EC2
Amazon S3
Amazon
SimpleDB
Amazon
DynamoDB
Traditional managed hosting
![Page 90: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/90.jpg)
Dropcam on AWS
Nexus (Scala)
Web (Python) Amazon
DynamoDB
Amazon S3
Cameras Users
HTTPS Streaming
CVR PostgreSQL
![Page 91: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/91.jpg)
Dropcam on Amazon DynamoDB
• camera_records – metadata about cameras
• cuepoints – metadata about activity
• recording_sessions – metadata about CVR data in S3
• user_sessions – session data for logged-in user
case class CameraRecord(
cameraId: Int, // hash key
ownerId: Int,
subscribers: Set[Int],
hoursOfRecording: Int,
...
)
case class Cuepoint(
cameraId: Int, // hash key
timestamp: Long, // range key
type: String,
...
)
![Page 92: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/92.jpg)
Example: Cuepoints • Activity detected PutItem
– camera_id, timestamp, type (motion, sound, etc.), other data…
• Activity recognized UpdateItem – category_id
• Available via API Query – Get all for camera_id, or
– Get all for camera_id BETWEEN start_time and end_time
• Expire after 7 days (or 30 days) BatchWriteItem (delete) – Periodically, per camera: query where timestamp < now - 7 days, then delete
– Spikes chew up IOPS self-throttling
– Another approach: table per week, DeleteTable after 5 weeks
Cuepoints
![Page 93: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/93.jpg)
...Not That Simple
• “Eventual consistency”
• Choosing hash key == sharding – Important up-front design decision
– Ensure uniform access over your key space!
• IOs are front-and-center – Actually, even harder: IOs Per Second
• Throttling behavior is opaque – Actively watch for throttled responses
• No built-in expiration
![Page 94: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/94.jpg)
Less is More
• The right set of tradeoffs
• Managed NoSQL – Focus on our own software and product
• Scales easily with our business
• High availability
• Very fast (SSDs)
• Predictable performance
• Inexpensive
![Page 96: Amazon DynamoDB Design Patterns for Ultra-High Performance Apps (DAT304) | AWS re:Invent 2013](https://reader034.vdocument.in/reader034/viewer/2022051412/54c666c44a7959342b8b4592/html5/thumbnails/96.jpg)
Please give us your feedback on this
presentation
As a thank you, we will select prize
winners daily for completed surveys!
DAT304