scale your app for the holidays with amazon dynamodb

© 2011 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Scale Your App for the Holidayswith DynamoDB

build high scale applications in daysfast | scalable | cost-effective

David Pearson, Business Development Manager


What Is NoSQL?

CAP theorem Not Only SQL

non-relationalunstructured

distributed

horizontally scalableschema-free

eventually consistent

BASE

key-value

open-source


scale-up =bigger servers

scalability

performanceRDBMS

versatile

change application to use… read slaves read caches data shards

scale-out = more servers

infrastructure scaling+

application scaling

big data challenge:the cost of scalability


NoSQL

specialized scale-out without application changes

scalability

performanceRDBMS

versatile

infrastructure scaling=

application scaling


NoSQL

General Characteristics

scalability

complexity

RDBMS


NoSQL @ Amazon

a story in three parts…


RDBMS used for “all kinds of access patterns”

NoSQL @ Amazonkey-value access

complex queries

transactions

analyticspart one - early days…

data (re-)partitioning

bigger hardware is tempting

trade-off with consistency

versatileeasy to learneasy to query

scalability

availability


replicated DHT with consistency management

NoSQL @ Amazon

part two - dynamo…

• Consistent hashing • Optimistic replication• “Sloppy quorum”• Anti-entropy mechanisms• Object versioning

specialized technology

limited query capabilities

simpler consistency



NoSQL @ Amazon


higher availability

incremental scalability

predictable performance

eventual consistency

no need to re-architect applications

just add hardware!

simpler query model



NoSQL @ Amazon


higher availability

incremental scalability

predictable performance

no strong consistency

scaling effort required

operational complexity

much better, but…

still required developers to be administrators…

install, patch, upgrade, balance clusters


NoSQL @ Amazon

part three - Amazon DynamoDB…

Fast & Predictable Performance

Seamless Scalability

Zero Administration

nosql database service

“Even though we have years of experience with large, complex NoSQL architectures, we are happy to be finally out of the business of managing it ourselves.” – Don MacAskill, CEO


• No compromise on durability or availability for performance• Express scale needs in simpler terms, not servers and disks• Scale will be our problem and not our customers • Extremely easy to use with no administration• Provide consistently low latencies

NoSQL @ Amazon

Design Philosophy

provisioned throughput

continuous replication across multiple AZ’s

backed by SSDs

consistent, disk-only writes (not memory)


November Traffic at Amazon.com

76% = wasted hardware

Capacity needed before DynamoDB

Actual traffic


Capacity needed before DynamoDB

Actual traffic

November Traffic at Amazon.com

Capacity we can provision with DynamoDB


Case Study

Stores user objects in cloud

Queries and object searches are served by DynamoDB

“We were excited by how fast we were able to put DynamoDB into production and how muchdeveloper time we saved. In addition, DynamoDB lets us scale up and down easily by simplyreserving increased throughput capacity when we need it and dialing it back when we don’t”

- Russel Dicker, Amazon CloudDrive

metadata in DynamoDB

objects in S3

list all the objects in my drive

find all my music albums

small objects move faster!


Goal = enable customers to see Cloud Drivephotos on their Kindle

Need = low latency access to the metadata, at any scale

Result = lower total cost of ownership (tco)lower admin effort required to scale

Cloud Drive Photos

hardware costs + operational costs + opportunity cost of feature development

future tco

#customers X#photos


image thumbnailsalbum cover art

links to S3

DynamoDB access pattern

list all images for photos uploaded by a customer


Two decisions + three clicks = ready for use

Getting started with DynamoDB…


Two decisions + three clicks = ready for use

primary keys +

provisioned level of throughput (easy to change)


• Reserve throughput for each table• Set at table creation, increased and decreased via an API call

$0.01 per hour for every 10 units of Write Capacity$0.01 per hour for every 50 units of Read Capacity$1.00 per GB-month of Storage

Provisioned Throughput

... for items up to 1KB in size

=10 writesper second…

=50 strongly consistentreads per second… free tier…

100MB storage + 5 writes/sec + 10 reads/sec each month

effectively assigns each tableits own set of servers


Items are indexed by primary keysingle hash keys and composite keys

key-value access =extreme performance

hash + range


deviceid = 21EC2020-3AEA-1069-A2DD-08002B30309D total = 25.00

deviceid = 74ED9134-3FEC-9902-E8BA-19733F49779C total = 50.00

table = collection of items

hash key

item (max size 64k) =collection of attributes

attribute (key value pair)string, number, binary


userid = 100 date = 2012-10-24-09-00-10 total = 25.00

userid = 100 date = 2012-10-24-09-00-11 total = 50.00

hash key range key

mapping 1:M relations


DynamoDBAPIs CreateTable

UpdateTable

DeleteTable

DescribeTable

ListTables

PutItem

GetItem

UpdateItem

DeleteItem

BatchGetItem

BatchWriteItem

Query

Scan

manage tables

query specific items OR scan the full table

read and write items

bulk select or update


Available for hash-and-range primary key tables

Retrieve all items by hash key

Range key conditions: ==, <, >, >=, <=, begins with, between

Counts. Top and bottom n values. Paged responses

Query Patterns


Designing a Photo Store

Getting started with DynamoDB…


Photo Store

use case usage patternsdata design

enable users to upload and view photos from any device


Photo Store


upload photos

view photos by time


Photo Store


table photos

hash userid

range timestamp +

photoid

attributes location,

resolution,

comments, tags, s3link

data model is optimizedfor retrieval performance

tags enable search integration

s3link points to raw files


Photo Store


table photos

hash userid

range timestamp +

photoid

attributes location,

resolution,

comments, tags, s3link

Potential Enhancements…• Albums• Search integration (CloudSearch)• Global caching and delivery of

media from S3 (CloudFront)


Enhancing relational DB performance… and cost

Extreme scalability…


Infrastructure Scaling

Read Scaling #1

Read Scaling #2 (hot keys)

Write Scaling

Scaling Relational DB’s

bigger hardware (scaling up)

read replicas (slaves)

read caching (memcached)

data sharding


Infrastructure Scaling

Read Scaling #1

Read Scaling #2 (hot keys)

Write Scaling

Scaling Relational DB’s

application changes required

bigger hardware (scaling up)

read replicas (slaves)

read caching (memcached)

data sharding


Scaling Relational DB’s with DynamoDB

Identify Design the newImplement code

candidates DynamoDB table changes in app

move simple, high-scale workloads

Tables with high transaction volume (esp writes)

Primary key-only is preferred (one non-PK index ok)

No dependencies (FK’s, triggers, procedures)

hopefully, forthe last time

maximize access performance


DynamoDB Scalability @ HalfBrick Studios

• Fruit Ninja Frenzy (facebook)

• Moved game data into DynamoDB

• Grew from 1 million to 8 million active monthly users in two weeks

8M

1M

“it’s really tough to quickly scale a normal database

system to handle that kind of rapid increase in load”

- Glen Arrowsmith, Systems Architect


DynamoDB Scalability

“Creating a table that can serve 100,000 writes/second is no more

work than creating a table that can serve 10 writes/second”

Werner Vogels, Amazon CTO


Super Bowl promotion

Millions of interactions over a relatively short period

Built the app in 3 days – from design to production-ready

Weatherbug app – lightning detection & alerting for 6M phones

Extending to 40M users/month

Developed and tested in weeks

“1/20th of the cost of the traditional DB approach”


DynamoDBFast

Scalable

Cost-Effective

• minimal development time and effort

• consistently low latency

• effortless scaling to meet workload demand

• Reduces DB costs and increases reliability

• Free tier reduces initial development costs


• Building Applications with DynamoDB (Matt Wood) google search = youtube "building applications with dynamodb"http://www.youtube.com/watch?v=4jZthAFKAE4http://www.slideshare.net/AmazonWebServices/building-applications-with-dynamodb

• From the Super Bowl to WeatherBug (Werner Vogels)http://www.allthingsdistributed.com/2012/06/amazon-dynamodb-growth.html

Recommended Resources

http://www.youtube.com/watch?v=4jZthAFKAE4

http://www.slideshare.net/AmazonWebServices/building-applications-with-dynamodb

http://www.allthingsdistributed.com/2012/06/amazon-dynamodb-growth.html


Questions?

aws.amazon.com/dynamodb

scale your app for the holidays with amazon dynamodb

Documents

express consent of

dynamoreplicated dht

versatile scalability

scalability big data

scalable keyvalue

keyvalue accessnosql

caches data shards

learnavailability easy