martin slides
TRANSCRIPT
-
8/3/2019 Martin Slides
1/27
Martin Kaufmann ETH ZrichMay 2nd, 2009 1
Building aDatabase on S3
Martin Kaufmann
Hot Topics in Data Management SystemsSeminar 2009
-
8/3/2019 Martin Slides
2/27
May 2nd, 2009 Martin Kaufmann ETH Zrich 2
Motivation
Database
-
8/3/2019 Martin Slides
3/27
May 2nd, 2009 Martin Kaufmann ETH Zrich 3
Motivation
Database
How can we do better?
-
8/3/2019 Martin Slides
4/27
May 2nd, 2009 Martin Kaufmann ETH Zrich 4
Motivation
Database
-
8/3/2019 Martin Slides
5/27
May 2nd, 2009 Martin Kaufmann ETH Zrich 5
Goal
Build a database system in the cloud Preserve advantages of utility computing service Availability
Scalability
Same characteristics as general-purpose database system Support for small objects and frequent updates
ACID properties
Is that possible?
-
8/3/2019 Martin Slides
6/27
May 2nd, 2009 Martin Kaufmann ETH Zrich 6
Overview
Motivation What is Amazon AWS? Amazon S3
Amazon SQS
Building a database on S3 CAP Theorem
Using S3 as a disk
Basic commit protocols (example why needed)
Evaluation of the paper
Related work Amazon SimpleDB
Conclusion
-
8/3/2019 Martin Slides
7/27May 2nd, 2009 Martin Kaufmann ETH Zrich 7
Amazon S3 (Simple Storage Service)
S3 is a distributed storage based on (key-value) pairs Possible to save objects between 1 byte and 5 GB
Advantages Infinite storage
Perfect scalability
~ 100% availability
Drawbacks Slow compared to local disk
Eventual consistency only
Can be quite expensive
Client
S3
(key, value)
-
8/3/2019 Martin Slides
8/27May 2nd, 2009 Martin Kaufmann ETH Zrich 8
Amazon S3 in more detail
Objects are organized in buckets Objects and buckets are identified by a URI
Access by means of a simple Web Service API
Most important operationsput(uri, bytestream)
get(uri)
get-if-modified-since(uri, timestamp)
Object 1
Object n
Bucket
k
S3
-
8/3/2019 Martin Slides
9/27May 2nd, 2009 Martin Kaufmann ETH Zrich 9
Amazon SQS (Simple Queueing Service)
SQS provides queues for storing messages
Message size to 256 KB, infinite number, always available
Each queue is identified by a URI, each message by an ID
Most important operationscreateQueue(uri)
sendMessage(uri, msg)
receiveMessage(uri, number-of-msg, timeout)
deleteMessage(uri, msg-id)
Queue k
Message 1
Message 2
...
Message nSQS
Client
send()
receive()
-
8/3/2019 Martin Slides
10/27May 2nd, 2009 Martin Kaufmann ETH Zrich 10
Overview
Motivation
What is Amazon AWS? Amazon S3
Amazon SQS
Building a database on S3 CAP Theorem
Using S3 as a disk
Basic commit protocols (example why needed)
Evaluation of the paper
Related work Amazon SimpleDB
Conclusion
-
8/3/2019 Martin Slides
11/27May 2nd, 2009 Martin Kaufmann ETH Zrich 11
CAP Theorem
Three goals: Availability, Consistency, Partition Tolerance
Eric Brewer`s CAP Theorem (2000)
Can only have 2 of them at a time in a distributed system!
Availability
ConsistencyPartition
Tolerance
RelationalDB
AmazonAWS
-
8/3/2019 Martin Slides
12/27May 2nd, 2009 Martin Kaufmann ETH Zrich 12
Using S3 as a disk
Observation: S3 has similary properties like a disk
Can`t we just exchange the disk with S3?
Can we store each tuple directly as an object in S3?
Database Database
S3
-
8/3/2019 Martin Slides
13/27May 2nd, 2009 Martin Kaufmann ETH Zrich 13
Let`s do some experiments
Store objects of different size on S3
Result: S3 very slow compared to disk
S3 needs a minimum of 100 ms
per request (for every size)
Problems This is too slow!
No consistency guarantee!0,001 s
0,01 s
0,1 s
1 s
10 s
10 KB 100 KB 1000 KB
Amazon S3
Harddisk
Conclusion S3 superior to local disk in terms of throughput
Need to read data in chunks larger than 100 KB,
cluster several records (= tuples) into pages
Need to take care of concurrent access
Comparison: S3 and harddisk with 7200rpm, 10ms
-
8/3/2019 Martin Slides
14/27May 2nd, 2009 Martin Kaufmann ETH Zrich 14
Architecture
Application Layer
Database client
Management Layer
Record Manager
Page Manager
Storage Layer
Amazon S3
Pages
Application Layer
Database client
Management Layer
Record Manager
Page Manager
Client 1: Client n:
Records Records
ManagementLayer
-
8/3/2019 Martin Slides
15/27May 2nd, 2009 Martin Kaufmann ETH Zrich 15
Record Manager
Record Relational tuple Record contains key and payload
Key is unique in one collection
Collection Relational table: Each record belongs to a collection
A collection is stored as an S3 bucket
Functions of the Record Manager Create new objects, read and objects, scan collections, commit/abort
Management Layer
Record Manager
Page Manager
Page i
Record a
S3
Record b
Bucketk
Page j
Record c
-
8/3/2019 Martin Slides
16/27May 2nd, 2009 Martin Kaufmann ETH Zrich 16
Page Manager
Page Single object in S3
Buffer pool for S3 pages
After commit All updates propagated to S3
Page is kept in buffer
After abort Delete modified pages from buffer pool
Management Layer
Record Manager
Page Manager
Page i
Record a
S3
Record b
Bucket
kPage j
Record c
-
8/3/2019 Martin Slides
17/27May 2nd, 2009 Martin Kaufmann ETH Zrich 17
Are we finished now?
Looks nice, BUT what happens if
two clients update different records in the same page?
Page i
Record a
S3
Record b
Bucketk
Page iRecord c
Client 1
Client 2
-
8/3/2019 Martin Slides
18/27May 2nd, 2009 Martin Kaufmann ETH Zrich 18
Basic commit protocols
Pending update (PU) queue for every page
Queues are implemented on Amazon SQS
Clients submit committed log records to PU queue
Checkpointing: PU queue written to S3 regularly
PU Queue iUpdate 1
Update 2
...
Update n
Client a
updateS3
Page i
checkpoint
Client b
update
-
8/3/2019 Martin Slides
19/27May 2nd, 2009 Martin Kaufmann ETH Zrich 19
Are we finished NOW ?
Checkpointing is done by clients!
What happens if two clients carry out a checkpoint on thesame page?
We have to introduce a lock queue for every PU queue.
Lock Queue i
Token
PU Queue i
Update 1
Update 2
...
Update n
Client S3
checkpoint
1. get token (i)
2. grant token (i)
3. execute checkpoint
-
8/3/2019 Martin Slides
20/27May 2nd, 2009 Martin Kaufmann ETH Zrich 20
Overview
Motivation
What is Amazon AWS? Amazon S3
Amazon SQS
Building a database on S3 CAP Theorem
Using S3 as a disk
Basic commit protocols (example why needed)
Evaluation of the paper
Related work Amazon SimpleDB
Conclusion
-
8/3/2019 Martin Slides
21/27May 2nd, 2009 Martin Kaufmann ETH Zrich 21
Evaluation
Strong points
Interesting idea to build a database in the cloud
Try to use textbook methods where it is possible
Working prototype
Provides database with infinite amount of storage Preserve adantages of utility computing
Weak points
Security problems not solved
Did not explore the possibility to use EC2
Did not consider aggregation functions
-
8/3/2019 Martin Slides
22/27May 2nd, 2009 Martin Kaufmann ETH Zrich 22
Overview
Motivation
What is Amazon AWS? Amazon S3
Amazon SQS
Building a database on S3 CAP Theorem
Using S3 as a disk
Basic commit protocols (example why needed)
Evaluation of the paper
Related work Amazon SimpleDB
Conclusion
-
8/3/2019 Martin Slides
23/27May 2nd, 2009 Martin Kaufmann ETH Zrich 23
Amazon SimpleDB
Architecture Domain elastic table, no support / need for schema
Item has one or many attributes, different attributes per item!
Attribute (name, {value}) pair, a name can have multiple values
Properties Indexing done atomatically
Simple, Flexible, Scalable
Similarities to DB on S3 paper
Store data on S3, no strict consistency
Differences Each tuple can have different attributes
An attribute name can have multiple values
Name
= Harry
Hobby= Quidditch
Name
= Ron
Friend
= Hermine,
Harry
Domain
ItemAttribute
-
8/3/2019 Martin Slides
24/27May 2nd, 2009 Martin Kaufmann ETH Zrich 24
Overview
Motivation
What is Amazon AWS? Amazon S3
Amazon SQS
Building a database on S3 CAP Theorem
Using S3 as a disk
Basic commit protocols (example why needed)
Evaluation of the paper
Related work Amazon SimpleDB
Conclusion
-
8/3/2019 Martin Slides
25/27
May 2nd, 2009 Martin Kaufmann ETH Zrich 25
Conclusion
Databases in the cloud have lots of advantages Infinity amount of storage
Perfect availability
Scalability
High throughput
There are also disadvantages High latency
Only eventual consistency (all solutions so far!)
If a DB in the cloud matches the needs dependson the application! Performance requirements, storage capacity,
number of concurrent users, cost,
-
8/3/2019 Martin Slides
26/27
May 2nd, 2009 Martin Kaufmann ETH Zrich 26
References
M. Brantner et al. Building a Database on S3.SIGMOD, 2008.
Eric Brewer. Towards rebust distributed systems.Principles of Distributed Systems 2000.
Amazon Web Services
http://aws.amazon.com/
-
8/3/2019 Martin Slides
27/27
The end
Any questions ?