tech talk: rocksdb slides by dhruba borthakur & haobo xu of facebook

The Story of RocksDB

Dhruba Borthakur & Haobo XuDatabase Engineering@Facebook

Embedded Key-Value Store for Flash and RAM

Monday, December 9, 13

A Client-Server Architecture with disks

Network roundtrip = 50 micro sec

Disk access = 10 milli seconds

Application ServerDatabase

Server

Locally attached Disks

Client-Server Architecture with fast storage

Network roundtrip = 50 micro secApplication Server

Database Server

Latency dominated by network

SSD RAM

100 microsecs

100 nanosecs

Architecture of an Embedded Database

Application Server

Database Server

Application Server

Database Server

Application Server

Database Server

SSD RAM

100 microsecs

100 nanosecs

Application Server

Database Server

Storage attached directly to application servers

SSD RAM

100 microsecs

100 nanosecs

Any pre-existing embedded databases?

Open Source

FB Proprietary

1. Berkeley DB2. SQLite3. Kyoto TreeDB4. LevelDB

Open Source

FB Proprietary

Key-value stores

1. High Performant2. No transaction log3. Fixed size keys

Open Source

FB Proprietary

Key-value stores

Comparison of open source databases

Random Reads

Random Writes

Random Reads

LevelDBKyoto TreeDBSQLite3

129,000 ops/sec151,000 ops/sec134,000 ops/sec

Random Writes

Random Reads

129,000 ops/sec151,000 ops/sec134,000 ops/sec

164,000 ops/sec88,500 ops/sec

9,860 ops/sec

Random Writes

HBase and HDFS (in April 2012)

Details of this experiment: http://hadoopblog.blogspot.com/2012/05/hadoop-and-solid-state-drives.html

Random Reads

HDFS (1 node)HBase (1 node)

93,000 ops/sec35,000 ops/sec

Log Structured Merge Architecture

Read Write data in RAM

Write Request from Application

Transaction log

Transaction log Read Only data in RAM on

Periodic Compaction

Write Request from ApplicationScan Request from Application

Periodic Compaction

Leveldb has low write ratesFacebook Application 1:

• Write rate 2 MB/sec only per machine

• Only one cpu was used

Leveldb has low write ratesFacebook Application 1:

• Write rate 2 MB/sec only per machine

• Only one cpu was used

We developed multithreaded compaction

10ximprovement on

write rate

100%of cpus are

in use+Monday, December 9, 13

Leveldb has stallsFacebook Feed:

• P99 latencies were tens of seconds

• Single-threaded compaction

Leveldb has stallsFacebook Feed:

• P99 latencies were tens of seconds

• Single-threaded compaction

We implemented thread aware compaction

Dedicated thread(s) to flush memtable Pipelined memtables

P99 reduced to less than a second

Leveldb has high write amplification•Facebook Application 2:

• Level Style Compaction

• Write amplification of 70 very high

Leveldb has high write amplification•Facebook Application 2:

• Level Style Compaction

• Write amplification of 70 very high

Two compactions by LevelDB Style Compaction

5 bytes

6 bytes

10 bytes

Level-0

Level-1

Level-2

Stage 1

11 bytes

10 bytes

Stage 2

10 bytes

Stage 3

Our solution: lower write amplification•Facebook Application 2:

• We implemented Universal Style Compaction

• Start from newest file, include next file in candidate set if

• Candidate set size >= size of next file

Our solution: lower write amplification•Facebook Application 2:

• We implemented Universal Style Compaction

• Start from newest file, include next file in candidate set if

• Candidate set size >= size of next file Single compaction by Universal Style Compaction

5bytes

6 bytes

10 bytes

Level-0

Level-1

Level-2

Stage 1

10 bytes

Stage 2

Write amplification reduced to <10Monday, December 9, 13

Leveldb has high read amplification

•Secondary Index Service:

•Leveldb does not use blooms for scans

•Secondary Index Service:

•Leveldb does not use blooms for scans

•We implemented prefix scans

• Range scans within same key prefix

• Blooms created for prefix

• Reduces read amplification

Leveldb: read modify write = 2X IOs

•Counter increments

• Get value, value++, Put value

• Leveldb uses 2X IOPS

•Counter increments

• Get value, value++, Put value

• Leveldb uses 2X IOPS

•We implemented MergeRecord

• Put “++” operation in MergeRecord

• Background compaction merges all MergeRecords

• Uses only 1X IOPS

Leveldb has a Rigid Design

•LevelDB Design

• Cannot tune system, fixed file sizes

•LevelDB Design

• Cannot tune system, fixed file sizes

•We wanted a pluggable architecture

• Pluggable compaction filter, e.g. TimeToLive

• Pluggable memtable/sstable for RAM/Flash

• Pluggable Compaction Algorithm

The Changes we did to LevelDB

Inherited from LevelDB

• Log Structured Merge DB

• Gets/Puts/Scans of keys

• Forward and Reverse Iteration

Inherited from LevelDB

• Log Structured Merge DB

• Gets/Puts/Scans of keys

• Forward and Reverse Iteration

RocksDB

• 10X higher write rate

• Fewer stalls

• 7x lower write amplification

• Blooms for range scans

• Ability to avoid read-modify-write

• Optimizations for flash or RAM

• And many more…

RocksDB is born!

•Key-Value persistent store

•Embedded

•Optimized for fast storage

•Server workloads

RocksDB is born!

•Key-Value persistent store

•Embedded

•Optimized for fast storage

•Server workloads

What is it not?

•Not distributed

•No failover

•Not highly-available, if machine dies you lose your data

What is it not?

•Not distributed

•No failover

•Not highly-available, if machine dies you lose your data

RocksDB API▪ Keys and values are arbitrary byte arrays.

▪ Data is stored sorted by key.

▪ The basic operations are Put(key,value), Get(key), Delete(key) and Merge(key, delta)

▪ Forward and backward iteration is supported over the data.

RocksDB Architecture

FlushCompaction

Active MemTable

ReadOnlyMemTable

sst sst

Switch Switch

Write Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

sst sst

Switch Switch

Write Request

Read Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

sst sst

Switch Switch

Write Request

Read Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

sst sst

Switch

Memory

Switch

Write Request

Read Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

sst sst

Switch

Memory Persistent Storage

Switch

Log Structured Merge Tree -- Writes

▪ Log Structured Merge Tree

▪ New Puts are written to memory and optionally to transaction log

▪ Also can specify log write sync option for each individual write

▪ We say RocksDB is optimized for writes, what does this mean?

RocksDB Write Path

FlushCompaction

Active MemTable

ReadOnlyMemTable

sst sst

Switch Switch

RocksDB Write Path

Write Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

sst sst

Switch Switch

Log Structured Merge Tree -- Reads

▪ Data could be in memory or on disk

▪ Consult multiple files to find the latest instance of the key

▪ Use bloom filters to reduce IO

RocksDB Read Path

FlushCompaction

Active MemTable

ReadOnlyMemTable

sst sst

Blooms

RocksDB Read Path

Read Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

sst sst

Blooms

RocksDB Read Path

Read Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

sst sst

Memory

Blooms

RocksDB Read Path

Read Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

sst sst

Blooms

RocksDB: Open & Pluggable

Pluggable Memtable

format in RAM

Blooms

CustomizableWAL

Pluggable Memtable

format in RAM

Blooms

CustomizableWAL

Pluggable Memtable

format in RAM

Blooms

CustomizableWAL

Pluggable Memtable

format in RAM

Blooms

CustomizableWAL

Pluggable Memtable

format in RAM

Transaction log

Blooms

CustomizableWAL

Pluggable Memtable

format in RAM

Transaction log

Blooms

CustomizableWAL

Pluggable Memtable

format in RAM

Transaction log

Blooms

CustomizableWAL

Pluggable Memtable

format in RAM

Transaction log Pluggable sst data format on storage

Blooms

CustomizableWAL

Pluggable Memtable

format in RAM

Pluggable Compaction

Blooms

CustomizableWAL

Pluggable Memtable

format in RAM

Write Request from ApplicationGet or Scan Request from Application

Pluggable Compaction

Blooms

CustomizableWAL

Example: Customizable WALogging

•In-house Replication solution wants to be able to embed arbitrary blob in the rocksdb WAL stream for log annotation

•Use Case: Indicate where a log record came from in multi-master replication

•Solution: A Put that only speaks to the log

Active MemTable

Replication Layer In one write batch:

PutLogData(“I came from Mars”)Put(k1,v1)

k1 v1 “I came from Mars”

Write Request

Active MemTable

Replication Layer In one write batch:

PutLogData(“I came from Mars”)Put(k1,v1)

k1 v1 “I came from Mars”

Example: Pluggable SST format

•One Facebook use case needs extreme fast response but could tolerate some loss of durability

•Quick hack: mount sst in tmpfs

•Still not performant:

• existing sst format is block based

•Solution: A much simpler format that just stores sorted key/value pairs sequentially

• no blocks, no caching, mmap the whole file

• build efficient lookup index on load

Example: Blooms for MemTable

•Same use case, after we optimized sst access, memtable lookup becomes a major cost in query

•Problem: Get needs to go through the memtable lookups that eventually return no data

•Solution: Just add a bloom filter to memtable!

RocksDB Read Path

FlushCompaction

Active MemTable

ReadOnlyMemTable

sst sst

Blooms

RocksDB Read Path

Read Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

sst sst

Blooms

RocksDB Read Path

Read Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

sst sst

Memory

Blooms

RocksDB Read Path

Read Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

sst sst

Blooms

Example: Pluggable memtable format

•Another Facebook use case has a distinct load phase where no query is issued.

•Problem: write throughput is limited by single writer thread

•Solution: A new memtable representation that does not keep keys sorted

Sort, Flush

Compaction

Unsorted MemTable

ReadOnlyMemTable

sst sst

Switch Switch

Write Request

Sort, Flush

Compaction

Unsorted MemTable

ReadOnlyMemTable

sst sst

Switch Switch

Write Request

Read Request

Sort, Flush

Compaction

Unsorted MemTable

ReadOnlyMemTable

sst sst

Switch Switch

Write Request

Read Request

Sort, Flush

Compaction

Unsorted MemTable

ReadOnlyMemTable

sst sst

Switch

Memory

Switch

Write Request

Read Request

Sort, Flush

Compaction

Unsorted MemTable

ReadOnlyMemTable

sst sst

Switch

Possible workloads for RocksDB?

▪ Serving data to users via a website

▪ A spam detection backend that needs fast access to data

▪ A graph search query that needs to scan a dataset in realtime

▪ Distributed Configuration Management Systems

▪ Fast serve of Hive Data

▪ A Queue that needs a high rate of inserts and deletes

Futures

•Scale linearly with number of cpus

• 32, 64 or higher core machines

• ARM processors

Futures

•Scale linearly with number of cpus

• 32, 64 or higher core machines

• ARM processors

•Scale linearly with storage iops

• Striped flash cards

• RAM & NVRAM storage

Come Hack with us

•RocksDB is Open Sourced

• http://rocksdb.org

• Developers group https://www.facebook.com/groups/rocksdb.dev/

Come Hack with us

•RocksDB is Open Sourced

• http://rocksdb.org

• Developers group https://www.facebook.com/groups/rocksdb.dev/

•Help us HACK RocksDB

tech talk: rocksdb slides by dhruba borthakur & haobo xu of facebook

ramtransaction log

applicationtransaction

clientserver architecture

rates facebook application

facebook feed

random reads hdfs

random readsdetails

high level

Technology

database in rust building a distributed...

rocksdb storage engine for mysql and mongodb

incentives presented by namami borthakur & k. gayatri

optimizing rocksdb for open-channel ssds

replicated rocksdb at pinterest @scale 2016 san jose

myrocks introduction - percona · engine=rocksdb is a...

pacman:coordinatedmemorycachingforparalleljobs ·...

matei zaharia , dhruba borthakur * , joydeep sen sarma * ,

rocksdb on open-channel ssds - · pdf filerocksdb on...

apache hadoop goes realtime at facebook borthakur, sarma

tokudb vs rocksdb - percona vs rocksdb.pdf · tokudb vs...

dissertation - university of virginia school of...

changpeng liu, intel xiaodong liu, intel jin yu, intel ·...

02-lixiaoyan performance tuning in bluestore&rocksdb

the hive think tank: rocking the database world with rocksdb

auto-tuning rocksdb by machine learning

rocksdb: evolution of development priorities in a key

assam public service commission - apsc.nic.in...

03 bluestore rocksdb optimization ceph v2 by...

fine-tuning rocksdb for nvme ssd