reducing the storage overhead of main-memory oltp ... · main-memory oltp databases with hybrid...

Reducing the Storage Overhead of Main-Memory OLTP Databases with

Hybrid IndexesHuanchen Zhang

You are running out of memory

2

Buy more ?

You are running out of memory

2

0 2M 4M 6M 8M 10M

20K

60K

Transactions Executed

Thro

ughp

utTPC-C on -Store

3

Mem

ory

(GB)

Disk tuplesIn-memory tuplesIndexes

4

8

0

Memory Limit = 5GB

The better way:Use memory more efficiently

5

Indexes are LARGEBenchmark % space for index

TPC-C

Voter

Articles

58%55%34%

Hybrid Index

34%41%18%

6

Our Contributions The hybrid index architecture

The Dual-Stage Transformation

Applied to 4 index structures- B+tree- Masstree

7

- Skip List- Adaptive Radix Tree (ART)

Performance Space30 – 70%

0 2M 4M 6M 8M 10M

20K

60K


Thro

ughp

ut (t

xn/s

)Did we solve this problem?

TPC-

C on

-Sto

re

Stay tuned

8

How do hybrid indexes achievememory savings ?

9

Static

dynamic stage static stage

Hybrid Index: a dual-stage architecture

10


write merge

Inserts are batched in the dynamic stage

11


Reads search the stages in order

12


read

A Bloom filter improves read performance

13


read

write merge

Memory-efficient

Skew-aware

14

~~~~~~~~


merge


15

Compaction

Reduction

Compression

The Dynamic-to-Static Rules

16

2 4

4

1 2

a b

6 8 10

3

c

4

d

5 5

e f

5

g

6

h

7

i

8

j

9

k

10

l

11

m

12

n

Compaction: minimize # of memory blocks

17

1 2 3

a b c

4 5 6

d h

7 8 9

i j k

10 11 12

l m n

e f g

3 6 9

17

Compaction: minimize # of memory blocks

1 2 3

a b c

4 5 6

d h

7 8 9

i j k

10 11 12

l m n

e f g

3 6 9

Reduction: minimize structural overhead

18

2 4

4

1 2

a b

6 8 10

3 4

c d

5 5

e f

5 6

g h

7 8

i j

9 10

k l

11 12

m n

1 2 3

a b c

4 5 6

d h

7 8 9

i j k

10 11 12

l m n

e f g

3 6 9

18

Reduction: minimize structural overhead


merge


19


merge


19

MergeQuestions:1. Partial?2. When?3. Blocking?

??

?

0 2M 4M 6M 8M 10M

20K

60K


Thro

ughp

ut (t

xn/s

)Did we solve this problem?

TPC-

C on

-Sto

re

B+tree

20

0 2M 4M 6M 8M 10M

20K

60K


Thro

ughp

ut (t

xn/s

)

60K

20K

Yes, we improved the DBMS’s capacity!TP

C-C

on-S

tore

B+tree

Hybrid

20


Thro

ughp

ut (t

xn/s

)

20K

60K

20K

60K

Mem

ory

(GB)

0 2M 4M 6M 8M 10M

TPC

-C o

n-S

tore

4

4

8

8 B+tree

Hybrid

B+tree

Hybrid


21


Thro

ughp

ut (t

xn/s

)

20K

60K

20K

60K

Mem

ory

(GB)

0 2M 4M 6M 8M 10M

TPC

-C o

n-S

tore

4

4

8

8 B+tree

Hybrid

B+tree

Hybrid


21

TakeAway:Largerworkingsetinmemory

Higherthroughput

Memorysavedbyindexes

This is just the BEGINNING

22

Conclusions

The hybrid index architecture


Applied to 4 index structures

GENERAL

PRACTICAL

USEFUL

23

- B+tree- Masstree

- Skip List- Adaptive Radix Tree (ART)

1-844-88-CMUDB

Toll-Free Hotline:

reducing the storage overhead of main-memory oltp ... · main-memory oltp databases with hybrid...

Documents