mio - a distributed skip graph based orderd kvs

Mioa distributed Skip Graph based orderd KVS

Cybozu Labs, Inc.Taro Minowa (Higepon)

Feb 26 2010 Mio - a Skip Graph based ordered KVS

Introduce myself

@higeponMona OS

http://www.monaos.org

MoshA fast Scheme interpreter

Outputzhttp://outputz.com

2

http://outputz.com

http://outputz.com


Summary

Mio is...a distributed orderd KVSmemcached + range searchSkip Graph basedWritten in Erlanghttp://github.com/higepon/mioIn alpha quality

3

http://github.com/higepon/mio





Background

4


RDBMS vs KVS

5

KVS

RDBMS

Scalability

High functionality

set/get

volatile

Transaction

SQL


RDBMS vs KVS

5

KVS

RDBMS

Scalability

High functionality

set/get

volatile

Transaction

SQL

Complement each other


Mio

6

KVS

RDBMS

Scalability

High functionality


Mio

6

KVS

RDBMS

Scalability

High functionality

Mio

+Range search


Mio

6

KVS

RDBMS

Scalability

High functionality

Mio

+Range search

Makes RDBMSlighter workload


Range search?

Querieslast 7 daysprev/nextTop 10 ranking

SQLSELECT * FROM photos WHERE date between xxx

and xx order by date limit 10

RDBMS handles these queires

7


Mio

8


The Challenges and Design Decisions

Range searchOrdered structureSkip Graphs algorithm

Scale-Outdistributed using Erlang functions

memcached compatible I/FVolatile

keep it simple

9


Skip GraphsJames Aspnes (2003)

10


Supported operations

search by keyinsert (join)removerange search by key1 and key2

11


Set of sorted doubly linked lists

12

Same as railway stationsAll keys (stations) consist doubly linked listKnows only his left and right stationKeep sorted by key

Search Shibuya start from UenoGo to left. O(n)

Shibuya Shinjuku UenoTamachi Yoyogi


Make an express lane

13

Skip some stationsUeno -> Shinjuku -> ShibuyaTamachi is placed on another express

lane


Shinjuku Ueno

Express

Local

Skip


Multiple lanes

Level 0 laneall keys are in the list

Level n (n > 0) laneexpress lanen + 1 lane is more express than n lane.

14


Level 0

Level 1

Level 2


Search

Start from highest to lower levelCan search from any stationsO(log n)

15


Level 0

Level 1

Level 2


Range Search

Search key1Collect matched on Level 0ex. Key1 = Ueno , Key2 = Shibuya

16


Level 0

Level 1

Level 2


Remove

Remove on each LevelUpdate neighbor’s linksHighest to lower

17

A B C A C

B


Insert

Insert on each LevelUpdate neighbors’s linksLowest to higher (in reverse order to remove)In which express lane is a new station insereted?

radomly locateduniform

18

A C

B

A B C


Easy to implement?

NoReally simple, but ...We should support concurrent insert/remove

If neighbor is removed when inserting?If someone inserts another to neighbor?Searching crash?

Fragile linked list We can’t find any perfect concurrent join

algorithm.

19


Our concurrent algorithm

Lock some nodesPlease read the source code :)Defined three invariants

20

A B C

A

B

C A

B

C


Implementation

21


Written in Erlang

A station(key, value) is a processgen_server processHold left/right on each levelFollow left/right = gen_server:call/2No distinction between local and remote process

Erlang is great!

Ditributed with -name optionerl -name name@FQDN

22


Performance

5000 qps on single nodereally slow on multiple nodes

need less communication between nodesneed better algorithm

23


Demo

24


Tips for practical Erlang

Max process option +PSet proper value. Don’t use MAX.

gerbage_collect()Fast enough, reduce memory usage.hibernate is slow...

refactorerlfprof on gen_server shows nothing

Use dynomite profile

25


Tips for practical Erlang

Common testCoverageload test

gen_server:call is slowUse mnesia for property access.Easy replication

Easy to runShould users run erl with many options?Shell script borrowed from RabbitMQ

26


Summary, Once more

Mio is...a distributed orderd KVSmemcached + range searchSkip Graph basedWritten in Erlanghttp://github.com/higepon/mioIn alpha quality

27



mio - a distributed skip graph based orderd kvs

Technology