Sadayuki Furuhashi

Kumofs

MessagePackProject

#msgpack　@frsyuki

Distributed key-value store

and e

About me:

Sadayuki Furuhashi (@frsyki)

Diskless network systemVIVER

kumofsDistributed key-value store

MessagePackCross-language communication library

University of Tsukuba

Master’s cource in Computer Science

supported by “Mitoh” project run by IPA, Information-technology Promotion Agancy, Japan

Projects:

About me:

Sadayuki Furuhashi

Diskless network systemVIVER

kumofsDistributed key-value store

MessagePackCross-language communication library

University of Tsukuba

Master’s cource in Computer Science

supported by “Mitoh” project run by IPA, Information-technology Promotion Agancy, Japan

Projects:

kumofs

kumo = 雲 = cloud

What’s kumofs?

• Distributed key-value store

• Get, Set, Delete and CAS (Compare-And-Swap)

• Linear scalability as servers added

• both read and write throughput scales-out

• dynamic rebalancing without any impacts on applications

• Optimized for read-latency

• always zero-hop

• No SPOF

http://itpro.nikkeibp.co.jp/article/COLUMN/20090617/331953/

“memcachedを超える成果も、Interopで若手技術者がクラウドを支える技術を競う”

http://gihyo.jp/dev/serial/01/alpha-geek/0041“小飼弾のアルファギークに逢いたい♥”

Shows and manages hundreds of photos smoothly> gives lots of accesses on metadata servers

http://yapcasia2009.ficia.com/pl/album/7BF0E66A-A35F-11DE-966B-860A873069EA

Why kumofs?

• Need to manage heavy random reads and writes

• Need low latency for user-experience

• Need elastic scalability

• builds small cluster at the first

• enlarges it when number of users increases

• “I won’t wakeup at midnight!”

• Systems that have SPOF rushes administrators to recover crashed servers even if it’s at midnight.

Archtectureof

kumofs

Manager

Gateway

kumo-manager:Watches kumo-servers to detect/detach crashed severs

Archtecture of kumofs

kumo-server:Stores and replicates data

kumo-gateway:Relays requests from applications to kumo-servers

Server

Application

ServerManager

Gateway

Manager

Duplicated(HA)

Server

Server

Server

Server

Server

Application

GatewayApplication

Gateway

Replication

Archtecture of kumofs Tokyo CabinetTokyo Cabinet

hash(key1)

Server A

Server B

Server C

Server D

Consistent Hashing

Server A

Server B

Server C

Server D

coordinated by Server B

Server A

Server B

Server C

Server D

coordinated by Server B

Server C

Server D

Server A

Server A

Server B

Server C

Server D

set

Replicate

Replication mechanism

Server A

Server B

Server C

Server D

getHigh-availability

Server A

Server B

Server C

Server D

getHigh-availability

Server A

Server B

Server C

Server D

getHigh-availability

Server A

Server B

Server C

Server D

coordinated by Server B

Server CServer D

Server A

Adding nodes dynamically

Server A

Server B

Server C

Server D

coordinated by Server B

Server C

Server A

Server E

Copies data

Adding nodes dynamically

Server A

Server B

Server C

Server D

Server E

Data are moving...

Adding nodes dynamically

Server A

Server B

Server C

Server D

Server E

setget

Adding nodes dynamically

Server A

Server B

Server C

Server D

Server E

setget

Adding nodes dynamically

getset

Consistent Hashing for Reading

Consistent Hashing for Writing

Dynamic rebalacing algorithm of kumofs（double-hash-space）

Application

Gateway

Gateway

Application

Server Server Server

memcached protocol

MessagePack-RPC

・Hides cluster configuration from applications. “memcached server on localhost”

localhost:11211 Asynchronous RPC library

MessagePack-RPC

Server Server Server

Tools

Administrator

Manager

Gets cluster configuration

Asynchronous RPC library

Tools

Written in Ruby

> automates operation tasks

> easy to implement

> easy to customize

kumoctl

kumostat

kumotop

MessagePack-RPCAsynchronous RPC libraryCross-language RPC

Tools

Written in Ruby

> automates operation tasks

> easy to implement

> easy to customize

kumoctl

kumostat

kumotop

MessagePack-RPCAsynchronous RPC libraryCross-language RPC

MessagePack

What’s MessagePack?

• Binary-based object serialization format

• fast and small

• Cross-language

• Java, C++, Ruby, Python, Erlang, Haskell, ...

• Dynamic typing like JSON, Avro, etc. (Unlike Thrift)

• type-system is compatible with JSON

• Streaming deserialization

MessagePack?

{“msgpack”:“json”, “hello”:“world”}

24% compact

MessagePack26 bytes

JSON34 bytes

“It's like JSON, but very fast and small.”

“It's like JSON, but very fast and small.”MessagePack?

Faster!

MessagePack for Java

require 'msgpack' # gem install “msgpack"

msg = [1,2,3].to_msgpack #=> "\x93\x01\x02\x03"

MessagePack.unpack(msg) #=> [1,2,3]

MessagePack?“It's like JSON, but very fast and small.”

require 'msgpack' # gem install “msgpack"

u = MessagePack::Unpacker.new($stdin)

u.each do |obj|

puts obj.to_json

end

MessagePack?“It's like JSON, but very fast and small.”

MessagePack?Annotation + dynamic code generation in Java

@MessagePackMessagepublic static class MyClass { public String str; public double num;

@MessagePackOptional public int flag = 0;}

MessagePack

• C++> Sadayuki Furuhashi

• Ruby> Sadayuki Furuhashi

• Java> Muga Nishizawa

• Python> INADA Naoki

• Haskell> Hideyuki Tanaka

• Lua> Nobuyuki Kubota

• Perl> tokuhirom, gfx, ...

• Erlang> UENISHI Kota

• Node.JS> Peter Griess

• JavaScript> uupaa

• PHP> advect

• D> repeatedly

Authors

MessagePack-RPC?

Rails app

C++ serverJava app

MessagePack-RPC

“Cross-language messaging library”

What’s MessagePack-RPC?

• Cross-language messaging library

• Java, C++, Ruby, Python, Erlang, Haskell, ...

• Simple protocol

• 3 messages: Request, Response and Notify

What’s MessagePack-RPC?

• Asynchronous RPC with parallel pipelining

• Concurrent communication over multiple servers

• Connection pooling

• omits overheads of reconnecting

• Concept of Future

• hides complicated event-driven I/O

• Concept of Session

• hides transport connections

• Dynamic typing

ProtocolMessagePack-RPC protocol

[0, msgid, method, params]

[1, msgid, error, result]

[2, method, params]

Request

Response

Notify

Message ID→Parallel Pipelining

Pipelining

Client Server PipeliningSends one request after another before receiving responses to speed-up.

Examples:・HTTP/1.1 pipelining・Thrift・SOAP

Client Server

Pipelining

“heavy” task“light” task

Examples:・HTTP/1.1 pipelining・Thrift・SOAP

“Heavy” task may keep “light” tasks waiting

> makes parallelism worse

must besorted

Client Server

Parallel pipelining

Example:・MessagePack-RPC

MessagePack-RPC doesn’t require responses to be sorted.

> Returns “light” tasks faster

Faster

“heavy” task“light” task

FutureClient Server1 Server2

call_async(method1)

call_async(method2)

future1

future2

Event-driven I/OServer-side event-driven I/O

Dispatcher

ServerClient

Client

Loop

multi-threaded event-driven I/O(C++, Java)

Servers deals with many clients efficiently.

Event-driven I/O

Client

Session Loop Server

Client

ServerSession Loop

Client-side event-driven I/O

Event-driven I/O

sharedevent loop

Client

Client

Server

Server

Loop

Session

Session

Client-side event-driven I/O

communications concurrently

Event-driven I/O

Session PoolServer

Server

pools these connectionsLoop

Session

Session

connection

Client-side event-driven I/O

Dispatcher

Event-driven I/O

sharedevent loop

Client

Server

Server

Loop

Session

Session

Client-side event-driven I/O

Client

sharedevent looptimer, signal handler,

other protocol, etc...

Dispatcher

Event-driven I/O

Client

Server

Loop

Session

Client-side event-driven I/O

Client

MessagePack-RPC

• C++ - Sadayuki Furuhashi• Erlang - UENISHI Kota• Haskell - Hideyuki Tanaka• Java - Muga Nishizawa• PHP - h0x10• Python - INADA Naoki• Ruby - Sadayuki Furuhashi

Authors

Users

• Ameba Now

• Microblog service; 18 million posts per month

• Apache Solr + MessagePack for realtime search

• Sedue

• Commercial distributed search engine

http://sedue.sc/Sedue Cloud Search: 簡単に使える高性能検索エンジン・レコメンドエンジンクラウドサービス

Conclusion

• Kumofs - Distributed key-value store> Optimized for low latency always zero-hop> No SPOF> Scales-out without any impacts on applications Gateway hides cluster configuration Consistency control algorithm

Conclusion

• MessagePack> Cross-language object serialization library> Supports optional fields

• MessagePack-RPC> Cross-language messaging library> Server/client-side event-driven I/O> Asynchronous RPC with parallel pipelining> Concept of Future and Session> Connection pooling

http://msgpack.org/

http://kumofs.sourceforge.net/