ZooKeeper: A Distributed Coordi-nation Service for Distributed Ap-

plications

Motivation• Large-scale distributed application

require different forms of coordina-tion:– Configuration– Group membership and leader election– Synchronization

Related Works• Amazon Simple Queue Service-queu-

ing• [25]-leader election• [27]-configuration• Chubby[6]-locking service with

strong synchronization

Zookeeper• Zookeeper is a distributed service for

distributed applications. It support:– Synchronization– Configuration management– Naming service

Why Do We Need Zookeeper• Zookeeper is simple• Zookeeper is replicated• Zookeeper is ordered• Zookeeper is fast

Data Model

• Regular Znode• Ephemeral Znode• Sequential flag

Watches

Zookeeper

Client1

Client4

crea

te/ex

ist/

(WAT

CH)

setDataNOTI

FICA

TION

Client API• create(path, data, flags)• delete(path, version)• exists(path, watch)• getData(path, watch)• setData(path, data, version)• getChildren(path, watch)• sync(path)

Zookeeper Service Architec-ture

• Read request is handled by local server• Write request is sent to the leader, the leader

broadcasts the change to the Zookeeper through Zab an atomic broadcast protocol.

Setup Zookeeper• Download:

http://www.apache.org/dyn/closer.cgi/zookeeper/

• Configure Zookeeper:– Standalone Mode– Replicated Mode

Standalone Mode• create file zoo.cfg with the content:

• Start server: bin/zkServer.sh start• Test with zookeeper client: bin/zk-

Cli.sh -server 127.0.0.1:2181

tickTime=2000 dataDir=/var/lib/zookeeper clientPort=2181

Standalone Mode (2)• ls /• get• Set

• …

Setup Zookeeper: Replicated mode

• Every server has the same configura-tion file.

• Create file named myid In the datadir directory. The content of myid file is an unique number. tickTime=2000

dataDir=/home/sdn/zookeeperclientPort=2181initLimit=5syncLimit=2server.1=192.168.0.94:2888:3888server.2=192.168.0.59:2888:3888…server.n=192.168.0.59:2888:3888

Use cases• Naming service• Configuration management• Synchronization• Message Queue• Notification system

Synchronization: Simple Lock• Client has smallest number have permission to access locked object• When the client finishes work with object, child node is deleted.

Another client has smallest number have permission to access locked object

App-Root

P(n)

P(n+1)

P(n+2)

P(n+3)

Zookeeper

Client1 Client

2Client

3

Client4

N=crea

te(P,

EPHEM

ERAL

|SEQ

UEN-

TIAL)

N=

crea

te(P

, EP

HEM

ERAL

|SEQ

UEN

-TI

AL)

N=

create(P,

EPHEM

ERAL|SEQUEN

-

TIAL)

N=create(P,

EPHEMERAL|SEQUEN-

TIAL)

Synchronization: Simple LockCheck existing of approot and create it.

Create child node with Sequential and Ephemeral flag And receive a number.

When receive the notification, check whether the number is smallest of child nodes number

Synchronization: Barrierpublic class SimpleLock implements Watcher

ZooKeeper zooKeeper = new ZooKeeper("192.168.0.94:2181", 3000, this);Stat res = zooKeeper.exists(root, true);if(res==null) String abc = zooKeeper.create(root, null, Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);

String createRes = zooKeeper.create(root+"/", null, Ids.OPEN_ACL_UNSAFE, CreateMod-e.EPHEMERAL_SEQUENTIAL);int number = Integer.parseInt(createRes.substring(root.length()+1));

while (true) {synchronized (root) {

//TODO check whether the number is smallest if(smallest) {dost();}else{root.wait();} }}

public void process(WatchedEvent event) { //Watcher eventsynchronized (root) {root.notify();}

}

private void dost(){System.out.println("Access System.out at "+System.currentTimeMillis());Thread.sleep(5000);

}

Synchronization: Simple Lock

public static void main(String[] args){new SimpleLock();new SimpleLock();new SimpleLock();new SimpleLock();

}

/simpleLock/0000000009:9Access System.out at 1416028020742/simpleLock/0000000010:10Access System.out at 1416028025765/simpleLock/0000000011:11Access System.out at 1416028030779/simpleLock/0000000012:12Access System.out at 1416028035806

Synchronization: Barrier• Every client creates child node of approot.• Whenever the number of child nodes is

enough, client will start work. AppRoot

P(n)

P(n+1)

P(n+2)

P(n+3)

Zookeeper

Client1 Client

2Client

3

Client4

N=crea

te(P,

EPHEM

ERAL

|SEQ

UEN-

TIAL)

N=

crea

te(P

, EP

HEM

ERAL

|SEQ

UEN

-TI

AL)

N=

create(P,

EPHEM

ERAL|SEQUEN

-

TIAL)

N=create(P,

EPHEMERAL|SEQUEN-

TIAL)

Synchronization: BarrierCheck existing of approot and create it.

Create child node with Sequential and Ephemeral flag And receive a number.

When receive the notification, check whether the number is smallest of child nodes number

Synchronization: Barrierpublic static void main(String[] args){

new Barrier(3).run();}

Run 3 instances:

/barrier/0000000022Starting at 1416048008944

/barrier/0000000023Starting at 1416048008947

/barrier/0000000024Starting at 1416048008948

Message Queue• Every client creates child node of approot.• Whenever the number of child nodes is

enough, client will start work.AppRoot

P(n)

P(n+1)

P(n+2)

P(n+3)

Zookeeper

sender1

sender2

re-ceiver

N=create(P, SEQUEN-TIAL)

N=create(P, SEQUEN-

TIAL)

get(smallestP)

Message Queuepublic class MessageQueue implements Watcherpublic static class Sender extends MessageQueue implements Runnablepublic static class Receiver extends MessageQueue implements Runnable

public void sendMessage() { //Watcher eventString sendMessage = "sendMessage at "+System.currentTimeMillis();zooKeeper.create(root+"/", sendMessage.getBytes(), Ids.OPEN_ACL_UNSAFE, Create-

Mode.PERSISTENT_SEQUENTIAL);}

public void process(WatchedEvent event) { //Watcher eventsynchronized (root) {root.notify();}

}

while (true) {synchronized (root) { List<String> childs = zooKeeper.getChildren(root, true); if(childs.size()==0){root.wait();}else{ //TODO sort the list

for (String child : childs) { byte[] data = zooKeeper.getData(root+"/"+child, false, new Stat()); System.out.println("readMessage:"+new String(data)); zooKeeper.delete(root+"/"+child, 0);} } }}

Message Queue:

Run 2 sender instances and 1 receiver instance:

readMessage:sendMessage at 1416066118908readMessage:sendMessage at 1416066118916readMessage:sendMessage at 1416066118926readMessage:sendMessage at 1416066118929readMessage:sendMessage at 1416066118939readMessage:sendMessage at 1416066118942readMessage:sendMessage at 1416066118952readMessage:sendMessage at 1416066118954readMessage:sendMessage at 1416066118964readMessage:sendMessage at 1416066118967readMessage:sendMessage at 1416066118976readMessage:sendMessage at 1416066118984readMessage:sendMessage at 1416066118989readMessage:sendMessage at 1416066118997readMessage:sendMessage at 1416066119001readMessage:sendMessage at 1416066119009readMessage:sendMessage at 1416066119016