apache zookeeper techtuesday
Post on 06-May-2015
5.719 Views
Preview:
TRANSCRIPT
Apache ZooKeeper
Andrei Savu @TechTuesday
Why use it? What to expect in the future?
Outline
Why use it?Crash CoursePractical Example
What to expect in the future (3.4.0 release)?GSoC 2010New ContribWork in Progress
Crash Course
What is ZooKeeper?
A highly available, scalable, distributed, configuration, consensus, group membership,
leader election, naming and coordination service.
What is ZooKeeper? (2)
replicated in memory tree data structuresomehow similar to a file systemno partial read / writesno renamesordered updatesstrong persistence guaranteesconditional updates (version)watches for data changesephemeral nodesgenerated file names
ZooKeeper Data Model
hierarchical namespaceeach znode has data and childrendata is read and written in its entirety
Basic ZooKeeper API
string create(path, data, acl, flags)
delete(path, expected_version)
stat set_data(path, data, expected_version)
(data, stat) get_data(path, watch)
stat exists(path, watch)
string[] get_children(path, watch)
ZooKeeper Service
Facts: 1) all servers store a copy of the data in memory 2) the leader is elected at startup 3) followers respond to clients 4) all updates go through the leader 5) responses are sent when a majority of servers have persisted the change
Practical Example
Distributed Queue (Python)
http://www.cloudera.com/blog/2009/05/building-a-distributed-concurrent-queue-with-apache-zookeeper/ http://github.com/henryr/pyzk-recipesRetry operation on ConnectionLoss:
http://github.com/andreisavu/pyzk-recipes
GSoC 2010
3 projects / 5 months
1. Monitoring & Web-based interface
Status: Committed to the trunk
1. JIRA: ZOOKEEPER-7012. Progress Tracking Wiki3. Monitoring for Ganglia, Nagios and Cacti
1. contrib / monitoring2. 'mntr' 4letter word
4. Web interface available as a Hue application1. contrib / huebrowser2. complete install instructions3. requirements: rest gateway, Hue 1.0
2. Read-Only Mode
Status: Under Review (Ready to be committed)
1. JIRA: ZOOKEEPER-7042. Progress Tracking Wiki3. Description: "When a ZooKeeper server loses contact with
over half of the other servers in an ensemble ('loses a quorum'), it stops responding to client requests. For some applications, it would be beneficial if a server still responded to read requests when the quorum is lost, but caused an error condition when a write request was attempted."
3. Failure Detector Model
Status: Under Review
1. JIRA: ZOOKEEPER-7022. Progress Tracking Wiki3. Detectors: Phi Accrual, Chen, Bertier, Fixed Heartbeat4. Why? Check the concluding remarks on the wiki. 5. Conclusion snippet: "in scenarios where we have a
changing network behavior, such in a WAN, the adaptive methods can be a good pick"
Contrib
Large Scale Pub/Sub (hedwig)
1. JIRA: ZOOKEEPER-7752. Uses ZooKeeper and BookKeeper3. Committed to the trunk4. Developed at Yahoo! Research5. Used for PNUTS cross data center replication6. http://vimeo.com/13282102
Work in Progress
only some interesting JIRAs
#834 Children for ephemerals
JIRA: ZOOKEEPER-834Allow ephemeral nodes to have children owned by the same session. Useful when publishing status information. No need to do serialization for basic data structures (hash tables)Similar to /proc in *nix systems.Examples: /agent-01/ip, /agent-01/memory, /agent-01/load
#829 /zookeeper/sessions/*
JIRA: ZOOKEEPER-829Requested by HBase developers: " we'd like the ability to forcible expire someone else's ZK session "
Plenty of bug fixes
Join the community!
Resources
http://hadoop.apache.org/zookeeper/http://wiki.apache.org/hadoop/ZooKeeper/ProjectDescriptionhttp://wiki.apache.org/hadoop/ZooKeeper/Tao
Thanks! Questions?
top related