all the little pieces
DESCRIPTION
Quick, what do memcache, MogileFS, and Gearman have in common? They are scalable, distributed technologies, and they can also interface with PHP, your ubiquitous web development language. Digg uses all 3 (and a few more) in its quest for social news domination, and this presentation shares what we’ve learned about them and how they are best utilized with PHP.TRANSCRIPT
![Page 1: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/1.jpg)
all the little piecesdistributed systems with PHP
Andrei Zmievski @ DiggDutch PHP Conference @ Amsterdam
Friday, June 12, 2009
![Page 2: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/2.jpg)
Who is this guy?
• Open Source Fellow @ Digg
• PHP Core Developer since 1999
• Architect of the Unicode/i18n support
• Release Manager for PHP 6
• Twitter: @a
• Beer lover (and brewer)
Friday, June 12, 2009
![Page 3: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/3.jpg)
Why distributed?
• Because Moore’s Law will not save you
• Despite what DHH says
Friday, June 12, 2009
![Page 4: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/4.jpg)
Share nothing
• Your Mom was wrong
• No shared data on application servers
• Distribute it to shared systems
Friday, June 12, 2009
![Page 5: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/5.jpg)
distribute…
• memory (memcached)
• storage (mogilefs)
• work (gearman)
Friday, June 12, 2009
![Page 6: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/6.jpg)
Building blocks
• GLAMMP - have you heard of it?
• Gearman + LAMP + Memcached
• Throw in Mogile too
Friday, June 12, 2009
![Page 7: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/7.jpg)
memcachedFriday, June 12, 2009
![Page 8: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/8.jpg)
background
• created by Danga Interactive
• high-performance, distributed memory object caching system
• sustains Digg, Facebook, LiveJournal, Yahoo!, and many others
• if you aren’t using it, you are crazy
Friday, June 12, 2009
![Page 9: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/9.jpg)
background
• Very fast over the network and very easy to set up
• Designed to be transient
• You still need a database
Friday, June 12, 2009
![Page 10: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/10.jpg)
architecture
client
client
client
memcached
memcached
memcached
Friday, June 12, 2009
![Page 11: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/11.jpg)
architecture
memcached
Friday, June 12, 2009
![Page 12: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/12.jpg)
slab #1
slab #2
slab #3
slab #4
152 bytes
152 bytes
456 bytes
1368 bytes
152 bytes
4104 bytes
152 bytes
152 bytes
456 bytes456 bytes
1368 bytes
Friday, June 12, 2009
![Page 13: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/13.jpg)
memory architecture
• memory allocated on startup, released on shutdown
• variable sized slabs (30+ by default)
• each object is stored in the slab most fitting its size
• fragmentation can be problematic
Friday, June 12, 2009
![Page 14: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/14.jpg)
memory architecture
• items are deleted:
• on set
• on get, if it’s expired
• if slab is full, then use LRU
Friday, June 12, 2009
![Page 15: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/15.jpg)
applications
• object cache
• output cache
• action flood control / rate limiting
• simple queue
• and much more
Friday, June 12, 2009
![Page 16: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/16.jpg)
PHP clients
• a few private ones (Facebook, Yahoo!, etc)
• pecl/memcache
• pecl/memcached
Friday, June 12, 2009
![Page 17: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/17.jpg)
pecl/memcached
• based on libmemcached
• released in January 2009
• surface API similarity to pecl/memcache
• parity with other languages
Friday, June 12, 2009
![Page 18: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/18.jpg)
• get
• set
• add
• replace
• delete
• append
• prepend
• cas
• *_by_key
• getMulti
• setMulti
• getDelayed / fetch*
• callbacks
API
Friday, June 12, 2009
![Page 19: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/19.jpg)
consistent hashing
A
B
IP2-1
IP1
IP3
IP2-2
Friday, June 12, 2009
![Page 20: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/20.jpg)
compare-and-swap (cas)
• “check and set”
• no update if object changed
• relies on CAS token
Friday, June 12, 2009
![Page 21: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/21.jpg)
compare-and-swap (cas)$m = new Memcached();$m->addServer('localhost', 11211);
do { $ips = $m->get('ip_block', null, $cas);
if ($m->getResultCode() == Memcached::RES_NOTFOUND) {
$ips = array($_SERVER['REMOTE_ADDR']); $m->add('ip_block', $ips);
} else {
$ips[] = $_SERVER['REMOTE_ADDR']; $m->cas($cas, 'ip_block', $ips); }
} while ($m->getResultCode() != Memcached::RES_SUCCESS);
Friday, June 12, 2009
![Page 22: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/22.jpg)
delayed “lazy” fetching
• issue request with getDelayed()
• do other work
• fetch results with fetch() or fetchAll()
Friday, June 12, 2009
![Page 23: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/23.jpg)
binary protocol
• performance• every request is parsed• can happen thousands times a
second
• extensibility
• support more data in the protocol
Friday, June 12, 2009
![Page 24: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/24.jpg)
callbacks
• read-through cache callback
• if key is not found, invoke callback, save value to memcache and return it
Friday, June 12, 2009
![Page 25: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/25.jpg)
callbacks
• result callback
• invoked by getDelayed() for every found item
• should not call fetch() in this case
Friday, June 12, 2009
![Page 26: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/26.jpg)
buffered writes
• queue up write requests
• send when a threshold is exceeded or a ‘get’ command is issued
Friday, June 12, 2009
![Page 27: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/27.jpg)
key prefixing
• optional prefix prepended to all the keys automatically
• allows for namespacing, versioning, etc.
Friday, June 12, 2009
![Page 28: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/28.jpg)
key locality
• allows mapping a set of keys to a specific server
Friday, June 12, 2009
![Page 29: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/29.jpg)
multiple serializers
• PHP
• igbinary
• JSON (soon)
Friday, June 12, 2009
![Page 30: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/30.jpg)
future
• UDP support
• replication
• server management (ejection, status callback)
Friday, June 12, 2009
![Page 31: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/31.jpg)
tips & tricks
• 32-bit systems with > 4GB memory:
memcached -m4096 -p11211memcached -m4096 -p11212memcached -m4096 -p11213
Friday, June 12, 2009
![Page 32: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/32.jpg)
tips & tricks
• write-through or write-back cache
• Warm up the cache on code push
• Version the keys (if necessary)
Friday, June 12, 2009
![Page 33: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/33.jpg)
tips & tricks
• Don’t think row-level DB-style caching; think complex objects
• Don’t run memcached on your DB server — your DBAs might send you threatening notes
• Use multi-get — run things in parallel
Friday, June 12, 2009
![Page 34: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/34.jpg)
delete by namespace
1 $ns_key = $memcache->get("foo_namespace_key");2 // if not set, initialize it3 if ($ns_key === false)4 $memcache->set("foo_namespace_key",5 rand(1, 10000));6 // cleverly use the ns_key7 $my_key = "foo_".$ns_key."_12345";8 $my_val = $memcache->get($my_key);9 // to clear the namespace:10 $memcache->increment("foo_namespace_key");
Friday, June 12, 2009
![Page 35: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/35.jpg)
storing lists of data
• Store items under indexed keys: comment.12, comment.23, etc
• Then store the list of item IDs in another key: comments
• To retrieve, fetch comments and then multi-get the comment IDs
Friday, June 12, 2009
![Page 36: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/36.jpg)
preventing stampeding
• embedded probabilistic timeout
• gearman unique task trick
Friday, June 12, 2009
![Page 37: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/37.jpg)
optimization
• watch stats (eviction rate, fill, etc)
• getStats()
• telnet + “stats” commands
• peep (heap inspector)
Friday, June 12, 2009
![Page 38: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/38.jpg)
slabs
• Tune slab sizes to your needs:
• -f chunk size growth factor (default 1.25)
• -n minimum space allocated for key+value+flags (default 48)
Friday, June 12, 2009
![Page 39: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/39.jpg)
slabs
slab class 1: chunk size 104 perslab 10082slab class 2: chunk size 136 perslab 7710slab class 3: chunk size 176 perslab 5957slab class 4: chunk size 224 perslab 4681...slab class 38: chunk size 394840 perslab 2slab class 39: chunk size 493552 perslab 2
Default: 38 slabs
Friday, June 12, 2009
![Page 40: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/40.jpg)
slabs
slab class 1: chunk size 1048 perslab 1000slab class 2: chunk size 1064 perslab 985slab class 3: chunk size 1080 perslab 970slab class 4: chunk size 1096 perslab 956...slab class 198: chunk size 9224 perslab 113slab class 199: chunk size 9320 perslab 112
Most objects: ∼1-2KB, some larger
memcached -n 1000 -f 1.01
Friday, June 12, 2009
![Page 41: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/41.jpg)
memcached @ digg
Friday, June 12, 2009
![Page 42: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/42.jpg)
ops
• memcached on each app server (2GB)
• the process is niced to a lower level
• separate pool for sessions
• 2 servers keep track of cluster health
Friday, June 12, 2009
![Page 43: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/43.jpg)
key prefixes
• global key prefix for apc, memcached, etc
• each pool has additional, versioned prefix: .sess.2
• the key version is incremented on each release
• global prefix can invalidate all caches
Friday, June 12, 2009
![Page 44: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/44.jpg)
cache chain
• multi-level caching: globals, APC, memcached, etc.
• all cache access is through Cache_Chain class
• various configurations:• APC ➡ memcached• $GLOBALS ➡ APC
Friday, June 12, 2009
![Page 45: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/45.jpg)
other
• large objects (> 1MB)
• split on the client side
• save the partial keys in a master one
Friday, June 12, 2009
![Page 46: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/46.jpg)
stats
Friday, June 12, 2009
![Page 47: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/47.jpg)
alternatives
• in-memory: Tokyo Tyrant, Scalaris
• persistent: Hypertable, Cassandra, MemcacheDB
• document-oriented: CouchDB
Friday, June 12, 2009
![Page 48: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/48.jpg)
mogileFriday, June 12, 2009
![Page 49: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/49.jpg)
background
• created by Danga Interactive
• application-level distributed filesystem
• used at Digg, LiveJournal, etc
• a form of “cloud caching”
• scales very well
Friday, June 12, 2009
![Page 50: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/50.jpg)
background
• automatic file replication with custom policies
• no single point of failure
• flat namespace
• local filesystem agnostic
• not meant for speed
Friday, June 12, 2009
![Page 51: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/51.jpg)
architecture
tracker
nodeapp tracker DB
node
node
node
tracker
node
Friday, June 12, 2009
![Page 52: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/52.jpg)
applications
• images
• document storage
• backing store for certain caches
Friday, June 12, 2009
![Page 53: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/53.jpg)
PHP client
• File_Mogile in PEAR
• MediaWiki one (not maintained)
Friday, June 12, 2009
![Page 54: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/54.jpg)
Example
$hosts = array('172.10.1.1', '172.10.1.2');$m = new File_Mogile($hosts, 'profiles');$m->storeFile('user1234', 'image', '/tmp/image1234.jpg');
...
$paths = $m->getPaths('user1234');
Friday, June 12, 2009
![Page 55: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/55.jpg)
mogile @ digg
Friday, June 12, 2009
![Page 56: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/56.jpg)
mogile @ digg
• Wrapper around File_Mogile to cache entries in memcache
• fairly standard set-up
• trackers run on storage nodes
Friday, June 12, 2009
![Page 57: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/57.jpg)
mogile @ digg
• not huge (about 3.5 TB of data)
• files are replicated 3x
• the user profile images are cached on Netscaler (1.5 GB cache)
• mogile cluster load is light
Friday, June 12, 2009
![Page 58: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/58.jpg)
gearmanFriday, June 12, 2009
![Page 59: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/59.jpg)
background
• created by Danga Interactive
• anagram of “manager”
• a system for distributing work
• a form of RPC mechanism
Friday, June 12, 2009
![Page 60: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/60.jpg)
background
• parallel, asynchronous, scales well
• fire and forget, decentralized
• avoid tying up Apache processes
Friday, June 12, 2009
![Page 61: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/61.jpg)
background
• dispatch function calls to machines that are better suited to do work
• do work in parallel
• load balance lots of function calls
• invoke functions in other languages
Friday, June 12, 2009
![Page 62: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/62.jpg)
architecture
gearmandclient
workerclient
client
worker
worker
Friday, June 12, 2009
![Page 63: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/63.jpg)
applications
• thumbnail generation
• asynchronous logging
• cache warm-up
• DB jobs, data migration
• sending email
Friday, June 12, 2009
![Page 64: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/64.jpg)
servers
• Gearman-Server (Perl)
• gearmand (C)
Friday, June 12, 2009
![Page 65: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/65.jpg)
clients
• Net_Gearman
• simplified, pretty stable
• pecl/gearman
• more powerful, complex, somewhat unstable (under development)
Friday, June 12, 2009
![Page 66: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/66.jpg)
Concepts
• Job
• Worker
• Task
• Client
Friday, June 12, 2009
![Page 67: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/67.jpg)
Net_Gearman
• Net_Gearman_Job
• Net_Gearman_Worker
• Net_Gearman_Task
• Net_Gearman_Set
• Net_Gearman_Client
Friday, June 12, 2009
![Page 68: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/68.jpg)
Echo Job
class Net_Gearman_Job_Echo extends Net_Gearman_Job_Common{ public function run($arg) { var_export($arg); echo "\n"; }} Echo.php
Friday, June 12, 2009
![Page 69: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/69.jpg)
Reverse Jobclass Net_Gearman_Job_Reverse extends Net_Gearman_Job_Common{ public function run($arg) { $result = array(); $n = count($arg); $i = 0; while ($value = array_pop($arg)) { $result[] = $value; $i++; $this->status($i, $n); }
return $result; }} Reverse.php
Friday, June 12, 2009
![Page 70: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/70.jpg)
Worker
define('NET_GEARMAN_JOB_PATH', './');
require 'Net/Gearman/Worker.php';
try { $worker = new Net_Gearman_Worker(array('localhost:4730')); $worker->addAbility('Reverse'); $worker->addAbility('Echo'); $worker->beginWork();} catch (Net_Gearman_Exception $e) { echo $e->getMessage() . "\n"; exit;}
Friday, June 12, 2009
![Page 71: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/71.jpg)
Client
require_once 'Net/Gearman/Client.php';
function complete($job, $handle, $result) { echo "$job complete, result: ".var_export($result, true)."\n";}
function status($job, $handle, $n, $d){ echo "$n/$d\n";} continued..
Friday, June 12, 2009
![Page 72: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/72.jpg)
Client
$client = new Net_Gearman_Client(array('lager:4730'));
$task = new Net_Gearman_Task('Reverse', range(1,5));$task->attachCallback("complete",Net_Gearman_Task::TASK_COMPLETE);$task->attachCallback("status",Net_Gearman_Task::TASK_STATUS);
continued..
Friday, June 12, 2009
![Page 73: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/73.jpg)
Client
$set = new Net_Gearman_Set();$set->addTask($task);
$client->runSet($set);
$client->Echo('Mmm... beer');
Friday, June 12, 2009
![Page 74: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/74.jpg)
pecl/gearman
• More complex API
• Jobs aren’t separated into files
Friday, June 12, 2009
![Page 75: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/75.jpg)
Worker$gmworker= new gearman_worker();$gmworker->add_server();$gmworker->add_function("reverse", "reverse_fn");
while (1){ $ret= $gmworker->work(); if ($ret != GEARMAN_SUCCESS) break;}
function reverse_fn($job){ $workload= $job->workload(); echo "Received job: " . $job->handle() . "\n"; echo "Workload: $workload\n"; $result= strrev($workload); echo "Result: $result\n"; return $result;}
Friday, June 12, 2009
![Page 76: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/76.jpg)
Client
$gmclient= new gearman_client();
$gmclient->add_server('lager');
echo "Sending job\n";
list($ret, $result) = $gmclient->do("reverse", "Hello!");
if ($ret == GEARMAN_SUCCESS) echo "Success: $result\n";
Friday, June 12, 2009
![Page 77: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/77.jpg)
gearman @ digg
Friday, June 12, 2009
![Page 78: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/78.jpg)
gearman @ digg
• 400,000 jobs a day
• Jobs: crawling, DB job, FB sync, memcache manipulation, Twitter post, IDDB migration, etc.
• Each application server has its own Gearman daemon + workers
Friday, June 12, 2009
![Page 79: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/79.jpg)
tips and tricks
• you can daemonize the workers easily with daemon or supervisord
• run workers in different groups, don’t block on job A waiting on job B
• Make workers exit after N jobs to free up memory (supervisord will restart them)
Friday, June 12, 2009
![Page 80: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/80.jpg)
ThriftFriday, June 12, 2009
![Page 81: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/81.jpg)
background
• NOT developed by Danga (Facebook)
• cross-language services
• RPC-based
Friday, June 12, 2009
![Page 82: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/82.jpg)
background
• interface description language
• bindings: C++, C#, Cocoa, Erlang, Haskell, Java, OCaml, Perl, PHP, Python, Ruby, Smalltalk
• data types: base, structs, constants, services, exceptions
Friday, June 12, 2009
![Page 83: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/83.jpg)
IDL
Friday, June 12, 2009
![Page 84: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/84.jpg)
Demo
Friday, June 12, 2009
![Page 85: All The Little Pieces](https://reader031.vdocument.in/reader031/viewer/2022013011/54b76a504a795957768b46a3/html5/thumbnails/85.jpg)
Thank You
http://gravitonic.com/talksFriday, June 12, 2009