GearmanDistributed Process Solution
Created by Google Drive Presentation.
Introduction$ whoami
########################################################
#### Name : Nguyen Duy Khanh ####
#### Job : Student of HCMUS, Part-timer at PNC ####
#### My skills : PHP, MySQL, JS ####
#### Love : New technology ####
########################################################
$|
Problems
HEAVY WORKLOADfunction main() {
...doSomeThing();doSomeThingBig();doSomeThingBigBig();doSomeThingVeryBig();…return result;
}
Problems
console.log(result);
Solution
Worker 2
Worker 1
Worker...
Worker n
Give me a job, my workers will do it for you.
Gearman❏ GEARMAN → MANAGER
❏ Open Source.
❏ Multi-language API.
❏ Multi-threaded ( 50k jobs / second ).
❏ Ultra fast ( written in C/C++ ).
❏ No limits on message size.
❏ Failover
Installation1. Download source code [ https://launchpad.net/gearmand ].2. Compile and install from tarball.
3. Starting gearman.
tar xzf gearmand-X.Y.tar.gz
cd gearmand-X.Y
./configure
make
make install
gearman -d
Gearman Architecture
Customer
Job Manager
Worker
How does it work ?
Client
Job Manager
Worker Python
Worker Node.JS
Worker PHP
Hey, I can send email. Let me do it !!!
How does it work ?
Client
Job Manager
Worker Python(send_email)
Worker Node.JS
Worker PHP
How does it work ?
Client
Job Manager
Worker Python
Worker Node.JS
Worker PHP
And I can push notification. Let me try !!!
How does it work ?
Client
Job Manager
Worker Python(send_email)
Worker Node.JS(push_notify)
Worker PHP
How does it work ?
Client
Job Manager
Worker Python
Worker Node.JS
Worker PHP
Haha, I can do them all !!!
How does it work ?
Client
Job Manager
Worker Python(send_email)
Worker Node.JS(push_notify)
Worker PHP(send_email, push_notify)
How does it work ?
Client
Job Manager
Worker Python(send_email)
Worker Node.JS(push_notify)
Worker PHP(send_email, push_notify)
I need send an email to [email protected] , please help me !
How does it work ?
Client
Job Manager
Worker Python(send_email)
Worker Node.JS(push_notify)
Worker PHP(send_email, push_notify)
OK, Let me see. Aaah … We have 2 workers can do it for you. But, Worker Python is busy, let Worker PHP do it.
How does it work ?
Client
Job Manager
Worker Python(send_email)
Worker Node.JS(push_notify)
Worker PHP(send_email, push_notify)
Hey Worker PHP, please send an email to [email protected]
How does it work ?
Client
Job Manager
Worker Python(send_email)
Worker Node.JS(push_notify)
Worker PHP(send_email, push_notify)
OK Boss, email has sent to [email protected]
How does it work ?
Client
Job Manager
Worker Python(send_email)
Worker Node.JS(push_notify)
Worker PHP(send_email, push_notify)
Hi client, your email has sent to [email protected].
Hello world !!!Install Node.js API
$ npm install node-gearman
Install PHP API$ pecl install gearman
Install Python API$ pip install gearman
or$ easy_install gearman
Hello world !!!worker1.js
Hello world !!!worker2.py
Hello world !!!client.php
Hello world !!!
● Python and C/C++ and a good choices for workers.
● PHP, Node.js should be client than worker. [ PHP Worker often crash after
1-2 hours because memory leak ]
Use-cases❏ Scatter/Gather
❏ Map - Reduce
❏ Asynchronous Queues
❏ Pipeline Processing
Scatter / Gather● Perform a number of task concurrently.
○ Ex : Db Query, Image Processing, Location Search,
Full-text Search…
● Take advantage of many tier.
● Speed up your web applications.
● Tasks don’t need to be related.
Scatter / Gather
Client
Db Query
Resize Image
Search
Map / Reduce
Asynchronous Queues
● Most popular use-case.
● Do something needn’t immediate processing.
○ Ex : send emails, push notification, index data,
crawling, ...
● Allows for batch operations.
Asynchronous Queuesfunction tweet(msg) {
// Insert to databasedb.insert(msg);// Background tasksgearman.push_notify();// Returnreturn true;
}
Pipeline Processing
● Do related thing.
○ Ex : search engine, data analytic, ...
● Chain workers to send data to next step.
Pipeline Processing
WorkerStep 1 Client Worker
Step 2 Client WorkerLast Step
Main Client Output
Examples
MongoDB Workers“parse_rss”
RSS Sources
ReloadRequest
Gearman Client End
Workers“crawling”
Have new items ?
No
Crawl new items
GEARMAN Auto News Crawling System
● Manager : GEARMAN
● Workers : Python
● Webserver : Node.JS
● Web Framework : Express
● Database : MongoDB
● Client Script : jQuery
Persistent Queues❏ By default, jobs are only stored in memory.
❏ Various contributions from community :
❏ MySQL / Drizzle
❏ PostgreSQL
❏ SQLite
❏ Tokyo Cabinet
❏ memcached
❏ Hope MongoDB (or an NOSQL DB) coming soon.
Gearman Admin GUI● Command Line Tool : http://gearman.info/bin/gearadmin.html
● Gearman-Monitor [ PHP ] : https://github.com/yugene/Gearman-Monitor
● GearmanUI [ PHP ] : http://rripado.info/gearmanui/
THE N THANK YOU !!! D
About UsAuthor : Khanh Nguyen DuyFind me at : [email protected]
Presentation made for “Javascript HoChiMinh City Meetup”
You can find us at :● http://meetup.com/JavaScript-Ho-Chi-Minh-City/● https://www.facebook.com/JavaScriptHCMC● https://plus.google.com/u/0/communities/116105314977285194967
