khanh-nguyen - gearman - distributed process solution
DESCRIPTION
Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. In other words, it is the nervous system for how distributed processing communicates.TRANSCRIPT
GearmanDistributed Process Solution
Created by Google Drive Presentation.
Introduction$ whoami
########################################################
#### Name : Nguyen Duy Khanh ####
#### Job : Student of HCMUS, Part-timer at PNC ####
#### My skills : PHP, MySQL, JS ####
#### Love : New technology ####
########################################################
$|
Problems
HEAVY WORKLOADfunction main() {
...doSomeThing();doSomeThingBig();doSomeThingBigBig();doSomeThingVeryBig();…return result;
}
Problems
console.log(result);
Solution
Worker 2
Worker 1
Worker...
Worker n
Give me a job, my workers will do it for you.
Gearman❏ GEARMAN → MANAGER
❏ Open Source.
❏ Multi-language API.
❏ Multi-threaded ( 50k jobs / second ).
❏ Ultra fast ( written in C/C++ ).
❏ No limits on message size.
❏ Failover
Installation1. Download source code [ https://launchpad.net/gearmand ].2. Compile and install from tarball.
3. Starting gearman.
tar xzf gearmand-X.Y.tar.gz
cd gearmand-X.Y
./configure
make
make install
gearman -d
Gearman Architecture
Customer
Job Manager
Worker
How does it work ?
Client
Job Manager
Worker Python
Worker Node.JS
Worker PHP
Hey, I can send email. Let me do it !!!
How does it work ?
Client
Job Manager
Worker Python(send_email)
Worker Node.JS
Worker PHP
How does it work ?
Client
Job Manager
Worker Python
Worker Node.JS
Worker PHP
And I can push notification. Let me try !!!
How does it work ?
Client
Job Manager
Worker Python(send_email)
Worker Node.JS(push_notify)
Worker PHP
How does it work ?
Client
Job Manager
Worker Python
Worker Node.JS
Worker PHP
Haha, I can do them all !!!
How does it work ?
Client
Job Manager
Worker Python(send_email)
Worker Node.JS(push_notify)
Worker PHP(send_email, push_notify)
How does it work ?
Client
Job Manager
Worker Python(send_email)
Worker Node.JS(push_notify)
Worker PHP(send_email, push_notify)
I need send an email to [email protected] , please help me !
How does it work ?
Client
Job Manager
Worker Python(send_email)
Worker Node.JS(push_notify)
Worker PHP(send_email, push_notify)
OK, Let me see. Aaah … We have 2 workers can do it for you. But, Worker Python is busy, let Worker PHP do it.
How does it work ?
Client
Job Manager
Worker Python(send_email)
Worker Node.JS(push_notify)
Worker PHP(send_email, push_notify)
Hey Worker PHP, please send an email to [email protected]
How does it work ?
Client
Job Manager
Worker Python(send_email)
Worker Node.JS(push_notify)
Worker PHP(send_email, push_notify)
OK Boss, email has sent to [email protected]
How does it work ?
Client
Job Manager
Worker Python(send_email)
Worker Node.JS(push_notify)
Worker PHP(send_email, push_notify)
Hi client, your email has sent to [email protected].
Hello world !!!Install Node.js API
$ npm install node-gearman
Install PHP API$ pecl install gearman
Install Python API$ pip install gearman
or$ easy_install gearman
Hello world !!!worker1.js
Hello world !!!worker2.py
Hello world !!!client.php
Hello world !!!
● Python and C/C++ and a good choices for workers.
● PHP, Node.js should be client than worker. [ PHP Worker often crash after
1-2 hours because memory leak ]
Use-cases❏ Scatter/Gather
❏ Map - Reduce
❏ Asynchronous Queues
❏ Pipeline Processing
Scatter / Gather● Perform a number of task concurrently.
○ Ex : Db Query, Image Processing, Location Search,
Full-text Search…
● Take advantage of many tier.
● Speed up your web applications.
● Tasks don’t need to be related.
Scatter / Gather
Client
Db Query
Resize Image
Search
Map / Reduce
Asynchronous Queues
● Most popular use-case.
● Do something needn’t immediate processing.
○ Ex : send emails, push notification, index data,
crawling, ...
● Allows for batch operations.
Asynchronous Queuesfunction tweet(msg) {
// Insert to databasedb.insert(msg);// Background tasksgearman.push_notify();// Returnreturn true;
}
Pipeline Processing
● Do related thing.
○ Ex : search engine, data analytic, ...
● Chain workers to send data to next step.
Pipeline Processing
WorkerStep 1 Client Worker
Step 2 Client WorkerLast Step
Main Client Output
Examples
MongoDB Workers“parse_rss”
RSS Sources
ReloadRequest
Gearman Client End
Workers“crawling”
Have new items ?
No
Crawl new items
GEARMAN Auto News Crawling System
● Manager : GEARMAN
● Workers : Python
● Webserver : Node.JS
● Web Framework : Express
● Database : MongoDB
● Client Script : jQuery
Persistent Queues❏ By default, jobs are only stored in memory.
❏ Various contributions from community :
❏ MySQL / Drizzle
❏ PostgreSQL
❏ SQLite
❏ Tokyo Cabinet
❏ memcached
❏ Hope MongoDB (or an NOSQL DB) coming soon.
Gearman Admin GUI● Command Line Tool : http://gearman.info/bin/gearadmin.html
● Gearman-Monitor [ PHP ] : https://github.com/yugene/Gearman-Monitor
● GearmanUI [ PHP ] : http://rripado.info/gearmanui/
THE N THANK YOU !!! D
About UsAuthor : Khanh Nguyen DuyFind me at : [email protected]
Presentation made for “Javascript HoChiMinh City Meetup”
You can find us at :● http://meetup.com/JavaScript-Ho-Chi-Minh-City/● https://www.facebook.com/JavaScriptHCMC● https://plus.google.com/u/0/communities/116105314977285194967