gearmanforbeginners-091102040759-phpapp02

8/2/2019 gearmanforbeginners-091102040759-phpapp02

1/63

Sunday, 25 October 2009


2/63

Gearman: a technologyfor distributed

computingGiuseppe MaxiaMySQL Community Team Lead

Sun Microsystems



3/63

Mainframe

Mainframe

operating system

hardware

application

terminal

client

USER FRIENDLINESS

0 100

terminal

terminal

terminal



4/63

Mini computers

Mainframe

operating system

hardware

application

client

USER FRIENDLINESS

0 100

Mini terminal

terminal

terminal

terminal

Mini



5/63

Networks

server

hardware

application

client

USER FRIENDLINESS

0 100

personalcomputer

personalcomputer

personalcomputer

personalcomputer

hardware

operatingsystem

operatingsystem

hardware



6/63

Web applications

web server

hardware

application

client

USER FRIENDLINESS

0 100

browser

browser

browser

browser

operatingsystem

hardware

operatingsystem

operatingsystem

hardware

operating

system



7/63

Cloud applications

web

server

hardware

application

client

USER FRIENDLINESS

0 100

browser

browser

browser

browser

operatingsystem

hardware

operatingsystem

operatingsystem

hardware

serviceprovider

applicationapplication

serviceproviderservice

providerserviceprovider



8/63

Some actors

memcached

gearman

Used in production by Facebook and Digg



9/63

MAG A NE R



10/63

MG A NER!=



11/63

MAG A NE R ?



12/63

M A GA N E R



13/63

server worker client

taskjob

request

http://gearman.orgSunday, 25 October 2009
http://gearman.org/http://gearman.org/http://gearman.org/http://gearman.org/http://gearman.org/


14/63

Distributed



15/63

Multiple operating systems



16/63

multiple languages



17/63

freedom ofchoice



18/63

redundancy



19/63

USING GEARMAN

Server:gearmand

Client libraries:

C/C++ Java

Perl PHP

PythonSunday, 25 October 2009


20/63

Simple usage

GEARMAN

Command line client and worker



21/63

/usr/local/sbin/gearmand -d

# started as daemon.# No feedback given on the command line



22/63

/usr/local/sbin/gearmand -v -vINFO Starting upINFO Listening on :::4730 (5)INFO Listening on 0.0.0.0:4730 (6)

# started as normal application# verbose output requested



23/63

gearman -w -h hostname -p 4730 \-f conta wc

# -w = act as worker# -f = function

# conta = function name# wc = command to execute when function# 'conta' is called



24/63

/usr/local/sbin/gearmand -v -vINFO Starting upINFO Listening on :::4730 (5)INFO Listening on 0.0.0.0:4730 (6)

INFO Accepted connection from 127.0.0.1:4994INFO [ 0] 127.0.0.1:4994 Connected



25/63

gearman -h hostname -p 4730 \-f conta < ~/.bashrc

57 135 2149 #


26/63

/usr/local/sbin/gearmand -v -v

INFO Starting upINFO Listening on :::4730 (5)

INFO Listening on 0.0.0.0:4730 (6)

INFO Accepted connection from 127.0.0.1:4994INFO [ 0] 127.0.0.1:4994 Connected

INFO Accepted connection from 127.0.0.1:5181

INFO [ 0] 127.0.0.1:5181 Connected

INFO [ 0] 127.0.0.1:5181 Disconnected



27/63

What happened

1 server start

listen to port 4730



28/63

What happened

2 worker starts

registers function 'conta' to server



29/63

What happened

3 client starts

requires function 'conta' from server

provides input data



30/63

What happened

4 server sends client request to worker

passes all input data to worker



31/63

What happened

5 worker receives request and data

processes input



32/63

What happened

6 worker returns processed data

server passes it to client



33/63

What happened

7 client receives processed data

client displays result



34/63

A simple Perl worker



35/63

A simple perl worker

1. add server

2. add function

3. loop

4. function definition



36/63

use strict;

use warnings;use Gearman::XS qw(:constants);use Gearman::XS::Worker;

my $host = '127.0.0.1';my $port = 4730;

my $worker = new Gearman::XS::Worker;

my $ret = $worker->add_server($host, $port);if ($ret != GEARMAN_SUCCESS) {printf(STDERR "%s\n", $worker->error());exit(1);

}Sunday, 25 October 2009


37/63

my $options = '';

$ret = $worker->add_function("reverse", # public function name0, # timeout

\&myreverse, # reference to function$options); # function arguments

if ($ret != GEARMAN_SUCCESS) {printf(STDERR "%s\n", $worker->error());

}



38/63

while (1) {

my $ret = $worker->work();if ($ret != GEARMAN_SUCCESS) {printf(STDERR "%s\n", $worker->error());

}

}



39/63

sub myreverse {

my ($job) = @_;

my $workload = $job->workload();my $result = reverse($workload);

printf("Job=%s F_Name=%s Workload=%s Result=%s\n",

$job->handle(),$job->function_name(),$job->workload(),$result);

return $result;}



40/63

A simple Perl client



41/63

A simple perl client

add server run a task



42/63

use strict;

use warnings;use Gearman::XS qw(:constants);use Gearman::XS::Client;

my $client = new Gearman::XS::Client;

my $host = '127.0.0.1';my $port = 4730;my $ret = $client->add_server($host, $port);if ($ret != GEARMAN_SUCCESS) {printf(STDERR "%s\n", $client->error());exit(1);

}



43/63

my $input = shift || 'teststring';

my ($return, $result) =$client->do("reverse", $input);

if ($return == GEARMAN_SUCCESS) {

printf("Result=%s\n", $result);}



44/63

A sample run



45/63

perl worker.pl



46/63

perl client.pl

Result=gnirtstset



47/63

perl worker.pl

Job=H:gmac3.local:4 F_Name=reverseWorkload=teststring Result=gnirtstset



48/63

more client functions

do_background add_task

run_tasks



49/63

Some advanced usage

DBIx::SQLCrosstab Data cubes

Perl only



50/63



51/63

Image processing

CPU intensive Large storage needed

Application is OS specific



52/63



53/63



54/63

Using GEARMAN

client server architecture

N servers

N workers

N clients

Supports multiple languages



55/63

worker1 worker2

worker4

func1

func2

worker3

worker5

func1

func1

func3

func3

func1

func3

func1

func3

worker0func1

func3

client

Gearmanserver

Gearmanserver



56/63

worker1 worker2

worker4

func1

func2

worker3

worker5

func1

func1

func3

func3

func1

func3

func1

func3

worker0func1

func3

Master

Slave2

Slave1

client

Gearmanserver

Gearmanserver

Gearmanserver


Remote sandbox


57/63

Remote sandbox

worker

add server

add functionloop

function definition



58/63

$ret = $worker->add_function(

"make_sandbox",0,\&make_sandbox,'');

if ($ret != GEARMAN_SUCCESS) {printf(STDERR "%s\n", $worker->error());}



59/63

sub make_sandbox {

my ($job, $options) = @_;

my $command = $job->workload();chomp $command;

print STDERR "\n";my $result = qx(make_sandbox $command) ;if ($?) {

print $result, "\n";return "Error ($!)\n";

}else {

return "ok\n"}

}Sunday, 25 October 2009

Client to install


60/63

Client to install

Remote sandboxes

add server

run remote installation on masterrun remote installation on each slave

run configuration queries on slaves



61/63

my $ret =

qx(echo "$MASTER_SANDBOX" |gearman -h $master_ip -f make_sandbox ) ;

# using the gearman command line utility



62/63

More on similar matters

FOSS.MY, Kuala Lumpur, Malaysia, 25-Oct-2009 CodeBits, Lisbon, Portugal, 3-4-5-Dec-2009

http://datacharmer.blogspot.com

http://gearman.org

http://gearman.org/http://gearman.org/http://gearman.org/http://gearman.org/http://datacharmer.blogspot.com/http://datacharmer.blogspot.com/


63/63

THANKS

Let's talk!
http://creativecommons.org/licenses/by-sa/3.0/

gearmanforbeginners-091102040759-phpapp02

Documents