distributed and concurrent programming with rabbitmq and eventmachine rails underground 2009
TRANSCRIPT
![Page 1: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/1.jpg)
“Divide and conquer riding rabbits and
trading gems”- a tale about distributed programming -
Paolo Negri @hungryblank
• http://www.slideshare.net/hungryblank
• http://github.com/hungryblank/rabbit_starter
![Page 2: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/2.jpg)
Resources
• http://www.slideshare.net/hungryblank
• http://github.com/hungryblank/rabbit_starter
![Page 3: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/3.jpg)
About meTime
GNU/Linux - Dbs“systems”
Perl
Python
Ruby
PHP
![Page 4: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/4.jpg)
Summary:
![Page 5: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/5.jpg)
DistributedConcurrentProgramming
![Page 6: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/6.jpg)
http://www.flickr.com/photos/myxi/448253580
rabbitMQ
![Page 7: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/7.jpg)
Control
![Page 8: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/8.jpg)
The problem
Given a dictionary of1.000.000 search phrases,
compare for each onethe first page of results ongoogle.com and bing.com
![Page 9: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/9.jpg)
Don’t do it.It’s a violation of terms
of service!(Sorry, I needed an example that required little or no explanation)
![Page 10: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/10.jpg)
Comparison
![Page 11: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/11.jpg)
The whole process
![Page 12: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/12.jpg)
Great! but...
• Fetching, parsing and analyzing 2.000.000 pages of results will take a long time
• we want to collect the statistics in a limited time and this sequential approach is not quick enough
![Page 13: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/13.jpg)
Distributed Computing
Is a method of solving computational problem by dividing the problem into
many tasks run simultaneously on many hardware or software systems
(Wikipedia)
![Page 14: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/14.jpg)
Map Reduce "Map" step
the master node takes the input, chops it up into smaller sub-problems, and distributes
those to worker nodes.(Wikipedia)
![Page 15: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/15.jpg)
Problems:
•How many nodes?
•How many workers?
•Distribution mechanism to feed the workers?
![Page 16: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/16.jpg)
What about queuing?
• the master node takes the input, chops it up into smaller sub-problems, and publishes them in a queue
• workers independently consume the content of the queue
![Page 17: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/17.jpg)
Here comes
• RabbitMQ is an implementation of AMQP, the emerging standard for high performance enterprise messaging
• It’s opensource
• Written in Erlang
![Page 18: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/18.jpg)
Erlang?
• general-purpose concurrent programming language designed by Ericsson
• first version written by J. Armstrong in 1986
• distributed
• fault tolerant
• soft real time
• high availability
![Page 19: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/19.jpg)
Erlang - is coming back• Projects
• CouchDB - RESTful document storage
• WebMachine - REST toolkit
• Nitrogen - web framework
• Mochiweb - web framework
• Ruby shops using it
• 37 Signal - Campfire
• Heroku
![Page 20: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/20.jpg)
+ Erlang
It’s messages all the way down
![Page 21: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/21.jpg)
Install it
• sudo apt-get install rabbitmq
• sudo gem install tmm1-amqp
Note: rabbitMQ must be v1.6.0 and amqp gem v 0.6.4to follow the code in the slides
![Page 22: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/22.jpg)
Do it! - master node
![Page 23: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/23.jpg)
Do it! - worker node
![Page 24: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/24.jpg)
Get for free
• Decoupling master/worker
• Workers take care of feeding themselves
• Flexible number of workers
![Page 25: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/25.jpg)
Behind the scenes
msg A Queue
Worker1
Worker2
Worker3
Master
Exchange
![Page 26: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/26.jpg)
Behind the scenes
Queue
Worker1
Worker2
Worker3
Master
Exchangemsg A
![Page 27: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/27.jpg)
Behind the scenes
Queue
Worker1
Worker2
Worker3
Master
Exchange msg A
![Page 28: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/28.jpg)
Behind the scenes
Queue
Worker1
Worker2
Worker3
Master
Exchange
msg A
![Page 29: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/29.jpg)
RabbitMQ
• Multiple exchanges (and multiple types of exchange)
• Multiple queues
• Queues are connected by bindings to exchanges
• Exchanges route messages to queues
![Page 30: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/30.jpg)
RabbitMQ
• Exchanges and queues have names
• BUT direct exchanges created implicitly are not public and don’t have name
• Queues and messages are resident in RAM and can be persisted on disk (at a performance price)
![Page 31: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/31.jpg)
What and where
RabbitMQ(Erlang) TCP/IP
Master(ruby)
Worker(ruby)
Worker(ruby)
Queue Exchange Worker(ruby)
![Page 32: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/32.jpg)
Problem #1
If a worker has a problem we might lose one or more messages
http://www.flickr.com/photos/danzen/2288625136/
![Page 33: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/33.jpg)
Solution - ACK in worker
![Page 34: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/34.jpg)
Acknoledgement
• messages - 1 or more depending by prefetch settings - are passed to the client
• the client processes and acknowledges the messages one by one
• if the client connection closes => unacknowledged messages go back in the queue
![Page 35: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/35.jpg)
Problem #2
No convenient way to control the workers
http://www.flickr.com/photos/streetfly_jz/2770586821/
![Page 36: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/36.jpg)
System queue - worker
![Page 37: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/37.jpg)
System queue - control
• save it as system_command.rb
• ruby system_command.rb halt
![Page 38: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/38.jpg)
System queue
Queue1
Queue3
Queue2
Worker1
Worker2
Worker3
Control script
Exchangemsg A
![Page 39: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/39.jpg)
System queue
Queue1
Queue3
Queue2
Worker1
Worker2
Worker3
Control script
Exchangemsg A
![Page 40: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/40.jpg)
System queue
Queue1
Queue3
Queue2
Worker1
Worker2
Worker3
Control script
Exchange
msg A
msg A
msg A
![Page 41: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/41.jpg)
EventMachine
![Page 42: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/42.jpg)
EventMachine
• Non blocking IO and lightweight concurrency
• eliminate the complexities of high-performance threaded network programming
Is an implementation of Reactor Pattern
![Page 43: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/43.jpg)
without EM
code
networkoperation
use network operation
result
Free
code
networkoperation
use network operation
result
Callback
Free
Free
Time
with EMRuby process Ruby process
![Page 44: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/44.jpg)
EventMachine
amqp gem is built on top of EventMachine => you’re in a
context where you can leverage concurrent
programming
![Page 45: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/45.jpg)
EM - Deferrables
“The Deferrable pattern allows you to specify any number of Ruby code blocks that will be
executed at some future time when the status of the Deferrable object changes “
![Page 46: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/46.jpg)
EM - Deferrables
![Page 47: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/47.jpg)
EM - Deferrables
![Page 48: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/48.jpg)
EM - Deferrables
![Page 49: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/49.jpg)
EM - Deferrables
![Page 50: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/50.jpg)
Deferrables
GooglePage
BingPage
without deferrables with deferrables
Time
GooglePage BingPage
![Page 51: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/51.jpg)
Problem #3
How many of them?
what are they doing?
http://www.flickr.com/photos/philocrites/341850461/
![Page 52: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/52.jpg)
Heartbeat - worker
![Page 53: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/53.jpg)
Heartbeat monitor
![Page 54: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/54.jpg)
Heartbeat queue
msg A
Queue
Worker1
Worker2
Worker3
MonitorExchange
msg B
![Page 55: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/55.jpg)
Heartbeat queue
Queue
Worker1
Worker2
Worker3
MonitorExchangemsg A
msg B
![Page 56: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/56.jpg)
Heartbeat queue
Queue
Worker1
Worker2
Worker3
MonitorExchange msg Amsg B
![Page 57: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/57.jpg)
Clustering
RabbitMQ - node C
RabbitMQ - node A RabbitMQ - node B
TCP/IP
TCP/IPTCP/IP
![Page 58: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/58.jpg)
async vs sync
• famoseagle/carrot
• celldee/bunny
Syncronous clients on github
![Page 59: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/59.jpg)
Rabbits on github
• danielsdeleo/moqueue Tests/stubbing
• Check out ezmobius/nanite“self assembling fabric of ruby daemons”
• Keep an eye on tonyg/rabbithub implementation of pubsubhubbub (PubSub over REST)
• auser/alice web app to monitor RabbitMQ
![Page 60: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/60.jpg)
More rabbits on github
• tmm1/em-spec
• eventmachine/eventmachine
• tmm1/amqp
• macournoyer/thin
![Page 61: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/61.jpg)
Q&A
?
![Page 62: Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009](https://reader031.vdocument.in/reader031/viewer/2022030316/58780e521a28ab971e8b6755/html5/thumbnails/62.jpg)
Thanks!(Thanks Mark!)
Paolo Negri
/ hungryblank