The Current State of Asynchronous

Processing in Ruby

Mathias Meyer, Peritor GmbH

self

• Software-Developer for Peritor GmbH in Berlin

• Code-Reviews, Scaling, Performance-Reviews and -Tuning, Refactorings

• Maintainer: acts_as_solr, run_later, heaps of stuff for Capistrano/Webistrano

What?

• Asynchronous Processing

• Just a fancy word for...

“Dude, this request is getting too long!”

“Okay, let’s move stuff into the background.”

In other words: What?

• A process puts a job into some queue

• Another process picks up the job and processes it

Why?

• Requests are taking too long

• Image uploads to S3

• Full-text search updates

• Generating PDFs, reports, etc.

• Sending emails (newsletters, etc.)

Why?

• Your app requires longer running tasks

• Compute daily statistics data

• Create reports

• Data crunching

Why?

• You need to run tasks at a certain time

• Scheduled tasks like invoicing, billing

• Sending out reminder emails

Why?

• Longer tasks

• Are harder to debug and monitor

• Block user and the application

Example

Rails

Client

Upload to S3

Response

!

Example - Much Better

RailsClient

Upload to S3

Response

Worker

The simplest Thing that could possibly work

Thread.newdoAccountMailer.deliver_signup(@user)end

Done.

Not so fast...

• Problems

• Not easy on resources

• Unreliable

• Not reproducible

A little better...

run_laterdoAccountMailer.deliver_signup(@user)end

run_later

• Borrowed from Merb, available as a Rails-Plugin

• Uses worker thread and a queue

• Simple solution for simple tasks

What you really want

• Reliable messaging

• Durability

• Scheduling

• Scalable processing

• Not necessarily all at the same time

Options, options

• Messaging Queues

• Polling

• Schedulers

• Oh my!

Protocols, oh my!

• AMQP

• STOMP

• JMS

• XMPP

• RestMS

Message Queues

Message Queues

• Publish/subscribe mechanism

• Usually require more than one new component in your infrastructure

• Broker middleware

• Subscribers, message listeners

Message Queues

• ActiveMQ

• RabbitMQ

• ActiveMessaging

• Amazon SQS

• Usually more code involved

Pollers

• Polling a database for new jobs

• Simple and domain-specific

• Only one new component

• Usually requires more effort to make them less prone to errors

Pollers

• Roll Your Own w/ or w/o daemons gem

• delayed_job - adds some infrastructure

• background_job

Pollers - RYO

create_table:jobsdo|t|t.string:klazz,:methodt.string:obj_idend

Pollers - RYOclassJob<ActiveRecord::Basedefself.schedule(obj,method)create(:klazz=>obj.class.name,:obj_id=>obj.id,:method=>method.to_s)enddefrunobj=klazz.constantize.find(obj_id)obj.send(self[:method])endend

Job.schedule(@user,:notify_signup)

Pollers - RYO

task:pollerdoJobPoller.new.runend

classJobPollerdefrunloopdojob=Job.find(:first,:lock=>true)nextunlessjobjob.runjob.deleteendendend

Pollers - delayed_job

User.find(params[:id]).send_later(:notify_signup)

rakejobs:work

Pollers - delayed_job

classSignupNotifier<Struct.new(:name)defperformuser=User.find_by_name(name)user.notify_signupendend

Delayed::Job.enqueueSignupNotifier.new("david")

Message Queues, Pollers

• Usually don’t directly support scheduling

Schedulers

Schedulers

• cron, cron, cron (oh, and rake)

• rufus-scheduler

• BackgrounDRb

• Quartz with JRuby (if you’re into that sort of thing)

• Roll Your Own

rufus-schedulerscheduler=Rufus::Scheduler.start_new

scheduler.cron'022**1‐5'doJob.new.runend

scheduler.in'20m'doputs"Getaflatwhite"end

Playing with the Big Guys

• Nanite - Jack of all trades

• Uses RabbitMQ, Erlang-based messaging server, AMQP implementation

• Distributes work across a network of workers

Nanite

• A self-assembling fabric of Ruby daemons

• Uses mappers and agents

• Mappers route message requests

• Agents handle messages

Nanite

Agent

Agent

Agent

Agent

RabbitMQ

Mapper

Mapper

Nanite

classReactorincludeNanite::Actorexpose:reactdefreact(payload)"reactingtomessagewithpayload:#{payload}"endend

Nanite

Nanite.request('/reactor/react','goodactingisreacting')do|r|prend

Nanite

• Agents register themselves, constantly pinging the mappers

• Mappers remove timed-out agents

• Work distributed based on agent load

• Unfortunately: Poor documentation

Other Stuff• BackgrounDRb

• AP4R

• job_fu

• spawn

• Starling/Workling

• Daemons

• beanstalkd

Careful now! Pitfalls

Race Conditions

Race Conditions

• Workers try accessing the same data

• Workers update data that is updated heavily from other layers

Race Conditions

• Make jobs repeatable

• Reduce proneness to errors coming from the database

• Let your broker take care of that

Queue Congestion

Queue Congestion

http://www.flickr.com/photos/lasgalletas/263909727/

Queue Congestion

• Workers can’t keep up with queue

• More work coming in than being processed

Queue Congestion

• Make it easy to add more workers

• Ensure they don’t steal each other’s work or do it twice (row-level locking, lock on fields)

Stalled Workers

http://www.flickr.com/photos/freeurmind/2941677586/

Stalled Workers

• Workers stopped processing jobs

• Got stuck on an a particular task

• Choked on an error

Stalled Workers

• Monitor your workers and queues

• Build in error reporting, report exceptions like in the rest of your code

• Make jobs repeatable

• Use Nanite (self-healing)

Recommendations

• Simple jobs

• Roll Your Own

• delayed_job

Recommendations

• Distributed

• Amazon SQS w/ ActiveMessaging or custom worker

• Works best on EC2

Recommendations

• Heavy load, scalable

• Nanite/RabbitMQ

The End

• Questions?

• @roidrage

• http://www.paperplanes.de

• http://github.com/mattmatt