Download - Message Queues and Drupal
Message Queues and Drupal
About Me
Name: Brian Altenhofel
IRC: VeggieMeat
Twitter: @BrianAltenhofel
Drupal: #550616
What I Do: Commerce, Third-party APIs, Migrations, Devops, anything with data...
Favorite Tools: Vim, Scotch, Padron Serie 1926 No. 1 Maduro
Commonly used message queues include Redis RabbitMQ IronMQ (service) Amazon SQS/SNS
What are message queues?
- a datastore for messages to be consumed like a mailbox
- used for parallel processing, ensuring that a task is performed without blocking the user experience
- scalability typically think vertical (server size) or horizontal (more servers) queues allow you to scale time and smooth spikes... great for rate-limited APIs
- decoupling
What are message queues?
ProducerProducerProducerQueueConsumerConsumerConsumer
PublisherPublisherPublisherQueueSubscriberSubscriberSubscriber
Many configurations possible... two most common
Producer/Consumer producers create messages, consumers claim them when they begin working on them, release them on failure, delete on completion i.e. Parallel processing
Publisher/Subscriber publishers create messages, subscribers consume them as long as they are in the queue, message eventually expires and disappears from queue i.e. Data feeds
You already use queues.
- uploading videos that need to be encoded
- GitHub uses queues extensively... most of the work their service does is in the background
- Drupal. Several modules implement queues (i.e. Notifications, Batch API) every time you enable a module, it gets added to a queue to check for updates
In Drupal...
Queues are pluggable
Backends available on Drupal.org:
AWS SQS http://drupal.org/project/aws_sqsBeanstalkd http://drupal.org/project/beanstalkdRedis http://drupal.org/project/redisSTOMP http://drupal.org/project/stomp
(And.... still in sandbox but ready to go...)Openstack Marconi - http://drupal.org/sandbox/BrianAltenhofel/2107851
Using a Different Queue
As easy as:
Global:
$conf['queue_default_class'] = 'MarconiQueue';
Specific queue
$conf['queue_class_{queue_name}'] = 'MarconiQueue';
Queues and Drush
drush queue-list
drush queue-run [queue_name]
Queue-list will list all available queues to run
Queue-run will run them (but will always delete items) advantage over cron is that cron processes can only be run 1 at a time queue-run can be invoked as much as you want
Code.
Code.
function my_module_create_queue() { $queue = DrupalQueue::get('my_queue'); $queue->createQueue();}
Code.
function my_module_create_queue_item() { $queue = DrupalQueue::get('my_queue'); $queue_data = array( 'my_key' => 'my_value', );
return $queue->createItem($queue_data);}
Code.
function my_module_count_queue_items() { $queue = DrupalQueue::get('my_queue'); return $queue->numberOfItems();}
Code.
function my_module_process_queue_item() { $queue = DrupalQueue::get('my_queue');
$item = $queue->claimItem(30);
if ($item && $item['my_key'] == 'my_value') { $queue->deleteItem($item); } else { $queue->releaseItem($item); }}
Code.
function my_module_cron_queue_info() { $queues['my_queue'] = array( 'worker callback' => 'my_module_process_queue_item', 'time' => 60, // Time for cron to spend on this queue, not // the lease time. );
return $queues;}
Note: this is only necessary if Drupal is also acting as the consumer/subscriber. This is not necessary is Drupal is just producing or publishing messages to a queue to be consumed by something else.
It's really that simple.
Debugging
watchdog() is a Drupaler's best friend
Log everything (for both your sanity and insanity)
(probably want to ship these logs if you're processing a lot of
items)
(if you want to get started on that):
http://www.youtube.com/watch?v=p0Av29yaBEI
Monitoring
Monitor queue depths with your favorite tool (Nagios, Zabbix,
Zenoss, etc.)
In a pinch, you can get an idea with
$queue->numberOfItems().
Take action if queues get too deep
Actions: Send alert with PagerDuty... spin up new workers...
Any questions?
@BrianAltenhofel
IRC: VeggieMeat (MANY channels)