2015 zendcon - do you queue

DO YOU QUEUE?ZendCon 2015

I AM MIKE WILLBANKS

• Father, Husband, Developer

• VP of Development at Packet Power

• Twitter : @mwillbanks

Task

Producer ConsumerMessages

Messages

Messages

WHAT IS A QUEUE?• Pub/Sub

• FIFO buffer

• Push / Pull

• A way to communicate between applications / systems.

• A way to decouple components.

• A way to offload work.

WHY QUEUE?

THE LONG RUNNING CASE

Request Long-process

Send ResponseResponse

THE INTEROP CASE

Request Conditioning

Service ResponseResponse

Web Service Call

Send Response

NOT YOUR DATABASE

Request Process

Response Send Response

DB

SO WHY QUEUE

• User experience

• System Security

• Load Distribution

• System Reliability

IN PRACTICEYou’ve seen this before…

WHAT YOU MIGHT QUEUE

COMMUNICATIONS

• Emails

• SMS

• Push Notifications

IMAGES

• Conversions

• Resize

• Thumbnail

• Watermark

VIDEO

• Conversion

• Resampling

• Audio overlay

IOT

• Receive messages from devices and process responses

PATTERNS

POINT TO POINT

Point to Point Channel

Receiver

Sender

PUBLISH / SUBSCRIBE

Publiser Subscribe Channel

Subscriber

Subscriber

Subscriber

MESSAGE BUSApplication

ApplicationApplication

Message Bus

PIPELINESender

Receiver

Point to Point Channel

Receiver

Point to Point Channel

INVALID MESSAGE

Channel

Receiver

Sender

X

Invalid Message Channel

PROTOCOLSOr implementations for that matter.

AMQP

• AMQP Working Group (Community and Vendor)

• Platform agnostic protocol.

• Completely open, interoperable and broadly applicable.

• Many severs available and many client libraries.

STOMP

• Simple protocol

• Behaviors follow very simple commands.

• Most message queues can communicate over STOMP.

Connect Send Disconnect

/queue/msg

PHP

STOMP

SERVER

Connect Subscribe Disconnect

/queue/msg

Read

Ack

SQS

• Fairly simple protocol

• Supports delays, timers, and multiple policies.

SPECIAL PURPOSE

Many queue implementations exist that don’t necessarily sit

under standards…

XMPP

• Best for real-time data.

• Leveraging pub/sub can turn it into more of a generic message system.

• Multiple libraries available.

SOCKET.IO

• New comer

• Real-time bidirectional event-based communication

• Largely leverages pub/sub

ZEROMQ

• The ultimate in message queue flexibility.

• Socket library that acts as a concurrency framework.

GEARMAN

• Application framework for farming out work.

• Job sever for asynchronous or synchronous messages.

BEANSTALKD

• Asynchronous Job Queue

• Supports delays

• Many PHP clients exist

CONSIDERATIONSHow do we evaluate our options…

PULL VS. PUSH

• Always PULL, whenever possible.

• Push eliminates several benefits.

DURABILITY

• Memory residence

• Persistence

• Restart survival

SECURITY

• Authentication

• Queue permissions / restrictions

DELIVERY

• Is the delivery guaranteed?

• If a message cannot be delivered how it it handled?

ROUTING

• Multiple routing scenarios

• Fanout

• Direct

• Topic

• Broadcast

BATCHING

• Do it later but in bulk (credit card processing)

• Can be done via scheduling (cron)

RECEIPT

• Do you get an acknowledgement of receipt?

IMPLEMENTING QUEUES

STARTING POINTS

PUSHING MESSAGES<?php

class UserService { public function save($user) { $this->db->save($user); $stomp = new Stomp('tcp://localhost:61613'); $stomp->send('/queue/email', [ 'to' => $user->getEmail(), 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ]); } }

HANDLING MESSAGES

<?php $stomp = new Stomp('tcp://localhost:61613'); $stomp->subscribe('/queue/email');

while (true) { if (!$stomp->hasFrame()) { sleep(2); continue ; } $stomp->readFrame(); $email = json_decode($frame->body);

mail($email->to, $email->subject, $email->message, $email->headers); }

MESSAGES

MESSAGE CONSIDERATIONS

• Message Format

• Message Contents

SERIALIZE

O:7:"Message":1:{s:7:"content";a:1:{s:3:"foo";a:1:{s:3:"bar";a:1:{i:0;s:3:"baz";}}}}

WORKERS

WORKER CONSIDERATIONS

• Should do ONE thing and ONE thing well.

• Should attempt to be as quick as possible in handling that type.

• Should be able to be scaled horizontally.

HANDLING WORKERS

• Prevent Memory Leaks

• memory_get_usage

• Handle Signals!

• pcntl_signal

ABSTRACTIONSWe want to make this easy…

<?php

interface QueueInterface { public function __construct(Stomp $stomp, $queue); public function dispatch(); public function publish(array $message); public function work(StompFrame $message); }

<?php

class AbstractQueue implements QueueInterface { protected $stomp; protected $queue; protected $signal;

public function __construct(Stomp $stomp, $queue) { $this->stomp = $stomp; $this->queue = $queue; }

protected function prepare() { if (php_sapi_name() != 'cli') { throw new RuntimeException('You cannot dispatch outside of the CLI'); }

if (function_exists('pcntl_signal')) { pcntl_signal(SIGTERM, array($this, 'signal')); pcntl_signal(SIGINT, array($this, 'signal')); pcntl_signal(SIGHUP, array($this, 'signal')); } }

protected function signal($signal) { $this->signal = $signal; }

public function dispatch() { $this->prepare();

while (true) { if ($this->signal) { break ; }

if (!$this->stomp->hasFrame()) { $this->wait(); continue ; }

$frame = $this->stomp->readFrame(); if ($this->validate($frame)) { $this->work($frame); }

$this->stomp->ack($frame); } }

protected function wait() { sleep(1); }

protected function validate(StompFrame $message) { return false; }

public function publish(array $message) { return $this->stomp->send($this->queue, json_encode($message)); }

<?php

class EmailQueue extends AbstractQueue { public function validate(StompFrame $message) { if (!array_key_exists('to', $message)) { return false; } return true; }

public function work(StompFrame $message) { $mail = json_decode($message);

mail($mail->to, $mail->subject, $mail->message); } }

BOOTSTRAPPINGLeverage your existing infrastructure as much as possible!

<?php declare(ticks=1);

include 'vendor/autoload.php'; $app = Zend\Mvc\Application::init(include 'config/application.config.php'); $sm = $app->getServiceManager();

if (!isset($argv[1])) { fprintf(STDERR, "Syntax: worker <name>\n\n"); exit(1); } $name = $argv[1];

try { echo "Starting worker: " . $name . ' as ' . get_current_user() . PHP_EOL; $consumer = $sm->get($name); $consumer->dispatch(); } catch (\Exception $e) { fprintf(STDERR, "%s\n", $msg); exit(1); } $consumer = null;

echo 'Shutdown ' . $name . ' worker gracefully.' . PHP_EOL; exit(0);

EVENTS

SERVICES TRIGGER EVENTS

<?php

use Zend\EventManager\EventManagerAwareTrait;

class UserService { use EventManagerAwareTrait;

public function save($user) { $this->db->save($user); $this->getEventManager()->trigger('save', null, ['user' => $user]); } }

ATTACH EVENTS

<?php

use Zend\ServiceManager\ServiceManager;

$sm = new ServiceManager(); $service = $sm->get('UserService'); $queue = $sm->get('EmailQueue'); $service->getEventManager()->attach('save', function($e) use ($queue) { $params = $e->getParams(); $queue->publish([ 'to' => $params['user']['email'], 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ]); });

HANDLING PROGRESS

• Keep track by using a generic handler

• Or, keep track via your database.

TOOLING

SUPERVISOR

• Daemon that runs on the server.

• Monitors programs and keeps them running in case of failure.

• Handles logging.

PAINLESS INSTALLATION

sudo easy_install supervisor sudo echo_supervisord_conf > /etc/supervisord.conf sudo service supervisor start

EXAMPLE PROGRAM CONFIGURATION

[program:emailworker] command=/usr/bin/php /var/www/worker "MyProject\Queue\Email"

process_name=%(program_name)s_%(process_num)d numprocs=2

numprocs_start=2 user=www-data autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=10 ; number of secs prog must stay running (def. 10) startretries=5 ; max # of serial start failures (default 3) log_stdout=true ; if true, log program stdout (default true) log_stderr=true ; if true, log program stderr (def false) redirect_stderr=true ; if true, redirect stderr to stdout stdout_logfile=/var/www/logs/worker-panoramaqueuekrpano.log stdout_logfile_maxbytes=10MB stdout_logfile_backups=15

SUPERVISORD MULTI SERVER MONITORING TOOL

https://github.com/mlazarov/supervisord-monitor

WHEN BAD THINGS HAPPEN

QUEUE BACKUP

• Most of all issues with queues are that a queue has backed up.

WORKER EXCEPTIONS

• Broken code causes hardships, but recoverable!

• https://pixabay.com/en/autobahn-accident-germany-car-road-837643/• https://pixabay.com/en/traffic-rent-a-car-traffic-jam-637118/• https://pixabay.com/en/airplanes-line-runway-military-713662/• https://pixabay.com/en/leo-animal-savannah-lioness-safari-350690/• https://pixabay.com/en/user-top-view-office-keyboard-154199/• https://pixabay.com/en/mechanics-engine-springs-mechanic-424130/• https://pixabay.com/en/laughter-fun-happiness-boy-child-449781/• https://pixabay.com/en/umbrellas-red-blue-patterns-205386/• https://pixabay.com/en/spot-runs-start-la-stadion-862274/• https://pixabay.com/en/artistic-the-art-of-abstraction-948588/• https://pixabay.com/en/boots-work-boots-shoes-647035/• https://pixabay.com/en/meerkat-watch-guard-cute-676944/• https://pixabay.com/en/broken-window-hole-glass-damage-960188/• https://pixabay.com/en/police-security-safety-protection-869216/• https://pixabay.com/en/parcel-package-packaging-box-575623/• https://pixabay.com/en/directory-signposts-trail-direction-494457/• https://pixabay.com/en/cookies-chocolate-chip-food-dessert-28423/• https://pixabay.com/en/phone-communication-call-select-735060/• https://pixabay.com/en/receipt-note-paper-bill-document-575750/• https://pixabay.com/en/tools-construct-craft-repair-864983/• https://pixabay.com/en/moore-oklahoma-tornado-disaster-112781/• https://pixabay.com/en/network-iot-internet-of-things-782707/• https://pixabay.com/en/mobile-phone-smartphone-app-426559/• https://pixabay.com/en/mr-site-build-crane-baukran-462074/• https://pixabay.com/en/film-projector-movie-projector-738806/• https://pixabay.com/en/calves-legs-human-standing-on-540519/• https://pixabay.com/en/notebook-pages-opened-paper-note-820078/• https://pixabay.com/en/letters-penpal-cards-leave-stack-566810/• https://pixabay.com/en/spray-household-surface-shine-315164/• https://pixabay.com/en/industry-crafts-gears-mechanical-94448/• https://pixabay.com/en/hornet-wasp-insect-sting-macro-11514/• https://pixabay.com/en/honey-bees-bees-hive-bee-hive-401238/• https://pixabay.com/en/temple-china-door-handle-840526/• https://pixabay.com/en/no-button-push-sign-icon-symbol-685042/

Image Credits

THANK YOU!http://joind.in/talk/view/15538