Working withAsynchronous Events

Jan Gregor Emge-Triebel (@jan0707)Dennis Oehme (@dennisoehme)

Garden of Concepts GmbH● Hanau / Munich

● In house CMS “.stone”

● In house CRM “Elvis”

● Various individual customer projects

● Symfony based development, supported by:

○ MongoDB, ElasticSearch, MySQL, RabbitMQ,

Redis, AngularJS, Twig, Jade, Gitlab, Docker,

Vagrant, Ansible

● Say hi @goc_mediastudio

Requests == Event

“Symfony is a request based framework.”

Fabien Potencier, Symfony Live 2014

@fabpot

Requests

Request Controller Logic 1 Logic 2 Response

Request Controller

Logic 1

Logic 2

Response

Synchronous

Asynchronous

Why do we need asynchronous event processing ?● Move long running logic

● Cut down reaction times

● Transfer load

● Microservices

● Exchange data/messages with other (sub-)systems

● Parallelization

● Scaling

Our journey● Gearman

● (PHP) Resque

○ Demo

● RabbitMQ

Gearman● Gearman Job Server (written in C)

● Runs as a Linux daemon

● Requires PECL extension for worker API

● Worker APIs exists for other Languages (C, Perl,...)

Gearman

http://gearman.org/examples/send-emails/

Gearman● Latest PECL-Extension version is 1.1.2 (August 2013)

● Latest Server-Release (spring 2014)

● No more process is being made

● Gearman daemon and extension are often incompatible

(PHP) Resque ● Is a Resque port to php:

○ Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple

queues, and processing them later.

● Needs Redis(-Cluster)

● Redis is capable of atomic transactions (FIFO)

Demo

https://github.com/Jan0707/async-symfony

(PHP) Resque ● Hardly any commits since 2012

● Last commit to Master on 2015-05-12

● chrisboulton/php-resque & michelsalib/BCCResqueBundle must be presumed

dead ?

RabbitMQ

Or why this talk should be named:“Working with Asynchronous Events in RabbitMQ”

RabbitMQ● Message Broker written in Erlang

● AMQP (Advanced Message Queuing Protocol)

● Open, but binary network protocoll

● Routing, message distribution

● Transactions

● Plus, more nice features○ TTL

○ DLX (Dead Letter eXchange)

○ PreFetching

Of publishers and consumers ...

Publisher Publish ConsumerEx-change QueueRoutes Consumes

There is a Bundle for it● https://github.com/php-amqplib/RabbitMqBundle

● Consumer and Producer as Symfony services

● Every queue Consumer runs as a Symfony command

● Configuration via app/config.yml○ Better: Create app/config/rabbitmq.yml and import in app/config/config.yml

app/config/rabbitmq.yml

Symfony Producer

Symfony Consumer

Lessons Learned: Consumer● PHP threads are not meant to be long running

○ New problems: memory leaks, ...

● Use supervisor to check and restart Consumers

● Open database connections, timeouts○ Better quit when inactive

● Cap maximum runtime per thread○ Automatically shutdown after processign x messages

● Memory limit: RabbitMQ-Bundle utilizes “soft-quota”

● Booting consumers takes rather long

● Prefetching (Warning: Order!)

Lessons Learned: Message● Compact/short messages

● Queues eat up a lot of storage / RAM

● Common format such as JSON or XML○ Do not serialize PHP objects or arrays

○ Ensure interoperability with other systems and languages

Lessons Learned: Message Transformation● Use JMS/Serializer to convert objects to JSON

○ Deserialize JSON to PHP objects in consumer

○ Objects can and should be validated (with symfony forms and asserts)

○ Reload entities / documents to keep them up to date (doctrine/unit of work)

● Alternative approach: PHPs JsonSerializable Interface

Lessons Learned: Message Header● Use message “Header” for content-type and version

○ Message formats can change

○ consumers must remain compatible

● Unqiue IDs make logging / debugging / live easier

● Other meta data:○ timestamp

○ user / session

○ App id

○ ...

Tip: Custom Consumer-/Producer-Classes● Message transformation

● Events

● Logging

Transactions● Messages must be brokered to exactly one Consumer

● Consumer must explicitly answer a message with “acknowledge”

● If not, the message will be marked “rejected” and be requeued○ Beware of requeuing:

■ Can lead to toxic messages

■ Can change the order of messages

○ Beware of final rejection:

■ Can lead to loss of data

Toxic Messages● If a message is not explicitly being answered, it will be requeued.

● If that happens multiple times :

● Congratulations on your infinite loop

● Congratulations on your infinite loop

● Congratulations on your infinite loop

● Congratulations on your infinite loop

● Congratulations on your infinite loop

● Congratulations on your infinite loop

● Congratulations on your infinite loop

● Congratulations on your infinite loop

● Congratulations on your infinite loop

Dead Letter Exchange● RabbitMQ comes with a Dead Letter Exchange (DLX)

● Every queue determines its own or shared DLX

● DLX is a separate queue, storing all rejected messages

○ (Reject, TTL, Prefetch)

app/config/rabbitmq.yml

Signal Handling● Graceful Shutdown

● pcntl_signal / _dispatch

● SIGTERM

● FATAL ERROR abfangen

● Logging

● Toxic Message-Handling

Shutdown

Deployment● Consumers need an up to date code base

● Consumers need a graceful shutdown

● Consumers must only be shutdown when idling○ Otherwise : Loss of message / data

Other challenges● Significant change of infrastructure

○ Choose your libraries and dependencies carefully!

● Moving load is not removing load

● Rethinking necessary○ Display of status / intermediate result

○ Websockets or Push instead of Polling

● Interdependencies of messages○ Especially when running parallel consumers!

● Performance (Doctrine ORM / ODM)

Questions ?Thank you :)

Find this and other talks at: https://www.gardenofconcepts.com/talks/

Feedback is always welcome

& please rate our talk !