messaging standards and systems - amqp & rabbitmq
DESCRIPTION
All Things Open 2014 - Day 1 Wednesday, October 22nd, 2014 Gavin Roy VP of Architecture for AWeber Communications Back Dev Messaging Standards and Systems - AMQP & RabbitMQ Find more by Gavin here: https://speakerdeck.com/gmrTRANSCRIPT
![Page 1: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/1.jpg)
Messaging Standards and Systems
AMQP & RabbitMQGavin M. RoyVP of ArchitectureAWeber CommunicationsTwitter: @Crad
All Things Open October 22, 2014
![Page 2: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/2.jpg)
About MeVP of Architecture
AWeber Communications
Blame me for pika, rabbitpy, pamqp, and a handful of
RabbitMQ plugins
Blog: https://gavinroy.com
Github: https://github.com/gmr
Book: http://manning.com/roy
![Page 3: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/3.jpg)
Advanced Message Queueing Protocol
❖ Open Standard
❖ Platform and Vendor Neutral
❖ Multiple Versions
![Page 4: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/4.jpg)
AMQP Timeline
2003
2005
2006
2008
2011Initial Specification
Working Group formed
AMQP 0-8
AMQP 0-9-1
OASISAMQP 1.0
![Page 5: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/5.jpg)
AMQP Working Group❖ JPMorgan Chase
❖ Cisco Systems
❖ IONA Technologies
❖ iMatrix
❖ RedHat
❖ TWIST
❖ Bank of America
❖ Barclays
❖ Credit Suisse
❖ Deutsche Börse Systems
❖ Goldman Sachs
❖ HCL Technologies
❖ Progress Software
❖ IIT Software
❖ INETCO Systems Ltd.
❖ Informatica Corporation
❖ Microsoft Corporation
❖ my-Channels
❖ Novell
❖ Solace Systems
❖ Tervala, Inc.
❖ VMWare
❖ WSO2
![Page 6: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/6.jpg)
Competing AMQP Standards
≠0-8&
0-9-11.0
![Page 7: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/7.jpg)
AMQP 0-9-1
❖ Currently has wider support than AMQP 1.0
❖ Multiple Broker Implementations: RabbitMQ, Apache Qpid, and SwiftMQ to name a few
❖ Specifies a Model and Protocol
![Page 8: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/8.jpg)
Native AMQP 0-8/0-9-1 Clients
C
Clojure
Cobol
Common Lisp
Delphi
Erlang
Go
Groovy
Haskell
Java
JavaScript
.NET
OCaml
Perl
PHP
Python
Ruby
Scala
![Page 9: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/9.jpg)
❖ Exchange Receives and route messages
❖ Message Queue Stores messages until they can be consumed
❖ Binding Defines the relationship between an Exchange and Queue and provides routing criteria
Advanced Message Queueing Model
X
Exchange
BindingMessage Queue
![Page 10: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/10.jpg)
Routing Keys
❖ Provided when publishing a message
❖ Compared against binding keys by exchanges
❖ Ideally provide context to the message:
❖ Connote the type of the message
❖ Categorize the content in the message
❖ Specify the type of consumer that should receive it
![Page 11: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/11.jpg)
Advanced Message Queueing Protocol
❖ Compact, binary frame format wire protocol
❖ Bi-directional RPC
❖ Commands consist of Classes and Methods:
❖ Example Request: Queue.Declare(name=“foo”)
❖ Response: Queue.DeclareOk(messages=0, consumers=0)
![Page 12: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/12.jpg)
Common AMQP Terms
❖ Broker A server that implements AMQP
❖ Producer or Publisher A client application that sends messages to a broker
❖ Consumer A client application that reads messages from a queue
![Page 13: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/13.jpg)
Publishers and Consumers
CP
![Page 14: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/14.jpg)
Multiple Publishers
CP
P
P
![Page 15: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/15.jpg)
Multiple Consumers
C
C
C
P
![Page 16: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/16.jpg)
AMQP Messages
❖ Comprised of 3 or more frames:
❖ Method FrameBasic.Publish, Basic.Deliver, etc
❖ Content Header with body size & message properties timestamp, message-id, app-id, etc
❖ n Body Frames with the opaque message payload
![Page 17: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/17.jpg)
AMQP 0-9-1 Issues & Gotchas
❖ Ambiguous
❖ Authentication
❖ Asynchronous
❖ Connection Negotiation
❖ Exceptions!
![Page 18: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/18.jpg)
RabbitMQ
❖ Open Source (MPL)
❖ Written in Erlang/OTP
❖ Developed/Maintained by Pivotal
❖ Multi-Protocol AMQP 0-9-1 & 1.0, MQTT, STOMP, XMPP, HTTP, Web-STOMP & More
❖ Roots in AMQP 0-8/0-9-1
![Page 19: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/19.jpg)
Who Uses It?Agora Games
Chef
Google AdMob
MeetMe
Mercado Libre
Mozilla
NASA
New York Times
National Science Foundation
Openstack
Rapportive
Soundcloud
(and many more)
![Page 20: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/20.jpg)
Why Use RabbitMQ?
❖ Create loosely-coupled applications
❖ Communicate across applications or platforms
❖ Tap into pre-existing message flows for new purposes
❖ Scale-out clustering for growth, throughput, and HA
❖ Federation for WAN latencies and network partitions
❖ Extensible plugin-in architecture
![Page 21: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/21.jpg)
RabbitMQ Extensions to AMQPAuthentication Failure
Exchange to Exchange Bindings
Delivery Confirmations
Basic.Nack
Consumer cancellations
Consumer priorities
Dead Letter Exchanges
Alternate Exchanges
Connection blocking
Message CC & BCC Routing
Queue Length Limits
Per Queue Message TTL
Per Message TTL
Queue TTL
Message User ID Validation
Auto-delete exchanges
![Page 22: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/22.jpg)
RabbitMQ Clustering
❖ LAN Only
❖ Adds highly-available queues
❖ Is cohesive, publish and consume from any node
❖ Leverages native Erlang clustering and communication
❖ Has multiple strategies for dealing with network partitions
❖ Manually configured via configuration or command line*
![Page 23: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/23.jpg)
RabbitMQ Pluginsauth-backend-amqp Use AMQP to respond to RabbitMQ auth requests
auth-backend-http Use a custom web service to provide authentication
autocluster-consul Automatically create clusters using Consul
influxdb-storage-exchange Store JSON messages as events in InfluxDB
presence-exchange Publishes messages upon binding changes
rabbitmq-toke Tokyo Cabinet backing store for queue messages
rabbitmq-topTop like view of RabbitMQ processes in the management UI
sharding Scale out RabbitMQ with automatic queue sharding
(and many more)
![Page 24: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/24.jpg)
Message Routing
![Page 25: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/25.jpg)
Built-In Exchange Types❖ Direct
String matching on the routing key
❖ FanoutNo routing key, messages delivered to all bound queues
❖ TopicPattern matching in the routing key
❖ HeadersNo routing key, value matching in the headers property
![Page 26: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/26.jpg)
Topic Exchange Binding KeysRouting Key: namespace.delimited.keys
#Receive all messages
namespace.# Receive all messages in namespace
namespace.delimited.* Receive all namespace.delimited messages
namespace.*.keys Receive all namespace messages ending with keys
![Page 27: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/27.jpg)
Exchange to Exchange Binding
X
X
X
Queue
Queue
Queue
Queue
Queue
Queue
Messages published into a topic exchange are routed to two other exchanges and a queue
Messages published into the consistent-hashing exchange are distributed amongst three queues
Messages published into a direct exchange routed to two queues
P
![Page 28: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/28.jpg)
Alternate Exchanges
P X
UnroutableMessagesX
Unroutable messages are sent to an alternate exchange
![Page 29: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/29.jpg)
Dead Letter Exchanges
P X
Reject Queue
Queue C
X
Consumer rejects the message without requeue
Dead-Letter Exchange routes the message
![Page 30: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/30.jpg)
Exchange PluginsConsistent Hashing Distribute messages via hashed value of routing key
Event Publishes messages on AMQP events such as queue creation
Random Distribute messages across all bound queues randomly
PostgreSQL LISTEN Subscribes to and publishes PostgreSQL notifications
Recent History Sends the last n messages to any newly bound queue
Reverse TopicAllows for routing patterns at publish time and not via binding
Riak StorageStores messages published through the exchange into Riak
Script Exchange Calls out to external scripts for message routing
![Page 31: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/31.jpg)
Performance Considerations
![Page 32: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/32.jpg)
Publishing Performance Scale
Persisted Messages
Transactions
HA Queues
Publisher confirms
No guarantees
Notification on failure
Alternate exchanges
HA Queues w/ Transactions
![Page 33: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/33.jpg)
Persisted Messages IO Model
Disk
Queue
Message Queue
Queue
Messages with delivery-mode 2 sent into RabbitMQ are persisted to disk
Pointers to the message are placed in the queue data structures
When the message is no longer in a queue, it is removed from disk
![Page 34: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/34.jpg)
HA Queues & Performance
Queue
Queue
Queue
P
RabbitMQ Cluster
A publisher sends to any node in a RabbitMQ cluster
The RabbitMQ servers in the cluster synchronize the state of the message in the queue
The message that was published is put in the queue and is stored on each server
![Page 35: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/35.jpg)
Consumer Performance Scale
Getting Messages
Consuming and using Transactions
Consume with Acknowledgements
Consuming with "No Ack Mode" enabled
Consuming with Acknowledgements and QoS > 1
![Page 36: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/36.jpg)
Operational Concerns
![Page 37: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/37.jpg)
Trending & Monitoring
❖ RabbitMQ Management Plugin provides internal stats
❖ Queue depths, connection counts, throughput, memory usage, etc
❖ Monitor with common tools such as Nagios or Sensu to services such as Boundary and NewRelic
❖ Stream based monitoring with Riemann for anomaly detection
![Page 38: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/38.jpg)
Thoughts on Configuration❖ Use configuration management!
❖ Even for exchanges, queues, and bindings if possible
❖ Helpful for disaster recovery
❖ Use RabbitMQ’s Policies when possible
❖ Exchanges and queues are immutable
❖ Deleting and redeclaring for changes can be disruptive
![Page 39: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/39.jpg)
Sample Usage Patterns
![Page 40: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/40.jpg)
Evolve Tightly-Coupled Applications
Application DB
![Page 41: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/41.jpg)
Decoupling Database Writes
Application C DB
![Page 42: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/42.jpg)
Listen for Database Notifications
X Queue
![Page 43: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/43.jpg)
Multi-Purposed Messages
C
C
C
P
Queue
DB
Cloud
GraphiteQueue
Queue
X
![Page 44: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/44.jpg)
RPC
C / PC / P
P / C
C / PResponse Queue
X
X
WorkerQueue
Front-end web app receives a request
Internet
App publishes RPC request
One of the subscribed worker consumers receives the RPC requestAfter processing the request,
the worker publishes the replyApp receives the reply from the response queue
App replies to client
![Page 45: Messaging Standards and Systems - AMQP & RabbitMQ](https://reader036.vdocument.in/reader036/viewer/2022062308/559af3991a28abbd708b473a/html5/thumbnails/45.jpg)
Delayed Messages
WARNING: This pattern is a hack and can cause undesirable situations should the wait queue overwhelm the system!
P X
X
Wait Queue
Ready Queue C
The wait queue is declared with a message TTL and a dead-letter exchangeMessages that will be delivered
later are routed to a wait queue
When a message times out due to the TTL, it is published to the DLX
The internally re-published message is routed to the queue with active consumers