scaling rabbitmq to 11
TRANSCRIPT
Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQto 11
Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQto 11
INTERACTIVE POLLS• http://pollev.com/gmr
• SMS: 22333
2
Gavin M. Roy @crad - Surge 2011
myYearbook.com• #1 Comscore Teen Destination, Top Trafficked Sites in US
• Use of RabbitMQ dates back to 2009
• Multiple clusters for different purposes
• Browser <-> Backend Message Bus
• Messaging, Event Processing
3
Gavin M. Roy @crad - Surge 2011
DISASTER PORN
Gavin M. Roy @crad - Surge 2011
ABOUT AMPQ• Platform, Vendor Neutral Messaging format
• 0.9.1 finalized in November 2008
• JP Morgan Chase, Red Hat, Rabbit Technologies, iMatrix,
IONA Technologies, Cisco Systems, Envoy Technologies,
Twist Process Innovations
• Members now include Bank of America, Barclays Banks,
Microsoft Corporation, Novell, VMWare
• 1.0 Final: October 2011
5
Gavin M. Roy @crad - Surge 2011
ABOUT RABBITMQ• Born in 2006, Released in 2007, Acquired in 2010
• Written in Erlang/OTP
• Talks AMQP 0.8 and 0.9.1
• Features include:
• Clustering, Active/Active HA Queues
• Management UI and API
• Plugin Architecture
6
Gavin M. Roy @crad - Surge 2011
ABOUT YOU & RABBITMQ
Gavin M. Roy @crad - Surge 2011
RABBITMQ& AMQPCONCEPTS
Gavin M. Roy @crad - Surge 2011
PUBLISHERS
Gavin M. Roy @crad - Surge 2011
CONSUMERS
Gavin M. Roy @crad - Surge 2011
QUEUES• Durability: Queue survives reboot
• Auto-Delete: Delete the queue when consumer goes away
• Exclusive: Only one consumer may consume
• x-expires: Auto-Delete after given duration
• x-message-ttl: Auto-discard a message after ttl
11
Gavin M. Roy @crad - Surge 2011
MESSAGES• Have client specified properties:
• Content type, Encoding, Timestamp, App-Id, User-Id,
Headers
• Delivery Mode:
• 1: Non-Persistent
• 2: Persistent
12
Gavin M. Roy @crad - Surge 2011
MESSAGE ROUTINGIt’s not just about Queues.
Gavin M. Roy @crad - Surge 2011
ROUTING KEYIn addition to the exchange, determines what is done with a
message.
Gavin M. Roy @crad - Surge 2011
CC & BCCMore routing options per message, but violates AMQP.
Gavin M. Roy @crad - Surge 2011
HEADERS EXCHANGEMore on this later.
Gavin M. Roy @crad - Surge 2011
EXCHANGES
Gavin M. Roy @crad - Surge 2011
FANOUT• 1:N Message Delivery Pattern
• No Routing Keys Involved
• Queues bound to a fanout exchange get all messages sent to
the exchange
18
Gavin M. Roy @crad - Surge 2011
DIRECT• 1:1 Message Delivery Pattern
• Routing Keys are direct, no wildcarding
• All queues bound to a routing key get the message
19
Gavin M. Roy @crad - Surge 2011
TOPIC EXCHANGES• Pattern Matching in Routing Keys
• Publish one message type to Surge.Sessions, another to
Surge.Mentions
• Period delimiter, * stays within the period scope, # does not
• Example patterns: *.Mentions, Surge.*, #
• Slower than Fanout and Direct
20
Gavin M. Roy @crad - Surge 2011
HEADERS EXCHANGES• Matches on Basic.Properties headers table values to the
arguments table specified when binding a queue
• Use x-match when declaring exchange to specify any or all
headers must match
• Slow implementation in Erlang
21
Gavin M. Roy @crad - Surge 2011
PLUGIN EXCHANGES• Consistent Hash Exchange: Round-robin distribution to queues
• External: RPC Exchange Plugin
• Script: Similar in intent to External
• Riak
• Last-Value Cache
• Recent History Exchange: Keeps last 20 messages routed
• Global Fanout: Sends to all queues, regardless of routing key
22
Gavin M. Roy @crad - Surge 2011
ONE MORE THING...Exchange-to-Exchange Bindings
Gavin M. Roy @crad - Surge 2011
CLUSTERS
Gavin M. Roy @crad - Surge 2011
STATS, DISK AND RAM NODES
Gavin M. Roy @crad - Surge 2011
WHERE DOES THE QUEUE LIVE?
Gavin M. Roy @crad - Surge 2011
CROSS-NODE PUBLISHING & CONSUMING
Gavin M. Roy @crad - Surge 2011
MAX CLUSTER SIZE~32 notes is a realistic maximum node count for a cluster
Gavin M. Roy @crad - Surge 2011
SCALING ISSUES?
Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQto 11
Gavin M. Roy @crad - Surge 2011
WHAT TO WATCH
Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQto 11
RABBITMQ
Y U NO TAKE MESSAGE?
Gavin M. Roy @crad - Surge 2011
TCP BACKPRESSUREMUCH?
Gavin M. Roy @crad - Surge 2011
MEMORY MANAGEMENT
Gavin M. Roy @crad - Surge 2011
[{RABBIT, [{VM_MEMORY_HIGH_WATERMARK, 0.4}]}].
40%
Gavin M. Roy @crad - Surge 2011
MESSAGE PERSISTENCE50-70% performance penalty compared to non-persisted
messages. * http://www.rabbitmq.com/faq.html#performance-persistent
Gavin M. Roy @crad - Surge 2011
CONSUMINGAck, Nack and Reject vs Basic.Consume(no_ack=True)
Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQto 11
Server Client
Basic.DeliverBasic.[Ack/Nack/Reject]
Basic.[Ack/Nack/Reject]Ok
Gavin M. Roy @crad - Surge 2011
Gavin M. Roy @crad - Surge 2011
BASIC.QOS
• Prefetch Size and Count
• How many messages the
client wants the RabbitMQ
to pre-deliver to the client
• Doesn’t work in no_ack
Gavin M. Roy @crad - Surge 2011
TOO MANY SHORT LIVED CONNECTIONS
Gavin M. Roy @crad - Surge 2011
APACHE + MOD_PHP
Gavin M. Roy @crad - Surge 2011
CONNECTION PERSISTENCE
Gavin M. Roy @crad - Surge 2011
100 SERVERS125 BACKENDS1 REQUEST/S12,500 CONN/S
Gavin M. Roy @crad - Surge 2011
AMQP CONNECTION SEQUENCE
Client Server
Protocol Header FrameConnection.Start
Connection.StartOk
Connection.TuneOkConnection.Tune
Connection.OpenOkConnection.Open
Channel.OpenChannel.OpenOk
Gavin M. Roy @crad - Surge 2011
685 BYTES PER8.16 MB/S IN STARTUP @12,500 CONN/S
Gavin M. Roy @crad - Surge 2011
VORPAL BUNNY
Gavin M. Roy @crad - Surge 2011
VORPAL BUNNY• Light-weight PHP client for RabbitMQ
• Used the experimental JSON-RPC Channel plugin
• Handles the overhead of the HTTP, JSON-RPC to AMQP
analog
• Fire and forget message publishing
• Can not be transparently load balanced
48
Gavin M. Roy @crad - Surge 2011
TOO MANY CONNECTIONS
Gavin M. Roy @crad - Surge 2011
VELOCITY TOO HIGH FOR ONE NODE?
Gavin M. Roy @crad - Surge 2011
MEMORYUTILIZATION DISPROPORTIONATE TO QUEUE DEPTHS
Gavin M. Roy @crad - Surge 2011
CLUSTERNODE FAILURES
Gavin M. Roy @crad - Surge 2011
UP TO 60 SECONDS TO DETECT
Gavin M. Roy @crad - Surge 2011
MESSAGES DROPPED ON THE FLOOR*
Gavin M. Roy @crad - Surge 2011
* 2.6.0 ADDEDACTIVE/ACTIVE QUEUES IN CLUSTERS
Gavin M. Roy @crad - Surge 2011
ACTIVE/PASSIVE FAILOVER?Recommendation is to use Pacemaker and DRBD or shared
storage for Mnesia databases.
Gavin M. Roy @crad - Surge 2011
RECOVERY?
Gavin M. Roy @crad - Surge 2011
CLUSTERING ALTERNATIVES
Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQ to 11
FEDERATION
• Exchange Plugin
• Provides ability to route all
messages to a x-federation
exchange to another broker
Gavin M. Roy @crad - Surge 2011
SHOVEL PLUGIN
• From Queue on one broker to an
exchange on another broker.
• Effective for cross-cluster data
broadcasting
Gavin M. Roy @crad - Surge 2011
TRENDING & MONITORING• RabbitMQ Message Velocity Rates Gotcha
• Management API Plugin exposes RESTful API including
monitoring data points
• 2.6.1 allows monitoring tag for users to allow data access
without full admin access
• Queue.Declare(passive=True) gets queue depths as well
• Plug in to Graphite, Nagios,or your favorite tool
61
Gavin M. Roy @crad - Surge 2011
PHOTO CREDITS• “skateboard kids 7”
© 2007 Volker Neumann
http://flickr.com/prawnpie
• “methodcn contest”
©2009 Erin O’Hara
http://flickr.com/erinohara73/
• USPS Truck Accident
Valley City Fire Department
http://www.valleycityfire.com
62
Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQto 11
QUESTIONS?