rabbitmq data ingestion at craft conf
DESCRIPTION
Presentation about RabbitMQ and FederationTRANSCRIPT
![Page 1: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/1.jpg)
Building a Distributed Data Ingestion System
with RabbitMQAlvaro Videla - RabbitMQ
![Page 2: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/2.jpg)
Alvaro Videla
• Developer Advocate at Pivotal / RabbitMQ!
• Co-Author of RabbitMQ in Action!
• Creator of the RabbitMQ Simulator!
• Blogs about RabbitMQ Internals: http://videlalvaro.github.io/internals.html!
• @old_sound — [email protected] — github.com/videlalvaro
![Page 4: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/4.jpg)
About this Talk
• Exploratory Talk
• A ‘what could be done’ talk instead of ‘this is how you do it’
![Page 5: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/5.jpg)
Agenda
• Intro to RabbitMQ
• The Problem
• Solution Proposal
• Improvements
![Page 6: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/6.jpg)
Intermission
![Page 7: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/7.jpg)
Intermission
History Lessons
![Page 8: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/8.jpg)
The Hungarian Connection
![Page 9: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/9.jpg)
The Hungarian Connection
• I’m from Uruguay
![Page 10: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/10.jpg)
The Hungarian Connection
• I’m from Uruguay
• I live in Switzerland
![Page 11: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/11.jpg)
The Hungarian Connection
• I’m from Uruguay
• I live in Switzerland
• I’m in Hungary right now
![Page 12: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/12.jpg)
WHAT?
![Page 13: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/13.jpg)
The Hungarian Connection
![Page 14: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/14.jpg)
The Hungarian Connection
Switzerland World Cup - 1954
![Page 15: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/15.jpg)
The Hungarian Connection
Switzerland World Cup - 1954
![Page 16: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/16.jpg)
The Hungarian Connection
Switzerland World Cup - 1954
Hungary 4 - Uruguay 2
![Page 17: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/17.jpg)
The Hungarian Connection II
Tivadar Puskás
Telephone Switch Inventor
![Page 18: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/18.jpg)
The Hungarian Connection III
Old Hungarian Alphabet
![Page 19: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/19.jpg)
The Hungarian Connection III
Erlang
![Page 20: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/20.jpg)
The Hungarian Connection III
As Neumann János Lajos said
![Page 21: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/21.jpg)
The Hungarian Connection III
As Neumann János Lajos said
Use Erlang
![Page 22: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/22.jpg)
What is RabbitMQ
![Page 23: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/23.jpg)
RabbitMQ
![Page 24: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/24.jpg)
RabbitMQ
![Page 25: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/25.jpg)
RabbitMQ• Multi Protocol Messaging Server
![Page 26: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/26.jpg)
• Multi Protocol Messaging Server!• Open Source (MPL)
RabbitMQ
![Page 27: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/27.jpg)
• Multi Protocol Messaging Server!• Open Source (MPL)!• Polyglot
RabbitMQ
![Page 28: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/28.jpg)
• Multi Protocol Messaging Server!• Open Source (MPL)!• Polyglot!• Written in Erlang/OTP
RabbitMQ
![Page 29: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/29.jpg)
Multi Protocol
http://bit.ly/rmq-protocols
![Page 30: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/30.jpg)
http://www.rabbitmq.com/community-plugins.html
Community Plugins
![Page 31: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/31.jpg)
Polyglot
![Page 32: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/32.jpg)
Polyglot
![Page 33: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/33.jpg)
Polyglot
• Java
![Page 34: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/34.jpg)
Polyglot
• Java!• node.js
![Page 35: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/35.jpg)
Polyglot
• Java!• node.js!• Erlang
![Page 36: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/36.jpg)
Polyglot
• Java!• node.js!• Erlang!• PHP
![Page 37: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/37.jpg)
Polyglot
• Java!• node.js!• Erlang!• PHP!• Ruby
![Page 38: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/38.jpg)
Polyglot
• Java!• node.js!• Erlang!• PHP!• Ruby!• .Net
![Page 39: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/39.jpg)
Polyglot
• Java!• node.js!• Erlang!• PHP!• Ruby!• .Net!• Haskell
![Page 40: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/40.jpg)
Polyglot
Even COBOL!!!11
![Page 41: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/41.jpg)
Some users of RabbitMQ
![Page 42: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/42.jpg)
Some users of RabbitMQ
![Page 43: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/43.jpg)
• Instagram!• Indeed.com
Some users of RabbitMQ
![Page 44: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/44.jpg)
• Instagram!• Indeed.com!• Telefonica
Some users of RabbitMQ
![Page 45: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/45.jpg)
• Instagram!• Indeed.com!• Telefonica!• Mercado Libre
Some users of RabbitMQ
![Page 46: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/46.jpg)
• Instagram!• Indeed.com!• Telefonica!• Mercado Libre!• NHS
Some users of RabbitMQ
![Page 47: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/47.jpg)
• Instagram!• Indeed.com!• Telefonica!• Mercado Libre!• NHS!• Mozilla
Some users of RabbitMQ
![Page 48: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/48.jpg)
The New York Times on RabbitMQ
This architecture - Fabrik - has dozens of RabbitMQ instances spread across 6 AWS zones in Oregon and Dublin.
Upon launch today, the system autoscaled to ~500,000 users. Connection times remained flat at ~200ms.
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2014-January/032943.html
![Page 49: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/49.jpg)
http://www.rabbitmq.com/download.html
Unix - Mac - Windows
![Page 50: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/50.jpg)
Messaging with RabbitMQ
A demo with the RabbitMQ Simulator
https://github.com/RabbitMQSimulator/RabbitMQSimulator
![Page 52: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/52.jpg)
RabbitMQ Simulator
![Page 53: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/53.jpg)
The Problem
![Page 54: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/54.jpg)
Distributed Application
App
App
App
App
![Page 55: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/55.jpg)
Distributed Application
App
App
App
App
![Page 56: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/56.jpg)
ConnectionFactory factory = new ConnectionFactory(); factory.setHost("localhost"); !
Connection connection = factory.newConnection(); !
Channel channel = connection.createChannel();
Data ProducerObtain a Channel
![Page 57: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/57.jpg)
channel.exchangeDeclare(EXCHANGE_NAME, "direct", true);
Data Producer
Declare an Exchange
![Page 58: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/58.jpg)
String message = "Hello Federation!"; channel.basicPublish(EXCHANGE_NAME, "", null, message.getBytes());
Data Producer
Publish a message
![Page 59: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/59.jpg)
ConnectionFactory factory = new ConnectionFactory(); factory.setHost("localhost"); !
Connection connection = factory.newConnection(); !
Channel channel = connection.createChannel();
Data ConsumerObtain a Channel
![Page 60: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/60.jpg)
channel.queueDeclare(QUEUE_NAME, true, false, false, null); channel.exchangeDeclare(EXCHANGE_NAME, "direct", true); channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, "");
Data ConsumerDeclare Queue and bind it
![Page 61: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/61.jpg)
QueueingConsumer consumer = new QueueingConsumer(channel); channel.basicConsume(QUEUE_NAME, false, consumer);
Data ConsumerStart a consumer
![Page 62: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/62.jpg)
while (true) { QueueingConsumer.Delivery delivery = consumer.nextDelivery(); String message = new String(delivery.getBody()); System.out.println("Received '" + message + "'"); !
channel.basicAck( delivery.getEnvelope(). getDeliveryTag(), false); }
Data ConsumerProcess messages
![Page 63: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/63.jpg)
Ad-hoc solution
![Page 64: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/64.jpg)
A process that replicates data to the remote server
![Page 65: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/65.jpg)
Possible issues• Remote server is offline
• Prevent unbounded local buffers
• Prevent message loss
• Prevent unnecessary message replication
• No need for those messages on remote server
• Messages that became stale
![Page 66: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/66.jpg)
Can we do better?
![Page 67: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/67.jpg)
RabbitMQ Federation
![Page 68: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/68.jpg)
RabbitMQ Federation• Supports replication across different administrative domains
• Supports mix of Erlang and RabbitMQ versions
• Supports Network Partitions
• Specificity - not everything has to be federated
![Page 69: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/69.jpg)
RabbitMQ Federation
![Page 70: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/70.jpg)
RabbitMQ Federation
![Page 71: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/71.jpg)
RabbitMQ Federation
![Page 72: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/72.jpg)
RabbitMQ Federation
• It’s a RabbitMQ Plugin
![Page 73: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/73.jpg)
RabbitMQ Federation
• It’s a RabbitMQ Plugin
• Internally uses Queues and Exchanges Decorators
![Page 74: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/74.jpg)
RabbitMQ Federation
• It’s a RabbitMQ Plugin
• Internally uses Queues and Exchanges Decorators
• Managed using Parameters and Policies
![Page 75: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/75.jpg)
Enabling the Plugin
rabbitmq-plugins enable rabbitmq_federation
![Page 76: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/76.jpg)
Enabling the Plugin
rabbitmq-plugins enable rabbitmq_federation
rabbitmq-plugins enable rabbitmq_federation_management
![Page 77: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/77.jpg)
Federating an Exchange
rabbitmqctl set_parameter federation-upstream my-upstream \ ‘{“uri":"amqp://server-name","expires":3600000}'
![Page 78: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/78.jpg)
Federating an Exchange
rabbitmqctl set_parameter federation-upstream my-upstream \ ‘{“uri":"amqp://server-name","expires":3600000}' !
rabbitmqctl set_policy --apply-to exchanges federate-me "^amq\." \ '{"federation-upstream-set":"all"}'
![Page 79: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/79.jpg)
Federating an Exchange
![Page 80: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/80.jpg)
Configuring Federation
![Page 81: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/81.jpg)
Config Optionsrabbitmqctl set_parameter federation-upstream \ name ‘json-object’
![Page 82: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/82.jpg)
Config Optionsrabbitmqctl set_parameter federation-upstream \ name ‘json-object’ !
json-object: { ‘uri’: ‘amqp://server-name/’, ‘prefetch-count’: 1000, ‘reconnect-delay’: 1, ‘ack-mode’: on-confirm }
http://www.rabbitmq.com/federation-reference.html
![Page 83: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/83.jpg)
Prevent unbound buffers
expires: N // ms. message-ttl: N // ms.
![Page 84: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/84.jpg)
Prevent message forwarding
max-hops: N
![Page 85: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/85.jpg)
Speed vs No Message Loss
ack-mode: on-confirm ack-mode: on-publish
ack-mode: no-ack
![Page 86: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/86.jpg)
AMQP URI:
amqp://user:pass@host:10000/vhost
http://www.rabbitmq.com/uri-spec.html
![Page 87: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/87.jpg)
Config can be applied via
• CLI using rabbitmqctl
• HTTP API
• RabbitMQ Management Interface
![Page 88: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/88.jpg)
RabbitMQ Federation
![Page 89: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/89.jpg)
Scaling the Setup
![Page 90: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/90.jpg)
The Problem
![Page 91: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/91.jpg)
The Problem
• Queues contents live in the node where the Queue was declared
![Page 92: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/92.jpg)
The Problem
• Queues contents live in the node where the Queue was declared
• A cluster can access the queue from every connected node
![Page 93: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/93.jpg)
The Problem
• Queues contents live in the node where the Queue was declared
• A cluster can access the queue from every connected node
• Queues are an Erlang process (tied to one core)
![Page 94: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/94.jpg)
The Problem
• Queues contents live in the node where the Queue was declared
• A cluster can access the queue from every connected node
• Queues are an Erlang process (tied to one core)
• Adding more nodes doesn’t really help
![Page 95: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/95.jpg)
Enter Sharded Queues
![Page 96: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/96.jpg)
Enter Sharded Queues
![Page 97: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/97.jpg)
Pieces of the Puzzle
• modulo hash exchange (consistent hash works as well)
• good ol’ queues
![Page 98: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/98.jpg)
Sharded Queues
![Page 99: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/99.jpg)
Sharded Queues
![Page 100: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/100.jpg)
Sharded Queues
![Page 101: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/101.jpg)
Sharded Queues
![Page 102: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/102.jpg)
Sharded Queues
• Declare Queues with name: nodename.queuename.index
![Page 103: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/103.jpg)
Sharded Queues
• Declare Queues with name: nodename.queuename.index
• Bind the queues to a consistent hash exchange
![Page 104: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/104.jpg)
Sharded Queues
• Declare Queues with name: nodename.queuename.index
• Bind the queues to a partitioner exchange
• Transparent to the consumer (virtual queue name)
![Page 105: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/105.jpg)
We need more scale!
![Page 106: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/106.jpg)
Federated Queues
![Page 107: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/107.jpg)
Federated Queues
• Load-balance messages across federated queues
• Only moves messages when needed
![Page 108: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/108.jpg)
Federating a Queue
rabbitmqctl set_parameter federation-upstream my-upstream \ ‘{“uri":"amqp://server-name","expires":3600000}'
![Page 109: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/109.jpg)
Federating a Queue
rabbitmqctl set_parameter federation-upstream my-upstream \ ‘{“uri":"amqp://server-name","expires":3600000}' !
rabbitmqctl set_policy --apply-to queues federate-me "^images\." \ '{"federation-upstream-set":"all"}'
![Page 110: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/110.jpg)
With RabbitMQ we can
![Page 111: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/111.jpg)
With RabbitMQ we can
• Ingest data using various protocols: AMQP, MQTT and STOMP
![Page 112: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/112.jpg)
With RabbitMQ we can
• Ingest data using various protocols: AMQP, MQTT and STOMP
• Distribute that data globally using Federation
![Page 113: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/113.jpg)
With RabbitMQ we can
• Ingest data using various protocols: AMQP, MQTT and STOMP
• Distribute that data globally using Federation
• Scale up using Sharding
![Page 114: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/114.jpg)
With RabbitMQ we can
• Ingest data using various protocols: AMQP, MQTT and STOMP
• Distribute that data globally using Federation
• Scale up using Sharding
• Load balance consumers with Federated Queues
![Page 115: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/115.jpg)
Credits
world map: wikipedia.org
federation diagrams: rabbitmq.com
![Page 116: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/116.jpg)
Questions?
![Page 117: RabbitMQ Data Ingestion at Craft Conf](https://reader037.vdocument.in/reader037/viewer/2022100309/554fb045b4c90586258b50c7/html5/thumbnails/117.jpg)
ThanksAlvaro Videla - @old_sound