white area lecture – messaging - cern wa 06-11-2013 - messaging.pdf · messaging software...

White Area Lecture – Messaging

Lionel Cons IT/SDC

6 November 2013

Part 1: what is messaging?

L. Cons – 6 Nov 2013 White Areas Lecture – Messaging 2

What is messaging?

§  “Message-oriented middleware (MOM) is software or hardware infrastructure supporting sending and receiving messages between distributed systems.” (source: Wikipedia)

§  Messaging is for software components what electronic mail is for people.


High-level overview

§  Information production and consumption can be decoupled (asynchronous) and use different: §  hosts (and operating systems) §  programming languages §  network protocols §  time


What is a message?

§  A message has a body and a header. §  The body (aka data):

§  is a byte string §  is opaque and immutable §  can contain JSON, XML, Thrift, protobuf…

§  The header (aka meta-data): §  is made of key/value pairs §  is also used for routing

§  This is very similar to electronic mail.


Messaging buzzwords

§  Concepts: acknowledgement, binding, body, channel, credit, encoding, exchange, exclusive, consumer, header, last value queue, link, message, node, persistence, queue, selector, session, (durable) subscription, topic, transaction, virtual destination…

§  API: JMS §  Protocols: AMQP, MQTT, OpenWire, STOMP,

XMPP… §  Software: ActiveMQ, Apollo, HornetQ, Qpid, MRG,

RabbitMQ, ZeroMQ…


JMS – Java Message Service

§  is a messaging standard that allows application components to create, send, receive, and read messages

§  is mature (1.1 spec. in 2002) and part of J2EE §  is (mainly) an API §  supports (mainly) two models:

§  point-to-point §  publish/subscribe


§  no consumers: message is kept §  multiple consumers: message is delivered only once

JMS point-to-point model (queue)


§  no consumers: message is discarded §  multiple consumers: message is delivered multiple times

JMS publish/subscribe model (topic)


§  producers send to an exchange §  consumers receive from a queue §  bindings are used to connect exchanges to queues

AMQP 0-* model


Messaging software


Broker Qpid MRG HornetQ Ac1veMQ Apollo RabbitMQ

Language Java C++ Java Java Scala Erlang

Main Protocols

AMQP AMQP proprietary STOMP

OpenWire STOMP AMQP MQTT

OpenWire STOMP AMQP MQTT

AMQP STOMP (MQTT)

Owner (*) Red Hat FuseSource (Progress)

VMware Red Hat

7 September 2012 : Red Hat completed its acquisition of FuseSource http://fusesource.com/redhat/

Messaging protocols & software

§  Many protocols exist, we recommend STOMP. §  Many brokers exist, we recommend ActiveMQ. §  These recommendations may change in the future. §  For more details, see EMI’s “Enterprise Messaging

Solutions Technical Evaluation”. (http://cern.ch/go/w9Qd)


Part 2: how to use messaging?


Sending a message seems easy…

use Net::STOMP::Client; $stomp = Net::STOMP::Client->new( uri => "stomp://mybroker:6163", ); $stomp->connect( login => "guest", passcode => "welcome", ); $stomp->send( destination => "/queue/test", subject => "this is a test", body => "Hello world!\n", ); $stomp->disconnect();"


… but it is not!

§  What if the broker is down? §  What if TCP’s connect hangs? §  What if STOMP’s connect hangs? §  What if a STOMP error frame is received? §  What if nothing comes back from the broker?

Is it a good sign or does this indicate a black hole? §  If you ask for a STOMP receipt, how long should

you wait for the receipt to come? §  …


Developing messaging clients

§  Asynchronous programming is tricky. §  Easy to have something working but

hard to have something working reliably. §  Diversity can lead to duplication:


Simplifying client development

§  Sound development principles: §  use abstraction §  factorize common code §  re-use existing code §  isolate technology dependent code from the rest

§  In practice: use our messaging blocks:


Message Queue Message Transfer Agent

""""""$msg = Messaging::Message->new( body => "Hello world!\n", …); $mq = Messaging::Message::Queue::DQS->new( path => "/some/where"); $mq->add($msg);""

Messaging blocks to produce


Message Queue (MQ)

§  MQ is a thin layer on top of Directory Queue (DQ). §  DQ is a filesystem based queue system

supporting multiple concurrent readers and writers. §  DQ is available for Perl, Python, Java, C/C++

(beta) and Erlang (experimental). §  MQ is available for Perl and Python (so far).


Message Transfer Agent (MTA)

§  MTA is capable of sending and receiving messages to and from MQ or broker.

§  MTA can run as a daemon or in “flush mode”. §  Available MTA implementations:

§  stompclt: speaking STOMP 1.0 – 1.2 §  amqpclt: speaking AMQP 0.9


Using messaging blocks

§  Advantages: §  completely decoupling application logic from messaging

protocol and messaging software §  less code to write and maintain §  proven code, used in production since several years §  available in EPEL

§  Drawbacks: §  additional software dependency §  performance penalty as messages hop through disk


Scalable messaging services


produce to any consume from all

""""""""mq = DQS(path='/some/where') for name in mq: if not mq.lock(name): continue msg = mq.get_message(name) mq.remove(name)""

Messaging blocks to consume


Messaging blocks flexibility


Reliable services through supervision

§  Supervision trees: §  workers perform the actual work §  supervisors monitor the workers §  all are defined in a supervision tree §  proven concept (Erlang/OTP)

§  We recommend to use the simplevisor: §  flexible implementation §  non intrusive §  available in EPEL


Supervision of messaging blocks


Part 3: messaging recommendations


Messaging is not a magic bullet

§  Messaging is not suited to: §  large data transfers §  low latency communication

§  Message brokers are not message stores §  better think of them as higher-level routers


Security considerations

§  You have to define who is allowed to do what. §  Brokers usually provide:

§  authentication: name + password or X.509 certificate §  authorization: per-destination granularity

§  Clients can also use: §  message encryption: to prevent snooping §  message signing: to ensure origin and integrity

§  Brokers and clients security levels must match the type of information carried by the messages.


Performance considerations

§  The number of messages per second is important …  but it is not the only parameter to consider

§  Other things to look at: §  persistence §  message size §  amplification factor §  number of concurrent connections §  number of new connections per second §  number of messages to keep in store §  distributions of these rates, sizes and counts


Trade-offs

§  Consider a “computer center monitoring” use case: §  120k clients sending 100 metrics every minute

§  1 metric per message → 200k msg/sec versus 100 metrics per message → 2k msg/sec §  drawbacks: bigger and harder to filter messages

§  Long-lived clients → 120k concurrent connections vs. flush every 5 minutes → 400 concurrent connections §  drawbacks: higher latency and 400 connections/sec

§  Message compression: §  CPU time versus network bandwidth and broker storage


Ballpark figures

§  Optimal message size (header + body): 1kB – 10kB §  Reasonable aggregated message rate: ≤ 1kHz

§  less if using persistent messages §  more if using topics without durable subscriptions

§  Reasonable new connections rate: ≤ 100Hz §  less if using X.509 authentication

§  A bit of maths: 10kB messages @ 1kHz represent: ≈ 100Mb/s or much more, depending on the protocol used ≈ 824GB/day to store on disk when consumers are down


Miscellaneous recommendations

§  Expertise is key to get the most out of messaging: §  train yourself and/or come to talk to us

§  Use mainstream broker features to avoid lock-in. §  Do not believe any benchmark results or

compliance statements: test by yourself! §  For practical guidelines:

§  http://cern.ch/messaging/user/guidelines.html


Connected (but still white) areas

§  Messaging can also be used without brokers: §  most widely used solution: ZeroMQ §  pros: lightweight, fast, low latency §  cons: limited security, messages are held by the

producers when the consumers are down §  Some programming languages support message

passing:

Pid ! Message

is enough to send a message in Erlang, even between different hosts!


Summary

§  For the appropriate use cases, messaging can bring many benefits.

§  Things are more complex than they seem. §  Messaging technology (like any computing

technology) will change. §  Our messaging blocks help both to simplify

development and to protect from changes. §  Security is important.


References

§  Messaging http://cern.ch/messaging

§  The WLCG Messaging Service and its Future http://cern.ch/messaging-chep2012

§  Making reliable distributed systems in the presence of software errors http://www.erlang.org/download/armstrong_thesis_2003.pdf

§  White Area Lectures https://twiki.cern.ch/twiki/bin/view/LCG/WhiteAreas


white area lecture – messaging - cern wa 06-11-2013 - messaging.pdf · messaging software...

Documents