MQTT as a Unified Message Bus for Infrastructure Services

Matthew TreinishOpen Source Developer Advocate -

IBMmtreinish@kortar.org

mtreinish on Freenode

Jeremy Stanleyfungi@yuggoth.org

fungi on Freenode

January 23, 2018

https://github.com/mtreinish/firehose/tree/lca2018

The problem

I The OpenStack community infrastructure operates >40 services on >250 serversI All the community infrastructure runs on donated public cloud resourcesI Some services depend on automation from othersI Several user facing services expose event busesI A real mess when you try to consume infrastructure events for any purpose

1 / 15

OpenStack Community Infrastructure Firehose

I An MQTT broker for the OpenStackcommunity infrastructure

I Has anonymous, read-only access viaMQTT on 1883/tcp

I SSL/TLS MQTT also available on8883/tcp

I Websockets supported (buttemporarily disabled)

2 / 15

MQTT

I Pub/sub messaging protocolI Formerly MQ Telemetry TransportI ISO/IEC 20922I Protocol dates back to 1999I Standard Maintained by OASISI Lightweight design, low bandwidth, and designed to handle unreliable networkingI Popular in IoT and sensor network applicationsI Large application ecosystem

3 / 15

MQTT Clients

I Bindings available for most languagesI https://github.com/mqtt/mqtt.github.io/wiki/librariesI Eclipse Paho project provides similar interfaces across multiple languages

MQTT Brokers

I Relies on a central brokerI Many different options: https://github.com/mqtt/mqtt.github.io/wiki/servers

4 / 15

MQTT Topics and Subscriptions

I Topics are generated dynamicallyI Topics are heirarchicalI Supports wildcarding

Examples:sensors/HOSTNAME/temperature/HDD_NAME

I sensors/sinanju/temperature/nvme0n1p1I sensors/+/temperature/+I sensors/sinanju/temperature/+I sensors/sinanju/#

5 / 15

QoS

I 3 QoS Levels:I 0: The broker/client will deliver the message once, with no confirmation.I 1: The broker/client will deliver the message at least once, with confirmation

required.I 2: The broker/client will deliver the message exactly once by using a four step

handshake.

I QoS of a message sent with each PUBLISHI Client sends QoS with SUBSCRIBE

6 / 15

The Firehose

I Runs Mosquitto MQTT brokerI Single broker instanceI Hardware Specs:

vCPUs 2CPU Frequency 2.6 GHz

RAM 2 GBswap 0 BDisk 40 GB

Bandwidth 200 Mbps

7 / 15

Mosquitto

I MQTT broker implemented in CI An Eclipse IoT projectI Support for MQTT v3.1 and v3.1.1

8 / 15

Services Using the Firehose

Service Base Topic Source of MessagesAnsible ansible Ansible MQTT Callback PluginGerrit gerrit germqtt

Launchpad launchpad lpmqttSubunit Gearman Worker gearman-subunit subunit-gearman-worker

Logstash Workers gearman-logstash logstash-gearman-worker

9 / 15

Typical Firehose Load

10 / 15

11 / 15

Manually Load Testing

12 / 15

CPU Usage: Memory Usage:

13 / 15

Use cases for Firehose

I 3rd Party CI OperatorsI Desktop Notifications:

I mqttwarn: https://github.com/jpmens/mqttwarnI Inter Service communication:

I gerritbot: https://git.openstack.org/cgit/openstack-infra/gerritbot/I Graphing metrics:

I mqtt_statsd: https://git.openstack.org/cgit/openstack-infra/mqtt_statsdI grafana: http://grafana.openstack.org/dashboard/db/mosquitto-status

14 / 15

Where to get more information

MQTT:I http://mqtt.org/I http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.htmlI https://mosquitto.org/I https://www.eclipse.org/paho/I #mqtt on FreenodeI https://github.com/mtreinish/pymqttbench

Firehose:I openstack-infra ML openstack-infra@lists.openstack.orgI #openstack-infra on FreenodeI http://docs.openstack.org/infra/system-config/firehose.htmlI https://docs.openstack.org/infra/system-config/firehose_schema.html

15 / 15