azure service bus : under the hood
TRANSCRIPT
IoT day 2015
Azure Service Bus : under the hood
AMQP protocol fundamentalsPaolo Patierno
Microsoft MVP for Windows Embedded
@ppatierno
IoT day 2015
Sponsor
Who am I ? Contacts
•Senior Software Engineer (Leonardo Ricerche S.r.l.)
•Microsoft MVP for Windows Embedded
”... constantly moving between the devices and the cloud ...”
•«DotNetCampania» member• https://paolopatierno.wordpress.com
•«TinyCLR.it» member• http://www.tinyclr.it
•«Embedded101» board of director member• http://www.embedded101.com/Blogs/PaoloPatierno.aspx
•Linkedin• http://it.linkedin.com/in/paolopatierno
•Contacts• [twitter] @ppatierno
• [email] [email protected]
• [skype] paolopat80
Agenda
•IoT communication patterns
•Azure Service Bus intro …• Queues
• Topics/Subscriptions
• Event Hubs
•… under the cover : AMQP protocol• Introduction
• Architecture
• Communication
• How it fits the patterns
•Demo
IoT Communication Patterns
Telemetry
Information flows
from device to other
systems for
conveying status
changes in the device
Inquiries
Requests from
devices looking to
gather required
information or asking
to initiate activities
Commands
Commands from
other systems to a
device or a group of
devices to perform
specific activities
Notifications
Information flows from
other systems to a
device or a group for
conveying status
changes in the world
1:N 1:N
Azure Service Bus … Queues
•Competing Consumers pattern• all consumers read from same stream (queue)
•Message consumed by a single consumer
Azure Service Bus … Topics
•Publish/Subscribe pattern• each consumer reads from its subscription (a copy of message on related
topic)
•Message consumed by more subscribers (use filters)
Subs
Azure Service Bus … Event Hubs
•Partitioned Consumers pattern
Proprietary messaging protocols
•Difficult to port applications• Requires re-coding all applications
•Difficult to integrate• Application level bridges to move messages
and translate message formats
•Restricted platform support• Limited to whatever vendor provides
Advanced Message Queuing Protocol 1.0
•Open, standard messaging protocol• OASIS (since 2012/10) and ISO/IEC (since 2014/05)
• Enables cross-platform apps to be built using brokers, libraries and frameworks from different vendors
•Features• Efficient : binary connection-oriented protocol
• Reliable : “fire and forget” to reliable, “exactly once” delivery
• Portable data representation : cross-platform, full-fidelity exchange
• Flexible : client-client, client-broker and broker-broker topologies
• Broker model independent : no requirements on broker internals
Microsoft said …
“Microsoft congratulates the AMQP community on approval of AMQP version 1.0 as an OASIS Standard. As an open and interoperable messaging protocol that can scale from mobile clients to the cloud, AMQP has benefitted from the participation of technical experts from around the world, and the achievement of this important milestone will lead to continued growth in the AMQP ecosystem. We look forward to working with the community to promote AMQP-based interoperability and innovation.”
-- Scott Guthrie
AMQP 1.0 client libraries
Language Library
C# Service Bus .NET Client Library, AMQP.Net Lite
JavaApache Qpid Java Message Service (JMS) client
IIT SwiftMQ Java client
C Apache Qpid Proton-C
PHP Apache Qpid Proton-PHP
Python Apache Qpid Proton-Python
Ruby Apache Qpid Proton-Ruby
Perl Apache Qpid Proton-Perl
NodeJS Node AMQP 1.0
AMQP : containers and nodes
•architecture• wire protocol
• data types system with compact binary encoding
•container• client contains producer and/or consumer
• broker contains entities (ex. queues, topics/subscriptions, event hubs)
•node• producer sends messages
• consumer receives messages
• entities store and/or forward messages
AMQP : transport
client
producer
(consumer)
broker
entity
container containerconnection
session
link
node node
multiplexing frames on sessions and links
transport indipendent
open (max frame size, channel max, timeout)
AMQP : communication Open
•Open TCP connection
•Open ”connection”
•Begin ”session”
•Attach ”link”
Container Container
header AMQP / SASL handshake
open (max frame size, channel max, …)
begin (in/out window, handle max, …)
begin (in/out window, handle max, …)
attach (name, source, target, handle, …)
attach (name, source, target, handle, …)
open TCP
connection
open
«connection»
begin «session»
attach «link»
header AMQP / SASL handshake
AMQP : communication Send
•Send messages• to queue
• to topic
• to event hub
Producer Container
attach (name, source, target, handle, …)
attach (name, source, target, handle, …) attach «link»
flow (in/out window, link credit, …) flow control
transfer (delivery id/tag, settled, …)
disposition (first, last, settled, state, …)
sending
message
transfer (delivery id/tag, settled, …)
disposition (first, last, settled, state, …)
sending
messagestransfer (delivery id/tag, settled, …)
transfer (delivery id/tag, settled, …)
AMQP : communication Receive
•Receive messages• from queue
• from subscription
• from event hub
Consumer Container
attach (name, source, target, handle, …)
attach (name, source, target, handle, …) attach «link»
flow (in/out window, link credit, …) flow control
transfer (delivery id/tag, settled, …)
disposition (first, last, settled, state, …)
receiving
message
transfer (delivery id/tag, settled, …)
disposition (first, last, settled, state, …)
receiving
messagestransfer (delivery id/tag, settled, …)
transfer (delivery id/tag, settled, …)
end
AMQP : communication Close
•Detach ”link”
•End ”session”
•Close ”connection”
•Close TCP connection
Container Container
detach (handle, close, …)
end
close
close
detach «link»
end «session»
close
«connection»
detach (handle, close, …)
AMQP : Telemetry
•Data transfer
•Messages settled
Device Server
transfer (<resource>)
transfer (<resource>)
transfer (<resource>)
settlement (based on QoS)
settlement (based on QoS)
flow
in_queue
AMQP : Inquiry
•Correlation between inquiry
request and response
Device Server
transfer (<info>, replyTo, messageId)
transfer (<info>, correlationId = messageId)
settlement (based on QoS)
settlement (based on QoS)
in_queue out_queue
AMQP : Command
•Correlation between command
request and response
Device
transfer (<cmd>, replyTo, messageId)
transfer (<result>, correlationId = messageId)
settlement (based on QoS)
settlement (based on QoS)
Serverin_queue out_queue
AMQP : Notification
•Notification received
•Messages settled DeviceServer
transfer (<notifiy>)
settlement (based on QoS)
transfer (<notify>)
settlement (based on QoS)
flow
in_queue
AMQP : Claim Based Security (CBS) token
•Build a token based on publisher name, expire time, URI, key, …
•Send the token to the $cbs endpoint and specify a ReplyTo
•Receive status code of the request from the above ReplyTo
Device Server
transfer (<token>, replyTo, messageId)
transfer (<status code>, correlationId = messageId)
settlement (based on QoS)
settlement (based on QoS)
$cbs <replyTo>
AMQP : messages
•message• header
• annotations
• system properties (ex. correlationId, replyTo, TTL, …)
• custom/user properties
• «opaque» body (encoded as you want, ex. binary, XML, JSON, …)
•message metadata
•most times body empty, all values as properties
•filter on properties
•properties can be changed «on fly»
AMQP : main features
•messaging middleware• asynchronous : produce and consumer decoupled
• credit based to receive
+5
AMQP : main features
•more messaging patterns• load balancing and competing consumers (ex. on a queue)
• publish/subscribe (ex. on topic/subscription)
• partitioned consumers (ex. on event hubs)
• messages redirection to queues based on filters
• request/response w/ «correlation id» and «replyTo»
•session and transactional message transfer
AMQP : advantages
•efficient• binary connection-oriented
• ”flow control” credit based
• packet size 60 bytes
•reliable• Quality of Service (best effort, at least once, exactly once)
•security• SSL/TLS
• SASL (Simple Authentication and Security Layer)• decople authentication from application layer
Netduino 3 WiFi
•STM32 Cortex-M4
•2 MB dual-bank flash
•256 KB RAM
•WiFi module (TI CC 3100)
•Go-Bus ports
•Micro SD
•.Net Micro Framework
•Visual Studio 2013
•Azure Enabled (SSL/TLS)
IoT day 2015
Access to Service Bus with AMQP
References
•Microsoft Azure• Service Bus : http://azure.microsoft.com/en-us/services/service-bus/
• Event Hubs : http://azure.microsoft.com/en-us/services/event-hubs/
•AMQP • Official web site : http://www.amqp.org
• AMQP .Net Lite : http://amqpnetlite.codeplex.com/
• Azure SB Lite : http://azuresblite.codeplex.com/
• Azure SB Lite Examples : https://github.com/ppatierno/azuresblite-examples
• Apache Qpid: https://qpid.apache.org/