scalable open-source iot solutions on microsoft azure

Open Source IoT Solutions on Azure

Stephane Verdy, Canonical

Sergey Demyanov, DataArt

Artyom Astafurov, DataArt

stephane.verdy@canonical.comsergey.demyanov@dataart.comastaff@dataart.com

Open Source IoT… when there’s always an option

Sensors

Devices

Gateways

Edge Processing

Cloud

Scalability

Data

Time series

Events

Analytics

Stage Hardware Environment

Prototyping Off the shelfDevelopment kit Developer’s Machine

Pilot Off the shelf / Commercial Development Cloud

Production Commercial gateways Production Cloud at Scale

Same design principles

Same core components

Maximize reuse

Continuous delivery

Edge computing matters

Our Project for Today

• Equipment monitoring• Infrastructure• Analytics

Prototype Run Scale

Core Components

• Gateway: Ubuntu Snappy Core• Gateway: DeviceHive IoT Toolkit• Cloud Infrastructure: Azure• Cloud: Device Management – DeviceHive• Cloud: Analytics/Processing – Apache Spark

Gateways:Ubuntu

Snappy Core

Gateways:DeviceHive IoT Toolkit

Microservices?

Message Bus

Service1 Service2 ServiceN

App1 App2 App3

Gateways

Message Bus

ZigBee Bluetooth LE Cloud

AllJoyn Connector

Data Aggregator App …

IoT Toolkit Overview

GPIO

ZigBee

BLE

AllJoyn Cloud

EnOcean

• Runs on any Linux • Open-source License• Program with Language of you Choice• Builds into snaps and deb packages• Easily extensible

Predictive Maintenance Demo

DeviceHive Cloud

IoT-[ , ]

Ubuntu Snappy

Web Sockets

SensorTagBLE

Streaming Zeppelin

SensorTagBLE

IoT-Toolkit [BLE, Cloud]

Demo: Source Code

# get dbus proxy objectble = dbus.Interface(dbus.SystemBus().get_object('com.devicehive.Bluetooth', '/com/devicehive/bluetooth'), 'com.devicehive.bluetooth‘

cloud = dbus.Interface(dbus.SystemBus().get_object('com.devicehive.cloud', '/com/devicehive/cloud'), 'com.devicehive.cloud') # connect to listenersble.connect_to_signal('PeripheralDiscovered', device_discovered)ble.connect_to_signal('PeripheralConnected', device_connected) # connect to discovered peripheral def device_discovered(mac, name, rssi): ble.Connect(mac, False) # enable acceleration sensor on connectdef device_connected(mac): ble.GattWrite(mac, 'F000AA1204514000b000000000000000', '01') ble.GattWrite(mac, 'F000AA1304514000b000000000000000', 'A0') ble.GattNotifications(mac, 'F000AA1104514000b000000000000000', True) cloud.SendNotification('Acceleration', json.dumps({ 'mac': mac, 'value': abs_value}), DEFAULT_PRIORITY)

Predictive Maintenance Demo

DeviceHive Cloud

IoT-Toolkit [BLE, Cloud]

Ubuntu Snappy

Web Sockets

SensorTagBLE

Streaming Zeppelin

SensorTagBLE

DeviceHive in Local Docker

Predictive Maintenance Demo

DeviceHive Cloud

IoT-[ , ]

Ubuntu Snappy

Web Sockets

SensorTagBLE

Streaming Zeppelin

SensorTagBLE

IoT-Toolkit [BLE, Cloud]

Cloud:DeviceHive

and Analytics

Overview

• From Gateways to the Cloud• Infrastructure in Azure• Services• IoT Data Pipeline• Demo

What Makes IoT Special

• High volume• Time-stamped• Immutable• Sensor• Event streams

Message Bus

Service1 Service2 ServiceN

App1 App2 App2

Microservices?

Message Bus

ZigBee Bluetooth LE Cloud

AllJoyn Connector

Data Aggregator App …

Gateways

Message Bus

Time Series DB

Device Management Rule Engine

Spark Job1 Spark Job2 Spark JobN

Cloud

Services

• Device management• Message bus• Stream processing• Batch processing• Storage

Services

• Device management: DeviceHive• Message bus: Apache Kafka• Stream processing: Spark Streaming• Storage: DataStax Cassandra

Infrastructure Core Principles

• Immutability• Containerization• Service Discovery • Resource Management• Scheduling

Infrastructure Core Principles

• Immutability: Juju• Containerization: Docker• Service Discovery: Consul• Resource Management: Mesos• Scheduling: Marathon

Apache Mesos

Apache Mesos

Service Discovery

Node 1 Node 2 Node 3

Container 1 Container 1 Container 1

Container 2

Consul Agent

Container 2 Container 2

Consul Agent Consul Agent

HashiCorp’s Consul:• RESTful API on each node• Service lookup via DNS

Scheduler

Infrastructure: Azure with JujuContainers + Registry(Docker)

Resource Management (Mesos)

Service Discovery (Consul: REST + DNS)

Logging(ELK)

Services: IoT PipelineDevice Management (DeviceHive)

Message Bus(Apache Kafka)

Stream/Batch Processing(Apache Spark)

IoT Cloud

Data Flow

Sensor

Gateway

BLE Framework

CloudFramework

Cloud

DeviceHive Kafka

SparkDashboard(Zeppelin)

Time Series(Cassandra)

Demo