architecting the internet of things darren hubert m256

Architecting the Internet of ThingsDarren Hubert M256

Collecting information from lots of devices is cool - but it is just telematics.

Merging perspectives between devices, systems, and humans to build a better understanding of the world around us.

But tying together insight with action —there lies the promise of IoT.

Objectives

Understand what IoT means for IT architects!Learn the technologies, and how to begin applying them to deliver a modern IoT project!Be better able to talk to customers about IoT!

Session OverviewWhat is IoT, and why is it important?

Technologies Architects need to know

Quick real world example(s)

Common patterns and practices

IoT from an Architect’s perspective

Expectations

Level 200 ARCHITECTURE SessionThere will be no codeThere will be no ”flashing light” demos

IoT: The “Internet of Things”…

“ ”

The network of physical objects that contain embedded technology to communicate and interact with their internal states or the external environment.

Source: Gartner

25Billion (2020)

Source: Gartner

what is it?

Unique objects connected to InternetDevices, not peopleBi-directional communicationLarge, complex data flowsNew types of insight

why is it important?

Worldwide market for IoT solutions to reach $7.2 trillion in 2020 (IDC)Economic value-add is forecast to be $1.9 trillion across sectors in 2020 (Gartner)

Leading Industry examples : utilities, insurance, agriculture, factory, automobiles, transport, consumer, etc

IoT 2010

Cell phone

VoIP phone

HVAC

Computer

Vending

Printer

Security

Media player

Oven

Automobile

Smart scale

Refrigerator

Television

Microwave

Coffee maker

Alarm clock

HOME HOMEWORKPLACE

Sleep tracking

Home security Home automation Leak detection

Smart appliances

Indoor navigation

Health monitoring

Smart lighting

Pet tracking

Information capture

Trip tracking and car health

Control

Child and eldermonitoring

Sports and fitness

Air conditioning and temperature control Environmental

sensors

Behavior modification

Garden, lawn and plant care

Food and nutrition tracking

Beacons and proximity

New devices and sensors

Object tracking

Identity Smart vending machines

Medication adherence

Bike ride stats and protection

Entertainment systems

Office equipment

IoT 2015

HOME HOMEWORKPLACE

The Gartner Hype Cycle

Technology Trigger

Peak of Inflated

Expectations

Trough of Disillusionmen

t

Slope of Enlightenme

nt

Plateau of Productivity

The Gartner Hype Cycle

An Architect’s Perspective

POV: IoT is at an Inflection Point

Connectivity is

proliferating

Hardware Is getting

cheap

Cloud cost, scale,

flexibility

M2M solutions are mainstream

Software is more

advanced

General Technical Requirements

Volumes of Data

Vague Security

RequirementsEnd to End IntegrationMany Devices Large Scale

Information Exchange PatternsBasics of IoT Communication

Telemetry

Information flowing from a device to other systems for conveying status of device and environment

Inquiries

Requests from devices looking to gather required information or to initiate activities

Commands

Commands from other systems to a device or a group of devices to perform specific activities

Notifications

Information flowing from other systems to a device (group) for conveying status changes

IoT SolutionsHow to Solve those Basic Patterns

EndpointsCommand

RoutingMessage passing

Message Security

Publish-Subscribe

IoT Challenges

Data VolumeScale ConnectivityAddressing Device Size

First Principles

Data VolumeScale ConnectivityAddressing Device Size

Cloud Storage AMQP/MQTT

CloudPub/Sub Queues

Reference Architecture

Presentation Device, Event and Data ProcessingData Transport

Devices andData

Sources

Solution PortalProvisioning API

Device Identity & Registry

Analytics

Data Visualization

& Presentation

Device Management

Gateway

Storage

Low power

devices

ingestion

IP capable devices

Existing IoT

devices Event Processing

What Do Architects Need to Know?

Producers Data Transport Storage Analysis Presentation & action

Event Hubs (Service Bus)

SQL DatabaseMachine Learning

Azure Websites

Heterogeneous client agents

Table/Blob Storage

HD Insight/Storm

Mobile Services

External Data Sources

DocumentDBStream Analytics

Notification Hubs

External Data Sources

Cloud Services Power BI

External Services

{ }

Microsoft Azure IoT Services

Depth?

Event HubsTelemetry Ingestion ServicesScalable publish-subscribe telemetry ingestorsProcesses massive amounts of data (Generally Available WW)• 1M Publishers• 1 GB/S Ingress• 1T Messages/Month (1k per message)TTP/AMQP Protocol Support Pluggable adapters for other cloud services

DocumentDBFully managed NoSQL document db service

{ }

NoSQL JSON and JavaScript DB ServiceJSON DB for rapid DevelopmentSchema free – for storage and queryAutomatic indexing of every document propertyCRUD access, query, and JavaScript processing Integrates with HDInsight, Azure Search, etc.

{ }

Azure HDInsight100% Apache Hadoop-based service in the cloudScales from terabytes to petabytes on demandProcesses unstructured or semi-structured data from devices and sensorsDeployable in Windows or LinuxConnects with on-premises Hadoop clusters Apache Storm for real-time events Apache Spark for in memory data analysis

Azure Stream AnalyticsProcess real-time data in Azure. “Data in Motion”Real Time AnalyticsIntake millions of events per second (up to 1 GB/s)Correlate between different streams, or with static data or modelsEasy processing on continuous streams of dataEnables the detection of anomalies. Ability to trigger an alert when a specific error or condition appears.

Azure Machine LearningCloud based predictive analytics engine

Designed for “applied” machine learning, Streamlined experience for data scientists, across multiple skill levelsDrag-and-drop, and data-flow graphs to set up experimentsBuild and test predictive models, predict future trends or behaviorPublish models as a fully managed web service (API)

Raw materials Acquire Raw Materials

Transform raw materials into “finished goods”

Deliver

Data Sources

Ingest Transform and Analyze Publish

Azure Notification HubsScalable mobile push notification engineTarget millions of devices/messages with single API call. Target audiences with dynamic tags. Tailor notifications by audience, language, and locationUse with any back end, in the cloud or on-premisesDynamically define and reach audience segments

Microsoft PowerBILive, single pane of glass dashboard solution for visualizations and KPIs Cloud based business analytics service:• Track data in real-time with support for streaming data• Drill through to underlying reports to explore and discover new

insight• Pin new visualizations and KPIs to monitor performance

IoT Cloud Patterns

Field Gateway

Device Connectivity & Management

IoT Device & Cloud PatternsD

evi

ces

RTO

S, L

inux,

Win

dow

s, A

ndro

id, i

OS

Cloud Gateway

Event Hubs

Field Gateway

Protocol Adaptation

Event Hubs• High scale telemetry ingestion

service• HTTP/AMQP protocol support• Each Event Hub supports

• 1 million publishers• 1GB/s ingress

• Generally available worldwide• 1 Trillion messages/month

Field Gateway



evi

ces

RTO

S, L

inux,

Win

dow

s, A

ndro

id, i

OS

Cloud Gateway

Event Hubs

Field Gateway

Protocol Adaptation

Additional IoT Requirements• Command & control• Device identity• Device registry• Device management

Field Gateway



evi

ces

RTO

S, L

inux,

Win

dow

s, A

ndro

id, i

OS

Cloud Gateway

Event Hubs&IOT Hub

Field Gateway

Protocol Adaptation

Protocol Adaptation

IoT Hub• Capability coming with Azure IoT

Suite• Bi directional D2C and C2D• Up to 10M devices• Telemetry ingestion• Command & control• Device registry & identity• Device Management• More…

Hot Path Business Logic

Service Fabric & Actor FrameworkField Gateway



evi

ces

RTO

S, L

inux,

Win

dow

s, A

ndro

id, i

OS

Cloud Gateway

Event Hubs&IOT Hub

Field Gateway

Protocol Adaptation

Protocol Adaptation

Analytics & Operationalized Insights

Batch “Cold Path” Analytics

Azure HDInsight, DocumentDB, Azure ML

Hot Path Analytics

Azure Stream Analytics, Azure HDInsight (Storm)

{ }

{ }





evi

ces

RTO

S, L

inux,

Win

dow

s, A

ndro

id, i

OS

Cloud Gateway

Event Hubs&IOT Hub

Field Gateway

Protocol Adaptation

Protocol Adaptation


Presentation & Business Connectivity

Presentation &

Business Connectivity

App Service, Websites

Dynamics, BizTalk Services, Notification Hubs



Hot Path Analytics


{ }

{ }


Service Fabric & Actor Framework


Pattern: Predictive MaintenanceD

evi

ces

RTO

S, L

inux,

Win

dow

s, A

ndro

id, i

OS

Cloud Gateway

Event Hubs&IOT Hub

Protocol Adaptation



Presentation &






Hot Path Analytics


{ }

{ }




Pattern: Service Delivery ManagementD

evi

ces

RTO

S, L

inux,

Win

dow

s, A

ndro

id, i

OS

Cloud Gateway

Event Hubs&IOT Hub

Field Gateway

Protocol Adaptation



Presentation &






Hot Path Analytics


{ }

{ }

Event Hub Stores

Streaming Data

Stream Analytics processes events as

they arrive in the EventHub

AML Model Web Service

BES endpoint

Power BI / D3

Dashboard

Data for

Real-time Processing

Aggregations

External Data

Azure Services

Azure SQLContains Historical

Data

Real time data stats

Azure Data Factory

Pipeline invokes AML Web Service

Real Ti

me

Batc

h

Example Architecture

Real Time Telemetry

Data

Azure Data Factory

Pipeline Moves Data

Batch updates of predictions

Data for


Real Time Energy

Consumption Data (Public

Source)

Event Hub Stores

Streaming Data

Stream Analytics processes events as

they arrive in the EventHub

AML Model Web Service

BES endpoint

Power BI / D3

Dashboard

Data Stream

Job

Hourly Prediction

Updates

External Data

Azure Services

Copy to Azure SQL

for batch predictions

Scrape Data

5 mins

Azure WebJob Runs jobs to scrape

data from public source

Azure SQLContains Historical

Energy Consumption Data

Real time data stats

Azure Data Factory

Pipeline invokes AML Web Service

Real Ti

me

Batc

h

Example Architecture

Old ways of Thinking can be dangerous Understand the business model Beware of new patterns: eventual

consistency, etc. Don’t focus on the device Avoid analysis paralysis. Get it done!

Risks of IoT

Examples

88 Acres. Darrel Smith, Dir. Energy and Sustainability, Microsoft RE&F

Architecture is at the centre of IoT IoT is Advanced ”Modern” Architecture IoT Projects are Complex - Teamwork is

necessary These projects are mission critical and

difficult We can’t learn everything - but we need

breadth Don’t be afraid - get started and learn

Summary

Complete your session evaluation now and win!

© 2015 Microsoft Corporation. All rights reserved.Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or

other countries.MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Real Time Telemetry

Data

Event Hub Stores

Streaming Data

Stream Analytics

processes events as they arrive in

the EventHub

AML Model (Published

Web Service)

Power BI (Dashboard)

Data for


Streaming Data

Near Real-time

Updates

IoT Clients Azure Services

IoT Clients Azure Services

ML Predictions consumed

through the RRS web service

interface

Example Architecture 1

Example Architecture 4External

DataAzure

Services

External Data

Azure Services

On Premise

SQL Server

Data Strea

m

Devices and

Systems

Azure Data Factory Pipeline

schedules On- Premise Data

Transfer

Scored Result

s

Copy Activity

Azure Storage Blob

Input

Azure Data Factory

Pipeline invokes Copy and Compute

Services

Compute

Activity

Copy Activity

Azure Storage Blob Output (ML

Scores)

AML Model Predictions consumed

through the web service interface

Consu

me

s V

ia

AD

F

Azure Data Factory Pipeline

transfers data to On-Premise data

storageCopy

Activity

Consumes

Power BI Dashboard

Data at rest

architecting the internet of things darren hubert m256

Documents

practices iot

iot solutions

promise of iot

modern iot project

technologies architects

nutrition tracking beacons

lots of devices

proximity new devices