building iot and big data solutions on azure
TRANSCRIPT
Building IoT and Big Data Solutions on Azure
Ido FlatowSenior Architect, Sela Group
Microsoft MVP & RD @idoflatow
Level: Intermediate
Modern Data – The Big Picture
IoT
User Data
Media Files
Documents
Machine Data
Log Files
IoT 2010
Cell phone
VoIP phone
HVAC
Computer
Vending
Printer
Security
Media player
Oven
Automobile
Smart scale
Refrigerator
Television
Microwave
Coffee maker
Alarm clock
HOME HOMEWORKPLACE
Sleep tracking
COMMUTE COMMUTE
Home security Home automation Leak detection
Smart appliances
Indoor navigation
Health monitoring
Smart lighting
Pet tracking
Information capture
Trip tracking and car health
Control
Child and eldermonitoring
Sports and fitness
Air conditioning and temperature control Environmental sensors
Behavior modification
Garden, lawn and plant care
Food and nutrition tracking
Beacons and proximity
New devices and sensors
Object tracking
Identity Smart vending machines
Medication adherence
Bike ride stats and protection
Entertainment systems
Office equipment
IoT 2016
HOME HOMEWORKPLACE
What We Need• An integrated data solution that will be:
– Able to process events from external sources– Able to walk data through different pipelines– Fast and responsive– Big-Data ready
In Other Words
Consume
BI Dashboards Applications
ProcessETL Aggregations Computation Analysis Querying
PersistHadoop SQL NoSQL
IngestStructured Data Un-Structured Data
Microsoft Azure Services forIoT and BigData
Devices Device Connectivity Storage Analytics Presentation & Action
Event Hubs SQL Database Machine Learning App Service
IoT Hub Table/Blob Storage Stream Analytics Power BI
External Data Sources DocumentDB HDInsight Notification Hubs
External Data Sources Data Factory Mobile Apps
BizTalk Services
{ }
Microsoft Azure Services forIoT and BigData
Devices Device Connectivity Storage Analytics Presentation & Action
Event Hubs SQL Database Machine Learning App Service
IoT Hub Table/Blob Storage Stream Analytics Power BI
External Data Sources DocumentDB HDInsight Notification Hubs
External Data Sources Data Factory Mobile Apps
BizTalk Services
{ }
Field Gateway
Device Connectivity & Management
Analytics & Operationalized Insights
IoT & Data Processing Patterns
Devic
esRT
OS, L
inux
, Wind
ows,
Andr
oid, iO
S
Protocol Adaptation
Batch Analytics & VisualizationsAzure HDInsight, AzureML, Power BI, Azure Data Factory
Hot Path AnalyticsAzure Stream Analytics, Azure HDInsight Storm
Hot Path Business LogicService Fabric & Actor Framework
Cloud GatewayEvent Hubs&IoT Hub
Field Gateway
Protocol Adaptation
Find insights to• Power new services• Improve your “things”
Operationalize your insights in real time
IoT Scale Object Models & Business Logic
Field Gateway
Device Connectivity & Management
IoT with Event HubsDe
vices
RTOS
, Lin
ux, W
indow
s, An
droid
, iOS
Cloud GatewayEvent Hubs
Field Gateway
Protocol Adaptation
Event Hubs• High scale telemetry ingestion service• HTTP/AMQP protocol support• Each Event Hub supports
• 1 million publishers• 1GB/s ingress
• Generally available worldwide• Billions of messages per day• TB of ingested data per day
IoT Hub Cloud Gateway Endpoints
device
Event processing(hot and cold path)
Device provisioning and management
Your IoT Hub
Device id
C2D queueendpoint
D2C send endpoint
Device …
Device …
Device …
D2C receive endpoint
C2D send endpoint
Msg feedback and monitoring endpoint
Device identity managementIoT Hub
management
Device business logic,Connectivity monitoring
Field GW /Cloud GW
IoT Hub Features • Connection
– Bidirectional communication– Reliable & secure channel– Per-device authentication– Multiplexing
• Features– Device to cloud telemetry– Cloud to device commands and notifications (with TTL & feedback)– File uploads/downloads– Monitoring devices (connection, activity, ...)– Multi protocols (AMQP, HTTP) IoT Protocol Gateway (MQTT)
IOT HUBDemo
Azure Stream AnalyticsMission critical reliability and scale
Enables rapid development
Fully managed real-time analytics
• Automatic recovery• Monitoring and
alerting• Scale on demand
• Managed Cloud Service
• Each unit handles 1MB/s
• Can scale up to 1GB/s
• SQL like language• temporal windowing
semantics• support for reference
data
Tumbling Windows• How many vehicles enter each toll booth
every 5 minutes?
SELECT TollId, COUNT(*) FROM EntryStream GROUP BY TollId, TumblingWindow(minute,5)
STREAM ANALYTICSDemo
What is Azure Data Factory?
Azure Data Factory is a managed service to produce trusted information from data stored in the cloud and on-premises. Easily create, orchestrate and schedule highly-available, fault tolerant work flows to move and transform your data at scale.
Evolving Approaches to Analytics
ETL Tool(SSIS, etc)
EDW(SQL Svr, Teradata, etc)
Extract
Original Data
Load
Transformed Data
Transform
OLTP
ERP LOB
…BI Tools
Devices
Web
Sensors
SocialIngestOriginal Data
Scale-out Storage & Compute
(HDFS, Blob Storage, etc)
Transform & Load
Data MartsData
Lake(s)Dashboards
Apps
Streaming data
Data Factory Concepts
Call Log Files
Azure Storage
On Premises Data Mart
Customer Table
Azure DB
Customer Churn Table
Visualize
Data Set(Collection of files, DB table, etc)
Activity: a processing step (Hadoop job, custom code, ML model, etc)
Pipeline: a sequence of activities (logical group)
…
Customer Call
Details
Customers Likely to Churn
Transform,
Combine, etc
Analyze Move
DATA FACTORYDemo
DocumentDB and Azure Data Services
fully managed, scalable, queryable, schema free JSON document database service for modern applications
fully featured RDBMStransactional processing
rich query managed as a service
elastic scale
internet accessible http/rest
schema-free data model
arbitrary data formats
Data size
Access
Updates
Structure
Integrity
Scaling
Hadoop vs. Relational DB
Hadoop Ecosystem and OSS vs.Azure IoT and BigData Services
Azure Services OSS SolutionsEvent Hubs KafkaIoT Hub Kafka + Mosquitto (MQTT broker)Stream Analytics StormHDInsight Hadoop
Map Reduce Map ReduceHive HiveSpark SparkHBase HBase
Azure ML MahoutData Factory PigDocumentDB MongoDB / Couchbase
DOCUMENTDB & HDINSIGHT
Demo
Azure IoT Hub vs Event HubArea IoT Hub Event Hubs
Communication patterns
Device-to-Cloud Cloud-to-Device
event ingress (device-to-cloud)
Device protocol support
AMQP, AMQP over WebSockets, HTTP, and MQTT
AMQP, AMQP over WebSockets, and HTTP
Security Per-device identity and revocable access control
Event Hubs-wide shared access policies, with limited revocation support
Operations monitoring
Rich set of device identity management Only aggregate metrics
File upload File notification endpoint for workflow integration
Manually request files from devices
Scale Millions of simultaneously connected devices
Limited number of simultaneous connections--up to 5,000 AMQP connections
Device SDKs C, Node.js, Java, .NET, Python Java, .NETC and Node.js in preview
The Bigger PicturePresentation and action
Storage andBatch Analysis
StreamAnalysis
IngestionCollectionEvent production
Event hubs
Cloud gateways(web APIs)
Field gateways
Applications
Legacy IOT (custom protocols)
Devices
IP-capable devices(Windows/Linux)
Low-power devices (RTOS)
Search and query
Data analytics(Power BI)
Web/thick client dashboards
SQL DB
DocumentDBPower BI
Storage
Stream Analytics
Devices to take action
MachineLearning
DataFactory
HDInsight
IoT Hub
Resources• Azure IoT Dev center
– http://www.azure.com/iotdev
• Azure Services– https://azure.microsoft.com/en-us/services/event-hubs– http://azure.microsoft.com/en-us/services/iot-hub– https://azure.microsoft.com/en-us/services/stream-analytics– https://azure.microsoft.com/en-us/services/data-factory– https://azure.microsoft.com/en-us/services/documentdb– https://azure.microsoft.com/en-us/services/hdinsight
• Microsoft and IoT– https://www.microsoft.com/en-us/cloud-platform/internet-of-things– https://blogs.microsoft.com/iot/
• My Info– @IdoFlatow // [email protected] // http://www.idoflatow.net/downloads