delivering real-time data with azure · 2019-10-16 · azure event hubs azure iot hub apache kafka...
Post on 02-Aug-2020
13 Views
Preview:
TRANSCRIPT
1@ChadGreen Delivering Real-Time Data with Azure
Delivering Real-Time Data with AzureDevUp
Chad GreenOctober 16, 2019
Silver Sponsors
Platinum Sponsors
Celebration Sponsor Lanyards Sponsor
Notebook Sponsor Registration Sponsor
3@ChadGreen Delivering Real-Time Data with Azure
u)
2
#
chadgreen@chadgreen.com
Chad GreenDirector of Software DevelopmentScholarRx
chadwickegreenChadGreenChadGreen.com
4@ChadGreen Delivering Real-Time Data with Azure
AgendaDelivering Real-Time Data with Azure
• Real-Time Data and Azure• Consuming Data Through Event Hubs• Analyzing Data with Stream Analytics• Ingesting Streaming Data into Power BI• Building Real-Time Visualizations with Power BI
5@ChadGreen Delivering Real-Time Data with Azure
Real-Time Data and AzureDelivering Real-Time Data with Azure
6@ChadGreen Delivering Real-Time Data with Azure
Real-Time ProcessingReal-Time Data and Azure
Data Sources
Data Storage
Ingestion Stream Processing
Analytical Data Store
Analytics and
Reporting
Batch Processing
Orchestration
7@ChadGreen Delivering Real-Time Data with Azure
• Ingest, process, and store messages in real-time• Cannot block the ingestion pipeline
• Act on data quickly
ChallengesReal-Time Data and Azure
8@ChadGreen Delivering Real-Time Data with Azure
ArchitectureReal-Time Data and Azure
Data Sources
Data Storage
Ingestion Stream Processing
Analytical Data Store
Analytics and
Reporting
Batch Processing
Orchestration
9@ChadGreen Delivering Real-Time Data with Azure
Technology ChoicesReal-Time Data and Azure
Data Sources
Data Storage
Ingestion Stream Processing
Analytical Data Store
Analytics and
Reporting
Batch Processing
Orchestration
Ingestion
Azure Event HubsAzure IoT HubApache Kafka
Data Storage
Azure Storage BlobsAzure Data Lake Store
Stream Processing
Azure Stream AnalyticsStorm
Spark Streaming
Analytical Data Store
SQL DWHbaseSparkHive
Cosmos DB
Analytics and
Reporting
Azure Analysis Services
Power BI
Microsoft Excel
10@ChadGreen Delivering Real-Time Data with Azure
Technology ChoicesReal-Time Data and Azure
Real-Time Message Ingestion Stream Processing Analytics and Reporting
Data Storage Analytical Data Store
Event Hubs Stream Analytics Power BI
Storage Blob Container Cosmos DB
11@ChadGreen Delivering Real-Time Data with Azure
Demo ScenarioReal-Time Data and Azure
• User answers practice test question• System records interaction• Dashboard showing real-time usage of test functionality
12@ChadGreen Delivering Real-Time Data with Azure
Demo ScenarioReal-Time Data and Azure
Clie
nt A
pplic
atio
n
13@ChadGreen Delivering Real-Time Data with Azure
Demo ScenarioReal-Time Data and Azure
Clie
nt A
pplic
atio
n
14@ChadGreen Delivering Real-Time Data with Azure
Demo ScenarioReal-Time Data and Azure
Clie
nt A
pplic
atio
n
15@ChadGreen Delivering Real-Time Data with Azure
Demo ScenarioReal-Time Data and Azure
Clie
nt A
pplic
atio
n
16@ChadGreen Delivering Real-Time Data with Azure
Demo ScenarioReal-Time Data and Azure
Clie
nt A
pplic
atio
n
17@ChadGreen Delivering Real-Time Data with Azure
Demo ScenarioReal-Time Data and Azure
Clie
nt A
pplic
atio
n
18@ChadGreen Delivering Real-Time Data with Azure
Consuming Data Through Event HubsDelivering Real-Time Data with Azure
19@ChadGreen Delivering Real-Time Data with Azure
Azure Event HubsSimple, secure, and scalable real-time data ingestion
Fully managed, real-time data ingestion service that is
simple, trusted, and scalable.
20@ChadGreen Delivering Real-Time Data with Azure
Simple Secure
Scalable Open
Azure Event HubsSimple, secure, and scalable real-time data ingestion
21@ChadGreen Delivering Real-Time Data with Azure
Event Producer Event Ingestion Event Consumer
Event Consumer
Event Consumer
Event-Driven ArchitectureSimple, secure, and scalable real-time data ingestion
22@ChadGreen Delivering Real-Time Data with Azure
Throughput UnitsAzure Event Hubs
Variable reserved capacities assigned to the Event Hub namespace
Ingress1-Mb or 1,000 events
Egress2-Mb or 4,096 events
Up to 20 TUs via PortalUp to 40 TUs via Support Ticket
23@ChadGreen Delivering Real-Time Data with Azure
ProvisioningConsuming Data Through Event Hubs
DemoProvision Azure Event Hubs
24@ChadGreen Delivering Real-Time Data with Azure
Sending Data to Event HubsConsuming Data Through Event Hubs
DemoSending Data to Even Hubs
25@ChadGreen Delivering Real-Time Data with Azure
Analyzing Data with Stream AnalyticsDelivering Real-Time Data with Azure
26@ChadGreen Delivering Real-Time Data with Azure
Azure Stream Analytics Serverless real-time analytics
Deliver powerful insights from your streaming data with
ease, in real time.
27@ChadGreen Delivering Real-Time Data with Azure
Potential Use CasesAzure Stream Analytics
• Analyze real-time telemetry streams from IoT devices
• Web logs/clickstream analytics
• Geospatial analytics for fleet management and driverless vehicles
• Remote monitoring and predictive maintenance of high value assets
• Real-time analytics on Point of Sale data for inventory control and anomaly detection
28@ChadGreen Delivering Real-Time Data with Azure
How does Stream Analytics workAzure Stream Analytics
29@ChadGreen Delivering Real-Time Data with Azure
Key Capabilities and BenefitsAzure Stream Analytics
Ease of Getting Started
Programmer Productivity
Fully Managed
Low Total Cost of
Ownership (TCO)
Reliability Performance
30@ChadGreen Delivering Real-Time Data with Azure
Key Capabilities and BenefitsAzure Stream Analytics
Ease of Getting Started
Programmer Productivity
Fully Managed
Low Total Cost of
Ownership (TCO)
Reliability Performance
31@ChadGreen Delivering Real-Time Data with Azure
Key Capabilities and BenefitsAzure Stream Analytics
Ease of Getting Started
Programmer Productivity
Fully Managed
Low Total Cost of
Ownership (TCO)
Reliability Performance
32@ChadGreen Delivering Real-Time Data with Azure
Key Capabilities and BenefitsAzure Stream Analytics
Ease of Getting Started
Programmer Productivity
Fully Managed
Low Total Cost of
Ownership (TCO)
Reliability Performance
33@ChadGreen Delivering Real-Time Data with Azure
Key Capabilities and BenefitsAzure Stream Analytics
Ease of Getting Started
Programmer Productivity
Fully Managed
Low Total Cost of
Ownership (TCO)
Reliability Performance
34@ChadGreen Delivering Real-Time Data with Azure
Key Capabilities and BenefitsAzure Stream Analytics
Ease of Getting Started
Programmer Productivity
Fully Managed
Low Total Cost of
Ownership (TCO)
Reliability Performance
35@ChadGreen Delivering Real-Time Data with Azure
Key Capabilities and BenefitsAzure Stream Analytics
Ease of Getting Started
Programmer Productivity
Fully Managed
Low Total Cost of
Ownership (TCO)
Reliability Performance
36@ChadGreen Delivering Real-Time Data with Azure
ProvisioningAnalyzing Data with Stream Analytics
DemoCreate a Stream Analytics Job
37@ChadGreen Delivering Real-Time Data with Azure
Stream Analytics Query LanguageAnalyzing Data with Stream Analytics
• Aggregate• Analytic• Array• GeoSpatial
• Input Metadata• Record• Windowing• Scalar
38@ChadGreen Delivering Real-Time Data with Azure
Stream Analytics Query LanguageAnalyzing Data with Stream Analytics
Events and Time
SELECT * FROM SensorReadings TIMESTAMP BY time
SELECT System.Timestamp AS Time FROM SensorReadings
39@ChadGreen Delivering Real-Time Data with Azure
Stream Analytics Query Language – WindowingAnalyzing Data with Stream Analytics
SELECT sensorId, COUNT(*) AS CountFROM SensorReadings TIMESTAMP BY timeGROUP BY sensorId, TumblingWindow(second, 10)
40@ChadGreen Delivering Real-Time Data with Azure
Stream Analytics Query Language – WindowingAnalyzing Data with Stream Analytics
SELECT sensorId, COUNT(*) AS Count, AVG(temp)FROM SensorReadings TIMESTAMP BY timeGROUP BY sensorId, HoppingWindow(second, 10 , 5)
41@ChadGreen Delivering Real-Time Data with Azure
Stream Analytics Query Language – WindowingAnalyzing Data with Stream Analytics
SELECT sensorId, MIN(temp) as tempFROM SensorReadingsTIMESTAMP BY timeGROUP BY sensorId, SlidingWindow(second, 5)HAVING MIN(temp) > 75
42@ChadGreen Delivering Real-Time Data with Azure
Stream Analytics Query Language – Joining Multiple SteamsAnalyzing Data with Stream Analytics
SELECT s1.time, s1.dspl, s1.hmdt as previousHmdt,s2.hmdt as newHmdt,datediff(ss, s1.time, s2.time) as secondsApart
FROM SensorData s1 timestamp by timeJOIN SensorData s2 timestamp by time
ON s1.dspl = s2.dsplAND DATEDIFF(s, s1, s2) BETWEEN 0 AND 5
WHERE (s2.hmdt – s1.hmdt >= .1) or (s1.hmdt – s2.hmdt >= .1)
43@ChadGreen Delivering Real-Time Data with Azure
Stream Analytics Query Language – Reference Data JOINAnalyzing Data with Stream Analytics
SELECT SensorReadings.sensorID, SensorReadings.tempFROM SensorReadings
JOIN thresholdRefDataON SensorReadings.sensorID = thresholdRefData.sensorID
WHERE SensorReadings.temp > thresholdRefData.value
44@ChadGreen Delivering Real-Time Data with Azure
Stream Analytics Query Language – Multiple OutputsAnalyzing Data with Stream Analytics
SELECT *INTO outputLogFROM SensorReadings
SELECT *INTO outputTempAlertFROM SensorReadingsWHERE temp > 75
45@ChadGreen Delivering Real-Time Data with Azure
QueryingAnalyzing Data with Stream Analytics
DemoQuery a Stream
46@ChadGreen Delivering Real-Time Data with Azure
Ingesting Streaming Data into Power BIDelivering Real-Time Data with Azure
47@ChadGreen Delivering Real-Time Data with Azure
Microsoft Power BIIngesting Streaming Data into Power BI
Provides interactive visualizations and business intelligence capabilities with an interface simple enough for end users to create their
own reports and dashboards.
48@ChadGreen Delivering Real-Time Data with Azure
Microsoft Power BIIngesting Streaming Data into Power BI
Create Collaborate and Share Access Insightsfrom Anywhere
49@ChadGreen Delivering Real-Time Data with Azure
Microsoft Power BIIngesting Streaming Data into Power BI
Create Collaborate and Share Access Insightsfrom Anywhere
50@ChadGreen Delivering Real-Time Data with Azure
Microsoft Power BIIngesting Streaming Data into Power BI
Create Collaborate and Share Access Insightsfrom Anywhere
51@ChadGreen Delivering Real-Time Data with Azure
Why Microsoft Power BIIngesting Streaming Data into Power BI
52@ChadGreen Delivering Real-Time Data with Azure
Why Microsoft Power BIIngesting Streaming Data into Power BI
53@ChadGreen Delivering Real-Time Data with Azure
Why Microsoft Power BIIngesting Streaming Data into Power BI
54@ChadGreen Delivering Real-Time Data with Azure
Why Microsoft Power BIIngesting Streaming Data into Power BI
55@ChadGreen Delivering Real-Time Data with Azure
Why Microsoft Power BIIngesting Streaming Data into Power BI
56@ChadGreen Delivering Real-Time Data with Azure
Why Microsoft Power BIIngesting Streaming Data into Power BI
57@ChadGreen Delivering Real-Time Data with Azure
Why Microsoft Power BIIngesting Streaming Data into Power BI
58@ChadGreen Delivering Real-Time Data with Azure
Why Microsoft Power BIIngesting Streaming Data into Power BI
59@ChadGreen Delivering Real-Time Data with Azure
Power BI Real-Time DatasetsIngesting Streaming Data into Power BI
• Support very fast dashboard tiles• Limited graphic options• Uses a Redis cache under the covers• Keeps approximately one hour of data
60@ChadGreen Delivering Real-Time Data with Azure
Real-Time DatasetsIngesting Streaming Data into Power BI
DemoCreating a Real-Time Dataset
with Stream Analytics
61@ChadGreen Delivering Real-Time Data with Azure
Real-Time DatasetsIngesting Streaming Data into Power BI
DemoPower BI Streaming Titles
62@ChadGreen Delivering Real-Time Data with Azure
Real-Time DatasetsIngesting Streaming Data into Power BI
DemoReal-Time Custom Report
Visualizations
63@ChadGreen Delivering Real-Time Data with Azure
u)
2
#
chadgreen@chadgreen.com
Chad GreenDirector of Software DevelopmentScholarRx
chadwickegreenChadGreenChadGreen.com
top related