azure stream analytics
TRANSCRIPT
About Me
Business Intelligence Consultant, in IT for 28 years Microsoft, Big Data Evangelist Owner of Serra Consulting Services, specializing in end-to-end Business Intelligence
and Data Warehouse solutions using the Microsoft BI stack Worked as desktop/web/database developer, DBA, BI and DW architect and
developer, MDM architect, PDW developer Been perm, contractor, consultant, business owner Presenter at PASS Business Analytics Conference and PASS Summit MCSE for SQL Server 2012: Data Platform and BI SME for SQL Server 2012 certs Contributing writer for SQL Server Pro magazine Blog at JamesSerra.com SQL Server MVP Author of book “Reporting with Microsoft SQL Server 2012”
Growth of internet connected devices vs. connected PCs, smartphones, tablets
2009 2015 20200
5
10
15
20
25
30B
illio
ns
http://www.gartner.com/newsroom/id/2905717 Forecast: The Internet of Things, Worldwide, 2013
The need for real-time processing
Smart grid CRM alerting sales Predictive maintenance
Real-time financial sales tracking
Real-time fraud detection
Click-stream analysis
Remote device monitoring
Connected car
Build more intelligent transportation
Reduce maintenance costs by receiving notifications when preventative maintenance is needed before equipment breaks down
Respond to and resolve traffic incidents and maintenance needs faster by integrating sensors, license plate reading systems, cameras, and social network data
Alleviate the hassle of city parking by providing citizens with an easy view of parking availability throughout the city and automatic payment systems
Increase revenue and incentivize off-peak travel by implementing dynamic toll and fare pricing based on license plate readings
Create more value withtoll and fare management
Simplify the parking experience
Increase traffic system efficiency
Optimize fleet and asset management
xx!
How do customers create a real-time streaming solution?
Infrastructure – procure and setup
Develop solution (code) for ingress, processing, and egress
Develop solutions to integrate with other components like ML, BI, etc.
Develop solutions to manage resiliency,
such as infrastructure failures
Develop solutions and infrastructure for increasing scale with business growth
Monitoring and troubleshooting of solution
Customers using Azure Stream AnalyticsFrom event or data streams to real time insights in less time with less people
resources
Development and operations resources Time
End-to-end stream processing on Microsoft AzureIngestor
(broker)Collection Presentation
and actionEvent producers
Transformation
Long-term storage
Event hubs
Storage adapters
Stream processingCloud gateways
(web APIs)
Field gateways
Applications
Legacy IOT (custom protocols)
Devices
IP-capable devices(Windows/Linux)
Low-power devices (RTOS)
Search and query
Data analytics (Excel)
Web/thick client dashboards
Service bus
Azure SQL DB
Azure storage
HDInsight
Stream Analytics
Devices to take action
Power BI
Introducing stream analytics
Mission critical reliability and scale
Enables rapid development
Fully managed real-time analytics
Intake millions of events per secondProcess data from connected devices/appsHighly-scalable publish-subscriber ingestor
At variable loadsScale that accommodates variable loads Preserves event order on per-device basis
Real-time analytics on continuous streams of data Detect patterns and anomalies in streaming dataTransform, augment, correlate, temporal operations
Correlate with reference data
Real-time analytics
Fully Managed
No hardware (PaaS offering)
Bypasses deployment expertiseUp and running in a few clicks (and within minutes)No software provisioning and maintainingNo performance tuning
Less HW and SW maintenanceSpin up any number of resources on demandUse and pay for resources only at times you need them
Expand your business globally
Procure HW Infrastructure and setup
Code for ingress, processing and egress
Plan for resiliency, such as HW failures
Design solution
Build Monitoring and Troubleshooting
Efficiently pay only for usageArchitected for multi-tenancy Small streaming jobs billed at low amountsNot paying for idle resources
Augment costs as neededLow startup costs Ability to incrementally add resourcesReduce costs when business needs changes
Lower costs
Introducing stream analytics
Mission critical reliability and scale
Enables rapid development
Fully managed real-time analytics
Guaranteed events deliveryGuaranteed not to lose events or incorrect outputGuaranteed “once and only once” delivery of eventAbility to replay events
Guaranteed business continuityGuaranteed uptime (three nines of availability)Auto-recovery from failures Built in state management for fast recovery
Mission critical reliability
Elasticity of the cloud for scale up or scale downSpin up any number of resources on demandScale from small to large when requiredDistributed, scale-out architecture
No challenges with scale
Introducing stream analytics
Mission critical reliability and scale
Enables rapid development
Fully managed real-time analytics
Only SQL queries neededDevelopers uses declarative SQL commandsSome functions take several lines of code versus thousands from other solutions
Rapid development
Versus 3 lines of code in Stream Analytics
Thousand lines of code in other solutions, such as Apache Storm
Implement temporal functionsTumbling WindowsHopping WindowsSliding Windows
Manage out-of-order events With configuration instead of code
Manage actions on late eventsUsing policy settings instead of code
Built in temporal semantics
Built-in monitoring View your system’s performance at a glanceHelp you find the cost-optimal way of deployment
Scheduling and monitoring built in
Microsoft real-time stream processing options
Complex event processing in SQL Server
Ease of development and operationalization
Flexibility and customizability
On-premises or Azure IaaS Azure PaaS Azure PaaS
No No Yes
.NET/LINQ SQL SCP.NET, Java, Python
Visual Studio Web browser Visual Studio
Demo Steps
1. Connect sensor to laptop with Bluetooth2. Create Azure EventHub 3. Create app to read sensor data and send it to
EventHub4. Create Azure Stream Analytics job5. Create Power BI reports
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing marketconditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
James [email protected]
Questions?