building high-scalable enterprise solutions,
TRANSCRIPT
![Page 1: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/1.jpg)
#SDF16
Building High-scalable Enterprise Solutions with Microsoft Azure and PowerBIThomas Claudius Huber – Principal Consultant@thomasclaudiush
![Page 2: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/2.jpg)
#SDF16
Thomas Claudius Huber
Principal Consultant
– Microsoft Most Valuable Professional (MVP)
Focus on C#, TypeScript, Angular and Azure
– Part of Cloud-Unit @Trivadis
Book- and Pluralsight-Author
– See also www.thomasclaudiushuber.com
![Page 3: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/3.jpg)
#SDF16
Agenda
The Big Picture
Azure Event Hubs
Stream Analytics and Power BI
![Page 4: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/4.jpg)
#SDF16
The Big Picture
![Page 5: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/5.jpg)
#SDF16
The big picture
![Page 6: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/6.jpg)
#SDF16
Agenda
The big picture
Azure Event Hubs
Stream Analytics and Power BI
![Page 7: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/7.jpg)
#SDF16
Azure Event Hubs
![Page 8: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/8.jpg)
#SDF16
Azure Event Hubs
Scales up to millions of events
Great approach for telemetry data
Great approach for sensor-data
![Page 9: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/9.jpg)
#SDF16
![Page 10: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/10.jpg)
#SDF16
Azure Event Hubs: Partitions
Event stream is segmented by partitions for scale
– Scaling for event consumers
– Scaling for event submitters
You can have between 2 and 32 partitions
Azure Support can enable up to 1024 or even more «under special conditions»
![Page 11: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/11.jpg)
#SDF16
Azure Event Hubs: Throughput Units
Throughput units (TUs) are important for billing. 1 Unit means
– Write maximum 1MB/sec and 1000 messages per second
– Read maximum 2MB/sec
Number of partitions has to be equal or greater than number of TUs
TUs are applied on namespace level
– You can have up to 10 event hubs in a namespace
– Namespace is used for the URI, for example https://<yourNamespace>.servicebus.windows.net
![Page 12: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/12.jpg)
#SDF16
Azure Event Hubs: Consumer Groups
Views on the Event Stream
– Similar to topic subscriptions that are also a «view» on an event stream
– Organizational anchor for checkpointing
$Default consumer group is always there
– Up to 20 named consumer groups are supported
Limitations
– 5 concurrent reading connections per consumer group
• It’s recommended to use a separate consumer group for each job
![Page 13: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/13.jpg)
#SDF16
Azure Event Hub vs. Azure IoT Hub
Quote: «It is not uncommon to use both IoT Hub and Event Hubs in the same solution»Official docmentation: https://azure.microsoft.com/en-us/documentation/articles/iot-hub-compare-event-hubs/
Event Hub IoT Hub
Communication Device to cloud Device to cloud and cloud to device
Security Event hub wide shared access policy Per device identity
Scale 5000 simultaneous connections Millions of simultaneous connections
SDKsSupported on .NET and C, also has AMQP and HTTP (REST) interfaces
Device SDKs for several platforms.
![Page 14: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/14.jpg)
#SDF16
But why not always using IoT Hub?
IoT Hub scales on devices
– a lot of concurrent connections
– but is expensive for a few thousand devices and a lot events
Event Hub scales on events
– but is limited to 5000 concurrent connections
![Page 15: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/15.jpg)
#SDF16
So what about pricing?
Event Hub
CHF 0.0253 per million events
1 Throughput Unit = CHF 0.0135/hr (~CHF 10/month)
– 1 Throughput Unit = 1MB / 1000 Messages Ingress, 2MB read
![Page 16: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/16.jpg)
#SDF16
Creating an Event Hub: Step 1
Go to the https://portal.azure.com
Select «New Internet of Things Event Hubs»
Enter Namespace-Name and Resource Group and click «Create»
![Page 17: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/17.jpg)
#SDF16
Creating an Event Hub: Step 2
So far you’ve created a namespace
– the namespace can have up to 10 event hubs
When you navigate to the namespace (via «Resources»), you can add a new event hub
![Page 18: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/18.jpg)
#SDF16
Demo: Create an Event Hub
![Page 19: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/19.jpg)
#SDF16
Connect to the Event Hub
To push events to an event hub, you have several options:
– From a .NET application you can use the NuGet-packageWindowsAzure.ServiceBus and the EventHubClient-class
• Samples: https://github.com/thomasclaudiushuber/EventHub.Clients
– You can also use a REST-API that is available automatically for each event hub
• https://github.com/thomasclaudiushuber/EventHub.RestClientGenerator
You need to setup a Shared Access policy for your event hub
To access the event hub, you need the URI and the shared access key
– both is part of a Connection String that you find in the Shared Access policy
![Page 20: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/20.jpg)
#SDF16
Pushing Events to the Event Hub from a .NET App
Use the NuGet-package WindowsAzure.Service
– It has an EventHubClient-class
Events are JSON-data items
![Page 21: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/21.jpg)
#SDF16
Consuming Events from the Event Hub in a .NET App
Same EventHubClient-class
Grab the consumerGroup and create the receivers for the partitionIds
![Page 22: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/22.jpg)
#SDF16
Demo: Access the Event Hub
![Page 23: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/23.jpg)
#SDF16
Agenda
The big picture
Azure Event Hubs
Stream Analytics and Power BI
![Page 24: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/24.jpg)
#SDF16
The big picture
![Page 25: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/25.jpg)
#SDF16
What is Stream Analytics
it’s a real-time event processing engine
Use a SQL-variant to query the event stream
High scalable
![Page 26: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/26.jpg)
#SDF16
Stream Analytics: Inputs and Outputs
![Page 27: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/27.jpg)
#SDF16
Stream Analytics: Windowed Aggregates
There are different windowed aggregates, one is the Tumbling Window
select DeviceName,max(value),count(*)into [SDFPBOutput]from [SDFEventHub99] Timestamp by Readtime group by devicename, TumblingWindow( Duration(second, 5), Offset(millisecond, -1))
![Page 28: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/28.jpg)
#SDF16
Demo: Setting up Stream Analytics
![Page 29: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/29.jpg)
#SDF16
Power BI as an Output
BI-tool to visualize your data
– Access it via https://www.powerbi.com
Stream Analytics can directly push its outupt to Power BI
In Power BI you can work with the Stream Analytics Data
![Page 30: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/30.jpg)
#SDF16
Build a Live Dashboard in Power BI
![Page 31: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/31.jpg)
#SDF16
Agenda
The big picture
Azure Event Hubs
Stream Analytics and Power BI
![Page 32: Building High-scalable Enterprise Solutions,](https://reader035.vdocument.in/reader035/viewer/2022062823/5871d21b1a28ab423c8b5dad/html5/thumbnails/32.jpg)
#SDF16
The big picture