business intelligence in azure - sql relay in...business intelligence in azure alex whittles alex...
TRANSCRIPT
PurpleFrogSystems.com
PurpleFrogSystems.com/blog
@PurpleFrogSys
Business Intelligence in Azure
Alex Whittles
Alex Whittles
Business Intelligence ConsultancyData Modelling ETL SystemsData Warehousing Reporting SystemsOLAP Cubes Managed Service
• SQL Relay Committee SQLRelay.co.uk
• SQL Bits Committee SQLBits.com
• Birmingham SQL UG SQLMidlands.com
• Birmingham Azure UG AzureBirmingham.uk
• MSc in Business Intelligence, CEng, CITP, FBCS, FIOEE, MIET, MIOD
• Run BI Consultancy
Alex Whittles
Business Intelligence ConsultancyData Modelling ETL SystemsData Warehousing Reporting SystemsOLAP Cubes Managed Service
• SQL Relay Committee SQLRelay.co.uk
• SQL Bits Committee SQLBits.com
• Birmingham SQL UG SQLMidlands.com
• Birmingham Azure UG AzureBirmingham.uk
• MSc in Business Intelligence, CEng, CITP, FBCS, FIOEE, MIET, MIOD
• Run BI Consultancy
Cortana Analytics Data Flow
Business Intelligence in Azure
Options:1. Migrating on-premise BI solutions to Azure2. Batch load BI systems in Azure3. Real-time BI systems in Azure
For each:Staging / ETL / Data Warehouse / Cube
On-Premise Batch Load BI
SQL ServerStaging Database
SQL ServerData Warehouse
SSASCube
Source Data
SSISETL
Azure IAAS Solution
SSISETL
Source Data
SSISETLSQL Server
Data Warehouse
SSASCube
[Infrastructure As A Service]
Azure Virtual Machine
SQL ServerStaging Database
On-Premise
Better Azure IAAS Solution
Source Data
SSISETL SQL Server
Data Warehouse
SSASCube
[Infrastructure As A Service]
Azure Virtual Machine
SQL ServerStaging Database
On-Premise
VPN GatewayOr ExpressRoute
PAAS / IAAS Hybrid
Source Data SSISETL
Azure SQL DB / SQL DWData Warehouse
SSASCube
[Platform As A Service]
Azure Virtual Machine
Azure SQL DBStaging Database
On-Premise
VPN GatewayOr ExpressRoute
Azure
PDW / APS / SQL DWPDW / APS = Parallel Data Warehouse / Analytics Platform System
appliance, on premiseSQL DW = Same concept but in Azure. Each node is an Azure SQL DB
Azure SQL Data Warehouse
PAAS / IAAS Hybrid
Source Data SSISETL
Azure SQL DB / SQL DWData Warehouse
SSASCube
[Platform As A Service]
Azure Virtual Machine
Azure SQL DBStaging Database
On-Premise
VPN GatewayOr ExpressRoute
Azure
PAAS Solution
Source Data
AzureData
Factory
ELT
Azure SQL DB / SQL DWData Warehouse
PowerBI
[Platform As A Service]
Azure SQL DBStaging Database
On-Premise Azure
Cheaper Staging – Blob/Table
Source Data
AzureData
Factory
ELT
Azure SQL DB / SQL DWData Warehouse
PowerBI
Azure Blob StorageStaging Database
On-Premise Azure
Blob / Table Storage
Blob• Azure File System• Any file, any format• Unstructured• Cheap
Table Storage• Simple tables• No real indexing• No foreign keys• Basic data types• Basic querying• Structured• Cheap
Integrate Data Lake
Source Data
AzureData
Factory
ETLT
Azure SQL DB / SQL DWData Warehouse
PowerBI
Data LakeStaging
On-Premise Azure
Data Lake
U-SQL (T-SQL + C#)
Processing Engine
Nod
e
Nod
e
Nod
e
Nod
e
Nod
e
Nod
e
Nod
e
Nod
e
Nod
e
Nod
e
Nod
e
Nod
e
Data Storage (HDFS)Structured Semi-Structured Unstructured
Dat
a La
ke
Anal
ytic
sD
ata
Lake
St
ore
Integrate Data Lake
SQL DWData Warehouse PowerBI
Data Lake Store
Staging
Data Lake AnalyticsTransform
Data Lake Analytics
Power User / Data Scientist‘big data’ analytics
Dashboards ExcelSelf-ServiceSQL Queries
/ Reports
HD Insight
Data Lake Analytics Vs HDInsightData Lake• U-SQL• (T-SQL + C#)
HDInsight• Hadoop• Map-Reduce
• Hive/Pig?
Integrate Data Lake
SQL DWData Warehouse PowerBI
Data Lake Store
Staging
Data Lake AnalyticsTransform
Data Lake Analytics
Power User / Data Scientist‘big data’ analytics
Dashboards ExcelSelf-ServiceSQL Queries
/ Reports
HD Insight
Real Time BI
SQL DWData Warehouse PowerBIData Lake
Staging & Transform
Data Lake AnalyticsOr HD Insight Power User / Data Scientist
‘big data’ analytics
Real-timeDashboards
ExcelSelf-ServiceSQL Queries
/ Reports
IoT or Event HubReal-time data
Real time
Batch Data
Stream AnalyticsReal-time insight
Machine Learning
SQL DWData Warehouse PowerBIData Lake
Staging & Transform
Data Lake AnalyticsOr HD Insight
IoT or Event HubReal-time data
Real time
Batch Data
Stream AnalyticsReal-time insight
Machine Learning
Azure Machine Learning
Cortana Analytics Suite
Cortana Intelligence
Cortana Intelligence - Batch
Cortana Intelligence - RealTime
Cortana Intelligence – Analytics
U-SQL / Data Lake Demo
U-SQL
U-SQL – User Defined Functions
U-SQL / Data Lake
U-SQL / Data Lake
Data Lake
U-SQL (T-SQL + C#)
Processing Engine
Nod
e
Nod
e
Nod
e
Nod
e
Nod
e
Nod
e
Nod
e
Nod
e
Nod
e
Nod
e
Nod
e
Nod
e
Data Storage (HDFS)Structured Semi-Structured Unstructured
Dat
a La
ke
Anal
ytic
sD
ata
Lake
St
ore
In Summary…
• Real-time or Batch?
• Virtual Machine with On-Prem solution?
• Or genuine Azure solution?
• U-SQL & Data Lake <- Everything to Everyone!
• Missing:
• SSIS Data flow equivalent
• Analysis Services as a Service
Business Intelligence in Azure
[email protected] www.PurpleFrogSystems.com
@PurpleFrogSys www.PurpleFrogSystems.com/blog