cloud scale analytics with azure · • power bi meets azure sql data warehouse: amplifying your...

Darwin Schweitzer

Cloud Scale Analytics with Azure

Frank J Sprague

Massachusetts Bay Transportation Authority

Boston Subway 1897

NY Subway 1904

Artificial Intelligence:

Changing the Game for Everyone

Metro

http://mapa-metro.com/en/Taiwan/Taipei/Taipei-MRT-map.htm

Start or continue your Data & AI Journey

Agenda

• Define Cloud Scale Analytics (CSA)

• Common architectural patterns

• Roles involved in building solutions

• Demo of Azure services used to build CSA

• Customer example of CSA

• Resources

© Microsoft Corporation

Today, 80% of

organizations adopt cloud-first

strategies

AI investment

increased by

300% in 2017

Data will grow to

44 ZB in 2020(1 ZB = 1 trillion GB)

Today, 80% of

organizations adopt cloud-first

strategies

AI investment

increased by

300% in 2017

Data will grow to

44 ZB in 2020(1 ZB = 1 trillion GB)

Cloud AIData

© Microsoft Corporation

Cloud

Data AI

Organizations that harness data, cloud, and AI outperform

AIIoT

ABCs of Cloud Scale Analytics

Infrastructure

Data

Apps

ABCs of Cloud Scale Analytics

© Microsoft Corporation

Advanced Analytics

Social

LOB

Graph

IoT

Image

CRM

Cloud

Ingest Store Prep & train Model & serve

Data orchestration and monitoring

Big Data store

Analytics engines

Data warehouse

BI + Reporting

Real Time Analyticsin Custom Apps

Big Data and data warehouse

Azure Data Factory Azure Blob storage

Azure Data Lake Store

Azure Databricks

Azure HDInsight

Azure Data Lake Analytics

Azure SQL Data Warehouse

Azure Analysis Services

© Microsoft Corporation

Implementing common architectural patterns

© Microsoft Corporation

Advanced analytics on Big Data

Ingest Store Prep & train Model & serve

Cosmos

DB

Business/custom

apps

(structured)

Files

(unstructured)

Media

(unstructured)

Logs

(unstructured)

Azure Blob StorageAzure Data Factory Power BI

SparkR

Azure Databricks

Microsoft Azure also supports other Big Data services like Azure HDInsight, Azure Machine Learning, and

Azure Data Lake to allow customers to tailor the above architecture to meet their unique needs.

Apps

Code-free data integration at scale for Azure SQL Data Warehouse with Azure Data Factory

© Microsoft Corporation

Ingest Store Prep & train Model & serve

Modern data warehouse

Azure Blob Storage

Logs

(unstructured)

Azure Data Factory

Azure Databricks

Microsoft Azure also supports other Big Data services like Azure HDInsight and Azure Data Lake

to allow customers to tailor the above architecture to meet their unique needs.

Media

(unstructured)

Files

(unstructured)

PolyBase

Business/custom

apps

(structured)

Azure SQL Data

Warehouse

Azure

Analysis

Services

Power BI

Code-free data integration at scale for Azure SQL Data Warehouse with Azure Data Factory

© Microsoft Corporation

Ingest Store Prep & train Model & serve

Real time analytics

Sensors and

IoT

(unstructured)

Apache Kafka for

HDInsight

Cosmos

DB

Files

(unstructured)

Media

(unstructured)

Logs

(unstructured)

Azure Blob StorageAzure Data Factory

Azure Databricks

Real-time

apps

Business/custom

apps

(structured)Microsoft Azure also supports other Big Data services like Azure IoT Hub, Azure Event Hubs,

Azure Machine Learning, and Azure Data Lake to allow customers to tailor the above

architecture to meet their unique needs.

HSBC PayMe

Build Cloud native apps using MySQL that scales up to 500M transactions a day

Azure MySQL

Cosmos DB

Click

Stream

APP DBs

APP DBs

AKSAzure APIM

Social

timeline

Azure

SQL

Analytical platform

Events

EventHub

EventHub

Azure

Storage

Azure Databricks

Model

development

& testing

Data engineers & scientist

(Python, R, Scala …)

Application platform

Tableau

Insights

generation

Azure MySQL AKS

> To APP

Grap

h

data

Technical Roles

• Developer

• Data Engineer

• Data Scientist

• Data Analyst

© Microsoft Corporation

Demo of Cloud Scale Analytics

Azure Databricks

Azure Data Factory

Azure Machine Learning service

Azure Machine Learning service

Build and Train Models

Automated ML

Validate

Containerize and test on Azure

Container Instance

Train Models

Azure ML Compute

Monitor

Monitor models with Azure ML

Deploy Models

Scale with Azure Kubernetes Service

CI/CD and model retraining

Azure DevOps integration

Azure SQL

Power BI

Seamless integration, everywhere

Business analysts

No code, low code

IT professionals, Data scientists

Low to high code

CDM folders

Azure Data Lake Storage

Visualize and report

Power BI

Power BI

Dataflows

Ingest

Orchestrate & move

Azure Data

Factory

Ingest

Model & serve

Azure SQL Data

Warehouse

Train & predict

Azure Machine

Learning

Advance data prep

Azure

Databricks

Azure MySQL

Cosmos DB

Click

Stream

APP DBs

APP DBs

AKSAzure APIM

Social

timeline

• Current platform supports all risk, regulatory and self-service reporting on all PayMe data.

• We also have a data science platform which does adhoc insights generation in batch mode (on near-realtime data).

• Graph-database modelled to store customer-merchant-transaction relationships catering to the social-timeline users.

Azure

SQL

Analytical platform

Events

EventHub

EventHub

Azure

Storage

Azure Databricks

Model

development

& testing

Data engineers & scientist

(Python, R, Scala …)

Application platform

Tableau

Insights

generation

Azure MySQL AKS

> To APP

Grap

h

data

© Microsoft Corporation

Retail &

consumer goods

Discrete

manufacturing

Government &

education

Professional

services

Banking &

financial services

Healthcare

Big Data + DW solutions in action

Workshop

Cloud Scale Analytics with Azure Workshop

Time : 2019-05-22 15:30 - 16:30

Workshop Azure Service

SQL Server 2019 Big Data Clusters

• Read and write directly to

HDFS using SQL or Spark

• Elastically scale compute

and storage with K8s

• Combine and cache data

from relational and non-

relational data sources

https://www.microsoft.com/en-us/sql-server/sql-server-2019

ResourcesBuild Sessions• Code-free data integration at scale for Azure SQL Data Warehouse with

Azure Data Factory

• Managing your ML lifecycle with Azure Databricks and Azure ML

• Azure AI: Making AI real for every developer and every organization

• Power BI Meets Azure SQL Data Warehouse: Amplifying your data stack with petabyte-scale analytics

• Deploying apps for on-premises and hybrid ML+AI on SQL Server 2019 big data clusters

• Build cloud native apps using MySQL that scale to 500 million transactions a day on Azure

GitHub Repos

• Cloud Scale Analytics Workshop

https://aka.ms/cloud.scale.analytics

• Azure Machine Learning Notebooks

https://aka.ms/aml.notebooks

• SQL 2019 Big Data Clusters

https://aka.ms/sqlworkshops

Not Matter what your Role

• Business Owner

• Architect

• Implementor

• End-user

You must Start. Your must Continue

THANK YOU!

Darwin Schweitzer | WW INTELLIGENT CLOUD – Big Data/AI Advanced Workload LeadWorldwide Commercial Business (WCB) – Intelligent Clouddarsch@microsoft.com | Twitter @DataSnowman | GitHub DataSnowmanPlease check out Data and AI and IoT resources at https://github.com/Azure/data-ai-iot