analytics on azure · pdf file big data lambda architecture azure storage polybase azure sql...

Click here to load reader

Post on 25-Apr-2020

8 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • Analytics on Azure What to Use When

    Cloud Solution Architect | Data & AI | One Commercial Partner | United Kingdom

    Christina E. Leo

    [email protected]

    @christinaleo

  • The Azure Data Landscape

    Big Data & Analytics

    HDInsight Stream Analytics

    Analysis

    Services

    People

    Automated Systems

    Apps

    Web

    Mobile

    Bots

    Data Sources

    Apps

    Sensors and devices

    IoT

    IoT Hub

    Prebuild Models

    Cognitive

    Services

    Dashboards &

    Visualizations

    Power BI

    Orchestration

    Data Catalog

    Data Factory

    Event Hubs

    Storage Blob

    Table Storage

    Integrations

    Services

    DataBricks

    Operationalization

    Bot

    Framework

    Functions &

    Serverless

    Azure Search

    Machine Learning

    Data Science &

    Deep Learning VM

    Studio

    Azure ML Services

    ML Development

    Environments

    Server

    Data Storage

    Data Lake Store

    CosmosDB

    SQL Server 2017

    SQL Data

    Warehouse

    SQL Azure

    SQL Managed

    Instance

    D A T A → S T O R A G E → I N T E L L I G E N C E → A C T I O N

  • Solution Scenarios Big Data & Advanced Analytics

    Modern Data Warehousing

    “We want to integrate all our data including ‘big data” with our data warehouse”

    Advanced Analytics

    “We are trying to predict when our customers churn.”

    Real-Time Analytics

    “We are trying to get insights from our devices in real-time, etc.”

  • Value

    Uses of Data

    BI AA AI Descriptive Diagnostic Predictive Prescriptive

    Data Engineering, Warehousing & Preparation

    Dashboards and Reports

    Detailed Statistics

    Machine Learning

    Deep Learning

    Analyst Data Scientist

    ML EngineerData Engineer

    AI Engineer

  • Modern Data Warehousing

    Modern Data Warehousing

    “We want to integrate all our data including ‘big data” with our data warehouse”

    Advanced Analytics

    “We are trying to predict when our customers churn.”

    Real-Time Analytics

    “We are trying to get insights from our devices in real-time, etc.”

    The Modern Data Warehouse extends the scope of the data warehouse to serve “big data” that is prepared with techniques beyond relational ETL.

  • M O D E R N D A T A W A R E H O U S I N G C A N O N I C A L O P E R A T I O N S

  • D A T A W A R E H O U S I N G P A T T E R N I N A Z U R E

    APPLICATIONS

    DASHBOARDSBUSINESS / CUSTOM

    APPS

    (STRUCTURED)

    LOGS, FILES AND MEDIA

    (UNSTRUCTURED)

    r AZURE DATABRICKS

    DATA LAKE

    STORE AZURE

    STORAGE

    HDINSIGHT

    AZURE SQL DW

    AAS

    Loading and preparing data for analysis with a data warehouse

    DATA

    FACTORY

    Azure

    Import/Export

    Service

    API’s, CLI &

    GUI Tools

    COSMOS DB

    COSMOS DB SQL DB

  • Real-Time Analytics Real-time Analytics (aka Stream Analytics) is the phenomenon of processing data as soon as it is generated, to derive very quick analysis/insight for timely action.

    Modern Data Warehousing

    “We want to integrate all our data including ‘big data” with our data warehouse”

    Advanced Analytics

    “We are trying to predict when our customers churn.”

    Real-Time Analytics

    “We are trying to get insights from our devices in real-time, etc.”

  • S T R E A M I N G - C A N O N I C A L O P E R A T I O N S

  • B I G D A T A S T R E A M I N G P A T T E R N W I T H A Z U R E

    REAL-TIME

    APPLICATIONS

    REAL-TIME

    DASHBOARDS BUSINESS / CUSTOM

    APPS

    (STRUCTURED)

    LOGS, FILES AND MEDIA

    (UNSTRUCTURED)

    r

    SENSORS AND IOT

    (UNSTRUCTURED)

    EVENT HUBS IoT HUB KAFKA on HDINSIGHT STREAM

    ANALYTICS

    STORM on

    HDINSIGHT AZURE DATABRICKS

    (Spark Streaming)

    AZURE ML STUDIO

    R SERVER AZURE DATABRICKS

    (Spark ML)

  • D A T A W A R E H O U S I N G P A T T E R N I N A Z U R E

    Big Data Lambda Architecture

    AZURE STORAGE

    PolyBase

    AZURE SQL DATA WAREHOUSE

    AZURE DATA FACTORY

    ANALYTICAL DASHBOARDS

    WEB & MOBILE APPS

    AZURE DATA FACTORY

    AZURE DATABRICKS

    (Spark ML, SparkR, SparklyR)

    AZURE DATABRICKS

    (Spark)

    AZURE HDINSIGHT

    (Kafka)

    AZURE COSMOS DB

    BUSINESS / CUSTOM

    APPS

    (STRUCTURED)

    LOGS, FILES AND MEDIA

    (UNSTRUCTURED)

    r

    event

    AZURE EventHub

    Stream Analytics

  • Advanced Analytics Advanced Analytics is the process of applying machine learning and/or deep learning techniques to data for the purpose of creating predictive/prescriptive insights.

    Modern Data Warehousing

    “We want to integrate all our data including ‘big data” with our data warehouse”

    Advanced Analytics

    “We are trying to predict when our customers churn.”

    Real-Time Analytics

    “We are trying to get insights from our devices in real-time, etc.”

  • A D V A N C E D A N A L Y T I C S - C A N O N I C A L O P E R A T I O N S

  • A D V A N C E D A N A L Y T I C S P A T T E R N I N A Z U R E

    APPLICATIONS

    DASHBOARDSBUSINESS / CUSTOM

    APPS

    (STRUCTURED)

    LOGS, FILES AND MEDIA

    (UNSTRUCTURED)

    r

    SENSORS AND IOT

    (UNSTRUCTURED)

    DATA LAKE

    STORE AZURE

    STORAGE

    HDINSIGHT AZURE DATABRICKS

    AZURE ML ML SERVER

    AZURE ML STUDIO

    SQL Server (In-database ML)

    AZURE DATABRICKS (Spark ML)

    DATA SCIENCE VM

    COSMOS DB

    SQL DB

    SQL DW

    AZURE ANALYSIS SERVICES

    COSMOS DB

    BATCH AI

    SQL DB

    Performing data collection/understanding, modeling and deployment

    DATA

    FACTORY

    AZURE CONTAINER

    SERVICE

    SQL Server (In-database ML)

    SQL DW

  • DEMO

  • T H

    IN G

    S T

    O N

    O T E

    There are no right or wrong solutions,

    only optimal solutions

    We lead with certain solutions and

    customise based on customer scenarios

    Customer voice and product and service

    maturity govern lead solutions

    Consider price and performance, ease of

    use, and ecosystem acceptance as factors

    Everything is fluid - a lead solution today

    might be non-optimal tomorrow, based on

    the factors above and new releases

  • Decision Points

    IaaS PaaS SaaS

    Customer:

    ❑ Regionality

    ❑ VNET requirements

    ❑ Speed of business

    ❑ Data Volumes

    ❑ Data Types

    ❑ Governance

    ❑ Team

    ❑ Skills

  • AI Decision Tree

    https://biz-excellence.com/2019/02/15/ai-dt/

View more