going beyond power bi: leveraging microsoft · empower your teams to securely and creatively work...

Going Beyond Power BI: Leveraging Microsoft Azure for Operational Analytics

Senior architect

Email: [email protected]

Masood Pehlvi

#HSCCATLANTA19

Principal Consultant

Email: [email protected]

Josh Reaves

#HSCCATLANTA19

Breakout Track 6: Power Platform

Overview

D365 Architectures

D365 Reporting Options

D365 FO Deep Dive

Conditions of Data

05

07

09

10

17

C O N T E N T S

Analytics Delivery Models

The Connected Customer

Optimized Architectures

Advanced Architectures

Key to Analytics Success

23

28

31

44

50

Pa r t 1 : Po w e r B I & D 3 6 5 Pa r t 2 : A z u r e A n a l y t i c s

/ 4Think Digital Customer Conference 2019

Part 1D365 Reporting - Power BI is Not Plug-n-Play


D365 Reporting Overview


Overview

1. Reporting Architectures (D365 FO and D365 CE)

2. Reporting Options (D365 FO and D365 CE)

3. Out-of-the-Box Power BI Reporting (D365 FO and D365 CE)

4. Unique Considerations for D365 FO Reporting Power BI Embedded

• Entity Store• Aggregate Measures --> Aggregate Data Entities

BYOD• Custom Entities --> Staging Tables in Azure SQL

Future State of Entities OData

• Performance Issues

Why is Power BI not plug-n-play in D365?


D365FO Reporting Architecture• Financial Reporting

• SSRS / PBIRS Paginated reporting

• Power BI Data Visualisation Ad Hoc analysis

• Excel Analytic reports & pivot tables

Dynamics365 BYODB Data Mart

Tactical / Strategic Reporting

OtherData

Std. Reporting

OperationalReporting

AggregationHistoric dataSnapshots

FinancialReporting Optional

Excel

Power BI

Real-time NearReal-time

Entity Store

NearReal-time

* BYOD requires additional Azure SQL DB


D365CE Reporting Architecture

• CRM Reporting (SSRS) Paginated reporting

• Power BI Data Visualisation Ad Hoc analysis

• Excel Analytic reports & pivot tables

Dynamics 365

Data Export Data Mart

Tactical / Strategic Reporting

OtherData

CRMReporting

OperationalReporting

AggregationHistoric dataSnapshots

Optional Excel Power BI

Real-timeNear

Real-time

* Data Export requires additional Azure SQL DB


D365 Reporting Options

1. Out-of-the-Box (Analytical Workspaces)

2. Reporting Services: SSRS ---> PBIRS

3. Excel

4. Financial Reporter

5. Power BI Embedded (Aggregate Measures)

6. Power BI Service

7. Complex Analytics

8. Power BI Dataflows – Coming Soon

1. Out-of-the-Box (Content Packs)

2. Reporting Services: SSRS

3. Excel

4. Personal Dashboards in D365 CE

• Embed Power BI Tiles

5. Power BI Service

6. Complex Analytics

7. Power BI Dataflows (CDS-A)

D365 FO D365 CE


Considerations for Reporting in D365FO


D365 FO Out-of-the-Box Reports The following Power BI solutions are available for D365FO and are based on the Entity Store:• Actual vs budget• Cash overview• Compensation and benefits• Cost accounting analysis• Credit and collections management• Employee competencies and development• Financial performance• Fixed asset management• Organizational training• Practice manager• Production performance• Purchase spend analysis• Recruiting• Sales and profitability performance• Vendor payments• Warehouse performance• Workforce metrics

Advantages• Power BI Content is prebuilt (Microsoft refreshes the

content frequently)• Near real-time access to data (based on Entity Store

refresh frequency)• Entity Store is optimized for reporting• Ability to drill through transactional data • Power BI license included• Standard AX package deployment *

Challenges• Prebuilt content rarely meets the unique needs of an

organization’s D365FO implementation• More D365FO Customization = Less Reliability on Out-of-

the-Box reports for Operational Reporting• Limits ability to integrate with other data sources• Limits customization of KPIs, data model, and design• Security is tied to D365FO, which may be too constricting• Standard AX package deployment *


D365FO Reporting ArchitectureD365FO

BYOD

PrimaryDB

SecondaryDB

Seconds

Entities

Entity Store

Minutes

OData

Azure SQL database

Power BI

Power BI

Direct Query

Operational Data Store

*Refresh

*Refresh

*Refresh

AutoRefresh

*Refresh

DataMart


D365FO Architecture: Integration Components

Data Entity (simplified projection of schema)

Azure SQL DatabaseRowStore & ColumnStore

OData(HTTPS/JSON/XML)

REST(HTTPS/JSON)

SOAP(HTTPS/XML)

X++ ClassX++ Class Table(s) Relational

QueryAggregate

Query

Single record orbatched records Data Management Framework (DMF)

Transform Validation Mapping

Asynchronous high volume interface

File Upload Data Staging

Application Server (AOS)

• Data management tasks, e.g.• Data migration / BYOD• Move data between AX instances (Test Prod)• Configuration & master data synchronization

• Recurring file-based integrations

Azure App Services(BizTalk or Logic Apps)Custom Apps

& integrations

2500+ entities (as of 8.0)

Synchronous interface


D365FO Reporting Architecture


D365FO Reporting ArchitectureD365FO

PrimaryDB

SecondaryDB

Seconds

Entities

Entity Store

Minutes

Power BI

Power BI

Operational Data Store

*Refresh

AutoRefresh

*Refresh

DataMart

BYOD

OData

Azure SQL database

Direct Query

*Refresh

*Refresh

ADLS


D365FO Reporting Options

1. Out of the Box

2. Reporting Services: SSRS ---> PBIRS

3. Excel

4. Financial Reporter

5. Power BI Embedded6. Power BI Service7. Complex Analytics 8. Power BI Dataflows – Coming Soon

Power BI Advantages:1. Flexible Data Access: OData / DirectQuery /

Composite2. Multiple Data Sources: Entity Store / Custom

Data Entities / Azure / On premise Related Systems

3. Creative Power BI builds: prebuilt Content Packs or self-service reporting

4. Tight Integration: Workspaces, Links / Pins5. Ease of Deployment: Publish / AX Package

Deployment / LCS6. Future Integration with the most common

business entities across the major functions such as marketing, sales, service, finance (through Dataflows and the Common Data Model schema)


Conditions of Data


Conditions of Data

Steps in creatinga report

1. Create

2. Prepare

3. Augment

4. Analyze

5. Visualize

6. Act


Conditions of Data

1. Create

2. Prepare

3. Augment

4. Analyze

5. Visualize

6. Act

D365FO / D365CE transactional data

Power BI report creation

Business / User action

Entity development• Data entities: BYOD (D365FO) or CDS-A

(D365CE)• D365FO Aggregate measures (Power BI

Embedded)


Power BI is the future

Power BI Embedded is included with D365 FO

Power BI can bring your enterprise BI together

But…Power BI is not plug-n-play for D365 FO or D365 CE

Considerations needed for entity development and report creation

Allow for effort & resources beyond D365 FO and D365 CE implementations!


Part 2Using Azure to Optimize Analytics Architectures


Overview

1. Azure Analytics Delivery Models

2. The Connected Customer

3. Optimized Azure Architectures

4. Advanced Azure Capabilities

5. Unlocking the Secret to Analytics Success

How can Azure technologies be leveraged best for enterprise analytics?


Azure Analytics Delivery Models

Gartner Analytics Maturity Model

5

In the past, this growth could have taken decades to achieve for an organization.

Now, this kind of transformation is possible in months.

Surprisingly, 87% of organizations still show low maturity.

Source: Gartner 2018

Why?

https://www.gartner.com/en/newsroom/press-releases/2018-12-06-gartner-data-shows-87-percent-of-organizations-have-low-bi-and-analytics-maturity

6 Key Elements of the Modern Analytics Maturity Model

Business Alignment Information derived from analytics and IoT is viewed as an integrated part of operations and strategy.

Common Business Language

Common information model standards, metrics, and rules are established and followed.

Governance Information policies and mandates are well documented, published, and utilized.

Information Culture

Information is considered an intangible enterprise asset and is used to maximize business goals and strengthen partnerships.

Sponsorship & Funding Senior level sponsorship maintains an enterprise budget.

Technology & Architecture

Formal IT management with shared architecture across services support a solid infrastructure. Cloud technologies are driving OCM.

/ 26

The Microsoft Azure EcosystemHitachi Solutions Provides Global Offerings across Microsoft Azure Technologies with Deep Expertise

Modern SolutionsCustom Applications whether they are web, mobile, or hybrid, we handle the entire process from idea to implementation and deployment. Capax does this with clear process, standards, and an unrivaled commitment to quality.

Cloud WorkplaceEmpower your teams to securely and creatively work together in the way that fits them best. We can shepherd you to the cloud reducing your costs, keep you safe, and increate your agility.

Data + AnalyticsWe are experts at Azure Data Services and all of the related components that make up a modern data estate. Our team of experts has built a strong foundation on solving enterprise problems and enabling the business.

Data Science + AIAt Capax Global, we've shown how data science initiatives can deliver significant value to enterprises when the right opportunities are identified, the business uses data to make decisions, and machine learning talent is embraced.

Business ApplicationsHitachi Solutions is a proven leader in the global Microsoft Dynamics channel, bringing a unique industry-first approach, backed by innovative Cloud IP, and world-class delivery.

Platform ModernizationChanging a technology that underpins an organization's applications and data platform. Capax has specific tooling for complex cases that reduce risk and help enable customers to adopt and innovate. Dynamics 365

/ 27

SharePoint

Teams

O365

IaaSManagedServices

IoT

AppDev

PowerPlatform

Analytics

Data & AIProductivity

Azure Infrastructure

Azure Platform

DigitalTransformation

Dynamics 365

Retail & POS

Finance & Operations

Unified Operations

• Leveraging D365 and Azure to drive Digital Transformation

• Use D365 and Azure capabilities to be more efficient and increase competitive advantage

• Optimizing Azure technologies to support, enhance, and extend D365 implementations

• Build an architectural foundation to facilitate advanced analytics to improve business outcomes:• Machine Learning

• Artificial Intelligence

• Develop a modern data platform

Microsoft Azure CapabilitiesDigital Transformation on the Azure Cloud

AI & ML

CognitiveServices


The Connected Customer

43.1° | 35.2°91.6° | 87.3°

MAINTENANCE INITIATED

us.hitachi-solutions.com

Hitachi Solutions: The Connected Customer

OPERATIO

NS

OPERATIONSCUSTOMER MASTERASSET MASTERDEVICECONNECTIVITY

STANDARDIZING CUSTOMER DATASTANDARDIZING ASSET DATA

ASSET LOCATIONSBOM AND SUB ASSEMBLIESEQUIPMENT CONNECTIVITY

EVENT MONITORINGEVENT RULES

INTELLIGENCEPREDICTIVE FAILURE RATESROOT CAUSE

IOT EVENT ANALYSISEQUIPMENT PERFORMANCE

EQUIPMENT MAINTENTANCEERROR CODES TO RESOLUTION

PREVENTIVE MAINTENANCEPREDICITVE MAINTENANCE

ROUTE OPTIMIZATIONRESOURCE OPTIMIZATION

INNOVATIONEQUIPMENT AS A SERVICEREMOTE RESOLUTIONREAL TIME PERFORMANCE

OUTCOME BASED MODELSPERFORMANCE MODELINGPREDICTIVE RESOLUTIONPREDICTIVE FAILUREEQUIPMENT OPTIMIZATIONSELF HEALING

PROCESSESCONTACT CENTER

MAINTENANCE PROCESSES

FIELD SERVICE

CASE & CONTACT CENTERRESOLUTION PROCESSKNOWLEDGE MANAGEMENTEQUIPMENT SPECFIC PROCESSESPARTS FORECASTINGRESOURCE FORECASTINGRMA & CLAIMS MANAGEMENTWORK ORDER PROCESSESAUGMENTED REALITYWARRANTY & CONTRACTS

us.hitachi-solutions.com

Provide End-to-End Solutions

Customer 360

Omni Channel

Agent Experience

Chat Bots

Warranty & Claims

Order Management

Self Service

Engineering

Management

MES (Including Lean)

Demand Planning

Inventory Management

Pricing & Trade

Management

Procurement

Vendor Management

Equipment

Management

Equipment Warranties

Work Order

Management

Depot Management

MRO

Advanced Timesheets

Preventative

Maintenance

Connected Devices

Predictive Maintenance

Predictive Failures

Automated Scheduling

Rules Based

Conditioning

Device Templates

Suggested Actions

Single Source of Truth

Master Data

Management

Data Modeling

Power BI

SSIS, SSRS, SSAS

Azure Architectures

Azure Data Factory

Machine Learning

Artificial Intelligence

Sales Force

Automation

Marketing Automation

Customer Portals

eCommerce

Configurator

Dealer Management

Account Planning &

Forecasting

Marketing & Sales

S E R V I C E O P E R A T I O N S F I E L D S E R V I C E

S A L E S

Customer & Call Center

Business Systems Equipment & Service

A N A LY T I C S

Business Analytics

I O T

Intelligent Devices


Optimized Azure Architectures

Benefits Choice of OS, dev environment

Choice of deployment

Lowest TCO

Strongest security

Best performance

Modernize applications across cloud and edge

DBaaS for lift & shift and modern applications

On premises

Fully managed database services

Intelligence, perf, scale, HA built-in

Advanced security

Most compliant

Choice of RDBMS flavors

Global distribution

Elastic scalability of storage and throughput

Single digit latency guaranteed, leading SLAs

5 consistency models

Support for most popular APIs

Azure SQL DBAzure MySQLAzure PostgreSQLAzure MariaDB

Build new, globally-distributed, cloud-native applications

Globally Distributed

Azure Cosmos DBSQL Server

Analytics

Modern data warehouse leveraging OSS and governance

Azure SQL DWAzure HD InsightAzure Data LakeAzure DatabricksAzure Data FactoryAzure Data CatalogAzure Stream AnalyticsPower BI

Global scale

Security & privacy

Best of open source & Microsoft AI innovation

Deploy ML models on premises, in the cloud or on the edge

Analytics for all, data democratization

Products

Azure Data Product StrategyLift & Shift and SaaS

Platform Services

Infrastructure Services

WebApps

MobileApps

API Apps

Notification Hubs

HybridCloud

Backup

StorSimple

Azure SiteRecovery

Import/Export

SQL Database CosmosDB

Redis Cache

AzureSearch

StorageTables

SQL DataWarehouse

Azure AD Health Monitoring

AD PrivilegedIdentity Management

OperationalAnalytics

Cloud Services

Batch RemoteApp

ServiceFabric

Visual Studio

ApplicationInsights

Azure DevOps

Domain Services

HDInsight MachineLearning Stream Analytics

Data Factory

EventHubs

Data LakeAnalytics Service

IoT Hub

Data Catalog

Security & Management

Azure ActiveDirectory

Multi-FactorAuthentication

Automation

Portal

Key Vault

Store/Marketplace

VM Image Gallery& VM Depot

Azure ADB2C

Scheduler

Xamarin

HockeyAppPower BI Embedded

SQL Server Stretch Database

MobileEngagement

Functions Cognitive Services Bot Framework Cortana

Security Center

Container Service

VM Scale Sets

Data Lake Store

BizTalkServices

Service BusLogic Apps

API Management

Content DeliveryNetwork

Media Services

Media Analytics

https://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0ahUKEwiy6vj9mtDNAhVB4mMKHUztCUcQjRwIBw&url=https://developer.microsoft.com/en-us/app-middleware-partners&psig=AFQjCNFdctU6OJlviMmzVX3AK3wCAgJKIw&ust=1467391908422760

https://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0ahUKEwijrpzbm9DNAhVU52MKHRcjAxMQjRwIBw&url=https://azure.microsoft.com/en-gb/services/power-bi-embedded/&psig=AFQjCNGAz6ASOEuOFwrvcGbgivFPur_t7w&ust=1467392103935851


Azure Storage and Processing

Azure Storage and ProcessingFoundational Building Blocks of AzureAzure Services: SQL Data Warehouse, HDInsight, Data Lake Store, Event Hubs, IoT Hubs…Microsoft Services: Office 365, OneDrive, XBox, Skype…

Hyper ScaleDurableSecureHighly AvailableOpenHybrid

Azure Storage +Data Platforms

Object Storage

Data Transport

File storage

Hybrid Storage

Lift and shift of legacy applications that require file

shares to the cloud

Secure, centralized storage target for

backup/disaster recovery

Move or migrate data into Azure

Secure, intelligent data tiering between on-premises and

cloud storage

Azure Blobs Azure FilesAzure NetApp Files

Azure Import/ExportAzure DataBox

Azure StorSimpleAzure File Sync

Avere

Disk Storage

Reliable, persistent, high performing storage for

Virtual Machines

PremiumStandard

Azure Data LakeAzure SQL DBAzure SQL DWAzure Blob StorageAzure Data FactoryDatabricks

A Z U R E D A T A L A K E S T O R EA highly scalable, parallel, file system in the cloud specifically optimized for big data Analytics

No limits on: number of files, size of individual files, total amount of data stored, how long data can be stored or ingestion throughput

Low latency and high throughput workloads can be used for ingesting streaming data.

Stores all data types Provides POSIX-style permissions for RBAC Integrates with AAD for authentication Optimized for Common Data Model (CDM)

Development and Integration with Dynamics 365 Backend Storage in Azure

Data Node Data Node Data Node Data Node Data NodeData Node

Shard Sha

rd Shard Sha

rd Shard Sha

rd Shard Sha

rd Shard Sha

rd

Shard Sha

rd Block Block Block Block Block Block

Block 1 Block 2 Block n …

Azure Data Lake Store File

/ 38

Azure Data Lake: The Core of Modern Data Architectures for Dynamics 365 and Power BI

Cognitive

Unified Operations

Customer Engagement

CDSApps


Azure Automation

/ 40

Azure Automation: Running Code in PowerShell Runbooks on Schedules

C H O O S I N G A Z U R E A U T O M A T I O NWhen Azure Automation can be a good option for cloud-based automation

When You Want .. Description

Process automation Automate frequent, time-consuming, and error-prone cloud management tasks by authoring runbooks graphically, in PowerShell, or Python.

Configuration management

Manage your desired state configuration (DSC) resources and apply configurations to virtual or physical machines in Azure. Monitor and automatically update machine configuration across physical and virtual machines, Windows or Linux, in the cloud or on-premises. Collect inventory about in-guest resources and track changes across services, daemons, software, registry, and files.

Update managementUpdate Windows and Linux systems across hybrid environments, and gain visibility of update compliance across Azure, on-premises, and other clouds. Schedule deployments to orchestrate installation of updates within a defined maintenance window.

Build and deploy resourcesDeploy Azure resources using Runbooks and Azure Resource Manager (ARM) templates. Integrate into development tools like Jenkins and Azure DevOps, ensuring continuous delivery and operations automation.

C H O O S I N G A Z U R E R E S O U R C E M A N A G E R T E M P L A T E S

When Azure Resource Manager (ARM) templates can be a good option for cloud-based automation

When You Want .. Description

Consistently and repeatedly deploy resources

ARM templates are composed of a JavaScript Object Notation (JSON) file that defines one or more resources, including any dependencies between them. This adds the benefit of treating your resources for a solution as a single unit, rather than independent components, making it easier to consistently deploy and manage the resources to development, test, staging, and production environments.

Manage your infrastructure through declarative templates rather than scripts

Declarative templates make it easier to define your resource parameters, dependencies, and infrastructure, compared to executing a series of scripts. Furthermore, you can apply tags to resources to logically organize all the resources in your subscription.

Include your infrastructure definition as part of your app source code

The template can become part of the source code for your app. Check it in to your source code repository and update it as your app evolves. Simply edit the template through Visual Studio or your favorite IDE.

Ensure your resources are deployed in the correct order

Resource dependencies are declaratively expressed within the ARM template. This ensures that components add their dependencies when they are provisioned, and that the resources within the template are created in the proper order. For instance, an ARM template that creates a VM and a VNet to which it is added, first creates the VNet, then creates the VM and associates the two.

What you can do with Azure Functions


Advanced Azure Capabilities


Advanced Azure Architectures

Advanced Azure Architectures

Modern Data Warehousing

“We want to integrate all our data including ‘big data” with our data warehouse”

Advanced Analytics

“We are trying to predict when our customers churn.”

Real-time Analytics

“We are trying to get insights from our devices in real-time, etc.”

Some analytics solutions require more complex, advanced Azure architectures to facilitate optimized, continuous integration and continuous deployments to support Modern Data Warehousing, Advanced Analytics like Machine Learning and AI, and Real-time Analytics from IoT devices.

M O D E R N D A T A W A R E H O U S E I N A Z U R E

APPLICATIONS

DASHBOARDSBUSINESS / CUSTOM APPS

(STRUCTURED)

LOGS, FILES AND MEDIA(UNSTRUCTURED)

rAZURE DATABRICKS

DATA LAKE STORE

AZURE STORAGE

HDINSIGHT DATA LAKE ANALYTICS

AZURE SQL DW

AAS

Loading and preparing data for analysis with a data warehouse

DATA FACTORY

AzureImport/Export

Service

API’s, CLI & GUI Tools

Azure Data Box

COSMOS DB

COSMOS DB SQL DB

A D V A N C E D A N A L Y T I C S I N A Z U R E

APPLICATIONS

DASHBOARDSBUSINESS / CUSTOM APPS

(STRUCTURED)


r

SENSORS AND IOT(UNSTRUCTURED)

DATA LAKE STORE

AZURE STORAGE

HDINSIGHT AZURE DATABRICKS

AZURE ML ML SERVER

DATA LAKE ANALYTICS

AZURE MLSTUDIO

SQL Server (In-database ML)

AZURE DATABRICKS(Spark ML)

DATA SCIENCE VM

COSMOS DB

SQL DB

SQL DW

AZURE ANALYSIS SERVICES

COSMOS DB

BATCH AI

SQL DB

Performing data collection/understanding, modeling and deployment

DATA FACTORY

AZURE CONTAINER SERVICE

SQL Server (In-database ML)

B I G D A T A S T R E A M I N G ( R E A L - T I M E ) I N A Z U R E

REAL-TIMEAPPLICATIONS

REAL-TIME DASHBOARDS

BUSINESS / CUSTOM APPS

(STRUCTURED)


r

SENSORS AND IOT(UNSTRUCTURED)

EVENT HUBS IoT HUB KAFKA on HDINSIGHT STREAM ANALYTICS

STORM on HDINSIGHT

AZURE DATABRICKS(Spark Streaming)

AZURE ML ML SERVER AZURE DATABRICKS(Spark ML)

Unlocking the Secret to Analytics Success


Azure Analytics Delivery ModelsGetting the Right Information

to the Right People

at the Right Time

via the Right Channels and Platforms


Q & A


Thanks!

going beyond power bi: leveraging microsoft · empower your teams to securely and creatively work...

Documents