discover data integration aspects of the sap data hub · 2019-11-07 · sap has no obligation to...

Discover Data Integration Aspects of the SAP Data Hub

Axel Schuller

SAP Data Intelligence Product Management

2PUBLIC© 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ

The information in this presentation is confidential and proprietary to SAP and may not be disclosed without the permission of SAP.

Except for your obligation to protect confidential information, this presentation is not subject to your license agreement or any other service

or subscription agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or any related

document, or to develop or release any functionality mentioned therein.

This presentation, or any related document and SAP's strategy and possible future developments, products and or platforms directions and

functionality are all subject to change and may be changed by SAP at any time for any reason without notice. The information in this

presentation is not a commitment, promise or legal obligation to deliver any material, code or functionality. This presentation is provided

without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a

particular purpose, or non-infringement. This presentation is for informational purposes and may not be incorporated into a contract. SAP

assumes no responsibility for errors or omissions in this presentation, except if such damages were caused by SAP’s intentional or gross

negligence.

All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from

expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates,

and they should not be relied upon in making purchasing decisions.

Disclaimer


Data Management in the Intelligent

Enterprise

▪ Use Cases

SAP Data Hub and SAP Data Intelligence

The approach for EIM and beyond:

▪ Native Data Integration

▪ Choregraphed workflows

▪ Metadata Management

Agenda


Challenges in Today's Enterprise LandscapesThe result is landscape complexity

Customer

Experience

Manufacturing

& Supply Chain

Digital

Core

People

Engagement

Network & Spend

Management

Applications

Data Marts

Third-Party Data

Data

Warehouses

Cloud

Datastores

Databases

Data

profiling

Data

cataloging

Data

masking

Speech

recognitionETL

Video

processing

Geospatial

processing

Data

cleansing

ELTTime

series

Text

analytics

Machine

learning

Data

quality

Data

ingestion

Data

replication

Event

Stream

processing

Streaming

analytics

Graph

processingImage

processing

Metadata

management


SAP Data Hub

SAP Data Intelligence

SAP Strategy for Diverse Landscapes Outlook for data integration and processing portfolio

SAP LT Replication Server

Data Quality Management

(DQaaS)

CPI – Data Services (CPI-DS)

Agile Data Preparation

(ADP)

Smart Data Integration

(SDI)

Smart Data Quality (SDQ)

SAP Data Services (DS)


Data Integration Technologies by Purpose

ETL / Data Quality Process

Orchestration

ETL / Replication Complex Event Processing Virtualization

SAP Data

Services

SAP Information

Steward

Batch

Batch

On P

rem

ise /

Clo

ud S

ourc

es

Clo

ud

Sourc

es

SAP Cloud Apps Third-Party Cloud Apps

On-Premise Third-Party On-Premise

PO

BPM BPM

APIs

APIs

CPI

Pre-Packaged Flows

AB

AP

sourc

es

On P

rem

ise/

Clo

ud S

ourc

es

Real-time Replication

SLT ABAP Targets

SAP HANA

Batch

Real-time Replication

SAP HANA

Incoming Streams

SAP HANA

Query Data RemotelyTable/

View

Table/

View

Remote Systems

…

Governance, Pipeline,

Orchestration

Intelligent Enterprises

(SAP Leonardo IoT, SAP Hybris, SAP Concur)

SAP Data Hub

Unstructured

Data

ML Data

Lakes

IoT Cloud

Storage

BW

LT

DS

Open

Source

Collect Prepare Enrich Refine Orchestrate

CPI-DSBatch Batch SAP IBP

On P

rem

ise

Sourc

es

Virtualization


Guiding Principles for EIM Portfolio

Integration of all tools to

enable new use cases and

scenarios.

REUSEINTEGRATE

Bring critical capabilities into

a single innovation

platform.

Protect existing landscapes

by leveraging investments

customer made.

EMBED


SAP Data HubKey capabilities

SAP Applications Distributed & External

Data Systems

SAP Data Hub

SAP HANA

Integration

Cloud Data

Integration*

ABAP

Integration

WorkflowsBW Process

Chains

Data Services

JobsHANA

Flowgraphs

…

SAP C/4HANA

SAP

NetWeaver + DMIS Addon

BW

Integration

SAC Push API

SAP BWSAP BW/4 HANA

SAP Analytics Cloud

(on-premise, cloud, multi cloud)

Standard

Connectors(open & native

protocols)

Cloud Storages

Hadoop / HDFS

Databases

3rd Party Applications

Streaming (e.g. IoT)

Public Clouds

SCI for process

integration

SAP Open

Connectors

SAP API

Business Hub

REST APIs

SAP Cloud

Platform

Connectors

3rd Party

Connectors

Data Pipelining & Processing

Data ingestion / Data Processing / Data Enrichment

Data Orchestration & Monitoring

Connection Management / Workflows / Scheduling

Data Governance

Data Discovery / Data Profiling / Metadata Cataloging

*This is the current state of planning and may be changed by SAP at any time without notice.


IoT Ingestion

and Orchestration

Data Science and

Machine LearningData WarehousingBusiness

Application

Transformation

Transform IoT event

streams into

enterprise-ready

data, and derive

actionable insights

Streamline data science

and machine learning,

from modeling and

development to

operations, across all

enterprise data assets

Build a multifaceted

data warehouse,

across diverse and

distributed data assets

Streamline innovation

initiatives around

Business Applications,

supporting enterprise

transformation programs

• Predictive maintenance

• Product as-a-service

• Intelligent logistics

• Smart manufacturing

• […]

• Smart energy management

• Fraud detection

• Customer churn prediction

• […]

• Consumer 360 view

• Marketing campaign

effectiveness

• Renewable energy

production simulation

• […]

• Customer Risk Intelligence

with S4 HANA Cloud

• Timesheet Analytics with

Fieldglass

• […]

Use SAP Data Hub to reimagine your business processes using …

• Proactive

Information/ Data

Governance

• Digital Data

Stewardship

• […]

Information/

Data

Governance

Understand ,

govern and

secure your data

to deliver trusted

insights


What is SAP Data Intelligence?

Create powerful data pipelines to

leverage your data projects and to

orchestrate the data processing

Harness the advanced machine learning

content to accelerate and scale and

automate your Data Science projects

Manage metadata across a

diverse data landscape and

create a metadata repository

SAP Data Intelligence is a mPaaS comprehensive solution to deliver

data-driven innovation and intelligence across the enterprise, unifying scalable enterprise AI and intelligent information management.

Access &

connect data

Govern &

discover data

Prepare &

label data

Build scalable

& flexible

data pipelines

Deploy & integrate

intelligent

applications

Monitor &

orchestrate

the lifecycle


▪ Central management of all system connections

▪ Data Workflows: Definition of cross-system

processes, e.g.

– Trigger Execution of SAP BW process chains

– Transfer data from from SAP BW

– Execute remote SAP Data Services jobs

– Submit Spark jobs to Hadoop clusters

▪ Scheduling of Data Workflows

▪ Extensive Monitoring

SAP Data Hub: Orchestration and Monitoring

Data Workflows


▪ Data Pipelines: Model flow-based applications

– Operators (independent computation units)

– Data (messages) flows between operators

– Containers constitute the operators’ execution

environments

▪ Over 250 pre-defined Operators in different

categories such Connectivity, Processing, Data

Quality, Computer Vision, Machine Learning

▪ Development of custom Operators e.g. in Python,

JavaSript, Go and R

▪ Content Lifecycle Management

SAP Data Hub: Pipelining & Processing

Pipeline Modeler


SAP Data Hub A pipeline-driven data integration, operations and governance

solution for disparate kinds of data (structured, unstructured,

streaming, cloud etc.), supporting both integration and processing in

a distributed fashion

SAP Data Services Moving application data from transactional sources to data

warehouses, including data quality processes and built-in Data

Integration transformations

Complementary solutions: SAP Data Services and SAP Data Hub

Key use cases

• BI / Traditional Data Warehousing, Data Migration, Data Quality

Characteristics

• ETL in a standalone heterogeneous landscape

• Centralized, on premise & server-based infrastructure

• Relational data focused

• Advanced data transformations & processing (e.g. Join, SQL, DQ…)

Key use cases

• Data science & Machine Learning, Big Data Warehousing, IoT, Data Tiering, Data Network

Characteristics

• Support multiple data ingestion methods

• Pipelining and orchestration of data processes in complex landscapes

• Cloud & on-premise deployment, distributed data processing & serverless computing via

Kubernetes

• Big data focused (tables/views, object storages, any data formats), for both data at rest and

data in motion

• Complex data refinery & processing (e.g. ML, Predictive, Image, custom code)

Files

Databases

Web Services

HANA

Applications

Databases

SAP Data

ServicesBatch Batch

<<Sources…>>

Data movement style Data processing and integration style

IoT Data Stream ML/Predictive Unstructured Data

Enterprise Data Data Warehouse Data Lake/

Object Storages

Interoperability

Orchestration SAP Data HubCollect Prepare Enrich Refine Orchestrate


SAP Data Services and SAP Data HubInteroperability: Example for combined scenario

Machine

Sensor

CRM

Database

SAP Data HubStreaming

OrchestrationSAP Data

Services

Extract

Transform

Deduplicate

Refine Machine Learning

Orchestration

Data

Stores

Cloud Hadoop

HDFSGCS | S3 | WASM

SAP BW/4HANA

12

3

54

Ingest large volumes of

data (e.g. distance, pace,

heartrate, location…)

from machine sensors by

using MQTT/Kafka

operator (SAP Data Hub)

Refine data according to

purpose and store it in

data stores (SAP Data

Hub)

Acquire additional

relevant structured data

(e.g. customers, sales,

behavioral, demographic)

into data stores by

remotely orchestrating

DS jobs (leveraging

existing SAP Data

Services investments)

Apply machine learning

algorithms (e.g.

classification, clustering,

identifying outliers, etc.)

on the data to discover

new insights about user

characteristics (SAP

Data Hub)

Invoke process chain to

ingest the results into

SAP BW/4HANA for

further data analysis and

reporting (SAP Data

Hub)

1 2 3 4 5

Collect Prepare Enrich Refine Orchestrate


Complementary solutions: SAP Cloud Platform Integration and SAP Data Hub

Key use cases and characteristics

• Integration between transactional (cloud and on-premise) applications

• A2A / B2B, B2G / Master Data Synchronization

• API focused (synch., async., business events)*

• Message-based processing (mapping, routing, monitoring)

• Transactional integrity (reliable messaging and error handling)

• Pre-packaged integration content for SAP/Non-SAP scenarios

Key use cases and characteristics

• Data science-driven pipelining and orchestration of disparate kinds of data in

complex landscapes

• Big Data Warehousing, IoT, Machine Learning & Predictive Analytics

• Data focused (table/view, storages, processing, libraries)*

• Distributed data processing & serverless computing

• High frequency event processing (e.g. via Kafka)

• Advanced data transformations & processing (e.g. ML, Predictive, Code)

Process integration style Data processing and integration style

IoT Data Stream ML/Predictive Unstructured Data

Enterprise Data Data WarehouseData Lake/

Object Storages

SAP Cloud AppsThird-Party

Cloud Apps Business Partner

SAP Applications Third-Party SystemsPublic Authorities

SAP Cloud Platform Integration

Interoperability

SAP Data Hub A pipeline-driven data integration, operations and governance

solution for disparate kinds of data (structured, unstructured,

streaming, cloud etc.), supporting both integration and processing in

a distributed fashion

SAP Cloud Platform IntegrationProvides service on cloud platform that facilitates the integration of

business processes and data across on-premise and cloud

applications

* Also table access possible by CPI (like CPI-DS) and API level access possible by SAP Data Hub, but not key focus

SAP Data HubCollect Prepare Enrich Refine Orchestrate


• Orchestration and landscape monitoring for

all data flows:

• Run SAP DS/CPI-DS with SAP Data Hub

• Reuse connectivity

• Share infrastructure

Direction SAP Data Services & SAP CPI-DS with SAP Data Hub

Monitoring Mockup

© 2019 SAP SE or an SAP affiliate company. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of

SAP SE or an SAP affiliate company.

The information contained herein may be changed without prior notice. Some software products marketed by SAP SE and its

distributors contain proprietary software components of other software vendors. National product specifications may vary.

These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or

warranty of any kind, and SAP or its affiliated companies shall not be liable for errors or omissions with respect to the materials.

The only warranties for SAP or SAP affiliate company products and services are those that are set forth in the express warranty

statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional

warranty.

In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or

any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation,

and SAP SE’s or its affiliated companies’ strategy and possible future developments, products, and/or platforms, directions, and

functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason

without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or

functionality. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ

materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, and they

should not be relied upon in making purchasing decisions.

SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered

trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. All other product and service names

mentioned are the trademarks of their respective companies.

See www.sap.com/copyright for additional trademark information and notices.

www.sap.com/contactsap

Follow us

https://www.sap.com/copyright

https://www.sap.com/registration/contact.html

https://www.linkedin.com/company/sap

https://www.youtube.com/user/SAP

https://twitter.com/sap

https://www.facebook.com/SAP

discover data integration aspects of the sap data hub · 2019-11-07 · sap has no obligation to...

Documents