0400 - data provisioning - direct extractor connection

SAP HANADirect Extractor Connection

© 2012 SAP AG. All rights reserved. 2

Agenda

OverviewRationale SAP HANA Direct Extractor Connection (DXC)SAP HANA Direct Extractor Connection DetailsSAP Business Content DataSource ExtractorsSAP HANA Direct Extractor Connection Setup & ConfigurationComparison with other SAP HANA Data Acquisition TechniquesAppendix: DXC Sidecar Variation


Agenda


Overview


OverviewSAP HANA Direct Extractor Connection

An additional data acquisition technique for working with data from SAP Business Suite systems has been added to the existing techniques for HANA data acquisition:

SLT Replication Data Services…and nowSAP HANA Direct Extractor Connection (DXC)


OverviewRationale

Simple, low-TCO data acquisition for SAP HANA leveraging existing delivered data models

Customer projects may face significant complexity in re-modeling entities from SAP Business Suite systems

Data from some areas in SAP Business Suite systems needs application logic to appropriately represent business documents


OverviewRationale

SAP Business Content DataSource Extractors have been available for many years as a basis for data modeling and data acquisition for SAP BW

SAP Business Content DataSource Extractors provide low-TCO data acquisition mechanism to use in SAP HANA data mart scenarios

In many use cases, batch-driven data acquisition is sufficient (e.g. every 15 min)


OverviewSAP HANA Direct Extractor Connection (DXC): Aspects

Leverage pre-existing foundational data models of SAP Business Suite entities for use in SAP HANA data mart scenarios

Significantly reduces complexity of data modeling tasks in SAP HANASpeeds up timelines for SAP HANA implementation projects

Provides Semantically Rich Data from SAP Business Suite to SAP HANAEnsures data appropriately represents the state of business documents from ERPApplication logic to “make sense of the data” already built into many extractors


OverviewSAP HANA Direct Extractor Connection (DXC): Aspects

Simplicity / Low TCORe-uses existing proprietary Extraction, Transformation, and Load mechanism built into SAP Business Suite systems, over a simple http(s) connection to SAP HANANo additional server or application needed in system landscape

Change Data Capture (delta handling)Efficient data acquisition – only bring new or changed data into SAP HANADXC provides a mechanism to properly handle data from all delta processing types


OverviewSAP HANA DXC Concept: Illustration

SAP ERP

Extractor

DataSource (flat structure)

ERP data

SAP HANA

Status

Data flow redirected from embedded BW: transferred over http connection

Data models in HANA are built using active data table of In-Memory DataStore object

InfoCubes

Data Store Objects

InfoObjects

Embedded BW

generic data transfer

DataSource

Generic data transfer

PSA

Activation

Queue

Active

Version

In memory DSO

Activation Processing

• Data load into HANA activation queue

• Separate activation step

• Scheduled batch jobs


Agenda


Rationale SAP HANA Direct Extractor Connection (DXC)


Rationale SAP HANA Direct Extractor Connection Pre-Existing Foundational Data Models of SAP Entities for use in SAP HANA

ChallengesData stored in many various tables, with high complexity in many modules of SAP Business Suite systems LT “real-time” approach Uses base tables in the SAP Business Suite as a basis for data modeling of SAP Business Suite entities Project solution: Model SAP entities from scratchIn some cases Big challenges b/c of high complexity in the SAP Business Suite system

DXC Benefits:Leverage SAP Delivered Business Content DataSources Which are existing foundational data models for key entities in SAP Business Suite systemsSignificantly reduces complexity of data modeling tasks in SAP HANASpeeds up timelines for customer’s implementation projects


ChallengesIn many modules of SAP Business Suite systems Application logic needed to have semantically rich data (data appropriately reflecting the state of business documents)LT “real-time” approach Uses base tables in the SAP Business Suite as a basis for data modeling Semantically rich data not provided “out of the box”

Project solution:Implement business logic from scratch to properly represent SAP Business Suite data It can be extremely challenging to determine proper application logic to implement on a project basis in SAP HANA (depending on use case)

DXC BenefitsDXC uses SAP DataSource Extractors provides semantically rich data “out of the box”

Ensures the data appropriately represents the state of business documents from ERPApplication logic to “make sense of the data” already built into many extractorsAvoid potentially difficult work of “reinventing the wheel” on a project basis in HANA -> reimplement application logic in HANA which is already provided in DataSource extractors

Rationale SAP HANA Direct Extractor Connection Provides Semantically Rich Data from SAP Business Suite to SAP HANA


ChallengesSome use cases require straightforward use of SAP ERP data Simple interface desiredSystem landscape impact of other data acquisition techniques

LT “real-time” Separate NetWeaver instance in system landscapeDataServices ETL tool, Separate BOE-instance in system landscape

DataServices Requires an SP from March 2011 on the ERP system to use SAP DataSource Extractors (1522554 /Note:1558737)

DXC BenefitsDXC provides a very simple, straightforward interface, re-uses existing extractors in SAP ERP

HTTP-Connection directly from SAP Business Suite to HANA

TCO advantages (Due to simplicity, minimal system landscape impact)DXC is available simply by applying a note (In most cases)Re-use widely available skill-sets

BW extraction and load is well-known in the industry

Rationale SAP HANA Direct Extractor Connection Simplicity / Low TCO


Rationale SAP HANA Direct Extractor Connection Activation Mechanism for Handling Delta Processing

ChallengesMany SAP Business Content DataSources offer delta processing, aka change data capture

Extraction only sends data created, changed, or deleted since the last extraction run Efficiency

Some DataSource extractor types deliver data with special properties It should not be simply loaded directly into a table in HANA This would cause incorrect results in reports

Delta processing types AIM, AIE, AIED, AIMD, ADD, ADDD and CUBE require data to be loaded into a DataStore Object (DSO) Activation processing is important for data correctnessStandard DSOs in BW include an activation mechanism, which handles the special properties of this data appropriately (e.g. after image only, overwrite, deletion flag, etc.)Without DXC, HANA standalone cannot properly handle data from the aforementioned types

DXC BenefitsDXC provides a special In-Memory DataStore Object (IMDSO) for use in HANA standalone

IMDSO Includes the same activation mechanism features as BW DSOsIMDSO properly handles special requirements of extractor types such as AIM and AIMD, which need DSO activation ensures correct data in reportsIMDSO ensures proper sequencing, overwrite, deletion of data, etc.


Agenda


SAP HANA Direct Extractor Connection Details


SAP HANA Direct Extractor Connection SAP HANA DXC Concept: Illustration

SAP ERP

Extractor


ERP data

SAP HANA

Status



InfoCubes

Data Store Objects

InfoObjects

Embedded BW


DataSource


PSA

Activation

Queue

Active

Version

In memory DSO







SAP HANA Direct Extractor Connection (DXC) utilizes the DataSource extractors that exist in SAP Business Suite systems

Originally developed for use with BWDXC is for use in HANA standalone (data mart) scenarios. BW on HANA is a separate topic

DXC utilizes the embedded BW to enable extraction, which exists inside all NetWeaver 7.0 or higher ABAP-based systems

With NetWeaver 7.0, BW is automatically included inside NetWeaver ABAPThis is true for ERP systems e.g. ECC 6.0 and higher

Also others, e.g. CRM 7.0 and higher, etc.


In typical business suite systems, the embedded BW is not utilized Customers typically have separate BW systems

DXC uses the embedded BW system to enable extraction and monitoring onlyData flow is redirected It gets sent to HANA Note: Modeling in the embedded BW is not part of the DXC solution

Note: An architectural variation available, which uses a “sidecar” BW instead of the embedded one. See appendix for details.



DXC is available for SAP Business Suite systems based on NetWeaver 7.0 or higher, such as ECC 6.0 or higher

Most customers already have the Support Package level needed on source Business Suite system, e.g. SP SAP NW 7.0 Support Package Stack 19 – first available Nov 2008DXC only requires applying a special SAP note

The extraction from the SAP Business Suite system -> controlled from the Data Warehousing workbench inside the embedded BW

When data is extracted from, the SAP Business Suite system, it is not loaded into the PSA of the embedded BW instead it is redirected and sent to HANA

It gets loaded into in-memory DSO’s activation queueThen activated into the active table of the in-memory

However, in the data load monitor of the embedded BW, the data load into the activation queue in the DSO in HANA appears like data is loading into the PSA in the embedded BW



Delta processing (aka “change data capture”)Works the same for DXC as it would if BW were the receiving system

If the DataSource is delta enabled, then delta-enabled data is available with SAP HANA Direct Extractor Connection

DXC internally in HANA uses the ICM (Internet Connectivity Manager) – receives XML packages over the http(s) connection

Mechanism written on the XS Engine (special component for HANA)Receives data packets from ICM, converts format Inserts the records into the activation queue of the in-memory DSOActivation processing records are go into the active table in proper sequence

Both the ICM and XS Engine components must be installed in SAP HANA to utilize DXC



Limitations for DXCBusiness Suite System based on NetWeaver 7.0 or higher (e.g. ECC) with at least the following SP level:

Release 700 SAPKW70021 (SP stack 19, from Nov 2008)Release 701 SAPKW70104 Release 702 SAPKW70201Release 730 SAPKW73001

DataSource must have a key field definedProcedure exists to define a key if one is not already defined

Certain DataSources may have specific limitations Inventory types, e.g. 2LIS_03_BF – data requires special features only available in BWCertain Utilities DataSources – can work with one and only one receiving systemSome DataSources are not delta enabled – not a specific issue for DXC or HANA, but something to take into account



SAP HANA Direct Extractor Connection SAP Business Suite – DataSource Extractors Example from Sales Order Item Content

HANA Data Models - virtual

SAP HANA

SAP ERP

Application Tables

Sales Order

Communication Structure

Setup Table

Update Methods

Update Mode

DataSource for transactional data2LIS_11_VAITM

In-Memory Data Store Object Tables:

Active table: /BIC/A 2LIS_11_VAITM00

Activation Queue table: /BIC/A 2LIS_11_VAITM40

Transfer Structure

Extract Strukture MC11VA0ITM

Application logic


Data flow redirected from embedded BW


Agenda


SAP Business Content DataSource Extractors


SAP Business Content DataSource ExtractorsOverview

SAP Business Suite

Proprietary Extraction TechnologyApplication based change data capture (aka delta capabilities)Extractors are application based and take data from the context of the application itselfExtraction is event/process driven and (in some cases) is accomplished with publishing new or changed data in ERP based Delta Queues for receiving systems (e.g. BW or HANA)Extract structures can easily be enhanced using append structures Transformations can be implemented at the time of extraction using Business Add Ins (BADIs)Extract Structures based on entities in the Business SuiteAsynchronous, mass data capable extraction


business documents master data

transactional data master data attributes master data text master data hierarchy

Transactionaldata Master data Texts Hierarchies

ERP & R/3 900 700 2300 100 4000

CRM 160 280 700 40 1180SRM 30 30 60 10 130Others 360 210 330 30 930GRC 20 30 120 10 180

1470 1250 3510 190 6420

SAP Business Content DataSource ExtractorsThousands of SAP Business Content DataSources Exist

Totals


Agenda


SAP HANA Direct Extractor Connection Setup & Configuration


All relevant information related to setup & configuration is provided by Note 1665602 - Setup & Config: SAP HANA Direct Extractor Connection (DXC)Note 1583403 - Direct extractor connection to SAP HANA

Setup & configuration Relevant Notes


Edit deamon.ini

Import XSA-

descriptor

Applicationserver

configuration

Test DXC application

setup

Create DXC user andschema

Setup & Configuration HANA System


ConfigureSAP_RSADDMIN_MAINTA

N

RSA5, selectdata sources

RSA1, activatesource-system

RSA1, replicsate

source-systemCreate info-packages

Optional: Create

process-chain

Setup & Configuration Setup Business Suite System


Agenda


Comparison with other SAP HANA Data Acquisition Techniques


Comparison with other SAP HANA Data Acquisition TechniquesContrast with Data Services

ETL type: features (data quality, metadata mgmt, transformations, etc.)

SAP DataSources: DataSources (Extractors), limited to the subset of DataSources released to Operational Data Provider

Support Package Requirement: source SAP Business Suite system that came out in March 2011

Delta handling (change except for SAP Business Content AIM, AIE, AIED, AIMD,

Software Requirement: DataServices required

Transformations: available in the DataServices ETL tool

ETL type: Sophisticated ETL tool with extensive valuable features (data quality, metadata mgmt, transformations, etc.)

SAP DataSources: available for SAP Business Content DataSources (Extractors), limited to the subset of DataSources released to Operational Data Provider

Support Package Requirement: SP must be applied to the source SAP Business Suite system that came out in March 2011 – see SAP note 1522554

Delta handling (change data capture): Yes, except for SAP Business Content DataSources with delta processing types AIM, AIE, AIED, AIMD, ADD, ADDD, CUBE

Software Requirement: BusinessObjects Enterprise and DataServices required

Transformations: extensive transformation capabilities available in the DataServices ETL tool

ETL type: Simple and straightforward ETL approach; no “premium” features

SAP DataSources: available for all SAP Business Content DataSources (Extractors) and Generic DataSources with a defined key; key can be defined if missing

Support Package Requirement: SP required in the source SAP Business Suite system that came out in March 2008; DXC is implemented by applying a special SAP note

Delta handling (change data capture): Yes, for all SAP Business Content DataSources and all delta processing types; uses an In-Memory DSO with activation processing

Software: Uses existing components in SAP HANA (XS Engine, ICM); configuration file imported into SAP HANA

Transformations: very limited - BADI (ABAP) in extraction exit available. When extensive transformations are required, it’s recommended to use DataServices

Direct Extractor Connection SAP BusinessObjects Data Services


Comparison with other SAP HANA Data Acquisition TechniquesContrast with SLT

Type: tables of SAP Business Suite systems

Realseconds

Delivered Foundational Data Models for SAP Generally no; some RDS content available

Semantically Rich Data: Semantics must be implemented on a project basis in SAP HANA

System Landscape: required

Project Available with some RDS packages

Transformations:

Type: Trigger-based table replication. Data comes from base tables of SAP Business Suite systems

Real-Time: Yes. Expected lag time typically less than 3 seconds

Delivered Foundational Data Models for SAP Entities: Generally no; some RDS content available

Semantically Rich Data: No. Semantics must be implemented on a project basis in SAP HANA

System Landscape: SAP NetWeaver 7.01 instance (SLT) required

Project Acceleration Building Data Marts in HANA: Available with some RDS packages

Transformations: Some available as of SP3

Type: Batch-driven ETL. Data comes from SAP delivered Business Content DataSource extractors

Real-Time: No. Once every 15 minutes approx. theoretical maximum (depends on DataSource)

Delivered Foundational Data Models for SAP Entities: Yes

Semantically Rich Data: Yes, via SAP delivered Business Content DataSource extractors

System Landscape: Nothing added, uses existing components in SAP HANA

Project Acceleration for Data Marts in HANA: Significant benefit

Transformations: Limited, BADI in extraction exit available

Direct Extractor Connection SAP Landscape Transformation


Comparison with other SAP HANA Data Acquisition Techniques Supported Capability Matrix – Part 1 – Data from Tables

Capability Log File Replication*Documentation

SLT Replication Documentation

Data Services 4.0Documentation Direct Extractor Connection*

Tabl

es

Minimum ERP Version ECC 6.0 ERP 4.6c ERP 4.6c ECC 6.0

Unicode/Non-Unicode Unicode Only Yes Yes Yes

MDMP No Yes Yes Yes – via how-to guide

Transparent Tables Yes Yes Yes Yes – via generic DataSource

Cluster & Pool Tables No Yes Yes Yes – via generic DataSource

Non-SAP ERP SourcesYes No Yes No

Compressed Values DB Table

DB2 (9.1, 9.5, 9.7) Yes Yes Yes – via generic DataSource

Row Compression DB TableDB2 9.7 only (no support for 9.1 or

9.5)Yes Yes Yes – via generic DataSource

SAP ERP DB SupportDB2 LUW/UDB Only* All DBs supported under

ERPAll DBs supported under ERP All DBs supported under ERP

SAP ERP OS SupportAIX, HP-UNIX, SUSE LINUX 10 & SUSE Linux 11, SPARC/Solaris, (see PAM for

version details)

All OS supported under ERP

All OS supported under ERP All OS supported under ERP

Transactional IntegrityYes No No Yes

Multi-Source SupportSingle Source System Multiple Source Systems Multiple Source Systems Multiple Source Systems

* See SAP Note 1513496 for official release limitations


Comparison with other SAP HANA Data Acquisition Techniques Supported Capability Matrix – Part 2 - Extractors

Capability Log File Replication*Documentation

SLT Replication Documentation

Data Services 4.0Documentation Direct Extractor Connection

Extr

acto

rs

Minimum ERP Version

N/A N/A

See SAP Note:1558737 ECC 6.0 or higher

Unicode/Non-Unicode Yes Yes

MDMPPartial (Not recommended) Project solution with how-to-guide

Transparent Tables N/A N/A

Cluster & Pool Tables N/A N/A

Non-SAP ERP Sources Available N/A

SAP ERP DB SupportAll DBs supported under ERP All DBs supported under ERP

SAP ERP OS SupportAll OS supported under ERP All OS supported under ERP

Multi-Source SupportMultiple Source Systems Multiple Source Systems – clients must

have unique logical system names

Semantically Rich Data Definitions

No No Yes Yes

* See SAP Note 1513496 for official release limitations


Agenda


Appendix: DXC Sidecar Variation


Appendix: DXC Sidecar VariationSAP HANA Direct Extractor Connection: Sidecar

RationaleCustomers with older Business Suite systems (lower than ones based on NetWeaver 7.0, e.g. ERP 4.7 or lower) or customers who do not want to use the embedded BW now have an alternative: The Sidecar variationCustomers with a BW system connected can use that BW as the “bridge” between ERP and HANA (via DXC) – the same mechanism as with the embedded BW, but it’s external from the source ERP system.

DXC utilizing a sidecar BW 7.x or ORANGE (BW on HANA)Data flow is redirected from within BW 7.x sidecar or Orange (BW on HANA) - Data is not loaded into BW – gets redirected to HANA

No concurrent consumption (Means – a DataSource can be used either in the connected BW or with DXC – cannot be used for both!)Apply SAP note 1583403 in BW Apply SAP note in ERP (minor enhancement, minimum risk)


Appendix: DXC Sidecar VariationSAP HANA DXC Concept: Illustration Embedded BW

SAP ERP

Extractor


ERP data

SAP HANA

Status



InfoCubes

Data Store Objects

InfoObjects

Embedded BW


DataSource


PSA

Activation

Queue

Active

Version

In memory DSO






Appendix: DXC Sidecar VariationSAP HANA DXC Concept: Illustration Standalone BW

SAP ERP

Extractor


ERP data

SAP HANA

Status

Data flow redirected to Hana: transferred over http connection

InfoCubes

Data Store Objects

InfoObjects

BW 7.x


DataSource


PSA

Activation

Queue

Active

Version

In memory DSO






Appendix: DXC Sidecar VariationSAP HANA DXC Concept: Illustration BW on HANA

SAP ERP

Extractor


ERP data

Non BW-schema

Status

Data flow redirected to non BW schema in Hana DB

InfoCubes

Data Store Objects

InfoObjects

BW schema


DataSource


PSA

Activation

Queue

Active

Version

In memory DSO





BW on Hana (ORANGE)


© 2012 SAP AG. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice.

Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.

Microsoft, Windows, Excel, Outlook, PowerPoint, Silverlight, and Visual Studio are registered trademarks of Microsoft Corporation.

IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, z10, z/VM, z/OS, OS/390, zEnterprise, PowerVM, Power Architecture, Power Systems, POWER7, POWER6+, POWER6, POWER, PowerHA, pureScale, PowerPC, BladeCenter, System Storage, Storwize, XIV, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, AIX, Intelligent Miner, WebSphere, Tivoli, Informix, and Smarter Planet are trademarks or registered trademarks of IBM Corporation.

Linux is the registered trademark of Linus Torvalds in the United States and other countries.

Adobe, the Adobe logo, Acrobat, PostScript, and Reader are trademarks or registered trademarks of Adobe Systems Incorporated in the United States and other countries.

Oracle and Java are registered trademarks of Oracle and its affiliates.

UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.

Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems Inc.

HTML, XML, XHTML, and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology.

Apple, App Store, iBooks, iPad, iPhone, iPhoto, iPod, iTunes, Multi-Touch, Objective-C, Retina, Safari, Siri, and Xcode are trademarks or registered trademarks of Apple Inc.

IOS is a registered trademark of Cisco Systems Inc.

RIM, BlackBerry, BBM, BlackBerry Curve, BlackBerry Bold, BlackBerry Pearl, BlackBerry Torch, BlackBerry Storm, BlackBerry Storm2, BlackBerry PlayBook, and BlackBerry App World are trademarks or registered trademarks of Research in Motion Limited.

Google App Engine, Google Apps, Google Checkout, Google Data API, Google Maps, Google Mobile Ads, Google Mobile Updater, Google Mobile, Google Store, Google Sync, Google Updater, Google Voice, Google Mail, Gmail, YouTube, Dalvik and Android are trademarks or registered trademarks of Google Inc.

INTERMEC is a registered trademark of Intermec Technologies Corporation.

Wi-Fi is a registered trademark of Wi-Fi Alliance.

Bluetooth is a registered trademark of Bluetooth SIG Inc.

Motorola is a registered trademark of Motorola Trademark Holdings LLC.

Computop is a registered trademark of Computop Wirtschaftsinformatik GmbH.

SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, StreamWork, SAP HANA, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries.

Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects Software Ltd. Business Objects is an SAP company.

Sybase and Adaptive Server, iAnywhere, Sybase 365, SQL Anywhere, and other Sybase products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Sybase Inc. Sybase is an SAP company.

Crossgate, m@gic EDDY, B2B 360°, and B2B 360° Services are registered trademarks of Crossgate AG in Germany and other countries. Crossgate is an SAP company.

All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG.

0400 - data provisioning - direct extractor connection

Documents

change data capture

semantically rich data

properly handle data

memory datastore object

multiple source systems

generic data transfer

sap business suite

extractor erp datastatus