real time data acquisition rda - powerpoint presentation

8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation

1/34

Andreas Keppler RIG EMEA BI,

SAP AG

SAP NetWeaver BIReal Time Data Acquisition


2/34

SAP AG 2005, Title of Presentation, Speaker Name /Version 1 2

Enterprise Data Warehousing Variants & Processes

Modeling the EDW

Data ModelingMeta Data

&

Document

Manage-

ment

Data Acquisition

Transformation

Data Distribution

Running the EDW

Administration & Monitoring

Data Flow Control

Performance Optimization

Information Lifecycle Management

User

Manage-

ment


3/34

Overview

Real-time Data Acquisition via the Service API

Real-time Data Acquisition via Web Services

Real-time Data Acquisition Daemon Monitor


4/34 SAP AG 2005, Title of Presentation, Speaker Name /Version 1 4

Motivation for real-time

Upload frequency for regular staging not sufficient Number of requests can not be handled by BI system

Reporting requirements of data with sub 1 hours actuality

Operational reporting

The clear distinct ion between tools for analytical applications and tools foroperational applications is more and more difficult.

See transactional data in reporting as it is created in source system (post & see )

Availability of data in source system not known Use Push mechanism (e.g. SAP NW XI)

Polling or Trigger data load process via event from source system

Remote access of data not feasible due to resource consumption insource system (and cross application reporting requirements)

Splitting of staging processes (EDW) General business rulesare processed during the day whereas scenario specific rules (definedby application, region, time zone, etc.) are processed at customerdefined times



What is real-time?

Real-time data warehousing is a framework for deriving

information from data as the data becomes available:

Lower time scale than for scheduled/batch data acquisit ion

Stream oriented

Near immediate availabili ty for reporting

In general, Real-time data warehousing supports tactical decision-making

Simon Terr: Real-time Data Warehousing 101, DM Review Online October 2003



External

System

Web ServicePush

Real-time Data Acquisition (RDA)

SAP Source System

Service API

ApplicationDelta Queue

Real-time

Update

SAP NetWeaver Business Intelligence

DataSource

Data Warehouse Layer

Operational Data

Store

Architected Data Marts

Daemon

Pull

~ 1/min

InfoPackage for Real-

Time Data Acquisit ion

DataTransfer Process for

Real-time Data Acquisi tionScheduled Data

Transfer Process

Scheduled Data

Transfer Process

Real-time

Update

PSA

DataStore Objects



RDA Scenarios

Real-time data acquisition can be used in two primary scenarios:

via the Service API (SAPI)

Incorporates usage of InfoPackage for Real-time Data Acquisition

(source to PSA)

Then leverages Data Transfer Process for Real-time Data Acquisition

(PSA to DataStore Object)

via a Web Service

Incorporates usage of Web Services to populate the PSA

Then leverages the Real-time DTP to transfer data to the DataStore

Object

A system Daemon is used to initiate and control data transfer in

both scenarios.



What is a Daemon ?

Daemon

System process fulfils a specific task at regular intervals

The SAP NetWeaver BI Daemon performs three steps (depending on

the scenario chosen):

Initiate Service-API (SAPI) data pull via InfoPackage for Real-t ime Data Acquisition into PSA

Track status of data transfer from source system

Init iate update of DataStore Object via DTP

Successful execution of each step is tracked in a control table

Allows restarting if necessary

Restart can be initiated so it starts at the next step after the last

successfully executed step.



Daemon Monitoring

Maintenance screen for demon control can be called from the Data

Warehousing Workbench

Transaction RSRDA


10/34

Overview





11/34


Scenario I: RDA via SAP applications

SAP Source System

Service API

ApplicationDelta Queue

Real-time

Update

SAP NetWeaver BI

DataSource

Operational Data

Store

Daemon

Pull

~ 1/min

InfoPackage for Real-time

Data Acquisition

Data Transfer Process for

Real-time Data Acquisi tion

PSA

DataStore Objects

Service API-based real-time dataacquisit ion is a two-stage

process:

Data is pulled into PSA by Real-

time InfoPackage

Data is transferred to DataStore

Object via Real-time DTP

In both cases the process of datamovement is initiated by the

system daemon


12/34


Application Activities on the Source System Side

Source system

(ERP, CRM ...) Application

Service API

1 2 3

Time 03:10:09

Delta Queue

D T i RDA P i


13/34


Daemon Triggers RDA Processing

Source system


Service API

BI

1 2 3

Daemon

pull ~ 1/min

PSA

Time 03:11:00

Confirmation Table

DataStore Object

Delta Queue

Have a look Data

transfer

TID 1

TID 2

TID 3

Log entries

Update PSA

1

2

3

TID 1 ok

TID 2 ok

TID 3 ok

CommitCommit

1 2 3

ata tore

DTP

Initiate

Step 1 completed

Step 2 compl.

R ti St I


14/34


Runtime Steps I

Step 1

Call source system for new records

Status update of transferred records in confirmation table

Update PSA

Commit

Update confirmation table and update PSA in one step Guarantees synchronization

R ti St II


15/34


Runtime Steps II

Step 2

Check for records in confirmation table

If records are available in confirmation table the corresponding records

exist as well in PSA

Daemon flags records in source system

Reply confirmation sent to SAP NetWeaver BI

Entries are flagged as processed in confirmation table

Initiate DTP after the records are confirmed in confirmation table

Commit

If not successful in BI, records will stay in confirmation table

Guarantees restart of step 2 even when the update in the source systemwas successful

Daemon Triggers RDA Processing every 3 minutes


16/34


Daemon Triggers RDA Processing every 3 minutes

Source system


Service API

BI

1 2 3

Daemon

pull ~ 1/min

PSA

Time 03:14:00

Confirmation Table

DataStore Object

Delta Queue

TID 1

TID 2TID 3

TID 1 ok

TID 2 ok

TID 3 ok

1 2 3

ata tore

DTP

Initiate

4 5

Next

data load cycle

1

2

34

5

1

2

3

4 5

TID 1 ok

TID 2 ok

TID 3 okTID 4 ok

TID 5 ok

TID 1 ok

TID 2 ok

TID 3 ok

Closure of open request


17/34


Closure of open request

Not executed on every data load cycle

After completion of data upload into PSA:

Entries in confirmation table are deleted once the request is closed

(threshold for number of records per request / time exceeded or daemon

stopped).

Before the next data load cycle is started:

The next request is opened for the next data load from source system

(determined by update period of the daemon)

Requests of InfoPackage and DTPs are synchronized via daemon

(i.e. request in PSA and DataStore Object are aligned).

Daemon closes request


18/34


Daemon closes request

BI

Daemon

pull ~ 1/min

PSA

Time 03:17:00

DataStore Objectata tore

DTP

Initiate

Step 0 completed

cont. with load

1

2

3

4

5

1

2

3Confirmation Table

TID 1 ok

TID 2 ok

TID 3 okTID 4 ok

TID 5 ok

Request closed due to

time or n of records

Request 1

Source system


Service API

1 2 3

Delta Queue

1 2 3 4 54 5

InfoPackage for Real-time Data Acquisition Creation


19/34


InfoPackage for Real-time Data Acquisition Creation

Real-time flag must be selected for the InfoPackage which the

daemon will monitor.

This option is only available for one InfoPackage attached to theDataSource

Note: If you want to use a DataSource with real-time capabilities,you have to install the PI_BASIS 2005.1 in the source system

InfoPackage for Real-time Data Acquisition Definition


20/34


InfoPackage for Real-time Data Acquisition Definition

Real-time extraction must be selected in the Adapter section of

the InfoPackage settings

InfoPackage for Real-time Data Acquisition DaemonS tti


21/34


Settings

Automatic Request Closure by # records

Periodicity is determined by Update Periods setting via Real Time Monitor (RSRDA).

Data Package Size: No. of Rows Allows control of number of records per data package (multiple data packages per request possible)

The deamon uploads at least one data package (even if number of actual records is smaller than

setting)

Request Size: Number of Rows Allows control of packets for downstream processing based on time

Automatic c losure of Request by number of records (1,000,000 shown here)

Max number of failed attempts

Max number of times Daemon attempts to request/commit data before throwing error condition

InfoPackage for Real-time Data Acquisition Update:I h tl D lt


22/34


Inherently Delta

Only Delta Processing is

available for Real-timeupdates (because the deltaqueue is leveraged for datastaging)

On the Schedule tab you can jumpvia Assign to the RDA Monitor.There, you can assign theInfoPackage to a deamon.


23/34

Overview




Scenario II: RDA via Web Services


24/34


External

System

Web Service

Push

Sce a o a eb Se ces

SAP NetWeaver Business Intelligence

DataSource

Operational Data

Store

Daemon

Pull

~ 1/min

Real-time

Update

PSA

DataStore Objects

InfoPackage for Real-time

Data Acquisition

Data Transfer Process for

Real-time Data Acquisi tion

Web Service-based Real-time data acquisition is also a two-stageprocess:

Data is pushed into the PSA via a Web Service

Data is transferred to DataStore Object via Real-time DTP

The daemon monitors the PSA and initiates the DTP on a regularbasis (via settings in Real-time InfoPackage)

Web Service Data Source


25/34


InfoPackage for Real-time Data Acquisition via Web Service


26/34


g q

An InfoPackage is needed for the Web Service to defineparameters such as t ime / record thresholds for request closure

and to allow the assignment of a daemon via the Real-time

Monitor.

Real-time requests in DataStore Object


27/34


q j

DataSource / PSA

ActivationQueue

ActiveData

ChangeLog

Per.1 Per. 2 Per.3

Req.ID, Pack ID, Rec. No.

Req.SID, Pack ID, Rec. No.

Document No.

Latency between source system transaction and availabil ity in DataStore Object isminimal due to Lean Staging .

Lean Staging means that the DataStore Object handles the data differentlyinternally less logging and activation activity is allowed when the DTP for Real-time Data Acquisit ion is connected to the DataStore Object

From an administrative point of view, the DataStore Object is a standard one.

The DataStore Object also handles the availability of this real-time data (e.g. datawhose request has not been closed) for reporting.

How to embed RDA in your data flow


28/34


DataSource

PSA

RegularDataStore Object

OLTP

BI

Delta Queue

2LIS_02_VAITM Appl ication

real-time

Update

Using one DataSource

DataSource

PSA

RDADataStore Object

RegularDataStore Object

OLTP

BI

Daemon

Pull~5/min

Delta Queue

2LIS_02_VAHDR2LIS_02_VAITM

Periodic

Scheduling

2LIS_02_VAITM

Appl ication

real-time

Update

Using two DataSources

Daemon

Pull~5/min

Well established Data Flow can

can be used

Separate (potentially) leaner

DataStore Object can be used

Reporting has to be enabled via

MultiProvider

Data has to be deleted frequently

from RDA DataStore Object to

avoid redundant data

Old data flow has to be replaced

by new real-time data flow

No issues with data alignment

from different DataSources

Can facilitate upload of huge

data volumes by spreading

data load over many smaller

data loads

Periodic

Scheduling

2LIS_02_VAITM

Extraction done via RDA and InfoPackage scheduling

( Normal Delta ) can currently not be done in parallelDataSource can only use a single extraction mechanism


29/34

Overview




Daemon Monitor


30/34


The RDA Daemon monitor provides an overview on the status of each daemonand attached

InfoPackage(s) for RDA Data Transfer Process(es) for RDA

Important to note that both loading to the PSA and the DataStore Object aremonitored within the Daemon Monitor

Transaction RSRDA

Real-time

DTP

Real-time

InfoPackage

DataSource

Daemon Status


31/34


The Daemons can have the following statuses (identified by the icon in front):

- Daemon active and running

- Daemon not active

- Daemon has an error

- Daemon is stopped (changes its status to not active or error)

Standard BI Monitor for InfoPackages


32/34


Granular details

of datamovement look

similar to Data

Load Monitor in

prior releases

DTP for RDA Monitor


34/34


No part of this publication may be reproduced or transmitted in any form or for any purpose without the express

permission of SAP AG. The information contained herein may be changed without prior notice.

Some software products marketed by SAP AG and its distributors contain proprietary software components of othersoftware vendors.

Microsoft, Windows, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation.

IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iSeries,

pSeries, xSeries, zSeries, z/OS, AFP, Intelligent Miner, WebSphere, Netfinity, Tivoli, and Informix are trademarks or

registered trademarks of IBM Corporation in the United States and/or other countries.

Oracle is a registered trademark of Oracle Corporation.UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.

Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered

trademarks of Citrix Systems, Inc.

HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C, World Wide Web Consortium,

Massachusetts Institute of Technology.

Java is a registered trademark of Sun Microsystems, Inc.

JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and

implemented by Netscape.

MaxDB is a trademark of MySQL AB, Sweden.

SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver and other SAP products and services mentioned herein

as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several othercountries all over the world. All other product and service names mentioned are the trademarks of their respective

companies. Data contained in this document serves informational purposes only. National product specifications may vary.

These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated

companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group

shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and

services are those that are set forth in the express warranty statements accompanying such products and services, if any.Nothing herein should be construed as constituting an additional warranty.

real time data acquisition rda - powerpoint presentation

Documents