bw edw workshop

BW Layered, Scalable Architecture (LSA) for EDW

BI Maturity & Data Logistic11

Model-driven SAP BW as DWH Best Practice

LSA Implementation

33

44

22

Agenda PDEBW1

Recap of LSA - LSA Related Topics55

BW Data Model and Data Integration66

Appendix - BI Organization & Governance77© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/

BI Maturity & Data Logistic

BI is the Top Priority of CIOs1.I1.I

11

Agenda PDEBW1

Data Logistic Types and BI Maturity - Overview1.II1.II

Simple Data Logistics – Data Mart Based1.III1.III

Advanced Data Logistics – Data Warehouse Based1.IV1.IV

BI Data Logistic Excellence – Hinderers & Enablers1.V1.V

© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/

The CIOs View and Priorities

InformationExcellence

ProcessExcellence


BI & Enterprise Application Excellence driveBI Data Logistic Excellence

Information Excellence(Prio 1: BI Applications)

Process Excellence(Prio 2/4: Enterprise Applications)

DataLogistic

BI

No BI-, CPM- andPlanning Excellence

without trustable,accurate, consistent Data

standardization &consolidation of BI data

logistic (EDW)

Standardization andconsolidation of EnterpriseApplications (processes)

standardization &consolidation of BI data

logistic (EDW)

BIData

Logistic

Simple Truth:Simple Truth:No BI without DataNo BI without Data


A Valid BI Data Logistic asPrerequisite for BI Value

BI BusinessValue Drivers

SAP ERP SAP CRM Other Legacy

Data-Infrastructure, Data-Logistic

BI Applications and FunctionalityStandard Reporting, Agile BI

Corporate Performance Management

BI Framework

ConsistencyFlexibility

EfficiencySuitability




11

Agenda PDEBW1






TDWI – Accelerating BI MaturityThe Chasm on The Way to The EDW Shore

Figure 1. The TDWI Maturity Model shows how many organizations evolve their BI solutions.The Gulf and Chasm represent places where many companies get stuck; the bell shaped curverepresents the number of companies in each stage; and the labels above the bell curverepresent the data structure dimension in the model.

Business ValueIntegration

Consolidation

3. Child 4. Teenager 5. Adult 6. Sage

GULF“ ManagementReporting ”

”

“ DataMarts ”

“ DataWarehouses ”

“DW ”

“ BI Services ”

1. Prenatal 2. Infant

GULFCHASM

“ManagementReporting”

“Spreadmarts”

“DataMarts”

“DataWarehouses”

“EnterpriseDW”

“BI Services”

Moving towards a centralized EDW based architecture means we haveto overcome profound challenges (chasm):Non-IT ones like politics, missing strategy and sponsorship andIT ones .....


Correlation of BI Value & Data Logistics ExcellenceThe Need of Centralization

OperationalReporting

Spreadmarts Data MartsData

WarehousesEnterprise

DWArchitecture

FinancialSystem

ExecutiveSystem

AnalyticalSystem

MonitoringSystem

StrategicSystem

BusinessService

Type ofSystem

“ Drive theBusiness ”

“ Drive theMarket ”

“ MonitorPerformance ”

“ EmpowerWorkers ”

“ InformExecutives ”

“ CostCenter ”Executive

Perception

Value

CostROI

Infant Child AdultTeenager Sage

Static Reports Spreadsheets OLAP/Ad hoc Reports

Dashboards/Scorecards

AnalyticalTools

PredictiveAnalytics

Customer BIEmbedded BI

Prenatal

ROI

AnalyticServicesOperational

ReportingSpreadmarts Data Marts

DataWarehouses

EnterpriseDW

Architecture

FinancialSystem

ExecutiveSystem

AnalyticalSystem

MonitoringSystem

StrategicSystem

BusinessService

Type ofSystem







Perception







Perception

Infant Child AdultTeenager Sage

Static Reports Spreadsheets OLAP/Ad hoc Reports

Dashboards/Scorecards

AnalyticalTools

Predictive

ROI

Beyond the Basics: Accelerating BI MaturityWayne W. EckersonDirector, TDWI ResearchThe Data Warehousing Institute 2007

Best practice: centralize before you federate‘...interestingly, organizations that try to federate development andadministration before they have centralized these tasks often havelimited success.’




11

Agenda PDEBW1






BI Data-Logistic Types‘Standalone’ Data Mart Architecture

Continental Europe:Sources Data

Marts

Business ValueSemantic IntegrationData Consolidation

1. Prenatal 2. Infant 3. Child 4. Teenager 5. Adult 6. Sage

GULF CHASM“ManagementReporting”

“Spreadmarts”

“DataMarts”


“EnterpriseDW”

“BI Services”


1. Prenatal 2. Infant 3. Child 4. Teenager 5. Adult 6. Sage


“Spreadmarts”

“DataMarts”


“EnterpriseDW”

“BI Services”“Management

Reporting”

“Spreadmarts”

“DataMarts”


“EnterpriseDW”

“BI Services”

A definition:

An implementation of an analyticsapplication serving a singledepartment, subject area, or limitedpart of the organization. Usuallyrefers to a physical platform onwhich summarized data is stored fordecision support.

Intra-Departmental:Complexity &Misalignment


‘Standalone’ Data Marts

Sales

Standalone Data Marts are independently built BI solutions oftenbased on different BI tools.

Different data models i.e. inconsistent semantics & valuesDifferent data management for Data Mart and different BI toolsInhomogeneous, multiple extractions & staging

High departmental acceptance but islands (silos, stovepipes)

HRMaterialManagement

Sources

StandaloneData Marts#

Employee

Material

#


BI Data-Logistic Types – Flier :Standalone Data Marts

Standalone Data Marts = Departmental BI1. Organization

Departmental, High daily involvement by local Business AnalystLimited IT support

2. Architecture, Reach of Data LogisticInhomogeneous, multiple extractions & stagingData Mart tools have often own data management, multi-dimensional, sql-generator, mdxDepartmental reach

3. Data ModelNo integration between Data Marts,Different data models i.e. inconsistent semantics & values

4. IssuesCross-departmental view: misalignment, islands, communication confusion, high costs,not pursuable, ...

5. AdvantagesDepartmental view: flexible, autonomy

6. Adequateness for Customer typesAdequate only from departmental perspective or for small customer (< 100) with ITknowledge




11

Agenda PDEBW1






BI Data-Logistic TypesData Warehouses

BI tools work on Data Marts or more general on special prepared sets of data

Because of the missing integration (islands) between Data Marts, DataWarehouses are established as a common integrated data foundation for DataMarts.

To show the new quality we call these kind of Data Marts ‘Architected’




”

“ DataMarts”


“DW”

“ BI Services ”



“Spreadmarts”

“DataMarts”


“EnterpriseDW”

“BI Services”


BI Data-Logistic TypesData Warehouse Architecture

Sources DataMarts

DataWarehouse

Org

Uni

t A

StagingArea

Bill Inmon - The Data Warehouse‘The Data Warehouse is

a subject-oriented, integrated, time-variant, non-volatilecollection of data used to support the strategic decision-making process forthe enterprise. It is the central point of data integration for businessintelligence and is the source of data for the data marts, delivering a commonview of enterprise data.’


A Data Warehouse as Integrated Foundationfor Data Marts – Architected Data Marts

MaterialManagement

Sales HR

Material

=Employee

=

Staging Area

Sources

employeematerial

customervendor

Data Warehouse

ArchitectedData Marts

Before Data Marts are loaded the data are integrated to common semanticsgiven by the dwh data model and the values are harmonized.The integrated results are stored persistently and build the data warehouse.Data Marts are filled from the Data Warehouse.

A Data Warehouse is a necessary condition for consistent BI.

Data Warehouse:Standardized extraction & loadIntegrate data with respect tointegrated semantics & valuesStore Integrated resultsLoad integrated data into DataMarts (Architected Data Marts)From this perspective thedata warehouse represents alayer in the informationsystem architecture


Bill Inmon’s Corporate Information Factory &The Data Warehouse

Copyright ©1999 by William H. Inmon

Conceptual DetailsSubject-orientedIntegratedhistorical completecomprehensiveapplication neutralgranularHigher organizational-unit ownednon-volatile…


Data Warehouse PrinciplesInformation Completeness, Reusability, Granularity

Customer Dimension Material DimensionA 4712

4713

Customer Material TimeAmount

CompanyCurrency

A 4712 200211 100A 4713 200211 150

Time Dimension200211

Customer Time DocNo Pos MaterialAmount

LocalCurrency

AmountCompanyCurrency

A 20021107-10am 1 10 4711 100 50A 20021107-3pm 1 10 4712 200 100A 20021107-4pm 2 10 4713 300 150

Customer Time DocNo Pos MaterialAmount

LocalCurrency

A 20021107-10am 1 10 4711 100 New bookingA 20021107-3pm 1 10 4712 200 Correction bookingA 20021107-4pm 2 10 4713 300 New booking

Sourcesystem

BW Data Warehouse Layer

daily

weekly dailymonthly

other InfoCubeother InfoCube

BW Architected Data Mart Layer

InfoCube

granularity : week granularity : daygranularity : month

Example:


DWH & Architected Data Mart Layer

Introducing a Data Warehouse Layer the architecture has two main Layer:

DWH Layerno business driven transformations just

cleansing, unification and integrationtransformations

Based on common definitions

Architected Data Mart Layerbusiness driven transformationsbusiness definedBWA area

BW

DW

HA

rchi

tect

edD

ata

Mar

ts


Bill Inmon ‘What is a Data Warehouse?’Data warehouses are significantly different from data marts.

From ‘Data Mart Does Not Equal Data Warehouse’, Bill Inmon, DM Direct, November 1999

Integrated, subject-orientedThe data warehouse data is integrated from the many legacy sources.Data warehouses are arranged around the corporate subject areas found in thecorporate data model.

Corporate owned or larger part of the organizationThe data warehouse represents a truly corporate/ higher organizational-unit effort.Usually the data warehouse is built and owned by centrally coordinated organizations

time-variant = complete history, non-volatile = permanent

GranularThe data warehouse contains the most granular data the corporation has. Data martdata is usually much less granular than data warehouse data. The volume of data foundin the data warehouse is significantly different from the data found in the data mart.

Application neutral, non-flavoredThe structure and the content of the data in the data warehouse do not reflect the bias ofany particular department, but represent the corporation's needs for data.

ComprehensiveThe data warehouse is broad in scope and keeps more data than actual requested bythe business/ data marts


BI Data-Logistic Types – Flier :Data Warehouse

Data Warehouse & Architected Data Marts: Integrated, Cross-organizational BI(on DWH definition level)

1. OrganizationIT managed sometimes BI Competency Center (BI CC)

2. Architecture, Reach of Data LogisticCommon extraction & staging techniques (ETL) for all cross departmental Data MartsData Integration results have own persistenceData Marts served by the data warehouse (architected, reliable data marts)Cross organizational on upper Org-unit

3. Data ModelCross Subject-area integrated

4. Issues, ChallengesBusiness-user view: Often IT-driven, missing business acceptance, needs business buy-in, time to market

5. AdvantagesData warehouse owner view: Integrated reporting, standards, consistency, less TCO thanpure Data Mart architectures

6. Adequateness for Customer typesFor independent operating higher organizational units


BI Data-Logistic TypesEnterprise Data Warehouse (EDW)




”

“ DataMarts ”


“DW ”

“ BI Services ”



“Spreadmarts”

“DataMarts”


“EnterpriseDW”

“BI Services”

Data Warehouses cover a certain part of an organization (country, businessunit, process e.g. FI), which means for larger companies there exist multipleData Warehouses.

Thus from a higher level point of view they suffer from similar issues likestandalone Data Marts. The Data Warehouses are often isolated.


The Multiple Data Warehouse Landscape Issue:Justification of an Enterprise Data Warehouse

Source

Source

Source

Source

Source

Source

Source Source

LocalSAP BW

LocalSAP BW

Source

Source

SourceLocal

SAP BWUnit 2

SAP BW

LocalSAP BW

LocalSAP BW

Stream BDWH

Stream CSAP BW

GroupSAP BW Stream A

SAP BW

Unit 1DWH


BI Data-Logistic TypesImpacts of Multiple Data Warehouses

Org

Uni

t A

Org

Uni

t B

Org

Uni

t C

Often we find multiple DWHsin an Organization

Complexity &Misalignment


Bill Inmon’s Corporate Information Factory &Enterprise Data Warehouse (EDW)

Copyright ©1999 by William H. Inmon

Enterprise Data Warehouse (EDW):A single instantiation of a datawarehouse layer for the entirecorporation or big parts of theorganization is often called theEnterprise Data Warehouse

EDW-Keywordsoffer a ‘single version of truth’extract once & deploy manysupport the ‘unknown’

re-buildnew-build

controlled redundancyprovide a corporate memory

Conceptual DetailsIntegratedhistorical completecomprehensiveapplication neutralgranularcorporate ownednon-volatile…


Treating Information as Corporate Asset -Flexibility is Key of Growth & Survival

The KnownDon’t limit yourself by just focusing on

current requirements

The UnknownBe prepared for

unpredictable future needs


The Whole Picture – The EDW

SAP BW @Henkel

Peter Hinzmann, CIO of Henkel, summarizes the motivation:

“We are increasingly focusing on the whole picture, which in turn raises theimportance of centralized management of the company. This process is activelysupported by IT.”

Werner Böttiger, IT project manager at Henkel’s business intelligence competencecenter:

“The enterprise data warehouse (EDW) layer does lead to a degree of redundantdata, but this can be kept under control.

All transactional data from the source systems need to pass the EDW layer before itis written to the final data destinations or data marts.

This enables us to safeguard data quality and create a single version of truth thatcontains all historical data,”


BI Data-Logistic Types – Flier :EDW

EDW & Architected Data Marts: Integrated, Corporate BI

1. OrganizationBI Competency Center – BI CC (IT, analysts, business)

2. Architecture, Reach of Data LogisticLike DWH but for entire Corporation (may be division)Single Point of Truth, historical complete, comprehensive, granular, integrated,Architected Data Marts built on EDW

3. Data ModelCross Company

4. Issues, ChallengesIT as driver, missing C-level sponsorshipMissing business acceptance, needs business buy-in, time to market

5. AdvantagesBI on entire value chain,Integration, standardization, new information culture, competitive advantagesEnabler for consistent ‘BI as service’

6. Adequateness for Customer typesFor all customers within highly competitive markets, High volume, Global organization


Ralph Kimball’s view on Data WarehouseLayer

Figure 1.1 The basic elements of the data warehouse.


Meta Group I


Bill Inmon's Corporate Information Factory andThe Enterprise Data Warehouse

DSS ApplicationsDepartmental Data Marts

EDW

MarketingAcctg Finance

Sales ERPERP

ERP

CRM

eComm.

Bus. Int.

ETL

GlobalODS

Oper.Mart

Explorationwarehouse/data mining

* Source: Bill Inmon

Stag

ing

Area

localODS

DialogueManager

CookieCognition

Preformatteddialogues

Cross mediaStorage Management

Near lineStorage

Web Logs

SessionAnalysis

Internet

ERPCorporate

Applications

ChangedData

GranularityManager

Archives




11

Agenda PDEBW1






Why are there Chasms & Gulfs ?




”

“ DataMarts”


“DW”

“ BI Services ”



“Spreadmarts”

“DataMarts”


“EnterpriseDW”

“BI Services”

zero high


Different Perspectives on BIBusiness & IT

Business perspective on Informationsolutionsolution, business driven BIbusiness driven BI

Flexible, time to marketAutonomy, agilityPerformanceConsistencyHigh availability

IT perspective on Informationdata, technology driven BIdata, technology driven BI

Manage BI (Development, Maintenance,Operation, Administration, Housekeeping)Lower TCO, TCD (Total Cost ofOwnership, Development)Reduce number of BI tools (BIConsolidation)



Data-Infrastructure, Data-Logistic

BI Applications and FunctionalityStandard Reporting, Agile BI


BI Framework




The preferred BI flavor/ tool differs fromemployee to employee, from departmentto department, from organizational-unit toorganizational-unit

Offering different, adequate BIfunctionalities to different business-usertypes is fine but often results in anuncontrolled bunch of BI-tools, whichmeans per se in high costs

But this is not the main issue:A dispersed BI landscape implies as

experiences show an island like,dispersed data infrastructure.

This results in even higher costs butmore important doubts the value of BI

Business Perspective on BIOrganizational Egoism

Group

Division A Division B Division C Division D

......BUUnit ....... .......

Sales FI HR

SpainUK

FranceNordic Different BI tools and

Data infrastructures


SAP AG 2007, SAP Skills 2007 Conference / B2 / 37

Local Business and Issues of Dispersed BI DataLogistics

Source

Source

Source

Source

Source

Source

Source Source

LocalSAP BW

LocalSAP BW

Source

Source

SourceLocal

SAP BWUnit 2

SAP BW

Organizational/ business and also IT egoisms cause dispersedBI data-logistics. They often hinder any evolutionary step

towards an architectured BI data-logistic.

LocalSAP BW

LocalSAP BW

Stream BDWH

Stream CSAP BW

GroupSAP BW Stream A

SAP BW

Unit 1DWH

Dispersed BI Data Logistic:Missing Service level definition

– Uncontrolled data flows– Uncontrolled extractions

Redundant developmentInconsistent data model, KPIsNo overall BI / DWH Strategy

This results in:Silos, IslandsUnreliable Information-basisUncontrolled redundancyHigh Costs

Dispersed BI Data Logistic:Missing Service level definition

– Uncontrolled data flows– Uncontrolled extractions

Redundant developmentInconsistent data model, KPIsNo overall BI / DWH Strategy

This results in:Silos, IslandsUnreliable Information-basisUncontrolled redundancyHigh Costs


Why are there Chasms & Gulfs ?Non-IT Reasons

Mature BI Data logistics are hindered by local interests:

If there is no organizational momentum toward a common goal, then the bestarchitecture, the best framework in the world is bound to fail.

W.H. Inmon“

But any advanced BI data logistic like an Enterprise Data Warehouse depends onestablishing a high degree of central governance. This again depends on a

Clear commitment of c- (orporate) management (sponsorship) – not CIO

Particular, Local interests overrule Group, Corporate interests

Delivering consistent information across the business depends on shareddefinitions and business rules for collecting, processing and delivering it.Any significant improvement in information – in terms of

quality, consistency, timeliness and accuracy– depends on gaining control of information processes and definitions.

a customer architecture council

“On the other hand side we find the awareness:


Is IT Ready for EDW?IT and Issues of BI Data Logistics

DW

H

DW

H

DW

H

data

mar

ts

Staging

gene

rate

dem

and

Source

Extraction

dem

and

supp

ly

Staging

Source

Extraction

volume

...

Cop

a

Staging

Source

Extraction

no of applications

We miss:Service level definitionsGuidelines & Best PracticesQuality-control in DevelopmentComprehensivenessCompleteness....

This results in:Increasing Complexity

– Increasing operational cost– Increasing maintenance costs

Low flexibility to answer ‘the unknown’– Increasing time-to-market

Consistency issues

We miss:Service level definitionsGuidelines & Best PracticesQuality-control in DevelopmentComprehensivenessCompleteness....

This results in:Increasing Complexity

– Increasing operational cost– Increasing maintenance costs

Low flexibility to answer ‘the unknown’– Increasing time-to-market

Consistency issues


Why are there Chasms & Gulfs ?Is IT Ready For EDW ?

BI is totally underestimated and misunderstood by IT! (and others)

BI is highly different than dealing with operative requirements.BI is daily business-related (at all org-levels) and thus highly volatile/ dynamic

BI data-logistics must be highly flexibleBI deals with historical data what creates high data volume growth

BI data-logistics have to manage often 10 times or more of dataBI never endsBI is implemented incrementally, BI-application after BI-application,

Data volume growth but just so important,Growth of meta-data

BI means steadily increasing maintenance-, administration- and operation-workloadBI data-logistics have to be highly transparent, standardized robust and automatedBI is steadily under time-to-market pressure

BI data-logistics cannot rely on today. They must ‘support the unknown’ the unforeseeable.

BI deals with data, operative-business with processes!As BI is underestimated, the EDW is underestimated!This is the best prerequisite to fail!


Crossing BI Data Logistic Gulfs & Chasms –Is IT Ready for a BI Excellence Framework? *



Infrastructure, Data-LogisticEnterprise Data Warehouse Layered, Scalable

Architecture

Applications and FunctionalityStandard Reporting, Agile BI


Organization Process

StrategyInformation as Corporate Asset

MethodologyBI Competency Centre

BI Framework*



Realization

Alignment

* BI Framework introduced by Gartner

BI Excellence is more than Infrastructure/ Data-Logistic Excellence:


Treat Information as Corporate Asset -Mature BI Data-Logistics (EDW)

Increasing awareness about importance of a proper SAP BW data-logistic for Data andMeta Data

Customer investments for an adequate SAP BW infrastructure to manage data and metadata consistently and in a comprehensive manner

Activities run under the title ‘Enterprise Data Warehouse’Most common to all activities is the concept of a layered architecture of information systemsintroduced by Bill Inmon The Corporate Information Factory (CIF)

SAP BWLayered, Scalable Architecture

(LSA)

Com

plex

ity &

Cos

t:A

dmin

istr

atio

n,M

aint

enan

ce

No of Application Scenarios (Data Marts)/ Volume

SAP BWwith no corporate standards


BW Layered, Scalable Architecture (LSA) for EDW



LSA Implementation

33

44

22

Agenda PDEBW1




BI Data-Logistic & Applications Modeling withSAP BW

Mod

elin

g A

ctio

nsO

perate/ Adm

inistrationBusiness requirements

DWH data model

Multi dimensional model

Physical Star Schema

Logical DWH & Staging

Physical DWH & Staging

Data Transfer,Transformation, Error-handling

Locate source &map data & model, extractor

Erwin, Visio, Eclipse (planned)

BCT BW Reference Data Model

BW InfoCube/ MultiCube

Star Schema/ BWA index

BW DSOs

Generated PSA, DSOs tables

InfoPackage, DTPs, DAPsRule-editor, Error-DTP

BCT DWH Data Model,Extractors: BCT, Generic, UDC

Process Design

Deploy BI applications

Monitor BI applications

Performance, Tuning

Run, Recover

Housekeeping

BW Process Chains

BW Transport

BW Monitoring & Statistics

BW Aggregates, BIA, NLS

BW Requests/ Process Chains

BW Housekeeping Proc Chains

BW models and implement all relevant BIdata-logistic areas. Data Warehousing withBW is end to end model-driven.This is a prerequisite for designing datawarehouses in a transparent, standardized,robust and best practice way.


Non-SAP-Sourcedata model

BW Model-Driven Data Warehousing based onBCT Data Warehouse Reference Data Model

Buildingblocks DWH Stores:

DTPs, DSOs

ArchitectedData Marts:InfoCubes/ BWA

SAP-Sourcedata model

Non-SAP Source BSAP Source A

BW Model-driven DWH:1. DWH data modeling

Reference data modelInfoObjects + relations

2. DWH structure modelingInfoProviders

3. DWH operations modelingExtract, load, transformTransfer, Error-handlingAdmin, Monitor

ETL/ StagingPSA, InfoPackagesmap models:

provided by BW

extractor

map models:user-defined

extractor

Subject-areas

BW is from all perspectives fully model-driven


SAP BW Model-Driven Data WarehousingCornerstones

Designing a Data-Logistic with SAP BW is from all perspectives fully model-driven:

The Reference Data Model (BCT)Provides mapping of SAP-source data model to BW reference data model (InfoObjects)For all Non-SAP sources

Customer does not start on a ‘blank’ sheet of paper (-> reference model)Mapping of source data model to reference data model by customerFlexible, expandable

The Meta-objects to design persistent & virtual elements of a data-logisticPersistent Data-containers (InfoProviders: InfoCube, InfoObject, DSO, PSA)Access Virtualization (InfoProviders: MultiProvider, Virtual InfoProvider, InfoSet,..)Staging Virtualization (DataSource, InfoSource)

The Meta-objects to design the operations between persistent elementsExtract, Transport, Load (InfoPackage)Transformation (Transformation Rules)Error-HandlingTransfers (DTPs)Complex data movements (Process chains)


Conformed dimensions enable virtual SAP BW MultiProvider scenarios

FACT Table

Material_Dimension_ID..........._Dimension_ID.............Dimension_ID.............Dimension_ID

Key figure 1Key figure 2InfoCube

B

Local Part ofMaterial Dimension

Material_Dimension_ID

0MATERIAL0PROD_HIERMaterial Dimension

Table

FACT Table

Material_Dimension_IDSalesOrg_Dimension_IDTime_Dimension_IDCustomer_Dimension_ID

Sales AmountQuantity InfoCube

A

Material_Dimension_ID

0MATERIALMaterial Dimension

Table

Gebiet 1 Geb iet 2 Gebiet 3

Bezirk 1

Gebiet 3a

Be zi rk 2

Regio n 1

Gebiet 4 Gebiet 5

Bezirk 3

Regi on 2

Geb iet 6

Bezirk 4

Gebie t 7 Geb iet 8

Bezirk 5

R egion 3

Vertriebsorg anisation

Material HierarchyTable

0MATERIALLanguage CodeMaterial Name

Material Text Table

Material Master Table

0MATERIALMaterial Type

Local Part ofMaterial Dimension

Conformed, SharedPart of

Material Dimension

Material Dimension:local & conformed part

BW Data Warehouse & Architected DMs:Conformed Dimensions - Preventing Silos


The SAP BW MultiProvider Concept

End-UserReporting & Analysis need

logical (dimensional) model

SAP BW InfoCube(s),DS-Objectphysical implementation

basic factsnot report specific

SAP BW MultiProvidervirtual Structures

basic factsnot report specific

optional

SAP BW BEx Queryvirtual Structures

complex KPIsnavigation behaviorreport (type) specific

SAP BW BEx Reportend-user requirement

specific

SAP BW

Concept of SAP BW MultiProvider

MultiProvider separate physical persistent SAP BW Structuresfrom Queries and Query dependent BEx-objects


BW Data Warehouse & Architected DMs:MultiProvider and Flexibility

+C1C1

Expand the scenario(C1) seamlessintroducing a newscenarioFlexibilitySave investments

C1C1

ManagingWorkload (logicalpartitions)PerformanceNew contentFlexibilityRebuildFlexibility

SAP BW Queries

virtualSAP BW MultiProvider

Separate optimalphysical DB Schemafrom logical SchemaPerformance

C1C1persistentSAP BW Structures


BW Data Warehouse & Architected DMs:Data Marts Built on BCT Reference Model

MaterialManagement

Sales HROLTP

Material

= =Employee

Staging Area:BW PSA

Load:BW InfoPackage

ArchitectedData Marts:

InfoCubes, MPs

Transfer:DTPs, Open Hub

BW

Achieving Data Mart harmonization applying the BW reference data modelNot necessarily explicit data warehouse persistency

TransformationRules

Non-SAP Extraction:BW Extractors,BW DBConnect,

BW Generic,FlatFile,

Univ.DataConnect

BW

Data-Logistic M

odeling

SAP

BW

Ref

eren

ce D

ata

Mod

el

0EMPLOYEE0MATERIAL

0CUSTOMER0VENDOR


BW Data Warehouse & Architected DMs:Limitations

The ‘just use’ SAP BW DWHfocused on InfoCubes,

InfoObject master datamore or less summarizedsupport the known

SAP BWs are built on today's end-user requirementsDesigned to answer the knownAnswering the unknown is limited by

Granularity of InfoCubes and DS-Objects (data content)– (historical completeness)

Business-scope driven extraction– (extracted fields – process comprehensiveness)

Business rules applied to original extracted dataAvailability of master data history

Limited Completeness & comprehensivenessthus overall flexibility is limited to react on time


BW Data Warehouse & Architected DMs:Summary

+ Usage of BCT reference data modelCommon model-based consistent Data Marts

+ Relying on the power of the BW model-driven approachEnables implementation of even most complex BI scenarios

Customers who have no explicit or corporate BI/ DWH strategy except ‘use SAPBW’ - Little awareness about the meaning of incremental BI

limited scalability, increasing TCO when BW grows

Solution focus on InfoCubes/ reporting - usage of intermediate storage (DSOs)only if required to cover concrete business scope

Limited flexibility

Little standards & guidelines, which exceeds the generic BW offering

We find the ‘just use SAP BW approach’ with a lot of customers. Theapproach works well until the BW-system reaches a certain size in terms ofdata-volume and number of BI applications. Then the TCO will increase andscalability, maintenance, performance and operational issues will likely occur.For corporate BI data warehouse architectures just relying on BWs genericdata warehouse approach is not sufficient!


Implementing an EDW-Based Architecture inSAP BW

I definitely need to gofor an EDW-based

layered architecturefor SAP BW!

But how to do it ?


BW Layered Scalable Architecture (LSA) for EDW



LSA Implementation

33

44

22

Agenda PDEBW1




The Layered, Scalable Architecture (LSA) for EDW

Reference LSA Overview3.I3.I

33

Agenda PDEBW1

LSA Building Blocks – Data Layer3.III3.III

LSA Building Blocks – Data Domains3.V3.V

LSA Building Blocks – Data Model3.VI3.VI

LSA Building Blocks – Layer & Master Data3.IV3.IV

Reference LSA & Customer LSA3.II3.II


Everything Seems to be Clear, Doesn‘t It?

• EDW &Layering• Chasms

?


The Broad View of BW on EDW

Traditional EDW SAP BW EDW

Companiessituation

operational world fragmentationno view across the business

Increasing operational worldharmonization

Yet limited view across the businessScope-Areas

Cross-/Corporate BI

In scopelittle information freshness (monthly)

In scopehigh information freshness (daily)high flexibility

Local- /depart. BI

Not in scope In scopestandard reporting with local adoptionsflexible roll out

Operational BI Not in scope Partly in scope

Evaluation Long time for ROIHigh Latency (e.g. monthly)High costsLow synergies for local/

departmental BIOverall acceptance problems

Incremental approach to EDWImmediate ROI (local BI)Std. DWH Latency (day) to Low (RDA)Standardization lowers costsHigh synergies for all BI flavorsIncreasing acceptance over time

Note: of course we find also the ‘traditional’ EDW approach using BW!© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/

Crossing the EDW Chasm:The BW Layered Scalable Architecture

SAP Customer Experiences & Feedback :

Nestlé

Kraft Foods

Arla Foods

Adidas

Edeka

Beiersdorf

HenkelJapan

TobaccoPhilips

Samsung

Novartis

Syngenta

BASF

LandHessen

ShellDownstream

Mc Kesson

Statoil

Best Practices &Best Practices &EDW PrinciplesEDW Principles

BW Layered Scalable Architecture (LSA) –The Reference Architecturefor BW on a Corporate Scale

LSA

Nike




33

Agenda PDEBW1







The BW Layered Scalable Architecture (LSA)describes the design of

service-level oriented, scalable, best practice BWarchitectures founded on accepted EDW principles*.

The BW LSA serves as a reference architectureto design transparent, complete, comprehensive

customer DWH architectures (Customer LSA).

The Customer LSA describes corporate standardsto build BI applications in a

performant, maintainable, flexible manner.

BW Layered Scalable Architecture (LSA)

* As introduced in Bill Inmon‘s Corporate Information Factory (CIF)© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/

BW LSA: The Reference Architecture

LSABuilding Blocks

Reference

Describescore structures &

definitions

LSAImplementation

Reference

Describesdesign standards tobuild BI applicationsfounded on building

blocks

LSAOperationsReference

DescribesSupport

Scenarios

Meta Data ManagementNaming ConventionsOrganization (InfoAreas)

Life CyclesInformationMeta ObjectLSA

AdministrationData BaseHousekeepingMonitoring

Transport

Security

Data Staging/ Management fortransactional & master data

Persistent ObjectsFlows - scheduled/ recoveryTransformationProgramming (Abap)Organization (Process Ch.)

BW LSA

Landmark Building BlocksLayerDomainsData Model &

Data IntegrationAssistant Building Blocks

Data QualityLandscapeETLStorageOwnership/ OrganizeDevelopment concept


LSA Building Blocks –The Architecture Cornerstones

The LSA Building Blocks

are the cornerstones of your futurearchitecture thus having a decisive influenceon the overall success of your future BW EDW

describe the general BW EDW layoutindependent from concrete BI applications andBI projects.

Landmark Building Blocks deal with architectureareas that need fundamental decisions beforedoing implementations – they play the same rolelike the supporting structures (pillars & ceilings) ofbuildings.

Assistant Building Blocks deal with architectureareas that are normally less prior with respect tothe point in time you should decide and roll outcorporate standards.

LSABuilding Blocks

Reference

Describescore structures &

definitions

Landmark Building BlocksLayerDomainsData Model &

Data IntegrationAssistant Building Blocks

Data QualityLandscapeETLStorageOwnership/ OrganizeDevelopment concept


LSA Landmark Building BlocksLayer & Domains

There are two areas to standardize the architecture of persistent data stores:

1. Structure the data stores in a flow from PSA to InfoCubes with respect totheir role and the offered services – define Data Layer

2. Structure (split/ collect) the data within the Layer to guarantee Layer andadvanced services – define Data Domains

LSA ArchitecturedNon-Architectured

Domain

Layer

data

flow


LSA Assistant Building Blocks I

ETL (Extraction, Transformation, Load)Do you expect extensively data from non-SAP sources?

If the answer is ‘yes’, it is meaningful investigating an ETL-tool like SAP Data IntegratorIf SAP systems are the initial and primary BW EDW sources, you just don’t care. May be later

Data QualityDo you have considerable data quality issues?

If the answer is ‘yes’, it makes sense thinking about a Data Quality tool.If integrated SAP systems are the initial and primary BW EDW sources, you don’t care.

Landscapeoften reduced to ‘Do I need a single or a multiple BW instance’ landscape. This topic hasbecome more and more an assistant one, because of the arrival of new technologies and thetransparent support by BW (BW Accelerator, Near Line Storage s. ‘Storage’).The landscape architecture comes into focus

if we have to support mission critical BI or to observe legal restrictions or with other customerspecific requirements

if it comes to agile BI and local autonomy (federated landscapes, SAP Newton appliancesand BW EDW)


LSA Assistant Building Blocks II

StorageTraditionally all data of a BW DWH are hosted by an RDBMS.This is for large scale BW EDWs not adequate (this applies also to smaller BWs)

Use RDBMS compression if available and usable (e.g. DB2 ‘deep compression’)Rarely used data should be hosted by Near Line Storage (NLS). NLS tools compress your

data up to 95% (e.g. SAND) without destroying your Service Level Agreements (SLAs).The BW Accelerator (BWA) must be part of the overall architecture

Ownership/ Organization

Designing, implementing and operating a BW EDW need strong governance.

A BI Competency Center (BICC) should be established if not existing yet.


The Layered Scalable Architecture Goals -Standardization, Transparency, Efficiency

architecturedarchitecturednonnon--architecturedarchitectured

Non-Architectured

D a

t a

f l o

w

D a

t a

f l o

w

LSAArchitectured

Large scale BWData Warehouses

(EDWs) shouldfollow architectureprinciples like we

can observeconstructing large

buildings:standardized,scalable, no

squiggles, efficientusage of materials,earth quake save.


DEPLOYING WEBI ON SAPWith large scale BWs there is only one way to be successful:

standardize, standardize, standardize ...in a way making your BW transparent, robust & in the broadest sense

scalable

The BW Layered Scalable Architecture is all about:What, Where, When & How to Standardize




33

Agenda PDEBW1







From LSA Reference Architecture toCustomer LSA

LSABuilding Blocks

Reference

LSAImplementation

Reference


SAP‘s LSAReference

Customer-LSABuildingBlocks

Customer-LSAImplementation

Standards

Customer-LSAOperationsStandards

Customer-LSAStandards


BI-Projects built on Customer LSA Standards

LSABuilding Blocks

Reference

LSAImplementation

Reference


SAP‘s LSAReference

Customer-LSABuildingBlocks

Customer-LSAImplementation

Standards

Customer-LSAOperationsStandards

Customer-LSAStandards

BI-Project-DesignBased on

Customer LSA

Plan-Build-RunApply: Individuell BI project design


Customer LSA ‘Handbook’Shell


Customer LSA ‘Handbook’Land Hessen


BI-Project-Design

Life Cycle of The Customer LSA

BW LSA: The Reference Architecture

Customer LSA : Standards - Handbook

BI-Project-DesignBI Project Design

Step 3:PerfectPerfect

Customer LSA

Step 4:UpdateUpdate

Customer LSA

Step 1:DesignDesign

Customer LSA

Step 2:ApplyApply

Customer LSA


Different Starting Points –Different Customer LSAs

BW on Corporate/ Divisional Level - BW EDW• Consolidation of multiple BWs into a single often global BW – (consolidation)• Reengineering of a BW getting on in years – (reengineering)• Replacement of multiple legacy DWHs by a single BW (replacement)• Allied set up of a BW with an (global )ERP roll-out (alliance)

BW1

BW2

BW3

BWn

Con

solid

atio

n

Pro

lifer

ated

BW

Ree

ngin

eerin

gDWH1

DWH2

DWH3

DWHn

Rep

lace

men

t

New

Cor

pora

teE

RP

(s)

Allia

nce


Reference LSA & Customer LSA

As background and targets of each customer differ, the preferred servicesdiffer and thus the Customer LSAs will differCustomers differ with respect to:

State of integration (master data, source systems (processes)Painful experience -> Control & Influence (outsourcing)SkillsOverall governance, sponsorship, commitment.In general expections with respect to:

Availability, error-tolerance, robustness, audits :Protection against human errors, Data flows and persistency's (DSO-

Objects) should be resistible against all kind of errorsReport availability at 8 am (local time), 24x7 operationsFast (SLA) recovery without touching source system

Scalability (application scalability)Fast new-build of BI-applications without accessing Sources

Scalability (performance)Flexibility, Protection of investment

manage seamless changes in OLTP world



Reference LSA & Customer LSA3.I3.I

33

Agenda PDEBW1





Reference LSA Overview3.II3.II


‘Layering’ means horizontal structuring/ modeling of BWThe data flows are organized in a unified, service-oriented way. Parameters are:

Coverage, comprehensiveness (Process, User demands)GranularityHistory (completeness)Reusability (BI application scalability)Recovery (robustness, availability)QualityIntegration statusAccess-rights (End-user)Life-cycle.....

LSA Landmark Building BlocksData Layer/ Layering of Data

Non-Architectured

D a

t a

f l o

w

D a

t a

f l o

w

Value of Data Layer:+ Highly Transparent & Flexible

+Development, Maintenance+Administration, Operations+Database-Integration

LSAArchitectured

Layer


Basic Design Aspects of BW LSAEDW & Architected Data Marts Layer

There are two main Layer:

EDW LayerSingle Point of Truthreusable

no business driven transformationsjust cleansing, unification andintegrationtransformations

based on common definitionsnaming

Corporate Memory orCorporate/ Group Data Repository

…

Architected Data Mart Layerbusiness driven transformationsbusiness definedBWA area

BW

EDW

Arc

hite

cted

Dat

a M

arts


LSA Reference Layer Quick Intro

LSA

Reporting Layer (Architected Data Marts)

Business Transformation Layer

Operational D

ata Store

Data Propagation Layer

Quality & Harmonisation Layer

CorporateMemory

Data Acquisition Layer

Virtualization Layer

EDW layerSingle Point of truthReusableCorporate ownedGranular

ArchitectedData Mart Layer

User

Sources


LSA Reference Layer Quick Intro

LSA



Operational D

ata Store



CorporateMemory



extractor inbox, 1:1 fromextraction, temporary

source system like service level,comprehensive, complete, master theunknown, long term

create harmonizedview, guaranteequality

EDW LayerSingle Point of truthReusableCorporate ownedGranular

BI Applications/Architected

Data Mart Layer

digestible, ready toconsume, integrated,unified data

apply business logic

reporting, analysis ready abstraction near real time,operational like


LSA Reference Layer BasicsFrom Acquisition to Reporting Layer

SourceSystems

EDW Layer

Data flow

Corporate Memory

BusinessTransformation

Layer

ReportingLayer

PropagationLayer

AcquisitionLayer

Data Mart LayerHarmonize/

QualityV-

Layer

DM1

DM3

DM2Layering applies to transactional & master data!To master data however in a simpler form


LSA Reference LayerAcquisition Layer

SourceSystems

EDW Layer

Data flow

Corporate Memory


Layer

ReportingLayer

PropagationLayer

AcquisitionLayer


QualityV-

Layer

DM1

DM3

DM2

Acquisition LayerPer DataSource per source system

DataSource should be comprehensive (offer allrelevant process information)

Fast inbound & outbound to targetsAccept data from extraction with as less aspossible overhead – no early checks, merges,transformations i.e. (1:1)

Stamps the extracted records uniquely forconsistent internal DWH processing and tracking(request handling)

Provides abstraction of DWH from sourcesProvides short term history of extracted data for

immediate / short term data inspection


LSA Reference LayerData Acquisition Layer Flier

Criteria Characteristics

potential sources Operational systems, other data warehouses, any source

potential targets Data Propagation, Harmonisation & Quality, Corporate Memory, ODS

reusability Yes

transformations None, 1:1; possible enrichment of reusable informationen

granularity single records, granularity of DataSource/ most granular

content DataSource comprehensive: more fields than actually requested by Data-Martse.g. all fields of a Business Content DataSource

history 2-4 weeks, short term storage

main services Decouple OLTP extraction cycle from internal BW data processingRobust & fast inbound and distribution

store & deploy Unmerged data (DataSource & source system specific) – copy of delta queueFast store: PSA Element, (Write Optimized DSO)Fast distribution, for large DWH: Scalability thru parallelism – single thread from

source system, scalable parallelism (logical/ semantical partitioning) toconsumers (targets)

update Request by request; no overwrite/ update of loaded records.

housekeeping Regular content deletion

Impl

emen

t.O

p.D

WH

Ser

vice

s


LSA Reference LayerCorporate Memory Layer I

SourceSystems

EDW Layer

Data flow

Corporate Memory


Layer

ReportingLayer

PropagationLayer

AcquisitionLayer


QualityV-

Layer

DM1

DM3

DM2

Corporate Memory (CM)‘Think about the Corporate Memory Layer as your DWH and BI lifeinsurance‘:Mastering tasks, which are unforeseeable (‘the unknown’) andmanageable only violating SLAs:

Avoiding re-init from source(s)(Disaster-) recoveryEnhancing and new build of Data MartsChange of source master data transformationsReorganizations

the CM layer offers a source system like Service Level:Granular dataComprehensive data (high coverage of underlying process)Complete history of loaded data


LSA Reference LayerCorporate Memory Layer II

SourceSystems

EDW Layer

Data flow

Corporate Memory


Layer

ReportingLayer

PropagationLayer

AcquisitionLayer


QualityV-

Layer

DM1

DM3

DM2

Corporate Memory (CM)‘Think about the Corporate Memory Layer as your DWH and BI lifeinsurance‘:

Mastering tasks, which are unforeseeable (‘the unknown’) andmanageable only violating SLAs

The CM layer offers a source system like Service Level1:1 from Acquisition as rule of thumbin addition harmonized data may be stored in a dedicated Corporate

Memory Object after complex harmonization/ transformationIf we have the same DataSource offered by multiple source-systems we

should go for a single CM DSO. (Data lifecycle management must exist!)NLS is the right place for CM data


LSA Reference LayerValue of a Corporate Memory

You can think about the Corporate Memory as the area, which protects your investments, whichprotects you against all kind of surprises. Surprises in an architecture are unwelcome but theywill happen.

you should never rely on what business ask you to deliverthe corporate memory stores more information (fields) about a process than you are actually asked for.In fact we recommend to store all available describing fields of a process. This is easy for SAP sourcesas we extract data using all fields of a delivered DataSource.– Thus the Corporate Memory is comprehensive.we do not aggregate information/ bookings. We store all extracted records.

– Thus the Corporate Memory is historical complete (e.g. for bookings we have the complete historyof a document)

you may believe that you can repeat the loads or do again an initial load if you experienceproblems or new requirements. This is often not realistic or just not possible.

Not realistic:– To repeat historic loads with delta loads– initial loads cause a downtime in the operative system what is often not acceptableNot possible– In the operative system the data will be archived

you should always be aware that when you transform/ map original data ( Quality &Harmonization Layer) the mapping rules may change.


LSA Reference LayerValue of a Corporate Memory - Summary

The CM is a layer of choiceto guarantee utmost possible flexibility to react on the unknown, the unforeseeableif downtime of operative systems for recovery purposes (new initialization) isunacceptable (24x7) or if archiving takes place in operative system. In mostrecovery situations the Corporate Memory will help.in general for robust and standardized recovery proceedingto keep more information than actually needed or which can actually be integratedwithout impacting daily operations


© SAP 2008 /

LSA Reference LayerCorporate Memory Flier


potential sources Data Acquisition Layer, (Harmonization Layer)

potential targets Data Propagation Layer (Business Transformation Layer, Reporting Layer)

reusability Yes

transformations Often 1:1, technical harmonization (compounding)

granularity All extracted records for delta loads (copy of delta queue)content comprehensive (all fields of a DataSource)

additional fields to simplify administration & access

history complete

main services source system like SLA‘Single Point of Truth’ of extracted data for delta/ changed data for fullultimate area for application recovery, new-build and DWH reorganizationmanage the unknown

store & deploy DataSource specificWrite-optimized (wo) DSO for delta loads, normal DSO + wo DSO for fullnot directly in flow to applications (branched out)

update Request by request; no overwrite/ update of loaded records

Data life cycle Should mainly reside on NLS (Near Line Storage)

DW

H S

ervi

ces

Impl

emen

tO

p.

© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 89

LSA Reference LayerHarmonization & Quality Layer

SourceSystems

EDW Layer

Data flow

Corporate Memory


Layer

ReportingLayer

PropagationLayer

AcquisitionLayer


QualityV-

Layer

DM1

DM3

DM2

Harmonization & Quality LayerThe services of this layer are data and data modelintegration and quality checked data e.g.

Technical harmonizationformat, lengthsimple content harmonization (e.g. date)integrating local master data to a single model (key

compounding, concatenation)Integration of local master data to group master data

(group data model)Checking Integrity of loaded dataUnification of data: adding e.g. load time, origin..Advanced content cleansing

e.g. detect duplicates, address cleansing etc. -> DataServices

...Harmonizing data and guaranteeing quality of data canmean high efforts or no efforts at all. Customers whoinvested in process integration and common coded/integrated master data will harvest now.


LSA Reference LayerHarmonisation & Quality Layer Flier


potential sources Acquisition Layer, CM

potential targets Data Propagation Layer, (occasionally Corporate Memory)

reusability Yes

transformations Yes, to achieve harmonized, quality-assured data

granularity single records, granular

content Those fields/ InfoObjects, which can be harmonized and/ or quality assured

history Often no persistency – Lookup-Mappings: long term

main services Provide consistent, harmonized data across multiple sourcesProvide quality assured data

store & deploy often only virtual as InfoSource or just as Transformation RuleIf explicit persitency needed ‚normal‘ DSO with overwriteLookup tables as DSOs/ InfoObjects/ Z-tables (history!)

update Request by request

housekeeping

Impl

emen

t.O

p.D

WH

Ser

vice

s


Detailing LSA Reference LayersPropagator Layer I

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

ReportingLayer

AcquisitionLayer

Data Mart LayersHarmonize/

QualityV-

Layer

DM1

DM3

DM2

Propagator LayerThe Propagator Layer is the interface to all business related Layers. It stands for ‘extract once deploy

many’ i.e. scalability of Data Mart developmentPropagator Layer Objects offer data that are easy to digest, ready to consume for all business

purposes (Data Marts internal & external)

PropagationLayer


PropagationLayer

Detailing LSA Reference LayersPropagator Layer II

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

ReportingLayer

AcquisitionLayer


QualityV-

Layer

DM1

DM3

DM2

Easy to digest means standardized, unified data :From Harmonization & Quality Layer transformations we get:

Compliance of data to corporate quality and consistency standardsIntegration of local master data to group master data for group reportingIntegration of local master data into a single BW data model (compounding)

Easy to digest means standardized, unified data :Propagator Layer adds:

Option to enhance or merge Data to simplify Data Mart building - but no business specifictransformations (no flavor for all EDW Layers)

Unified Data transfer behavior out of Propagator Objects (e.g. additive delta)Suitable portions of data (-> Domains) to fulfill SLAs related to local autonomy, report

availability, robustness, recovery, administration and operations


Detailing LSA Reference LayersPropagation Layer & Trimming Data

DataSource ASource 2

DataSource ASource 1

‘DataSource A’Propagator

3. Collect

DataSource BDataSource A

‘DataSource A & B’Propagator

2.Merge

DataSource A

‘DataSource A +’Propagator

1.ExtendAdddata

DataSource A

‘DataSource A’Propagator 1

4.Split

‘DataSource A’Propagator 2

Propagator DSOs: Unified BI application experience – additive delta

viaqueue

viageneric

viafull

moving

via full

completefull

incompletefull ?

5.Unify


Detailing LSA Reference LayerPropagation Layer & Digestible Data

The core service of a Propagation Layer is to offer digestible data to applications.

Digestible may mean

harmonized data in the broadest sense (for more details s. Chapter on Data Model)integrating data: common semantics, common valuessmoothing data: common semantics, technically unified values (e.g. compounding)

harmonize behavior of extractors (additive delta to Data Marts)

trimmed to fit DataSources and Data persistency‘s toReduce data complexity for applications

1. Extending DataSources by looking up information, which applications frequently askfor. Note: introduced parent-child relationship must be stable otherwise realignmentissues!

2. Merging different but highly related DataSources and store data in a single propagator,If application always or frequently request them together (e.g. HR InfoTypes, avoidingextractor enhancements)

Provide sound data portions for better support of application services (availability etc)3. Collecting data from the same (or similar) DataSource but from different source

systems to less or a single source system independent propagator (s) ( LSAdomains)

4. Splitting data from a DataSources into multiple persistency’s with identical structure( LSA domains)


Data flow

2LIS_11_VASTI

2LIS_11_VASCL

YGTCS_SUMMARY

2LIS_12_VCSCL

Acquisition

Corp Memory

EDWpropagationR/3

Corp Memory

Corp Memory

Corp Memory

EDW LayerCustomer Example


LSA Reference LayerData Propagation Layer Flier


potential sources Data Acquisition Layer, Harmonization Layer, Corporate Memory

potential targets Business Transformation Layer, Reporting Layer

reusability Yes

transformations Additional, stable fields to increase (re-)useability & accessibility.No application-specific rules!

granularity single records, granularity defined by DataSource business-keycontent DataSource specific

as comprehensive as possible, if propagator is expecting volatile requiermentsMerge of different DataSources to reduce complexity changes

history Minimum history defined by requirements of target-applications/ dependent fromCorporate Memory existence

main services ‘Single Point of Truth’ for BI applications (Business Transf. & Reporting Layer)Provide digestible (additive delta, content, performance) data for BI applicationsapplication recovery, rebuilt

store & deploy ‚normal‘ DSO in overwritesemantical/ logical partitioned for large scale DWH/ time-zone support

update driven by BI application requirements (report availability)

housekeeping Regular delete of DSO change-log content

DW

H S

ervi

ces

Impl

.O

p.


LSA Reference LayersData Mart Layers

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

ReportingLayer

PropagationLayer

AcquisitionLayer


QualityV-

Layer

DM1

DM3

DM2


Data flow

2LIS_11_VASTI

2LIS_11_VASCL

YGTCS_SUMMARY

2LIS_12_VCSCL

Acquisition

Corp Memory

InfoSource

InfoSource

ReportingEDW

propagationR/3Appl specific

Transformation

Corp Memory

Corp Memory

Corp Memory

Acquisition/Pass-Thru

Corporate Mem.

Propagation

Transformation

Reporting

Multi-provider

Corporate Mem.

Business Transformation LayerCustomer Example


LSA Reference LayerBusiness Transformation Layer Flier


potential sources Data Propagation Layer

potential targets Reporting Layer

reusability No

transformations Applications specific rules, aggregation, disaggregation, lookups

granularity Defined by application needscontent Defined by application needshistory Defined by application needs

main services area where complex merging of propagators objects will take placearea where complex calculations/ enrichment (lookups) will take placeguarantees reporting layer consistency (e.g. Synchronization of DataSources)

store & deploy ‚normal‘ DSO in overwrite, if any: often only virtual as InfoSource/Transf. Rulesemantical/ logical partitioned for large scale DWH/ time-zone support


housekeeping Regular delete of DSO change-log content

DW

H S

ervi

ces

Impl

.O

p.


Detailing LSA Reference LayersSub-Layers of Reporting/ Data Mart Layer

Access Abstraction-/Virtual Layer

Reporting/ Data Mart Layer

Analytics-/Dimensional Layer

Flexible Reporting/Granular Reporting

Layer

highly granularhighly comprehensiveshort life cycleflat, multidimensional

less granularless comprehensivelong life cyclemultidimensionaloptimized performance

abstract from physics‘virtual’flexible ‘view’ generationprotect front-end investments

Planning Layer

dedicated for planningdirect input structures


Detailing the Reporting LayerCustomer Example

FlexibleLayer

DimensionalLayer

VirtualLayer

ReportingGranular

ReportingPerformance

Long term

AbstractionFlexible

Data flow

Reporting Layer


© SAP 2008 /

LSA Reference LayerReporting Layer Flier


potential sources Data Propagation Layer, Business Transformation Layer

potential targets -

reusability No

transformations Applications specific rules, aggregation, disaggregation, lookups

granularity Defined by application needscontent Defined by application needshistory Defined by application needs

main services reporting & analysissave investments in front-end area: separation of physical Objects (InfoCube,

DSO) and virtual Objects( Virtual InfoProvider, InfoSet) by using logical views(MultiProvider)

different services of Reporting Layer often modeled by sub-LayerAccess Abstraction Layer (MultiProvider logical view)Analytic Layer (InfoCubes)Flexible Reporting (granular InfoCubes, DSOs)

store & deploy InfoCubes, DSO-Objects, InfoSets, Virtual InfoProvider, MultiProviderQueries work always on MutliProvidersemantical/ logical partitioned for large scale DWH/ time-zone support


housekeeping Archiving, NLS, Compression

Impl

.O

p.D

WH

Ser

vice

s

© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 103

LSA Reference Layer –Operational Data Store (ODS)

LSA



Operational D

ata Store



CorporateMemory



EDW layerSingle Point of truthReusableCorporate ownedGranular

ArchitectedData Mart Layer

User

Sources


3rd PartyLandscape

SAP

BW

PIPEPOS Inbound Processing Engine

Upload of POS Data to SAP BWvia POS Inbound Processing Engine

mySAPLandscape

Sales Analyses of POS Data

StoreContr.

PromoAnalysis

LossPrevent

BI-R

epor

ting

Tem

plat

es

Outbound Interfaces

Retail High Volume SAP BW ODS – POS DataManagement


3rd PartyLandscape

SAP BW

mySAPLandscape

Reporting-Templates

SAP BWMasterdata

ExchangeInfrastructure

R/3 POS Inbound

Card Payment

Billing

Forecast &Replenisment

InventoryManagement

SAP BW

Custom-BuiltInterface

IDocs

StandardsSales

TransactionsVMIXML

Transactiondatabase

Sales Audit• Monitor• Analyses• Editor

BAPI

PIPE Access Module

Overview POS Inbound Processing Engine(PIPE)


Example for Dedicated SAP BW OperationalData Stores

SAP BW

PIPE-ODS

APA

SAP BW

PIPE-ODS

Americas

SAP BW

PIPE-ODS

UK

SAP BW

PIPE-ODS

EMEA

CentralSAP BW

CentralSAP for Retail

Master DataPOS

TransactionData


Hybrid ProviderReal-Time Business Intelligence

HybridProviderCombines mass data with latest deltainformation at query runtimeConsists of a DataStore object and aInfoCube with automatically generateddata flow in betweenDSO object can be connected to a realtimedata acquisition DataSource/DTPIf the DataSource can provide appropriatedelta information in direct access mode aVirtualProvider can be used instead of theDSO.Facilitates replication of DSO-/VirtualProvider data to SAP NetWeaverBW Accelerator by switching off databasepersistency of the InfoCube

(RDA) DTP

ReplicateDSO

latest data mass data

InfoCube

HybridProvider

OLTP

Implement operational reporting scenarios leveraging newmodeling objects

BWA

optionally


LSA Reference LayerOperational Data Store Layer Flier


potential sources Data Acquisition Layer

potential targets application specific (Reporting Layer), external (e.g. Pipe)

reusability Limited

transformations simple

granularity extracted records granularitycontent application specific

history application specific

main services near real time reportingmass data management (e.g. POS)

store & deploy normal DSO (Hybrid Provider)mass data

Write-optimized (wo) DSONo PSA

Pipe (point of sale inbound processing engine)update RDA, request by request

DW

H S

ervi

ces

Impl

emen

tO

p.


LSA Reference LayerAcquisition Layer and Subsequent Layer

Why are there different paths thru LSA? The general answer is simple:In a BW we have to fulfill different competing SLAs (Service Level Agreements)Experience shows that a single staging approach with single persistent stores for allpurposes cannot achieve this (RDBMS limits) ! This applies the more challenging theconditions are for BW.Note: This means on the other hand side that if we found less complex conditions the LSABuilding Blocks set up can be simpler. This applies as well for an overall customer LSA set upas for specific data sources within a full blown customer LSA !

What are complex conditions andchallenging requirements?

24x7, time zoneshigh volume, not split able loadssmall recovery window (e.g. 6h)out sourcing (skills?)off shoring (skills?)high operational robustnesshigh report availability (e.g. >96%)high application flexibility....

LSA

Reporting Layer (Architected Data Marts) –Shared Master Data (InfoObjects)


Operational D

ata Store


Harmonization LayerCM



LSA Layers @CustomerSummary

The LSA describes services that are relevant at large scale BWimplementations. The services are grouped by Layers.

The LSA suggested Layers are a basic offering. Based on customersituation and preferences additional ones may be introduced evenlater on (Customer LSA).

As with all LSA areas customer adaptation of Layers follows the80:20 rule i.e. concentrate on services first, which offer the mostbenefit for the majority of data

Not all Layers have necessarily own persistent InfoProviders for alldata flows. Whether and when persistent InfoProviders exist are givenin the LSA Implementation standards

The Customer LSA Layer definitions apply throughout the BW EDW– No exceptions without approval!


BI-Projekt-Design

The Layered Scalable ArchitectureCustomer Adaptation & Utilization

SAP LSA: The Reference Architecture

Customer LSA : Standards - Handbook

BI-Projekt-DesignCustomer BI Projects (Plan-Build-Run)

Step 3:PerfectPerfect

Customer LSA

Step 4:UpdateUpdate

Customer LSA

Step 1:DesignDesign

Customer LSA

Step 2:ApplyApply

Customer LSA


LSA Building BlocksReference LSA & Customer LSA – Example I

LSA

Reporting Layer (ArchitectedData Marts)


Operational D

ata Store

Data PropagationLayer

Quality &Harmonisation Layer

Corporate

Memory


From Reference LSA to Customer LSA:Not all LayerAdditional LayerDifferent visualization and branding of Layer

Reference LSA Customer LSA

SAP

Sour

ces

Oth

erSo

urce

s

Operational Data Store

Dat

a A

cqui

sitio

n

Dat

a Pr

opag

atio

n

Qua

lity

Tran

sfor

mat

ions

Bus

ines

s Tr

ansf

orm

atio

n

Flex

ible

Rep

ortin

g

SubsidiaryReporting &

Analytics

Business-UnitSpanning

Reporting &Analytics

ScorecardsGroup

Near real timeReporting

Acc

ess

Abs

trac

tion

Laye

r

Proj

ect-d

efin

ed R

epor

ting

& A

naly

tics

End-

user

Rep

ortin

g &

Ana

lytic

s

CorporateMemory

EnterpriseData Warehouse BI Application Area

Data flow


LSA Building BlocksReference LSA & Customer LSA – Example I

LSA

Reporting Layer (ArchitectedData Marts)


Operational D

ata Store

Data PropagationLayer

Quality &Harmonisation Layer

Corporate

Memory


From Reference LSA to Customer LSA:Detailing


AcquisitionLayer

CorporateMemoryLayer

PropagationLayer


Layer

FlexibleLayer

DimensionalLayer

VirtualLayer

(YADSSnnn)

YCDSSnnn

YPDSSnnn YBAPPnnn

YFAPPnnn

YVAPP1nn

YVAPP2nn

YDAPPnnn

D a t a f l o wlookup

1:1Unlink

UnflavoredIntegratedGranular

Ready to consume

Applybusiness-logic

Reporting

Granular

ReportingPerformance

Long term

AbstractionFlexible

CompleteComprehensiveMost granular


LSA Building BlocksReference LSA & Customer LSA – Example II

LSA

Reporting Layer (Architected DataMarts)


Operational D

ata Store


Quality & HarmonisationLayer

Corporate

Memory


From Reference LSA to Customer LSA:DetailingNot all LayerAdditional LayerDifferent visualization and branding of LayerIndividual domains


Data flow

2LIS_11_VASTI

2LIS_11_VASCL

YGTCS_SUMMARY

2LIS_12_VCSCL

Acquisition

Corp Memory

ReportingEDWpropagationR/3

Appl specificTransformation

Corp Memory

Corp Memory

Corp MemoryCorporate Mem.




33

Agenda PDEBW1







Situation of Master Data

With master data we have similar challenges as with transaction data and additional ones

LSA topics1. High volume master data tables have a load impact

1. With full loads for InfoObjects (even if delta-load possible), which are used with extractorenhancements to catch the changes of an enhancement ( merging DataSources inPropagator Layer)

2. With change run (aggregate maintenance)2. Scenario reporting readiness depends on master data load time

1. Transaction data can only be processed into target subsequent to master data loads2. Different scenario or market reporting readiness requires different points in time of master

data readiness3. Recovery of InfoCubes or new ones often need master data history

BIA topics1. Change Run2. High volume master data tables (shared dimensions) have a negative impact on query

performance as with large fact tables3. Limitations of external hierarchies on high volume master data


LSA & Master DataSituation

Master data have two main tasks to fulfill:– Being target of lookups during transactional data load (referential integrity, adding

information– Being the shared dimensions of InfoCubes for reporting (MultiProvider

Today InfoObject hosted master data (P,Q,S,X,Y tables) serve for both purposes.

InfoObjectMaster data

Shared Dimension(Navigational Attributes)

Integrity,Lookups

Transactional loads


Issues with Data Mart Level Managed Master Data

Process C

hainD

ata MartA

Transactionalloads

With non-architected BWs the master data loadsare often managed on BI application/ Data Martlevel what leads to redundant, uncontrolledmaster data processing.This is unacceptable for large scale BWs

Data Mart level managed master data:

Master dataLoad e.g.

0CUST_SALES+ Change Run

3:00

Process ChainMaster Data



Transactionalloads

First Step: BW level managed master data:

Master data should be collected/ prioritized andloaded across the Data Marts thus become BWmanaged

Transactionalloads



Process C

hainD

ata MartB

1:00 1:15

Transactionalloads

Process ChainData Mart B

Process ChainData Mart A

2:301:00 1:15


LSA ImplementationLayering Master Data

LSA

Reporting Layer(Architected Data Marts)

Propagation/CM Layer



InfoObject tables

Master DataDSO

Shared Dimensionx,y,p,q,s tables

(Navigational Attributes)

Integrity,Lookups

Picture shows master data with delta load, master data withfull loads need additional ‘assistant DSO’ to determine delta

Large scale BWs should layerthe master data:

1.Separation of staging andreporting tasks

InfoObject tables forreportingPropagator Master DataDSOs for staging

2.Storing master data history(introducing ‘active from’‘active-to’ time-slice inPropagator Master DataDSO)


LSA ImplementationDecoupling EDW & Data Mart Layer Loads

Process ChainsMaster Data

InfoObjectChange RunPropagator/

Corporate MemProcess ChainData Mart A

LSA managed master data loads

EDW transactional & master data loads are decoupled from Data Mart &InfoObject loadsIt still remains challenging at what point in time to schedule master dataloads in a global system

Process ChainsEDW Master Data

Propagator for0CUST_SALES

Process ChainsEDW Transactional

Data

Data Mart LayerB Trans Layer

InfoObjectUpdate

EDW Layers Data Mart Layers



multiple loads

per day

loads by

business

requirements


LSA ImplementationReal Time Data Acquisition (RDA) for Master Data

Process ChainsMaster Data

InfoObjectChange Run

Propagator/Corporate Mem

Process ChainData Mart A

LSA managed master data loads (RDA)

With NW BW 7.30 there will be Real Time Data Acquisition (RDA) forMaster Data .This will increase the robustness of global BW EDWs considerably

RDAEDW Master Data

Propagator for0CUST_SALES

Process ChainsEDW Transactional Data


InfoObjectUpdate

EDW Layers Data Mart Layers



multiple loads

per day loads by

business

requirements

continuous loads


LSA and Master DataKeep master data history

Customer CustomerGroup

A Y after 2nd Load

Customer CustomerGroup

A X 1st Load: 20020101A Y 2nd Load: 20020401

InfoObject Master Data Table

Master Data from Source or Master Data Server

InfoObject master data keep only the actual version of master data:


LSA and Complete History of Master Data

Corporate Memory DS-Objects store all versions of master data

customer mother-compcustomer mother-comp

AAA YBBB YCCC X

InfoObject Customer Master

customer ActiveFrom mother-compcustomer ActiveFrom mother-comp

AAA 19980101 XAAA 19981101 YBBB 19980101 YCCC 19980101 X

I.Simple Customer Master CM

DS-Object

customer mother-comp

AAA XBBB YCCC X

customer mother-comp

AAA Y

load from 19980101

load from 19981101

customer ActiveTo ActiveFrom mother-compcustomer ActiveTo ActiveFrom mother-comp

AAA 19981031 19980101 XAAA 99991231 19981101 YBBB 99991231 19980101 YCCC 99991231 19980101 X

II.Time slice Customer Master CM

DS-Object


Alternatives Managing Master Data in LSA

1. Slim Staging: direct load into InfoObject (like in normal BW) for all Master Data notaddressed in 2 or 3.

2. Complete history of master data changes in a Propagator/ Corporate Memory* viatime-stamping records – (restricted usability)

Flexible stagingDelta loads direct into CM DSOFull loads via intermediate normal DSO (special type of Pass Thru) for delta determination

Same handling for transactional full loads

3. Complete history of master data changes in a Propagator/ Corporate Memory* viatime-sliced history (easy accessable in loockup)

Flexible stagingPropagator/ CM normal DSO with active-from, active-to (via function module set) slice, active-to is part of key

Delta loads direct into Propagator/ CM DSOFull loads via intermediate normal DSO (special type of Pass Thru) for delta determination

* whether you make it a Propagator or a CM member is a matter of taste© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/

SAP BW EDW Functions and ModelingDetermine Time slice of Master Data I

InfoObject

Characteristic Active-to Active-from Origin Attribute1 Attr24711 99991231 20030501 S1 Y A4711 20030430 20030401 S1 X A4712 99991231 20030401 S1 X B

Characteristic Origin Attribute1 Attr24711 S1 X->Y A4712 S1 X B

Characteristic Attribute1 Attr24711 Y A4712 X B

Transformation Rule:• Read DSO-Object with 4711 and 99991231• Update 99991231 record with new values and Active-fr• Use Return table to create new record

Characteristic Attribute1 Attr24711 X->Y A4712 X B

Corporate Memory‘normal DSO’

‘Pass Thru’‘normal DSO’

Load (here full)


LSA and Master Data

Completeness of Master Data History

At least for the ‘big entities’ all historical master data versionsshould be stored in the SAP BW Data Warehouse Layer usingflexible master data staging.For less important entities it is sufficient to keep the actualversions of master data in the InfoObject master data table thususing direct staging.




33

Agenda PDEBW1







Incremental EDWReasons & Challenges

Incremental set up is targeting to:

Align actual business requirements & BI/ EDW excellence

Reduce complexity

Step by step process & organizational excellence

...

Nevertheless an incremental approach is challenging as we have to guaranteeconsistency & scalability during the stony path of roll-out:

Consistency– Between the various Data Marts (FI, CO, LO, CRM...)

– consistent development of Data Marts– Sometimes between different SAP BW systems

– consistent deployment of Data MartsScalability


EDW Life Cycle DimensionsIncremental Implementation & Scalability

2. Incremental in terms of organizational coverage – organizational roll-out

Dem

and Planning

Generating D

emand

Procure 2 P

ay

CO

PA

..... .....

EDW

nBI Application Coverage & EDW

Org U

nit A

Org U

nit B

Org U

nit C

Org U

nit N

..... .....

EDW

nOrganizational Coverage & EDW

Needs Scalabilitythat is addressed by

(EDW)- layers

Needs Scalabilitythat is addressed by

domains (& datamodel)

1. Incremental in terms of functional coverage – BI application/ Data Mart roll-outAn EDW implementation is always an incremental one, never a big-bang


LSA Reference Layers Highly Simplified -Layering Does Not Resolve All Challenges

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

ReportingLayer

PropagationLayer

AcquisitionLayer


QualityV-

Layer

How to support:a world wide continuous roll-outBW-wide load scalability24x7 operations & administrationdifferent reliability of sourceslocal autonomyoverall local & group SLAs...

?© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/

LSA Landmark Building BlocksData Domains

Non-Architectured

D a

t a

f l o

w

D a

t a

f l o

w

LSAArchitectured

Layer

Domain

Domains means structuring/ modeling of data within the layers:Transparent, disjunct structuring of transactional data using stable criterion.Target is the support of:

Independency/ autonomy of organizations24x7, time-zonesScalability / performance/ low latency(parallel vers sequential)Challenging recovery-windowEmbedding into RDBMSImplementation & Operational robustness

Value of Data Domains:+ Transparency & Flexibility

+Development, Maintenance+Administration, Operations

+ Scalability & Robustness+Application+Load/ Query Performance+Database-Integration


LSA Domain Concept - Strategic BW PartitioningAccross the Layers – For All Flows I

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

ReportingLayer

PropagationLayer

AcquisitionLayer


QualityV-

Layer

DM1

Domains apply to transactional dataNormally not to master data!

2LIS_11_VAITM

Belongs to Domain A

Belongs to Domain B

Belongs to Domain C

Single ER

P


LSA Domain Concept - Strategic BW PartitioningAccross the Layers – For All Flows II

SourceSystems

EDW Layers

Data flow

Corporate Memory


Layer

ReportingLayer

PropagationLayer

AcquisitionLayer


QualityV-

Layer

DM1

Domains apply to transactional dataNormally not to master data!

Belongs to Domain A

Belongs to Domain B

Belongs to Domain C

2LIS_11_VAITM

Multiple E

RP


Domain US

LSA Building Blocks &Extract Once - Deploy Many

EDW

Propagators

Data Marts

DataSources

Ord

er

Del

iver

y

Sup

ply

Cha

in

Ord

er In

fo

1:n

Del

iver

y In

fo

Domain AMS

Ord

er

Del

iver

y

Sup

ply

Cha

in

Ord

er In

fo

Del

iver

y In

fo

Domain APJ

Ord

er

Del

iver

y

Sup

ply

Cha

in

Ord

er In

fo

Del

iver

y In

fo

Filling Domains with respect to domain criteria


LSA Data DomainsBW Landscape Consolidation (Consolidation BW)

EuropeJapan

Asia Pacific

ERPERPERPERPERPERP

ERPERP

ERPERP

BW BW

BW

BW

BW ERPERP

North America

South America

A BW EDW replaces a bunch of existing BWs and/ or legacy DWHs(BI Consolidation) spread across the organization

To enable comparable services like we had in a distributed, multiple DWH instance world(yes, there are some nice things) we introduce Data Domains in a BW EDW that divide

the transactional data but use identical meta data & master data.

BW

Using Domains in a BW EDW stands for manageability & flexibilityDomains allow SLAs in a BW EDW like in a distributed BW world

X

X

Domain AmericasDomain Americas

XX

Domain EuropeDomain Europe

X

X

Domain Asia PacificDomain Asia PacificBW EDWBW EDW


LSA Data DomainsBW in Parallel to ERP Rollout (Primary BW)

EuropeJapan

Asia Pacific

North America

South America

A single BW EDW shall offer standard reporting & analytics for all organizational unitsin a global ERP rollout.

To enable comparable services like we had in a distributed, multiple BW instanceworld we introduce Data Domains in a BW EDW that

divide the transactional data but use identical meta data & master data.


AMSAMS GermanyGermany APAAPAUSUS EMEAEMEA

Global ERPGlobal ERP

BW EDWBW EDW


LSA Data DomainsDivide Data by Sources-‘Quality’ (Integration BW)

BW EDWBW EDW

mainERP

remoteERP 1

remoteERP 2

Main DomainMain Domain Remote DomainRemote Domain

less stable, no controlstable, controlled



1st Summary: Domains Targets to Keep Large BWEDWs Manageable & Scalable

Dom

ain

Wes

tD

omai

n W

est

Dom

ain

Cen

tral

Dom

ain

Cen

tral

Dom

ain

East

Dom

ain

East

BW EDWBW EDWCompany

is operatingIn North America


North America

South America

Domain SouthDomain SouthAMSAMS

Company isexpanding to

South America

Dom

ain

Dom

ain

Wes

tW

est

Dom

ain

Dom

ain

Cen

tral

Cen

tral

Dom

ain

Dom

ain

East

East

BW EDWBW EDW

EuropeNorth America

South America

Domain SouthDomain SouthAMSAMS

Dom

ain

Dom

ain

Wes

tW

est

Dom

ain

Dom

ain

Cen

tral

Cen

tral

Dom

ain

Dom

ain

East

East

DomainDomainEUEU

Company isexpanding to

Europe

BW EDWBW EDW


LSA Data Domains BackgroundEvolutionary BW EDW

An BW EDW that has to provide ‘local’ BI services facesby far more databy far more and more complete BI applications – finally the entire value-chainmore source systemshigher query workload

than any other DWH in the organization before!

But with service expectations necessary to support daily business, like:High performance, robustness, high availability, low latency, organizational autonomy, fastrecovery ......

In addition BW EDWs on a global scale have to observe time-zones and24x7 operations

These challenges cannot be answered by just layering data: The LSA answeris the Data Domain concept:

LSA Data Domains divide (transactional) data in the BW Layer in a disjunctiveas stable as possible manner enabling scalable EDW services for most BI

needs across the business© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/

LSA Building BlocksReference LSA & Customer LSA – Example

Acquisition/PSA Layer

CorporateMemoryLayer

PropagationLayer


Layer

FlexibleLayer

DimensionalLayer

VirtualLayer

(YADSS100)

YCDSS100

YPDSS1DX

YPDSS1GX

YPDSS1WX

YPDSS1UX

YBAPP1AX

YBAPP1DX

YBAPP1GX

YBAPP1WX

YBAPP1UX

YFAPP1AX

YFAPP1DX

YFAPP1GX

YFAPP1WX

YFAPP1UX

YDAPP1AX

YDAPP1UX

YVAPP1XX

YVAPP1XXYDAPP1WX

YDAPP1DX

YDAPP1GX

Lookup-tables

Asia

Europe

Americas

Germany

US

YPDSS1AX

Controltables

LSA

Reporting Layer (Architected DataMarts)


Operational D

ata Store


Quality & HarmonisationLayer

Corporate

Memory


From Reference LSA to Customer LSA:Individual Domains



Transparent, Scalable Structuring of BW:LSA Domains & Layers

LSA





Corporate

Memory


Access Abstraction Layer(MultiProvider)

Single Source System (Layer)

LSA





Corporate

Memory


Access Abstraction Layer(MultiProvider)

Multiple Source Systems (Layer)

Distributionactively designed:

Domains

Distributioninherited

LSA Domains distribute the transactional data for the entire BW EDW in adisjunctive manner. The meta data of domains are identical.

The LSA addresses an evolutionary EDW approach introducing DataDomains enabling ‘local’ BI services, 24x7 operations andadministration without neglecting the broader EDW picture.


LSA Data Domains and LayerProperties of LSA Domains

As a rule of thumb:

Domains organize data by a general criteria driven by Reporting, BI andManageability requirements i.e. domains often differ from operational dataorganizationDomains are from transactional data point of view disjunct – harmonized masterdata are sharedDomains use identical meta data definitionsCross views are achieved via MultiProviders or dedicated persistent InfoProvidersDomains should be stableDomains should be consistent across all Layer (across all flows)Domains should be general for all Layer, exceptions:

The Acquisition Layer inherits the structuring from source systems/ extractorsDomains do not apply on the Corporate Memory


Criteria Defining LSA Domains

Golden rule defining domains:

as many domains as necessary – as less domains as possible

Defining Domains – rules of thumb

Often a geography driven domain concept works wellOften one basic domain per continent makes sense ( rough time zone handling)e.g. APA, EMEA, Americastime zone aspects may lead to e.g. 3 Asian domains East-Asia, Mid-Asia, West-AsiaE.g. for continental BW EDW a starting point could be East, Middle, West...

Expected Volume (realistic volume estimates are a key input defining domains )Large APA volume contribution & large (for your business important) countries may get anown domain e.g. China and US

Independency (special service level agreement) for certain countriesImportant countries/ markets get an own domain

Robustness: take Quality/ stability of different sources into consideration(potential) instable data offerings from sources may lead to an own domain

Each customer may have additional criteria, normally it is a mixture of multiple items


LSA Domain Criteria & Business View

Domains should be as stable as possible (TCO) on the one hand side but onthe other hand side from the overall service perspective it makes sensedefining domains with respect the business view on data, but this is notnecessarily stable as it refers to organization.

We may find a business view of data that is not reflected by organizationalunits of the sources (e.g. sources know Company-Code, planning area etc butbusiness view is ‘market’ what is not reflected by the sources)

Domains offer the opportunity resolving this using domains in an EDW forbetter BI services (reporting) and more transparency (domains)

Note: As domains are a mixture of application & technical dimensions weshould not overload domains with a purely business view -> observe Goldenrule


Domains and Source SystemsDefining Adequate Domains

...... ......

......

......Acquisition

Propagation

Acquisition

Propagation

DataSource from single corporate source-system

DataSource from multiple source-systems

?

??

Nodomains

Nodomains

adequatedomains

adequatedomains

Too manydomains


LSA Domains Definitions & EDW Roll-OutManaging Volume Uncertainty

Especially at the beginning of an EDW roll-out there is often a large degree ofuncertainty with respects to volume expected in the future.

But there is normally at least a classification of organizational units that drive thedomains possible in terms of t-shirt sizes (S,M,L)

S & M organizational units reside in standard domains (ASIA; AMERICAS; EMEA)

L organizations/ markets get an own domain

Possible challenges during roll-out:

Domains will become too big e.g. EMEA -> Create an additional Domain EMEA2

Large Domains -> think about additional splitting criteria (note: time-partitioning helpsnormally little with load issues)

This leads to initial domain concept for your EDWe.g. Asia, EMEA, AMERICAS, CHINA, US

Note: Existing Domains must continuously be observed with respect to SLA compliance


LSA DomainsDomain DWH Characteristic

BW EDWAll data

Domain ‘B’EMEA

Domain ‘A’APA

Domain ‘C’AMS

Country/ MarketE.g. ‘F’

E.g. ‘UK’

,,,,,,,,

Organizational-unit0COMPCODE = FR01

0COMPCODE = FR02

In BW EDW all loadedrecords will be qualified/‘stamped’ assigning a stable‘domain- driving’characteristic like market/country to organizationalcriteria like in this example0COMPCODE coming fromsource system.

This allows easy redistribution of adomain data if service levelscannot be kept

assign

assign


LSA DomainsDomain DWH Characteristic – New Domain

BW EDWAll data

Domain ‘B’EMEA – ‘F’

Domain ‘A’APA

Domain ‘C’(AMS)

E.g. ‘UK’

,,,,,,,,

Organizational-unit0COMPCODE = FR01

0COMPCODE = FR02

Domain ‘F’Country ‘F’

France ‘F’ gets an owndomainAll records in Domain ‘B’with country ‘F’ are movedto the new domain


LSA & Importance of Volume Estimates


bw edw workshop

Documents

log contentdwh servicesimpl

practicelsa implementation334422agenda pdebw1recap

zone supportupdate driven

ac qui si

layerac qui si

perspectives fully model

bi competency center

housekeeping regular delete