© 2012 wellesley information services. all rights reserved. a comprehensive guide to building or...

© 2012 Wellesley Information Services. All rights reserved.

A Comprehensive Guide to Building or Improving Your Enterprise Data Warehousewith SAP NetWeaver BW 7.3

Dr. Bjarne BergCOMERIT

2

In This Session …

• Take a look at the latest enterprise data warehousing capabilities in SAP NetWeaver® BW 7.3

• Examine new data modeling capabilities

• Understand SAP’s vision for SAP NetWeaver BW, and its evolution into an in-memory EDW solution. Gain insight into the deployment of SAP HANA data marts.

• Get insight into the new ETL interface, new management monitoring tools, the hybrid provider options for real-time data warehousing in SAP NetWeaver BW 7.3, and the capabilities of the Layered Scalable Architecture (LSA)

• Understand how in-memory technology impacts data warehousing capabilities, existing SAP NetWeaver BW customers, and the SAP data warehousing roadmap

3

What We’ll Cover …

• New Administrative Cockpit Capabilities• The New Semantic Partition (SPO) Options • The HybridProvider and Real-Time Data• Faster Data Load and New Design Options• Front-End and Security Benefits• Using SAP NetWeaver BW 7.3 for Performance - Real Examples• New In-Memory Possibilities with SAP NetWeaver BW 7.3 and SAP

NetWeaver BW Accelerator 7.2• Wrap-up

4

New 7.3 Performance and Admin Monitoring Capabilities

• New monitors include: Monitor of database usage and object sizes (i.e., InfoCubes, DSOs) Query usage statistics are more visible (similar to RSRT, RSRV, RSTT) We can see more of the use of SAP NetWeaver BW Accelerator and

sizes Monitor for the actual use of OLAP/MDX Cache and hit ratios You can now selectively delete internal statistics in RSDDSTATWHM by

date through the updated RSDDSTAT_DATA_DELETE ABAP program There is also a new MDX Editor for coding and syntax assistance

Solution Manager has been updated to take advantage of these

new monitors.

5

New 7.3 Performance and Admin Monitoring Capabilities (cont.)• Other new monitors include:

DEAMON update information (i.e., RDA capacity status, usage) A performance monitoring workbench for performance trends Process chain monitoring (new transaction: RSPCM) with error

and active chain monitoring, user specific displays, and performance threshold monitoring (i.e., for SLAs)

5

NLS: In SAP NetWeaver BW 7.3, the Near Line Storage has been enhanced to include archiving, support for write optimized DSOs, and database support for tools such as DB2 Viper (v9.5) and more

6

New: Specific Database Support — IBM DB2

• IBM DB2: 7.3 supports specific database features: DB2 v9.5 supports PSA, DSO, and fact table compressions for

reduced disk volume (integrates with DB2 storage management)

Support for MDC clustering in the DB Cockpit. This is available in v9.5.2. or higher and is the default for all DSO tables and PSA in version 9.7.

Much faster request deletion if MDC clustering is used DB2 v9.7 supports Index compressions for reduced disk

volume Support for IBM DB2’s DPF (Database Partitioning Feature)

6

In the past, SAP NetWeaver BW has been unable to take advantage of vendor specific database features. This is changed, and fewer workarounds are needed.

7




8

Semantic Partitioned Objects (SPO)

• When data stores and InfoCubes are allowed to grow over time, the data load and query performance suffers

• Normally objects should be physically partitioned when the numbers of records exceed 100 million However, this may be different depending on the size of your hardware and

the type of database you use• In SAP NetWeaver BW 7.3 we get an option to create a Semantic Partitioned

Object (SPO) through wizards You can partition based on fields such as calendar year, region, country, etc.

9

Semantic Partitioned Objects (SPO) (cont.)

• When an SPO is created, a reference structure keeps track of the partitions. The structure is placed in the MultiProvider for querying.

SPO Wizards create all Data Transfer Processes (DTP), transformations, filters for each data store, and a process chain automatically.

Source: SAP AG, 2011

10




11

HybridProvider

Query (real time)

RDAReal-Time

Data Acquisition

ECC and

ExternalSystems

BWA based InfoCube

Virtual InfoCube

Transaction Data

Direct reads

Real-Time Data Acquisition

Indexing

Direct Access

The New HybridProvider and Real-Time Data

• The “HybridProvider” (HP) is new in SAP NetWeaver BW 7.3. The core idea is to link the historical data inside BW with real-time data.

• There are two ways of implementing an HP: HP based on a DSO HP based on a Virtual InfoCube

12

HybridProvider

Query (real time)

RDAReal-Time

Data Acquisition

ECC and

ExternalSystems

BWA based InfoCube

Virtual InfoCube

Transaction Data

Direct reads


Indexing

Direct Access

This solution provides for really fast queries, but delta logic has to be custom designed

Option 1: The DSO-Based HybridProvider

• Core features: Real-time data is in the DSO and historical data in the SAP NetWeaver BW

Accelerator-based InfoCube The DSO use real-time data acquisition (RDA) to load data SAP NetWeaver BW automatically creates a process chain for the

HybridProvder’s data flow The process chain is executed for every closed request

13

This is a good option if you have a low volume of new records and a high number of queries or operational dashboards

Option 1: The DSO-Based HybridProvider (cont.)

• This solution provides for really fast queries, but delta logic has to be custom designed and may be complex. However, the solution allows for high-frequency updates and very rapid query response.

14

HybridProvider

Query (real time)

RDAReal-Time

Data Acquisition

ECC and

ExternalSystems

BWA based InfoCube

Virtual InfoCube

Transaction Data

Direct reads


Indexing

Direct Access

Virtual cubes with many users may place high-stress on the ERP system

Option 2: The Virtual Cube-Based HybridProvider

• Core features: Data is read in real-time from SAP ECC, while historical data is read from

SAP NetWeaver BW Accelerator The difference depends on how often SAP NetWeaver BW Accelerator is

loaded Non-complex data logic can be applied DTP is permitted if you do not filter the data set

15




Hint for Experts: The search feature in the SAP NetWeaver BW 7.3 Admin Workbench has been improved so that it is easier to search more items such as process chains.

16

Faster Data Load and New Design Options — Activation

• During activation, SAP NetWeaver BW 7.0 has to lookup in the NIRV table to see if the object already exists This can be a slow process

• In SAP NetWeaver BW 7.0 we may buffer the number ranges to compare the data load with records in-memory This speeds up data activation

• However, in SAP NetWeaver BW 7.3, the data activation is changed from single lookups to package fetch of the active table, resulting in faster activation and less locks on the lookup tables The new method may result in 15-30% faster data activation (20-

40% in lab tests)

17

Faster Data Load and New Design Options — Activation (cont.)• For data transformations, the option “Read from DataStore” for a

faster data lookup is also available• Additionally, the use of navigational attributes as sources in

Masterdata transformations reduce overhead for lookups Combined, this may lead to an additional 10-20% improvement

The 7.3 initial load runtime option “Insert only” and the “Unique data records only” prevents all lookups during activation

18

New Data Design Options — Delta, Init, and Web Services

• In SAP NetWeaver BW 7.3 you can create generic delta extraction for the Universal Data (UD) and Database Connect (DB) options, as well as for flat files

• Additionally, you can use the new DataSource adapter “Web Service Pull” to load data from external Web services You can even create generic Web services delta loads and load the new data

straight into the staging area of SAP NetWeaver BW 7.3• While Web services does not support hierarchies yet, there is now integration

of hierarchies into the standard process flow such as transformation and DTPs, as well as being able to load hierarchies from flat files using a new DataSource

When you use delta loads, the first time SAP NetWeaver BW 7.3 automatically defines it as an “init load.” After that, it automatically switches to “delta” as the InfoPackage mode (no need to define it anymore!)

19

New Options — ETL Interface and Shut-Downs

• SAP NetWeaver BW 7.3 has a new interface to create and manage data flows graphically in a “drag-and-drop” manner, including: Data Flow Copy to copy data

flows and process chains Data Flow Migration to

migrate, or reverse migrate, data flows with automatic adoption of InfoPackages and process chains

Use “RS_SYSTEM_SHUTDOWN” to restart an application server and suspend RDA and process chains and re-start them in a controlled manner. Data loads are not in “flux” and you do not have to clean up partial loads.

Use “RS_SYSTEM_SHUTDOWN” to restart an application server and suspend RDA and process chains and re-start them in a controlled manner. Data loads are not in “flux” and you do not have to clean up partial loads.


20

• SAP NetWeaver BW 7.3 has a new, step-by-step wizard that allows you to generate data flows from flat files or existing data sources

• A great benefit is that the wizards work against any InfoProvider; i.e., you can use the wizards to create loads from DSOs to DSOs or InfoCubes

This wizard reduces the number or manual steps needed to load data. It also simplifies the development process and makes ETL work much easier.

This wizard reduces the number or manual steps needed to load data. It also simplifies the development process and makes ETL work much easier.

The New DataFlow Generation Wizard

21

Layered Scalable Architecture (LSA) – New Templates

Source: sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/24800

• The LSA consists logically of: Acquisition layer Harmonization/quality layer Propogation layer Business transformation layer Reporting layer Virtualization layer

SAP BW 7.3 SP-3 has a set of 10 templates to help build a layered data architecture for large-scale data warehousing

22




23

Security Changes and New Mass Change Feature

• When companies upgraded from SAP NetWeaver BW version 3.5 to 7.0, security migration was optional and many did only a technical upgrade and kept the “obsolete authorization” concept. A security migration is now required.

SAP recommends you migrate to the new concept before upgrading a 7.0 system. (Notes: 931898; 938871; 946724, 958665; 1001652)

Great BW 7.3 Security Admin Feature: You can make mass changes to authorizations instead of one-by-one. This can be done by cut-and-paste in a worklist, hierarchy nodes, and you can also add users to multiple analysis authorizations.

2424

Security Features in SAP NetWeaver BW 7.0 and 7.3

• Some companies did not do a security conversion in SAP NetWeaver BW 7.0 since items such as 0TCTAUTHH did not migrate (manually reassigned for the hierarchies) and passwords became case sensitive. Now it is required.

• SAP NetWeaver BW 7.0 still has the ABAP migration tool. It can be used in SA38 (RSEC_MIGRATION), and migration can occur before the upgrade.

Authorizations are building blocks of the 7.0/7.3 reporting concept and security contains both the data value and hierarchy restrictions.

Security can be built using the “RSECADMIN” transaction

SAP BW 3.5 and BW 7.0 without security

conversion

BW 7.0 and 7.3 with security conversion

Technical Foundation

Authorization Objects Analysis authorizations

Maintenance Not changeable ChangeableNumber of InfoObjects

10 fields at 10 characters Unlimited

Navigational attributes

On global basis Individually

Hierarchy authorization

Via GUID 0TCTAUTHH Equivalent to value authorizations

Composition of Authorizations

Only intersection of authorization objects are permitted

Union ('as expected')

Authorization Relevance

Per InfoCube and InfoObject

Per InfoObject only

25

BW 7.0 and 7.3 Authorization Objects

After the required security conversion you

have access to lower security granularity for developers and users.

Authorization objects for the Data Warehousing Workbench:S_RS_DS: For the DataSource or its sub objects (BI 7.x)S_RS_ISNEW: For new InfoSources or their sub objects (NW BI 7.x)S_RS_DTP: For the data transfer process and its sub objectsS_RS_TR: For transformation rules and their sub objectsS_RS_CTT: For currency translation typesS_RS_UOM: For quantity conversion typesS_RS_THJT: For key date derivation typesS_RS_PLENQ: Authorizations for maintaining or displaying the lock settingsS_RS_RST: Authorization object for the RS trace toolS_RS_PC: For process chainsS_RS_OHDEST: Open Hub Destination

Authorization objects for the Business Explorer:S_RS_DAS: For Data Access ServicesS_RS_BTMP: For BEx Web templatesS_RS_BEXTX: Authorizations for the maintenance of BEx texts

Authorization objects for the Admin of analysis authorizationsS_RSEC: Authorization for assignment and administration of analysis authorizationsS_RS_AUTH: Authorization object to include analysis authorizations in roles

Changed Authorization Objects:S_RS_ADMWB (Data Warehousing Workbench: Objects):

New sub objects:CONT_ACT – Installing Business Content; USE_DND - Drag & Drop to InfoAreas and application components; CNG_RUN - Attribute change run

26




27

How To Leverage SAP NetWeaver BW 7.3 for Performance

• In this section we will explore post implementation reviews from a few real companies and see how version 7.3 may help in achieving faster load, query, and response times

• All examples come from real companies, but their names have been removed

SAP NetWeaver BW 7.3 has many performance benefits that can be retrofitted to earlier

implementations.

The trick is to use the upgrade as an opportunity fix older designs.

SAP NetWeaver BW 7.3 has many performance benefits that can be retrofitted to earlier

implementations.

The trick is to use the upgrade as an opportunity fix older designs.

28

Background – BW System Overview

• This system has the following characteristics:

• The system is medium sized in terms of data volume, but has a substantial hardware configuration

Database: Oracle version 10.2.0BW system: BW version 7.0 with content 7.0.4; Patch level 7 BPC system: BPC content 7.0; Patch level 8Operation systems: Linux 2.6 for BWA; AIX 6.1 for one app server and Solaris

10 for two app servers.Kernel: 701, patch level 94 (newer patches existed)

Area Value Trend

Active Users 106 steady

Avg. Availability per Week 100 % up

DB Size 3514.24 GB steadyLast Month DB Growth 131.88 GB up

This company should upgrade to SP-9, it contained performance improvements. Lessons learned: Keep an eye on new ServicePacks

Real Example

29

More System Background

• The system has been in production for several years and has was upgraded two years ago

• Currently, there are over 700 InfoProviders in the system

29

Most InfoCubes followed standard development guidelines, but some had abnormalities such as InfoCubes feeding DSOs.

Lessons learned: structured design review sessions should be undertaken as part of every project to assure that this does not continue.

0

50

100

150

200

250

300

350

400

11395

11 0 939

101

353

10

InfoProvidersReal Example

30

Users, Security, and Waits

• In January, there were 474 users logged on to the system (including developers and basis staff)

• Of these, 16 executed more than 400 navigation steps (high usage) 90 had moderate activity 368 had low activity

• 2,271 users are allowed to see all reports in the system, and can also list the content of all database tables. This is an unusual security configuration that should be revisited as soon as possible.

The customer should work with Network admin to determine capacity towards the servers. During dialog with the servers, the average user experienced a GUI wait time of almost 220 ms, indicating slow network connectivity to the servers.

Real Example

31

2500

2600

2700

2800

2900

3000

3100

3200

3300

3400

3500

3600

Oracle DB (Mb)

-300

-200

-100

0

100

200

300

400

Feb-2010 Mar-2010 Apr-2010 May-2010 Jun-2010 Jul-2010 Aug-2010 Sep-2010 Oct-2010 Nov-2010 Dec-2010 Jan-2011 Feb-2011

Oracle DB (Mb) Growth

Database Growth and Cleanup

• The database has grown between 12 and 344 Gb each month for the last year

• Three months of the year saw data, logs, and PSA cleaned. Data volume declined between 63 and 275 Gb those months

31

The database has grown by 732Gb (26%) in the last year, and the growth is uneven.

To do: Schedule “housekeeping” jobs. Better management of cleanup would result in more predictable patterns.(i.e. we found PSA data that had 10 months of load history).

Real Example

32

Database Performance — RSA1 Manage InfoCubes Performance

• Database statistics are used by the database optimizer to route queries.Outdated statistics leads to performance degradation.

• Outdated indexes can lead to very poor search performance in all queries where conditioning is used (i.e., mandatory prompts)

• The current sampling rates (3% for some cubes) are too low, and statistics should only be run after major dataloads, and be scheduled weekly

Most database statistics are outdated and cause database performance to perform significantly poorer than otherwise would be the case. Sampling should be changed and process chains should be re-scheduled.

Real Example

33

DSO Design and Locks on Large Oracle Tables

Many of the very large DSOs are not partitioned, and several objects have over 250 million recordsAdditionally, 101 DSO objects are flagged as being reportable. This results in System IDs (SIDs) being created during activation. I.e., 87 million SIDs for 0CO_ITEM_NO and 65 million SIDs for 0DOC_NUMBER

Combined, these result in frequent locks on the Oracle database and failed parallel activation jobs

-25 50 75

100 125 150 175 200 225 250 275 300 325 350 375 400 425

Mill

ions

DSO Number of Records

FIX: Partition DSOs. The lock on very large DSOs during parallel loads are well known and SAP has issued several notes on the topic:634458 'ODS object: Activation fails - DEADLOCK' 84348 'Oracle deadlocks, ORA-00060.'

Real Example

34

InfoCubes should be performance tuned if the number of records exceeds 30 million and partitioned before they are approaching 100+ million records. This creates faster loads, better query performance, and easier management.

-25 50 75

100 125 150 175 200 225 250 275 300 325 350 375 400 425

Mill

ions

InfoCubes Number of RecordsReal Example

BW 7.3 InfoCube Design

• Many InfoCubes are very large and not partitioned

• Several have over 100,000,000 records and one is approaching 0.5 billion (a stats cube where much of the data should be removed)

• We strongly recommend that most of these cubes are physically partitioned, and that a multiproviders with hints are used instead

• This can also be implemented in SAP NetWeaver BW Accelerator 7.2 if the relevant cubes are indexed.

35

Recommended Use of MultiProvider Hints in 7.3 (non-SPO)

• If a query has restrictions on this characteristic, the OLAP processor is already checked to see which part of the cubes can return data for the query. The data manager can then completely ignore the remaining cubes.

Problem: To reduce data volume in each InfoCube, data is partitioned by Time period.

A query must now search in all InfoProviders to find the data. This is very slow.

Solution: We can add “hints” to guide the query execution. In the RRKMULTIPROVHINT table, you can specify one or several characteristics for each MultiProvider, which are then used to partition the MultiProvider into BasicCubes.

An entry in RRKMULTIPROVHINT only makes sense if a few attributes of this characteristic (that is, only a few data slices) are affected in the majority of, or the most important, queries (SAP Notes: 911939. See also: 954889 and 1156681). Alternatively, some of the MP joins may be included on BWA v. 7.2

2002 2003 2004 2005 2006 2007 2008

Real Example

36

InfoCube Design — High Cardinality Flags

• High-Cardinality flag for large InfoCubes with more than 10 million rows

• There are currently 11 InfoCubes with a ratio of more than 20% of the records in the dimensions vs. fact table

• SAP recommends for Indexing and performance reasons to flag these as “high-cardinality” dimensions. However, it has minor impact to smaller cubes.

• Currently there are four medium and large InfoCubes that are not following the basic SAP design guidelines, and subsequently exhibit slow performance

Redesign large InfoCubes with high-cardinality to take advantage of the standard performance enhancements available.

InfoCube Number of rowsEntries in dimension compared for F table

FIUC_C03 12,859,780 37%ZGAT_C01 20,793,573 46%FIIF_C02 68,090,967 102%FIGL_C01 156,738,973 88%

Real Example

37

SAP NetWeaver BW 7.3 System Statistics — Data Volume Reduction• There are many statistical cubes with significant volume and no real benefits.

During the BW upgrade, most of these were not cleared and are now creating poor system performance. For example:

0TCT_C02 has 408 million rows; others also have millions of rows Stats are collected for over 1,900 objects, queries, infoProviders,

templates, and workbooks There are 7 aggregates with over 2 million rows on the stats cubes Creating aggregates on stats cubes inserts 1.9 million rows and takes

35.6 minutes for refresh each night High-cardinality flags are set for small cube with only one million rows

(0TCT_C21) Run-time stats are not collected for all InfoCubes and cannot be analyzed

37

To do: goto RSDDSTAT and select “Delete Data” for old stats and also schedule periodic jobs using standard process chains.

Real Example

3838

BW 7.3 hint: Explore the use of line item dimensions for fields that are frequently conditioned in queries. This model change can yield faster queries.

Use of Line Item Dimensions for 7.3 InfoCubes not in SAP NetWeaver BW Accelerator

• Line item dimensions are basically fields that are transaction oriented

• Once flagged as a line item dimension, the field is actually stored in the fact table and has no table joins

• This may result in improvements to query speeds for cubes not in SAP NetWeaver BW Accelerator

• Today, there are very few InfoCubes that used this feature (mostly TCT cubes)

39

Re-Design 7.3 Data Loads

• On an average daily dataextraction, transformation, and load process takes 44.8 hours if run sequentially

• A substantial amount of the time is spent on data transformation (51%) and lookups are often done on large DSOs without secondary indexes.

Consider upgrading as soon as possible to a newer ECC version that allows for better integration of standard BW 7.x extractors. Also, revisit extractor design for lookups on source system instead of inside BW.

Total Load time (sequential) - hrs 44.8 100% - Time spent on source extraction - hrs 11.1 24.7% - Time spent on error filtering - hrs 0.5 1.1% - Time spent on transformations - hrs 22.9 51.1% - Time spent on target - hrs 10.4 23.1%Number of records extracted from source 371,407,881Number of records written to target 125,102,791 33.7%

ETL Statistics (average per day)Real Example

• Of the 371 million records extracted from the source, only 33.7% are written to disk

• This is due to lack of ability to do delta processing for some files and also a substantial amount of transform and lookup logic in some of the ABAP rules

40

B-Tree Indexes on Large Objects

• InfoCubes that are not flagged as high-cardinality use a Bitmap index instead of a classical b-tree index for the joins.

• This type of index does not get “unbalanced” since it uses pointers instead of “buckets”

• When updating these large InfoCubes, dropping and recreating Indexes in the process chain can be very time consuming and actually take longer than the inserts

• It can also result in locks when the objects are very large (100 million+ records) and when attempting to do this in parallel (see ORA-0060)

Rebuilding bitmap indexes in load processing for large objects should not be a default answer for all designs. The process chains that do that, need to be revisited.

Real Example

41

Legacy SAP BW 7.0 Performance in 7.3 - Data Activation

This results in many reads on the NRIV table that slows down data activation and process chains (see notes:

857998, 141497, 179224 and 504875)

InfoCube Dimension Rows (thousands)

Buffer range level (thousands)

FIGL_C012 138,286 163,271 FIGL_C016 47,005 53,133 FICP_C053 10,172 29,655 FICP_C059 6,268 18,808

FICP_C02 FICP_C021 3,281 15,740 FICP_C013 9,149 46,308 FIAR_C014 3,804 64,388 FIAP_C017 19,139 25,844

FIIF_C02 FIIF_C022 69,264 79,758

FIGL_C01

FICP_C05

FICP_C01

Start buffering of number ranges of dimensions and InfoObjects (see tables above) Or use 7.3 data activation instead

Objects: BID0003554; 4020; 3531; 4016; 2364; 2313; 2450; 3834 and 2309

Table InfoObject Rows (thousands)

Buffer range level (thousands)

/BIC SGOBJKEY 12,792 18,699SCO_ITEM_NO 87,423 93,099SDOC_NUMBER 65,184 71,713SAC_DOC_NO 16,814 19,693SREF_DOC_NO 15,321 18,663SBILL_NUM 10,389 14,704SBA_BELNR 9,433 13,421SCO_DOC_NO 11,468 12,282SMAT_DOC 10,427 12,040STCTSTEPUID 9,951 11,793

/BI0

Real Example• During activation, there is limited use of buffering of number ranges for dimensions and InfoObjects, even when the number of entries are large.

42

PSA Cleanup in SAP NetWeaver BW 7.3

• For this company there were over 1.6 billion rows in the PSA

• An estimated 27Gb could be freed up if data older than one month is removed

• A formal retention policy that is communicated and enforced should be implemented ASAP

Lessons learned: Start PSA cleanup ASAP and schedule regular jobs that take care of this in the future

AGE of PSA Number of Records PercentEstimated

Gb <1 MONTH 591,774,063 35% 14.9 1-3 MONTHS 738,204,015 44% 18.6 3-6 MONTHS 144,476,834 9% 3.6 6-12 MONTH 173,651,469 10% 4.4 1-2 YEARS 35,421,883 2% 0.9

Total 1,683,528,264 100% 42.3

Real Example

43

Aggregates Are Not Needed with BW 7.3 and BWA 7.2

• At this company, there are 11 aggregates in the system• Four are related to the cube ZIR_C01

and seldom used and (two has never been used by any query)

• The 7 other aggregates are used only by the statistical cubes.

• Every day, 1.9 million records are inserted into the aggregates and take 35.6 minutes of processing time

InfocubeNumber of records in

Aggregate0TCT_C01 895,0700TCT_C21 420,3140TCT_C01 313,5400TCT_C01 188,8750TCT_C21 87,7710TCT_C22 82,0510TCT_C23 69,725

To do: delete unused aggregates. Also, by reducing the data volume in the underlying statistical cubes (cleanup), the remaining

aggregates will reduce in size and processing time. If ZIR_C01 was in BWA, all four related aggregates can be deleted.

Real Example

44

The Memory Cache Size Utilization

• The OLAP Cache is by default 100 MB for local and 200 MB for global use

• The current system at this company was consuming no more than 80MB on average

• This means that most queries were re-executing the same data

Real Example

45

Since many of the queries are using CKF and sums and sorts extensively, the cache read mode for most queries should be turned on ASAP at this company

Real Example

SAP NetWeaver BW 7.3 OLAP Cache — Turned Off?

• At one client, the OLAP cache was turned off for 131 out of 690 queries (excluding 4 planning queries in BW-IP)

• The cache was also turned off for 24 out of 256 InfoCubes• The OLAP Cache mode for their IP queries should have been

stored as “Binary Large Objects (BLOB),” that would speed up caching and very large reads, as is done in these queries

46

Broadcast to Pre-Fill the Cache — MDX Based SAP BusinessObjects Reporting

46

To do: Set up Java connectivity ASAP and use the Broadcasting feature to prefill the MDX cache for BI analytical processing intensive functions such as CKF, Sorts, Exceptions, Conditions

Real Example

• This company’s Java Stack did not communicate properly with SAP NetWeaver BW, and multiple logons were required

• As a result, broadcasting could not be used until the connectivity was set up correctly

47

ConformedReportable DSO

Write OptimizedDSO

BPC on SAP NetWeaver BW 7.3 — Current System

• This BPC system was experiencing substantial load performance issues

• Some of this was due to underlying SAP BW configuration, while some was due to the technical configuration of the data store architecture and data flow inside SAP BW

Production Issues include:1) Dependent jobs not running

sequentially, i.e., load from Summary cube to Staging cube is sometimes executed before the summary cube data is loaded and activated, resulting in zero records in the staging cube.

2) Long latency with 6 layers of PSA, DSOs, and InfoCubes before consolidation processes can be executed.

FIGL_D15S FIGL_D10S FIGL_D08FIGL_D13S FIGL_D11S

FIGL_D21 FIGL_D17 FIGL_D14FIGL_D20 FIGL_D18

GL Summary Cube

(FIGL_C03)

BPC Staging Cube

(BPC_C01)

Consolidation Cube

(OC_CON)

ECC 6.0

Asia-

Pacific

ECC 6.0

North-America

ECC 4.7

Latin-America

R/3 3.1i

EU will

move to ECC6.0

ECC 4.7

ASIA Shut down in 2011

Persistent Staging Area (PSA)

Consolidation Processes:1) Clearing2) Load3) Foreign Exchange4) Eliminations5) Optimizations

Real Example

48

Write OptimizedDSO

BPC on SAP NetWeaver BW 7.3 — Possible Future System

• Long-term the BPC system on SAP NetWeaver BW 7.3 should be simplified. This includes removing the conformed DSO layer, with BEx flags for data stores that are never reported on.

• Also, the BPC staging cube serves little practical purpose since the data is already staged in the GL Summary cube and the logic can be maintained in the load from this cube directly to the consolidation cube.

FIGL_D15S FIGL_D10S FIGL_D08FIGL_D13S FIGL_D11S

GL Summary Cube

(FIGL_C03)

Consolidation Cube

(OC_CON)

ECC 6.0

Asia-

Pacific

ECC 6.0

North-America

ECC 4.7

Latin-America

R/3 3.1i

EU will

move to ECC6.0

ECC 4.7

ASIA Shut down in 2011

Persistent Staging Area (PSA)

Consolidation Processes:1) Clearing2) Load3) Foreign Exchange4) Eliminations5) Optimizations

Long-term benefits include reduced data latency, faster data activation, less data replication, smaller system backups as well as simplified system maintenance. Take advantage of the 7.3 features!

Real Example

49




50

In-Memory Processing — SAP NetWeaver BW Accelerator 7.2 and SAP NetWeaver BW 7.3 • BIG DEAL! - In SAP NetWeaver BW

7.3, you can also build analytical indexes These are simply “data

containers” from APD that is stored in SAP NetWeaver BW Accelerator

You can build queries on them • Using the Index Designer in

SAP BusinessObjects Data Services, you can also index data from external data files and data bases and query it in SAP NetWeaver BW 7.3

SAP BW

InfoCubes

DSOs

1. Indexing and compression stored on a file system

2. Indexes copied into RAM on blades

BI Analytical Engine

3. Queries are routed to BWA by the Analytical engine

Business Object Data Services

Files

Databases

Index DesignerExternal data

DataIntegrator

It is important to note that currently, the SAP

BusinessObjects indexes are on dedicated hardware

blades.

51

CompositeProviders: Unions of Indexes in SAP NetWeaver BW Accelerator• With 7.3 you get a tool to create a new “CompositeProvider” to join indices

inside the SAP NetWeaver BW Accelerator system• This works similar to MultiProviders, but can be modeled ad-hoc and the joins

are executed inside SAP NetWeaver BW Accelerator and not inside the BW Analytical Engine (this is much faster)

51

All BW 7.3 CompositeProviders, DSOs, MultiProviders & InfoCubes are available as InfoProviders for querying. Even queries can be defined as InfoProviders.

All BW 7.3 CompositeProviders, DSOs, MultiProviders & InfoCubes are available as InfoProviders for querying. Even queries can be defined as InfoProviders.


52

The New SAP NetWeaver BW 7.3 Features — Modeling in SAP NetWeaver BW Accelerator 7.2

With SAP NetWeaver BW 7.3 and SAP NetWeaver BW Accelerator 7.2, you can complete many of the joins and the aggregation that previously were conducted in the BI Analytical engine, and was therefore hard to performance tune

It also reduces the data being transferred to the app server and therefore is much faster as well

53

The New SAP NetWeaver BW Accelerator 7.2/SAP NetWeaver BW 7.3 — Faster Performance• The major benefit of SAP NetWeaver BW Accelerator 7.2 on SAP NetWeaver

BW 7.3 is that you can move more of the calculation over to the accelerator and materialize the result in-memory

• This improves the reads, and calculations in the OLAP/BI analytical engine on the SAP NetWeaver BW side are substantially reduced


The following calculations is included in BW 7.3/BWA 7.2SUM MIN MAX CNT (count all detailed values)

CN0 (Count all values not zero or null) AVG (average)AV0 (Average for all values not zero or null)

54

The SAP NetWeaver BW 7.3/SAP NetWeaver BW Accelerator 7.2 Limitations and Query Settings • There are still some limitations. For example, you cannot use the

exception aggregation for single key figures in SAP NetWeaver BW Accelerator if it uses: Virtual key figures Conversion before aggregation Formula calculation before aggregation Non-cumulative key figures Key figures with elimination of internal business volume

You also get more detailed control on each query as to how it reads SAP NetWeaver BW Accelerator. Think of it as similar to Query read modes.(more details at SDN : http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/70950003-f7ef-2d10-b1bc-ee483800b25c?QuickLink=index&overridelayout=true

With BW 7.3 SP-5 is able to run completely on the HANA database (in-memory processing database instead of traditional RDBMSs)

With BW 7.3 SP-5 is able to run completely on the HANA database (in-memory processing database instead of traditional RDBMSs)

55

SAP HANA — Loading the Application and Performance

• You can load the application based on the logs in the source system, ETL-based (Extract Transform and Load) loads, and SAP trigger-based replication

Log-based replication is possible on IBM DB 2 LUW/UDB and planned for MSFT SQL Server Enterprise Edition, Oracle Enterprise Edition, and Sybase ASE (as of Dec. 2011)

Log-based replication is possible on IBM DB 2 LUW/UDB and planned for MSFT SQL Server Enterprise Edition, Oracle Enterprise Edition, and Sybase ASE (as of Dec. 2011)

Max Min AverageBank 1 : 6.3 521.6 258.8 369.5HANA POC (SAP) 1 : 5.2 484.3 301.4 350.3

Query speed improvementsCompression (data)

Project

Some Reported SAP HANA Performance Achivements

Tool PurposeBusinessObjects Data Services 4.0 ETL-based replicationSybase replication server & Load Controller Log-based replicationSAP Landscape Transformation (LT) Trigger-based replication

56

Persistence Layer

Looking Inside SAP HANA — In-Memory Computing Engine (IMCE)

Disk Storage

Data

VolumesPage Mgmt.

BusinessObjects Data Services

Log

Volumes

Logger

Metadata

Manager

Authorization

Manager

Transaction

Manager

Relational Engine

-Row Store-Column Store

Load

Controller

SQL Script

Calculation

Engine

Replication Server

SQL Parser

MDX

Session Manager

Inside the Computing Engine of SAP HANA, we have many different components that manage the access and storage of the data. This includes MDX

and SQL access, as well as Load Controller (LC) and the Replication Server.

Inside the Computing Engine of SAP HANA, we have many different components that manage the access and storage of the data. This includes MDX

and SQL access, as well as Load Controller (LC) and the Replication Server.

57

Enterprise Data Warehouse – SAP BW

SAP HANA — Virtual Marts and Applications

• Virtual data marts and new applications were built that run on SAP NetWeaver BW, which is again enabled by SAP HANA in-memory processing

ERP

Database

HANA (in-the works)

Virtual

Data

Marts

Applications

Databases

Virtual

Data

Marts

Virtual

Data

Marts

Virtual

Data

Marts

BI Solutions

Files

This provides much tighter integration with the source system, (less data latency) and much faster query response time for high-volume analysis

This provides much tighter integration with the source system, (less data latency) and much faster query response time for high-volume analysis

Applications developed by SAP1. Profitability analysis 2. Dynamic cash management3. Strategic workforce planning4. Smart meter analytics (power companies)5. Planning & consolidation6. Customer revenue performance mgmt7. Predictive segmentation & targeting8. Trade promotion management9. Merchandise & assortment planning10.Sales & operations planning (SOP)11.Demand signal repository

58




59

Additional Resources

• SAP NetWeaver 7.3 on SAP Developer Network www.sdn.sap.com/irj/sdn/nw-73

• LSA Templates and Architecture in SAP NetWeaver BW 7.3 www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/24800

• Features list for SAP NetWeaver BW 7.3 www.sdn.sap.com/irj/sdn/edw?rid=/library/uuid/304444f7-e02d-

2d10-9c97-d5e3ecf09882• SAP NetWeaver 7.3 Business Content

http://help.sap.com/saphelp_nw73/helpdata/en/ca/6fbd35746dbd2de10000009b38f889/frameset.htm

• Roadmap – SAP NetWeaver BW 7.3 www.sdn.sap.com/irj/sdn/bw-73?rid=/library/uuid/300347b5-

9bcf-2d10-efa9-8cc8d89ee72c

60

7 Key Points to Take Home

• SAP NetWeaver BW 7.3 is a stable version you can use now• The upgrade is somewhat technical, and strong skills are required• When you start, plan for how much of the new functionality you

will deploy. Include redesign and training as well.• You will need to spend time on security conversion if you have

not already done so• The new features are unique, so don't rely on past experiences• Complete a technical review before you start leveraging 7.3• Make sure you have access to experts and decide early how much

risk you can live with before starting “open heart surgery”

Your Turn!

How to contact me:Dr. Bjarne [email protected]

61

Continue the conversation! Post your questions in the BI-BW Forum on Insider Learning Network*

*bit.ly/BI-BWForum

http://bit.ly/BI-BWForum

http://bit.ly/BI-BWForum

http://bit.ly/yTZkvP

62

Disclaimer

SAP, R/3, mySAP, mySAP.com, SAP NetWeaver®, Duet™®, PartnerEdge, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies. Wellesley Information Services is neither owned nor controlled by SAP.

© 2012 wellesley information services. all rights reserved. a comprehensive guide to building or...

Documents

new data modeling capabilities

new transaction

new etl interface

new mdx editor

new design options frontend

realtime data warehousing

db2 v9

enterprise data warehouse