307: bw performance tuning - queries / data loads ramesh sampath mike maddox suresh kandoor vj...

307: BW Performance Tuning - Queries / Data Loads

Ramesh SampathMike Maddox

Suresh KandoorVJ Sudarsan

http://www.codongroup.com

http://www.codongroup.com/

Objectives

To provide a comprehensive understanding of the factors affecting data load and query performance and to discuss the strategies to help identify, monitor and resolve performance issues.

2

• Why Load Performance & Query Run-Times are important ?

• Query Performance• Identifying long running Queries• Query tuning Techniques

• Data Load Performance• Identifying long running loads• Improving Data Load times

• Questions

Agenda

3

Why Query Run-Times ?

• More Reporting Functions moving from R/3 to BW

• User Frustration with Data Warehouse Promise

• Impacts Number of Analysis Performed

• User Productivity is affected by Query Run-Times

• Quick response times, reduces Concurrent Users

• Affects the bottom line !! 4

Query Performance

• Identifying and Isolating the slow Queries – Using BW Statistics– Using SAP Transactions

• Exploring Possible Solutions to achieve better query run-times

5

Identifying long running Queries

Using BW Statistics

Average Run-Time of Query in Seconds

6


Using BW Statistics

7


Using BW Statistics

8


Using SAP Transactions• ST22 – Query Run Time exceeded set Limit

for queries

9


Using SAP Transactions• Analyze SM50, SM66 to identify real-time query

runs / users affected

10

Why Load Performance & Query Run-Times are important ?

Query Performance Identifying long running Queries Query tuning Techniques

• Data Load Performance• Identifying long running loads• Improving Data Load times

• Questions

Agenda

11

Query Performance

• Query Tuning Techniques– Analyze Query Plan– Identify degenerated indexes or incorrect

statistics– Design Considerations (Data Modeling)– Queries on InfoCube and ODS– Selections on Hierarchies– Do’s and Don’ts in Query Building

12

Query Tuning Techniques

Displays Query Plan

Displays List of Aggregates applicable to query based on selection

Analyze Query Plan (RSRT)

Check Read Mode of Queries

13


Specify Database Hints

14


Specify Database Hints

Query Cost Estimate

Check the SQL Generated, to Identify Secondary Indexes on Dimension Tables, Master Data Tables

15


Full Table Scan on Fact Table – May lead to bad performance

Are the Statistics data Accurate ?

16


Identify degenerated indexes or incorrect statistics: (RSRV)

Check DB Parameter Settings

Check Degenerated Indexes

Check Database Statistics 17


Transaction: RSRV in BW 3.XCheck DB Parameter Settings

18


• Design Considerations for– Dimension Tables– Fact Tables– Aggregates– ODS Objects– Master Data

19


• Design Considerations for Dimension Tables:– Smaller Dimension Tables - Dimension Table size

< 5% of Fact Table– Avoid M:N relationship– Logical grouping of objects in dimension– Fewer Dimension Tables per Cube– Avoid Near Line Item Dimensions– For large dimension tables, change index type to B-

tree from Bitmap (default)– Create Secondary Indexes on Dimension tables

20


• Design Considerations for Fact Tables:– Partition large fact tables (over 10 million records)– Create Physical Partitioning (one Million records per

partition)– Compression– Aggregates– Avoid Virtual Key Figures– Avoid exception aggregation

21


Logical Partitioning of Cubes

Consolidated view of all data

Parallel SELECT Statements

MultiProvider

Europe

2000

America

2001

Asia

2002

The Power of Parallel Processing!

Query against MultiProvider

Sub-queries: Run against small structures (pruning)

Year

Partiton by Region

22


• Design Considerations for Aggregates:– Data Selected from DB:Reported > 20:1– Best on Delta Capable Cubes – Use Nav. Attributes in Aggregate (Pro / Con). -

Avoid adding frequently changing attributes– Create Aggregates on Hierarchy nodes– Create Base Aggregate and summarized

Aggregate from Base– Avoid Virtual Key Figures / Virtual Characteristics– Avoid "Before Aggregation" Formulas in Queries

23


• Design Considerations for ODS Objects:– Create Secondary Indexes – Physical Partitioning using DB Tools– Locally managed indexes on Partitioned ODS– Use Multi-provider to split the Data– Create summarized cubes – High level analysis on Cubes, Detailed level

analysis run queries or Jump to ODS queries. – Avoid Compounded Info Objects in ODS – Inefficient

filtering by SAP– Index on Master Data P & Q Tables

24


• Design Considerations for Master Data:– Additional Over head with Time dependent Master data

and hierarchies– Hierarchy vs. Navigational Attributes. Navigation

attributes perform better than hierarchies.– Create additional index on the attributes of the master

data that are used in query selections. X, Y Tables are used in Cube based Query filters. X, Y, P, Q Tables are used in ODS Based query filters. Create secondary indexes on Master data tables based on its usage

25


• Query Selection on Hierarchies– Temporary tables generated by SAP when querying based Hierarchy nodes. – Solution: Flatten the hierarchy nodes as attributes and select on them.

HIERNODE

26


• Do’s and Don’ts in Query Building:– Avoid generic queries – Break down queries by summary level & detail level analysis– Always have mandatory variables on time component – Take advantage of Partition Pruning – Limit the Use of hierarchies in Query Selections, use them

extensively in Query Output– Use trailing Wild cards in query selections – Index!– Create indexes on MD, Dimension table objects used in

query selections– Limit the use of ‘Before Aggregation’ Formulas

27


• Summary of Query Enhancing Techniques– Aggregates– Compression– Partitioning– Secondary Indexes– Accurate Database Statistics– Multi-Providers– Data Class / Table Space assignment for Cubes /

ODS Objects and Indexes

28



Data Load Performance• Identifying long running loads• Improving Data Load times

• Questions

Agenda

29

Why Load Performance ?

• With BW Evolving as Corporate Data Warehouse, more Functional Areas (Logistics / Financials / HR / …) are being serviced by BW

• Small Window to Load Large Amounts of Data

• Shrinking Data Load Batch Windows

• Request for Frequent Data Loads. Middle of day data loads, hourly …

30

Data Load Performance

• Identifying and Isolating the Load Performance:

– Using BW Statistics– Using SAP Standard Transactions

• Exploring Possible Solutions to improve load run times

31

BW Statistics – Load Times By Data Targets

Identifying long running loads

Record Count Load Time

32

Detailed Analysis for Info Cube Loads


Slow Update Rules

Inserts to Cube Slow

# of Records processed

33

BW Statistics – Load Times By InfoSource


Load Time

Record Count

34

Detailed Analysis to identify cause on all Loads (Cube / ODS / Master Data)


Slow R/3 Extractor

Slow Update Rules

Slow Transfer Rules

# of Records Processed

35


SM37 on BW system – ‘BI_ODSA*’ for ODS Activation Times

SM50, SM66 – Process Overview

ST04 – Database Analysis Overview

36


• Summary of factors affecting data loads– Inefficient Extract Programs– Inefficient logic in User Exits– Slow Data Transformation Services

– Transfer Rules– Update Rules

– Slow Data Loading Services– Loads into Info Cubes– Loads into ODS Objects

37



Data Load Performance Identifying long running loads Improving Data Load times

• Questions

Agenda

38

Improving Data Load Times

• Strategies to improve Load Performance:– Extractor Performance– Info Cube Data Loads– Data Transformation Services (DTS)– ODS Data Loads– Master data Loads– Flat Files Data Loads

39


• Strategies to improve Load Performance: (Contd.)

– Perform Run-time Analysis on Extractor to identify Processing Times by ABAP Logic & Database Selects

– Check Select Statements on Large R/3 Tables– Ensure that Selects are based on Primary Keys or

secondary indexes– Check the usage of Run-time ABAP Memory by Internal

Tables.

– Schedule set up job in parallel 40


• Every thing has been checked, but extract is still slow ?– Does the selection parameters entered at the

Info package facilitate the use of Indexes !

Selection Parameters

41


• Strategies to improve InfoCube Data Loads:– Load Master Data before Transactions to pre-create SID’s.– Number range Buffering on Large or Near Line item

Dimensions for Large data loads– Packet Size – reduce the number of records per data packet– Secondary Indexes – Drop indexes on complete re-loads or

loading significant amount of data (over 25% of existing cube data)

– ‘Turn-off’ Archival Logs before Large Initial Loads– On complete re-load of cube, do not delete dimension table

entries, if you do not anticipate any changes to dimension entries.

42


• Strategies to improve InfoCube Data Loads: (Contd.)– Incremental Data Loads

– Do not drop Secondary Indexes on Incremental Loads– Do not select Refresh statistics after load

43


• Strategies to improve DTS– Eliminate Single Selects on Update Routines &

Transfer Routines– Utilize Start Routines– Avoid Transfer Rules to enhance Array Inserts

Examine Update Routines – Info Object Level

Utilize Start Routines – Data Packet Level

44


• Strategies to improve ODS Data Loads– ‘Turn-off’ Bex Reporting– Mark line item characteristics viz., doc Number as

Attributes only– Large Initial Loads to ODS

– ‘Turn-off’ Database Archival Logs – Delete Secondary Indexes

– First time load to ODS (ODS data is empty) – Load up to 1 Million Records, but do not exceed it. – Activate After the First load to facilitate Bulk Inserts

45


Activating data in ODS Object

Setting ins SPRO

46


• Strategies to improve subsequent data loads to ODS:– Activation – Single Record Inserts vs. Mass Insert when ODS is

empty– Enable Unique records, if true. Bulk Inserts.

– Data Packet Size (<10 Data Packets per Request)– Eliminate unused indexes.– Focus on smaller data loads for fewer records per activation

to reduce the commit interval.

New in BW 3.X

47


• Why does deletion of an Active Request from ODS takes a long time even for few thousand records ?– Data base partitioning on Active table – No Index on ODS Primary Key on ODS Change Log Table.

Add a Index to Change log Table (SE11)

ODS Primary Key

48


• Strategies to improve Master Data Loads: – Smaller Data Packet Size to minimize commit

intervals– Convert full loads to delta extract based on date

selection in the extract tables– Create accurate DB statistics– ‘Turn-off’ Consistency Check.

49


• Strategies to improve Flat File Data Loads:– Use Logical File Names / Application Server Files– Run in Background Batch Mode– Break Large Files into Smaller Files and load them in parallel– Use Fixed Length Files in place of CSV files

50


• Summary of Data Load Enhancing Techniques– Extractor Performance on Source System– Info Cube Data Loads (Initial / Delta Loads)– Data Transformation Services (Transfer, Update

Rules)– ODS Data Loads (Initial, Subsequent Loads)– Master Data Loads– Flat File Loads

51

Questions ?

Questions

Ramesh Sampath [email protected] Sudharsan [email protected] Maddox [email protected] Kandoor [email protected]

http://www.codongroup.com

mailto:[email protected]




http://www.codongroup.com/

Thank you for attending!

Please remember to complete and return your evaluation form following this session.

Session Code: [307]

307: bw performance tuning - queries / data loads ramesh sampath mike maddox suresh kandoor vj...

Documents

load performance query

query building

better query

query plan rsrt check

master data tables

statistics data accurate

mode of queries

slow queries