tuning sap oracle aix v.1.2

IBM Americas Advanced Technical Support

© 2003 and © 2008 International Business Machines, Inc.

IIBBMM SSAAPP TTeecchhnniiccaall BBrriieeff

TTuunniinngg SSAAPP®® wwiitthh OOrraaccllee®® oonn AAIIXX®®

MMaarrkk GGoorrddoonn

IIBBMM SSoolluuttiioonnss AAddvvaanncceedd TTeecchhnniiccaall SSuuppppoorrtt

VVeerrssiioonn:: 11..22 DDaattee:: NNoovveemmbbeerr 2255,, 22000088



1. ACKNOWLEDGEMENTS ................................................................................................................ 4 1.1. VERSION 1.2 ................................................................................................................................... 4 1.2. INITIAL VERSION............................................................................................................................. 4

2. DISCLAIMERS ................................................................................................................................... 4

3. TRADEMARK..................................................................................................................................... 5

4. FEEDBACK ......................................................................................................................................... 5

5. VERSION UPDATES.......................................................................................................................... 5

6. INTRODUCTION................................................................................................................................ 5

7. ANALYZING A PROBLEM WITH A SPECIFIC PROGRAM OR TRANSACTION .............. 8 7.1. COMPONENTS OF SAP RESPONSE TIME ........................................................................................... 8 7.2. MAJORITY OF TIME IS CPU ON APPLICATION SERVER ..................................................................... 9

7.2.1. Summary of Majority of time is CPU on application server ............................................... 19 7.3. MAJORITY OF TIME IS DATABASE REQUEST TIME ......................................................................... 19

7.3.1. Types of problems causing high DB request time................................................................ 20 7.3.1.1. Indexes do not support predicates................................................................................ 20 7.3.1.2. Misuse of SAP Data Model ......................................................................................... 20 7.3.1.3. SELECT in LOOP instead of FOR ALL ENTRIES.................................................... 20 7.3.1.4. Data skew causes wrong access path to be chosen ...................................................... 21 7.3.1.5. Invalid or missing Oracle statistics .............................................................................. 21 7.3.1.6. Unnecessary SQL......................................................................................................... 21 7.3.1.6.1. Table could be buffered on application server but is not buffered .............................. 21 7.3.1.6.2. SAP instance buffers are too small .............................................................................. 21 7.3.1.6.3. FOR ALL ENTRIES with empty internal table .......................................................... 22

7.3.2. MIRO Example..................................................................................................................... 22 7.3.2.1. Summary of MIRO ...................................................................................................... 29

7.3.3. Program performs too many SQL operations...................................................................... 30 7.3.3.1. Summary of Program performs too many SQL operations ......................................... 33

7.4. MAJORITY OF TIME IS NOT CPU OR DB REQUEST TIME................................................................. 34

8. SYSTEM PERFORMANCE PROBLEMS ..................................................................................... 34 8.1. PERFORM SQL CACHE ANALYSIS................................................................................................. 34

8.1.1. Indicator of inefficient SQL access ...................................................................................... 34 8.1.2. SQL Bget ratios that are generally not problems ................................................................ 36 8.1.3. Local predicates do not match indexes................................................................................ 36 8.1.4. Reasons that the predicates do not match indexes............................................................... 38 8.1.5. Actions when predicates do not match indexes.................................................................... 39 8.1.6. High Impact SQL caused by missing indexes on custom table............................................ 40

8.1.6.1. Summary of missing custom index.............................................................................. 42 8.1.7. High-impact SQL caused by incorrect use of SAP data model ........................................... 42

8.1.7.1. Summary of incorrect use of data model ..................................................................... 46 8.1.8. High-impact SQL when optimizer takes wrong access path................................................ 46



8.1.8.1. Summary of wrong choice of access path.................................................................... 50 8.1.9. High-impact SQL is a symptom of SAP buffer problem ...................................................... 51

8.1.9.1. Summary of symptom of SAP buffer problem............................................................ 57 8.1.10. High Impact Unnecessary SQL............................................................................................ 57

8.1.10.1. Summary of High Impact Unnecessary SQL............................................................... 59 8.1.11. High Impact SQL is a symptom of Oracle statistics problem.............................................. 59

8.1.11.1. Summary of access path problem caused by statistics................................................. 61 8.2. CREATE CANDIDATE LIST FROM ST03N........................................................................................ 62

9. SYSTEM HEALTH CHECK ........................................................................................................... 63 9.1. CPU ACTIVITY .............................................................................................................................. 63 9.2. I/O ACTIVITY ................................................................................................................................ 65

9.2.1. Good I/O performance in Oracle......................................................................................... 66 9.2.2. Symptom of I/O hotspot on disk ........................................................................................... 67 9.2.3. May be AIX constraint or disk performance problem ......................................................... 68

9.3. ORACLE REVIEW........................................................................................................................... 69 9.3.1. Concurrent I/O..................................................................................................................... 69 9.3.2. Database Hit Rate................................................................................................................ 70 9.3.3. Database delays ................................................................................................................... 70

9.4. SAP BUFFER SETTINGS.................................................................................................................. 72 9.5. SAP BUFFERED TABLE STATISTICS................................................................................................ 73

9.5.1. Not buffered but could be buffered ...................................................................................... 73 9.6. EVALUATE MEMORY IN AIX ......................................................................................................... 74

9.6.1. AIX 6.1 Restricted Tunables concept ................................................................................... 74 9.6.2. AIX Alternative Memory Management ................................................................................ 75 9.6.3. AIX paging ........................................................................................................................... 76 9.6.4. Evaluating increasing memory for Oracle or SAP.............................................................. 78

10. FOUR GUIDELINES FOR AVOIDING PERFORMANCE PROBLEMS............................. 80 10.1. USE THE SAP DATA MODEL ...................................................................................................... 80 10.2. USE ARRAY OPERATIONS ON THE DATABASE............................................................................. 81 10.3. CHECK WHETHER THE DATABASE CALL CAN BE AVOIDED......................................................... 81 10.4. WRITE ABAP PROGRAMS THAT ARE LINE-ITEM SCALABLE ...................................................... 81

11. APPENDIX 1: SUMMARY OF PERFORMANCE MONITORING TOOLS........................ 82 11.1. SAP .......................................................................................................................................... 82

11.1.1. DB02 Transaction................................................................................................................ 82 11.1.2. SE11 Transaction................................................................................................................. 82 11.1.3. SE30 Transaction................................................................................................................. 82 11.1.4. SM12 Transaction................................................................................................................ 82 11.1.5. SM50 Transaction................................................................................................................ 82 11.1.6. SM51 Transaction................................................................................................................ 83 11.1.7. SM66 Transaction................................................................................................................ 83 11.1.8. STAT Transaction ................................................................................................................ 83 11.1.9. STAD Transaction................................................................................................................ 83



11.1.10. ST02 Transaction............................................................................................................. 83 11.1.11. ST03N Transaction .......................................................................................................... 83 11.1.12. ST04 Transaction............................................................................................................. 84 11.1.13. ST05 Transaction............................................................................................................. 84 11.1.14. ST06 Transaction............................................................................................................. 84 11.1.15. ST10 Transaction............................................................................................................. 84 11.1.16. RSINCL00 Program......................................................................................................... 84 11.1.17. SQLR Transaction............................................................................................................ 84 11.1.18. RSTRC000 Program ........................................................................................................ 84

11.2. AIX .......................................................................................................................................... 84 11.2.1. nmon..................................................................................................................................... 84 11.2.2. ptx......................................................................................................................................... 85 11.2.3. iostat..................................................................................................................................... 85 11.2.4. vmstat ................................................................................................................................... 85

11.3. ORACLE .................................................................................................................................... 85 11.3.1. STATSPACK ........................................................................................................................ 85 11.3.2. Trace and tkprof................................................................................................................... 85 11.3.3. Automatic Workload Repository .......................................................................................... 86

12. APPENDIX 2: REFERENCE MATERIALS.............................................................................. 87 12.1. ORACLE MANUALS ................................................................................................................... 87 12.2. IBM MANUALS ......................................................................................................................... 87

1. Acknowledgements

1.1. Version 1.2 Thank you to Damir Rubic and Steve Poon, who contributed new material on AIX and Oracle.

1.2. Initial Version Thank you to Walter Orb of IBM Germany, who was for several years the pSeries SAP performance lead for IBM Americas Solutions ATS, and who showed me many of the processes used in this paper. Thank you to Marty Carangelo, Dale Martin, and Ralf Schmidt-Dannert, for their contributions on Oracle® and pSeries performance. Thank you to Phil Hardy and Damir Rubic, who reviewed the paper and offered many improvements.

2. Disclaimers IBM has not formally reviewed this paper. While effort has been made to verify the information, this paper may contain errors. IBM makes no warranties or representations with respect to the content hereof and specifically disclaims any implied warranties of merchantability or fitness for any particular purpose. IBM assumes no responsibility for any errors that may appear in this document. The information



contained in this document is subject to change without any notice. IBM reserves the right to make any such changes without obligation to notify any person of such revision or changes. IBM makes no commitment to keep the information contained herein up to date. The paper contains examples from systems ranging from SAP 4.6 and Oracle 8.1.7 and AIX 4.3 up to SAP 7.00 Oracle 10.2 and AIX 5.3. The processes and guidelines in this paper are the compilation of experiences analyzing performance on a variety of SAP systems. Your results may vary in applying them to your system. Examples have been edited to clarify points for the paper.

3. Trademark • SAP®, R/3®, and SAP Netweaver® are the trademark(s) or registered trademark(s) of SAP AG in

Germany and in several other countries. • IBM®, RS/6000®, System p®, pSeries® and AIX® are registered trademarks of IBM

Corporation. • Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be

trademarks of their respective owners. • OraPerf.com is a trademark of OraPerf.com

4. Feedback Please send comments or suggestions for changes to [email protected].

5. Version Updates • Version 1.0 – initial version • Version 1.1 – add SM04 Sessions • Version 1.2 –

o Rename document to “Tuning SAP with Oracle on AIX” o added Section 7.3.1 (and sub-sections) – types of DB access problems o added SQL cache example Section 8.1.6 o added invalid statistics example Section 8.1.11 o removed BDCPV example of access path choice problem, and replaced with Section 8.1.8

and LTAP/LTAK example o added ST06N and AIX virtualization CPU display in Section 9.1 o Added Section 8.1.10 – empty FOR ALL ENTRIES internal table o Added SAP memory management (Sections 9.6.1 and 9.6.2) o Added Oracle related information in Sections 9.3.1 and 11.3.3.

6. Introduction There are two intended audiences for this paper – Oracle® DBAs and SAP BASIS administrators. Either may be doing performance analysis on an SAP system with Oracle on AIX. The goal of this paper is to provide each audience with material that is useful and new: An SAP Basis administrator experienced with

mailto:[email protected]



other databases should find the ORACLE specific tuning tools and techniques helpful, while the experienced ORACLE adminstrator is presented with SAP specific tuning tools and techniques. This paper covers the two most common types of performance problems – database performance and inefficient ABAP coding. While there are other causes of problems in SAP (e.g. network performance, external RFC interfaces, SAP instance configuration, SAP sort, etc), database and ABAP performance are the most common and generally have the biggest impact. In order to provide the most benefit in the smallest paper, these other issues are not included in this paper. This paper has a process-based approach, where different goals are pursued via different processes and tools.

• To fix a problem reported for a specific program, we will perform elapsed time analyses of programs, determine where time is spent, and optimize these long running parts. This includes interpretation of ST03N and STAT/STAD records, and using ST05 and SE30. The paper will demonstrate how to use the SAP stats to obtain database performance statistics, identify I/O bottlenecks and SAP problems, etc. The benefit of this approach is that it is focused on an area that has been identified as a business problem.

• To check for inefficient use of DB resources and improve overall database server performance, we will use ST04 statement cache analysis. The value of this approach is that it offers a very big potential payoff in reducing resource usage and increasing system efficiency. The disadvantage is that one may be finding and solving problems that no end-user cares about. For example, if we can improve the elapsed time of a batch job from 2 hours to 10 minutes, but the job runs at 2:00 AM, and nobody needs the output until 8:00 AM, it may not really be a problem. Even if it is not a business problem, it may still be beneficial to address a problem of this type as part of optimizing resource consumption, in order to reduce the computing resources required to support business requirements.

• To do a system health check, review AIX paging and CPU usage, Oracle I/O statistics, ST10 and ST02 buffering. The operating environment needs to be running well for good performance, but problems in these areas can be symptoms of other problems. For example, inefficient SQL can cause high CPU usage or high I/O activity. Therefore, a health check should be done together with analysis of SQL and ABAP problems.

This paper has many examples, and it describes what is good or bad in each example. There are not always specific rules given on what is good or bad, such as “Database request time” over 40% of “elapsed time” is bad and under 40% is good. Rather, this paper tries to focus on an opportunity-based approach, such as:

• Look for where a program (or the SAP and database system) spends time. • Ask “If I fix a problem in this area, will people notice and care that it has been fixed?”

It will discuss how to estimate the impact of solving a problem. System wide performance analysis (such as a statement of cache analysis, or ST03 analysis) will generally turn up several candidates. By estimating the impact of fixing these problems, one can decide which to address first. When doing this analysis, it is important to identify and track specific issues. Often, a performance issue may not have enough impact to merit a new index, or an ABAP change. In this case, we want to track that



we have analyzed it, and chosen not to do anything, so that we don’t waste time discovering it again next year. This paper refers to a number of SAP Notes. An OSS userid, or userid that allows access to service.sap.com, is a prerequisite for anyone doing performance analysis on an SAP system, whether the person is an Oracle DBA, AIX administrator, or SAP BASIS Administrator. This paper breaks performance management into three parts, which are discussed in Sections 7, 8, and 9:

• Analyzing a problem with a specific program or transaction • System performance problems • System Health Check.

Since AIX-level symptoms such as paging, excessive CPU use, and high I/O rates can be symptoms of application and SQL problems, one needs to start with reviewing the SAP and Oracle indicators, before taking action based on the AIX level indicators.


7. Analyzing a problem with a specific program or transaction If a problem has been identified, then the first step is to determine where most of the response time is spent. Based on this first step, there are different tools that are used to monitor the ABAP, database, RFCs, etc.

7.1. Components of SAP response time SAP note 364625 describes the different components of SAP dialog step response time. This paper is focused on programs that have high database request time, or high CPU time. These are the two most frequent causes of performance problems.

Figure 1: STAD transaction display of time components



7.2. Majority of time is CPU on application server As an example, assume that we have been asked to investigate the performance of VA05. It runs over an hour for some users. We check ST03N.

Figure 2: ST03N VA05

Average times are somewhat slow (“0 Response” is 3,933 ms per dialog step), with CPU on the application server about twice the length of database request time. Average times can hide problems with a few long running dialog steps.



Check the ST03N “Hit lists”, to see what the elapsed time components are when it runs for a long time. The longest running dialog steps are saved in the ST03N hit lists.

Figure 3: ST03N VA05 top time

There are some VA05 programs in the top time list, and the CPU time is over four times as long as DB request time. High CPU time on the application server can point to problems in the ABAP code. Since CPU time is the majority of elapsed time, use SE30 to trace the ABAP runtime. We arrange for an end-user (ID MOO in the examples below) to run VA05 so we can trace it.



RUN SE30 and select the TMP variant.

Figure 4: SE30 TMP variant selected



Then press change to change the TMP variant options.

Figure 5: SE30 gather stats on internal tables

Enable statistics for internal tables. By default these are off, but they are the most common source of high CPU use in ABAP programs.



In the “Duration/Type” tab, select “by call”.

Figure 6: SE30 Aggregation level

Select Aggregation level by call, so that different calls on the same table are reported separately. (Aggregation level “None” will cause the trace to grow very big very quickly.) Press save on the screen in Figure 6 and green arrow to go back to the SE30 main screen shown in Figure 4. Then press “Enable/Disable” on the SE30 main screen to show the list of work processes here in Figure 7.



Figure 7: SE30 start/end measurement

Select the process to be traced, and press start.

Figure 8: SE30 active



After tracing for a while, press “End measurement”, and green arrow back to the main SE30 screen shown in Figure 4. Select the trace file, and press “analyze”.

Figure 9: SE30 analysis

The program spent over 90% of elapsed time processing ABAP on the application server.



Press “Hit list” and then sort the list by “Net” to see where the time was spent.

Figure 10: SE30 sorted by Net time

Here, the majority of time is spent on a “READ TABLE” ABAP statement. Calculate the time per call as (544,981,426 microseconds/14,526 READ TABLE calls). Each READ TABLE takes over 35 ms, which is very long. Read table can be very fast, depending on the size of the table and the options used in defining the table. Note that the “Read Table IT_2396” statement takes about 30 microseconds per call (461269 microseconds/14526 calls).



Select the slow “Read Table” statement and press ‘Display source code” to see the ABAP.

Figure 11: SE30 source code

Goto > attributes – to display the owner of the program .

Figure 12: SE30 program attributes

This is SAP code. We will open an OSS message against it.



Performance problems reading internal tables are very common performance and scalability issues. In the SE30 transaction, press the “Tips and tricks” button to see various suggestions on ways to improve ABAP programming. “Tips and tricks” contains several examples of common problems with internal tables.

Figure 13: SE30 tips and tricks



One can review “Tips and tricks” for suggestions on ways to improve ABAP programs. In this case, the problem appears to be that the ABAP is doing a linear search of the internal table.

Figure 14: SE30 binary search

When a large internal table is read without the ‘BINARY SEARCH’ option, the program becomes much slower as the number of items processed grows and the table grows. The slow internal table read access in Figure 14 took 203 microseconds. In our trace of the program shown in Figure 10, we saw that internal table read accesses took over 35 milliseconds – a couple orders of magnitude longer.

7.2.1. Summary of Majority of time is CPU on application server Check the characteristics of the program. If a majority of time is spent in CPU time on the application server, use SE30 to trace the program. This is an SAP program, so we would open an OSS message to SAP. If the program were a custom program, then we would send this to the development team to investigate ways to speed up processing the internal tables, such as “BINARY SEARCH” option on READ TABLE.

7.3. Majority of time is Database Request time If the majority of program or transaction elapsed time is DB time, the first question we need to ask is whether the DB time is high because of a problem in retrieving the data. There are a number of different problems that manifest themselves as high elapsed DB time.



7.3.1. Types of problems causing high DB request time

7.3.1.1. Indexes do not support predicates Local predicates are the “column operator value” clauses in SQL. In order to execute these efficiently, the column must be in an index. If the column is not in an index, then Oracle must read the table to check whether the row contains the value which is sought. Join predicates are the table1.columnA = table2.columnB clauses in SQL. Nested Loop Join is the most common join method used with SAP, and in order to process an efficient NLJ, the inner table must have an index containing the columns in the join predicates. There is an example of this in Section 7.3.2 and Section 8.1.6.

7.3.1.2. Misuse of SAP Data Model SAP often stores data redundantly in more than one table. For example, a delivery document might contain the order number. And an order document might contain the associated delivery number. But, the delivery tables are indexed by delivery number, and order tables are indexed by order number, so if a program has the value of an order number and wants to retrieve the associated delivery, the program could use either table. But, if the program selects from the delivery table using order number, there is no index for order number and the select is slow. If the program selects from the order table, there is an index and the select is fast.

There are three SAP notes (185530, 191492, and 187906) that describe common errors in use of the data model and how to fix them.

There is an example of this problem in Section 8.1.7.

7.3.1.3. SELECT in LOOP instead of FOR ALL ENTRIES If an ABAP program has a list of keys in an internal table, there are two ways to use the table to select rows from the database: LOOP AT internal_table SELECT … from DB_TABLE where COLUMN = internal_table-column ENDLOOP

Figure 15: LOOP with select

Each SELECT call in the LOOP will make a call between the application server, and the database server. Instead, an array operation should be used:

© 2003 and © 2008 International Business Machines, Inc. SELECT … from DB_TABLE for all entries in internal_table



WHERE COLUMN=internal_table-column The LOOP AT will make one call to the DB for each value, the FOR ALL ENTRIES will group the values together, and SAP will create an IN list (there is one column=value in the WHERE) or an OR (if there is more than one column=value clause in the WHERE. There is an example of this in Section 7.3.3.

7.3.1.4. Data skew causes wrong access path to be chosen When Oracle estimates the number of rows that will be returned on a select, it uses the column cardinality and table cardinality to estimate the number of rows with each column value. For example if a column AUFNR has 1,000 distinct values, and its table has 10,000 rows, then Oracle assumes that each value specified for AUFNR will return 10 rows. However, if many rows contain the same value, then the data is skewed -- that is the distribution of values in rows is not uniform. If data is skewed, Oracle requires additional histogram statistics, combined with ABAP HINT, in order to use values at execution time to determine the skew, and choose a better access path. There is an example of this in Section 8.1.8.

7.3.1.5. Invalid or missing Oracle statistics Cost based optimizers collect statistics about tables and indexes (number of rows, distinct values of columns, etc) that are used to optimize SQL statements. If the statistics are missing, or not collected in the correct way, the optimizer may choose the wrong access path. There is an example of this issue in Section 8.1.11.

7.3.1.6. Unnecessary SQL

7.3.1.6.1. Table could be buffered on application server but is not buffered Tables that are seldom changed, and where the application can tolerate a small interval between when rows are changed, and the changes are available on all application servers can be buffered on the SAP application server to offload the database. There is an example of this issue in Section 9.5.1.

7.3.1.6.2. SAP instance buffers are too small If the SAP generic or single record buffers are too small, then tables that could be buffered on the application cannot fit into buffer, and must be read from the DB server. There is an example of this problem in Section 8.1.9.


7.3.1.6.3. FOR ALL ENTRIES with empty internal table The ABAP FOR ALL ENTRIES statement uses an internal table with a list of keys to be retrieved for a column or columns. If the internal table is empty, the local predicates on the columns are not generated in the SQL, which generally causes the program to retrieve rows that it does not need. There is an example of this problem in Section 8.1.10.

7.3.2. MIRO Example We have been asked to investigate the performance of the MIRO transaction. As a first step, check the response time characteristics of MIRO using ST03N.

Figure 16: MIRO ST03N times

Figure 16 shows that the average MIRO dialog step runs about 1 second (“0 Response” column), and about 2/3 of the elapsed time (“0 DB” column) is database request time. Next, select the “Database” tab to see if individual SQL calls are slow.



Figure 17: MIRO ST03N Database

In Figure 17, note that the average sequential read time is rather long -- nearly 35 ms. This can be normal for array fetch operations which retrieve many rows, but may also be a symptom of inefficient SQL. In general, average sequential read times in ST03N should be less than 5-10 ms. If average sequential read times are long, check the STAT/STAD records for the transaction, to determine whether many rows are retrieved with each sequential read. If there are many rows per read, use the time per row when evaluating performance.



Since DB request time is the majority of time (see Figure 16), and individual SQL calls may be long, trace the transaction with ST05. Here is the output of an ST05 SQL trace.

Figure 18: MIRO ST05 trace

In Figure 18, note that there are some slow SELECTs on RBKP – each takes over 300 ms and returns no rows. (The duration is in microseconds.) In general, a statement that matches the indexes well will take a few ms per row. Press the “ABAP display” button, which is the sheet of paper button in Figure 18, in order to see the ABAP source code.



Figure 19: MIRO ABAP display

The cursor will be positioned at the source line in the ABAP. In Figure 19, the ABAP is SAP dynamic SQL, which has created the SQL statement in the ‘code’ variable at runtime.



Figure 20: MIGO program attributes

After pressing the “Attributes” tab in Figure 19, the program attributes in Figure 20 show that the owner of this code is SAP.

Review the access path used by Oracle -- press the Explain SQL button in Figure 18.

© 2003 and © 2008 International Business Machines, Inc. Figure 21: MIRO RBKP table scan


“TABLE ACCESS FULL” in Figure 21 means that no index on the table is used. Get the local predicates (the WHERE “column operator value” clauses) from the statement in Figure 21. We will compare them to the columns in the indexes, to see if the predicates reference indexed columns.

• MANDT = • LIFNR = • WAERS = • RMWWR = • BUKRS = • BLDAT = • RBSTAT = • XBLNR =

Drill into the table name in Figure 21, to display the indexes and indexed columns.

Figure 22: RBKP has three indexes

In the index display in Figure 22, Oracle will apply predicates to the index columns starting from the top column of an index. With the predicates in the statement, MANDT is the only matching column in these indexes, since USNAM and ERFNAM, which are adjacent to MANDT, are not in



the local predicates. Thus RBSTAT, which is the third column in the RBKP~3 and RBKP~4 indexes, cannot be processed as matching column – Oracle would match MANDT, and then scan the index to check RBSTAT values. So, the problem seems to be that the predicates in the SQL do not match the indexes on the table. Since Figure 20 showed that the program is SAP code, we would:

• First, check the SAP data dictionary definitions • Second, search SAP notes for this problem • Third, open an OSS message

Check the indexes already defined in the data dictionary using: SE11 > display > indexes.

Figure 23: RBKP DDIC has 5 secondary indexes

The data dictionary contains three indexes (1, 2 and 4) that were not in the list of indexes in Figure 22. These are SAP indexes that by default are not enabled on the DB, but that can be created on the database if they are required for the business processes being used.



Select index in the list in Figure 23, and press ‘Choose’.

Figure 24: RBKP index in data dictionary

Comparing the columns in this index to the predicates in Figure 21, the predicates match MANDT, XBLNR, LIFNR, and BUKRS. It looks like creating the RBKP~2 index will solve the problem.

If the index is created, and does not solve the problem, then one would go to service.sap.com in order to search SAP notes, and then open an OSS message if no solution was found.

7.3.2.1. Summary of MIRO The transaction was identified as a candidate to review. We checked the runtime characteristics using ST03N, and found that the majority of time was DB time, and that per-call DB times were rather slow. The transaction was traced with ST05, and slow SQL was found. The slow SQL statement was initiated from an SAP ABAP program. We checked the data dictionary, and found that there were indexes on RBKP that were defined in the data dictionary, but not defined on the database. We create the index using the SAP definition.



Since the program is an SAP program, if the index had not been found in the data dictionary, we would search OSS for notes about this problem, and if none were found, would open an OSS message to SAP.

7.3.3. Program performs too many SQL operations A different variety of inefficient SQL is when a program uses single row selects instead of array selects. SAP has a construct (FOR ALL ENTRIES) that uses an internal table in an ABAP to specify the key values to be fetched from a table in Oracle. The FAE can be executed as an IN list (if there is only one variable column in the internal table) or as an OR (if there are two or more variable columns in the internal table. In this example, we have been asked to review the performance of a program that is slow. In checking the stats for the program, we see that database request time (“0 DB”) is long, but the average time for each sequential read is short, just 2.5 ms. (You may also see low average read times with long running programs if the counters for database time fill or wrap. See SAP note 99584 for details.)

Figure 25: ZMM program - lots of fast DB accesses

Average CPU time (“0 CPU”) is about ¼ of database request time, so we need to look at the database accesses with ST05 to investigate.



We trace the transaction with ST05, and then list the trace.

Figure 26: ZMM ST05

The program behavior is interesting. It repeatedly selects rows from the same table, EBKN. Next, summarize the trace (goto > summary)

Figure 27: ST05 summarized trace



Choose the starting point and end point in the summary, and then press the summary button to compress the trace by table and operation.

Figure 28: ZMM program compressed trace

Now we can see that almost all time in the database is single row fetches of EBKN. From the ST05 trace in Figure 26, display the ABAP source.



Figure 29: ZMM program source

The program is looping through an internal table (LOOP AT irep_req). For each entry in the table, the program selects a single row from EBKN. If it is possible to change the program to do a single FOR ALL ENTRIES select on EBKN from the internal table, the program would probably run much faster, since reading N rows at once from oracle is generally faster than reading 1 row N times. Since the program is custom code (Zxx), we send it back to the ABAP team, to review whether the program could be restructured to use SAP array fetch operations.

7.3.3.1. Summary of Program performs too many SQL operations ST03N statistics showed long database time and relatively small CPU time, but per-call sequential DB request times were good (2.5 ms). Since the STAT/STAD statistics may have wrapped or filled for such a long job, we used ST05 to trace the SQL and also confirm the SQL response times. The trace showed that the same table was repeatedly accessed. The ABAP program is looping through an internal table, fetching one row at a time from the database. The program needs to be re-evaluated, to determine if the single-row access to the database can be converted into an array operation, such as FOR ALL ENTIRIES.



7.4. Majority of time is not CPU or DB request time There are a number of SAP response time categories that can also indicate performance problems that are not covered in this paper. SQL and ABAP, which are covered in this paper, are the source of the vast majority of performance problems. The paper ‘Tuning SAP / DB2 / zSeries” on IBM Techdocs (www.ibm.com/support/techdocs) contains detailed explanations about additional SAP sources of program delays (such as ENQ delay, SAP sort, local I/O, commit work and wait, RFC, GUI, network, etc), their symptoms and possible fixes.

8. System performance problems

8.1. Perform SQL Cache Analysis In order to find inefficient SQL that may have a system-wide impact, because of using excessive CPU or doing excessive I/O, review the SQL cache (ST04 >detail analysis > SQL request).

8.1.1. Indicator of inefficient SQL access The primary indicator shown in ST04 of inefficient SQL access is when both ‘Bgets/exec’ and ‘Bgets/row’ are high. A Bget (buffer get) is when oracle references a page of memory in Oracle database memory. A high ratio is a sign that Oracle must search a large amount of data to find the results. Searching many pages will consume extra CPU and may cause extra I/O operations. Normally, it might take 5-20 buffer gets to retrieve a single row, and it can be much less when performing array operations that retrieve many rows with a single call to the DB. Newer versions of SAP and Oracle display CPU and elapsed time statistics, as well as the efficiency (bufgets/row and bufgets/exec) statistics. The CPU and elapsed time data is useful for estimating the impact of the statement on the system or program, but they are not needed to locate and analyze inefficient SQL.

Figure 30: ST04 display of SQL cache for Oracle 10


http://www.ibm.com/support/techdocs


Figure 31: ST04 SQL cache - big problem example

In Figure 31, the last statement in the list looks like it is very inefficient. The SQL cache statements were reset and then evaluated since reset. There were 497,407,252 buffer gets over the interval, and one statement has executed over 105,576,852 buffer gets – 20% of the total. Both Bgets/exec and Bgets/row are over 95,000, which is much higher than the 5-20 Bgets/row that one might see if the SQL were efficiently accessing the data

Figure 32: ST04 SQL cache - moderately inefficient SQL

Even when the Bgets/row is lower than the extreme example shown in Figure 31, if a statement is executed frequently enough, it can cause an important impact on the system with fewer Bgets/row. The statements circled in Figure 32 are each 2-3% of buffer gets during the interval. By addressing and fixing several of these problems, one can make a measurable decrease in CPU usage on the system.




The higher ‘Bgets/row’ and ‘Bgets/exec’ are, the more likely that the SQL is inefficient, and Oracle is doing extra work to retrieve the data. If a statement is in the top 20 in the SQL cache ordered by buffer gets (or Disk reads) during a peak time, and the Bget/exec > 50 and Bgets/row> 50, the statement should be examined to determine if it can be improved.

8.1.2. SQL Bget ratios that are generally not problems • When Bgets/exec is high and Bgets/row is low, SAP is doing array operations, which affect

many rows per database call. This is generally an efficient way to access the database. • When Bgets/exec is low and Bgets/row is high, this shows that the SQL seldom retrieves

rows. If “Proc rows” is 0, check whether the table contains data, and if the statement is needed in the program.

• When Bgets/exec is low, and Bgets/row is low, then the data has been retrieved efficiently.

8.1.3. Local predicates do not match indexes As described above in Section 7.3.1, one of the main causes of inefficient SQL is that the local predicates do not match the available indexes. Local predicates are the WHERE “COLUMN operator value” clauses in the SQL. If the predicates don’t contain columns that are in indexes, or if the indexed columns are not selective (that is they return many rows) then Oracle generally cannot access the data efficiently.


Figure 33: Sample explain

In Figure 33, the local predicates are: • MANDT = • VGBEL = • ROWNUM <=

ROWNUM <= is a predicate that is inserted by SAP DBSL to stop fetching rows before the complete result set has been retrieved as soon as the ROWNUM count has been reached. MANDT is usually automatically inserted into the SQL by SAP DBSL. The program specified VGBEL=. One can drill into the table or indexes shown in Figure 33, to see the indexes on the table.



Figure 34: Display indexes from explain

Now that we have predicates and index columns, one can compare to determine if the local predicate columns are contained in indexes. Comparing Figure 33 and Figure 34, we see that the only indexed predicate column is MANDT. Since there is only one value of MANDT, Oracle must look at all the rows of the table, in order to find the rows matching the ‘VGBEL =’ predicate.

8.1.4. Reasons that the predicates do not match indexes There are two basic scenarios:

• The program is not using the SAP data model correctly, and • The business process may require a new index on the table

Many kinds of information are redundantly stored in SAP. For instance, the transaction data for a billing document may contain columns with the document number for the delivery or sales order



being billed. But just because the information is present does not mean that is the right way to retrieve it. Billing documents, to continue the example above, are indexed to support lookup by billing document number. If a program tries to access the billing tables using the delivery column, in order to find the associated billing document, the column with the delivery number will not be indexed, and access will be slow. If this is the situation, the solution is not to create an index, but to use the SAP data model correctly. There is an example of incorrect use of the SAP data model below in section 8.1.6. SAP has several SAP notes that describe wrong and right ways to use the SAP data model to retrieve application information:

• MM – SAP note 191492 • SD – SAP note 185530 • PP – SAP note 187906

And if we look at SAP note 185530, we find this SQL and a solution:

Figure 35: Note 185530 - VBRP where VGBEL

8.1.5. Actions when predicates do not match indexes It can occur that specific business processes require new indexes on tables. Here are some guidelines on how to approach the problem when you find a program where the predicates do not match the indexes: ABAP Creator Table Creator Action when predicates do not match indexes SAP SAP • Check data dictionary for standard indexes which

are not active in DB • Check OSS notes • Open OSS message to SAP

Custom Custom • Rewrite program if possible • Evaluate table access patterns, whether table can be

buffered • Create index

Custom SAP • Check data dictionary for standard indexes that are not active in DB

• Review use of data model (will generally fix problem)

• Change program • Create index (should very seldom be necessary)



8.1.6. High Impact SQL caused by missing indexes on custom table SAP now externalizes more information from the Oracle SQL cache, such as elapsed time and CPU utilization statistics. In this example, the SQL cache is sorted by elapsed time, to bring the statements with longest total execution time to the top. Note that the top statement performs over 60,000 bufgets for each row processed, which is inefficient. With proper indexing, selecting a single row from at table will take 4-5 bufgets. Each execution is over 8 seconds (the time is in microseconds), and with proper indexing, selecting a row would normally take a few milliseconds.

Figure 36: Custom table SQL

We select the statement and press the explain button (wrench) to see the access path.

Figure 37: Custom table explain

And we see that Oracle is reading the entire ZSD table each time this statement is executed. From the SQL list in Figure 36, we can also display the ABAP text by pressing the “Display ABAP text” (paper and eyeglasses) button.



Figure 38: Custom program for custom table

The ABAP is custom development – the name starts with Y or Z. If you are not sure if the program is custom development, you can display the attributes (goto > attributes) to see who created the program. If created and modified by SAP, it is standard. In the explain on Figure 37, one can also display information about the table and its indexes by clicking on the name of the table. We compare the local predicates in the SQL statement

• MANDT = • ZSALES_DELIVERY = • ROWNUM <= (added by SAP DBSL for select single, to limit the rows in the result)

With the columns contained in the index • MANDT (which has one distinct value, so is likely not filtering) • ZTRANNUM (which has almost 2 million distinct values, but is not referenced in the SQL)

And now we can see the source of the problem – there is no index containing the ZSALES_DELIVERY column.

Figure 39: Custom table display indexes

At this time, we need to determine what action should be taken. From the table in Section 8.1.5, we see that possible actions might be to rewrite the SQL, or buffer the table in SAP, or create an



index. In this case, let us assume that we cannot re-write the SQL to match the available index, and buffering the table is not useful since the primary index does not contain the column ZSALES_DELIVERY. So, we will create a new index (MANDT, ZSALES_DELIVERY) on the ZSD table.

8.1.6.1. Summary of missing custom index The statement was identified by sorting the SQL cache by elapsed time. The top statement performed over 60,000 bufgets per row, which is inefficient. Each execution took over 8 seconds, which was about 1,000 times as long as it would take with a proper index. When we explained the statement, and compared the local predicates to the available indexes, we found that there is no index that supported the local predicate for ‘ZSALES_DELIVERY = gtab-vbeln’. The solution was to create a new index (MANDT, ZSALES_DELIVERY) for this statement.

8.1.7. High-impact SQL caused by incorrect use of SAP data model Sort the SQL cache by buffer gets, to bring the high impact statements to the top.

Figure 40: VBRP high impact statement

The first statement in the list is very, very inefficient – it performs over 100,000 Buffer gets per execution, and retrieves one row per execution.



Explain the statement to see what access path is used.

Figure 41: VBRP explain

Note the predicates, which we will need later. • MANDT = • VGBEL = • ROWNUM <=

The statement is doing an index range scan, so it is using the index, but we cannot tell yet how many index columns are matched in the SQL. Drill in on the table VBRP to check the indexes to see which columns from the predicates are in the indexes.



Figure 42: VBRP table and index

The only local predicate column that matches is MANDT. Since there is only one MANDT value, this does no filtering on the index. Oracle checks the index, and also has to go to the table and look at every row that satisfied the MANDT= condition to check the value of VGBEL. Use the “Display call point” button in Figure 40 to look at the source code.

Figure 43: Program accessing VBRP

Goto > attributes to see the owner of the program.



Figure 44: Program accessing VBRP not from SAP

So, we see that the owner of the program is not SAP. From the table in Section 8.1.5, we know that the first action is to examine if the program is using the data model correctly. SAP note 185530 describes several incorrect and correct lookups for SD. This problem is contained in the note:

Figure 45: SAP note with examples of using the SD data model



The action for this problem is to send it back to the developer. In this proposed fix, for SD documents there is a table VBFA (document flow) that contains the predecessor and successor documents for sales documents. Given a sales order number, one can find all the subsequent documents related to the sales order, or given an invoice number, one can find all the predecessor documents that lead to that invoice document.

8.1.7.1. Summary of incorrect use of data model Statement is found in ST04 SQL cache with very high Bgets/exec and high Bgets/row. Check the predicates and indexes, only one predicate (MANDT=) matches an index. Check the owner of the program. It is not SAP, so we suspect that the programmer is looking in the wrong place for the information. Check SAP note 185530, since the problem here is in use of the SD tables, and find the proposed fix. Send the program back to developer to re-work.

8.1.8. High-impact SQL when optimizer takes wrong access path While the Oracle cost–based optimizer generally does a good job of determining the best way to access tables based on the predicates in the SQL, it can go wrong. Here the SQL cache has been sorted by Disk reads, to find statements that run a long time and cause lots of I/O.

Figure 46: ST04 SQL cache for LTAP/LTAK example

The statement does about 170 buffer gets per row, which is inefficient. With proper indexes, a single table access is about 5 bufgets, and a two table join 10. Explain the statement, to see what it is doing.



Figure 47: LTAP LTAK explain

There are several things to note in the explain in Figure 47. First, Oracle estimates that 187,391 rows will be retrieved, but from the SQL execution statistics in Figure 46, we know that on the average there are about 560 rows retrieved on each execution. For some reason, Oracle is overestimating the number of rows that will be retrieved. Second, Oracle is choosing MERGE JOIN (where both tables are read and the rows are merged together) rather than the much more common (with SAP) NLJOIN, which reads one table first, and joins rows that qualify in the first table to the second table. Get the local predicates, which will be needed later:

• MANDT = • LGNUM = • PQUIT =

All three are on LTAP (called T00 in the SQL). The ST04 statistics tell us that this cannot be a good way to access the data, since each execution takes 170 buffer gets to return one row.



Next, check if there are indexes that support the local predicates on MANDT, LGNUM, and PQUIT.

Figure 48: LTAP indexes

Note that there is an index (LTAP~M) with (MANDT, LGNUM, PQUIT), but it was not chosen by oracle. Why would this happen? The default way that statements are prepared by SAP is that parameters are used. This enables a statement to be shared by many different threads, but has the disadvantage that Oracle cannot use histogram statistics to compare the values used at execution time, to determine when an infrequently occurring value is being sought. SAP provides an option to use an ABAP hint to change this default. See SAP note 129385 regarding how to prepare a statement with literals or values. If no values are used, then Oracle will estimate filtering by using the number of distinct values in the columns. In this example, the SQL is selecting

• MANDT = (one distinct value) • LGNUM = (24 distinct values) • PQUIT = (two distinct values)

So, on the average Oracle would calculate that 1/48th of the rows (1*24*2) would be retrieved, for each (MANDT, LGNUM, PQUIT) combination. We can check if the distribution of values is skewed, by counting the number of rows with each value, for example:



Figure 49: LTAP count PQUIT

One would check the distribution of all the columns with local predicates. Next, we need to determine what the parameters are set to at runtime.

• Read the ABAP to check for literals • Run ST05 and trace the program at runtime to check values

From Figure 46, we can display the ABAP source.

Figure 50: LTAP LTAK source

We would check the program attributes, to confirm that this is SAP code. It is a SAP standard program, so we go to Service Marketplace, to check for SAP notes.



Figure 51: LTAP LTAK SAP note

We find that there is a note, which adds a hint to the program and specifies that histogram statistics are needed. It is necessary that both steps are done to fix the problem – the histogram statistics gather the data that shows oracle that there is skew in values of a column, and the ABAP HINT passes the value used at execution time, so that Oracle can compare the table statistics to the execution value, to determine whether a frequently occurring or infrequently occurring value is being retrieved.

8.1.8.1. Summary of wrong choice of access path Statement is found in SQL cache with many buffer gets per execution, many buffer gets per row, and many disk reads per execution. Explain the statement, and we see that Oracle has overestimated the rows returned, and used MERGE JOIN. Check the predicate columns, and find that filtering is on PQUIT. Check the source code, and see that it is SAP. Check OSS, and find there is a SAP note to add hint and gather histogram statistics.



8.1.9. High-impact SQL is a symptom of SAP buffer problem Here, the SQL cache has been sorted by buffer gets, to highlight high impact statements.

Figure 52: A616 in SQL cache

The highlighted statement performs over 200 Bgets/exec, and on average retrieves less than one row per execution. Since the statement is executed so frequently, it is one of the top statements in total Buffer gets.



Figure 53: A616 explain

Explain shows that index A616~0 is being used to access the table. The statement predicates are:

• MANDT = • KAPPL = • KSCHL = • DATBI >= • DATAB <= • KUNNR = • ROWNUM <=

Drill into the A616 table in Figure 53 to display the indexes.



Figure 54: A616 index

Comparing the index columns with the predicates, MANDT, KAPPL, KSCHL match, but since KNUMA_AG is not in the SQL, then KUNNR and DATBI cannot be matched in the index, but can be processed via range scan.



In Figure 52, press the button ‘Display call point in ABAP program’.

Figure 55: A616 ABAP source

The program is SAP code, not in the customer Z* or Y* namespace. While the problem is not exactly that the predicates do not match indexes (five predicates are present in the index, not all with matching index access), we check the A616 table DDIC settings. We would check for undefined indexes as shown in Section 7.3.1, and also check technical settings. Using SE13, check the technical settings.



Figure 56: A616 technical settings

The table is configured as buffered on the application server, but there must be some sort of problem, since the SQL cache statistics in Figure 52 show over 200,000 SQL calls. When a table is buffered on the application server, there should be very few calls to the database server to retrieve rows from the table. It may be that the table is frequently changed and invalidated and then re-loaded or that the table does not fit in the application server buffer. These can be checked in Figure 58 below. The table may also be read by the ABAP in a way that bypasses buffering on the application server. See SAP note 47239 regarding buffered table behavior.



So, check the table buffers using ST02.

Figure 57: ST02 buffers with swaps

There are swaps, which means that the buffer has filled, and may not be able to hold all the objects referenced by the programs running on the application server. Drill in on the ‘Generic key’ line, then press the ‘Buffered objects’ button to see the next screen.

Figure 58: ST02 buffered objects

See SAP note 3501 for information on the meaning of the buffered status. Error means that there was an error loading the table into the buffer. This usually means that the buffer did not have enough space to hold the table.



There are very few changes on the table, so the problem is not that the table is being invalidated and re-loaded. So, we have now determined that the frequent calls to A616 are not a problem with inefficient SQL, but are caused by something else – that the generic buffers on the application servers are not large enough.

8.1.9.1. Summary of symptom of SAP buffer problem There is an SQL statement that is rather high in Bgets/exec and that is one of the highest statements in total Buffer gets, which means that it is having a high impact on database resource use. Look at the program source, and it is SAP code. So, we check the table attributes and indexes. The table technical settings show that the table should be buffered. Check the buffers on the application server, and find that the generic buffer is too small. The frequent calls to A616 are a symptom of another problem, the small generic buffer size. There are other situations where the SQL can be a symptom of another problem, as when buffered tables are accessed from ABAP in a way that bypasses the SAP buffering (SAP note 47239).

8.1.10. High Impact Unnecessary SQL In this example, we have ordered the SQL cache by I/O, in order to bring high-impact statements to the top of the list. The top statement looks OK, based on the efficiency metrics – the ratio of BUFGETS/rows processed is good. But, the question we ask is why would a program need to retrieve 14 million rows?

Figure 59: ZMR program retrieves 14 million rows

We explain the SQL to see what the program is doing.



Figure 60: Explain ZEN table

In the explain, we see something unusual. The only local predicate is “MANDT =”, so all the rows in the table for the productive MANDT are being selected. Usually we would only see SQL with MANDT= and no other local predicates in SQL that loads fully buffered tables. We need to check the program that is issuing the SQL.

Figure 61: ZMR program source

We see that the program is a custom program, and that the ABAP has selection conditions (where sales_order EQ …) that are not in the SQL. ABAP SQL works like a template for building the DB SQL. When the WHERE condition is converted to SQL local predicates, if there is no value specified for a variable or if an internal table is empty, that column is not included in the SQL. Here, the problem is that the internal table i_salesord is empty, so the DBSL does not create a local predicate on sales_order. Since MANDT is implicitly added to the SQL, we get an SQL statement with only “MANDT =”. The internal table used in FOR ALL ENTRIES places restrictions on the rows to be retrieved, so if the internal table is empty, then all rows are retrieved. In this case, the fix is to modify the ABAP to check for whether i_salesord is empty.



8.1.10.1. Summary of High Impact Unnecessary SQL The SQL in cache is efficient (low BUFGETs / row) but many rows are processed. We check the SQL, and the only local predicate is “MANDT =” – there are no other selection conditions. Checking the ABAP, there is a WHERE clause with sales_order column referencing an internal table. If the internal table is empty, then the DBSL does not create a local predicate on SALES_ORDER. The fix is to modify the ABAP to check whether the table is empty, before the SELECT.

8.1.11. High Impact SQL is a symptom of Oracle statistics problem In this example, we have ordered the SQL cache by total elapsed time, in order to find statements that have the largest impact on runtime (at the system level). Note that the top statement has over 6,000 bufgets per execution, but 0 bufgets per row. This is because no rows were returned. See Section 8.1.2 regarding ratios of bufgets to executions and rows. This example is taken from a test system. In general, problems with statistics on productive systems are encountered soon after new tables or new SAP functionality are transported into prod, or when errors have been made in existing procedures for gathering DB statistics.

Figure 62: D010INC high bgets/Exec

When we explain the statement, then drill into the table name in the explain to display the table, we see it is being read without an index, using ‘TABLE ACCESS FULL’.



Figure 63:D010INC explain

Click on the table name (D010INC) to see the indexes and index statistics.

Figure 64:D010INC indexes



SAP retrieves the statistics here from Oracle. At the time the statistics were analyzed, the table was empty. But the table is clearly no longer empty, since each execution of the SQL (see Figure 62) currently reads over 6,000 pages of data from the table. We can analyze the table to generate new statistics (note the “Analyze” button in Figure 64), and after the statistics are generated, when we explain the statement, the index is used.

If this were a productive system and the table were an SAP business table, we would need to review the statistics gathering process for the table, to verify that it was correct, and we would need to analyze statistics for the table, in order for Oracle to choose a better access path.

8.1.11.1. Summary of access path problem caused by statistics We sorted the SQL cache to find statement with high elapsed time. The top statement did over 6,000 bufgets per execution, and never returned a row. This is inefficient. We explained the statement and examined the table and index statistics, and found that the table had statistics gathered when it was empty, and the table had grown. See SAP notes 932975 and 1020260 (and other note they reference) regarding Oracle tables that require special statistics to ensure proper optimization.



8.2. Create candidate list from ST03N Run ST03N, sort by total database time.

Figure 65: ST03N by DB time

Transactions which have the longest DB time, and thus probably the largest impact on the database server, are at the top of the list. Next, click on the “Database” tab, to see the per-call times.

Figure 66: ST03N by DB time database tab




Check the average times for sequential read (‘0 Seq read time’ column). If this is long, it usually means one of two things:

• Each read retrieves many rows (this is not a problem, if the program needs the data), or • The SQL access is inefficient, and it takes a long time for a few rows (this is a problem).

At this point, we cannot tell which is occurring, but one can trace the programs with ST05, as shown in Section 7.3, to determine whether there is a problem or not.

9. System Health Check

9.1. CPU activity As discussed above, SQL problems can contribute to high CPU use on the DB server, and ABAP coding problems can contribute to high CPU use on the application server. If the system has high CPU utilization, then first search for SQL and application problems that may be causing high CPU use. For native or dedicated LPAR environments, SAP ST06 (or OS07)transaction has statistics on recent use, and on historical CPU use. Since ST06 has statistics based on hourly averages, it will not show CPU constraints until they are very severe. It is generally better to monitor CPU use using an AIX based tool, and gather statistics on 5 or 10 minute intervals, in order to be able to calculate average and peak activity. For SPLPAR environments, ST06N (or OS07N) is required together with an updated saposcol, in order for SAP correctly report CPU activity. See SAP note 994025 regarding SAP and AIX requirements for SAP monitoring in a virtualized AIX environment. Important Note: The older OS monitors in SAP (ST06 and OS07) will give misleading and incorrect output, when used in a virtualized AIX environment.


Figure 67: ST06N with AIX virtualization support

One can use nmon, sar, ptx, or other tools to gather and report on the utilization. While it may be acceptable for an application server running batch work to run at 100% utilization, if an application server supporting dialog or critical interfaces is frequently hitting 100% utilization, it can have a significant impact on response time and end-user satisfaction. If the DB server has high CPU activity:

• Evaluate the impact of inefficient SQL and applications as in Section 8.1. • If there is a SAP application instance on the DB server, investigate whether it can be moved

off. • If no work can be moved off, evaluate moving some batch jobs to run outside of peak time. • Acquire more CPU power for the DB server

If one application server has high CPU activity

• Evaluate the impact of inefficient ABAP code on excessive CPU use as in Section 7.2. • Evaluate changes in the SAP login groups to balance the workload differently • Change the mix of batch/update/dialog to reduce the amount of work on the server • Acquire more CPU power




9.2. I/O activity I/O activity should be monitored at the Oracle level from SAP using ST04, or as an alternative, one can use STATSPACK to monitor it at the Oracle level directly from Oracle. Oracle STATSPACK has the benefit that it can be configured to automatically gather statistics, which can be used for historical reporting and comparison. ST04 file statistics are gathered by logging in to view the current statistics, or can be manually reset for an interval. ST04 and STATSPACK both report average read times, which are more useful than the read rates available with most AIX monitors. Another reason for starting with the Oracle view of I/O activity is that there are several different configuration problems that can cause the symptom of slow I/O at the Oracle level, though physical I/O at the AIX level is fast. The actual problem is serialization on a resource (e.g. AIO server processes, filesystem buffers, or filesystem i-nodes) in AIX after the I/O is issued by Oracle, but before AIX sends the I/O to the disk. ROTs for sites using Oracle database on JFS filesytems

• Configure AIO max servers to be (oracle data files * 1.25), to prevent shortages of AIO servers • Set vmtune/vmo numfsbufs to at least 300, to prevent shortage of fsbufs • Limit the size of each datafile to 2 GB, to reduce i-node serialization problems

ROTs for sites using Oracle database on JFS2 filesystems

• Configure AIO max servers to be (oracle data files * 1.25), to prevent shortages of AIO servers • Set vmtune/vmo numfsbufs to at least 300, to prevent shortage of fsbufs • Mount the Oracle database filesystems with the cio option, to avoid i-node serialization

problems There are also queue sizes on disk adapters and hdisks that can be checked and set in the ODM via SMIT. In our experience, the defaults work fine with SAP. ST04 > detail analysis menu > file system requests can be used to view read and write activity and times, as viewed by Oracle. With AIX AIO, the write times are often unreliable, so it is generally best to focus on the read times as an indicator of performance problems.


9.2.1. Good I/O performance in Oracle In the following section of an Oracle v$filestat report (ST04 > detail analysis menu > file system requests) note that the average I/O response times are small single digit, and they are not widely skewed. Here, the I/O response times are good. (Though there could still be performance problems due to low hit-rate in Oracle, or other factors.)

Figure 68: ST04 file system requests with good response times



9.2.2. Symptom of I/O hotspot on disk Figure 69 is an ST04 “file system request” report (Oracle v$filestat) that has been downloaded to Excel.

Figure 69: ST04 file system requests

It has widely varying I/O response times on the most frequently read datafiles. When there is skew in the response time of read-intensive files, it is a common symptom of an I/O constraint at the physical disk level, where the data files are not spread across enough disks. (A disk may be called a vpath, hdiskpower, LUN, etc – it is the AIX representation of the physical location where the data is stored). In order to prevent this from happening, we recommend using a database layout where each LV is configured to reside on many hdisks. In SMIT, this would be done by creating the LV on several hdisks, and specifying “Range of physical volumes maximum”. See the paper “Configuring the Enterprise Storage Server (ESS) for Oracle OLTP Applications”, which is document number WP100319 at http://www.ibm.com/support/techdocs, for more details about configuring a “stripe and spread” layout for the database.


http://www.ibm.com/support/techdocs


The solution for I/O hotspots is to analyze the I/O at the physical level in the storage system, and if overloaded spots are found, to spread the datafile across more disks:

• Determine name(s) of file(s) with slow response time – ST04 or STATSPACK • Convert filename to filesystem – using ls or find commands • Convert filesystem to LV – using df or lsfs • List PVs (or vpaths) in the LV – using lslv –p • If ESS, list LUNs for the vpaths – using lsess • Use ESS expert to check I/O activity for clusters, ranks, LUNs

9.2.3. May be AIX constraint or disk performance problem Since Oracle I/O statistics contain the time for the I/O to go through AIX and then be issued to disk, there are configuration problems that can masquerade as I/O bottlenecks. The ROTs for avoiding the most common configuration problems were listed in Section 9.2.

The AIX serialization problem mentioned here is not common, but can occur on a large system when a specific workload is being processed in parallel, so that many jobs are accessing and changing the same tables and same Oracle datafiles. This might occur when doing parallel IDOC processing, or running any business process (delivery due, billing, forecasting, etc) via many parallel batch jobs.

Figure 70: ST04 file system requests slow on high write files

The average I/O times are good on some files, but the files that have the most write activity have very slow I/O times. This could occur for several reasons, such as AIX i-node contention, or write cache filling on the disk system. I-node serialization occurs on JFS structured datafiles. When the file is being written, it is locked to serialize access and preserve integrity. This serialization blocks readers and writers from using the datafile.


When there are slow Oracle I/O times on frequently changed datafiles, the actions to resolve the problem might be: • Examine disk statistics (e.g. ESS expert) for signs of disk or cache overload related to high

write activity, if none found, then • Open a perfpmr to confirm whether AIX i-node serialization is a problem • If i-node serialization is a problem, then

o Move data to JFS2 filesystems with AIX 5.2, and mount with cio option, which gets rid of i-node serialization, or



o If JFS2 is not an option, and datafiles are currently larger than 2 GB, rebuild the slow tablespaces with a max 2 GB datafile size, or

o If JFS2 is not an option, convert to raw LV structured DB.

9.3. Oracle Review There are a couple quick checks one can do, as part of the system health check. See the SAP note 618868 for more information on oracle performance FAQs.

9.3.1. Concurrent I/O In section 9.2, cio, or concurrent I/O was mentioned as one way of improving I/O performance. Concurrent I/O is a feature of JFS2 provided in AIX 5.2 ML01 and newer versions. Cio includes the performance benefits of direct I/O, plus the additional performance benefit from eliminating contention on the inode lock. Concurrent I/O is intended for use by applications that do their own data serialization, such as Oracle. Do not place Oracle binaries on filesystems mounted with the cio mount option. Oracle 9i does not support opening of database files with the O_CIO option; therefore, the database filesystems must be mounted with the cio mount option in order to use concurrent I/O. Oracle 10g and 11g will open database files located on the JFS2 file system with the O_CIO option if the filesystemio_options initialization parameter is set to either directIO or setall. (The setall option should be used, as asynchronous I/O is disabled with the directIO option.) With Oracle 10g or 11g, the filesystems may also be mounted with the cio option. For more information, refer to:

• SAP Note 948294 - AIX JFS2: mount options to use with Oracle 10g • Oracle Doc ID: Note: 257338.1, Direct I/O (DIO) and Concurrent I/O (CIO) on AIX 5L.

For optimal performance when using direct I/O and concurrent I/O, the JFS2 blocksize (agblksize) must be set according to the Oracle db_block_size:

• For database files (.dbf): − For db_block_size ≥ 4096, use agblksize = 4096. − For db_block_size = 2048, use agblksize = 2048.

• For redo log and control files: Put the redo log and control files on a separate, dedicated filesystem and use agblksize = 512.

The agblksize is specified during filesystem creation time. To check agblksize, use the lsfs –q command as root user. For example, # lsfs -q /orafs Name Nodename Mount Pt VFS Size Options Auto Accounting /dev/oralv -- /orafs jfs2 165847040 rw yes no (lv size: 165847040, fs size: 165847040, block size: 4096, sparse files: yes, inline log: no, inline log size: 0, EAformat: v1, Quota: no, DMAPI: no, VIX: yes, EFS: no, ISNAPSHOT: no)


9.3.2. Database Hit Rate ST04 can be used to review the hit rate in the database. Since SAP often executes many SQL operations per dialog step, the goal should generally be to have a hit rate in the high 90s. A 96% hit rate is two times the misses of a 98% hit rate. A 94% hit rate is three times the misses of a 98% hit rate. The impact of cache misses quickly grows as hit rate decreases. Since inefficient SQL can cause the symptom of low hit rates, the SQL cache analysis process in Section 8.1 should be the first action, when a low hit rate is seen. After the SQL analysis, one can review adding more memory to Oracle (in order to increase hit rate) by using the process in Section 9.6.4.

Figure 71: ST04 database overview

9.3.3. Database delays The oracle v$system_event view can be used to review the causes of delay in oracle. These delay statistics are recorded in STATSPACK. STATSPACK reports can be uploaded to the web site www.oraperf.com for an analysis. As with hit-rates, many Oracle delays (buffer busy waits, enqueue, etc) can be symptoms of SQL or application design problems, and may not fixed with database parameters.


http://www.oraperf.com/


Normally, db file sequential read is the largest source of delay. See SAP note 619188 for more information on Oracle wait events.

Figure 72: ST04 - Oracle v$system_event delay statistics

While a workload is running, you can use “ST04 > detail analysis > oracle sessions”, to display wait events for the Oracle processes. Associating an Oracle wait event with its SQL statement can make it easier to determine the cause of the delay.

Figure 73: ST04 Oracle sessions



9.4. SAP buffer settings When SAP buffers fill and swap, it can cause performance problems for transactions and batch jobs. One example of this was shown in Section 51, where the generic buffer area was filled. In addition to this, another common problem is when the program buffer fills, causing swaps and impacting response time. One can check ST03N, to see the impact of program swaps overall.

Figure 74: ST02 swaps

ST02 has swaps on programs, CUA, screen. We check ST03N, to evaluate the system-wide impact.

© 2003 and © 2008 International Business Machines, Inc. Figure 75: ST03N overview


In Figure 75, ST03N data that has been downloaded to Excel. Note that load and generate time is about 5% of average dialog response time. However, this impact will not be evenly distributed. When the programs and screens referenced by a program are in SAP buffers, the program will run quickly. If they are not present, then it might take several seconds longer than normal. If the application servers are memory constrained, then one may choose to live with this impact. If one wants to be able to have more reliable transaction response times, then one would increase the size of the buffers that are swapping.

9.5. SAP buffered table statistics There are two quick checks that can be done:

• Tables that are not buffered, but which should be, and • Tables which are buffered, but whose settings should be changed.

9.5.1. Not buffered but could be buffered The most common problem with buffered tables is tables that could be buffered, but are not. Generally, these are custom tables. Most SAP tables have the appropriate settings already enabled. Figure 76 is a segment of an ST10 report for not buffered tables, ordered by calls to the database. Impact on the database is more a function of the buffer gets per row (as shown in Section 8.1) and not calls, but calls or rows are the only orders available with ST10.

Figure 76: ST10 not buffered tables




There are three tables in this list that may be buffering candidates: • YMSESPIV02 • OICDC • YVGMDRCADDR

Since OICDC is a SAP standard table (it does not start with Y or Z), one can check in SE13 if buffering is allowed but not currently enabled. If it is allowed but not enabled, we can turn buffering on. The two Y* tables are custom tables, so we must check:

• How large is the table • Will it be changed frequently (it looks like not, from ST10 stats) • Can the application tolerate a short period where the buffered versions are different, when

the table is changed in the database and changes are propagated to application servers. Since the tables are all read with direct read, if the table is reasonably small (e.g. <5 MB), not changed much, and the application can tolerate inconsistency, then they could be fully buffered. If the tables are very large, then they might be single record buffered, to save buffer space by buffering only the referenced rows.

SELECT SINGLE (SAP direct read) can be read from either the generic or single record buffer. SAP SELECT can only be read from the generic buffer.

9.6. Evaluate memory in AIX There are three different SAP memory management options, which are described in more detail below. The latest version supported for your SAP kernel should be used. The three options are

• ES/TABLE = UNIX_STD – should not be used with AIX 5 or AIX 6 • ES/TABLE = SHM_SEGS – for SAP kernels w/o support for ES/SHM_SEGS_VERSION=2 • ES/TABLE = SHM_SEGS & ES/SHM_SEGS_VERSION=2 – latest and best version

See below for information on kernel levels required for each. SAP Instance memory parameter settings can have an impact on AIX memory use. For example, the latest version of SAP EM (using ES/SHM_SEGS_VERSION = 2) has more efficient disclaim processing, which can reduce the amount of real memory needed for SAP EM. If there is paging in AIX, please review the latest versions of the SAP notes referenced in Section 9.6.2.

9.6.1. AIX 6.1 Restricted Tunables concept Since AIX 5.3, six tuning commands (vmo, ioo, schedo, raso, no and nfso) are available to tune the operating system for the specific workload. Beginning with AIX 6.1, some tunables are now classified as restricted use tunables. The default values for restricted tunables are considered to be optimum for most AIX environments and should only be modified at the recommendation of IBM support professionals. As these parameters are not recommended for user modification, they are no longer displayed by default, but can be displayed with the –F option (force). The –F option forces the display of restricted tunable parameters when the options –a, -L, or –x are specified alone on



the command line to list all tunables. When –F is not specified, restricted tunables are not included in a display unless specifically named in association with a display option.

9.6.2. AIX Alternative Memory Management In the previous versions of SAP kernels (4.XX and 6.XX), extended memory bottleneck would occur if the high numbers of user contexts existed on the system. To alleviate that problem, SAP implemented a new AIX memory management (ES/TABLE=SHM_SEGS) that allowed individual user context and overall extended memory size to grow beyond several GB. New memory management was based on the several different AIX memory allocation functions like disclaim(), late swap allocation etc… Several benefits are achieved using this method:

• Eliminating potential memory bottlenecks, • User context change is much faster, particularly for large contexts, • A much higher throughput is achieved, especially for higher context change rates (local

updates …)

Variable PSALLOC=EARLY must never be set if this Extended Memory implementation is activated. The profile parameter that would initiate the utilization of the new Extended Memory Management is:

• ES/TABLE=SHM_SEGS

Additional parameters are required and important for the successful implementation of the new memory management:

• EM/TOTAL_SIZE_MB: allows a maximum of 16 GB extended memory and can be larger, if required. This can be used to limit the total amount of Extended Memory, to prevent a swap bottleneck.

• ES/SHM_PROC_SEG_COUNT: specifies the number of shared memory segments that are used as shared and privately.

• ES/SHM_MAX_PRIV_SEGS: specifies the number of shared memory segments that are used privately.

• ES/SHM_USER_COUNT: the maximum number of users in all open sessions. • ES/SHM_SEG_COUNT: the maximum number of AIX system segments that ESSHM

uses. • ES/SHM_SEG_SIZE: size of the AIX system segments (in megabytes)

To properly implement the AIX alternative memory management following notes should be reviewed:

• OSS Note: 445533 (up to SAP 4.6D), • OSS Note: 789477 (as of 6.10).

In the specific situation that is primarily related to the large memory requirements based on the new AIX memory management, the memory management performance anomaly can occur. This is primarily related to the method that is used to map and disclaim the AIX memory pages used by



the SAP kernel. To improve the performance of the memory management the following variable should be included in the profile of the associated application server:

• ES/TABLE = SHM_SEGS • ES/SHM_SEGS_VERSION = 2

To get more details about the memory mgmt. process used in this case, please check the following notes:

• OSS Note: 856848, • OSS Note: 1088458.

9.6.3. AIX paging The standard tools (nmon, ptx, sar) can be used for monitoring paging. The page space page-ins and page-outs (paging to disk) are the important indicator to check. If you encounter AIX paging, a quick check would be to look at the virtual memory manager parameters. AIX memory pages are categorized as being computational or file pages. AIX will always try to utilize available physical memory (subject to vmo parameter settings). What is not required to support computational page demand will tend to be used for filesystem cache. In AIX 5L, a number of the default “vmo” settings in AIX 5.3 are not optimized for Oracle database workloads and should be modified. (The defaults have been changed in AIX 6.1 and should not require modification.) Check the settings for the following vmo parameters using the vmo –a command (as root user):

− minperm% = 3 − maxperm% = 90 − maxclient% = 90 − lru_file_repage = 0

Refer to SAP Note 973227 - AIX Virtual Memory Management: Tuning Recommendations. Note: In AIX 5.2 environments with large physical memory, set maxperm%, maxclient% such as not to exceed 24GB of memory. If you encounter AIX paging on a system with an SAP instance, and you have reviewed that the correct EM configuration is used, consider paging first as a symptom of an application problem, and approach it from the application statistics in SAP. Since SAP memory use can vary greatly, as programs allocate and free SAP memory, ABAP problems such as the slow internal table access described in Section 7.2 can contribute to paging problems. If a report processing many line items runs quickly, while it is running it will acquire more memory, and then release it. If the program runs too long because of inefficient ABAP coding, then the program will need the memory for longer than it should. When several programs do this, it can cause paging. If the programs were coded efficiently, then they would quickly finish, and this would reduce the likelihood that they would all be running simultaneously.


After having found the programs and users with large memory requirements, one can consider running the reports in batch on a server specially configured for large memory requirements, or the huge reports can be automated to run at night, when there may be less demand for memory. If the total SAP demand for memory is larger than available memory, one can reduce SAP memory quotas for EM and heap, but if the workload is unchanged, this will cause programs to go into PRIV mode (when EM is exhausted) or to abend when running out of heap. Check for running programs using large amounts of memory with ‘SM04 > goto > memory’:

Figure 77: SM04 > goto > memory

If a user is running several transactions, only one of the transactions will be displayed in SM04. Choose a user, and press “Sessions” to see the names of the transactions.

Figure 78: SM04 Sessions



One can then examine these transactions further to determine which use the memory. Use table TSTC to see transaction name and code. VA05 is “List of Sales Orders”.

ST03N has historical statistics on memory usage.

Figure 79: ST03N memory use statistics

9.6.4. Evaluating increasing memory for Oracle or SAP If you have evaluated SAP or Oracle memory use, and would like to increase the buffer sizes in Oracle or SAP, how do you determine whether there is memory available, or whether adding memory will cause paging? The free page information on commands such as vmstat does not always reflect how much memory is really available to be added to SAP or Oracle buffers. If vmtune shows free memory, then there are truly free memory pages that can be added to SAP or Oracle buffers. Check vmstat (or ST06) over the course of several days, and if it consistently shows free memory, then use the minimum free pages to determine the amount of memory that is available to add to Oracle or SAP. Do not try to allocate all free memory to SAP or Oracle.



Figure 80: ST06 displays free memory

There are situations, such as with a database server or NFS server where files are mapped to memory, where ST06 or AIX vmstat may show almost no free pages, but there is still a lot of memory that can be added to SAP or Oracle. Unless a file is opened for DIO (direct I/O) AIX maps file pages into available physical memory, in order to help performance. There are AIX parameters on the vmtune (AIX 5 vmo) command that set the limits of memory-mapped files. On a database server, first check that SAP note 78498 or 973227 (depending on SAP and AIX version) has been implemented to establish limits on real memory used for memory-mapped files.

Check the amount of real memory that is being used for memory-mapped files. This can be done with the command svmon –G.



Figure 81: svmon -G output

The total of AIX memory mapped pages is the sum of “pers” and “clnt” columns, for both “pin” and “inuse” rows. In this example, 234+1,585,658=1,585,892

Check the memory-mapped file page limits with the vmtune (vmo) command:

Figure 82: vmtune maxperm display

Note in Figure 82 that vmtune has been used to limit maxperm to 12%, or 473,836 pages. Figure 81 showed that actual perm storage in memory was 1,585,892 pages. When strict_maxperm is 0, AIX can expand perm storage use past the maxperm setting. 1,585,892 - 473,836 = approximately 1,000,000 pages (4GB) that could be potentially given to Oracle or SAP. Rather than having Oracle have a cache miss and do I/O that AIX fulfills from memory-mapped file cache, it is usually more efficient to give the memory to Oracle. Likewise, if additional SAP buffer memory can reduce database calls, then it is generally good to give the memory to SAP.

Since SAP and Oracle memory use can vary widely from hour to hour, run the svmon command periodically for several days or a week, to determine the minimum number of (persistent + client) pages over the period. This minimum (pers+clnt pin+inuse) should be compared with vmtune (vmo) maxperm, to determine if there is available memory.


Make changes gradually, and don’t over-allocate the memory in Oracle and SAP buffers, as that can cause AIX paging. Don’t add memory to Oracle or SAP just because AIX shows that there are available pages. If Oracle hit rates are low, or SAP buffers need to be increased to support the workload, then increasing the size of the Oracle and SAP buffers is reasonable.

10. Four guidelines for avoiding performance problems As seen in the examples above, there are a few general rules for avoiding performance problems.

10.1. Use the SAP data model If you’re evaluating the performance of custom code, and it runs slowly because the predicates don’t match the indexes on SAP tables, the odds are very good that the program is looking for the data in the



wrong place. Most SAP business documents (e.g. sales order, purchase order, delivery note, etc) can be found using the document number with a standard SAP table and standard SAP index. In addition to the example in Section 8.1.6, See SAP notes 185530, 187906, and 191492 for examples of incorrect and correct use of the SAP data model. The symptom of this problem is in the ST04 SQL cache - high buffer gets per exec and high buffer gets per row. This happens when the predicates on the SQL do not match the index columns on the table and Oracle has to read many blocks of data to retrieve the result.

10.2. Use array operations on the database If the program builds internal tables that contain keys for selects on other tables, evaluate whether an array operation such as FOR ALL ENTRIES can be used to perform array selects, rather than using LOOP AT with individual database calls. The symptom of this problem is seen in ST05 traces, where a program makes frequent calls to a table, and each call accesses few rows.

10.3. Check whether the database call can be avoided There are several versions of this problem:

• A table is set as buffered, but the application server generic or single record buffers are too small to hold the table rows. The cure for this is to increase the size of the SAP buffers.

• A table is not set to be buffered, but should be. In this case, the table is usually read-only, and the application can tolerate a small interval when the data is not in synch on all the application servers. In this case, the table attributes should be changed to buffered.

• A program is repeatedly fetching the same information from the database. This problem can be detected by using the ‘duplicate selects’ function in ST05. The program needs to be examined to see how it can be changed so that it does not have to repeatedly go back to the database for the same information

10.4. Write ABAP programs that are line-item scalable If the program will process many lines in a report:

• Use BINARY SEARCH on the “READ TABLE from itab” statements • Evaluate whether internal tables need to be defined as SORTED

The symptom of this problem is high CPU use for a program, where CPU use does not scale with additional line items – e.g. a 100 line report takes 1 second CPU, but a 1000 line item report takes more than 10 seconds CPU, and a 10,000 line report takes much more than 100 seconds CPU. These scalability problems get worse as the report lines increase.



11. Appendix 1: summary of performance monitoring tools A quick summary of key tools and their main functions in performance monitoring follows.

11.1. SAP

11.1.1. DB02 Transaction DB02 is used to display information about tables and indexes in the database, such as space usage trends, size of individual tables, etc.:

• Display all indexes defined on tables (DB02 > detail analysis) • Check column cardinality (DB02 > detail analysis > enter table name > select table >

press table columns)

11.1.2. SE11 Transaction SE11 is used to gather information about data dictionary and database definition of tables, indexes, and views:

• Display table columns and indexes (SE11 > enter table name > display > extras > database objects > check)

• Display secondary indexes defined in data dictionary (SE11 > enter table name > display > indexes). There may be data dictionary indexes that are not active on the database.

• Display view definitions

11.1.3. SE30 Transaction When STAT or ST03 shows that most of a program’s elapsed time is CPU, SE30 is used to investigate where CPU time is spent in an ABAP program

11.1.4. SM12 Transaction SM12 > extras > statistics can be used to view lock statistics:

• High percentages of rejects can point to a concurrency problem (multiple programs trying to enqueue the same SAP object) that may be solved via SAP settings such as “late exclusive material block” in the OMJI transaction. There are different SAP settings for different parts of the business processes, so these changes would be implemented with SAP functional experts.

• High counts of error can point to a problem where the enqueue table is too small. Compare “peak util” with “granule arguments” and “granule entries” to check for the table filling.

11.1.5. SM50 Transaction SM50 is an overview of activity on an SAP instance. If many processes are doing the same thing (e.g. access same table, ENQ, CPIC, etc), this can point to where further investigation is required.



11.1.6. SM51 Transaction SM51 can be used to check instance queues (goto > queue information)

• If there are queues for DIA, UPD, UP2, etc, there will be “wait for work process” in STAT and ST03.

• If there are queues for ENQ, there is a problem with enqueue performance.

11.1.7. SM66 Transaction Gives an overview of running programs on an SAP system. If many processes are doing the same thing (e.g. access same table, ENQ, CPIC, etc), this can point to where further investigation is required.

11.1.8. STAT Transaction Displays STAT records for a single SAP instance.

11.1.9. STAD Transaction Is used to displays STAT records for an interval from all instances on an SAP system.

11.1.10. ST02 Transaction ST02 is used to monitor the activity in SAP managed buffer areas, such as program buffer, generic buffer, roll, and EM.

11.1.11. ST03N Transaction ST03 is not a tool for solving performance problems. It is a tool that is mainly useful for tracking historical activity. One can monitor average response times for individual transactions and for the system as a whole, and get counts of dialog steps to use for trend analysis.

There are a few limited ways that it might be used in performance monitoring: • As a filter for inefficient programs. Use the ST03 “transaction” profile, sort the list by elapsed

time, and look for transactions which use very little CPU relative to elapsed time, e.g. 10% or less of elapsed time is CPU on the application server. These may have problems such as inefficient database access, slow RFC calls, etc.

• As a filter for problems that occur at a certain time of the day. Run ST03, and select “dialog” process display. Use the ST03 “times” profile, press the right arrow to go to the screen that displays average direct read, sequential read, and change times. Look for hours of the day when the average time goes up. This could point to a time when there is an I/O constraint, or CPU constraint on the DB server.

• Use as a filter for database performance problems, in very limited circumstances. If average “sequential read” times are over 10 ms for dialog, and commit time is over 25-30 ms, there may be some sort of database performance problem. Check SQL cache with ST04, look for I/O constraints and other database problems.



11.1.12. ST04 Transaction ST04 has many functions, the most important for performance are: viewing the SQL cache, checking Oracle delays using v$system_event, and monitoring database buffer activity and monitoring Oracle shadow processes.

11.1.13. ST05 Transaction ST05 is one of the most important tools for SAP performance, among its functions are: • Trace calls to database server to check for inefficient SQL when program is known. • Compress and save SQL traces for regression testing and comparisons. • Trace RFC, enqueue, and locally buffered table calls.

11.1.14. ST06 Transaction Display AIX-level stats for the application server – paging, CPU usage, and disk activity.

11.1.15. ST10 Transaction ST10 is used to monitor table activity, and table buffering in SAP on the application server: • Check for tables that are candidates for buffering in SAP • Check for incorrectly buffered tables

11.1.16. RSINCL00 Program Expand ABAP source and include files, with cross-reference of table accesses. This is useful when examining ABAP source, as it gives an overview of the whole program.

11.1.17. SQLR Transaction Merge an ST05 trace with STAT records, to determine which dialog step executed which statements. This is useful when tracing a transaction made up of many dialog steps, to join the trace to the dialog step that issued the problematic SQL. Depending on the SAP version, this may be available as a transaction SQLR, program /SQLR/0001, or program SQLR0001.

11.1.18. RSTRC000 Program Lock a user mode into a work process. This is useful for doing traces (such as OS level or Oracle level traces) where one needs to know the PID to establish the trace. Once the user is locked into a work process, one can determine the PID or the work process and Oracle shadow process, and use them to establish the trace.

11.2. AIX

11.2.1. nmon Can be used to record and report many different AIX indicators. It is very useful for tracking and trending CPU, I/O activity, or paging. The I/O activity is reported as rates, not rate and response time, so one cannot use it to determine I/O delay. Oracle statistics should be used for I/O delays.



11.2.2. ptx Can be used to record and report many different AIX indicators. It is very useful for tracking and trending CPU, I/O activity, or paging activity. The I/O activity is reported as rates, not rate and response time, so one cannot use it to determine I/O delay. Oracle statistics should be used for I/O delays.

11.2.3. iostat Shows I/O rates, but not response times. No easy way to record and play back. Not too useful in diagnosing problems.

11.2.4. vmstat Shows CPU use, and paging activity. No easy way to record and play back. Some caveats on vmstat:

• fre is not really free memory at an application level, since AIX maps file pages into memory, and will tend to fill real memory with anything that has been touched by a program. If you are checking for available memory on a DB or application server, in order to increase the size of Oracle or SAP buffers, see the process in section 9.6.4.

• In the page section, pi is most important, the rest of the page indicators are much less important. Pi and po are real page activity to disk. Page-ins cause delay to a program that is waiting on the page fault. Page-outs may just be AIX moving infrequently referenced pages out to disk.

11.3. Oracle The most important Oracle performance tool, SQL cache analysis, is in SAP ST04 transaction.

11.3.1. STATSPACK STATSPACK can be used to monitor trend information, such as I/O activity, Oracle delays (the v$system_event view), and CPU use. Since it cannot link SQL statements to their SAP program source line, as can be done with ST04 SQL cache, the STATSPACK list of high impact SQL is of limited use compared to ST04 SQL cache. STATSPACK shows the program name, but one then has to search for the statement in the program. Statspack setup is described in Oracle metalink DOCID 149124.1.

11.3.2. Trace and tkprof Since Oracle does not show all delay statistics (the v$system_event waits) in the SQL cache, one needs to use oracle trace to evaluate the causes of delay for a specific SQL statement. This is seldom needed, since locking problems can be seen at runtime with SM50 and SM66, and I/O counts and statement hit-rate can be seen with ST04 SQL cache. Lock the user running the test in an SAP work process as discussed in section 11.1.18, then trace the PID of the user’s oracle process and then tkprof the trace.



SAP note 654176 has information on using TKPROF. Oracle metalink also describes the process for using trace and tkprof in DOCID 142898.1.

11.3.3. Automatic Workload Repository Oracle 10g collects statistics for the system, sessions, and individual SQL statements and stores these statistics in the Automatic Workload Repository (AWR). By default, AWR snapshots are taken every 60 minutes and retained for 7 days. AWR snapshots may also be generated manually. The AWR statistical information is used for self tuning and problem determination by Oracle’s performance advisors (SQL Tuning Advisor, SQL Access Advisor) and Automatic Database Diagnostic Monitor (ADDM). AWR provides a comprehensive report (awrrpt) similar to STATSPACK, with additional information not available in STATSPACK. To generate an AWR reports, run the following script and select two snapshots covering the period of interest:

$ORACLE_HOME/rdbms/admin/awrrpt.sql Note that awrrpt requires Diagnostic Pack license. STATSPACK does not require additional license and is available and supported in Oracle 10g. For further information, refer to Oracle MetaLink Doc ID 276103.1, Performance Tuning using 10g Advisors and Manageability Features.



12. Appendix 2: Reference Materials

12.1. Oracle Manuals • 9i Database Performance Tuning Guide and Reference – A96533-02

12.2. IBM manuals • AIX 5L Performance Tools Handbook – SG24-6039



Figure 1: STAD transaction display of time components 8 Figure 2: ST03N VA05 9 Figure 3: ST03N VA05 top time 10 Figure 4: SE30 TMP variant selected 11 Figure 5: SE30 gather stats on internal tables 12 Figure 6: SE30 Aggregation level 13 Figure 7: SE30 start/end measurement 14 Figure 8: SE30 active 14 Figure 9: SE30 analysis 15 Figure 10: SE30 sorted by Net time 16 Figure 11: SE30 source code 17 Figure 12: SE30 program attributes 17 Figure 13: SE30 tips and tricks 18 Figure 14: SE30 binary search 19 Figure 15: LOOP with select 20 Figure 16: MIRO ST03N times 22 Figure 17: MIRO ST03N Database 23 Figure 18: MIRO ST05 trace 24 Figure 19: MIRO ABAP display 25 Figure 20: MIGO program attributes 26 Figure 21: MIRO RBKP table scan 26 Figure 22: RBKP has three indexes 27 Figure 23: RBKP DDIC has 5 secondary indexes 28 Figure 24: RBKP index in data dictionary 29 Figure 25: ZMM program - lots of fast DB accesses 30 Figure 26: ZMM ST05 31 Figure 27: ST05 summarized trace 31 Figure 28: ZMM program compressed trace 32 Figure 29: ZMM program source 33 Figure 30: ST04 display of SQL cache for Oracle 10 34 Figure 31: ST04 SQL cache - big problem example 35 Figure 32: ST04 SQL cache - moderately inefficient SQL 35 Figure 33: Sample explain 37 Figure 34: Display indexes from explain 38 Figure 35: Note 185530 - VBRP where VGBEL 39 Figure 36: Custom table SQL 40 Figure 37: Custom table explain 40 Figure 38: Custom program for custom table 41 Figure 39: Custom table display indexes 41 Figure 40: VBRP high impact statement 42 Figure 41: VBRP explain 43 Figure 42: VBRP table and index 44 Figure 43: Program accessing VBRP 44 Figure 44: Program accessing VBRP not from SAP 45



Figure 45: SAP note with examples of using the SD data model 45 Figure 46: ST04 SQL cache for LTAP/LTAK example 46 Figure 47: LTAP LTAK explain 47 Figure 48: LTAP indexes 48 Figure 49: LTAP count PQUIT 49 Figure 50: LTAP LTAK source 49 Figure 51: LTAP LTAK SAP note 50 Figure 52: A616 in SQL cache 51 Figure 53: A616 explain 52 Figure 54: A616 index 53 Figure 55: A616 ABAP source 54 Figure 56: A616 technical settings 55 Figure 57: ST02 buffers with swaps 56 Figure 58: ST02 buffered objects 56 Figure 59: ZMR program retrieves 14 million rows 57 Figure 60: Explain ZEN table 58 Figure 61: ZMR program source 58 Figure 62: D010INC high bgets/Exec 59 Figure 63:D010INC explain 60 Figure 64:D010INC indexes 60 Figure 65: ST03N by DB time 62 Figure 66: ST03N by DB time database tab 62 Figure 67: ST06N with AIX virtualization support 64 Figure 68: ST04 file system requests with good response times 66 Figure 69: ST04 file system requests 67 Figure 70: ST04 file system requests slow on high write files 68 Figure 71: ST04 database overview 70 Figure 72: ST04 - Oracle v$system_event delay statistics 71 Figure 73: ST04 Oracle sessions 71 Figure 74: ST02 swaps 72 Figure 75: ST03N overview 72 Figure 76: ST10 not buffered tables 73 Figure 77: SM04 > goto > memory 77 Figure 78: SM04 Sessions 77 Figure 79: ST03N memory use statistics 78 Figure 80: ST06 displays free memory 79 Figure 81: svmon -G output 80 Figure 82: vmtune maxperm display 80

tuning sap oracle aix v.1.2

Documents