ims performance - general analysis
TRANSCRIPT
Richard [email protected] PerformanceApril 25th, 2016
IMS Performance: General Analysis
Agenda
Performance Evaluation Process
Resource Measurement Facility for z/OS (RMF)
IMS Performance Analyzer for z/OS (PA)
2
Performance Evaluation Process
Performance Evaluation Process
§ External Throughput Rate (ETR)
§ Internal Throughput Rate (ITR)– A measure of CPU efficiency or path length– ITR = Transactions per second / CPU % Busy
§ Transaction rate projection when CPU is 100% busy
§ Higher ITR is GOOD and indicates– workload is more efficient– Overall service time per transaction is lower
4
Stress/Measure
Analysis
Create/Customize Environment
IMS PerformanceEvaluation Cycle
Performance Evaluations
§ New IMS versions– New functions, enhancements
§ Languages– COBOL, Java
§ z Systems– z13, zEC12
§ DASD– DS8870, DS8800
§ z/OS Operating Systems– z/OS 2.2, 2.1
5
Current Hardware Environment
§ Processors– IBM System zSystems z13 Model 2964-7B1 (NE1)– IBM System zEnterprise EC12TM Model 2827-785 (HA1)
§ Storage– IBM System Storage® DS8000® series latest model:
§ DS8870 (16 FICON® channel paths) § DS8800 (8 FICON® channel paths)
§ Coupling Facilities– Internal Coupling Facilities (ICFs) with Coupling Facility
Control Code (CFCC) level 20.
6
IBM z13 Model 2964-7B1 (NE1)
IBM zEC12 Model 2827-785 (HA1)
IMS Performance Measurement Procedure
1. Start environment
2. Start workload and ramp up to steady state
3. Clear SMF datasets
4. Start RMF I with an interval set
5. Start RMF III with the same interval as RMF I
6. Wait for one or two RMF intervals
7. Switch SMF dataset
8. IMS - /SWITCH OLDS
9. IMS - /CHECKPOINT STATISTICS
10. IMS - /TRACE SET ON MONITOR ALL
11. Wait for measurement interval
12. IMS - /TRACE SET OFF MONITOR ALL
13. IMS - /CHECKPOINT STATISTICS
14. IMS - /SWITCH OLDS
15. Switch SMF dataset
16. Dump SMF dataset
17. Run RMF post processor report
18. Run IMS PA against OLDS used between the IMS /SWITCH OLDS period
19. Run IMS PA against MONITOR dataset
7
Resource Measurement Facility for z/OS (RMF)
What is RMF?
§ Performance management tool that measures system activity and generates reports based on data collected in System Management Facility (SMF) records
§ Provides performance analysis and problem determination by reporting:– LPAR CPU busy %– DASD device activity– Storage summary
§ RMF data gatherers:– Monitor I (historical)– Monitor II (snap shot monitoring)– Monitor III (sysplex analysis)
§ RMF Postprocessor: reporting tool for SMF records
9
RMF Address Spaces
10
RMF Control
S RMF
Monitor I (zz session)
F RMF,S ZZ
RMF Monitor III Gatherer
F RMF,S III
SYS1.PROCLIB(RMF) SYS1.PROCLIB(RMFGAT)
SYS1.PARMLIB(ERBRMF00)
SYS1.PARMLIB(ERBRMF04)
Monitor I and III Data Gathering
11
1. MeasurementsCACHE SMF 74.5CHANNEL SMF 73CPU SMF 70.1CRYPTO SMF 70.2DEVICE SMF 74.1ENQ SMF 77IOQ SMF 78.3FCD SMF 74.7ESS SMF 74.8PAGESP SMF 75PAGING SMF 71TRACE SMF 76VSTOR SMF 78.2WKLD SMF 72.3IRLM SMF 79.152. TimingCYCLE(1000)NOSTOPSYNC(SMF)3. Reporting / RecordingRECORDREPORT(REALTIME)SYSOUT(A)4. User ExitsNOEXITS
Report Classes
§ Report classes are split by main components:– IMS Control region– IMS DL/I region– IMS DBRC region– IMS Dependent regions (e.g. MPP, IFP, BMP, JMP JBP, etc.)– IMS Connect– DB2 Master– DB2 DDF– IRLM– …
12
RMF Monitor I Overview
RMF Monitor I Parameters
§ Start RMF Monitor I using command: S RMF,,,MEMBER(xx)
14
SYS1.PARMLIB(ERBRMFxx)
RMF Monitor I CPU Activity Report
§ Number of processors
§ LPAR busy % (Total/Average), used to calculate ITR– ITR = ETR / LPAR busy %
§ Machine type and model – z13 Model 2964-7B1 (NE1)
15
RMF Monitor I Partition Data Report
§ DED – processors are dedicated to this partition (for repeatability)
§ DMX – processors are mixed dedicated and shared
16
RMF Monitor I Channel Path Activity Report
§ To display channel path types, use command D M=CHP
§ zHPF (z high performance FICON)
17
RMF Monitor I DASD Activity Report
§ AVG RESP TIME – Response time (ms): sum of IOSQ, PEND, DISC and CONN time. If RESP Time is high, look into these components
§ AVG PEND TIME – Average time an I/O request is waiting for hardware
§ Monitor the volumes on which IMS OLDS and WADS are defined to check their response times
18
RMF Monitor I Virtual Storage Activity Report
§ Most volatile is SQA and CSA both above and below the 16MB line
§ System CSA Usage for the Period
§ Key 7 IMS CSA Usage for the Period
19
RMF Monitor III Overview
RMF Monitor III Parameters
§ Start RMF monitor III via modify command: F RMF,START III, options… – e.g. F RMF,START III, NOOPTIONS, CYCLE(250), MINTIME(60), SYNC(1M),
CFDETAIL, NOSTOP
§ Start with member parameter:– F RMF,START,MEMBER(xx)
§ CFDETAIL - Coupling Facility activity data will be collected
§ SYNC(xM) – Synchronizes the report to a specified minute within the hour
21
SYS1.PARMLIB(ERBRMF04)
RMF Monitor III Parameters
§ The RMF Post Processor is used to generate the RMF Workload Activity Report from the SMF log data sets
22
RMF Monitor III Summary Report
§ RMF Post Processor control statement - SUMMARY(INT,TOT)
23
RMF Monitor III Workload Activity Report
§ IMS Transaction Rate per second (ETR) – END / S
§ Service Unit (SU) – represents unit of work
§ Service Time (ST) – processor CPU time
§ SU / transaction – (SU/ sec) / ETR
§ ST / transaction – ST / ENDED
§ Set PROJECTCPU = YES in IEAOPTxx member to see how much processing can be offloaded onto zIIP or zAAP processors
24
RMF Monitor III Serialization Delay Report
§ Helps analyzing serialization related performance problems, provides contention information on system and address space level
§ REPORT(SDELAY): This report is only available in XML output format. You will need to specify XPRPTS DD name in postprocessor job
§ Generates both serialization delay summary and details. Summary provides information on system locks showing avg. contention time and GRS latch/enqueue statistics
25
RMF Monitor III Serialization Delay Report
§ Serialization delay details: provides information on lock details
§ For each lock type, shows max of 20 address spaces with the longest contention time reported
26
RMF Monitor III Coupling Facility Activity Report: Usage summary
27
RMF Monitor III Coupling Facility Activity Report: Usage summary
28
RMF Monitor III Coupling Facility Activity Report:Structure Activity§ Detailed information on each structure broken out by system and total
§ CHNGD and DELAYED REQUESTS, it could indicate shortage of subchannels
29
RMF Monitor III Coupling Facility Activity Report:Subchannel Activity§ Shows number and type of links
§ PTH BUSY: Number of CF requests rejected from this system because all paths were busy
30
IMS Performance Analyzer for z/OS (IMS PA)
What is IMS PA?
§ Key performance tool used to generate resource utilization and performance reports that can be used to analyze the performance of IMS systems.
§ We generally produce IMS PA log reports against the IMS log data sets, however IMS Monitor, IMS Connect Event Data, and OMEGAMON data and can be used as input to produce reports.
32
IMS PA Reports
§ Specific Reports– Transit Time Analysis– CPU Usage Report– DLI Call Statistics– Scheduling Statistics– Latch Statistics– Logger Statistics– Summary Report
33
IMS PA JCL for Batch Processing
§ Sample IMS PA JCL to generate a list of reports using IMS OLDS as input
§ IMS PA accepts IMS Monitor, IMS Connect Event Data, OMEGAMON ATF and TRF
34
IMS PA Log Information Report
§ Log information report shows statistics for various IMS log records
§ IMS x3730 record shows the transaction rate per sec (1,979 TPS)
§ IMS x5050 record shows the number of database updates
35
IMS PA DL/I Call Statistics
§ Similar information for transaction rate (ETR) can be found in DL/I statistics report under Transactions process by above Programs
§ A large value for Programs that reached Termination indicates no program reuse
36
IMS PA: Analyzing Full Function Transaction Response Time
End User vs. Host Response Time
§ End User Response Time
§ Host Response Time
38
Transit Time Analysis Report
§ The Total column under the Average Transit time category shows the average host response time
§ Transit Time Analysis By Transaction Code Report:
39
Transaction Flow
40
Input Queue Time
§ Time spent in the input queue or on the message queue buffers waiting for message region
§ Major part of response time in busy systems
41
Input Queue Time Tuning Considerations
§ Time waiting for scheduling
§ Processing time of scheduling
42
Analyzing Input Queue Time
§ Time waiting for scheduling:– Are there enough dependent regions to process the arriving transactions?
§ IMS PA: Internal Resource Usage, Dispatcher Statistics– Are the parameters concerning transaction scheduling properly tuned?
§ TRANSACT macro parameters§ EXEC parameters of MPP regions
§ Processing time of scheduling:– I/O to the message queue data set
§ IMS PA: Internal Resource Usage - Message Queue Pool Statistics– I/O to the ACBLIB data set
§ IMS PA: Total System IWAIT Detail – Check for scheduling failures, some useful commands
§ /DISPLAY commands§ IMS PA: Internal Resource Usage – Scheduling Statistics
43
Time Waiting for Scheduling
§ Are there enough dependent regions to process the arriving transactions?– What is the arriving transaction rate?– What is the average transaction processing time?– How many regions are available for scheduling?
44
Time Waiting for Scheduling
§ Are the parameters concerning transaction scheduling properly tuned?– TRANSACT macro parameters:
§ MSGTYPE= (CLASS=) Are there a sufficient number of regions defined with the proper transaction class?
§ PRTY= Used to determine scheduling priority of this transaction.§ PROCLIM= Number of messages of the transaction code a program can process in
a single scheduling.§ PARLIM= Threshold value of queued transactions used to schedule a new region.
– EXEC parameters of MPP regions:§ CLn=
45
Processing Time of Scheduling
§ I/O to the message queue data set– IMS PA: Internal Resource Usage - Message Queue Pool Statistics
§ High value of “Locate calls satisfied in the Pool” indicates Message Queue Pool is satisfying most requests without the need for I/O
46
Processing Time of Scheduling
§ I/O to the ACBLIB data set– IMS PA: Total System IWAIT Detail ACBLIB block loading IWAITs and ACBLIB
miscellaneous IWAITs.
§ Check for scheduling failures, some useful commands:– /DISPLAY PROGRAM – /DISPLAY TRANSACTION – /DISPLAY POOL (PSBW/DMBP/PSBP/EPCB) – /DISPLAY ACTIVE REGION
§ System definition considerations:– APPLCTL macro parameter, check SCHDTYPE=– DBPCB parameter, PROCOPT=
47
Processing Time of Scheduling
§ To check for scheduling failures, use the following reports:– Scheduling Statistics– Intent Failure Summary – Pool Space Failure Summary
48
Processing Time Events
§ Preparation of PCB
§ Initialization of the application program
§ Issuing the message prime GU call
§ Updating the database (if any)
§ Issuing the ISRT IOPCB
§ Sync point processing (The program updates the database physically)
§ Releasing the scheduled resources
49
Processing Time
§ This is a time from the first GU IOPCB call issued by the application to the completion of program processing after the syncpoint
§ The PGM EXEC column under Avg. Transit Time shows the average processing time per transaction code
50
Processing Time
§ Processing time of scheduling to first DL/I call
§ Application processing time
§ The Scd-DLI column of this report shows the average scheduling to first DL/I call time of each transaction code.
§ The Elapsed column of this report shows the average processing time of each PSB and transaction code.
51
Time to Schedule First DL/I Call
§ Processing time of scheduling to first DL/I call tuning considerations
§ Reducing the program loading time by:– Preloading the high use application and language modules
§ Reducing the I/O to PGMLIB– Using the LLA function of z/OS to reduce directory I/O– Using the VLF function of z/OS to reduce module I/O
§ Reconsider the concatenation sequence of STEPLIB on MPP execution JCL to optimize searching
§ Checking the usage of DASD containing the PGMLIB to optimize I/O time
52
Application Processing Time
§ Application processing time tuning considerations
§ Reducing the I/O IWAIT elapsed time by:– Reducing each I/O IWAIT elapsed time
§ Check the accessing rate to the DASD containing the databases and change the allocation if needed
§ If the I/O rate to the message queue data set is high, consider increasing the size of the message queue buffer (QBUF).
– Reducing the number of I/O IWAIT per CALL§ Reorganize the database to improve space utilization.§ Check the DBD and change the CI size or other parameters if needed.§ Check the logic of the randomizer if the database is HDAM or PHDAM.§ Increase the database buffers because the buffer shortage results in the re-reading
of the CIs or blocks which have already been read in the previous calls.
53
Output Queue Time
§ The time from the program termination to the completion of sending a message
§ The output column under Avg. Transit Time report shows the average output queue time for each transaction code
§ The output queue time should be close to 0 if the QPOOL is optimized
54
Summary Report
§ Summary Report for Full Function workload
§ Same fields as in the transit analysis report can be requested (Input queue, process time, etc.) but many other values are shown such as schedule time, syncpoint time etc.
§ Helps narrow down long input queue and processing times
55
IMS PA: Analyzing Fast Path Transaction Response Time
Transaction Flow
§ Transit time is the time spent before the scheduling of the FP router to the output message
57
Total Transit Time
§ The Total column under the Average Transit time category shows the average transit time
§ Fast Path (EMH) Transit Time Analysis By Transaction Code Report:
58
Input Queue Time
§ The time from scheduling the fast path input edit/routing exit routine to issuing of the GU call by the application program
§ The Input Queue column under the Average Transit Time category shows the average input queue time for each transaction code
59
Region Occupancy
§ Fast Path transactions are processed as wait for input, input queue time is equal to the time waiting for an IFP region
– Are there enough dependent regions to process the arriving transactions?§ How high is the transaction arriving rate?§ How long is the average transaction processing time?§ How many IFPs are available for scheduling?
– IMS PA IFP Region Occupancy report shows the region utilizations. High region utilization may cause long input queue times.
60
Processing Time
§ The time from GU call of the application program to the sync point
§ Database is updated, if the program is doing so, and sync point processing is done
§ The Pgm Exec column under the Average Transit Time category shows the average program execution time for each transaction code
61
Processing Time
§ Program execution time is mainly the time to access the database, pick the transactions to be analyzed
§ FP Resource Usage and Contention report shows DEDB calls, ADS I/O calls, common buffer usage and contention
§ Analyze transactions which access the ADS more than the number of DEDB calls.– (DEDB Calls Reads Avg + DEDB Calls Updates Avg < ADS I/O Reads Avg)
§ This indicates the situation when the segment is contained in the overflow area (dependent or independent) or the database has long synonyms chains. The reorganization of the database accessed by the transaction can improve the processing time.
62
Processing Time
§ Transactions which includes the process of CI contention. (Contentions CI/Sec > 0)– This indicates when the segment is contained in the overflow area (dependent or
independent). Or the multiple programs which access the same root segment can run simultaneously. The reorganization of the database accessed by the transaction can improve the processing time.
§ Transactions which includes the process of buffer steal. (Common Buffer Usage Stl > 0)– This indicates the invocation of buffer steal caused by the shortage of NBA buffer.
Recommendation is to increase the number of NBA.
§ Transactions which includes the process of buffer wait. (Common Buffer Usage Wts > 0)– This indicates the invocation of buffer steal caused by the shortage of whole Fast Path
buffers. Recommendation is to increase the allocation of DBFX.
§ Transactions which includes the process of buffer wait. (Contentions Tot OBA > 0)– This indicates the invocation of OBA wait caused by the shortage of NBA.
Recommendation is to increase the allocation of NBA.
63
Output Queue Time
§ Time from syncpoint to entry to the output buffer
§ The Output Queue column under the Average Transit Time category shows the average output queue time per each transaction code.
§ Output queue time is mainly the time needed to write the logs to disk, so it depends on the disk where the WADS and OLDS are located.
64
Output Queue Time
§ IMS Logging Rate = BLKSIZE (bytes) * OLDS Writes initiated per sec
§ Average WADS/OLDS I/O write times
65
Summary
§ Performance Evaluation Process
§ RMF– http://www-03.ibm.com/systems/z/os/zos/features/rmf/
§ IMS PA– http://www.ibm.com/support/knowledgecenter/SSAVHQ_4.4.0/ipi-product_landing.dita– http://www.redbooks.ibm.com/abstracts/sg248071.html?Open
66