ibm i performance tools for application developers

© 2018 Dawn May Consulting, LLCUser Groups November 2018

IBM i Performance Tools for Application Developers

Dawn May [email protected] @DawnMayiCan

!1

mailto:[email protected]


Check PTFs Before Collecting Performance Data

• Some PTFs are required before certain performance data can be collected

• The following support documents should be checked for PTFs for performance data:

• For 7.3

• Job Watcher, Performance Explorer (PEX), and Disk Watcher PTFs for IBM i 7.3

• For 7.2

• Job Watcher, Performance Explorer (PEX), and Disk Watcher PTFs for IBM i 7.2

!2

http://public.dhe.ibm.com/services/us/igsc/idoctor/html/downloadsV7R3.html

http://www-01.ibm.com/support/docview.wss?uid=nas8N1020091


Performance Instrumentation and Data Collection

• The Advantage

• IBM develops the software stack, top to bottom

• Instruments the software with component-specific performance metrics

• IBM develops the performance data collectors that harvest those performance metrics

• IBM i has an integrated database – Db2 - this is a BIG DEAL

• Performance data is stored in the database automatically

• No “add on” application is necessary – it’s all in the Operating System

• IBM provides the graphical analysis tools

• Analysis of the performance data in the DB2 files using SQL

#1

!3

IBM i has the best performance instrumentation and data collection capabilities in the industry!


Introduction to Wait Accounting

!4

© 2018 Dawn May Consulting, LLCCOMMON Performance Management Webinar

“All computers wait at the same speed.”

!5


What is Wait Accounting?

• Wait Accounting is the ability to determine what a job is doing when it is not using the CPU i Exclusive! Patented technology built into IBM i.

Stop!

https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_73/rzahx/rzahxbasicwaitaccounting.htm

!6




Wait Accounting Overview

• When a job is not running (i.e., not using CPU), it is waiting

• But what is it waiting for?

• Waits may be normal

• Waits may not be normal

• Wait Accounting helps to determine what the wait is and if it is a problem

• IBM i has instrumented most of the key wait conditions

• Wait information is automatically collected by Collection Services and Job Watcher

!7


Wait States

• A job, thread, or task is either running or waiting

• Wait information is tracked for every job, thread and task on the system

• There are three wait states:

• CPU Queued

• Ready to use processor, but waiting for a processor to become available

• Idle

• Normal condition, waiting for work

• Blocked

• Waiting for some resource

• Typically the most interesting waits to analyze

!8


Types of Waits

• Idle waits

• Typically a normal wait condition; for example:

• Waiting for the “Enter” key to be pressed on a 5250 display session

• Waiting for data from the network

• Blocked waits

• Waits that occur while performing a work request

• Serialized access to a shared resource

• Contention is an example

• Blocked waits are are what we want to look at more closely

!9


• Blocked waits are expected

• A record lock may be expected …. but one that lasts for a very long time is unexpected

• If waits can be reduced or eliminated, CPU can be used more efficiently

• One wait may be reduced or eliminated, only to have another wait surface

• Impossible to remove all wait times

• When is a wait bad?

• Is there a business impact? Are users complaining?

• “It depends” but jobs that wait more than 25% of their total run time may need additional analysis

Additional Consideration about Blocked Waits

!10

… … …


Wait Accounting - Buckets

• Wait Buckets

• “Wait groups” instrumented in the operating system

• 32 buckets

• Collected by Collection Services and Job Watcher

• Waits can be analyzed to determine where a job is spending its time

• Waits can be viewed at a system-level or at an individual job/thread/task level

• Can also be grouped by generic job name, subsystem, user profile, pool ID, etc…

!11


Wait Buckets

• The system keeps track of wait information automatically

• Wait information for every job/thread/task is tracked

• A wait bucket is a pair:

• Count: The number of times this wait has been encountered

• Time: The accumulated amount of time spent in the wait

• In addition to the accumulated count/wait pair, interval details are also tracked

• Current wait

• Time in the current wait

!12


32 Wait Buckets

1. Time dispatched on a CPU 2. CPU queuing 3. Reserved 4. Other waits 5. Disk page faults 6. Disk non-fault reads 7. Disk space usage contention 8. Disk operation start contention 9. Disk writes 10. Disk other 11. Journaling 12. Semaphore contention 13. Mutex contention 14. Machine level gate serialization 15. Seize contention 16. Database record lock contention

17. Object lock contention 18. Ineligible waits 19. Main storage pool contention 20. Journal save while active 21. Reserved 22. Reserved 23. Reserved 24. Socket transmits 25. Socket receives 26. Socket other 27. IFS 28. PASE 29. Data queue receives 30. Idle/waiting for work 31. Synchronization Token contention 32. Abnormal contention

!13


Wait Buckets

!14

Help!!Abnormal

Contention?

Semaphore

Contention?

Disk Space

Usage

Contention?

Seize Contention?

CPU

Queueing?


Wait Accounting Documentation

• IBM i Wait AccountingGeared toward average user http://www.ibm.com/developerworks/ibmi/library/i-ibmi-wait-accounting/

• Job Waits White Paper Geared toward the performance expert http://public.dhe.ibm.com/services/us/igsc/idoctor/Job_Waits_White_Paper.pdf

!15

http://www.ibm.com/developerworks/ibmi/library/i-ibmi-wait-accounting/

http://public.dhe.ibm.com/services/us/igsc/idoctor/Job_Waits_White_Paper.pdf


Common Waits that Applications Use

• Disk Waits

• Semaphores, Mutexes, Synchronization Tokens

• Journaling

• Database record locks

• Object locks

• Sockets

!16


Example of Waits that make up a Bucket

• Database record lock contention has several different enumerated wait types:

• Read

• Update

• Weak

• Transfer

• Check

• Conflict exit

!17


Holders, Waiters and Call Stacks

• IBM i keeps track of who is holding a resource, and if applicable, who is waiting to access that resource

• A Holder is the job/thread/task that is holding the serialized resource

• A Waiter is the job/thread/task that wants to access the serialized resource

• IBM i maintains call stacks for every job/thread/task

• The combination of

• Who - holders and waiters … who has it? who wants it?

• What – object being waited on

• When - interval time

• Where and How - call stacks

Delivers a powerful solution for analyzing wait conditions

!18


Why Use Wait Accounting?

• Identify what the job (application) is doing when not using the CPU

• Determine where to focus your effort and investment

• Is there a contention point? On what resource?

• CPU, memory, I/O, other wait types?

• Focus effort where collected data demonstrates wait time

• Identify bottlenecks that prevent scaling

• Use wait analysis when testing applications

• Gain insight into potential performance issues before rolling out application changes

!19


Wait Accounting – “Run-wait” signature

• Wait accounting identifies the time the thread/task was running vs the time the thread/task was waiting

• Consider the following:

CPU 140 min

CPU queuing 50 min

Wait 170 min

Elapsed time 6 hours (360 mins)

Run-wait signature

!20


You can ask questions for each type of wait, such as: • Disk Page Faults Time:

• Are my pool sizes appropriate? • What objects are being faulted into memory?

• Disk Writes Time: • Is the write cache being overrun? • Is the application doing writes synchronously?

• Journal Time: • Are all the journals optimally configured? • Are unnecessary objects being journaled?

• Object Lock Time: • Am I locking records or objects unnecessarily?

Wait Accounting – “Run-wait” Signature

Wait time - 170 minutes

Disk Page Faults 100 mins

DB Record

Locking 15 mins

Journal 15 mins

Disk Writes 30 mins

Object Lock

10 mins

!21


Metrics for Waits

• IBM i collects the count and time spent in each wait bucket

• Measured, not sampled

Disk Reads Disk Writes Record Locks Journal

Total Count 3523 17,772 355 5741

Total Time 42 seconds 73 seconds 45 seconds 44 seconds

Average Time per Wait

0.012 seconds 0.004 seconds 0.126 seconds 0.008 seconds

!22


Work with System Activity - Wait Detail

• You can display wait times with WRKSYSACT

• F11 (3 times) to display total wait time

• Option 6 becomes available to display the wait detail

!23

Wait Detail COMMON1 10/08/18 09:37:45 Job or Task: QP0ZSPWP User: QSECOFR Number: 622745 Wait Wait Category Pct Time dispatched on a CPU .2 Disk writes .2 Socket other 947.9 PASE .4 Idle / waiting for work 1.3

Type options, press Enter. 1=Monitor job 5=Work with job 6=Wait detail Job or --------Storage-------- Total Opt Task User Number Thread Allocated Deallocated Wait QP0ZSPWP QSECOFR 653467 000002AC 1935 338 .0 DAWNMAYA1 DAWNM 653283 000003F6 292 0 98.6 QP0ZSPWP QSECOFR 622745 00000003 34 0 950.0 QUSRWRK QSYS 621915 00000001 0 0 903.0 SMIORESTSK 0 0 999.9


Collecting Wait Data

• Collection Services - always on

• Collects data automatically at regular intervals

• Typically 5 or 15 minute intervals

• System and job level data

• Starting point

• Job Watcher

• Needs to be started/stopped

• Typically 5 or 10 second intervals

• Additional detailed data such as call stacks, object waited on, holder

• Frequently needed to solve performance issues

!24


Why Should Developers Use Wait Accounting?

• Wait accounting can:

• Help you understand application characteristics

• Is it CPU bound? I/O bound?

• Help you to understand where to focus your effort and investment

• Is there a bottleneck on CPU, Memory, I/O, Contention time?

• Invest resources where greatest benefit will be

• Offer insight into potential performance issues before end-users are affected

• Leverage wait accounting data during test

• Eliminate surprises

• Identify bottlenecks that prevent scaling

• Provide valuable clues to help analyze performance issues as they arise

!25


Performance Data

• The performance data collection capability is built into the operating system

• Everyone can collect all types of IBM i performance data

• The visualization of the data may require additional products

• Visualization is the display of charts or tables of performance data

• The Performance Data Investigator is part of the operating system

• Everyone can visualize Collection Services data

!26


Wait Accounting Graphical Analysis Tools

• Performance Data Investigator (PDI) • Geared to average user

• Component in IBM Navigator for i Web console

• Nothing to install, can view Collection Services for free

• Disk Watcher, Performance Explorer, Database and Batch Model with 5770-PT1

• 5770-PT1 Job Watcher option for Job Watcher data

• IBM iDoctor for i • Geared to advanced user

• Microsoft Windows based client

• Requires Job Watcher yearly license

• IBM Service Offering

!27

http://www.ibm.com/developerworks/ibmi/library/i-pdi/index.html

https://www.ibm.com/it-infrastructure/services/lab-services/idoctor


IBM i Performance Data Investigator

• The IBM i Performance Data Investigator is included with the IBM i operating system • Collection Services, Health Indicators, Monitors, and Graph History

• Performance Tools, 5770-PT1 Manager feature must be installed for • Database • Batch Model • Disk Watcher • Performance Explorer

• Performance Tools, 5770-PT1 Job Watcher feature must be installed for Job Watcher

!28


Job Watcher

!29


Job Watcher

• Job watcher collects detailed wait information

• Objects being waited on

• Holder of object

• Job Watcher does not collect everything that Collection Services collects

• It does not always collect information about every thread

• Thread must use CPU during interval

• Thread must exist for entire interval

• It does not collect memory pool or detailed I/O statistics

• Data is written to (undocumented) Db2 files

!30


Job Watcher…

• Job Watcher collects more detailed performance data than Collection Services

• Call Stacks

• SQL Statements

• Socket and TCP data

• Activation Group data

• Java data for IBM Technology for Java VM Jobs

• More frequent intervals

• .1 second to 1 hour

• As fast as possible

!31


Job Watcher Usage

• Use Job Watcher when you need detailed performance data to resolve a problem

• Typically review the problem first with Collection Services data

• Consider using a definition like the IBM-supplied Q10SECSQL as a starting point

• 10 second intervals, gathers call stacks and SQL if it can

• For problem determination Job Watcher can be run on specific jobs

• Caution: When using Job Watcher on specific job(s), you may not get detailed Holder information

• Multiple collections can be run at the same time

• Need to manage the amount of data collected

• Specify Maximum Disk Space to consume on Start

• Will stop collecting if specified ASP threshold is reached

• iDoctor Monitors clean up after themselves

!32


Job Watcher Data Collection Steps

• Create the definition to identify what data will be collected

• Several IBM-supplied definitions come with the operating system

• Start the Job Watcher collection

• Run the collection until the problem has occurred

• Stop the Job Watcher collection

• Analyze the data

• Analysis can begin as soon as the data from the first interval has been collected

• There may be times when you want to run Job Watcher for long periods of time

• Consider 1 hour collection size

• Manage the size of the data

!33


Job Watcher Authority Requirements

• To use the Job Watcher commands (ADD/RVMJWDFN, STR/ENDJW)

• You must have *SERVICE special authority

• -OR- Be authorized to the Job Watcher function of the operating system

• Change Function Usage (CHGFCNUSG) command with a function ID of QIBM_SERVICE_JOB_WATCHER

• CHGFCNUSG FCNID(QIBM_SERVICE_JOB_WATCHER) USER(<usrprofile>) USAGE(*ALLOWED)

• Job Watcher Definitions:

• Additional authority is needed to see the definitions

• They are shipped with public authority *EXCLUDE

• To see the shipped definitions users need authority to the QAPYJWDFN file in QUSRSYS

!34

http://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/IBM%20i%20Technology%20Updates/page/Authority


Performance Tools Job Watcher Option

• Navigator for i Web provides a GUI for Job Watcher • Requires the Performance Tools LPP, Job Watcher option to be

installed

!35


Job Watcher in Navigator for i

• With Navigator, you can

• Manage your definitions

• Start and Stop Job Watcher Collections

• Manage your Job Watcher Collections

• Investigate Job Watcher Data

!36


IBM-Supplied Job Watcher Definitions

• Several IBM-supplied Job Watcher definitions are provided

• Recommendations: • Collect with Call Stacks and SQL

• Use 10 second intervals for general analysis (Q10secsql) • Use 5 second intervals for complex or intermittent issues, or for

contention related problems (Q5secsql)

!37


IBM Supplied Job Watcher Definitions

If you are using iDoctor, you will have additional Job Watcher definitions

!38


Running Job Watcher Continuously

• If you have a problem but Job Watcher is not running, you will need to recreate the problem to capture Job Watcher data

• Many clients choose to run Job Watcher 24x7 • Ensures you have diagnostic data the first time the problem occurs

• Options: • QMGTOOLS has the ability to run Job Watcher continuously

• http://www-01.ibm.com/support/docview.wss?uid=nas8N1020478 • iDoctor monitors

• Your own CL program

!39



Job Watcher Collections

!40


Run Job Watcher with the Commands

• Add Job Watcher Definition (ADDJWDFN) to define the collection

• Identifies the performance data that is to be collected

• Note: Job Watcher Definitions can only be displayed through the GUI

• Remove Job Watcher Definition (RMVJWDFN) to remove a definition

• Start Job Watcher (STRJW) to start the collection

• End Job Watcher (ENDJW) to end the collection (optionally)

!41


CPU Utilization and Waits Overview

• Disk Time, Lock Contention Time, Operating System Contention Time, CPU Queueing

!42


Interval Details

!43


Wait Accounting Analysis: Strategy

• Understand the “big picture” first

• Understand overall partition characteristics

• Identify any system bottlenecks

• Typically done using Collection Services data

• Drill down to job level

• Waits can be analyzed in various useful ways

• Continue detailed analysis using Job Watcher

• Narrow focus to interesting timeframes or jobs

• Many more job level details available

!44


Analyzing Job Wait Data

!45


Collection Services: Where to start


• Combines related waits into higher level buckets

Waits Overview

• All individual “blocked” wait buckets shown

!46


Run-Wait Signature

• CPU Utilization and Waits Overview

• Combination of CPU utilization and the wait buckets

• Very beneficial in assessing the overall health of your partition

• Stacked bars are wait times

• Line is partition CPU utilization

!47


Dispatched CPU Time

• Efficient System with little waiting?

• Or is something using too much CPU?

!48


Processor Bound System

• Dispatched CPU and CPU Queuing

!49


I/O Bound System

• Disk Time

!50


Lock Contention

• Job Watcher data is typically needed to solve lock related issues

!51


Investigate Job Watcher Data

!52


Job Watcher: Where to Start

Perspectives similar to Collection Services

!53


Job Watcher – Waits Overview

• Same wait buckets as Collection Services, but more granular intervals

!54


Reviewing wait times with Job Watcher

What is consuming CPU time?

!55


Cpu Utilization and Waits Overview

!56


Drill down to find the Thread or Task using CPU

!57


Investigate that Thread

!58


Display the SQL Statement for that Database Server Job

!59


Reviewing wait times with Job Watcher

What is causing disk wait time?

!60


• CPU Utilization and Waits Overview from a contrived workload shows disk time at the beginning of the test

• What caused it?

• Select the beginning and ending of the timeframe for investigation

Job Watcher and Disk Waits

!61


Drill Down to Disk Waits Overview

• It is mostly page faults time, with some non-fault read time

!62


Find the Jobs that are Faulting

!63


All Waits for One Thread

!64


Interval Details and What Job Watcher can Reveal

• Walk thru, interval by interval, and see the objects being faulted into memory

!65


With Collection Services…

• A similar investigation could be done with Collection Services data

• Collection Services could tell us…

• The timeframe

• The jobs and the job user profile

• And for server jobs, the job’s current user profile

!66

But Collection Services cannot show the detailed wait information, nor the

object waited on


Viewing Waits with Job Watcher Example of Object Lock Contention

!67



• Observe the pervasive gray Lock Contention Time

!68


Drill into Seizes and Locks Waits Overview

!69


Drill down into All Waits by Thread or Task Sorted by Seizes and Lock Waits

!70


The victim is the thread that is waiting for the lock

!71


The Job Watcher - Interval Details

!72


Holder Chase

• LOOP3 job was waiting on LOOP2

• Show hold to see LOOP2 shows that LOOP2 is waiting on LOOP4

• Show holder to see LOOP4, who is holding the lock on the object

!73


Collection Services

!74


User Defined Categories

• Allows you to store data from your application in Collection Services

• Develop a program to collect performance data for a new category in Collection Services

• Register your new category and specify the data collection program

• Collection Services includes your category in the list of available collection categories

• Add the category to your Collection Services profile, and then cycle Collection Services

• Your customized collection program is called at each collection interval

• Your data is stored in the Collection Object

• You access your data directly from the Collection Object with the Management Collection Object APIs

!75

http://www-01.ibm.com/support/knowledgecenter/ssw_ibm_i_73/rzahx/rzahxcollservuserdef.htm

https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_73/apis/perfmgmt_colsvc2.htm

https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_73/apis/perfmgmt_colsvc2.htm


User Defined Transactions and Collection Services

• Integrate application transaction data in Collection Services

• Instrument your application for transactions

• Using APIs, you can integrate transaction data into Collection Services sample data

• Start Transaction (QYPESTRT, qypeStartTransaction) API

• End transaction (QYPEENDT, qypeEndTransaction) API

• Configure Collection Services to to collect the User-defined Transactions (*USRTNS) category

• Collection Services collects the transaction data

• Collection Services also collects the transaction response time

• Create Performance Data exports this data to the user-defined transaction performance database file, QAPMUSRTNS

• Collection Services creates one record for each type of transaction for each unique job

• More later on user defined transactions and Performance Explorer….

!76

http://www-01.ibm.com/support/knowledgecenter/ssw_ibm_i_73/rzahx/rzahxcollservusertran.htm

https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_73/apis/qypestrt.htm

https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_73/apis/qypeendt.htm

http://www-01.ibm.com/support/knowledgecenter/ssw_ibm_i_73/rzahx/rzahxqapmusrtns.htm?lang=en-us


Database Full Opens

• Database full opens are expensive

• General recommendation is to keep Native Full Opens (RLA) per second < 1000

!77


Database Full Opens by Job

!78


Collection Services Data Files for Jobs

• QAPMJOBMI

• Machine Data for all tasks and threads

• Only contains data for those items that used CPU

• CPU, transitions, Physical I/Os, faults, overflows, thread, journal, IFS, SSL, storage, current user, …

• QAPMJOBOS

• OS Data for every job for each interval

• Job attributes, transactions, Logical I/Os, print stats, connection info, comm, IFS, submitter, server, …

• QAPMJOBL

• Logical view combining the two

• QAPMJSUM

• Selected QAPMJOBL data summarized by type of job

!79


Collection Services - Interesting Data not in PDI

• Collection Services contains a lot of interesting data that can be helpful for application developers

• Much of which is not externalized with PDI

• The following are some job examples (QAPMJOBMI and QP|APMJOBOS):

• Activation Group Creations / Programs Activated

• Who submitted a job? How many spooled files did a job create?

• Number of object and record locks for a job

• Number of socket send & receives, bytes sent & received

• Commit and journal operations

• Many IFS metrics

• Short-lived threads

!80


Short-Lived Threads and Tasks

• Collection Services captures performance data for every job, task, and secondary thread that used processor time during an interval

• Threads or tasks that are short-lived do not have their data collected at each interval • Rather, a count of short-lived threads and tasks is tracked

• As a whole, it could be a significant factor in total system utilization

• The QAPMCONF file reports the short lifespan thresholds used during collection.

• Environment variables can be used to a change the thresholds

• QPM_TASK_SL_THRESHOLD - control short lifespan task processing

• QPM_THREAD_SL_THRESHOLD - control short lifespan secondary thread processing

• The values are obtained at the start of a collection

• You must cycle an active collection for changed values to be used

!81

https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_73/rzahx/rzahxshortlifespanthreadsandtasks.htm


Viewing Short-Lived Threads

• Data is in the QAPMJOBMI file

• Short Lifespan Entry Count field

• Short lifespan tasks are reported for the processor node they were associated with

• Short lifespan secondary threads are reported for the job to which they belong

!82


Activation Group Creations, Programs Activated

• Collection Services has activation group statistics

• QAPMJOBMI file

• Following fields reported for the primary thread only

• JBAGCRT - Activation groups created. The number of activation groups created by all threads within this job.

• JBPGMSACT - Program and service program activations created. The number of ILE program and service program activations created by all threads within this job.

• PDI does not graph these metrics

• Query the Collection Services QAPMJOBMI file

!83


Example SQL to Summarize Activation InformationSELECT JBNAME CONCAT(JBUSER) CONCAT(JBNBR) AS JobName, SUM(JBAGCRT) AS AGs_Created, SUM(JBPGMSACT) AS PGMsActivated FROM qpfrdata.qapmjobmi GROUP BY JBNAME, JBUSER, JBNBR

!84


Example SQL, ORDER BY Activation Groups CreatedSELECT JBNAME CONCAT(JBUSER) CONCAT(JBNBR) AS JobName, SUM(JBAGCRT) AS AGs_Created, SUM(JBPGMSACT) AS PGMsActivated FROM qpfrdata.qapmjobmi GROUP BY JBNAME, JBUSER, JBNBR ORDER BY SUM(JBAGCRT) DESC;

!85


Group by JBNAME only to get a “Generic” Job ViewSELECT JBNAME AS JobName, SUM(JBAGCRT) AS AGs_Created, SUM(JBPGMSACT) AS PGMsActivated FROM qpfrdata.qapmjobmi GROUP BY JBNAME ORDER BY SUM(JBAGCRT) DESC;

!86


• Understand authority look ups before making a security implementation change

• QAPMSYSTEM file

• SYAUTH - The number of times that authority was checked for objects

• SYNUAL - The number of times that a noncached user authority lookup was performed

QAPMSYSTEM - Object Authority Lookups

!87

select intnum, syauth, synual from QAPMSYSTEM order by INTNUM


Performance Explorer

!88


• Performance Explorer is the most sophisticated IBM i performance tool

• Collects more detailed information about a specific application, program, or resource

• Every I/O operation, every task switch…

• Hundreds of events collected

• More overhead at collection time

• More storage required for resulting data

• Most complex to use

• Research the problem with other tools first

• PEX is used to identify the causes of performance problems that cannot be resolved using one of the other performance data collectors

• Generally used by IBM performance analysis experts

• iDoctor is required for advanced PEX Analysis


!89



• Performance Explorer is typically used for two main reasons:

• Detailed performance trace data is needed to identify the performance problem

• Analyzing the performance of applications

!90


Performance Explorer – “Trace Profile”

• “Trace-profile” is an easy and fairly light-weight PEX collection

• Useful when trying to diagnose high CPU issues

• Reports CPU usage at a program/module/procedure level

• Make sure you have latest PEX PTFs appliedhttps://www-01.ibm.com/support/docview.wss?uid=ibm10731315

• The PEX definition, which can be run over a subset of jobs is:ADDPEXDFN DFN(TPROF) TYPE(*TRACE) JOB((*ALL *ALL)) TASK(*ALL) MAXSTG(4000000) INTERVAL(1) TRCTYPE(*SLTEVT) SLTEVT(*YES) MCHINST(*NONE) BASEVT((*PMCO *NONE *FORMAT2))

!91

https://www-01.ibm.com/support/docview.wss?uid=ibm10731315


Performance Explorer – “Trace profile”

• Collect data

• Start Performance Explorer (STRPEX)

• Run the command, program, or workload that you want to analyze

• End Performance Explorer (ENDPEX)

• Analyze the data

!92


Performance Explorer TPROF reports – PDI

!93


Performance Explorer – “Statistics”

• PEX Stats can identify applications and programs that use excessive CPU or high I/O rates

• Relatively small amount of data

• Higher run-time overhead

• PEX removes the overhead from the reported data

• ADDPEXDFN DFN(MYSTATS) TYPE(*STATS) JOB(*ALL) TASKS(*ALL) DTAORG(*FLAT)

• PRTPEXRPT can be used to print the statistics collection

!94


Performance Explorer APIs

• Add Trace Point (QYPEADDT, qypeAddTracePoint)

• Records application-defined trace data.

• End Transaction (QYPEENDT, qypeEndTransaction)

• Indicates the end of a user-defined transaction.

• Log Transaction (QYPELOGT, qypeLogTransaction)

• Generates a transaction log record in the PEX trace data.

• Retrieve PEX Information (QYPERPEX, qypeRetrievePexInfo)

• Returns a list of active Performance Explorer collections

• Start Transaction (QYPESTRT, qypeStartTransaction)

• Called at the start of a user-defined transaction.

!95

http://www.apple.com

https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_73/apis/qypeaddt.htm?view=kc

https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_73/apis/qypeendt.htm?view=kc

https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_73/apis/qypelogt.htm?view=kc

https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_73/apis/qyperpex.htm?view=kc

https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_73/apis/qypestrt.htm?view=kc


PEX Log Transaction API

• PEX has the ability to collection transaction data

• Log Transaction (QYPELOGT, qypeLogTransaction) API

• Called any time between the Start and End Transaction APIs

• Allows you to trace the progress of your transactions

• Provides trace-type performance data collected by PEX

• Collection Services ignores this API

• When PEX is running, this API generates a log transaction trace record

• Captures the data supplied on the API

• PEX captures performance data associated with the current thread

• CPU, I/O, seize/lock activity

• Application-supplied data is written to the QAYPEMIUSR file, QMUDTA field

!96

https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_73/apis/qypelogt.htm


PEX Add Trace Point API

• Add Trace Point API allows you to record application-specific data

• Similar to the Log Transaction API, but more general purpose

• Add Trace Point (QYPEADDT, qypeAddTracePoint) API

• Called any time between the Start and End Transaction APIs

• Allows you to capture application-defined trace data

• Collection Services ignores this API

• When PEX is running, this API generates a generic application trace record

• Captures the data supplied on the API

• PEX captures performance data associated with the current thread

• CPU, I/O, seize/lock activity

• Application-supplied data is written to the QAYPEMIUSR file, QMUDTA field

!97

https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_73/apis/qypeaddt.htm?view=kc


More on PEX

• PEX Definitions from IBM Support • http://www-01.ibm.com/support/docview.wss?uid=nas8N1014671

!98



General Recommendation: Keep a Baseline and be Proactive

• Use Collection Services data for overall baseline data

• Weekly, end-of-month, end-of-year, prior to any hardware, software, or configuration change

• Use Job Watcher data as a baseline for major application changes

• The baseline

• Shows the expected performance characteristics over a period of time

• Makes it easier to recognize change

• Know your typical run-wait signatures at the partition level and job level

Before After

!99


About Dawn May

Dawn May Consulting, LLC

Customized on-site education • General IBM i • Systems Management • Performance tools and management • Work management

Performance Consulting • System performance review • Recurring performance assessments • Performance diagnostics

!100

Photo Credit: David Bowman @ IBM // Rochester, MN January 2018

[email protected] DawnMayi.com

https://twitter.com/DawnMayiCan@DawnMayiCan

https://www.linkedin.com/in/dawnmay/

http://ibmsystemsmag.com/blogs/i-can/


www.ibm.com/power/i

!101


Thank You!

The following have donated servers and services to the IBM i Community for demonstration, demo and educational uses:

• iInTheCloud, LLC • Pete Massiello and Larry Bolhuis • www.iInTheCloud.com • Hosting and network services

• Frankeni Technology Consuling, LLC • Larry Bolhuis • www.frankeni.com • Server components, licensing, and support services

• FRS • Carl Novit • www.gofrs.com • The Power System and server components

!102

http://www.iinthecloud.com/

http://www.frankeni.com/

http://www.gofrs.com/


IBM i Performance FAQ - a MUST read!http://www.ibm.com/common/ssi/cgi-bin/ssialias?subtype=WH&infotype=SA&appname=STGE_PO_PO_USEN&htmlfid=POW03102USEN&attachment=POW03102USEN.PDF

!103

http://www.ibm.com/common/ssi/cgi-bin/ssialias?subtype=WH&infotype=SA&appname=STGE_PO_PO_USEN&htmlfid=POW03102USEN&attachment=POW03102USEN.PDF

http://www.ibm.com/common/ssi/cgi-bin/ssialias?subtype=WH&infotype=SA&appname=STGE_PO_PO_USEN&htmlfid=POW03102USEN&attachment=POW03102USEN.PDF


IBM i Performance on developerWorkshttps://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/IBM%20i%20Technology%20Updates/page/IBM%20i%20Technology%20Updates

!104


i Can http://www.ibmsystemsmag.com/Blogs/i-Can/

For a simple list of all blogs on one page:

”i Can” Blog of Blogs

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power Systems/page/i Can Blog of Blogs

!105

QSYS2.ACTIVE_JOB_INFO Service Enhancements: QTEMP_SIZE and Much MoreIBM i Access Client Solutions - IFS PropertiesWork with Active Prestart JobsPrestart Jobs and Server JobsSubsystem Configuration for Prestart JobsDisplay Active Prestart JobsPrestart Job Configuration SettingsIBM i System Supplied Prestart Job EntriesIntroduction to Prestart JobsSearch the Job Log of an Active JobIBM i QSYS2.Services_Info() and Navigator for iImproved Temporary Storage Tracking for IBM i (Part 9)Copy Performance Data using ACS IFS TasksEditing a File in the IFS with Navigator for iWorking with the IFS: Access Client Solutions vs Navigator for iHow to Save 5250 Configurations as Default ProfileWorkload Groups and Performance ConsiderationsIBM i 7.2 Improved Temporary Storage Tracking (Part 8)Subsystem Management with Navigator for i, Part 2Active Memory Pools with Navigator for i

https://www.ibm.com/developerworks/community/wikis/home?lang=en

https://www.ibm.com/developerworks/community/wikis/home?lang=en

http://ibmsystemsmag.com/blogs/i-can/september-2018/qsys2-active_job_info-service-enhancements-qtemp_s/

http://ibmsystemsmag.com/blogs/i-can/september-2018/ibm-i-access-client-solutions-ifs-properties/

http://ibmsystemsmag.com/blogs/i-can/september-2018/work-with-active-prestart-jobs/

http://ibmsystemsmag.com/blogs/i-can/august-2018/prestart-jobs-and-server-jobs/

http://ibmsystemsmag.com/blogs/i-can/august-2018/subsystem-configuration-for-prestart-jobs/

http://ibmsystemsmag.com/blogs/i-can/july-2018/display-active-prestart-jobs/

http://ibmsystemsmag.com/blogs/i-can/july-2018/prestart-job-configuration-settings/

http://ibmsystemsmag.com/blogs/i-can/june-2018/ibm-i-system-supplied-prestart-job-entries/

http://ibmsystemsmag.com/blogs/i-can/june-2018/introduction-to-prestart-jobs/

http://ibmsystemsmag.com/blogs/i-can/may-2018/search-the-job-log-of-an-active-job/

http://ibmsystemsmag.com/blogs/i-can/may-2018/ibm-i-qsys2-services_info-and-navigator-for-i/

http://ibmsystemsmag.com/blogs/i-can/april-2018/improved-temporary-storage-tracking-for-ibm-i-(par/

http://ibmsystemsmag.com/blogs/i-can/april-2018/copy-performance-data-using-acs-ifs-tasks/

http://ibmsystemsmag.com/blogs/i-can/april-2018/editing-a-file-in-the-ifs-with-navigator-for-i/

http://ibmsystemsmag.com/blogs/i-can/march-2018/working-with-the-ifs-access-client-solutions-vs-na/

http://ibmsystemsmag.com/blogs/i-can/march-2018/how-to-save-5250-configurations-as-default-profile/

http://ibmsystemsmag.com/blogs/i-can/february-2018/workload-groups-and-performance-considerations/

http://ibmsystemsmag.com/blogs/i-can/february-2018/ibm-i-7-2-improved-temporary-storage-tracking-(par/

http://ibmsystemsmag.com/blogs/i-can/january-2018/subsystem-management-with-navigator-for-i-part-2/

http://ibmsystemsmag.com/blogs/i-can/december-2017/active-memory-pools-with-navigator-for-i/


iDoctor versus Performance Data Investigator

• You have two graphical interfaces for performance data analysis…Which should you use? It depends….

Feature iDoctor PDIInterface Windows client Browser

Wait Analysis Yes YesCollection Services Yes Yes

Job Watcher Yes YesDisk Watcher Yes Yes

Performance Explorer Yes Profile collections onlyDatabase Yes Yes

Job Watcher Monitors Yes NoCustomizable Yes Yes

User Defined graphs and queries

Yes YesUpdate Frequency Monthly Twice Yearly

Support Defect only Standard SWMA

Chargeable Yearly license▪ Collection Services at no additional charge with i▪Disk Watcher, Database, and Performance Explorer included with

base PT1 product▪ Job Watcher is an additional option of PT1 and has an additional Experimental Features Yes (e.g., VIOS

Investigator)No

Multinational language support No Yes

!106

ibm i performance tools for application developers

Documents