aix performance monitoring - ocean user group of · pdf fileaix performance monitoring ocean...

© 2009 IBM Corporation

AIX Performance Monitoring OCEAN Tech Conference Irvine - July 2009

Mike Rede

Field Technical Sales Specialist

Power Systems

[email protected]


2

Agenda

• Topas/NMON

• LPARSTAT

• LSLPARUTIL

• LPAR2RRD

• PM for Power Systems


3

CPU Monitoring tools

• lparstat (new in 5.3, interval count) – > lparstat 1 5 > lparstat -i

• mpstat (new in 5.3, show CPU utilization) – > mpstat 1 5

• procmon (new in 5.3) • topas ( display process info)

– > topas -P

• sar – > sar 1 5 > sar -P ALL 1 3

• iostat – > iostat -t 1 5

• vmstat (show CPU utilization) – > vmstat 1 5

• ps • trace

– > trace -a; ls; trcstop (then) > ls -l /var/adm/ras/trcfile

• curt • splat • truss • gprof, pprof, prof, tprof • time, timex


4

Agenda

• Topas/NMON

• LPARSTAT

• LSLPARUTIL

• LPAR2RRD



5

• nmon integration into topas

– nmon function has been integrated into topas infrastructure including the ability to use nmon analyzer to process topas output

– Installed by default with AIX & fully supported by AIX Support

$ nmon script /usr/bin/nmon /usr/bin/topas_nmon

or

$ topas and hit ~

– Identical to nmon “classic” V12

– Including saving nmon data to file

•nmon –f –s300 –c 288 just works as normal

– Additional smitty support to make saving a daily nmon file easy

– For older AIX releases, continue to use nmon “classic”

•Which is supported via the AIX Wiki/Performance Tools Forum

AIX Enhancements


6

Topas/nmon SMIT Menus‏

© 2008 IBM Corporation 7

topas - CEC monitoring screen

Split screen accessible with -C or the "C" command Upper section shows CEC-level metrics Lower sections shows sorted list of shared and dedicated partitions

Configuration info retrieved from HMC or specified from command line

c means capped, C - capped with SMT u means shared, U - uncapped with SMT S means SMT

Use topas –R to record to /etc/perf/topas_cec.YYMMDD and topasout to display recording

Sample Full-Screen Cross-Partition Output Topas CEC Monitor Interval: 10 Wed Mar 6 14:30:10 2005 Partitions Memory (GB) Processors Shr: 4 Mon: 24 InUse: 14 Mon: 8 PSz: 4 Shr_PhysB: 1.7 Ded: 4 Avl: 24 Avl: 8 APP: 4 Ded_PhysB: 4.1 Host OS M Mem InU Lp Us Sy Wa Id PhysB Ent %EntC Vcsw PhI --------------------------shared------------------------------------------ ptools1 A53 u 1.1 0.4 4 15 3 0 82 1.30 0.50 22.0 200 5 ptools5 A53 U 12 10 1 12 3 0 85 0.20 0.25 0.3 121 3 ptools3 A53 C 5.0 2.6 1 10 1 0 89 0.15 0.25 0.3 52 2 ptools7 A53 c 2.0 0.4 1 0 1 0 99 0.05 0.10 0.3 112 2 -------------------------dedicated---------------------------------------- ptools4 A53 S 0.6 0.3 2 12 3 0 85 0.60 ptools6 A52 1.1 0.1 1 11 7 0 82 0.50 ptools8 A52 1.1 0.1 1 11 7 0 82 0.50 ptools2 A52 1.1 0.1 1 11 7 0 82 0.50

http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.doc/doc/base/aixinformation.htm


8

Agenda

• Topas/NMON

• LPARSTAT

• LSLPARUTIL

• LPAR2RRD



9

LPARSTAT command

• LPARSTAT command usage:

• The "lparstat" command shows the available physical processors in the shared pool on a micropartitioned server.

• The lparstat command can be run from any logical partition that has enabled "Allow shared processor pool utilization authority".

• Use the values of “app” and “psize” that are reported in the output to calculate the overall CPU utilization percent.


10

LPARSTAT command

P595-1

----------

# lparstat

System configuration: type=Shared‏‏‏mode=Uncapped‏‏‏smt=On‏‏‏lcpu=4‏‏‏mem=12288‏‏‏psize=64‏‏‏ent=0.50

%user‏‏%sys‏‏%wait‏ %idle‏‏physc %entc‏‏lbusy‏‏‏app‏‏‏‏vcsw phint

----- ‏----‏‏‏‏‏----‏‏‏------ ‏‏‏‏-----‏‏‏‏‏‏-----‏‏‏‏-----‏‏‏‏‏‏-----‏‏‏‏‏‏----‏‏‏‏‏‏‏-----

74323912 ‏3188322977‏‏‏51.50‏ ‏‏‏‏1.1 ‏‏‏‏‏‏1.8 ‏‏‏‏‏0.01‏ ‏‏97.8 ‏‏‏‏‏‏0.6‏ ‏‏‏‏‏0.9 ‏‏‏‏‏0.7‏‏

• “psize” is the size of the shared processor pool.

• “app” is the number of available physical processors.


11

LPARSTAT command

P595-2

----------

# lparstat


%user‏‏%sys‏‏%wait‏%idle‏‏physc %entc‏‏lbusy‏‏‏app‏‏‏‏vcsw ‏‏ phint

----- ‏‏‏-----‏‏‏‏-----‏‏‏------‏ ‏‏‏‏-----‏‏‏‏‏-----‏‏‏------‏‏‏‏------‏‏‏‏‏-----‏‏‏‏‏-----

603098954‏‏20060323490‏‏‏‏37.68‏ ‏‏‏‏‏3.3‏ ‏‏‏‏‏7.4 ‏‏‏‏‏0.09 ‏‏92.5 ‏‏‏‏‏‏0.7 ‏‏‏‏‏1.4‏‏‏‏‏‏‏5.4‏‏




12

LPARSTAT command

P595-3

----------

# lparstat


%user‏‏%sys‏‏%wait‏‏%idle physc %entc‏‏lbusy‏‏‏app‏‏‏‏vcsw‏‏‏‏‏‏‏‏‏ phint

----- ‏‏‏‏‏‏‏‏‏‏‏‏-----‏‏------‏‏‏------‏‏‏‏‏----- ‏‏‏‏----- ‏‏------‏‏‏‏------‏‏‏‏‏----- ‏‏‏‏‏-----

1134448816‏‏‏37820201839‏‏‏41.90 ‏‏‏‏‏2.4 ‏‏‏‏‏‏8.2 ‏‏‏‏‏0.16 ‏‏‏91.7 ‏‏‏‏0.8 ‏‏‏‏‏‏2.5‏‏‏‏‏‏‏5.0‏‏




13

LPARSTAT command

• The "lparstat" command output shows:

– the size of the shared pool is 64 CPU's (psize) on P595-3 and P595-1.

– the size of the shared pool is 56 CPU's (psize) on P595-2.

• The overall CPU utilization percentage for each machine can be calculated this way:

– Overall CPU utilization = (psize - app) / psize

– P595-1 has a CPU utilization percentage of 19.5 % = (64 - 51.5) / 64)




14

LPARSTAT command

• These results are representative of the AVERAGE CPU Utilization for the last couple seconds.

• To compare the DAILY AVERAGE using the “lparstat” command – and to compare to the results of the “lpar2rrd” graph – you would have to collect data for a 24 hour period like this:

• # lparstat 86400 1


15

Agenda

• Topas/NMON

• LPARSTAT

• LSLPARUTIL

• LPAR2RRD



16

LSLPARUTIL command

• Calculate the CPU utilization also as a percent.

• Use the "lslparutil" command to look at the "total_pool_cycles" and "utilized_pool_cycles" for each server.

• Secure login (SSH) to the HMC through a PuTTY session.

• > lslparutil –r pool –m (servername) –F total_pool_cycles utilized_pool_cycles

– NOTE: You can also get the same information - reported by “lslparutil” – by using the HMC GUI screen and “View”ing the Utilization Data.


17

LSLPARUTIL command - example

> lslparutil –r pool –m (server name) –F total_pool_cycles utilized_pool_cycles

158257737255071‏8967037181413078

• Notice that the “utilized_pool_cycles” value is 15 digits.

– 158257737255071

• Notice that the “total_pool_cycles” value is 16 digits.

– 8967037181413078

• If you divide “utilized” by “total” (1.58 by 89) you can get a rough estimate of the CPU utilization. In this case it is equal to 1.77 %.


18

LSLPARUTIL command - examples

• P595-1 - 64 active cpu, 453GB

– utilized_pool_cycles = 174287276922931140

– total_pool_cycles =913273813303689687‏‏‏‏‏

• P595-2 - 56 active cpu out of 64 installed, 1TB



• P595-3 - 64 active cpu, 1TB




19

LSLPARUTIL command

• Using the first two digits (and rounded) from utilized_pool_cycles and total_pool_cycles and doing the math shows that:

• P595-1 has a Utilization percentage of 17/91 = 19%




20

LPARSTAT comparison

• P595-1 has a CPU utilization percentage of 19.5 %

= (64 - 51.5) / 64


= (56 - 37.68) / 56


= (64 - 41.9) / 64

• So, "lparstat" and "lslparutil" CPU utilization percentages match up.


21

Agenda

• Topas/NMON

• LPARSTAT

• LSLPARUTIL

• LPAR2RRD



22

LPAR2RRD Tool

• LPAR2RRD graphical tool:

• http://www.ibm.com/developerworks/wikis/display/virtualization/lpar2rrd+tool

• "lpar2rrd" tool gets its data directly from the "lslparutil" command.


23

LPAR2RRD Tool

The CPU Utilization graph for the P595-1 shows the average utilization taken over time on Monday and into Tuesday.


24

LPAR2RRD Tool

There are 64 available processors = 49 Configured + 15 Borrowed. So the CPU utilization was calculated as 10.39 / 64 = 16.2 % . But that was for the last day.


25

LPAR2RRD – last month CPU utilization


26

LPAR2RRD Tool – last year


27

Agenda

• Topas/NMON

• LPARSTAT

• LSLPARUTIL

• LPAR2RRD



28

PM for Power Systems

• An automated capacity planning and performance analysis report and graphs offering.

• Designed to help you plan for and manage the growth and performance of your system.

• Summary level

• Detailed level


29

PM for Power Sytems

• Summary level:

– The summary level report offering is available for no additional charge. It provides you with summary level measurements on key performance metrics about your system or specific partition being monitored.

– Also included is a projection, based on your actual data, of your utilization and growth against the capacities of the system or partition.

– To qualify for the no additional charge option, the system has to be under processor warranty or on an IBM Hardware Maintenance Agreement.


30

PM for Power Sytems

• Detailed level:

– For a nominal additional charge, an IBM Global Services fee service provides all of the features of the no charge offering plus additional information.

– To substantiate the data on the summary graphs, up to 26 additional more detailed reports and graphs are available. These graphs are designed to help you begin to understand the detail behind some of the summary averages.

– As a client, you also have access to the interactive graphing function of the offering. This allows you to drill down into selected graphs to see the data at an hourly or daily level; and you can scroll backwards in time to see the same graphs for previous months to help create a visual comparison.


31

IBM Performance Management for Power Systems - - What is it and what are the options?

• PM for Power Systems (AIX support): an offering to simplify your systems

management by helping you plan for and manage the growth and performance of the

IBM Power System family

• Previously known as PM for System p

• Supports AIX (Version 5.2 or later)

• Note: PM for Power Systems also supports the IBM i operating system

• Two offerings for customers to choose from: (both available worldwide) http://www.ibm.com/systems/power/support/pm/ or

http://www.ibm.com/systems/p/support/pm/

• No Charge service….. A no additional charge option that

provides web delivered interactive access to key performance and

growth data plus a projection of remaining growth of the system or partition

- System must be under IBM warranty or on an IBM hardware

maintenance agreement to be eligible

• Full function service….. An IBM Global Technology Services (billable) report

set that provides multiple detail reports on an ongoing basis depicting the growth

and performance of the system

• Available either as a stand alone fee offering or as part of an IBM premium

service offering like the IBM Enhanced Technical Support offering

http://www.ibm.com/systems/p/support


32

PM for Power Systems data and workflow

Workload Estimator

integration…for sizing

future options

A rolling history of 24 months

of client performance and

growth data is securely

stored at IBM


33

All clients receive a record for each of their partitions reporting data on

the Server Information page

Select the report icon on the left nav for the respective system or partition

A tab is available showing IBM i systems; another shows IBM p systems


34

Interactive Management Summary Graphs

The No Additional

Charge graphs

This data is available for no additional charge if the system is

under warranty or on an IBM Hardware Maintenance

Agreement.

The individual graphs are accessed

interactively by the user and are then printable as Individual

graphs.

The user does have the ability to go ‘backwards in time’ to

select the month of interest. (up to 24 mos)


35

AIX CPU daily graph


36

Selecting other views of the data (interactive graphing example)

Please select a graph from the list

Average Processor Utilization in % per day

Usage by the Hour/Average during the period

Peak Processor Utilization

RUN Q > Limit in % per day

RUN Q > Limit by Hour during the period


37

AIX Usage by Hour – monthly view (interactive graphing example continued)


38

Peak processor utilization


39

Memory utilization per hour


40

Disk Space usage – daily (hard disk and file system)


41

PM for Power Systems

• Remember most everyone can take advantage of these offerings and tools for no additional charge. Step one is to get started with PM for Power Systems. Simply visit the following URL, select your operating system interest and then follow the ‘getting started’ instructions.

• http://www.ibm.com/systems/power/support/pm/


Thank you!


43

Useful Links – 1 of 3

•IBM POWER Systems – http://www-03.ibm.com/systems/power/

•Toolbox for IT – http://unix.ittoolbox.com/groups/technical-functional/ibm-aix-

l/

•AIX and UNIX Forums – http://www.ibm.com/developerworks/forums/dw_auforums.jsp

a

•AIX and POWER Portal – http://rootvg.net/

•Technotes & Tips – http://www-

03.ibm.com/support/techdocs/atsmastr.nsf/Web/Technotes

•IBM Systems Information Center – http://publib.boulder.ibm.com/eserver/


44


•Fix Central

– http://www-912.ibm.com/eserver/support/fixes/fixcentral/main/pseries/aix

•Contacting IBM (Support)

– http://techsupport.services.ibm.com/guides/pseries/contacting.shtml

•IBM - Software Support Handbook

– http://www14.software.ibm.com/webapp/set2/sas/f/handbook/home.html

•IBM developerWorks: Wiki

– http://www-941.haw.ibm.com/collaboration/wiki/

– http://www.ibm.com/developerworks/wikis/dashboard.action

•Training

– http://www.mux-ed.com/uglypink.pdf

http://www.mux-ed.com/uglypink.pdf




45


• AIX User Group Community

– http://www.ibm.com/community/usergroups

•IBM Redbooks – http://www.redbooks.ibm.com/

Power Virtualization: http://www.redbooks.ibm.com/abstracts/sg247940.html?Open

Workload Partition Mgmt: http://www.redbooks.ibm.com/abstracts/sg247431.html?Open

LPAR Simplification Tools Handbook: http://www.redbooks.ibm.com/abstracts/sg247231.html?Open

Power VM - Managing and Monitoring: http://www.redbooks.ibm.com/abstracts/sg247590.html?Open

AIX Enterprise Edition SA Guide: http://www.redbooks.ibm.com/redpieces/abstracts/sg247738.html?Open

AIX 6.1 Differences Guide: http://www.redbooks.ibm.com/abstracts/sg247559.html?Open

WLAR Mgmt in AIX: http://www.redbooks.ibm.com/abstracts/sg247656.html?Open

Multi-Tenant Utility Computing on Power Systems and AIX: http://www.redbooks.ibm.com/abstracts/sg247681.html?Open

•AIX Performance articles (first in a series (with links to others)):

http://www.ibm.com/developerworks/aix/library/au-aixoptimization/index.html

•Official IBM Visio Stencils: – http://www.visiocafe.com/ibm.htm

http://www.ibm.com/community/usergroups

http://www.redbooks.ibm.com/


46

This document was developed for IBM offerings in the United States as of the date of publication. IBM may not make these offerings available in

other countries, and the information is subject to change without notice. Consult your local IBM business contact for information on the IBM

offerings available in your area.

Information in this document concerning non-IBM products was obtained from the suppliers of these products or other public sources. Questions

on the capabilities of non-IBM products should be addressed to the suppliers of those products.

IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give

you any license to these patents. Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive, Armonk, NY

10504-1785 USA.

All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives

only.

The information contained in this document has not been submitted to any formal IBM test and is provided "AS IS" with no warranties or

guarantees either expressed or implied.

All examples cited or described in this document are presented as illustrations of the manner in which some IBM products can be used and the

results that may be achieved. Actual environmental costs and performance characteristics will vary depending on individual client configurations

and conditions.

IBM Global Financing offerings are provided through IBM Credit Corporation in the United States and other IBM subsidiaries and divisions

worldwide to qualified commercial and government clients. Rates are based on a client's credit rating, financing terms, offering type, equipment

type and options, and may vary by country. Other restrictions may apply. Rates and offerings are subject to change, extension or withdrawal

without notice.

IBM is not responsible for printing errors in this document that result in pricing or information inaccuracies.

All prices shown are IBM's United States suggested list prices and are subject to change without notice; reseller prices may vary.

IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.

Many of the features described in this document are operating system dependent and may not be available on Linux. For more information,

please check: http://www.ibm.com/systems/p/software/whitepapers/linux_overview.html

Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are

dependent on many factors including system hardware configuration and software design and configuration. Some measurements quoted in this

document may have been made on development-level systems. There is no guarantee these measurements will be the same on generally-

available systems. Some measurements quoted in this document may have been estimated through extrapolation. Users of this document

should verify the applicable data for their specific environment.

Revised January 19, 2006

Special notices

http://www.ibm.com/systems/p/software/whitepapers/linux_overview.html


47

The following terms are registered trademarks of International Business Machines Corporation in the United States and/or other countries: AIX, AIX/L,

AIX/L(logo), alphaWorks, AS/400, BladeCenter, Blue Gene, Blue Lightning, C Set++, CICS, CICS/6000, ClusterProven, CT/2, DataHub, DataJoiner, DB2,

DEEP BLUE, developerWorks, DirectTalk, Domino, DYNIX, DYNIX/ptx, e business(logo), e(logo)business, e(logo)server, Enterprise Storage Server,

ESCON, FlashCopy, GDDM, i5/OS, IBM, IBM(logo), ibm.com, IBM Business Partner (logo), Informix, IntelliStation, IQ-Link, LANStreamer, LoadLeveler,

Lotus, Lotus Notes, Lotusphere, Magstar, MediaStreamer, Micro Channel, MQSeries, Net.Data, Netfinity, NetView, Network Station, Notes, NUMA-Q,

Operating System/2, Operating System/400, OS/2, OS/390, OS/400, Parallel Sysplex, PartnerLink, PartnerWorld, Passport Advantage, POWERparallel,

Power PC 603, Power PC 604, PowerPC, PowerPC(logo), PowerPC 601, Predictive Failure Analysis, pSeries, PTX, ptx/ADMIN, RETAIN, RISC

System/6000, RS/6000, RT Personal Computer, S/390, Scalable POWERparallel Systems, SecureWay, Sequent, ServerProven, SpaceBall, System/390,

The Engines of e-business, THINK, Tivoli, Tivoli(logo), Tivoli Management Environment, Tivoli Ready(logo), TME, TotalStorage, TURBOWAYS, VisualAge,

WebSphere, xSeries, z/OS, zSeries.

The following terms are trademarks of International Business Machines Corporation in the United States and/or other countries: Advanced Micro-

Partitioning, AIX 5L, AIX PVMe, AS/400e, Chipkill, Chiphopper, Cloudscape, DB2 OLAP Server, DB2 Universal Database, DFDSM, DFSORT, e-

business(logo), e-business on demand, eServer, Express Middleware, Express Portfolio, Express Servers, Express Servers and Storage, GigaProcessor,

HACMP, HACMP/6000, I5/OS (logo), IBMLink, IBM TotalStorage Proven, IMS, Intelligent Miner, iSeries, Micro-Partitioning, NUMACenter, ON DEMAND

BUSINESS logo, OpenPower, POWER, Power Architecture, Power Everywhere, Power Family, Power PC, PowerPC Architecture, PowerPC 603, PowerPC

603e, PowerPC 604, PowerPC 750, POWER2, POWER2 Architecture, POWER3, POWER4, POWER4+, POWER5, POWER5+, POWER6, POWER6+,

Redbooks, Sequent (logo), SequentLINK, Server Advantage, ServeRAID, Service Director, SmoothStart, SP, System i, System i5, System p, System p5,

System Storage, System z, System z9, S/390 Parallel Enterprise Server, Tivoli Enterprise, TME 10, TotalStorage Proven, Ultramedia, VideoCharger,

Virtualization Engine, Visualization Data Explorer, X-Architecture, z/Architecture, z/9.

A full list of U.S. trademarks owned by IBM may be found at: http://www.ibm.com/legal/copytrade.shtml.

UNIX is a registered trademark in the United States, other countries or both.

Linux is a trademark of Linus Torvalds in the United States, other countries or both.

Microsoft, Windows, Windows NT and the Windows logo are registered trademarks of Microsoft Corporation in the United States and/or other countries.

Intel, Intel Xeon, Itanium and Pentium are registered trademarks or trademarks of Intel Corporation in the United States and/or other countries.

AMD Opteron is a trademark of Advanced Micro Devices, Inc.

Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States and/or other countries.

TPC-C and TPC-H are trademarks of the Transaction Performance Processing Council (TPPC).

SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPEC OMP, SPECviewperf, SPECapc, SPEChpc, SPECjvm, SPECmail, SPECimap and

SPECsfs are trademarks of the Standard Performance Evaluation Corp (SPEC).

NetBench is a registered trademark of Ziff Davis Media in the United States, other countries or both.

AltiVec is a trademark of Freescale Semiconductor, Inc.

Other company, product and service names may be trademarks or service marks of others.

Revised January 19, 2006

Special notices (cont.)


Thank you!

aix performance monitoring - ocean user group of · pdf fileaix performance monitoring ocean...

Documents