understanding the aix performance data in an ibm apv partition

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Understanding the AIX Performance Data in a PowerVM Partition

Pete Weilnau

[email protected]

Philadelphia CMGFeb 29, 2008


Agenda

Review of PowerVM

Instrumentation

Understanding the Instrumentation

Putting it into practice


Disclaimer

All trademarks referenced in this presentation

are the properties of their respective companies.

Some diagrams are from IBM Redbook SG24-

7940-02 Advanced POWER Virtualization on

IBM System p5: Introduction and Configuration


PowerVM

Available on System p5 and higher

A marketing umbrella term for a number of hardware and software technologies:

• Dynamic LPAR (DLPAR)• Shared Processor Pool (Micro-partitioning)• Virtual IO Server (VIO)• Simultaneous Multi-Threading (SMT)• Virtual LAN (VLAN)• Integrated Virtualization Manager (IVM)• Capacity on Demand (CoD)• Live Partition Mobility (POWER6 only)• Shared Dedicated Capacity (POWER6 only)


System p overview

AIX

LPAR

Linux

LPAR

VIO

Server

LPAR

AIX

LPAR

Linux

LPAR

Hypervisor firmware

HMC

intranetintranet

remote

admin


POWER Hypervisor

System firmware

• introduced w/ POWER5

• always enabled (think “NO BASIC MODE”)

Key features

• Dynamic LPAR (DLPAR)

allows dynamic reallocation of resources

... but only if the OS supports it!

• shared processor pool (Micro-partitioning)

AIX 5.3 & Linux only (no i5/OS support)

allows fractional physical CPU entitlement

• virtual device support (VLAN, SCSI, consoles)


POWER Hypervisor

source: Advanced POWER Virtualization on IBM System p5 – SG24-7940-02


Virtual IO Server

a software appliance that runs in a dedicated LPAR

as the name implies, it provides virtual IO services to other (“client”) LPARs

• virtual disks

• virtual Ethernet adapters

• shared Ethernet adapters (SEA)

Integrated Virtualization Manager (IVM)

• feature added w/ VIO Server 1.2

• provides a browser-based subset of HMC function

• main restriction – all resources are owned by VIO, no dedicated resources


Hardware Management Console (HMC)

hardware appliance to manage one or more

managed systems (CECs)

includes partition management and many other

management functions

provides optional web interface for remote

access


Web-based System Manager (WebSM)


Processor allocations


SMT and PURR



Physical, virtual, and logical processors



Focus of this research:

What can we learn about LPAR level activity

from an AIX Point of View?


Let’s talk Instrumentation


Old Faithfuls

iostat, vmstat, sar and time:

• Use PURR based statistics whenever SMT is

enabled %user, %system, %iowait and %idle.

When executing on a shared-processor partition:

• pc or %physc represent % physical processor

consumed.

• ec or %entc represent % of entitled capacity

consumed.


iostat example

What does %user, %sys, %idle, %iowait really represent in a shared-processor environment?

% of what? Isn’t it a moving target?


Old Faithfuls

Sar –p ALL:

• Use PURR based statistics whenever SMT is enabled

%usr, %sys, %wio and %idle.

When executing on a shared-processor partition:

• physc represent % physical processor consumed.

• %entc represent % of entitled capacity consumed.

lcpu = shows number of logical CPUs

ent = shows entitlement


Sar –P ALL example

What does that ‘U’ record represent?

‘-’ normally shows the total


sar –P ALL

U row is reported if running in shared processor pool.

It shows the UNUSED system-wide capacity.

• System-wide = LPAR.

If in uncapped mode:

• The individual processor rows (phyc) are calculated against physical consumption

Unused is computed against entitlement and not physical consumption.

Source: man sar


Old Faithfuls

mpstat -s:

• Displays both virtual and logical processors

proc0 – represents a Virtual CPU

cpu0 and cpu1 represent logical CPUs


Other Commands

Mpstat –a

• Provides a lot of detail info about processor

activity, well beyond the scope of this

presentation.

• cpu0 and cpu1 represent logical CPUs


lparstat - New

lparstat:

• Shows configuration and performance information for the

partition in which it is run.

4 Modes:

• lparstat :monitoring mode

• lparstat –h :hypervisor summary

• lparstat –H :hypervisor calls

• lparstat –i :Configuration info


lparstat

• Psize = processors in shared pool

• Lbusy = percent occupation of the logical CPUs at the system and user level

• Vcsw = virtual context switches

• Phint = phantom interrupts that the partition received


lparstat -h

• Adds:

• %hypv – the amount of time spent in the Power hypervisor.

• hcalls – the number of calls AIX made to the Power hypervisor (per sec)Note: requires root access


lparstat -H

Note: requires root access


lparstat -i


So far all the instrumentation discussed has

been related to the current LPAR

Is there a way to see other LPARs?


topas - cecdisp

topas:

• Is a real time monitor with significant

capability

• -cecdisp has been added for LPAR support


topas -cecdisp


topas –cecdisp - Another


Understanding the Instrumentation


Experiment


stress –c 1 –t 60 &

• One thread pushes a physical cpu to 100% and we’ve hit 100% of our entitlement.

Note: physc is shown in processors, not a percentage.• Note the lbusy is only 50%, meaning only 1 (of the 2) logical cpus is being used.• Note that SMT mode doesn’t limit the ability to fully consume a physical CPU.• App = the available physical processors in the shared pool.


stress –c 2 –t 60 &

• 2 threads still produces physical CPU = 100% and we’ve still hit 100% of our entitlement.

• Note the lbusy is now 100%, meaning both logical CPUs are now saturated.

• SMT is allowing 2 logical CPUs to run.

• In an uncapped environment, why didn’t it get another physical processor involved?


Let’s give this LPAR another VCPU


Rerun stress –c 1 –t 60

• Note we now have 4 logical CPUs. Entitlement of 1.00 has not changed.

• But we now have 2 more logical CPUs (smt=On)

• When we run stress with 1 thread we still get 100% of a physical CPU

• Entitled capacity is also at 100% which. But note lbusy is now at 25%.



• For the first time we have 2 physical processors saturated.

• Entitlement has finally exceeded 100%!

• lbusy is 50%

What will happen when we go to 3 threads?



• Still have 2 physical processors saturated. Entitlement still 200.

• lbusy is now 75.

What’s holding us back?


Let’s add yet another VCPU…


Test 3 Threads again…

Want to make a prediction?

• Physical consumption goes to 3 physical processors.

• lbusy is now 50. Note we now have 6 logical CPUs.

What will happen when we go to 4 threads?


Stress –c 4 –t 60

• Physical consumption and entitlement do not change.

• At 3 threads we were getting all the resource that was available. With 4 threads we are simply slicing the same resources across an additional thread, so no more work can actually get done.


Putting it into practice


Before PowerVM support

oprah

AIX

5.3

harpo

AIX

5.3

karl

AIX

5.3

marx

VIO

Server

groucho

Linux

(RedHat)

gummo

Linux

(SuSE)

System

Viewed


CPU utilization

If this was running on a MicroPartition, what

does 100% represent?


Process Utilization

Same Question


Before PowerVM Support

oprah

AIX

5.3

harpo

AIX

5.3

Karl

AIX

5.3

marx

VIO

Server

groucho

Linux

(RedHat)

gummo

Linux

(SuSE)


With PowerVM Support

oprah

AIX 5.3

1 CPU

Dedicated

harpo

AIX 5.3

Shared

karl

AIX 5.3

Shared

marx

VIO

Server

Shared

groucho

Linux(RedHat)

Shared

gummo

Linux(SuSE)

Shared

Shared Pool


Demo Environment

oprah

AIX 5.3

1 CPU

Dedicated

harpo

AIX 5.3

Shared

karl

AIX 5.3

Shared

marx

VIO

Server

Shared

groucho

Linux(RedHat)

Shared

gummo

Linux(SuSE)Shared

Shared Pool

System

Viewed


LPAR Configuration Info


harpo

AIX 5.3

Shared

karl

AIX 5.3

Shared

marx

VIO

Server

Shared

groucho

Linux

(RedHat)

Shared

gummo

Linux

(SuSE)

Shared

oprah

AIX 5.3

1 CPU

Dedicated

Shared Pool

What can we see about the

CEC?

Physical view


Physical CPU utilization


Shared pool utilization


oprah

AIX 5.3

1 CPU

Dedicated

harpo

AIX 5.3

Shared

karl

AIX 5.3

Shared

marx

VIO

Server

Shared

groucho

Linux

(RedHat)

Shared

gummo

Linux

(SuSE)

Shared

Shared Pool

Shared processor pool

What about LPARs competing for the

Shared Pool?


Now we see a very different view of

harpo’s real CPU usage!

Shared pool utilization


Memory allocation


Context switches


oprah

AIX 5.3

1 CPU

Dedicated

harpo

AIX 5.3

Shared

karl

AIX 5.3

Shared

marx

VIO

Server

Shared

groucho

Linux

(RedHat)

Shared

gummo

Linux

(SuSE)

Shared

Shared Pool

Dedicated LPARs

Can I see Dedicated LPARs?


Dedicated CPU utilization


oprah

AIX 5.3

1 CPU

Dedicated

harpo

AIX 5.3

Shared

karl

AIX 5.3

Shared

marx

AIX 5.3

VIO

Server

Shared

groucho

Linux

(RedHat)

Shared

gummo

Linux

(SuSE)

Shared

Shared Pool

LPAR details

What about the LPAR that is the PerfMan managed system?


Physical CPU utilization


Virtual CPU


In Summary

PowerVM is a sophisticated virtualization platform

modeled after zSeries partitioning.

Respectable instrumentation is available from an

AIX guest, but it requires a knowledgeable

consumer.


References

Advanced POWER Virtualization on IBM

System p5: Introduction and Configuration (IBM

Redbook SG24-7940-02)

IBM System p Advanced POWER Virtualization

Best Practices (IBM Redpaper)

System p Logical Partitioning Guide (SA76-

0098-00)

understanding the aix performance data in an ibm apv partition

Documents

physical processor

lparstat new lparstat

ibm system p5 sg24

power hypervisor source

lparstat h note

advanced power virtualization

mode lparstat h

sharedprocessor environment