understanding the aix performance data in an ibm apv partition

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Understanding the AIX Performance Data in a PowerVM Partition

Pete Weilnau

pete.weilnau@perfman.com

Philadelphia CMGFeb 29, 2008

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Agenda

Review of PowerVM

Instrumentation

Understanding the Instrumentation

Putting it into practice

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Disclaimer

All trademarks referenced in this presentation

are the properties of their respective companies.

Some diagrams are from IBM Redbook SG24-

7940-02 Advanced POWER Virtualization on

IBM System p5: Introduction and Configuration

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

PowerVM

Available on System p5 and higher

A marketing umbrella term for a number of hardware and software technologies:

• Dynamic LPAR (DLPAR)• Shared Processor Pool (Micro-partitioning)• Virtual IO Server (VIO)• Simultaneous Multi-Threading (SMT)• Virtual LAN (VLAN)• Integrated Virtualization Manager (IVM)• Capacity on Demand (CoD)• Live Partition Mobility (POWER6 only)• Shared Dedicated Capacity (POWER6 only)

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

System p overview

AIX

LPAR

Linux

LPAR

VIO

Server

LPAR

AIX

LPAR

Linux

LPAR

Hypervisor firmware

HMC

intranetintranet

remote

admin

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

POWER Hypervisor

System firmware

• introduced w/ POWER5

• always enabled (think “NO BASIC MODE”)

Key features

• Dynamic LPAR (DLPAR)

allows dynamic reallocation of resources

... but only if the OS supports it!

• shared processor pool (Micro-partitioning)

AIX 5.3 & Linux only (no i5/OS support)

allows fractional physical CPU entitlement

• virtual device support (VLAN, SCSI, consoles)

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

POWER Hypervisor

source: Advanced POWER Virtualization on IBM System p5 – SG24-7940-02

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Virtual IO Server

a software appliance that runs in a dedicated LPAR

as the name implies, it provides virtual IO services to other (“client”) LPARs

• virtual disks

• virtual Ethernet adapters

• shared Ethernet adapters (SEA)

Integrated Virtualization Manager (IVM)

• feature added w/ VIO Server 1.2

• provides a browser-based subset of HMC function

• main restriction – all resources are owned by VIO, no dedicated resources

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Hardware Management Console (HMC)

hardware appliance to manage one or more

managed systems (CECs)

includes partition management and many other

management functions

provides optional web interface for remote

access

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Web-based System Manager (WebSM)

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Processor allocations

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

SMT and PURR

source: Advanced POWER Virtualization on IBM System p5 – SG24-7940-02

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Physical, virtual, and logical processors

source: Advanced POWER Virtualization on IBM System p5 – SG24-7940-02

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Focus of this research:

What can we learn about LPAR level activity

from an AIX Point of View?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Let’s talk Instrumentation

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Old Faithfuls

iostat, vmstat, sar and time:

• Use PURR based statistics whenever SMT is

enabled %user, %system, %iowait and %idle.

When executing on a shared-processor partition:

• pc or %physc represent % physical processor

consumed.

• ec or %entc represent % of entitled capacity

consumed.

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

iostat example

What does %user, %sys, %idle, %iowait really represent in a shared-processor environment?

% of what? Isn’t it a moving target?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Old Faithfuls

Sar –p ALL:

• Use PURR based statistics whenever SMT is enabled

%usr, %sys, %wio and %idle.

When executing on a shared-processor partition:

• physc represent % physical processor consumed.

• %entc represent % of entitled capacity consumed.

lcpu = shows number of logical CPUs

ent = shows entitlement

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Sar –P ALL example

What does that ‘U’ record represent?

‘-’ normally shows the total

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

sar –P ALL

U row is reported if running in shared processor pool.

It shows the UNUSED system-wide capacity.

• System-wide = LPAR.

If in uncapped mode:

• The individual processor rows (phyc) are calculated against physical consumption

Unused is computed against entitlement and not physical consumption.

Source: man sar

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Old Faithfuls

mpstat -s:

• Displays both virtual and logical processors

proc0 – represents a Virtual CPU

cpu0 and cpu1 represent logical CPUs

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Other Commands

Mpstat –a

• Provides a lot of detail info about processor

activity, well beyond the scope of this

presentation.

• cpu0 and cpu1 represent logical CPUs

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

lparstat - New

lparstat:

• Shows configuration and performance information for the

partition in which it is run.

4 Modes:

• lparstat :monitoring mode

• lparstat –h :hypervisor summary

• lparstat –H :hypervisor calls

• lparstat –i :Configuration info

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

lparstat

• Psize = processors in shared pool

• Lbusy = percent occupation of the logical CPUs at the system and user level

• Vcsw = virtual context switches

• Phint = phantom interrupts that the partition received

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

lparstat -h

• Adds:

• %hypv – the amount of time spent in the Power hypervisor.

• hcalls – the number of calls AIX made to the Power hypervisor (per sec)Note: requires root access

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

lparstat -H

Note: requires root access

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

lparstat -i

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

So far all the instrumentation discussed has

been related to the current LPAR

Is there a way to see other LPARs?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

topas - cecdisp

topas:

• Is a real time monitor with significant

capability

• -cecdisp has been added for LPAR support

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

topas -cecdisp

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

topas –cecdisp - Another

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Understanding the Instrumentation

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Experiment

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

stress –c 1 –t 60 &

• One thread pushes a physical cpu to 100% and we’ve hit 100% of our entitlement.

Note: physc is shown in processors, not a percentage.• Note the lbusy is only 50%, meaning only 1 (of the 2) logical cpus is being used.• Note that SMT mode doesn’t limit the ability to fully consume a physical CPU.• App = the available physical processors in the shared pool.

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

stress –c 2 –t 60 &

• 2 threads still produces physical CPU = 100% and we’ve still hit 100% of our entitlement.

• Note the lbusy is now 100%, meaning both logical CPUs are now saturated.

• SMT is allowing 2 logical CPUs to run.

• In an uncapped environment, why didn’t it get another physical processor involved?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Let’s give this LPAR another VCPU

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Rerun stress –c 1 –t 60

• Note we now have 4 logical CPUs. Entitlement of 1.00 has not changed.

• But we now have 2 more logical CPUs (smt=On)

• When we run stress with 1 thread we still get 100% of a physical CPU

• Entitled capacity is also at 100% which. But note lbusy is now at 25%.

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Rerun stress –c 2 –t 60

• For the first time we have 2 physical processors saturated.

• Entitlement has finally exceeded 100%!

• lbusy is 50%

What will happen when we go to 3 threads?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Rerun stress –c 3 –t 60

• Still have 2 physical processors saturated. Entitlement still 200.

• lbusy is now 75.

What’s holding us back?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Let’s add yet another VCPU…

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Test 3 Threads again…

Want to make a prediction?

• Physical consumption goes to 3 physical processors.

• lbusy is now 50. Note we now have 6 logical CPUs.

What will happen when we go to 4 threads?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Stress –c 4 –t 60

• Physical consumption and entitlement do not change.

• At 3 threads we were getting all the resource that was available. With 4 threads we are simply slicing the same resources across an additional thread, so no more work can actually get done.

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Putting it into practice

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Before PowerVM support

oprah

AIX

5.3

harpo

AIX

5.3

karl

AIX

5.3

marx

VIO

Server

groucho

Linux

(RedHat)

gummo

Linux

(SuSE)

System

Viewed

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

CPU utilization

If this was running on a MicroPartition, what

does 100% represent?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Process Utilization

Same Question

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Before PowerVM Support

oprah

AIX

5.3

harpo

AIX

5.3

Karl

AIX

5.3

marx

VIO

Server

groucho

Linux

(RedHat)

gummo

Linux

(SuSE)

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

With PowerVM Support

oprah

AIX 5.3

1 CPU

Dedicated

harpo

AIX 5.3

Shared

karl

AIX 5.3

Shared

marx

VIO

Server

Shared

groucho

Linux(RedHat)

Shared

gummo

Linux(SuSE)

Shared

Shared Pool

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Demo Environment

oprah

AIX 5.3

1 CPU

Dedicated

harpo

AIX 5.3

Shared

karl

AIX 5.3

Shared

marx

VIO

Server

Shared

groucho

Linux(RedHat)

Shared

gummo

Linux(SuSE)Shared

Shared Pool

System

Viewed

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

LPAR Configuration Info

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

harpo

AIX 5.3

Shared

karl

AIX 5.3

Shared

marx

VIO

Server

Shared

groucho

Linux

(RedHat)

Shared

gummo

Linux

(SuSE)

Shared

oprah

AIX 5.3

1 CPU

Dedicated

Shared Pool

What can we see about the

CEC?

Physical view

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Physical CPU utilization

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Shared pool utilization

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

oprah

AIX 5.3

1 CPU

Dedicated

harpo

AIX 5.3

Shared

karl

AIX 5.3

Shared

marx

VIO

Server

Shared

groucho

Linux

(RedHat)

Shared

gummo

Linux

(SuSE)

Shared

Shared Pool

Shared processor pool

What about LPARs competing for the

Shared Pool?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Now we see a very different view of

harpo’s real CPU usage!

Shared pool utilization

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Memory allocation

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Context switches

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

oprah

AIX 5.3

1 CPU

Dedicated

harpo

AIX 5.3

Shared

karl

AIX 5.3

Shared

marx

VIO

Server

Shared

groucho

Linux

(RedHat)

Shared

gummo

Linux

(SuSE)

Shared

Shared Pool

Dedicated LPARs

Can I see Dedicated LPARs?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Dedicated CPU utilization

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

oprah

AIX 5.3

1 CPU

Dedicated

harpo

AIX 5.3

Shared

karl

AIX 5.3

Shared

marx

AIX 5.3

VIO

Server

Shared

groucho

Linux

(RedHat)

Shared

gummo

Linux

(SuSE)

Shared

Shared Pool

LPAR details

What about the LPAR that is the PerfMan managed system?

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Physical CPU utilization

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

Virtual CPU

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

In Summary

PowerVM is a sophisticated virtualization platform

modeled after zSeries partitioning.

Respectable instrumentation is available from an

AIX guest, but it requires a knowledgeable

consumer.

© COPYRIGHT 2008 THE INFORMATION SYSTEMS MANAGER, INC. (ISM)

References

Advanced POWER Virtualization on IBM

System p5: Introduction and Configuration (IBM

Redbook SG24-7940-02)

IBM System p Advanced POWER Virtualization

Best Practices (IBM Redpaper)

System p Logical Partitioning Guide (SA76-

0098-00)