v sphere apis for performance monitoring

7/28/2019 V Sphere Apis for Performance Monitoring

1/92

vSphere APIs for performance monitoringLondon Workshop

October 2010

Balaji Parimi, Staff Engineer, Ecosystem Performance, VMware, Inc.

Ravi Soundararajan, Senior Staff Engineer, Performance, VMware, Inc.


2/92

Motivation

To debug performance, why deal with this...?


3/92

Motivation

When you can deal with this instead?


4/92

More motivation

Why look at data like this?

Before memhog: no guest swapping After memhog, guest swaps,but Host does not!


5/92

More motivation

When you can look at it like this?


6/92

Even more motivation

Why compare resource pool performance like this?


7/92

Even more motivation

When you can compare them like this?


8/92

Why?

vSphere gives you awesome, helpful charts

But you dont have to rely solely on these charts

Do you want to learn how to make your own charts?

Keep watching


9/92

Goal

Teach you how to use our APIs for performance monitoring


10/92

Agenda

What sorts of stats are useful? How does vSphere retrieve them?

How can you get these stats and use them yourself?


11/92

Useful stats

Basics of performance monitoring in virtual infrastructure Find underperforming resources

Find overcommitted resources

Identify issues due to resource sharing among VMs

Resources we will look at

CPU

Memory

Disk

Network


12/92

Resources that we often look at

CPU Memory

Disk

Network


13/92

CPU basics

ESX

CPU0 CPU1 CPU2 CPU3

VM0 VM1 VM2 VM3

VM4

Run (accumulating used time)Ready (wants to run, no physical CPU available)Wait: blocked on I/O or voluntarily descheduled

VM5VM6

Run

Ready

Wait/Idle


14/92

Why is my VM slow?

CPU saturated (cpu.usage.average)

Ready time? (cpu.ready.summation)

Latency to be swapped in? (cpu.swapwait.summation)


15/92

CPU saturation

2 vCPUs2.2GHz/CPU~4.4GHz used

(Look at lefty-axis)


16/92

Small ready time

Ready time vCPU1: 150msReal-time chart: refresh 20s150ms / 20s = 0.75% (No big deal)Righty-axis is relevant


17/92

Now, turn on CPU burner on same host

CPU burner~100% of 1 vCPU


18/92

And see what happens to original VMs ready time

SpecJBB ready time~2000ms = 10%

(ps. SpecJBB perf. dropped by 10%)


19/92

Latency to load in VM: cpu.swapwait.average

Sometimes there is a latency to load VM data from disk: cpu swapwait

CPU takes 20s to load in data before VM can run!


20/92

CPU issues: Summary

CPU saturated? High Ready time

Problematic if it is sustained for high periods

Sample rule of thumb: > 20% per vCPU investigate further

Possible contention for CPU resources among VMs

Workload Variability? Fix with VMotion/DRS

Resource limits on VMs? Check Limits, reservations and shares

Actual over commitment? Fix with Vmotion/DRS/more CPUs

High SwapWait time

Consider setting memory reservation (see next section, Memory)


21/92


CPU Memory

Disk

Network


22/92

Memory

ESX must balance memory usage Page sharing to reduce memory footprint of Virtual Machines

Ballooning to relieve memory pressure in a graceful way

Host swapping to relieve memory pressure when ballooning insufficient

Compression to relieve memory pressure without host-level swapping

ESX allows over commitment of memory

Sum of configured memory sizes of virtual machines can be greater than

physical memory if working sets fit Memory also has limits, shares, and reservations

Host swapping can cause performance degradation


23/92

VM1

Ballooning, compression, and swapping (1)

Ballooning: Memctl driver grabs pages and gives to ESX

Guest OS choose pages to give to memctl (avoids hot pages if possible): either freepages or pages to swap

Unused pages are given directly to memctl

Pages to be swapped are first written to swap partition within guest OS and thengiven to memctl

Swap partition w/inGuest OS

ESX

VM2

memctl

1. Balloon

2. Reclaim

3. Redistribute

F


24/92

SwapPartition(w/in guest)

Ballooning, swapping, and compression (2)

Swapping: ESX reclaims pages forcibly

Guest doesnt pick pagesESX may inadvertently pick hot pages (possible VMperformance implications)

Pages written to VM swap file

VM1

ESX

VM2

VSWP(external to guest)

1. Force Swap2. Reclaim3. Redistribute


25/92

ESX

CompressionCache

Ballooning, swapping and compression (3)

Compression: ESX reclaims pages, writes to in-memory cache

Guest doesnt pick pagesESX may inadvertently pick hot pages (possible VMperformance implications)

Pages written in-memory cache faster than host-level swapping

SwapPartition(w/in guest)

VM1 VM2

1. Write to Compression Cache2. Give pages to VM2


26/92

Ballooning, swapping, and compression

Bottom line: Ballooning may occur even when no memory pressure just to keep memory

proportions under control

Ballooning is preferable to compression and vastly preferably to swapping

Guest can surrender unused/free pages With host swapping, ESX cannot tell which pages are unused or free and may

accidentally pick hot pages

Even if balloon driver has to swap to satisfy the balloon request, guest chooses whatto swap

Can avoid swapping hot pages within guest

Compression: reading from compression cache is faster than reading from disk


27/92

Swapping in Guest! = Swapping in Host

DVDstore benchmark: SQL DB benchmark uses lots of memory

About to start memory hogger program in guest


28/92

Force Guest swapping: No Host-level swapping

Before memhog: no guest swapping After memhog, guest swaps, butHost does not!


29/92

Viewing Host-level swapping with performance charts

Setup: 2 VMsone dvdstore, one memhog, competing for host memoryHost swaps out dvdstore VM memory to fulfill memhog VM requestsHost swaps in dvdstore VM memory to fulfill dvdstore VM requests


30/92

Using Swap Rate Counters: Remember CPU SwapWait?

Cpu.swapwait.summation: CPU is waiting for memory to be swapped in


31/92

Absolute Swap Counters

Swapin, swapout (KB) show some activity but hard to detect


32/92

And Swap Rate Counters

SwapinRate, SwapoutRate (KBps) show activity much more clearlyRule of thumb: host swapping > 1MBps is cause for concern

R h f l k


33/92


CPU Memory

Disk

Network

ESX t t k


34/92

ESX storage stack

Different latencies for local disk vs. SAN (caching, switches, etc.) Queuing within kernel and in hardware

vSphere shows

Total Command Latency

Kernel Latency

Device Latency

Bandwidth/IOPS

Di k f bl 101


35/92

Disk performance problems 101

What should I look for to figure out if disk is an issue?

Am I getting the IOPs I expect?

Am I getting the bandwidth (read/write) I expect?

Are the latencies higher than I expect?

Where is time being spent?

What are some things I can do?

Make sure devices are configured properly (caches, queue depths)

Use multiple adapters and multipathing Check networking settings (for iSCSI/NAS)

Another disk example: Slow VM power on


36/92

Another disk example: Slow VM power on

Trying to Power on a VM

Sometimes, powering on VM would take 5 seconds

Other times, powering on VM would take 5 minutes!

Where to begin?

Powering on a VM requires disk activity on host Check disk metrics for host

Lets look at the vSphere client


37/92

Let s look at the vSphere client

Max Disk Latencies range from 100ms to 1100msvery high! Why?

(counter name: disk.maxTotalLatency.latest)

Rule of thumb:latency > 20ms isBad.

Here:1,100msREALLY BAD!!!

High disk latency: Mystery solved


38/92

High disk latency: Mystery solved

Host events: disk has connectivity issues high latencies!

Bottom line: monitor disk latencies; issues may not be related tovirtualization!



39/92


CPU

Memory

Disk

Network

Network performance problems 101


40/92

Network performance problems 101

What should I look for to figure out if network is an issue?

Am I getting the packet rate that I expect?

Am I getting the bandwidth (read/write) I expect?

Is all traffic on one NIC, or spread across many NICs?

[more advanced not available through counters]: out-of-order packets?

What are some things I can do?

Check host networking settings

Full-duplex/Half-duplex

10Gig network vs 100Mb network?

Firewall settings

Check VM settings: all VMs on proper networks?

Network performance troubleshooting


41/92

Network performance troubleshooting

Customer complains about slow network

Shes running netperf on a GigE Link

She sees only 200Mbps

Why? I bet its that VMware stuff!!

Note to reader: Please dont blame VMware first

Where do we start?

All VMs using same NIC (VM network)


42/92

All VMs using same NIC (VM network)

All VMs using VM Network and sharing 1 physical NIC

Where do we begin? Check VM bandwidth


43/92

Where do we begin? Check VM bandwidth

Measure VM Bandwidth (net.transmitted.average)

200 Mb/s

Screenshot from the vSphere client

Check Host Bandwidth


44/92

Measure Host Bandwidth (net.transmitted.average)

Host sees around 900Mbpswhy is VM at 200Mbps?

Hmm are we sharing this NIC with multiple VMs?

All traffic is going through one NIC!


45/92

g g g

Measure per-physical-NIC traffic

Hmm all VM traffic is going through 1 NIC

Lets split the VMs across NICs

All traffic through oneNIC on this host

Split VMs across multiple NICs. Bingo!


46/92

p p g

Network issues: Configuration woes


47/92

Network adapter set to fullduplex, 100 Mbps:< 0.1Mbps!

Specificcombo of switch andadapter caused thisperformance degradation!

Lesson: Check specs &configuration!

Network adapter set to

autonegotiate: 90Mbps

Agenda


48/92

What sorts of stats are useful?

How does vSphere retrieve them?


Stats infrastructure in vSphere


49/92

ESX

VM VM VM VM VM

vCenter Server(vpxd, tomcat)

ESX

VM VM VM VM VM

ESX

VM VM VM VM VM

DB

1. Collect 20s

and 5-min hostand VM stats2. Send 5-minstats tovCenter

3. Send 5-minstats to DB

4. Rollups


50/92

Anatomy of a stats query: Past-hour (RealTime) Stats


51/92

Client

ESX

VM VM VM VM VM


ESX

VM VM VM VM VM

ESX

VM VM VM VM VM

DB

1. Query

2. Get statsfrom host

3. Response

No calls to DBNote: Same code path for past-day stats within last 30 minutes

Anatomy of a stats query: Archived stats


52/92

Client

ESX

VM VM VM VM VM


ESX

VM VM VM VM VM

ESX

VM VM VM VM VM

DB

1. Query

3. Response

No calls to ESX host (caveats apply)Stats Level = Store this stat in the DB

2. Get Stats

Agenda


53/92

What sorts of stats are useful?

How does vSphere retrieve them?


Phew! Ok, How do I get these stats?


54/92

You want a chart like this?

PowerCLI

CPU Usage for a VM for last hour:

$vm = Get-VM Name Foo

Get-Stat Entity $vm Realtime Maxsample 180 Statcpu.usagemhz.average

Grab appropriate fields from output, use graphing program, etc.

Looks simple Whats going on behind the scenes?


55/92

To get stats, this is what is going on FOR EACH GET-STAT CALL

Retrieve PerformanceManager

QueryPerfProviderSummary $vm Says what intervals are supported

QueryAvailablePerfMetric $vm Describes available metrics

QueryPerfCounter Verbose description of counters

Create PerfQuerySpec Query specification to get the stats

QueryPerf Get stats

Bottom line: The PowerCLI toolkit spares you detailsEasy to use!

PowerCLI Is so easy Why use Java / C#?


56/92

PowerCLI is great for scripting

Stateless

Hides details

But with Java / C#

You can squeeze out more performance!

Much higher scalability

Pseudo code


57/92

Get MOREF

for each Get-Stat {

QueryAvailablePerfMetric();

QueryPerfCounter();

QueryPerfProviderSummary();

create PerfQuerySpec();

QueryPerf();

}

Get MOREF


QueryPerfCounter();



for each Get-Stat {

QueryPerf();

}

PowerCLI Java

perfCounter propertyOf

PerformanceManager

Performance implications: Need to write scalable scripts!


58/92

Entities(cpu.usagemhz.average)

PowerCLI(Time in secs)

Java(Time in secs)

1 VM 9.2 14

6 VMs 11 14.5

39 VMs 101 16

363 VMs 2580 (43 minutes) 50

Java provides opportunities for scalable, ongoing stats collectionLets examine Java code in more detail

A Nave script that works for small environments may not be suitablefor large environments

Highly-tunedJava StatsCollector

GetPerfStats Main method


59/92

Get MOREF

Get CounterIds

QueryAvailablePerfMetric

QueryProviderSummary

create PerfQuerySpec

QueryPerf

Get MOREF


QueryPerfCounter();



for each Get-Stat {

QueryPerf();

}

perfCounter

GetPerfStats


60/92

Get MOREF


QueryPerfCounter();



for each Get-Stat {

QueryPerf();

}

Get MOREF


61/92

Get the entity MOREF


62/92

Get CounterIds


63/92

Get available counterIDsfrom perfCounter property of

PerformanceManager

Map human-readable stat name to counterID(e.g., cpu.usagemhz.average 101)QueryPerf () requires counterID

GetPerfStats


64/92

Get MOREF


QueryPerfCounter();



for each Get-Stat {

QueryPerf();

}


65/92


66/92

Create PerfQuerySpec


67/92

Use wild card

CSV output format

GetPerfStats


68/92

Get MOREF


QueryPerfCounter();



for each Get-Stat {

QueryPerf();

}

QueryPerf


69/92

So, what is Java / C# buying us?

Avoiding redundant work


70/92

Avoiding redundant work

More compact return format (CSV vs. objects)

Low-overhead tracking of ongoing inventory changes

Etc.

If we dig deeper, we can optimize even more

Digging deeper: The PerfQuerySpec architecture

To grab counters:


71/92

To grab counters:

QueryPerf(PerfQuerySpec[] querySpec)

PerfQuerySpec: Specifies which counters to grab

PerfQuerySpec[]: [pQs1, pQs2, pQs3, ]

Array of PerfQuerySpec objects pQs1, pQs2, pQs2

Can grab multiple stats using single QueryPerf call

Entity (host,VM)

Format(CSV,normal)

MetricId StartTime EndTime IntervalID(20s, 300s)

maxSample

Complexities of QueryPerf

How Does vSphere Process QueryPerf(querySpec[])?


72/92

How Does vSphere Process QueryPerf(querySpec[])?

1. vCenter receives queryPerf request with querySpec[]

2. vCenter takes each querySpec one at a time

3. vCenter gets data for each querySpec before processing next one

Options for querySpec[]:

1. 1 entry 1 stat or set of stats for a single entity (e.g., all CPU)

2. Multiple entries. Examples:

Each entry for a different entity

Each entry for a different stat type, same entity

VM1,cpu.* VM2,cpu.* H3,mem.*

VM1,cpu.* VM1,net.* VM1,mem.*

pQs1 pQs2 pQs3

Implications of QuerySpec

Format of QuerySpec Allows Multiple Client Options


73/92

Format of QuerySpec Allows Multiple Client Options

1. Grab each stat one at a time

2. Grab a group of stats per entity at once

3. Grab all stats for all entities at once

4. Grab stats for a subset of entities at once

Some Tradeoffs:

1. Network processing (large result sets vs. small result sets)

2. Client aggregation overhead

3. vCenter processing (Each QueryPerf handled in a single thread)

What about in-guest stats?

Using VIX APIs:


74/92

Using VIX APIs:

Create a script that can get what ever stats you are interested in.

Make the script write the stats to a file.

Copy file from the guest.

Session covering this topic

PPC-15 Guest Operations using VMware VIX APIs and Beyond

Back to the Future (1)

Now I know how to I convert this (many metrics on different charts)


75/92


To This (CPU, Memory, Disk, and Network on the same chart)


76/92

Combining metrics across VMs & Hosts


77/92



78/92

Comparing resource pools


79/92

Use VIX API + vSphere counters to get RP performance data

What about VMs running on a Host?

Memory usage of VMs on a Host


80/92

Summary, Part 1: Some useful Counters to monitor

Resource Metric Host or Description


81/92

VM?CPU Usage Both CPU % used

Ready VM Ready to run, but limit or no available physical CPU

SwapWait VM CPU time spent waiting for host-level swap-in

Memory Swapin,swapinrate

Both Memory ESX host swaps in from disk (per VM, orcumulative over host)

Swapout,swapoutrate

Both Memory ESX host swaps out to disk (per VM, orcumulative over host)

Disk commands Both Operations done during stats refresh interval

totalLatency Host End-to-end disk latency (available for reads & writes)

Usage Both Disk bandwidth utilized (available for reads & writes)

Network Packetsreceived,transmitted

Both Operations done during stats refresh interval

Usage Both Network bandwidth used (available for reads & writes)

For completenessVM memory metrics

Metric Description


82/92

Memory Active (KB) Physical pages touched recently by a virtual machine

Memory Usage (%) Active memory / configured memory

Memory Consumed(KB)

Machine memory mapped to a virtual machine,including its portion of shared pages. Does NOTinclude overhead memory.

Memory Granted (KB) VM physical pages backed by machine memory. Maybe less than configured memory. Includes shared

pages. Does NOT include overhead memory.Memory Shared (KB) Physical pages shared with other virtual machines

Memory Balloon (KB) Physical memory ballooned from a virtual machine

Memory Swapped (KB) Physical memory in swap file (approx. swap out

swap in). Swap out and Swap in are cumulative.

Overhead Memory (KB) Machine pages used for virtualization

Host memory metrics

Metric Description


83/92

p

Memory Active (KB) Physical pages touched recently by the host

Memory Usage (%)* Active memory / configured memory

Memory Consumed(KB) Total host physical memory free memory on host.Includes Overhead and Service Console memory.

Memory Granted (KB) Sum of memory granted to all running virtualmachines. Does NOT include overhead memory.

Memory Shared (KB) Sum of memory shared for all running VMs

Shared common (KB) Total machinepages used by shared pages

Memory Balloon (KB) Machinepages ballooned from virtual machines

Memory Swap Used

(KB)

Physical memory in swap files (approx. swap out

swap in). Swap out and Swap in are cumulative.Overhead Memory (KB) Machine pages used for virtualization

*For a cluster, mem.usage.average = (consumed + overhead)/total mem

Summary, Part 2: Cheat sheet

Rules of Thumb


84/92

Ready Time > 20% sustained is undesirable

Host-level swapping is bad, > 1MBps is especially bad

Disk latencies > 20 ms BAD

Use IOmeter to assess disk bandwidth and latency

Network

run netperf to get network baselines

Summary, Part 3: SDK/API Tips and tricks

Collect static data once


85/92

CounterIDs, metricIDs, MOREFs etc.

Use Views to keep this data up to date.

Reuse PerfQuerySpec as much as possible

Use CSV format Reduces serialization cost and the size of metadata

Choose metrics and query intervals carefully

Query the real-time stats at a slower rate than the refresh rate

Choose correct stats levels

Use parallelism (multi-threaded clients)

Conclusion

vSphere gives a bunch of awesome charts


86/92

If you want to see the data differently, use the API

PowerCLI is great for simple scripts

When designing for scalability, consider Java / C#

Resources

Developer Support

D di d f i i h b ildi l i i S h


87/92

Dedicated support for your organization when building solutions using vSphereAPIs, PowerCLI, vSphere Web Services SDKs and many more VMware SDKs

http://vmware.com/go/sdksupport

PowerCLI Training

2 day instructor led training, 40% lecture, 60% lab

http://vmware.com/go/vsphereautomation

VMware Developer Community

SDK Downloads, Documentation, Sample Code, Forums, Blogs

http://developer.vmware.com

Technology Alliance Partner (TAP) Program

Updated partner benefits

http://www.vmware.com/partners/alliances/programs/

Disclaimer

This session may contain product features that are

tl d d l t


88/92

currently under development.

This session/overview of the new technology representsno commitment from VMware to deliver these features inany generally available product.

Features are subject to change, and must not be included incontracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new technologies or featuresdiscussed or presented have not been determined.

These features are representative of feature areas under development. Feature commitments aresubject to change, and must not be included in contracts, purchase orders, or sales agreements ofany kind. Technical feasibility and market demand will affect final delivery.


89/92

What about VMs across resource pools?


90/92


To This (CPU, Memory, Disk, and Network on the same chart)


91/92



92/92

v sphere apis for performance monitoring

Documents