Download - VSP1999 esxtop for Advanced Users
![Page 1: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/1.jpg)
VSP1999esxtop for Advanced Users
Name, Title, Company
![Page 2: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/2.jpg)
2
Disclaimer
This session may contain product features that are currently under development.
This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product.
Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
Technical feasibility and market demand will affect final delivery.
Pricing and packaging for any new technologies or features discussed or presented have not been determined.
![Page 3: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/3.jpg)
3
Before we dive in…
![Page 4: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/4.jpg)
4
vSphere Performance Management Tools (1 of 2)
vCenter Alarms• Relies on static thresholds
• Alarm trigger may not always indicate an actual performance problem
vCenter Operations• Aggregates metrics into workload,
capacity and health scores
• Relies on dynamic thresholds
vCenter Charts• Historical trends
• Post mortem analysis, comparing metrics
![Page 5: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/5.jpg)
5
vSphere Performance Management Tools (2 of 2)
esxtop/resxtop• For live troubleshooting and root cause analysis
• esxplot, perfmon and other tools can be used for offline analysis
![Page 6: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/6.jpg)
6
Performance Snapshot
For complicated problems• Technical support may ask you for a performance snapshot for offline
analysis
![Page 7: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/7.jpg)
7
About This Talk
This talk will focus on the esxtop counters using illustrative examples
esxtop manual:• http://www.vmware.com/pdf/vsphere4/r41/vsp_41_resource_mgmt.pdf
Interpreting esxtop statistics• http://communities.vmware.com/docs/DOC-11812
Previous vmworld talks:• VMworld 2008 - http://vmworld.com/docs/DOC-2356
• VMworld 2009 - http://vmworld.com/docs/DOC-3838
• VMworld 2010 - http://www.vmworld.com/docs/DOC-5101
![Page 8: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/8.jpg)
8
esxtop Screens
Screens• c: cpu (default)• m: memory• n: network
• d: disk adapter
• u: disk device (added in ESX 3.5)
• v: disk VM (added in ESX 3.5)• i: Interrupts (added in ESX 4.0)
• p: power management (added in ESX 4.1) VMkernel
CPUScheduler
MemoryScheduler
VirtualSwitch
vSCSI
c, i, p m d, u, vn
VM VM VMVM
![Page 9: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/9.jpg)
9
New counters in ESX 5.0
![Page 10: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/10.jpg)
10
vCPU and VM Count
World, VM and vCPU count
![Page 11: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/11.jpg)
11
VMWAIT
%WAIT - %IDLE
More about this later…
![Page 12: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/12.jpg)
12
CPU Clock Frequency in Different P-states
CPU clock frequency in
different P-states
P-states are visible to ESX only when power management setting in the BIOS is set to “OS Controlled”
More about this later…
![Page 13: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/13.jpg)
13
Failed Disk IOs
Failed IOs are now accounted separately from successful IOs
![Page 14: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/14.jpg)
14
VAAI: Block Deletion Operations
New set of VAAI stats for tracking
block deletion
VAAI : vStorage API for Array Integration
![Page 15: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/15.jpg)
15
Low-Latency Swap (Host Cache)
Low-Latency (SSD) Swap
![Page 16: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/16.jpg)
16
Understanding CPU counters
![Page 17: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/17.jpg)
17
CPU State Times
IDLE
WAIT
SWPWT blocked
VMWAIT
RUNRDY
MLMTD
Elapsed Time
CSTP
Guest I/O
![Page 18: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/18.jpg)
18
CPU Usage Accounting
USED = RUN
RUN
SYS
OVRLPSystem Service
USED could be < RUN if the CPU is not running at its rated clock frequency
+ SYS - OVRLP
![Page 19: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/19.jpg)
19
Impact of P-States
P-States %RUN %UTIL %USED
P0 (2400 Mhz) 100% 100% 100%
P1 (1700 Mhz) 100% 100% 70%
P2 (1200 Mhz) 100% 100% 50%
P3 (800 Mhz) 100% 100% 33%
%USED: CPU usage with reference to rated base clock frequency%UTIL: CPU utilization with reference to current clock frequency%RUN: CPU occupancy time
![Page 20: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/20.jpg)
20
Factors That Affect VM CPU Usage Accounting
Chargeback• %SYS time
CPU frequency scaling• Turbo boost
• USED > (RUN – SYS)
• Power management• USED < (RUN – SYS)
Hyperthreading
![Page 21: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/21.jpg)
21
Poor performance due to power management
![Page 22: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/22.jpg)
22
CPU Usage: With CPU Clock Frequency Scaling
VM is running all the time but uses only 75% of the clock frequency. Power savings
enabled in BIOS.
![Page 23: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/23.jpg)
23
Poor performance due to core sharing
![Page 24: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/24.jpg)
24
Hyperthreading
PCPU
Core
HT Off HT On
ESX scheduler tries to avoid sharing the same core
![Page 25: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/25.jpg)
25
CPU Usage: Without Core Sharing
Two VMs running on different cores
USED is > 100 due to Turbo Boost
![Page 26: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/26.jpg)
26
CPU Usage: With Core Sharing
Two VMs sharing the same core
%LAT_C counter shows the CPU
time unavailable to due to core sharing
![Page 27: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/27.jpg)
27
Performance Impact of Swapping
![Page 28: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/28.jpg)
28
Performance Impact of Swapping
Some swapping activity
Time spent in blocked state due
to swapping
![Page 29: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/29.jpg)
29
How to identify storage connectivity issues
![Page 30: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/30.jpg)
30
NFS Connectivity Issue (1 of 2)
I/O activity to NFS datastore
System time charged for NFS activity
![Page 31: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/31.jpg)
31
NFS Connectivity Issue (2 of 2)
VM blocked, connectivity lost
to NFS datastore
No I/O activity on the NFS datastore
VM is not using CPU
![Page 32: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/32.jpg)
32
Poor performance during snapshot revert
![Page 33: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/33.jpg)
33
Snapshot Revert
Reads in MB from VM check point file
Not accounted in VM disk I/O traffic
But can be seen in adapter view
![Page 34: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/34.jpg)
34
Wide-NUMA behavior in ESX 5.0
![Page 35: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/35.jpg)
35
Wide-NUMA Support in ESX 5.0
2 x 16G NUMA Nodes
24G vRAM exceeds one NUMA node
1 home NUMA node assigned
1 vCPU VM
![Page 36: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/36.jpg)
36
Wide-NUMA Support in ESX 5.0
8 vCPUs, exceeds one NUMA node
2 x 16G NUMA Nodes
24G vRAM exceeds one NUMA node 2 Home NUMA
nodes assigned
![Page 37: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/37.jpg)
37
Network packet drops due to CPU resource issue
![Page 38: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/38.jpg)
38
Network Packet Drops
Max CPU limited
Excessive Ready time
Packet drops at the vSwitch
![Page 39: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/39.jpg)
39
Understanding esxtop disk counters
![Page 40: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/40.jpg)
40
Disk I/O Latencies
ApplicationGuest OS
ESX StorageStack
VMM
Driver
KAVG
iostat/perfmon
DAVG
GAVG
QAVG
KAVG = GAVG – DAVG
Array SPFabric
vSCSI
HBA
Time spent in ESX storage stack is minimal, for all practical purposesKAVG ~= QAVG
In a well configured system QAVG should be zero
![Page 41: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/41.jpg)
41
Disk I/O Queuing
GQLEN – Guest QueueAQLEN – Adapter QueueWQLEN – World QueueD(/L)QLEN – LUN QueueSQLEN – Array SP Queue
DQLEN
AQLEN
SQLEN
GQLEND(/L)QLEN can change dynamically when SIOC is enabled
Reported in esxtop
ApplicationGuest OS
VMM
Driver
Array SPFabric
vSCSI
HBA
WQLENESX Storage
Stack
![Page 42: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/42.jpg)
Max IOPS = Max Outstanding IOs / Latency
For example, with 64 outstanding IOs and 4msec average latency
Max IOPS = 64/4ms = 16,000
![Page 43: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/43.jpg)
43
Identifying Queue bottlenecks
![Page 44: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/44.jpg)
44
Disk I/O Queuing – Device Queue
Device Queue length, modifiable via driver
parameter
IO commands in Flight
IO commands waiting in
Queue
![Page 45: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/45.jpg)
45
Disk I/O Queuing – World Queue
World ID
World Queue Length – modifiable
Disk.SchedNumRequestOutstanding
![Page 46: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/46.jpg)
46
Device Queue Full
KAVG is non-zero
Queuing issue
LUN Queue
depth is 32
32 IOs in flight and
32 Queued
![Page 47: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/47.jpg)
47
Disk I/O Queuing – Adapter Queue
Different adapters have different queue
size
Adapter Queue can come into play
if the total outstanding IOs
exceeds the adapter queue
![Page 48: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/48.jpg)
48
A few takeaways…
![Page 49: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/49.jpg)
49
Takeaways esxtop is great for troubleshooting a diverse set of problems You can do root-cause analysis by co-relating statistics from
different screens Good understanding of the counters is essential for accurate
troubleshooting esxtop is not designed for performance management There are various other tools for vSphere performance
management
![Page 50: VSP1999 esxtop for Advanced Users](https://reader033.vdocument.in/reader033/viewer/2022061514/5681641a550346895dd5d2e6/html5/thumbnails/50.jpg)
50
Thank You!