procedure to follow during vm hung

2
Procedure: VM Hung -Check out the output of 'ps -s | grep < VM's name >' multiple times. -It will display a list of worlds. Check whether for a single world, the state field (3 and 4th field from left) shows "WAIT" and "SCSI" respectively consistently in all the outputs of the above command 1. vmdumper -l to get the world id for the VM 2. vmdumper < wid-for-a-thread > backtrace < -- log backtrace in vmkernel.log 3. vmdumper < wid-for-a-vm > samples_on 4. vmdumper < wid-for-a-vm > unsync < -- check if vmmcores-unsync-< N >.gz is generated 5. vmdumper < wid-for-a-vm > sync < -- check if vmmcores-< N >.gz is generated 6. vmdumper < wid-for-a-vm > vmx < -- check if vmx-zdump.* is generated 7. vmdumper < wid-for-a-vm > vmx_force < -- check if vmx-zdump.* is generated 8. vmdumper < wid-for-a-vm > suspend_vm < -- check if .vmss is generated 9. vmdumper < wid-for-a-vm > samples_off 10. If step 8 fails, will try to collect the suspended state from the vSphere Client or using the vm-support command (see Suspending a virtual machine on ESX/ESXi to collect diagnostic information (2005831) ESXi 5.x section ) 11. Capture host logs with performance snapshots in batch mode using the cmd: esxtop -a -b -d 5 -n 120 > esxtop.csv and vm-support -p 12. Capture Guest OS logs 13. Capture VMRC/Remotemks logs (C:\Users\< USER >\AppData\Local\VMware\ Monsanto Company Confidential Page 1 of 2 Printed: 8/21/2022

Upload: ravi-d

Post on 15-Jan-2016

489 views

Category:

Documents


3 download

DESCRIPTION

VM troubleshooting step during hung

TRANSCRIPT

Page 1: Procedure to Follow During VM Hung

Procedure: VM Hung-Check out the output of 'ps -s | grep < VM's name >' multiple times.

-It will display a list of worlds. Check whether for a single world, the state field (3 and 4th field from

left) shows "WAIT" and "SCSI" respectively consistently in all the outputs of the above command

1. vmdumper -l to get the world id for the VM

2. vmdumper < wid-for-a-thread > backtrace < -- log backtrace in vmkernel.log

3. vmdumper < wid-for-a-vm > samples_on

4. vmdumper < wid-for-a-vm > unsync < -- check if vmmcores-unsync-< N >.gz is generated

5. vmdumper < wid-for-a-vm > sync < -- check if vmmcores-< N >.gz is generated

6. vmdumper < wid-for-a-vm > vmx < -- check if vmx-zdump.* is generated

7. vmdumper < wid-for-a-vm > vmx_force < -- check if vmx-zdump.* is generated

8. vmdumper < wid-for-a-vm > suspend_vm < -- check if .vmss is generated

9. vmdumper < wid-for-a-vm > samples_off

10. If step 8 fails, will try to collect the suspended state from the vSphere Client or using the vm-

support command

(see Suspending a virtual machine on ESX/ESXi to collect diagnostic information (2005831) ESXi 5.x

section )

11. Capture host logs with performance snapshots in batch mode using the cmd:

esxtop -a -b -d 5 -n 120 > esxtop.csv

and

vm-support -p

12. Capture Guest OS logs

13. Capture VMRC/Remotemks logs (C:\Users\< USER >\AppData\Local\VMware\vpx)

14. Migrate all the virtual machine from affected host, except virtual machine which is hung.

15.

a. Enable the VMFS file core dumping:

- The VMFS file dump has to be turned on and the core dump has to be redirected to a file. KB article

that explains the process:

http://kb.vmware.com/kb/2077516

b. Then run this command to get the live dump:

localcli --plugin-dir /usr/lib/vmware/esxcli/int/ debug livedump perform

Monsanto Company Confidential Page 1 of 2 Printed: 4/21/2023

Page 2: Procedure to Follow During VM Hung

16. Crash the host using NMI and collect the dump / ESX host log bundle.

See Using hardware NMI facilities to troubleshoot unresponsive hosts

(http://kb.vmware.com/kb/1014767) Can be done from the HP iLO

Monsanto Company Confidential Page 2 of 2 Printed: 4/21/2023