citrix xenserver 5.5 troubleshooting

Citrix® XenServer Webcast 26. März 2010 Thomas Krampe Citrix Technology Professional Advanced Troubleshooting

Upload: thomas-krampe

Post on 20-May-2015




6 download


Citrix XenServer 5.5 Troubleshooting Deep Dive.


Page 1: Citrix XenServer 5.5 Troubleshooting

Citrix® XenServer

Webcast 26. März 2010

Thomas KrampeCitrix Technology Professional

Advanced Troubleshooting

Page 2: Citrix XenServer 5.5 Troubleshooting

204/19/10 (c) 2009 Thomas Krampe




Birth and Death of a VM

Object modell & CLI

Unattended Setup & Configuration

Question & Answers

Page 3: Citrix XenServer 5.5 Troubleshooting

304/19/10 (c) 2009 Thomas Krampe



XenServer Architecture

Xen Domain0

Guest OS Internals


Storage Manager


GUI (XenCenter)

Page 4: Citrix XenServer 5.5 Troubleshooting

404/19/10 (c) 2009 Thomas Krampe

XenServer Architecture

Virtualized HardwareVirtualized Hardware








Page 5: Citrix XenServer 5.5 Troubleshooting

504/19/10 (c) 2009 Thomas Krampe

Xen Domain0

Dom0 kernel and initrd passed on start up

Dom0 has direct access to HW




[Ctrl-A] 3 times will trap into the Xen debug console

Page 6: Citrix XenServer 5.5 Troubleshooting

604/19/10 (c) 2009 Thomas Krampe

Guest OS internals

PV Guest

Modified Kernel (“Xenified”)

Kernel name will have “xs” somewhere in the filename

Guest agent reports VM stats

Use pygrub bootloader

HVM Guest (Windows)

Unmodified Kernel

Need PV drivers

Event Channel, Network and Block

Guest agent uses WMI

Page 7: Citrix XenServer 5.5 Troubleshooting

704/19/10 (c) 2009 Thomas Krampe

XAPI /opt/xensource/bin/xapi

Service, autostarted on boot

Implements the Xen API spec

Talks to Storage Manager (sm) for SR control operations

Stores configuration in sqlite DB


sqlite3 /var/xapi/state.db

DB is replicated on all hosts in a pool

Sets up SR defaults on firstboot

Failing firstboot scripts might cause issues

/var/xapi/generate-* and /var/xapi/firstboot-*

Defaults from /etc/xensource-inventory

Page 8: Citrix XenServer 5.5 Troubleshooting

804/19/10 (c) 2009 Thomas Krampe

Storage Manager

Manages SR control operations to various backends


Plugin model

One plugin for each SR type


Error codes listed in


xe sm-list lists all supported drivers

Page 9: Citrix XenServer 5.5 Troubleshooting

904/19/10 (c) 2009 Thomas Krampe


Local or Remote

Talks to XAPI

Bash auto completion

xe.exe (for Windows)

xe help or xe help --all

Page 10: Citrix XenServer 5.5 Troubleshooting

1004/19/10 (c) 2009 Thomas Krampe


Uses Microsoft .NET framework

Uses https/xml-rpc to communicate with XAPI

VNC Client for guest frame buffer

VNC traffic is tunneled over https

RDP client for Windows guests

Page 11: Citrix XenServer 5.5 Troubleshooting

1104/19/10 (c) 2009 Thomas Krampe


TroubleshootingServer Install

Useful Linux Commands

Server Diagnostics & Inventory

Saving System State

XS Server Log Files

XenSource Agent

XenCenter Logging


Page 12: Citrix XenServer 5.5 Troubleshooting

1204/19/10 (c) 2009 Thomas Krampe

Server Install After Product CD Boots

Alt-F2 displays Linux login prompt

Login: root (no password required)

Alt-F3 displays install event log

Alt-F1 displays original install menu

Support (creates /tmp/support.tar.bz2)

Support tarball is created in /tmp on install error

Support tarball is saved in /root after successful install

Page 13: Citrix XenServer 5.5 Troubleshooting

1304/19/10 (c) 2009 Thomas Krampe

Useful Linux Commands


List Disks detected

fdisk –l


Show NIC configuration

ifconfig eth0 [eth1 ..]

ifup eth0

TCP/IP Tools ping, route -n

dhclient eth0

Page 14: Citrix XenServer 5.5 Troubleshooting

1404/19/10 (c) 2009 Thomas Krampe

Useful Linux Commands

Process Management

ps –Af



Page 15: Citrix XenServer 5.5 Troubleshooting

1504/19/10 (c) 2009 Thomas Krampe

Server Diagnostics and Inventory

Server Diagnostics


Run on the XS Console

Records all log files

Writes compressed file for submission to L2 support


Page 16: Citrix XenServer 5.5 Troubleshooting

1604/19/10 (c) 2009 Thomas Krampe

Server Diagnostics and Inventory

Server Diagnostics

Automatic Upload for Support

xe host-bugreport-upload

Can be run from admin system

Runs xen-bugtool and uploads to XS Support

Note unique ID of report for support

Page 17: Citrix XenServer 5.5 Troubleshooting

1704/19/10 (c) 2009 Thomas Krampe

Server Diagnostics and Inventory

Server Inventory | less

Interactive script that captures all non-log state

Hardware devices & Device drivers

Storage repository and VM configurations

Page 18: Citrix XenServer 5.5 Troubleshooting

1804/19/10 (c) 2009 Thomas Krampe

Saving System State

xen-bugtool archives state and configuration files

Network config

Install options

Install logs

Storage Repository metadata

VM configs


Page 19: Citrix XenServer 5.5 Troubleshooting

1904/19/10 (c) 2009 Thomas Krampe

XS Server Log Files

XenSource XAPI Agent


Contains other system logging


Xen/Domain 0 Xen Boot Messages

xe host-dmesg

Domain 0 Boot Messages


Page 20: Citrix XenServer 5.5 Troubleshooting

2004/19/10 (c) 2009 Thomas Krampe

Debug Logging for XAPI Agent

Enable debug logging for all subsystems

xe log-set-output level=debug output=file:/tmp/debug-all

Debug for only a certain subsystem

xe log-set-output level=debug key=storage output=file:/tmp/storage-


List Available Subsystems

xe log-get-keys

Restore normal logging xe log-set-output level=debug output=nil

xe log-set-output level=debug key=storage output=nil

Page 21: Citrix XenServer 5.5 Troubleshooting

2104/19/10 (c) 2009 Thomas Krampe

XenSource Agent

XS Agent runs on top Xen Hypervisor API and XenStore

Can be restarted without harming VMs or other members

of the pool

service xapi restart

If the UI seems to get stuck or goes into a weird state, restart

the agent and reconnect the UI

Another simular command is xe-toolstack-restart

Page 22: Citrix XenServer 5.5 Troubleshooting

2204/19/10 (c) 2009 Thomas Krampe

XenCenter Logging


{user home}\Application Data\XenSource\XenCenter\logs\



Log files will get rolled


Increase Log Level C:\Program Files\XenSource\XenCenter directory

<level value=“DEBUG” />

Page 23: Citrix XenServer 5.5 Troubleshooting

2304/19/10 (c) 2009 Thomas Krampe


xen-bugtoolgenerates XE Server diagnostic file for submission to Technical Support Representative.

xe host-bugreport-upload

runs xen-bugtool and uploads the result to XenSource support.

xe host-bugtool

can be used to save Domain 0 state, to assist during emergency recovery

Use Linux commands (network, storage, modules) to troubleshoot server install.

Page 24: Citrix XenServer 5.5 Troubleshooting

2404/19/10 (c) 2009 Thomas Krampe

Birth and Death of a VM

Birth and Death of a VM

VM Installation Internals

Behind the Scenes

Debugging VMs

Page 25: Citrix XenServer 5.5 Troubleshooting

2504/19/10 (c) 2009 Thomas Krampe

VM Installation Internals

Debian Sarge/Etch

Root file system is extracted into the VDI

Debian Xen kernel

RHEL (3.x,4.x,5) / SLES (9.x, 10.x)

Uses regular repositories

RHEL / SLES Xen kernel

Windows (all) Installs from CD/ISO bits

Unmodified Windows kernel

Page 26: Citrix XenServer 5.5 Troubleshooting

2604/19/10 (c) 2009 Thomas Krampe

Behind the Scenes

Paravirtualized VMs use pygrub

Simple python based bootloader for Xen

Finds kernel and initrd from VM’s boot device

CPU then jumps to this kernel to start this VM

HVM VMs use hvmloader

hvmloader creates a virtual BIOS and jumps to it. This is like a

BIOS in a physical machine

qemu-dm does device emulation, provides disks, NICs, serial

ports, USB devices etc to this VM

Follows standard boot process to boot from disk

Page 27: Citrix XenServer 5.5 Troubleshooting

2704/19/10 (c) 2009 Thomas Krampe

Debugging Windows VMs

Have client Windows machine ready, install WinDbg

Add /DEBUGPORT=com1 to boot.ini in VM, shut down VM

xe vm-param-add uuid=<VM UUID> param-name=other-config:hvm_serial=tcp:<client IP>:7001

sockpipe my_pipe 7001 on client machine

windbg -k com:pipe,port=\\.\pipe\my_pipe,resets=0

Start VM

Refer to WinDbg manual for commands

Page 28: Citrix XenServer 5.5 Troubleshooting

2804/19/10 (c) 2009 Thomas Krampe

Object Model and CLI

Object Model and CLI

Object Model

CLI Command Format


Constarining Information

Interesting Commands

Looking inside a log file

Page 29: Citrix XenServer 5.5 Troubleshooting

2904/19/10 (c) 2009 Thomas Krampe

Object Model

Page 30: Citrix XenServer 5.5 Troubleshooting

3004/19/10 (c) 2009 Thomas Krampe

CLI Command Format

Object commands

xe <class>-<TAB>

Object Param commands xe <class>-param-<TAB>

Some fields are marked <expensive

Data Types Single Valued/Multi-Valued

Read Only/Read-Write


Page 31: Citrix XenServer 5.5 Troubleshooting

3104/19/10 (c) 2009 Thomas Krampe


Setting Map values[root@localhost] xe vm-param-set uuid=VM uuid other-config:foo=baa

Getting Map values

[root@localhost] xe vm-param-get uuid=VM-uuid param-name=other-config mac_seed: 6fjafhahffhadfbadfad

[root@localhost] xe vm-param-get uuid=VM-uuid param-name=other-config param-key=mac_seed 6fjafhahffhadfbadfad

Page 32: Citrix XenServer 5.5 Troubleshooting

3204/19/10 (c) 2009 Thomas Krampe

Constraining Information


vm-list HVM-boot-policy="BIOS order" power-state=halted

Parameterization xe vm-list params=name-label

Minimal Output xe vm-list --minimal

Parameterization and filtering can be combined

Page 33: Citrix XenServer 5.5 Troubleshooting

3304/19/10 (c) 2009 Thomas Krampe

Interesting Commands

Wait for a VM to run

xe event-wait class=vm name-label=myvm power-state=running

List of CPUs on a host xe host-cpu-list

CPU Flags must match across all hosts in pool

List of subsystems xe log-get-keys

Can be used to isolate subsystem logs

Page 34: Citrix XenServer 5.5 Troubleshooting

3404/19/10 (c) 2009 Thomas Krampe

Looking inside a log file

Each Entry looks like

[timestamp][hostname][level] [ref-1] [(num-1) unix_rpc] text-1 text2 text3….

[timestamp] Timestamp of log entry

[hostname] Host name (in a pool)

[level] Debug level

[ref-1] ID of the task on whose behalf this is created

num-1 Thread ID, local to the current process

unix_rpc Unix Domain Socket (as opposed to inet_rpc=TCP)

text-1 Subsystem

text2/text3 more information on action performed

Page 35: Citrix XenServer 5.5 Troubleshooting

3504/19/10 (c) 2009 Thomas Krampe

Looking inside a log file

xensource.log[20100209 16:58:28.688|debug|localhost|3078703|SR scanner D:dfd9ef0b708d|helpers] login done[20100209 16:58:28.921|debug|localhost|3078706 unix-RPC|SR.scan R:1ac783270499|taskhelper] task destroyed[root@localhost log]#

Xenstored-access.log[20100209 16:49:05.381] D35 write data/meminfo_free 551040[20100209 16:49:05.381] D35 write data/meminfo_total 1035760[20100209 16:49:05.381] D35 write data/updated 1

messagesFeb 9 17:04:58 fragmgrid2001 xapi: [ info|localhost|3078916|SR scanner D:dfd9ef0b708d|dispatcher] spawning a new thread to handle the current taskFeb 9 17:04:58 fragmgrid2001 xapi: [ info|localhost|3078917 unix-RPC|session.slave_login D:58b671575b1e|xapi] Session.create trackid=ea252f3e59cc6f0d2d3de1345967d54b pool=true uname= is_local_superuser=true auth_user_sid=Feb 9 17:04:58 fragmgrid2001 xapi: [ info|localhost|3078919 unix-RPC|dispatch:SR.scan D:64e76e160ebc|taskhelper] task SR.scan R:98637b78c34f (uuid:6e8469dc-724d-0260-a584-44d36bca2953) created (trackid=ea252f3e59cc6f0d2d3de1345967d54b) by task D:dfd9ef0b708d

Page 36: Citrix XenServer 5.5 Troubleshooting

3604/19/10 (c) 2009 Thomas Krampe

Unattend Installation

Possible over TFTP - PXE

use a single xml configuration file

Page 37: Citrix XenServer 5.5 Troubleshooting

3704/19/10 (c) 2009 Thomas Krampe

Questions & Answers