troubleshooting an integrated xendesktop, pvs and xenserver environment karen sciberras, escalation...

Troubleshooting an integrated XenDesktop, PVS and XenServer environmentKaren Sciberras, Escalation Engineer & Keith Mclaughlin, Lead Escalation EngineerTuesday, May 11th 2010

Agenda

Architecture Overview

Citrix Confidential - Do Not Distribute

XenDesktop Architecture

Architecture Overview Boot Virtual Desktop Clients


VDA Clients

Desktop Delivery Controller

Provisioning

ServiceMAPI

XenServerXAPI

Xen Server

Boot Virtu

al Machine

Boot Virtual Machines

PXE Boot

• Creates and manages the image

• Creates and manages Domain Machine Accounts

• Delivers image to VDA Clients


Provisioning Service Responsibilities

XenDesktop Responsibilities

Creates Desktop Groups and VDA clientsThis is easily obtain using the XenDesktop Setup Wizard

Manage the Virtual Machines by: Handling the Power Management for 'managed' desktops

Maintaining Pools of idle desktops

This is handled by Pool Management Service on DDC

XenDesktop Setup Wizard



Allows an administrator to quickly create a set of virtual desktop

It is installed on the Provisioning Services Server and communicates with:• XenDesktop DDC (Desktop Delivery Controller)

• Provisioning Services

• Virtual Infrasturcture (Citrix XenServer, Microsoft HyperV or VMWare ESX)

What is needed:• Virtual Machine Template on the hosting infrastructure (XenServer, ESX or HyperV)

• A base OS provided by Provisioning Services


Virtual Infrastructure

Provisioning Service (PVS)

Desktop Delivery Controller

XenDesktop Setup Tool

1. Connect to XenServer Pool2. Obtain list of Templates3. Select a Template4. Create X number of VMs

•A MAC address created for each VM•Corresponds to the Virtual NIC of VM

1. Select base OS2. Add Target Device in PVS3. Each Target Device identified by MAC4. PVS adds Target Device to AD5. Obtains list of SIDs for VMs

1. Creates Desktop Group2. Add Virtual Machines to Group3. Maps UUID to SID

Creating Machine Accounts in the domain


VDA Clients


Provisioning

ServiceMAPI

Xen Server

Domain ControllerSQL Database

Create VDA Target

SQL DB

VDA1VDA1

Creat

e VDA T

arge

t

Add VDA

Target

Pool Management Service

• Feature that controls the power states of ‘machines’

• Service that contains the logic for the power state transitions

• We have three plug-ins which all talk to hypervisors:• XenServer• VMware ESX• Microsoft Hyper-V

• Idle pool• Powers on machines in advance so that users don’t have to wait for them• Powers down machines when they are not required

What is Pool Management?

Desktop Life cycles

Off

Free

In Use Disconnected

Tainted

Idle pool

Log on

Disconnect

Reconnect

Log off

Suspended

Log off

7 Minutes

5 Minutes

Taint Action[immediate]

PooledAssigned

Some hidden dials that can be tweaked (with care!!)

LogoffActionDelay (7 mins) / DisconnectActionDelay (5 mins)Lets the user change their mind without waiting for a new VM to boot...

MUST be > RegistrationTimeout

RegistrationTimeout (3 mins)How long a desktop gets to register (before showing error)

RegistrationForceShutdownTimeout (3hrs)How long a desktop gets to register (before we force a power-off)

ShutdownTimeout (10 mins)How long we give a VM to shutdown gracefully, before pulling the plug

General Issues and Troubleshooting


Known Issues

• Provisioning Services and Antivirus

• Configuration for PVS to handle machine passwords

• Virtual machines are not mapped to AD account names

• Hypervisor is Overwhelmed

Provisioning Services and Antivirus

• Whitelist PVS Filters

• Do not scan system drive

• Antivirus updates

Provisioning Services and Antivirus best practices

• Limit Antivirus updates to the target.

• Disable scanning of the write cache location especially if caching on server.

• Do not scan I/O in real time.

• Exclude scanning low level PVS drivers• BNNS.sys, BNNF.sys, BNPort.sys, and bnistack.sys.BNDevice.exe

• Install Antivirus before Provisioning Services test updates on a staging image.

• More information can be found:• http://support.citrix.com/article/ctx124185

Known Issues





Configuration for PVS to handle machine passwords

Issue: Users were able to log into their machines but now cannot.

Administrator is able to log into the vdisk in private mode.

Resolution: Points to the AD machine account not being setup properly in AD

AD machine account password has expired

Group Policy setting: Computer Configuration -> Windows Settings -> Security Settings -> Local Policies -> Security Options

Domain member: Disable machine account password changes: Enable

Known Issues





Identifying Virtual Machines

Hosting Infrastructure Identifiers look like this:Microsoft path/to/VM

VMWare Name-UniqueID

XenServer GUID

XenDesktop usesActive Directory Machine Identity [Machine SID]

We store this mapping as VM meta-data

VM Meta Data

CTXGuestOSID This is the Guest OS SID.

Data is written by the DDC Farm Master.

Referenced by the DDC to keep the virtual machine and the Guest SID aligned for the PVS service.

CTXGuestMGTInfoData is written by the DDC Farm Master.

Keeps the virtual machine and the Pool management service aligned.

This data identifies which pool a machine belongs to

Allows the DDC to query power state of the virtual machines in that pool.

Virtual machines not mapped to AD account names

VM in XenServerDesktop in Active

Directory

Problems [mouse over]

Known Issues





Hypervisor Overwhelmed

By default, the Pool Management Service will start-up 10% of the default pool size.

In large environments, this may be more than the hypervisor/ Provisioning server will be able to handle.

To prevent this, the pool management can be configured to stagger the start-up of the Virtual Machine.

This is configured in the config file of the Pool Management Service

Hypervisor Overwhelmed

Open C:\Program Files\Citrix\VmManagement\CdsPoolMgr.exe.config

Add setting, for example:

<?xml version="1.0" encoding="utf-8" ?><configuration> <appSettings> <add key="LogToCdf" value ="1"/> <add key="LogFileName" value ="C:\cdslogs\VMManager.log"/> <add key="LogDebug" value="1"/> <add key="MaximumTransitionRate" value="20"/> </appSettings></configuration>

Restart the Pool Management Service

Tools Used

CDF Tracing Pool management Logs

Setup Logs

PVS Soap Logs

Network Tracing e.g. Wireshark

Enable CDF Tracing on XenDesktop

Only available in XenDesktop

CDF trace information can be written to logfiles in plain text

Need to edit a text file to enable

Can also use CDF Control (CTX111961)

Modules that can log

Filename Path Location

CdsImaProxy.exe.config Citrix\Desktop Delivery Controller Desktop Delivery Controller

CdsPoolMgr.exe.config Citrix\VmManagement Desktop Delivery Controller

SetupToolApplication.exe.config Citrix\XenDesktop Setup Wizard Provisioning Services

How to Enable Logging

• Manually create a directory where to store the log

• Edit the config file with the following values: Configure the value LogToCDF from 0 to 1<add key=“LogToCDF” value=“1” /> Add the location where log file will be stored:<add key=“LogFileName” value=“<location.log>”

• Restart the service

Article CTX117452 provides further information

INF:(9/30/2009 12:31:02 PM):Retrieving poolName for Pool at address http://10.90.144.102. INF:(9/30/2009 12:31:02 PM):Pool name for PoolMaster at address http://10.90.144.102 is Karen XenEnv INF:(9/30/2009 12:31:02 PM):Cloned Machine XDVDA1 (UUID : 27568153-1d53-c7c1-bcbb-438a761ff565) ,

MAC : 92:45:30:22:a8:bf INF:(9/30/2009 12:31:02 PM): XenManager.CloneVm method. - ExitINF:(9/30/2009 12:31:02 PM):Vm cloning for desktop XDVDA1 was successful.INF:(9/30/2009 12:31:02 PM):Vm disk provisioning for desktop XDVDA1 beginning.INF:(9/30/2009 12:31:02 PM):Calling ProvisionDisk of diskManager with args: clonedVmName = XDVDA1.INF:(9/30/2009 12:31:02 PM):-> PVSmanager.ProvisionDisk - EntryINF:(9/30/2009 12:31:02 PM):Entering IDiskManager.ProvisionDisk method.INF:(9/30/2009 12:31:02 PM):The VM XDVDA1 will be provisioned using diskTemplate XP machine.INF:(9/30/2009 12:31:02 PM):Adding new machines to provisioning server. INF:(9/30/2009 12:31:02

PM):Adding new machines to provisioning server.INF:(9/30/2009 12:31:02 PM):-> PVSmanager50.GetSharedDiskTemplates - EntryINF:(9/30/2009 12:31:02 PM):-> PVSManager50.GetSites - EntryINF:(9/30/2009 12:31:02 PM):Adding site XD with Id 05d0bb98-2bfa-4663-8b42-1c3c6085688aINF:(9/30/2009 12:31:02 PM):All 1 sites successfully retrieved.

XenDesktop Setup Wizard Logs

Provisioning Services: Enable SOAP Logs

Provisioning Services: Adding Device to Farm

010-04-13 10:49:47,609 [12] DEBUG SoapServer.ServiceMain - ** Start Command 'Add Device'2010-04-13 10:49:47,625 [12] DEBUG SoapServer.ServiceMain - Username: XDS\Administrator2010-04-13 10:49:47,625 [12] DEBUG SoapServer.ServiceMain - deviceName=XD3VDA12010-04-13 10:49:47,625 [12] DEBUG SoapServer.ServiceMain - collectionId=f368555d-4d66-487c-ad1c-c1bdb04a9bbe2010-04-13 10:49:47,625 [12] DEBUG SoapServer.ServiceMain - deviceMac=7e-d8-03-55-99-e62010-04-13 10:49:47,625 [12] DEBUG SoapServer.ServiceMain - description=2010-04-13 10:49:47,625 [12] DEBUG Mapi.CommandType - in CommandAddDevice.Execute 2010-04-13 10:49:47,625 [12] DEBUG Mapi.Command - Add to table Device2010-04-13 10:49:47,625 [12] DEBUG Mapi.Command - sqlStatement = <INSERT INTO [Device] ([deviceId],[deviceName],[collectionId],[deviceMac],[description]) SELECT DISTINCT @v1,@v2,@v3,@v4,@v5>2010-04-13 10:49:47,625 [12] DEBUG Mapi.Command - parameter values are <@v1 = 235ab677-f89a-4339-9efe-317c927054c5, @v2 = XD3VDA1, @v3 = f368555d-4d66-487c-ad1c-c1bdb04a9bbe, @v4 = 7ED8035599E6, @v5 = >2010-04-13 10:49:47,625 [12] DEBUG SoapServer.ServiceMain - Command 'Add Device' returned:2010-04-13 10:49:47,625 [12] DEBUG SoapServer.ServiceMain - return code: 02010-04-13 10:49:47,625 [12] DEBUG SoapServer.ServiceMain - ** End Command

Provisioning Services: SOAP Logs - Adding Device to Domain

2010-04-13 10:49:48,859 [12] DEBUG SoapServer.ServiceMain - Username: XDS\Administrator

2010-04-13 10:49:48,859 [12] DEBUG SoapServer.ServiceMain - MacroSet 'Set Device'

2010-04-13 10:49:48,859 [12] DEBUG SoapServer.ServiceMain - deviceId=235ab677-f89a-4339-9efe-317c927054c5

2010-04-13 10:49:48,859 [12] DEBUG SoapServer.ServiceMain - adTimestamp=1271152188

2010-04-13 10:49:48,859 [12] DEBUG SoapServer.ServiceMain - adSignature=8899

2010-04-13 10:49:48,859 [12] DEBUG Mapi.CommandType - authGroups = <4c5b1faa-0bc7-478a-a45c-50f3e72d4549>

2010-04-13 10:49:48,859 [12] DEBUG Mapi.CommandType - parameters preVal = <deviceId=235ab677-f89a-4339-9efe-317c927054c5>

2010-04-13 10:49:48,859 [12] DEBUG Mapi.CommandType - record fields preConv = <adTimestamp=1271152188, adSignature=8899>

2010-04-13 10:49:48,859 [12] DEBUG Mapi.CommandType - pre-ValidParms = <deviceId=235ab677-f89a-4339-9efe-317c927054c5>

2010-04-13 10:49:48,859 [12] DEBUG Mapi.CommandType - post-ValidParms = <deviceId=235ab677-f89a-4339-9efe-317c927054c5>

2010-04-13 10:49:48,859 [12] DEBUG Mapi.CommandType - pre-ConvertFields = <adTimestamp=1271152188, adSignature=8899>

2010-04-13 10:49:48,859 [12] DEBUG Mapi.CommandType - post-ConvertFields = <adTimestamp=1271152188, adSignature=8899>

2010-04-13 10:49:48,859 [12] DEBUG Mapi.Command - Set in table Device

2010-04-13 10:49:48,859 [12] DEBUG Mapi.Command - sqlStatement = <UPDATE [Device] SET [adTimestamp] = @v1,[adSignature] = @v2 FROM

[Device] d WHERE d.[deviceId] = @v3>

2010-04-13 10:49:48,859 [12] DEBUG Mapi.Command - parameter values are <@v1 = 1271152188, @v2 = 8899, @v3 = 235ab677-f89a-4339-9efe-317c927054c5>

2010-04-13 10:49:48,859 [12] DEBUG SoapServer.ServiceMain - Command 'MacroSet' returned:

2010-04-13 10:49:48,859 [12] DEBUG SoapServer.ServiceMain - return code: 0

Packet Sniffers

SUM306: Citrix Provisioning Services stream process architecture and advanced troubleshooting Today at 2:30-3:20.

troubleshooting an integrated xendesktop, pvs and xenserver environment karen sciberras, escalation...

Documents

distributecitrix confidential

domaincitrix confidential

ddccitrix confidential

vda clientscitrix confidential

virtual machine template

power management

set of virtual desktopit

vda clientsthis