troubleshooting xendesktop 5 deployments

Troubleshooting XenDesktop 5 DeploymentsBaptiste Duflos, Escalation Manager & Ken Baldwin, Escalation EngineerTuesday, May 24th 2011

Introduction and objectives

Case study for MCS fails to create pooled machines

• Fully integrated provisioning into the XenDesktop 5 console

• Desktop lifecycle support and image roll-back capability

• Leverages and supports all 3 major Hypervisors

Machine Creation Services introduces:

Citrix Confidential - Do Not Distribute

VM

Master Disk

VM VM

Diff DiskId Disk

Diff DiskId Disk

Diff DiskId Disk

Storage

One copy of the base image shared by all VMs

One copy of the base image shared by all VMs

Pooled image will reset back to initial state after reboot

Pooled image will reset back to initial state after reboot

Persistent Identity disk provides AD computer account info

Persistent Identity disk provides AD computer account info

Each VM consists of a Difference disk and an Identity disk

Each VM consists of a Difference disk and an Identity disk

VMs can be created in pooled or private mode

VMs can be created in pooled or private mode

BrokerBroker

Hypervisorsand

Storage

Hypervisorsand

Storage

HCL

Machine Identity Service

Data Access

Active Directory

Active Directory

AD Identity ServiceData

Access

HypervisorsHypervisors

HCL

Machine Creation Service

Data AccessSQL

Infrastructure Service

Host Service

ConfigurationService


Machine Creation ServiceMachine Creation Service

Reproducing the error: failed to create Catalog

Data Access

Storage

HCL HypervisorsHypervisors

Network

SQL

The Catalog could not be loaded due to the following errors:There are no master images associated with this Catalog

See CTX127068 for resolutions to this problem


http://support.citrix.com/article/CTX127068

Troubleshooting Methodology – initial first look


• Verify the Certs and Proxy.xml - CTX125578

• Validate the Hypervisor permissions - CTX127546• Configure and test

multiple host connections

• Validate the Hypervisor is configured correctly

• Check the image

• Try using another virtual image for creation

• Check the master image snapshot wasn’t deleted

• Check permissions if storage path is not using local attached storage

Troubleshooting Methodology – Logs and Traces

• Service Logging - CTX127492

• SQL Trace - CTX127257

• CDF Control - CTX111961SQL





CitrixMachineCreationService:-> Citrix.XDServiceBase.LogicBase.GetRemoteServiceInstances - EntryCitrixMachineCreationService:Returning cached service instancesCitrixMachineCreationService: Citrix.XDServiceBase.LogicBase.GetRemoteServiceInstances - ExitCitrixMachineCreationService:Sorting the ServiceInstances.CitrixMachineCreationService:Using the next service instance http://xd5-lab.local/Citrix/HostingUnitService/IServiceAPI CitrixMachineCreationService:Conversion error in Property Resolver. Exception is System.NullReferenceException: Object reference not set to an instance of an object.at HostingUnitServiceClient.HusClient.TranslateHostingUnit(HostingUnitInternal hostingUnit)at HostingUnitServiceClient.HusClient.GetHostingUnitDetails(Guid uid)at Citrix.DesktopUpdateManager.SDK.SDKLogic.GetHostingUnit(Guid uid)at Citrix.XDServiceBase.PropertyResolver`2.Resolve(TInput toResolve)CitrixMachineCreationService:Exception caught in PostProvTask, HostingUnit not found, not adding prefix

Machine Creation Service Log Analysis


MachineCreationServiceLog:2:1:Queued task RunTask-580c3ed9-d6ac-44a1-94e2-442e015c531c, current queue length=1, high priority=0, no-op=0"MachineCreationServiceLog:2:1:VMware: Begin copy disk lenir-012603_S4B4-1-baseDisk, task RunTask-580c3ed9-d6ac-44a1-94e2-442e015c531c"MachineCreationServiceLog:2:1:Dequeued task RunTask-580c3ed9-d6ac-44a1-94e2-442e015c531c, current queue length=0, high priority=0, no-op=0"MachineCreationServiceLog:2:1:Queued task RunTask-b5e8b09e-5568-41eb-86e1-2acae9b98358, current queue length=1, high priority=0, no-op=0"MachineCreationServiceLog:2:1:VMware: Begin copy disk lenir-012603_S4B4-1-baseDisk, task RunTask-b5e8b09e-5568-41eb-86e1-2acae9b98358"MachineCreationServiceLog:2:1:Dequeued task RunTask-b5e8b09e-5568-41eb-86e1-2acae9b98358, current queue length=0, high priority=0, no-op=0"MachineCreationServiceLog:2:1:EndCopyDisk: task RunTask-b5e8b09e-5568-41eb-86e1-2acae9b98358"MachineCreationServiceLog:2:1:EndCopyDisk: task RunTask-580c3ed9-d6ac-44a1-94e2-442e015c531c"

MCS Log Analysis



SQL Profile trace Analysis

• On the SQL Profile trace make sure to select the following:

• “Security Audit”

• “Stored Procedures”

• Look through the trace and check for any permission errors or any failures for running a stored procedure

• For our case everything looked normal so we need to focus on the CDF analysis

• With CDF Control you can download the public TMF files which will allow you to parse the CDF trace and troubleshoot your issue

Using CDF Control


• Parsing the CDF trace and enabling the expert shader feature allows us to quickly find exceptions which are typically highlighted in orange

High level failure is: “Failed to copy all master images to all of the Hosts. No machines have been added to the Catalog.”

c

MachineCreationServiceLog:1:1:Converting to a return code, an exception of type: Citrix.Cds.DAL.DALDataStoreException and message: General database error: XML parsing: line 1, character 331, illegal name character.“MachineCreationServiceLog:2:1:The DALDataStoreException, has an inner Sql exception with the Number set as 9421.“MachineCreationServiceLog:1:1:Creating a new provisioning scheme failed with error ServiceStatusInvalidDB.“MachineCreationServiceLog:1:1:System.InvalidOperationException: ServiceStatusInvalidDBAt Citrix.DesktopUpdateManager.SDK.NewProvisioningSchemeSupport.NewProvisioningSchemeLogic.DoCommitScheme(NewProvisioningSchemeWorkflow context)MachineCreationServiceDAL:8:5:DAL >>> WorkflowAddMetadata(2bcc068d-a5b0-42c0-933b-38958a7a74bb, Citrix_DesktopStudio_ExtraWarnings, Failed to copy all master images to all of the Hosts. No machines have been added to the Catalog.)“

CDF Trace Log Analysis


Root Cause Analysis


• This issue resulted in Citrix adding a check in the code for each call to path with improved error handling when illegal characters are discovered in the storage naming scheme.

• The change has been checked into XenDesktop 5 SP1.

Resolution


Troubleshooting XenDesktop 5 Session Launch using Pass-through Authentication

• XenDesktop 5 sessions fail to launch when using pass-through authentication

Steps to Reproduce:1. Launch XenDesktop session from a domain-joined Windows PC2. Desktop Viewer opens, and the progress wheel spins..3. VDA Windows logon screen is seen brieflyExpected Results:

The session logon process completes, and the Windows desktop is presented.Actual Results:

The session closes immediately after flashing the Windows Logon screen

Problem Definition

• XenDesktop 5 in a POC environment, XenDesktop 4 is already deployed and is in production

• XenDesktop 4 sessions prompt for credentials at the Windows logon screen from the same endpoint

• Explicit authentication works for both XD4&5


Background on the issue

Three main components involved in session launch

Narrowing Down the Issue


SQL

XenDesktop Authentication Methods

Explicit Authentication Pass-through Authentication• User name and password are presented directly to Web Interface site

• Allows Broker to validate and authenticate VDA session launch request

• Useful for non-domain joined endpoint authentication

• User identity is verified by IIS using NTLM or Kerberos

• Allows Broker to validate the user for desktop enumeration

• Requires endpoint device to provide credentials directly to the ICA Server


EndpointEndpoint

XenDesktop 5 BrokerXenDesktop 5 Broker


Explicit Authentication

Web Interface

ICA

XML Services

Controller

VDAVDA

SQL

WCF

HTTP(S)

EndpointEndpoint



Pass-through Authentication

Controller

ICA

XML Services

WCF

ICA File

Web Interface

IIS

SQL

HTTP(S)

VDAVDA

Reproduce the Issue

Test Cases Test Results

1. XenDesktop 4 environment using Pass-through authentication

2. XenDesktop 5 environment using Pass-through authentication

3. XenDesktop 4/5 environments using explicit authentication

1. Reached the Windows logon screen, where I was able to login

2. Session launch fails at the Web Interface Site

3. Worked with both XD4 & XD5


EndpointEndpoint



Session Launch Fails at Web Interface

Controller

XML Services

Web Interface

VDAVDA

IIS

SQL

An error occurred while making the requested connection

Troubleshooting the Broker

• Service Logging - CTX127492• CDF Control - CTX111961• XDPing - CTX123278• Powershell SDK - CTX127254• WCF Diagnostics- MS732009






http://msdn.microsoft.com/en-us/library/ms732009.aspx

CdsXmlServices:2:1:ProcessCredentials: exception Citrix.Xms.XmlSupport.CredentialsException: ID only credentials received but TrustRequestsSentToTheXmlServicePort=false at Citrix.Xms.XmlSupport.CredentialsProcessor.ProcessCredentials(CommonCredentials RequestCredentials, CredentialType SupportedCredentials, CredentialOptions ProcessingOptions)CdsXmlServices:2:1:GetErrorIdFromCredentialsException: AccessDenied -> not-trustedCdsXmlServices:2:1:Credential Exception, reason AccessDenied: Citrix.Xms.XmlSupport.CredentialsException: ID only credentials received but TrustRequestsSentToTheXmlServicePort=false at Citrix.Xms.XmlSupport.CredentialsProcessor.ProcessCredentials(CommonCredentials RequestCredentials, CredentialType SupportedCredentials, CredentialOptions ProcessingOptions) at Citrix.Cds.Xms.Wpnbr.BaseTransaction.ProcessCredentials(CredentialType SupportedCredentials, CredentialOptions ProcessingOptions) at Citrix.Cds.Xms.Wpnbr.AddressTransaction.HandleRequest(IXmlMultiplexer multiplexer) at Citrix.Xms.XmlSupport.XmlPerf.WrapTransaction(Type t, Action transaction) at Citrix.Cds.Xms.Wpnbr.WpnbrServer.HandleRequest(HttpListenerRequest request, WindowsIdentity identity)CdsXmlServices:2:1:GetErrorIdFromCredentialsException: AccessDenied -> not-trusted

Broker CDF Analysis


• Searched Citrix KB for XML Service issues in XD5

• Found that XD5 broker requires XML service to trust ID-Only credentials (CTX128328)

• Also required for SSO to work through Access Gateway

• Configure using XenDesktop 5 Powershell SDK (CTX127254)


Troubleshooting: Broker Components



EndpointEndpoint



Session Launch Fails During Session Initialization

Controller

ICA

XML Services

WCF

ICA File

Web Interface

VDAVDA

IIS

SQL

HTTP(S)

• PortICA Service Logs (CTX118837)

• Workstation Agent Service Logs (CTX127492)

• CDF Trace Modules:CdsWorkerAgentICA ServiceMF_Session_WfshellMF_DLL_CtxginaMF_Library_System

Troubleshooting VDA: Session Launch


Portica.ICA.IcaClientStack.GetCredentials

CdsWorkerAgent:2:1:Validate no credentials returned

Portica.BizLogic.TakeOwnershipOfCredentials

Portica.GinaServer.SendAutoLogonMessage

Utils.Kernel32.UnmanagedBuffer.SafeDisposeObj ThreadID=7, disposing=True, pointer=32C60E8, size=1568, source=Citrix.Portica.GinaServer.SendAutoLogonMessage

Portica.GinaServer.ProcessGinaMsg Received message of type: CancelIcaConnection

Portica_DLL_PICACredProviderPortica_DLL_PICADisplayManagerPortica_DLL_PICASessionHelperPortica_Library_picaCPHelper

http://support.citrix.com/article/ctx118837


• Enforce Auto Logon (CTX127392)

• Requires credentials to be passed, or the session is canceled

• Enabled by default in XD5 for security purposes

• Can be manually set on VDA• Create DWORD value on the VDA called 'EnforceAutoLogon' in HKLM\

Software\Policies\Citrix, and set it to 0


Troubleshooting: VDA Components


Troubleshooting Online Plugin

• ICA Logging - CTX115304• CDFControl - CTX124934• DebugView - BB896647• Client Policies - EDocs


Directory must exist, and be writable

Enable LogEvidence for CST



http://technet.microsoft.com/en-us/sysinternals/bb896647

http://support.citrix.com/proddocs/index.jsp?topic=/online-plugin-121-windows/ica-import-icaclient-template-v2.html

ICA Log Analysis


• Desktop Group

• ICA Address

• Auto-Logon Allowed

• Desktop Viewer

• Single Sign-On

[KB-Win7-x32RTM]Address=10.54.67.97:1494AutologonAllowed=ONBrowserProtocol=HTTPonTCPConnectionBar=1InitialProgram=#WinXP 32-bit $P8Launcher=WILaunchReference=EE2998E87E058B78E1CAF7050FB40ESessionsharingKey=-R7YM1LL1qw5bcb7LTq21sCUseLocalUserAndPassword=On

• Searched Citrix KB for UseLocalUserAndPassword

Pass-through Authentication Requirements


http://support.citrix.com/proddocs/index.jsp?topic=/ica-settings/ica-settings-uselocaluserandpassword2.html

Pass-through Authentication Client Policy Settings


Pass-through Authentication CST Override


Allows all regions except Restricted

• Collects and analyzes ‘evidence’ from session launch details

• Classifies ICA sessions into one of four regions:• oidTrustedRegion • oidIntranetRegion • oidInternetRegion• oidRestrictedRegion

• Checks WI Site against Internet Explorer security zones

• Blocks certain ICA Client actions (such as Pass-through) based on region settings (CTX124871)

• Requires CST registry keys to be present (CTX128775)

Client Selective Trust (CST)


ICA Log Analysis - CST Evidence


• Collect

• Inspect

• Select

• Authorize

ICA Client connection initializedAddEvidence InitialProgram=#KB-Win7-x32RTMRegion All RegionsAddEvidence ICAFileAddress=XenDesktop.get.services.citrite.net:1494Region Trusted RegionAddEvidence ServerAddress=XenDesktop.GET.SERVICES.CITRITE.NETRegion Trusted RegionAddEvidence CGPEnabled=TrueRegion All RegionsAddEvidence ServerIPAddress=10.54.67.220Region All RegionsEvidenceRequest Connection Authorisation (event: Open connection to Citrix Server) Granted

CTX124921

• CST evaluates Initial Program value as evidence

• Requires the desktop group name to be added to the CST whitelist if ‘Allow pass-through for all connections’ is not enabled

• Used DebugView output to determine what evidence was being evaluated

Desktop Viewer CST Requirements


CST Whitelist


KB-Win7-x32RTM]Address=10.54.67.97:1494AutologonAllowed=ONBrowserProtocol=HTTPonTCPConnectionBar=1InitialProgram=#WinXP 32-bit $P8Launcher=WILaunchReference=EE2998E87E058B78E1CAF7050FB40ESessionsharingKey=-R7YM1LL1qw5bcb7LTq21sCUseLocalUserAndPassword=On

Wildcards don’t work here

EndpointEndpoint



Pass-through Authentication

Controller

ICA

XML Services

WCF

ICA File

Web Interface

IIS

SQL

HTTP(S)

VDAVDA


Root Cause Analysis

• Provided a private binary that instead evaluates the ICA address, which supports wildcards

• Client Selective Trust is being replaced by ICA File Signing

• Recommending ICA File Signing as a replacement (eDoc)

Resolution


Resources discussed

For More Information

• CTX127492 - How to enable Controller Service Logging in XenDesktop 5

• CTX128075 - XDDBDiag: XenDesktop 5 Database Diagnostics

• CTX128909 - XenDesktop 5 Logon Process and Communication Flow

• CTX127969 - Desktop Studio Logging Options

• CTX127587 - XenDesktop 5 Reference Architecture

• CTX128190 - How to Change Virtual Channel Priority in XenDesktop 5

• CTX127254 - XenDesktop 5 SDK PowerShell Cmdlet Help

Questions and wrap up

troubleshooting xendesktop 5 deployments

Documents

distributecitrix confidential

objectivescitrix confidential

problemcitrix confidential

lookcitrix confidential

base image

virtual image

image roll

vmspooled image