case studies on troubleshooting xendesktop 5
DESCRIPTION
citrix documentTRANSCRIPT
Troubleshooting XenDesktop 5 DeploymentsBaptiste Duflos, Escalation Manager & Ken Baldwin, Escalation EngineerTuesday, May 24th 2011
Introduction and objectives
Case study for MCS fails to create pooled machines
• Fully integrated provisioning into the XenDesktop 5 console
• Desktop lifecycle support and image roll-back capability
• Leverages and supports all 3 major Hypervisors
Machine Creation Services introduces:
Citrix Confidential - Do Not Distribute
VM
Master Disk
VM VM
Diff DiskId Disk
Diff DiskId Disk
Diff DiskId Disk
Storage
One copy of the base image shared by all VMs
One copy of the base image shared by all VMs
Pooled image will reset back to initial state after reboot
Pooled image will reset back to initial state after reboot
Persistent Identity disk provides AD computer account info
Persistent Identity disk provides AD computer account info
Each VM consists of a Difference disk and an Identity disk
Each VM consists of a Difference disk and an Identity disk
VMs can be created in pooled or private mode
VMs can be created in pooled or private mode
BrokerBroker
Hypervisorsand
Storage
Hypervisorsand
Storage
HCL
Machine Identity Service
Data Access
Active Directory
Active Directory
AD Identity ServiceData
Access
HypervisorsHypervisors
HCL
Machine Creation Service
Data AccessSQL
Infrastructure Service
Host Service
ConfigurationService
Citrix Confidential - Do Not Distribute
Machine Creation ServiceMachine Creation Service
Reproducing the error: failed to create Catalog
Data Access
Storage
HCL HypervisorsHypervisors
Network
SQL
The Catalog could not be loaded due to the following errors:There are no master images associated with this Catalog
See CTX127068 for resolutions to this problem
Citrix Confidential - Do Not Distribute
Troubleshooting Methodology – initial first look
Citrix Confidential - Do Not Distribute
• Verify the Certs and Proxy.xml - CTX125578
• Validate the Hypervisor permissions - CTX127546• Configure and test
multiple host connections
• Validate the Hypervisor is configured correctly
• Check the image
• Try using another virtual image for creation
• Check the master image snapshot wasn’t deleted
• Check permissions if storage path is not using local attached storage
Troubleshooting Methodology – Logs and Traces
• Service Logging - CTX127492
• SQL Trace - CTX127257
• CDF Control - CTX111961SQL
Citrix Confidential - Do Not Distribute
CitrixMachineCreationService:-> Citrix.XDServiceBase.LogicBase.GetRemoteServiceInstances - EntryCitrixMachineCreationService:Returning cached service instancesCitrixMachineCreationService: Citrix.XDServiceBase.LogicBase.GetRemoteServiceInstances - ExitCitrixMachineCreationService:Sorting the ServiceInstances.CitrixMachineCreationService:Using the next service instance http://xd5-lab.local/Citrix/HostingUnitService/IServiceAPI CitrixMachineCreationService:Conversion error in Property Resolver. Exception is System.NullReferenceException: Object reference not set to an instance of an object.at HostingUnitServiceClient.HusClient.TranslateHostingUnit(HostingUnitInternal hostingUnit)at HostingUnitServiceClient.HusClient.GetHostingUnitDetails(Guid uid)at Citrix.DesktopUpdateManager.SDK.SDKLogic.GetHostingUnit(Guid uid)at Citrix.XDServiceBase.PropertyResolver`2.Resolve(TInput toResolve)CitrixMachineCreationService:Exception caught in PostProvTask, HostingUnit not found, not adding prefix
Machine Creation Service Log Analysis
Citrix Confidential - Do Not Distribute
MachineCreationServiceLog:2:1:Queued task RunTask-580c3ed9-d6ac-44a1-94e2-442e015c531c, current queue length=1, high priority=0, no-op=0"MachineCreationServiceLog:2:1:VMware: Begin copy disk lenir-012603_S4B4-1-baseDisk, task RunTask-580c3ed9-d6ac-44a1-94e2-442e015c531c"MachineCreationServiceLog:2:1:Dequeued task RunTask-580c3ed9-d6ac-44a1-94e2-442e015c531c, current queue length=0, high priority=0, no-op=0"MachineCreationServiceLog:2:1:Queued task RunTask-b5e8b09e-5568-41eb-86e1-2acae9b98358, current queue length=1, high priority=0, no-op=0"MachineCreationServiceLog:2:1:VMware: Begin copy disk lenir-012603_S4B4-1-baseDisk, task RunTask-b5e8b09e-5568-41eb-86e1-2acae9b98358"MachineCreationServiceLog:2:1:Dequeued task RunTask-b5e8b09e-5568-41eb-86e1-2acae9b98358, current queue length=0, high priority=0, no-op=0"MachineCreationServiceLog:2:1:EndCopyDisk: task RunTask-b5e8b09e-5568-41eb-86e1-2acae9b98358"MachineCreationServiceLog:2:1:EndCopyDisk: task RunTask-580c3ed9-d6ac-44a1-94e2-442e015c531c"
MCS Log Analysis
Citrix Confidential - Do Not Distribute
Citrix Confidential - Do Not Distribute
SQL Profile trace Analysis
• On the SQL Profile trace make sure to select the following:
• “Security Audit”
• “Stored Procedures”
• Look through the trace and check for any permission errors or any failures for running a stored procedure
• For our case everything looked normal so we need to focus on the CDF analysis
• With CDF Control you can download the public TMF files which will allow you to parse the CDF trace and troubleshoot your issue
Using CDF Control
Citrix Confidential - Do Not Distribute
• Parsing the CDF trace and enabling the expert shader feature allows us to quickly find exceptions which are typically highlighted in orange
High level failure is: “Failed to copy all master images to all of the Hosts. No machines have been added to the Catalog.”
c
MachineCreationServiceLog:1:1:Converting to a return code, an exception of type: Citrix.Cds.DAL.DALDataStoreException and message: General database error: XML parsing: line 1, character 331, illegal name character.“MachineCreationServiceLog:2:1:The DALDataStoreException, has an inner Sql exception with the Number set as 9421.“MachineCreationServiceLog:1:1:Creating a new provisioning scheme failed with error ServiceStatusInvalidDB.“MachineCreationServiceLog:1:1:System.InvalidOperationException: ServiceStatusInvalidDBAt Citrix.DesktopUpdateManager.SDK.NewProvisioningSchemeSupport.NewProvisioningSchemeLogic.DoCommitScheme(NewProvisioningSchemeWorkflow context)MachineCreationServiceDAL:8:5:DAL >>> WorkflowAddMetadata(2bcc068d-a5b0-42c0-933b-38958a7a74bb, Citrix_DesktopStudio_ExtraWarnings, Failed to copy all master images to all of the Hosts. No machines have been added to the Catalog.)“
CDF Trace Log Analysis
Citrix Confidential - Do Not Distribute
Root Cause Analysis
Citrix Confidential - Do Not Distribute
• This issue resulted in Citrix adding a check in the code for each call to path with improved error handling when illegal characters are discovered in the storage naming scheme.
• The change has been checked into XenDesktop 5 SP1.
Resolution
Citrix Confidential - Do Not Distribute
Troubleshooting XenDesktop 5 Session Launch using Pass-through Authentication
• XenDesktop 5 sessions fail to launch when using pass-through authentication
Steps to Reproduce:1. Launch XenDesktop session from a domain-joined Windows PC2. Desktop Viewer opens, and the progress wheel spins..3. VDA Windows logon screen is seen brieflyExpected Results:
The session logon process completes, and the Windows desktop is presented.Actual Results:
The session closes immediately after flashing the Windows Logon screen
Problem Definition
Citrix Confidential - Do Not Distribute
• XenDesktop 5 in a POC environment, XenDesktop 4 is already deployed and is in production
• XenDesktop 4 sessions prompt for credentials at the Windows logon screen from the same endpoint
• Explicit authentication works for both XD4&5
Citrix Confidential - Do Not Distribute
Background on the issue
Three main components involved in session launch
Narrowing Down the Issue
Citrix Confidential - Do Not Distribute
SQL
XenDesktop Authentication Methods
Explicit Authentication Pass-through Authentication• User name and password are presented directly to Web Interface site
• Allows Broker to validate and authenticate VDA session launch request
• Useful for non-domain joined endpoint authentication
• User identity is verified by IIS using NTLM or Kerberos
• Allows Broker to validate the user for desktop enumeration
• Requires endpoint device to provide credentials directly to the ICA Server
Citrix Confidential - Do Not Distribute
EndpointEndpoint
XenDesktop 5 BrokerXenDesktop 5 Broker
Citrix Confidential - Do Not Distribute
Explicit Authentication
Web Interface
ICA
XML Services
Controller
VDAVDA
SQL
WCF
HTTP(S)
EndpointEndpoint
XenDesktop 5 BrokerXenDesktop 5 Broker
Citrix Confidential - Do Not Distribute
Pass-through Authentication
Controller
ICA
XML Services
WCF
ICA File
Web Interface
IIS
SQL
HTTP(S)
VDAVDA
Reproduce the Issue
Test Cases Test Results
1. XenDesktop 4 environment using Pass-through authentication
2. XenDesktop 5 environment using Pass-through authentication
3. XenDesktop 4/5 environments using explicit authentication
1. Reached the Windows logon screen, where I was able to login
2. Session launch fails at the Web Interface Site
3. Worked with both XD4 & XD5
Citrix Confidential - Do Not Distribute
EndpointEndpoint
XenDesktop 5 BrokerXenDesktop 5 Broker
Citrix Confidential - Do Not Distribute
Session Launch Fails at Web Interface
Controller
XML Services
Web Interface
VDAVDA
IIS
SQL
An error occurred while making the requested connection
Troubleshooting the Broker
• Service Logging - CTX127492• CDF Control - CTX111961• XDPing - CTX123278• Powershell SDK - CTX127254• WCF Diagnostics- MS732009
Citrix Confidential - Do Not Distribute
CdsXmlServices:2:1:ProcessCredentials: exception Citrix.Xms.XmlSupport.CredentialsException: ID only credentials received but TrustRequestsSentToTheXmlServicePort=false at Citrix.Xms.XmlSupport.CredentialsProcessor.ProcessCredentials(CommonCredentials RequestCredentials, CredentialType SupportedCredentials, CredentialOptions ProcessingOptions)CdsXmlServices:2:1:GetErrorIdFromCredentialsException: AccessDenied -> not-trustedCdsXmlServices:2:1:Credential Exception, reason AccessDenied: Citrix.Xms.XmlSupport.CredentialsException: ID only credentials received but TrustRequestsSentToTheXmlServicePort=false at Citrix.Xms.XmlSupport.CredentialsProcessor.ProcessCredentials(CommonCredentials RequestCredentials, CredentialType SupportedCredentials, CredentialOptions ProcessingOptions) at Citrix.Cds.Xms.Wpnbr.BaseTransaction.ProcessCredentials(CredentialType SupportedCredentials, CredentialOptions ProcessingOptions) at Citrix.Cds.Xms.Wpnbr.AddressTransaction.HandleRequest(IXmlMultiplexer multiplexer) at Citrix.Xms.XmlSupport.XmlPerf.WrapTransaction(Type t, Action transaction) at Citrix.Cds.Xms.Wpnbr.WpnbrServer.HandleRequest(HttpListenerRequest request, WindowsIdentity identity)CdsXmlServices:2:1:GetErrorIdFromCredentialsException: AccessDenied -> not-trusted
Broker CDF Analysis
Citrix Confidential - Do Not Distribute
• Searched Citrix KB for XML Service issues in XD5
• Found that XD5 broker requires XML service to trust ID-Only credentials (CTX128328)
• Also required for SSO to work through Access Gateway
• Configure using XenDesktop 5 Powershell SDK (CTX127254)
Citrix Confidential - Do Not Distribute
Troubleshooting: Broker Components
EndpointEndpoint
XenDesktop 5 BrokerXenDesktop 5 Broker
Citrix Confidential - Do Not Distribute
Session Launch Fails During Session Initialization
Controller
ICA
XML Services
WCF
ICA File
Web Interface
VDAVDA
IIS
SQL
HTTP(S)
• PortICA Service Logs (CTX118837)
• Workstation Agent Service Logs (CTX127492)
• CDF Trace Modules:CdsWorkerAgentICA ServiceMF_Session_WfshellMF_DLL_CtxginaMF_Library_System
Troubleshooting VDA: Session Launch
Citrix Confidential - Do Not Distribute
Portica.ICA.IcaClientStack.GetCredentials
CdsWorkerAgent:2:1:Validate no credentials returned
Portica.BizLogic.TakeOwnershipOfCredentials
Portica.GinaServer.SendAutoLogonMessage
Utils.Kernel32.UnmanagedBuffer.SafeDisposeObj ThreadID=7, disposing=True, pointer=32C60E8, size=1568, source=Citrix.Portica.GinaServer.SendAutoLogonMessage
Portica.GinaServer.ProcessGinaMsg Received message of type: CancelIcaConnection
Portica_DLL_PICACredProviderPortica_DLL_PICADisplayManagerPortica_DLL_PICASessionHelperPortica_Library_picaCPHelper
• Enforce Auto Logon (CTX127392)
• Requires credentials to be passed, or the session is canceled
• Enabled by default in XD5 for security purposes
• Can be manually set on VDA• Create DWORD value on the VDA called 'EnforceAutoLogon' in HKLM\
Software\Policies\Citrix, and set it to 0
Citrix Confidential - Do Not Distribute
Troubleshooting: VDA Components
Troubleshooting Online Plugin
• ICA Logging - CTX115304• CDFControl - CTX124934• DebugView - BB896647• Client Policies - EDocs
Citrix Confidential - Do Not Distribute
Directory must exist, and be writable
Enable LogEvidence for CST
ICA Log Analysis
Citrix Confidential - Do Not Distribute
• Desktop Group
• ICA Address
• Auto-Logon Allowed
• Desktop Viewer
• Single Sign-On
[KB-Win7-x32RTM]Address=10.54.67.97:1494AutologonAllowed=ONBrowserProtocol=HTTPonTCPConnectionBar=1InitialProgram=#WinXP 32-bit $P8Launcher=WILaunchReference=EE2998E87E058B78E1CAF7050FB40ESessionsharingKey=-R7YM1LL1qw5bcb7LTq21sCUseLocalUserAndPassword=On
• Searched Citrix KB for UseLocalUserAndPassword
Pass-through Authentication Requirements
Citrix Confidential - Do Not Distribute
Pass-through Authentication Client Policy Settings
Citrix Confidential - Do Not Distribute
Pass-through Authentication CST Override
Citrix Confidential - Do Not Distribute
Allows all regions except Restricted
• Collects and analyzes ‘evidence’ from session launch details
• Classifies ICA sessions into one of four regions:• oidTrustedRegion • oidIntranetRegion • oidInternetRegion• oidRestrictedRegion
• Checks WI Site against Internet Explorer security zones
• Blocks certain ICA Client actions (such as Pass-through) based on region settings (CTX124871)
• Requires CST registry keys to be present (CTX128775)
Client Selective Trust (CST)
Citrix Confidential - Do Not Distribute
ICA Log Analysis - CST Evidence
Citrix Confidential - Do Not Distribute
• Collect
• Inspect
• Select
• Authorize
ICA Client connection initializedAddEvidence InitialProgram=#KB-Win7-x32RTMRegion All RegionsAddEvidence ICAFileAddress=XenDesktop.get.services.citrite.net:1494Region Trusted RegionAddEvidence ServerAddress=XenDesktop.GET.SERVICES.CITRITE.NETRegion Trusted RegionAddEvidence CGPEnabled=TrueRegion All RegionsAddEvidence ServerIPAddress=10.54.67.220Region All RegionsEvidenceRequest Connection Authorisation (event: Open connection to Citrix Server) Granted
CTX124921
• CST evaluates Initial Program value as evidence
• Requires the desktop group name to be added to the CST whitelist if ‘Allow pass-through for all connections’ is not enabled
• Used DebugView output to determine what evidence was being evaluated
Desktop Viewer CST Requirements
Citrix Confidential - Do Not Distribute
CST Whitelist
Citrix Confidential - Do Not Distribute
KB-Win7-x32RTM]Address=10.54.67.97:1494AutologonAllowed=ONBrowserProtocol=HTTPonTCPConnectionBar=1InitialProgram=#WinXP 32-bit $P8Launcher=WILaunchReference=EE2998E87E058B78E1CAF7050FB40ESessionsharingKey=-R7YM1LL1qw5bcb7LTq21sCUseLocalUserAndPassword=On
Wildcards don’t work here
EndpointEndpoint
XenDesktop 5 BrokerXenDesktop 5 Broker
Citrix Confidential - Do Not Distribute
Pass-through Authentication
Controller
ICA
XML Services
WCF
ICA File
Web Interface
IIS
SQL
HTTP(S)
VDAVDA
Citrix Confidential - Do Not Distribute
Root Cause Analysis
• Provided a private binary that instead evaluates the ICA address, which supports wildcards
• Client Selective Trust is being replaced by ICA File Signing
• Recommending ICA File Signing as a replacement (eDoc)
Resolution
Citrix Confidential - Do Not Distribute
Resources discussed
For More Information
• CTX127492 - How to enable Controller Service Logging in XenDesktop 5
• CTX128075 - XDDBDiag: XenDesktop 5 Database Diagnostics
• CTX128909 - XenDesktop 5 Logon Process and Communication Flow
• CTX127969 - Desktop Studio Logging Options
• CTX127587 - XenDesktop 5 Reference Architecture
• CTX128190 - How to Change Virtual Channel Priority in XenDesktop 5
• CTX127254 - XenDesktop 5 SDK PowerShell Cmdlet Help
Questions and wrap up