Ramon Scott – Lead Escalation Engineer

Configuring & Troubleshooting XenDesktop SitesSUM408

May 2013

Presentation Goals

2

Provide an Understanding of the Provide an Understanding of the

ArchitectureArchitecture

Instruct on How to ConfigureInstruct on How to Configure

Provide Proven Troubleshooting Provide Proven Troubleshooting

Methodologies and ResourcesMethodologies and Resources

High-Level XenDesktop Database High-Level XenDesktop Database And Services ArchitectureAnd Services Architecture

DatabaseDatabase

• Supported Databases:• SQL Server 2008 SP1 / 2008R2

(including Express)

• Database Schema• Full Relational Schema

• Tables, Views, Stored Procedures• Single Database (for core product)• Multiple SQL ‘Schemas’ in Database

• ‘Schemas’ map onto Windows services running on Broker

XenDesktop 5 Database Overview

Broker

Broker

Database

Setup Process

XD Console

Single Admin

Broker1. Schema

Database

2. Schema

XD Admin 3. Verify

XD Admin credentials used

XD Admin credentials used

Separate Admins

XD Console Broker1. Schema

Database3. Schema

XD Admin

4. Verify

SQL Server Console

SQL Admin

2. Schema

“Export”(SQL script)

SQL Admin credentials used

SQL Admin credentials used

Database Access

• Security Access Model

ᵒ Network Service Account

“NT AUTHORITY\NETWORK SERVICE”

ᵒ Computer Account

“DOMAIN\MACHINE$”

• SQL Login per Broker

• Restricted permission setᵒ Brokers do not have rights to change schema

Controller

DatabaseController

Broker Service

Controller

DatabaseController

Broker Service

Database

• Broker is critically dependant on Database• Existing connections not impacted• Creating new connections and reconnecting to desktops

impacted

• Database Failure = Broker Failure

• Supported Database H/A Options: (expected popularity order)

1.SQL Mirror2.Virtual Machine H/A3.SQL Cluster

Database High-Availability

Citrix Confidential - Do Not Distribute

Database Schema Roles and Permissions

XenDesktop Service Database Role

AD Identity Service (Acct) ADIdentitySchema_ROLE

Broker Service (Broker) chr_Broker

chr_Controller

Central Configuration Service (Config) ConfigurationSchema_ROLE

Machine Creation Service (PvsVM) DesktopUpdateManagerSchema_ROLE

Hosting Management Service (Hyp) HostingUnitServiceSchema_ROLE

Machine Identity Service (Prov) MachinePersonalitySchema_ROLE

Health Checks: XDDBDiag

• Provided consistency data check on the data

• Provides connectivity verification

It also provides the following:ᵒ Virtual Desktop Agent Informationᵒ Hypervisor Connections Informationᵒ Policy Informationᵒ Controller Informationᵒ Desktop Groups Informationᵒ SQL Informationᵒ Current Connections / Connection Log

ServicesServices

Machine CreationServices

BrokerService

InfrastructureServices

XenDesktop 5 Services Architecture

12

Controller

Broker Service

Machine CreationService

AD Identity Service

Machine Identity Service

Host Service

ConfigurationService

Virtual Desktop

Agent (VDA)

WinRM 2.0[5985/5986]

Desktop Studio

WCF [80]

PowerShell Desktop Director

WCF [80]

PowerShell

SQL Server

Windows Communication Foundation (WCF)

Windows Communication Foundation (WCF)

Service Status

XenDesktop Service PowerShell Cmdlet

AD Identity Service (Acct) Get-AcctServiceStatus

Broker Service (Broker) Get-BrokerServiceStatus

Central Configuration Service (Config) Get-ConfigServiceStatus

Machine Creation Service (Prov) Use Get-ProvServiceStatus

Hosting Management Service(Hyp) Get-HypServiceStatus

Machine Identity Service (PvsVM) Get-PvsvmServiceStatus

Machine CreationMachine Creation

Desktop Catalog models

• Existing

• Dedicated

• Pooled

• Pooled with personal vDisk

• Streamed

• Streamed with personal vDisk

Base ImageBase Image

App

Profile

App

Profile

App

Profile

PvDPvD

PvDPvD

PvDPvD

ImageImage

ImageImage

ImageImage

ProfileProfile

ProfileProfile

ProfileProfile

ProfileProfile

ProfileProfile

ProfileProfile

Base Image with Apps

Base Image with Apps

Base Image with Apps

Base Image with Apps

Streamed Base Image with Apps

Streamed Base Image with AppsStreamed

Base Image

Streamed Base Image

App

Profile

App

Profile

App

Profile

PvDPvD

PvDPvD

PvDPvD

*Image Streamed from Citrix Provisioning Server

(PVS)

*Image Streamed from Citrix Provisioning Server

(PVS)

*Image created with Machine Creation Services

(MCS)

*Image created with Machine Creation Services

(MCS)

*Image created outside of XenDesktop

*Image created outside of XenDesktop

Desktop Catalog models

* Behaves like pooled-static

* Behaves like pooled-static

MCS – ID Disk, Difference Disk, Base VM

Virtual Desktop 1Diff Disk ID DiskVHD Chain

Windows 7 Master

This is what the user sees as Drive C:\

This is hidden from the users view

Virtual Desktop 2Diff Disk ID DiskVHD Chain

Virtual Desktop xDiff Disk ID DiskVHD Chain

Storage Subsystem

MCS with PvD – ID Disk, Diff Disk, Base VM, PVDisk

Virtual Desktop 1

VHD Chain

Windows 7 Master

Diff Disk ID Disk

Personal vDisk• This part is hidden from user• Merged with the Diff Disk• Seen by user as Drive C:\• E.g. Installed apps

• Seen by the user as Drive P:\• USERDATA e.g. My Documents• Free space is the split allocation

• PVDisk auto-created during catalog creation by copying PvD template from Base VM

• 10GB by default with 50 / 50 split for App Data / User Data

PVS – Streamed vDisk, Cache, Base VM

Virtual Desktop 1StreamedvDisk

Write Cache

PVS Stream

Windows 7 Master

This is what the user sees as Drive C:\

Visible file on another disk, typically D:\

Virtual Desktop 2StreamedvDisk

Write Cache

PVS Stream

Virtual Desktop xStreamedvDisk

Write Cache

PVS Stream

Storage Subsystem

PVS with PvD–Streamed vDisk, Cache, Base PVS with PvD–Streamed vDisk, Cache, Base VM, PvDisk

Virtual Desktop 1

PVS Stream

Windows 7 Master

Streamed vDisk

Write Cache

Personal vDisk• This part is hidden from user• Seen by user as Drive C:\• E.g. Installed apps

• Seen by the user as Drive P:\• USERDATA e.g. My Documents• Free space is the split allocation

• PvDisk auto-created during catalog creation by copying PvD template from Base VM

• 10GB by default with 50 / 50 split for App Data / User Data

Where are some of the common Issue ?

• Hypervisor communication• Domain permissions• Previously failed attempts still present

in database• Host Connection configured with

incorrect storage• Naming convention on the host

What logs do we need for this issue ?

22

Machine CreationServices

BrokerService

InfrastructureServices

Broker

Broker Service

Machine CreationService

AD Identity Service

Machine Identity Service

Host Service

ConfigurationService

Desktop Studio

WCF [80]

PoSH

SQL Server

Troubleshooting Methodology

23

• Understand issue history• Verify configuration, error logs and alerts

• Gather and review log data of issues• Compare data to working environment

Enabling Log from the Command Line

Service –LogFile –LogFile <Location>Citrix.ADIdentity.SdkWcfEndpoint.exe -LogFile c:\xdlogs\AD.log

Citrix.MachineCreation.SdkWcfEndpoint.exe -LogfileLogfile “c:\xdlogs\MCS-PVSvm.log”

Citrix.MachineIdentity.SdkWcfEndpoint.exe -LogFile c:\xdlogs\mi.log

Case Study 1Machine Creation Services

Case Study Walk Through

Background:

•New Deployment

•Latest Hotfixes

•Full Administrator account used

•Worked before they rebuilt environment

Case Study 1: MCS Fails after wizard

Log Analysis: Desktop Studio LogsCase Study 1: Machine Creation Service fail after wizard

24/04/13 02:37:10.7603 : DesktopStudio: [6] Script SetActionMetaData(402): [RES] Value: Failed to copy all master images to all of the Hosts. No machines have been added to the Catalog.

Search Terms: [Time of Issue]Fail | Error | Exception | Denied

Search Terms: [Time of Issue]Fail | Error | Exception | Denied

Search Terms: [Time of Issue]Fail | Error | Exception | Denied

Search Terms: [Time of Issue]Fail | Error | Exception | Denied

Case Study 1: Machine Creation Service fail after wizardLog Analysis: Machine Creation Service Logs

Failed to copy disk. Reason : SR_HAS_NO_PBDS

ManagedMachineException: Failed to copy disk. Reason : SR_HAS_NO_PBDS

Concluding job d5ea54c6-b7f1-4d45-ac08-2e2abae39e48 with state DiskConsolidationFailed.

WorkflowAddMetadata(, Citrix_DesktopStudio_ExtraWarnings, Failed to copy all master images to all of the Hosts. No machines have been added to the Catalog.)

Root Cause analysis: Misconfiguration

29

• Failed to copy disk Reason : SR_HAS_NO_PBDS

• Hypervisor Connection’s did not include correct storage for the Master Image

• Target device disk could not be copied due to this Hypervisor -Storage misconfiguration

*Definitions: SR - Storage RepositoriesPBD - Physical Block Devices

*Definitions: SR - Storage RepositoriesPBD - Physical Block Devices

VDA Startup and Registration

Controller

DDC

Broker Service

VDA Registration

VDA

Desktop Service

VDA

Active Directory Controller

WCF

LDAP

Database

Registered

Troubleshooting VDA Startup and Registration

• XDPing Log• Basic Checks• Logs:ᵒ Workstation Agent Logsᵒ Broker Logs

• Network TraceController

Broker Service

VDA

Desktop Service

1011011010 SSL 1011011010 SSL 1011011011011010 SSL 1011011010 SSL 1011011011011010 SSL 1011011010 SSL 1011011011011010 SSL 1011011010 SSL 101101

XDPING

• Can be run on both the DDC and VDA

• Used to collect data related to basic components

• Will verify if the components are working correctlyᵒ Verify Domain Membershipᵒ Network Interfacesᵒ WCF Endpointsᵒ Servicesᵒ DNS lookupᵒ Time difference between machine and

Domain Controller

Basic Checks

• Check the Network: Ping , Telnet and NetStat, Firewall

• Ensure Services started without errors

• Listening on the correct port

• Check time

• Check configured list of DDCs in registry

Case Study 2Startup and Registration

Case Study Walk Through Background:

•Locked down environment

•Special configuration needed to manually enable needed services

•Worked in the Proof of Conference Lab but failed in production

Case Study 2: New Catalog Fail to Register

Failed to register with http://FTLRSCOTT2RHONE.lab.net:80/Citrix/CdsController/IRegistrar.

WCF Fault with detail CallbackCommunicationError, message 'Fail worker callback using SPN host/RS2-SynPool01.lab.net and IP address 10.19.196.945'

Register FAILURE: HighAvailabilityActive = False, InHighAvailabilityMode = False, _firstRegistrationAttemptTime = 05/18/2013 13:54:31, HighAvailabilityRegistrationTimout = 00:05:00

Message following Error pattern

Could not register with any controllers. Waiting to try again in 9407 ms

Search Terms: [Time of Issue]Fail | Error | Exception | Denied

Search Terms: [Time of Issue]Fail | Error | Exception | Denied

Case Study 2: New Catalog Fail to RegisterLog Analysis: Workstation Agent Service Logs

37

Broker:TestWorkerComms failed for worker S-1-5-21-1123877020-465626563-

3648135752-1267 caught exception:

System.ServiceModel.Security.SecurityNegotiationException: The caller was not

authenticated by the service. ---> System.ServiceModel.FaultException: The

request for security token could not be satisfied because authentication failed.

Search Terms: [Time of Issue]Fail | Error | Exception | Denied

Search Terms: [Time of Issue]Fail | Error | Exception | Denied

Case Study 2: New Catalog Fail to RegisterLog Analysis: Broker Service Logs

38

Root Cause analysis: Misconfiguration

39

• The DDC was not authorized the initiate a connection to the VDA

• “Access To Compute From The Network” Computer Policy did not have an entry for the Controlled and the default everyone was removed in production.

Resolution: Customer added explicit entry to a Group that included all the Brokers as members

• PVD maintains logs in the base of the volume attached to the VM◦ (alongside the VHD containing the PVD user-installed applications)

• These logs contain a wealth of information that should be captured and provided to support/engineering if you experience problems

• Most frequently seen PVD support cases …◦ Failure of PVD to start virtualization (PVD can’t locate volume/VHD, etc.. …)◦ Customers trying to install unsupported apps ◦ Customers trying to move PVDs between VMs

41

• Desktop Director has helpdesk-facing PVD metrics and support◦ % of application area in use / total size◦ % of user profile area in use / total size◦ PVD reset

• PVD reset allows the helpdesk to reset the application area while leaving the user’s data intact◦ Aka “revert to factory default”◦ Useful to reset PVDs that become wedged due to users installing broken applications

VDA Launch

VDA Launch

Controller #1Controller #1

BrokerServiceBrokerService

DDC

VDAVDA

Desktop ServiceDesktop Service

VDA

WI

Idle

Launch RequestLaunch Request

SQL

WCF

XML broker queries DB for a ready workerXML broker queries

DB for a ready worker

Broker signals worker to Prepare

for a Session

Broker signals worker to Prepare

for a SessionUser Clicks to launch sessionUser Clicks to launch session

ICA Service

ICA Service

PolicySettings

Preparing New Session

Controller #1Controller #1

BrokerServiceBrokerService

DDC

VDAVDA

Desktop ServiceDesktop Service

VDA

WI

SQL

WCF

Work State: ActiveWork State: ActiveWork State: ConnectedWork State: Connected

Request to Validate Ticket sent Controller

Request to Validate Ticket sent Controller

ICA file is sent to Endpoint

ICA file is sent to Endpoint

ICA Service

ICA Service

PolicySettings

1. Validates Ticket 2. Validates License3. Policies

1. Validates Ticket 2. Validates License3. Policies

Ticket is ValidAuthNTicket

Ticket is ValidAuthNTicket

Connected

Portica gets License

Portica gets License

Active

Troubleshooting VDA Launch

• Event Logs (Web Interface, Controller, Storefront)

• Desktop Studio

• Broker Logs

• Workstation Agent

• Portica Logs

• Network Packet tracing

Case Study 3VDA Launch

Case Study Walk Through Background:

•They recently converted all images to a Citrix PVS image

•The original image worked

•All streamed images including the golden image failed to launch

Case Study 3: Launch Failure 1030

Search: Prepare

Troubleshooting :VDA Launch• Search Strings:

Checkpoint|connectionaccept|WaitforincomingConnection|sessionicaconnect

Troubleshooting :VDA Launch• Search Strings:

Checkpoint|connectionaccept|WaitforincomingConnection|sessionicaconnect

Root Cause analysis: MFAphook Module Failed to Load

53

• Conversion via provisioning server had changes the long name format of the drive

• mfaphook failed to load and this is needed for interaction with the OS.

Resolution: Add back short name to system see CTX133773 for more information

Tools

XD Tools

• HDX Monitor• CDF Control• Citrix Scout• Site Checker• Desktop Director

HDX Monitor

• Thinwire (Graphics)

• Direct 3D (Graphics)

• Media Stream (aka RAVE)

• Flash

• Audio

• USB Devices

HDX Monitor

• Mapped Client Drives (CDM)

• Branch Repeater

• Printer

• Client

• Smart Card

• Scanner

• System

Citrix Scout / XD Collector (CTX130147)

58

• Push button easy data collection system

• Makes data collection and upload push button easy

• Integrates data collected by Scout with the Citrix Tools as a

Service (TaaS) backend

• Simplifies data collection & analysis

#CitrixSummit

CDF Control: CTX111961

Tip:

•Use this tool to remotely

enable and collect CDF traces

when system are non

persistent

59

Site Checker Tool: CTX133767

• Enumerate Environment• Checks Services Status • Checks service instances registration status

• Reset Controllers Services instances into Database

Desktop Director

61

• Web Based

• Unified view of apps and desktops

• End-user details empower the help desk

• Includes HDX Monitor

• Access to personal vDisk tasks

Resources discussed

Optimal deployment recommendations• CTX124087 - XenDesktop Modular Reference Architecture

• CTX127939 - XenDesktop 5 Database Sizing and Mirroring Best Practices

• CTX123244 - High Availability for Desktop Virtualization - Reference

Architecture

• CTX120760 - XenDesktop - Design Handbook

• CTX128700 - XenDesktop Planning Guide - XenDesktop Scalability

• Whitepaper - Benchmarking Citrix XenDesktop using Login Consultants VSI

64

For More Information

• CTX132536 - Worker Unregisters at Session Launch

• CTX130147 - Citrix Scout

• CTX111961 - CDFControl

• CTX127492 - How to enable Controller Service Logging in XenDesktop 5

• CTX128075 - XDDBDiag: XenDesktop 5 Database Diagnostics

• CTX128909 - XenDesktop 5 Logon Process and Communication Flow

65

For More Information

• Vmware – Using VMware with XenDesktop

• SCVMM Using Microsoft SCVMM 2008 with XenDesktop

• CTX127538: How to Reconfigure a XenDesktop Site to Use a Mirrored

Database

• CTX127998 : Database Access and Permission Model for XenDesktop 5CTX133160 - LSQuery - License Server Data Collection Tool CTX127314 - How to Collect Data for Troubleshooting Licensing Issues

66

Takeaways

Presentation Goals RecapGoals Recap

68

Provide an understanding of the Provide an understanding of the

architecturearchitecture

Instruct On How To ConfigureInstruct On How To Configure

Provide Troubleshooting ResourcesProvide Troubleshooting Resources

Q & A

Before you leave…

70

• Conference surveys are available online at www.citrixsynergy.com starting Friday, May 24 at 9:00 a.m. PTᵒ Provide your feedback by 4:00 p.m. PT that day and you’ll receive a $30 Amazon.com

gift card via email

• Download presentations starting Monday, June 3, from your My Conference Planning tool located within the My Account section

Work better. Live better.Work better. Live better.