debugging xenapp & xendesktop

Debugging XenApp & XenDesktop Lalit Kaushal Escalation Engineer EMEA

Upload: camilla-todd

Post on 31-Dec-2015




4 download


Debugging XenApp & XenDesktop. Lalit Kaushal Escalation Engineer EMEA. Agenda. Overview of Common Components Troubleshooting Utilities Common Issues Troubleshooting Tips. Overview of Common Components. Putting It All Together.  Find “best” virtual desktop. - PowerPoint PPT Presentation


Page 1: Debugging XenApp & XenDesktop

Debugging XenApp & XenDesktopLalit KaushalEscalation Engineer EMEA

Page 2: Debugging XenApp & XenDesktop

• Overview of Common Components

• Troubleshooting Utilities

• Common Issues

• Troubleshooting Tips


Page 3: Debugging XenApp & XenDesktop

Overview of Common Components

Page 4: Debugging XenApp & XenDesktop

Putting It All Together





Active Directory with roaming profiles

DesktopDelivery Controller

Virtual Machines


Find “best” virtual desktop

Start VM

PXE-boot VM and stream OS


Connect using ICA

Acquire license and determine settings

Log in

Apply profile Deliver apps

Full range of authentication methods supported through web interface technology

Full support for SmartAccess and ICA session policies

Page 5: Debugging XenApp & XenDesktop

Common Components

• ICA Client

• Web Interface

• Active Directory



• DDC/ZDC (Although roles are a bit different)

Page 6: Debugging XenApp & XenDesktop

Troubleshooting Utilities

Page 7: Debugging XenApp & XenDesktop

• Understand the problem

• Where is the problem• Network• Server (all servers / one server)• Client (one client machine/ one client version/client type)• Data Store problem (corruption / inconsistency / configuration)

Before you begin

Page 8: Debugging XenApp & XenDesktop

Where to start?

• Collect Information• Frequency? Can I reproduce?• Determine Possible Causes/Effects• Get dumps, logs

• Tools• Determine necessary tools• Create a Setup

• Debug • Tools and Information to solve problem

Page 9: Debugging XenApp & XenDesktop

Solving the problem

• Determine accurate reproduction steps

• Find appropriate starting point to debug

• Crashes – Determine state (using global, stack, etc.)

• Debug against working model

• Use appropriate tools

Page 10: Debugging XenApp & XenDesktop

What tools are available?

• WINDBG – Windows Debugger

• CDFControl – CDF Tracing

• FILEMON – File Monitoring

• REGMON – Registry Monitoring

• PROCEXP – Process Explorer


Page 11: Debugging XenApp & XenDesktop


Page 12: Debugging XenApp & XenDesktop

• Combines Filemon and Regmon

Process Monitor

Page 13: Debugging XenApp & XenDesktop

• Process Explorer shows handles and DLLs processes

• Helpful to troubleshoot: • Memory Optimization issues• Application Streaming• Access issues

• Process Explorer is available from Microsoft

Process Explorer

Page 14: Debugging XenApp & XenDesktop

Sync Packet (SYN) Start of TCP session. Three way handshake (Syn, Syn-Ack, Ack) ICA session initialisation packets are transmitted next

Reset Packet (RST)Something has gone wrong, TCP session failed, unhandled closure of session

Finish Packet (FIN) Session is been closed in a handled manner

Push Packet (PSH) Data is been sent to receiving process directly

Ack Packet (ACK) Packet was received successfully by the remote device

Network Trace - Packets

Page 15: Debugging XenApp & XenDesktop

Session (Network trace)

Start of a session.

• • • End of session

Page 16: Debugging XenApp & XenDesktop


Page 17: Debugging XenApp & XenDesktop

• User Mode versus Kernel Mode

• The Windows operating system can be conceptually divided into 2 parts:• User Space (User Mode)• Kernel Space (Kernel Mode)

• Applications run in User Mode

• System drivers run in Kernel Mode (Privileged Mode)


Page 18: Debugging XenApp & XenDesktop















Page 19: Debugging XenApp & XenDesktop

• Microsoft definition: BSOD is a Fatal Exception Error or System failure

• Fatal exception errors:• Access to an illegal instruction has been

encountered • Invalid data or code has been accessed • The privilege level of an operation is invalid

• In most cases the exception is non-recoverable

• Dumps system memory to a file for debugging • Memory.dmp is placed on the System Drive• Requires free space equivalent to physical RAM + approx 12MB

Dump and Logs - BSOD

Page 20: Debugging XenApp & XenDesktop

• User dump – process memory • Live dump (snapshot) • Post-mortem dump (after crash)

• Kernel dump – OS kernel memory• Manual dump• Post-mortem dump (after BSOD)

• Complete dump – physical memory (kernel memory + processes)• Manual dump• Post-mortem dump (after BSOD)

Dumps & Logs - Types of Dumps

Page 21: Debugging XenApp & XenDesktop

• Dr Watson• Debugger generates a log file (Drwtsn32.log) & User Dump (user.dmp) when an

application exception or program error occur • Log file is cumulative, user.dmp overwritten• Set as the default debugger: drwtsn32.exe –I

• User Dump• Generates memory dump of specific process• Microsoft Knowledge Base Article – 241215

User Dump


Page 22: Debugging XenApp & XenDesktop

TestDefaultDebugger - CTX111901

Page 23: Debugging XenApp & XenDesktop

• Can generate a dump from a session

• No keyboard required

• Command line option available

• 32 / 64 bit

SystemDump - CTX111072

Description saved in dump

Page 24: Debugging XenApp & XenDesktop

• Citrix DumpCheck (Explorer Extension)

DumpCheck - CTX108825

Page 25: Debugging XenApp & XenDesktop

Common Issues

Page 26: Debugging XenApp & XenDesktop

Common Problems

• Server\Application Crash

• Server\Application Hang

• CPU Spikes

• Web Interface Debugging

Page 27: Debugging XenApp & XenDesktop

Server\Application Crash

Page 28: Debugging XenApp & XenDesktop

• Some method of capturing the fault is needed• Ntsd - • Windbg - • Userdump - • Dr Watson – • WER -

• Verify your chosen method works• TestDefaultDebugger –

• Have one of these methods enabled

Capturing Application Crash Dumps

Page 29: Debugging XenApp & XenDesktop

• Use tool analyze crash dumps•• Latest version is part of WDK (620mb download)• Earlier version are available as standalone download

Debugging tools for Windows

Page 30: Debugging XenApp & XenDesktop


Symbols – Huh?

Page 31: Debugging XenApp & XenDesktop

• .PDB – Program Database • Generated during compilation of the application by the vendor• Necessary to translate memory into something human readable..

•11010101001010101 = helloworld()

• Microsoft symbols Server - Essential•

• Citrix symbols ••  • SRV*c:\symcache*;SRV*c:\



Page 32: Debugging XenApp & XenDesktop

Tell Windbg where to find the symbols

Page 33: Debugging XenApp & XenDesktop

• Can use similar method for Kernel or User Dump analysis• !analyze –v• lmv m suspicousmodule• Update suspiciousmodule to latest version• Search if known stack trace

• Look at stack functions• Understand what the code was trying to do when it crashed

Analyzing crashes

Page 34: Debugging XenApp & XenDesktop

Systemdump_400000 makes a call into USER32

• Component names•DLL•EXE•SYSTEM Driver

Systemdump_400000 makes a call into ntdll Read upwards

Page 35: Debugging XenApp & XenDesktop

• The top of the stack is the last function executed• What caused the crash

• Look for non core OS components• Core OS module are usually not the fault• Closest to the top of the stack• Treat them as suspicious

• Find out via lmv command• Version

• Owner

• Timestamp

Review of the stack

Case Study: Using WinDbg to analyze IMA CrashCase Study: Using WinDbg to analyze IMA Crash

Page 36: Debugging XenApp & XenDesktop

Case Study

• Issue ReportedIMA frequently stopped unexpectedly on several server in the farm

• Data Collected• Collected User Dump

Page 37: Debugging XenApp & XenDesktop
Page 38: Debugging XenApp & XenDesktop

Case Study

• Issue ReportedIMA frequently stopped unexpectedly on several server in the farm

• Data Collected• Collected User Dump

• Analysis Done• !analyze –v• lmv m <modulename>

• Resolution• Uninstall Oracle Client 9.2 and update to 10.2

Page 39: Debugging XenApp & XenDesktop

Server\Application Hangs

Page 40: Debugging XenApp & XenDesktop

Server Hangs

• Dumps are not created automatically• Full memory dumps are most useful

• Need to force a dump• Systemdump -

• If server is not fully hung• Keyboard -• Hardware NMI Switch• Configure for full memory dump instead of kernel

Page 41: Debugging XenApp & XenDesktop

Analyzing Server Hangs

• Automatic analysis• !analyze –v –hang

•Not 100% reliable for full memory dumps• Lmv m suspectmodulename

• Check for locks• The 3 step programme with two new commands• !locks• Look for exclusive waiters

•Notice contention count• Look at the owner thread code

• !thread <threadID>

Page 42: Debugging XenApp & XenDesktop

• Force a crash of process• userdump.exe - • Vista/2008 – Available from Task Manager

• Same windbg commands again

• Automatic analysis usually good• !analyze –v –hang• Try and understand what code is doing from function names• Might have to chase the hang from one process to another

Analyzing application hangs

Case Study: Using WinDbg to analyze Server hangCase Study: Using WinDbg to analyze Server hang

Page 43: Debugging XenApp & XenDesktop

Case Study

• Issue Reported• XenApp server is hanging during logon

• Data Collected• Collected Kernel Dump

Page 44: Debugging XenApp & XenDesktop
Page 45: Debugging XenApp & XenDesktop

Case Study

• Issue Reported• XenApp server is hanging during logon

• Data Collected• Collected Kernel Dump

• Analysis Done• !analyze –v -hang• !locks

• Resolution• Involved Microsoft and recommended relevant Microsoft Hotfix

Page 46: Debugging XenApp & XenDesktop

CPU Spikes

Page 47: Debugging XenApp & XenDesktop

CPU Spikes

• Try to define a pattern (leverage perfmon)

• Determine offending Thread ID causing the spike (Process Explorer, QSlice)

• Obtain UserDump of offending process immediately after (Userdump.exe, WinDbg.exe)

• !runaway•WinDbg command to view thread times•Topmost thread is one to investigate

• Use application spy to look at what the application is doing (TracePlus, Logger)

ProcDump – New Microsoft Tool!!!

Page 48: Debugging XenApp & XenDesktop

• Microsoft command-line utility• To monitor an application for CPU Spikes• Generate a dump during spikes

• usage: procdump [-64] [[-c CPU usage] [-u] [-s seconds]] [-n exceeds] [-e] [-h] [-m commit usage] [-ma] [-o] [-r] [-t] < <process name or PID> [dump file]] | [-x <image file> <dump file> [arguments]>

• C:\>procdump -c 20 -n 3 -o pnamain c:\dump\pnamain


Page 49: Debugging XenApp & XenDesktop

ProcDump Demo

Page 50: Debugging XenApp & XenDesktop

Debugging WI

Page 51: Debugging XenApp & XenDesktop

Problem categories - UI


- Verify html code- Firebug- IE inspector- IE developer tools



- “This looks wrong”- Accessibility- Section 508- Browsers’ incompatibilities

Page 52: Debugging XenApp & XenDesktop

Problem categories - Logic

UI LogicCommunic


Weird or counter-intuitive behaviour Spec says different thing Configuration issues WI trace file Event log Live http headers

Page 53: Debugging XenApp & XenDesktop

Problem categories - Communication

UI LogicCommun


When it’s not WI’s fault New features Performance issues Capturing traffic (Wireshark, Fiddler) Capturing ICA file

Page 54: Debugging XenApp & XenDesktop

Troubleshooting Tips

Page 55: Debugging XenApp & XenDesktop

• Isolate the problem!• Does the issue affect Farm / Server / User ?

•Farm – Try new farm / Data Store•Server – Try different server / Clean build•User – Try new user or Administrator

• Does the issue affect ALL users ?• Is it the same in Fixed Window as Seamless ?• Does the problem happen via RDP ?

• 5 Why’s?

Troubleshooting Tips

Page 56: Debugging XenApp & XenDesktop

Authentication Issues

• What type of Authentication is configured?

• Is Application Enumeration works?

• Is Explicit\Prompt authentication works?

• Is Kerberos enabled?

• Capture Network Traffic

Page 57: Debugging XenApp & XenDesktop

Licensing Issues

• What’s the SA Date and is it valid for current Product?

• License are not Product specific (2007.0131)

• What’s LMC and ‘LMSTAT –a’ command showing?

• Are you able to Telnet LS from XenApp box and vice versa?

• Is customer using Citrix Option (Citrix.opt) file?

• Is product going into Grace period? If not, what’s the error?

• License Acquisition Error 500?

Page 58: Debugging XenApp & XenDesktop

Session Disconnection - Define the issue

Is the issue server-side or client-side ? Where is the network reset packet originating from (client/server)?Use Network and/or CDF tracing to identify the type of disconnection which is occurringIf issue is intermittent enable Connection auditing events to define the rate of the issue and follow up with usersIf client-side; check for network outages and process or device failures.

Narrow down steps required to reproduce the issue.

Rule out Keep-alives, other components, and timeouts. Map out patterns. (users always disconnected when shadowing, etc.) Identify if issue is related to users or group , subset of servers, network segment, build, or reported outages.

Page 59: Debugging XenApp & XenDesktop

Session Disconnection

WsxBrokenConnection: Reason=2, Source=2

What side of the connection the disconnection:

1 = Client 2 = Server

Reason why the disconnection occurred. Reason why the disconnection occurred.

wsxica 17896 04/17/2007 08:02:18.636 wsxica.c 2242 CDF_INFO WsxBrokenConnection: From WD: RequestedBPP: 0, SessionBPP: 0, Reason: 0 (overwriting context SessionBPP: 0)wsxica 17896 04/17/2007 08:02:18.636 wsxica.c 2260 CDF_INFO WsxBrokenConnection: Reason=2, Source=2wsxica 17896 04/17/2007 08:02:18.636 wsxica.c 2292 CDF_INFO WSXICA: BrokenConnection not terminatewsxica 17896 04/17/2007 08:02:18.636 wsxica.c 2299 CDF_INFO WsxBrokenConnection: open event Global\WFSHELL_DISCONNECT_1wsxica 17896 04/17/2007 08:02:18.636 wsxica.c 2313 CDF_INFO WSXICA: SetEvent disconnectwsxica 17896 04/17/2007 08:02:18.636 wsxica.c 2323 CDF_INFO WSXICA: release disconnect semaphore: Global\CPSVC_DISCONNECT_1wsxica 17896 04/17/2007 08:02:18.636 license.c 335 CDF_INFO ReleaseLicense: saved LogonId=1, fLimitChecksDone=1

User’s session was disconnect from CMCUser’s session was disconnect from CMC

Page 60: Debugging XenApp & XenDesktop

CDF Tracing (wsxBrokenConnection)

Disconnection matrix

Page 61: Debugging XenApp & XenDesktop

• BTG to the rescue!

• Topics include all current Citrix Products

• Ensures basic information is collected

• Helps to narrow down technical issues

• Faster resolution times

• Your feedback counts!

Brief Troubleshooting Guide

Page 62: Debugging XenApp & XenDesktop

Additional Information

• Citrix XenApp 5.0 for Windows Server 2008 Administrator's GuideKB Article CTX115519

• Getting Started with Citrix XenApp 5.0 KB Article CTX116418

• Brief Troubleshooting GuideKB Article CTX106727

• Troubleshooting Tools For Your Citrix EnvironmentKB Article CTX107572

• Citrix XenApp 5.0 Installation GuideKB Article CTX116573


Page 63: Debugging XenApp & XenDesktop

• Session surveys are available online at starting Thursday, 7 October• Provide your feedback and pick up a complimentary gift card at the registration


• Download presentations starting Friday, 15 October, from your My Organiser Tool located in your My Synergy Microsite event account

Before you leave…

Page 64: Debugging XenApp & XenDesktop