satisfy your technical curiosity hardcore production debugging ingo rammer

Satisfy Your Technical Curiosity

Hardcore Hardcore Production Debugging Production Debugging

Ingo RammerIngo Rammer

http://www.thinktecture.comhttp://www.thinktecture.com

Satisfy Your Technical CuriositySatisfy Your Technical Curiosity

Ingo Rammer and Ingo Rammer and Support and consulting for software architects and Support and consulting for software architects and developersdevelopers

Application Optimization and TuningApplication Optimization and TuningDeveloper-Coaching and -MentoringDeveloper-Coaching and -MentoringArchitecture and Code ReviewsArchitecture and Code ReviewsPrototyping and Architectural ConsultingPrototyping and Architectural Consulting

http://www.thinktecture.com [email protected]


TodayToday

ToolchestToolchestWinDbg/CDB + SOSWinDbg/CDB + SOSMdbg + MdbgSampleMdbg + MdbgSampleADPlusADPlusGFlagsGFlagsLogViewerLogViewerSOS Assist & DumpGenSOS Assist & DumpGenHawkeyeHawkeye


TodayToday

ProcessesProcessesTracing .NET memory leaksTracing .NET memory leaksAPI Call LoggingAPI Call LoggingKD and WinDbg to trace sync elements (mutex)KD and WinDbg to trace sync elements (mutex)Programmatically creating memory dumpsProgrammatically creating memory dumpsRemote debugging with WinDbg Remote debugging with WinDbg Service debugging with WinDbg and GflagsService debugging with WinDbg and GflagsCatching CLR Exceptions in WinDbgCatching CLR Exceptions in WinDbgAnd more …And more …

ScenariosScenarios


XCOPY ToolsXCOPY Tools

XCOPY – you don‘t have to install anything on a serverXCOPY – you don‘t have to install anything on a serverDefinitely not Visual StudioDefinitely not Visual StudioAll tools are freeAll tools are freeDebuggers are from Microsoft, rest is optionalDebuggers are from Microsoft, rest is optional

Easy to put it on most machines even in restrictive Easy to put it on most machines even in restrictive environments environments I carry them on a USB stickI carry them on a USB stick

Note: Rest of the slides are mainly for your reference. Note: Rest of the slides are mainly for your reference. We‘ll now switch to the tools!We‘ll now switch to the tools!


MDbgMDbg

Managed Code DebuggerManaged Code DebuggerSuccessor of cordbg (Common Object Runtime Successor of cordbg (Common Object Runtime Debugger)Debugger)Two different version!Two different version!

MDBG in the SDK MDBG in the SDK SupportedSupported

„„MDBG Sample“ as C# SourceMDBG Sample“ as C# SourceIL DisassemblyIL DisassemblyGUIGUIUnsupportedUnsupported, but that’s the MDbg , but that’s the MDbg youyou want! want!


MDbg CommandsMDbg Commands

ruru -> Run Application -> Run Applicationatat -> Attach to running process -> Attach to running processgg -> Go (Like F5 in Visual Studio) -> Go (Like F5 in Visual Studio)lolo -> Load Module (“LO ILDASM”) -> Load Module (“LO ILDASM”) ca exca ex -> Catch Exceptions -> Catch Exceptionsww -> Where am I? (Stack Trace) -> Where am I? (Stack Trace)tt -> Thread-List -> Thread-Listt <#>t <#> -> Thread selection (“t 3”) -> Thread selection (“t 3”)pp -> Print (local variable) -> Print (local variable)


MDbg with swallowed ExceptionsMDbg with swallowed Exceptions

Drag and Drop of the EXE to MDbgDrag and Drop of the EXE to MDbgOr: start Mdbg and use Or: start Mdbg and use RunRun or or AttachAttach

ca exca ex (Catch Exceptions)(Catch Exceptions)gg (Go) (Go)When the exception is caught:When the exception is caught:

ww (Stack Trace)(Stack Trace)gg (If needed: Go to continue)(If needed: Go to continue)


WinDbgWinDbg

Low Level Debugger (also Kernel Debugging)Low Level Debugger (also Kernel Debugging)Doesn‘t know about .NET, only unmanaged codeDoesn‘t know about .NET, only unmanaged codeExtension module SOS (Son Of Strike) gives access to Extension module SOS (Son Of Strike) gives access to internal .NET data structuresinternal .NET data structuresQuite complex, but extremely powerful:Quite complex, but extremely powerful:

Memory leak analysis (Memory leak analysis (all existing objectsall existing objects))Analysis of memory dumpsAnalysis of memory dumpsRemote debugging over TCP/IP (also with TCP forwarder as Remote debugging over TCP/IP (also with TCP forwarder as relay)relay)


WinDbg FundamentalsWinDbg Fundamentals.loadby sos mscorlib.loadby sos mscorlib -> load SOS from the directory, from which mscorlib -> load SOS from the directory, from which mscorlib

has been loaded. Only works *after* CLR has beenhas been loaded. Only works *after* CLR has beeninitialized in the debuggee.initialized in the debuggee.

GG -> go (F5)-> go (F5)CTRL+BREAKCTRL+BREAK -> Break-> Break~ ~ -> Thread list-> Thread list~#s~#s -> Thread selection (z.B. ~4s)-> Thread selection (z.B. ~4s)!clrstack!clrstack -> Stack trace-> Stack trace!help [cmd]!help [cmd] -> List of SOS commands-> List of SOS commands.hh [cmd].hh [cmd] -> Help for WinDbg commands-> Help for WinDbg commandsqq -> Quit-> Quitqdqd -> Quit, but detach first-> Quit, but detach first


WinDbg and ExceptionsWinDbg and ExceptionsWinDbg by default breaks at a few unmanaged exceptions (like Access Violations)WinDbg by default breaks at a few unmanaged exceptions (like Access Violations)

SXE <event>SXE <event> -> Break-> BreakSXI <event>SXI <event> -> Ignore-> IgnoreSXN <event>SXN <event> -> Output-> Output

SXE clrSXE clr -> Break on all CLR Exceptions-> Break on all CLR Exceptions

Important: NullReferenceException, DivideByZero are Important: NullReferenceException, DivideByZero are notnot CLR Exceptions, but CLR Exceptions, but regularregular unmanaged Exceptions unmanaged Exceptions

SXE avSXE av -> Break at Access Violation (null ref, default)-> Break at Access Violation (null ref, default)SXE dzSXE dz -> Break at Divide by Zero (default)-> Break at Divide by Zero (default)


WinDbg and Memory LeaksWinDbg and Memory LeaksWinDbg doesn‘t know about .NET and can therefore see WinDbg doesn‘t know about .NET and can therefore see moremore than any managed code than any managed code debuggerdebugger

!dumpheap!dumpheap -> All existing objects (reachable or not!)-> All existing objects (reachable or not!)!gcroot!gcroot -> garbage collection paths to an object-> garbage collection paths to an object!do <adr>!do <adr> -> Dump object-> Dump object!da <adr>!da <adr> -> Dump array-> Dump array!dso!dso -> Dump stack objects (not precise, but very important) -> Dump stack objects (not precise, but very important)

!dumpheap –stat!dumpheap –stat!dumpheap –type <className>!dumpheap –type <className>!gcroot <objectAddress>!gcroot <objectAddress>

Use together with HawkEye, to identify the class name of GUI elementsUse together with HawkEye, to identify the class name of GUI elements

.dump /ma c:\dumps\mydump.dmp.dump /ma c:\dumps\mydump.dmp -> Dump (/ma: mini dump with “All” option) -> Dump (/ma: mini dump with “All” option)


Memory Dump AnalysisMemory Dump Analysis

ADPlus ADPlus (Auto Dump Plus)(Auto Dump Plus)Creates memory dumps immediately Creates memory dumps immediately oror when a when a certain event (unmanaged) happens certain event (unmanaged) happens Dumps can be multiple 100 MBs in size, but can Dumps can be multiple 100 MBs in size, but can usually be ZIPped quite nicely usually be ZIPped quite nicely Dumps contain the complete process memory Dumps contain the complete process memory and and more (handle information)more (handle information)Also allow reconstruction of all loaded DLLs and Also allow reconstruction of all loaded DLLs and EXEsEXEs


ADPlusADPlus

C:\> C:\> adplus –hang –p <PID>adplus –hang –p <PID>C:\> C:\> adplus –hang –pn <processname>adplus –hang –pn <processname>C:\> C:\> adplus –crash –p <PID>adplus –crash –p <PID>


Global Flags (Gflags.exe)Global Flags (Gflags.exe)

Tool of Tool of Debugging Tools for WindowsDebugging Tools for WindowsAllows the immediate start of a debugger Allows the immediate start of a debugger when an application startswhen an application startsFor services: For services: Interact with DesktopInteract with Desktop has to be has to be activated if debugger is running as frontendactivated if debugger is running as frontend

You can use local You can use local remote debugging remote debugging if this is not if this is not desirabledesirable


KD.EXE KD.EXE

Kernel debugging for .NET applications … well?Kernel debugging for .NET applications … well?Find out about blocking situation regarding Kernel sync Find out about blocking situation regarding Kernel sync elementselementsImportant: current symbols!Important: current symbols!

C:\> mkdir c:\symbolsC:\> mkdir c:\symbolsC:\> set _NT_SYMBOL_PATH=SRV*C:\symbols*C:\> set _NT_SYMBOL_PATH=SRV*C:\symbols*http://msdl.microsoft.com/download/symbols


KD for kernel sync constructsKD for kernel sync constructsCommand: !process <processNumber> <flags> <name> (Command: !process <processNumber> <flags> <name> ( not PID!) not PID!)

C:\> KD /kl C:\> KD /kl (( local Kernel debugging) local Kernel debugging)

lkd> !process 0 0 <processName> lkd> !process 0 0 <processName> (( basic information about all basic information about all processes of this name)processes of this name)

PROCESSPROCESS 8900e020 8900e020 SessionId: 0 Cid: 1cd0 Peb: 7ffde000 ParentCid: 12a0SessionId: 0 Cid: 1cd0 Peb: 7ffde000 ParentCid: 12a0

lkd> !process lkd> !process 8900e0208900e020 2 2 (( 2: thread information with 2: thread information with locks)locks)

THREAD 89d86da8 Cid 1cd0.1ba0 Teb: 7ffda000 Win32Thread: 00000000 WAIT: (DelayExecution) THREAD 89d86da8 Cid 1cd0.1ba0 Teb: 7ffda000 Win32Thread: 00000000 WAIT: (DelayExecution) UserMode AlertableUserMode Alertable

89d86e98 NotificationTimer 89d86e98 NotificationTimer

THREAD 87ba5020 Cid 1cd0.17e0 Teb: 7ffd6000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode THREAD 87ba5020 Cid 1cd0.17e0 Teb: 7ffd6000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode AlertableAlertable

89d48fc0 Mutant - owning thread 89d86da8 89d48fc0 Mutant - owning thread 89d86da8

THREAD 88827360 Cid 1cd0.1d88 Teb: 7ffad000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode THREAD 88827360 Cid 1cd0.1d88 Teb: 7ffad000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode AlertableAlertable

89d48fc0 Mutant - owning thread 89d86da889d48fc0 Mutant - owning thread 89d86da8

In WinDbg/SOS: use Cid from KD to match !threads command‘s OSID column to find the offending threadIn WinDbg/SOS: use Cid from KD to match !threads command‘s OSID column to find the offending thread


Creating memory dumpsCreating memory dumps

DbgHelp.dll of the Debugging Tools for DbgHelp.dll of the Debugging Tools for Windows is redistributableWindows is redistributable

[DllImport("DbgHelp.dll", SetLastError=true)] private static extern bool MiniDumpWriteDump( IntPtr hProcess, int processId, IntPtr fileHandle, int dumpType, IntPtr excepInfo, IntPtr userInfo, IntPtr extInfo);

public static void CreateMiniDump(int pid, string outputFileName) {

using (FileStream stream = new FileStream(outputFileName, FileMode.Create, FileAccess.ReadWrite)) { using (Process proc = Process.GetProcessById(pid)) { int dumpType = (int)0x00000306; MiniDumpWriteDump(proc.Handle, proc.Id, stream.Handle, dumpType, IntPtr.Zero, IntPtr.Zero, IntPtr.Zero); } } }


Remote Debugging with WinDbgRemote Debugging with WinDbg

WinDbg/CDB can be started to act as a server and will listen WinDbg/CDB can be started to act as a server and will listen on a TCP endpoint for commandson a TCP endpoint for commands

CDB -server tcp:port=1234 CDB -server tcp:port=1234 <Application.exe><Application.exe>WinDbg –remote tcp:server=localhost,port=1234WinDbg –remote tcp:server=localhost,port=1234Or: WinDbg menu Or: WinDbg menu FileFileConnect to Remote SessionConnect to Remote Session

Can also be used with Gflags!Can also be used with Gflags!

Warning: there is no built-in securityWarning: there is no built-in security


Stopping on CLR ExceptionsStopping on CLR Exceptions

Note: the currently public versions of SOS.DLL expose critical Note: the currently public versions of SOS.DLL expose critical memory leaks which consume a few MB of RAM for each CLR memory leaks which consume a few MB of RAM for each CLR exception which is running through a filter like the following exception which is running through a filter like the following (even if the filter is NOT triggered)(even if the filter is NOT triggered)

!StopOnException –create System.IO.DirectoryNotFoundException 1!StopOnException –create System.IO.DirectoryNotFoundException 1


DownloadsDownloadsMDbg Sample MDbg Sample

Google: „mdbg sample“Google: „mdbg sample“

WinDbg, CDB, GFlagsWinDbg, CDB, GFlagsGoogle: „debugging tools for windows“Google: „debugging tools for windows“

Hawkeye Hawkeye Google: hawkeye .netGoogle: hawkeye .net

satisfy your technical curiosity hardcore production debugging ingo rammer

Documents