satisfy your technical curiosity hardcore production debugging ingo rammer
Post on 17-Jan-2016
219 Views
Preview:
TRANSCRIPT
Satisfy Your Technical Curiosity
Hardcore Hardcore Production Debugging Production Debugging
Ingo RammerIngo Rammer
http://www.thinktecture.comhttp://www.thinktecture.com
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
Ingo Rammer and Ingo Rammer and Support and consulting for software architects and Support and consulting for software architects and developersdevelopers
Application Optimization and TuningApplication Optimization and TuningDeveloper-Coaching and -MentoringDeveloper-Coaching and -MentoringArchitecture and Code ReviewsArchitecture and Code ReviewsPrototyping and Architectural ConsultingPrototyping and Architectural Consulting
http://www.thinktecture.com ingo.rammer@thinktecture.com
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
TodayToday
ToolchestToolchestWinDbg/CDB + SOSWinDbg/CDB + SOSMdbg + MdbgSampleMdbg + MdbgSampleADPlusADPlusGFlagsGFlagsLogViewerLogViewerSOS Assist & DumpGenSOS Assist & DumpGenHawkeyeHawkeye
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
TodayToday
ProcessesProcessesTracing .NET memory leaksTracing .NET memory leaksAPI Call LoggingAPI Call LoggingKD and WinDbg to trace sync elements (mutex)KD and WinDbg to trace sync elements (mutex)Programmatically creating memory dumpsProgrammatically creating memory dumpsRemote debugging with WinDbg Remote debugging with WinDbg Service debugging with WinDbg and GflagsService debugging with WinDbg and GflagsCatching CLR Exceptions in WinDbgCatching CLR Exceptions in WinDbgAnd more …And more …
ScenariosScenarios
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
XCOPY ToolsXCOPY Tools
XCOPY – you don‘t have to install anything on a serverXCOPY – you don‘t have to install anything on a serverDefinitely not Visual StudioDefinitely not Visual StudioAll tools are freeAll tools are freeDebuggers are from Microsoft, rest is optionalDebuggers are from Microsoft, rest is optional
Easy to put it on most machines even in restrictive Easy to put it on most machines even in restrictive environments environments I carry them on a USB stickI carry them on a USB stick
Note: Rest of the slides are mainly for your reference. Note: Rest of the slides are mainly for your reference. We‘ll now switch to the tools!We‘ll now switch to the tools!
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
MDbgMDbg
Managed Code DebuggerManaged Code DebuggerSuccessor of cordbg (Common Object Runtime Successor of cordbg (Common Object Runtime Debugger)Debugger)Two different version!Two different version!
MDBG in the SDK MDBG in the SDK SupportedSupported
„„MDBG Sample“ as C# SourceMDBG Sample“ as C# SourceIL DisassemblyIL DisassemblyGUIGUIUnsupportedUnsupported, but that’s the MDbg , but that’s the MDbg youyou want! want!
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
MDbg CommandsMDbg Commands
ruru -> Run Application -> Run Applicationatat -> Attach to running process -> Attach to running processgg -> Go (Like F5 in Visual Studio) -> Go (Like F5 in Visual Studio)lolo -> Load Module (“LO ILDASM”) -> Load Module (“LO ILDASM”) ca exca ex -> Catch Exceptions -> Catch Exceptionsww -> Where am I? (Stack Trace) -> Where am I? (Stack Trace)tt -> Thread-List -> Thread-Listt <#>t <#> -> Thread selection (“t 3”) -> Thread selection (“t 3”)pp -> Print (local variable) -> Print (local variable)
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
MDbg with swallowed ExceptionsMDbg with swallowed Exceptions
Drag and Drop of the EXE to MDbgDrag and Drop of the EXE to MDbgOr: start Mdbg and use Or: start Mdbg and use RunRun or or AttachAttach
ca exca ex (Catch Exceptions)(Catch Exceptions)gg (Go) (Go)When the exception is caught:When the exception is caught:
ww (Stack Trace)(Stack Trace)gg (If needed: Go to continue)(If needed: Go to continue)
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
WinDbgWinDbg
Low Level Debugger (also Kernel Debugging)Low Level Debugger (also Kernel Debugging)Doesn‘t know about .NET, only unmanaged codeDoesn‘t know about .NET, only unmanaged codeExtension module SOS (Son Of Strike) gives access to Extension module SOS (Son Of Strike) gives access to internal .NET data structuresinternal .NET data structuresQuite complex, but extremely powerful:Quite complex, but extremely powerful:
Memory leak analysis (Memory leak analysis (all existing objectsall existing objects))Analysis of memory dumpsAnalysis of memory dumpsRemote debugging over TCP/IP (also with TCP forwarder as Remote debugging over TCP/IP (also with TCP forwarder as relay)relay)
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
WinDbg FundamentalsWinDbg Fundamentals.loadby sos mscorlib.loadby sos mscorlib -> load SOS from the directory, from which mscorlib -> load SOS from the directory, from which mscorlib
has been loaded. Only works *after* CLR has beenhas been loaded. Only works *after* CLR has beeninitialized in the debuggee.initialized in the debuggee.
GG -> go (F5)-> go (F5)CTRL+BREAKCTRL+BREAK -> Break-> Break~ ~ -> Thread list-> Thread list~#s~#s -> Thread selection (z.B. ~4s)-> Thread selection (z.B. ~4s)!clrstack!clrstack -> Stack trace-> Stack trace!help [cmd]!help [cmd] -> List of SOS commands-> List of SOS commands.hh [cmd].hh [cmd] -> Help for WinDbg commands-> Help for WinDbg commandsqq -> Quit-> Quitqdqd -> Quit, but detach first-> Quit, but detach first
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
WinDbg and ExceptionsWinDbg and ExceptionsWinDbg by default breaks at a few unmanaged exceptions (like Access Violations)WinDbg by default breaks at a few unmanaged exceptions (like Access Violations)
SXE <event>SXE <event> -> Break-> BreakSXI <event>SXI <event> -> Ignore-> IgnoreSXN <event>SXN <event> -> Output-> Output
SXE clrSXE clr -> Break on all CLR Exceptions-> Break on all CLR Exceptions
Important: NullReferenceException, DivideByZero are Important: NullReferenceException, DivideByZero are notnot CLR Exceptions, but CLR Exceptions, but regularregular unmanaged Exceptions unmanaged Exceptions
SXE avSXE av -> Break at Access Violation (null ref, default)-> Break at Access Violation (null ref, default)SXE dzSXE dz -> Break at Divide by Zero (default)-> Break at Divide by Zero (default)
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
WinDbg and Memory LeaksWinDbg and Memory LeaksWinDbg doesn‘t know about .NET and can therefore see WinDbg doesn‘t know about .NET and can therefore see moremore than any managed code than any managed code debuggerdebugger
!dumpheap!dumpheap -> All existing objects (reachable or not!)-> All existing objects (reachable or not!)!gcroot!gcroot -> garbage collection paths to an object-> garbage collection paths to an object!do <adr>!do <adr> -> Dump object-> Dump object!da <adr>!da <adr> -> Dump array-> Dump array!dso!dso -> Dump stack objects (not precise, but very important) -> Dump stack objects (not precise, but very important)
!dumpheap –stat!dumpheap –stat!dumpheap –type <className>!dumpheap –type <className>!gcroot <objectAddress>!gcroot <objectAddress>
Use together with HawkEye, to identify the class name of GUI elementsUse together with HawkEye, to identify the class name of GUI elements
.dump /ma c:\dumps\mydump.dmp.dump /ma c:\dumps\mydump.dmp -> Dump (/ma: mini dump with “All” option) -> Dump (/ma: mini dump with “All” option)
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
Memory Dump AnalysisMemory Dump Analysis
ADPlus ADPlus (Auto Dump Plus)(Auto Dump Plus)Creates memory dumps immediately Creates memory dumps immediately oror when a when a certain event (unmanaged) happens certain event (unmanaged) happens Dumps can be multiple 100 MBs in size, but can Dumps can be multiple 100 MBs in size, but can usually be ZIPped quite nicely usually be ZIPped quite nicely Dumps contain the complete process memory Dumps contain the complete process memory and and more (handle information)more (handle information)Also allow reconstruction of all loaded DLLs and Also allow reconstruction of all loaded DLLs and EXEsEXEs
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
ADPlusADPlus
C:\> C:\> adplus –hang –p <PID>adplus –hang –p <PID>C:\> C:\> adplus –hang –pn <processname>adplus –hang –pn <processname>C:\> C:\> adplus –crash –p <PID>adplus –crash –p <PID>
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
Global Flags (Gflags.exe)Global Flags (Gflags.exe)
Tool of Tool of Debugging Tools for WindowsDebugging Tools for WindowsAllows the immediate start of a debugger Allows the immediate start of a debugger when an application startswhen an application startsFor services: For services: Interact with DesktopInteract with Desktop has to be has to be activated if debugger is running as frontendactivated if debugger is running as frontend
You can use local You can use local remote debugging remote debugging if this is not if this is not desirabledesirable
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
KD.EXE KD.EXE
Kernel debugging for .NET applications … well?Kernel debugging for .NET applications … well?Find out about blocking situation regarding Kernel sync Find out about blocking situation regarding Kernel sync elementselementsImportant: current symbols!Important: current symbols!
C:\> mkdir c:\symbolsC:\> mkdir c:\symbolsC:\> set _NT_SYMBOL_PATH=SRV*C:\symbols*C:\> set _NT_SYMBOL_PATH=SRV*C:\symbols*http://msdl.microsoft.com/download/symbols
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
KD for kernel sync constructsKD for kernel sync constructsCommand: !process <processNumber> <flags> <name> (Command: !process <processNumber> <flags> <name> ( not PID!) not PID!)
C:\> KD /kl C:\> KD /kl (( local Kernel debugging) local Kernel debugging)
lkd> !process 0 0 <processName> lkd> !process 0 0 <processName> (( basic information about all basic information about all processes of this name)processes of this name)
PROCESSPROCESS 8900e020 8900e020 SessionId: 0 Cid: 1cd0 Peb: 7ffde000 ParentCid: 12a0SessionId: 0 Cid: 1cd0 Peb: 7ffde000 ParentCid: 12a0
lkd> !process lkd> !process 8900e0208900e020 2 2 (( 2: thread information with 2: thread information with locks)locks)
THREAD 89d86da8 Cid 1cd0.1ba0 Teb: 7ffda000 Win32Thread: 00000000 WAIT: (DelayExecution) THREAD 89d86da8 Cid 1cd0.1ba0 Teb: 7ffda000 Win32Thread: 00000000 WAIT: (DelayExecution) UserMode AlertableUserMode Alertable
89d86e98 NotificationTimer 89d86e98 NotificationTimer
THREAD 87ba5020 Cid 1cd0.17e0 Teb: 7ffd6000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode THREAD 87ba5020 Cid 1cd0.17e0 Teb: 7ffd6000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode AlertableAlertable
89d48fc0 Mutant - owning thread 89d86da8 89d48fc0 Mutant - owning thread 89d86da8
THREAD 88827360 Cid 1cd0.1d88 Teb: 7ffad000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode THREAD 88827360 Cid 1cd0.1d88 Teb: 7ffad000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode AlertableAlertable
89d48fc0 Mutant - owning thread 89d86da889d48fc0 Mutant - owning thread 89d86da8
In WinDbg/SOS: use Cid from KD to match !threads command‘s OSID column to find the offending threadIn WinDbg/SOS: use Cid from KD to match !threads command‘s OSID column to find the offending thread
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
Creating memory dumpsCreating memory dumps
DbgHelp.dll of the Debugging Tools for DbgHelp.dll of the Debugging Tools for Windows is redistributableWindows is redistributable
[DllImport("DbgHelp.dll", SetLastError=true)] private static extern bool MiniDumpWriteDump( IntPtr hProcess, int processId, IntPtr fileHandle, int dumpType, IntPtr excepInfo, IntPtr userInfo, IntPtr extInfo);
public static void CreateMiniDump(int pid, string outputFileName) {
using (FileStream stream = new FileStream(outputFileName, FileMode.Create, FileAccess.ReadWrite)) { using (Process proc = Process.GetProcessById(pid)) { int dumpType = (int)0x00000306; MiniDumpWriteDump(proc.Handle, proc.Id, stream.Handle, dumpType, IntPtr.Zero, IntPtr.Zero, IntPtr.Zero); } } }
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
Remote Debugging with WinDbgRemote Debugging with WinDbg
WinDbg/CDB can be started to act as a server and will listen WinDbg/CDB can be started to act as a server and will listen on a TCP endpoint for commandson a TCP endpoint for commands
CDB -server tcp:port=1234 CDB -server tcp:port=1234 <Application.exe><Application.exe>WinDbg –remote tcp:server=localhost,port=1234WinDbg –remote tcp:server=localhost,port=1234Or: WinDbg menu Or: WinDbg menu FileFileConnect to Remote SessionConnect to Remote Session
Can also be used with Gflags!Can also be used with Gflags!
Warning: there is no built-in securityWarning: there is no built-in security
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
Stopping on CLR ExceptionsStopping on CLR Exceptions
Note: the currently public versions of SOS.DLL expose critical Note: the currently public versions of SOS.DLL expose critical memory leaks which consume a few MB of RAM for each CLR memory leaks which consume a few MB of RAM for each CLR exception which is running through a filter like the following exception which is running through a filter like the following (even if the filter is NOT triggered)(even if the filter is NOT triggered)
!StopOnException –create System.IO.DirectoryNotFoundException 1!StopOnException –create System.IO.DirectoryNotFoundException 1
Satisfy Your Technical Curiosity
Satisfy Your Technical CuriositySatisfy Your Technical Curiosity
DownloadsDownloadsMDbg Sample MDbg Sample
Google: „mdbg sample“Google: „mdbg sample“
WinDbg, CDB, GFlagsWinDbg, CDB, GFlagsGoogle: „debugging tools for windows“Google: „debugging tools for windows“
Hawkeye Hawkeye Google: hawkeye .netGoogle: hawkeye .net
Satisfy Your Technical Curiosity
top related