debugging net applications with windbg

Debugging .NET Applications with WinDBG

Cory Foyhttp://coryfoy.com | @cory_foy

Agenda

.NET Fundamentals Debugging Crashes Debugging Memory

Leaks Debugging Hangs WinDBG Tips and

Tricks Other Tools

.NET Fundamentals

CLR Fundamentals Memory Management Basics Debugging Fundamentals

CLR Fundamentals

Managed Assembly Layout

PE Header

CLR Header

IL (Intermediate Language)

Metadata Header

Metadata

CLR Fundamentals

.NET Runtime in a Process

Demo: Viewing Managed Objects


Viewing Managed Objects

Start up the app and click on the Managed Objects button

Start up WinDBG and attach to the executable (click “No” if it asks you to save the workspace) then load SOS by typing “.loadby sos mscorwks” and hitting enter. Also load symbols by typing “.symfix C:\symbols” and hitting enter, then typing “.reload” and hitting enter


View All Threads by typing in “~” and hitting enter. View all managed threads by typing in “!threads” and hitting enter. Note there are 5 native threads, but only 2 managed threads. Also note that thread 2 is marked as (Finalizer) – that’s the thread responsible for finalization

Change to the main thread by typing “~0s” (tilde zero s). This tells WinDBG to switch to thread 0. Then type “!clrstack” to see where we are in the application. Note we are in the Main method right now.


Next, let’s look at the objects on the heap. Type “!dumpheap –stat”. You’ll see many objects. You can filter by typing things like -min or –max as parameters to dumpheap.

Now find all of our objects on the heap by typing “!dumpheap –type WinDBG”. This filters for any object with the text “WinDBG” in it’s name. The top list are the actual objects and their memory locations, while the bottom list are the type names. You can figure out which is which by matching up the MethodTable addresses (MT Column)


We can view a specific object using “!dumpobj” (or “!do” for short). We’ll look at the HelloWorld Object by typing “!do <address>” where address is from the !dumpheap command earlier. We can see we have a string field called “hello” and a world property.

We can then look at the string by typing “!do <address>”. In this example, the address would be 263a000. We can see that field contains the string “Hello”.


When we did a !do on the HelloWorld object, one piece of information was the MethodTable address. We can use the !dumpmt command to see what methods the object exposes. Type “!dumpmt –md <methodtableaddress>”. If the JIT Column has “PreJIT” then the method came from an Ngen’d object. If it has JIT, then the method has been called and JIT’d. If it has NONE, the method hasn’t been called.

You can dissassemble IL that has been JIT’d by passing in the MethodDesc address to “!U”


If we’ve attached to the process doing a live debug (which we’re doing here), then you can set breakpoints using !bpmd. For example, we can have it breakpoint just before the MessageBox shows up by passing in the assembly name and fully qualified type. You can use bl to see the breakpoints.

We then type “g” to release the current break we have on the app, and click the Managed Objects button again.

We’ll then see WinDBG hit the breakpoint. We can run !clrstack to see what led us to that call

CLR Fundamentals

Threads Managed Threads is an object – it lives

on native system threads CLR Threads▪ Finalizer▪ Garbage Collector (Server GC)▪ Debugger▪ Timer▪ Threadpool (I/O and Worker)

CLR Fundamentals

Just-In-Time compilation Program makes a call to a method The .NET Runtime checks to see if the

method has been called before If so, it executes the JIT’d code If not, it compiles the IL code and stores

it in memory, updating the MethodDesc

.NET Fundamentals

CLR FundamentalsMemory Management Basics Debugging Fundamentals

Memory Management

Stacks versus Heaps Stack – First in / First Out Heap – access by address

Garbage Collector Heap Where all objects are stored Broken into 3 generations and one Large

Object Heap Large Object > 85,000 bytes

Memory Management

Garbage Collector Sequence Suspend Execution Engine Mark objects without roots Plan (budgets, fragmentation) Sweep (delete market objects) Compact (move leftover objects to back

of heap) Restart Execution Engine

Memory Management

Memory Leaks Possible in both Managed and

Unmanaged Code Use Perfmon to check for symptoms Unmanaged Leak▪ Private Bytes Increase, #Bytes In All Heaps

stays flat Managed Leak▪ Both Private Bytes and Bytes In All Heaps

increase▪ Need multiple dump files

Memory Management

http://blogs.msdn.com/tess/archive/2008/03/17/net-debugging-demos-lab-6-memory-leak-review.aspx

Memory Management

Why do we leak managed memory? Objects not being released “Pinned” memory Finalized Objects (Destructors on

Managed Objects)▪ Finalized objects require an additional GC

cycle to be cleaned since they have to go in the finalizer thread to run

Memory Management

Exception Handling Workflow Exception Occurs (create an exception

object) Notify Debugger (1st Chance Exception) Look for a handler by walking up the call

stack If handler found, let it handle exception If not, throw a 2nd Chance Exception and

terminate the process

.NET Fundamentals

CLR Fundamentals Memory Management BasicsDebugging Fundamentals

Debugging Fundamentals

Typical Problems in Production System hangs or deadlocks Fatal Exceptions Data Loss or inconsistency Performance Problems Excessive Memory Usage App Pool Restarts (slow access to

ASP.NET pages)


Approaching the problem Be the application (visualize what

could cause the problem) Use Application and System Logs Try to reproduce in Development or

Staging environments Create a hypothesis and use

WinDBG to validate


Debugging Production Apps Typically don’t have Visual Studio

installed, or access to the remote debugger

Attaching a debugger freezes all threads

Capture memory dumps▪ At time of crash / exception▪ Over time to troubleshoot hangs / leaks

Agenda

.NET FundamentalsDebugging

Crashes Debugging Memory


Tricks Other Tools

Debugging Crashes

Application seems to work fine, but something happens Unhandled Exception Dialog App “disappears”

Steps Desktop App: Attach and stop on

Exceptions Web App: Use DebugDiag to capture

memory dump

Demo: Debugging Crashes


Debugging Crashes

Open Application and click the “Crash” button”. Unhandled Dialog should appear and kill the app. Start app back up and attach to it with WinDBG, then load SOS. Once SOS is loaded, type “sxe clr” to tell WinDBG to break on all .NET Exceptions, then type “g” and click on the Crash button again.

Because we’ve enabled exception notification, WinDBG breaks on the first-chance exception.

Debugging Crashes

The first place to look is the exception object itself. We can either do a “!do” on the object address listed, or simply type “!pe”. You can also pass an address to !pe if you need to view other than the last exception thrown.

We can see there is a file not found exception. Let’s see where we were at.

So it looks like the app calls “ReadAllText” from a method called “Crash_Click” from the Form1 object. If we have access to Source, we’d start there. If not, we can find the method address and dissassemble

Debugging Crashes

Note that if we click “g” at this point, WinDBG breaks again. This is because the exception we first saw was *not* the crash reason. Like the previous exception, this one is listed as a First Chance Exception.

If we click “g” again, we’ll see WinDBG breakpoint again. Note that it clearly tells us this is a second chance exception – the next thing that will happen is process termination.

Demo: Debugging Crashes on Application Startup


Debugging Crashes On Startup

Open the App and check the “Crash on Startup” option, then close the app and restart. Notice it immediately crashes.

The challenge is that we won’t have time to attach a debugger. You can set registry keys to automatically launch a debugger, or we could launch the app from the debugger, which we’ll do here.

Start up WinDBG and go to File->Open Executable. Browse to the WinDBGDemo executable and select it.


At this point, we are in the PE load of the application, meaning it hasn’t even decided if this is a native or managed app yet. Which also means no .NET libraries are loaded, so we can’t load SOS.

What we can do is set a breakpoint to fire when the .NET runtime is loaded, at which point we can set the handlers we need. Type “sxe ld mscorwks” which means set WinDBG to break when mscorwks is loaded. We can now load SOS and set the CLR Exception breakpoints. Now type “g” till our app breaks. You can now debug it as a crash.


If you have a Windows Service which is crashing on startup, then you’ll need to modify some registry keys and permissions on the service. For more information, see the following KB article, or the blog post from our site

(Note: When you finish this demo, go to your user directory\AppData\Local\Microsoft\WinDBGDemo.exe\1.0.0.0 and modify the user.config file to have CrashOnStartup to be False)

http://support.microsoft.com/kb/824344

http://blog.coryfoy.com/2008/06/debugging-a-net-windows-service-that-is-crashing-on-startup/

Agenda

.NET Fundamentals Debugging CrashesDebugging

Memory Leaks Debugging Hangs WinDBG Tips and

Tricks Other Tools

Debugging Memory Leaks

Memory Usage Continues to grow in the app

May cause app pool restarts in ASP.NET

Out of Memory exceptions when you still have memory free

Several Reasons Unreleased objects “Pinned” memory You have plenty of memory! (GC only

runs when it feels memory pressure)


Diagnosing Memory Leaks PerfMon to determine if it is a managed

leak Take multiple memory dumps to

determine what is happening For the objects on the heap, find out why

they are sticking around Check the GC Generation for the objects

Demo: Debugging Memory Leaks



Start the app and task manager, then click on the Memory Leak button. Note that the memory is still increasing.

Fire up Perfmon to see if this is a managed or native leak. Start->Run->Perfmon. You can remove the Processor Time counter and add the following counters: .NET CLR Memory -> #Bytes in all Heaps -> WinDBGDemo and Process -> Private Bytes -> WinDBGDemo.

You should see that both are growing at about the same rate indicating a managed leak


To debug this, we’ll need to take two snapshots of the application far enough apart to see memory difference. You can do this right from Task Manager. Create two, about 15 seconds apart. You can now kill the application.

Open two instances of WinDBG and open each memory dump file you created in a different instance of WinDBG. Load up SOS in each one as well.


What we’re wanting to do is compare the two memory dumps to see if we can spot the culprit. Using !VMStat on each file we can see that memory usage is certainly increasing.

And if we do a “!dumpheap –stat” on each one, we can see something quite startlingly – 230,000 additional objects are on the heap!


Normally you’d have to dig through the heap to find which objects increased, but we see something interesting. An object called “LeakForm” has had 900 instances created. Perhaps those aren’t being cleaned up?

Let’s look at one. We’ll see all the instances by doing a “!dumpheap -type WinDBGDemo.LeakForm” then picking any object at random to do a !do on

With that object address, let’s see what is holding on to it with “!GCRoot <address>”


We see that the LeakForm is being held on to an object, which is of type System.Object[]. If we do a !do on it, we don’t see much of interest

But if we compare the arrays between the first dump and the second dump using the “!da <address>” command, we find a clue. Between the first and second memory dumps, all of the extra LeakForms were added to this array!

At this point, we examine the source code to see what is going on.

Agenda


LeaksDebugging

Hangs WinDBG Tips and

Tricks Other Tools

Debugging Hangs

Two types of hangs Low CPU Hang▪ Generally caused by a deadlock

High CPU Hang▪ App in a tight loop, or under heavy load

Debugging Hangs

Diagnosing Hangs High CPU Hang▪ Multiple memory dumps▪ Compare which thread is getting processor

time Low CPU Hang▪ Walk through the waiting threads and match

locks▪ Very tedious process

Demo: Debugging High CPU Hangs


Debugging High CPU Hangs

Start the app and Task Manager. Click on the High CPU Hang. You should see a single processor spike (100% on 1CPU). In this example, I have a 4CPU system, so a CPU of 25% indicates it is fully utilizing one processor.

Take two memory dumps from Task Manager about 15 seconds apart. You can then kill the application.


Open the two memory dump files in different WinDBG instances and load SOS.

The first thing we need to know is if a single thread is using up all of the time, indicating a possible loop. We can use the command !runaway to see which threads are getting CPU time

We can see from this that thread 0 was using up all of the time in between the two memory dumps


So what is thread 0 doing? By running !clrstack on each memory dump, we can see that the call stacks are the same, possibly indicating that this is the culprit.

I say possibly because if a bunch of calls are happening to different methods, we very well could have gotten “lucky” and seen the same method. Always verify your assumptions.

In this case, looking at the source (either real source, or through reflector) we see the following. I think we have our culprit

Agenda


Leaks Debugging HangsWinDBG Tips and

Tricks Other Tools

Demo: WinDBG Tips and Tricks


WinDBG Tips and Tricks

You can get quite a bit of help with both WinDBG and SOS.

For the WinDBG help content, type “.hh” and it will open the WinDBG CHM file. You can also type “.hh <command>” and it will take you to that section

For SOS, you can type “!help” to see all available commands, or “!help <command>” to see a specific one


You can use the shell command to run the output of a WinDBG command to an external process and the results are displayed back into the UI. For example, to quickly find the shared domain, you can run ‘.shell –ci “!dumpdomain” find /i “shared domain”’

You can also take advantage of this to do integrations with PowerShell


When viewing stacks, you can use !clrstack to view the managed stack. You can also use the command “k” to view the native stack

You can also output the stack for all threads by combining the “~” command with a star (for all threads) and the command. For example “~*k” or “~*e!clrstack” (which means execute the command !clrstack)


You can do a lot with automation of WinDBG. For example, you can have WinDBG execute commands when certain breakpoints are hit

You can also automate what happens when CLR Exceptions are hit by using the –c option with sxe. In this example, every time we hit an exception we would spit out the exception and CLR stack trace.


You can also loop through various elements. For example, if we wanted to dump out all strings greater than 5k that were on the heap, we could do the following with the “.foreach” command

Finally, if you want to adjust the colors, you can do that in View->Options

Agenda



TricksOther Tools

Additional Tools

DebugDiag Used to automate the capture of

exceptions for IIS processes ADPlus

Used to capture dumps from command line

Managed Debuggers (CorDbg, MDbg) Managed Debugging Assistants

(Visual Studio) Profilers

More Information

http://blogs.msdn.com/tess microsoft.com/whdc/DevTools/

Debugging http://windbg.info http://www.coryfoy.com foyc at cory foy dot com @cory_foy on Twitter Slides will be posted on CoryFoy.com

debugging net applications with windbg

Technology

debugging

net runtime

chance exception

memory dumps

windbg tips

crash button

hitting enter

dumpheap stat