crash (or) hang dump analysis using windbg in windows platform by k.s.shanmugasundaram

Crash Dump Analysis using WinDbg

By K.S.Shanmuga sundaram

2013


About Training

+

2013


Session - 1

2013


Agenda – Session1

Understanding Dump File 1

Varieties of Dump File 2

Creation of Dump File 3

Terminologies for analyzing of Dump File 4

Introduction to WinDbg 5

2013


Varieties of Bugs

Bugs

Arithmetic

Logical

syntax

Resource Multi

Threaded

Perform -ance

User Interface

2013


Impact of Bug on Application

Application may Malfunction

Application may Crash

Application may Hang

2013


Application crash

Arises due to unhandled exception

2013


Application Hang

May arise due to Deadlock. May arise due to High CPU usage.

2013


How to Fix Crash / Hang Issue?

1. Store Application state i.e.

Process(s) Info, Thread(s) Info, Loaded Module Info, Register Info, Memory Info, Handle Info, Callstack Info, Etc.,

2. Start Analyze to find the root cause of the issue.

Process Dump

Information

2013


What is Dump File?

Definition:- Static snap shot of an application at any given time.

Fact sheet

File Format Binary File Format

File Extension

.DMP - Stands for Dump

Contents Process, Thread, Stack,Callstack, Handles, Modules, etc.,

Usage (i) Debugging Crashing application. (ii) Debugging Hanging application.

Other terms

Crash dump, Core Dump, Hang Dump, Heap Dump, Memory Dump, Process dump or just dump.

2013


Agenda – Session1






2013


Physical Memory Layout

User Space

Kernel Space

Kernel Process

User Process

Kernel Process

Kernel Process

User Process

User Process

Access Restricted

2013


Physical Memory Layout – 32 bit

User Space

Kernel Space

4 GB

2 GB

0

2 32

2013


Physical Memory Layout – 32 bit

User Space

Kernel Space

4 GB

3 GB

0

with /3GB switch in boot.ini file

2 32

2013


Types of Dump File

2013


Agenda – Session1






2013


Dump File creation

2013


Dump File creation using Win32 API

BOOL MiniDumpWriteDump (

HANDLE hProcess,

DWORD ProcessId,

HANDLE hFile,

MINIDUMP_TYPE DumpType,

PMINIDUMP_EXCEPTION_INFORMATION ExceptionParam,

PMINIDUMP_USER_STREAM_INFORMATION UserStreamParam,

PMINIDUMP_CALLBACK_INFORMATION CallbackParam

);

2013



API Name MiniDumpWriteDump

DLL Dbghelp.dll

Header file Dbghelp.h

Arguments

HANDLE hProcess Target process handle.

DWORD ProcessId Target process ID.

HANDLE hFile Dump file handle.

MINIDUMP_TYPE DumpType Type of information to be written dump file

PMINIDUMP_EXCEPTION_INFORMATION ExceptionParam,

Pointer to Exception info like ThreadID, Exception pointers.

PMINIDUMP_USER_STREAM_INFORMATION UserStreamParam

Pointer to user defined information. [Optional]

PMINIDUMP_CALLBACK_INFORMATION CallbackParam

Callback to receive extended dump information. [Optional].

2013



Argument MINIDUMP_TYPE DumpType

Description

Enumerator constants. It can have any one of the values

MiniDumpNormal

MiniDumpWithDataSegs

MiniDumpWithFullMemory

MiniDumpWithHandleData

MiniDumpFilterMemory

MiniDumpScanMe

MiniDumpWithUnloadedModules

MiniDumpWithIndirectlyReferencedMe

mory

MiniDumpFilterModulePaths

MiniDumpWithProcessThreadData

MiniDumpWithPrivateReadWriteMe

mory

MiniDumpWithoutOptionalData

MiniDumpWithFullMemoryInfo

MiniDumpWithThreadInfo

MiniDumpWithCodeSegs

2013


Kernel Dump creation using System settings

2013


Agenda – Session1






2013


Symbols

Global variables Local variables Function names and the addresses of their entry points Frame pointer omission (FPO) records Source File name and line number for each Instruction

Functions (except for functions declared static) Global variables specified as extern (and any other global variables visible across multiple object files)

Also called as Full Symbol

Also called as Stripped Symbol

Stores Debugging information about a program

2013


Symbols in detail

Name of the item Address of the item in virtual memory Frame pointer omission (FPO) records for each function Data type of each variable, structure, and function Types and names of the parameters for each function Scope of each local variable Symbols associated with each line in each source file

The name of the item. The address of the item in the virtual memory space of its module. For a function, this is the address of its entry point. Frame pointer omission (FPO) records for each function.

2013


Symbols Relation

Public symbol data can be thought of as a subset of the private symbol .

2013


Symbol File Format

PDB stands for “Program Database”

Fact sheet

Microsoft proprietary file format (.pdb)

Extracted from Source files ( .cpp,.c, ,cs, etc.,).

PDB file information extracted by using the DIA (Debug Interface Access) SDK.

A .NET PDB file only contains only Source Filename, Line number , Local variable names as .NET Metadata already contains rest of the symbols.

Visual studio 5.0 generates .dbg file format.

2013


Dump file relation

• DMP

• PDB • DBG

• C • CPP • H • CS

• EXE • DLL • LIB

2013


Application Dependency

2013


Dump File Dependency

IMAGE SYMBOLS SOURCES

Required Files For

Dump File

Analysing

2013


Process & Threads

User Space

Kernel Space

User Process

Thread 1

Thread 2

Thread N

1 MB Stack Memory

1 MB Stack Memory

1 MB Stack Memory

2013


Stack

User

Space

Kernel Space

Stack Memory – Thread 1

Committed

Guard

Reserved Stack Memory – Thread 2

Stack Memory – Thread 3

Stack Growth

2013


Callstack

Call Stack

Method2() Frame 2

Method1 Frame 1

Main() Frame 0

Void Method1(int) { Method2(40,50); } Void Method2(int,int) { Do something… } Void main() { Method1(20); }

2013


C compiler Name Mangling

Function Name

Function Address

Method1() 0x1234

Method1() 0x3423

Method2() 0x1654

Void Method1(int) { Do something… } Void Method1(char) { Do something… } Void Method2(int,int) { Do something… } Void main() { Method1(20); Method1(‘a’); Method2(40,50); }

Typical Representation

__Method1_0x1234

__Method1_0x3423

__Method2_0x1654

2013


C++ compiler Name Mangling

Void Method1(int) { Do something… } Void Method1(char) { Do something… } Void Method2(int,int) { Do something… } Void main() { Method1(20); Method1(‘a’); Method2(40,50); }

Typical Representation

__Method1_1_@INT_0x1234

__Method1_1_@CHAR_0x3423

__Method2_2_@INT_@INT_0x1654

Function Name

No of Param

Param Type

Function Address

Method1() 1 Int 0x1234

Method1() 1 Char 0x3423

Method2() 2 Int,Int 0x1654

2013


Calling convention

2013


Calling convention comparison

2013


Exception Dispatching

Debugger

Frame Handlers

Operation System

Default Post Mortem Debugger

Windows Error Reporting

First Chance

exception

Second Chance

exception

Unhandled exceptions

1 2

3

4

6

8

Exception

5 7

9

2013


C++ Exception Test

void TestMethod { int Temp = 100; Temp = Temp /(Temp-Temp); } void main() { try { TestMethod(); }

catch(…) { printf(“ Exception Caught”); } }

Is Program

will crash?

2013


C++ Exception Answer

C/C++ --> Code Generation --> Enable C++ Exceptions

C++ Exceptions

Command line Option

Remarks

No No Exception handled.

Yes /EHsc Only C++ Exception handled.

With SEH /EHa Both C++ and SEH exception handled. (Destructor will not be called on stack unwinding)

2013


Terminologies

2013


Agenda – Session1






2013


WinDbg – A Introduction

Microsoft Windows Debugger

GUI Debugger.

Both user and kernel mode debugger.

Non-Invasive Debugger.

Freeware.

Light weight

2013


WinDbg – Command Types

Command Types Description

Regular Commands Used debug processes

Meta or Dot-Commands usually to control the behavior of the debugger

Extension Commands implemented as exported functions in extension DLLs

2013


WinDbg – Commands

Command Types Examples

Regular Commands K

Meta or Dot-Commands .cls, .reload, .time

Extension Commands !analyze, !locks, !teb, !peb

2013


WinDbg Commands

Command Description

!analyze –v Analyze Exceptions.

!analyze -hang –v Analyze Hanging.

!gle Get Last Error.

.lastevent Get last exception Event.

.ecxr Display Exception context.

!sym noisy While loading symbols displays error information. Be default it is OFF.

K Display Call stack.

ld * Load all Modules.

!for_each_frame Display call stack with Frame number.

.frame N Set “N” Frame as context

2013


WinDbg Commands

Command Description

.reload /f /v /i Load all modules forecefully with verbose

!teb Thread Environment block ( Formatted output)

!peb Process Environment block ( Formatted output)

!runaway Displays information about time consumed by each thread.

dt nt!_TEB Full Thread Environment block

~ Display all threads Info

|| Display process Info

!wow64exts.sw Switches 64bit dump file into 32bit.

dt ModulName!* Display all available data type in a module.

dt ModulName!Type Display declaration for a given data type.

2013


WinDbg Commands

Command Description

.symopt Display Current Symbol loading settings

.symopt +0x40 Forcefully try to match PDB with Target Binary. set's flag of "SYMOPT_LOAD_ANYTHING". It mean it will not check for timestamp

!locks Display critical section Info

!handles Display handles Info

2013


Any Questions?

Session - 1

2013


Session - 2

2013


Agenda – Session2

Postmortem Debugger Settings 1

WinDbg Setup 2

Dump File creation using Tools 3

Dump File creation using Win32 API 4

Dump File Analyzing case study 5

2013


Agenda – Session2

Postmortem Debugger Setup 1

WinDbg Setup 2




2013


Registry Path

\\HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT \CurrentVersion \ AeDebug

Registry Values

Value Name Remarks

Debugger Debugger Executable full path

Auto

0 - message box will be displayed prior to postmortem debugging. 1 - No message box.

Postmortem Debugger Setup Registry Settings - Windows XP and prior OS.

2013


Registry Path

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps

In case of 64 bit set corrresponding registry locations (wow64).

Registry Path

Value Name Value Data

DumpFolder Dump file complete path

DumpCount Total Dump file count

DumpType 0 - custom, 1 - Mini, 2 – Full

CustomDumpFlags

Valid if DumpType=0, Possible hexadecimal values = 0,1,2,4,8,10,20,40,80,100,200,400,800,1000,2000,4000,10000, 20000,40000,80000,100000,1fffff. Refer MSDN for details.

Postmortem Debugger Setup Registry Settings - Windows Server 2008, Vista with SP1 and 7.0.

2013


Postmortem Debugger Setup Default Enabling

Tool Settings

Dr.Watson drwtsn32 –I

WinDbg WinDbg –I

Visual Studio Tools -> options --> Debugging --> Just-In-Time Enable All

2013


Agenda – Session2


WinDbg Setup 2




2013


WinDbg Setup Environment Variables

_NT_SYMBOL_PATH

_NT_ALT_SYMBOL_PATH

_NT_DEBUGGER_EXTENSION_PATH

_NT_EXECUTABLE_IMAGE_PATH

_NT_SOURCE_PATH

2013


WinDbg Setup Symbol, Image & Source path Setup

Path Type

File type Files

Symbol path

Program database files (.pdb) Target application, Windows binaries, and all dependents binaries.

Source path

Source files (.cpp & .h, .cs) Target application, CRT libraries, and all dependent binaries.

Image Path

Binary files (.exe, .dll) Target application, Windows dll, and all dependents.

2013


Agenda – Session2


WinDbg Setup 2




2013


Dump File Creation Using Tools

Tool Remarks

Process Explorer

Right Click Create Dump

ProcDump procdump.exe -ma Target.exe

WinDbg WinDbg -pv -pn Target.exe -c ".dump /ma DumpFileName.dmp;q"

Task Manager

Right Click Create Dump

Dr.Watson Drwatson.exe (Enable Create Dump File)

2013


Agenda – Session2


WinDbg Setup 2




2013


Dump File Creation Using Win32 API

Refer Sample MFC_1

2013


Agenda – Session2


WinDbg Setup 2




2013


Case Study - 1

Exception type Access violation exception

Sample code 1 – Refer Sample MFC_1

int* i = NULL; *i = 100;

Sample code 2

Printf(“%s%s%s%s%s%s%s%s%s%s”);

2013


Case Study - 2

Exception type Stack Overflow


Void TestMethod() { unsigned char Temp[1024*1024*1024]; }

Sample code 2

Void TestMethod() { TestMethod(); }

2013


Case Study - 3

Exception type Divide by Zero

Sample code 1 - Refer Sample MFC_1

Void TestMethod() { int nTemp = 100; nTemp = nTemp / (nTemp - nTemp ) }

2013


Case Study - 4

Exception type High CPU usage


UINT ThreadProc1(LPVOID pParam_in) { while(1); eturn 1; } UINT ThreadProc2(LPVOID pParam_in) { for(long i=0;i<200000;i++); return 1; } UINT ThreadProc3(LPVOID pParam_in) { for(long i=0;i<900000;i++); return 1; }

2013


Case Study - 5

Exception type Deadlock


UINT ThreadProc1(LPVOID pParam_in) { CommonMethod(); return 1; } UINT ThreadProc2(LPVOID pParam_in) { CommonMethod(); return 1; } void CommonMethod() { EnterCriticalSection(&CriticalSection); //LeaveCriticalSection(&CriticalSection); }

2013


References

"Microsoft Windows Internals” by Mark

Russinovich and David Solomon 5th Edition

"Advanced Windows Debugging" by Mario Hewardt and Daniel

Pravat

“Memory Dump Analysis Anthology“ by Dmitry Vostokov

2013


Any Questions?

Session - 2

2013


Questions?

For more details, please contact

http://www.linkedin.com/in/shanmugasundaramks

[email protected]

K.S.Shanmuga sundaram

Software Architect,

Bangalore, India



mailto:[email protected]

crash (or) hang dump analysis using windbg in windows platform by k.s.shanmugasundaram

Education

memory dump

dump file4introduction

process dump

heap dump

core dump

othertermscrash dump

shanmuga sundaramhow

shanmuga sundaramwhat