crash dump analysis
DESCRIPTION
DebuggingTRANSCRIPT
-
Crash Dump AnalysisSystem Debugging in Linux
Vtzslav BtrlaMartin DckJakub Jerm
-
CrashDumpAnalysisMFFUKSystemDebugginginLinux 2
Overview
Userspacedebuggingtools
Mostlyalreadymentioned gdbandfriends
Kerneldebuggingtools
Historicallylessofficialysupportedtools(invanillakernel) Differentdeveloper'sattitude printk()astheultimatedebuggingtool
-
CrashDumpAnalysisMFFUKSystemDebugginginLinux 3
LinuxKernelCrashDump
LKCDtoolchain(http://lkcd.sourceforge.net/)
Kernelpatchesforsimplecrashdumpgenerationonkernelpanic SupportedonlyonSCSIdevices
Inkerneldebuggerandpostmortemdebugger SimilarfeaturesasmdbK
Heuristicalanalysis
Activedevelopmentduring2.5.xtimeframe Lastsupportedreleasefrom2006
-
CrashDumpAnalysisMFFUKSystemDebugginginLinux 4
Standarddebuggingmeans
MagicSysRqhotkeys
Fordealingwithhangsandsecurityissues Operator'sinterventiontotherunningsystem Canbeenabled/disabledby/proc/sys/kernel/sysrq
Alt+SysRq+0..9 setconsolelogginglevel Alt+SysRq+C activatekexecandgenerateacrashdump Alt+SysRq+B immediatereboot Alt+SysRq+O immediateshutdown Alt+SysRq+S syncallmountedfilesystems Alt+SysRq+U remountallfilesystemsreadonly
-
CrashDumpAnalysisMFFUKSystemDebugginginLinux 5
Standarddebuggingmeans(2) Alt+SysRq+P dumpregisterstoconsole Alt+SysRq+T dumpprocessinformationtoconsole Alt+SysRq+L dumpstacktracesofrunningthreads Alt+SysRq+M dumpmemorystatisticstoconsole Alt+SysRq+D dumplockedlockstoconsole Alt+SysRq+K killallprocessesonthecurrentconsole Alt+SysRq+E terminateallprocessesexceptinit Alt+SysRq+I killallprocessesexceptinit Alt+SysRq+F executetheOOMkiller Alt+SysRq+N resetnicelevelofallrealtimeprocesses Alt+SysRq+X switchoffrawkeyboardmode
-
CrashDumpAnalysisMFFUKSystemDebugginginLinux 6
Standarddebuggingmeans(3) Activatefromcommandlinebywritinginto/proc/sysrqtrigger
Activateovernetworkbyaspecialsysrqdserver
RaisingElephantsIsSoUtterlyBoringRebootEvenIfSystemUtterlyBroken
Rawkeyboard SendSIGTERMtoallprocesses SendSIGKILLtoallprocesses Syncdatatodisk Remountallfilesystemsreadonly Reboot
-
CrashDumpAnalysisMFFUKSystemDebugginginLinux 7
Standarddebuggingmeans(4)
kprobes
Basickernelinstrumentation Anexecutionofanykernelfunctioncantriggeracallbackfunction
kexec
Bootinganew(Linux)kernelwithoutphysicalreboot Physicalmemorycanbe(moreorless)retainedandanalyzedbythenewkernel(asacrashdump)
-
CrashDumpAnalysisMFFUKSystemDebugginginLinux 8
UserModeLinux
UML
Specialpseudohardwarearchitecture Otherwisecompatiblewiththetargetarchitecture RunningLinuxkernelasauserspaceprocess
Originallyavirtualizationeffort
Greatfordebuggingandkerneldevelopment Aplainstandardgdbcanbeusedtoattachtotherunningkernel GuestthreadsarethreadsoftheUMLprocess
Slightlymorecomplicatedtofollowprocesses
-
CrashDumpAnalysisMFFUKSystemDebugginginLinux 9
KDB
Linuxkerneldebugger
http://oss.sgi.com/projects/kdb Asetofpatchestothevanillakernel(veryunprobablytobeintegratedsoon)
Forlocaldebugging
PossibilitiesandfeaturesverysimilartomdbK Activationin/proc/sys/kernel/kdb BreakinbythePausekey(Ctrl+Aonserialconsole)
md memorydisplay mm memorymodify
-
CrashDumpAnalysisMFFUKSystemDebugginginLinux 10
KDB(2) rd registerdump rm registermodify bt stacktrace bp setbreakpoint...
-
CrashDumpAnalysisMFFUKSystemDebugginginLinux 11
KGDB
KernelGDB
http://kgdb.sourceforge.net/
Since2.6.26integratedinthevanillakernel(finally!) Client/serverdebuggerforremotekerneldebugging
Overseriallineorethernet Client:Plaingdb
Assemblyandsourcelinedebugging Symbolinformationfromvmlinuxbinaryontheclient
Activation AfterloadingtheI/Omoduleandconnectingtotheserver Justafterbootwhenkgdbwaitkerneloptionisused
-
CrashDumpAnalysisMFFUKSystemDebugginginLinux 12
KGDB(2)
Features Memoryandregisteranalysisonkerneloopsorpanic Limitedsinglesteppingoftherunningkernel
%gdb./vmlinux(gdb)setremotebaud115200(gdb)targetremote/dev/ttyS0
%gdb./vmlinux(gdb)targetremoteudp:192.168.1.1.:6443
Slide 1Slide 2Slide 3Slide 4Slide 5Slide 6Slide 7Slide 8Slide 9Slide 10Slide 11Slide 12