BRUCE DAWSONVALVE

GETTING STARTED DEBUGGING ON LINUX(MAKING IT EASY IN LESS THAN AN HOUR)

Linux DebuggingChallenges:Default debugger is intimidating to new usersTough to get symbols and source to show upMany tricks needed for efficient debuggingYou can be productive on Linux, quickly

Main TopicsChoosing a debuggerGetting symbols to show upGetting source code to show upTips and tricks

Choosing a Debugger: gdb

Choosing a Debugger: cgdb

Choosing a debugger: VisualGDBIntegrates into VisualC++Remote build/debugUsed by some Valve developersCommercial product

Choosing a Debugger: QtCreator

QtCreator DemoCreating a projectBuildingFixing errorsDebuggingCode exploration

Install from http://qt-project.org/downloads#qt-creatorLatest version is 3.0.0Must mark the .run file as executable before runningGetting QtCreator

Install from http://qt-project.org/downloads#qt-creatorLatest version is 3.0.0Must mark the .run file as executable before runningGetting QtCreator

Install from http://qt-project.org/downloads#qt-creatorLatest version is 3.0.0Must mark the .run file as executable before runningGetting QtCreator

QtCreatorCan use for full edit/build/run/debug cycleFile-> New File or Project-> Import Project-> Import Existing ProjectImports all files from the specified directoryWill run make in that directory, assumes makefileCan use cmake or run any custom build command you want

QtCreator buildingSummarizes warnings and errors in Issues tabCan double-click to jump to location of error/warning

QtCreator DebuggingVS compatible keyboard shortcuts (F5, F10, F11, etc.)Important exception: Ctrl+F5Can load core files, attach to processes, launch processes, etc.Other debug windows available from Window-> ViewsThreads windowRegisters windowDebugger log (for invoking raw gdb commands)

QtCreator: Go-to AnythingCtrl+K is the universal Go-To commandSimilar to Ctrl+, in VS 2010+ goes to source files: goes to C++ classes, enums, and functionsl goes to line numberFuzzy matchingF2 goes to the definition of a symbolAlt left/right goes back/forward through navigation history

QtCreator Debug EnvironmentLD_LIBRARY_PATH needed for many gamesRequired for Steam runtimerun.sh sets up runtime environment, lets print it:$ run.sh printenv LD_LIBRARY_PATH/data/valve/steam-runtime/bin/../runtime/amd64/lib/x86_64-linux-gnu:/data/valve/steam-runtime/bin/../runtime/amd64/lib:/data/valve/steam-runtime/bin/../runtime/amd64/usr/lib/x86_64-linux-gnu:/data/valve/steam-runtime/bin/../runtime/amd64/usr/lib:/data/valve/steam-runtime/bin/../runtime/i386/lib/i386-linux-gnu:/data/valve/steam-runtime/bin/../runtime/i386/lib:/data/valve/steam-runtime/bin/../runtime/i386/usr/lib/i386-linux-gnu:/data/valve/steam-runtime/bin/../runtime/i386/usr/lib:/data/clients/tf2/game/bin:/data/valve/steam-runtime/bin/../runtime/amd64/lib/x86_64-linux-gnu:/data/valve/steam-runtime/bin/../runtime/amd64/lib:/data/valve/steam-runtime/bin/../runtime/amd64/usr/lib/x86_64-linux-gnu:/data/valve/steam-runtime/bin/../runtime/amd64/usr/lib:/data/valve/steam-runtime/bin/../runtime/i386/lib/i386-linux-gnu:/data/valve/steam-runtime/bin/../runtime/i386/lib:/data/valve/steam-runtime/bin/../runtime/i386/usr/lib/i386-linux-gnu:/data/valve/steam-runtime/bin/../runtime/i386/usr/lib:/data/valve/steam-runtime/bin/../runtime/amd64/lib/x86_64-linux-gnu:/data/valve/steam-runtime/bin/../runtime/amd64/lib:/data/valve/steam-runtime/bin/../runtime/amd64/usr/lib/x86_64-linux-gnu:/data/valve/steam-runtime/bin/../runtime/amd64/usr/lib:/data/valve/steam-runtime/bin/../runtime/i386/lib/i386-linux-gnu:/data/valve/steam-runtime/bin/../runtime/i386/lib:/data/valve/steam-runtime/bin/../runtime/i386/usr/lib/i386-linux-gnu:/data/valve/steam-runtime/bin/../runtime/i386/usr/lib:Copy LD_LIBRARY_PATH setting to QtCreator

QtCreator Debug EnvironmentLD_LIBRARY_PATH needed for many gamesRequired for Steam runtimerun.sh sets up runtime environment, lets print it:$ run.sh printenv LD_LIBRARY_PATH/data/valve/steam-runtime/bin/../runtime/amd64/lib/x86_64-linux-gnu:/data/valve/steam-runtime/bin/../runtime/amd64/lib:/data/valve/steam-runtime/bin/../runtime/amd64/usr/lib/x86_64-linux-gnu:/data/valve/steam-runtime/bin/../runtime/amd64/usr/lib:/data/valve/steam-runtime/bin/../runtime/i386/lib/i386-linux-gnu:/data/valve/steam-runtime/bin/../runtime/i386/lib:/data/valve/steam-runtime/bin/../runtime/i386/usr/lib/i386-linux-gnu:/data/valve/steam-runtime/bin/../runtime/i386/usr/lib:/data/clients/tf2/game/bin:/data/valve/steam-runtime/bin/../runtime/amd64/lib/x86_64-linux-gnu:/data/valve/steam-runtime/bin/../runtime/amd64/lib:/data/valve/steam-runtime/bin/../runtime/amd64/usr/lib/x86_64-linux-gnu:/data/valve/steam-runtime/bin/../runtime/amd64/usr/lib:/data/valve/steam-runtime/bin/../runtime/i386/lib/i386-linux-gnu:/data/valve/steam-runtime/bin/../runtime/i386/lib:/data/valve/steam-runtime/bin/../runtime/i386/usr/lib/i386-linux-gnu:/data/valve/steam-runtime/bin/../runtime/i386/usr/lib:/data/valve/steam-runtime/bin/../runtime/amd64/lib/x86_64-linux-gnu:/data/valve/steam-runtime/bin/../runtime/amd64/lib:/data/valve/steam-runtime/bin/../runtime/amd64/usr/lib/x86_64-linux-gnu:/data/valve/steam-runtime/bin/../runtime/amd64/usr/lib:/data/valve/steam-runtime/bin/../runtime/i386/lib/i386-linux-gnu:/data/valve/steam-runtime/bin/../runtime/i386/lib:/data/valve/steam-runtime/bin/../runtime/i386/usr/lib/i386-linux-gnu:/data/valve/steam-runtime/bin/../runtime/i386/usr/lib:Copy LD_LIBRARY_PATH setting to QtCreator

Loading Files from the TerminalTo open in an existing QtCreator instance:$ qtcreator.sh -client Can wrap this in an alias:$ alias qtedit='~/qtcreator-3.0.0/bin/qtcreator.sh -client'

Symbols!

Symbol Sanity: Others CodeGetting symbols for libc6$ sudo apt-get install libc6-dbg (or libc6-dbg:i386)Puts symbols in /usr/lib/debug/CopyOfSoPathGetting symbols for the Steam run-timeRun Steam from the terminal like this:STEAM_RUNTIME=debug steamWill download and use debug version of runtime (symbols and source)Or, download steam-runtime SDK

Assume libc6 is in /lib/x86_64-linux-gnu/libc-2.15.soInstalled symbols go to /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.15.sogdb and other debuggers automatically look there

Symbol StrippingOur convention is:.so is code and minimal symbols.so.dbg is code and full debug info (everything).so is shipped, .so.dbg is archivedArchiving a file with full symbols and full code is usefulOne file provides everythingWorks with tools that cant handle stripped symbols

Symbol StrippingCopy symbols from .so to .so.dbg$ objcopy .so .so.dbgOptionally add --only-keep-debug to strip codeAdd a debug link from .so to .so.dbg$ objcopy --add-gnu-debuglink=.so.dbg .soRemove debug information from .so$ strip -S .soOptionally add -x to strip more informationSee gendbg.sh in the source-sdk-2013 for examples

Symbol Sanity: Your CodeYou can put your symbols in /usr/lib/debug/SoPathOr side-by-side with your .so filesOr leave debuginfo in your .so filesBetter yet, use a symbol server*

* apologies for Microsoft-speak

Symbol Servers (on Linux)Just a file server and a conventionBased on build IDs (40 hex digits)Step 1: tell gdb to look for symbols in a second locationAssume symbol server directory is /mnt/syms(gdb) set debug-file-directory /usr/lib/debug:/mnt/symsPut command in ~/.gdbinit

Adding to Symbol ServersExtract the build ID $ readelf -n .so Build ID: 6d5f7575de387ed72286 (shortened for slide purposes)Copy the .so.dbg file somewhere and make a link to it$ cp .so.dbg (somewhereonserver)$ mkdir -p /mnt/syms/.build-id/6d$ ln -s (somewhereonserver) /mnt/syms/.build-id/6d/5f7575de387ed72286$ ln -s (somewhereonserver) /mnt/syms/.build-id/6d/5f7575de387ed72286.debug

Our Symbol ServerFile paths made from product name, file name, build ID, then file name againArchived files contain both binary code and debug info (symbols)Two links point to each file

/mnt/syms/.build-id/6d/5f7575de387ed72286/mnt/syms/.build-id/6d/5f7575de387ed72286.debug/mnt/syms/tf2/client.so.dbg/6d5f7575de387ed72286/client.so.dbg

Symbol Server UsesDebuggers automatically retrieve binary and debug infoYou can put libc6 symbols in symbol serverYou can write scripts to retrieve unstripped symbol filesHandy for tools that cant handle stripped symbols or ignore symbol servers

Source SanitySource for locally built binaries will just workBuild machine binaries need remapping(gdb) set substitute-path /home/buildbot/tf2/build/src /data/clients/tf2/srcPut in ~/.gdbinitGet libc6 source and add to gdb search paths:$ apt-get source libc6(gdb) directory /data/home/bruced/libcsource/eglibc-2.15/stdio-common/(gdb) directory /data/home/bruced/libcsource/eglibc-2.15/malloc/Put in ~/.gdbinit

Tips and Tricks

Linux Library Loadinglddprints shared library dependenciesUsed to diagnose why a module wont loadtf2/game$ ldd hl2_linuxlinux-gate.so.1 => (0xf7780000)libtcmalloc_minimal.so.4 => not foundlibdl.so.2 => /lib/i386-linux-gnu/libdl.so.2libc.so.6 => /lib/i386-linux-gnu/libc.so.6/lib/ld-linux.so.2Often fixed by setting LD_LIBRARY_PATH

Linux Library LoadingLD_PRELOAD specify shared objects to load first, can override symbols$ LD_PRELOAD="/usr/lib/libtcmalloc.so lsLD_DEBUG debug process loading. Example:$ LD_DEBUG=all ls 2>out.txt

Better On LinuxValGrind runs process on a virtual CPU, analyzes every memory access. Finds leaks, overruns, and uninitialized variablesstrace trace system calls. Sample usage:$ strace -p $(pidof procname)Attach to process$ strace -o out.txt lsLaunch process

Dumpbin Replacementsnm list symbols in a shared objectobjdump -d disassemble an object file

More Tips and TricksForcing old compilers to add build IDs:-Wl,--build-idGetting build IDs from a core file (Linux crash dump)eu-unstrip -n --core corefile

Attaching to processes may require root privileges or disabling of ptrace hardeningptrace hardening is a security feature to stop debuggers from attaching to running processesEither elevate gdb before attaching or disable ptrace hardening:$ sudo -i# echo 0 > /proc/sys/kernel/yama/ptrace_scope

Ptrace hardening (security)

List all files opened by a particular process:$ lsof -p $(pidof steam)List all processes that have a file open$ lsof /lib/i386-linux-gnu/libc-2.15.solsof LiSt Open Files

ReferencesBlogging about symbols: http://randomascii.wordpress.com/2013/01/19/symbols-on-linux-part-two-symbols-for-other-versions/QtCreator: http://qt-project.org/downloads#qt-creator http://richg42.blogspot.com/2013/10/a-shout-out-to-qtcreator-28x-on.html http://richg42.blogspot.com/2013/10/qtcreators-python-debug-visualizers.html http://linux-debugger-bits.blogspot.com/2014/01/qtcreator-projects.htmlSteam run-time SDK: https://github.com/ValveSoftware/steam-runtime/blob/master/sdk/README.txtSymbol servers on Linux: http://fedoraproject.org/wiki/Releases/FeatureBuildId#Find_files_by_build_IDPtrace hardening: https://wiki.ubuntu.com/SecurityTeam/Roadmap/KernelHardening#ptrace_Protection

Questions?bruced@valvesoftware.comAsk questions nowOr drop by the Linux break-out session at 5:00 in this room (6C)

Extra slides follow

QtCreator Disassembly QuirksDisassembly is shown when Operate by Instruction is selectedDisassembly may omit some instructions especially callDisassembly may not show correct rangeCan work around by right-clicking on EIP in registers window and selecting Open Disassembler at 40057c: mov edi,0x400747400581: mov eax,0x0400586: call 0x400440 40058b: mov esi,0x10400590: mov edi,0x40075d400595: mov eax,0x040059a: call 0x400440 40059f: movabs rax,0x400921fb54442d184005a9: mov QWORD PTR [rbp-0x8],rax4005ad: call 0x400544 4005b2: movsd xmm0,QWORD PTR [rbp-0x8]40057c: mov edi,0x400747400581: mov eax,0x0 12 printf("kFooBar == %u\n", kFooBar);40058b: mov esi,0x10400590: mov edi,0x40075d400595: mov eax,0x0 14 double pi = 3.14159265358979323;40059f: movabs rax,0x400921fb54442d184005a9: mov QWORD PTR [rbp-0x8],rax 16 InlineDebugTest();4005b2: movsd xmm0,QWORD PTR [rbp-0x8]

Ive been developing on Microsoft Windows for about twenty years, including eight years where I worked at Microsoft. But about a year ago I decided to join Valves Linux team.

I initially found debugging on Linux to be intimidating for a developer accustomed to Visual Studio on Windows. But Ive learned how to make it easy sometimes easier than development on Windows and today I want to shared this with you. This talk explains how to get comfortable debugging on Linux, including which debuggers Valve has found useful, how to get symbols and source code to show up, and how to confidently investigate typical bugs. I also talk about some other Linux tools that are commonly needed by developers.

This talk is targeted at clearing debugging obstacles for new Linux users

I only started working on Linux about a year ago, and I found the learning curve frustrating. The wealth of unfamiliar choices some of them with interfaces that seem alien to a long-time Visual Studio user made it hard to be productive.

My hope is that this talk will help new Linux users by giving some suggestions on best practices.

There is no single best way to do Linux development so these suggestions will seem wrong or misguided to some people, especially more experienced Linux developers. Thats fine. Im not saying anybody has to follow these recommendations, but they have worked for me.

This talk assumes a Debian based OS translate as needed

This is the rough outline of the topics that I want to cover, since these represent some of the biggest productivity obstacles when I started.One line of source shown, need to learn command line options to step, set breakpoints, view more source, view locals, view registers, view disassembly, etc. Big learning curve.

And, for me, command-line debuggers do not make me as productive.

Better I really like seeing source context.

But, cryptic commands still needed.

Some people like gdb or cgdb. To them I say, if you like your debugger, you can keep your debugger. But I want something that works better for me. I want something that gives me a rich display of the state of my program, and doesnt require learning cryptic commands.

This is more like it. Source window (can flip to disassembly), locals, call stack, breakpoints, registers, threads, program output, etc.Can also browse and edit source code and build. Its a full IDE, and its not just for creating Qt apps.

Impressive, no? And, some Valve developers have been using it heavily on industrial strength projects so we can say with some confidence that it works reliably enough to use for real work.

How much would you expect to pay?

Impressive, no? And, some Valve developers have been using it heavily on industrial strength projects so we can say with some confidence that it works reliably enough to use for real work.

How much would you expect to pay?

Impressive, no? And, some Valve developers have been using it heavily on industrial strength projects so we can say with some confidence that it works reliably enough to use for real work.

How much would you expect to pay?

Windows users are used to symbol servers. Point _NT_SYMBOL_PATH at Microsofts symbol server and your internal symbol server and your symbols (also known as debug information) are automatically loaded by VS, windbg, xperf, and every other tool that needs them.

On Linux the situation is more complex, but it is quite possible to get to a very similar state.objdump -d (disassembling a binary) is one example of a tool that doesnt support separated symbolsThe first three alternatives work for many scenarios, but they fall down if you are looking at crash-dumps/core-files from older builds, such as getting crash reports from the field. They also dont answer the question of how you find the right symbol files to put in /usr/lib/debug or side-by-side.

A symbol server is a way of automatically finding the symbols.

Symbol server is Microsoft-speak. Linux speak would be standard way to find debuginfo by build id.

Could also read the build ID from the .so.dbg file its there also.We also put symbols for the steam-runtime and libc6 on our symbol server, so it doesnt matter whether individual developers set them up.

gdb will look for .build-id in all debug directories and will follow links to magically load the correct symbols

Finding source does not work as cleanly.

Source indexing any interest?Documentation for adding build IDs: http://linux.die.net/man/1/ld and http://fedoraproject.org/wiki/Releases/FeatureBuildIdeu-unstrip comes from the elfutils package, so sudo apt-get install elfutils will get it

Disassembly also fails to show all of some functions Im unsure why.