bud17-tr04: kernel debug stories

ENGINEERS AND DEVICESWORKING TOGETHER

ENGINEERS AND DEVICES

WORKING TOGETHER

●○○○

●○○○○○

●○

●


●○○○ printk()

●○○○

●○○○○

●○


0x0 0x0000ffffffffffff 256TB

0xffff000000000000 0xffff000008000000 128MB

0xffff000008000000 0xffff00000807ffff 512KB

0xffff000008080000 10MB+

Kernel image specific 0xffff7dffbfff0000 ~126TB

0xffff7dfffe7fd000 0xffff7dfffec00000 4MB + 12KB

0xffff7dfffee00000 0xffff7dffffe00000 16MB

0xffff7e0000000000 0xffff800000000000 2048G

0xffff800000000000 0xffffffffffffffff 128TB


●○ ⇒○ ⇒○ ⇒○ ⇒

●○ printk○ git grep printk

●○

○

○


●○ FRAME_WARN○ SCHED_STACK_END_CHECK

●○ slub_debug=

■ SLUB_DEBUG

○ DEBUG_PAGEALLOC○ PAGE_POISONING

●○ DEBUG_SPINLOCK○ DEBUG_MUTEXES○ DEBUG_ATOMIC_SLEEP○ DEBUG_LOCK_ALLOC○ LOCKUP_DETECTOR

●


●○○○

●○○

●○○ PROVE_LOCKING○ KASAN


PROVE_LOCKING

lock(subsys) lock(driver)lock(driver) lock(subsys)

DEBUG_SPINLOCKDEBUG_MUTEXES

lock(subsys)lock(driver)unlock(driver)unlock(subsys)lock(driver)lock(subsys)

PROVE_LOCKING

PROVE_LOCKING


WORKING TOGETHER

●○○○

●○○○○○

●○

●


●

●

●

●●

●


make cscope git grep 'struct foo {'

printk() [ 0.001636] xyz: Found 10 widgets⇒ git grep "Found .* widgets"

I can’t reproduce butmy customer can


●

○○○

●○○ ⇒○ quiet

LOG_BUF_SHIFT log_bug_len=● pr_debug()

○ <8> <7>○○ DYNAMIC_DEBUG




SP

LR (r30)

FP (r29)

r19..r28

r18

IP0/IP1 (r16-r17)

r9..r15

r8

r0..r7


git grep REGMAP_ALLOW_WRITE_DEBUGFS


config DEBUG_FSbool "Debug Filesystem"select SRCUhelp debugfs is a virtual file system that kernel developers use to put debugging files into. Enable this option to be able to read and write to these files.

If unsure, say N.



<release_thread>:stp x29, x30, [sp,#-16]!mov x29, spmov x0, x30bl ffff000008092ea0 <_mcount>ldp x29, x30, [sp],#16ret

<release_thread>:stp x29, x30, [sp,#-16]!mov x29, spmov x0, x30nopldp x29, x30, [sp],#16ret

<release_thread>:stp x29, x30, [sp,#-16]!mov x29, spmov x0, x30bl ftrace_callerldp x29, x30, [sp],#16ret


●●

○ ftrace=function ftrace_filter=mydrv_* tp_printk○ ftrace=function_graph ftrace_graph_notrace=rcu*,*lock,*spin*

●cd /sys/kernel/debug/tracingecho function > current_tracerecho 1 > tracing_on

●○○○ ftrace_dump_on_oops○

● trace_printk()



/proc/vmcore

console=ttyS0,1115200 … … crashkernel=128M

-p

./kexec -p vmlinux --dtb=xxx.dtb --append="root=/dev/mmcblk0p9 rw 1 maxcpus=1 reset_devices"



●

●

●

●

●

●

●printk()


●

●●

●○○

○ echo 'sched:*' > /sys/kernel/debug/tracing/set_event

My widget missedits deadline


●●●●●●

My widget missedits deadline


●●●●

.


●●

●

●

●

●


●○○ initcall_debug○ initcall_debug clk_disable_unused() clk_ignore_unused○ DEBUG_SPINLOCK DEBUG_MUTEX LOCKUP_DETECTOR○

●○○

○

○○

My board juststopped dead


●○

●○

○

○

●○○○

●○○



●

●●

○○

○●●

●




[ 277.592103] PCSR of cluster0 cpu1 is 0xffff0000080c4f78[ 277.592103] PCSR of cluster0 cpu1 is 0xffff0000080c4f78[ 277.600833] PCSR of cluster0 cpu2 is 0xffff00000809a934[ 277.609564] PCSR of cluster0 cpu3 is 0xffff00000809a934

http://archive.arm.linux.org.uk/lurker/message/20170213.061135.c97560eb.en.html




●

●

●

●

○○○


git bisect start <bad_sha1> <good_sha1>git bisect run compile_kernel_and_run_test.sh

compile_kernel_and_test.sh


I’m sure thisused to work


ktest.pl


Documentation/ tools/testing/ktest/sample.conftools/testing/ktest/examples

I’m sure thisused to work


●●

●

●

●

●


My board justrandomly failed


WORKING TOGETHER

●○○○

●○○○○○

●○

●


WORKING TOGETHER

Graphic by Jost, CC0-PD

mailto:[email protected]

http://andrewhennigan.com/

https://pixabay.com/en/grunge-stamp-graphic-design-design-1069198/

https://creativecommons.org/publicdomain/zero/1.0/

https://pixabay.com/en/grunge-stamp-graphic-design-design-1069198/


WORKING TOGETHER

●○○○

●○○○○○

●○

●

ENGINEERS AND DEVICESWORKING TOGETHERom: http://linaro.co/bud17-kdb

http://linaro.co/bud17-kdb

mailto:[email protected]

http://www.linaro.org

http://www.linaro.org

bud17-tr04: kernel debug stories

Technology