BUD17-TR04: Kernel Debug Stories

1. ENGINEERS AND DEVICES WORKING TOGETHER

2. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ ○ ○ ● ○ ○ ○ ○ ○ ● ○ ●

3. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ ○ ○ printk() ● ○ ○ ○ ● ○ ○ ○ ○ ● ○

4. ENGINEERS AND DEVICES WORKING TOGETHER 0x0 0x0000ffffffffffff 256TB 0xffff000000000000 0xffff000008000000 128MB 0xffff000008000000 0xffff00000807ffff 512KB 0xffff000008080000 10MB+ Kernel image specific 0xffff7dffbfff0000 ~126TB 0xffff7dfffe7fd000 0xffff7dfffec00000 4MB + 12KB 0xffff7dfffee00000 0xffff7dffffe00000 16MB 0xffff7e0000000000 0xffff800000000000 2048G 0xffff800000000000 0xffffffffffffffff 128TB

6. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ ⇒ ○ ⇒ ○ ⇒ ○ ⇒ ● ○ printk ○ git grep printk ● ○ ○ ○

7. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ FRAME_WARN ○ SCHED_STACK_END_CHECK ● ○ slub_debug= ■ SLUB_DEBUG ○ DEBUG_PAGEALLOC ○ PAGE_POISONING ● ○ DEBUG_SPINLOCK ○ DEBUG_MUTEXES ○ DEBUG_ATOMIC_SLEEP ○ DEBUG_LOCK_ALLOC ○ LOCKUP_DETECTOR ●

8. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ ○ ○ ● ○ ○ ● ○ ○ PROVE_LOCKING ○ KASAN

9. ENGINEERS AND DEVICES WORKING TOGETHER PROVE_LOCKING lock(subsys) lock(driver) lock(driver) lock(subsys) DEBUG_SPINLOCK DEBUG_MUTEXES lock(subsys) lock(driver) unlock(driver) unlock(subsys) lock(driver) lock(subsys) PROVE_LOCKING PROVE_LOCKING

11. ENGINEERS AND DEVICES WORKING TOGETHER ● ● ● ● ● ●

12. ENGINEERS AND DEVICES WORKING TOGETHER make cscope git grep 'struct foo {' printk() [ 0.001636] xyz: Found 10 widgets ⇒ git grep "Found .* widgets" I can’t reproduce but my customer can

13. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ ○ ○ ● ○ ○ ⇒ ○ quiet LOG_BUF_SHIFT log_bug_len= ● pr_debug() ○ <8> <7> ○ ○ DYNAMIC_DEBUG I can’t reproduce but my customer can

14. ENGINEERS AND DEVICES WORKING TOGETHER I can’t reproduce but my customer can SP LR (r30) FP (r29) r19..r28 r18 IP0/IP1 (r16-r17) r9..r15 r8 r0..r7

15. ENGINEERS AND DEVICES WORKING TOGETHER git grep REGMAP_ALLOW_WRITE_DEBUGFS I can’t reproduce but my customer can config DEBUG_FS bool "Debug Filesystem" select SRCU help debugfs is a virtual file system that kernel developers use to put debugging files into. Enable this option to be able to read and write to these files. If unsure, say N.

16. ENGINEERS AND DEVICES WORKING TOGETHER I can’t reproduce but my customer can <release_thread>: stp x29, x30, [sp,#-16]! mov x29, sp mov x0, x30 bl ffff000008092ea0 <_mcount> ldp x29, x30, [sp],#16 ret <release_thread>: stp x29, x30, [sp,#-16]! mov x29, sp mov x0, x30 nop ldp x29, x30, [sp],#16 ret <release_thread>: stp x29, x30, [sp,#-16]! mov x29, sp mov x0, x30 bl ftrace_caller ldp x29, x30, [sp],#16 ret

17. ENGINEERS AND DEVICES WORKING TOGETHER ● ● ○ ftrace=function ftrace_filter=mydrv_* tp_printk ○ ftrace=function_graph ftrace_graph_notrace=rcu*,*lock,*spin* ● cd /sys/kernel/debug/tracing echo function > current_tracer echo 1 > tracing_on ● ○ ○ ○ ftrace_dump_on_oops ○ ● trace_printk() I can’t reproduce but my customer can

18. ENGINEERS AND DEVICES WORKING TOGETHER /proc/vmcore console=ttyS0,1115200 … … crashkernel=128M -p ./kexec -p vmlinux --dtb=xxx.dtb --append="root=/dev/mmcblk0p9 rw 1 maxcpus=1 reset_devices" I can’t reproduce but my customer can

19. ENGINEERS AND DEVICES WORKING TOGETHER ● ● ● ● ● ● ● printk()

20. ENGINEERS AND DEVICES WORKING TOGETHER ● ● ● ● ○ ○ ○ echo 'sched:*' > /sys/kernel/debug/tracing/set_event My widget missed its deadline

21. ENGINEERS AND DEVICES WORKING TOGETHER ● ● ● ● ● ● My widget missed its deadline

22. ENGINEERS AND DEVICES WORKING TOGETHER ● ● ● ● .

24. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ ○ initcall_debug ○ initcall_debug clk_disable_unused() clk_ignore_unused ○ DEBUG_SPINLOCK DEBUG_MUTEX LOCKUP_DETECTOR ○ ● ○ ○ ○ ○ ○ My board just stopped dead

25. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ ● ○ ○ ○ ● ○ ○ ○ ● ○ ○ My board just stopped dead

26. ENGINEERS AND DEVICES WORKING TOGETHER ● ● ● ○ ○ ○ ● ● ● My board just stopped dead

27. ENGINEERS AND DEVICES WORKING TOGETHER My board just stopped dead [ 277.592103] PCSR of cluster0 cpu1 is 0xffff0000080c4f78 [ 277.592103] PCSR of cluster0 cpu1 is 0xffff0000080c4f78 [ 277.600833] PCSR of cluster0 cpu2 is 0xffff00000809a934 [ 277.609564] PCSR of cluster0 cpu3 is 0xffff00000809a934

28. ENGINEERS AND DEVICES WORKING TOGETHER ● ● ● ● ○ ○ ○

29. ENGINEERS AND DEVICES WORKING TOGETHER git bisect start <bad_sha1> <good_sha1> git bisect run compile_kernel_and_run_test.sh compile_kernel_and_test.sh compile_kernel_and_test.sh I’m sure this used to work

30. ENGINEERS AND DEVICES WORKING TOGETHER ktest.pl compile_kernel_and_test.sh Documentation/ tools/testing/ktest/sample.conf tools/testing/ktest/examples I’m sure this used to work

32. ENGINEERS AND DEVICES WORKING TOGETHER My board just randomly failed

35. ENGINEERS AND DEVICES WORKING TOGETHER Graphic by Jost, CC0-PD

37. ENGINEERS AND DEVICES WORKING TOGETHERom: http://linaro.co/bud17-kdb

BUD17-TR04: Kernel Debug Stories

Linaro

BUD17-TR04: Kernel Debug Stories