Mechanisms and tools of development and monitoring in Linux Kernel
A brief presentation about the tools and mechanisms about development for the Linux Kernel to help understanding of what is required.
Tools that create an execution profile or provide instrumentation through static or dynamic methods, the Linux Kernel code, will be presented.
Also discussed will be the GDB debugger and how through a remote virtual serial connection to a virtual machine it can be used to debug a live Kernel and Linux Kernel modules. Also demonstrated will be how a deeper understanding of the code can be attained by attaching the memory locations used by the Kernel module to the GDB session.
Lastly, some of the Kernel execution contexts, such as, interrupts, deferrable work, context, etc. are presented.
1. Energia Open Source
Monitoring and profiling in user space
Profiling in can be achieved using gprof / library
Static instrumentation
Monitoring with SystemTap
Dynamic instrumentation
Overwriting some symbols in Global Offset Table
LD_PRELOAD environment variable
Can be achieved using ptrace (process trace) support
gdb, strace, etc.
Communication with kernel space
ProcFs, SysFs, DebugFs
netlink sockets
ioctl
27.02.2012 Network Monitoring by Process id 1
2. Energia Open Source
Kernel Development
Downloading the kernel sources and C compiling tools
Linux Cross Reference site (good for browsing code)
cscope good program to cross information of different C files
C language (useful some knowledge of assembly and gcc
extensions)
Understand how the user space interacts with kernel space
Useful virtualised environment
(Virtualbox, vmware, kvm) to boot the new kernel
Debug a remote kernel (with gdb)
27.02.2012 Network Monitoring by Process id 2
3. Energia Open Source
Debug a remote kernel
Using a virtual machine and a virtual serial connection
On host machine
Uncompressed kernel image (on the guest)
Virtual serial connection (can be done with socat command)
gdb
On guest
If the kernel already has support (kgdb or kgdb light)
Patch it with kgdb
Sysrq + g key or write to /proc/sysrq the 'g' value
27.02.2012 Network Monitoring by Process id 3
4. Energia Open Source
Debug remote kernel II
On the host machine it will call target remote ttyS0
The two machines are now linked with virtual serial console
for debugging
From now the guest machine is completely over the control
of the host gdb
What can we do at this stage?
Can set breakpoints, inspect data, do backtraces, etc.
The usual gdb session
27.02.2012 Network Monitoring by Process id 4
5. Energia Open Source
Can I debug a local kernel ?
Local debugging of a kernel is also possible
Still needs the uncompressed kernel image, gdb and
/proc/kcore file
Can I still stop my machine with breakpoints?
No, you can't.
It's only possible to inspect some global data on the kernel
Kernel modules information (metadata), etc.
27.02.2012 Network Monitoring by Process id 5
6. Energia Open Source
Debugging / profiling tools production systems
Ltt / Lttng
Oprofile
Kprobes
Ftrace
printk (the information is written on the kernel log)
/var/log/ directory (using Ksyslog)
dmesg command
Maybe DTrace is coming to Linux...?
27.02.2012 Network Monitoring by Process id 6
7. Energia Open Source
Debug kernel module
Inspect kernel module code
/sys/module/”module name”/sections
text section
data section, bss section (uninitialized data), etc.
In gdb do add-symbol-file “module-name” .text_address = ...
objdump tool
Disassemble the kernel module
27.02.2012 Network Monitoring by Process id 7
8. Energia Open Source
Kernel Contexts
Interrupts
Top half and bottom half
Top half needs to be very brief and if needed schedule a bottom
half
Bottom halves (deferrable work)
Soft irq
Tasklet
Work queues
Process Context
System calls
The kernel runs on behalf of user programs
27.02.2012 Network Monitoring by Process id 8
9. Energia Open Source
Kernel Development
Doing kernel development is not as difficult as it once was
As mentioned many tools can help us to debug and profile
our modules
This was just a brief overview of some information gathered
during the development of my master thesis
27.02.2012 Network Monitoring by Process id 9