Some basic knowledges required for beginners in writing linux kernel module - with a description of linux source tree, so that the idea of where and how develops. The working of insmod and rmmod commands are described also.

1. Introduction
To
Kernel Modules
- Dibyajyoti Ghosh

2. Introduction
What is kernel
Kernel Space and User Space
Kernel Categories
Linux Kernel Source
Build your own Kernel
Kernel Files
Kernel Modules
Kernel Module Build System
Few Special Files
Use Cases
Kernel Module Vs Kernel Built-in
Kernel Module Vs User Application
How “insmod” works
How “rmmod” works
Outline

3. Introduction
Let us start our discussion with a diagrammatic
representation of a Linux system
User Space Applications
Hardware
Operating System
Kernel
Kernel modules

4. What is Kernel
What is kernel?
- Central core of an Operating System
- Kernel is loaded first during booting and stays till the system is up.
So image size should be minimum.
- Usually loaded into a protected memory area – Kernel Space.
- Kernel and BIOS are completely separate entity.

5. What is Kernel (Contd.)
Main Roles of kernel?
- Kernel executes jobs or handles interrupts etc. in Kernel Space.
- Kernel provides a set of portable, architecture and hardware
independent Kernel APIs to allow user space applications to use the
hardware resources.
- User Space applications request for basic services (Memory
management; Process management; I/O management etc.) through
system calls – Services provided by kernel.
- Kernel handles concurrent access and usage of hardware
resources.

6. Kernel Space and User Space

Kernel codes [including kernel modules] execute in a separate
address space [protected from being overwritten by external
programs] with super-user privilege : This environment [Address
space + privilege] is called Kernel Space

User applications run in another separate address space with lowest
privilege mode : This environment is called User Space

Linux system can switch from User Space to Kernel Space,
whenever an application issues a system call, or an hardware
interrupt suspends the application process

Kernel code executing a system call executes in process context;
whereas the code handling interrupts [Interrupt Handlers or
Interrupt Service Routines] works asynchronously in Interrupt
context.

7. Kernel Categories
Micro kernel:
- Provides minimal services, such as defining Memory Address Space, IPC
and CPU management
- All other services such as Hardware management etc. are implemented
separately as User Space Processes.
- Examples: AIX, Mac OS X, MINIX etc.
Monolithic kernel:
- Contains all the core functions of OS and device drivers
- Some can load modules dynamically to extend kernel features on demand
- Examples: Linux, FreeBSD etc.
Hybrid kernel:
- Similar to micro kernel, except that they include additional code in kernel
space so that such code can run more swiftly compared to if those code
were made to run from User Space.
- Examples: Windows etc.
Exo kernel:
- Still experimental

8. Linux Kernel Source
arch/<ARCH>:
- Architecture specific code
block/:
- Block layer core
COPYING:
- Linux copying conditions (GNU GPL).
CREDITS:
- Linux main contributors
crypto:
- Cryptographic libraries
Documentation/:
- Kernel documentation. Don't miss it!
drivers/:
- All device drivers except sound ones (usb, pci...)
firmware/:
- Legacy: firmware images extracted from old drivers

9. Linux Kernel Source (Contd)
fs/:
- Filesystems (fs/ext3/, etc.)
include/:
- Kernel headers
include/linux/:
- Linux kernel core headers
include/uapi/:
- User space API headers
init/:
- Linux initialization (including main.c)
ipc/:
- Code used for process communication
Kbuild:
- Part of the kernel build system

10. Linux Kernel Source (Contd)
Kconfig:
- Top level description file for conguration parameters
kernel/:
- Linux kernel core (very small!)
lib/:
- Misc library routines (zlib, crc32...)
MAINTAINERS:
- Maintainers of each kernel part. Very useful!
Makefile:
- Top Linux Makefile (sets arch and version)
mm/:
- I Memory management code (small too!)
net/:
- Network support code (not drivers)
README:
- Overview and building instructions

11. Linux Kernel Source (Contd)
REPORTING­BUGS:
- Bug report instructions
samples/:
- Sample code (markers, kprobes, kobjects...)
scripts/:
- Scripts for internal or external use
security/:
- Security model implementations (SELinux...)
sound/:
- Sound support code and drivers
tools/:
- Code for various user space tools (mostly C)
usr/:
- Code to generate an initramfs cpio archive
virt/:
- Virtualization support (KVM)

12. Build your own Kernel
[root@asl­host169]# make defconfig
/** Creates a default .config file for kernel configuration, based on underlying
system
*/
[root@asl­host169]# make menuconfig
/** Opens a text based GUI, where we can custom-configure the kernel as
per our requirement. The previous .config file is kept as backup with
name .config.old, and a new .config file is created along with the
modifications. Put a [*] against “Enable loadable module support”
*/
[root@asl­host169]# make bzImage modules modules_install
install
/** Builds the kernel and its modules; Prepares various files like
vmlinux, vmlinux.bin, vmlinuz, zImage, bzImage,
System.map, initrd.img etc.
*/

13. Kernel Files
vmlinux:Binary image of Linux kernel in a Statically Linked
Executable File Format.
vmlinux.bin: Same as vmlinux, but in a bootable raw binary file
format. All symbols and relocation information is discarded.Binary
image of Linux kernel in a Statically Linked Executable File Format.
Generated from vmlinux by objcopy ­O binary vmlinux
vmlinux.bin.
vmlinuz:Binary compressed [with zlib/LZMA/bzip2] vmlinux file.
During bootup, it is decompressed to get the boot image.

14. Kernel Files (Contd.)
zImage:Prepared with make zImage. This is an old format for small
kernels.
bzImage:Prepared with make bzImage. This big zImage was
created while the kernel grew bigger enough, and accordingly the size
of the binary image.
System.map:Static kernel symbol table.
initrd.img:A small file that does some initiations, and extracts and
executes the actual kernel file.

15. Kernel Modules
- Linux Kernel Module (LKM) are nothing but a piece of kernel code,
that can be loaded as and when required to the running kernel. However,
kernel must should “Enable loadable module support”
[CONFIG_MODULES=y] for this.
- The prototype for the init and exit function of a module:
init function: static int __init <init_fn_name>(void)
exit function: static void __exit <exit_fn_name>(void)
- The __init and __exit Macros:
__init macro: causes the init function to be discarded and its memory
freed once the init function finishes for built-in codes, but not loadable
modules.
__exit macro: causes the omission of the function when the module is
built into the kernel, and has no effect for loadable modules.
These macros are defined in linux/init.h

16. Kernel Modules (Contd.)
Fine, it’s time for an example:
Step1: Write a HelloWorld module
#include <linux/init.h>
#include <linux/module.h>
MODULE_LICENSE("Dual BSD/GPL");
static int hello_init(void)
{
printk(KERN_ALERT "Hello, worldn");
return 0;
}
static void hello_exit(void)
{
printk(KERN_ALERT "Goodbye, cruel worldn");
}
module_init(hello_init);
module_exit(hello_exit);

17. Kernel Modules (Contd.)
Step2: Prepare a Makefile to build it to HelloWorld.ko kernel object
KERNELDIR=/lib/modules/$(shell uname ­r)/build
obj­m += HelloWorld.o
all:
make ­C $(KERNELDIR) M=$(PWD) modules
clean:
make ­C $(KERNELDIR) M=$(PWD) clean
Step3: Keep the HelloWorld.c and the Makefile in same directory, and
issue make
[root@asl­host169 my_module]# make
make[1]: Entering directory `/home/kernel/linux­3.19'
CC [M] /home/dibyajyoti/Study_mats/my_module/HelloWorld.o
Building modules, stage 2.
MODPOST 1 modules
CC
/home/dibyajyoti/Study_mats/my_module/HelloWorld.mod.o
LD [M] /home/dibyajyoti/Study_mats/my_module/HelloWorld.ko
make[1]: Leaving directory `/home/kernel/linux­3.19'

18. Kernel Modules (Contd.)
Step4: Load the module
[root@asl­host169 my_module]# insmod HelloWorld.ko
[root@asl­host169 my_module]# dmesg ­c
[706754.319134] Hello, world
[root@asl­host169 my_module]#
Step5: Unload the module
[root@asl­host169 my_module]# rmmod HelloWorld
[root@asl­host169 my_module]# dmesg ­c
[706891.948856] Goodbye, cruel world
[root@asl­host169 my_module]#
Step6: Bingo! The journey of our module is end :-)

19. Kernel Module Build System
Ref: <Kernel_Home>/Documentation/kbuild/modules.txt
- "kbuild" is the build system used by Linux kernel (Our example uses it)
- Modules' build system should comply with this for easy compatibility to
changes in build infrastructure and pick right flags to gcc
- Functionality for both in-tree and out-of-tree module building are
provided
- External modules are supplied with Makefiles to hide complexity
- The command to build an external module is:
$ make -C <path_to_kernel_src> M=$PWD
- To build against the running kernel use:
$ make -C /lib/modules/`uname -r`/build M=$PWD
- Then to install the module(s) just built, add the target
"modules_install" to the command:
$ make -C /lib/modules/`uname -r`/build M=$PWD modules_install

20. Kernel Module Build System (Contd.)
- make -C $KDIR M=$PWD
($KDIR refers to the path of the kernel source directory.)
- “-C $KDIR”
The directory where the kernel source is located. "make" will actually
change to the specified directory when executing and will change back
when finished.
- “M=$PWD”
Informs kbuild that an external module is being built. The value given
to "M" is the absolute path of the directory where the external module
(kbuild file) is located.

21. Kernel Module Build System (Contd.)
- When building an external module, make line looks like (generally):
make -C $KDIR M=$PWD [target]
“[target]” is optional, and can have following values:
- “modules”
Default target for external modules. It has the same functionality as if
no target was specified.
- “modules_install”
Installs the external module(s). Default location is
/lib/modules/<kernel_release>/extra/, but a prefix may be added with
INSTALL_MOD_PATH
- “clean”
Removes all generated files in the module directory only.
- “help”
Lists the available targets for external modules.

22. Kernel Module Build System (Contd.)
- “obj-<y/m> := <module_name>.o”
kbuild system will build <module_name>.o from <module_name>.c,
and, after linking, will result in the kernel module <module_name>.ko.
The above line can be put in either a "Kbuild" file or a "Makefile.
NOTE: Many a times, we may encounter lines like:
“obj-$(CONFIG_BT) += bluetooth/”
Means, value of the tri-state “CONFIG_BT” ['m' if module; 'y' if built-
in; 'n/<NULL>' if not defined] is appended to make the line obj-m /
obj-y / not-defined correspondingly.

23. Kernel Module Build System (Contd.)
- When the module is built from multiple sources, an additional line is
needed listing the files:
“<module_name>-objs := <src1>.o <src2>.o ...”

24. Few Special Files
Once the module is built, many files are generated, like:
- <module_name>.ko : Actual kernel module file to “insmod”. “ko”
Stands for Kernel Object.
- Module.symvers : When your module “exports” symbols using
EXPORT_SYMBOL or its variants, those symbols are listed here along
with the EXPORTer module name
- modules.order : This file tells the modules should be loaded in which
order [Top-down ordering]

25. Use case
I built kernel 3.3 with mmc_core as module and kernel 3.9 with
mmc_core as built-in.
Result:
The Module.symvers of both contain the symbols.
However, System.map of 3.9.0 contains symbols from mmc_core sub-
section, while that of 3.3.0 does not contain the symbols.
- The file /proc/kallsyms in proc filesystem is the dynamic list of all the
kernel symbols

26. Use case
Most of us probably faced:
insmod: ERROR: could not insert module <module_name>.ko: Unknown
symbol in module
- The above occurs for modules, but never originates for kernel built-in.
Reason:
- Kernel built-in are built during kernel compilation phase, and symbols
are resolved when preparing the kernel binary image. At this phase, all
the symbol references are resolved
- Kernel modules are external agent, that refers to kernel symbols. When
we call insmod utility to load the module, first the symbols are
resolved. During this phase, if the kernel symbol is not available in
/proc/kallsyms file, we face the above error (irritation!).

27. Kernel Modules Vs. Kernel built-in
Similarity:
- Both seats and executes in kernel space completely
- Both has direct access to whole kernel
- Once loaded, both have equal rights and responsibilities
Dissimilarity:
- Unlike module, kernel built-in is part of the kernel binary image
- kernel built-in can access the whole source [if not static function]. But
modules can access kernel code only if they are any of macro / inline fn
or symbol exported by kernel using EXPORT_SYMBOL or its variants
- Kernel built-in has lesser memory footprint compared to module
- Kernel built-in usually outperforms kernel modules

28. Kernel Modules Vs. User Applications
Full of Dissimilarities:
- Unlike Applications, Kernel modules do not define a main() function
- Kernel modules link only to kernel; Aplications link to libraries
- Kernel modules use different set of header files than user applications
- Kernel modules should avoid global variables, as those must be unique
kernel-wide.
- Kernel module may be hardware specific; applications are generally not
- Kernel modules can be dynamically loaded.
- Kernel module runs in kernel space; application runs in user space
- Kernel modules typically are not sequential in nature.
- Kernel modules typically can be interrupted.
- Faults may be fatal to system for kernel codes.

29. How “insmod” works
Steps in brief:
- Allocates a buffer, and scales it up if required
- Opens the <module_name>.ko file [O_RDONLY mode], and copies
[read] the entire file to the userspace buffer
- Now it issues init_module system call; C Compiler supported ABI
assigns a unique number [128 for init_module] to it - eventually it maps
to kernel call sys_init_module
Ref: File: arch/x86/include/generated/uapi/asm/unistd_32.h
#define __NR_init_module 128
File: arch/x86/include/generated/asm/syscalls_32.h
__SYSCALL_I386(128, sys_init_module, sys_init_module)
- However, sys_init_module code does not directly find any. It is defined
by using the following macro [defined in kernel/module.c]
SYSCALL_DEFINE3(init_module, void __user *, umod, unsigned long, len, const char __user
*, uargs);

30. How “insmod” works (Contd.)
- This call does the following major steps to actually load the module
- copy_module_from_user() : Allocates memory in kernel space [vmalloc()];
& copies the userspace buffer to kernel space [copy_from_user()]
- load_module() : Does the following
1. various sanity checks [elf header etc.]
2. some basic setup [refcnt; MODINFO_ATTR field etc] (setup_modinfo())
3. some preparation for relocation
4. copies the arguments
5. set module state as MODULE_STATE_COMING (complete_formation())
6. setup of link to sysfs and
7. call do_init_module() : This actually calls the module initialization
function (<fn>) that is specified by our module using module_init(<fn>).
Additionally, the module state is set to MODULE_STATE_LIVE

31. How “rmmod” works
Steps in brief:
- It determines the module name and checks if it is in use presently
- Now it issues delete_module system call; C Compiler supported ABI
assigns a unique number [129 for delete_module] to it - eventually it
maps to kernel call sys_delete_module
Ref: File: arch/x86/include/generated/uapi/asm/unistd_32.h
#define __NR_delete_module 129
File: arch/x86/include/generated/asm/syscalls_32.h
__SYSCALL_I386(129, sys_delete_module, sys_delete_module)
- However, sys_delete_module code does not directly find any. It is
defined by using the following macro [defined in kernel/module.c]
SYSCALL_DEFINE2(delete_module, const char __user *, name_user, unsigned int, flags);

32. How “rmmod” works (Contd.)
- This call does the following major steps to actually load the module
-Copies the module name from userspace
- Traverse the list of modules and find the specified module by name
- Checks if other modules are dependent on it
- Synchronizes all asynchronous function calls. This function waits until all
asynchronous functiona calls are done
- free_module(): Frees up a module, its memory, removes from list etc.
1. unlinks the sys file system from the module
2. set module state tp MODULE_STATE_UNFORMED
3. Removes dynamic debug info.
4. Arch specific cleanup
5. Clear the unload stuff from module
6. Frees up any allocated parameters
7. Some more free up are done; Frees up the core containing the module
structure
8. Frees up module memory – the kernel space buffer allocated during load