This presentation is about the main tasks which Linux kernel platform engineers take care of. The talk includes real-life cases which help understand the role of respective specialists and might be helpful to those who consider such change in their careers.
The talk was delivered by Sam Protsenko (Software Engineer, Consultant, GlobalLogic) at GlobalLogic Embedded Career Day #2 on February 10, 2018.
More about GlobalLogic Embedded Career Day #2: https://www.globallogic.com/ua/events/globallogic-kyiv-embedded-career-day-2-materials
4. 4
Confidential
Tasks
• Board bring-up
• Porting kernel (and bootloader) to a new board
- Writing the BSP
- Drivers development
• Migrating to a new kernel version
• Bug fixing
• Boot time optimizations
• Upstreaming
• Sometimes:
- Bootloader related work
- Low-level work in rootfs (Android, OpenEmbedded, etc.)
- Hardware debugging
5. 5
Confidential
Tasks
Make board minimally functional,
step by step, for all required
components. Make sure it works.
• Check power and reset lines
• Check clock signals
• Debug SoC issues (caches, etc.)
• Configure pin muxes
• Bring-up RAM (DDR), serial
console (UART), peripheral
(eMMC, USB, LCD, etc.)
Board bring-up
6. 6
Confidential
Tasks
Make kernel and bootloader fully
functional on a new board.
• Bootloader bring-up
• Device Tree file
• Configuration file
• SoC support
• Write missing peripheral drivers
Porting kernel to a new board
7. 7
Confidential
Tasks
Modify board related files to work in
a new kernel version.
• Sometimes company doesn’t
want to upstream its kernel
• Migration work:
- board files -> device tree
- new API for kernel frameworks
- config options
- sometimes back-porting is needed
• Automated approach exists: see
“Prequel” by Julia Lawall
Migrating to a new kernel version
8. 8
Confidential
Components and Tools
• Linux kernel
• Bootloader (U-Boot, UEFI/edk2)
• RootFS (Android, Debian, OpenEmbedded,
BusyBox)
• C language
• OS principles
• Microcontrollers basics (+schematics)
• Linux user experience
Components
• Software
- toolchain (ARM GCC): gcc, gdb, ld, as
- binutils (add2line, strings, objdump, nm)
- Git
- Linux CLI tools
- board specific and task specific tools
(fastboot, dfu-util, omapconf, minicom, adb)
• Hardware
- JTAG
- logical analyzer
- oscilloscope
- soldering iron
Tools
Knowledge
11. 11
Confidential
Story #1: NOR flash support and XIP boot
Task highlights:
• Provide NOR flash chip support:
- in kernel
- in U-Boot
• Provide XIP boot:
- for U-Boot
- for kernel
12. 12
Confidential
Overview: NOR flash
• NOR flash is XIP capable (eXecution In Place)
• XIP from NOR is often used for reducing the boot time
• Can be useful in automotive and mobile areas
• It’s possible to run U-Boot and kernel from NOR flash
(Android or Linux rootfs is stored on eMMC)
• Concerns:
- drivers (GPMC, MTD, CFI, ELM)
- Device Tree definition
- timings
- U-Boot/kernel configuration for XIP
15. 15
Confidential
Overview: NOR and NAND flash comparison
Parameter NOR NAND
Capacity Small (64 KB - 512 MB) Huge (16 MB - 1 TB)
XIP (code execution) Yes No
Block erase time Very slow (1 sec) Fast (500 μsec)
Write time Slow (10 μsec / word) Fast (200 μsec / page)
Read time Fast (100 nsec / word) Fast (50 μsec / page)
Erase cycle range 10,000 to 100,000 1,000,000 with ECC
Access method Random Sequential
Price High (4x / MB) Very low (x / MB)
17. 17
Confidential
Sub-task 1: Timings
● GPMC controller must be programmed with correct timings
● Timings must be calculated from NOR chip characteristics
● There are a lot of them
● Usually board manufacturer should provide you with correct timings
● If NOR flash hangs - most likely timings are wrong
18. 18
Confidential
Sub-task 1: Timings
• cs-on
• cs-rd-off
• cs-wr-off
• oe-on
• oe-off
• we-on
• we-off
• rd-access
And much more else timings...
GPMC: Timings and parameters
22. 22
Confidential
Sub-task 2: Clocks and DPLLs
● GPMC clock must be enabled and be of correct frequency
● Check correct value (calculated for NOR timings)
● Check actual value (from kernel DebugFS)
● Fix clock if needed
● Clock derivation path can be seen from SoC TRM
27. 27
Confidential
Sub-task 2: Clocks and DPLLs
● Enable CONFIG_COMMON_CLK_DEBUG
● Mount DebugFS (in example below, it’s mounted to /d)
● Print clocks frequencies:
/d/clk/.../dpll_core_x2_ck/dpll_core_h12x2_ck # cat clk_rate
532000000
/d/clk/.../dpll_core_x2_ck/dpll_core_h12x2_ck/l3_iclk_div # cat clk_rate
532000000
But we see from TRM that L3_ICLK must be CORE_X2_CLK / 2
Check clock freq in DebugFS
29. 29
Confidential
Sub-task 2: Clocks and DPLLs
● Print clocks frequencies:
/d/clk/.../dpll_core_x2_ck/dpll_core_h12x2_ck # cat clk_rate
532000000
/d/clk/.../dpll_core_x2_ck/dpll_core_h12x2_ck/l3_iclk_div # cat clk_rate
266000000
So GPMC functional clock frequency now is 266 MHz, which is a desired frequency.
Check clock freq in DebugFS after fix
30. 30
Confidential
Sub-task 3: Kernel support
● NOR flash and eMMC were sharing the lines (found from schematic)
● So there was only 512 KiB of NOR flash accessible in kernel
● Means no kernel XIP
● But we can still:
○ read/write NOR from kernel
○ do U-Boot XIP
● NOR support wasn’t ready in our kernel at the time (k3.8)
● But it was ready in mainline kernel (3.11)
31. 31
Confidential
Sub-task 3: Kernel support
● Patches often have dependencies
● It’s hard to find all actual dependencies between branches
● Kernel maintainers have dedicated trees/branches for features like GPMC
● Use smart cherry-picking
Backporting patches
35. 35
Confidential
Sub-task 4: U-Boot XIP
● For XIP boot we don’t need MLO (U-Boot SPL), only u-boot.bin
● Boot sequence:
○ U-Boot is started by ROM code
○ U-Boot copies all files needed to RAM (dtb, kernel, ramdisk)
○ U-Boot switches to eMMC (e.g. via GPIO line)
○ U-Boot starts kernel from RAM; now we are not in XIP boot anymore
● Problem: U-Boot hangs when running XIP from NOR
36. 36
Confidential
Sub-task 4: U-Boot XIP
● U-Boot is too big to figure out why it’s not working
● We can’t even be sure XIP can work at all
● Let’s shrink U-Boot to minimal code and see if it works
● XIP “Hello world” baremetal firmware was implemented:
○ start assembler code (shrinked)
○ linker script (minimal: text/rodata/data/bss, all in SRAM)
○ BSS clear
○ enable UART clock + dependency clocks
○ pin muxes
○ naiive UART driver
○ print “Hello world” to UART (serial console)
https://gitlab.com/joeskb7/dra7xx-hello
Approach the problem
37. 37
Confidential
Sub-task 4: U-Boot XIP
● Minimal app worked, but U-Boot won’t work
● Now we know NOR flash + XIP is functional and there is
some bug in U-Boot
● Use JTAG to investigate the issue
Investigation
39. 39
Confidential
Sub-task 4: U-Boot XIP
● git bisect-ing helped to find the actual bug
● We were given modified U-Boot with code like this
(in drivers/mmc/Makefile):
CFLAGS := $(subst -Os, -O0, $(CFLAGS)) -fPIC
Root cause (2)
41. 41
Confidential
Sub-task 4: U-Boot XIP
● So only MMC related files were built with -O0 -fPIC
● But other U-Boot files were build with -Os
● That led to NULL pointer dereference
● Removing that line fixed the problem
Fix
42. 42
Confidential
Learnings:
● Always check mainline kernel and mailing lists for existing patch
● Git is your friend:
○ cherry-pick thoughtfully
○ bisect often helps to find the regression
● If some task is repeated often -- develop some tool or script
○ Share it if possible
● Isolating the problem helps to find the root cause
● JTAG is an ultimate tool for debugging
44. 44
Confidential
Story #2: Updating MAX732x driver
Task highlights:
• Update MAX732x driver to use Device Tree
• Provide bindings documentation
• Make it functional and bug-free
• Upstream changes
45. 45
Confidential
Overview: Chip description
• I2C I/O Expander
• Often used in automotive
• Have one-directional and
bidirectional ports
• Sends interrupt to SoC when input
event occurred
48. 48
Confidential
Sub-task 1: Device tree support
Platform data doesn’t exist in case of OF provider.
- chip->irq_base = pdata->irq_base;
+ if (pdata)
+ chip->irq_base =
pdata->irq_base;
Check pdata
50. 50
Confidential
Sub-task 2: Interrupt controller
● GPIO framework is capable of handling IRQ controller code for you
● Remove IRQ domain code / manual IRQ calculation and use gpiochip_* API
config GPIO_MAX732X_IRQ
select GPIOLIB_IRQCHIP
#include <linux/gpio/driver.h>
gpiochip_add()
gpiochip_remove()
gpiochip_irqchip_add()
gpiochip_set_chained_irqchip()
GPIOLIB_IRQCHIP
51. 51
Confidential
Sub-task 3: DT definition and bindings
● We want to I2C address of MAX7325 to be 0x5d
● As per datasheet:
Pin Line
----------
AD2 V+
AD0 V+
Check I2C address on schematics
57. 57
Confidential
Learnings:
● If in doubt -- check schematics
● Check out existing code in kernel for examples
● Check if there is existing framework for that problem
● Take responsibility (technical debt)
● Ask your customer about upstreaming
60. 60
Confidential
Story #3: Flashing EEPROM
Task highlights:
• EEPROM chip wasn’t flashed with Board ID
• It led to 2 issues:
- U-Boot was unable to find correct DTB file
- "board_rev" variable were missing in U-Boot environment
• Fix:
- Disable “Read only” mode for EEPROM chip
- Figure out how to work with it
- Write correct Board ID value to EEPROM
- Set back “Read only” mode