To introduce a robust, secure and reliable platform for the industrial environments is a key challenge; moreover, the platform needs to survive for a long time (more than 10+ years). There are many good solutions aiming to meet these requirements, such as LTSI (Long Term Support Initiative) and CIP (Civil Infrastructure Platform). However, it still needs a high amount of maintenance and development costs in handling SoC/ hardware board in-house patch, non-upstream driver and keep source code consistent with different SoC and platform afterwards.
In this presentation, SZ Lin will introduce how to operate long-term maintenance model of embedded industrial Linux distribution. In addition, he will also address the building, deploying and testing architecture and workflow for producing a robust, secure and reliable platform.
Axa Assurance Maroc - Insurer Innovation Award 2024
Long-term Maintenance Model of Embedded Industrial Linux Distribution
1. Long-term Maintenance Model of
Embedded Industrial Linux
Distribution
LINUXCON + CONTAINERCON + CLOUDOPEN CHINA 2017
SZ Lin (林上智)
MXcore, Software Supervisor
06/19 2017
2. Confidential
About Me
SZ LIN (林上智)
Software Engineer at Moxa
• Industrial-grade Linux Distribution
Debian Developer
Blog - https://szlin.me
2
16. Confidential
SoC Board Support Package Kernel
• Kernel version depends on SoC vendors
– Well made but not well maintained
• Contain lots of in-house patches
– Errata patches
– Specific feature patches
– …
• Different SoC might use different versions of
kernel
• The lifetime is unsure
16
17. Confidential
LTS: Long Term Stable Kernel [1]
Extend software uptime for stable kernel
• Only accept bug fixes and security fixes
img: https://www.kernel.org/category/releases.html
17
18. Confidential
LTSI: Long Term Support Initiative [2]
• Linux Foundation collaborative project
– Based on LTS
– Add another chance to include further patches on top
of LTS
– Auto Test framework
– Same lifetime with LTS (yearly release and 2 years life
time)
18
19. Confidential
CIP (Civil Infrastructure Platform) [3]
• Linux Foundation collaborative project
– Support kernel and core package
– Auto Test framework
– Maintenance period
• 10 years and more (10-20 years)
19
24. Confidential
Good system security [5]
Everything is open
Usually, fixed packages are uploaded
within a few days
Stability
unstable → testing → stable
Scalability
Server, Desktop,
Laptop, Embedded devices
Long term support
5 more years by Debian-LTS project
(i386, amd64, armel and armhf)
Multiple architectures
alpha, amd64, armel, armhf,
aarch64, hppa, i386, ia64, mips,
mipsel, powerpc, s390, and spar
Why We Choose Debian [4]
Incredible amounts
of software
Debian comes with over 43000
different pieces
of software with free
24
26. Confidential
Unified Kernel
• Kernel supports multi-platform
• Centralized topology
– Use the same kernel version
• Long-term maintenance
– Apply patches through a single entry point
• R&D resources focus
– Modify Kernel configuration
– Edit Device tree
– Prepare SoC-related patches
26
33. Confidential
How to Find Out In-house Patch By
Using Your Own Custom Kernel
• Get hardware component errata/ datasheet
• Get hardware component official driver if
possible
– It always surprises you
• Refer to mainline kernel
– It might be fixed already
• Time to backport
• Discuss the issue in subsystem mailing list or
forum
– Make the patch
33
36. Confidential
Regular Update Mechanism
1
Well-testing
Stable, stable and stable
2
Backported patches
Include device drivers and new kernel features
Upstream first
Do not create in-house patch after fixing bug
3
Minimal modification
Apply the patches only what we need
4
Bug fixes
New feature patches
Security fixes
New device drivers
Backported patches
…
36
37. Confidential
Backported Patches in Maintenance
Phase
• Try not to modify *ANY API* (if possible…)
• Try not rename/ remove any member of structure
– “Add” behavior is also risky
• Regression test
• Unit test
• Do not apply lots of commits
– Handle it carefully
• git cherry-pick
• git blame
• git bisect
• …
37
44. Confidential
24/ 7 Long-term Platform Test
Robustness
Robustness is the ability
of a computer system to
cope with errors during
execution and cope with
erroneous input [12]
Reliability
Reliability is enhanced
by features that help to
avoid, detect and repair
hardware faults [11]
Security
Quick response in
resolving CVE/
vulnerabilities and
attacks in platform
Longevity
Long-term support at
least 10 years life cycle
with bug fixes, new
features and new
hardware components
44
Endurance test
Compatibility test
…
45. Confidential
24/ 7 Long-term Platform Test
Robustness
Robustness is the ability
of a computer system to
cope with errors during
execution and cope with
erroneous input [12]
Reliability
Reliability is enhanced
by features that help to
avoid, detect and repair
hardware faults [11]
Security
Quick response in
resolving CVE/
vulnerabilities and
attacks in platform
Longevity
Long-term support at
least 10 years life cycle
with bug fixes, new
features and new
hardware components
45
Fuzz testing
[14][15][16]
…
46. Confidential
24/ 7 Long-term Platform Test
Reliability
Reliability is enhanced
by features that help to
avoid, detect and repair
hardware faults [11]
Security
Quick response in
resolving CVE/
vulnerabilities and
attacks in platform
46
Power failure test
Reboot test
…
47. Confidential
24/ 7 Long-term Platform Test
Security
Quick response in
resolving CVE/
vulnerabilities and
attacks in platform
47
Daily test for CVE [25]
…