Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Design, Build,and Maintain the Embedded Linux Platform

344 vues

Publié le

Using open source software to build an embedded Linux platform from scratch.
Building an embedded Linux platform is like a puzzle; placing the suitable software components in the right positions will constitute an optimal platform. However, selecting suitable components is difficult since it depends on different application scenarios. The essential components of an embedded Linux platform include the bootloader, Linux kernel, toolchain, root filesystem; it also needs the tools for image generation, upgrades, and testing. There are abundant resources in the Linux ecosystem with these components and tools; however, selecting the suitable modules and tools is still a key challenge for system designers.

Publié dans : Technologie
  • DOWNLOAD THE BOOK INTO AVAILABLE FORMAT (New Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THE can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THE is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBOOK .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookBOOK, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, EBOOK, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THE Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THE the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THE Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

Design, Build,and Maintain the Embedded Linux Platform

  1. 1. Embedded Linux Platform 林上智 (SZ LIN) szlin@debian.org Date: 2020/7/15
  2. 2. 2 林上智 (SZ LIN) • Software Engineer • Embedded Linux Design and Development • Kernel Contributor • Civil Infrastructure Platform TSC Member • Open Source Development and Governance • Debian Developer • OpenChain Project Governing Board Member • Cybersecurity • CISSP – ISSAP, CSSLP • ISA/ IEC 62443 Cybersecurity Expert About Me 178F 8338 B314 01E3 04FC 44BA A959 B38A 9561 F3F9
  3. 3. 3 RoutersEthernet Switches Wireless LAN Network Infrastructure Embedded Computers Computing Serial Connectivity I/O Connectivity Video Connectivity Edge Connectivity Device
  4. 4. Before Using Embedded Linux Something You Should Know 4
  5. 5. 5 Copyright Patent A patent gives its owner the right to exclude others from making, using, selling, and importing an invention for a limited period of time, usually twenty years. src: https://en.wikipedia.org/wiki/Patent Copyright is a legal right, that grants the creator of an original work exclusive rights to determine whether, and under what conditions, this original work may be used by others src: https://en.wikipedia.org/wiki/Copyright
  6. 6. 6 Copyright Patent Identify key recommended processes for effective open source management [1]. It is a shared defensive patent pool with the mission to protect Linux [2].
  7. 7. Target Application Align with business 7
  8. 8. 8 Longevity + Stability + Security Performance Real-time SafetyResource Limited
  9. 9. 9 Longevity + Stability + Security Performance Real-time SafetyResource Limited Performance Real-time SafetyResource Limited SoftwareHardware
  10. 10. 10 Image: https://upload.wikimedia.org/wikipedia/commons/5/5b/Linux_kernel_map.png
  11. 11. 11 Hardware Survey – Key Components Short LifecycleLong Lifecycle (> 10 years) SoC e.g., Intel, ARM, MIPS Non-volatile memory e.g., NAND flash, eMMC, SD, SSD DRAMRTC Watchdog PMIC Cellular module Wi-Fi module PHY Switch module UART module WPAN module External LAN controller
  12. 12. 12 Lifecycle of Embedded Linux Platform Maintenance PhaseDevelopment Phase System Preparation Regular UpdateBuild and Testing 4321 Long-term Test
  13. 13. 13 13 Maintenance PhaseDevelopment Phase System Preparation Regular UpdateBuild and Testing 4321 Long-term Test More info: Manage Vulnerabilities in Open Source Components in ICS Industrial Control Systems (ICS) Cyber Security Conference Singapore, 2020 [3]
  14. 14. Development Phase Platform design and customization 14
  15. 15. 15 GNU/ Linux User Space Kernel Space Toolchain
  16. 16. Choose Proper Bootloader 16
  17. 17. 17
  18. 18. 18 Category License Supported Platforms Supported UEFI Maintainer Das U-Boot [4] GPL-2+ 68k, ARM, Blackfin, MicroBlaze, MIPS, Nios, SuperH, PPC, RISC-V, x86 (on top of Coreboot) Y DENX Software Engineering Coreboot [5] GPL-2 IA-32, x86-64, ARMv7, ARMv8, MIPS, RISC-V, POWER8 Y coreboot.org GRUB GPL-3 IA-32, x86-64, IA-64, ARM, PowerPC, MIPS and SPARC Y GNU Project rEFInd [6][7] GNU GPLv3, Modified BSD License (original program), additional components released under various licenses x86, x86-64, or ARM64 Y Roderick W. Smith
  19. 19. 19 Kernel Space
  20. 20. Choose Proper Kernel Based on the application requirement 20
  21. 21. 21 Linux Kernel Comparison Table Category Latest version Target Application Maintainer Linux kernel 5.7 • Performance • Resource Limited [8] [9] Kernel.org Preempt RT kernel 5.6 • Real-time • Functional safety • Resource Limited Real Time Linux collaborative project
  22. 22. 22 SoC Board Support Package Kernel • Kernel version depends on SoC vendors – Well made but not well maintained • Contain lots of in-house patches – Errata patches – Specific feature patches – … • Different SoC might use different versions of kernel • The lifetime is unsure
  23. 23. 23 LTS: Long Term Stable Kernel [10] Extend software uptime for stable kernel • Only accept bug fixes and security fixes img: https://www.kernel.org/category/releases.html
  24. 24. 24 LTSI: Long Term Support Initiative [11] • Linux Foundation collaborative project – Based on LTS – Add another chance to include further patches on top of LTS – Auto Test framework – Same lifetime with LTS (yearly release and 2 years life time)
  25. 25. 25 CIP (Civil Infrastructure Platform) [12] • Linux Foundation collaborative project – Support kernel and core package – Auto Test framework – Maintenance period • 10 years and more (10-20 years) More info: Activities of Super Long Term Support Kernel Workgroup in Civil Infrastructure Platform Project Embedded Linux Conference Europe, 2019[51]
  26. 26. 26 Linux Kernel Source Comparison Table Version Maintenance Period (years) Features Latest Version Supported Real- time kernel Maintainer SoC BSP kernel ? Bug fixes ? N SoC vendor kernel team LTS kernel 2 ~ ? • Bug fixes • Security fixes 5.4 N Kernel.org LTSI kernel 2 ~ ? • Bug fixes • Security fixes • Specific features • New features 4.14 N LTSI (Linux Foundation Projects) CIP kernel 10 + • Bug fixes • Security fixes • Specific features • New features 4.19 Y CIP (Linux Foundation Projects)
  27. 27. 27 ELISA: Safety-Critical Systems [13] • Linux Foundation collaborative project – Build and certify Linux-based safety-critical applications – Define and maintain a common set of tools and processes • SIL2LinuxMP [14] project and the Linux Foundation’s Real-Time Linux project – IEC 61508
  28. 28. 28 Longevity + Stability + Security Performance Real-time SafetyResource Limited Mutually Exclusive ?
  29. 29. Multiple Kernel In Single Platform To fulfill multiple user scenarios 29
  30. 30. 30 FIT (Flattened Image Tree) • Tree data structure • Handle multiple types of image – kernel : kernel image – fdt : dtb file – ramdisk : root file system • Image hashing – md5 – sha1 • Image signing • Each node in configurations has their image configuration in booting stage
  31. 31. 31 More info.: http://git.denx.de/?p=u-boot.git;a=blob_plain;f=doc/uImage.FIT/source_file_format.txt;hb=HEAD
  32. 32. 32 User Space
  33. 33. Choose Proper C Library and Toolchain 33
  34. 34. 34 C Library and Toolchain Comparison Table Category License Features Target Application Maintainer User glibc [15] LGPL 2.1 • Stable ABI • Backward compatibility • Fully symbol versioning • Stack smashing protection/ heap corruption detection • Profiling • Performance • Security GNU uClibc-ng [16] LGPL 2.1 • No-MMU architecture support • Tiny size • Resource Limited uclibc-ng.org Musl [17] MIT • Stable ABI • Backward compatibility • Stack smashing protection/ heap corruption detection • Resource Limited • Security musl-libc.org * Be aware of year 2038 problem [19]Other option [72]
  35. 35. 35 Year 2038 Problem [20] • The time_t datatype is a data type in the ISO C library and kernel structure defined for storing system time values. • 32-bit system can represent dates from  Dec 13 1901  Jan 19th 2038 • It causes integer overflowing on 03:14:08 UTC 19 January 2038
  36. 36. Init System 36
  37. 37. 37 Init System Comparison Table Category License C Library User Note busybox GPL 2.0 uClinux-ng Glibc musl ProteanOS PiBox Resource-limited application sysvinit GPL 2.0+ uClinux-ng glibc musl Devuan systemd LGPL 2.1+ glibc Arch, CentOS, CoreOS, Debian, Fedora, Mint, OpenSUSE, Redhat, Ubuntu Linux only openrc 2-clause BSD musl glibc Gentoo Alpine Linux upstart GPL 2.0 glibc Chromium OS Linux only
  38. 38. Choose proper RFS (Root filesystem) 38
  39. 39. 39 Root Filesystem Comparison Table Category Maintenance Period (years) Number of packages C Library Security Tracker CI Busybox ? 300 ~ 400 applets • uClibc • glibc ? ? Yocto Latest release the previous two releases It depends on meta- * • glibc • musl Y Y Buildroot 1 2000+ [42] • glibc • musl • uClibc-ng Y Y Debian 3 + 2 (i386, amd64, armel, armhf and arm64) 51000+ • glibc • musl Y Y
  40. 40. System Development Tools 40
  41. 41. 41 System Development Tools Comparison Table Root filesystem System Development Tools Toolchain System Development Tools License Busybox Yocto OE-Core MIT Yocto Yocto OE-Core MIT Buildroot Buildroot Buildroot GPL 2.0+ Debian ISAR Debian toolchain Metadata: MIT Others: GPL 2.0 ELBE Debian toolchain GPL 3.0+ Yocto Deby (meta-debian) OE-Core MIT Live-build Debian toolchain GPL 3.0+
  42. 42. 42 Why to Choose Debian [40] Good system security [41] Everything is open Usually, fixed packages are uploaded within a few days Stability unstable → testing → stable Scalability Server, Desktop, Laptop, Embedded devices Long term support 5 more years by Debian-LTS project (i386, amd64, armel, armhf and arm64) Multiple architectures alpha, amd64, armel, armhf, aarch64, hppa, i386, ia64, mips, mipsel, powerpc, s390, and sparc Open source compliance All packages in the Debian archive must to comply with the Debian Free Software Guidelines (DFSG)
  43. 43. Upstream First 43
  44. 44. v1.0 Security fixes New features v1.1 v2.0 Project inside the organization v1.0 v1.1 v2.0 In-house security or bug patches Bug fixes Upstream 44
  45. 45. •The project shares its results with the upstream •The project fulfills longer time maintenance and security fixes •The project develops their code very quickly •The project faces difficulties to backport upstream patches due to conflicts as time goes by 45
  46. 46. 46 46 Maintenance PhaseDevelopment Phase System Preparation Regular UpdateBuild and Testing 4321 Long-term Test More info: Building, Deploying and Testing an Industrial Linux Platform Open Source Summit Japan, 2017 [42]
  47. 47. 47
  48. 48. 48 CI/ CD Automatic Release Pipeline Building Testing Deploying 432 1 Release Continuous Delivery Continuous DeploymentCode Continuous Integration
  49. 49. 49 CI/ CD Automatic Release Pipeline Building Testing Deploying 432 1 Release Code Continuous Integration
  50. 50. 50
  51. 51. 51 Static Testing Cases Management - Jenkins Static analysis #1 Static analysis #2 … Static analysis #n
  52. 52. 52 Image: https://c1.staticflickr.com/5/4030/4438139050_04604b4908.jpg
  53. 53. 53 Distributed Compiler • Software – Icecream/ IceCC was created by SUSE based on distcc [43][44] • Improve performance of compile jobs in parallel • Add dynamic scheduler of the compilation jobs • Support multiple platform • Support cross compiling • Hardware - for each node – SSD – Large capacity memory – Gigabit LAN
  54. 54. 54 CI/ CD Automatic Release Pipeline Building Testing Deploying 432 1 Release Continuous Delivery Code Continuous Integration
  55. 55. 55
  56. 56. 56 Continuous Delivery – LAVA [45][46]
  57. 57. 57
  58. 58. 58 Fuego [47][48] • Test framework for testing embedded Linux – Official automated test framework for the LTSI project. • BSD 3-Clause license in default • Over 100 pre-packaged tests – Ability for 3rd parties to initiate or schedule tests on our hardware, and the ability to share our test results with others.
  59. 59. 59
  60. 60. Maintenance Phase Long-term Testing and Regular Update 60
  61. 61. More info: Long-term Maintenance Model of Embedded Industrial Linux Distribution Open Source Summit China 2017 61 61 Maintenance PhaseDevelopment Phase System Preparation Regular UpdateBuild and Testing 4321 Long-term Test
  62. 62. 62 Notification Test Results * Test cases are managed by test framework
  63. 63. 63 For Stable Kernel Maintenance • Automated Linux Kernel Testing [49][50] – Detect, bisect, report and fix regressions on upstream Kernel trees before release – Short tests on many configurations KernelCI
  64. 64. 64 Reproducible Builds [51] • Create an independently-verifiable path from source to binary – Ensure builds have identical results – Act as part of a chain of trust – Prove the source code has not been tampered/modified
  65. 65. 65 Open Source Testing Tools Continuous Integration • Jenkins [55] • Jenkins X [56] Continuous Delivery/ Deployment • LAVA 2 [45] Distributed compiler service • icecc • GOMA [57][58] • distcc [51] Test Case Management • Jenkins • LAVA 2 • Fuego [47][48] Version Control • Git with gitlab [64] Static Program Analysis • Coding style • OWASP • Infer [53] • Sonarqube [54] Dynamic Program Analysis • Gcov [59] • Valgrind [60] • Profiling tools [61] Vulnerability Scanning • OpenVAS [63] • Vuls [65] Fuzzing Testing • Syzkaller [66] • Trinity [68] • OSS-fuzz [69] • The Fuzzing Project [70]
  66. 66. 66 CI/ CD/ LT are concepts of software engineering instead of tools or procedures
  67. 67. 67 67 Maintenance PhaseDevelopment Phase System Preparation Regular UpdateBuild and Testing 4321 Long-term Test More info: Software Update for Embedded Systems COSCUP Taiwan 2019
  68. 68. 68 Why We Need Software Update? Bug fixes Security fixes New userspace program New kernel features
  69. 69. 69 The Components Might Be Updated Components Size Update frequency Risk Peripheral devices firmware < 10 MB Rarely Mid Bootloader (including SPL) < 1 MB Rarely High Device tree <100 kB Rarely High Linux kernel < 10 MB Regularly High Root file system Variant Regularly High System configuration < 1 MB Rarely Low Application Variant Often Low
  70. 70. 70 Software Update Requirements Basic Features Fail-safe Roll-back Size reduction Signatures Multiple storage type support (e.g., NOR/NAND flash, eMMC) Build system integration Remote access (e.g., OTA)
  71. 71. 71 Software Update Requirements Additional Features Online and offline updates Encryption Delta-updates Successful update detection Proactive updating
  72. 72. 72 Update Approaches Components Size Complexity Time Cost Image/ block based Large Low Very High File based Variant Low Variant Package based (e.g., deb, rpm) Variant Low Variant Delta based Low Very High Low
  73. 73. 73 Partition Architecture
  74. 74. 74 Asymmetric/ Symmetric Firmware Updates [71]
  75. 75. 75 Comparison - Features Category Fail-Safe Roll-Back Delta- Updates Signatures Multiple Storage Type Support Build System Integration SWUpdate Y Y librsync Y •NOR NAND flashes •UBI volumes •SD / eMMC Yocto/ Buildroot RAUC Y Y casync Y •NOR NAND flashes •UBI volumes •SD / eMMC Yocto/ Buildroot OSTree N Y archive- z2 Y ? Yocto
  76. 76. 76 Comparison - Others Method Asymmetric/ Symmetric Image Updates Type Language License SWUpdate Both Image-based File-based C99 GPLv2 With openssl exception RAUC Both Image-based File-based C LGPLv2.1 OSTree Asymmetric File-based C/C++ MPL 2.0 /LGPLv2+
  77. 77. 77 Conclusion Community Collaboration Different approach for multiple target applications Preparedness Planning Longevity, stability and security
  78. 78. 78 Thank you
  79. 79. 79 References [1] https://www.openchainproject.org [2] https://www.openinventionnetwork.com/ [3] https://icss20.sched.com/event/ZjMw/managing-vulnerabilities-in-open-source- components-in-ics [4] https://en.wikipedia.org/wiki/Das_U-Boot [5] https://en.wikipedia.org/wiki/Coreboot [6] http://www.rodsbooks.com/refind/ [7] https://en.wikipedia.org/wiki/REFInd [8] https://tiny.wiki.kernel.org/start [9] https://bootlin.com/pub/conferences/2017/jdll/opdenacker-embedded-linux-in-less- than-4mb-of-ram/opdenacker-embedded-linux-in-less-than-4mb-of-ram.pdf [10] https://www.kernel.org/category/releases.html [11] https://ltsi.linuxfoundation.org/ [12] https://www.cip-project.org/ [13] https://elisa.tech/
  80. 80. 80 References [14] http://www.osadl.org/SIL2LinuxMP.sil2-linux-project.0.html [15] https://www.gnu.org/software/libc/ [16] https://uclibc-ng.org/ [17] http://events.linuxfoundation.org/sites/events/files/slides/uclibc-still-makes-sense-brodkin- elce2017.pdf [18] https://www.musl-libc.org/ [19] https://en.wikipedia.org/wiki/Year_2038_problem [20] http://elinux.org/images/6/6e/End_of_Time_--_Embedded_Linux_Conference_2015.pdf [21] https://en.wikipedia.org/wiki/Linux_startup_process [22] http://upstart.ubuntu.com/faq.html [23] https://en.wikipedia.org/wiki/Systemd [24] https://sysdfree.wordpress.com/2019/03/09/135/ [25] https://wiki.gentoo.org/wiki/Comparison_of_init_systems [26] https://elinux.org/images/6/69/Demystifying_Systemd.pdf [27] http://proteanos.com/ [28] https://www.piboxproject.com/ [29] https://lists.debian.org/debian-devel/2016/02/msg00122.html [30] https://busybox.net/FAQ.html#libc [31] https://wiki.yoctoproject.org/wiki/Stable_branch_maintenance
  81. 81. 81 References [32] https://www.debian.org/social_contract#guidelines [33] https://bootlin.com/pub/conferences/2018/elc/petazzoni-buildroot-whats-new/petazzoni-buildroot- whats-new.pdf [34] https://events.static.linuxfound.org/sites/events/files/slides/libc-talk.pdf [35] http://events17.linuxfoundation.org/sites/events/files/slides/ELC%202016%20- %20Designing%20a%20distro%20from%20scratch%20using%20OpenEmbedded.pdf [36] https://github.com/meta-debian/meta-debian [37] https://events.linuxfoundation.org/wp-content/uploads/2017/12/ELCE2018_Debian-Yocto-State-of- the-Art_r6_Kazuhiro-Hayashi.pdf [38] https://events.linuxfoundation.org/wp-content/uploads/2017/12/Buildroot-vs-Yocto-Differences-for- Your-Daily-Job-Luca-Ceresoli-AIM-Sportline.pdf [39] https://events.static.linuxfound.org/sites/events/files/slides/belloni-petazzoni-buildroot-oe_0.pdf [40] https://www.debian.org/intro/why_debian.en.html [41] https://www.debian.org/security/index.en.html [42] http://events.linuxfoundation.org/sites/events/files/slides/Building%2C%20Deploying%20and%20Testing% 20an%20Industrial%20Linux%20Platform.pdf [43] https://github.com/icecc [44] https://www.slideshare.net/szlin/distributed-compiler-icecc
  82. 82. 82 References [45] https://validation.linaro.org/static/docs/v2/# [46] http://elinux.org/images/3/35/LAVA_Project_Update.pdf [47] https://elinux.org/Fuego [48] http://fuegotest.org/ [49] https://kernelci.org/ [50] https://reproducible-builds.org/ https://fosdem.org/2019/schedule/event/kernelci_a_new_dawn/attachments/slides/3300/export/events/attachments/ker nelci_a_new_dawn/slides/3300/gtucker_kernelci_fosdem_2019_v2_3_1024x768.pdf [51] https://github.com/distcc/distcc [52] https://wiki.jenkins-ci.org/display/JENKINS/Plugins [53] http://fbinfer.com/ [54] https://www.sonarqube.org/ [55] https://jenkins.io [56] https://jenkins.io/projects/jenkins-x/ [57] https://chromium.googlesource.com/infra/goma/server/ [58] https://chromium.googlesource.com/infra/goma/client [59] https://gcc.gnu.org/onlinedocs/gcc/Gcov.html [60] http://valgrind.org/ [61] https://perf.wiki.kernel.org/index.php/Main_Page [62] http://linux-test-project.github.io/ [63] http://www.openvas.org/
  83. 83. 83 References [64] https://about.gitlab.com/ [65] https://vuls.io/ [66] https://github.com/google/syzkaller [67] https://kselftest.wiki.kernel.org [68] http://codemonkey.org.uk/projects/trinity/ [69] https://github.com/google/oss-fuzz [70] https://fuzzing-project.org/ [71] https://mkrak.org/wp-content/uploads/2018/04/FOSS-NORTH_2018_Software_Updates.pdf [72] https://android.googlesource.com/platform/bionic [73] https://events.linuxfoundation.org/wp-content/uploads/2017/12/Strategies-for-Developing-and- Deploying-your-Embedded-Applications-and-Images-Mirza-Krak-Mender.io_.pdf [74] http://events17.linuxfoundation.org/sites/events/files/slides/ELC2017_SWUpdate.pdf [75] https://events.linuxfoundation.org/wp-content/uploads/2017/12/ELCE-2018-Update-Tools-BoF_Jan- Lubbe.pdf [76] https://events.linuxfoundation.org/wp-content/uploads/2017/12/ELCE-2018-Update-Tools-BoF_Jan- Lubbe.pdf [77] https://elinux.org/images/f/f5/Embedded_Systems_Software_Update_for_IoT.pdf [78] https://rauc.readthedocs.io/en/latest/ [79] http://elinux.org/images/6/6e/End_of_Time_--_Embedded_Linux_Conference_2015.pdf

×