Talk given at Pipeline conference in London 2017. “A lot of software development falls naturally into the Continuous Delivery mindset, where any potential fix should be done on master and customers can naturally take the latest version of the software. We believe this is the the ideal approach to software development; it results in reduced cycle times, less work in progress, fewer queues and rework, and reduced testing costs. However, not all software falls into this category. Some industries require that a given release be supported for decades. Some industries require that algorithms do not change, even for the better. But of course, any bugs must be fixed. Some industries require that any software that is qualified has a complete audit trail. This presentation will share our experience in doing Continuous Delivery in the industrial embedded software world. It will describe how we can do component-based development for multi-product configurations, how to create a release from any version of our software and how we make sure we can maintain any software we have released to a customer. It will cover version-control strategy, a binary artifact management approach, together with a build and dependency management solution that addresses this.” Mike is a Partner at Praqma, a Continuous Delivery consulting company based in Scandinavia. He has over 13 years of experience delivering software in various cultures and industries. He helps organize several community events and conferences, including CoDe Academy which teaches Continuous Delivery to university students. Mike is a trustee on the cyber-dojo foundation.

1. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Maintainable Embedded Software
Continuous Delivery of
Mike Long

2. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
“Continuous Delivery is the
ability to get changes of all
types...into production, or into
the hands of users, safely and
quickly in a sustainable way.”
https://continuousdelivery.com

3. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
What is Embedded Software?

4. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Where are the Continuous Delivery Challenges?
Access to Production-like environments Rare
Support lifespan Up to 20 years
Safety critical Often
Regulated markets Often
Release cost High
Deployment Custom
Audit trail requirements (full traceability) Often
Deployment/Product matrix Complex

5. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
http://code-maturity.praqma.com https://github.com/Praqma/VersionedBinaryArtifacts

6. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Build the foundations
Step 1
Begin with the standard Continuous Delivery playbook

7. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
http://blog.codinghorror.com/the-f5-key-is-not-a-build-process/
If your "build process" is the
F5 key, you have a
problem…..Get your build
process out of the IDE and into
a build script.

8. Continuous Delivery of Maintainable Embedded Software
www.praqma.com

9. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Development Principles
Component principles:
● Integration and testing of the latest should be automatic
● All binaries should be fully traceable
● Artifacts should be managed
● Metadata should be built into the binary
Release principles:
● Release is a manual decision
● Release binaries can only depend on released components.
● Releases must be maintainable

10. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Build step 1 Know what you built
build.h
// Generated Tue Mar 14 10:45:56 GMT 2017
#ifndef _BUILD_H_
#define _BUILD_H_
#define BUILD_NUMBER 177
#define BUILD_SHA a1b2080ea2692f536eb95577bca2b9f6d944a018
#define BUILD_JOB "http://jenkins.mycompany.com/job/myjob"
#endif
// Generated Tue Mar 14 10:45:56 GMT 2017
#ifndef _VERSION_H_
#define _VERSION_H_
#define PRODUCT_NAME "VBA_demo"
#define MAJOR_VERSION 1
#define MINOR_VERSION 0
#define PATCH_VERSION 3
#define PRE_RELEASE_VERSION "-SNAPSHOT"
version.h

11. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Build step 2 Control how it is built
Put the build tools in a safe place. Artifact management systems are a good place for this.
Libs

12. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Build step 3 Reproducible Builds
Use the same tools everywhere, in development, CI, and release. Defined as code and version controlled.
mycompany/my-cross-compiler:1.3-release
libs
gcc
linker
#Dockerfile
FROM
busybox:1.26.2-glibc
ADD tools.tar.gz
RUN make all

13. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Tales from the trenches Leif Ove

14. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Increase safety
Step 2
Build quality into the product

15. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
“In complex systems, adding
more inspection steps and
approval processes actually
increases the likelihood of future
failures.”
The DevOps Handbook
p. 32

16. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Build Quality In Test Driven Development
module.c
lib.a

17. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Build Quality In Hardware in the
loop system testing
This is an investment, requiring unique engineering.

18. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Build Quality In Static and Dynamic Analysis

19. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Build Quality In What kind of testing do I need?
This is an investment, requiring unique engineering.

20. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Increase development efficiency
Step 3
Optimise the flow

21. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Reuse Managing product variants

22. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Reuse 1 Build and test avoidance
Reusing components reduces test volume and engineering time

23. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Component Build

24. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Component Release

25. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Component Maintenance

26. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Component Maintenance release

27. A Baseline is a container for a configuration of
components chosen from the dev repository
17.1-1.tar.gz
DSP APP
Com
ms
...
Dev

28. Baseline reside in baseline repositories
17.1-1.tar.gz
DSP App
Com
ms
...
Testing
17.1-1.tar.gz

29. Repository per environment
Dev Test Qualification Commercialization
Contains
Components
Contains Baselines

30. Baselines are promoted through environments
Dev Test Qualification Commercialization

31. Baseline Repository Layout
Dev
Test
17.0-1.zip
17.0-2.zip
16.4-4.zip
17.0-6.zip
17.0-8.zip
Qualification
16.4-4.zip
17.0-6.zip
17.0-8.zip
Commercialization
16.4.zip
17.0.zip

32. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Extra credit
Step 4
For the high achievers

33. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Component Auto-incrementing release builds

34. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
ASIC Quality – Continuous Delivery

35. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Final thoughts

36. Continuous Delivery of Maintainable Embedded Software
www.praqma.com

37. Continuous Delivery of Maintainable Embedded Software
www.praqma.com

38. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
The researchers praised Tesla
for the way the vehicle
handled the bogus commands and
how the car maker responded to
their findings.

39. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
In late July, Fiat Chrysler issued a
recall for more than 1.4 million
vehicles after hackers Charlie Miller
and Chris Valasek showed how to use
bugs in the onboard software of the
Jeep Cherokee to shut it down remotely.

40. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
Build the foundations
Increase safety
Increase development efficiency
Summary

41. Continuous Delivery of Maintainable Embedded Software
www.praqma.com
The Continuous Delivery and DevOps Company