Continuous Deployment is all over software companies. We will look at how to transfer some of the methods to hardware startups. Exploring embedded devices as an iterative process rather than from a traditional engineering approach. - why is hardware so hard - hardware test types - over the air updates - A case for continuous deployment - vendors hate you - libvirt test host hardware abstraction - A quick look at a pratical jenkins setup

1. Continuous Deployment for Massive Scale Embedded Devices (IoT)

2. Continuous Deployment for Massive Scale
Embedded Devices
(Internet of Things, Hype hype!)
Intro

3. - Introduction to massive scale hardware
- Libvirt hub agent
- OTA CD
- Build environments for embedded linux
- Vendors
Intro

4. particle.ioNest
Examples of “Internet of things” hype
Raspberry pi
Internet Connected Hardware

5. 3,370,000,000
Residential internet gw
Actual economically relevant internet hardware
500,000
Teslas
6,800,000,000
smartphones
Internet Connected Hardware

6. bankruptcysoftwaremanufacture
Hardware is Hard
- Hardware is done at massive scale
- Hence it used to be a waterfall business
- Fixing anything costs millions and takes years
- Hence test suits are massive and take forever
designspecify

7. Embedded Software Testing
Bringup
Acceptance
Compliance
EMI
Functional
Integration
Regression
Smoke
Fresh from vendor Developing Software Building millions
Aging
Functional
Calibration
Weardown
Testing is driven by hardware developers.
Some typical examples:

8. Embedded Software is Hard
- work on very few, very
expensive prototypes
- Test requires specialised jigs and
probes
- Manual reset required when
hardware gets stuck
- All the tools are sudo and bad

9. Embedded Software is Hard
Typical test script you get from vendors:
- sudo apt-get -y install some-stuff-that-doesnt-exist
- sudo echo > /dev/something #never worked
- sudo someflashtool-binary --device=/dev/ttyUSB0
- sudo ./usbthing --find-device-automagically
- sudo scp 192.168.1.2:/test.xml /home/android/meh.
cvs
- echo probably worked. like i care. lol

10. The 2000 Solution
- TTY to LAN
- Allows sharing hardware within teams
- Conflict resolution: “shout across the
office”
- I can’t believe people still do this
- Usb controllable PSU
- To “reboot” during test cycles

11. Embedded Software Testing
Give me continuous integration plz

12. Mocking Hardware
Emulating/Mocking hardware for faster tests
➢ Nah.
➢ Buy more hardware

13. DOCKERIZE IT!!11
Damage control bad software? docker!

14. … or something...
- Anything using kernel namespaces
- Or in this case libvirt

15. Hub Agent at Nokia
1 usb hub
=
1 virtual machine

16. Hub Agent
udev
Hub agent
New device on hubNew usb hub
Spawn agent VM
Register agent
Attach device to agent
Set agent label

17. Hub Agent at Superscale
Docker now supports usb.
Yey!

18. Over The Air Updates
CD Pipeline would be incomplete without delivery
Don’t build devices without internet in 2016.
Continuously deliver updates

19. Over The Air Updates
1,000,000 SCADA systems
are exposed on the internet
- Used in Factories, Nuclear plants,
Hospitals,...
- 700+ vulnerabilities
- No OTA = no Fix
- => Have OTA, plz

20. Over The Air Updates
Typical memory layout
system
user data / config state
bootloader
calibration
Vendor code from hell
Done in factory. Don’t touch.
Stuff to delete in CI.
Always test from clean state
OTA goes here

21. Over The Air Updates
Why OTA is full-image
➢ This is hardware, it doesn’t change.
➢ We must test a known state
➢ OTA scales to billions

22. Over The Air Updates
Classic staged rollouts before CD
The thing
git push git pull
Build & test
Roll to 1000 customersRelease for real

23. Continuous Hardware
git push build
OTA cloud
Dude
your stuff is broketh
Roll this?

24. Build Environments
OBS
Meego
yocto
openwrt
buildroot
android
package based system based hacks
scratchbox2
Some words on scalability of build environments

25. Vendors
They don’t care.

26. Vendors
They don’t care.
No one reads your test instructions. Barely anyone speaks english.

27. Vendors
Hardware vendors are “agile” too.
memory chip origin: “found box in garage”
➢ Yield test != works
➢ Have every single variant in your test rack
➢ Write down chip versions on incoming
batches
➢ Beware of changes even within batches
➢ Thanks to continuous deployment, we actually
CAN test all variants on git push in parallel

28. Vendors
Destruction tests you want on every variant
➢ Flash memory until death (must survive
specified number of cycle)
➢ Max that cpu, forever, until it desolders
➢ Battery charging cycles.
➢ Extreme environment conditions
➢ Find creative ways of destroying stuff
➢ (yeah, you, don’t trust that vendor)

29. Exit
Internet hardware is hard, implement CD early
➢ Flash memory until death (must survive
specified number of cycle)
➢ Max that cpu, forever, until it desolders
➢ Battery charging cycles.
➢ Extreme environment conditions
➢ Find creative ways of destroying stuff