ITRenew is selling recertified OCP servers under the Sesame brand, those servers come either with their original UEFI BIOS or with LinuxBoot. The LinuxBoot project is pushing the Linux kernel inside bios flash and using userland programs as bootloader. To achieve quality on our software stack, as any project, we need to test it. Traditional BIOS are tested by hand, this is 2019 we need to do it automatically! We already presented the hardware setup behind the LinuxBoot CI, this talk will focus on the software. We use u-root for our userland bootloader; this software is written in Go so we naturally choose to use Go for our testing too. We will present how we are using and extending the Go native test framework `go test` for testing embedded systems (serial console) and improving the report format for integration to a CI. Julien Viard de Galbert

1. Testing Firmware the DevOps way

2. System Firmware
The firmware that starts your computer
BIOS
UEFI
LinuxBoot

3. LinuxBoot on OCP hardware
ITRENEW is selling Open Compute Project hardware under the SESAME brand.
Most of our hardware comes from hyper-scale datacenter decommissioning.
The recertification process includes updating the firmwares.
One option for firmware is LinuxBoot.
ITRENEW historical activity is decommissioning

4. LinuxBoot ?
What
Linuxboot is placing a Linux kernel in the BIOS flash and using userland to implement a bootloader.
Presented last year at ER2018 by Trammel Hudson.
We use u-root and it's bootloader implementations
Why
Maintainability
Security
microcode update
Reduice/Remove UEFI footprint
Linux is already part of our security audit...
Extra features (nvme, infra integration)
Replacing the UEFI (proprietary) firmware, with an open source one gives us those advantages.

5. Developing LinuxBoot need Testing !
Testing requirements
Automated testing with CI, like devops!
Keeping the full logs from the console for debugging
Up to 3 logs to watch:
test output
serial console
IPMI console
Keep all details on first run (long tests)
Simple/summary view by default
IPMI provides a Serial Over LAN access to the console.

6. Testing Hardware setup
This setup was presented last year at ER2018 by Jean-Marie Verdun.
The hardware for testing is composed of
servers used as DUT,
controlling servers used as CI runners.
Each DUT has:
a relay for controlling power,
a flash emulator to load firmware.
So today we will focus on the testing software.

7. Let's look at it!
DEMO time!
If you are reading the slides: Features are presented live here, explanations comes later...

8. Why use Golang to write tests ?
u-root is written in Go
Go has it's test framework: go test and the testing package.
BUT
Go test output does not directly fulfil our requirements

9. go test
Files named *_test.go are compiled only for testing.
Functions named Test*(t) are test case automatically run by the framework.
Inside test functions subtest can be created by calling t.Run.
Short example:
func TestMyFeature(t *testing.T) {
got := MyFeature(/* ... */)
want := "OK" /* expected result */
if got != want {
t.Fatalf("test failed, got %v, want %v", got, want)
}
}

10. go test output
Success has very minimal output:
ok MyPackage 0.001s
Error is more verbose:
--- FAIL: TestMyFeature (0.00s)
prez_test.go:13: test failed, got KO, want OK
FAIL
FAIL MyPackage 0.001s
Verbose mode: messages on success, error gets flooded :(
=== RUN TestMyFeature
--- PASS: TestMyFeature (0.00s)
PASS
ok command-line-arguments 0.001s

11. go test -json
There is a JSON output for further processing
an event log with one JSON event per line
{"Action":"run","Test":"TestMyFeature","Time":"2019-08-12T11:55:41.174693884+02:00"}
{"Action":"output","Test":"TestMyFeature","Output":"=== RUN TestMyFeaturen","Time":"2
{"Action":"output","Test":"TestMyFeature","Output":"--- FAIL: TestMyFeature (0.00s)n","
{"Action":"output","Test":"TestMyFeature","Output":" prez_test.go:13: test failed, go
{"Action":"fail","Test":"TestMyFeature","Elapsed":0,"Time":"2019-08-12T11:55:41.17486883
{"Action":"output","Output":"FAILn","Time":"2019-08-12T11:55:41.174877223+02:00"}
{"Action":"output","Output":"FAILtMyPackaget0.001sn","Time":"2019-08-12T11:55:41.1750
{"Action":"fail","Elapsed":0.001,"Time":"2019-08-12T11:55:41.175039291+02:00"}

12. GoTestWeb in a nutshell
html5/css/js application
Parses go test json output client side
Integrates asciinema-player for showing console (serial, IPMI) logs
parses output event to get asciinema filename
events timestamp are used to seek the video on click
Renders the verbose test results folded if OK, open if Failed.
Keep test hierarchy (renders subtests inside tests)

13. GoTestWeb development
Disclaimer: I'm not a front dev!
github: GoTestWeb
techno:
Bootstrap
jQuerry
Ractive.js
RequireJS

14. Enough of JS, more of Go!

15. Tastevin
Tastevin is grouping the Go code used to build the tests.
Provides packages to ease writing test for remote platform.
Includes a gotest command to wrap go test -json save the json and display the summary.
Named as a wine tool to go with fiano the project to analyse and build LinuxBoot images from UEFI images.

16. GoExpect
goexpect is an implementation of Expect in go.
expect allows to script a behavior on a terminal:
wait for a regex
send a string
wait for another regex
send another string
...

17. GoExpect Spawners
command
ssh
generic (to be customized)

18. GoExpect custom Spawners
serial Spawner
connect a serial port to the generic spawner
IPMI Spawner
wrap ipmitool sol activate in a command spawner
QEMU Spawner
wrap qemu in a command spawner
can be extended as needed!

19. GoExpect Batcher
A sequence of actions (expect/send) can be presented in a go slice
BExp expect a regexp with the default timeout
BExpT expect a regexp with a given timeout
BSnd send a string
example for login in a server:
[]expect.Batcher{
&expect.BExp{R: "username:"},
&expect.BSnd{S: *user + "n"},
&expect.BExp{R: "password:"},
&expect.BSnd{S: *pass + "n"},
}

20. BExpTLog: BExpT with Log output
This custom element can be used in Batcher.
It provides the same feature as BExpT:
a matching regex and
a timeout
Adds a Log output to be send when the text is matched.
The goal: this log will be a seek point for the asciinema view.

21. Asciinema Writer
goexpect can duplicate the terminal output to a writer interface.
it's actually a writecloser interface, it will close at the end of the expect session.
The asciinema writer
gets it's data for the writer interface
adds the timestamp for the current event
save it in asciicast v2 format
logs message on open/close so that GoTestWeb can catch the details.

22. Scriptreplay Writer
The scriptreplay writer
log the data to a plain file
record the timestamp in a timing file
can be played with scriptreplay -t file.tim file.log
logs message on open/close so that GoTestWeb can generate download links.

23. Tastevin packages
Includes:
goexpect spawner for serial, IPMI, qemu
goexpect batcher extension: BExpTLog
console writer: asciicast and scriptreplay
control package for the CI system: em100, relay
a basic testsuite for LinuxBoot startup
a few utility packages

24. Tastevin command
gotest run
a wrapper to go test -json
saves the json to a file
output the package test summary (without error detail)
or output the verbose go test output
gotest serve
for developer: given a JSON, locally serve GoTestWeb to render it
https://github.com/gotestyourself/gotestsum can convert json to junit.

25. CI integration
gotest can be used directly in CI systems to save the json and asciicasts
gotest gen can generate the html and export the GoTestWeb application as well
simply place the json, asciicast, html and GoTestWeb in a CI artifact for later rendering in the browser!

26. Developer live mode
DEMO
gotest live
cd gotest-live-demo
gotest live go test -json .

27. Developer live mode
gotest live
run tests as gotest run
serve the results as gotest serve
use websocket to stream the results between gotest live and the browser
asciinema-player can only play plain files so this is unfortunately not streamed.

28. Tastevin and GoTestWeb are ready
For you to use and contribute to!
https://github.com/JulienVdG/tastevin/
https://github.com/JulienVdG/gotestweb/
Or contact me for more informations:
Julien Viard de Galbert julien.viard-de-galbert@itrenew.com

29. Questions ?

30. Links
Tastevin and GoTestWeb on github:
https://github.com/JulienVdG/tastevin/
https://github.com/JulienVdG/gotestweb/
More on ITRENEW vision by Ali Fenn at OCPSummit19
Unlocking the Power of Circular Datacenters
Introducing Sesame Open Hardware Compute
Me: Julien Viard de Galbert julien.viard-de-galbert@itrenew.com