Igalia is a free and open source software company that develops in areas like kernels, virtualization, and graphics. The document discusses using virtual hardware for testing low-level software like firmwares and drivers to save resources and time compared to relying solely on real hardware. It describes Igalia and CERN's efforts to support a FMC-TDC board in Linux, including developing drivers, libraries, and tests while facing issues with unavailable hardware and evolving specifications. Virtualizing the hardware allowed parallel development and easier debugging to minimize these issues. The document concludes with discussing using virtual hardware to test components of the WhiteRabbit project and opportunities to improve the virtual models and testing suite.

1. Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
7th WhiteRabbit Workshop - November 2012

2. Agenda
About Igalia
Virtual vs Real
Linux support and making of FMC TDC board
Testing, Virtual Hardware and WhiteRabbit
Technical details and demo
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

3. Who is Igalia?
Free and open software company
50 hackers around the globe
Hacking in different areas
kernel
virtualization
graphics/rendering
compilers
distros
...
www.igalia.com
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

4. Virtual vs Real
Using virtual HW is all about having the possibility to test
low level software, such as firmwares, HDL cores or
kernel/drivers, without real HW. It is not about replacing
real hardware at all.
The idea is analysing every test case on real hardware in
order to know if we can run it on virtual hardware. This
approach saves resources and time.
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

5. Linux support and making of FMC TDC board
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

6. FMC TDC Linux support
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

7. FMC TDC Linux support
Time-To-Digital-Converter (TDC) board
Used for measuring the time of arrival of each incoming
pulse in order to save the timestamp into the memory,
ready to be read later
CERN/Igalia effort to support FMC-TDC board in Linux
drivers, user-space library and test cases developed
Integration with ZIO framework, use FMC bus and SPEC
drivers
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

8. Making of FMC TDC Linux support
We face the typical issues when developing drivers for non
mature hardware
Hardware not available
Buggy firmwares delaying development
Is it a hardware or software bug? Where is it?
Flashing firmware, running tests... are time-intensive tasks
Hardware specification evolving along the project
...
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

9. Making of FMC TDC Linux support
In order to minimize the previous issues we explored the
virtual way with the following results ...
No real hardware bottle-neck, parallel development
No buggy firmwares issues
Got a virtual baseline in order to test old and new
specifications/firmwares, bug identification really easy
Saved time in real flashing and testing approaches
Developed continuous integration and automatic testing
tooling for software (kernel, libraries, etc) on top of virtual
hardware
Got good-enough virtual models (buses, I/O spaces,
memory maps, etc) to cover goals.
Further emulation (VHDL, firmware, etc) possible but it
wasn’t required for this project
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

10. Testing, Virtual Hardware and WhiteRabbit
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

11. Testing, Virtual Hardware and WhiteRabbit
Some thoughs ...
It is not possible catching all bugs in a complex system
A complex system based on hardware/software require
different, but integrated, testing approaches
Pragmatic testing is a must. Keep all possible testing in the
soft side. It is more affordable, fast and flexible
It doesn’t matter where the bug is (software or hardware).
If it appears in a critical system then it is a critical bug
In the long, continuous integration and automatic testing
is the way
Flexible and robust testing approach is key to get great
releases
At the end, final testing should always run in the real
system
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

12. Testing, Virtual Hardware and WhiteRabbit
Potential WhiteRabbits components to test
Tools and compilers (ARM, LM32)
Builroot
Barebox (bootloader)
IPL
Linux kernel and drivers (versions, configurations, etc)
User space applications (ptpd, hal, libs, etc)
VHDL and LM32 binaries
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

13. Testing, Virtual Hardware and WhiteRabbit
Testing with Virtual Hardware would be possible on the
following components
No virtual Hardware required
Tools and compilers (ARM, LM32)
Builroot
Virtual Hardware required
Linux kernel and drivers code (similar FMC TDC approach)
LM32 binaries (real-time)
VHDL (behaviour)
Not explored
Barebox (bootloader)
IPL
User space applications (ptpd, hal, libs, etc)
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

14. Technical details and demo
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

15. Our work
Virtualized models
Industry Pack
TEWS TPCI-200.
GE IP-OCTAL-232.
FMC TDC
SPEC board (only needed bits)
FMC TDC board
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

16. FMC TDC board
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

17. FMC TDC board
It was needed to test error conditions on the driver
Generic stuff
The virtual board in being detected by the driver as a
genuine device.
Normal mode: no error injection.
DMA
DMA error: TDC_DMA_STAT_ERR and
TDC_DMA_STAT_ABORT.
DMA timeout.
Input pulses
Filter events with pulse width lower than 100 ns
(requirement).
Test different input pulse arrival rate.
Test timestamp threshold setup.
Test time threshold setup.
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

18. What was virtualized?
SPEC
PCI: (sub)Vendor ID, (sub)Device ID. No MSI support.
Memory Map: BAR addresses.
FMC TDC Firmware
Memory map: status and configuration registers, Event’s
circular buffer.
DMA core: memory map, registers.
Input pulse generation
Different parameters setup: input pulse arrival rate, pulse
width, etc.
Error injection
As it was previously mentioned.
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

19. How we did that
QEMU (http://wiki.qemu.org)
Open source machine emulator and virtualizer.
Buildbot (http://trac.buildbot.net/)
Continuous integration system designed to automate the
build/test cycle.
Python. Web interface to check the logs.
Our own testing suite
Based on Sam’s experience working with PTS in the past.
Needed more flexibility: different setups for the same test.
Developed in Python, as PTS.
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

20. Testing suite design
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

21. Testing suite design
Launch QEMU with some setups:
Choose memory size, virtual machine image, network, etc.
Snapshot mode.
Shared folders to interchange data.
No screen.
The kernel is pass as a parameter to QEMU -> Not using
the one in the virtual machine.
Timeout. If QEMU is still running after a given time,
something wrong happened.
Forward kernel logs to QEMU’s serial port. We can read
dmesg info from host.
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

22. The result (I)
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

23. The result (II)
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

24. The result (III)
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

25. The result (IV)
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

26. Potential improvements
Improve the virtual model.
Do a more realistic model of the device.
Add VHDL virtualization.
QEMU configuration: use config files?
Buildbot: fix issues.
Testing suite
Add new features (hooks, etc).
Choose Red Hat’s autotest framework?
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

27. Testing and HW Virtualization
Questions & Answers
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

28. Testing and HW Virtualization
Thank you!
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez

29. Links
http://www.ohwr.org/projects/spec
http://www.ohwr.org/projects/fmc-tdc
http://www.ohwr.org/projects/fmc-tdc-sw
http://blogs.igalia.com/berto/2012/10/03/
industrypack-qemu-and-linuxcon/
http://events.linuxfoundation.org/images/
stories/pdf/lceu2012_garcia.pdf
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez