One of the killer features of Xen is the ability to contain qemu in a minimal stubdomain. But even though qemu-upstream has been supported by Xen for a long time, stubdomains are compatible only with the ancient qemu-traditional. There were multiple approaches to this problem discussed over time (rumprun, Linux, ...), including some PoC patches. In this presentation I'll explain why we've chosen the Linux solution in Qubes OS and what challenges we faced to make it really work.

1. A bit of history
QEMU upstream
Linux-based Device Model Stubdomains in Qubes
OS
Marek Marczykowski-G´orecki, Invisible Things Lab
June 22, 2018
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

2. A bit of history
QEMU upstream
A bit of history
1 Qubes 1.0 (2012) - PV domains only
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

3. A bit of history
QEMU upstream
A bit of history
1 Qubes 1.0 (2012) - PV domains only
2 Qubes 2.0 (2014) - initial HVM support, MiniOS based
stubdomain for qemu-traditional (the only one at that time)
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

4. A bit of history
QEMU upstream
A bit of history
1 Qubes 1.0 (2012) - PV domains only
2 Qubes 2.0 (2014) - initial HVM support, MiniOS based
stubdomain for qemu-traditional (the only one at that time)
3 Qubes 4.0 (2018) - most PVHv2 domains, HVM with
linux-based and qemu upstream stubdomains where needed
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

5. A bit of history
QEMU upstream
MiniOS based stubdomains in Qubes
No qemu in dom0 (hard requirement) - patched libxl
Display using qubes-gui-agent (port for qemu)
DHCP server based on LWIP
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

6. A bit of history
QEMU upstream
Why change?
Hard to debug and develop
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

7. A bit of history
QEMU upstream
Why change?
Hard to debug and develop
Lack of newer device support (audio, vbkd, . . . )
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

8. A bit of history
QEMU upstream
Why change?
Hard to debug and develop
Lack of newer device support (audio, vbkd, . . . )
Hope to get better security support for currently maintained
qemu (as we don’t consider PV bulletproof anymore)
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

9. A bit of history
QEMU upstream
Why change?
Hard to debug and develop
Lack of newer device support (audio, vbkd, . . . )
Hope to get better security support for currently maintained
qemu (as we don’t consider PV bulletproof anymore)
MiniOS build system gives a lot of headache. . .
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

10. A bit of history
QEMU upstream
Why change?
Hard to debug and develop
Lack of newer device support (audio, vbkd, . . . )
Hope to get better security support for currently maintained
qemu (as we don’t consider PV bulletproof anymore)
MiniOS build system gives a lot of headache. . .
Direct kernel boot for HVM
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

11. A bit of history
QEMU upstream
What we need?
No qemu in dom0
PCI passthrough
Custom GUI (instead of VNC/SDL)
DHCP server (nice to have)
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

12. A bit of history
QEMU upstream
Which stubdomain?
Rumprun, not progressing at that time, dead now
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

13. A bit of history
QEMU upstream
Which stubdomain?
Rumprun, not progressing at that time, dead now
Linux, deployed by OpenXT (use OpenEmbedded, and v4v for
communication)
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

14. A bit of history
QEMU upstream
Which stubdomain?
Rumprun, not progressing at that time, dead now
Linux, deployed by OpenXT (use OpenEmbedded, and v4v for
communication)
Linux, patches by Anthony Perard, later revived by Eric
Shelton
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

15. A bit of history
QEMU upstream
Stubdomain build blocks
dracut-based build - use build host binaries (busybox, glibc
etc)
minimal Linux kernel (based on make tinyconfig)
recent upstream QEMU build (2.10.1 as of today)
udhcpd (busybox)
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

16. A bit of history
QEMU upstream
Make it work
Load kernel + initramfs (can be bundled into one binary later,
but simpler to develop when separate)
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

17. A bit of history
QEMU upstream
Make it work
Load kernel + initramfs (can be bundled into one binary later,
but simpler to develop when separate)
Pass qemu command line via xenstore (as for MiniOS) -
putting qemu-xen variant there
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

18. A bit of history
QEMU upstream
Make it work
Load kernel + initramfs (can be bundled into one binary later,
but simpler to develop when separate)
Pass qemu command line via xenstore (as for MiniOS) -
putting qemu-xen variant there
-append can contain spaces. . . , workaround: use
x1b separator and set FS=
x1b
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

19. A bit of history
QEMU upstream
Make it work
Load kernel + initramfs (can be bundled into one binary later,
but simpler to develop when separate)
Pass qemu command line via xenstore (as for MiniOS) -
putting qemu-xen variant there
-append can contain spaces. . . , workaround: use
x1b separator and set FS=
x1b
Different disk configuration: format=host device
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

20. A bit of history
QEMU upstream
Make it work
Load kernel + initramfs (can be bundled into one binary later,
but simpler to develop when separate)
Pass qemu command line via xenstore (as for MiniOS) -
putting qemu-xen variant there
-append can contain spaces. . . , workaround: use
x1b separator and set FS=
x1b
Different disk configuration: format=host device
No direct access to QMP socket - pass selected commands via
xenstore (as for MiniOS), then convert to QMP commands
inside (a script)
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

21. A bit of history
QEMU upstream
Problems
PCI passthrough related problems
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

22. A bit of history
QEMU upstream
Problems
PCI passthrough related problems
RAM usage, CPU usage
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

23. A bit of history
QEMU upstream
Problems
PCI passthrough related problems
RAM usage, CPU usage
read-only disks (IDE vs AHCI vs SCSI), supported by
Windows installer by default (right now we use mptsas1068)
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

24. A bit of history
QEMU upstream
Problems
PCI passthrough related problems
RAM usage, CPU usage
read-only disks (IDE vs AHCI vs SCSI), supported by
Windows installer by default (right now we use mptsas1068)
no migration / save+restore without qemu in dom0 (only one
console)
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

25. A bit of history
QEMU upstream
Next steps
Cleanup libxl patches
Xenconsoled support for secondary consoles, fix
save/migration
Better design for QMP access
PVH stubdomain?
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

26. A bit of history
QEMU upstream
Next steps
Cleanup libxl patches
Xenconsoled support for secondary consoles, fix
save/migration
Better design for QMP access, how fragile is libxl parsing
QMP response?
PVH stubdomain?
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

27. A bit of history
QEMU upstream
Resources
github.com/QubesOS/qubes-vmm-xen
github.com/QubesOS/qubes-vmm-xen-stubdom-linux
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS

28. A bit of history
QEMU upstream
Questions
Questions?
Marek Marczykowski-G´orecki, Invisible Things Lab Linux-based Device Model Stubdomains in Qubes OS