EmbeddedXEN is a particularly efficient virtualization framework tailored to ARM-based core embedded systems. While security and OS isolation are key features of conventional virtualizuation frameworks, the main concerns for EmbeddedXEN are device heterogeneity and realtime aspects, which are particularly important in the embedded world. EmbeddedXEN mainly relies on the original XEN architecture but with major differences in the way guest OS are handled: the hypervisor has been simplified, and only two guest OS (dom0 and domU) can run simultaneously; while dom0 is used to manage the native OS with drivers (original and backend splitted drivers), a paravirtualized OS (domU) can be cross-compiled on a different ARM device, and user applications can run seamlessly on the (virtualized) host device. Another important difference is that no user space tools are required to manage the VMs; the framework produces a compact single binary image containing both dom0 and domU guests, which can be easily deployed. The Xenbus architecture has been adapted to that context. EmbeddedXEN therefore allows the porting of an OS and its applications from an ARM embedded device to last generation ARM hardware, such as HTC Smartphone for example.

1. Dealing with Hardware Heterogeneity Using
a Virtualization Framework Tailored to ARM Based
Embedded Systems
Prof. Daniel Rossier, PhD
HEIG-VD
Institut REDS, Reconfigurable & Embedded Digital Systems
rte Cheseaux 1, 1400 Yverdon-les-Bains
http://www.reds.ch/

2. Outline
• Background
• Overview of EmbeddedXEN
• Protection & Memory Isolation
• Domain Interactions
• Device Heterogeneity
• Conclusions & Future Work
2 REDS@HEIG-VD

3. Background
• HEIG-VD
• University of Applied Sciences in Yverdon, Switzerland (CH)
• Reconfigurable Embedded Digital System Institute
 Hardware Design, FPGA
 Embedded Execution Environment, Drivers, BSP, OS/RTOS, etc.
• Applied Research & Development
• ARM based Microcontrolers (v4, v5, v6, v7)
• Low-level interactions between computing cores (CPU, DSP, FPGA, etc.)
• Towards Cortex-A15 HVM
3 REDS@HEIG-VD

4. Background
• Embedded Virtualization on ARM
• XEN: free & easy access to a stable & evolving hypervisor source code
• Early port of XEN on ARM in 2007 in the context of a Diploma Project
• Necessity to get a simple, thin, fast, robust, easy-to-deployed virtualization
framework
• Re-use of Linux file organization & build system (Makefiles, scripts, …)
• Focus on realtime aspects (Linux/Xenomai as RTOS)
• Different sources of inspiration
• "Fast Secure Virtualization for the ARM Platform", Daniel Ferstay,
Master Thesis, The University of British Columbia, 2006
• XEN ARM port, George G. Davis, MontaVista, 2007
• Secure Xen on ARM Project, Sang-bum SUH, Samsung, 2007
4 REDS@HEIG-VD

5. Background
• Focus on heterogeneity of embedded devices
• Idea to re-use OS & applications from old devices to recent devices
• Time-to-market migration to new hardware generation
• Dealing with various cross-compiled binaries (ARM v5-v7)
• Dealing with different peripherals
• Less emphasis on security aspects
• Publicly available
• https://sourceforge.net/projects/embeddedxen
5 REDS@HEIG-VD

6. Background
• Hardware constraints
• Low latency, reactivity (response time)
• ARM cores do not support virtualization mechanisms
 not easy to deal with various levels of execution modes
• Para-virtualization remains attractive
• About 30 files to be (slightly) adapted
• Low execution overhead
• Efficient processing of downcalls/upcalls with support of domain interactions
 Physical interrupts are quickly processed in dom0.
6 REDS@HEIG-VD

7. Overview of EmbeddedXEN
Primary Guest OS Secondary Guest OS
known as Dom0 known as DomU
Xen-guest Xen-guest
(pv) (pv)
- Full privilege - Full privilege
- Dom0 original drivers
- DomU original drivers
- Backend drivers
- Native drivers - Frontend drivers
• Limited use of hypercalls
• ARM execution modes (USR/SVC) –
pseudo-user mode with double stack handling / downcalls & upcalls are
simple jumps to specific (callback) addresses
• No subtle use of domain access control (ARM DACR) to protect memory
• Memory sharing facilitated between primary and secondary OS
Domain Prioritized
VCPU Creation and EmbeddedXEN Upcalls
sched_flip Setup Handling Migration
Hypervisor Hypercalls
Manager
Scheduler Handling
Hardware
7 REDS@HEIG-VD

8. Overview of EmbeddedXEN
• Linux-like source tree and build system (Makefiles)
Config & build embeddedxen
system
tools
xen-guest
environnement.conf linux-2.6.26-domU
Common
part
xenbus core include console
include xen-guest arch
linux-2.6.32-dom0
hypervisor-4.0.2
include xen-guest arm
arch/arm
xen-guest
mach-msm mach-mx35
xen-guest include arch
Secondary mach-mx35
xen OS
mx35_fab4.c
xen-guest arm
Hypervisor arch/arm
Primary
kernel mm
OS mach-msm
8 REDS@HEIG-VD

9. Overview of EmbeddedXEN
• Single binary (multi-kernel) image
• Automatic parsing & image relocation during hypervisor bootstrap
EmbeddedXEN
Boot Head
1 (Dom-U)
0 (Dom-0)
Hypervisor
EOD
EOD
DOM-0 DOM-U
vmlinux vmlinux.dom0 vmlinux.domU
uImage
Dom-0 Filesystem /home/root/squeezeos.rootfs.img: • Stored separately in
sdcard, for example
Dom-U Filesystem
9 REDS@HEIG-VD

10. Protection & Memory Isolation
• Memory isolation between domains relies on different address space
isolation.
• No further advanced mechanisms to protect hypervisor and guest memory
• The guest OS kernel runs at the same privilege as the hypervisor.
• Each domain receives its own (contiguous) physical memory region
during domain set up.
• No pagination of the kernel linear address space is performed.
• Paravirt of memory management is kept minimal.
• Guest OS kernel has access to the whole memory.
• No protection mechanism for strong isolation of VM (but is it really necessary?)
10 REDS@HEIG-VD

11. Protection & Memory Isolation
• Virtual & Physical Memory Layouts
Initial Virtual Address dom0 Virtual Address
Space Physical RAM Space
Interrupt Vectors Interrupt Vectors
0xffff0000
frame table frame table
XEN heap (2 MiB) XEN heap (2 MiB)
boot allocator boot allocator
domU
Hypervisor code Hypervisor code
Linear
1st-level page table mapping 1st-level page table
0xFF000000
I/O I/O
VMALLOC_END
dom0
domU
dom0
dom0
Linear
PAGE_OFFSET mapping
0xC0000000
Hypervisor
3GiB 3GiB
0x00000000
11 REDS@HEIG-VD

12. Domain Interactions
• Virtualization of peripherals in EmbeddedXEN is quite similar
to the existing mechanisms in XEN.
• Driver split with frontend & backend drivers
• Communication with xenbus
• Use of grant tables for sharing/copying pages between domains
• However, a revisited (simplified) implementation of these
mechanisms have been achieved in EmbeddedXEN.
• XEN store is dynamically allocated at boot time of guest OSes.
• No user space tools are required to manage XEN store entries or peripherals
configs.
• Hotplugs & dynamic configs of peripherals are less relevant to embedded
systems.
12 REDS@HEIG-VD

13. Domain Interactions
• domU is passed information during bootstrap under control of dom0.
xen-guest OS xen-guest OS
xenbus xenstore subsys Subsys subsys Subsys
thread thread
event_channel xenstore
A/B xenbus
thread
Page Dom0 Page DomU
prod prod
cons cons
backend drivers frontend drivers
event_channel C/D
Hypervisor
via unpause_domU(store_mfn, domU
store_evtchn) start_info->store_mfn
start_info->store_evtchn
13 REDS@HEIG-VD

14. Domain Interactions
• Grant tables are used in a different way
• Shared pages are possible only in the vmalloc'd area.
• Kernel linear addresses are not shareable; contents needs to be copied using
temporary mappings.
Dom0 RAM DomU
Hypervisor Hypervisor
0xFF000000 0xFF000000
gnttab(domU)
VMALLOC_END has VMALLOC_END
has references to
references to gnttab(domU) gnttab
gnttab_foreign[1] gnttab(domU) domU
gnttab(dom0) gnttab_foreign[0]
gnttab gnttab(dom0)
shared pages
has
references to has
VMALLOC_START VMALLOC_START
gnttab(dom0) references to
mem_map_foreign mem_map_foreign(domU) dom0 mem_map_foreign(dom0) mem_map_foreign
0xC0000000 0xC0000000
shared pages
3GiB 3GiB
0x00000000
Hypervisor 0x00000000
14 REDS@HEIG-VD

15. Device Heterogeneity
• Different levels of heterogeneity
• At CPU level: various instructions sets (locks, cache, etc.), various PTEs
(MMU) flags, various co-processors, etc.
 Compatibility ensured via hypercalls
• At peripherals level: not the same hardware
 Compatibility ensured via backend driver processing hypervisor-4.0.2
include
arch/arm
xen-guest
mach-msm mach-mx35
xen
built-in.o cache-v6.S cache-v7.S Kconfig Makefile mm.h
proc-macros.S tlb-v6.S tlb-v7.S mm
arch/arm
kernel mm
15 REDS@HEIG-VD

16. Device Heterogeneity
• Example of ARMv6 running on ARMv7 CPU (iMX35 -> HTC Desire HD)
Original version Paravirt version
linux-2.6.26-domU/arch/arm/mm/cache-v6.S: linux-2.6.26-domU/arch/arm/mm/cache-v6.S:
ENTRY(v6_flush_kern_cache_all) .extern xen_flush_kern_cache_all
mov r0, #0 ENTRY(v6_flush_kern_cache_all)
#ifdef HARVARD_CACHE b xen_flush_kern_cache_all
mcr p15, 0, r0, c7, c14, 0 @ D cache clean+invalidate #if 0 /* paravirt */
#ifdef CONFIG_SMP mov r0, #0
mcr p15, 0, r0, c7, c5, 0 @ I+BTB cache invalidate
#else …
b v6_icache_inval_all
#endif mov pc, lr
#else #endif /* 0 */
mcr p15, 0, r0, c7, c15, 0 @ Cache clean+invalidate
#endif
mov pc, lr
linux-2.6.26-domU/xen-guest/hypervisor.c: hypervisor-4.0.2/arch/arm/mm/cache_v7.S:
void xen_flush_kern_cache_all(void) ENTRY(xen_flush_kern_cache_all)
{ stmfd sp!, {r4-r5, r7, r9-r11, lr}
struct mmuext_op op; bl xen_flush_dcache_all @ much more complex!!
mov r0, #0
op.cmd = MMUEXT_FLUSH_CACHE; mcr p15, 0, r0, c7, c5, 0 @ I+BTB cache invalidate
HYPERVISOR_mmuext_op(&op, 1, NULL, DOMID_SELF); ldmfd sp!, {r4-r5, r7, r9-r11, lr}
} mov pc, lr
ENDPROC(xen_flush_kern_cache_all)
16 REDS@HEIG-VD

17. Device Heterogeneity
• Example of framebuffer device heterogeneity
• Configuration of framebuffer is retrieved from Dom0 via xenbus
xen-guest Dom0 xen-guest DomU
User space
applications User space
xenstore (/dev/fb0) xenbus applications
thread (/dev/fb0)
xenbus
thread
framebuffer
core
framebuffer
device-specific core
framebuffer
msmfb_pan_update()
Query framebuffer
xenfb backend xenfb frontend
params
fb_event_dom_register domU fb memory
dom0 fb memory fb_event_dom_switch
FB config is retrieved (allocated by domU)
(allocated by dom0) from dom0
17 REDS@HEIG-VD

18. Device Heterogeneity
• Example of audio device heterogeneity
• DomU audio buffers are accessed from Dom0 via shared pages.
xen-guest
Dom0 xen-guest
DomU
User space applications User space applications
xenstore xenstore
/dev/msm_pcm_out /dev/pcm/snd/pcm0c0d0p
xenbus - or - xenbus
thread /dev/pcm/snd/pcm0c0d0p thread
pcm subsystem pcm subsystem
xen-audio backend
xenvaud_pcm_start xen-audio frontend
xenvaud_pcm_stop
Audio buffers allocated in
domU
arch-specific audio driver
Sound device (headset, speakers, etc.)
18 REDS@HEIG-VD

19. Conclusions & Future Work
• EmbeddedXEN is an embedded virtualization framework which puts
emphasis on efficient and heterogeneous hardware.
• Application environments can be re-used "as such" on modern
platforms (Android-based for example) taking advantage of last
generation hardware.
• EmbeddedXEN relies on the main principles of XEN, with a revisited
lightweight, but less secure architecture.
• A single multikernel binary image, easy to deploy on the target
platform without additional tools, makes EmbeddedXEN well
tailored to embedded systems.
19 REDS@HEIG-VD

20. Conclusions & Future Work
• Further investigation projects:
• Elaboration of a domU using a graphical desktop for user
applications
• Support of multicore ARM CPUs (cortex-A9, cortex-A15)
• Live migration of domU using remote NFS-filesystem (migration
within a cloud)
• Support of hard realtime OS (RTEMS-paravirt)
20 REDS@HEIG-VD

21. • Thanks for your attention!
• Further information: daniel.rossier@heig-vd.ch
21 REDS@HEIG-VD