1. Linux symposium 2012
Jiun-Hung Ding, Chang-Jung Lin, Ping-Hao Chang
Chieh-Hao Tsang, Wei-Chung Hsu, Yeh-Ching Chung
ARMVISOR
1

2. Hypervisor on ARM architecture
ARMVISOR
2

3. KVM
 Linux Kernel module
 Free, open
 Many developers maintain it!!!
 QEMU
3

4. System Architecture
Guest
QEMU
Device
Driver Linux KVM
Hardware
4

5. Trap & emulate
Guest OS
User space trap
Kernel space
VMM
5

6. Challenges
 ARM is non-virtualizable.
 Sensitive instructions
 Privileged instructions
6

7. Non-virtualizable
2531 sensitive
sensitive but
25 critical
non-privileged
instructions
instructions
instructions
6 privileged
instructions
7

8. Challenges
 ARM is non-virtualizable
 Critical instructions
 Possible solutions
 Hardware support
 Dynamic binary translation (DBT)
 Para-virtualization
8

9. Implementation
 CPU virtualization
 Memory virtualization
 IO virtualization
 Optimization
9

10. User space Kernel space Guest Mode
1. VM initialization
2. Return to QEMU
3. Run VM
4. Enter Guest
5. Exit Guest
Lightweight trap
Enter Guest
Heavyweight trap
Exit Guest
Return to QEMU
Run VM
Enter Guest
QEMU KVM Guest OS

11. CPU Virtualization
11

12. CPU Virtualization
 ARM: non-virtualizable ISA
 Para-virtualization
 Manually insert software interrupts (SWI) into
guest OS
12

13. Para-virtualization
…
mov r0, r0
add sp, sp
movs pc, lr
…
13

14. Para-virtualization (cont.)
.macro virt_svc_movs, inst
SWI 0x190
… inst
mov r0, r0
.endm
add sp, sp
virt_svc_movs “movs pc, lr”
…
14

15. UND ABORT SWI IRQ/FIQ
KVM Trap Entry
KVM/Guest
Host Trap Handler Context Switch
Unit
KVM Trap Dispatcher
Instruction MMU Exception/Interrupt QEMU I/O
Emulation Emulation Emulation Emulation
15

16. KVM Vector
oxffff1000
0x1C FIQ
0x18 IRQ
0x14 (Reserved*)
0xffff001c 0x10 Data Abort
Kernel 0x0C Prefetch Abort
Vector 0x08 Software Interrupt
oxffff0000 0x04 Undefined Instruction
0x00 Reset
16

17. KVM Vector
The KVM trap
Interface
oxffff1000
0xffff001c
KVM
oxffff0000 Vector
17

18. Memory Virtualization
18

19. GVA
Guest
PTB
GPA
19

20. Shadow Page Tables
 Map guest virtual address to host physical
address
 For each guest page table (GPT), ARMvisor
will allocate a shadow page table (SPT) to
map it.
20

21. Guest
PTB GVA
GPA
Host
HVA PTB
New SPTE !!!
HPA
21

22. Emulation Flow
PABT/DABT
Hidden protection fault
trap
guest Guest MMIO Shadow Shadow
page table permission access page table page table
walker checker checker mapping update
True permission fault Hidden translation fault
True Translation fault MMIO emulation
22

23. Shadow Page Tables
 Map guest virtual address to host physical
address
 For each guest page table (GPT), ARMvisor
will allocate a shadow page table (SPT) to
map it.
 How to keep coherence between SPT and
GPT?
23

24. Synchronization
 Write protect page table page
 Modification would cause a protection fault.
 Reverse map (RMAP) : Record reverse mapping
form guest physical page to SPT entries
25

25. Permission Emulation (1)
 Guest OS in non-privileged mode
 Some instructions access kernel space with
user permission
 LDRBT, LDRT, STRBT, STRT
 Double shadow page table
26

26. Permission Mapping
 Permission mechanism on ARM
 Permission bits
 Domain
 No access
 Client
 Manager
28

27. Using Domain Mechanism
GUD GKD
Virtual User Space CLIENT NA
Virtual Kernel Space CLIENT CLIENT
29

28. Optimizations
30

29. Virtualization Overhead -CPU
 CPU virtualization
 Frequent lightweight traps result lots of context
switch
 Try to reduce…
 number of traps
 Overhead of emulation
31

30. CPU Optimization
 Shadow file register (SFR)
 Map VCPU’s shadow state of the register file into
memory region that is both accessible for the
VMM and guest with RW permission.
33

31. mcr cpsr, r1
oxffff2000
VCPU
Register Sync
File Shadow Read/Write
Register File Instructions
KVM/Guest
Context Switch
Interface
oxffff1000
KVM vector
oxffff0000 34

32. CPU Optimization
 Shadow file register (SFR)
 Map VCPU’s shadow state of the register file into
memory region that is both accessible for the
VMM and guest with RW permission.
 Para-virtualization: Fast instruction trap
 Sets of pre-defined macros which is composed of
encoded information of the replaced instructions.
36

33. Virtualization Overhead –MEM
 Memory virtualization
 Synchronization model
 Try to reduce…
 Protection faults
37

34. Memory Optimization
 Para-virtualization: hyper calls for PT
modifications
 When Guest OS sets PTEs
 When Guest OS is going to free a L2 page table
38

35. Implementation Status
39

36. Experiment Board
 ARMvisor supports ARMv6 & ARMv7
architecture in host
ARM v6 11mpcore ARM v7 cortex-a8
40

37. Environment
 Host OS: Linux 2.6.35 Ubuntu
 Guest OS: Linux 2.6.31
 LMBench
41

38. Experiment Data
100
80
60
40
20
0
base cpu opt mem opt all opt
42

39. 16
Experiment Data
15.20
14 12.46 12.17
12 11.24 10.81
9.37 9.94
10 8.45 7.97 8.04
8 7.18
5.78 5.66
6 4.63
4.39 4.41
4
1.89 1.43
2 1.23
0
all opt
43

40. Related Work
 VMware MVP
 Xen-ARM
 B LABS CODEZEOR
 OK Labs OKL4
 Virtual Open System
44

41. Conclusion
 ARMvisor supports ARMv6 & ARMv7
architecture in host
 Support RealView EB and Beagle Board
 CPU and memory optimization gain lots of
performance improvement
 Reduce traps
 Decrease protection faults
45

42. Future Work
 I/O optimization
 Support ARM hardware virtualization
extensions
 Multi-VM
 Multi-core
 …
46

43. Contact Us
 SSLAB, NTHU, Taiwan
 Website
 https://sites.google.com/a/sslab.cs.nthu.edu.tw/ar
mvisor/
 Thanks For Your Attention
47

44. IO Virtualization Overhead
Guest
Driver
QEMU
Device
Driver Linux KVM
Hardware
49