1. ARMvisor
Peter Chang
This side is licensed under CC-BY-NC-SA
姓名標示─非商業性─相同方式分享
(http://creativecommons.org/licenses/by-nc-sa/3.0/tw/legalcode)

2. Who am I?

3. Who am I?
• Peter Chang

4. Who am I?
• Peter Chang
• One of the current developers of ARMvisor

5. Who am I?
• Peter Chang
• One of the current developers of ARMvisor
• http://tw.linkedin.com/in/peterchangtw

6. What is ARMvisor?

8. ARM

9. ARM

10. ARM

11. What is KVM?

12. What is KVM?
• A.k.a. Kernel-based Virtual Machine

13. What is KVM?
• A.k.a. Kernel-based Virtual Machine
• Type-II Virtual Machine Monitor

14. What is KVM?
• A.k.a. Kernel-based Virtual Machine
• Type-II Virtual Machine Monitor
• A module of Linux kernel

15. What is KVM?

16. What is KVM?
• Officially support x86/x64, PowerPC, S390

17. What is KVM?
• Officially support x86/x64, PowerPC, S390
• No official support for ARM architecture

18. What is ARMvisor?

19. What is ARMvisor?
• ARM架構上的KVM

20. What is ARMvisor?
• ARM架構上的KVM
• Para-virtualization

21. What is ARMvisor?
• ARM架構上的KVM
• Para-virtualization
• Trap & Emulation

22. What is ARMvisor?
• ARM架構上的KVM
• Para-virtualization
• Trap & Emulation
• Dynamic Memory Allocation

23. What is ARMvisor?
• ARM架構上的KVM
• Para-virtualization
• Trap & Emulation
• Dynamic Memory Allocation
• virtio & IRQchip-in-kernel

24. Guest OS: Linux 2.6.35
QEMU 0.14 Driver
Device
Driver ARMvisor
Host OS: Linux 2.6.38
Hardware: ARM Cortex-A8

25. 2012
2011
2010
2009

26. 2012
2011
2010
2009

27. 2012
2011
2010
Starting Point
2009

28. 2012
2011
2010
Starting Point
2009

29. 2012
2011
ARMvisor Prototype
2010
Starting Point
2009

30. 2012
2011
ARMvisor Prototype
2010
Starting Point
2009

31. 2012
CPU Opt & Mem Opt
2011
ARMvisor Prototype
2010
Starting Point
2009

32. 2012
CPU Opt & Mem Opt
2011
ARMvisor Prototype
2010
Starting Point
2009

33. I/O Opt
2012
CPU Opt & Mem Opt
2011
ARMvisor Prototype
2010
Starting Point
2009

34. Supported Hardware
ARM Realview-eb
ARM11
ARMv6 ISA

35. Supported Hardware
TI BeagleBoard
Cortex-A8
ARMv7 ISA

36. Supported Software
• Patched Host OS:
• Linaro Linux 2.6.38
• Host Root Filesystem
• Ubuntu/Debian RFS (CLI or GUI)
• QEMU 0.14

37. Supported Software
• Patched Guest OS:
• Linux 2.6.35 (running on ARMv6 ISA)
• Guest Root Filesystem:
• Ubuntu/Debian RFS (CLI or GUI)

38. System model of ARMvisor

39. CPU virtualization

40. CPU virtualization
• ARM is non-virtualizable CPU
• Patch guest OS
• “Trap and emulation”

41. ARMv6 ISA
1.Branch instructions
2.Data-processing instructions
3.Multiply instructions
4.Parallel addition and subtraction instructions
5.Extend instructions
6.Miscellaneous arithmetic instructions
7.Other miscellaneous instructions
8.Status register access instructions
9.Load and store instructions
10.Load and Store Multiple instructions
11.Semaphore instructions
12.Exception-generating instructions
13.Coprocessor instructions

42. ARMv6 ISA
1.Branch instructions
Sensitive
2.Data-processing instructions Instructions ?
3.Multiply instructions
4.Parallel addition and subtraction instructions
5.Extend instructions
6.Miscellaneous arithmetic instructions
7.Other miscellaneous instructions
8.Status register access instructions
9.Load and store instructions
10.Load and Store Multiple instructions
11.Semaphore instructions
12.Exception-generating instructions
13.Coprocessor instructions

43. ARMv6 ISA
1.Branch instructions
2.Data-processing instructions
3.Multiply instructions
4.Parallel addition and subtraction instructions
5.Extend instructions
6.Miscellaneous arithmetic instructions
7.Other miscellaneous instructions
8.Status register access instructions
9.Load and store instructions
10.Load and Store Multiple instructions
11.Semaphore instructions
12.Exception-generating instructions
13.Coprocessor instructions

44. Sensitive Instructions
Data-processing instructions S-BIT: MOVS, ...
Status register access instructions MRS, MSR, CPS, SETEND
Load and store instructions T-BIT: LDRT, STRT, ...
Load and Store Multiple
instructions LDM(2), LDM(3), STM(2)
Exception-generating instructions SWI, BKPT
Coprocessor instructions MCR, MRC, MCRR, ...

45. Observation from
Guest Linux Code
• MOVS
• (MRS, MSR, CPS)
• (LDRBT, LDRT, STRBT, STRT)
• (LDM(2), LDM(3), STM(2))
• SWI
• (MCR, MRC, MCRR)
15 sensitive instructions
used in the guest linux code

46. “Trap & emulation”
Guest%OS
User%space% trap
%
Kernel%space
VMM

47. How to “trap”?
…
mov r0, r0
add sp, sp
movs pc, lr
…

48. How to “trap”?
…
mov r0, r0
add sp, sp
virt_svc_movs “movs pc, lr”
…

49. How to “trap”?
…
mov r0, r0
add sp, sp
virt_svc_movs “movs pc, lr”
…

50. How to “trap”?
.macro virt_svc_movs, inst
SWI 0x190
inst
… .endm
mov r0, r0
add sp, sp
virt_svc_movs “movs pc, lr”
…

51. How to “emulate”?

52. oxffff1000
0xffff001c
Kernel Vector
oxffff0000

53. oxffff1000
0xffff001c
Kernel Vector
oxffff0000

54. The KVM trap
Interface
oxffff1000
0xffff001c
Kernel Vector
oxffff0000

55. UND ABORT SWI IRQ/FIQ
KVM
Trap
Entry
KVM/Guest
Host
Trap
Handler
Context
Switch
Unit
KVM
Trap
Dispatcher
Instruction
MMU
Exception/Interrupt
QEMU
I/O
Emulation Emulation Emulation Emulation

56. User space Kernel space Guest Mode
QEMU KVM Guest OS

57. User space Kernel space Guest Mode
1. VM initialization
QEMU KVM Guest OS

58. User space Kernel space Guest Mode
1. VM initialization
2. Return to QEMU
QEMU KVM Guest OS

59. User space Kernel space Guest Mode
1. VM initialization
2. Return to QEMU
3. Run VM
QEMU KVM Guest OS

60. User space Kernel space Guest Mode
1. VM initialization
2. Return to QEMU
3. Run VM
4. Enter Guest
QEMU KVM Guest OS

61. User space Kernel space Guest Mode
1. VM initialization
2. Return to QEMU
3. Run VM
4. Enter Guest
5. Exit Guest
QEMU KVM Guest OS

62. User space Kernel space Guest Mode
1. VM initialization
2. Return to QEMU
3. Run VM
4. Enter Guest
5. Exit Guest
Lightweight trap
QEMU KVM Guest OS

63. User space Kernel space Guest Mode
1. VM initialization
2. Return to QEMU
3. Run VM
4. Enter Guest
5. Exit Guest
Lightweight trap
6. Enter Guest
QEMU KVM Guest OS

64. User space Kernel space Guest Mode
1. VM initialization
2. Return to QEMU
3. Run VM
4. Enter Guest
5. Exit Guest
Lightweight trap
6. Enter Guest
7. Exit Guest
QEMU KVM Guest OS

65. User space Kernel space Guest Mode
1. VM initialization
2. Return to QEMU
3. Run VM
4. Enter Guest
5. Exit Guest
Lightweight trap
6. Enter Guest
7. Exit Guest
8. Return to QEMU
QEMU KVM Guest OS

66. User space Kernel space Guest Mode
1. VM initialization
2. Return to QEMU
3. Run VM
4. Enter Guest
5. Exit Guest
Lightweight trap
6. Enter Guest
Heavyweight trap
7. Exit Guest
8. Return to QEMU
QEMU KVM Guest OS

67. User space Kernel space Guest Mode
1. VM initialization
2. Return to QEMU
3. Run VM
4. Enter Guest
5. Exit Guest
Lightweight trap
6. Enter Guest
Heavyweight trap
7. Exit Guest
8. Return to QEMU
9. Run VM
QEMU KVM Guest OS

68. User space Kernel space Guest Mode
1. VM initialization
2. Return to QEMU
3. Run VM
4. Enter Guest
5. Exit Guest
Lightweight trap
6. Enter Guest
Heavyweight trap
7. Exit Guest
8. Return to QEMU
9. Run VM
10. Enter Guest
QEMU KVM Guest OS

69. VCPU
oxffff2000
Register
Sync
File
The KVM trap
Interface
oxffff1000
0xffff001c
Kernel Vector
oxffff0000

70. VCPU
oxffff2000
Register
Shadow
Register
File
Sync
File
The KVM trap
Interface
oxffff1000
0xffff001c
Kernel Vector
oxffff0000

71. mcr
cpsr,
r1
VCPU
oxffff2000
Register
Shadow
Register
File
Sync
File
The KVM trap
Interface
oxffff1000
0xffff001c
Kernel Vector
oxffff0000

72. mcr
cpsr,
r1
VCPU
oxffff2000
Register
Shadow
Register
File
Sync
File
The KVM trap
Interface
oxffff1000
0xffff001c
Kernel Vector
oxffff0000

73. mcr
cpsr,
r1
VCPU
oxffff2000
Read/Write
Register
Shadow
Register
File
Sync Instructions
File
The KVM trap
Interface
oxffff1000
0xffff001c
Kernel Vector
oxffff0000

74. mcr
cpsr,
r1
VCPU
oxffff2000
Read/Write
Register
Shadow
Register
File
Sync Instructions
File
The KVM trap
Interface
oxffff1000
0xffff001c
Kernel Vector
oxffff0000

75. Memory virtualization

76. PTB Virtual
Address
Physical
Address

77. GVA
GPA
HVA
HPA

78. Guest
PTB
GVA
GPA
HVA
HPA

79. Guest
PTB
GVA
GPA
HVA
HPA

80. Guest
PTB
GVA
GPA
Host
PTB
HVA
HPA

81. Guest
PTB
GVA
GPA
Host
PTB
HVA
New
SPTE
!!!
HPA

83. PABT/DABT
trap

84. PABT/DABT
trap
guest
page
table
walker

85. PABT/DABT
trap
guest
page
table
walker
True
Translation
fault

86. PABT/DABT
trap
guest
Guest
page
table
permission
walker checker
True
Translation
fault

87. PABT/DABT
trap
guest
Guest
page
table
permission
walker checker
True
permission
fault
True
Translation
fault

88. PABT/DABT
trap
guest
Guest
MMIO
access
page
table
permission
checker
walker checker
True
permission
fault
True
Translation
fault

89. PABT/DABT
trap
guest
Guest
MMIO
access
page
table
permission
checker
walker checker
True
permission
fault
True
Translation
fault MMIO
emulation

90. PABT/DABT
trap
guest
Guest
Shadow
page
MMIO
access
page
table
permission
table
checker
walker checker mapping
True
permission
fault
True
Translation
fault MMIO
emulation

91. Hidden
protection
fault
PABT/DABT
trap
guest
Guest
Shadow
page
MMIO
access
page
table
permission
table
checker
walker checker mapping
True
permission
fault
True
Translation
fault MMIO
emulation

92. Hidden
protection
fault
PABT/DABT
trap
guest
Guest
Shadow
page
MMIO
access
page
table
permission
table
checker
walker checker mapping
True
permission
fault Hidden
translation
fault
True
Translation
fault MMIO
emulation

93. Hidden
protection
fault
PABT/DABT
trap
guest
Guest
Shadow
page
MMIO
access
Shadow
page
page
table
permission
table
checker table
update
walker checker mapping
True
permission
fault Hidden
translation
fault
True
Translation
fault MMIO
emulation

94. I/O virtualization

95. I/O virtualization
• Emulation by QEMU
• virtio
• IRQ chip in kernel

96. Emulate by QEMU

97. Guest OS: Linux 2.6.35
QEMU 0.14 Driver
Device
ARMvisor
Driver Host OS: Linux 2.6.38
Hardware: ARM Cortex-A8

98. Guest OS: Linux 2.6.35
1
QEMU 0.14 Driver
Device
ARMvisor
Driver Host OS: Linux 2.6.38
Hardware: ARM Cortex-A8

99. Guest OS: Linux 2.6.35
1
QEMU 0.14 Driver
Device
2
ARMvisor
Driver Host OS: Linux 2.6.38
Hardware: ARM Cortex-A8

100. Guest OS: Linux 2.6.35
1
QEMU 0.14 Driver
Device
3 2
ARMvisor
Driver Host OS: Linux 2.6.38
Hardware: ARM Cortex-A8

101. Guest OS: Linux 2.6.35
1
QEMU 0.14 Driver
Device
3 2
ARMvisor
Driver Host OS: Linux 2.6.38
4
Hardware: ARM Cortex-A8

102. Guest OS: Linux 2.6.35
QEMU 0.14 Driver
Device
ARMvisor
Driver Host OS: Linux 2.6.38
Hardware: ARM Cortex-A8

103. Guest OS: Linux 2.6.35
QEMU 0.14 Driver
Device
ARMvisor
Driver Host OS: Linux 2.6.38
5
Hardware: ARM Cortex-A8

104. Guest OS: Linux 2.6.35
QEMU 0.14 Driver
Device
6
ARMvisor
Driver Host OS: Linux 2.6.38
5
Hardware: ARM Cortex-A8

105. Guest OS: Linux 2.6.35
QEMU 0.14 Driver
Device
7
6
ARMvisor
Driver Host OS: Linux 2.6.38
5
Hardware: ARM Cortex-A8

106. Guest OS: Linux 2.6.35
8
QEMU 0.14 Driver
Device
7
6
ARMvisor
Driver Host OS: Linux 2.6.38
5
Hardware: ARM Cortex-A8

107. virtio

108. virtio
Vir,o
Driver
Guest
Vir,o
AMBA
Controller
Vring Transport
Vir,o
AMBA
Controller
QEMU
Vir,o
Device

109. irq_chip in kernel

110. Opera,ng
System Interrupt
Controller
Deliver
Interrupt
Get
IRQ
number
1
Ack
IRQ
number
Mask
IRQ
2
End
of
IRQ
3
Unmask
IRQ
4

111. irq_chip in kernel
Guest
Deliver7
GIC QEMU IRQ7
Control7
GIC7
Device Device Device
IOCTL7
ARMvisor

112. irq_chip in kernel
Guest
Deliver$
IRQ$
QEMU
Device Device Device Control$
GIC$
GIC$in$Kernel KVM

113. Future Works

114. Future Works

115. Future Works
• Support for ARM’s Virtualization extension

116. Future Works
• Support for ARM’s Virtualization extension
• Cortex-A15 and beyond

117. Future Works
• Support for ARM’s Virtualization extension
• Cortex-A15 and beyond
• SMP for host and guest

118. Future Works
• Support for ARM’s Virtualization extension
• Cortex-A15 and beyond
• SMP for host and guest
• AArch64 Support

119. Future Works
• Support for ARM’s Virtualization extension
• Cortex-A15 and beyond
• SMP for host and guest
• AArch64 Support
• ARMv8

120. By the way, ...

121. OpenSource

122. OpenSource
• We HAVE opened source in late August,
2012.

123. OpenSource
• We HAVE opened source in late August,
2012.
• GNU GPLv2

124. OpenSource
• We HAVE opened source in late August,
2012.
• GNU GPLv2
• Source code of Host and Guest OS

125. OpenSource
• We HAVE opened source in late August,
2012.
• GNU GPLv2
• Source code of Host and Guest OS
• https://github.com/SSLab-NTHU

126. Q &A