Hypervisor Security and steps that must be taken to protect against breakouts Video here: https://www.youtube.com/watch?v=y8L6B6Q5EdI

1. Robert Clark
Lead Security Architect
HP Cloud
Hypervisor Security

3. About the Speaker

4. OpenStack Security Group
• Established 18-24 months ago
• Issues OpenStack Security Notes
• Consults on OpenStack Security Advisories
• Security Initiatives
• Nearly 100 members

5. OpenStack Security Guide
http://docs.openstack.org/security

6. OpenStack Security Guide

7. Virtualization
Overview

8. Virtualization Technologies
• Hosted OS Virtualization – VMware Desktop
Solutions
• Para Virtualization – The guest needs to know
it’s running in a virtualized environment
• Full Virtualization – The guest is un-aware
that it is running on a virtualized platform.

9. Virtualization Stack
Compute Host
Alice
VM
Alice
VM
Alice
VM
Hardware
Hypervisor
Device Emulation

10. Simplified KVM
Compute Host
Alice
VM
Alice
VM
Alice
VM
Hardware CPU VIRT
Linux Kernel
KVM
QEMU
Linux OS

11. Simplified Xen
Compute Host
Dom0 Alice
VM
Hardware
Xen Hypervisor
Alice
VM
QEMU

12. Generalized Virtualization Stack
Compute Host
Alice
VM
Alice
VM
Alice
VM
Hardware
Hypervisor / Host OS / Dom0
QEMU
Compute Instances
Device Emulation
/ Paravirt
Hardware Interfacing
/ Enabling
Hardware
Memory, Disk, CPU etc

13. Attack
Vectors

14. Introducing ‘Mal’
Mal
VM

15. Compute Host Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM

16. Compute Instance Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM
Bob
VM
Bob
VM

17. Compute Instance Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM
Bob
VM
Bob
VM
KVM / XEN
QEMU

18. Compute Instance Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM
Bob
VM
Bob
VM
KVM / XEN
QEMU
Dom0
Linux Kernel
Linux OS

19. Compute Instance Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM
Bob
VM
Bob
VM
Mal
VM KVM / XEN
QEMU
Dom0
Linux Kernel
Linux OS

20. Compute Instance Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM
Bob
VM
Bob
VM
Mal
VM
QEMU
Basic VM to VM network Attacks
KVM / XEN
QEMU
Dom0
Linux Kernel
Linux OS

21. Compute Instance Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM
Bob
VM
Bob
VM
Mal
VM KVM / XEN
QEMU
VM to hypervisor attacks
KVM / XEN
QEMU
Dom0
Linux Kernel
Linux OS

22. Compute Instance Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM
Bob
VM
Bob
VM
Mal
VM KVM / XEN
QEMU
VM to QEMU / Device attacks
KVM / XEN
QEMU
Dom0
Linux Kernel
Linux OS

23. Dom0
Compute Instance Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM
Bob
VM
Bob
VM
Mal
VM
QEMU
Linux Kernel
Linux OS
VM to QEMU
KVM / XEN
QEMU

24. Dom0
Compute Instance Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM
Bob
VM
Bob
VM
Mal
VM
QEMU
Linux Kernel
Linux OS
VM to QEMU
KVM / XEN
QEMU

25. Dom0
Compute Instance Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM
Bob
VM
Bob
VM
Mal
VM
QEMU
Linux Kernel
Linux OS
1.
VM to QEMU
KVM / XEN
QEMU
2.

26. Dom0
Compute Instance Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM
Bob
VM
Bob
VM
Mal
VM
QEMU
Linux Kernel
Linux OS
1.
VM to QEMU
KVM / XEN
QEMU
2.

27. Dom0
Compute Instance Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM
Bob
VM
Bob
VM
Mal
VM
QEMU
Linux Kernel
Linux OS
VM to hypervisor attacks
KVM / XEN
QEMU

28. Dom0
Compute Instance Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM
Bob
VM
Bob
VM
Mal
VM
QEMU
Linux Kernel
Linux OS
VM to hypervisor attacks
KVM / XEN
QEMU

29. Dom0
Compute Instance Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM
Bob
VM
Bob
VM
Mal
VM
QEMU
Linux Kernel
Linux OS
VM to hypervisor attacks
KVM / XEN
QEMU

30. Dom0
Compute Instance Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM
Bob
VM
Bob
VM
Mal
VM
QEMU
Linux Kernel
Linux OS
VM to OS / Management / Linux Kernel / Dom0
KVM / XEN
QEMU

31. Dom0
Compute Instance Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM
Bob
VM
Bob
VM
Mal
VM
QEMU
Linux Kernel
Linux OS
VM to OS / Management / Linux Kernel / Dom0
KVM / XEN
QEMU

32. Dom0
Compute Instance Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM
Bob
VM
Bob
VM
Mal
VM
QEMU
Linux Kernel
Linux OS
1.
2.
VM to OS / Management / Linux Kernel / Dom0
KVM / XEN
QEMU

33. Dom0
Compute Instance Attack Vectors
Compute Host [Nova]
Alice
VM
Alice
VM
Bob
VM
Bob
VM
Mal
VM
QEMU
Linux Kernel
Linux OS
1.
2.
3.
VM to OS / Management / Linux Kernel / Dom0
KVM / XEN
QEMU

34. Cloud Issues
Compute Host [Nova]
Alice
VM
Bob
VM

35. Cloud Issues - Scale
Compute Host [Nova]
Cher
VM
Dave
VM
Compute Host [Nova]
Alice
VM
Bob
VM

36. Cloud Issues - Scale
Compute Host [Nova]
Cher
VM
Dave
VM
Compute Host [Nova]
Alice
VM
Bob
VM
Compute Manager
Block Storage
Network Nodes
Operations Systems
Object Storage

37. Cloud Issues - Scale
Compute Host [Nova]
Cher
VM
Dave
VM
Compute Host [Nova]
Alice
VM
Bob
VM
Compute Manager
Block Storage
Network Nodes
Operations Systems
Object Storage
Mal
VM

38. Cloud Issues – Flat Exploitation
Compute Host [Nova]
Cher
VM
Dave
VM
Compute Host [Nova]
Alice
VM
Bob
VM
Compute Manager
Block Storage
Network Nodes
Operations Systems
Object Storage
Mal
VM

39. Cloud Issues – Flat Exploitation
Compute Host [Nova]
Cher
VM
Dave
VM
Compute Host [Nova]
Alice
VM
Bob
VM
Compute Manager
Block Storage
Network Nodes
Operations Systems
Object Storage
Mal
VM

40. Cloud Issues – Service Trust
Compute Host [Nova]
Cher
VM
Dave
VM
Compute Host [Nova]
Alice
VM
Bob
VM
Compute Manager
Block Storage
Network Nodes
Operations Systems
Object Storage
Mal
VM

41. Cloud Issues – Service Trust
Compute Host [Nova]
Cher
VM
Dave
VM
Compute Host [Nova]
Alice
VM
Bob
VM
Compute Manager
Block Storage
Network Nodes
Operations Systems
Object Storage
Mal
VM

42. Cloud Issues – Nova RPC
Compute Host [Nova]
Cher
VM
Dave
VM
Compute Host [Nova]
Alice
VM
Bob
VM
Compute Manager
Block Storage
Network Nodes
Operations Systems
Object Storage
Mal
VM

43. What about
side channels?

44. Cross-VM Side Channel Attacks
• Web Servers providing SSL
• VOIP providers
• Cloud VPN
• Chat Applications
• Secure File Storage
• Virtually any service doing anything useful

45. Cross-VM Side Channel Attacks
Alice
Client
Compute Host [Nova]
Bob
VM
TLS/SSL
CPU
L1 Cache
• Disrupting or observing system operation

46. Cross-VM Side Channel Attacks
Alice
Client
Compute Host [Nova]
Bob
VM
TLS/SSL
Stealing the bits!
Mal
MITM
CPU
L1 Cache

47. Cross-VM Side Channel Attacks
Alice
Client
Compute Host [Nova]
Bob
VM
Mal
VM
TLS/SSL
Mal
MITM
CPU
L1 Cache
Stealing the bits!

48. Cross-VM Side Channel Attacks
Alice
Client
Compute Host [Nova]
Bob
VM
Mal
VM
TLS/SSL
Mal
MITM
CPU
L1 Cache
Stealing the bits!

49. Cross-VM Side Channel Attacks
Alice
Client
Compute Host [Nova]
Bob
VM
Mal
VM
TLS/SSL
Mal
MITM
CPU
L1 Cache
Stealing the bits!

50. Isn’t this all a bit
theoretical?

51. CloudBurst
• Date: 2008
• Type: OS Virtualization - VMWare
• Result: Full Breakout
• Author: Kostya Kirtchinsky, Immunity Inc

52. Xen Ownage Trilogy
• Date: 2011
• Type: Xen
• Result: Full Breakout
• Author: Joanna Rutkowska

53. VirtuNoid
• Date: 2011
• Type: Kernel Side Full Virtualization - KVM
• Result: Full Breakout
• Author: Nelson Elhage
• CVE-2011-1751

54. SYSRET-64
• Date: 2012
• Type: Para Virtualization - Xen
• Result: Full Breakout
• Author: Rafal Wojtczuk
• US-CERT #649219

55. VMDK Has Left The Building
• Date: 2012
• Type: ESXi File Handling Logic
• Result: Data Leakage / Loss
• Author: Friedwart Kuhn

56. KVM IOAPIC, SET MSR, TIME
• Date: 2013
• Type: Full Virtualization - KVM
• Result: Denial of Service, Potential Breakout
• Author: Andrew Honig
• IOAPIC: CVE-2013-1798
• TIME: CVE-2013-1797
• SET MSR: CVE-2013-1796

57. Virtualization Security Trends
IBM X-Force 2010 Mid-Term Report

58. Virtualization Security Trends
Attack Vector Xen KVM
Virtual CPUs 5 (8.5%) 8 (21.1%)
SMP 1 (1.7%) 3 (7.9%)
Software MMU 4 (6.8%) 2 (5.3%)
Interrupt and Timer Mechanisms 2 (3.4%) 4 (10.5%)
I/O and Networking 11 (18.6%) 10 (26.3%)
VM Exits 4 (6.8%) 2 (5.3%)
Hypercalls 2 (3.4%) 1 (2.6%)
VM Management 7 (11.9%) 2 (5.3%)
Remote Management Software 9 (15.3%) 1 (2.6%)
Hypervisor add-ons 5 (8.5%) 0 (0.0%)
TOTAL 59 38

59. Time to unplug?
Go home cloud, you’re drunk!

60. Protections – Compiler Hardening
• RELocation Read-Only
• Stack Canaries
• Never eXecute (NX) / (DEP)
• Position Independent Executable
• Address Space Layout Randomization
• QEMU:
CFLAGS="-arch x86_64 -fstack-protector-all -Wstack-protector --param
ssp-buffer-size=4 -pie -fPIE -ftrapv -D_FORTIFY_SOURCE=2 O2 -Wl,-z,relro,-
z,now"

61. Protections – Reduce Attack Surface
• Out of the box you probably support
– 3D Graphics
– Multiple Network Devices
– Sound
– Bluetooth!?
• Compile them out!

62. Protections – Mandatory Access
Controls
• Limit the capabilities of a successful exploit
• Define and constrain with QEMU should be
doing
• Provide isolation for VM processes (KVM)
• SELinux
• AppArmour

63. Protections – Mandatory Access
Controls

64. Protection
• Reduce Attack Surface
• Harden Compilation
• Isolate, detect and alert on exploitation
through MAC
• Harden your base OS/Dom0 using the same
techniques
• Apply MAC to other OpenStack components

65. OpenStack Security Guide
• http://docs.openstack.org/sec
• Chapter 26 – Securing OpenStack Networking
Services
• Chapter 40 – Hypervisor Selection
• Chapter 41 – Hardening the Virtualization
Layers
• Chapter 43 – Security Services for Instances

66. Thank You
Please consider contributing to the
OpenStack Security Group

67. References
• Directly Referenced / Informed This Talk
– http://www.insinuator.net/2013/05/analysis-of-hypervisor-breakouts/
– https://www.ernw.de/download/ERNW_DCVI-HypervisorsToClouds.pdf
– https://www.hashdays.ch/downloads/slides/jonathan_sinclair_vm_state.pdf
– ftp://public.dhe.ibm.com/linux/pdfs/LXW03004-USEN-00.pdf
– http://blog.cryptographyengineering.com/2012/10/attack-of-week-cross-vm-
timing-attacks.html
– http://www.vupen.com/blog/20120904.Advanced_Exploitation_of_Xen_Sysre
t_VM_Escape_CVE-2012-0217.php
– http://www.symantec.com/avcenter/reference/Virtual_Machine_Threats.pdf
– http://invisiblethingslab.com/resources/bh08/part1.pdf
– http://blogs.gartner.com/neil_macdonald/2011/01/26/yes-hypervisors-are-
vulnerable/
– ftp://public.dhe.ibm.com/common/ssi/ecm/en/wgl03003usen/WGL03003USE
N.PDF