Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Nested Virtualization with OpenNebula
(and PCI Passthrough)
A Practical Approach
Venko Moyankov
OpenNebulaConf 2019
Barcel...
The Problem
KVM KVM KVM
KVM KVM KVM
KVM KVM KVM
KVM KVM KVM
H-V H-V H-V
XEN XEN XEN
KVM
KVM
StorPool Lab
KVM KVM KVM
KVM KVM KVM
KVM KVM KVM
KVM KVM KVM
also and more ...
H-V H-V H-V
XEN XEN XEN
The Technologies Behind
● Hardware virtualization (VT-x)
● Nested Virtualization
● VMCS Shadowing
● IOMMU (PCI Passthrough...
Hardware Virtualization
VT-x
AMD-V or SVM
All CPUs, but may need to enable it in BIOS
Nested Virtualization
Enables hardware virtualization in the guest
KVM feature
VMCS Shadowing
● Hardware feature
● Accelerates nested virtualization
● Available in most CPUs since 2013 (Haswell)
Source...
PCI Passthrough
Allows guests to have direct
exclusive access to PCI devices
● I/O MMU virtualization (IOMMU)
VT-d
AMD-Vi
...
SR-IOV
● Single Root I/O Virtualization
● One physical device appears as
multiple virtual functions (VF)
● Allows differen...
ACS and IOMMU Groups
Host
GPU1
Hypervisor / VMM
VM
Host
driver
driver
VMVM
driver
VM
driver
GPU2 GPU3GPU0
IOMMU / VT-dRAM
...
Let's Do It
Host
Hardware Virtualization
Check it is available with: lscpu | grep vmx
Nested Virtualization
kvm-intel.conf: options kv...
PCI Passthrough & SR-IOV
PCI Passthrough
Enabled via kernel options:
Intel:
intel_iommu=on iommu=pt
AMD:
amd_iommu=pt
SR-I...
ACS and IOMMU Groups
● Check ACS is supported
lspci -vv | egrep “Access Control Services”
https://heiko-sieger.info/iommu-...
Congratulations!
You are ready to run
Nested Virtualization !
Almost
Enable PCI passthrough in OpenNebula
/var/lib/one/remotes/im/kvm-probes.d/pci.rb
/var/lib/one/remotes/etc/im/kvm-probes.d/...
Tweak domain.xml
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0xd8' slot='0x...
OpenNebula
Summary
CPU BIOS Kernel KVM OpenNebula
VT-x 🗸 🗸
Nested Virt. 🗸
VMCS Shadowing 🗸 🗸 🗸
IOMMU (PCI Passthrough) 🗸 🗸 🗸 🗸
SR-IOV...
Q&A
Venko Moyankov
venko@storpool.com
StorPool Storage
www.storpool.com
@storpool
Thank you!
OpenNebulaConf2019 - Nested Virtualization with OpenNebula: A Practical Approach - Venko Moyankov - StorPool
Prochain SlideShare
Chargement dans…5
×

OpenNebulaConf2019 - Nested Virtualization with OpenNebula: A Practical Approach - Venko Moyankov - StorPool

49 vues

Publié le

In this talk, StorPool will present the concept of nested virtualization and PCI pass-through. Nested virtualization is when you have virtual machines running inside other virtual machines. If done right, nested virtualization is extremely suitable for testing large virtualized environments, without the need for having hundreds of physical servers.

This approach empowers system administrators by reducing the time and effort needed to run/test multiple dynamic environments. This is done by having easier automation/provisioning, API control and much more. In a nested virtualization solution, the physical hardware is sliced and diced fast and easy, which gives you a large set of opportunities to experiment without the need for huge investments in physical hardware or time and effort needed to reinstall/set-up software.

Join this talk to learn more about why and how to do nested virtualization with OpenNebula and achieve an inception moment of having virtual machines, inside virtual machines, inside virtual machines…

Publié dans : Logiciels
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

OpenNebulaConf2019 - Nested Virtualization with OpenNebula: A Practical Approach - Venko Moyankov - StorPool

  1. 1. Nested Virtualization with OpenNebula (and PCI Passthrough) A Practical Approach Venko Moyankov OpenNebulaConf 2019 Barcelona, October 21-22
  2. 2. The Problem KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM H-V H-V H-V XEN XEN XEN
  3. 3. KVM KVM StorPool Lab KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM also and more ... H-V H-V H-V XEN XEN XEN
  4. 4. The Technologies Behind ● Hardware virtualization (VT-x) ● Nested Virtualization ● VMCS Shadowing ● IOMMU (PCI Passthrough) ● SR-IOV ● ACS (IOMMU groups) ● OpenNebula PCI Passthrough ● libvirt Domain XML tweaks (VF Net only) Host OpenNebula
  5. 5. Hardware Virtualization VT-x AMD-V or SVM All CPUs, but may need to enable it in BIOS
  6. 6. Nested Virtualization Enables hardware virtualization in the guest KVM feature
  7. 7. VMCS Shadowing ● Hardware feature ● Accelerates nested virtualization ● Available in most CPUs since 2013 (Haswell) Source: https://software.intel.com/en-us/blogs/2014/12/12/enabling-virtual-machine-control-structure-shadowing-on-a-nested-virtual-machine
  8. 8. PCI Passthrough Allows guests to have direct exclusive access to PCI devices ● I/O MMU virtualization (IOMMU) VT-d AMD-Vi Mostly used for GPU and NIC Host GPU1 Hypervisor / VMM VM Host driver driver VM driver VM driver GPU2 GPU3GPU0 PCIe IOMMU / VT-d
  9. 9. SR-IOV ● Single Root I/O Virtualization ● One physical device appears as multiple virtual functions (VF) ● Allows different VMs to share a single PCIe hardware ● Mostly used for NIC Host NIC VF1 Hypervisor / VMM VM Host driver driver VM driver VM driver VF2 VF3PF PCIe IOMMU / VT-d
  10. 10. ACS and IOMMU Groups Host GPU1 Hypervisor / VMM VM Host driver driver VMVM driver VM driver GPU2 GPU3GPU0 IOMMU / VT-dRAM ACS PCIe
  11. 11. Let's Do It
  12. 12. Host Hardware Virtualization Check it is available with: lscpu | grep vmx Nested Virtualization kvm-intel.conf: options kvm-intel nested=1 Check it in the guest with: lscpu | grep vmx VMCS Shadowing kvm-intel.conf: options kvm-intel enable_shadow_vmcs=1
  13. 13. PCI Passthrough & SR-IOV PCI Passthrough Enabled via kernel options: Intel: intel_iommu=on iommu=pt AMD: amd_iommu=pt SR-IOV ● Ensure SR-IOV and VT-d are enabled in BIOS. ● Setup VFs: # echo '8' > /sys/class/net/eth3/device/sriov_numvfs
  14. 14. ACS and IOMMU Groups ● Check ACS is supported lspci -vv | egrep “Access Control Services” https://heiko-sieger.info/iommu-groups -what-you-need-to-consider/ ● Check IOMMU groups for a in /sys/kernel/iommu_groups/*; do find $a -type l; done | sort --version-sort
  15. 15. Congratulations! You are ready to run Nested Virtualization ! Almost
  16. 16. Enable PCI passthrough in OpenNebula /var/lib/one/remotes/im/kvm-probes.d/pci.rb /var/lib/one/remotes/etc/im/kvm-probes.d/pci.conf :filter: '15b3:1018' :short_address: [] :device_name: []
  17. 17. Tweak domain.xml <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0xd8' slot='0x00' function='0x5'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0'/> </hostdev> <interface managed="yes" type="hostdev"> <driver name="vfio" /> <mac address="02:00:11:ab:cd:01" /> <source> <address bus="0xd8" domain="0x0000" function="0x5" slot="0x00" type="pci" /> </source> <address bus="0x01" domain="0x0000" function="0" slot="0x01" type="pci" /> </interface> https://github.com/OpenNebula/addon-storpool/blob/master/docs/ advanced_configuration.md#vms-domain-xml-tweaking
  18. 18. OpenNebula
  19. 19. Summary CPU BIOS Kernel KVM OpenNebula VT-x 🗸 🗸 Nested Virt. 🗸 VMCS Shadowing 🗸 🗸 🗸 IOMMU (PCI Passthrough) 🗸 🗸 🗸 🗸 SR-IOV 🗸 🗸 ACS (IOMMU groups) 🗸 libvirt Domain XML (VF Net only) 🗸
  20. 20. Q&A
  21. 21. Venko Moyankov venko@storpool.com StorPool Storage www.storpool.com @storpool Thank you!

×