Support for Open vSwitch with DPDK has been introduced on CloudStack version 4.12. In this talk we will go through the motivations and advantages of using DPDK on KVM hosts, high level architecture of the feature, what is needed and how can it be used on production environments and future works.

1. Open vSwitch and DPDK
support in CloudStack
NicolasVazquez
nicolas.Vazquez@shapeblue.com - @nicovazquez90
ApacheCon North America 2019 – LasVegas

2. About me
• Senior Software Engineer at ShapeBlue
• Design, implementation of new features
• 7+ years experience in Software Development
• Joined CloudStack community in 2015
• Became committer in 2016

3. Agenda
• Introduction
• OVS
• DPDK
• OVS-DPDK
• CloudStack high level support
• AdditionalVM configurations
• DPDK Support
• 4.12
• Agent configuraton
• DPDK enabledVMs creation
• 4.13
• DPDK vHost User mode selection
• DPDK enabled hosts detection
• Live migrations
• Conclusions

4. Introduction

5. Introduction - OVS
• Open source vSwitch solution, mostly used in SDN
• ConnectVMs on the same host or different hosts
• Operates in userspace
• Supported in KVM hypervisor by CloudStack
• Networking alternative to native bridges

6. Introduction – OVS-DPDK
• DPDK:
• Open source set of libraries for fast packet processing
• Operates in user space
• Integrated with OVS:
• Huge increase in packet throughput – 10x performance
over native OVS
• Lower latency

7. AdditionalVM
configurations

8. Additional VM configurations
• Pass additionalVM metadata configuration toVMs on start
• Introduced in version 4.12
• Supported by three major hypervisors:
• KVM
• Vmware
• XenServer
• Flexibility – arbitrary metada can be passed
• Additional configurations can be added/updated when:
• Deploying a newVM
• Updating a stoppedVM

9. Additional VM configurations
• Format depends on hypervisor:
• KVM: domain XML
• Vmware: key-value pair toVMX file
• XenServer: XAPI to add parameters
IMPORTANT:
• Configurations must be separated by new lines
• Each configuration can have an optional name, ending in ‘:’
• The whole set of configurations must be URL-encoded and
passed as ‘extraconfig’ parameter
• Configurations are stored as VM details with prefix
‘extraconfig’
• New parameter ‘extraconfig’ on existing APIs:
• deployVirtualMachine
• updateVirtualMachine
• This parameter MUST be URL-encoded

10. Additional VM configurations
general-metadata:
<metadata>
<app1:foo
xmlns:app1="http://app1.org/app1/">Test1</ap
p1:foo>
<app2:bar
xmlns:app2="http://app1.org/app2/">Test2</ap
p2:bar>
</metadata>
URL-encode
general-
metadata%3A%0A%3Cmetadata%3E%0A%20%20
%20%20%3Capp1%3Afoo%20xmlns%3Aapp1%3D
%22http%3A%2F%2Fapp1.org%2Fapp1%2F%22%
3ETest1%3C%2Fapp1%3Afoo%3E%0A%20%20%20
%20%3Capp2%3Abar%20xmlns%3Aapp2%3D%22
http%3A%2F%2Fapp1.org%2Fapp2%2F%22%3ETe
st2%3C%2Fapp2%3Abar%3E%0A%20%20%3C%2F
metadata%3E%0A
CS API
• VM details updated on CloudStack database
• Configurations added to the VM domain XML
at VM startup

11. Additional VM configurations
• Configurations stored as part of the service offering
• Introduced in version 4.13
• Security
• Only administrators can create service offerings
• Users select the predefined configurations
• ‘extraconfig’ parameter on createServiceOffering API
• Additional configurations in two ways:
• VM details
• Service offerings
• Additive configurations
• In case of configurations with the same name
• VM detail takes precedence over service
offering detail

12. Additional VM configurations
Demo

13. DPDK Support

14. DPDK Support – Agent configuration
• Prerrequisites:
• DPDK
• OVS networking with DPDK support (can be obtained
on virt SIG repo:
http://mirror.centos.org/centos/7/virt/x86_64/ovirt-
4.2/
• Bridges created with datapath=netdev
• libvirt version supporting huge pages (> 1.2.5)
• libvirt supporting huge pages (set HUGE_PAGES=1 on
/etc/default/qemu-kvm)
• Huge pages enabled

15. DPDK Support – Agent configuration
• New properties can be added on the agent.properties file:
• ‘openvswitch.dpdk.enabled’
• Expected values: true or false
• Enables OVS-DPDK support on the host
• ‘openvswitch.dpdk.ovs.path’
• Expected values: directories in the filesystem
• OVS location expected – usually ‘/var/run/openvswitch’
• DPDK ports will be created under this directory

16. DPDK Support – Create DPDK enabled VMs
• DPDK enabledVMs require additional configurations:
• Huge pages:
• Name: ‘dpdk-hugepages’
• CPU NUMA:
• Name: ‘dpdk-numa’
• Example:
<memoryBacking>
<hugePages/>
</memoryBacking>
dpdk-numa:
<cpu mode="host-passthrough">
<numa>
<cell id="0" cpus="0" memory="9437184"
unit="KiB" memAccess="shared"/>
</numa>
</cpu>
VM start
OVS-DPDK enabled
KVM hosts

17. DPDK Support – Create DPDK enabled VMs
• CloudStack KVM agent receives additional
configurations
• Creates DPDK enabledVMs when:
• ‘openvswitch.dpdk.enabled’ = true
• VM additional configuration contains:
• ‘extraconfig-dpdk-hugepages’
• ‘extraconfig-dpdk-numa’
• OVS driver extended to create OVS ports with
DPDK support
• DPDK ports created with prefix:
• ‘csdpdk-N’ – N incremental
• VM interfaces:
• Created as type=vhostuser
• Path indicates port created with name
‘csdpdk-N’ on
‘openvswitch.dpdk.ovs.path'

18. DPDK Support – Create DPDK enabled VMs
Demo

19. DPDK Support – DPDK vHost User mode selection
• Basic support in 4.12 – vHost user client mode
by default
• Since 4.13, administrators are allowed to
select vHost user mode:
• Client
• Server
• vHost user mode included as part of service
offerings details – ‘DPDK-VHOSTUSER’
• Note: vHost user mode is not an additional
configuration – MUST NOT be URL-encoded
‘DPDK-VHOSTUSER’ = client ‘DPDK-VHOSTUSER’ = server
Source: https://software.intel.com/en-us/articles/data-plane-
development-kit-vhost-user-client-mode-with-open-vswitch

20. DPDK Support – DPDK vHost User mode selection
• CloudStack KVM agent uses server mode by
default (if parameter is not specified)
• OVS ports creation vsVM interfaces:
‘DPDK-VHOSTUSER’ = client ‘DPDK-VHOSTUSER’ = server
Source: https://software.intel.com/en-us/articles/data-plane-
development-kit-vhost-user-client-mode-with-open-vswitch
vHost user mode OVS port type VM interface type
server dpdkvhostuser server
client dpdkvhostuserclient Client

21. DPDK Support – DPDK enabled hosts detection
• Limitations in 4.12:
• CloudStack not aware of which hosts are
DPDK enabled
• DPDK enabledVMs failure on non-DPDK
enabled hosts
• Since 4.13:
• DPDK capability included as part of the
ready command on agent connection
with the management server
• Non-DPDK enabledVMs allowed on
DPDK-enabled hosts
• Suitable hosts selected for deploying DPDK
enabledVMs
• Extension of non-DPDK enabled userVMs on
DPDK-enabled hosts.
• Initially only systemVMs were allowed

22. DPDK Support – DPDK live migrations
• Since the management server is aware of
DPDK enabled hosts:
• Find suitable hosts for migration of DPDK
enabledVMs
• DPDK enabledVMs can only be migrated
to DPDK enabled hosts
• DPDK enabledVM when:
• DPDK required configurations are onVM
details and/orVM service offering
• VM running in a DPDK enabled host
VM live migration

23. Conclusions

24. Conclusions
• Further reading:
• https://www.dpdk.org
• http://docs.openvswitch.org/en/latest/howto/dpdk/
• https://access.redhat.com/documentation/en-
us/red_hat_enterprise_linux/7/html/virtualization_deployme
nt_and_administration_guide/sect-
editing_a_guest_virtual_machines_configuration_file-
adding_multifunction_pci_devices_to_kvm_guest_virtual_m
achines
• http://docs.cloudstack.apache.org/en/latest/plugins/ovs-
plugin.html#dpdk-support
• Pull requests:
• https://github.com/apache/cloudstack/pull/2839
• https://github.com/apache/cloudstack/pull/3153
• https://github.com/apache/cloudstack/pull/3365
• Initial support in 4.12
• Extended support in 4.13
• Future work:
• Include UI – onlyAPI currently
• AdditionalVM configurations
• Service offerings details
• DPDK required configurations
• Support different types of storage
• DPDK as a new host type?
• DPDK enabledVM type?
• And more to be defined…

25. Questions?

26. Thank you!