XPDS14 - vNUMA in Xen - Wei Liu, Citrix

•

1 like•3,180 views

This document discusses vNUMA support in the Xen hypervisor. It motivates vNUMA by explaining how cross-NUMA node memory access is expensive and how vNUMA can avoid this. It describes the history and current status of vNUMA in Xen, the design which involves providing guest VMs with virtual NUMA topology information, and challenges with features like ballooning. Preliminary benchmarks show performance improvements with vNUMA enabled in PV guests. Future work includes improving vNUMA support and addressing compatibility issues.

Technology

Agenda Motives History and status Design Problems Benchmark Future Work
Agenda
I Motives
I History and status
I Design
I Problems
I Preliminary benchmark results
I Future work
Chicago { August 18, 2014 vNUMA in Xen 2 / 19

Agenda Motives History and status Design Problems Benchmark Future Work
Motives
I Cross NUMA node memory access is expensive
I Need to avoid cross node memory access
I Xen is NUMA aware
I NUMA aware scheduling
I NUMA aware guest memory placement
I Operating system like Linux is NUMA aware
I NUMA aware scheduling
I NUMA aware memory allocation / migration
I The missing bits
I Memory layout information
I CPU topology
Chicago { August 18, 2014 vNUMA in Xen 3 / 19

Agenda Motives History and status Design Problems Benchmark Future Work
History and status
I PV vNUMA presented in Xen Summit 2010 by Dulloor Rao
http://slidesha.re/1AXsFbu
I HVM vNUMA patches posted by Andre Przywara circa 2010
I Elena U

mtseva has been working on upstreamable PV
vNUMA since 2013
Chicago { August 18, 2014 vNUMA in Xen 4 / 19

Agenda Motives History and status Design Problems Benchmark Future Work
Design: PV and PVH
I Toolstack puts enlightenment information in hypervisor
I Guest memory allocation in accordance with enlightenment
information
I Guest retrieves enlightenment information via hypercall during
boot up
Chicago { August 18, 2014 vNUMA in Xen 5 / 19

$Agenda Motives History and status Design Problems Benchmark Future Work Design: PV and PVH Enlightenment information structure struct vnuma_info { nr_vnodes; vdistance[nr_vnodes * nr_vnodes]; vcpu_to_vnode[nr_vnodes]; vnode_to_pnode[nr_vnodes]; vmemrange[nr_vnodes]; } Chicago { August 18, 2014 vNUMA in Xen 6 / 19$

Agenda Motives History and status Design Problems Benchmark Future Work
Design: HVM
I Toolstack puts enlightenment information in hypervisor
I Toolstack arranges ACPI tables
I Guest memory allocation in accordance with enlightenment
information
I Guest retrieves layout information via ACPI tables during boot
up
Chicago { August 18, 2014 vNUMA in Xen 7 / 19

Agenda Motives History and status Design Problems Benchmark Future Work
Problems: vNUMA and other features
PV PVH HVM
Ballooning Y Y* N
PoD N/A ? N
Chicago { August 18, 2014 vNUMA in Xen 8 / 19

Agenda Motives History and status Design Problems Benchmark Future Work
Problems: CPU topology
Chicago { August 18, 2014 vNUMA in Xen 9 / 19

Agenda Motives History and status Design Problems Benchmark Future Work
Benchmark
I Host
I 2 sockets, 12 PCPUs, HT disabled
I 36GB RAM, 2 NUMA nodes
I NUMA balancing enabled
I Guest
I 12 VCPUs
I 16GB RAM, 2 virtual NUMA nodes
I vnodes mapped to dierent pnodes, vcpu pinned to pnode
I NUMA balancing enabled
I Benchmarks to run
I Autonuma
I SPECJBB
I STREAM
Chicago { August 18, 2014 vNUMA in Xen 10 / 19

Agenda Motives History and status Design Problems Benchmark Future Work
Benchmark: Autonuma
Chicago { August 18, 2014 vNUMA in Xen 11 / 19

Agenda Motives History and status Design Problems Benchmark Future Work
Benchmark: Autonuma
Chicago { August 18, 2014 vNUMA in Xen 12 / 19

Agenda Motives History and status Design Problems Benchmark Future Work
Benchmark: Autonuma
Chicago { August 18, 2014 vNUMA in Xen 13 / 19

Agenda Motives History and status Design Problems Benchmark Future Work
Benchmark: SPECJBB
Chicago { August 18, 2014 vNUMA in Xen 14 / 19

Agenda Motives History and status Design Problems Benchmark Future Work
Benchmark: SPECJBB
Chicago { August 18, 2014 vNUMA in Xen 15 / 19

Agenda Motives History and status Design Problems Benchmark Future Work
Benchmark: STREAM
Chicago { August 18, 2014 vNUMA in Xen 16 / 19

Agenda Motives History and status Design Problems Benchmark Future Work
Benchmark: Conclusion
I vNUMA improves performance for PV guest
I vNUMA has weird result in SPECJBB for PVH guest, but
other two benchmarks have good results
Chicago { August 18, 2014 vNUMA in Xen 17 / 19

More from The Linux Foundation

XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...

The Linux Foundation

In recent years unikernels have shown immense performance potential (e.g., boot times of only a few ms, image sizes of only hundreds of KBs).The fundamental drawback of unikernels is that they require that applications be manually ported to the underlying minimalistic OS, needing both expert work and often considerable amount of time. The Unikraft project provides a unikernel code base and build system that significantly simplifies the building of unikernels. In addition to support for a number CPU architectures, languages and frameworks, Unikraft provides debugging and tracing features that are generally sorely missing from unikernel projects. In this talk we will talk about these features, show a set of preliminary performance numbers, and provide a roadmap for the project's future.

XPDDS19 Keynote: Unikraft Weather Report

The Linux Foundation

XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...

The Linux Foundation

This talk will introduce Dom0-less: a new way of using Xen to build mixed-criticality solutions. Dom0-less is a Xen feature that adds a novel approach to static partitioning based on virtualization. It allows multiple domains to start at boot time directly from the Xen hypervisor, decreasing boot times dramatically. Xen userspace tools, such as xl and libvirt, become optional. Dom0-less extends the existing device tree based Xen boot protocol to cover information required by additional domains. Binaries, such as kernels and ramdisks, are loaded by the bootloader (u-boot) and advertised to Xen via new device tree bindings. The audience will learn how to use Dom0-less to partition the system. Uboot and device tree configuration details will be explained to enable the audience to get the most out of this feature. The talk will include a status update and details on future plans.

XPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, Xilinx

The Linux Foundation

As the number of contributions grow, reviewer bandwidth becomes a bottleneck; and maintainers are always asking for more help. However, ultimately maintainers must at least Ack every patch that goes in; so if you're not a maintainer, how can you contribute? Why should anyone care about your opinion? This talk will try to lay out some advice and guidelines for non-maintainers, for how they can do code review in a way which will effectively reduce the load on maintainers when they do come to review a patch.

XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...

The Linux Foundation

This talk is a follow-up to our Summit 2017 presentation in which we covered our plans for Intel VMFUNC and #VE, as well as related use-cases. This year, we will provide a report on what we have accomplished in Xen 4.12, and what remains to be addressed. We will also give a brief status update of VMI on AMD hardware. The session will end with some real-world numbers of the Hypervisor Introspection solution running on Citrix Hypervisor 8.0 with #VE enabled.

XPDDS19: Memories of a VM Funk - Mihai Donțu, Bitdefender

The Linux Foundation

Safety certification is one of the essential requirements for software to be used in highly regulated industries. Besides technical and compliance issues (such as ISO 26262 vs IEC 611508) transitioning an existing project to become more easily safety certifiable requires significant changes to development practices within an open source project. In this session, we will lay out some challenges of making safety certification achievable in open source and the Xen Project. We will outline the process the Xen Project has followed thus far and highlight lessons learned along the way. The talk will primarily focus on necessary process, tooling changes and community challenges that can prevent progress. We will be offering an in-depth review of how Xen Project is approaching this challenging goal and try to derive lessons for other projects and contributors.

OSSJP/ALS19: The Road to Safety Certification: Overcoming Community Challeng...

The Linux Foundation

Safety certification is one of the essential requirements for software to be used in highly regulated industries. The Xen Project, a secure and stable hypervisor that is used in many different markets, has been exploring the feasibility of building safety certified products on top of Xen for a year, looking at key aspects of its code base and development practices. In this session, we will lay out the motivation and challenges of making safety certification achievable in open source and the Xen Project. We will outline the process the project has followed thus far and highlight lessons learned along the way. The talk will cover technical enablers, necessary process and tooling changes and community challenges offering an in-depth review of how Xen Project is approaching this exciting and and challenging goal.

OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making...

The Linux Foundation

2018 saw fundamental shifts in security boundaries which were previously taken for granted. A lot of work has been done in the past 2 years, and largely in secret under embargo, but there is plenty more work to be done to strengthen the existing mitigations and to try to recover some performance without reopening security holes. This talk will look at speculative execution sidechannels, the work which has already been done to mitigate the security holes, and future work which hopes to bring some improvements.

XPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, Citrix

The Linux Foundation

The Arm architecture provides a set of guidelines that any software should abide by when accessing the memory with MMU off and update page-tables. Failing to do so may result in getting TLB conflicts or breaking coherency. In a previous talk ("Keeping coherency on Arm"), we focused on updating safely the stage-2 (aka P2M) page-tables. This talk will focus on the boot code and Xen memory management. During this session, we will introduce some of the guidelines and when they should be used. We will also discuss how Xen boot sequence needs to be reworked to avoid breaking the guidelines.

XPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltd

The Linux Foundation

For many years the QEMU codebase has contained PV backends for Xen guests, giving them paravirtual access to storage, network, keyboard, mouse, etc. however these backends have not been configurable as QEMU devices as their implementation did not fully adhere to the QEMU Object Model (QOM). Particularly the PV storage backend not using proper QOM devices, or qdevs, meant that the QEMU block layer needed to maintain legacy code that was cluttering up the source. This was causing push-back from the maintainers who did not want to accept any patches relating to that Xen backend until it was 'qdevified'. In this talk, I'll explain the modifications I made to QEMU to achieve 'qdevification' of the PV storage backend, how compatibility with the libxl toolstack was maintained, and what the next steps in both QEMU and libxl development should be.

XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...

The Linux Foundation

PCI is a local computer bus for attaching hardware devices in a computer, and is the main peripheral bus on modern x86 systems. As such, having a proper way to emulate it is crucial for Xen to be able to expose both fully emulated devices or passthrough devices to guests. This talk will focus on the current status of PCI emulation in Xen, how and where it is used, what are its main limitations and future plans to improve it in order to be more robust and modular.

XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D

The Linux Foundation

XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems

The Linux Foundation

Xen is a very powerful hypervisor with a talented and diverse developers community. Despite the fact it's almost everywhere (from the Cloud to the embedded world), it can be difficult to set up and manage as a system administrator. General purpose distros have Xen packages, but that's just a start in your Xen journey: you need some tooling and knowledge to have a working and scalable platform. XCP-ng was built to overcome those issues: by bringing Xen to the masses with a fully turnkey distro with Xen as its core. It's the logical sequel to the XCP project, with a community focus from the start. We'll see how it happened, what we did, and what's next. Finally, we'll see the impact of XCP-ng on the Xen Project.

XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...

The Linux Foundation

XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...

The Linux Foundation

High level toolstacks for server and cloud virtualization are very mature with large communities using and supporting them. Client virtualization is a much more niche community with unique requirements when compared to those found in the server space. In this talk, we’ll introduce a client virtualization toolstack for Xen (redctl) that we are using in Redfield, a new open-source client virtualization distribution that builds upon the work done by the greater virtualization and Linux communities. We will present a case for maturing libxl’s Go bindings and discuss what advantages Go has to offer for high level toolstacks, including in the server space.

XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...

The Linux Foundation

Today Xen is scheduling guest virtual cpus on all available physical cpus independently from each other. Recent security issues on modern processors (e.g. L1TF) require to turn off hyperthreading for best security in order to avoid leaking information from one hyperthread to the other. One way to avoid having to turn off hyperthreading is to only ever schedule virtual cpus of the same guest on one physical core at the same time. This is called core scheduling. This presentation shows results from the effort to implement core scheduling in the Xen hypervisor. The basic modifications in Xen are presented and performance numbers with core scheduling active are shown.

XPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSE

The Linux Foundation

The use of Virtual GPUs (vGPUs) has widely grown in server farms to give Virtual Machines (VMs) dedicated graphics. Software rendering with virtual CPUs can only take us so far and even with Intel-GVT, which uses integrated graphics, there isn't enough power to do the fun stuff. In this presentation, Jon Farrell will be talking about the process of implementing AMD MxGPU on Xen, challenges that he encountered while doing it, and discussing performance metrics of bare metal and vGPU VM on popular benchmarks like 3D Mark* and The Witcher 3. To wrap up his presentation, Jon will share his thoughts about future research and where this technology can take us.

XPDDS19: Implementing AMD MxGPU - Jonathan Farrell, Assured Information Security

The Linux Foundation

Current support of nested virtualization with Xen is limited to fully emulated devices for the L1 hypervisor (L0 hypervisor being the one running on the physical machine). For being able to let L2 dom0 make use of L1 PV devices several new interfaces are needed. In this design session I'll present my ideas how to add support of PV devices for L2 dom0. There are several possibilities how to do the work which I'd like to discuss.

XPDDS19: Support of PV Devices in Nested Xen - Jürgen Groß, SUSE

The Linux Foundation

In today's public and private cloud markets, availability is a very important metric for all cloud service providers. COLO is an ideal Application-agnostic Solution for Non-stop service in the cloud. Our solution can protect user service even from physical network or power interruption. And the the switching process is difficult for users to perceive (TCP connection will not be terminated). Under COLO mode, both primary VM (PVM) and secondary VM (SVM) are running parallel. The COLO has more than ten times performance increase compared with previous solution (like Remus). Current COLO codes has been merged in QEMU community, we can use COLO in upstream without any other addition patches. In this talk, we will talk about the COLO implementation in QEMU and Xen, the new designed COLO-Proxy, discussing on problems we've met while developing COLO. and report the latest progress from Intel.

XPDDS19: Application Agnostic High Availability Solution On Hypervisor Level ...

The Linux Foundation

More from The Linux Foundation (20)