Noboru Iwamatsu: Paravirtualized USB Support

1. Paravirtualized USB Support for Xen:
Status Update
Noboru Iwamatsu
n_iwamatsu@jp.fujitsu.com
FUJITSU LABORATORIES LTD.
Xen Summit at Oracle Feb 24-25, 2009

2. Contents
 Background and Motivation
 Development Status
 PVUSB Design
 Demo (Video)
 Test Results
1 Copyright 2009 FUJITSU LIMITED

3. Background: Client virtualization and USB
 In client virtualization, special Service-VMs work in the
background of User-VM, and provide various functions.
 Both User-VM and Service-VMs require using the USB
devices at the same time by the same USB host controller.
User VM Service VM Service VM
Usage example of
client virtualization User OS Service OS Service OS
Office/Music VoIP/
Authentication
apps Messenger
Hypervisor
USB Host Controller
Laptop
USB devices
2 Copyright 2009 FUJITSU LIMITED

4. Motivation
 USB support for Xen today
 Qemu-dm UHCI emulation
• USB 1.1 only
 PCI pass-through with IOMMU
• Works well. But the entire controller is assigned to a single domain and can’t
be used from other domains.
 Current options are not suitable for client virtualization!
 We have started development to achieve a good performance and
flexibility!
Aug. 2008 Started development.
Proposed in XCI (Xen Client Initiative).
“PV USB support” was on the roadmap of Xen 3.4.
Sep. 2008
Nov. 2008 Just started working!
Xen Summit @Tokyo 2008
3 Copyright 2009 FUJITSU LIMITED

5. Development Status
CHANGES from the last summit
 What we did
 Emulation of the CLEAR_HALT command added in the backend.
 RING operations and queue usage changed.
 Initial hotplug code added.
 Results
 Various devices worked (HID, Storage, Serial, Ethernet)
 Performance improved (6MB/s to 33MB/s on USB storage)
 Hotplug worked, but unplug not worked yet :-)
 What we left
 No code posted yet!
 Disconnection and canceling functions not completed.
 Not well tested.
4 Copyright 2009 FUJITSU LIMITED

6. PVUSB Design: Driver Architecture
Driver domain kernel-space Guest domain kernel-space
free or reuse
usb_alloc_urb()
usbback USB Function Driver
urb urb
Xen backend interface
USB Core
xenbus RING .urb_enqueue() usb_hcd_giveback_urb()
usb_alloc_urb()
Virtual USB
free Cloned
Cloned USB Function Driver Host Controller Driver
urb urb
usb_submit_urb () urb->complete
RING xenbus
USB Core
Xen frontend interface
Host Controller Driver usbfront
Hardware
(USB Host Controller) urb (USB Request Block structure):
In linux kernel, all USB drivers communicate
with urb. (described in include/linux/usb.h).
 Frontend driver (usbfront) - Implemented as a USB host controller driver
 Backend driver (usbback) - Implemented as a USB function driver
5 Copyright 2009 FUJITSU LIMITED

7. PVUSB Design: Hotplug
 Hotplug-rule is set from sysfs interface in the backend driver.
 The hotplug-rule format
• <usbbusname>:<domid>:<vusb number>:<virtual port number>
 Example settings
% echo 1-2.3:1:0:3 > /sys/bus/usb/drivers/usbback/new_vport
% echo 1-4:2:0:1 > /sys/bus/usb/drivers/usbback/new_vport
% echo 1-2.1:2:0:2 > /sys/bus/usb/drivers/usbback/new_vport
Claim the device, Guest VM 1 Guest VM 2
Dom 0
and notify by xenbus. (domain ID:1) (domain ID:2)
usbback Hotplug notification usbfront usbfront
Host Controller Host Controller
Host Controller
(vusb-0) (vusb-0)
(usb1)
Virtual Virtual
roothub
roothub roothub
1-4
device
hub
device device device device
1-2.1 1-2.3
device device
6 Copyright 2009 FUJITSU LIMITED

8. PVUSB Design: Source Code Tree
Require no change in existing kernel code!
Xen USB public I/O header
usbif header file
Xen USB backend driver
Backend interface management
Makefile
USB backend RING operations and URB transmissions
usbback header file
USB stub driver – grabbing and managing USB devices
Xenbus operations
Xen USB frontend driver
Makefile
usbfront header file
Debugging functions for Host Controller
USB2.0 Host Controller interface
Root Hub emulations for Host Controller
USB frontend RING operations and URB transmissions
Xenbus operations
7 Copyright 2009 FUJITSU LIMITED

9. Demo (Video)
 Hotplug Keyboard, Mouse, Flash drive, HDD and Ethernet.
 All devices successfully connected to guest domain and work fine.
8 Copyright 2009 FUJITSU LIMITED

10. Results: Tested Devices (USB2.0)
Device Transfer
Name Manufacturer Driver Status
Type Type
Flash drive RUF2-R2GS Buffalo Bulk usb-storage
Flash drive RUF-C1G/U2 Buffalo Bulk usb-storage
Flash drive DataTraveler DTI/1G Kingston Bulk usb-storage
HDD HDCN-U500 IO DATA Bulk usb-storage
Memory Card
BSCRA38U2 Buffalo Bulk usb-storage
Reader/Writer
Ethernet
ETX2-US2 IO DATA Bulk/Intr pegasus
(100Base-TX)
Ethernet
ETG2-US2 IO DATA Bulk/Intr asix
(1000Base-T) [1]
USB2.0 SVGA
VGA KAIREN Bulk/Intr sisusbvga
Adapter [2]
Works Works with issues Not work
[1] Interrupt URB fails, and the device goes into a stall.
[2] No URB fails, but fails to output the correct screen.
9 Copyright 2009 FUJITSU LIMITED

11. Results: Tested Devices (USB1.1)
Device Transfer
Name Manufacturer Driver Status
Type Type
Keyboard FMV-KB333 Fujitsu Intr usbhid
Keyboard FKB-108-EU FILCO Intr usbhid
Cordless Notebook
Mouse Logitech Intr usbhid
Mouse
Mouse Optical Mouse USB Logitech Intr usbhid
FDD USB-FDU Y-E DATA Bulk/Intr usb-storage
Serial USB-RSAQ5 IO DATA Bulk/Intr pl2303
Webcam WebCam 3 USB Creative Labs Isoc ov511 [3]
Webcam WebCam NX pro Creative Labs Isoc gspca [4]
Works Works with issues Not work
[3] No URB fails, but incorrect horizontal stripes are into the video image.
[4] No URB fails, but the video image has incorrect blinking and flickers.
10 Copyright 2009 FUJITSU LIMITED

12. Results: Performance (USB Storage)
USB Mass Storage Read/Write Throughput (MB/s)
0.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 40.0
92% vs native
29.0
Read
USB Flash Drive
31.5
82% vs native
20.3
Write
24.9
99% vs native
33.3
Read
33.8
USB HDD
99% vs native
24.9 Paravirtualized USB
Write
25.2
Dom0 nativedriver
Test environment:
Read/Write 1GB block using dd command.
CPU: Intel Core 2 Duo 6700 2.66GHz
Kernel: linux-2.6.18-xen (c/s 789), x86_64
USB Flash Drive: Buffalo RUF2-R2GS
Xen: 3.3.0 release, x86_64
USB HDD: IODATA HDCN-U500
11 Copyright 2009 FUJITSU LIMITED

13. Results: Performance (USB Ethernet)
USB Ethernet(100Base-TX) TCP Throughput (Mbps)
0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 100.0
86% vs native
77.7
Rx
89.9
USB Ethernet
67% vs native
62.2
Tx
92.5
Paravirtualized USB
Dom0 native driver
Test environment:
TCP throughput measured by ttcp command.
CPU: Intel Core 2 Duo 6700 2.66GHz
Kernel: linux-2.6.18-xen (c/s 789), x86_64
USB Ethernet: IODATA ETX2-US2
Xen: 3.3.0 release, x86_64
12 Copyright 2009 FUJITSU LIMITED

14. Results: CPU Utilization (USB Storage)
CPU utilization: USB HDD Read/Write
50
x1.9 vs native
Paravirtualized USB
Domain 0 native driver
40
Guest
x2.6 vs native
30
CPU(%)
Guest
20
Dom0
Dom0
Dom0
10
Dom0
0
Paravirtualized Dom0 native Paravirtualized Dom0 native
USB driver USB driver
Read Write
Test environment:
Xentop average while Read/Writ with dd command.
CPU: Intel Core 2 Duo 6700 2.66GHz
Kernel: linux-2.6.18-xen (c/s 789), x86_64
USB HDD: IODATA HDCN-U500
Xen: 3.3.0 release, x86_64
13 Copyright 2009 FUJITSU LIMITED

15. Results: CPU Utilization (USB Ethernet)
CPU utilization: USB Ethernet Rx/Tx
90
x3 vs native Paravirtualized USB
80
Domain 0 native driver
70 Guest
x3.4 vs native
60
Guest
CPU(%)
50
40
Dom0
30
Dom0
20
Dom0
10 Dom0
0
Paravirtualized Dom0 native Paravirtualized Dom0 native
USB driver USB driver
Rx Tx
Test environment:
Xentop average while using ttcp command.
CPU: Intel Core 2 Duo 6700 2.66GHz
Kernel: linux-2.6.18-xen (c/s 789), x86_64
USB Ethernet: IODATA ETX2-US2
Xen: 3.3.0 release, x86_64
14 Copyright 2009 FUJITSU LIMITED

16. TODO & Future Work
 TODO (for posting the code)
 Complete the device disconnection and the URB canceling functions.
• Current PVUSB can hotplug, but can’t unplug, and can’t cancel the URBs that
already transferred to the backend.
• When the device fails and actually stops, current PVUSB can do nothing.
 Add Xend support.
We take a month to post the code.
 Future Work (after initial posted)
 Isochronous transfer support (for WebCAM)
• Driver has no errors, the problem seems timing related.
 Some optimizations
• Reducing the buffer allocation overhead
• Better handling of the large buffer transfer
• Netchannel2 support
15 Copyright 2009 FUJITSU LIMITED

17. Conclusion
 Paravirtualized USB achieved:
 Various devices worked.
Worked devices
Keyboard(2), Mouse(2), FDD(1), Serial(1), Flash drive(3), HDD(1), Ethernet(1)
 Performance improved.
• USB Storage: 99 % of the native driver
• USB Ethernet: 67% - 86% of the native driver
 Hotplug worked.
 Toward the initial release
 Complete the several essential functions.
 More test
• SMP(Quad-core), multiple-domain, heavy load, …
16 Copyright 2009 FUJITSU LIMITED

18. 17 Copyright 2009 FUJITSU LIMITED

19. Appendix: dmesg output
dmesg output after usbfront loaded.
18 Copyright 2009 FUJITSU LIMITED

20. Appendix: lsusb output
lsusb output after connected the devices.
19 Copyright 2009 FUJITSU LIMITED

21. Appendix: Webcam(ov511) output
xawtv video streaming output
20 Copyright 2009 FUJITSU LIMITED

22. Appendix: USB-VGA(sisusbvga) output
USB-VGA output: CentOS 5.2 default login screen (res. 1024x768)
21 Copyright 2009 FUJITSU LIMITED