1. Evolving New SR-IOV
Configuration Tools
Mitch Williams
Intel LAN Access Division
Intel® Ethernet

2. SR-IOV in One Minute
VM1 VM2 VMn
PV DD PV DD PV DD
VMM
Virtual Switch
NIC HW
Network
Emulated NICs
Intel® Ethernet
*Other names and brands may be claimed as the property of others
2 Copyright © 2010 Intel Corporation. All products, dates and figures are preliminary,
for planning purposes only and are subject to change without notice.

3. SR-IOV in One Minute
VM1 VM2 VMn VM1 VM2 VMn
PV DD PV DD PV DD PV DD PV DD NIC DD
VMM VMM
Virtual Switch Virtual Switch
NIC HW NIC HW VT-D
NIC
Network Network
Emulated NICs Direct Assigned NIC
Intel® Ethernet
*Other names and brands may be claimed as the property of others
3 Copyright © 2010 Intel Corporation. All products, dates and figures are preliminary,
for planning purposes only and are subject to change without notice.

4. SR-IOV in One Minute
VM1 VM2 VMn VM1 VM2 VMn VM1 VM2 VMn
PV DD PV DD PV DD PV DD PV DD NIC DD VF DD VF DD VF DD
VMM VMM
VMM
Virtual Switch Virtual Switch
VF1 VF2 VFn
SR
NIC HW NIC HW VT-D IOV
NIC NIC Switch
NIC (PF)
Network Network Network
Emulated NICs Direct Assigned NIC SR-IOV
Intel® Ethernet
*Other names and brands may be claimed as the property of others
4 Copyright © 2010 Intel Corporation. All products, dates and figures are preliminary,
for planning purposes only and are subject to change without notice.

5. Intel SR-IOV NICs
• Intel® 82576 Gigabit Ethernet Controller
– Up to 8 VFs per port
– Dual Port / Quad Port
– Copper / Fiber / Backplane
– igb / igbvf driver
– Upstream kernel / RHEL 5.4 / XCP 0.1.1
• Intel® 82599 10 Gigabit Ethernet Controller
– Up to 64 VFs per port
– Single Port / Dual Port
– Copper / Fiber / DA / Backplane
– ixgbe / ixgbevf
– Upstream kernel
Intel® Ethernet
*Other names and brands may be claimed as the property of others
5 Copyright © 2010 Intel Corporation. All products, dates and figures are preliminary,
for planning purposes only and are subject to change without notice.

6. Mitch’s SR-IOV Pain
• MAC addresses assigned randomly
– Reassigned each time host reboots
– Causes net device proliferation in VMs
– Annoying for DHCP users
– FATAL for fixed IP addresses
Intel® Ethernet
*Other names and brands may be claimed as the property of others
6 Copyright © 2010 Intel Corporation. All products, dates and figures are preliminary,
for planning purposes only and are subject to change without notice.

7. Mitch’s SR-IOV Pain
• MAC addresses assigned randomly
– Reassigned each time host reboots
– Causes net device proliferation in VMs
– Annoying for DHCP users
– FATAL for fixed IP addresses
Intel Test Subject
#796:
Seen here using
fixed IP addresses
with SR-IOV.
Results Inconclusive.
Intel® Ethernet
*Other names and brands may be claimed as the property of others
7 Copyright © 2010 Intel Corporation. All products, dates and figures are preliminary,
for planning purposes only and are subject to change without notice.

8. Mitch’s SR-IOV Pain Reliever
• Step 1: Have a meeting
• Step 2: Send a flurry of email
• Step 3: Fix it!
– Kernel PCI subsystem
– Kernel netdevice layer
– Kernel RT-netlink interface
– igb driver
– iproute2 package (AKA the “ip” command)
Intel® Ethernet
*Other names and brands may be claimed as the property of others
8 Copyright © 2010 Intel Corporation. All products, dates and figures are preliminary,
for planning purposes only and are subject to change without notice.

9. How It Works
root@avec:~# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:30:48:cc:6d:ee brd ff:ff:ff:ff:ff:ff
vf 0: MAC ee:d4:38:30:91:14
vf 1: MAC 2a:da:93:d7:7d:ec
vf 2: MAC aa:c6:38:a5:8b:a5
vf 3: MAC 4a:d2:64:7c:a8:17
root@avec:~# ip link set dev eth0 vf 0 mac 00:11:22:33:44:55
root@avec:~# ip link set dev eth0 vf 0 vlan 10
root@avec:~# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:30:48:cc:6d:ee brd ff:ff:ff:ff:ff:ff
vf 0: MAC 00:11:22:33:44:55, vlan 10
vf 1: MAC 2a:da:93:d7:7d:ec
vf 2: MAC aa:c6:38:a5:8b:a5
vf 3: MAC 4a:d2:64:7c:a8:17
root@avec:~#
Intel® Ethernet
*Other names and brands may be claimed as the property of others
9 Copyright © 2010 Intel Corporation. All products, dates and figures are preliminary,
for planning purposes only and are subject to change without notice.

10. Other Pain-Reducing Features
• VF VLAN setting
– Put a VF on VLAN, without the VM noticing
• TX Rate Limiting
– Set max transmit rate in MBps
– Coming soon…
Intel® Ethernet
*Other names and brands may be claimed as the property of others
10 Copyright © 2010 Intel Corporation. All products, dates and figures are preliminary,
for planning purposes only and are subject to change without notice.

11. Near Term Needs Part 1
• 10 Gig Support
– Done, in validation
– Plan for submittal to 2.6.35
• Update Distros
– Backport patches to XCP and RHEL 6
– Add support to VM management tools
• Add Migration Support
– Use channel bonding + hotplug in guests
– Add (more) support to VM management tools
Intel® Ethernet
*Other names and brands may be claimed as the property of others
11 Copyright © 2010 Intel Corporation. All products, dates and figures are preliminary,
for planning purposes only and are subject to change without notice.

12. Near-Term Needs Part 2
• VMs with emulated NICs
can’t talk to VMs with VF
VMA VMB VM1 VM2 VMn
NICs. VNIC VNIC VF DD VF DD VF DD
• Need to add MAC
VMM Virtual Switch
addresses of emulated
NICs to PF NIC switch. SR
VF1 VF2 VFn
• No mechanism in place for
IOV
NIC Switch
this. NIC (PF)
Network
Intel® Ethernet
*Other names and brands may be claimed as the property of others
12 Copyright © 2010 Intel Corporation. All products, dates and figures are preliminary,
for planning purposes only and are subject to change without notice.

13. Call For Comments
• Please help us!
• What do you need to make SR-IOV work for you?
• What do you want to make SR-IOV easy for you?
• mitch.a.williams@intel.com
• gregory.v.rose@intel.com
Intel® Ethernet
*Other names and brands may be claimed as the property of others
13 Copyright © 2010 Intel Corporation. All products, dates and figures are preliminary,
for planning purposes only and are subject to change without notice.