With the development of virtualization, there are more device assignment requirements. Based on VT-d interrupt remapping, Intel introduces VT-d interrupt posting as a more enhanced method to handle interrupts in the virtualization environment. The Posted Interrupts (PI) on CPU side has been already supported in Intel CPUs, with VT-d Posted Interrupt we can get some additional advantages, it can directly deliver external interrupts to running vCPUs without hypervisor involvement, decease the interrupt migration complexity, differentiate between urgent and non-urgent external interrupt, and avoid consuming host-vector for each interrupt to vCPU. In this presentation, Feng will talk about the mechanism of VT-d PI and its advantages, as well as some performance data of I/O intensive workload in Xen, which will show the performance gain after using VT-d PI.

1. VT-d Posted Interrupts
Feng Wu, Jun Nakajima <Speaker>
Intel Corporation

2. Agenda
• Motivation
• Difference btw CPU-based and VT-d Posted Interrupts
• Architecture
• Implementation Details
• Performance
• Summary
2

3. Motivation
• Interrupt virtualization efficiency
• Interrupt migration complexity
• Big requirement of host vector for different assigned devices
3

4. CPU-based Posted-Interrupt in Xen
• External interrupt handling
4
VCPU0
running on
Physical CPU 0
External
interrupt
VCPU1
running on
physical CPU 1
Set bit for guest Vector in
Posted-interrupt requests (PIR)
of VCPU1
VM Exit
Guest IPI:
- No need to kick VCPU1
- Notification Event is handled by
Notification Event
VCPU1 in non-root mode
- vIRR is updated by H/W and
delivered by “virtual interrupt
delivery” mechanism
- VM-Exit is eliminated
Xen

5. Key Data Structures for CPU-based Posted-
Interrupt Processing
• Posted-interrupt notification vector:
− Send virtual interrupts to guests w/o VM exit
5
− If physical vector == Posted-interrupt
notification vector (VMCS field)
• PIR (Posted-interrupt requests)
− Set bits for guest vectors in advance
• ON (Outstanding Notification)
− If this bit is set, there is a notification outstanding
for one or more posted interrupts
Virtual-APIC
Page
PI Desc.
Address
Posted-interrupt
notification vector
PIR
ON
Posted-interrupt
Descriptor
5
VMCS

6. CPU-based Posted-Interrupt in Xen – cont’d
6
VCPU1
running on
physical CPU 1
Set bit for guest Vector in
Posted-interrupt requests (PIR)
of VCPU1
• Virtual interrupts from QEMU
Guest IPI:
Notification Event
Virtual Devices
QEMU
Virtual interrupts
- Virtual MSI
- Virtual IOAPIC
Xen
- No need to kick VCPU1
- Notification Event is handled by
VCPU1 in non-root mode
- vIRR is updated by H/W and
delivered by “virtual interrupt
delivery” mechanism
- VM-Exit is eliminated

7. What’s new for VT-d Posted-Interrupts
7
VCPU1
running on
physical CPU 1
Notification Event
VT-d Engine
with
Posted-Interrupt support
External Interrupts
Direct-assigned
Device
- No need to kick VCPU1
- Notification Event is handled by
VCPU1 in non-root mode
- vIRR is updated by H/W and
delivered by “virtual interrupt
delivery” mechanism
- VM-Exit is eliminated
No VMM overhead
AT ALL!

8. VT-d Posted-Interrupts Architecture
8
Virtual Processor A
VMCS
vAPIC Page
Posted Descriptor
Virtual Processor B
VMCS
vAPIC Page
Virtual Processor C
CPU 0 CPU 1
VMCS
Posted Descriptor
Interrupt
Remapping Table
IRTE for Interrupt Z
IRTE for Interrupt Y
IRTE for Interrupt X
IRTE for Interrupt N
External Interrupts
Notification
Event
Notification
Event
Posted Descriptor
vAPIC Page
Notification
Event
CPU 0 Host-Vectors
Other Vectors (e.g. Hyp. IPIs)
Other Vectors (e.g. Timer)
Notification Vector for vCPU B
Notification Vector for vCPU A
CPU 1 Host-Vectors
Other Vectors (e.g. Hyp. IPIs)
Other Vectors (e.g. Timer)
Notification Vector for vCPU C

9. Xen Implementation Details:
• Update IRET according to guest’s modification to the
interrupt configuration (MSI address, data)
• Interrupt migration during VCPU scheduling
9

10. Xen Implementation: IRTE update
10
QEMU
Xen
VCPU
Interrupt configuration update
(e.g. MSI/MSIx)
Hypercall
XEN_DOMCTL_bind_pt_irq!
Update the guest information in
IRTE
- Guest vector
- Posted-interrupt descriptor
address
No changes needed
for this part
Changes happen here

11. Xen Implementation: VCPU Scheduling
‘SN’ – Set
‘NV’ –Notification vector
11
RUNSTATE_running
‘SN’ – Clear
‘NDST’ – New
physical CPU of the
VCPU
Posted-Interrupt Descriptor
‘SN’ – Set
‘NV’ –Wakeup vector
‘SN’ – Set
‘NV’ –Wakeup vector
RUNSTATE_runnable RUNSTATE_blocked
‘SN’ – Set
‘NV’ –Notification vector
NV
PIR [0 - 255]
SN
NV – Host vector for Notification Event
SN – If set, suppress Notification Event

12. Summary
• VT-d Posted-interrupts advantages
– External interrupts from direct-assigned devices are delivered to guest
12
running in non-root mode directly
– Improve Interrupt virtualization efficiency, e.g. Less VM-Exits.
– Simplify interrupt migration
– Consume less physical interrupts
• Performance
• The Specification will be published very soon
– Can be found in Intel website

13. 13
Thank YOU!
Q & A
Or contact
Feng Wu <feng.wu@intel.com>

14. 14
Back up

15. VT-d Posted-Interrupts Support
• Interrupt-remap-table-entry (IRTE) enhanced as follows:
- An existing reserved bit claimed to indicated Posted-interrupt (PST)
- Software may choose to “remap” or “post” each interrupt independently
• IRTEs with ‘PST’ set are interpreted per below format
- New Fields
15
• Descriptor Address: the address of the posted-interrupt descriptor
• Virtual Vector: the guest vector of the interrupt
• URG: indicates if the interrupt is urgent
- Other fields continue to have the same meaning

16. Posted-interrupt Descriptor
16

17. VT-d: Steps for Interrupt Posting
17
Read IRTE
Interrupt
Remapping
‘PST’ == 1
N
Y
Read Posted-interrupt descriptor (locking the cache-line)
Set guest virtual in PIR
(‘ON’ == 0)
&&
(‘URG’ == 1 || ‘SN’ == 0)
Y
Y
Set ‘ON’ and Generate notification event
Unlock cache-line
N
Unlock cache-line