Event Channels are Xen's mechanism for paravirtualized interrupts. These were limited to only 4096 which then limits the number of guests that a host may support to around 300 to 500. This presentation will give a brief introduction to event channels, a detailed look at the new, innovative FIFO-based event channel ABI that increases the limit to over 100,000 as well as having several other useful features (e.g., multiple priorties). Some of the key performance measurments of the new ABI will be shown.

1. “Unlimited” Event Channels
David Vrabel
24 October 2013

2. What are Event Channels?
Paravirtual interrupts
Edge triggered
Bidirectional
Directed at a single VCPU
24 October 2013
“Unlimited” Event Channels
2 / 13

3. How do Event Channels Work?
Domain A
1
Domain B
Notify
Xen
24 October 2013
Shared Memory
2
4
Set Pending
“Unlimited” Event Channels
Call Handlers
3
Upcall
3 / 13

4. Original (2-Level) Design
Per-VCPU Shared Memory
64 bits
Selector
...
1
Shared Memory
64 bits
Pending
...
Masked
0
1
0
0
...
...
0
1
0
...
...
...
4096 bits
24 October 2013
“Unlimited” Event Channels
4 / 13

5. Problems
Too few
No priorities
Unfair
24 October 2013
“Unlimited” Event Channels
5 / 13

6. Additional Requirements
Identical ABI between 32-bit and 64-bit guests
Reasonable memory usage
Easily extensible
24 October 2013
“Unlimited” Event Channels
6 / 13

7. New (FIFO-based) Design
Bit 15
READY
1
0
Bit 31
1
Port 0
1
(Reserved)
30
29
16
0
P
M
L
LINK
1
0
1
6
1
0
1
0
1
0
1
3
2
3
HEAD
Priority 0
5
1
...
Per-VCPU shared memory
24 October 2013
6
7
...
1
15
4
Up to 132,072 words
Shared memory
“Unlimited” Event Channels
7 / 13

8. Raising an Event
function raise(q, p)
1. Set pending
E[p].pending = 1
if not E[p].masked and not E[p].linked
E[p].linked = 1
if T[q] != p
linked = link(T[q], p)
else
linked = false
if not linked
C.head[q] = p
T[q] = p
24 October 2013
2. Unmasked and not in list?
Add to list
3. Set LINK of tail
or
Set HEAD in control block
4. Advance tail
“Unlimited” Event Channels
8 / 13

9. Handling an Event
function handle_one_event(q)
1. Get local head
p = H[q]
if p == 0
p = C.head[q]
link = unlink(p)
H[q] = link
if E[p].pending and not E[p].masked
handle(p)
2. List was empty?
Get new HEAD from
control block
3. Clear LINKED and LINK
4. Advance local head
5. Pending and unmasked?
Handle IRQ
24 October 2013
“Unlimited” Event Channels
9 / 13

10. Event Channel Fairness
3000
2-Level
FIFO-based
Latency (ns)
2500
2000
1500
1000
500
0
75
80
85
90
Event Channel Port
24 October 2013
“Unlimited” Event Channels
10 / 13

11. Current Status
Xen support in 4.4
Linux support in 3.14
24 October 2013
“Unlimited” Event Channels
11 / 13

12. Further Work
Investigate how priorities may be used
Add support for limiting number of event channels to other toolstacks (XAPI,
libvirt)
Add support in other OSes
Xen event lock scalability
Linux PIRQ setup/teardown refactoring
24 October 2013
“Unlimited” Event Channels
12 / 13

13. Questions?
Design Document
http://xenbits.xenproject.org/people/dvrabel/event-channels-F.pdf
24 October 2013
“Unlimited” Event Channels
13 / 13