As users demand greater scalability from Citrix XenServer, the transmission of performance data from guests via xenstore is increasingly becoming a bottleneck. Future use of service domains is likely to make this problem worse. A simple, efficient way of transmitting time-varying datasets between userspace components in different domains is required. This talk will propose a lock-free mechanism to allow interdomain reporting of performance data without relying on continuous xenstore usage, and describe how it fits into the XAPI toolstack.

1. Efficient interdomain transmission of
performance data
John Else
john.else@citrix.com

2. XAPI project
XenAPI
XAPI
XCP/XenServer toolstack

3. XAPI project
High level OO API around Xen
HTTP interface
Collects performance data from dom0/guests

4. Performance data
Numeric
Changes regularly - requires sampling
If we sometimes lose a sample - that's OK

5. Interdomain performance data
Currently one metric per domU: free memory
Transmitted via xenstore
Even this has been enough to cause performance problems!

6. Improvements so far
Data collection caused laggy API response from XAPI
-> XAPI disaggregation: rrdd, networkd and xenopsd

7. Improvements so far
High xenstore load reduced bootstorm performance
-> Watch xenstore instead of polling

8. What next?
Take xenstore out of the data path
Create a generic API for:
Driver domain monitoring
Application monitoring
...

9. rrdd plugins - dom0

10. rrdd plugins - interdomain

11. rrdd plugins - protocol v1
header 10 bytes
data length (ASCII) 8 bytes
md5sum (ASCII) 32 bytes
json data inc. timestamp ? bytes

12. rrdd plugins - protocol v1
DATASOURCES <- header
00000a79 <- data length
6cc0472a94896d245dcea04a37c26474 <- data md5sum
{
"timestamp": 1407715758,
"datasources": {
"io_throughput_read_9d0e83ec": {
"description": "Data read from the SR, in MiB/s",
"owner": "host",
"value": "0.55",
"value_type": "float",
"type": "absolute_to_rate",
"units": "MiB/s",
"min": "0.00",
"max": "inf"
}
}
}

13. rrdd plugins - protocol v1
DATASOURCES <- header
00000a79 <- data length
6cc0472a94896d245dcea04a37c26474 <- data md5sum
{
"timestamp": 1407715758,
"datasources": {
"io_throughput_read_9d0e83ec": {
"description": "Data read from the SR, in MiB/s",
"owner": "host",
"value": "0.55",
"value_type": "float",
"type": "absolute_to_rate",
"units": "MiB/s",
"min": "0.00",
"max": "inf"
}
}
}

14. rrdd plugins - protocol v2
Separate data from metadata
Pack as much as possible as binary

15. rrdd plugins - protocol v2
header 10 bytes
data crc32 4 bytes
metadata crc32 4 bytes
datasource count (int32) 4 bytes
timestamp (int64) 8 bytes
datasource values n * 8 bytes
metadata length (int32) 4 bytes
json metadata ? bytes

16. rrdd plugins - protocol v2
header 10 bytes
data crc32 4 bytes
metadata crc32 4 bytes
datasource count (int32) 4 bytes
timestamp (int64) 8 bytes
datasource values n * 8 bytes
metadata length (int32) 4 bytes
json metadata ? bytes

17. rrdd plugins - protocol v2
DATASOURCES <- header
.... <- data CRC
.... <- metadata CRC
.... <- datasource count
........ <- timestamp
........ <- datasource value
.... <- metadata length
{
"datasources": {
"io_throughput_read_9d0e83ec": {
"description": "Data read from the SR, in MiB/s",
"owner": "host",
"value_type": "float",
"type": "absolute_to_rate",
"units": "MiB/s",
"min": "0.00",
"max": "inf"
}
}
}

18. rrdd plugins - protocol v2
DATASOURCES <- header
.... <- data CRC
.... <- metadata CRC
.... <- datasource count
........ <- timestamp
........ <- datasource value
.... <- metadata length
{
"datasources": {
"io_throughput_read_9d0e83ec": {
"description": "Data read from the SR, in MiB/s",
"owner": "host",
"value_type": "float",
"type": "absolute_to_rate",
"units": "MiB/s",
"min": "0.00",
"max": "inf"
}
}
}

19. rrdd plugins - protocol v2
V2 protocol gives about a 10x speedup in the "usual" case, i.e. set of
exported datasources doesn't change.

20. Demo

21. Further info
Protocol:
Frontend:
Slides:
IRC: freenode #xen-api
Email: john.else@citrix.com
github.com/xapi-project/rrd-transport
github.com/xapi-project/ocaml-rrdd-plugin
johnelse.github.io/xen-summit-2014