Explain direct cache access technology by focusing on implementations and specifications

1. © 2011 VMware Inc. All rights reserved
Revisit DCA, PCIe TPH and DDIO
VMware
Hisaki Ohara
Dec. 2014

2. Terminology
 Direct cache access
• Technology such that CPU can hit cache data of IO device on demand to
2
avoid memory access
 DCA
• Adopted since Xeon 5100 (Woodcrest)
• As a part of Intel IOAT
• OS/driver intervention required
 PCIe TLP Processing Hints (TPH)
• PCI-SIG: PCI Express 2.1 Protocol Extensions
 DDIO
• Adopted since Xeon E5 (SandyBridge)
• Likely to be a subset of PCIe TPH
• No OS/driver intervention required

3. DCA
 Hardware Prefetch on DMA Write
3
• Memory access cannot be avoided
• Need to avoid cache eviction before CPU uses
 DCA flag := CPUID(EAX:1):ECX[18]
• Require driver’s intervention
 TLP format for DCA
82599 datasheet: Fig.7.29
82599 datasheet: Fig.7.30 Available for multiple sockets

4. PCIe TLP Processing Hint (TPH)
 “TPH is a performance feature that allows CPU to prefetch or keep
certain PCIe writeback data in LLC for quick core consumption.”
 Steering Tag (ST) field: Identify target resources
• Three modes to derive ST value
4
• No ST Mode
• Interrupt Vector Mode
• Device Specific Mode

5. 5

6. 6

7. 7

8. PCIe Port after SandyBridge generation
8

9. DDIO: How to enable/disable
9

10. Differences of DDIO from DCA
 DDIO based on PCIe specification
• Not certain the usage of ST on DDIO from the public information
 DDIO accelerate only for local socket, while DCA does not matter
• I don’t think that DCA contributed the performance for remote sockets
10

11. References
 PCI Express Base Specification Revision 3.0
 PCI-SIG Engineering Change Notice (ECN): TLP Processing Hints
• https://www.pcisig.com/specifications/pciexpress/specifications/ECN_TPH_11
11
Sept08.pdf
 IDF 2009: TCIS006: PCI Express 3.0 Technology: Device
Architecture Optimizations on Intel Platforms
 Intel Xeon Processor E5-2600 v2 Datasheet, Vol.2
• http://www.intel.com/content/www/us/en/processors/xeon/xeon-e5-v3-
datasheet-vol-2.html
 Intel Data Direct I/O: Frequently Asked Questions
• http://www.intel.com/content/dam/www/public/us/en/documents/faqs/data-direct-
i-o-faq.pdf