Getting Linux to 20Mpps

1. eXpress Data Path
Getting Linux to 20Mpps
Brenden Blanco

2. Getting Started with XDP
Pick the minimum viable feature set
Packet read/write, drop/pass/forward, bpf table access
Pick a candidate driver
mlx4 40G (because I had it on hand)
Hack it together

3. Benchmark Setup
Receiver
Xeon E5-1630 @3.
70GHz
Mellanox MT27520
ConnectX-3 Pro
Sender
Xeon E5645 @2.
40GHz
Mellanox MT27520
ConnectX-3 Pro
40G

4. Benchmark Setup
Receiver
Xeon E5-1630 @3.
70GHz
Mellanox MT27520
ConnectX-3 Pro
Sender
Xeon E5645 @2.
40GHz
Mellanox MT27520
ConnectX-3 Pro
40G

5. XDP Benchmarks (single core)
Generated using pktgen - (scripts in $kernelsrc/samples/pktgen)
ip routing drop: ~3.6 Mpps
tc clsact drop using bpf: ~4.2 Mpps
xdp drop using bpf: 20 Mpps (<10 % cpu util)
xdp forward with rewrite: 10 Mpps

6. Specific Challenges in mlx4
Need to find where to run BPF program - Before SKB Allocation
Multiple packets per page
good for memory density, bad for fast-reuse of pages
Pages are dma-mapped as read-only
XDP_TX feature needs lockless access to hardware tx ring
Fast enough packet rate stresses linux page allocator

7. XDP Upstream Status
Initial API done!
Support for mlx4 driver (Mellanox Connect{3, 3-Pro})
Prototypes for mlx5, ixgbe, e1000 for testing
Ready in Linux 4.8
1200 LOC
Available now in http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git

8. Trying XDP For Yourself
https://github.com/iovisor/xdp-vagrant
vagrant up
vagrant ssh
sudo /usr/share/bcc/examples/networking/xdp/xdp_drop_count.py eth1