Disruptive IP Networking with Chelsio 40GbE H/W Offload NIC 09 Dec, 2013 SAKURA Internet Research Center Senior Researcher / Naoto MATSUMOTO

1. 09 Dec, 2013
SAKURA Internet Research Center
Senior Researcher / Naoto MATSUMOTO

2. Enabling ToDo
1) Download from support web
http://service.chelsio.com/
2) Install Drivers
#
#
#
#
#
tar xzvf ./ChelsioUwire-2.8.0.0.tar.gz
cd ChelsioUwire-2.8.0.0
make UNAME_R=2.6.32-358.el6
make UNAME_R=2.6.32-358.el6 install
reboot
© Copyright 2013 Chelsio Communications

3. Linux Kernel module check
# modinfo t4_tom
filename:
/lib/modules/2.6.32-358.6.1.el6.x86_64/updates/kernel/drivers/net/offload/t4_tom/t4_tom.ko
version:
2.8.0.0
license:
GPL
author:
Chelsio Communications
description:
TCP offload module for Chelsio T4/T5-based network cards
srcversion:
677C1016F4208F8F816F0FF
depends:
toecore,ipv6,cxgb4
vermagic:
2.6.32-358.6.1.el6.x86_64 SMP mod_unload modversions
parm:
activated:whether to enable TOE at init time or not (int)
parm:
unsupported_allow_unload:allow UNSUPPORTED unloading of module (int)
parm:
cop_managed_offloading:all connection offloading decision managed by COP (int)
parm:
offload_vlan:Only Offload connections on the indicated VLAN (int)
parm:
tp_multi_chan:allow active opens to use TP c-channel mask (int)
parm:
send_page_order:order of page allocation for sendmsg (int)

4. ethtool check
# ethtool eth1
Settings for eth1:
Supported ports: [ ]
Supported link modes: 40000baseSR4/Full
Supported pause frame use: No
Supports auto-negotiation: No
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: 40000Mb/s
Duplex: Full
Port: Other
PHYAD: 255
Transceiver: external
Auto-negotiation: off
Supports Wake-on: bg
Wake-on: d
Current message level: 0x000000ff (255)
drv probe link timer ifdown ifup rx_err tx_err
Link detected: yes

5. NIC Dirver Tuning w/ script
# /opt/ChelsioUwire-2.8.0.0/tools/t4_perftune.sh
Discovering Chelsio T4/T5 devices ...
Configuring Chelsio T4/T5 devices ...
Tuning eth1
IRQ table length 12
Writing 1 in /proc/irq/66/smp_affinity
Writing 2 in /proc/irq/67/smp_affinity
Writing 4 in /proc/irq/68/smp_affinity
Writing 8 in /proc/irq/69/smp_affinity
Writing 10 in /proc/irq/70/smp_affinity
Writing 20 in /proc/irq/71/smp_affinity
Writing 40 in /proc/irq/72/smp_affinity
Writing 80 in /proc/irq/73/smp_affinity
Writing 100 in /proc/irq/74/smp_affinity
Writing 200 in /proc/irq/75/smp_affinity
Writing 400 in /proc/irq/76/smp_affinity
Writing 800 in /proc/irq/77/smp_affinity
eth1 now up and tuned
:

6. Build Packet Generator /w NETMAP
1) Creat Bootable USB memory for Packet Generator/Receiver
Download Bin image from http://info.iet.unipi.it/~luigi/netmap/
Burn to USB Memory using Win32 Disk Imager for multiple Traffic Generators.
2) Check vnstat on Reciver side (40GbE H/W NIC)
# vnstat -i eth1 -l
Monitoring eth1... (press CTRL-C to stop)
rx:
17.68 Gbit/s 35352503 p/s
tx:
2.54 Mbit/s 5087 p/s
Source: SAKURA Internet Research Center. 11/2012: Project THORN

7. 10/40GbE-NIC Performance check
10/40GbE-NIC 64Byte Short Packet Processing Bechmark (RX)
1.39Gbit/s
Need more Traffic Generators
1.41Gbit/s
2.81Mpps
1.39Gbit/s
2.62Mpps
Corei7-3930K CPU @ 3.20GHz / 32GB DDR3-DIMM / PCI Express 3.0 using linux standard pkt-gen with NETMAP.
Mpps
(Million Packet/sec)
SOURCE: SAKURA Internet Research Center. 12/2013 Project THORN.

8. 10/40/100GbE Networking GOAL

9. Thanks for your interest.
SAKURA Internet Research Center.