How to Speak Intel DPDK KNI for Web Services. 31 Oct, 2013 SAKURA Internet Research Center Senior Researcher / Naoto MATSUMOTO

1. 31 Oct, 2013
SAKURA Internet Research Center
Senior Researcher / Naoto MATSUMOTO

2. Our LAB’s Goal:
To research and deploy about near future technology*
for global IT industory.
*The timescale is between in 3 to 5 years.
Source: SAKURA Internet Research Center 10/2013, Project THORN.

3. Intel® DPDK KNI Overview
1)
2)
3)
4)
Intel® DPDK source codes for linux were released at End of 2012.
6WIND Announces dpdk.org Open-Source Project at April 2013.
Added KNI(Kernel Network Interface) function at Mid 2013.
dpdk-1.5.0r1 was released at Oct 2013.
Web Services
HTTPD
(without App modification)
User
Kernel
Linux Sockets
vEth0_0
Intel DPDK KNI
Intel DPDK (PMD drivers)
Physical NIC(s)
FYI: Disruptive IP Networking with Intel DPDK on Linux @SlideShare
http://www.slideshare.net/naotomatsumoto/disruptive-ip-networking-with-intel-dpdk-on-linux
Source: SAKURA Internet Research Center 10/2013, Project THORN.

4. Buildup Intel® DPDK KNI Env
1) Install Intel® DPDK
# uname –a
Linux 2.6.32-358.6.1.el6.x86_64 #1 SMP Tue Apr 23 16:15:13 CDT 2013 x86_64 x86_64 x86_64 GNU
# cd /opt; tar xzvf ./dpdk-1.5.0r1.tar.gz; cd dpdk-1.5.0r1
# make config T=x86_64-default-linuxapp-gcc
# make install T=x86_64-default-linuxapp-gcc
# make
# rmmod ixgbe
# modprobe uio; insmod build/kmod/igb_uio.ko; insmod build/kmod/rte_kni.ko kthread_mode=multiple
# mkdir -p /hugetlb
# mount -t hugetlbfs nodev /hugetlb
# echo 4096 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
2) Intel ® DPDK PMD Testing
# build/app/testpmd -c7 -n3 -- -i --nb-cores=2 --nb-ports=2
3) Configure Intel® DPDK KNI module
#
#
#
#
export RTE_SDK=/opt/dpdk-1.5.0r1
cd /opt/dpdk-1.5.0r1/examples/kni
make
./build/kni -c 0xf0 -n 4 -- -p 0x3 --config="(0,4,6,8),(1,5,7,9)" &
4) Intel® DPDK KNI Operation
# ifconfig vEth0 192.168.0.1
# ethtool -d vEth0
# tcpdump -i vEth0
Source: SAKURA Internet Research Center 10/2013, Project THORN.

5. Intel® DPDK KNI boot sequence
# ./build/kni -c 0xf0 -n 4 -- -p 0x3 --config="(0,4,6,8),(1,5,7,9)“
EAL:
EAL:
EAL:
:
EAL:
EAL:
EAL:
:
EAL:
EAL:
:
APP:
KNI:
APP:
KNI:
Detected lcore 0 as core 0 on socket 0
Detected lcore 1 as core 1 on socket 0
Detected lcore 2 as core 2 on socket 0
Setting up memory...
Ask a virtual area of 0x2097152 bytes
Virtual area found at 0x7f1d8a000000 (size = 0x200000)
Web Services
HTTPD
TSC frequency is ~3299939 KHz
Master core 4 is ready (tid=8b3bc800)
Initialising port 0 ...
pci: 0c:00:00
8086:10fb
Initialising port 1 ...
pci: 0c:00:01
8086:10fb
Figure 2. Kernel NIC Application Packet Flow, 8.0 Kernel NIC Interface Sample Application
Intel® Data Plane Development Kit (Intel® DPDK) Sample Applications User Guide September 2013
Checking link status
..........................................................................................done
Port 0 Link Up - speed 10000 Mbps - full-duplex
Port 1 Link Down
APP: Lcore 5 is reading from port 1
APP: Lcore 6 is writing to port 0
APP: Lcore 7 is writing to port 1
APP: Lcore 4 is reading from port 0
Source: SAKURA Internet Research Center 10/2013, Project THORN.

6. Intel® DPDK KNI Operation
# ip link show
21: vEth0_0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
22: vEth1_0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
# ip addr add 192.168.0.1/24 dev vEth0_0
# ethtool vEth0_0
Settings for vEth0_0:
Supported ports: [ FIBRE ]
Supported link modes:
10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: No
Advertised link modes: 10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: external
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
:
Link detected: yes
Source: SAKURA Internet Research Center 10/2013, Project THORN.

7. Intel® DPDK KNI Monitoring
# vnstat -l -i vEth0_0
Monitoring vEth0_0...
rx:
(press CTRL-C to stop)
800.90 Mbit/s 1601806 p/s
tx:
0 kbit/s
0 p/s
# tcpdump -i vEth0_0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vEth0_0, link-type EN10MB (Ethernet), capture size 65535 bytes
00:55:46.407878 IP 192.168.0.9.bre > 192.168.0.1.spytechphone: UDP, length 22
00:55:46.407884 IP 192.168.0.9.bre > 192.168.0.1.spytechphone: UDP, length 22
00:55:46.407886 IP 192.168.0.9.bre > 192.168.0.1.spytechphone: UDP, length 22
00:55:46.407887 IP 192.168.0.9.bre > 192.168.0.1.spytechphone: UDP, length 22
00:55:46.407889 IP 192.168.0.9.bre > 192.168.0.1.spytechphone: UDP, length 22
00:55:46.407890 IP 192.168.0.9.bre > 192.168.0.1.spytechphone: UDP, length 22
00:55:46.407891 IP 192.168.0.9.bre > 192.168.0.1.spytechphone: UDP, length 22
00:55:46.407892 IP 192.168.0.9.bre > 192.168.0.1.spytechphone: UDP, length 22
# dropwatch
Initalizing null lookup method
dropwatch> *Does not Effect…
Source: SAKURA Internet Research Center 10/2013, Project THORN.

8. Intel® DPDK KNI perf check
# perf record ./build/kni -c 0xf0 -n 4 -- -p 0x3 --config="(0,4,6,8),(1,5,7,9)“
# perf report
Samples: 662K of event 'cycles', Event count (approx.): 660643513353
41.29% kni kni
[.] kni_allocate_mbufs
30.80% kni kni
[.] main_loop
9.24% kni kni
[.] ixgbe_recv_pkts
8.93% kni kni
[.] rte_delay_us
5.22% kni kni
[.] rte_kni_tx_burst
:
# perf stat ./build/kni -c 0xf0 -n 4 -- -p 0x3 --config="(0,4,6,8),(1,5,7,9)"
^C./build/kni: Interrupt
Performance counter stats for './build/kni -c 0xf0 -n 4 -- -p 0x3 --config=(0,4,6,8),(1,5,7,9)':
123704.616399
274
13
9,996
493,863,586,483
309,423,280,316
164,745,543,981
585,543,402,812
task-clock
context-switches
CPU-migrations
page-faults
cycles
stalled-cycles-frontend
stalled-cycles-backend
instructions
99,577,432,345 branches
116,202,928 branch-misses
43.487826969 seconds time elapsed
#
2.845 CPUs utilized
#
0.002 K/sec
#
0.000 K/sec
#
0.081 K/sec
#
3.992 GHz
#
62.65% frontend cycles idle
#
33.36% backend cycles idle
#
1.19 insns per cycle
#
0.53 stalled cycles per insn
# 804.961 M/sec
#
0.12% of all branches
[83.33%]
[83.33%]
[66.67%]
[83.33%]
[83.33%]
[83.33%]
Source: SAKURA Internet Research Center 10/2013, Project THORN.

9. Intel® DPDK KNI CPU check
# ./build/kni -c 0xf0 -n 4 -- -p 0x3 --config="(0,4,6,8),(1,5,7,9)“ &
# pkt-gen –i ix0 -f tx -l 64 -D 00:0C:BD:00:E8:15 (on Traffic Generator side)
# top
top - 18:31:52 up 20 min, 4 users, load average: 4.17, 3.31, 1.98
Tasks: 337 total,
3 running, 334 sleeping,
0 stopped,
0 zombie
Cpu0 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu8 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu9 : 0.0%us, 97.0%sy, 0.0%ni, 3.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu10 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu11 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32831124k total, 9119152k used, 23711972k free,
18796k buffers
Swap: 7565304k total,
0k used, 7565304k free,
370952k cached
Source: SAKURA Internet Research Center 10/2013, Project THORN.

10. Thanks for your interest.
SAKURA Internet Research Center.
Source: SAKURA Internet Research Center 10/2013, Project THORN.