This document discusses experiments on detecting and handling "elephant" traffic flows that use disproportionate network resources. It tested using Open vSwitch to detect large flows via per-flow statistics and signaling them using DSCP. Cumulus Linux then reacted by lowering drop thresholds or steering flows to alternate queues/paths. Results showed detecting elephants allowed mice flows to complete faster with few drops, while elephants achieved similar bandwidth. Next steps include refining detection and testing different reaction methods.

1. v
Elephants and Mice
vSwitch Elephant Detection and Underlay Elephant
Handling
Justin Pettit, Kanna Rajagopal, JR Rivers and Martin Casado
20140515

2. Of Elephants and Mice
Large data transfers “elephants” can
consume disproportionate network
resources (buffer/queues) which causes
packet drops in other traffic, greatly
increasing the mean-time-to-completion
(mttc) of smaller data transfers “mice”.
 mice mttc disproportionately affect application mttc
cumulusnetworks.com 2

3. Project Goal
Evaluate the effects of predictive signaling
and distributed reaction on the
elephant/mice problem
Detect and signal elephants via Open
vSwitch
React to elephants in HW via Cumulus Linux
cumulusnetworks.com 3

4. Elephant Detection and Signaling
 Detect via…
 per flow statistics (current)
• explicitly measure flow rate over time
 TCP Segmentation Offload buffer size (pending)
• indicates socket is sending large amount of data
 Signal via IP Differentiated Services Code Point (DSCP)
 carried through entire IP fabric
cumulusnetworks.com 4
vswitch
elephants
mice
nic
vm
vm
vm
vnic
vnic
vnic

5. Elephant Reaction - Escalating Affects
Lower drop/wred threshold in a single queue
 diffserv AF behavior
Alternative queue on same physical port
Alternative network path
cumulusnetworks.com 5
?
drop-threshold
queue-select
alternate-path

6. Test Topology
Sources
 VMs connected via Open vSwitch
• 10G connection to network
Network Paths
 1G “normal” link
• easy to congest with VM traffic sources
 10G “alternative” link
Sink
 bare metal server
• 10G connection from network
cumulusnetworks.com 6
1G
Open vSwitch
n
vm
vni
c
10G
10G
10G
vmvni
c
vm
vni
c

7. Traffic Generation and Result Measurement
Generators
 elephants – nuttcp
• fixed time transfers, 4M window
 mice – small (10ms) interval pings
• mimics tcp-acks, lock release, small db transations
Results
 elephants
• realized bandwidth, drops
 mice
• mean-time-to-completion, drops
cumulusnetworks.com 7

8. Results – flow statistic detection & alternate queue reaction
test case
(120 sec period)
elephant mouse
Mbps drops Latency (ms) drops
elephant only 941 63 N/A N/A
mouse only N/A N/A 0.444 0
mouse vs elephant
no detection
941 61 3.055 0
mouse vs elephant
w/detection
937 1223 0.401 0
cumulusnetworks.com 8

9. Results – flow statistic detection & alternate queue reaction
cumulusnetworks.com 9
0
1
2
3
4
5
6
7
8
9
10
500
550
600
650
700
750
800
850
900
950
1000
1 11 21 31 41 51 61 71 81 91 101 111 121 131
Latencyms)
Bandwidth(Mbps)
Time (Secs)
Mice vs Elephants (Detection off)
Elephant
Mice

10. Results – flow statistic detection & alternate queue reaction
cumulusnetworks.com 10
0
1
2
3
4
5
6
7
8
9
10
500
550
600
650
700
750
800
850
900
950
1000
1 11 21 31 41 51 61 71 81 91 101 111 121 131
Latency(ms)
Bandwidth(Mbps)
Time (Secs)
Mice vs Elephants (Detection on, OVS timer off)
Elephant
Mice

11. Next Steps
TSO based detection
 pre-test with Linux kernel patches
 retest with updated Open vSwitch from vmware
Various reactions
 alternate drop threshold
 alternate path
cumulusnetworks.com 11

12. Thank You!
cumulusnetworks.com 12