Jumbo frames allow for larger packet sizes than the standard Ethernet maximum transmission unit (MTU) of 1500 bytes. In OpenStack, enabling jumbo frames involves configuring the global, physical, and logical network MTUs. A use case demonstrates attaching a volume to an instance with jumbo frame support, which reduces traffic and potential packet drops compared to standard MTU sizes. Path MTU discovery allows dynamically determining the optimal MTU for a network path to avoid fragmentation.

1. Jumbo
Mumbo in
OpenStack
Vikram Hosakote
Cisco Systems
vhosakot@cisco.com
OpenStack Summit
Barcelona

2. Agenda
• What is MTU?
• IP Fragmentation
• Jumbo Frames in OpenStack
• Use Case
• Path MTU Discovery
• Advantages & Disadvantages of Jumbo Frames
• Q&A

3. What is MTU?

4. • MTU (Maximum Transmission Unit) is the size (in bytes) of the largest packet that can be
transferred by an interface without IP fragmentation
• MTU is
• Fixed before transmitting data (Ethernet)
• Negotiated during handshake/connect time (point-to-point serial links)
• Dynamically determined on-the-fly while transmitting data
• Default MTU is 1500 bytes
• Jumbo MTU is 9000 bytes
• What is MTU?

5. • What is MTU?

6. IP Fragmentation

7. • “IP fragmentation is an process that breaks datagrams into smaller pieces (fragments), so that
packets may be formed that can pass through a link with a smaller maximum transmission unit (MTU)
than the original datagram size. The fragments are reassembled by the receiving host.”
• https://en.wikipedia.org/wiki/IP_fragmentation
• Adds additional processing overhead on the interface
• IP Fragmentation

8. • IP Fragmentation (DF bit in IP header)

9. Jumbo Frames in OpenStack

10. In neutron.conf
[DEFAULT]
global_physnet_mtu = 9000
advertise_mtu = true
In openvswitch_agent.ini
[ovs]
bridge_mappings = provider1:eth1,provider2:eth2,provider3:eth3
In ml2_conf.ini
[ml2]
physical_network_mtus = provider2:4000,provider3:1500
path_mtu = 9000
• Jumbo Frames in OpenStack

11. • Enable the DHCP MTU option (26) in /etc/neutron/dnsmasq-neutron.conf.
• dhcp-option-force=26,1454
• Restart dnsmasq
• MTU values only apply to new network resources
• network_device_mtu in nova.conf is deprecated in Juno
• Jumbo Frames in OpenStack

12. Use Case

13. • Boot a nova instance
• Create a cinder volume using ceph backend
• Attach volume to instance and mount it
• SSH into instance and write lots of data in this volume
• Measure traffic drop on ceph nodes
• Use Case

16. [mc-compute-1 ~]# ip a | grep mtu
2: mx: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000
3: t: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000
4: p: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000
7: br-inst: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN
8: br-prov: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN
9: br-int: <BROADCAST,MULTICAST> mtu 9000 qdisc noop state DOWN
10: phy-br-inst@int-br-inst: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000
11: int-br-inst@phy-br-inst: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000
12: phy-br-prov@int-br-prov: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000
13: int-br-prov@phy-br-prov: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000
MC-UCS-FI-A(nxos)# show queuing interface | grep MTU
q-size: 360640, HW MTU: 9216 (9216 configured)

17. [mc-block-1 ~]# ping -M do -s 8972 10.7.8.2
PING 10.7.8.2 (10.7.8.2) 8972(9000) bytes of data.
8980 bytes from 10.7.8.2: icmp_seq=1 ttl=64 time=0.118 ms
8980 bytes from 10.7.8.2: icmp_seq=2 ttl=64 time=0.066 ms
--- 10.7.8.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.062/0.082/0.118/0.022 ms
-M used to select Path MTU Discovery strategy. May be either do (prohibit
fragmentation, even local one), want (do PMTU discovery, fragment locally when packet
size is large), or dont (do not set DF flag)

18. Path MTU Discovery

19. • Path MTU Discovery (PMTUD) is a technique to determining the MTU of the network path between
source and destination.
• PMTUD works by setting the Don't Fragment (DF) flag bit in the IP headers of outgoing packets.
Then, any device along the path whose MTU is smaller than the packet will drop it, and send back
an Internet Control Message Protocol (ICMP) Fragmentation Needed (Type 3, Code 4) message
containing its MTU, allowing the source host to reduce its Path MTU appropriately.
• The process is repeated until the MTU is small enough to traverse the entire path without
fragmentation.
• Path MTU Discovery

20. Advantages & Disadvantages

21. • Advantages
• Greater efficiency
• Lesser packet drop
• Interfaces process fewer packets
• Disadvantages
• Affects per-packet processing
• Bigger packets on the wire may increase lag and latency
• Packet corruption causes huge retransmits and congestion

22. Q & A