SlideShare a Scribd company logo
1 of 41
Download to read offline
TCP and BBR
Geoff Huston
APNIC
Networking is all about moving
Data
• The way in which data movement is controlled is a key characteristic of
the network architecture
• The Internet Protocol architecture passed all controls to the end
systems, and treated the network as a passive switching environment
– But that was many years ago, and since then we have seen the deployment of
active middleware in networks and control traffic flows
• All this is changing again as we see a what could well be a new
generation of flow control algorithms being adopted in the Internet
• Lets look at one of the more interesting initiatives: Bottleneck
Bandwidth and Round-Trip Time (BBR)
2
Let’s talk about speed
• How fast can we push a single session to move data
through the network?
• Session speed is the result of a combination of:
• available transmission speeds,
• transmission bit error rate,
• end-to-end latency,
• host buffer size and
• protocol efficiency
– All of these factors are critical
3
The Evolution of Speed
1980’s
– TCP rates of Kilobits per second
1990’s
– TCP rates of Megabits per second
2000’s
– TCP rates of Gigabits per second
2010’s
– TCP rates of Gigabits per second
4
80’s 90’s 00’s 10’s
K
M
G
Today
• Optical transmission speeds are approaching Terrabit
capacity
• But peak TCP session speeds are not keeping up
• What’s going on?
5
80’s 90’s 00’s 10’s
K
M
G
T
optical
transport
TCP speed
TCP
• The Transmission Control Protocol is an end-to-end
protocol that creates a reliable stream protocol from the
underlying IP datagram device
• TCP operates as an adaptive rate control protocol that
attempts to operate fairly and efficiently
TCP Design Objectives
To maintain an average flow which is Efficient and Fair
• Efficient:
– Minimise packet loss
– Minimise packet re-ordering
– Do not leave unused path bandwidth on the table!
• Fair:
– Do not crowd out other TCP sessions
– Over time, take an average 1/N of the path capacity when there are N
other TCP sessions sharing the same path
It’s a Flow Control process
• Think of this as a multi-
flow fluid dynamics
problem
• Each flow has to gently
exert pressure on the
other flows to signal
them to provide a fair
share of the network,
and be responsive to
the pressure from all
other flows
TCP Control
TCP is an ACK Pacing protocol
Data sending rate is matched to the
ACK arrival rate
TCP Control
ACK pacing protocols relate to a past network state, not
necessarily the current network state
– The ACK signal shows the rate of data that left the network at the
receiver that occurred at ½ RTT back in time
– So if there is data loss in the forward path, the ACK signal of that loss
is already at least ½ RTT old!
• So TCP should react quickly to ‘bad’ news
– If there is no data loss, that is also old news
• So TCP should react conservatively to ‘good’ news
“Classic TCP” – TCP Reno
• Additive Increase Multiplicative Decrease (AIMD)
– While there is no packet loss, increase the sending rate by one
segment (MSS) each RTT interval
– If there is packet loss decrease the sending rate by 50% over the
next RTT Interval
• Start Up
– Each RTT interval, double the sending rate
– We call this “slow start” – probably because its anything but slow!!!
Idealised TCP Reno
Time
Slow Start
Rate Doubles
each RTT
Interval
Congestion Avoidance
Rate increases by 1 MSS per RTT
Rate halves on Packet Loss
Notification of Packet Loss
via Duplicate ACKs causes
RENO to halve its sending
rate
TCP RENO and Idealized Queue
Behaviour
Total Queue Capacity
(Onset of Packet Loss)
Link Capacity Capacity
(Onset of Queuing)
Network Buffers Fill
Network Buffers Drain
Reno is too slow
• TCP Reno tries to oscillate between sending rates R and 2R that
span the actual link capacity
• It increases its sending rate slowly so it’s really lousy when trying
to run at very high speed over long delay networks
• It over-corrects on loss and leaves available path capacity idle
– 10Gbps rates over 100ms RTT demands a packet loss rate of less than
0.000003%
– An average 1% loss rate over a 100ms RTT can’t operate faster than
3Mbps
Faster than Reno?
• Can we make TCP faster and more efficient by changing
the way in which the sending rate is inflated?
• Of course!
Refinements to RENO
• There have been many efforts to alter RENO’s flow control
algorithm
• In a loss-based AIMD control system the essential parameters
are the manner of rate increase and the manner of loss-based
decrease
– For example:
MulTCP behaves as it it were N simultaneous TCP sessions: i.e. increase by N segments
each RTT and rate drop by 1/N upon packet loss
• What about varying the manner of rate increase away from AI?
Enter CUBIC
• CUBIC is designed to be useful for high speed sessions while still
being ‘fair’ to other sessions and also efficient even at lower speeds
• Rather than probe in a linear manner for the sending rate that triggers
packet loss, CUBIC uses a non-linear (cubic) search algorithm
CUBIC and Queue formation
Total Queue Capacity
(Onset of Packet Loss)
Link Capacity Capacity
(Onset of Queuing)
Network Buffers Fill
Network Buffers Drain
CUBIC assessment
• Can react quickly to available capacity in the network
• Tends to sit for extended periods in the phase of queue
formation
• Can react efficiently to long fat pipes and rapidly scale up
the sending rate
• Operates in a manner that tends to exacerbate ‘buffer bloat’
conditions
Can we do even better?
• Lets look at the model of the network once more, and observe that there
are three ‘states’ of flow management in this network:
– Under-Utilised – where the flow rate is below the link capacity and no queues form
– Over-Utilised – where the flow rate is greater that the link capacity and queues form
– Saturated – where the queue is filled and packet loss occurs
• Loss-based control systems probe upward to the Saturated point, and back
off quickly to what they guess is the Under-Utilised state in order to the let
the queues drain
• But the optimal operational point for any flow is at the point of state change
from Under to Over-utilised, not at the Saturated point
RTT and Delivery Rate with Queuing
Under-Utilised Over-Utilised Saturated
How to detect the onset of
queuing?
• By carefully measuring the Round Trip Time!
BBR Design Principles
• Probe the path capacity only intermittently
• Probe the path capacity by increasing the sending rate for a short
interval and then drop the rate to drain the queue:
– If the RTT of the probe equals the RTT of the previous state then there is
available path bandwidth that could be utilised
– If the RTT of the probe rises then the path is likely to be at the onset of
queuing and no further path bandwidth is available
• Do not alter the path bandwidth estimate in response to packet loss
• Pace the sending packets to avoid the need for network buffer rate
adaptation
Idealised BBR profile
sending rate
network queues
BBR Politeness?
• BBR will probably not constantly pull back when
simultaneous loss-based protocols exert pressure on the
path’s queues
• BBR tries to make minimal demands on the queue size,
and does not rely on a large dynamic range of queue
occupancy during a flow
From Theory to Practice
• Lets use BBR in the wild
• I’m using iperf3 on Linux platforms (Linode)
– The platforms are dedicated to these tests
• It’s the Internet
– The networks paths vary between tests
– The cross traffic is highly variable
– No measurement is repeatable to a fine level of detail
Cubic vs BBR over a 12ms RTT 10G
circuit
Wow!
• That was BRUTAL!
• As soon as BBR started up it collided with CUBIC, and BBR
startup placed pressure on CUBIC such that CUBIC’s
congestion window was reduced close to zero
• At this stage CUBIC’s efforts to restart its congestion
window appear to collide with BBR’s congestion control
model, so CUBIC remains suppressed
– The inference is that BBR appears to be operating in steady state
with an ability to crowd out CUBIC
BBR vs Cubic – second attempt
Same two endpoints, same
network path across the public
Internet
Using a long delay path AU to
Germany via the US
BBR vs Cubic
BBR(1)starts
Cubicstarts
BBR(2)starts
Cubicends
BBR(2)ends
The Internet is capable of
offering a 400Mbps capacity
path on demand!
In this case BBR is apparently
operating with filled queues,
and this crowds out CUBIC
BBR does not compete well
with itself, and the two sessions
oscillate in getting the majority
share of available path capacity
BBR and Loss
Recovery
31
Packet loss causes
retransmission that
appears to occur in
addition to the stable link
capacity model used by
BBR.
Once loss is reduced, BBR
maintains a more
consistent sending model
Parallel
BBR
Streams
32
We used 50 parallel BBR
streams between the
same two endpoints
(200ms RTT)
In this larger setup the
majority of sessions
managed to equilibrate
between each other and
evenly share the path
bandwidth
So what can we say about BBR?
It’s “interesting” in so many ways:
– It’s a move away from the more common loss-based flow control
protocols
– It looks like it will operate very efficiently in a high-speed small-buffer
world
• High speed small buffer switching chips are far cheaper, but loss-based TCP
reacts really badly to small buffers by capping its flow rate
– It will operate efficiently over ECMP paths, as it is relatively impervious to
packet re-ordering
– It also looks as if it will operate efficiently in rate policed environments
– Unlike AIMD systems, it will scale from Kbps to Gbps over long delay
paths very efficiently
– It resists the conventional network-based traffic control mechanisms
Why use BBR?
• Because it achieves
• Its incredibly efficient
• It makes minimal demands on network buffer capacity
• It’s fast!
Why not use BBR?
• Because it over achieves!
• The classic question for many Internet technologies is scaling
– “what if everyone does it?”
– BBR is not a scalable approach in competition with loss-based flows
– It works so well while it is used by just a few users, some of the time
– But when it is active, BBR has the ability to slaughter concurrent
loss-based flows
– Which sends all the wrong signals to the TCP ecosystem
• The loss-based flows convert to BBR to compete on equal terms
• The network is then a BBR vs BBR environment, which is unstable
Is this BBR experiment a failure?
Is it just too ‘greedy’ and too ‘insensitive’ to other flows to be
allowed out on the Internet to play?
– Many networks have been provisioned as a response to the
aggregate behaviours of loss-based TCP congestion control
– BBR changes all those assumptions, and could potentially push
many networks into sustained instability
– We cannot use the conventional network control mechanisms to
regulate BBR flows
• Selective packet drop just won’t create back pressure on the flow
Is BBR an outstanding success?
• We can’t achieve speed if we need also large high speed buffers
in network routers
– Loss-based flow-control systems have a sloppy control loop that is
always ½ RTT late
– Small buffers in the switches exacerbate this problem
• We can use small buffers in switches if we use flow control
systems that are sensitive to the onset of queue formation (rather
than being sensitive to packet loss resulting from a full queue)
• BBR points to an approach that does not require large buffer
pools in switches
37
Where now?
BBR 2.0
– Alter BBR’s ‘sensitivity’ to loss rates, so that it does not persist with an
internal bandwidth delay product (BDP) that exceeds the uncongested BDP
This measure would moderate BBR 1.0’s ability to operate for extended periods with
very high loss levels
– Improve the dynamic sharing fairness by moderating the Bandwidth Delay
Product by using an estimated ‘fair’ proportion of the path BDP
– Accommodate the signal distortion caused by ACK stretching middleware
– Place an upper bound on the volume of in-flight data
– Alter the +/- 25% probe factors dynamically (i.e. allow this to be less than
25% overload)
The new Network Architecture
• We are seeing a shift in end systems to assert edge-centric
control and hide from network-level active middleware in
the Internet
• QUIC and BBR are instances of a recent push back from
the network-level QoS bandwidth control mechanisms, and
result in greater levels of autonomous control being passed
back to the end hosts
• For better or worse!
39
What is all this telling us?
• The Internet still contains a large set of important unsolved
problems
• Moving large data sets over high speed networks requires a
different approach to what we are doing today
• BBR seems to be a step in an interesting direction,
particularly for very high speed networking
• But it still calls for more research and more testing at scale
40
That’s it!
Questions?

More Related Content

What's hot

Distributed stream processing with Apache Kafka
Distributed stream processing with Apache KafkaDistributed stream processing with Apache Kafka
Distributed stream processing with Apache Kafkaconfluent
 
IAB-5039 : MQTT: A Protocol for the Internet of Things (InterConnect 2015)
IAB-5039 : MQTT: A Protocol for the Internet of Things (InterConnect 2015)IAB-5039 : MQTT: A Protocol for the Internet of Things (InterConnect 2015)
IAB-5039 : MQTT: A Protocol for the Internet of Things (InterConnect 2015)PeterNiblett
 
Lecture 6- Deadlocks.pptx
Lecture 6- Deadlocks.pptxLecture 6- Deadlocks.pptx
Lecture 6- Deadlocks.pptxAmanuelmergia
 
CoAP protocol -Internet of Things(iot)
CoAP protocol -Internet of Things(iot)CoAP protocol -Internet of Things(iot)
CoAP protocol -Internet of Things(iot)Sabahat Nowreen Shaik
 
EnOcean Alliance: Energy Harvesting Wireless Sensors to Cloud - Graham Martin
EnOcean Alliance: Energy Harvesting Wireless Sensors to Cloud - Graham MartinEnOcean Alliance: Energy Harvesting Wireless Sensors to Cloud - Graham Martin
EnOcean Alliance: Energy Harvesting Wireless Sensors to Cloud - Graham MartinWithTheBest
 
RIP RTCP RTSP
RIP RTCP RTSPRIP RTCP RTSP
RIP RTCP RTSPDev Heba
 
Mac protocols of adhoc network
Mac protocols of adhoc networkMac protocols of adhoc network
Mac protocols of adhoc networkshashi712
 

What's hot (20)

Distributed stream processing with Apache Kafka
Distributed stream processing with Apache KafkaDistributed stream processing with Apache Kafka
Distributed stream processing with Apache Kafka
 
Framing Protocols
Framing ProtocolsFraming Protocols
Framing Protocols
 
IAB-5039 : MQTT: A Protocol for the Internet of Things (InterConnect 2015)
IAB-5039 : MQTT: A Protocol for the Internet of Things (InterConnect 2015)IAB-5039 : MQTT: A Protocol for the Internet of Things (InterConnect 2015)
IAB-5039 : MQTT: A Protocol for the Internet of Things (InterConnect 2015)
 
Lecture 6- Deadlocks.pptx
Lecture 6- Deadlocks.pptxLecture 6- Deadlocks.pptx
Lecture 6- Deadlocks.pptx
 
CoAP protocol -Internet of Things(iot)
CoAP protocol -Internet of Things(iot)CoAP protocol -Internet of Things(iot)
CoAP protocol -Internet of Things(iot)
 
Real Time Operating Systems
Real Time Operating SystemsReal Time Operating Systems
Real Time Operating Systems
 
AMQP
AMQPAMQP
AMQP
 
Chapter07
Chapter07Chapter07
Chapter07
 
Amqp Basic
Amqp BasicAmqp Basic
Amqp Basic
 
EnOcean Alliance: Energy Harvesting Wireless Sensors to Cloud - Graham Martin
EnOcean Alliance: Energy Harvesting Wireless Sensors to Cloud - Graham MartinEnOcean Alliance: Energy Harvesting Wireless Sensors to Cloud - Graham Martin
EnOcean Alliance: Energy Harvesting Wireless Sensors to Cloud - Graham Martin
 
Ipv4 header
Ipv4 headerIpv4 header
Ipv4 header
 
Congestion control and quality of service
Congestion control and quality of serviceCongestion control and quality of service
Congestion control and quality of service
 
How to choose an RTOS?
How to choose an RTOS?How to choose an RTOS?
How to choose an RTOS?
 
An Introduction to OMNeT++ 5.1
An Introduction to OMNeT++ 5.1An Introduction to OMNeT++ 5.1
An Introduction to OMNeT++ 5.1
 
RIP RTCP RTSP
RIP RTCP RTSPRIP RTCP RTSP
RIP RTCP RTSP
 
Aca2 01 new
Aca2 01 newAca2 01 new
Aca2 01 new
 
Layered Architecture
Layered ArchitectureLayered Architecture
Layered Architecture
 
Token ring
Token ringToken ring
Token ring
 
Tcp
TcpTcp
Tcp
 
Mac protocols of adhoc network
Mac protocols of adhoc networkMac protocols of adhoc network
Mac protocols of adhoc network
 

Similar to AusNOG 2019: TCP and BBR

RIPE 76: TCP and BBR
RIPE 76: TCP and BBRRIPE 76: TCP and BBR
RIPE 76: TCP and BBRAPNIC
 
NZNOG 2020: Buffers, Buffer Bloat and BBR
NZNOG 2020: Buffers, Buffer Bloat and BBRNZNOG 2020: Buffers, Buffer Bloat and BBR
NZNOG 2020: Buffers, Buffer Bloat and BBRAPNIC
 
RIPE 80: Buffers and Protocols
RIPE 80: Buffers and ProtocolsRIPE 80: Buffers and Protocols
RIPE 80: Buffers and ProtocolsAPNIC
 
High performance browser networking ch1,2,3
High performance browser networking ch1,2,3High performance browser networking ch1,2,3
High performance browser networking ch1,2,3Seung-Bum Lee
 
Congestion control
Congestion controlCongestion control
Congestion controlAman Jaiswal
 
integrated and diffrentiated services
 integrated and diffrentiated services integrated and diffrentiated services
integrated and diffrentiated servicesRishabh Gupta
 
Congestion control 1
Congestion control 1Congestion control 1
Congestion control 1Aman Jaiswal
 
CN Module 5 part 2 2022.pdf
CN Module 5 part 2 2022.pdfCN Module 5 part 2 2022.pdf
CN Module 5 part 2 2022.pdfMayankRaj687571
 
Congection control and Internet working
Congection control and Internet workingCongection control and Internet working
Congection control and Internet workingTharuniDiddekunta
 

Similar to AusNOG 2019: TCP and BBR (20)

RIPE 76: TCP and BBR
RIPE 76: TCP and BBRRIPE 76: TCP and BBR
RIPE 76: TCP and BBR
 
NZNOG 2020: Buffers, Buffer Bloat and BBR
NZNOG 2020: Buffers, Buffer Bloat and BBRNZNOG 2020: Buffers, Buffer Bloat and BBR
NZNOG 2020: Buffers, Buffer Bloat and BBR
 
RIPE 80: Buffers and Protocols
RIPE 80: Buffers and ProtocolsRIPE 80: Buffers and Protocols
RIPE 80: Buffers and Protocols
 
QoSintro.PPT
QoSintro.PPTQoSintro.PPT
QoSintro.PPT
 
High performance browser networking ch1,2,3
High performance browser networking ch1,2,3High performance browser networking ch1,2,3
High performance browser networking ch1,2,3
 
lect10_interconnect.ppt
lect10_interconnect.pptlect10_interconnect.ppt
lect10_interconnect.ppt
 
Quality of service
Quality of serviceQuality of service
Quality of service
 
Routing Protocols
Routing ProtocolsRouting Protocols
Routing Protocols
 
Congestion control
Congestion controlCongestion control
Congestion control
 
Qo s 09-integrated and red
Qo s 09-integrated and redQo s 09-integrated and red
Qo s 09-integrated and red
 
integrated and diffrentiated services
 integrated and diffrentiated services integrated and diffrentiated services
integrated and diffrentiated services
 
Congestion control 1
Congestion control 1Congestion control 1
Congestion control 1
 
Lecture set 1
Lecture set 1Lecture set 1
Lecture set 1
 
CN Module 5 part 2 2022.pdf
CN Module 5 part 2 2022.pdfCN Module 5 part 2 2022.pdf
CN Module 5 part 2 2022.pdf
 
9_Network.ppt
9_Network.ppt9_Network.ppt
9_Network.ppt
 
Part9-congestion.pptx
Part9-congestion.pptxPart9-congestion.pptx
Part9-congestion.pptx
 
Congestion Control
Congestion ControlCongestion Control
Congestion Control
 
Lecture05
Lecture05Lecture05
Lecture05
 
Congection control and Internet working
Congection control and Internet workingCongection control and Internet working
Congection control and Internet working
 
Bg4101335337
Bg4101335337Bg4101335337
Bg4101335337
 

More from APNIC

Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...APNIC
 
APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85
APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85
APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85APNIC
 
NANOG 90: 'BGP in 2023' presented by Geoff Huston
NANOG 90: 'BGP in 2023' presented by Geoff HustonNANOG 90: 'BGP in 2023' presented by Geoff Huston
NANOG 90: 'BGP in 2023' presented by Geoff HustonAPNIC
 
DNS-OARC 42: Is the DNS ready for IPv6? presentation by Geoff Huston
DNS-OARC 42: Is the DNS ready for IPv6? presentation by Geoff HustonDNS-OARC 42: Is the DNS ready for IPv6? presentation by Geoff Huston
DNS-OARC 42: Is the DNS ready for IPv6? presentation by Geoff HustonAPNIC
 
APAN 57: APNIC Report at APAN 57, Bangkok, Thailand
APAN 57: APNIC Report at APAN 57, Bangkok, ThailandAPAN 57: APNIC Report at APAN 57, Bangkok, Thailand
APAN 57: APNIC Report at APAN 57, Bangkok, ThailandAPNIC
 
AINTEC 2023: Networking in the Penumbra!
AINTEC 2023: Networking in the Penumbra!AINTEC 2023: Networking in the Penumbra!
AINTEC 2023: Networking in the Penumbra!APNIC
 
CNIRC 2023: Global and Regional IPv6 Deployment 2023
CNIRC 2023: Global and Regional IPv6 Deployment 2023CNIRC 2023: Global and Regional IPv6 Deployment 2023
CNIRC 2023: Global and Regional IPv6 Deployment 2023APNIC
 
AFSIG 2023: APNIC Foundation and support for Internet development
AFSIG 2023: APNIC Foundation and support for Internet developmentAFSIG 2023: APNIC Foundation and support for Internet development
AFSIG 2023: APNIC Foundation and support for Internet developmentAPNIC
 
AFNOG 1: Afghanistan IP Deployment Status
AFNOG 1: Afghanistan IP Deployment StatusAFNOG 1: Afghanistan IP Deployment Status
AFNOG 1: Afghanistan IP Deployment StatusAPNIC
 
AFSIG 2023: Internet routing and addressing
AFSIG 2023: Internet routing and addressingAFSIG 2023: Internet routing and addressing
AFSIG 2023: Internet routing and addressingAPNIC
 
AFSIG 2023: APNIC - Registry & Development
AFSIG 2023: APNIC - Registry & DevelopmentAFSIG 2023: APNIC - Registry & Development
AFSIG 2023: APNIC - Registry & DevelopmentAPNIC
 
Afghanistan IGF 2023: The ABCs and importance of cybersecurity
Afghanistan IGF 2023: The ABCs and importance of cybersecurityAfghanistan IGF 2023: The ABCs and importance of cybersecurity
Afghanistan IGF 2023: The ABCs and importance of cybersecurityAPNIC
 
IDNIC OPM 2023: IPv6 deployment planning and security considerations
IDNIC OPM 2023: IPv6 deployment planning and security considerationsIDNIC OPM 2023: IPv6 deployment planning and security considerations
IDNIC OPM 2023: IPv6 deployment planning and security considerationsAPNIC
 
IDNIC OPM 2023 - Internet Number Registry System
IDNIC OPM 2023 - Internet Number Registry SystemIDNIC OPM 2023 - Internet Number Registry System
IDNIC OPM 2023 - Internet Number Registry SystemAPNIC
 
PacNOG 32: Resource Public Key Infrastructure (RPKI) in 30 minutes or less
PacNOG 32: Resource Public Key Infrastructure (RPKI) in 30 minutes or lessPacNOG 32: Resource Public Key Infrastructure (RPKI) in 30 minutes or less
PacNOG 32: Resource Public Key Infrastructure (RPKI) in 30 minutes or lessAPNIC
 
RIPE 87: On Low Earth Orbit Satellites (LEOs) and Starlink
RIPE 87: On Low Earth Orbit Satellites (LEOs) and StarlinkRIPE 87: On Low Earth Orbit Satellites (LEOs) and Starlink
RIPE 87: On Low Earth Orbit Satellites (LEOs) and StarlinkAPNIC
 
VNNIC Internet Day 2023: On LEOs and Starlink
VNNIC Internet Day 2023: On LEOs and StarlinkVNNIC Internet Day 2023: On LEOs and Starlink
VNNIC Internet Day 2023: On LEOs and StarlinkAPNIC
 
40th TWNIC Open Policy Meeting: APNIC PDP update
40th TWNIC Open Policy Meeting: APNIC PDP update40th TWNIC Open Policy Meeting: APNIC PDP update
40th TWNIC Open Policy Meeting: APNIC PDP updateAPNIC
 
40th TWNIC Open Policy Meeting: A quick look at QUIC
40th TWNIC Open Policy Meeting: A quick look at QUIC40th TWNIC Open Policy Meeting: A quick look at QUIC
40th TWNIC Open Policy Meeting: A quick look at QUICAPNIC
 
40th TWNIC OPM: On LEOs (Low Earth Orbits) and Starlink Download
40th TWNIC OPM: On LEOs (Low Earth Orbits) and Starlink Download40th TWNIC OPM: On LEOs (Low Earth Orbits) and Starlink Download
40th TWNIC OPM: On LEOs (Low Earth Orbits) and Starlink DownloadAPNIC
 

More from APNIC (20)

Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
 
APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85
APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85
APNIC Update and RIR Policies for ccTLDs, presented at APTLD 85
 
NANOG 90: 'BGP in 2023' presented by Geoff Huston
NANOG 90: 'BGP in 2023' presented by Geoff HustonNANOG 90: 'BGP in 2023' presented by Geoff Huston
NANOG 90: 'BGP in 2023' presented by Geoff Huston
 
DNS-OARC 42: Is the DNS ready for IPv6? presentation by Geoff Huston
DNS-OARC 42: Is the DNS ready for IPv6? presentation by Geoff HustonDNS-OARC 42: Is the DNS ready for IPv6? presentation by Geoff Huston
DNS-OARC 42: Is the DNS ready for IPv6? presentation by Geoff Huston
 
APAN 57: APNIC Report at APAN 57, Bangkok, Thailand
APAN 57: APNIC Report at APAN 57, Bangkok, ThailandAPAN 57: APNIC Report at APAN 57, Bangkok, Thailand
APAN 57: APNIC Report at APAN 57, Bangkok, Thailand
 
AINTEC 2023: Networking in the Penumbra!
AINTEC 2023: Networking in the Penumbra!AINTEC 2023: Networking in the Penumbra!
AINTEC 2023: Networking in the Penumbra!
 
CNIRC 2023: Global and Regional IPv6 Deployment 2023
CNIRC 2023: Global and Regional IPv6 Deployment 2023CNIRC 2023: Global and Regional IPv6 Deployment 2023
CNIRC 2023: Global and Regional IPv6 Deployment 2023
 
AFSIG 2023: APNIC Foundation and support for Internet development
AFSIG 2023: APNIC Foundation and support for Internet developmentAFSIG 2023: APNIC Foundation and support for Internet development
AFSIG 2023: APNIC Foundation and support for Internet development
 
AFNOG 1: Afghanistan IP Deployment Status
AFNOG 1: Afghanistan IP Deployment StatusAFNOG 1: Afghanistan IP Deployment Status
AFNOG 1: Afghanistan IP Deployment Status
 
AFSIG 2023: Internet routing and addressing
AFSIG 2023: Internet routing and addressingAFSIG 2023: Internet routing and addressing
AFSIG 2023: Internet routing and addressing
 
AFSIG 2023: APNIC - Registry & Development
AFSIG 2023: APNIC - Registry & DevelopmentAFSIG 2023: APNIC - Registry & Development
AFSIG 2023: APNIC - Registry & Development
 
Afghanistan IGF 2023: The ABCs and importance of cybersecurity
Afghanistan IGF 2023: The ABCs and importance of cybersecurityAfghanistan IGF 2023: The ABCs and importance of cybersecurity
Afghanistan IGF 2023: The ABCs and importance of cybersecurity
 
IDNIC OPM 2023: IPv6 deployment planning and security considerations
IDNIC OPM 2023: IPv6 deployment planning and security considerationsIDNIC OPM 2023: IPv6 deployment planning and security considerations
IDNIC OPM 2023: IPv6 deployment planning and security considerations
 
IDNIC OPM 2023 - Internet Number Registry System
IDNIC OPM 2023 - Internet Number Registry SystemIDNIC OPM 2023 - Internet Number Registry System
IDNIC OPM 2023 - Internet Number Registry System
 
PacNOG 32: Resource Public Key Infrastructure (RPKI) in 30 minutes or less
PacNOG 32: Resource Public Key Infrastructure (RPKI) in 30 minutes or lessPacNOG 32: Resource Public Key Infrastructure (RPKI) in 30 minutes or less
PacNOG 32: Resource Public Key Infrastructure (RPKI) in 30 minutes or less
 
RIPE 87: On Low Earth Orbit Satellites (LEOs) and Starlink
RIPE 87: On Low Earth Orbit Satellites (LEOs) and StarlinkRIPE 87: On Low Earth Orbit Satellites (LEOs) and Starlink
RIPE 87: On Low Earth Orbit Satellites (LEOs) and Starlink
 
VNNIC Internet Day 2023: On LEOs and Starlink
VNNIC Internet Day 2023: On LEOs and StarlinkVNNIC Internet Day 2023: On LEOs and Starlink
VNNIC Internet Day 2023: On LEOs and Starlink
 
40th TWNIC Open Policy Meeting: APNIC PDP update
40th TWNIC Open Policy Meeting: APNIC PDP update40th TWNIC Open Policy Meeting: APNIC PDP update
40th TWNIC Open Policy Meeting: APNIC PDP update
 
40th TWNIC Open Policy Meeting: A quick look at QUIC
40th TWNIC Open Policy Meeting: A quick look at QUIC40th TWNIC Open Policy Meeting: A quick look at QUIC
40th TWNIC Open Policy Meeting: A quick look at QUIC
 
40th TWNIC OPM: On LEOs (Low Earth Orbits) and Starlink Download
40th TWNIC OPM: On LEOs (Low Earth Orbits) and Starlink Download40th TWNIC OPM: On LEOs (Low Earth Orbits) and Starlink Download
40th TWNIC OPM: On LEOs (Low Earth Orbits) and Starlink Download
 

Recently uploaded

WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024Jan Löffler
 
Computer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a WebsiteComputer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a WebsiteMavein
 
Bio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptxBio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptxnaveenithkrishnan
 
Discussing Potential of Submarine Cables Causing Internet Blackout in Ghana
Discussing Potential of Submarine Cables Causing Internet Blackout in GhanaDiscussing Potential of Submarine Cables Causing Internet Blackout in Ghana
Discussing Potential of Submarine Cables Causing Internet Blackout in GhanaDesmond Israel
 
Test Automation with Gen AI_Final_Presentation
Test Automation with Gen AI_Final_PresentationTest Automation with Gen AI_Final_Presentation
Test Automation with Gen AI_Final_PresentationUiPathCommunity
 
Aligning Testing Objectives with Overall Project Goals for Successful Outcome...
Aligning Testing Objectives with Overall Project Goals for Successful Outcome...Aligning Testing Objectives with Overall Project Goals for Successful Outcome...
Aligning Testing Objectives with Overall Project Goals for Successful Outcome...Anju21552
 
Internet 2.0 Conference (Event Information Deck | Dec'24 - Mar'25)
Internet 2.0 Conference (Event Information Deck | Dec'24 - Mar'25)Internet 2.0 Conference (Event Information Deck | Dec'24 - Mar'25)
Internet 2.0 Conference (Event Information Deck | Dec'24 - Mar'25)Internet 2.0 Conference
 
Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024Shubham Pant
 
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDSTYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDSedrianrheine
 
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdfLESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdfmchristianalwyn
 
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASSLESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASSlesteraporado16
 
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced HorizonsVision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced HorizonsRoxana Stingu
 

Recently uploaded (12)

WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
 
Computer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a WebsiteComputer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a Website
 
Bio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptxBio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptx
 
Discussing Potential of Submarine Cables Causing Internet Blackout in Ghana
Discussing Potential of Submarine Cables Causing Internet Blackout in GhanaDiscussing Potential of Submarine Cables Causing Internet Blackout in Ghana
Discussing Potential of Submarine Cables Causing Internet Blackout in Ghana
 
Test Automation with Gen AI_Final_Presentation
Test Automation with Gen AI_Final_PresentationTest Automation with Gen AI_Final_Presentation
Test Automation with Gen AI_Final_Presentation
 
Aligning Testing Objectives with Overall Project Goals for Successful Outcome...
Aligning Testing Objectives with Overall Project Goals for Successful Outcome...Aligning Testing Objectives with Overall Project Goals for Successful Outcome...
Aligning Testing Objectives with Overall Project Goals for Successful Outcome...
 
Internet 2.0 Conference (Event Information Deck | Dec'24 - Mar'25)
Internet 2.0 Conference (Event Information Deck | Dec'24 - Mar'25)Internet 2.0 Conference (Event Information Deck | Dec'24 - Mar'25)
Internet 2.0 Conference (Event Information Deck | Dec'24 - Mar'25)
 
Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024
 
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDSTYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
 
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdfLESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
 
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASSLESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
 
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced HorizonsVision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
 

AusNOG 2019: TCP and BBR

  • 1. TCP and BBR Geoff Huston APNIC
  • 2. Networking is all about moving Data • The way in which data movement is controlled is a key characteristic of the network architecture • The Internet Protocol architecture passed all controls to the end systems, and treated the network as a passive switching environment – But that was many years ago, and since then we have seen the deployment of active middleware in networks and control traffic flows • All this is changing again as we see a what could well be a new generation of flow control algorithms being adopted in the Internet • Lets look at one of the more interesting initiatives: Bottleneck Bandwidth and Round-Trip Time (BBR) 2
  • 3. Let’s talk about speed • How fast can we push a single session to move data through the network? • Session speed is the result of a combination of: • available transmission speeds, • transmission bit error rate, • end-to-end latency, • host buffer size and • protocol efficiency – All of these factors are critical 3
  • 4. The Evolution of Speed 1980’s – TCP rates of Kilobits per second 1990’s – TCP rates of Megabits per second 2000’s – TCP rates of Gigabits per second 2010’s – TCP rates of Gigabits per second 4 80’s 90’s 00’s 10’s K M G
  • 5. Today • Optical transmission speeds are approaching Terrabit capacity • But peak TCP session speeds are not keeping up • What’s going on? 5 80’s 90’s 00’s 10’s K M G T optical transport TCP speed
  • 6. TCP • The Transmission Control Protocol is an end-to-end protocol that creates a reliable stream protocol from the underlying IP datagram device • TCP operates as an adaptive rate control protocol that attempts to operate fairly and efficiently
  • 7. TCP Design Objectives To maintain an average flow which is Efficient and Fair • Efficient: – Minimise packet loss – Minimise packet re-ordering – Do not leave unused path bandwidth on the table! • Fair: – Do not crowd out other TCP sessions – Over time, take an average 1/N of the path capacity when there are N other TCP sessions sharing the same path
  • 8. It’s a Flow Control process • Think of this as a multi- flow fluid dynamics problem • Each flow has to gently exert pressure on the other flows to signal them to provide a fair share of the network, and be responsive to the pressure from all other flows
  • 9. TCP Control TCP is an ACK Pacing protocol Data sending rate is matched to the ACK arrival rate
  • 10. TCP Control ACK pacing protocols relate to a past network state, not necessarily the current network state – The ACK signal shows the rate of data that left the network at the receiver that occurred at ½ RTT back in time – So if there is data loss in the forward path, the ACK signal of that loss is already at least ½ RTT old! • So TCP should react quickly to ‘bad’ news – If there is no data loss, that is also old news • So TCP should react conservatively to ‘good’ news
  • 11. “Classic TCP” – TCP Reno • Additive Increase Multiplicative Decrease (AIMD) – While there is no packet loss, increase the sending rate by one segment (MSS) each RTT interval – If there is packet loss decrease the sending rate by 50% over the next RTT Interval • Start Up – Each RTT interval, double the sending rate – We call this “slow start” – probably because its anything but slow!!!
  • 12. Idealised TCP Reno Time Slow Start Rate Doubles each RTT Interval Congestion Avoidance Rate increases by 1 MSS per RTT Rate halves on Packet Loss Notification of Packet Loss via Duplicate ACKs causes RENO to halve its sending rate
  • 13. TCP RENO and Idealized Queue Behaviour Total Queue Capacity (Onset of Packet Loss) Link Capacity Capacity (Onset of Queuing) Network Buffers Fill Network Buffers Drain
  • 14. Reno is too slow • TCP Reno tries to oscillate between sending rates R and 2R that span the actual link capacity • It increases its sending rate slowly so it’s really lousy when trying to run at very high speed over long delay networks • It over-corrects on loss and leaves available path capacity idle – 10Gbps rates over 100ms RTT demands a packet loss rate of less than 0.000003% – An average 1% loss rate over a 100ms RTT can’t operate faster than 3Mbps
  • 15. Faster than Reno? • Can we make TCP faster and more efficient by changing the way in which the sending rate is inflated? • Of course!
  • 16. Refinements to RENO • There have been many efforts to alter RENO’s flow control algorithm • In a loss-based AIMD control system the essential parameters are the manner of rate increase and the manner of loss-based decrease – For example: MulTCP behaves as it it were N simultaneous TCP sessions: i.e. increase by N segments each RTT and rate drop by 1/N upon packet loss • What about varying the manner of rate increase away from AI?
  • 17. Enter CUBIC • CUBIC is designed to be useful for high speed sessions while still being ‘fair’ to other sessions and also efficient even at lower speeds • Rather than probe in a linear manner for the sending rate that triggers packet loss, CUBIC uses a non-linear (cubic) search algorithm
  • 18. CUBIC and Queue formation Total Queue Capacity (Onset of Packet Loss) Link Capacity Capacity (Onset of Queuing) Network Buffers Fill Network Buffers Drain
  • 19. CUBIC assessment • Can react quickly to available capacity in the network • Tends to sit for extended periods in the phase of queue formation • Can react efficiently to long fat pipes and rapidly scale up the sending rate • Operates in a manner that tends to exacerbate ‘buffer bloat’ conditions
  • 20. Can we do even better? • Lets look at the model of the network once more, and observe that there are three ‘states’ of flow management in this network: – Under-Utilised – where the flow rate is below the link capacity and no queues form – Over-Utilised – where the flow rate is greater that the link capacity and queues form – Saturated – where the queue is filled and packet loss occurs • Loss-based control systems probe upward to the Saturated point, and back off quickly to what they guess is the Under-Utilised state in order to the let the queues drain • But the optimal operational point for any flow is at the point of state change from Under to Over-utilised, not at the Saturated point
  • 21. RTT and Delivery Rate with Queuing Under-Utilised Over-Utilised Saturated
  • 22. How to detect the onset of queuing? • By carefully measuring the Round Trip Time!
  • 23. BBR Design Principles • Probe the path capacity only intermittently • Probe the path capacity by increasing the sending rate for a short interval and then drop the rate to drain the queue: – If the RTT of the probe equals the RTT of the previous state then there is available path bandwidth that could be utilised – If the RTT of the probe rises then the path is likely to be at the onset of queuing and no further path bandwidth is available • Do not alter the path bandwidth estimate in response to packet loss • Pace the sending packets to avoid the need for network buffer rate adaptation
  • 24. Idealised BBR profile sending rate network queues
  • 25. BBR Politeness? • BBR will probably not constantly pull back when simultaneous loss-based protocols exert pressure on the path’s queues • BBR tries to make minimal demands on the queue size, and does not rely on a large dynamic range of queue occupancy during a flow
  • 26. From Theory to Practice • Lets use BBR in the wild • I’m using iperf3 on Linux platforms (Linode) – The platforms are dedicated to these tests • It’s the Internet – The networks paths vary between tests – The cross traffic is highly variable – No measurement is repeatable to a fine level of detail
  • 27. Cubic vs BBR over a 12ms RTT 10G circuit
  • 28. Wow! • That was BRUTAL! • As soon as BBR started up it collided with CUBIC, and BBR startup placed pressure on CUBIC such that CUBIC’s congestion window was reduced close to zero • At this stage CUBIC’s efforts to restart its congestion window appear to collide with BBR’s congestion control model, so CUBIC remains suppressed – The inference is that BBR appears to be operating in steady state with an ability to crowd out CUBIC
  • 29. BBR vs Cubic – second attempt Same two endpoints, same network path across the public Internet Using a long delay path AU to Germany via the US
  • 30. BBR vs Cubic BBR(1)starts Cubicstarts BBR(2)starts Cubicends BBR(2)ends The Internet is capable of offering a 400Mbps capacity path on demand! In this case BBR is apparently operating with filled queues, and this crowds out CUBIC BBR does not compete well with itself, and the two sessions oscillate in getting the majority share of available path capacity
  • 31. BBR and Loss Recovery 31 Packet loss causes retransmission that appears to occur in addition to the stable link capacity model used by BBR. Once loss is reduced, BBR maintains a more consistent sending model
  • 32. Parallel BBR Streams 32 We used 50 parallel BBR streams between the same two endpoints (200ms RTT) In this larger setup the majority of sessions managed to equilibrate between each other and evenly share the path bandwidth
  • 33. So what can we say about BBR? It’s “interesting” in so many ways: – It’s a move away from the more common loss-based flow control protocols – It looks like it will operate very efficiently in a high-speed small-buffer world • High speed small buffer switching chips are far cheaper, but loss-based TCP reacts really badly to small buffers by capping its flow rate – It will operate efficiently over ECMP paths, as it is relatively impervious to packet re-ordering – It also looks as if it will operate efficiently in rate policed environments – Unlike AIMD systems, it will scale from Kbps to Gbps over long delay paths very efficiently – It resists the conventional network-based traffic control mechanisms
  • 34. Why use BBR? • Because it achieves • Its incredibly efficient • It makes minimal demands on network buffer capacity • It’s fast!
  • 35. Why not use BBR? • Because it over achieves! • The classic question for many Internet technologies is scaling – “what if everyone does it?” – BBR is not a scalable approach in competition with loss-based flows – It works so well while it is used by just a few users, some of the time – But when it is active, BBR has the ability to slaughter concurrent loss-based flows – Which sends all the wrong signals to the TCP ecosystem • The loss-based flows convert to BBR to compete on equal terms • The network is then a BBR vs BBR environment, which is unstable
  • 36. Is this BBR experiment a failure? Is it just too ‘greedy’ and too ‘insensitive’ to other flows to be allowed out on the Internet to play? – Many networks have been provisioned as a response to the aggregate behaviours of loss-based TCP congestion control – BBR changes all those assumptions, and could potentially push many networks into sustained instability – We cannot use the conventional network control mechanisms to regulate BBR flows • Selective packet drop just won’t create back pressure on the flow
  • 37. Is BBR an outstanding success? • We can’t achieve speed if we need also large high speed buffers in network routers – Loss-based flow-control systems have a sloppy control loop that is always ½ RTT late – Small buffers in the switches exacerbate this problem • We can use small buffers in switches if we use flow control systems that are sensitive to the onset of queue formation (rather than being sensitive to packet loss resulting from a full queue) • BBR points to an approach that does not require large buffer pools in switches 37
  • 38. Where now? BBR 2.0 – Alter BBR’s ‘sensitivity’ to loss rates, so that it does not persist with an internal bandwidth delay product (BDP) that exceeds the uncongested BDP This measure would moderate BBR 1.0’s ability to operate for extended periods with very high loss levels – Improve the dynamic sharing fairness by moderating the Bandwidth Delay Product by using an estimated ‘fair’ proportion of the path BDP – Accommodate the signal distortion caused by ACK stretching middleware – Place an upper bound on the volume of in-flight data – Alter the +/- 25% probe factors dynamically (i.e. allow this to be less than 25% overload)
  • 39. The new Network Architecture • We are seeing a shift in end systems to assert edge-centric control and hide from network-level active middleware in the Internet • QUIC and BBR are instances of a recent push back from the network-level QoS bandwidth control mechanisms, and result in greater levels of autonomous control being passed back to the end hosts • For better or worse! 39
  • 40. What is all this telling us? • The Internet still contains a large set of important unsolved problems • Moving large data sets over high speed networks requires a different approach to what we are doing today • BBR seems to be a step in an interesting direction, particularly for very high speed networking • But it still calls for more research and more testing at scale 40