1.
Shaowen Ma, shaowen@google.com
Group Product Manager
June 2022
Cloud IPv6
Innovation
“Without networking, there is no cloud.”

2.
Agenda
Underlay B2/B4/Jupiter/Andromeda
IPv6 ULA and VPC Design
Cloud Migrate to IPv6
DNS, Interconnect, LB,
IPv6 K8S and Hybrid Cloud
IPv6 Cloud Products
Many Cloud IPv6 Innovations
Summary

3.
Global IPv6 adoption
China
IPv6 adoption 20.6%
33 20.6% China
https://www.google.com/intl/en/ipv6/statistics.html

4.
Why IPv6? IoT+5G
By 2025, over 4.5 billion
IoT connections, of which
majority will be 5G
4.5
billion
Smart City
Video Surveillance
Smart Lighting
Waste Management
Driver Tracking
ADAS
Healthcare
Remote Monitoring
Virtual Care
Ambulance Health
COVID Response Site
Connectivity
Stadium / Campus
In-Building
Wind / Solar farms
AR/VR
Gaming / Sporting
Search / Ads
Remote training
Connected workforce
5G | WiFi | Fixed
2^128=3.4x10^38
2^32=4.2B(10^9)
Sources: Omdia, Gartner, Ericsson
IPv4 IPv6

5.
Cloud Global Network, IPv6 Underlay
Private Internet for Public Cloud
PLCN (HK, LA) 2019
Faster (US, JP, TW) 2016
Unity (US, JP) 2010
Dunant (US, FR) 2020
Monet (US, BR) 2017
Junior (Rio, Santos) 2018
Tannat (BR, UY, AR) 2018
SJC (JP, HK, SG) 2013
Indigo (SG, ID, AU) 2019
HK-G (HK, GU) 2020
JGA (AU, GU) 2019
Curie (CL, US) 2019
Havfrue (US, IE, DK) 2019
Equiano (PT, NG, ZA) 2021
Edge points
of presence
CDN nodes
Munich
New York
Paris
Hamburg
Chicago
Mumbai
Singapore
Kuala Lumpur
Sydney
Tokyo
Chennai
Taipei
Seattle
San Francisco
Montréal
Zurich
Madrid
London
Hong
Kong
Osaka
Toronto
Los Angeles
Denver
Dallas
Miami
Atlanta
Washington DC
Rio de Janeiro
São Paulo
Buenos Aires
Milan
Marseille
Amsterdam
Stockholm
Frankfurt
Dedicated Interconnect
Current regions
and number of zones
Future regions
and number of zones
Mumbai
Singapore
Jakarta
Sydney
Tokyo
Osaka
Hong Kong Taiwan
Oregon
Los Angeles
São Paulo
Finland
Frankfurt
Zurich
Seoul
3
3
3
3
3
3 3
3
3 3
3
3
3
3
4
3
3
Iowa
S. Carolina
N. Virginia
Montréal 3
Belgium
London
Netherlands
3
3
3
Salt Lake City
3 3
Las Vegas
3
Network
29
REGIONS
AVAILABLE IN
200+
COUNTRIES AND TERRITORIES
88
ZONES
146
EDGE LOCATIONS
Internet
Traffic
Scale

6.
Cloud SubSea network overview

7.
Subsea cables Worldwide
Echo in 2023
Echo will run from Eureka, California to Singapore, with a
stop-over in Guam, with plans to also land in Indonesia.
Additional landings are possible in the future
Grace Hopper in 2022
16 fiber pairs (32 fibers), a significant upgrade to the
internet infrastructure connecting the U.S. with Europe.
Equiano in 2021
subsea cable from Portugal to South Africa
Curie in 2020
Equipped with four 18 Tbps fiber-optic pairs and running
from the United States to Chile, Curie is 10,500
kilometers long and delivers 72 Tbps of much needed
bandwidth to South America
Fiber Capacity
16x400Gx48 channel = 307T
Single Fiber pair 18T~19T

8.
Proprietary + Confidential
A snapshot for IPv6 Innovation
Google innovations in networking
2006
Google
Global
Cache
2008
Watchtower
Freedome
Maglev
Software Defined
Load Balancer
BwE
B4
Software
Defined
WAN
2010
Onix
2012
Jupiter
Software Defined
Data Centers
2014
Andromeda
Software Defined
Network
Virtualization
2015
QUIC
2017
Espresso
Software
Defined Edge
2020
SWIFT
Cloud RDMA
2021
Orion
Software
Defined Network
Control Plane
2016
gRPC
TCP-BBR
Aquila
A unified,
low-latency
fabric
2022

9.
(50th percentile)
Modern Protocols: QUIC RFC-9000

10.
Proprietary + Confidential
Reducing latency with BBR
TCP before BBR
● Packet Loss = Network Congestion
● Resulting in slow down
TCP BBR
● Respond to actual congestion not packet loss
● Consider how fast the network is delivering data

11.
Proprietary + Confidential
Being Fast is a Feature
Quic:RFC 9000 HTTP3: RFC9114
● TLS 1.3 - reduces the handshake round-trips required (3 → 2) for
cold connections, which directly improves performance. We’re
the only CDN with TLS 1.3 on by default for all customers.
● gQUIC - a UDP-based transport protocol used at Google.
Supported widely in Chrome & Firefox, with client-side support
opt-in on Android, and via Cronet SDK on iOS and other
Android-based platforms. We’ve supported this for years, and
customers like Snap see significant benefits to their users.
● IETF QUIC + HTTP/3 - IETF standardization of QUIC (L4
transport protocol) and HTTP/3 (HTTP/2 over IETF QUIC).
client-side support will be much wider as a result of
standardization - already in iOS14, Chrome M85, Firefox and
libcurl.

12.
Google Confidential and Proprietary

13.
GPU Direct
Cloud RDMA
Edge Fabric
Espresso
Google
Edge POP
Google
Edge POP
Google
Edge POP
Global
Backbone
Network
B4 DCI
B2
WAN
Cloud Load
Balancing
Cloud
CDN
ISP
Cloud Global Network IPv6 Transformation
Data Center Cluster
K8S/Storage/GPU
Overlay Services
IPv6-only
IPv6 Only, 6to4
IPinIP, RoCEv2
Global Backbone, B2/B4
Edge Pop/LB/CDN/DNS
MPLS/SR/
6to4/Dual Stack
MPLSoGRE IPv6
SD-WAN/Interconnect IPSec BGP IPv6
Applications,
Search/Gmail/Youtube etc IPv4/IPv6

14.
Proprietary + Confidential
Key IPv6 Benefit: No NAT, no Performance downgrade
NAT44
Common
VPC design
NAT64/46
IPv6 to IPv4 translation
NAT66
No NAT for IPv6 VPC
🔼 No more NAT (Network Address Translation)
🔼 Auto-configuration
🔼 No more private address collisions
🔼 Better multicast routing
🔼 Simpler header format
🔼 Simplified, more efficient routing
🔼 True quality of service (QoS), also called "flow labeling"
🔼 Built-in authentication and privacy support
🔼 Flexible options and extensions
No NAT

15.
Proprietary + Confidential for Charter Communications
Cloud IPv6 Network Products
Internet
Access
private
Only
Cloud
Storage Private Cloud
On-Premise
Cloud
CDN
Cloud Load
Balancing
Cloud
DNS
Cloud
Armor
Firewall
Cloud
VPN
Cloud
Interconnect
Admin
User

16.
Proprietary + Confidential for Charter Communications
IPv6 VPC Still need Private Address(ULA)!
Internet
Access
private
Only
Cloud
Storage
Load
Balancer
Private Cloud
On-Premise
● Frontend: Only less 1%( around 100 VMs) need public IPv6 GUA(global
unicast Address) address.
● Backend: 99% VM(few 15K+) only ask for Internal/ULA address, include
Database, Storage. customer don’t want to expose those application to
internet. Prefer ULA address, can leverage BYOIP GUA for now.
● All VM on GCP and On-premise may need talk to each others. Private cloud
already use ULA address, can allocate ULA block for GCP

17.
Proprietary + Confidential for Charter Communications
IPv6 Addressing, GUA/ULA
Global unicast Address/Unique Local Addressing
fd20::/20
ULA IP block
/48
Subnet
VPC(working
in Progress)
/64 /96
VM
GUA IP block
/48
Subnet
VPC
/64 /96
VM
2600:1900:/28

18.
Proprietary + Confidential for Charter Communications
IPv6 NAT66? (Other Cloud Provider Solution)
● All VM only have single internal IP address( possible ULA)
● Ingress: User to Cloud traffic use Anycast LB, already
address translation(similar with NAT)
● Egress: VM use ULA as Src IP, and Andromeda replace
Src address with GUA, 1:1 NAT, GUA not visible to VM.
● Easier to connect to On-Premise Private Cloud ULA via
Private BGP peering.
● IPv4 public cloud providers first use Public IPv4, then they
facing security and shortage of IPv4, later all cloud
provider changed to only allocate RFC1918 to VM, and
NAT on vRouter.
● Only provide ULA on VM, NAT on virtual Router
Internet
Access
Internal
Only
Cloud
Storage
Load
Balancer
Application/Service
IPv6
GUA
IPv6
ULA

19.
Proprietary + Confidential for Charter Communications
Dual NIC solution with VPN and Internet Access
Internet
Access
VPC
Private
Only VPC
Application/Service
Dest : EA
Dest : IA
Dest : EA
Src : EA
Dest : IA
Src : IA
Src : EA
• EA: external Address, GUA
• IA: internal Address, ULA or GUA
IPv6 Google GUA
NIC1
Private Cloud
On-Premise
Interconnect
Access
Public Internet
Access
Src : IA
IPv6 ULA/BYOIP
NIC0
LB

20.
Global LB for IPv6 and IPv4 clients
GCP Datacenters
Region us-east
myapp.com
Compute Engine
Autoscaling
Region us-central
myapp.com
Compute Engine
Autoscaling
Cloud Load
Balancing
Cloud
DNS
User in Iowa
2001:db8::1
Google
Edge POP
Google
Network
10.10.20.0/24
10.10.10.0/24
IPv6 LB: 2600:1901::1
User in New York
100.1.1.1
IPv4 LB: 200.1.1.1
1
2
1
2
1
1
2
IPv6 clients path
IPv4 clients path
2
1 2
www.myapp.com
200.1.1.1
2600:1901::1
IPv6 Forwarding rules(FRs) i.e. VIPs are free, customers
only pay for IPv4 FRs. Rest of the pricing is same.

21.
Cloud GW IPv6 Support
2001:2345::/64
Cloud network Customer network
Google
Peering Fabric
Customer
peering router
Computing
services
Cloud
Router
External data center
HQ
Now support IPv4 BGP session with IPv6 AF.
Later can support IPv6 BGP session with IPv6 AF
Colocation facility
2001:DB8::2/125
2001:3456::/64
2001:1234:1::/64
2001:DB8::1/125
BGP v4
GR/BFD
Interconnect
GW
route
OS V1
OS V2
IPv6 BGP
2001:1234:2::/64

22.
Kubernetes and IPv6 Networking
What fundamentals are needed to be considered an enterprise
container platform?
● Overlay networking and Pod reachability
● Integrated load balancing (L4, L7, TLS, metrics)
● Egress
● In-cluster firewalling & security policy
Areas of differentiation
● Multi-cluster networking and security
● Hybrid networking & automation
● In-cluster traffic observability
internet

23.
Scalable & Dynamic
Advanced Traffic Management
Scale from 0 to N with traffic-based autoscaling. Distribute global
traffic based on Service capacity and health, allowing you to
serve traffic
Scalable IP addressing
IPv6 support and flexible IP address management through
multi-Pod CIDR allows you to run more apps with less IPs.
15,000 Node Clusters
GKE supports the highest scale clusters through networking
innovations such as EndpointSlices, container-native load
balancing, and scalable IP addressing.
Global Service Discovery
GKE supports the highest scale clusters through networking
innovations such as EndpointSlices, container-native load
balancing, and scalable IP addressing.
Scale up to 15,000 nodes per cluster
Scale from 1 to N pods dynamically based on
traffic, CPU, custom metrics, and predictions
Scale IP addressing with IPv6
and multi-Pod CIDR.
2001:581:d1::/64
Scale traffic across clusters with:
● multi-cluster Ingress
● multi-cluster Services
● multi-cluster autoscaling
● global service discovery
● globally routable Pods
Scale traffic automatically across Pods and clusters with
Google’s fully managed load balancing infrastructure

24.
Proprietary + Confidential
Iptables upgrade to eBPF

25.
eBPF Overview

26.
GKE Private Addresses
gcloud compute networks subnets update SUBNET_NAME
--stack-type=IPv4_IPv6
--ipv6-access-type=EXTERNAL
gcloud compute networks subnets create SUBNET_NAME
--network=NETWORK
--range=IPV4_RANGE
--stack-type=IPV4_IPV6
--ipv6-access-type=INTERNAL
--region=REGION

27.
Dual Stack (IPv4/IPv6)
# Valid IP family policies:
# - SingleStack
# - PreferDualStack
# - RequireDualStack
ipFamilyPolicy: PreferDualStack
ipFamilies:
- IPv6
- IPv4
clusterIP: 198.51.100.2
clusterIPs:
- 192.51.100.2
- 2001:D88:1::2
podIP: 192.0.2.2
podIPs:
- ip: 192.0.2.2
- ip: 2001:D88:2::2
podCIDR: 192.0.2.0/24
podCIDRs:
- ip: 192.0.2.2/24
- ip: 2001:D88:2::2/36

28.
Dual Stack IPv4/IPv6 Hybrid Cloud
1. Flat Network - Service Mesh , Multi-Cluster Connectivity without NAT or Proxy
2. Migration from on-premises IPv6 deployment to GCP
3. IPv6 standardization - Telcos and SPs
GCP
Customer’s VPC
GKE Private Cluster
Node Node Node
On-prem
IPv6 space
Static/Dynamic
Route
Advertisement
VPN/interconnect
IPv6 space

29.
Proprietary + Confidential
IPv6 Enable GDC for Applications and Network Functions
Telco Edge
Google Pop
Enterprise
Smart Retail Smart Factory Smart City Healthcare Smart Venue
Telco Solutions
Closed
Loop
Telco Fiber,Telco
Cellular & Google Fiber
GCP Services
Google Marketplace
Centralized Control(s)
AI/ML Training
Operations Management
RAN Network Planner & SAS
Google Distributed Cloud
Managed Platform
Network Services
5G Core & Network Functions
Private Cellular
IoT & SD-WAN
Security
Security
&
Compliance
Edge Services
AI/ML Inferencing
Vision, Video, NLP, Translation,
Retail, Healthcare, Care, {custom}
Immersive AR/VR
Search, Ads, Training,
Collaboration, Gaming, other
Enterprise Services
Computer Vision
Business Process
Asset Intelligence
Caching & Media
Data
Control
29 Regions
146+ POP

30.
Thank you