Kubernetes networking can be complex to scale due to issues like growing iptables rules, but newer solutions are helping. Pod networking uses CNI plugins like flannel or Calico to assign each pod an IP and allow communication. Service networking uses kube-proxy and iptables or IPVS for load balancing to pods. DNS is used to resolve service names to IPs. While Kubernetes networking brings flexibility, operators must learn the nuances of their specific CNI plugin and issues can arise, but the ecosystem adapts quickly to new needs and changes don't impact all workloads.

1. Kubernetes at Datadog Scale

2. Datadog is a monitoring
and analytics platform that
helps companies improve
observability of their
infrastructure and
applications

3. Ara Pulido
Developer Advocate at Datadog
@arapulido
ara.pulido@datadoghq.com

5. 19 major releases since
2015
+90k commits
+2k contributors

6. Kubernetes popularity since 2016

7. Kubernetes popularity since 2016

8. Kubernetes vs Mariah Carey

9. My journey with Kubernetes
2 YEARS FULL TIME IN KUBERNETES PROJECTS
MAINTAINER OF A YOUTUBE KUBERNETES CHANNEL
CKA; PART OF THE TEAM THAT CREATED CKAD

12. Dozens of clusters
+1K nodes / cluster
Multicloud

13. Survey: number of nodes / cluster
Source: https://github.com/MagalixCorp/kubernetes-autoscale-survey

14. What are the
common
scalability issues?

15. Networking

16. Kubernetes
networking
is not easy

17. Kubernetes Networking 101
POD NETWORKING
DNS
SERVICE NETWORKING

18. Pod Networking
Pod
172.17.0.2
Pod
172.17.0.3
EVERY POD GETS A UNIQUE IP
EVERY POD CAN TALK TO ANY OTHER POD IN THE CLUSTER

19. Service Networking
Client
172.17.0.6
Pod
172.17.0.4
Pod
172.17.0.5
Service
10.107.122.52
kube-proxy

20. DNS
Service
myservice.default.svc.cluster.local 10.97.239.227

21. Pod Networking

22. Pod Networking
Pod
172.17.0.2
Pod
172.17.0.3
EVERY POD GETS A UNIQUE IP
EVERY POD CAN TALK TO ANY OTHER POD IN THE CLUSTER

23. Pod
172.17.0.2
Pod
172.17.0.3
CNI Container Network
Interface

24. Overlays (flannel, weavenet)
Pod
10.244.2.1
10.244.1.0/24
10.244.0.0/16
10.244.2.0/24
10.244.0.0/16
Pod
10.244.1.1
192.168.0.1 192.168.0.2
cni0
flannel0
cni0
flannel0

25. Overlays (flannel, weavenet)
Pod
10.244.2.1
10.244.1.0/24
10.244.0.0/16
10.244.2.0/24
10.244.0.0/16
Pod
10.244.1.1
192.168.0.1 192.168.0.2
cni0
flannel0
cni0
flannel0
I WAS HERE

26. Overlays (flannel, weavenet)
Pod
10.244.2.1
Pod
10.244.1.1
192.168.0.1 192.168.0.2
10.244.1.0/24
10.244.0.0/16
10.244.2.0/24
10.244.0.0/16
cni0
flannel0
cni0
flannel0

27. Overlays (flannel, weavenet)
Pod
10.244.2.1
Pod
10.244.1.1
192.168.0.1 192.168.0.2
10.244.1.0/24
10.244.0.0/16
10.244.2.0/24
10.244.0.0/16
cni0
flannel0
cni0
flannel0

28. Overlays
Pod
10.244.2.1
10.244.1.0/24
10.244.0.0/16
10.244.2.0/24
10.244.0.0/16
cni0
flannel0
cni0
flannel0
Pod
10.244.1.1
192.168.0.1 192.168.0.2
Src: 10.244.1.1
Dst: 10.244.2.1

29. Overlays
10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.0
10.244.0.0/16 dev flannel proto kernel scope link src 10.244.1.1
Pod
10.244.2.1
10.244.1.0/24
10.244.0.0/16
10.244.2.0/24
10.244.0.0/16
cni0
flannel0
cni0
flannel0
Pod
10.244.1.1
192.168.0.1 192.168.0.2
Src: 10.244.1.1
Dst: 10.244.2.1

30. Overlays
Src: 192.168.0.1
Dst: 192.168.0.2
Src: 10.244.1.1
Dst: 10.244.2.1
Pod
10.244.2.1
10.244.1.0/24
10.244.0.0/16
10.244.2.0/24
10.244.0.0/16
cni0
flannel0
cni0
flannel0
Pod
10.244.1.1
192.168.0.1 192.168.0.2
10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.0
10.244.0.0/16 dev flannel proto kernel scope link src 10.244.1.1

31. BGP (Calico)
Pod
10.244.162.131/32 10.244.143.129/32
Pod
10.244.162.131
192.168.0.1 192.168.0.2
BGP BGP
10.244.143.129
calixxxx calixxxx

32. BGP (Calico)
10.244.162.131 dev cali12d4a061371 scope link
10.244.143.129 via 192.168.0.2 dev eth0
10.244.143.129 dev cali12d4bf881 scope link
10.244.162.131 via 192.168.0.1 dev eth0
Pod
10.244.162.131/32 10.244.143.129/32calixxxx calixxxx
Pod
10.244.162.131
192.168.0.1 192.168.0.2
BGP BGP
10.244.143.129

33. BGP (Calico)
10.244.162.131 dev cali12d4a061371 scope link
Pod
10.244.162.131/32 10.244.143.129/32calixxxx calixxxx
Pod
10.244.162.131
192.168.0.1 192.168.0.2
BGP BGP
10.244.143.129

34. BGP (Calico)
10.244.143.129 via 192.168.0.2 dev eth0
Pod
10.244.162.131/32 10.244.143.129/32calixxxx calixxxx
Pod
10.244.162.131
192.168.0.1 192.168.0.2
BGP BGP
10.244.143.129

35. Direct Pod Routing (AWS, Lyft)
Pod
10.0.0.3
Pod
10.0.0.4
SDN

36. Direct Pod Routing (AWS, Lyft)
Pod
10.0.0.3
Pod
10.0.0.4
SDN
DATADOG

37. Direct Pod Routing (AWS, Lyft)
Pod
10.0.0.3
Pod
10.0.0.4
SDN

38. Be ready to become an
“expert” in the CNI plugin
of your choice

39. Service
Networking

40. Client
172.17.0.6
Pod
172.17.0.4
Pod
172.17.0.5
Service
10.107.122.52
kube-proxy
Service Networking

41. iptables mode (default)
Client
172.17.0.6
Pod 1
172.17.0.4
Pod 2
172.17.0.5
Iptables
10.107.122.52
kube-proxy API
Watch svc, endpoints

42. iptables mode (default)
Client
172.17.0.6
Pod 1
172.17.0.4
Pod 2
172.17.0.5
Iptables
10.107.122.52
kube-proxy API
Watch svc, endpoints
I WAS HERE

43. iptables mode (default)
Client
172.17.0.6
Pod 1
172.17.0.4
Pod 2
172.17.0.5
Iptables
10.107.122.52
kube-proxy API
Watch svc, endpoints

44. iptables mode (default)
Chain KUBE-SERVICES (2 references)
KUBE-SVC-4N57TFCL4MD7ZTDA tcp -- anywhere 10.107.122.52
/* default/nginx: cluster IP */ tcp dpt:www
Chain KUBE-SVC-4N57TFCL4MD7ZTDA (1 references)
target prot opt source destination
KUBE-SEP-UZXILYFQQ2IZUWN5 all -- anywhere anywhere
statistic mode random probability 0.50000000000
KUBE-SEP-43IWXJI557JKCKCF all -- anywhere anywhere
Chain KUBE-SEP-43IWXJI557JKCKCF (1 references)
target prot opt source destination
KUBE-MARK-MASQ all -- 172.17.0.5 anywhere
DNAT tcp -- anywhere anywhere tcp
to:172.17.0.5:80

45. iptables mode (default)
Chain KUBE-SERVICES (2 references)
KUBE-SVC-4N57TFCL4MD7ZTDA tcp -- anywhere 10.107.122.52
/* default/nginx: cluster IP */ tcp dpt:www

46. iptables mode (default)
Chain KUBE-SERVICES (2 references)
KUBE-SVC-4N57TFCL4MD7ZTDA tcp -- anywhere 10.107.122.52
/* default/nginx: cluster IP */ tcp dpt:www
Chain KUBE-SVC-4N57TFCL4MD7ZTDA (1 references)
target prot opt source destination
KUBE-SEP-UZXILYFQQ2IZUWN5 all -- anywhere anywhere
statistic mode random probability 0.50000000000
KUBE-SEP-43IWXJI557JKCKCF all -- anywhere anywhere

47. iptables mode (default)
Chain KUBE-SERVICES (2 references)
KUBE-SVC-4N57TFCL4MD7ZTDA tcp -- anywhere 10.107.122.52
/* default/nginx: cluster IP */ tcp dpt:www
Chain KUBE-SVC-4N57TFCL4MD7ZTDA (1 references)
target prot opt source destination
KUBE-SEP-UZXILYFQQ2IZUWN5 all -- anywhere anywhere
statistic mode random probability 0.50000000000
KUBE-SEP-43IWXJI557JKCKCF all -- anywhere anywhere
Chain KUBE-SEP-43IWXJI557JKCKCF (1 references)
target prot opt source destination
KUBE-MARK-MASQ all -- 172.17.0.5 anywhere
DNAT tcp -- anywhere anywhere tcp
to:172.17.0.5:80

48. GROWS LINEARLY WITH PODS AND SERVICES
THERE IS NO ROOM FOR MORE FEATURES
FOR EVERY CHANGE, THE WHOLE TABLE NEEDS
TO BE RESYNC
iptables mode (default)

49. GA SINCE 1.11
IPVS mode

50. GA SINCE 1.11
IPVS mode
DATADOG

51. ATOMIC CHANGES (NO NEED TO RESYNC EVERY RULE)
O(1) COMPLEXITY
DIFFERENT LOAD BALANCING ALGORITHMS
ipvsadm -ln --tcp-service 10.107.122.52:80
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.107.122.52:80 rr
-> 172.17.0.4:80 Masq 1 0 37
-> 172.17.0.5:80 Masq 1 0 44
GA SINCE 1.11
IPVS mode

52. GA SINCE 1.11
IPVS mode

53. Many large deployments
are already moving to
IPVS and newer solutions
are being developed
(eBPF based)

54. DNS

55. Service
10.97.239.227myservice.default.svc.cluster.local

57. NodeLocal DNSCache is now
part of Kubernetes (Beta 1.15)

58. DNS Local Cache
` Client
DNS
Cache CoreDNS
DATADOG

59. Take aways

60. The bad

61. KUBERNETES IS FLEXIBLE FOR DEVS, COMPLEX FOR OPS
YOU WILL HIT BUGS
CNI PLUGINS WORK *VERY* DIFFERENTLY. LEARN YOURS.

62. The good

63. THE ECOSYSTEM PICKS UP QUICKLY
DEVELOPMENT EXPERIENCE DOESN’T CHANGE (MUCH)
CHANGES DON’T HAVE TO HAPPEN FOR ALL
WORKLOADS DIRECTLY

64. Thank you!