Kubernetes (K8s) is a powerful, flexible and portable open source framework for distributed containerized applications delivery and management. An important part of the services provided by most Kubernetes clusters is the containers’ networking stack. In most cases and for many applications it “just works”, but this seeming simplicity is backed by a complex stack of technologies that provide many capabilities beyond the basics. This presentation accompanies the meetup and webinar where Oleg Chunikhin, CTO at Kublr, shows how Kubernetes networking stack works, describes main components, interfaces and extensibility options. What is covered: - general notions of Kubernetes networking - Pods and Network Policies - implementation of Kubernetes networking - CNI, CNI plugins, and Linux network namespaces - some Kubernetes CNI providers: Calico, Weave, Flanel, and Canal - K8S networking extensibility for advanced and “exotic” use-cases with Multus CNI plugin as an example

1. Kubernetes Networking 101
Oleg Chunikhin | CTO, Kublr

2. Introductions
Oleg Chunikhin
CTO, Kublr
• 25 years in software architecture &
development
• Working w/ Kubernetes since its release in 2015
• Software architect behind Kublr—an enterprise
ready container management platform
• Twitter @olgch

3. Enterprise Kubernetes Needs
Developers SRE/Ops/DevOps/SecOps
• Self-service
• Compatible
• Conformant
• Configurable
• Open & Flexible
• Governance
• Org multi-tenancy
• Single pane of glass
• Operations
• Monitoring
• Log collection
• Image management
• Identity management
• Security
• Reliability
• Performance
• Portability
@olgch; @kublr

4. @olgch; @kublr
Automation
Ingress
Custom
Clusters
Infrastructure
Logging Monitoring
Observability
API
Usage
Reporting
RBAC IAM
Air Gap TLS
Certificate
Rotation
Audit
Storage Networking Container
Registry
CI / CD App Mgmt
Infrastructure
Container Runtime Kubernetes
OPERATIONS SECURITY &
GOVERNANCE

5. Kubernetes Networking
@olgch; @kublr
• Kubernetes overview / refresher
• Network Policies - in-cluster firewall
• Kubernetes Networking Architecture and CNI

6. Kubernetes Cluster
K8S Architecture Refresher: Components
The Master, agent, etcd, API, overlay network, and DNS
Master
API Server
etcd data
controller
manager
scheduler etcd
kubectl
Worker
kubelet
container
runtime
overlay
network
cluster
DNS
kube-proxy
@olgch; @kublr
CNI

7. Cluster
K8S Architecture: Compute & Network
Nodes, pods, services, addressing
Node 1
172.16.0.1
Node 2
172.16.0.2
Pod A-1
10.0.0.3
Cnt1
Cnt2
Pod A-2
10.0.0.5
Cnt1
Cnt2
Pod B-1
10.0.0.8
Cnt3
SrvA
10.7.0.1
SrvB
10.7.0.3
@olgch; @kublr

8. Network Policy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: my-network-policy
namespace: default
spec:
podSelector:
matchLabels:
app: net-srv
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app: net-pinger
ports:
- protocol: TCP
port: 8080
egress:
- {}
@olgch; @kublr
1. Network Policies are per-namespace
2. Network Policies select pods based on labels
a. Isolated Pods - if selected by at least one policy
Only traffic allowed by union of all selecting policies
b. Non-isolated Pods - not matched by any policy
All traffic is allowed
3. Network Policies are additive, never conflict
4. For traffic between pods to be allowed, egress on the source,
and ingress on the target must be allowed
5. Policy type may be Ingress, Egress, or both
6. If no policy type is specified, then Ingress is always set, and
Egress is set if there are egress rules defined
7. May include any number of ingress and egress rules
1
2
5
7
7

9. Network Policy Anatomy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: my-network-policy
namespace: default
spec:
podSelector: { ... }
policyTypes:
- Ingress
- Egress
ingress:
- { ... } # ingress rule 1
...
- { ... } # ingress rule N
egress:
- { ... } # egress rule 1
...
- { ... } # egress rule M
@olgch; @kublr
Metadata
podSelector:
matchLabels:
key1: value1
matchExpression:
- key: key2
operator: In # NotIn, Exists, DoesNotExist
values: [val1, val2]
● pod selector is a standard Label Matcher
● podSelector is required
● empty selector matches any pod
● requirements are AND’ed
# for ingress rules
from: [ peer1, ... , peerN ]
# for egress rules
to: [ peer1, ... , peerN ]
# for both ingress and egress
ports: [ port1, ... , portM ]
Pod Selector
Ingress and Egress Rules
ipBlock:
cidr: 10.0.0.0/24
except: [10.24.0.0/16, ...]
namespaceSelector: { ... }
podSelector: { ... }
NetworkPolicy Peer
● peers and ports are OR’ed
● empty or missing field matches all
port: 8000
endPort: 32000
protocol: TCP # UDP, SCTP
NetworkPolicy Port
● protocol defaults to TCP
● endPort is optional
● endPort is beta (on by default) since K8S 1.22
● SCTP is stable since K8S 1.20
● pod and namespace selectors are standard
Label Matchers
● if no namespaceSelector, podSelector
matches policies in the same namespace

10. Test applications
apiVersion: v1
kind: Service
metadata:
name: net-srv
spec:
selector: { app: net-srv }
ports:
- { port: 8080, protocol: TCP, targetPort: 8080 }
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: net-srv
spec:
replicas: 1
selector:
matchLabels:
app: net-srv
template:
metadata: {labels: {app: net-srv}}
spec:
terminationGracePeriodSeconds: 1
containers:
- name: echo-server
image: inanimate/echo-server:latest
ports:
- name: nc
containerPort: 8080
stdin: true
tty: true
@olgch; @kublr
Test Server
apiVersion: apps/v1
kind: Deployment
metadata:
name: net-pinger
spec:
replicas: 1
selector:
matchLabels:
app: net-pinger
template:
metadata: {labels: {app: net-pinger}}
spec:
terminationGracePeriodSeconds: 1
containers:
- name: echo-server
image: alpine
stdin: true
tty: true
command:
- sh
- -c
- |
while true; do
wget www.google.com --spider -q -T 1 &>/dev/null &&
echo -n "google OK, " || echo -n "google NA, "
wget http://net-srv:8080 --spider -q -T 1 &>/dev/null &&
echo -n "net-srv OK, " || echo -n "net-srv NA, "
date
sleep 1
done
Test Client

11. Network Policy examples
kind: NetworkPolicy
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
@olgch; @kublr
1. Deny all traffic by default
kind: NetworkPolicy
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress:
- {}
egress:
- {}
2. Allow all traffic by default
kind: NetworkPolicy
spec:
podSelector:
matchLabels:
app: net-pinger
policyTypes:
- Egress
egress:
- {}
3. Allow all ingress/egress
to specific pods
kind: NetworkPolicy
spec:
podSelector:
matchLabels:
app: net-pinger
policyTypes:
- Egress
egress:
- {}
4. Allow specific traffic for
specific pods
kind: NetworkPolicy
spec:
podSelector:
matchLabels:
app: net-srv
policyTypes:
- Ingress
ingress:
- {}
kind: NetworkPolicy
spec:
podSelector:
matchLabels:
app: net-srv
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: net-pinger
ports:
- protocol: TCP
port: 8080

12. Node (Virtual Machine) 1
K8S Architecture: Compute & Network
Pod 1 net ns
@olgch; @kublr
root
net ns
eth0
ctr 1a
eth0
veth0
ctr 1b
veth1
Pod 2 net ns
ctr 2a
eth0
ctr 2b
cbr0
Node 2
Pod 3 net ns
root
net ns
eth0
ctr 3a
eth0
veth0
ctr 3b
veth1
Pod 4 net ns
ctr 4a
eth0
ctr 4b
cbr0
kubelet
kube-proxy
containerd
kubelet
kube-proxy
containerd

13. CNI Plugins
@olgch; @kublr
Provider Network Model Network
Policies
Mesh Datastore Encryption
Calico Encapsulated (VXLAN or IPIP)
Unencapsulated (BGP)
Yes Yes K8S API No
Canal Encapsulated (VXLAN) Yes Yes K8S API No
Weave Encapsulated Yes Yes No Yes
Flannel Encapsulated (VXLAN) No No K8S API No
AWS Unencapsulated Yes No K8S API No

14. Technologies Used
• Network namespaces (container runtime, CNI)
• Bridge (container runtime, CNI)
• Encapsulation (VXLAN, IPIP)
• Routing (BGP)
@olgch; @kublr

15. References
https://kubernetes.io/docs/concepts/cluster-administration/networking/
https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/
https://github.com/containernetworking/cni
https://kubernetes.io/docs/reference/kubernetes-api/policy-resources/network-policy-v1/
https://sookocheff.com/post/kubernetes/understanding-kubernetes-networking-model/
@olgch; @kublr

16. Q&A
@olgch; @kublr

17. Oleg Chunikhin
CTO
oleg@kublr.com
@olgch
Kublr | kublr.com
@kublr
Signup for our newsletter
at kublr.com
@olgch; @kublr