Trust is good, control is better – A short story about Network Policies. Abstract: Probably everybody who uses Kubernetes in a productive environment with multiple users possibly has looked at policies. Often the operators of the cluster(s) just trust the policies but in some cases it might be useful to control if the policies actually have taken action and often there are just to many Policies in the cluster setup to manually test them all (and obviously you don’t want to do this). Testing the effectiveness of the Network Policies can be done in different approaches. In this talk we will show you the benefits and drawbacks of different approaches and what solution we finally chose. Also we will show you some other tools and how they complement our solution. As a takeaway you will get an overview of different testing strategies for policies, as well as understanding challenges in testing policies in general and the Kubernetes ecosystem. Event: ContainerDays 2019 Datum: 26.06.2019 Speaker: Johannes M. Scheuermann, Maximilian Bischoff (beide inovex) Mehr Tech-Vorträge: inovex.de/vortraege Mehr Tech-Artikel: inovex.de/blog

1. Trust is good, control is better
A short story about Network Policies
Maximilian Bischoff,
Johannes M. Scheuermann
Hamburg, 26. June 2019

2. Maximilian Bischoff
Unofficial: Chaos Monkey
@johscheuer
2
Cloud Platform Engineer
Cloud Platform Engineer
Johannes M. Scheuermann

3. What can you expect ?
● Get an overview about challenges with network policies
● Get an overview on different aspects of testing /
validating your setup
3

4. What about you?
4

5. Why should I test my
network policies?
5

6. Why should I test my policies ?
Many adjustment screws
https://www.pexels.com/photo/colorful-toothed-wheels-1711986

7. Why should I test my policies ?
Kubernetes doesn’t implement the policies
7
kube-apiserver
Netpol
CNI Plugin
Read NetPol
Implements them

8. Why should I test my policies ?
Kubernetes doesn’t implement the policies
8
kube-apiserver
CNI Plugin
Read NetPol
No Feedback !
Netpol
Implements them

9. Why should I test my policies ?
I choose you !
This list is not complete!9

10. ...
spec:
ingress:
- from:
- namespaceSelector:
matchLabels:
team: operations
podSelector:
matchLabels:
type: monitoring
Why should I test?
Hard to read policies
https://github.com/ahmetb/kubernetes-network-policy-recipes/blob/master/07-allow-traffic-from-some-pods-in-another-namespace.md10
...
spec:
ingress:
- from:
- namespaceSelector:
matchLabels:
team: operations
- podSelector:
matchLabels:
type: monitoring
and or

11. Node
Why should I test my policies ?
Component updates
11
kube-proxy
iptables CNI-Plugins
Node
kube-proxy
ipvs CNI-Plugins
Node
kube-proxy
iptables
CNI-Plugins
v1
Node
kube-proxy
iptables
CNI-Plugins
v2

12. Node
Why should I test my policies ?
Component updates
12
kube-proxy
iptables CNI-Plugins
Node
kube-proxy
ipvs CNI-Plugins
Node
kube-proxy
iptables
CNI-Plugins
v1
Node
kube-proxy
iptables
CNI-Plugins
v2
Is everything still working after an update?

13. Node
Why should I test my policies ?
Component updates
13
kube-proxy
iptables CNI-Plugins
Node
kube-proxy
ipvs CNI-Plugins
Node
kube-proxy
iptables
CNI-Plugins
v1
Node
kube-proxy
iptables
CNI-Plugins
v2
Conformance tests don’t test network policies!

14. What to test
14

15. What to test
15
Policy
Policy
Policy
SDN
Control
Plane
Data Plane
Conformance - Effect of policies
- Synchronisation

16. How to test
16

17. How to test
17
Policy
Policy
Policy
SDN
Control
Plane
Data Plane
Conformance - Effect of policies
- Synchronisation

18. Testing strategies
Copy-pod
18
kind: pod
apiVersion: v1
metadata:
name: foo
namespace: default
labels:
app: foo
spec:
containers:
- name: foo
image: foo:latest
...
kind: pod
apiVersion: v1
metadata:
name: foo-test-copy
namespace: default
labels:
app: foo
testing.framework: “”
spec:
containers:
- name: test
image: test/runner:latest
...
copy

19. Testing strategies
Docker networking
19
foo
pausefoo
test-
runner

20. Testing strategies
Linux namespaces
20
pause
(of pod
foo)
test-
runner
underlying OS
...
Network
IPC
CGroup
...
IPC
CGroup

21. kubectl exec -it foo -- wget -qO - --timeout=2
http://bar.default
wget: download timed out
Manually
https://github.com/ahmetb/kubernetes-network-policy-recipes/blob/master/03-deny-all-non-whitelisted-traffic-in-the-namespace.md21
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: default-deny-all
namespace: default
spec:
podSelector: {}
ingress: []
default
foo barbar

22. node 1
netassert
https://github.com/controlplaneio/netassert22
default
foo barbar
config.yaml
---
k8s:
deployment:
default:foo:
default:bar: TCP:80
ssh
docker run
--net ...
test.js
nmap
netassert

23. node 1
illuminatio
23
default
foo barbar
API Server
nsenter
results
test.py
nmap
illuminatio

24. kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: demo
namespace: default
spec:
podSelector:
matchLabels:
app: prometheus
ingress:
...
Test case generation
Preface
24
isolated from every pod
except for those matching

25. Test case generation
Two kinds of tests
25
pod A
app=prometheus
pod B
app=grafana
pod C
...
spec:
podSelector:
matchLabels:
app: prometheus
ingress:
- from:
- podSelector:
matchLabels:
app: grafana

26. pod A
app=prometheus
Test case generation
Multiple policies
26
pod B
team=ops
app=foo
...
spec:
podSelector:
matchLabels:
app: prometheus
ingress:
- from:
- podSelector:
matchLabels:
app: grafana
...
spec:
podSelector: {}
ingress:
- from:
- podSelector:
matchLabels:
team: ops
namespaceSelector:
{}
?

27. Wrap up
27

28. How do these tools complement
28
Policy
Policy
Policy
kubeaudit/
SDN
Control
Plane
Data Plane
netassert/illuminatio

29. Recap
● Test your assumptions!
● Regression testing makes your life easier
● Network Policies are still hard to get right
○ Missing feedback
○ Does it work for Services and Pods?
29

30. Thank You
Maximilian Bischoff
IT Engineering &
Operations
inovex GmbH
Ludwig-Erhard-Allee 6
76131 Karlsruhe
maximilian.bischoff@
inovex.de
Johannes Scheuermann
IT Engineering &
Operations
inovex GmbH
Ludwig-Erhard-Allee 6
76131 Karlsruhe
johannes.scheuermann@
inovex.de