K8s security best practices

1. K8S security – Best practices
By: Sharon Vendrov

2. 0
500
1000
1500
2000
2500
Mac Os X Windows 7 Windows XP Windows 8.1 Windows 10
CVE Sum
CVE Sum
2
Total Number Of Vulnerabilities in 2017 – Source:
CVEdetails.com

3. 3

4. 4

5. Sharon Vendrov
Sr. DevOps Engineer
About Me
5
Storm-runner functional

6.  Infrastructure protection
 K8s internal security
 Authentication & Authorization options
 Network
 Secrets
 Container runtime Security
 Some other security tools and considerations
6
Agenda

7. Infrastructure protection
7
 Limit SSH access to your cluster
 Use hardened images for your cluster ( )
 Encrypt your storage volume
 Avoid from exposing your cluster to the internet
 Limit the access to the K8s API (consider to use bastion machine)
 Create dedicated cluster for each environment (Prod, Stg, Dev)
 Separate sensitive pods into different nodes

8. Kubernetes internal security
8
 Use minimal base docker image
 Don’t use arbitrary base images
 Separate sensitive workloads across instances (using anti-affinity,
taints and tolerations)
 Use namespaces for isolation
 Enforce resource quota (CPU, Memory, Storage)
Image Name node:latest ubuntu:latest alpine:latest scratch
Image Size 670MB~ 110MB~ 4.1MB~ 0

9. Secure kubelet
9
 curl -sk https://<nodeIP>:10250/run/<namespace>/<pod-name>/<container-name> -d
"cmd=ls -la /“
 Protect kubelet by enable authentication and authorization:
start the apiserver with --kubelet-client-certificate and --kubelet-client-key flags
/usr/local/bin/kubelet
--anonymous-auth=false
--authorization-mode=Webhook
--allow-privileged=true
--kubeconfig=/var/lib/kubelet/kubeconfig
--client-ca-file=/var/lib/kubernetes/ca.pem
• Enable kubelet certification rotation (1.8 beta)

10. Authentication & Authorization

11. 11

12. 12

13. Authentication
13
 Static password/token file
 Client certificates x509
 Proxy + headers
 OpenID Connect
 Custom (Web hook)
password,user,uid,”group1,group2,group3”

14. Authentication
14
 Service accounts
 Default service account have full permissions over the cluster, use custom SA instead
 Set “automountServiceAccountToken : false” in your pod spec – when possible

15. Authorization
15
 ABAC
 Difficult to manage and understand
 Requires ssh and root filesystem access on the master
 For permission changes to take effect the cluster API server must be restarted
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1",
"kind": "Policy",
"spec": {
"user": "bob",
"namespace": "projectSpaceX",
"resource": "pods",
"readonly": true
}
}

16. Authorization
16
 RBAC (stable 1.8)
Service Account
User
Role binding Role

17. 17
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: read-pods
namespace: default
subjects:
- kind: User
name: Bob
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: default
name: pod-reader
rules:
- apiGroups: [""]
group
resources: ["pods"]
verbs: ["get", "watch", "list"]

18. Authorization
18
 Custom (Web hook)
 Node
Restrict kubelet to perform R/W operation only to his bound pods
--authorization-mode=Node,RBAC
--admission-control=NodeRestriction

19. Network

20. 20

21. Netwok
21
 Limit the access to cloud provider metadata
(http://169.254.169.254/latest/meta-data/)

22. 22
$ curl -s 169.254.169.254/latest/meta-data/iam/security-
credentials/kubernetes-worker-iam-policy
{
"Code" : "Success",
"LastUpdated" : "2017-12-25T00:00:00Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "MyAccessKeyID",
"SecretAccessKey" : "MySecretAccessKey",
"Token" : "MySessionToken",
"Expiration" : "2017-12-25T04:00:00Z"
} @bradgeesaman

23. 23
# Place credentials in ENV vars
$ export AWS_REGION=us-east-1
$ export AWS_ACCESS_KEY_ID=MyAccessKeyID
$ export AWS_SECRET_ACCESS_KEY=MySecretAccessKey
$ export AWS_SESSION_TOKEN=MySessionToken
$ aws ec2 … @bradgeesaman

24. The solution
24
• For AWS use kube2iam or kiam (using docker proxy for requests to the
metadata)
• For GCE use k8s-metadata-proxy
• Limit egress with network policy

25. 25
 Use network policy (GA from 1.7) https://goo.gl/HRtn5B
 Egress rules are beta from 1.8
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: access-nginx
spec:
podSelector:
matchLabels:
run: nginx
ingress:
- from:
- podSelector:
matchLabels:
access: "true"
• Istio

26. Network policy guidelines
26
 Label your workloads properly
 Isolate workloads from each other
 Restrict income traffic to the kube-system (except kube-dns)
 Consider limit egress to the internet

27. “The definition of Secret—
something you tell everybody to
tell nobody.”
– The universe

28. Treat your secrets with respect
28
 Don’t store your secrets on Git, it will remain in history even If you
delete it.
 Create dedicated secrets for dev and prod environments
 Secrets are stored at etcd as base64 (almost like plain text) 
encrypt your secrets (K8S encryption –alpha 1.7)
 Use Vault as you secret management (starting from Vault 0.8.3)

29. Security Context
A security context defines privilege and access control settings for a Pod or Container
29
 Discretionary Access Control: Permission to access an object, like a file, is
based on user ID (UID) and group ID (GID).
 Security Enhanced Linux (SELinux): Objects are assigned security labels.
 Running as privileged or unprivileged.
 Linux Capabilities: Give a process some privileges, but not all the privileges
of the root user.
 AppArmor: Use program profiles to restrict the capabilities of individual
programs.
 Seccomp: Limit a process’s access to open file descriptors.
 AllowPrivilegeEscalation: Controls whether a process can gain more
privileges than its parent process.

30. Example: RunasNonRoot
30
apiVersion: v1
kind: Pod
metadata:
name: security-context-demo
spec:
containers:
- name: sec-ctx-demo
image: gcr.io/google-samples/node-hello:1.0
securityContext:
runAsNonRoot : true

31. 31

32. Example: readOnlyRootFilesystem
32
apiVersion: v1
kind: Pod
metadata:
name: security-context-demo
spec:
containers:
- name: sec-ctx-demo
image: gcr.io/google-samples/node-hello:1.0
securityContext:
runAsNonRoot : false
readOnlyRootFilesystem : true

33. 33

34. 34

35. Other security tools and considerations
35
 Scan your docker images for vulnerabilities, (Clair CoreOS /Quay.io,
Docker Security Scanning, aqua, Twistlock).
 Use kube-bench (aqua security) or kubernetes-auto-analyzer
(nccgroup) to execute CIS Kubernetes Benchmark
 Enforce cluster wide security policy w/podSecurityPolicy
 Use only trusted private docker registry
 Always tag your images avoid from using “latest”
 Audit events and store them on external storage (beta 1.8)
 Consider using kubeaudit to audit security issue

36. 36

37. Other security considerations
37
 Specify an image with its digest (SHA256)
 Keep up with K8S stable releases
 Implement monitoring and set alerts
 Don’t run “kubectl create –f <some unknown URL to some unknown
yamls>
 Keep updated with new security vulnerabilities from the google
group “kubernetes-announces”
https://groups.google.com/forum/#!forum/kubernetes-announce

38. 38

39. Thanks and credit
39
 My Wife 
 All K8s contributors
 Hacking and Hardening Kubernetes Clusters by Example [I] - Brad Geesaman -
https://goo.gl/komeXN
 Running containers securely with Google Container Engine, Alex Mohr and
Jessica Frazelle - https://goo.gl/AFhTyp
 Shipping in Pirate-Infested Waters: Practical Attack and Defense in Kubernetes
[A] - Greg Castle - https://goo.gl/WFDrrv
 Compliance and Identity Management in Kubernetes [I] Marc Boorshtein -
https://goo.gl/Jf7Rkh
 Securing K8s Microservices with Calico Network Policies, Vadim Solvey -
https://goo.gl/rWGGXM

40. Thank You.