Kubernetes Secrets - Can Kubernetes Keep a Secret
•Download as PPTX, PDF•
3 likes•3,626 views
We’ve all experienced it: you’re working on a task, adding some code, and then you need to store some sensitive configuration value. It could be an API key, client secret or an encryption key ― something that’s highly sensitive and must be kept secret. And this is where things get messy. Usually, secret storage is highly coupled with how the code is deployed, and different platforms have different solutions. Kubernetes has a promise to simplify this process by using the native secret object, which, as the name implies, can be used to store secrets or sensitive configurations. Unfortunately, Kubernetes secrets are fundamentally broken, and a developer who tries to use them will definitely have some issues. But no need to worry ― there are solid alternatives for storing secrets securely on Kubernetes platform. One solution is to use Kamus, an open-source, git-ops solution, that created by Soluto, for managing secrets on Kubernetes. Kamus can encrypt a secret so it can be decrypted only by your app on runtime - and not by anyone else. The first part of this session will cover the challenges faced when using Kubernetes secrets (from a usability and security point of view). The second part will discuss some of the existing solutions (Sealed Secrets, Helm Secrets and others), their pros, and cons, and then feature Kamus: how it works, what problems it solves, how it differs from other solutions, and what threats it can help mitigate (and what threats it can’t). The talk will cover all that is required to know so you can run Kamus on your own cluster and use it for secret management. Join me for this session to learn how you can build a Kubernetes cluster than can keep a secret ― for real.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Kubernetes Secrets - Can Kubernetes Keep a Secret
Similar to Kubernetes Secrets - Can Kubernetes Keep a Secret (20)
More from Soluto
More from Soluto (20)
Recently uploaded
Recently uploaded (20)
Kubernetes Secrets - Can Kubernetes Keep a Secret
- 1. @omerlh Can Kubernetes Keep a Secret? Omer Levi Hevroni AppSec Cali 2019
- 5. @omerlh I OWASP • Zap contributor • Proud member • Glue project leader
- 6. @omerlh
- 7. @omerlh Super-Devs: Full Responsibility ● Writing Code ● Deploying to Production ● Monitoring https://www.imdb.com/title/tt4016454/mediaviewer/rm2380811776
- 8. @omerlh Super-Devs Need Help ● Good tools to support them ● Make it harder to do mistakes ● Secure by design
- 9. @omerlh
- 10. @omerlh Manifests Files Code A GitOps Solution Kubernetes Icons Source: Kubernetes Community, Apache 2 license
- 11. @omerlh How do we manage secrets?
- 12. @omerlh Manifests Files Code Secret A GitOps Solution Kubernetes Icons Source: Kubernetes Community, Apache 2 license
- 13. @omerlh Requirements GitOps Kubernetes native Secure “One-way encryption”
- 14. @omerlh Pod is out of scope ● Who can “SSH” into it? ● What is running on the pod? ● Does the code leaked the secrets?
- 16. @omerlh First iteration – Kubernetes Secrets
- 18. @omerlh Requirements GitOps Kubernetes native Secure
- 19. @omerlh@omerlh
- 20. @omerlh Let’s take a deeper look… Producing (dev) Consuming (application) Kubernetes Icons Source: Kubernetes Community, Apache 2 license
- 21. @omerlh Producing - File Manifest?
- 22. @omerlh Well, that complicates things… http://i.imgur.com/5ebYy62.jpg @omerlh
- 23. @omerlh Producing – Naive Approach
- 24. @omerlh Producing - Encrypted Secrets? ● Secrets that can be committed ● Transparent for the application ● Multiple solutions ○ Helm Secrets ○ Sealed Secrets
- 26. @omerlh@omerlh
- 27. @omerlh Issues ● Key Management ○ Sealed Secret – single key-pair in the cluster ○ Helm Secret – based on Mozilla mops (AWS/GCP KMS support) ● Coupling to a specific cluster/deployment method ● Any change to the secret requires decryption
- 28. @omerlh Let’s take a deeper look… ⓧ Producing (dev) Consuming (application)
- 29. @omerlh Consuming – Environment Variables
- 30. @omerlh The Environment Variable Dispute https://i0.wp.com/www.rogerogreen.com/wp-content/uploads/2015/06/Disputation.jpg
- 31. @omerlh • Some log libraries collects env vars • Accessible via /proc • Visible when inspecting docker image • RCE – run env to leaked all env vars. Simpler than finding all sensitive files and exporting them (even with LFI) • Harder to commit accidently • Simpler than files • If you can access /proc or inspect docker images, you can inspect mounted volumes • Better permissions model on windows (thanks @swisshttp!) • Leaked files (thanks @sporkmonger!) The Environment Variable Dispute Cons Pros https://tvtropes.org/pmwiki/pmwiki.php/Main/GoodAngelBadAngel
- 33. @omerlh Consuming – Volume Mount
- 34. @omerlh Consuming/Producing: Configuration Files Configuration File Base64 Encoder Secret Manifest
- 35. @omerlh Let’s take a deeper look… ⓧ Producing (dev) ⓧ Consuming (application) Kubernetes Icons Source: Kubernetes Community, Apache 2 license
- 36. @omerlh Requirements GitOps – under some serious limitations Kubernetes native Secure – depend on usage
- 37. @omerlh Second iteration – Hashicorp Vault
- 38. @omerlh What? ● Secure secrets storage ● Native Kubernetes integration ● Seamless consuming ○ Side-car to generate config files https://www.vaultproject.io/
- 39. @omerlh Workflow /my-app/super-sensitive Kubernetes Icons Source: Kubernetes Community, Apache 2 license
- 40. @omerlh @omerlh
- 41. @omerlh Workflow Access Control /my-app/super-sensitive Kubernetes Icons Source: Kubernetes Community, Apache 2 license
- 42. @omerlh Manifests Files Code Secret Vault - Threat Modeling Kubernetes Icons Source: Kubernetes Community, Apache 2 license Kubernetes Icons Source: Kubernetes Community, Apache 2 license
- 43. @omerlh Imperfect solution ●Separate storage of secrets and deployment files ○ No single source of truth ●External Permission Model ●Deployment ○ Cloud vendor alternatives (Azure KeyVault, AWS secret manager) ○ Vault users authn/authz
- 44. @omerlh Requirements ⓧ GitOps Kubernetes native Secure – depend on usage
- 45. @omerlh
- 47. @omerlh Eureka! http://theunprofessionalblog.blogspot.com/2016/04/whatsapp-this-is-killing-me.html
- 48. @omerlh Third iteration – Kamus Travis secret encryption – for Kubernetes
- 49. @omerlh Kamus?
- 51. @omerlh A perfect solution? GitOps Kubernetes native Secure
- 52. @omerlh Let’s talk about security
- 53. @omerlh Permission Model Encrypt Decrypt User Yes (Can be limited) No Pod Yes Only it’s own secrets
- 54. @omerlh Kamus – Threat Model Encryptor Decryptor Kubernetes Icons Source: Kubernetes Community, Apache 2 license
- 55. @omerlh Mitigations: User ● Secure by default permission model ● Secured CLI ○ Enforce HTTPS ○ Support for certificate pinning
- 56. @omerlh Mitigations: Git ● Strong encryption (using Azure KeyVault/GCP KMS) ○ HSM protection ○ IP Filtering ● One-way encryption
- 57. @omerlh Mitigations: Pod ● Secure by default permission model ● In-Memory volume for decrypted files Kubernetes Icons Source: Kubernetes Community, Apache 2 license
- 58. @omerlh Mitigations: Kamus API ● Separate pods ● Authentication support for encryptor ● Security tests ○ SAST (Checkmarx) ○ DAST (Zaproxy) ○ Packages scan (Snyk)
- 59. @omerlh Accepted Risks ●Clear text traffic inside the cluster ●Any pod in the same namespace can mount any service account ○ Pod impersonation ●Service account token never expires
- 62. @omerlh Kamus - A perfect solution GitOps Kubernetes native Secure
- 63. @omerlh How can I use it? ● Simply using helm: helm install kamus soluto/kamus ● Checkout the install guide for a secure installation ● Blog post - https://bit.ly/2T2Nhgs
- 64. @omerlh Kamus Roadmap ● AWS support ● Custom Resource Descriptor ● Rolling encryption keys ● Quality – improve test coverage ● FaaS
- 66. @omerlh Solutions GitOps Kubernetes Native Secure Kubernetes Secrets It depends Yes It depends Vault No Yes Yes Kamus Yes Yes Yes
- 69. @omerlh Can Kubernetes Keep a Secret? @omerlh
- 70. @omerlh Kamus Enable Super-Devs to Fly Higher https://www.imdb.com/title/tt4016454/mediaviewer/rm2380811776
- 71. @omerlh Thank You! Omer Levi Hevroni AppSec Cali 2019
Editor's Notes
- Hey, good morning everyone My name is Omer I want to start this talk by showing gratitude First, to all the people who worked hard on organizing this conf and all the people who are working today so we all could enjoy it - thank you Second, I want to thank the organizers who choose me to speak here, so thank you. It is a big honor <pause> Can kubernetes keep a secret? <pause> Why? Raise you’re hand if you ever worked on a project and you had to deal with credentials: API Key, client secret, certificates etc
- What you need to do is pass these credentials securely to the platform running your code, so the app in production can use it. Different platforms have different solutions to the problem (sometime good, sometime bad). This talk will focus on Kubernetes – how we can tell a secret to Kubernetes, and make sure only Kubernetes will know it? But first – let me introduce myself quickly, so you could understand what are my credentials and where I’m coming from.
- I’m a builder, this is what I love doing and doing it from a really early age Doing it professionally for the last 8 years I’m from Israel, married etc Who else is a builder? This talk is for you!
- Today I’m working at Soluto, our missing is to help people with their technology My job is DevSecOps, or as I see it - helping the entire team to build a more secure software I’m achieving it via many approaches, including education, reviewing and threat modeling – but what I love the most is threat modeling
- Big part of my work is OWASP, I’m enthusiast and familiar to many project. I contributed code to projects, mainly Zap and Glue and I’m a paid memember and project leader of Glue. Glue is a tool that helps to integrate security tools into the CI/CD pipeline – I will not have time to dive into the tool, but come talk with me later about it – I have stickers
- What you need to do is pass these credentials securely to the platform running your code, so the app in production can use it. Different platforms have different solutions to the problem (sometime good, sometime bad). This talk will focus on Kubernetes – how we can tell a secret to Kubernetes, and make sure only Kubernetes will know it?
- Explain why it is a challenge – you cant expect one person to manage all secrets Why not solved it manually
- Explain why it is a challenge – you cant expect one person to manage all secrets Why not solved it manually
- And that’s why we love GitOps: Git is a tool that all devs are familiar with.
- And we started to look for solutions. It want not an easy path, and today I want to share with you the process we want through. So, let’s start we talking on what we want.
- Choose one sentence
- Security is what we all here love
- Security features like encryption at rest
- Encoding is not encrypting Adding native approach
- Add meme
- Add slide with links
- Laugh at my bad english
- Example of 3 items/JSON representatiom
- Security is what we all here love
- Add the user here
- Add the user here
- Valut policies, policy assignment etc
- Security is what we all here love
- Make it more visual
- Security is what we all here love
- Battle tested
- Add attributation
- Add headlines – encryptor & decryptor
- We really love Kamus, we’re been using it in production for the past 6 months
- End of journey meme/image
- Today I discussed 3 different solutions for secret management on Kubenretes. All are good solutions, depend on your requirments.
- I started the talk by asking “Can Kubernetes keep a secret?” Now you that yes – Kubernetes can. You just need to use the right tool for you’re use case.
- For us, it was Kamus What Kamus can do for you?