Secrets are any sensitive piece of information (like a password, API token, TLS private key) that must be kept safe. This presentation is a practical guide covering what we've done at Cloud Posse to lock down secrets in production. It includes our answer to avoid the same pitfalls that Shape Shift encountered when they were hacked. The techniques presented are compatible with automated cloud environments and even legacy systems.

1. Managing Secrets in
Production
A practical guide to what we’ve done at Cloud Posse
Erik Osterman
erik@cloudposse.com

2. Story Time
“Once upon time… there was company called Shape Shift”
Inside job led to theft of $250K of virtual currency
SSH keys compromised multiple times
Hacker paid for secrets
Entirely avoidable

3. What is this talk?
Practical guide to how we lock down secrets
Our answer to Shape Shift’s problem
Incomplete - huge problem space

4. What this talk is not...
Silver Bullet / “One True Answer”
General Container/Hypervisor/OS-level Security
Technical Implementation Details

5. What are secrets?
Any sensitive piece of information (like a password, API token, TLS private key)
that must be kept safe

6. Examples of Secrets
AWS Credentials (e.g. AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
API Keys (e.g. Datadog, Sumologic, Mailgun) - Third Party Integrations
TLS/SSL Keys (e.g. SSH, OpenVPN, Kubernetes, CA)
Database Credentials (E.g. MySQL, Postgres, Redis)

7. Why do we care?
1. Unauthorized data access
2. Identity spoofing
3. Regulatory fines
4. Hacking Profitable → Prolific

8. Scope of the Problem
1. Applications depend on external resources (e.g. databases, APIs)
2. These resources are accessed using credentials (aka secrets)
3. Cloud environments seek a balance between automation and security
4. Not all secrets are under our control
(E.g. ssh keys or credentials we cannot rotate programmatically)

9. Goal
Operate a secure production infrastructure
Objectives:
1. Use practical techniques that do not require massive code changes
2. Compatible with automated cloud environments
3. Compatible with legacy systems

10. What not to do...
1.Store secrets in git repository
2.Hardcode secrets in configurations
3.Write them in plain-text
4.Manually distributed them
5.Reuse/share keys across users and apps
6.Build homegrown systems to protect secrets
(* unless you’re Netflix, Hashicorp or Google)

11. How Cloud Posse protects secrets...
SSH Keys (avoid the Shape Shift pitfall)
AWS Credentials
Bootstrap Secrets
Application Secrets
Organizational Secrets (aka “Shared” secrets)

12. Protecting SSH Secrets
You can’t protect the private key
You can add multiple factors (a.k.a. MFA)
Our Solution
Use Github Public Key API to distribute public keys
https://github.com/cloudposse/github-authorized-keys
Use Duo for MFA Push Notifications + Geofencing
https://github.com/cloudposse/bastion
Other tricks:

13. Protecting AWS Secrets (Account-level)
Multiple AWS Accounts (production, staging, dev) - Share nothing
Single Identity/Consolidated Billing Account
Federated IAM User Accounts
Assumed Roles (e.g. read-only, admin, dba)
MFA required to assume roles - to devalue credentials

14. Protecting AWS Secrets (Client-side)
Client Side (e.g. Terraform, AWS Cli)
IAM User Account Access Keys (never shared!)
Assumed Roles (limit scope)
Temporary Sessions with STS (expire after 1 hour)
MFA (devalue credentials)
Solution: https://github.com/cloudposse/aws-assumed-role

15. Protecting AWS Secrets (Server-side)
Server Side (e.g. EC2 Instance, Docker Container)
IAM Instance Profiles with Assumed Roles
Use Kube2IAM with Kubernetes (kops)
https://github.com/cloudposse/charts/tree/master/incubator/kube2iam-kops
Temporary AWS credentials
Drop-in Compatiblity with all official AWS client library

16. Application Secrets
We use Kubernetes Secrets
Interface/Abstraction for defining secrets and referencing them
Not encrypted at rest (only obfuscated)
Referenced using Environment Variables (e.g. MAILGUN_API_KEY)
Want to use Vault
Need to implement kubernetes controller for single-use pod tokens

17. Bootstrap Secrets
Secrets you need to provision new clusters - “Infrastructure Layer”
Run terraform, awi cli, kops inside of Container
Private S3 Configuration Bucket
Encrypted Bucket Objects
Mount S3 Bucket inside container (S3FS)
Use /dev/shm for caching

18. Organizational Secrets
Shared by teams where no better alternative exists.
e.g. AWS root account, company credit card, or webhooks (like Slack, Datadog, Sentry)
Our Solution:
1. 1Password for Teams
2. Teams for groups of engineers
3. Vaults for different collections of passwords / tokens
4. MFA Everywhere

19. Cool Stuff
Vault Controller for Kubernetes
https://github.com/Boostport/kubernetes-vault
Envconsul
https://github.com/hashicorp/envconsul
Docker Secrets Management
https://docs.docker.com/engine/swarm/secrets/
Netflix Bless Teleport SSH
https://github.com/Netflix/bless https://gravitational.com/teleport/

20. __EOF__
Everything you need to protect secrets already exists
Cloud Posse will help you get there!
Erik Osterman, Founder
Cloud Posse, LLC
hello@cloudposse.com
https://cloudposse.com/

21. Questions for the Audience
Vault - how are you currently using it?
CVE Scanning in the Container Era - who’s doing it and how?