Secret-based protocols are the most popular methods for establishing trust in authentication. Unfortunately, they are also one of the first attack surfaces to be probed when system compromise is attempted. Today’s digital services often focus on scalability, high-availability, and fault tolerance, leading to a shift towards microservices on cluster-based architectures. Secret management has evolved as well, leading to the development of cluster-compatible, open-source SM tools such as HashiCorp’s Vault. This talk is designed to help SecOps professionals leverage security concepts such as spatial and temporal attack surfaces, trust, and risk acceptance to secure their cluster credential management.

1. Secure Secret Management
on a Budget
Mary Racter
BSides Cape Town
2 December 2017
Reasoning about SM with Vault on DC/OS and
Open Source Tools

2. whois
Hello! I’m Mary. I’m a Security Engineer at Praekelt.org, an Non-Profit
Organisation that provides digital solutions to support quality of life worldwide.
Previously a penetration tester at MWR InfoSecurity.
medium/@racter

3. Purpose of this Talk
• An introduction to secret
management
• Introduction to some cybersecurity
tools and primitives for secret
management at scale
• Some learnings about how and how
not to handle secrets

4. Intro: What is a Secret?

5. A secret is some knowledge, or piece of data, that is
hidden from entities who are not entitled to know it.
Knowledge of a secret is commonly used to validate
an entity’s identity.
Examples of secrets in computing:
● Passwords
● RSA Private Keys
● Encryption Keys
● API Tokens

6. There are a lot of ways to infiltrate a software
system, but compromising secrets are one of the
most reliably low-tech methods that present a
good chance of success.
That’s why guessing passwords forms a part of
every hacker and pentester’s offensive arsenal.

7. Anatomy of a Secret (for hackers)

8. “A full 80% of data breaches are caused
by silly mistakes by those responsible for
managing secrets - It’s not that the
adversaries are so sophisticated.”
- Rashmi Jha, DigiCert Security Summit 2017

9. A Story of Secrets at
(medium) Scale:
The Harsh Tale of
Praekelt.org

10. Open-source software form the backbone of
Praekelt.org’s software infrastructure.
We run our Python web applications as Docker
Containers on the open-source version of
Mesosphere’s DC/OS, a cluster-based container
orchestration platform.
We host our codebases on GitHub.

12. Our containers run webapps that need stateful
services like databases and message queues. How
do these webapps get access to the stateful
services?
Why, by authenticating against them with a
secret, of course.

13. At the moment, we create and configure stateful
services on persistent hosts using Puppet.
Any usernames and passwords required on
those services are described in the Puppet
config, which sits in GitHub.

14. We then…
Copypasta those credentials…
...into environment variables.
USERNAME=’admin’
PASSWORD=’admin’
By the way, the database
you need to connect to is
on 10.0.0.5
It’s called ‘postgres’
Your username is ‘admin’
and your password is
‘admin’

15. This is pretty risky.

16. • If someone manages to break into your container’s environment,
reading the environment variables to gain access to secrets is
trivial.
• Environment variables are commonly exposed in application logs.
• Many webapp frameworks’ DEBUG mode will display environment
variables by default.
• Credential leaks could happen if your process forks to interact
with a third-party application with access to the child environment.
RISKS OF PASSING SECRETS AS ENVIRONMENT VARIABLES

17. • Git is designed to preserve history. Unrevoked credentials in
Git history are a point of exposure.
• If someone at GitHub *really* wanted our secrets, they could
get them.
• It is quite possible to make a private repo public by accident.
• Easy to expose more secrets than needed to third-party
contractors and interns
RISKS OF STORING SECRETS IN GITHUB (IN A PRIVATE REPO)

18. • Easy for those with access to fork/clone secrets wholesale.
• Coarse access control - can control access to repositories, but one
repository could contain secrets of varying sensitivities
• You need to rotate and revoke credentials manually, which becomes
more tedious the more stateful services you need to manage.
Importantly…
It does not allow for security best practice at scale.
RISKS OF STORING SECRETS IN GITHUB (IN A PRIVATE REPO)

21. Secret Management Tasks
• Create secrets
• Store secrets
• Distribute secrets
• Manage secret life cycles

22. We’re migrating
our secret
creation and
storage to
HashiCorp’s
Vault.

23. It is well-maintained, open-source, and designed with
high-availability and container orchestrators in mind
Vault is a Secret
Management
Solution

24. “Vault generates, secures, stores, and
controls access to tokens, passwords,
certificates, API keys, and other secrets.
It handles leasing, key revocation, key
rolling, and auditing.
Through a unified REST API, users can
access an encrypted Key/Value store and
network encryption-as-a-service, or
generate AWS IAM/STS credentials,
SQL/NoSQL databases, X.509 certificates,
SSH credentials, and more.”

25. Vault is only one piece of the puzzle. It can generate
and store secrets securely for you, but it does not
handle secure distribution.

26. Secret Management Tasks
• Create secrets ✓
• Store secrets ✓
• Distribute secrets
० Get the secret from Vault to the correct consumer
० Keep the secret safe from exfiltration during transit
० Scaling secret distribution with large number of consumers
• Manage secret life cycles

27. Security Management Primitive: Trust
• Trust between software actors refers to
waiving the frequency or rigour with which
authorisation routines are conducted for
privileged requests, if some preconditions
are satisfied
० Example preconditions: same network
trust zone (VPNs, internal networks),
valid session token
• Trust can be one-way or mutual
• Mutual trust + degree of interdependence
= coupling

28. $$$ Solutions
• Many enterprise container orchestration platforms have components
coupled with a secret store which inject secrets into containers at the
time of launch, in env or mounted volumes (eg. DC/OS EE,
Kubernetes/GCE)
• There are also open-source tools that leverage this paradigm as well
but require ecosystem buy-in (eg. EnvConsul)
What do you do if you don’t have control over your scheduler logic,
can’t afford an enterprise license of a product, and/or don’t want to
invest in a new orchestration ecosystem?

29. Naive Distribution Workflow: Container asks Vault for
Secrets After Launch
• If the container authenticates to Vault with a secret, how does that
secret get there in the first place?
• How does Vault confirm the identity and permissions of the client
container?
• Solving this by trusting all actors who can connect over a private
interface is too coarse-grained

30. The Secure Introduction
Problem

31. • If we can securely get the initial secret granting the
container access to Vault, then the container can
securely fetch all subsequent secrets.
• But how do we fetch this first secret?
THE SECURE INTRODUCTION PROBLEM

32. General Solution: Using a
Secure Introduction Agent
Jeff Mitchell, Secure Introduction At Scale: Think Like A Vault Developer,
ContainerDays NYC 2016 Talk: https://www.youtube.com/watch?v=R-jJXm3QGLQ

33. A Secure Introduction Agent is closely coupled with
the cluster scheduler, and maintains a mapping of
container properties (such as app name) to Vault
policies.
To minimise the attack surface of the initial
secret, we use wrapped tokens.

34. Secret Distribution Primitive: Wrapped Tokens
• A single-use token whose sole purpose is to
encapsulate the true token value
• Once the true token value is extracted, the
wrapping token is useless
० Lowers risk of passing them as
environment variables
० Lowers risk of exposure through logs of
intermediary services

47. Excellent...But…
admin:admin? Again? For all clients connecting to
that resource?
Let’s try kill this bird (and another one) next.

48. Secret Management Tasks
• Create secrets ✓
• Store secrets ✓
• Distribute secrets ✓
• Manage secret life cycles
० Revoke secrets from entities no longer requiring them
० Revoke compromised secrets and issuing new ones (key-rolling)
० Destroy invalid secrets
० Prevent re-use of secret value

49. Goals of Secret Lifecycle Management
● Reduce validity period of secret to narrow its temporal attack surface
● Reduce algorithmic attack surface by not exposing expired credentials with
the same generation method
● Reduce usefulness of compromised credentials to malicious parties
● Automating this at scale

50. What Does Automated Lifecycle Management Look Like?
A couple of primitives and some glue ;)

51. Vault Secret Management Primitive: Dynamic Secrets
• Dynamic secrets are lazily generated when they are needed from one
“master” secret
• Prevents hard-coding of secrets
• Prevents secret re-use by automating new secret generation
• Supports automated renewal and rotation of secrets
• Scales well for unique passwords in 1:∞ resource:client scenarios

52. Dynamic Secrets Example: Postgres
● The secret management service holds a master secret (username +
password) to a Postgres database
० This master secret is authorised to create new roles
● When a consumer needs access to that database, it requests a new set of
dynamic secrets from Vault
● Vault authenticates to the Postgres database with the master secrets, then
runs queries to create the dynamic secret
● Vault wraps the new secret with some metadata and returns it to the
consumer

53. Vault Secret Management Primitive: Leases
● Leases are metadata for issued secrets that describe their validity
० Every dynamic secret and auth token issued by Vault has a lease ID
० A lease contains info on the validity period (Time to Live) of a secret,
renewability, etc.
● Leases allow the validity period of secrets to be extended or for secrets to be
revoked by referencing the lease ID
● Leases with a short TTL forces consumers to check in with Vault
continuously to keep the secrets from expiring
● Dynamic secrets that are no longer used are revoked automatically with the
lease expiry mechanism

54. Putting it Together
● At container launch, a helper process in
the container fetches the required secrets
(to a file or to the environment)
● The helper process also makes calls to
Vault to renew the leases on those secrets
● As long as the container is alive, the
secrets remain valid
● If the container dies for any reason, the
helper process stops renewing leases and
lets those secrets expire

55. In conclusion
● Secret management in medium-scale, open-source system still relatively
unexplored
● In a pinch, you can use your scheduler as an identity server for clients that
consume secrets
● Moving beyond storing secrets in cloud repositories is possible without paying
fiat currency
● Most secret management solutions for container orchestration platforms
exploit trust and couplings to distribute secrets - see if you can spot where it
happens!

56. Open Source Tools
• HashiCorp Vault - https://www.vaultproject.io
• Vault-Gatekeeper-Mesos -
https://github.com/ChannelMeter/vault-gatekeeper-mesos
• Vaultkeeper - https://github.com/praekeltfoundation/vaultkeeper
• EnvConsul - https://github.com/hashicorp/envconsul

57. Thank you!
Questions?
Feel free to drop me mail about Praekelt or
secret management at mary@praekelt.org