The document provides an overview of secret management solutions and architectures. It discusses what secrets are and why secret management is important. Some key points: - Secrets include authentication credentials, API keys, passwords, and certificates that need access control. As services increase, so do secrets. - An ideal secret management solution provides security, encryption, access control, auditing, ease of use, and integration with other tools. - Version control systems and orchestration tools like Kubernetes can be used for secrets but have limitations compared to dedicated secret management solutions. - AWS offers Parameter Store, Secrets Manager, and KMS for secret management. Parameter Store is generally recommended, while Secrets Manager is better for database

1. Overview of Secret
management Solutions and
Architectures
Yuechuan Chen

2. About this talk
Three goals:
1. Raise awareness of good secret management practices, what it is and why it’s important
2. Identify characteristics of a good solution
3. Overview of solutions on AWS, Azure and Kubernetes
Daniel Summerfield’s talk “Turtles all the way down”

3. What’s a secret
- A secret is anything that you want to control access to, such as API keys, passwords, certificates,
and more.
- More services = more authentication = more secrets
- People tend to take insecure shortcuts: hardcoding secrets in source code, container images,
configuration files. Credentials are shared via email, slack, and shared folders.

4. What is secret management
Secret management concerns with:
1. Storage
2. Secret lifecycle (creation, modification, distribution, destruction,
auditing - tracking secrets back in time)
3. Recovery and remediation

5. What’s an ideal secret management solution
- Security
- Encryption ( at rest/ in transit)
- Fine grained access control
- Good access logs
- Easy to manage and cheap to operate
- Central location to operate on secrets
- No more remembering where secrets are kept
- Easy to integrate, scalable

6. Version control & orchestrator
tools

7. SM with Version
Control
- Pros:
1. Easy to get started
2. Encryption at rest + in transit
3. Some compartmentalization
- Cons:
1. No access history
2. Difficult to rotate secret
3. Difficult to rotate encryption key
4. Require key management
5. Require additional protection
against tampering the repository
Bottom line: Only good for small projects

8. Orchestrator based solutions
Pro:
- No code change necessary, apps access secrets same way
as before.
- No need to provision decryption keys to individual nodes
- Can offload key management to KMS services, e.g. K8S
offers KMS plugin API since v1.10.0 [1]
- Access to the secrets can be audited
- Single source of truth
Cons:
- Trust between components need to be bootstrapped
- Orchestrator lock in, different tools offers different
features.

9. AWS Sec. Management
> Parameter Store
> Secrets Manager
> KMS

10. Parameter Store
- Strong Encryption
- Strong Access Control via
IAM
- No secret zero required
- Logging integrated with
CloudTrail possible SIEM
integrations

11. AWS Parameter Store with IAM role
Using IAM roles with Parameter Store is nice because it does not require
maintaining additional authentication tokens.
{
"Sid": "",
"Effect": "Allow",
"Action": "ssm:GetParameters",
"Resource": [
"arn:aws:ssm:*:*:parameter/SERVICE_N
AME/*",
]
},
● Secrets are namespace separated by `/`
● Grant access to a particular namespace:
"arn:aws:ssm:*:*:parameter/SERVICE_N
AME/db/*"

12. AWS Parameter Store pro and cons
Pro:
- Secure and scalable with no single point of failure.
- No server to manage
- Secrets are stored under paths, can grant permission to all secrets under a path
- Integrate with many services: EC2, ECS, Lambda, CloudFormation, CodeDeploy etc.
- Integrates with CloudWatch Events and Lambda trigger ( allows an event driven workflow)
- Secret rotation can be implemented as separate lambda functions
Con:
- 10k parameters per account and 4kbyte per secret
- Restricted by AWS KMS limits
- No rotation out of the box

13. AWS Secrets Manager
- New service
- Encrypted by default
- Support secret rotation via Lambda
- RDS secret rotation is supported by default
- Encryption backed by Keys in KMS
- Promotes programmatic retrieval of secrets
- Access control via IAM

14. AWS Secrets Manager
Pro:
- Application pull secrets at runtime
- Full automated RDS rotation
- Powerful access control with IAM policies
- Enforce TLS in transit and use KMS keys for encryption at rest
- Much cheaper than managing a Highly Available Hashicorp Vault cluster
Con:
- Application is locked into AWS ecosystem by having a dependency on ASM
- Limited auditing capabilities: CloudTrail only captures secret management events but not data
access events
- Does not offer much extra compare to parameter store

15. AWS KMS
- Backed by HSM
- More control over the key type
and storage
- No limit on key size and number
of secrets
- Require a lot of work

16. Recommendation:
Use Parameter store in most scenarios:
- Deploy to AWS
- Integrated services
- Does not mind the 10k secret and 4k size limit
Use Secrets Manager when:
- Working primarily with RDS databases ( credential rotation comes out of the box)

17. Azure Sec. Management
- Azure Key Vault Secret

18. Azure key Vault Secret
- Similar to Parameter store, Key Vault is the hosted secret management alternative in Azure. It’s
roughly equivalent to Parameter store + ACM + AKS in AWS
- Key Vault Secret can be encrypted by two types of keys: Software keys and Hardware keys.
- Integrates with many Azure services
- Max 25k bytes per secret
- Warning from Microsoft to keep highly sensitive data out of Key Vault ??!

19. Azure VM, Function and App Service
Same procedure for Azure VM, Function and App
Services
Associated services
- Azure Key Vault Secret
- Service Principal (SP) and Managed Service
Identity (MSI)
Example with Azure Function is shown below, VM and
App service also work similarly

20. Using Key Vault Secret with external apps and
services
Create Service Principal:
1. Create a Service Principal via Azure Active Directory > App registrations > New Application
Registration
2. Provide the app name, and an sign-on url to create the application.
3. Note down the application ID and create a new password
1
2
34. Grant permission to Key Vault

21. const Azure = require('azure'); // require the Azure SDK
const MsRest = require('ms-rest-azure');
MsRest.loginWithServicePrincipalSecret(
'7d5f93e7-b528-490d-925f-d80778538a8a', //app id
'ZVaIui1QaM+5oAT4iZIEv7mRLU+vIecLgTu3M41jly0=', // should be obtained dynamically
'motorolasolutions.microsoft.com', //app domain
(err, credentials) => {
if (err) throw err
let client = new KeyVault.KeyVaultClient(cred)
return client.getSecret('https://xxx.vault.azure.net/', 'secret', '')
})
.then( secret => { /*use secret */} )

22. Kubernetes Sec. Management
- K8S Secret
- H Vault integration using open source
projects
- H Vault integration using K8S Auth
Method

23. Kubernetes Overview
Master node is responsible for
coordinating the cluster, usually has the
following components:
- API Server
- Scheduler
- Controller
- ETCD Key-Value DB
Slave nodes runs containers.
Deploy on AWS with Kops on Azure with
AKS, Kubeadm. Minikube

24. Secret management solutions
Ways to manage and inject secrets to containers:
- Kubernetes Secrets
- Hashicorp Vault + Secret Initialization Container (kubernetes-vault, qubite implementation)
- Storing secrets in a secret object file is safer and more flexible than putting in a pod definition.

25. Kubernetes Secrets flow
1. Admin creates a secret via kubectl, that
makes create secret request to
the API Server
2. Secrets are written to database
3. Secrets are provisioned to the slave
node that’s running the container
4. Secrets are mounted as volume or
injected to the environment variable of
the target container
* detail here, example

26. Kubernetes secrets and some gotchas
- Secrets can be provisioned to a container or a namespace, containers under the namespace have
access to the secrets under the same NS.
- Secrets are written to a tempFS which are deleted on pod terminition.
- Secrets are size limited to 1Mb
- Make sure all secrets are created before referencing in containers, otherwise the Pod will hang
because container has trouble mounting secret volume
- Only possible to mount one secret per directory. Mounting a secret will mask the content of the
directory.

28. Some considerations
- Lock down API Server via access control (RBAC) mechanism from pods and human admins.
- By default, any user who can access the API Server can read all secrets
- More on on “controlling access to API Server”
- Use TLS for all API Server access
- ETCD database:
- Write access to ETCD is equivalent to gaining root on the kubernetes cluster.
- Secrets are, by default, stored as plaintext in etcd. enable encryption on etcd. *how-to
- Manage the symmetric encryption key by leveraging Azure Key Vault, AWS Parameter store. Etc.
- Enforce TLS between etcd cluster and API Server
- Restrict access to etcd
- Lock down access to the slave nodes.
- Anyone with root access on a node can read secret from the API Server by impersonating the kubelet.
- Lock down Kubelet: disable https-anonymous-auth, possible attack scenario
- Unless you specify some flags on Kubelet, it’s default mode of operation is to accept unauthenticated API
requests.
- Version control kubernetes configurations and store them securely ( git-secret or git-crypt for
example)

29. Kubernetes Secrets summary
- Secret auditing with Kubernetes Audit
- Revocation and rotation can be done by deleting and recreating secrets
- Easy to use and tightly integrated to kubernetes

30. Hashicorp
Vault

31. K8S-Vault*
Link to example
Link to project

32. General Recommendations
Use kubernetes Secret if:
- Secrets does not change often and are used exclusively within kubernetes
Use Vault with K8S Authentication method if:
- secrets need to be used outside of kubernetes containers

33. Solution comparison chart
AWS Parameter
store
AWS Secret
Manager
AWS KMS Azure Key Vault
Secret
K8S Secret Vault with K8S Vault with K8S
Auth method
Do we need to
provision secret
zero to our
app/cluster in order
to bootstrap trust?
No, IAM task role is
used to obtain a
STS token used to
transparently
authenticate to
parameter store1
No, IAM task role
does the wonder.
No, IAM task role is
used to obtain a
STS token used to
transparently
authenticate to
KMS
No, Managed
Service Identity
(MSI) is used to
obtain a secure
token to
transparently
authenticate to Key
Vault 1
Yes, database
encryption key and
tls certs need to be
provisioned to
setup the K8S
cluster via other
means
K8S Vault
Controller need to
be authenticated
with Vault.
Authentication
need to be setup
between Vault and
K8S.
Is it cloud provider
agnostic?
No, limited
advantage outside
of AWS
No No, limited
advantage outside
of AWS
No, limited
advantage outside
of Azure
Yes, but limited to
app running in
containers
managed by K8S
Yes, but relies on
Hashicorp Vault
Relies on
Hashicorp Vault
Amount of effort to
integrate into
application
Little Little A lot Little Little, K8S cluster
need to be
secured
Moderate, require
Vault and
Controllers
Little
Recommended
scenarios
Best for
deployments in an
AWS only
environment (
lambda, EC2
instances, ECS...)
RDS heavy use
cases
This approach is
generally not
recommended.
Useful for: A large
number/size of
secrets are
required. Scenario
where self
managed secret
and keys are
required.
Best for
deployments in an
Azure only
environment (VM,
Functions, App
Services, VSTS
etc.)
Good for
containers
orchestrated by
Kubernetes.
Secrets are not
used anywhere
else. Require other
means to provision
database
encryption key.
Good for secrets
required to be
shared across
platforms.
Integration with
K8S is possible via
open source
projects.
Best for secrets
required to be
shared across
platforms.
Simple integration
makes this the
best way to
manage secrets
on K8S

34. Thank you