Chickens & Eggs: Managing secrets in AWS with Hashicorp Vault

Chickens & Eggs
Managing secrets in AWS with Hashicorp Vault
Jeff Horwitz
Jornaya
jhorwitz@jornaya.com

Applications need secrets
• Where do we store the secrets?
• How do we manage the secrets?
• How do servers/applications obtain the secrets?

Jornaya Environment
• Applications run in AWS in Autoscaling groups
• Conﬁguration management via Chef server
• No PCI, HIPAA or other compliance requirements
• Secrets include passwords, private keys, API keys

Vault
“Vault secures, stores, and tightly controls access to tokens,
passwords, certificates, API keys, and other secrets in modern
computing. Vault handles leasing, key revocation, key rolling, and
auditing. Through a unified API, users can access an encrypted
Key/Value store and network encryption-as-a-service, or generate
AWS IAM/STS credentials, SQL/NoSQL databases, X.509
certificates, SSH credentials, and more.”
https://www.vaultproject.io/

Vault
✓ Platform independent
✓ High availability
✓ User & server authentication options
✓ Fine-grained access control
✓ Good language and tooling support
✓ Under active development

Other Options
• Instance userdata
• Chef encrypted databags
• Chef vault
• Citadel
• KMS

Vault Server
• Responds to client requests
• Interacts with backends
• storage, authentication, secret, audit
• Encrypts/Decrypts secrets with master key
• Master key is never stored on disk

Playing with Vault
~$ vault server --dev
==> WARNING: Dev mode is enabled!
In this mode, Vault is completely in-memory and unsealed.
Vault is configured to only have a single unseal key. The root
token has already been authenticated with the CLI, so you can
immediately begin using the Vault CLI.
The only step you need to take is to set the following
environment variables:
export VAULT_ADDR='http://127.0.0.1:8200'
The unseal key and root token are reproduced below in case you
want to seal/unseal the Vault or play with authentication.
Unseal Key: e95cf9d02c044c67dfd5d379d3a56cee7a0209f8cc681fca1435b6c022fcf028
Root Token: 611f6670-f871-ef2d-2110-87a54261d407

Talking to Vault
• Vault provides an HTTP(S) RESTful API
• JSON responses
• The vault command is a user-friendly wrapper
• Modules available for various languages
• Conﬁg management (e.g. Chef via Ruby gem)

Reading & Writing Secrets
~$ export VAULT_ADDR=http://127.0.0.1:8200
~$ vault write secret/phillydevops/luggage-combo value=12345
Success! Data written to: secret/phillydevops/luggage-combo
~$ vault read secret/phillydevops/luggage-combo
Key Value
--- -----
refresh_interval 720h0m0s
value 12345

HTTP API
~$ curl -X POST -d '{"value": "12345"}'  
-H 'X-Vault-Token: 611f6670-f871-ef2d-2110-87a54261d407' 
http://127.0.0.1:8200/v1/secret/phillydevops/luggage-combo

HTTP API
~$ curl -k -H 'X-Vault-Token: 611f6670-f871-ef2d-2110-87a54261d407'  
http://127.0.0.1:8200/v1/secret/phillydevops/luggage-combo | jq
{
"request_id": "ac71da58-cc11-431b-598e-7eb3fe3e37a3",
"lease_id": "",
"renewable": false,
"lease_duration": 2592000,
"data": {
"value": "12345"
},
"wrap_info": null,
"warnings": null,
"auth": null
}

Shamir’s Secret Sharing
• https://en.wikipedia.org/wiki/Shamir's_Secret_Sharing
• Splits a key into n shards (unseal keys)
• k shards needed to derive original key (k < n)
• No one person can easily obtain the original key
• No need to revoke shards if people leave

Unsealing Process
• Vault server does not store the master key on disk
• Given the master key via the unsealing process
• Need minimum number of shards to unseal
• Key is only stored in memory
• Restarting will lose the key and “seal” the vault.

Unsealing Demo
~$ vault read secret/phillydevops/luggage-combo
Error reading secret/phillydevops/luggage-combo: Error making API request.
URL: GET http://127.0.0.1:8200/v1/secret/phillydevops/luggage-combo
Code: 503. Errors:
* Vault is sealed
~$ vault unseal
Key (will be hidden): ********
Sealed: false
Key Shares: 1
Key Threshold: 1
Unseal Progress: 0

Backends
Storage backends
Secret backends
Authentication backends
Audit backends

Storage Backends
• Physical storage for secrets
• Only see encrypted secrets
• High availability features

Storage Backends
Backend Support HA?
Consul Official Yes
Zookeeper Community Yes
etcd Community Yes
DynamoDB Community Maybe
S3 Community No
Swift Community No
Azure Community No
mysql Community No
postgresql Community No
inmem Official No
file Official No

Clustering
• Storage backend must support high availability
• Active and standby servers
• State maintained by Consul (storage backend)
• Use Consul DNS or API to discover active server
• Standby will redirect to active by default

Cluster Status
$ vault status
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Version: Vault v0.6.1
Cluster Name: vault-cluster-abcd0123
Cluster ID: deadbeef-beef-beef-beef-deadbeefcafe
High-Availability Enabled: true
Mode: active
Leader: https://vault-1.example.com:8200

Secret Backends
• Static or dynamic secrets
• Backend conﬁguration
• Multiple backend mountpoints

Generic Backend
• Default backend
• Mounted at secret
• Stores and retrieves static secrets as k/v pairs
• Great for:
• username/password
• RSA private keys
• API keys

Tokens
• Authentication in vault is performed via tokens
• UUIDs returned by authentication backends
• Tokens can expire and must be renewed
• Tokens can be revoked
• A single “root” token never expires

Authentication Backends
• Returns a token based on credentials
• Tokens may have a TTL (backend-dependent)
• Options to ﬁt different workﬂows and platforms

Authentication Backends
• AppRole (App ID)
• AWS EC2
• GitHub
• LDAP
• TLS Certiﬁcates
• Tokens
• Username & Password

LDAP Backend
~$ vault auth -method ldap username=jhorwitz
Password (will be hidden):
Successfully authenticated! You are now logged in.
The token below is already saved in the session. You do not
need to "vault auth" again with the token.
token: deadbeef-cafe-beef-beef-deadbeafcafe
token_duration: 28799
token_policies: [admins, default]

Lookup a Token
~$ vault token-lookup deadbeef-cafe-beef-beef-deadbeafcafe
Key Value
--- -----
accessor 5776eb4b-05b1-3bab-98d3-08d34040a806
creation_time 1480366497
creation_ttl 28800
display_name ldap-jhorwitz
explicit_max_ttl 0
id deadbeef-cafe-beef-beef-deadbeafcafe
meta map[policies:admins,default username:jhorwitz]
num_uses 0
orphan true
path auth/ldap/login/jhorwitz
policies [admins default]
renewable true
ttl 27379

aws-ec2 Backend
• Good for server-level authentication
• Fixes the chicken & egg problem on EC2 instances
• Let AWS do the dirty work for you

Instance Identity
$ curl http://169.254.169.254/latest/dynamic/instance-identity/document
{
"privateIp" : “10.0.0.1”,
"devpayProductCodes" : null,
"availabilityZone" : "us-east-1a",
"accountId" : "1234567890",
"version" : "2010-08-31",
"instanceId" : "i-deadbeef",
"billingProducts" : null,
"instanceType" : "t2.nano",
"pendingTime" : "2016-09-09T17:30:35Z",
"architecture" : "x86_64",
"imageId" : "ami-cafebeef",
"kernelId" : null,
"ramdiskId" : null,
"region" : "us-east-1"
}

Identity Signature
$ curl http://169.254.169.254/latest/dynamic/instance-identity/pkcs7
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAaCAJIAEggGjewog
ICJwcml2YXRlSXAiIDogIjEwLjUwLjAuNjgiLAogICJkZXZwYXlQcm9kdWN0Q29kZXMiIDogbnVs
bCwKICAiYXZhaWxhYmlsaXR5Wm9uZSIgOiAidXMtZWFzdC0xYSIsCiAgImFjY291bnRJZCIgOiAi
Mjk4Nzg1NDUzNTkwIiwKICAidmVyc2lvbiIgOiAiMjAxMC0wOC0zMSIsCiAgImluc3RhbmNlSWQi
IDogImktYzRkMDQwZjUiLAogICJiaWxsaW5nUHJvZHVjdHMiIDogbnVsbCwKICAiaW5zdGFuY2VU
…

Whitelisting
• Vault maintains a whitelist of instance IDs
• Cannot reauthenticate using an ID in the whitelist
• You can turn this off or remove ID from whitelist
• Use a nonce to prevent replay attacks

Roles
• Clients specify a role with authentication requests
• Roles are “bound” to instance properties
• IAM role or instance proﬁle
• AMI ID
• Vault cross-checks with EC2 before returning token

Creating a Role
$ vault write auth/aws-ec2/role/deployer  
bound_iam_role_arn=arn:aws:iam::1234567890:instance-profile/deployer
policies=deployer 
Success! Data written to: auth/aws-ec2/role/deployer

Policies
• ACLs that are applied to roles, users and groups
• Applied at token creation
• Tokens can have multiple policies

Sample User Policy
path "secret/test/*" {
policy = "write"
}
path "secret/*" {
policy = "read"
}
path "secret/classified/*" {
capabilities = ["deny"]
}

Sample Server Policy
path "secret/env/prod/*" {
policy = "read"
}
path "secret/shared/*" {
policy = "read"
}
path "secret/ssl/private-key" {
policy = "read"
}

Integration
• Chef authenticates instance to aws-ec2 backend
• Chef stores token
• local ﬁlesystem (for scripts and apps)
• node run state (for Chef recipes)

Get PKCS7 Signature
def instance_identity_pkcs7
uri = URI.parse('http://169.254.169.254/latest/dynamic/instance-identity/pkcs7')
resp = Net::HTTP.get_response(uri)
resp.body.delete("n", '')
end

Generate Nonce
nonce = SecureRandom.uuid

Send Login Request
def aws_ec2_login(address, role, pkcs7, nonce)
# find the leader, since Net::HTTP doesn't handle redirects
uri = URI.parse("#{address}/v1/sys/leader")
resp = Net::HTTP.get_response(uri)
leader = JSON.parse(resp.body)['leader_address']
uri = URI.parse("#{leader}/v1/auth/aws-ec2/login")
data = {
'role' => role, 'pkcs7' => pkcs7, 'nonce' => nonce
}
req = Net::HTTP::Post.new(uri, 'Content-Type' => 'application/json')
req.body = data.to_json
resp = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) do |http|
http.request(req)
end
…

Write Token to File
file node['leadid_vault']['client']['token_file'] do
content lazy { node.run_state['token'] ||
node['leadid_vault']['client']['token'] }
owner node['leadid_vault']['client']['uid']
group node['leadid_vault']['client']['gid']
mode '0440'
sensitive true
action :create_if_missing
only_if { node['leadid_vault']['client']['enabled'] }
end.run_action(:create)

Using the Token
• Chef recipes use the vault gem
• Other apps use consul-template

Conﬁgure Vault Chef gem
def configure(opts = {})
require 'vault'
Vault.configure do |config|
config.address = opts[:address]
config.ssl_ca_cert = opts[:ssl_ca_cert]
config.ssl_verify = opts[:ssl_verify]
if opts[:token]
config.token = opts[:token]
elsif opts[:token_file]
config.token = File.read(opts[:token_file])
else
raise 'must specify either token or token_file'
end
end
end

Helper to read secrets
def read(path)
require 'vault'
Vault.logical.read(path)
end

Reading secrets in Chef
Chef::Recipe.send(:include, VaultHelpers)
if node['leadid_vault']['client']['enabled']
# Grab the Datadog API and Application Key from vault
api_key = read(
"#{node['leadid_base']['datadog']['secret_path']}" +
"/datadog/api-key").data
node.default['datadog']['api_key'] = api_key[:value]
end
…
include_recipe 'datadog::dd-agent'
include_recipe 'datadog::dd-handler'

consul-template
$ consul-template
-consul consul.example.com
-template "/opt/jornaya/app.yml.ctmpl:/opt/jornaya/app.yml:service nginx restart"

consul-template
database:
host: db.example.com
dbname: myapp
{{with secret "secret/leadid/env/prod/db-user"}}
user: {{.Data.username}}
password: {{.Data.password}}
{{end}}

consul-template
database:
host: db.example.com
dbname: myapp
user: donald
password: WeHaveTheBestPasswords

Learnings
• Don't use a self-signed certiﬁcate. There is pain.
• For SSL, name your servers or use static IP addresses.
• Clustering was the easiest part. Do it.
• Set up your path hierarchy ahead of time.
• Be prepared to RTFM.
• Different backends behave differently.
• Tokens do not live forever.

Chickens & Eggs: Managing secrets in AWS with Hashicorp Vault

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (20)

Similaire à Chickens & Eggs: Managing secrets in AWS with Hashicorp Vault

Similaire à Chickens & Eggs: Managing secrets in AWS with Hashicorp Vault (20)

Dernier

Dernier (20)

Chickens & Eggs: Managing secrets in AWS with Hashicorp Vault