Slides from my talk at DevOpsPro Moscow 2019

1. Vault configuration as code via Terraform:
stories from trenches
Andrey Devyatkin

2. Big thanks
to
DevOpsPro team!

4. I’m Andrey
● Enjoying life as technology
specialist, father and endurance
athlete
● 10+ years in the industry
● Writing tools
● Fixing automation, projects and
organisations
● Meetups/conferences organizer
● Trainer
● Certified this and that

5. Why this presentation? What to expect?
● Not pretending to be an expert just sharing what worked and what didn’t
● Share what worked for us and hopefully save some time for some of you
● A lot of technical details and references
● Slides will be available online - you don’t have to remember/photo everything

6. Agenda
● Context
● Deployment
● Configuration
● Integration
● Access control and development workflow
● Unexpected findings

7. Why Vault?
● Centrally Manage Secrets to Reduce
Secrets Sprawl
● Shift from static secrets to short-time
dynamically generated ones
● Protect Sensitive Data Across Clouds
and Private Datacenters

8. Where do we start?
Collect requirements and clarify context

9. Questions to ask - deployment and operations
● Where to deploy? VM? Container? Baremetal?
● Patch or scratch?
● How to access? VPN? Public? Service mesh?
● How to auto-unseal?
● How to get in initial secrets? (Ex. TLS)
● What storage is available?
● Where to stream logs?
● Where to stream telemetry?
● How to extract audit files?
● HA? DR?
● One per env or one for all?

10. Look for best practices and templates
● Why not to make use of https://github.com/hashicorp/terraform-aws-vault, right?
● With some small tweaks
● And little more tweaks
● And some more tweaks
(TLS certs handling, consul-less, dynamic configuration depending on environment,
rolling upgrades via multiple auto scaling configurations, audit files sync, etc)
Another option - https://github.com/hashicorp/vault-helm
Some inspiration
https://learn.hashicorp.com/vault/operations/ops-reference-architecture

11. Vault production (min) readiness checklist
● TLS termination
● Vault HA storage - ACL and encryption
● Local storage encryption
● Auto-unseal using KMS
● Stripped down image, infra as code,
encryption, minimal exec rights
● No ssh or other kind of remote access, NACL
for outgoing traffic
● IDS
● Backups and DR
● Logs and telemetry export from the node
● Audit on, sync audit files to remote storage,
integrity check for audit files
● Sync audit files to archive
● Audit files parsing and anomaly detection
● Availability/performance monitoring and
alerting
More here https://learn.hashicorp.com/vault/operations/production-hardening

12. Context - before Vault
● Applications running in containers
● Orchestrated by Kubernetes
● Running in AWS
● Secrets in configmaps/secrets
● Apps require database connection and connection to other cloud services (i.e.
database creds and cloud access creds), other static secrets
● Developers pulling secrets from k8s secrets

13. Deployment
● EC2, classic ELB, auto-scaling group per AZ (for rolling updates)
● Immutable infra
● Initial secrets baked in as encrypted archive that could be un-encrypted only with key
accessible to Vault instances
● Behind VPN
● Auto-unseal with KMS
● HA storage in DynamoDB with point in time restore
● Logs in CloudWatch
● Telemetry in Prometheus

14. It would be a good idea
to
split deployment terraform spec and
configuration terraform spec

15. Vault is up and running. What is next?

16. To start configuring Vault via Terraform we need...
● Vault URL configured as VAULT_ADDR env variable
● Vault token (root token will do for the start but revoke it afterwards together with the
rest of the root tokens)
● A good idea what are you after…
More here https://www.youtube.com/watch?v=fOybhcbuxJ0 and here
https://www.terraform.io/docs/providers/vault/index.html

17. One slide Vault intro
LDAP
k8s
App
Role
AWS
...
Auth methods
Vault
token
AWS
Data
base
Secret Engines
Rabbit
MQ
PKI
Database login credentials
AWS access keys
RabbitMQ logic credentials
Certificates
Lease
Audit device
More here https://www.youtube.com/watch?v=VYfl-DpZ5wM
KV
Transit Encrypted data
Secret value
Vault
policies

18. Token
Be aware of TTL and Max TTL

19. Auth methods and policies
Boring but very important

20. You probably need more than one...
● Humans - operators and developers
● Machines - CI/CD, bots, etc
● Things - Apps, Infra etc
A good idea is to use MFA for humans, limit from where auth methods could be invoked

21. Auth -> Role -> Token with policy
Ex.
LDAP -> LDAP Group -> Token with policy

22. LDAP
● Leverages existing IAM setup
● Delegates credentials validation
● Used my humans
● Would be a good idea to simplify login procedure for your users
More here https://www.vaultproject.io/docs/auth/ldap.html

23. LDAP
resource "vault_ldap_auth_backend" "ldap" {
path = "ldap"
url = "ldaps://dc-01.example.org"
userdn = "OU=Users,OU=Accounts,DC=example,DC=org"
userattr = "sAMAccountName"
upndomain = "EXAMPLE.ORG"
binddn = "${var.binddn}"
bindpass = "${var.bindpass}"
discoverdn = false
groupdn = "OU=Groups,DC=example,DC=org"
groupfilter = "(&(objectClass=group)(member:1.:={{.UserDN}}))"
}
https://www.terraform.io/docs/providers/vault/r/ldap_auth_backend.html

24. Things to consider
● token_no_default_policy
● token_bound_cidrs
● token_ttl
● token_max_ttl

25. LDAP role (backend group actually)
resource "vault_ldap_auth_backend_group" "group" {
groupname = "dba"
policies = ["dba"]
backend = "${vault_ldap_auth_backend.ldap.path}"
}
https://www.terraform.io/docs/providers/vault/r/ldap_auth_backend_group.html

26. Policy
data "vault_policy_document" "example" {
rule {
path = "secret/*"
capabilities = ["create", "read", "update", "delete", "list"]
description = "allow all on secrets"
}
}
resource "vault_policy" "example" {
name = "example_policy"
policy = "${data.vault_policy_document.example.hcl}"
}
https://www.terraform.io/docs/providers/vault/d/policy_document.html

27. Policy
● You will need a policy to manage policy...
● Deny by default
● Do not have to match LDAP group name but easier for users if it
does
● Member of multiple groups gets multiple policies
More here https://learn.hashicorp.com/vault/getting-started/policies

28. AppRole if you really have to...
● If you don’t have a better way
● Mostly used for CI
● Initial secret issue
● No good way to audit access
More here https://www.vaultproject.io/docs/auth/approle.html

29. AppRole
resource "vault_auth_backend" "approle" {
type = "approle"
}
resource "vault_approle_auth_backend_role" "example" {
backend = vault_auth_backend.approle.path
role_name = "test-role"
token_policies = ["default", "dev", "prod"]
}
https://www.terraform.io/docs/providers/vault/r/approle_auth_backend_role.html

30. AppRole

31. AppRole
resource "vault_approle_auth_backend_role_secret_id" "secret" {
backend = "approle"
role_name = "${vault_approle_auth_backend_role.role.role_name}"
}
locals {
kv = {
role_id = "${vault_approle_auth_backend_role.role.role_id}"
secret_id = "${vault_approle_auth_backend_role_secret_id.secret.secret_id}"
}}
resource "vault_generic_secret" "kv" {
path = "${vault_mount.kv.path}/approle"
data_json = "${jsonencode(local.kv)}"
}

32. Cloud IAM, K8S, etc
● Better way for non-interactive auth
● Leverages existing entities
● Delegates entity validation

33. AWS IAM
resource "vault_auth_backend" "aws" {
description = "Auth method for CI engines"
type = "aws"
}
resource "vault_aws_auth_backend_role" "ci_builder" {
backend = "${vault_auth_backend.aws.path}"
role = "ci-builder"
auth_type = "iam"
bound_iam_principal_arns = ["${data.aws_iam_role.ci_builder.arn}"]
token_ttl = 3600
token_max_ttl = 3600
token_policies = ["${vault_policy.ci_builder.name}"]
}
https://www.terraform.io/docs/providers/vault/r/aws_auth_backend_role.html

34. AWS IAM
resource "aws_iam_user" "ci_builder" {
name = "${module.iam_auth_for_ci_user_name.qualified_name}"
tags = "${module.tags.default}"
}
resource "aws_iam_access_key" "ci_builder" {
user = "${aws_iam_user.ci_builder.name}"
}
resource "aws_iam_user_policy" "ci_builder" {
name = "Allow-Vault-to-look-up-users-for-iam-auth"
user = "${aws_iam_user.ci_builder.name}"
policy = "${data.aws_iam_policy_document.ci_builder.json}"
}

35. AWS IAM
# https://www.vaultproject.io/docs/secrets/aws/index.html#example-iam-policy-for-vault
data "aws_iam_policy_document" "iam_auth_for_ci" {
statement {
effect = "Allow"
actions = ["iam:GetUser","iam:GetRole",]
resources = ["arn:aws:iam::${data.aws_caller_identity.i.account_id}:*"]
}
statement {
effect = "Allow"
actions = ["sts:AssumeRole"]
resources = ["arn:aws:iam::${data.aws_caller_identity.i.account_id}:role/vault-cluster*"]
}
}

36. AWS IAM
resource "vault_aws_auth_backend_client" "ci_builder" {
backend = "${vault_auth_backend.ci_builder.path}"
access_key = "${aws_iam_access_key.ci_builder.id}"
secret_key = "${aws_iam_access_key.ci_builder.secret}"
}

37. K8S
resource "kubernetes_service_account" "vault_auth" {
metadata {
name = "vault-auth"
namespace = "${var.k8s_namespace}"
}
automount_service_account_token = "true"
}
data "kubernetes_secret" "sa" {
depends_on = ${kubernetes_service_account.vault_auth}
metadata {
name = "${kubernetes_service_account.vault_auth.default_secret_name}"
}
}

38. K8S
resource "kubernetes_cluster_role_binding" "vault_auth" {
metadata { name = "role-tokenreview-binding" }
role_ref {
api_group = "rbac.authorization.k8s.io"
kind = "ClusterRole"
name = "system:auth-delegator"
}
subject {
kind = "ServiceAccount"
name = "${kubernetes_service_account.vault_auth.metadata.name}"
namespace = "${var.k8s_namespace}"
api_group = ""
}
}

39. K8S
resource "vault_kubernetes_auth_backend_config" "config" {
backend = "${vault_auth_backend.kubernetes.path}"
kubernetes_host = "${var.k8s_api}"
kubernetes_ca_cert = "${data.kubernetes_secret.sa.data.ca.crt}"
token_reviewer_jwt = "${data.kubernetes_secret.sa.data.token}"
}
resource "vault_kubernetes_auth_backend_role" "role" {
backend = "kubernetes"
role_name = "${var.app_name}"
bound_service_account_names = ["${kubernetes_service_account.app.metadata.0.name}"]
bound_service_account_namespaces = ["${var.k8s_namespace}"]
token_ttl = "${var.ttl}"
token_policies = ["${vault_policy.read-psql.name}, "${vault_policy.read-aws.name}"]
}

40. K8S
Here are more details if you are interested
https://www.youtube.com/watch?v=t6ZKhY0-_cA

41. I got token. What is next?

42. KV
AWS SSM Parameters Store -> KV for humans
Configmaps/Secrets -> KV for Apps

43. KV
Human initiated secret storage for static secrets
and
Machine initiated-readable storage for static secrets

44. AWS
resource "aws_iam_user" "user" {
name = "vault-aws-admin"
tags = "${module.tags.default}"
}
resource "aws_iam_access_key" "key" {
user = "${aws_iam_user.user.name}"
}
resource "aws_iam_user_policy" "policy" {
name = "Allow-Vault-to-create-temp-users"
user = "${aws_iam_user.user.name}"
policy = "${data.aws_iam_policy_document.document.json}"
}

45. AWS
https://www.vaultproject.io/docs/secrets/aws/index.html#example-iam-policy-for-vault
"iam:AttachUserPolicy" "iam:ListGroupsForUser"
"iam:CreateAccessKey" "iam:ListUserPolicies"
"iam:CreateUser" "iam:PutUserPolicy"
"iam:DeleteAccessKey" "iam:RemoveUserFromGroup"
"iam:DeleteUser"
"iam:DeleteUserPolicy"
"iam:DetachUserPolicy"
"iam:ListAccessKeys"
"iam:ListAttachedUserPolicies"

46. AWS
resource "vault_aws_secret_backend" "aws" {
description = "AWS secret engine so operators can get temporary keys"
path = "aws"
region = "${data.aws_region.r.name}"
access_key = "${aws_iam_access_key.key.id}"
secret_key = "${aws_iam_access_key.key.secret}"
default_lease_ttl_seconds = "28800"
max_lease_ttl_seconds = "86400"
}

47. AWS
resource "aws_iam_role" "admin" {
name = "admin"
max_session_duration = "28800"
assume_role_policy = "${data.aws_iam_policy_document.trust.json}"
tags = "${module.tags.default}"
}
resource "vault_aws_secret_backend_role" "access-aws-admin-role" {
backend = "${vault_aws_secret_backend.aws.path}"
name = "access-aws-admin-role"
role_arns = ["${aws_iam_role.admin.arn}"]
credential_type = "assumed_role"
}

48. AWS
# https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.html
data "aws_iam_policy" "admin" {
arn = "arn:aws:iam::aws:policy/SystemAdministrator"
}
resource "aws_iam_role_policy_attachment" "admin" {
role = "${aws_iam_role.admin.name}"
policy_arn = "${data.aws_iam_policy.admin.arn}"
}

49. AWS
data "aws_iam_policy_document" "trust" {
statement {
effect = "Allow"
actions = ["sts:AssumeRole"]
principals {
type = "AWS"
identifiers = ["${aws_iam_user.user.arn}"]
}
}
}

50. AWS
Works in the same way for apps and humans!

51. AWS
Use temporary AWS creds to generate sign-in AWS console URL! No SSO needed!
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_enable-console
-custom-url.html

52. For more inspiration https://youtu.be/Y0er4UCmqiA

53. Database creds
● Creation and revocation statements are hard
● Use RDS IAM auth if possible

54. Secrets rotation
DB_SECRET_ENGINE_MOUNTS=$(vault secrets list -format=json | jq -r '. | to_entries[] | select(.value.type |
startswith("database")) | .key')
for DB_SECRET_ENGINE_MOUNT in ${DB_SECRET_ENGINE_MOUNTS}; do
DB_CONNECTION_NAMES=$(vault list -format=json ${DB_SECRET_ENGINE_MOUNT}config | jq --raw-output .[])
for DB_CONNECTION_NAME in ${DB_CONNECTION_NAMES}; do
vault write -force ${DB_SECRET_ENGINE_MOUNT}rotate-root/${DB_CONNECTION_NAME}
done
done

55. Secrets rotation
AWS_USERS=$(aws iam list-users --query "Users[?starts_with(UserName, 'vault-aws-')].UserName" --output text)
for AWS_USER in ${AWS_USERS}; do
KEYS_ID=$(aws iam list-access-keys --user-name ${AWS_USER} --query "AccessKeyMetadata[*].AccessKeyId" --output
text)
for KEY_ID in ${KEYS_ID}; do
aws iam delete-access-key --access-key-id ${KEY_ID} --user-name ${AWS_USER}
done
done
terraform apply
Note! Keys are still in Terraform state - encrypt state storage and state itself!

56. Unexpected findings

57. KV state issue
● Terraform provider for Vault in some cases(?) does not re-read KV and newly added
values are not readable/found
● terraform state rm data-source
data "vault_generic_secret" "rundeck_auth" {
path = "secret/rundeck_auth"
}
provider "rundeck" {
url = "http://rundeck.example.com/"
auth_token = "${data.vault_generic_secret.rundeck_auth.data["auth_token"]}"
}

58. Vault and Terraform not always play together
resource "vault_database_secret_backend_connection" "postgres" {
count = "${var.enable_postgresql}"
backend = "${vault_mount.db.path}"
name = "${var.postgresql_db_name}"
allowed_roles = ["${var.postgresql_role_name}", "${local.read_only_role_name}", "${local.admin_role_name}"]
data = {
username = "${var.postgresql_db_username}"
password = "${var.postgresql_db_username_password}"
}
postgresql {
connection_url =
"postgres://{{username}}:{{password}}@${var.postgresql_db_endpoint}:${var.postgresql_db_port}/${var.postgresql_db_name}"
max_open_connections = "${var.postgresql_max_open_connections}"
}
lifecycle {
ignore_changes = ["data.password"]
}
}

59. Vault and Terraform not always play together
resource "aws_iam_access_key" "key" {
user = "${aws_iam_user.user.name}"
}
resource "aws_iam_user_policy" "policy" {
name = "Allow-Vault-to-create-temp-users"
user = "${aws_iam_user.user.name}"
policy = "${data.aws_iam_policy_document.document.json}"
}
resource "vault_aws_secret_backend" "aws" {
description = "AWS secret engine for operators to get temporary keys"
path = "$humans-aws"
region = "${data.aws_region.r.name}"
access_key = "${aws_iam_access_key.key.id}"
secret_key = "${aws_iam_access_key.key.secret}"
default_lease_ttl_seconds = "28800"
max_lease_ttl_seconds = "86400"
}

60. Some misconceptions
data "vault_generic_secret" "db_creds" {
path = "${module.vault_config.database_path}/creds/${var.db_role_name}"
depends_on = ["module.vault_config"]
}
locals {
db_username = "${data.vault_generic_secret.db_creds.data["username"]}"
db_password = "${data.vault_generic_secret.db_creds.data["password"]}"
}
locals {
psql_url = "postgres://${local.db_username}:${local.db_password}@${module.db.connection_string}?sslmode=require”
}
resource "vault_generic_secret" "config" {
path = "${module.vault_config.kv_path}/config"
data_json = <<EOF
{
"Psql_url": "${local.psql_ur}"
}
EOF
}

61. How should I do with Lambdas?

62. PSQL creds revocation

63. Thanks!
Questions?
@andrey9kin
info@andreydevyatkin.com
https://www.linkedin.com/in/andreydevyatkin/