Learn how a non-developer was able to write a Ruby app using a tool built with the HashiCorp stack.

1. A tale of application
development
Nicolas Corrarello - HashiConf 2018

4. Disclaimer
• Argentinian by birth, Italian by blood
• English is not my first language
• I’m not a developer
• Don’t criticise my programming language choices, I
basically had Christmas week last year to write this :D.
Client libraries available in lots of languages (https://
www.vaultproject.io/api/libraries.html / https://
www.consul.io/api/libraries-and-sdks.html)

5. (What’s the story) Morning
glory?
• Spending lots of time authoring documents
• Write an app to automate the process
• Don’t have time to maintain it so make it as readable as
possible, zero touch maintenance as possible

7. 1. IMPORTANT - READ BEFORE SIGNING. Are there Consul / Vault Tokens readily
available on the system (see Nomad or the myriad of Vault Authentication Methods
available, then use Vault to get a Consul token :D)
Are you provisioning this yourself or someone else doing it for you?
J. Developer

8. AbstractionsNot HashiCorp specific (duh)
“The best thing to come out of Java” - Variety

9. Abstract your code from lock-in
Minimal effort, greater portability
Using existing libraries (datamapper, dal.py,
hibernate) may help. Also may make things way
more complicated (in certain cases, see Rails)

10. require 'json'
require './models/vault'
##
# Persistence layer. Implemented in S3, but the interface is generic enough so
it can be migrated to other platforms.
class Persistence
##
# Create a persistence layer. Initialises a HashiCorp Vault session and obtains
credentials from it using the existing abstraction.
def initialize
@vault = LocalVault.new
@awscreds = self.awsClient
@data = {}
end
##
# Store something in the persistence layer. Uses namespace (which translates to
an s3 bucket), and a simple key/value. Object is stored as JSON.
def Store(key,value,namespace)
s3 = @awscreds
begin
obj = s3.bucket(namespace).object(key)
obj.put(body: value.to_json)
return true, nil
rescue
return false, "Error persisting object"
end
end
##
# Retrieve something from the persistence layer. Uses namespace (which
translates to an s3 bucket), and a key. Returns the full object in it's native
format.
def Retrieve(key,namespace)
[…]
end
##
# List objects by "glob". Returns a list of objects available. In this case as
S3 is somewhat hierarchical, glob would be 'customers' to to retrieve customer
objets, or 'pov' to return POV documents.
def List(glob,namespace)
[…]
end
end
Initialize the object
Store something

12. Let’s talk secrets… lots…
• AWS API Credentials for Deployment
• AWS Credentials for the application to read/write the
bucket
• JWT Issuer / Secret
• Google API Keys for logins
• Crypto

13. Crypto?
Not the kind that your
coworker keeps talking
about, you know who,
the self named Bitcoin
expert….

18. ##
# Creates a customer in the object, requires an sfdcAccountId (unique)
and a customer name. Information is encrypted and persisted, but kept in
plaintext in memory.
def Create(sfdcAccountId,name,domain)
unless sfdcAccountId.nil? || name.nil?
begin
cyphertext = @vault.logical.write("transit/encrypt/#{key}",
plaintext: Base64.encode64(value).gsub('n',''), context:
Base64.encode64(context).gsub('n',''))
cyphername = cyphertext.data[:ciphertext]
rescue
return false, "Error encrypting data"
end
customer = {
:id => sfdcAccountId,
:name => cyphername,
:contacts => nil,
}
customerm = {
:id => sfdcAccountId,
:name => name,
:contacts => nil,
}
key = 'customers/' + sfdcAccountId + '/customer.json'
begin
@persistence.Store(key,customer,"jamo-#{domain}")
if @data[domain] == nil
@data[domain] = {}
end
@data[domain][sfdcAccountId] = customerm
return true, nil
rescue
return false, "error persisting customer"
end
end
end
Encrypt a string
Persist it

19. ##
# Creates a customer in the object, requires an sfdcAccountId (unique)
and a customer name. Information is encrypted and persisted, but kept
in plaintext in memory.
def Create(sfdcAccountId,name,domain)
unless sfdcAccountId.nil? || name.nil?
begin
cyphername = @vault.encrypt(name,'foo', @jwt_secret)
rescue
return false, "Error encrypting data"
end
customer = {
:id => sfdcAccountId,
:name => cyphername,
:contacts => nil,
}
customerm = {
:id => sfdcAccountId,
:name => name,
:contacts => nil,
}
key = 'customers/' + sfdcAccountId + '/customer.json'
begin
@persistence.Store(key,customer,"jamo-#{domain}")
if @data[domain] == nil
@data[domain] = {}
end
@data[domain][sfdcAccountId] = customerm
return true, nil
rescue
return false, "error persisting customer"
end
end
end
##
# Encrypt a value with a defined key and
context for symmetric encryption.
def encrypt(value,key,context)
cyphertext =
@vault.logical.write("transit/encrypt/#{key}",
plaintext:
Base64.encode64(value).gsub('n',''), context:
Base64.encode64(context).gsub('n',''))
return cyphertext.data[:ciphertext]
end

20. require 'vault'
require 'base64'
##
# This object abstract the crypto / secrets management functions of HashiCorp Vault
class LocalVault
def initialize
@vault = Vault::Client.new
end
##
# Obtain a set of credentials to access the S3 Bucket used by the persistence layer
def getAwsCreds
credentials = @vault.logical.read("aws/creds/s3-bucket")
return credentials
end
def getGoogleCredentials
credentials = @vault.logical.read("secret/gsuite")
return credentials
end
def getJWTsecret
jwtsecret = @vault.logical.read("secret/JWT")
return jwtsecret
end
##
# Encrypt a value with a defined key and context for asymmetric encryption.
def encrypt(value,key,context)
cyphertext = @vault.logical.write("transit/encrypt/#{key}", plaintext:
Base64.encode64(value).gsub('n',''), context:
Base64.encode64(context).gsub('n',''))
return cyphertext.data[:ciphertext]
end
##
# Decrypt a value with a defined key and context.
def decrypt(value,key,context)
plaintext = @vault.logical.write("transit/decrypt/#{key}", ciphertext: value,
context: Base64.encode64(context).gsub('n',''))
return Base64.decode64(plaintext.data[:plaintext]).gsub('n','')
end
end
Initialize the object
Get dynamically generated AWS Tokens
Get static GSuite API Keys
Get info to sign JWTs
Encrypt a string
Decrypt a string

21. 2. How is this application getting its Vault / Nomad / Consul token?
How is this application supposed to get it’s runtime configuration?
J. Developer

22. group "webs" {
count = 1
task "jamo" {
vault {
policies = ["jamo"]
change_mode = "signal"
change_signal = "SIGHUP"
}
driver = "docker"
env {
VAULT_ADDR = “https://vault.service.consul:8200"
MEMCACHE_ADDR = "memcache.service.consul:11211"
}
[...]
}
Let Nomad get me a Token
What to do if that token is rotated
Environment variables.
DNS resolved by Consul

23. What if I’m using K8s?
• The good option, is using the K8s authentication method
and doing a login in the Dockerfile so the pod gets it
VAULT_TOKEN and then consume secrets
• The not bad option, is using consul-template to deploy a
helm chart with the secrets, doing authentication in the
pipeline that deploys it (see https://www.hashicorp.com/
resources/how-to-share-secrets-pipeline or http://
nicolas.corrarello.com/general/vault/security/ci/
2017/04/23/Reading-Vault-Secrets-in-your-Jenkins-
pipeline.html)

24. Perfect > Good Enough
• Getting credentials out of Vault is super easy, so
it’s easy to fall on the trap of having very short TTL
on secrets.
• Don’t underestimate complexity in recovering from
failures in running state
• Focus on business logic! (Always with reasonable
abstractions)

25. get '/' do
begin
client = Mysql2::Client.new(:host =>
mysqladdr, :username =>
mysqlvars.data[:username], :password =>
mysqlvars.data[:password])
mysqlstatus = client.query("SHOW STATUS")
rescue
puts "Asking for new credentials"
mysqlvars = getmysqlcreds(localvault)
client = Mysql2::Client.new(:host =>
mysqladdr, :username =>
mysqlvars.data[:username], :password =>
mysqlvars.data[:password])
mysqlstatus = client.query("SHOW STATUS")
end
##
# Returns an S3 client
protected
def awsClient
awscreds = self.awsCreds
begin
s3 =
Aws::S3::Resource.new(access_key_id:
awscreds[:access_key], secret_access_key:
awscreds[:secret_key], region: 'us-east-1')
return s3
rescue
awscreds = self.awsCreds
s3 =
Aws::S3::Resource.new(access_key_id:
awscreds[:access_key], secret_access_key:
awscreds[:secret_key], region: 'us-east-1')
return s3
end
end

26. 3. Are you supposed to provide credentials / configuration to the application owner? Do
application teams chuck releases over the fence to you?
J. Developer

27. Friction-less alternatives
• Consul-template
• Great for PKI & Existing servers (see https://www.yet.org/
2018/10/vault-pki/ by Sebastién Braun)
• Envconsul
• Similar pattern, but with environment variables

29. ##
# Wait for lock to be acquired before returning form
data
sessionid = Diplomat::Session.create({:hostname =>
"server1", :ipaddress => "4.4.4.4"})
lock_acquired = Diplomat::Lock.wait_to_acquire("/
key/to/lock", sessionid)
##
# Set a lock for the record that is about to
start being modified
sessionid = Diplomat::Session.create({:Node
=> "consul.service.consul", :Name =>
"locking session"})
lock_acquired = Diplomat::Lock.acquire("/
jamo/locks/#{docid}", sessionid)
# Do stuff
[…]
# Release lock
Diplomat::Lock.release("/jamo/locks/
#{docid}", sessionid )

30. Load balancing
• Fabio is awesome! I don’t even consider it’s there, it just
works!
• Not the only great pattern, see https://www.hashicorp.com/
resources/favorite-consul-customer-success-story

31. Not used in this project but
really cool….
• K/V Store
• Blocking queries
• Prepared queries
• Leader election

32. Don’t do this at home…
##
# Get Docker Hub Webhook
post '/dockerhubwebhook' do
callback = request.body.read
docker = JSON.parse(callback)
puts docker['push_data']['tag']
if docker['push_data']['tag'] != 'latest'
puts 'Deploying new Jamo Version' + docker['push_data']['tag']
nomadjob = erb :jsonnomad, :locals => { :tag =>
docker['push_data']['tag'] }
Nomad.job.create(nomadjob)
end
end

33. So what was
accomplished?
• 100% of the code is business code (alright, minus the
HTTP interface). Decoupled logical objects, from HTTP API,
from presentation layer that can be maintained individually
• Data encrypted at rest and in transit
• All credentials are ephemeral / short lived
• Microservices loosely coupled, discovered via Consul
• Cloud agnostic

34. What’s next?
• Already have a scheduler, stop running long task on
runtime and just schedule things! (I can get Nomad tokens
from the Vault API)
• Stop TLS Nightmare / port forwarding. Have Consul
Connect handle that

35. Stop TLS nightmare
▪ Service Access Graph
▪ Certificate Authority (built-in, Vault, Others)
▪ Application Integration

36. Thank you