Learn how Decisiv provides secure access to developers and deals with compliance hurdles. Senior Engineer Hunter Madison will talk about how Decisiv needed to quickly solve the pain of scaling the engineering team, migrating to AWS, maintaining ISO 27002 compliance, and a few of his key learnings from his two-year journey using Teleport.

1. Teleport at Decisiv
Hunter Madison

2. What we will cover
Who is Hunter?
What does Decisiv do?
Securing Developer Access
Implementation Decisions
Q&A Session

3. Security Audits

4. Auditable Standards
● ISO 27002
● ISO 9000
● PCI DSS
● FedRamp
● NIST 500-292
● GDPR
● CCPA

5. Two Core Questions
● Who can do what, when?
● Why did someone do something then?

6. Designing Secure Systems
● When Security Gets in the Way - Interactions, volume 16, issue 6: Norman, D.
○ “The audience, either not understanding the rationale or simply
disagreeing with the necessity for the procedures imposed upon
them, see these as impediments to accomplishing their jobs.”
● Music Software & Interface Design: Steinberg's Dorico - Tantacrul
○ “The reality of modern life is that we are now required to keep
learning software all the time. It’s overwhelming, and a designer’s job
should be to try and reduce that pain as much as possible.”
● It's not good enough to be secure
● It has to be usable

7. Users

8. Users Don’t Change...
● End user workflows are generally set in stone
○ Workflows don’t change as the company scales up
○ What is ok with 5 developers isn't ok with 60
● Anything change made that an end user doesn’t see
immediate benefit in is hard to sell

9. ...But The World Does
● Your company will grow in size and attack surface
○ Laptops will get stolen
○ You will need more cloud resources
● What works for five developers won’t work for sixty,
six hundred, or six thousand

10. The User Role

11. The User Role
● Creates user accounts
● Adds ssh keys
● (Sometimes) tries to keep the UIDs consistent
● (Sometimes) sets up a .bash_profile
● (Sometimes) configures sudoers

12. The User Role Has Problems
● What happens when new people join and need access?
○ Hopefully, their key is provided to you the day they start
○ And they don’t need access immediately
○ Script needs to get run everywhere
● What happens when people leave?
○ Script needs to run everywhere again
○ Revocations don’t happen as fast as they should
● What happens when access is used to change application or server state
improperly?
○ Installing apps onto boxes scheduled for decommissioning
○ App consoles
● What happens when one developer really wants to connect their blackberry to
the vpn and ssh into boxes?

13. How We Fixed It

14. Teleport At A High Level
● It’s a highly available cluster of authentication and proxy
servers which create an auditable and IDP secured SSH
bastion host
● It’s also X.509 Certificate Authority
● It can store its state locally or in services like S3 and
DynamoDB
○ For this talk, we are assuming that Teleport is
configured to use S3 and DynamoDB
● It records end user actions into multiple auditable forms

15. IDP

16. Configure your IDP
● Teleport benefits from having a good ontology inside
of your IDP
● Your IDP pushes...
○ Groups which become Teleport roles
○ Attributes which are interpolated when evaluating
roles
● All of this data is accessible to you at login time
● Making good use of it cuts down on the administrative
headache significantly

17. First Time Provisioning Workflow

18. Use Labels and Metadata
Interpolation
commands:
- name: instance_id
command: ['/bin/curl', 'http://169.254.169.254/latest/meta-data/instance-id']
period: 24h0m0s
- name: account_id
command: ['/bin/sh', '-c', 'curl -s
http://169.254.169.254/latest/dynamic/instance-identity/document|jq -r .accountId']
period: 24h0m0s
- name: public_ip
command: ['/bin/curl', 'http://169.254.169.254/latest/meta-data/public-ipv4']
period: 24h0m0s
- name: app_owner
command: ['/opt/bin/get_tag’, ‘App_Owner’]
period: 24h0m0s

19. Labels in the Teleport UI

20. Use Labels and Metadata Interpolation
kind: role
metadata:
name: developers
spec:
allow:
logins:
- developers # '{{ external["ssh_user"] }}'
node_labels:
app_owner: '{{ external["team"] }}'
deny:
logins: null
node_labels:
app_owner: operations
options:
cert_format: standard
client_idle_timeout: 8h0m0s
enhanced_recording:
- command
- network
forward_agent: false
max_session_ttl: 8h0m0s
port_forwarding: true
version: v3

21. SSH

22. It’s Just SSH
Host proxy.example.com
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
CheckHostIP no
CertificateFile
~/.tsh/keys/example.com/%u@example.com-cert.pub
IdentityFile ~/.tsh/keys/example.com/%u@example.com
Port 3023
Host *.apps.example.com
Port 3022
ProxyCommand ssh %r@proxy.example.com -s proxy:%h:%p
StrictHostKeyChecking no

23. Really, Just SSH
● Your tools that use SSH and can read ~/.ssh/ssh_config will work with Teleport
too!
○ tsh login and go!
○ Provided it supports certificate authentication (IDEA-216138)
● I’ve tested
○ Ansible (Parminko)
○ Inspec (Train)
○ Capistrano (Net::SSH)
○ OpenSSH
● Windows
○ Enable OpenSSH for Windows
○ tsh.exe only provides the signed certificates

24. Who started this?

25. /proc/<pid>/environ
● Teleport sets environment variables for each session
○ The two to know TELEPORT_SESSION and
SSH_TELEPORT_USER
● If you are unsure who started a process (like a tmux or
screen session) check the environ and find the
TELEPORT_SESSION

26. Terminal Example

27. Looking at Sessions, Proactively

28. Session Recording
● Teleport records every session for playback
● These are great to watch, but hard to search through
at scale

29. Events Table
● With “Enhanced Session Recording” enabled, a new
“session.command” event becomes available.
● It’s worth getting this data into your SIEM/Logging solution
○ NEW_IMAGE
● This gives you a really quick and easy way to find and log
“problem commands”
○ Screen
○ Tmux
○ psql

30. SIEM Integration

31. Not Just SSH

32. You Have A X509 CA
● Every time a user logs in with tsh, they get a newly refreshed X509 certificate
● You can use these client certs to authenticate with a lot of tools that don’t
necessarily support SSO out of the box
○ OpenVPN
○ Postgres
○ Mariadb
● For server certs start with `tctl get --with-secrets cert_authority`
● Same session expiration rules apply
● Role support is application dependent
○ You can see groups (as organization), valid logins (as locality) and the
username (as common name) in the subject
○ Your mileage will vary if the application supports parsing that data

33. Demo

34. Implementation Decisions

35. Teleport Deployment

36. “User” Tier
● End Users
○ Everywhere!
○ We are a global, remote company
○ Okta as an IDP
● Nodes
○ Lots of AWS accounts
○ We do account vending
○ Join tokens via cross account STS/SSM

37. Cluster Mode
● Tunneled
○ Single cluster
○ Nodes connect via the internet
○ Single SAML SP
● Trusted
○ Each account gets is own cluster
○ Single SAML SP shared via a primary cluster
○ We used this up to 4.0
○ When it breaks, it *hurts*
○ Users need to be aware of cluster switches

38. Load Balancing Tier
● We run Teleport in a HA setup
● Application Load Balancers
○ Change your timeout to get the web console to work.
○ You can let teleport generate its own self signed SSL cert. ALBs don’t check SSL.
● Network Load Balancers
○ You will see constant errors in the logs because of the heartbeat.

39. Application Tier
● We run Teleport via an autoscaling group with one host per AZ
● We stack the Auth and Proxy components onto the same hosts
● Use SSM for your “break glass” mechanism

40. Database Tier
● We use S3 to store sessions and Dynamo to store state and events
● This makes our auth/proxy hosts stateless
○ Really nice for upgrades
● Events in Dynamo open up SIEM integrations
○ DyanmoDB streams with NEW_IMAGE

41. Teleport Deployment

42. Any Questions?

43. Recommended Next Steps
Download Teleport
https://gravitational.com/teleport/download
Join Teleport Community
https://community.gravitational.com
Read the Teleport Admin Guide
https://gravitational.com/teleport/docs/

44. Teleport at Decisiv
Hunter Madison