4. Seattle | September 16-17, 2019
I work at LifeOmic
where all data is sensitive data
5. Seattle | September 16-17, 2019
The Sucker’s Choice
• Don’t pick between security or speed
• Get more security and more speedSecurity
Speed
Where you think the boundary is
The actual boundary
You are here
6. Seattle | September 16-17, 2019
Tools we use
• AWS
• Git (Bitbucket)
• Jenkins
• Jira
• Snyk
7. Seattle | September 16-17, 2019
From idea to production
CM Ticket
Bitbucket PR
Code Review / Approval
CM Approvals
Deploy to AWS dev
Deploy to AWS Prod
Local development
8. Seattle | September 16-17, 2019
Implementation
CM Ticket
Bitbucket PR
Code Review / Approval
CM Approvals
Deploy to AWS dev
Deploy to AWS Prod
Local development
Implementation of change and test automation
on the developers machine
Push change to a Git branch, test automation
runs in Jenkins
9. Seattle | September 16-17, 2019
Review and deploy
CM Ticket
Bitbucket PR
Code Review / Approval
CM Approvals
Deploy to AWS dev
Deploy to AWS Prod
Local development
After approval, changes are merged into
Git master branch and Jenkins deploys to
AWS dev account
Team members review code and click ‘Approve’
10. Seattle | September 16-17, 2019
Request production deployment
CM Ticket
Bitbucket PR
Code Review / Approval
CM Approvals
Deploy to AWS dev
Deploy to AWS Prod
Local development
Humans review the new changes being
deployed and approve if the appropriate
processes were followed
Developer creates a Jira issue to request
deployment of specific project and build to
production
11. Seattle | September 16-17, 2019
Production deployment
CM Ticket
Bitbucket PR
Code Review / Approval
CM Approvals
Deploy to AWS dev
Deploy to AWS Prod
Local development
Jenkins watches for the Jira issue and deploys
the new project version when approved
12. Seattle | September 16-17, 2019
Process requirements
• Nobody can act alone to modify production
• Changes to production are auditable
• Security scans are run
13. Seattle | September 16-17, 2019
Automation > Humans
CM Ticket
Bitbucket PR
Code Review / Approval
CM Approvals
Deploy to AWS dev
Deploy to AWS Prod
Local development
Automate review of process compliance
Automate detailed change description
collection for Jira issue
14. Seattle | September 16-17, 2019
Old Change Management Ticket Flow
Developer runs job to promote
Jira emails sent to approvers
Humans review and approve
Jira ticket moves to implementing
Jira CM issue is created
Deploy job waits for approval
Deploy job resumes
15. Seattle | September 16-17, 2019
Automation possibilities
Creating the ticket
• What has changed
• Has anything been removed
Reviewing changes
• Bitbucket PRs for all changes
• All PRs were reviewed by peers
• Security scans have been run
16. Seattle | September 16-17, 2019
New Change Management Ticket Flow
Developer runs job to promote
Jira emails sent to approvers
CM Bot Updates
Jira ticket moves to implementing
Jira CM issue is created
Deploy job waits for approval
Deploy job resumes
Humans Review
Fewer details
17. Seattle | September 16-17, 2019
CM Bot execution
Jira Issue Created Webhook
CM Description updated
Approval (or not)
Bitbucket / Jenkins / Jira Analysis
CM Verdict in comment
18. Seattle | September 16-17, 2019
Git Commits
Commit 1 Commit 2 Commit 3 Commit 4 Commit 5
Merge 1 Merge 2 Merge 3
Master Branch
• Every commit must be from a LifeOmic Bitbucket user
• All commits into master are through merge commits
• No Fast Forwards or squashes
19. Seattle | September 16-17, 2019
Pull requests
PR 2 PR 3PR 1
• Every merge into master must be associated with a Bitbucket pull request
• Every pull request must have approvers who are not the committer
20. Seattle | September 16-17, 2019
Jenkins Builds
Build 1 Build 2 Build 3
• Every merge into Master starts a Jenkins build for test and deploy
21. Seattle | September 16-17, 2019
Change Management Jira Issues
CM 1 CM 2 CM 3
• Production deployment requests refer to a Jenkins build
22. Seattle | September 16-17, 2019
Verifying a promotion request
Commit 4 Commit 5
Merge 3
PR 3
Build 3
CM 1 CM 2 CM 3
Master Branch
• Find previously approved CM issue
• Find all new artifacts associated with the
request
• Verify the processes were followed
• Leave a comment in the CM issue with
with changes detected
23. Seattle | September 16-17, 2019
Automation
• Good at finding and listing every
single change in the new build
• Good at filling CM tickets with
details
• Very fast
Developers
• Good at remembering what they
did recently
• Good at listing just enough details
to get the CM ticket approved
• Comparatively slow
26. Seattle | September 16-17, 2019
Automation
• Good at checking every single change
for peer reviews and security scans
• Good at catching edge cases which
can be hard to detect manually
• Very fast
• Reviewing at odd hours
Human Approvers
• Vulnerable to being rushed and
sometimes spot checking changes
• Good at thinking big picture for
security and compliance
• Comparatively slow
• Sleeping and spending time with family
27. Seattle | September 16-17, 2019
Git merge edge cases
• Detecting any changes in the merge commits
• Handling cases for PRs with multiple committers
• Detecting removed commits
• Squash merges
28. Seattle | September 16-17, 2019
Changing culture
• Worst case is fallback to human review
• Teams eagerly follow review process to avoid waiting on human approvals
• Security team doesn’t have to play the bad guy, the tool is the bad guy
• Motivation to automate production changes
• Shame for squash merges is a right of passage for new hires
29. Seattle | September 16-17, 2019
The future
• Detecting risky Terraform changes
• Allowing automatic rollbacks
• More types of security scans
30. Seattle | September 16-17, 2019
Continuous delivery and HIPAA compliance is possible
• We’ve open-sourced a library to help
• Utilizes JupiterOne for data collection
• https://github.com/JupiterOne/jupiter-change-management-client
I'm Matt Lavin
I work at LifeOmic
I hate waiting.
I hate delays to getting bug fixes and features to customers.
- want get feedback quickly
- want happy customers
- want security fixes out
- Please take my word that I'm better at building software than making slides or presenting
been to conferences and seen the cool companies moving quickly
amazing continuous delivery pipelines
constantly churn out changes
their speed allows experimentation
They have big teams
I work at LifeOmic
All our data is sensitive data because we manage patient data for doctors and researchers.
We have to maintain HIPAA / HITRUST compliance
tempting to say that our data is so sensitive that fast production change is not possible or realistic.
We take security very seriously and yet we want to develop new software as fast as possible to help people
We are a small company
If you haven't read read the book, Crucial Conversations, you should.
I first heard the idea of the “Sucker’s Choice” from that book. Knowing about the sucker’s choice can avoid disagreements between dev and sec by asking “how can we have it all”. We don't need to pick between security or speed. We can have both security and speed.
Side note, I think this same graphic could be applied to quality and speed too. If you start asking “how can we have two things at once” instead of either/or then good ideas come to the surface
!! So I built a tool that gives us both !!
Pretty standard workflow
We expect test automation for all changes, that's key to continuous delivery with or without security involved
Here are the two steps that caused me the most pain.
Can't automate code writing or code reviews (until AI takes over)
Deployments are automated already
So the biggest slowdown was writing the details of the CM ticket
and waiting on a human to review the details and click approve.
We could do better!
Suffix on commit message
Jenkins API for each build includes the git commit hashes that were built -- to detect new changes
Also includes the logs -- security scans
The automation really can be both more secure and faster at the same time