How frequently does a good agile team deploy to production? Not every team is capable of deploying "on every commit". What does it take for a team to even start deploying at the end of each sprint, or each week, or each day?
Most companies don't realize that deploying more frequently often requires both significant technical change as well as cultural change. In this talk, I'll guide you through what it takes to deploy more frequently, both from the technical side of setting up pipelines as well as the organizational side of removing red tape. I'll draw on the unique challenges that teams must overcome at each step of the way, from deploying once a month all the way down to full continuous delivery. If your team has been struggling to go faster, come see how you can change to get there. And if you already are at full continuous delivery, come see how to go even faster than that!
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
So you-want-to-go-faster
1. So You Want To Go Faster?
A Journey to Continuous Deployment
2. • 10+ years software
development experience
• Work at Excella Consulting
(~5 years)
• I love testing, all things
agile, cats
• If any of those things
interest you…
About Me
Dan Davis
14. • Works, but how fast?
• Top Speed: limited by…
– how frequently I check the repository
– how quickly I can perform the deployment
• Probably not a great process IMHO
Not really Continuous Deployment
Code
Repository
Deploy to
Production
15. • Need some kind of automation
How can we go faster?
Code
Repository
Deploy to
Production
Bottlenecks
16. Automation Tools
• 3 basic characteristics
• Execute a script / code (in a task/job)
• Trigger task/job
– On demand
– On a timer
– On an event, like a code commit
• Start other jobs / tasks when completed
– Avoid one big script, create logical separations
– Assembly line, but for code!
23. Setting up Infrastructure
How do I keep track of what’s been approved?
Provision
Non-prod
server
Deploy to
non-
production
Acceptance
Test
Provision
Production
Deploy to
production
Deploy to
production
24. Creating Build Artifacts
Can I test sooner than post-deploy?
Artifact
Repo
Provision
Non-prod
server
Deploy to
non-
production
Acceptance
Test
Provision
Production
Deploy to
production
Create Build
Artifacts
25. Run Unit Tests
Provision
Non-prod
server
Deploy to
non-
production
Acceptance
Test
Provision
Production
Deploy to
production
Create Build
Artifacts
Unit Tests /
Static
Analysis
How do I know it continues to work after I deploy?
27. Generic Pipeline
• This is the basic machinery
• Speed is an important variable
– Depending on speed we may choose different tools
• Continuous Deployment is a speed, not a process or tool
– To achieve that speed, we have to work up to it
– Think of it like shifting gears
Unit Test /
Static
Analysis
Create a
build artifact
Provision
Non-prod
server
Deploy to
non-
production
Acceptance
Test
Provision
Production
Deploy to
production
Monitoring
28. • Once Per Month
• Once Per Sprint (2 weeks)
• Once Per Week
• Every 2-3 days
• Every Commit (multiple
times per day)
5 Gears of Production Deployment
30. • Basic Continuous Integration
– Run unit tests when code is committed
• Manual Elements
– Acceptance Testing performed by a QA team
– Deployments performed by a deployment engineer
– Servers are manually provisioned as needed
1st Gear Teams
Unit Test /
Static
Analysis
Create a
build artifact
Provision
Non-prod
server
Deploy to
non-
production
Acceptance
Test
Provision
Production
Deploy to
production
Monitoring
31. • Deployments prone to failure
– Knowledge siloed in handful of people
• Deployments require downtime
– Must be done on weekends or late at night (with little support)
• Lack of consistency between environments
– Leads to deployment problems and bugs
1st Gear Pain Points
32. Do You Want To Go
Faster?
(Deploying Every Sprint)
34. • Main Limiting Factor: Deployment failure rate
– High failure rate will prevent you from speeding up…
• Focus on
– Standardizing the deployment process (though automation)
– Building out a pipeline
– Learning from build failures
2nd Gear Focus Areas
Unit Test /
Static
Analysis
Create a
build artifact
Provision
Non-prod
server
Deploy to
non-
production
Acceptance
Test
Provision
Production
Deploy to
production
Monitoring
36. • Basic SSH wrappers around shell commands
Fabric / Capistrano
Server
Command
Command
Command
Manual
Execution
Server
Command
Command
CommandFabric /
Capistrano
File
Version
Control
37. Benefits
• Simple approach, one step
above shell scripts
• Low learning curve
• Does just enough…
– Multi-host deployment
– Organize into roles
• Great for lightweight
automation
38. • Now we can establish a basic pipeline
• Next Challenge: only deploy from pipeline
• When pipeline breaks, prevents deployments until fixed
• FAQ: Why should I care about failures in the pipeline?
2nd Gear Cultural Changes
Unit Test /
Static
Analysis
Create a
build artifact
Provision
Non-prod
server
Deploy to
non-
production
Acceptance
Test
Provision
Production
Deploy to
production
Monitoring
39. • When you see the check engine
light, what do you do?
– Take it to mechanic
– Don’t keep driving!
– Letting it fester will only make
it worse!
• Treat your pipeline like your car!
– When it breaks fix it with a
sense of urgency
– Minimize the possible changes
that caused it to break in the
first place
Pipeline Failures
40. • Now have a somewhat reliable build process
– Our build engineers should be a lot happier
– Easier to recruit new people for deployments
• Lots of manual steps, but should be fast enough to
deploy every sprint…
The power of 2nd gear
Unit Test /
Static
Analysis
Create a
build artifact
Provision
Non-prod
server
Deploy to
non-
production
Acceptance
Test
Provision
Production
Deploy to
production
Monitoring
41. So You Want To Go Faster?
(Deploying every week)
42. • Main Limiting Factor: Environments
– Deploy once per week: start thinking about impact of downtime
• Focus On
– Improving consistency between environments
– Standing up infrastructure
– Eliminating reasons to SSH to servers
Shifting into 3rd gear
Provision
Non-prod
server
Deploy to
non-
production
Acceptance
Test
Provision
Production
Deploy to
production
Create Build
Artifacts
Unit Tests /
Static
Analysis
Monitoring
43. • Deployments often require service interruption
– Stop processes/services
– Drain queues
– Update packages
• Can try to reduce deployment time, but not worth it
• Better Approach: Blue/Green Deployment
Why Downtime Matters
50. • Shift in mindset: Infrastructure as Code
– Virtualization means no more owning physical servers
– No more dedicated hardware
Culture Changes
Yes, this is a thing you can buy…
51. • Orchestrator + Config mgmt + Deployment
• Reliable deployments with no downtime!
– Consistency between environments
– Blue/Green deployments
• Can even start deploying to non-prod more frequently
• Team should settle into a pretty good groove with
weekly deployments
Power of 3rd Gear!
Provision
Non-prod
server
Deploy to
non-
production
Acceptance
Test
Provision
Production
Deploy to
production
Create Build
Artifacts
Unit Tests /
Static
Analysis
Monitoring
52. Do you want to go faster?
(Deploying every 2-3 days)
53. • Ironically, one of the
hardest gears
• Traditionally manual
processes become
unsustainable
• More cultural battling than
technology
Shifting into 4th gear
54. • Main Limiting Factor: Unready / Unfinished code
– Barely enough time to develop
– No time to manually test
• Focus On
– Improving confidence in automated testing
– Eliminating manual testing
– Using Feature Toggles
Shifting into 4th gear
Provision
Non-prod
server
Deploy to
non-
production
Acceptance
Test
Provision
Production
Deploy to
production
Create Build
Artifacts
Unit Tests /
Static
Analysis
Monitoring
55. One Week Deployment Cycle
Coding Coding Coding
Testing / Bug
Fixing
Deployment /
Happy Hour
Day 1 Day 2 Day 3 Day 4 Day 5
(Tu)(Wed)
56. 2 – 3 Day Deployment Cycle
Coding
Testing / Bug
Fixing
Deploy
ment?
Day 1 Day 2 Day 3
Bottleneck
We must eliminate the “code freeze” period before deployment
57. • Acceptance Testing (i.e. New Functionality)
• Regression Testing (i.e. Old Functionality)
• Both are highly predictable
– Great candidates for automation
Cutting down testing time
58. Tools for Automating Acceptance Tests
“Web Driver”
• Mimics the actions of a
user in a browser
59. • Who writes the automation code?
– Testers don’t typically have the skillset
– Developers are better suited for it
• Developers must now write feature code AND test
code?
– Is that fair?
Culture
60. • Engineers Day
• Deeper appreciation for
what you build
• Encourage
thoughtfulness about
how to design parts
Automotive Engineers
61. • Developers should have a stake in the quality of the final
product
– Developers should learn what types of bugs get through and
how to prevent them
• Encourage mindfulness of code
– Hard to test? Find ways to make it easier
• Promote a better collaboration with Product Owner
– Work closer with acceptance criteria
Developers Responsible for Quality
63. 2 – 3 Day Deployment Cycle
Coding /
Continuous Testing
Deploy
ment?
Day 1 Day 2 Day 3
Is 2.5 days enough time to develop a useful feature?
64. • Can’t complete in everything 2-3 days
• Catch 22: Can’t merge it until all stories are completed
• Maybe we should slow down…
Useful Functionality
Useful Feature
Story 1 Story 2 Story 3 Story 4
65. • A little bit of conditional logic that can enable/disable
functionality
• Often stored in a cache or database
Feature Toggles
67. Deploying With Feature Toggles
Non-Prod Production
Story
1
Story
2
Story
3
Story
4
Story
1
Story
2
Story
3
Story
4
68. • Decoupling code deployment with releasing
functionality
– At previous speeds, deploying to production meant turning on
functionality
– Replaced with turning on a feature toggle
• Culture Change: Deploying to production is an
unremarkable event
– The same scrutiny is no longer needed
– “Rollbacks” are now effortless, just un-toggle
Culture Change
69. • Automated our testing process to reduce testing time to
minutes
• Using feature toggles allows our team to work extremely
efficiently
– Ship what’s ready, keep working on what’s not
• Should be able to comfortably deploy every 2-3 days
– There’s very little in the way of going faster…
The Power of 4th Gear
Provision
Non-prod
server
Deploy to
non-
production
Acceptance
Test
Provision
Production
Deploy to
production
Create Build
Artifacts
Unit Tests /
Static
Analysis
Monitoring
70. Do you want to go faster?
(Deploying on Every Commit!)
the most fast???
^
71. • Main Limiting Factors: Efficiency
– Creating new infrastructure is slow
– Push button deploys are getting in the way
• Focus On
– Shore up testing with monitoring and alerting
– Immutable Infrastructure
– Flip the switch to automatically deploy to production
Shifting into 5th gear
Provision
Non-prod
server
Deploy to
non-
production
Acceptance
Test
Provision
Production
Deploy to
production
Create Build
Artifacts
Unit Tests /
Static
Analysis
Monitoring
72. • Spending a lot of time configuring servers
Immutable Infrastructure
All Configs
Base OS
New Server
Code
Image
73. • Spending a lot of time configuring servers
Immutable Infrastructure
All Configs
Base OS
New Server
Image
Code
Image w/
Code (v1.0)
75. • New code means building a new image
– Creates a “build artifact”
• Rather than updating, just destroy and recreate
Immutable Infrastructure
76. • Even with perfect deployments, gap in our testing
• What happens after we deploy?
– A flood of users might hit the server all at once
– A memory leak could start sapping performance
– Networking could fail
Monitoring / Alerting
77. • Alerts: when certain conditions are met, send a
notification
• If we see a sudden spike in errors
– Alert the developers
– Let the Product Owner know
– CC the security team
• Be proactive!
– Don’t just wait for users to tell you it’s broken
Monitoring / Alerting
79. • Cultural Challenge: Is it ok
to automatically deploy to
production??
Culture Change
80. • Continuous Deployment takes Bravery
– Trust that your automated tests will catch show stopper bugs
– Dedicate yourself to fixing bugs quickly (because you can)
• Continuous Deployment is a speed, there are trade-offs
– Only achievable with a dedicated and competent team
• You’re not the first
– I run a team at Excella that is doing this
– We built on the shoulders of others before us
Culture Change
86. • Once every commit deploys to
production, you may start to redline
• Limited by
– # of commits per day
– Longest step in pipeline
• Most teams won’t hit upper limit
– ~5 min / step == 100 deployments
(8 hour day)
• Consider breaking into microservices
Going Faster
