Recipe for DevOps Success: Capital One Style

1. Recipe for DevOps Success:
Capital One Style

2. • Tell you about Capital One’s DevOps journey
• Describe the successes and challenges
• Give you at least 1 actionable item to take away
Goals

3. • Mark Andersen (mark.andersen2@capitalone.com)
• Alma Mater: University of Illinois – Urbana Champaign
• At Capital One for last 2 years
• Director of Devops for Capital One Auto Finance
• Capital One cloud account lead for Plano
Who am I?

4. • More than just credit cards (Auto Lending, Home Lending, Retail Banking,
Commercial Banking, etc.)
• Startup in bank years
• All in on devops
• All in on the public cloud
• Open sourcing some of our own products (Hygieia / Cloud Custodian)
Who is Capital One?

6. • Inconsistent releases
• Many handoffs between many teams for even the smallest change
• Manual steps documented in spreadsheet
• Environments took many months to setup and had considerable drift
Capital One Life Before DevOps

7. The Capital One DevOps Transformation
Phase1
On Prem Use of
Configuration
Management
Automation Tools
(Chef)
Automation of
Building out
Middleware and
Application Software
Phase2
Cloud Journey – Full
Infrastructure
Automation into
Delivery Pipelines
Current
Implementing Robust
Pipelines with
Increased Quality
Checks and
Automation
Pre-Approved
Releases

8. • Started with 2 legacy applications. Got a focused SWAT Team together in 1 room
for 2 months.
• SWAT Team had developers, infrastructure, and production support. Forced
cross team / cross responsibility collaboration.
• Added a few professional services resources to address knowledge gaps.
• Automated the application build, infrastructure deployment, and application
deployment.
• Tools: Jenkins, AWS CloudFormation, Chef
• Moved on to address 4 more applications of increasing complexity.
• Completed an additional 4 applications before having the application teams drive
themselves. (retired the SWAT Team)
The Capital One DevOps Transformation

9. Automation Really Does
Work and Provides Value
• When testing our first
application, we had a
bad performance test.
Changed a property to
make the servers 1
size bigger. Ran the
deployment job. 20
minutes later we were
resized and had a
successful test.
SWAT Team Cross-
Functional Collaboration
Provided Huge Dividends
• Removed many
handoffs between
teams
• Developed shared
goals for deliverables
– More “We” instead
of “They”
• People learned other
roles and learned to
respect the complexity
– i.e. App Teams
creating infrastructure
automation
Speed of Delivery Increased
• Less waiting on other
teams
• Faster feedback loop
for teams
• Test and Fail
What worked?

10. Tried to Automate
Everything all at once
• Created long pipelines
with long deploy
cycles. Also required
too many new
problems to be solved
at the same time.
Automation handoff to
application team had issues
• As automation was
returned back to
applications team, they
didn’t have some of the
training / knowledge to
grow it and support it.
Tools were setup to
support non-devops
environment
• Tools needed to be
updated to support
devops processes and
availability needs.
What didn’t work well

11. 10. Don’t just have the devops engineer know the automation. DevOps engineers need to teach
the POD how things work and how to fix things.
• The goal is for the developers to be self sufficient.
• The more the developers understand, the more then can fix their own issues without help and waiting on someone.
• Treat the automation like a car. Teach the application team to drive it, change the oil, change the windshield fluid,
etc. Not how to build the engine.
9. Prepare yourself for on prem / legacy software to be challenging to automate. You will have to
be creative.
• Cluster discovery tends to be tougher
• Consider the software you are trying to automate and deploy
Top 10 Ingredients for DevOps Success

12. 8. Don’t try to do everything at once. Focus on one or two thing first.
• If you try to do CI / CD / Environments / Cloud all at once, it will be too much.
• Smaller batches (where have I heard that?)
• Smaller chunks means you get to celebrate more. Do it
• Celebrate like Dude Perfect
Top 10 Ingredients for DevOps Success (cont.)

13. 7. If you are picking (or building) tools, make sure they have API’s.
• API’s are the enabler for devops (for cloud too)
• Even if it has a bad API, it is better than no API.
• Be API first for your tools / scripts. Build your UI after that.
6. Speed matters. Long feedback loops are bad feedback loops.
• When you automate everything the first time, you’ll find some things take a
really long time.
• Look to parallelize everything (especially functional tests)
• If you pipeline takes 1 hour to complete, are you going to wait for it for
feedback?
Top 10 Ingredients for DevOps Success (cont.)

14. 5. Unit testing is still needed.
• Don’t just focus on functional tests. They tend to be really slow and brittle.
• Unit tests are fast and stable.
• Consider mocking out your functional dependencies for speed and stability.
4. Networking and security is hard. It is even harder for developers. Training and tools are
needed.
• Centralize your log collection for everything (applications and infrastructure). Let everyone see them. Create
dashboards in “developer” language for the information they need.
Top 10 Ingredients for DevOps Success (cont.)

15. 3. Avoid creating a Devops silo to replace your
infrastructure silo.
2. Focus on removing / reducing the handoffs. Create
self service for things you can’t give developers direct
access to.
• If you can’t give developers access to create / modify them
(LDAP groups, Security Groups, etc.), give them a tool to do it
the right way
Top 10 Ingredients for DevOps Success (cont.)

16. 1. Don’t let great get in the way of being very good.
• Deliver something, get feedback, do some more.
• If you are trying to wait until it is perfect, you’ll never deliver.
• Unicorns don’t really exist (or they are tough to find). Strive to be a horse.
Top 10 Ingredients for DevOps Success (cont.)

17. Thank You

18. Appendix

19. Main site Open Source Project Site = https://developer.capitalone.com/
Hygieia (DevOps visualization tool) = http://www.capitalone.io/Hygieia/getting_started.html
Cloud Custodian (cloud management) = https://developer.capitalone.com/opensource-
projects/cloud-custodian/
My email = mark.andersen2@capitalone.com
Capital One Open Source Projects