My presentation at Gartner IOSS 2016

1. Starting and Scaling DevOps at Enterprise —
Lessons Learned
Template version: 8/29/2012

2. ‹#›
Tapabrata “Topo” Pal
Director, Individual Contributor, DevOps Evangelist
tapabrata.pal@capitalone.com
@TopoPal
Past:
• PhD in Semiconductor Physics
• Over 20 years of IT experience as Developer, Architect,
System Engineer
• Experience in Retail, Healthcare and Finance industries

3. Relative to our peers, we are a start-up
Most of you know Capital One is a Credit Card company. We are one of the largest in the US with over 70 million accounts. Many know that we are also one
of the nations largest banks. Fewer, however, realize that we are a Founder led 20 years old Technology Company.
Our “youngest” competitor is 108 years old
We respect our elders…
But we disrupted the credit industry with our leader’s vision….

4. “Every single person in the US had
the same credit card.
It made absolutely no sense.”
Rich Fairbank
Capital One Founder & CEO

5. Information Based Strategy (IBS)
Data Technology Data Science
Test & Learn
+ +
How we fulfilled on that vision was truly innovative
We used data, technology, and data science to execute an Information Based Strategy:
… design products based on customer's needs, passions, and life stages – not “one size fits all”
… make adjustments to products or presentation and see the impact in the data
… predict business results before full-scale market deployment
… we were “doing” Big-Data before even the name Big-Data came into the picture.

6. The Digital Revolution
Today our vision is to change banking for good. We are he nation’s largest Digital Bank. Digital is the new Bank branch. What is striking is the degree at
which Mobile has become the preferred channel for Banking.

7. "Ultimately the winners in banking will have
the capabilities of a
world-class software company.“
Rich Fairbank
Capital One Founder & CEO
As Rich Fairbank said: Ultimately the winners in banking will have the capabilities of a world-class software company.

8. About Five Years Ago…
Until about 5 years ago, we built software just like any other traditional non-technology big enterprise..
Mostly out sourced..
100% Waterfall
We released new software carefully, slowly, with lots of planning, documentation, meetings, approvals, tickets, …
Build, Deploy, Test was all manual…
Our success rate was quite high - even though we lacked the speed.

9. About Five Years Ago…
• Mostly outsourced
Until about 5 years ago, we built software just like any other traditional non-technology big enterprise..
Mostly out sourced..
100% Waterfall
We released new software carefully, slowly, with lots of planning, documentation, meetings, approvals, tickets, …
Build, Deploy, Test was all manual…
Our success rate was quite high - even though we lacked the speed.

10. About Five Years Ago…
• Mostly outsourced
• Waterfall
Until about 5 years ago, we built software just like any other traditional non-technology big enterprise..
Mostly out sourced..
100% Waterfall
We released new software carefully, slowly, with lots of planning, documentation, meetings, approvals, tickets, …
Build, Deploy, Test was all manual…
Our success rate was quite high - even though we lacked the speed.

11. About Five Years Ago…
• Mostly outsourced
• Waterfall
• Quarterly Releases
Until about 5 years ago, we built software just like any other traditional non-technology big enterprise..
Mostly out sourced..
100% Waterfall
We released new software carefully, slowly, with lots of planning, documentation, meetings, approvals, tickets, …
Build, Deploy, Test was all manual…
Our success rate was quite high - even though we lacked the speed.

12. About Five Years Ago…
• Mostly outsourced
• Waterfall
• Quarterly Releases
• Manual Processes
Until about 5 years ago, we built software just like any other traditional non-technology big enterprise..
Mostly out sourced..
100% Waterfall
We released new software carefully, slowly, with lots of planning, documentation, meetings, approvals, tickets, …
Build, Deploy, Test was all manual…
Our success rate was quite high - even though we lacked the speed.

13. First Proof of Concept
Our first proof of concept was for a Java based project. We used Subversion for source control, Hudson for CI, Maven as build tool and Nexus as binary
repository.

14. Seed of “DevOps”
That Proof of concept was like a small camp fire…
Within a few months, we were able to show the improvement - from days to minutes - literally.
Then we shared the success story with other teams and started more of these little camp fires within other teams.
Collectively we then presented our success stories to the executive leaders.
In the mean time we started to change our development methodology from Waterfall to Agile.
You can not do justice to Agile without applying DevOps principle and in that sense we were all set for our next step in the journey.
Before moving this forward, we went to the drawing board and started creating a strategy around it.

15. Seed of “DevOps”
That Proof of concept was like a small camp fire…
Within a few months, we were able to show the improvement - from days to minutes - literally.
Then we shared the success story with other teams and started more of these little camp fires within other teams.
Collectively we then presented our success stories to the executive leaders.
In the mean time we started to change our development methodology from Waterfall to Agile.
You can not do justice to Agile without applying DevOps principle and in that sense we were all set for our next step in the journey.
Before moving this forward, we went to the drawing board and started creating a strategy around it.

16. Seed of “DevOps”
That Proof of concept was like a small camp fire…
Within a few months, we were able to show the improvement - from days to minutes - literally.
Then we shared the success story with other teams and started more of these little camp fires within other teams.
Collectively we then presented our success stories to the executive leaders.
In the mean time we started to change our development methodology from Waterfall to Agile.
You can not do justice to Agile without applying DevOps principle and in that sense we were all set for our next step in the journey.
Before moving this forward, we went to the drawing board and started creating a strategy around it.

17. Seed of “DevOps”
That Proof of concept was like a small camp fire…
Within a few months, we were able to show the improvement - from days to minutes - literally.
Then we shared the success story with other teams and started more of these little camp fires within other teams.
Collectively we then presented our success stories to the executive leaders.
In the mean time we started to change our development methodology from Waterfall to Agile.
You can not do justice to Agile without applying DevOps principle and in that sense we were all set for our next step in the journey.
Before moving this forward, we went to the drawing board and started creating a strategy around it.

18. BYOD

19. Build Your Own DevOps

20. ‹#›
Development
• Architecture
• Design
• Code
• Test
As far as our understanding of DevOps goes..
Agile is about getting business teams closer to the development and collaboratively develop software in a faster pace.
But this generates a bottleneck - how do we build these newer versions of software with new features, where do we deploy them, how do we test them,
how do we manage the lifecycle in this fast environment? So, now, we need to move Operations closer to the Development (Shift-Left). And since we are
serious about security, we needed to move security also closer to the development - DevOpsSec.

21. ‹#›
Business
• Requirements
• Feature Request
• Roadmap
Development
• Architecture
• Design
• Code
• Test
As far as our understanding of DevOps goes..
Agile is about getting business teams closer to the development and collaboratively develop software in a faster pace.
But this generates a bottleneck - how do we build these newer versions of software with new features, where do we deploy them, how do we test them,
how do we manage the lifecycle in this fast environment? So, now, we need to move Operations closer to the Development (Shift-Left). And since we are
serious about security, we needed to move security also closer to the development - DevOpsSec.

22. ‹#›
Business
• Requirements
• Feature Request
• Roadmap
Development
• Architecture
• Design
• Code
• Test
Operations
• Infrastructure
• Platforms
• Environment
• Deployment
• Incident Mgmt
• Change & Release Mgmt.
As far as our understanding of DevOps goes..
Agile is about getting business teams closer to the development and collaboratively develop software in a faster pace.
But this generates a bottleneck - how do we build these newer versions of software with new features, where do we deploy them, how do we test them,
how do we manage the lifecycle in this fast environment? So, now, we need to move Operations closer to the Development (Shift-Left). And since we are
serious about security, we needed to move security also closer to the development - DevOpsSec.

23. ‹#›
Business
• Requirements
• Feature Request
• Roadmap
Development
• Architecture
• Design
• Code
• Test
Operations
• Infrastructure
• Platforms
• Environment
• Deployment
• Incident Mgmt
• Change & Release Mgmt.
Information Security
Application Security Security Testing
Information Security Infrastructure Security
As far as our understanding of DevOps goes..
Agile is about getting business teams closer to the development and collaboratively develop software in a faster pace.
But this generates a bottleneck - how do we build these newer versions of software with new features, where do we deploy them, how do we test them,
how do we manage the lifecycle in this fast environment? So, now, we need to move Operations closer to the Development (Shift-Left). And since we are
serious about security, we needed to move security also closer to the development - DevOpsSec.

24. ‹#›
Business
• Requirements
• Feature Request
• Roadmap
Development
• Architecture
• Design
• Code
• Test
Operations
• Infrastructure
• Platforms
• Environment
• Deployment
• Incident Mgmt
• Change & Release Mgmt.
Information Security
Application Security Security Testing
Information Security Infrastructure Security
DevOpsSec
As far as our understanding of DevOps goes..
Agile is about getting business teams closer to the development and collaboratively develop software in a faster pace.
But this generates a bottleneck - how do we build these newer versions of software with new features, where do we deploy them, how do we test them,
how do we manage the lifecycle in this fast environment? So, now, we need to move Operations closer to the Development (Shift-Left). And since we are
serious about security, we needed to move security also closer to the development - DevOpsSec.

25. Three pillars of DevOpsSec
We based our DevOpsSec transformation on three major pillars..

26. Three pillars of DevOpsSec
Automate Everything
We based our DevOpsSec transformation on three major pillars..

27. Three pillars of DevOpsSec
Automate Everything
Shift-Left
We based our DevOpsSec transformation on three major pillars..

28. Three pillars of DevOpsSec
Automate Everything
Shift-Left
Dashboard Everything
We based our DevOpsSec transformation on three major pillars..

29. Automate Everything - a.k.a “Tools Explosion”
This is a huge pain point of DevOps - tools explosion. We started with just three tools and now we have a whole bunch of tools. And this is just going to
be like this for a long time to come!

30. Code Build
Release
MonitorDeploy + Test Execution
All these tools are to create a continuous delivery pipeline
1. Everything as code
2. Security and compliance at every step
3. Heavy use of test automation including security testing
But there is a problem… how do we get all the infrastructure to run this type of pipeline?

31. Code Build
Release
MonitorDeploy + Test Execution
App
Test
Infra
All these tools are to create a continuous delivery pipeline
1. Everything as code
2. Security and compliance at every step
3. Heavy use of test automation including security testing
But there is a problem… how do we get all the infrastructure to run this type of pipeline?

32. Code Build
Release
MonitorDeploy + Test Execution
App
Test
Infra
DEV INT QA PERF PROD
DEV INT
SEC
QA SEC PERF PROD
DEV INT QA SEC PERF PROD
All these tools are to create a continuous delivery pipeline
1. Everything as code
2. Security and compliance at every step
3. Heavy use of test automation including security testing
But there is a problem… how do we get all the infrastructure to run this type of pipeline?

33. Code Build
Release
MonitorDeploy + Test Execution
App
Test
Infra
DEV INT QA PERF PROD
DEV INT
SEC
QA SEC PERF PROD
DEV INT QA SEC PERF PROD Infra
App
All these tools are to create a continuous delivery pipeline
1. Everything as code
2. Security and compliance at every step
3. Heavy use of test automation including security testing
But there is a problem… how do we get all the infrastructure to run this type of pipeline?

34. Code Build
Release
MonitorDeploy + Test Execution
App
Test
Infra
DEV INT QA PERF PROD
DEV INT
SEC
QA SEC PERF PROD
DEV INT QA SEC PERF PROD Infra
App
Flow
All these tools are to create a continuous delivery pipeline
1. Everything as code
2. Security and compliance at every step
3. Heavy use of test automation including security testing
But there is a problem… how do we get all the infrastructure to run this type of pipeline?

35. Code Build
Release
MonitorDeploy + Test Execution
App
Test
Infra
DEV INT QA PERF PROD
DEV INT
SEC
QA SEC PERF PROD
DEV INT QA SEC PERF PROD Infra
App
Flow Feedback
All these tools are to create a continuous delivery pipeline
1. Everything as code
2. Security and compliance at every step
3. Heavy use of test automation including security testing
But there is a problem… how do we get all the infrastructure to run this type of pipeline?

36. Building a server in a Datacenter
Steps = 62
Days = 60
Cost = $25k
Pain = @#$%^&*
Opportunity cost = ?????
Who has not felt the pain in getting a server in a data center?
We did! All our developers did, all our Operators did, everyone involved in getting a server felt various levels of pain, frustration…
Plus it became a serious bottleneck in the pipeline.

37. We had been running our Dev and QA Environments on public cloud for a while. This year we have put some of our critical production load on public
cloud.
We have also put containerized application… and we created a lot of the tooling around that ourselves.

38. Shift-Left: Mindset Shift
With Automation and Shift to a Cloud paradigm comes a Shift in mindset. A lot of activities that traditionally takes place at the end of a delivery lifecycle
now started happening at the beginning - we built teams with mix of engineers who can write code for application, test and infrastructure. And the same
team can implement the whole pipeline from the get go.

39. Shift-Left: Mindset Shift
Test Automation
Security Scan
Environment Build
Software Build Setup
Deployment Setup
Production Support
With Automation and Shift to a Cloud paradigm comes a Shift in mindset. A lot of activities that traditionally takes place at the end of a delivery lifecycle
now started happening at the beginning - we built teams with mix of engineers who can write code for application, test and infrastructure. And the same
team can implement the whole pipeline from the get go.

40. Shift-Left: Mindset Shift
Test Automation
Security Scan
Environment Build
Software Build Setup
Deployment Setup
Production Support
With Automation and Shift to a Cloud paradigm comes a Shift in mindset. A lot of activities that traditionally takes place at the end of a delivery lifecycle
now started happening at the beginning - we built teams with mix of engineers who can write code for application, test and infrastructure. And the same
team can implement the whole pipeline from the get go.

41. Shift-Left: Mindset Shift
Test Automation
Security Scan
Environment Build
Software Build Setup
Deployment Setup
Production Support
Planning
Architecture
Design
Coding
Unit Testing
With Automation and Shift to a Cloud paradigm comes a Shift in mindset. A lot of activities that traditionally takes place at the end of a delivery lifecycle
now started happening at the beginning - we built teams with mix of engineers who can write code for application, test and infrastructure. And the same
team can implement the whole pipeline from the get go.

42. http://github.com/capitalone/Hygieia
Dashboard Everything
Dashboard everything… Hygieia DevOps Dashboard…
We built it ourselves and we Open Sourced it. This product has taken off since its launch last summer and has own the “Open Source Rookies of the Year
2015” award.

43. Hygieia has a team level dashboard where you can visualize the teams workitems, coding activities, build status, static code analysis, unit test, security scan, functional test results, deployment status across environments - with drill downs to get more details. It also has
a “product” level dashboard..

44. Where you can see a roll up view of all teams dashboards. You can track every commit from commit stage to production and see quality trends. This helps us determining where the bottlenecks are… so that the teams can start the discussion as to how to remove the
bottlenecks.

45. 23
How do we scale
How do we do this at a large scale?
Community of Practices on various subject areas: DevOps, Security, Cloud, Architecture, Design, Programming Languages, Test Automation
Office Hours by horizontal service provider teams - come ask questions, get help, solve problems.
Voice of Customer: Horizontal service teams hear from their own customers - development teams, test engineers, product owners etc..
Pulse: Our internal social site.
Tech Goo: We have dedicated team to remove “Goo’s”: People come and complain vote up/down on proposals and things get changed or done. Goo’s
subject can be anything - tools, process,

46. 23
How do we scale
How do we do this at a large scale?
Community of Practices on various subject areas: DevOps, Security, Cloud, Architecture, Design, Programming Languages, Test Automation
Office Hours by horizontal service provider teams - come ask questions, get help, solve problems.
Voice of Customer: Horizontal service teams hear from their own customers - development teams, test engineers, product owners etc..
Pulse: Our internal social site.
Tech Goo: We have dedicated team to remove “Goo’s”: People come and complain vote up/down on proposals and things get changed or done. Goo’s
subject can be anything - tools, process,

47. 23
How do we scale
How do we do this at a large scale?
Community of Practices on various subject areas: DevOps, Security, Cloud, Architecture, Design, Programming Languages, Test Automation
Office Hours by horizontal service provider teams - come ask questions, get help, solve problems.
Voice of Customer: Horizontal service teams hear from their own customers - development teams, test engineers, product owners etc..
Pulse: Our internal social site.
Tech Goo: We have dedicated team to remove “Goo’s”: People come and complain vote up/down on proposals and things get changed or done. Goo’s
subject can be anything - tools, process,

48. 23
How do we scale
How do we do this at a large scale?
Community of Practices on various subject areas: DevOps, Security, Cloud, Architecture, Design, Programming Languages, Test Automation
Office Hours by horizontal service provider teams - come ask questions, get help, solve problems.
Voice of Customer: Horizontal service teams hear from their own customers - development teams, test engineers, product owners etc..
Pulse: Our internal social site.
Tech Goo: We have dedicated team to remove “Goo’s”: People come and complain vote up/down on proposals and things get changed or done. Goo’s
subject can be anything - tools, process,

49. 23
How do we scale
How do we do this at a large scale?
Community of Practices on various subject areas: DevOps, Security, Cloud, Architecture, Design, Programming Languages, Test Automation
Office Hours by horizontal service provider teams - come ask questions, get help, solve problems.
Voice of Customer: Horizontal service teams hear from their own customers - development teams, test engineers, product owners etc..
Pulse: Our internal social site.
Tech Goo: We have dedicated team to remove “Goo’s”: People come and complain vote up/down on proposals and things get changed or done. Goo’s
subject can be anything - tools, process,

50. 23
How do we scale
How do we do this at a large scale?
Community of Practices on various subject areas: DevOps, Security, Cloud, Architecture, Design, Programming Languages, Test Automation
Office Hours by horizontal service provider teams - come ask questions, get help, solve problems.
Voice of Customer: Horizontal service teams hear from their own customers - development teams, test engineers, product owners etc..
Pulse: Our internal social site.
Tech Goo: We have dedicated team to remove “Goo’s”: People come and complain vote up/down on proposals and things get changed or done. Goo’s
subject can be anything - tools, process,

51. 23
How do we scale
How do we do this at a large scale?
Community of Practices on various subject areas: DevOps, Security, Cloud, Architecture, Design, Programming Languages, Test Automation
Office Hours by horizontal service provider teams - come ask questions, get help, solve problems.
Voice of Customer: Horizontal service teams hear from their own customers - development teams, test engineers, product owners etc..
Pulse: Our internal social site.
Tech Goo: We have dedicated team to remove “Goo’s”: People come and complain vote up/down on proposals and things get changed or done. Goo’s
subject can be anything - tools, process,

52. 23
How do we scale
How do we do this at a large scale?
Community of Practices on various subject areas: DevOps, Security, Cloud, Architecture, Design, Programming Languages, Test Automation
Office Hours by horizontal service provider teams - come ask questions, get help, solve problems.
Voice of Customer: Horizontal service teams hear from their own customers - development teams, test engineers, product owners etc..
Pulse: Our internal social site.
Tech Goo: We have dedicated team to remove “Goo’s”: People come and complain vote up/down on proposals and things get changed or done. Goo’s
subject can be anything - tools, process,

53. 24
Our First ever Software Engineering
Conference
2 days
1,200+ attendees
13 Learning Tracks
28 Tech Expo Booths
52 Sessions
0 Vendor
Our mission was to promote the culture of sharing, collaborating and building relationship; and also let people learn something new. Attendees came
from Richmond and McLean, VA; San Francisco, New York, Chicago, Texas, Seattle, UK.

54. 25
Key Learnings

55. 25
Key Learnings
Start small, prove success, learn from mistakes

56. 25
Key Learnings
Start small, prove success, learn from mistakes
Executive Support is a must

57. 25
Key Learnings
Start small, prove success, learn from mistakes
Executive Support is a must
Empower your engineers

58. 25
Key Learnings
Start small, prove success, learn from mistakes
Executive Support is a must
Empower your engineers
No tools discussion

59. 25
Key Learnings
Start small, prove success, learn from mistakes
Executive Support is a must
Empower your engineers
No tools discussion
Embrace Open Source and Open Technologies

60. 25
Key Learnings
Start small, prove success, learn from mistakes
Executive Support is a must
Empower your engineers
No tools discussion
Embrace Open Source and Open Technologies
Share, share, share…

61. 25
Key Learnings
Start small, prove success, learn from mistakes
Executive Support is a must
Empower your engineers
No tools discussion
Embrace Open Source and Open Technologies
Share, share, share…
It is endless!