Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Automation and Developer Infrastructure
Empowering Engineers to Move from Idea to Production
Alex Thurlow
Engineering Manager
I help
people
get
jobs.
Indeed is the #1
external source of hire
Unique Visitors (millions)
Million unique visitors
2009 2011 2012 2013 2014 2015
...
Alex Thurlow
Plus many more!
What should we build?
What’s best for the jobseeker?
How do we know?
Data-Driven Product Design
go.indeed.com/ddpd
Learn Measure
Build
Lean Startup Cycle
Deliver
Learn Measure
Data driven product design requires
quick feedback
Engineering Velocity
go.indeed.com/evtt
Engineering Velocity
go.indeed.com/evtt
Developer
Productivity
Software
Architecture
Release
Process
Engineering Velocity
go.indeed.com/evtt
Developer
Productivity
Software
Architecture
Release
Process
Service-Oriented Architecture
Make small, independent applications
Deliver
Learn Measure
Quickly!
Deliver
Learn Measure
Engineering Velocity
go.indeed.com/evtt
Developer
Productivity
Software
Architecture
Release
Process
Jon Hanks
Software Engineer
I help
people
get
jobs.
Hobo
Automated developer provisioning
First goal for a new developer
Push code to production in week 1
Deliver
Learn Measure
1:1 Mentorship
Setup = 2-3 Days
1-2 days left to get to production
Team Rotation
Helps new developer understand Indeed
Rotate every 4 weeks through 3 teams
Rotate every 4 weeks through 3 teams and face the
same setup challenges in each team.
Frustration!!!
Experienced devs face a similar
problem working on multiple projects
or switching projects
Service-Oriented Architecture
Job Database
Search Webapp
Job Service Resume service Location Service
Search Webapp
Job Search Index Resume Database GeoIP Database
Need to run several
interconnected applications
Total Developers Producing Code By Month
2014 Apr Jul Oct 2015 Apr Jul Oct 2016
1:1 mentorship didn’t scale
Use technology to speed
developer provisioning
Vagrant automates creating virtualized
development environments
Job Service Resume service Location Service
Search Webapp
Job Search Index
Resume
Database
GeoIP Database
Vagrantfile
Virt...
Success!
(kind of)
Reproducible environments
Our Solution—The Good
Our Solution—The Good
Sharable between devs
Our Solution—The Good
<1 day setup time
Our Solution—The Bad
5 Only Linux support
4 Limited support for different application types
1 Slow to create environments....
Our Solution—The Bad
5 Only Linux support
4 Limited support for different application types
1 Slow to create environments....
Download full machine image
Download and install required software
Start every application
Our Solution—The Bad
5 Only Linux support
4 Limited support for different application types
1 Slow to create environments....
Job Service Resume service Location Service
Search Webapp
Job Search Index
Resume
Database
GeoIP Database
Vagrantfile
Virt...
Job Service
Resume
service
Location
Service
Employer
Webapp
Job Search
Index
Resume
Database
GeoIP
Database
Virtual Machin...
Our Solution—The Bad
5 Only Linux support
4 Limited support for different application types
1 Slow to create environments....
Shared RAM
Remaining
Memory
Virtual
Machine
Our Solution—The Bad
5 Only Linux support
4 Limited support for different application types
1 Slow to create environments....
Our Solution—The Bad
5 Only Linux support
4 Limited support for different application types
1 Slow to create environments....
We’re all getting Macbooks!
Hooray?
Our Solution—The Bad
5 Only Linux support
4 Limited support for different application types
1 Slow to create environments....
Hobo (hō-bō)
synonyms: vagrant
Hobo goals
1
2
3
4
5
6
Support all types of applications
Easy to share
Simple for developers to use for development and de...
Hobo goals
1
2
3
4
5
6
Support all types of applications
Easy to share
Simple for developers to use for development and de...
Hobo goals
1
2
3
4
5
6
Support all types of applications
Easy to share
Simple for developers to use for development and de...
Hobo goals
1
2
3
4
5
6
Support all types of applications
Easy to share
Simple for developers to use for development and de...
Hobo goals
1
2
3
4
5
6
Support all types of applications
Easy to share
Simple for developers to use for development and de...
Hobo goals
1
2
3
4
5
6
Support all types of applications
Easy to share
Simple for developers to use for development and de...
What is Docker?
Hypervisor
Host OS
Guest
OS
Bins &
Libs
Guest
OS
Bins &
Libs
VM VM
Server with Virtual Machines
Bins & Libs
Host OS
Server...
Docker terminology
Image
An OS filesystem
Docker terminology
Container
A running instance of an image
Docker terminology
Registry
A server for storing images
Hobo goals
1
2
3
4
5
6
Support all types of applications
Easy to share
Simple for developers to use for development and de...
Hobo goals
1
2
3
4
5
6
Support all types of applications
Easy to share
Simple for developers to use for development and de...
Hobo goals
1
2
3
4
5
6
Support all types of applications
Easy to share
Simple for developers to use for development and de...
Our solution
Create and publish a base image
for each application type
Java
CentOS
Java
CentOS
hobo-java
Publish
Docker allows “Inheriting” Images
Java
CentOS
Tomcat
hobo-java
Tomcat
hobo-tomcat
Publish
Ready to use Hobo Images
T O M C A T
Hobo goals
1
2
3
4
5
6
Support all types of applications
Easy to share
Simple for developers to use for development and de...
Job Service Resume service Location Service
Search Webapp
Job Search Index Resume Database GeoIP Database
Virtual Machine
Docker is designed to have one
process per container
Search Webapp
Job Service Resume service Location Service
Job Search Index Resume Database GeoIP Database
Search Webapp
Job Service Resume service Location Service
Job Search Index Resume Database GeoIP Database
Employer Webapp
Networking is easy with One machine
Many machines = we need
service discovery
Service Discovery
Enabling services to find other
services automatically
What is Consul?
A key/value store geared towards service discovery
Consul has both an HTTP interface and a DNS interface
PUT /v1/catalog/register
{"Node": "user-service", "Address": "10.0.0.3"}
$ dig +short user-service.node.consul
10.0.0.3
Now all config files can reference
other containers by name!
How do we start a stack of multiple
dependent applications?
[HOBO-SETTINGS]
depends = user-service search-service location-service
[job-service]
type = application
[job-service-mysql...
[HOBO-SETTINGS]
depends = user-service search-service location-service
[job-service]
type = application
[job-service-mysql...
[HOBO-SETTINGS]
depends = user-service search-service location-service
[job-service]
type = application
[job-service-mysql...
Hobo Config File
user-service
[HOBO-SETTINGS]
depends =
[user-service]
type = application
[user-service-mysql]
type = reso...
Hobo goals
1
2
3
4
5
6
Support all types of applications
Easy to share
Simple for developers to use for development and de...
Hobo goals
1
2
3
4
5
6
Support all types of applications
Easy to share
Simple for developers to use for development and de...
src
test
README
build.xml
hobo
src
test
README
build.xml
hobo
job-service
job-service-mysql
job-service-mongo
hobo_config
job-service
config
Dockerfile
hobo
run.sh
Dockerfile
FROM dockerreg.indeed.net/hobo/hobo-tomcat
job-service
hobo
config
tomcat.xml
mysql-conn.properties
job-service.properties
job-service
hobo
config
myconfig
job-service.properties
tomcat.xml
mysql-conn.properties
job-service.properties
job-service
config
Dockerfile
hobo
run.sh
run.sh
Start/Stop commands
Environment variables
run.sh
Volume mounts
run.sh
What are volume mounts?
They mount parts of the host filesystem
into the container
What are volume mounts?
They mount parts of the host filesystem
into the container
What do we mount?
What do we mount?
Built code
What do we mount?
Configuration files
What do we mount?
Log directories
What do we mount?
Data directories
Hobo goals
1
2
3
4
5
6
Support all types of applications
Easy to share
Simple for developers to use for development and de...
git clone repository
ant hobo-deploy
Using Hobo
Results
2014 Apr Jul Oct 2015 Apr Jul Oct 2016 Apr
Active Usage Increase Over Time
Projects Configured For Hobo
330
“Sometimes I need to run apps from
other teams and the first thing I look
for is ‘Is this hobo’d?’. If so I’m happy!”
- In...
15 minutes 30 seconds
10 times per week means
120 hours per year for each developer
Lessons Learned and Takeaways
Lessons Learned and Takeaways
Volume mounts are useful
Lessons Learned and Takeaways
Make it easy to use for the default
case but fully customizable
Lessons Learned and Takeaways
Docker is moving quickly
Engineering Velocity
go.indeed.com/evtt
Developer
Productivity
Software
Architecture
Release
Process
Control Tower
Automating Release Management
Develop Features
Push Git
Branch
Create Code
Review
Mark Issue as
Resolved
Take Code
Review
Feedback
Creates Pull
Request
...
Complex and manual
1
2
3
4
Goals of Our Release Process
Understand which features are being released
Understand cross-product and cross-team ...
Goals of Our Release Process
1
2
3
4
Understand which features are being released
Understand cross-product and cross-team ...
Goals of Our Release Process
1
2
3
4
Understand which features are being released
Understand cross-product and cross-team ...
Goals of Our Release Process
1
2
3
4
Understand which features are being released
Understand cross-product and cross-team ...
Goals of Our Release Process
1
2
3
4
Understand which features are being released
Understand cross-product and cross-team ...
Goals of Our Release Process
4 Record release details for tracking, analysis, and repeatability
3 Quickly address bugs in ...
Develop Features
Push Git
Branch
Create Code
Review
Mark Issue as
Resolved
Take Code
Review
Feedback
Creates Pull
Request
...
The life of a feature
Test Deploy
& Record
Develop
Feature
Define
Release
We use standard development and
project management tools
&
Git: with great power comes
great complexity
Git flow
master
master
jira/IND-123
master
jira/IND-123
Develop Features
Push Git Branch
Create Code
Review
Mark Issue as
Resolved
Take Code Review
Feedback
Creates Pull
Request
...
Feature code can also be in libraries
common-webapp common-database jobservice-api
Search Webapp
Develop Features
Push Git Branch
Create Code
Review
Mark Issue as
Resolved
Take Code Review
Feedback
Creates Pull
Request
...
Release Managers Execute Releases
Goals of Our Release Process
4 Record release details for tracking, analysis, and repeatability
3 Quickly address bugs in ...
Develop Features
Push Git
Branch
Create Code
Review
Mark Issue as
Resolved
Take Code
Review
Feedback
Creates Pull
Request
...
Develop Features
Push Git
Branch
Create Code
Review
Mark Issue as
Resolved
Take Code
Review
Feedback
Creates Pull
Request
...
master
release
common-webapp common-database jobservice-api
Search Webapp
common-webapp
1.2.6
common-database
3.1.5
jobservice-api
1.0.1
Search Webapp
Build Run Tests Deploy to QA
Goals of Our Release Process
1
2
3
4
Understand which features are being released
Understand cross-product and cross-team ...
What’s in the box?
Look at a dashboard in Jira with everything
that’s supposed to be in the release.
Look at Git log to see commits for features
Look at Git log for each new library version
Look at code reviews for each feature
to make sure they’re complete
Goals of Our Release Process
1
2
3
4
Understand which features are being released
Understand cross-product and cross-team ...
Use Jira for Release Tracking
Bug Bug Feature
Fix Version
Bug Bug Feature
Fix Version
Deploy
Ticket
Bug Bug Feature
Fix Version
Other App
Feature
Deploy
Ticket
Bug Bug Feature
Fix Version
Other App
Feature
Deploy
Ticket
Depends upon
Bug Bug Feature
Fix Version
Other App
Feature
Deploy
Ticket
Depends upon
Depends upon
Test
Goals of Our Release Process
1
2
3
4
Understand which features are being released
Understand cross-product and cross-team ...
Develop Features
Push Git
Branch
Create Code
Review
Mark Issue as
Resolved
Take Code
Review
Feedback
Creates Pull
Request
...
Deploy!
Goals of Our Release Process
1
2
3
4
Understand which features are being released
Understand cross-product and cross-team ...
Develop Features
Push Git
Branch
Create Code
Review
Mark Issue as
Resolved
Take Code
Review
Feedback
Creates Pull
Request
...
master
release
jira/IND-123
master
deploy/2016-06-22
jira/IND-123
jira/IND-123
master
deploy/2016-06-22
master
deploy/2016-06-22
JIRA updates
Close released issues
JIRA updates
Close deploy ticket
JIRA updates
Mark Fix Version as “Released”
Develop Features
Push Git
Branch
Create Code
Review
Mark Issue as
Resolved
Take Code
Review
Feedback
Creates Pull
Request
...
Goals of Our Release Process
4 Record release details for tracking, analysis, and repeatability
3 Quickly address bugs in ...
Problems
1 Release management took a lot of time
2 It was hard to understand what exactly was in a release
3 Lots of manua...
Problems
1 Release management took a lot of time
2 It was hard to understand what exactly was in a release
3 Lots of manua...
Problems
1 Release management took a lot of time
2 It was hard to understand what exactly was in a release
3 Lots of manua...
Problems
1 Release management took a lot of time
2 It was hard to understand what exactly was in a release
3 Lots of manua...
Clicks and Git Commands
>100
Release with 4 features
Extra Actions Per Feature
~13
2014 Apr Jul Oct 2015 Apr Jul Oct 2016
Total Developers Producing Code By Month
2014 Apr Jul Oct 2015 Apr Jul Oct 2016
Unique projects deployed per month
We needed to automate
Why automate rather than simplify?
Data History Transparency
We needed to automate
Control Tower
Goals of Our Release Process
1
2
3
4
Understand which features are being released
Understand cross-product and cross-team ...
Goals of Our Release Process
1
2
3
4
Understand which features are being released
Understand cross-product and cross-team ...
What feature code is in libraries?
common-webapp
1.2.6
common-database
3.1.5
jobservice-api
1.0.1
Indeed Webapp
Goals of Our Release Process
1
2
3
4
Understand which features are being released
Understand cross-product and cross-team ...
Develop Features
Push Git
Branch
Create Code
Review
Mark Issue as
Resolved
Take Code
Review
Feedback
Creates Pull
Request
...
Develop Features
Push Git
Branch
Create Code
Review
Mark Issue as
Resolved
Take Code
Review
Feedback
Creates Pull
Request
...
Goals of Our Release Process
1
2
3
4
Understand which features are being released
Understand cross-product and cross-team ...
Develop Features
Push Git
Branch
Create Code
Review
Mark Issue as
Resolved
Take Code
Review
Feedback
Creates Pull
Request
...
master
release
jira/IND-123
master
release
jira/IND-123
IND-123 commits In release IND-123 commit Not in release
master
release
jira/IND-123
Goals of Our Release Process
1
2
3
4
Understand which features are being released
Understand cross-product and cross-team ...
Develop Features
Push Git
Branch
Create Code
Review
Mark Issue as
Resolved
Take Code
Review
Feedback
Creates Pull
Request
...
Develop Features
Push Git
Branch
Create Code
Review
Mark Issue as
Resolved
Take Code
Review
Feedback
Creates Pull
Request
...
Develop Features
Push Git
Branch
Create Code
Review
Mark Issue as
Resolved
Take Code
Review
Feedback
Creates Pull
Request
...
Test DeployDevelop
Feature
Define
Release
Results
Problems
1 Release management took a lot of time
2 It was hard to understand what exactly was in a release
3 Lots of manua...
Clicks and Git Commands
>100
Release with 4 features
Control Tower Clicks
<15
Release with 4 features
Extra Actions Per Feature
~13
Extra Actions Per Feature
0
Jul Oct 2015 Apr Jul Oct 2016 Apr
Deploys per month
Deploys Per Project
1.9X
Problems
1 Release management took a lot of time
2 It was hard to understand what exactly was in a release
3 Lots of manua...
Problems
1 Release management took a lot of time
2 It was hard to understand what exactly was in a release
3 Lots of manua...
More knowledge = Less risk
of developers are release managers
57%
To start doing releases
~3 months
Lessons Learned and Takeaways
Lessons Learned and Takeaways
Everything can be automated
Lessons Learned and Takeaways
Engineers don’t like mystery
Lessons Learned and Takeaways
Automate the process you have, not
the process you want
Where do we go from here?
Too much human interaction
Train machines to handle releases
Engineering blog & talks - indeed.tech
Open Source - opensource.indeedeng.io
Careers - indeed.jobs
Twitter - @IndeedEng
Le...
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production
Prochain SlideShare
Chargement dans…5
×

Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production

Link to video: https://youtu.be/aHHfq4WK9Jw

At Indeed, we're growing quickly, from our engineer headcount to the number of features we deploy. Over the last three years, we’ve had a 6x increase in engineers, and a 15x increase in number of deploys. We’re currently deploying over 700 new features each week. In this talk, we'll describe the infrastructure built to support, scale and automate our software development and product releases, and how any organization can use these tools and techniques to improve release velocity in the face of rapid growth. Specifically, we will discuss Hobo — an easy, standardized way for developers to run our application stacks in Docker. We’ll also describe Control Tower, which manages software releases by unifying all of the information about application features into a single interface. These tools allow engineers to focus on product development, while moving their work from idea to production as efficiently as possible.

  • Soyez le premier à commenter

Automation and Developer Infrastructure — Empowering Engineers to Move from Idea to Production

  1. 1. Automation and Developer Infrastructure Empowering Engineers to Move from Idea to Production
  2. 2. Alex Thurlow Engineering Manager
  3. 3. I help people get jobs.
  4. 4. Indeed is the #1 external source of hire Unique Visitors (millions) Million unique visitors 2009 2011 2012 2013 2014 2015 0 20 40 60 80 100 120 140 160 180 2010 180M 180 million unique users 80.2M unique US visitors per month 16M jobs 60+ countries 28 languages 64% of US job searchers search on indeed each month
  5. 5. Alex Thurlow
  6. 6. Plus many more!
  7. 7. What should we build?
  8. 8. What’s best for the jobseeker?
  9. 9. How do we know?
  10. 10. Data-Driven Product Design go.indeed.com/ddpd
  11. 11. Learn Measure Build Lean Startup Cycle
  12. 12. Deliver Learn Measure
  13. 13. Data driven product design requires quick feedback
  14. 14. Engineering Velocity go.indeed.com/evtt
  15. 15. Engineering Velocity go.indeed.com/evtt Developer Productivity Software Architecture Release Process
  16. 16. Engineering Velocity go.indeed.com/evtt Developer Productivity Software Architecture Release Process
  17. 17. Service-Oriented Architecture
  18. 18. Make small, independent applications
  19. 19. Deliver Learn Measure Quickly!
  20. 20. Deliver Learn Measure
  21. 21. Engineering Velocity go.indeed.com/evtt Developer Productivity Software Architecture Release Process
  22. 22. Jon Hanks Software Engineer
  23. 23. I help people get jobs.
  24. 24. Hobo Automated developer provisioning
  25. 25. First goal for a new developer Push code to production in week 1
  26. 26. Deliver Learn Measure
  27. 27. 1:1 Mentorship
  28. 28. Setup = 2-3 Days
  29. 29. 1-2 days left to get to production
  30. 30. Team Rotation
  31. 31. Helps new developer understand Indeed
  32. 32. Rotate every 4 weeks through 3 teams
  33. 33. Rotate every 4 weeks through 3 teams and face the same setup challenges in each team. Frustration!!!
  34. 34. Experienced devs face a similar problem working on multiple projects or switching projects
  35. 35. Service-Oriented Architecture
  36. 36. Job Database Search Webapp
  37. 37. Job Service Resume service Location Service Search Webapp Job Search Index Resume Database GeoIP Database
  38. 38. Need to run several interconnected applications
  39. 39. Total Developers Producing Code By Month 2014 Apr Jul Oct 2015 Apr Jul Oct 2016
  40. 40. 1:1 mentorship didn’t scale
  41. 41. Use technology to speed developer provisioning
  42. 42. Vagrant automates creating virtualized development environments
  43. 43. Job Service Resume service Location Service Search Webapp Job Search Index Resume Database GeoIP Database Vagrantfile Virtual Machine
  44. 44. Success! (kind of)
  45. 45. Reproducible environments Our Solution—The Good
  46. 46. Our Solution—The Good Sharable between devs
  47. 47. Our Solution—The Good <1 day setup time
  48. 48. Our Solution—The Bad 5 Only Linux support 4 Limited support for different application types 1 Slow to create environments. 15 minutes or more! 2 Hard initial setup especially for large application stacks 3 Virtual machine memory overhead
  49. 49. Our Solution—The Bad 5 Only Linux support 4 Limited support for different application types 1 Slow to create environments. 15 minutes or more! 2 Hard initial setup especially for large application stacks 3 Virtual machine memory overhead
  50. 50. Download full machine image
  51. 51. Download and install required software
  52. 52. Start every application
  53. 53. Our Solution—The Bad 5 Only Linux support 4 Limited support for different application types 1 Slow to create environments. 15 minutes or more! 2 Hard initial setup especially for large application stacks 3 Virtual machine memory overhead
  54. 54. Job Service Resume service Location Service Search Webapp Job Search Index Resume Database GeoIP Database Vagrantfile Virtual Machine
  55. 55. Job Service Resume service Location Service Employer Webapp Job Search Index Resume Database GeoIP Database Virtual Machine Job Service Resume service Location Service Search Webapp Job Search Index Resume Database GeoIP Database Virtual Machine
  56. 56. Our Solution—The Bad 5 Only Linux support 4 Limited support for different application types 1 Slow to create environments. 15 minutes or more! 2 Hard initial setup especially for large application stacks 3 Virtual machine memory overhead
  57. 57. Shared RAM
  58. 58. Remaining Memory Virtual Machine
  59. 59. Our Solution—The Bad 5 Only Linux support 4 Limited support for different application types 1 Slow to create environments. 15 minutes or more! 2 Hard initial setup especially for large application stacks 3 Virtual machine memory overhead
  60. 60. Our Solution—The Bad 5 Only Linux support 4 Limited support for different application types 1 Slow to create environments. 15 minutes or more! 2 Hard initial setup especially for large application stacks 3 Virtual machine memory overhead
  61. 61. We’re all getting Macbooks! Hooray?
  62. 62. Our Solution—The Bad 5 Only Linux support 4 Limited support for different application types 1 Slow to create environments. 15 minutes or more! 2 Hard initial setup especially for large application stacks 3 Virtual machine memory overhead
  63. 63. Hobo (hō-bō) synonyms: vagrant
  64. 64. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  65. 65. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  66. 66. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  67. 67. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  68. 68. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  69. 69. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  70. 70. What is Docker?
  71. 71. Hypervisor Host OS Guest OS Bins & Libs Guest OS Bins & Libs VM VM Server with Virtual Machines Bins & Libs Host OS Server with Docker Containers DockerEngine Container1 Container2 Container3
  72. 72. Docker terminology Image An OS filesystem
  73. 73. Docker terminology Container A running instance of an image
  74. 74. Docker terminology Registry A server for storing images
  75. 75. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  76. 76. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  77. 77. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  78. 78. Our solution Create and publish a base image for each application type
  79. 79. Java CentOS
  80. 80. Java CentOS hobo-java Publish
  81. 81. Docker allows “Inheriting” Images
  82. 82. Java CentOS Tomcat
  83. 83. hobo-java Tomcat hobo-tomcat Publish
  84. 84. Ready to use Hobo Images T O M C A T
  85. 85. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  86. 86. Job Service Resume service Location Service Search Webapp Job Search Index Resume Database GeoIP Database Virtual Machine
  87. 87. Docker is designed to have one process per container
  88. 88. Search Webapp Job Service Resume service Location Service Job Search Index Resume Database GeoIP Database
  89. 89. Search Webapp Job Service Resume service Location Service Job Search Index Resume Database GeoIP Database Employer Webapp
  90. 90. Networking is easy with One machine
  91. 91. Many machines = we need service discovery
  92. 92. Service Discovery Enabling services to find other services automatically
  93. 93. What is Consul? A key/value store geared towards service discovery Consul has both an HTTP interface and a DNS interface
  94. 94. PUT /v1/catalog/register {"Node": "user-service", "Address": "10.0.0.3"} $ dig +short user-service.node.consul 10.0.0.3
  95. 95. Now all config files can reference other containers by name!
  96. 96. How do we start a stack of multiple dependent applications?
  97. 97. [HOBO-SETTINGS] depends = user-service search-service location-service [job-service] type = application [job-service-mysql] type = resource [job-service-mongo] type = resource Hobo Config File
  98. 98. [HOBO-SETTINGS] depends = user-service search-service location-service [job-service] type = application [job-service-mysql] type = resource [job-service-mongo] type = resource Hobo Config File
  99. 99. [HOBO-SETTINGS] depends = user-service search-service location-service [job-service] type = application [job-service-mysql] type = resource [job-service-mongo] type = resource Hobo Config File
  100. 100. Hobo Config File user-service [HOBO-SETTINGS] depends = [user-service] type = application [user-service-mysql] type = resource
  101. 101. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  102. 102. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  103. 103. src test README build.xml
  104. 104. hobo src test README build.xml
  105. 105. hobo job-service job-service-mysql job-service-mongo hobo_config
  106. 106. job-service config Dockerfile hobo run.sh
  107. 107. Dockerfile FROM dockerreg.indeed.net/hobo/hobo-tomcat
  108. 108. job-service hobo config tomcat.xml mysql-conn.properties job-service.properties
  109. 109. job-service hobo config myconfig job-service.properties tomcat.xml mysql-conn.properties job-service.properties
  110. 110. job-service config Dockerfile hobo run.sh
  111. 111. run.sh Start/Stop commands
  112. 112. Environment variables run.sh
  113. 113. Volume mounts run.sh
  114. 114. What are volume mounts? They mount parts of the host filesystem into the container
  115. 115. What are volume mounts? They mount parts of the host filesystem into the container
  116. 116. What do we mount?
  117. 117. What do we mount? Built code
  118. 118. What do we mount? Configuration files
  119. 119. What do we mount? Log directories
  120. 120. What do we mount? Data directories
  121. 121. Hobo goals 1 2 3 4 5 6 Support all types of applications Easy to share Simple for developers to use for development and debugging Support complex application stacks simply OS X support Quick startup
  122. 122. git clone repository ant hobo-deploy Using Hobo
  123. 123. Results
  124. 124. 2014 Apr Jul Oct 2015 Apr Jul Oct 2016 Apr Active Usage Increase Over Time
  125. 125. Projects Configured For Hobo 330
  126. 126. “Sometimes I need to run apps from other teams and the first thing I look for is ‘Is this hobo’d?’. If so I’m happy!” - Indeed Developer
  127. 127. 15 minutes 30 seconds
  128. 128. 10 times per week means 120 hours per year for each developer
  129. 129. Lessons Learned and Takeaways
  130. 130. Lessons Learned and Takeaways Volume mounts are useful
  131. 131. Lessons Learned and Takeaways Make it easy to use for the default case but fully customizable
  132. 132. Lessons Learned and Takeaways Docker is moving quickly
  133. 133. Engineering Velocity go.indeed.com/evtt Developer Productivity Software Architecture Release Process
  134. 134. Control Tower Automating Release Management
  135. 135. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version Yes Does the feature meet all release criteria? Yes
  136. 136. Complex and manual
  137. 137. 1 2 3 4 Goals of Our Release Process Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  138. 138. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  139. 139. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  140. 140. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  141. 141. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  142. 142. Goals of Our Release Process 4 Record release details for tracking, analysis, and repeatability 3 Quickly address bugs in releases 2 Understand cross-product and cross-team dependencies 1 Understand which features are being released
  143. 143. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version Yes Does the feature meet all release criteria? Yes
  144. 144. The life of a feature
  145. 145. Test Deploy & Record Develop Feature Define Release
  146. 146. We use standard development and project management tools
  147. 147. &
  148. 148. Git: with great power comes great complexity
  149. 149. Git flow
  150. 150. master
  151. 151. master jira/IND-123
  152. 152. master jira/IND-123
  153. 153. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Develop Feature =
  154. 154. Feature code can also be in libraries
  155. 155. common-webapp common-database jobservice-api Search Webapp
  156. 156. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Develop Feature =
  157. 157. Release Managers Execute Releases
  158. 158. Goals of Our Release Process 4 Record release details for tracking, analysis, and repeatability 3 Quickly address bugs in releases 2 Understand cross-product and cross-team dependencies 1 Understand which features are being released
  159. 159. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version Yes Does the feature meet all release criteria? Yes
  160. 160. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  161. 161. master release
  162. 162. common-webapp common-database jobservice-api Search Webapp
  163. 163. common-webapp 1.2.6 common-database 3.1.5 jobservice-api 1.0.1 Search Webapp
  164. 164. Build Run Tests Deploy to QA
  165. 165. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  166. 166. What’s in the box?
  167. 167. Look at a dashboard in Jira with everything that’s supposed to be in the release.
  168. 168. Look at Git log to see commits for features
  169. 169. Look at Git log for each new library version
  170. 170. Look at code reviews for each feature to make sure they’re complete
  171. 171. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  172. 172. Use Jira for Release Tracking
  173. 173. Bug Bug Feature Fix Version
  174. 174. Bug Bug Feature Fix Version Deploy Ticket
  175. 175. Bug Bug Feature Fix Version Other App Feature Deploy Ticket
  176. 176. Bug Bug Feature Fix Version Other App Feature Deploy Ticket Depends upon
  177. 177. Bug Bug Feature Fix Version Other App Feature Deploy Ticket Depends upon Depends upon
  178. 178. Test
  179. 179. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  180. 180. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  181. 181. Deploy!
  182. 182. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  183. 183. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  184. 184. master release jira/IND-123
  185. 185. master deploy/2016-06-22 jira/IND-123
  186. 186. jira/IND-123 master deploy/2016-06-22
  187. 187. master deploy/2016-06-22
  188. 188. JIRA updates Close released issues
  189. 189. JIRA updates Close deploy ticket
  190. 190. JIRA updates Mark Fix Version as “Released”
  191. 191. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  192. 192. Goals of Our Release Process 4 Record release details for tracking, analysis, and repeatability 3 Quickly address bugs in releases 2 Understand cross-product and cross-team dependencies 1 Understand which features are being released
  193. 193. Problems 1 Release management took a lot of time 2 It was hard to understand what exactly was in a release 3 Lots of manual steps = possibility for error 4 Only senior engineers knew enough to handle releases
  194. 194. Problems 1 Release management took a lot of time 2 It was hard to understand what exactly was in a release 3 Lots of manual steps = possibility for error 4 Only senior engineers knew enough to handle releases
  195. 195. Problems 1 Release management took a lot of time 2 It was hard to understand what exactly was in a release 3 Lots of manual steps = possibility for error 4 Only senior engineers knew enough to handle releases
  196. 196. Problems 1 Release management took a lot of time 2 It was hard to understand what exactly was in a release 3 Lots of manual steps = possibility for error 4 Only senior engineers knew enough to handle releases
  197. 197. Clicks and Git Commands >100 Release with 4 features
  198. 198. Extra Actions Per Feature ~13
  199. 199. 2014 Apr Jul Oct 2015 Apr Jul Oct 2016 Total Developers Producing Code By Month
  200. 200. 2014 Apr Jul Oct 2015 Apr Jul Oct 2016 Unique projects deployed per month
  201. 201. We needed to automate
  202. 202. Why automate rather than simplify? Data History Transparency
  203. 203. We needed to automate
  204. 204. Control Tower
  205. 205. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  206. 206. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  207. 207. What feature code is in libraries?
  208. 208. common-webapp 1.2.6 common-database 3.1.5 jobservice-api 1.0.1 Indeed Webapp
  209. 209. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  210. 210. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  211. 211. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  212. 212. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  213. 213. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  214. 214. master release jira/IND-123
  215. 215. master release jira/IND-123 IND-123 commits In release IND-123 commit Not in release
  216. 216. master release jira/IND-123
  217. 217. Goals of Our Release Process 1 2 3 4 Understand which features are being released Understand cross-product and cross-team dependencies Quickly address bugs in releases Record release details for tracking, analysis, and repeatability
  218. 218. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  219. 219. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  220. 220. Develop Features Push Git Branch Create Code Review Mark Issue as Resolved Take Code Review Feedback Creates Pull Request Local development Define Release Check code review statuses for issues Check Git branch merge status Check with developer Check dependent issues of each feature Check Issue Statuses Add to list of features for release Determine which library upgrades are available or needed Merge desired features to release Upgrade library dependencies Push to origin Create release branch Update issue statuses Create fix version Create deploy ticket Add fix version to issues in release Link other dependent issues to deploy ticket Test Release Run unit tests in Jenkins Send bugs back to developer Test release in QA environment Build Release in Jenkins Deploy to production Developer fixes bugs Merge feature branch to release branch Jenkins deploys in QA environment Are there bugs? Yes Does the feature meet all release criteria? Yes Record Release Delete release branch Merge release to integration branch Tag Release Close Issues Delete feature branches Merge release to production branch Close Deploy ticket Release fix version
  221. 221. Test DeployDevelop Feature Define Release
  222. 222. Results
  223. 223. Problems 1 Release management took a lot of time 2 It was hard to understand what exactly was in a release 3 Lots of manual steps = possibility for error 4 Only senior engineers knew enough to handle releases
  224. 224. Clicks and Git Commands >100 Release with 4 features
  225. 225. Control Tower Clicks <15 Release with 4 features
  226. 226. Extra Actions Per Feature ~13
  227. 227. Extra Actions Per Feature 0
  228. 228. Jul Oct 2015 Apr Jul Oct 2016 Apr Deploys per month
  229. 229. Deploys Per Project 1.9X
  230. 230. Problems 1 Release management took a lot of time 2 It was hard to understand what exactly was in a release 3 Lots of manual steps = possibility for error 4 Only senior engineers knew enough to handle releases
  231. 231. Problems 1 Release management took a lot of time 2 It was hard to understand what exactly was in a release 3 Lots of manual steps = possibility for error 4 Only senior engineers knew enough to handle releases
  232. 232. More knowledge = Less risk
  233. 233. of developers are release managers 57%
  234. 234. To start doing releases ~3 months
  235. 235. Lessons Learned and Takeaways
  236. 236. Lessons Learned and Takeaways Everything can be automated
  237. 237. Lessons Learned and Takeaways Engineers don’t like mystery
  238. 238. Lessons Learned and Takeaways Automate the process you have, not the process you want
  239. 239. Where do we go from here?
  240. 240. Too much human interaction
  241. 241. Train machines to handle releases
  242. 242. Engineering blog & talks - indeed.tech Open Source - opensource.indeedeng.io Careers - indeed.jobs Twitter - @IndeedEng Learn More

×