Presentation by Nicole Forsgren Velasquez | Jez Humble | Nigel Kersten | Gene Kim

1. DevOps
Means Business
Presented by
Nicole Forsgren Velasquez | Jez Humble | Nigel Kersten | Gene Kim
Findings From Puppet Labs State Of DevOps 2014
Velocity Conference, Santa Clara
June 26, 2014
Puppet Labs Webinar 2014 State of DevOps

2. Puppet Labs Webinar 2014 State of DevOps
Agenda
● Demographics
● Key Findings
● Survey Methodology and Analysis
● IT Performance
● Organizational Culture
● Job Satisfaction
● Big Surprises
● Future Plans
● Q&A

3. Gene
Kim
Multiple award-
winning CTO,
researcher and co-
author of The Phoenix
Project: A Novel About
IT, DevOps, and
Helping Your Business
Win. First time Google
Slides user.
Presenters
Nicole Forsgren
Velasquez
Professor at the
Huntsman School of
Business at Utah State
University. IT impacts,
knowledge
management, and
analytics consultant
and researcher.
Stats ninja.
Jez
Humble
Principal at
Thoughtworks,
Lecturer at UC
Berkeley, co-author of
Continuous Delivery,
and accomplished
procrastinator.
Nigel
Kersten
CIO at Puppet Labs
and responsible for
IT/Operations there.
Almost fully recovered
sysadmin.
Puppet Labs Webinar 2014 State of DevOps

4. Puppet Labs Webinar 2014 State of DevOps
9,200+
Respondents from
110 countries,
across all industries
Demographics
@botchagalupe#devops survey early observations #4 Retail, Banking,
Finance and Healthcare respondents LT 15% total (typically the risk averse
sectors)

5. 27% of
responses
from companies
with 500 to 9,999
employees
Size of Organization
Puppet Labs Webinar 2014 State of DevOps

6. Puppet Labs Webinar 2014 State of DevOps
Size of Infrastructure
51% of
respondents
said their
infrastructure
included
<500 Servers
@botchagalupe#devops survey early observations #1
Only 40% of respondents have more than 500 servers

7. Puppet Labs Webinar 2014 State of DevOps
Departments
16%
identified as
DevOps
Department

8. 2012
2013
2014
DevOps Grows Up
What is devops?
DevOps adoption is accelerating.
Holy cow! DevOps works!
Puppet Labs Webinar 2014 State of DevOps

9. Why I Care:
IT Performance Matters
Puppet Labs Webinar 2014 State of DevOps

10. Cross-population Studies
Tradeshows
Industry Press
Blogosphere
Vendors
Consultants
Industry Analyst
Anecdote
Peer
Recommendation
Prior Experience
ROI Story Best Practice
Benchmark
Population Study
Academic Study
Intuitive Analytical
We Are Here

11. High-performing IT Orgs are
More Agile
Puppet Labs Webinar 2014 State of DevOps
30x 8,000x
more frequent
deployments
faster lead times
than their peers

12. High-performing IT Orgs are
More Reliable
Puppet Labs Webinar 2014 State of DevOps
2x 12x
the change
success rate
faster mean time to
recover (MTTR)

13. Firms with high-performing IT
orgs are winning
more likely to
exceed
profitability,
market share
& productivity
goals
Puppet Labs Webinar 2014 State of DevOps
2x higher market
capitalization
growth over 3
years*
50%

14. How We Collected Responses
Puppet Labs Webinar 2014 State of DevOps
Snowball Sampling

15. Questions We Asked
Puppet Labs Webinar 2014 State of DevOps
Likert-type questions give us a more nuanced
measurement.

16. Latent Constructs
Puppet Labs Webinar 2014 State of DevOps
Some things can’t be measured directly, like happiness.
“Happiness Statistics” by Andrew Tarvin is licensed under CC BY 2.0

17. Quantifying IT Performance
Puppet Labs Webinar 2014 State of DevOps
Deploy
30X
more frequently
with
50%
fewer failures
● Deploy Frequency
● Mean Time to Recover (MTTR)
● Lead Time for Changes

18. Categorizing Companies: High,
Medium, Low IT Performance
Puppet Labs Webinar 2014 State of DevOps
Deploy Frequency
Mean Time to
Recover (MTTR)
Lead Time for
Changes
MTTR
Lead Time
Deploy Frequency
High-performing IT orgs
Medium-performing IT orgs
Low-performing IT orgs

19. Puppet Labs Webinar 2014 State of DevOps
Practices Correlated with IT
Performance Metrics
Throughput Metrics Stability Metrics
Deployment Frequency
● Continuous Delivery
● Version Control
Mean Time to Recover (MTTR)
● Version Control
● Monitoring System and
Application Health
Lead Time for Changes
● Version Control
● Automated Testing
Change Fail Rate
● Not strongly correlated with
specific practices.
● Significant differences between
groups with high, medium and
low change fail rates. High
performing IT orgs have 50%
lower change fail rates than
their peers.(mention Roy
Rapoport!)

20. Puppet Labs Webinar 2014 State of DevOps
Practices Correlated with IT
Performance Metrics
Throughput Metrics Stability Metrics
Deployment Frequency
● Continuous Delivery
● Version Control
Mean Time to Recover (MTTR)
● Version Control
● Monitoring System and
Application Health
Lead Time for Changes
● Version Control
● Automated Testing
Change Fail Rate
● Not strongly correlated with
specific practices.
● Significant differences between
groups with high, medium and
low change fail rates. High
performing IT orgs have 50%
lower change fail rates than
their peers.(mention Roy
Rapoport!)

21. Puppet Labs Webinar 2014 State of DevOps
Practices Correlated with IT
Performance Metrics
Throughput Metrics Stability Metrics
Deployment Frequency
● Continuous Delivery
● Version Control
Mean Time to Recover (MTTR)
● Version Control
● Monitoring System and
Application Health
Lead Time for Changes
● Version Control
● Automated Testing
Change Fail Rate
● Not strongly correlated with
specific practices.
● Significant differences between
groups with high, medium and
low change fail rates. High
performing IT orgs have 50%
lower change fail rates than
their peers.(mention Roy
Rapoport!)

22. Puppet Labs Webinar 2014 State of DevOps
Practices Correlated with IT
Performance Metrics
Throughput Metrics Stability Metrics
Deployment Frequency
● Continuous Delivery
● Version Control
Mean Time to Recover (MTTR)
● Version Control
● Monitoring System and
Application Health
Lead Time for Changes
● Version Control
● Automated Testing
Change Fail Rate
● Not strongly correlated with
specific practices.
● Significant differences between
groups with high, medium and
low change fail rates. High
performing IT orgs have 50%
lower change fail rates than
their peers.(mention Roy
Rapoport!)

23. Puppet Labs Webinar 2014 State of DevOps
Top Predictors of IT
Performance
• Peer-review of production changes (vs.
external change approval)
• Version control of all production artifacts
• Proactive monitoring of the production
environment
• High trust culture
• Win-win relationship between Dev, Ops and
Infosec
• High job satisfaction

24. Puppet Labs Webinar 2014 State of DevOps
Feedback Loops
Source: Elisabeth Hendrickson: Care And Feeding Of Feedback Loops

25. Puppet Labs Webinar 2014 State of DevOps
Organizational Culture
15% 52% 33%

26. Puppet Labs Webinar 2014 State of DevOps
Organizational Culture
Resources
● http://www.thisamericanlife.org/radio-archives/episode/403/nummi
● http://sloanreview.mit.edu/article/how-to-change-a-culture-lessons-from-nummi/
● Read Toyota Kata by Mike Rother

27. Puppet Labs Webinar 2014 State of DevOps
Job Satisfaction
Job satisfaction is the # 1
predictor of organizational
performance!
Top Predictors
of Job Satisfaction
★ High-trust organizational
culture
★ Climate of learning
★ Win-win relationships
between ops, dev and infosec
teams
★ Proactive monitoring and
autoscaling
★ Use of version control for all
production artifacts
★ Automated testing

28. Gene
• Version control of the environment is more
important than version control of the code!
• There’s never been a better time to know
statistics
Jez
• Whether or not you have an integration or
stabilization phase has zero impact on IT perf
• Peer review is more effective than CAB
Surprises
Puppet Labs Webinar 2014 State of DevOps

29. Can Large Orgs Be High
Performers?
Puppet Labs Webinar 2014 State of DevOps
Yes.
But orgs with 10,000+
employees 40% less likely
to be high performing vs.
500 employee orgs…

30. Can Large Orgs Adopt
These Practices?
Puppet Labs Webinar 2014 State of DevOps
Yes!
(Automated testing,
Continuous integration,
proactive monitoring…)
The only practice not being
adopted is Peer Review vs.
Change Approval!

31. Nicole
• DevOps practices and IT performance impact
organizational performance
• Change fail rate wasn’t part of IT performance
Nigel
• Forming new DevOps teams and giving people
DevOps titles is successful in practice
Surprises
Puppet Labs Webinar 2014 State of DevOps

32. • Collect more stock tickers!
• 2014-15 research goals
• What do you want to know?
• Put your ideas and problems you want to
explore in this Google+
Community:http://bit.ly/FutureStateOfDevOps
Puppet Labs Webinar 2014 State of DevOps
Future Plans

33. bit.ly/puppetquestions

34. Sept 3-4 | FlowCon
FlowCon brings together technologists and industry leaders passionate about
innovation through continuous delivery, continuous design, and lean product
development. Register at http://flowcon.org/
Sept 23-24 | PuppetConf
Join us for the IT automation event of the year. Register now to save $249.
http://puppetconf.com
Oct 21-23 | DevOps Enterprise Summit
A conference for horses, by horses, featuring Macy’s Disney, GE Capital, and
more. Register at http://devopsenterprise.io/
Nov 9-14 | LISA ‘14
Systems engineering and ops professionals share real-world knowledge about
designing, building & maintaining critical systems.
https://www.usenix.org/conference/lisa14
Puppet Labs Webinar 2014 State of DevOps
Upcoming Events

35. Resources
• Download the 2014 State of DevOps Report:
http://puppetlabs.com/2014-devops-report
• Pre-order Jez’s forthcoming book Lean
Enterprise: Adopting Continuous Delivery,
DevOps, and Lean Startup at Scale:
http://amzn.com/1449368425
•Sign up for Gene’s newsletter to receive a
sample chapter of his forthcoming book DevOps
Cookbook:
http://itrevolution.com/blog