The document summarizes findings from the State of DevOps report on IT performance. It discusses how high-performing IT organizations were twice as likely to exceed goals in areas like profitability and productivity. Key metrics for IT performance included lead time for changes, release frequency, time to restore service, and change fail rate. The document also discusses how surveys and log data were analyzed, with a focus on establishing validity and reliability of measures. Demographics showed little difference among large enterprises. High-performing organizations had significantly better throughput and shorter lead times. Security was also found to be addressed more effectively by high-performing teams.

2. Agenda
• A brief history of the State of DevOps Report
• Theory, survey design, demographics and psychographics
• IT and security performance findings
• Continuous integration/delivery redux

3. About the authors
Jez Humble
@jezhumble
Nicole
Forsgren,
PhD
@nicolefv
Gene Kim
@realgenekim
Nigel
Kersten
@nigelkersten
Alanna
Brown
@alannapb

4. DevOps Grows Up
2012:What is devops?
2013:DevOps adoptionis
accelerating.
2014:Holy cow! DevOps
works!
2015:IT goes lean.
2016:Shifting left.

5. IT performance matters!
Firms with high-performing IT organizations were twice as likely to exceed their
profitability,market share and productivity goals.
http://bit.ly/2015-devops-report/http://bit.ly/2014-devops-report/

6. IT performance
• Lead time for changes
• Release frequency
• Time to restore service
• Change fail rate

7. Not all data is created equal
• Who here thinks surveys are sh*t?
[Nicole should probably turn around]
• Who here LOVES the data from their log files?
And who has seen sh*t data in a log file?

8. What is a latent construct?

9. We use
PSYCHOMETRICS
to make our survey data good*
*or give us a reasonable assurance that it’s telling us what we
think it’s telling us (& some of this can also apply to your log
data)

10. Psychometrics includes:
Construct creation (manual)
• When possible:use previously validated
constructs
• Based on definitions and theory, carefully
and precisely worded,card sorting task,
pilot tested
Construct evaluation (statistics)
• EstablishingValidity:discriminant and
convergent
• Establishing Reliability

11. Analysis methods
Statistics bingo!
• Measure assessment:measures exhibit good psychometric properties (composite reliability,
AVE, EFA measures load well and do not cross-load,etc)
• Model assessment:PLS was used to assess the structured equation models

12. Analysis methods
A note about prediction:one of three conditions must be met:
1. Longitudinal (no, this is cross-sectional)
2. Randomized,experimental design(no, this is a non-experimental)
3. Theory-based design
When this conditionwas not met,only correlations were tested and reported.

13. Demographics
* We don’t’see significant differences among enterprise organizations

14. Demographics

15. Demographics

16. High-performing organizations are
decisively outperforming their lower-
performing peers in terms of
throughput.

17. “What is your lead time for changes?”
“How long does it take to go from code committed to code
successfully running in production?”

20. IT Performance Over theYears
Deploy Frequency Change LeadTime MeanTime to Recover

21. Employees in high-performing
organizations are 2.2 times more
likely to recommend their
organization as a great place to
work.
More likelyto recommend their
organization to a friend.

22. Because they address security at every
stage,high-performing teams spend less
time fixing security issues.

23. Capital One: DevOpsSec
Information Security
Business Development Operations
• Application Security
• Information Security
• Security Security
• Infrastructure Security
• Requirements
• Feature Request
• Roadmap
• Architecture
• Design
• Code
• Test
• Infrastructure
• Platforms
• Environment
• Deployment
• Incident Mgmt.
• Change & Release Mgmt.
DevOpsSec

24. • Conduct security review for all major
features
• Integrate Information Security into the
entire software delivery lifecycle
• Test security requirements as part of
automated testing process.
• Ensure that Information Security
defines pre-approved,easy-to-
consume libraries,packages,toolchains
and processes

25. Security is a priority in my organization
These industries responded highest (75%
Agree or Strongly Agree)
• Financial Services
• Government
Strongly agree
Strongly disagree
Neutral
These departments responded highest
(71-73%Agree or Strongly Agree)
• Professional Services
• Release Engineering

26. High performers spend 29% more time
on new work than low performers,and
22% less time on unplanned work and
rework.
21%

27. Impact of continuous delivery
Effective test data management
Comprehensive, fast and reliable test and
deployment automation
Trunk based development and
continuous integration
Applicationcode and app & system
configurationall in versioncontrol
Incorporating security(and
securityteams) into the delivery
process
Together the factors onthe
left model continuous
delivery whichleads to
Less rework
Lower levels of development pain
Generative performance-oriented culture
(per Westrum’s Model)
Higher levels of IT performance (higher
throughput and stability)
Identifying strongly with the
organization you work for
Higher level of org
performance (productivity,
market share,profitability)
Lower change fail rates

28. “Identifying strongly with the organization”
• I am glad I chose to work for this organization rather than another company.
• I talk of this organization to my friends as a great company to work for.
• I am willing to put in a great deal of effort beyond what is normally expected to help my
organization to be successful.
• I find that my values and my organization's values are very similar.
• In general, the people employed by my organization are working toward the same goal.
• I feel that my organization cares about me.
These items are adapted from Kankanhalli, Atreyi, Kwok-Kee Wei, and Bernard C.Y.Tan (2005)

29. Lean product management
Gathering, broadcasting,and implementing
customer feedback
Splitting work into small batches and
making visible the flow of work through
the delivery process
Together the factors onthe
left model continuous
delivery whichleads to
Generative performance-oriented culture
(per Westrum’s Model)
Higher levels of IT performance (higher
throughput and stability)
Identifying strongly with the organization
youwork for
Higher level of org
performance (productivity,
market share,profitability)

30. Thank you!