What is the best way to measure DevOps performance? And, how can it be done in a scientific way? In this webinar, Dr. Nicole Forsgren will present the frameworks and methodologies uniquely suited to evaluating the way we build and scale software applications. She’ll highlight lessons learned through a four-year research project presented in her upcoming book, Accelerate, written along with Jez Humble and Gene Kim.
2. 2
Housekeeping
▪ This webinar is being recorded
▪ Links to the slides and the
recording will be made available
after the presentation
▪ You can post questions via the
GoToWebinar Control Panel
4. 4
The Highlights
▪ Measuring Performance
− Common Mistakes
▪ Our Approach: Software Delivery Performance
− Where are you?
▪ Maturity Models
▪ Does Software Delivery Performance Matter?
▪ How to Improve Performance
▪ You can help
6. 6
Common Mistakes
▪ Outputs vs. Outcomes
▪ Individual/local vs. Team/global
▪ Some common examples:
− Lines of code
− Velocity
− Utilization
7. 7
Common Mistakes: Lines of Code
▪ More is better?
− Bloated software
− Higher maintenance costs
− Higher cost of change
▪ Less is better?
− Cryptic code that no one can read
▪ Ideal: solve business problems with most efficient code
8. 8
Common Mistakes: Velocity
▪ Agile: problems are broken down into stories, which are
assigned “points” of estimated effort to complete
▪ At end of sprint, total points signed off by customer is
recorded = velocity
▪ Velocity is a capacity planning tool. NOT a productivity tool.
▪ Why doesn’t this work for productivity?
− Velocity is a relative measure, not absolute. So: bad for comparing
teams
− Gaming by inflating estimates
− Focus on team completion at the expense of collaboration (a global
goal)
9. 9
Common Mistakes: Utilization
▪ Utilization is only good up to a point
▪ Higher utilization is better?
− High utilization doesn’t allow slack for unplanned work
− Queue theory: as utilization approaches 100%, lead times approach
infinity
− Once you hit higher and higher levels of utilization (a poor goal of
productivity), teams will take longer and longer to get work done
11. 11
Measuring Software Delivery Performance
▪ Focus on both Outcomes and Global measures:
- Deploy frequency (when business demands)
- Lead Time for Changes
- Mean Time to Recover (MTTR)
- Change Fail Rate
13. 13
High Performing DevOps Teams
More agile
More frequent
Code deployments
46x
That’s the difference between multiple
times per day and once a week or less.
Faster lead time from commit to
deploy
440x
That’s the difference between less than
an hour and more than a week.
14. 14
High Performing DevOps Teams
More reliable
Faster mean time to
recover from downtime
96x
That means high performers recover in
less than an hour instead of several days
As likely that changes will
fail
1/5x
That means high performers changes fail 0-15% of
the time, compared to 31-45% of the time.
23. 23
DevOps is Good for Technology
▪ Measuring DevOps and Software delivery performance
- Deploy frequency (when business demands)
- Lead Time for Changes
- Mean Time to Recover (MTTR)
- Change Fail Rate
26. 26
High Performing technology organizations are twice
as likely to achieve or exceed
Commercial Goals
• Productivity
• Profitability
• Market Share
• # of customers
2x
27. 27
High Performing technology organizations are twice
as likely to achieve or exceed
Commercial Goals
• Productivity
• Profitability
• Market Share
• # of customers
Non-commercial Goals
• Quantity of products or services
• Operating efficiency
• Customer satisfaction
• Quality of products or services
• Achieving organizational or mission goals
2x
28. 28
High Performing technology organizations are
twice as likely to achieve or exceed
Commercial Goals
• Productivity
• Profitability
• Market Share
• # of customers
Non-commercial Goals
• Quantity of products or services
• Operating efficiency
• Customer satisfaction
• Quality of products or services
• Achieving organizational or mission goals
50%
Higher market cap
growth over 3 years*
2x
30. 30
Using an outcomes-based, capability-
focused approach, you can focus your
efforts on the right capabilities to improve
software delivery performance.
31. 31
We know there are key capabilities that drive
Software Delivery Performance
▪ They fall into four categories:
− Technology and automation
− Process
− Measurement/monitoring
− Culture
32. 32
Technology and Automation
▪ Version control
▪ Deployment
automation
▪ Continuous
integration
▪ Trunk-based
development
▪ Test automation
▪ Test data
management
▪ Shift left on security
▪ Continuous delivery
▪ Loosely-coupled
architecture
▪ Architect for
empowered teams
33. 33
Process
▪ Gather and implement customer feedback
▪ Work in small batches
▪ Lightweight change approval process
▪ Team experimentation
34. 34
Measurement and Monitoring
▪ Visual management
▪ Monitoring for business decisions
▪ Check system health proactively
▪ WIP limits
▪ Visualizations
35. 35
Culture
▪ Westrum organizational culture
▪ Climate for learning
▪ Collaboration among teams
▪ Make work meaningful
▪ Transformational leadership
36. 36
Where should I start?
▪ “It depends.” Everyone is different
▪ Patterns I see often:
− Architecture is highest contributor to Continuous Delivery (SODR 2017)
and shows up for very many teams (DORA: as the need for loosely-
coupled architecture or trunk-based development)
− Lightweight change approval process is a constraint for most teams
(DORA)
− Continuous integration (DORA – and its full complement)
37. 37
So What to Do?
1. Identify your constraints. Pick “a few.”
2. Work to eliminate those constraints.
3. Re-evaluate your environment and system.
4. Rinse and repeat.
40. 40
Your Role in this
▪ Be the transformational leaders. Own this.
▪ Start by measuring a few things
− Focus on outcomes: Software delivery performance (speed & stability)
− Drive performance improvements through capability improvements–
both with tech and with not tech
▪ Share your stories! Leverage community
41. 41
For More Information:
For our ROI whitepaper, case
studies, the State of DevOps
Reports & peer-reviewed
research, visit
devops-research.com
46. 46
DevOps is
Technical practices
seen in Continuous Delivery,
Management practices
seen in Lean and Agile principles, and
Organizational Culture and Leadership
Research shows that these drive Organizational Performance
and Technology Performance
@nicolefv
49. 49
Remember it’s “Tech Plus”
▪ CAMS -- the original definition, coined by Damon Edwards
and John Willis at DOD Mountain View 2010
▪ 2014 – 2017 State of DevOps Reports
− Technology + Lean Management + Culture
▪ Bessen (2017): “Real Reason Superstar Firms are Pulling
Ahead”
− Because of IT combined with other things, like management,
brand, or IP
− Strategic use of technology explains revenue and productivity gains
more than M&A and entrepreneurship
51. 51
Technology / technology stack doesn’t matter
▪ Low performers are more likely to:
− be working on software developed by an outsourcing partner
− be working on mainframe system
BUT
▪ Working on a mainframe system was not statistically correlated
with performance.
▪ Working on greenfield or brownfield (or any other system)
wasn’t correlated with performance, either.
52. 52
Architectural outcomes: can my team…
▪ …make large scale changes to the design of its system without the
permission of someone outside the team, or depending on other teams?
▪ ...complete its work without fine-grained communication and
coordination with people outside the team?
▪ ...deploy and release its product or service on demand, independently of
other services the product or service depends upon?
▪ ...do most of its testing on demand, without requiring an integrated test
environment?
▪ ...perform deployments during normal business hours with negligible
downtime?
53. 53
Conway’s Law
“organizations which design systems …
are constrained to produce designs which
are copies of the communication
structures of these organizations”
56. 56
Leadership Necessary but Not Sufficient
• Teams with the least transformational leaders (the bottom third)
were one-half as likely to be high IT performers
• Leaders cannot do it alone! Teams with the top 10% of
transformational leaders performed no better than the median
58. 58
Good practices and smart investment make the
work better, too!
▪ The work?
− Less deployment pain
− Less burnout
− Higher employee Net Promoter Score
59. 59
Employees in high performing
organizations are 2.2 times
more likely to recommend
their organization as a
great place to work
Notes de l'éditeur
Historically: most measures of performance have focused on productivity
Historically: most measures of performance have focused on productivity
Historically: most measures of performance have focused on productivity
Historically: most measures of performance have focused on productivity
At one point, World of Warcraft came out and Level 60 was amazing and as high as you could go. A Level 60 mage or warlock was awesome and it was freaking HARD to get there. So this was a serious GOAL. So if you use a maturity model…. 40 is moderate, 50 is super high, 60 is epic. And once you achieve this, you are amazing and you are set.
(go)
OH WAIT. Except the world changed. The competition changed…. And now you can get to level 110. Suddenly being a Level 60 just isn't’ very cool anymore, and frankly, it isn’t good enough. Technology is the same way. And so is business. The best, most innovative companies know this. Their competition is getting better, the world is changing, and customers demand more. So instead of picking a destination, like in a maturity model, they point in a a direction,
Aka the DevOps
Aka the DevOps
Complex systems mean you will elevate a constraint and a new one will appear.
Continuous improvement paradigm