Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Measuring Performance: See the Science of DevOps Measurement in Action

What is the best way to measure DevOps performance? And, how can it be done in a scientific way? In this webinar, Dr. Nicole Forsgren will present the frameworks and methodologies uniquely suited to evaluating the way we build and scale software applications. She’ll highlight lessons learned through a four-year research project presented in her upcoming book, Accelerate, written along with Jez Humble and Gene Kim.

  • Identifiez-vous pour voir les commentaires

Measuring Performance: See the Science of DevOps Measurement in Action

  1. 1. Measuring Performance: See the Science of DevOps Measurement in Action February 27, 2018 Nicole Forsgren, PhD
  2. 2. 2 Housekeeping ▪ This webinar is being recorded ▪ Links to the slides and the recording will be made available after the presentation ▪ You can post questions via the GoToWebinar Control Panel
  3. 3. 3 Your Hosts Dr. Nicole Forsgren @nicolefv Tim Buntel @tbuntel
  4. 4. 4 The Highlights ▪ Measuring Performance − Common Mistakes ▪ Our Approach: Software Delivery Performance − Where are you? ▪ Maturity Models ▪ Does Software Delivery Performance Matter? ▪ How to Improve Performance ▪ You can help
  5. 5. Common Mistakes
  6. 6. 6 Common Mistakes ▪ Outputs vs. Outcomes ▪ Individual/local vs. Team/global ▪ Some common examples: − Lines of code − Velocity − Utilization
  7. 7. 7 Common Mistakes: Lines of Code ▪ More is better? − Bloated software − Higher maintenance costs − Higher cost of change ▪ Less is better? − Cryptic code that no one can read ▪ Ideal: solve business problems with most efficient code
  8. 8. 8 Common Mistakes: Velocity ▪ Agile: problems are broken down into stories, which are assigned “points” of estimated effort to complete ▪ At end of sprint, total points signed off by customer is recorded = velocity ▪ Velocity is a capacity planning tool. NOT a productivity tool. ▪ Why doesn’t this work for productivity? − Velocity is a relative measure, not absolute. So: bad for comparing teams − Gaming by inflating estimates − Focus on team completion at the expense of collaboration (a global goal)
  9. 9. 9 Common Mistakes: Utilization ▪ Utilization is only good up to a point ▪ Higher utilization is better? − High utilization doesn’t allow slack for unplanned work − Queue theory: as utilization approaches 100%, lead times approach infinity − Once you hit higher and higher levels of utilization (a poor goal of productivity), teams will take longer and longer to get work done
  10. 10. Our Approach: Software Delivery Performance
  11. 11. 11 Measuring Software Delivery Performance ▪ Focus on both Outcomes and Global measures: - Deploy frequency (when business demands) - Lead Time for Changes - Mean Time to Recover (MTTR) - Change Fail Rate
  12. 12. 12 We see More throughput More stability In tandem. Without the tradeoffs that some suggest are necessary.
  13. 13. 13 High Performing DevOps Teams More agile More frequent Code deployments 46x That’s the difference between multiple times per day and once a week or less. Faster lead time from commit to deploy 440x That’s the difference between less than an hour and more than a week.
  14. 14. 14 High Performing DevOps Teams More reliable Faster mean time to recover from downtime 96x That means high performers recover in less than an hour instead of several days As likely that changes will fail 1/5x That means high performers changes fail 0-15% of the time, compared to 31-45% of the time.
  15. 15. 15
  16. 16. Maturity Models
  17. 17. Maturity Models Don’t Work
  18. 18. 18 Maturity models are for CHUMPS
  19. 19. 19 Maturity models are for CHUMPS
  20. 20. 20 Industry changes: Year over year What is good enough in one year is out of date the next Speed Stability
  21. 21. Does Software Delivery Performance Matter?
  22. 22. 22 “IT doesn’t matter.” -- Nicholas Carr, 2003
  23. 23. 23 DevOps is Good for Technology ▪ Measuring DevOps and Software delivery performance - Deploy frequency (when business demands) - Lead Time for Changes - Mean Time to Recover (MTTR) - Change Fail Rate
  24. 24. 24 Software Delivery Performance is comprised of throughput and stability, and both are possible without tradeoffs
  25. 25. 25 DevOps is good for organizations
  26. 26. 26 High Performing technology organizations are twice as likely to achieve or exceed Commercial Goals • Productivity • Profitability • Market Share • # of customers 2x
  27. 27. 27 High Performing technology organizations are twice as likely to achieve or exceed Commercial Goals • Productivity • Profitability • Market Share • # of customers Non-commercial Goals • Quantity of products or services • Operating efficiency • Customer satisfaction • Quality of products or services • Achieving organizational or mission goals 2x
  28. 28. 28 High Performing technology organizations are twice as likely to achieve or exceed Commercial Goals • Productivity • Profitability • Market Share • # of customers Non-commercial Goals • Quantity of products or services • Operating efficiency • Customer satisfaction • Quality of products or services • Achieving organizational or mission goals 50% Higher market cap growth over 3 years* 2x
  29. 29. How to Improve: You can accelerate your journey
  30. 30. 30 Using an outcomes-based, capability- focused approach, you can focus your efforts on the right capabilities to improve software delivery performance.
  31. 31. 31 We know there are key capabilities that drive Software Delivery Performance ▪ They fall into four categories: − Technology and automation − Process − Measurement/monitoring − Culture
  32. 32. 32 Technology and Automation ▪ Version control ▪ Deployment automation ▪ Continuous integration ▪ Trunk-based development ▪ Test automation ▪ Test data management ▪ Shift left on security ▪ Continuous delivery ▪ Loosely-coupled architecture ▪ Architect for empowered teams
  33. 33. 33 Process ▪ Gather and implement customer feedback ▪ Work in small batches ▪ Lightweight change approval process ▪ Team experimentation
  34. 34. 34 Measurement and Monitoring ▪ Visual management ▪ Monitoring for business decisions ▪ Check system health proactively ▪ WIP limits ▪ Visualizations
  35. 35. 35 Culture ▪ Westrum organizational culture ▪ Climate for learning ▪ Collaboration among teams ▪ Make work meaningful ▪ Transformational leadership
  36. 36. 36 Where should I start? ▪ “It depends.” Everyone is different ▪ Patterns I see often: − Architecture is highest contributor to Continuous Delivery (SODR 2017) and shows up for very many teams (DORA: as the need for loosely- coupled architecture or trunk-based development) − Lightweight change approval process is a constraint for most teams (DORA) − Continuous integration (DORA – and its full complement)
  37. 37. 37 So What to Do? 1. Identify your constraints. Pick “a few.” 2. Work to eliminate those constraints. 3. Re-evaluate your environment and system. 4. Rinse and repeat.
  38. 38. 38 Additional Resources
  39. 39. You Can Help!
  40. 40. 40 Your Role in this ▪ Be the transformational leaders. Own this. ▪ Start by measuring a few things − Focus on outcomes: Software delivery performance (speed & stability) − Drive performance improvements through capability improvements– both with tech and with not tech ▪ Share your stories! Leverage community
  41. 41. 41 For More Information: For our ROI whitepaper, case studies, the State of DevOps Reports & peer-reviewed research, visit devops-research.com
  42. 42. Questions?
  43. 43. Thank You
  44. 44. EXTRA SLIDES HERE
  45. 45. 45 Transformations need Technology AND Process AND Culture
  46. 46. 46 DevOps is Technical practices seen in Continuous Delivery, Management practices seen in Lean and Agile principles, and Organizational Culture and Leadership Research shows that these drive Organizational Performance and Technology Performance @nicolefv
  47. 47. 47 “IT doesn’t matter.” -- Nicholas Carr, 2003
  48. 48. 48
  49. 49. 49 Remember it’s “Tech Plus” ▪ CAMS -- the original definition, coined by Damon Edwards and John Willis at DOD Mountain View 2010 ▪ 2014 – 2017 State of DevOps Reports − Technology + Lean Management + Culture ▪ Bessen (2017): “Real Reason Superstar Firms are Pulling Ahead” − Because of IT combined with other things, like management, brand, or IP − Strategic use of technology explains revenue and productivity gains more than M&A and entrepreneurship
  50. 50. 50 Architecture matters… technology doesn’t
  51. 51. 51 Technology / technology stack doesn’t matter ▪ Low performers are more likely to: − be working on software developed by an outsourcing partner − be working on mainframe system BUT ▪ Working on a mainframe system was not statistically correlated with performance. ▪ Working on greenfield or brownfield (or any other system) wasn’t correlated with performance, either.
  52. 52. 52 Architectural outcomes: can my team… ▪ …make large scale changes to the design of its system without the permission of someone outside the team, or depending on other teams? ▪ ...complete its work without fine-grained communication and coordination with people outside the team? ▪ ...deploy and release its product or service on demand, independently of other services the product or service depends upon? ▪ ...do most of its testing on demand, without requiring an integrated test environment? ▪ ...perform deployments during normal business hours with negligible downtime?
  53. 53. 53 Conway’s Law “organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations”
  54. 54. Leadership Matters
  55. 55. 55 @nicolefv
  56. 56. 56 Leadership Necessary but Not Sufficient • Teams with the least transformational leaders (the bottom third) were one-half as likely to be high IT performers • Leaders cannot do it alone! Teams with the top 10% of transformational leaders performed no better than the median
  57. 57. 57 Relationship between transformational leadership and performance
  58. 58. 58 Good practices and smart investment make the work better, too! ▪ The work? − Less deployment pain − Less burnout − Higher employee Net Promoter Score
  59. 59. 59 Employees in high performing organizations are 2.2 times more likely to recommend their organization as a great place to work

×