By Baruch Sadogursky
Devops is usually viewed from a traditional perspective of a collaboration of Dev, Ops and QA, driven by the change in Culture, People and Process. But how do you know where you stand and were to move? As in almost any field, data and metrics give you the gauges and instruments. In this talk we’ll talk about the key measurements for the DevOps transformation process and provide you with 3 metrics you can start measuring tomorrow.
5. Poll time!
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
How many of you are software engineers?
How many of you are optimists?
How many of you are self-confident in their work?
6. Dunning-Kruger Effect a.k.a. ”optimism”
People suffer from illusory superiority,
mistakenly assessing their cognitive ability as
greater than it is.
Wikipedia
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
“
7. Second-system effect a.k.a. “self-confidence”
The tendency of small, elegant, and
successful systems, to be succeeded by over-
engineered, bloated systems, due to inflated
expectations and overconfidence.
The Mythical Man-Month
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
“
21. Velocity to agile is like ? To devops…
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
?
22. Velocity to agile is like NPS To devops…
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
NPS
23. Also, it’s a lot like profits
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Metric Easily
understandable
Unity actionable
Profit
✓ ✓ ✗
Velocity
✓ ✓ ✗
NPS
✓ ✓ ✗
24. Also, it’s a lot like profits
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Metric Easily
understandable
Unity actionable
Profit
✓ ✓ ✗
Velocity
✓ ✓ ✗
NPS
✓ ✓ ✗
25. Also, it’s a lot like profits
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Metric Easily
understandable
Unity actionable
Profit
✓ ✓ ✗
Velocity
✓ ✓ ✗
NPS
✓ ✓ ✗
26. Poll time!
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Who knows what velocity is?
Who knows what burndown chart is?
Who has a burndown chart?
Who looks at the burndown chart?
Who trusts the burndown chart?
Who knows what to do if it doesn't look right?
31. This is devops*
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Ops
QADev
Common goals,
tools, culture
Deep
specialization
*Unless you’re netfliX
**You’re not
35. How do metrics collaborate?
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected
36. How do metrics collaborate?
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected
37. How do metrics collaborate?
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected
38. How do metrics collaborate?
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected
39. How do metrics collaborate?
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected
40. How do metrics collaborate?
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected
41. How do metrics collaborate?
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected
52. How effective are we?
Samples are good enough for that
As long as they are representative
Need to be collected over time
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
55. Development affects operational costs
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Central container images repository
Developers keep pushing images
Storage prices skyrocket
Not all PoPs need all images
57. Development affects QA
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Anecdotes about test suite stability
shared at daily startup
Special issue type “test suite
stability”
59. Development influences Operations trust
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Operations want to predict costs and
be trustworthy
Will application updates generate
more load?
72. Step into data-driven life
@jbaruch www.jfrog.com/shownotes #codemash #datadrivendevops
Removes blame game
Builds accountability and trust
Creates common base for discussion