Devops is usually viewed from a traditional perspective of a collaboration of Dev, Ops, and QA, driven by the change in Culture, People, and Process. But how do you know where you stand and where to move? As in almost any field, data and metrics give you the gauges and instruments. In this talk, we’ll talk about the key measurements for the DevOps transformation process and provide you with 3 metrics you can start measuring tomorrow.

1. #DataDrivenDevOps
How to cook a precise
software in a modern world
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

2. About me
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
@jbaruch

3. shownotes
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
jfrog.com/shownotes
Slides
Video (tomorrow)
Links
Ratings, comments
Raffle

4. Cobbler's
children have
no shoes
@jbaruch #QConLondon #datadrivendevops
jfrog.com/shownotes

5. Poll time!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
How many of you are software engineers?
How many of you are optimists?
How many of you are self-confident in their work?

6. Dunning-Kruger Effect a.k.a. ”optimism”
People suffer from illusory superiority,
mistakenly assessing their cognitive ability as
greater than it is.
Wikipedia
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
“

7. Second-system effect a.k.a. “self-confidence”
The tendency of small, elegant, and
successful systems, to be succeeded by over-
engineered, bloated systems, due to inflated
expectations and overconfidence.
The Mythical Man-Month
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
“

8. The consequences
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

9. disillusion
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Software I like
Software I
know really
well

10. overconfidence
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Every software is 80% ready 80% of the time
Every developer, ever
“

11. Hard to give up
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

12. @jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

13. @jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

14. How departments measure
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Sales
Marketing
Support
HR
Finance
Anecdotes
from daily
standup
Engineering

15. How departments measure
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Sales
Marketing
Support
HR
Finance
Anecdotes
from daily
standup
Engineering

16. @jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

17. @jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

18. @jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

19. Engineering metrics in pre-devops era
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Dev
• Velocity
• Architecture
• Compliance
QA
• Incidents
• Defects
• MTTR
• Code coverage
• Test suite
stability
Ops
• SLA
• Cost env setup
• Average
customer cost
• Tools stability

20. DEVOPS!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
?

21. Velocity to agile is like ? To devops…
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
?

22. Velocity to agile is like profit To devops…
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Profit

23. Very much alike!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Metric Easily
understandable
Unity actionable
Profit
✓ ✓ ✗
Velocity
✓ ✓ ✗

24. Very much alike!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Metric Easily
understandable
Unity actionable
Profit
✓ ✓ ✗
Velocity
✓ ✓ ✗

25. Very much alike!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Metric Easily
understandable
Unity actionable
Profit
✓ ✓ ✗
Velocity
✓ ✓ ✗

26. Poll time!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Who knows what velocity is?
Who knows what burndown chart is?
Who has a burndown chart?
Who looks at the burndown chart?
Who trusts the burndown chart?
Who knows what to do if it doesn't look right?

27. What’s the problem?
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Velocity

28. True for both!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Metric Easily
understandable
Unity actionable
Profit
✓ ✓ ✗
Velocity
✓ ✓ ✗

29. Too coarse!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Profit

30. Too coarse!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
DevOps

31. This is devops*
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Ops
QADev
Common goals,
tools, culture
Deep
specialization
*Unless you’re netfliX
**You’re not

32. This is devops
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Ops
QADev

33. Same for metrics!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Ops
QADev
profitSpecialized
metrics

34. Same for metrics!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Ops
QADev

35. How do metrics collaborate?
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected

36. How do metrics collaborate?
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected

37. How do metrics collaborate?
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected

38. How do metrics collaborate?
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected

39. How do metrics collaborate?
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected

40. How do metrics collaborate?
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected

41. How do metrics collaborate?
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
dev ops qa
dev • Time to build
• Release pipelines
stability
• Artifact replication
topology
• False reopen count
• QA coverage
ops • Avg cost of customer • Smoke tests quality
qa • Test suite stability
• Incoming defect
velocity
• Test execution time
• Time to create new
QA lab
influenced
affected

42. Cooking
with
metrics
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

43. Metrics
categorization
@jbaruch #QConLondon #datadrivendevops
jfrog.com/shownotes

44. Why?
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

45. ROI – is this activity is the right thing to do?
@jbaruch #QConLondon #datadrivendevops
jfrog.com/shownotes

46. Continuous improvement
@jbaruch #QConLondon #datadrivendevops
jfrog.com/shownotes

47. trust
@jbaruch #QConLondon #datadrivendevops
jfrog.com/shownotes

48. sampling
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

49. Complete sample
@jbaruch #QConLondon #datadrivendevops
jfrog.com/shownotes

50. @jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

51. Representative sample
@jbaruch #QConLondon #datadrivendevops
jfrog.com/shownotes

52. How effective are we?
Samples are good enough for that
As long as they are representative
Need to be collected over time
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

53. Predicting the future
@jbaruch #QConLondon #datadrivendevops
jfrog.com/shownotes

54. Meet scrum team “avengers”
@jbaruch #QConLondon #datadrivendevops
jfrog.com/shownotes

55. anecdotes from the daily scrum stand-ups
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
We never get enough testing environments
from Ops, QA is suffering.
Scrum team “Avengers”
“

56. Let’s gather some data!
0
2
4
6
8
10
week 1 week 2 week 3 week 4 week 5 week 6 week 7 week 8 week 9 week 10 week 11 week 12 week 13 week 14
Environment Utilization
Utilized Available
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

57. That doesn’t look good! But why?
0
5
10
15
20
25
Raj Jane Christina Helen Lee Baruch John Scott Nataly Ruppert Josh Patrick Michael
1
2
3
6
7
10
13 13
14
15
16
18
25
Days the environment stays checked out
Days checked out
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

58. Continuous
improvement
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
0
2
4
6
8
10
12
14
16
week 1 week 3 week 5 week 7 week 9 week 11 week 13 week 15 week 17 week 19 week 21 week 23
Environment Utilization
Utilized Available

59. anecdotes from the daily scrum stand-ups
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Tests are failing, because the test suite is not
stable, QA makes our builds fail without
reason.
Scrum team “Avengers”
“

60. Let’s gather some data!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

61. Return on
investment
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

62. anecdotes from the daily scrum stand-ups
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Those developers want to deploy new GC, its
tons of work, without any profit.
Scrum team “Avengers”
“

63. Let’s gather some data!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

64. Trust and
predictability
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
0:00
0:40
1:20
2:00
2:40
3:20
4:00
4:40
5:20
6:00
6:40
7:20
8:00
8:40
9:20
10:00
10:40
11:20
12:00
12:40
13:20
14:00
14:40
15:20
16:00
16:40
17:20
18:00
18:40
19:20
20:00
20:40
21:20
22:00
22:40
23:20
GC time
GC time

65. Dos and Don’ts

66. Measuring the right thing
@jbaruch #QConLondon #datadrivendevops
jfrog.com/shownotes

67. LoC LOL, anyone?
@jbaruch #QConLondon #datadrivendevops
jfrog.com/shownotes

68. Avoid data overload
@jbaruch #QConLondon #datadrivendevops
jfrog.com/shownotes

69. Use reliable data
@jbaruch #QConLondon #datadrivendevops
jfrog.com/shownotes

70. Common vocabulary is important
@jbaruch #QConLondon #datadrivendevops
jfrog.com/shownotes
”medium” In russia”medium” In brazil

71. Metrics
generate
incentives, be
aware!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

72. Tweak as you go
@jbaruch #QConLondon #datadrivendevops
jfrog.com/shownotes

73. Elevate!
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Raw Data
Product
Specific
Dashboards
Integrated
Dashboards

74. @jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

75. @jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

76. @jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes

77. How departments measure
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Sales
Marketing
Support
HR
Finance
Anecdotes
from daily
standup
Engineering

78. Step into data-driven life
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
Removes blame game
Builds accountability and trust
Creates common base for discussion

79. Q&A and shownotes
@jbaruch #QConLondon #datadrivendevops jfrog.com/shownotes
@jbaruch
#QconLondon
#datadrivendevops
www.jfrog.com/shownotes