Creating a Scalable Monitoring System That Everyone Will Love (Velocity Conf)

@molly_struve
1
Creating A Scalable Monitoring
System That Everyone Will
❤
@ThePracticalDev | @molly_struve | dev.to/molly_struve

@molly_struve
Monitoring Mistakes
Overhauling the
System
The Payoff
2

@molly_struve
Monitoring Mistakes
3
Overhauling the
System
The Payoff

@molly_struve
Monitoring Mistakes
5
Overhauling the
System
The Payoff

@molly_struve
Monitoring Mistakes
Overhauling the
System
6
Overhauling the
System
The Payoff

@molly_struve
Monitoring Mistakes
Overhauling the
System
The Payoff
7

@molly_struve
Monitoring Mistakes
Overhauling the
System
The Payoff
8

@molly_struve
Monitoring Mistakes Overhauling the System The Payoff
9

@molly_struve
10

@molly_struve
11

@molly_struve
12

@molly_struve
13

@molly_struve
14

@molly_struve
15

@molly_struve
16

@molly_struve
17

@molly_struve
18

@molly_struve
19

@molly_struve
20

@molly_struve
21

@molly_struve
22

@molly_struve
23

@molly_struve
24

@molly_struve
25

@molly_struve
Incredibly Inconsistent
26

@molly_struve
Inconsistent Alerts
Required no actionReported data
27

@molly_struve
Inconsistent Alerts
Required no actionReported data
Immediate action required
28

@molly_struve
Manual Monitoring
29

@molly_struve
Manual Monitoring
30

@molly_struve
31

@molly_struve
32

@molly_struve
33

@molly_struve
34

@molly_struve
35

@molly_struve
make on-call devs miserable
36

@molly_struve
37

@molly_struve
38

@molly_struve
39

@molly_struve
Coverage doesn’t matter if you have
no idea what is going on!
40

@molly_struve
Monitoring Mistakes
Overhauling the
System
The Payoff
41

@molly_struve
Monitoring Mistakes
Overhauling the
System
The Payoff
42

@molly_struve
Consolidate Monitoring To a Single Place
Monitoring Must Haves
1
2
3
4
43
5

@molly_struve
"
# $
%
&
44

@molly_struve
45

@molly_struve
46

@molly_struve
47

@molly_struve
48

@molly_struve
49

@molly_struve
50

@molly_struve
51

@molly_struve
Make ALL Alerts Actionable
1
2
3
4
52
5

@molly_struve
53
Action

@molly_struve
54
Action

@molly_struve
Action Required No action Needed
55

@molly_struve
#ops_alerts
#dev_alerts
56

@molly_struve
#ops_alerts
#ops_reporting
#dev_alerts
#dev_reporting
57

@molly_struve
Make Sure Alerts Are Mutable
1
2
3
4
58
5

@molly_struve
1
2
3
4
59
5

@molly_struve
30 60 90
minutes
60

@molly_struve
61

@molly_struve
62
Miss new alerts

@molly_struve
63

@molly_struve
64

@molly_struve
65

@molly_struve
66

@molly_struve
67

@molly_struve
Track Alert History
1
2
3
4
68
5

@molly_struve
Tracking Alert History
69

@molly_struve
70

@molly_struve
71

@molly_struve
Track Alert History
1
2
3
4
72
5 Remove ALL Manual Monitoring

@molly_struve
73

@molly_struve
Manual Monitoring
🙅(
Scale
74

@molly_struve
75

@molly_struve
76
Automatic Alerts

@molly_struve
Track Alert History
1
2
3
4
77

@molly_struve
Monitoring Mistakes
Overhauling the
System
The Payoff
78

@molly_struve
Monitoring Mistakes
Overhauling the
System
The Payoff
79

@molly_struve
On boarding is a breeze
80

@molly_struve
3 On-boarding steps:
1
2
3
81

@molly_struve
Show them the monitoring setup1
2
3
82

@molly_struve
2
3
If an alert goes off you have to address it
83

@molly_struve
2
3
How to mute a triggered alert
84
If an alert goes off you have to address it

@molly_struve
On boarding is a breeze
85

@molly_struve
Happier on-call developers
86

@molly_struve
All alerts must be actionable
87

@molly_struve
88

@molly_struve
No more noise
89

@molly_struve
90

@molly_struve
Alerts must be mutable
91

@molly_struve
Alerts must be mutable
92

@molly_struve
Diagnosing alerts is faster and easier
93

@molly_struve
94

@molly_struve
95
High Priority Job Queue Backed Up

@molly_struve
96
Today?

@molly_struve
97
Today?
Going on
longer?

@molly_struve
98
Today?
Going on
longer?

@molly_struve
99
Today?
Going on
longer?Alert History

@molly_struve
100

@molly_struve
Developers began helping to
improve our monitoring system
101

@molly_struve
102

@molly_struve
All alerts must be actionable
103

@molly_struve
30-40 Alerts
104

@molly_struve
>90 Alerts
105

@molly_struve
106
Basic Alerts

@molly_struve
107
More Granular Alerts
Basic Alerts

@molly_struve
108
vs
High MySQL load High MySQL load
on db 1 for client 5

@molly_struve
Developers
❤
monitoring system
109

@molly_struve
Monitoring Must Have Benefits
1
2
3
110
4

@molly_struve
On boarding is a breeze.1
2
3
111
4

@molly_struve
On-call developers are a lot happier
1
2
3
On boarding is a breeze.
112
4

@molly_struve
1
2
3
113
4

@molly_struve
Developers helping to improve your
monitoring systems
1
2
3
114
4

@molly_struve
Track Alert History
1
2
3
4
115

@molly_struve
❤❤❤
116

@molly_struve
Questions?
117

Rate today ’s session
Session page on conference website O’Reilly Events App

Creating a Scalable Monitoring System That Everyone Will Love (Velocity Conf)

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

Creating a Scalable Monitoring System That Everyone Will Love (Velocity Conf)