Grier Johnson worked as a platform engineer and site reliability engineer at LinkedIn where he helped build out their metrics collection and alerting systems. Some key lessons he learned include: (1) don't reinvent the wheel and leverage existing open source solutions for metrics stores and alerting, (2) plan for redundancy and distribute stores close to data sources, and (3) make metrics discovery and customizing alerts and graphs easy for users.

1. Metrics and Monitoring
Infrastructure
Lessons Learned Building Metrics at LinkedIn

2. “If you can’t measure it,
you can’t manage it.”
- W. Edwards Deming

3. “It is wrong to suppose that if
you can’t measure it, you can’t
manage it – a costly myth.”
- W. Edwards Deming

4. Who Am I?

5. Grier Johnson
Platform Engineer

6. Grier Johnson
Production Engineer

7. Grier Johnson
Site Reliability Engineer

8. Grier Johnson
Service Engineer

9. Grier Johnson
Production Engineer

10. Grier Johnson
All these titles mean about the same thing

11. Built out some metrics
collection at LinkedIn

13. Architected the
alerting system

15. So what makes a
“Metrics Infrastructure”

16. The Metrics Store

17. Metric stores have
high writes, low read

18. The read requirements for
historical data are even
lower

19. Metrics Transport

20. You probably just want
to use Kafka

21. Now your metrics are
pub/sub

22. Add something like
Hadoop for data analysis
maybe?

23. Metrics Emission

24. Standardize…
Your metrics

25. Standardize…
Your intervals

26. Standardize…
Your protocols

27. Standardize…
Your namespace

28. Metrics Types

29. Allow for custom
metrics

30. Prefer counters

31. Histograms

32. Protect your
namespace, seriously

33. Metrics Presentation

34. Allow for
customization
Graph Size

35. Allow for
customization
Start and end times

36. Allow for
customization
Graph type

37. Allow for
customization
Legends

38. Allow for
customization
Color palette

39. Static Graphs and
Links

40. Speed Matters

41. Update frequency
matters

42. Give a lot of thought to the
colors. No really. People
really care about this.

45. Ok, now how about
monitoring and alerting…

46. Data Sources

47. Metrics

48. Centralized Checks

49. Decentralized /
Distributed Checks

50. Stream Processing

51. Defining Your Alerts

52. What data are we
looking at

53. What does it mean for the
data to be good or bad

54. What do we do if it’s
bad?

55. Processing

56. Bringing together data
and definition

57. Read your data
sources

58. Read your definitions

59. Apply the definitions to
the data

60. Do something! Or
don’t.

61. Alerting Actions

62. Nothing, nothing at all.

63. E-mail

64. Alert!

65. Run a script

66. … wait what?

67. Use your monitoring
system to respond.

68. Automation is better
than alerting

69. UI/UX

70. At-a-glance visual for
site/service health

71. Point-and-click alert
suppression

72. CLI and API for
automation tasks

73. Bulk actions, please

74. Stopping Alerts

75. “Suppression”
or acknowledge

76. “Suppression”
or sleep

77. “Suppression”
or quiet

78. “Suppression”
or silence

79. Allow suppression
scheduling

80. Cancelling
suppressions

83. So what did I learn…

84. Lesson 1
Writing your Metrics Infrastructure from scratch

85. Don’t Invent It Here

86. Use Open Source

87. Metrics Stores
• Netflix’s Atlas (https://github.com/Netflix/atlas)
• Prometheus (https://prometheus.io)
• RackSpace’s Blueflood (http://blueflood.io)
• OpenTSDB (http://opentsdb.net)

88. Alerting
• Sensu (https://sensuapp.org)
• Riemann (http://riemann.io)
• Zabbix (http://www.zabbix.com)
• Nagios (http://actually-dont.com)

89. Hybridize

90. Contribute

91. Lesson 2
When you build your metrics system from scratch
anyhow…

92. Redundancy doesn’t
matter, until your first
outage with data loss

93. Care about thin
bandwidth pipes

94. Distribute your stores
close to the metrics
creators

95. Aggregate distributed
metrics at the
presentation layer

96. Cache what you can

97. Use Kafka

98. Lesson 3
Graphing UIs

99. DPI Matters

100. When your graph is 500px
wide, how many of the 750
data points can you show?

101. Show the highest data
points?

102. The lowest?

103. The average?

104. They’re all trade-offs and
someone will hate you for
it.

105. Javascript is slow

106. It looks good though

107. Hard to e-mail dynamic
javascript graphs

108. Remember to plan for
caching

109. Outages test your
frontend’s performance

110. Lesson 4
Metrics Discovery

111. Why have 100M metrics
if you can’t find them

112. Even when you find them,
you can’t makes sense of
1000 metrics for one service

113. Standard names help.
Namespace helps.
Not having 100M metrics helps

114. Lesson 5
Alerting On Metrics

115. Alerting on absolute
values is bad

116. Use standard
deviation

117. Use rate of change

118. But remember DST and
Holidays ruin everything

119. Lesson 6
Alerting Overload

120. Alerts are for humans

121. Low friction for alert
suppression

122. Low friction for alert
changes and
customization

123. Lesson 7
Alerting Levels

124. There is no “Warning”
level for alerts

125. If it is worth alerting
you, it’s critical

126. Rate of change
monitoring will help with
most useful cases here

127. Lesson 8
Suppression Times

128. Unlimited suppression
time = Regret

129. Less than unlimited is
OK

130. Low friction alert
modifications

131. Lesson 9
Processing your metrics from streams for alerts

132. Don’t re-read your kafka
stream to build up metrics
history.

133. It’s right there in your
metrics store

134. Lesson 10
Alerting on Logs

135. Don’t

136. No really, logs are for
humans, rethink your
monitoring strategy

137. Lesson 11
Alerting on Exceptions

138. Maybe

139. When exceptions are
rare, alerting is fine

140. There’s a special place in
hell for people that alert
on rates of exceptions

141. I’m sure there’s more
But if I’m not over time by this point I talked REALLY FAST

142. Questions?
• Grier Johnson
• grierj@gmail.com
• @grierj on Twitter for DMs
• https://www.linkedin.com/in/grierjohnson