As the amount of metrics, software that produce and process them, and people involved in them continue to increase, we need better ways to organize them, to make them self-describing, and do so in a way that is consistent. Leveraging this, we can then automatically build graphs and dashboards, given a query that represents an information need, even for complicated cases. We can build richer visualizations, alerting and fault detection. This talk will introduce the concepts and related tools, demonstrate possibilities using the Graph-Explorer interface, and lay the groundwork for future work.

1. rethinking metrics:
metrics 2.0

5. instagram.com/wrongrob

6. vimeo.com/43800150

10. problems
Metrics 2.0 concepts
implementations
uses & ideas

11. Mostly
graphite

12. terminology
sync

13. (1234567890, 82)
(1234567900, 123)
(1234567910, 109)
(1234567920, 77)
db15.mysql.queries_running
host=db15 mysql.queries_running

14. Problems

15. Vimeo.com
pagerequests/s?
server X disk write?

16. stats.hits.vimeo_com
stats_counts.hits.vimeo_com
stats.*.vimeo_requests
collectd.db.disk.sda1.disk_time.write

17. Understanding metrics
Terminology? Meaning?
Prefix?
Unit?
Aggregation?
Source?

19. Unclear, inconsistent
terminology, format
tightly coupled
lack information

20. http://litlquest.com/forest-trees/see-forest-trees-2

21. O(S*P*A*C)
S = # Sources
P = # People
A = # Aggregators
C = #Complexity

23. Graphs and
dashboards are a
huge time sink.

24. metrics 2.0
concepts

25. Self-describing
Standardized
Orthogonal dimensions

26. stats.timers.dfs5.
proxy-server.object.GET.200.
timing.upper_90

27. {
server: dfvimeodfsproxy5,
http_method: GET,
http_code: 200,
unit: ms,
metric_type: gauge,
stat: upper_90,
swift_type: object
}

28. MB/s Err/d
Req/h ...
B Err Warn Conn
File Req …
SI + IEC

29. allow more characters
unit: Req/s,
site: vimeo.com,
...

30. Metadata
meta: {
src: proxy.py:458,
from: diamond
}

31. metrics20.org

32. Immediate understanding
of metrics
Minimize time to graphs,
alerting, troubleshooting
compatibility & flexibility
in tooling

33. Implementations
getting the data

35. Source formats
…
service=foo instance=host unit=B 123
1234567890
{s}foo.{i}host.{u}B 123 1234567890
<uuid> 125 1234567890 #seperate data
…

36. Carbon-tagger
…
stats.gauges.host.foo 125 1234567890
service=foo instance=host
target_type=gauge unit=B 123 1234567890
…

38. Statsdaemon
unit=B
unit=B
...
unit=ms
unit=ms
...
unit=B/s
unit=ms stat=mean
unit=ms stat=upper_90
...

39. Keep metric
tags in sync
with data

40. Implementations
Graphing & dashboarding
Visualization
Alerting

41. Graphing &
Dashboarding

42. Graph
Explorer

44. Graph-Explorer queries 101
proxy-server swift
server:regex unit=ms
(AND)

52. upper_90 (or stat=upper_90)
from <datetime>
to <datetime>
avg over <timespec>
(5M, 1h, 3d, ...)

54. Compare object put/get
stack …
http_method:(PUT|GET)
swift_type=object
avg by http_code,server

56. Comparing servers
http_method:(PUT|GET)
group by unit,target_type
avg by http_code,
swift_type,http_method

58. transcode unit=Job/s
avg over <time>
from <datetime> to <datetime>

59. Note: data is obfuscated

60. Bucketing
sum by zone:eu-west|us-east|
ap-southeast|us-west|
sa-east|vimeo-df|vimeo-lv
group by state

61. Note: data is obfuscated

62. Compare job states per region
group by zone

63. Note: data is obfuscated

64. Unit conversion
unit=Mb/s network
server:regex
sum by server

67. Integration
Metric unit=B/s
Query unit=TB

69. Deriving
Metric unit=B
Query unit=GB/d

71. Highly extensible
Equal rights for
all tags
→ real world use
drives spec

72. Err/s Anything/s
MAnything/d
B Err Conn File Req
Anything
SI + IEC

76. Minimize time from
information need to
insights.

77. Future
Work

78. Faced-based suggestions
Custom hierachies

79. Tag insights

80. ●Storage aggregation rules
● graphite API functions such as
cumulative, summarize and
smartSummarize
●consolidateBy & Graph
renderers

83. stat=upper/lower/mean/...
(assume avg otherwise)

84. Visualizations

85. From: dygraphs.com

86. bin=10
bin=20
bin=30
bin=40
bin=50
bin=100

88. Alerting

90. unit=Err/s

95. Classifying
clusters of
cause & effect

96. Different algos
for different
metric categories

97. Alert criticality
& routing based
on tags

98. integrating
logs & metrics

101. Algorithms
leverage both
logs and metrics

102. Conclusion
structured
self-describing
standardized
metrics = enabler

103. Conclusion
Concerns? Ideas? Advice?
Ready for early adopters!
Work with me on next-gen
telemetry!

104. Seen in this presentation:
metrics20.org
vimeo.github.io/graph-explorer
github.com/vimeo/timeserieswidget
github.com/vimeo/carbon-tagger
github.com/vimeo/statsdaemon
github.com/graphite-ng/carbon-relay-ng
github.com/Dieterbe/anthracite

105. You might also like:
github.com/vimeo/graphite-influxdb
github.com/vimeo/graphite-api-influxdb-docker
Github.com/vimeo/whisper-to-influxdb
github.com/Dieterbe/influx-cli
github.com/graphite-ng/graphite-ng
Github.com/vimeo/smoketcp
Github.com/vimeo/tailgate

106. Stay in touch!
Metrics20 google group
it-telemetry google group
twitter.com/Dieter_be
dieter.plaetinck.be
dieter@plaetinck.be
dieter@vimeo.com
Lisa labs office hours after lunch
Q & A

107. Bonus round

108. Dashboard definition
queries = [
'cpu usage sum by core',
'mem unit=B !total group by type:swap',
'stack network unit=Mb/s',
'unit=B (free|used) group by =mountpoint'
]

110. Catchall plugins
stats.dfvimeocliapp2.twitter.error
{
“n1”: “dfvimeocliapp2”,
“n2”: “twitter”,
“n3”: “error”,
“plugin”: “catchall_statsd”,
“source”: “statsd”,
“target_type”: “rate”,
“unit”: “unknown/s”
}

111. Equivalence
servers.host.cpu.total.iowait → “core” : “_sum_”
servers.host.cpu.<core-number>.iowait
servers.host.loadavg.15