Most metrics systems link timeseries to a string key, some add a few tags. They often lack information, use inconsistent formats and terminology, and are poorly organized. As the amount of people and software generating, processing, storing and visualizing metrics grows, this approach becomes very cumbersome and there is a lot to be gained from taking a step back and re-thinking metric identifiers and metadata. Metrics 2.0 is a set of conventions around metrics: With barely any extra work metrics become self-describing and standardized. Compatibility between tools increases dramatically, dashboards can automatically convert information needs into graphs, graph renderers can present data more usefully, anomaly detection and aggregators can work more autonomously and avoid common mistakes. Result: less micromanaging of software and configuration, quicker results, more clarity. Less frustration and room for errors. This talk will also cover the tools that turn this concept into production-ready reality: Graph-Explorer is an application that integrates with Graphite. Enter an expression that represents an information need and it generates the corresponding graphs or alerting rules, automatically applying unit conversion, aggregation, processing, etc. Statsdaemon is an aggregation daemon like Etsy's Statsd that expresses performed aggregations and statistical operations by updating the metrics tags, making sure that the metric metadata always corresponds to the data. Dieter Plaetinck is a systems-gone-backend engineer at Vimeo.

2.
Credit: user niteroi @ panoramio.com

3.
vimeo.com/43800150

11.
1 Metrics 2.0 concepts
2 Implementation
3 Advanced stuff

12.
“Dieter” ?

13.
Peter Deter→

14.
Terminology sync

15.
(1234567890, 82)
(1234567900, 123)
(1234567910, 109)
(1234567920, 77)
db15.mysql.queries_running
host=db15 mysql.queries_running

17.
How many pagerequests/s is vimeo.com
doing?

18.
● stats.hits.vimeo_com
● stats_counts.hits.vimeo_com

20.
stats.<host>.requesthostport.vimeo_
com_443

21.
stats.timers.dfs5.proxy­
server.object.GET.200.timing
.upper_90

22.
O(X*Y*Z)
X = # apps
Y = # people
Z = # aggregators

23.
How long does it take to retrieve an object from swift?

24.
stats.timers.<host>.proxy­
server.<swift_type>.<http_method>.
<http_code>.timing.<stat>
stats.timers.<host>.object­
server.<http_method>.
timing.<stat>
target=stats.timers.dfs*.object*GET*timing.mean ?
target=groupByNode(stats.timers.dfs*.proxy
­server.object.GET.*.timing.mean,2,"avg")
target=stats.timers.dfs*.object­
server.GET.timing.mean

25.
swift_type=object stat=mean timing GET avg by http_code

28.
O((DxV)^2)
D = # dimensions
V = # values per dim

29.
collectd.db.disk.sda1.dis
k_time.write

32.
What should I name my metric?

33.
10
100
1000
10000
100000
1000000

35.
Metrics 2.0

36.
Old:
● information lacking
● fields unclear & inconsistent
● cumbersome strings / trees
● forbidden characters
New:
● Self­describing
● Standardized
● all dimensions in orthogonal tag­space
● Allow some useful characters

37.
stats.timers.dfs5.proxy­server.object.GET.200.timing.upper_90
{
“server”: “dfvimeodfsproxy5”,
“http_method”: “GET”,
“http_code”: “200”,
“unit”: “ms”,
“target_type”: “gauge”,
“stat”: “upper_90”,
“swift_type”: “object”
“plugin”: “swift_proxy_server”
}

38.
Main advantages:
● Immediate understanding of metric meaning (ideally)
● Minimize time to graphs, dashboards, alerting rules

39.
github.com/vimeo/graph­explorer/wiki

40.
SI + IEC
B Err Warn Conn Job File Req ...
MB/s Err/d Req/h ...

41.
{
“site”: “vimeo.com”,
“port”: 80,
“unit”: “Req/s”,
“direction”: “in”,
“service”: “webapp_php”,
“server”: “webxx”
}

43.
Carbon­tagger:
...
service=foo.instance=host.target_type=gauge.type=calculatio
n.unit=B 123 1234567890
…
Statsdaemon:
..unit=B..unit=B... unit=B/s→
..unit=ms..unit=ms.. unit=ms stat=mean→
→ unit=ms stat=upper_90
→ ...

46.
Graph­Explorer queries 101
site:api.vimeo.com unit=Req/s
requesthostport api_vimeo_com

48.
Smoothing
avg over 10M
avg over ...

50.
Aggregation, compare port 80 vs 443
avg by <dimension>
sum by <dimension>
sum by server

52.
Compare 80 traffic amongt servers
site:api.vimeo.com unit=Req/s port=80 group by none avg
over 10M

54.
Graph­Explorer queries 201
proxy­server swift server:regex upper_90 unit=ms from
<datetime> to <datetime> avg over <timespec>

59.
Compare object put/get
Stack .. http_method:(PUT|GET) swift_type=object avg by
http_code,server

61.
Comparing servers
http_method:(PUT|GET) avg by
http_code,swift_type,http_method group by none

63.
Compare http codes for GET, per swift type
http_method=GET avg by server group by swift_type

65.
transcode unit=Job/s avg over <time> from <datetime> to
<datetime>

66. Note: data is obfuscated

67.
Bucketing
!queue sum by zone:ap­southeast|eu­west|us­east|us­
west|sa­east|vimeo­df|vimeo­lv group by state

68. Note: data is obfuscated

69.
Compare job states per region (zones bucket)
group by zone

70. Note: data is obfuscated

71.
Unit conversion
unit=Mb/s network dfvimeorpc sum by server

74.
unit=MB

77.
{
server=dfvimeodfs1
plugin=diskspace
mountpoint=_srv_node_dfs5
unit=B
type=used
target_type=gauge
}

78.
server:dfvimeodfs unit=GB type=free srv node

80.
unit=GB/d group by mountpoint

87.
Dashboard definition
queries = [
'cpu usage sum by core',
'mem unit=B !total group by type:swap',
'stack network unit=b/s',
'unit=B (free|used) group by =mountpoint'
]

89.
stats.dfvimeocliapp2.twitter.error
{
“n1”: “dfvimeocliapp2”,
“n2”: “twitter”,
“n3”: “error”,
“plugin”: “catchall_statsd”,
“source”: “statsd”,
“target_type”: “rate”,
“unit”: “unknown/s”
}

90.
Two hard things in computer science

91.
stats.gauges.files.
id_boundary_7day
stats.gauges.files.
id_boundary_ceil

92.
unit=File id_boundary_7d
{
“unit”: “File”,
“n1”: “id_boundary_7d”,
}

93.
{
“intrinsic”: {
“site”: “vimeo.com”,
“unit”: “Req/s”
},
“extrinsic”: {
“agent”: “diamond”,
“processed_by”: “statsd1”,
“src”: “index.php:135”,
“replaces”: “vimeo_com_reqps”
}
}

94.
site=vimeo.com unit=Req/s
processed_by=statsd1
src=index.php:135 added_by=dieter
123 1234567890

96.
Equivalence
servers.host.cpu.total.iowait “core” : “_sum_”→
servers.host.cpu.<core­number>.iowait
servers.host.loadavg.15

97.
Rollups & aggregation

98.
/etc/carbon/storage­aggregation.conf
[min]
pattern = .min$
aggregationMethod = min
[max]
pattern = .max$
aggregationMethod = max
[sum]
pattern = .count$
aggregationMethod = sum
[default_average]
pattern = .*
aggregationMethod = average

100.
2 kinds of graphite users

101.
Self­describing metrics
stat=upper/lower/mean/...
target_type=counter..

102.
● stats.timers.render_time.histogram.bin_0.01
● stats.timers.render_time.histogram.bin_0.1
● stats.timers.render_time.histogram.bin_1 unit=Freq_abs bin_upper=1→
● stats.timers.render_time.histogram.bin_10
● stats.timers.render_time.histogram.bin_50
● stats.timers.render_time.histogram.bin_inf
● stats.timers.render_time.lower unit=ms stat=lower→
● stats.timers.render_time.mean unit=ms stat=mean→
● stats.timers.render_time.mean_90 ...→
● stats.timers.render_time.median
● stats.timers.render_time.std
● stats.timers.render_time.upper
● stats.timers.render_time.upper_90

103.
Also..
● graphite API functions such as "cumulative", "summarize"
and "smartSummarize"
● Graph renderers

105.
From: dygraphs.com

111.
Facet based suggestions

113.
Metric types
● gauge
● count & rate
● counter
● timer

118.
gauge
● Multiple values in same interval
● “sticky”

120.
Count & Rate

121.
Counter

122.
Timer..

124.
http://janabeck.com/blog/2012/10/12/lessons­learned­from­100/

125.
Timer..

126.
● What should a metric be?
● Stickyness?
● Behavior on no packets received
● Behavior on multiple packets received

127.
My personal takeaways

128.
Conclusion
● Building graphs, setting up alerting cumbersome
● Esp. changing information needs (troubleshooting, exploring, ..)
● Esp. Complicated information needs
→ PAIN
● Structuring metrics
● Self­describing metrics
● Standardized metrics
● Native metrics 2.0
● → BREEZE

129.
Conclusion
● Metrics can be so much more usable and useful. Let's talk about
tagging, standardisation, retaining information throughout the
pipeline.
● Converting information needs into graph defs, alerting rules
● Graph­Explorer, carbon­tagger, statsdaemon, …
● Graphite­ng (native metrics 2.0)
● Metrics 2.0 in your apps, agents, aggregators?
● Build out structured metrics library

130.
github.com/vimeo
github.com/Dieterbe
twitter.com/Dieter_be
dieter.plaetinck.be