Do you gather metrics from your application? Can you combine them and easily generate custom graphs out of them? Can your developers measure whatever they want at any point of your application without breaking it or making it slower? In our next itnig friday, Víctor Martínez will show us how easy it is to roll on your own Graphite installation and how to use Etsy's statsd collector to flush your metrics. You will learn what Graphite is, how all of its components work, how to get your real time&historic metrics into Carbon, Graphite's database, and how to plot them in different manners. Víctor will show us some Graphite dashboards, alternative statds implementations, detailed common Graphite configuration gotchas, design limitations and how to deal with them. <a>Visit details</a>

2. Collecting metrics with
Graphite and StatsD
Víctor Martínez
@knoopx
http://github.com/knoopx
http://knoopx.net

3. What is Graphite?

4. What is Graphite?
An all-in-one solution for storing and visualizing
realtime time-series data in an efficient manner.
Key features:
Efficient storage and ultra-fast retrieval.
http://graphite.wikidot.com/

5. Graphite Architecture
Graphite Web
The frontend of
Graphite. It provides a
dashboard for retrieval
and visualization of our
metrics and a powerful
plotting API.
Carbon
The core of Graphite. It
consists of set of three
processes that take
care of receiving
incoming data,
aggregating it and
persisting it to disk.
Whisper
The current (stable)
data persistence
storage; an efficient
time-series based
database.
* There's a new persistence
layer called Ceres in the
works

6. Feeding in your data
PORT=2003
SERVER=graphite.your.org
echo "stats.app.logins.count 1 `date +%s`" | nc ${SERVER} ${PORT};
PORT=2003
SERVER=graphite.your.org
echo "stats.app.logins.count 1 `date +%s`" | nc ${SERVER} ${PORT};
PORT=2003
SERVER=graphite.your.org
echo "stats.app.logins.count 1 `date +%s`" | nc ${SERVER} ${PORT};
schema.application_name.namespace.metric_name.metric_type
Value
Name
Timestamp

7. Understanding how Carbon works
storage-schemas.conf
Allows you to define how your data
should be persisted (retention rules)
and in which order (priority).
storage-aggregation.conf
Allows you to define how your
metrics should be aggregated.
Key configuration files

8. Understanding how Carbon works
storage-schemas.conf
What does retention mean?
In a timespan of X time, in how much granularity I
want to store my data

9. Understanding how Carbon works
storage-schemas.conf
● For a timespan of 6 hours I want to store my data in
intervals of 10 seconds.
● For a timespan of 7 days I want to store my data in
intervals of 1 minute.
● For a timespan of 1 year I want to store my data in
intervals of 1 hour.
● For a timespan of 5 year I want to store my data in
intervals of 24 hours.

10. Understanding how Carbon works
storage-schemas.conf

11. Understanding how Carbon works
storage-aggregation.conf
Some examples:
stats.app.users.logins.count
stats.app.emails.sent.count
stats.app.orders.amount.max
stats.app.orders.amount.min
stats.app.response.time

12. Understanding how Carbon works
storage-aggregation.conf

13. Understanding how Carbon works
Carbon limitations
What happens if I send two metrics at
the same timespan (1h)?
1 new user – 6 minutes ago
2 new users – 5 minutes ago

14. Understanding how Carbon works
Carbon limitations
Carbon discards everything
but the last one
2 users

15. Understanding how Carbon works
Carbon limitations
● Increase schema retention: 1s:12h,[...]
and never, ever, push metrics older than
12 hours/push more than once per
second
● Pre-aggregate your data before pushing
● Use StatsD
Possible Solutions:

16. Understanding how Carbon works
Pre-aggregate your data
https://gist.github.com/knoopx/4738209

17. Understanding how Carbon works
Use StatsD
What is StatdD?
A network daemon that runs on the Node.js platform
and listens for statistics, like counters and timers, sent
over UDP and sends aggregates to one or more
pluggable backend services (e.g., Graphite).
Key features: Aggregates the data for us, fire and
forget communication protocol (UDP)
https://github.com/etsy/statsd/

18. Understanding how Carbon works
https://github.com/Shopify/statsd-instrument
StatsD.increment('revenue', 23.99)
StatsD.measure('worker.time') do
Worker.perform(value)
end
StatsD Instrument Gem
Important: Carefully read StatsD documentation and
ensure that your Carbon configuration explicitly
matches StatsD naming conventions

19. Visualizing your metrics
Using the Graphite dashboard
Common functions:
● summarize
● timeShift
● keepLastValue
● integral
● drawAsInfinite
● sumSeries
● alias
● aliasByNode
Full Documentation: http://graphite.readthedocs.
org/en/1.0/functions.html

20. Visualizing your metrics
Using the Graphite dashboard
Time-shifting and aliasing:
alias(timeShift(camaloon.sidekiq.workers.success.count,"24h"),"Yesterday")
alias(camaloon.sidekiq.workers.success.count,"Today")

21. Visualizing your metrics
Using the Graphite dashboard
Summarizing data:
summarize(camaloon.sidekiq.workers.success.count,"1h")
1h

22. Visualizing your metrics
Using the Graphite dashboard
Aliasing and combining series:
aliasByNode(camaloon.sidekiq.workers.*.count, 3)

23. Visualizing your metrics
Tasseo https://github.com/obfuscurity/tasseo
Cubism.js https://github.com/square/cubism

24. Visualizing your metrics
Graphene https://github.com/jondot/graphene

25. Visualizing your metrics
Your own solution
/render/?target=camaloon.users.count&format=json
We internally use Highchart Highstock

26. Other resources
Deployment
https://github.com/manasg/chef-graphite
https://github.com/gingerlime/graphite-fabric
https://github.com/knoopx/puppet-graphite
http://warwickp.com/2012/03/vagrant-box-with-graphite-statsd-gdash-nginx
https://github.com/obfuscurity/backstop
Learning
http://obfuscurity.com/Tags/Graphite

27. Questions?