This document discusses scaling Graphite, an open source monitoring tool. It describes how to feed data into Graphite using tools like StatsD and Collectd. It then discusses how to scale these data collection tools to handle high volumes of metrics. The document reviews several open source projects for scaling Graphite like Carbonapi, Carbonzipper, and go-carbon. It provides an overview of how these projects improve Graphite's flexibility, redundancy, and performance when dealing with large numbers of metrics.

1. Scaling out
Metrics Flow and Graphite
Vladimir Mevzos
PRESENTED BY:
Senior DevOps , Appsflyer

2. Measure Anything,Everything

3. Graphite Original Stack

4. Feeding Graphite with Data
· $ echo "foo.bar 1 `date +%s`" | nc localhost 2003
· $ echo [(foo.bar, (`date +%s`, 1)), …] | nc localhost 2003

5. Feeding Graphite with Data
· Plaintext protocol
· Pickle protocol
· StatsD
· Collectd
· Sensu
· Telegraf

6. Use your Data
· Grafana - UI for graphite
· Sensu – monitoring framework
· Autoscale/Healer – Homebrew ruby scripts

7. Aggregate Everything First

8. Scaling Collectd

9. Scaling StatsD
· AWS shared ENA (master/slave)
· Keepalived for HA and LB
· LVS – Linux Virtual Server UDP (DR ,TUN)
· StatsD Proxy vs statsite

10. Scaling Statsd

11. StatsD Kernel Trick
· net.ipv4.udp_mem > 10 MB
· net.core.wmem_max > 10 MB

12. Looks Good ...
But,if metrics > 30k/s
· Scaling
· Redundancy
· Performance
· Backup
· Management

13. What is other people do?
OpenSource projects
•Carbonapi
•Carbonzipper
•Gocarbon
•Carbon-c-relay
•Carbon-relay-ng
•Bucky
•Buckytools

14. Routing and Filtering

15. Tribute to carbon-c-relay
· Flexibility
· IOPS: Ephemeral / RAM / EBS
· Filtering by Regexp
· Buffer
· Clone

16. New Graphite Stack

17. Tribute to carbonapi

18. Tribute to go-carbon
· Performance
· Flexibility
· Easy to manage

19. Go-carbon Key Points
· IOPS: max-updates-per-second
· IOPS: scan-frequency
· CPU: max-cpu ,workers
· Cache: max-size,write-strategy

20. Measure Anything
Measure Everything

21. Questions?
vladimir@appsflyer.com