Scaling prometheus to a million machines.

1. Breaking Prometheus
Scaling prometheus
to a million machines

2. About Me
• Technical Lead Digital Ocean
• Microservices in GO Book
• Lives in Bangkok

3. Dark Days
• Graphite
• InfluxDB
• OpenTSDB (*sigh*)

4. Manual Prometheus

5. Manual Issues
• Lots of Prometheus servers
• Mismatched versions
• New machines = update config
• Missing matches

6. Consul + Prometheus = Peanut butter + Jelly

8. Datacenter wide
10,000s of nodes
Stage 2:

9. Prometheus Per Region
Prometheus
Grafana
Prometheus Prometheus Prometheus
Prometheus
Prometheus

10. I/O Problems
• Modify retention windows
• Drop metrics from node_exporter
• Larger and larger machines

11. Tuning options
storage.local.retention
storage.local.memory-chunks
storage.local.max-chunks-to-persist
storage.local.checkpoint-interval
storage.local.checkpoint-dirty-series-limit

12. Sharding

14. rate(node_cpu{instance=“server12345.digitalocean.com:9100”}[2m])
Shard on red
Prometheus Proxy

15. Alert Manager

16. Shard Problems
• Shard redistribution
• Over provisioning
• Data loss
• Limited data windows

17. Kafka Bus

19. Million VMs

20. Digital Ocean
Agent
• Installable Metrics Agent
• Authenticated Push Gateway
• “Reverse node exporter”

21. Query Api
• Customer facing API
• GRPC / Json
• Authenticated per customer
• Prometheus queries

22. https://github.com/digitalocean/vulcan
Introducing Vulcan

23. Vulcan
• Prometheus Api
• Cassandra storage
• Kafka incoming
• Standard Scrapers

24. Cassandra Store

25. Downsampling
• In memory shared promethues
• Driven from data in kafka
• Reusable for Alerting

26. Downsampling / Alerting

27. Future
• New Scrape Sources (Kafka)
• Per series expiry TTLs
• Plugin Storage Model (In Memory)
• Alerting High Availability

28. Questions?
We’re Hiring!
Matthew Campbell
hyper@hyperworks.nu
@kanwisher
github.com/mattkanwisher