Slides from #PromCon2018 Munich. https://promcon.io/2018-munich/talks/thanos-prometheus-at-scale/ Bartłomiej Płotka Fabian Reinartz The Prometheus Monitoring system has been thriving for several years. Along with its powerful data model, operational simplicity and reliability have been a key factor in its success. However, some questions were still largely unaddressed to this day. How can we store historical data at the order of petabytes in a reliable and cost-efficient way? Can we do so without sacrificing responsive query times? And what about a global view of all our metrics and transparent handling of HA setups? Thanos takes Prometheus' strong foundations and extends it into a clustered, yet coordination free, globally scalable metric system. It retains Prometheus's simple operational model and even simplifies deployments further. Under the hood, Thanos uses highly cost-efficient object storage that's available in virtually all environments today. By building directly on top of the storage format introduced with Prometheus 2.0, Thanos achieves near real-time responsiveness even for cold queries against historical data. All while having virtually no cost overhead beyond that of the underlying object storage. We will show the theoretical concepts behind Thanos and demonstrate how it seamlessly integrates into existing Prometheus setups.

2. ●
●
●
●

11. Remote
write

12. ●
●
●

16. service Store {
rpc Series(SeriesRequest) returns (stream SeriesResponse);
rpc LabelNames(LabelNamesRequest) returns (LabelNamesResponse);
rpc LabelValues(LabelValuesRequest) returns (LabelValuesResponse);
}
message SeriesRequest {
int64 min_time = 1;
int64 max_time = 2;
repeated LabelMatcher matchers = 3;
}

20. ●
●

22. T-10hT-16h T-4h T-2h T

23. T-10hT-16h T-4h T-2h T

25. --storage.tsdb.max-block-duration=2h
--storage.tsdb.retention=12h

26. Store API

27. ●
●
●

33. Store
Gateway

35. Bucket
…
Bucket
Bucket
Cluster A
(master)
Cluster B
Cluster C
Federation (through Store API)

36. Bucket

41. ●
●

42. ●
●
●
● rate()

43. 10x 12x

44. chunk
count sum min max counter
chunk...

45. count sum min max counter
count_over_time(requests_total[1h])

46. count sum min max counter
sum_over_time(requests_total[1h])

47. count sum min max counter
min(requests_total)
min_over_time(requests_total[1h])

48. count sum min max counter
max(requests_total)
max_over_time(requests_total[1h])

49. count sum min max counter
rate(requests_total[1h])
increase(requests_total[1h])

50. count sum min max counter
requests_total
avg(requests_total)
...
*
avg

51. ●
●
●