Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Time Series in Prometheus
Fabian Reinartz – Engineer, SoundCloud Ltd.
prometheus.io
...
http_requests_total{status="200",method="GET"} @1434317560938 94355
http_requests_total{status="200",method="GET"} @14...
● 1 million time series
● 10 second sample resolution
● 64bit timestamp + 64bit value
Requirements
100,000 samples/sec
time [~weeks]
series
[~millions]
Writes
The Fundamental Problem
Orthogonal write and read patterns.
Reads
...
http_requests_total{status="200",method="GET"} @1434317560938 94355
http_requests_total{status="200",method="GET"} @14...
Ingestion
PromQL
Storage
in-memory data
append(series, time, value)
series iterators
HDD / SSD
LevelDB
Encode
Decode
Compr...
http_requests_total{status="200",method="GET"}
Prometheus Metrics
Metric name Labels
{__name__="http_requests_total",statu...
Prometheus Metrics
Learning the hard way
__name__ = http_requests_total
status = 200
method = GET
fnv(sort( )
fnv( __name_...
1KB chunks
chunk in memory
[complete and immutable]
head chunk
[incomplete]
Sample
Ingestion
append(series, time, value)
m...
Series maintenance
memory
disk
older than retention time
Chunk preloading
memory
disk
PromQL
series iterator
basetime
Anatomy of a chunk [v0]5bytesheader
basevalue
valuetime
valuetime
valuetime
valuetime
... (one per timestamp)
......
basetime
Anatomy of a chunk [v1]5bytesheader
basevalue
valuetime
valuetime
valuetime
valuetime
... (one per timestamp)
......
basetime
Anatomy of a chunk [v2]5bytesheader
basevalue
valuetime
valuetime
valuetime
valuetime
... (one per timestamp)
......
basetime
Anatomy of a chunk [v2]5bytesheader
basevalue
valuetime
valuetime
valuetime
valuetime
... (one per timestamp)
......
rate(prometheus_local_storage_ingested_samples_total[1m])
Checkpointing
On shutdown and regularly to limit data loss in case of a crash.
memory
disk
checkpoint file
Prometheus Storage
Prometheus Storage
Prometheus Storage
Prometheus Storage
Prometheus Storage
Prochain SlideShare
Chargement dans…5
×

Prometheus Storage

5 412 vues

Publié le

A brief overview of time series in Prometheus and how they are stored.

Publié dans : Technologie
  • Soyez le premier à commenter

Prometheus Storage

  1. 1. Time Series in Prometheus Fabian Reinartz – Engineer, SoundCloud Ltd.
  2. 2. prometheus.io
  3. 3. ... http_requests_total{status="200",method="GET"} @1434317560938 94355 http_requests_total{status="200",method="GET"} @1434317561287 94934 http_requests_total{status="200",method="GET"} @1434317562344 96483 http_requests_total{status="404",method="GET"} @1434317560938 38473 http_requests_total{status="404",method="GET"} @1434317561249 38544 http_requests_total{status="404",method="GET"} @1434317562588 38663 http_requests_total{status="200",method="POST"} @1434317560885 4748 http_requests_total{status="200",method="POST"} @1434317561483 4795 http_requests_total{status="200",method="POST"} @1434317562589 4833 http_requests_total{status="404",method="POST"} @1434317560939 122 ... Prometheus Metrics Metric name Labels Timestamp Sample Value
  4. 4. ● 1 million time series ● 10 second sample resolution ● 64bit timestamp + 64bit value Requirements 100,000 samples/sec
  5. 5. time [~weeks] series [~millions] Writes The Fundamental Problem Orthogonal write and read patterns. Reads
  6. 6. ... http_requests_total{status="200",method="GET"} @1434317560938 94355 http_requests_total{status="200",method="GET"} @1434317561287 94934 http_requests_total{status="200",method="GET"} @1434317562344 96483 http_requests_total{status="404",method="GET"} @1434317560938 38473 http_requests_total{status="404",method="GET"} @1434317561249 38544 http_requests_total{status="404",method="GET"} @1434317562588 38663 http_requests_total{status="200",method="POST"} @1434317560885 4748 http_requests_total{status="200",method="POST"} @1434317561483 4795 http_requests_total{status="200",method="POST"} @1434317562589 4833 http_requests_total{status="404",method="POST"} @1434317560939 122 ... Prometheus Metrics Key-Value store (with BigTable semantics) seems suitable. Metric name Labels Timestamp Sample Value VALUEKEY
  7. 7. Ingestion PromQL Storage in-memory data append(series, time, value) series iterators HDD / SSD LevelDB Encode Decode Compress Decompress
  8. 8. http_requests_total{status="200",method="GET"} Prometheus Metrics Metric name Labels {__name__="http_requests_total",status="200",method="GET"} Labels
  9. 9. Prometheus Metrics Learning the hard way __name__ = http_requests_total status = 200 method = GET fnv(sort( ) fnv( __name__ = http_requests_total ) fnv( status = 200 ) fnv( method = GET ) ⊕ ⊕
  10. 10. 1KB chunks chunk in memory [complete and immutable] head chunk [incomplete] Sample Ingestion append(series, time, value) memory disk evictable chunks (LRU) chunk on disk [complete and immutable] PromQL series iterator one file per time series series hash:
  11. 11. Series maintenance memory disk older than retention time
  12. 12. Chunk preloading memory disk PromQL series iterator
  13. 13. basetime Anatomy of a chunk [v0]5bytesheader basevalue valuetime valuetime valuetime valuetime ... (one per timestamp) ... (one per value) 1000000 1441558420098 1001050 1441558432221 1002040 1441558444311 1002040 1441558444311 1000000 1441558420098 1001050 1441558432221
  14. 14. basetime Anatomy of a chunk [v1]5bytesheader basevalue valuetime valuetime valuetime valuetime ... (one per timestamp) ... (one per value) 1000000 1441558420098 1050 12123 2040 24213 3100 36313 4250 48500 1002040 1441558444311 1000000 1441558420098 1001050 1441558432221 +12123 +1050 +12090 +1000
  15. 15. basetime Anatomy of a chunk [v2]5bytesheader basevalue valuetime valuetime valuetime valuetime ... (one per timestamp) ... (one per value) 1002040 1441558444311 1000000 1441558420098 1001050 1441558432221 1000000 1441558420098 1050 12123 -60 -33 -50 -56 +50 -8 +12123 +1050 +12090 +1000
  16. 16. basetime Anatomy of a chunk [v2]5bytesheader basevalue valuetime valuetime valuetime valuetime ... (one per timestamp) ... (one per value) 13:14 < nostrovsk> Hey guys, Looking for a sanity check here 13:15 < nostrovsk> 500 machines per server, each running node and jmx exporters, for 1 week is only 30gb of data? 13:36 <@ bbrazil> what's your scrape rate and how heavy are those jmx exporters? 13:37 <@ bbrazil> doesn't sound implausible to me 13:42 <@ bbrazil> we're 25GB/two weeks with ~5k samples/s 13:45 <@ beorn7> Compression, it works... ;) 13:53 < fish_> beorn7: nothing says better 'good job' than people coming to this channel because they can't believe that things are soo good :)
  17. 17. rate(prometheus_local_storage_ingested_samples_total[1m])
  18. 18. Checkpointing On shutdown and regularly to limit data loss in case of a crash. memory disk checkpoint file

×