5 tips to better use the Service Discovery feature of Prometheus

1. PromCon Online
Julien Pivotto
@roidelapluie
5 tips to better use
Prometheus Service Discovery
July 15th, 2020

2. • Used to discover targets
• Listens to events from target (e.g. consul)
• Queries targets at a certain interval (e.g. DNS)
Service Discovery in Prometheus
@roidelapluie

3. scrape_configs:
- job_name: node
digitalocean_sd_configs:
- bearer_token: "<replace with a Personal Access Token>"
relabel_configs:
- source_labels: [__meta_digitalocean_tags]
regex: '.*,monitoring,.*'
action: keep
- source_labels: [__meta_digitalocean_public_ipv6]
target_label: __address__
replacement: '[$1]:9100'
Classic pattern: Service discovery + Relabel con g.
Relabel con g = Change labels found by SD to alert scrape con g.
Classic exemple
@roidelapluie

4. alerting:
alertmanagers:
digitalocean_sd_configs:
- bearer_token: "<replace with a Personal Access Token>"
relabel_configs:
- source_labels: [__meta_digitalocean_tags]
regex: '.*,monitoring,.*'
action: keep
Automatically discover alertmanager instances.
Tip 1: Alerting uses service discovery
@roidelapluie

5. scrape_configs:
- job_name: node
digitalocean_sd_configs:
- bearer_token: "<Personal Access Token1>"
- bearer_token: "<Personal Access Token2>"
- bearer_token: "<Personal Access Token3>"
relabel_configs:
- source_labels: [__meta_digitalocean_tags]
regex: '.*,monitoring,.*'
action: keep
- source_labels: [__meta_digitalocean_public_ipv6]
target_label: __address__
replacement: '[$1]:9100'
You can use multiple sources in one job.
Tip 2: SD con gs are lists
@roidelapluie

6. scrape_configs:
- job_name: node
static_configs:
- targets: [127.0.0.1:9090]
gce_sd_configs:
- zone: europe-west1-b
project: myproject
digitalocean_sd_configs:
- bearer_token: "<Personal Access Token1>"
- bearer_token: "<Personal Access Token2>"
You can use multiple kinds of SD for one job.
Tip 3: Multiple SD for one job
@roidelapluie

7. - job_name: node
digitalocean_sd_configs:
- bearer_token: "<Personal Access Token1>"
relabel_configs:
- source_labels: [__meta_digitalocean_private_ipv4]
target_label: __address__
replacement: '[$1]:9100'
- job_name: cadvisor
digitalocean_sd_configs:
- bearer_token: "<Personal Access Token1>"
relabel_configs:
- source_labels: [__meta_digitalocean_private_ipv4]
target_label: __address__
replacement: '[$1]:8080'
Tip 4: Reuse SD for multiple jobs
@roidelapluie

8. Prometheus will REUSE the same API calls/connection if the SD con g is
exactly identical.
Check in DEBUG logs:
caller=manager.go:226 component="discovery manager scrape" msg="Starting
provider" provider=*digitalocean.SDConfig/0 subs="[node cadvisor]"
Tip 4: Reuse SD for multiple jobs (cont)
@roidelapluie

9. scrape_configs:
- job_name: node
file_sd_configs:
- files:
- targets.yml
Tip 5: le_sd les can contain con g labels
@roidelapluie

10. targets.yml:
- labels:
__scheme__: https
__metrics_path__: /metrics2
__param_foo: bar
targets: [localhost:9100]
- labels:
__param_foo: bar2
targets: [localhost:9101]
https://localhost:9100/metrics2?foo=bar
http://localhost:9101/metrics?foo=bar2
Tip 5: le_sd les can contain con g labels (cont)
@roidelapluie

11. Julien Pivotto
@roidelapluie
roidelapluie@inuits.eu
Essensteenweg 31
2930 Brasschaat
Belgium
Contact:
info@inuits.eu
+32-3-8082105