How to monitor Kubernetes and your applications with prometheus-operator and kube-prometheus.

1. Monitoring Kubernetes
with prometheus-operator
Lili Cosic

2. Who am I?
● Lili Cosic
● @lilic on GitHub
● @lilicosic on Twitter
● Principal Software Engineer
● Engineer in the OpenShift in-cluster monitoring team.
● Maintainer and contributor to prometheus operator, kube prometheus, kube-
state-metrics and member of SIG Instrumentation.

3. Prometheus briefly

4. Credit: Prometheus official docs

5. prometheus-operator org
● https://github.com/prometheus-operator
● Consists of two projects right now:
○ Prometheus operator - the operator
○ kube-prometheus - collection of manifests for monitoring
kubernetes
● Independent organization with maintainers from
multiple companies
● 5.5k GitHub stars
● Adopters from various companies (add yourself if
you are using it!)

6. Prometheus Operator

7. Prometheus operator
● https://github.com/prometheus-operator/prometheus-operator
● One of the first Kubernetes operators created by CoreOS
● Simplifies managing, operating and configuring monitoring components within
your Kubernetes clusters
● Provides multi tenancy features
● Self service monitoring

8. Prometheus operator
● Custom Resources
○ Prometheus
○ Alertmanager
○ ServiceMonitor & PodMonitor
○ PrometheusRule
○ ThanosRuler
○ Probe
○ AlertmanagerConfig

9. Prometheus Custom Resource
● Configure the Prometheus
deployment in your Kubernetes
clusters
● Fields to know:
○ selectors
○ alerting
○ resources
○ replicas

10. Alertmanager Custom Resource
● Configure the Alertmanager
StatefulSet deployment in your
Kubernetes clusters

11. ServiceMonitor & PodMonitor Custom Resources
● Configure targets to be monitored in your cluster
● Difference between ServiceMonitor and
PodMonitor
○ ServiceMonitor -> selects pod(s) via Services
○ PodMonitor -> directly selects pod(s)
● Some interesting fields to look out for:
○ namespaceSelector
○ sampleLimit
○ targetLimit

12. ServiceMonitor & PodMonitor Custom Resources
● How does it work?
○ ServiceMonitor or PodMonitor get created by user
○ Operator picks the resource
○ Operator creates a Secret resource with the content of target discovery in prometheus
specific configuration
○ Config-reloader sidecar watches the secrets and reloads Prometheus if there are any
changes

13. PrometheusRule Custom Resource
● Create Alerting and Recording rules
● Alerting rules - define alert conditions
based on Prometheus expression
language expressions and to send
notifications about firing alerts to an
external service
● Recording rules - precompute
frequently needed or computationally
expensive expressions and save their
result as a new set of time series.

14. PrometheusRule Custom Resource
● How does it work?
○ Create PrometheusRule in namespace that prom-operator
watches
○ Operator picks that custom resource
○ Operator bin packs into ConfigMaps
○ Mounts ConfigMaps into Prometheus pod
○ config-reloader sidecar reloads prometheus

15. Probe Custom Resource
● Configure how groups of ingresses
or static targets should be
monitored.
● Operator automatically generates
Prometheus scrape configuration
● Deploy something like
blackbox_exporter

16. AlertmanagerConfig Custom Resource
● Configure subsections of Alertmanager
configuration
● Useful for routing alerts to custom receivers
● Setting inhibit rules
● Great in a multi tenant environment where you
don’t want to give admin access to
Alertmanager Custom Resource to everyone

17. ThanosRuler Custom Resource
● Configure, connect and deploy
Thanos Ruler
● Thanos Rule is a component in
Thanos that evaluates Prometheus
recording and alerting rules against
chosen query API.
● Useful for multi tenant
environments where multiple
Prometheus instances are
deployed

18. Cool overlooked features of prometheus-operator
● Automated Sharding - Specify in the Prometheus spec by setting the number
of shards to distribute targets onto.
● enforcedNamespaceLabel - great for multi tenancy
● Thanos sidecar - configure object storage

19. kube-prometheus

20. kube-prometheus project
● https://github.com/prometheus-operator/kube-prometheus
● Easily monitor your Kubernetes cluster infra workloads out of the box
● Building blocks of Kubernetes cluster monitoring
● You can customize the experience with jsonnet - we do this in OpenShift
clusters
○ Jsonnet - a data templating language that extends JSON
● We do not maintain the helm chart but it is widely used

21. What components?
● Prometheus Operator Deployment
● Highly available Prometheus - 2 replicas
● Highly available Alertmanager - 3 replicas
● kube-state-metrics - metrics about Kubernetes resources
● Prometheus node_exporter - metrics about nodes
● Prometheus Adapter for Kubernetes Metrics APIs
● Grafana + dashboards
● Monitoring Kubernetes cluster components
● Alerting and Recording rules about Kubernetes and monitoring components

22. What you get if you apply the manifests repo
Pods deployed within the cluster Targets being monitored

23. How to monitor your own applications

24. Example app manifests

25. Troubleshooting - targets page
Go to prometheus UI and access the /targets page to see all the targets that
Prometheus could discover or those it could not

26. Troubleshooting
● Set debug log level on prometheus-operator to see which ServiceMonitors
or PodMonitors it picked up.
● kubectl -n monitoring get secret prometheus-k8s -ojson |
jq -r '.data["prometheus.yaml.gz"]' | base64 -d | gunzip
| grep "my-service-monitor-name"
● po-lint is helper binary that decodes and validates your Custom Resources

27. Conclusion

28. Help and docs
● https://prometheus-operator.dev/ <- new website (thanks metalmatze!)
● We also have troubleshooting docs
● Slack channel -> #prometheus-operator channel on Kubernetes slack
● Open issue on GitHub
● Useful docs links:
○ Custom resources and fields docs
○ List of metrics from kube-state-metrics
○ Runbooks for alerts (please contribute more!)
○ Alerting
○ Monitor external etcd
○ Customize kube prometheus experience

29. Thank you!
Lili Cosic
@LiliCosic - Twitter
@lilic - GitHub