SlideShare une entreprise Scribd logo
1  sur  22
Code instrumentation in Py
with Prometheus &
Grafana
Francois SCHMIDTS
Vlad ZLOTEANU
DOLEAD
Contents
- Prometheus & Grafana
- Code instrumentation example
- 3 use cases
- (Dolead’s) push client
Prometheus + Grafana = ❤
Metrics
retrieval
Target 2
Target 1
Target N
Querying
PromQL
TimeSeriesDB
- Multidimensional data
model
Exporter
Grafana
Pulls
Queries
Alert
Manager
Instrum
entation
Pulls
Dashboard
Prometheus
- TSDB
- Open Source
- Incubated by CNCF (After Kubernetes)
- Adapted to VM/containers monitoring
- Autodiscovery
- Pull model
- Multidimensional data
- Includes alerting
Grafana
- OS metric analytics / visualisation
- multiple providers: CloudWatch, Prometheus, InfluxDb, ES, ..
- multiple dashboards already available
- in coop with Prometheus exporters
Node exporter + Grafana dashboard
MongoDB exporter + Grafana dashboard
Case study: RR Stats import
● Metric: Duration of execution
Labels
● Result
○ success/failure
● Source
○ Google Ads, Fb Ads, Bing Ads, Taboola, etc.
● Category
○ Account, Campaign, Keyword, ..
○ Today vs Past
● Node
Instrumentation - Code example
1. Debugging / Gain insight
"Where does the problem come from / What is going on?"
● Segment by sources (Google Ads, Fb Ads, Bing Ads, Taboola, etc.)
○ Did they slow down? Error rate gone up? Are they unavailable?
● Segment by category
○ Did we introduce a bug on that code?
● Segment by node
○ do I have a problem on that node?
All successful stats downloads
All successful stats downloads - vs Bing
1. Debugging / Gain insight
Combination with external data / corroboration
- deployments
- CPU/Ram/Load on the node
- “can we corroborate with a slow query increase in Mongodb?”
Example: Sync activity vs machine load
2. Alerting
- Grafana alerts:
- alerts based on configured data sources
- Prometheus AlertManager:
- can alert based on PromQL query
- Infrastructure as Code
Instrument now, decide later
2. Alerting - Example
2. Alerting - Graph
3. Trends / Scale
● Trends over time, drive scale (technical) / business decisions
○ Capacity planning
○ "Will I (when will I) have a problem in the future?"
● SLA / QoS
And all this is available thanks to this code:
Push (vs pull)
- Async, short-lived processes
- The prometheus way => send metrics to a push gateway
- One push gateway per process !
- More infrastructure to setup
- Our way, the prometheus-distributed-client => send metrics to a database
- Available from everywhere
- Consistent in case of concurrent calls
- Use either
Conclusion
- Try to always instrument your code
- Limite the cardinality of the metrics you use
- Make nice graphs !
- Use Our lib : https://github.com/dolead/prometheus-distributed-client
Thank you!
Questions?

Contenu connexe

Similaire à Code instrumentation in Py with Prometheus and Grafana

Moving Towards a Streaming Architecture
Moving Towards a Streaming ArchitectureMoving Towards a Streaming Architecture
Moving Towards a Streaming Architecture
Gabriele Modena
 
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic SystemTimely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
Accumulo Summit
 
Tooling for Machine Learning: AWS Products, Open Source Tools, and DevOps Pra...
Tooling for Machine Learning: AWS Products, Open Source Tools, and DevOps Pra...Tooling for Machine Learning: AWS Products, Open Source Tools, and DevOps Pra...
Tooling for Machine Learning: AWS Products, Open Source Tools, and DevOps Pra...
SQUADEX
 

Similaire à Code instrumentation in Py with Prometheus and Grafana (20)

Smartblitzmerker
SmartblitzmerkerSmartblitzmerker
Smartblitzmerker
 
Telemetry indepth
Telemetry indepthTelemetry indepth
Telemetry indepth
 
System monitoring
System monitoringSystem monitoring
System monitoring
 
Prometheus with Grafana - AddWeb Solution
Prometheus with Grafana - AddWeb SolutionPrometheus with Grafana - AddWeb Solution
Prometheus with Grafana - AddWeb Solution
 
Prometheus - Intro, CNCF, TSDB,PromQL,Grafana
Prometheus - Intro, CNCF, TSDB,PromQL,GrafanaPrometheus - Intro, CNCF, TSDB,PromQL,Grafana
Prometheus - Intro, CNCF, TSDB,PromQL,Grafana
 
Moving Towards a Streaming Architecture
Moving Towards a Streaming ArchitectureMoving Towards a Streaming Architecture
Moving Towards a Streaming Architecture
 
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic SystemTimely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
 
Data Engineer's Lunch #60: Series - Developing Enterprise Consciousness
Data Engineer's Lunch #60: Series - Developing Enterprise ConsciousnessData Engineer's Lunch #60: Series - Developing Enterprise Consciousness
Data Engineer's Lunch #60: Series - Developing Enterprise Consciousness
 
Monitoring using Prometheus and Grafana
Monitoring using Prometheus and GrafanaMonitoring using Prometheus and Grafana
Monitoring using Prometheus and Grafana
 
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
 
Real time big data analytics with Storm by Ron Bodkin of Think Big Analytics
Real time big data analytics with Storm by Ron Bodkin of Think Big AnalyticsReal time big data analytics with Storm by Ron Bodkin of Think Big Analytics
Real time big data analytics with Storm by Ron Bodkin of Think Big Analytics
 
Nexmark with beam
Nexmark with beamNexmark with beam
Nexmark with beam
 
OSDC 2019 | Democratizing Data at Go-JEK by Maulik Soneji
OSDC 2019 | Democratizing Data at Go-JEK by Maulik SonejiOSDC 2019 | Democratizing Data at Go-JEK by Maulik Soneji
OSDC 2019 | Democratizing Data at Go-JEK by Maulik Soneji
 
Influx/Days 2017 San Francisco | Dan Cech
Influx/Days 2017 San Francisco | Dan Cech Influx/Days 2017 San Francisco | Dan Cech
Influx/Days 2017 San Francisco | Dan Cech
 
Streaming meetup
Streaming meetupStreaming meetup
Streaming meetup
 
ML in Production at FunTech Meetup (Feb 2019)
ML in Production at FunTech Meetup (Feb 2019)ML in Production at FunTech Meetup (Feb 2019)
ML in Production at FunTech Meetup (Feb 2019)
 
Monitoring as Software Validation
Monitoring as Software ValidationMonitoring as Software Validation
Monitoring as Software Validation
 
Tooling for Machine Learning: AWS Products, Open Source Tools, and DevOps Pra...
Tooling for Machine Learning: AWS Products, Open Source Tools, and DevOps Pra...Tooling for Machine Learning: AWS Products, Open Source Tools, and DevOps Pra...
Tooling for Machine Learning: AWS Products, Open Source Tools, and DevOps Pra...
 
Monitoring MySQL with Prometheus and Grafana
Monitoring MySQL with Prometheus and GrafanaMonitoring MySQL with Prometheus and Grafana
Monitoring MySQL with Prometheus and Grafana
 
OSMC 2017 | Monitoring MySQL with Prometheus and Grafana by Julien Pivotto
OSMC 2017 | Monitoring  MySQL with Prometheus and Grafana by Julien PivottoOSMC 2017 | Monitoring  MySQL with Prometheus and Grafana by Julien Pivotto
OSMC 2017 | Monitoring MySQL with Prometheus and Grafana by Julien Pivotto
 

Dernier

Call Now ≽ 9953056974 ≼🔝 Call Girls In New Ashok Nagar ≼🔝 Delhi door step de...
Call Now ≽ 9953056974 ≼🔝 Call Girls In New Ashok Nagar  ≼🔝 Delhi door step de...Call Now ≽ 9953056974 ≼🔝 Call Girls In New Ashok Nagar  ≼🔝 Delhi door step de...
Call Now ≽ 9953056974 ≼🔝 Call Girls In New Ashok Nagar ≼🔝 Delhi door step de...
9953056974 Low Rate Call Girls In Saket, Delhi NCR
 
VIP Call Girls Ankleshwar 7001035870 Whatsapp Number, 24/07 Booking
VIP Call Girls Ankleshwar 7001035870 Whatsapp Number, 24/07 BookingVIP Call Girls Ankleshwar 7001035870 Whatsapp Number, 24/07 Booking
VIP Call Girls Ankleshwar 7001035870 Whatsapp Number, 24/07 Booking
dharasingh5698
 
Cara Menggugurkan Sperma Yang Masuk Rahim Biyar Tidak Hamil
Cara Menggugurkan Sperma Yang Masuk Rahim Biyar Tidak HamilCara Menggugurkan Sperma Yang Masuk Rahim Biyar Tidak Hamil
Cara Menggugurkan Sperma Yang Masuk Rahim Biyar Tidak Hamil
Cara Menggugurkan Kandungan 087776558899
 
Call Girls in Ramesh Nagar Delhi 💯 Call Us 🔝9953056974 🔝 Escort Service
Call Girls in Ramesh Nagar Delhi 💯 Call Us 🔝9953056974 🔝 Escort ServiceCall Girls in Ramesh Nagar Delhi 💯 Call Us 🔝9953056974 🔝 Escort Service
Call Girls in Ramesh Nagar Delhi 💯 Call Us 🔝9953056974 🔝 Escort Service
9953056974 Low Rate Call Girls In Saket, Delhi NCR
 
VIP Call Girls Palanpur 7001035870 Whatsapp Number, 24/07 Booking
VIP Call Girls Palanpur 7001035870 Whatsapp Number, 24/07 BookingVIP Call Girls Palanpur 7001035870 Whatsapp Number, 24/07 Booking
VIP Call Girls Palanpur 7001035870 Whatsapp Number, 24/07 Booking
dharasingh5698
 

Dernier (20)

University management System project report..pdf
University management System project report..pdfUniversity management System project report..pdf
University management System project report..pdf
 
(INDIRA) Call Girl Aurangabad Call Now 8617697112 Aurangabad Escorts 24x7
(INDIRA) Call Girl Aurangabad Call Now 8617697112 Aurangabad Escorts 24x7(INDIRA) Call Girl Aurangabad Call Now 8617697112 Aurangabad Escorts 24x7
(INDIRA) Call Girl Aurangabad Call Now 8617697112 Aurangabad Escorts 24x7
 
Call Now ≽ 9953056974 ≼🔝 Call Girls In New Ashok Nagar ≼🔝 Delhi door step de...
Call Now ≽ 9953056974 ≼🔝 Call Girls In New Ashok Nagar  ≼🔝 Delhi door step de...Call Now ≽ 9953056974 ≼🔝 Call Girls In New Ashok Nagar  ≼🔝 Delhi door step de...
Call Now ≽ 9953056974 ≼🔝 Call Girls In New Ashok Nagar ≼🔝 Delhi door step de...
 
VIP Model Call Girls Kothrud ( Pune ) Call ON 8005736733 Starting From 5K to ...
VIP Model Call Girls Kothrud ( Pune ) Call ON 8005736733 Starting From 5K to ...VIP Model Call Girls Kothrud ( Pune ) Call ON 8005736733 Starting From 5K to ...
VIP Model Call Girls Kothrud ( Pune ) Call ON 8005736733 Starting From 5K to ...
 
Intze Overhead Water Tank Design by Working Stress - IS Method.pdf
Intze Overhead Water Tank  Design by Working Stress - IS Method.pdfIntze Overhead Water Tank  Design by Working Stress - IS Method.pdf
Intze Overhead Water Tank Design by Working Stress - IS Method.pdf
 
Call Girls Wakad Call Me 7737669865 Budget Friendly No Advance Booking
Call Girls Wakad Call Me 7737669865 Budget Friendly No Advance BookingCall Girls Wakad Call Me 7737669865 Budget Friendly No Advance Booking
Call Girls Wakad Call Me 7737669865 Budget Friendly No Advance Booking
 
(INDIRA) Call Girl Meerut Call Now 8617697112 Meerut Escorts 24x7
(INDIRA) Call Girl Meerut Call Now 8617697112 Meerut Escorts 24x7(INDIRA) Call Girl Meerut Call Now 8617697112 Meerut Escorts 24x7
(INDIRA) Call Girl Meerut Call Now 8617697112 Meerut Escorts 24x7
 
VIP Call Girls Ankleshwar 7001035870 Whatsapp Number, 24/07 Booking
VIP Call Girls Ankleshwar 7001035870 Whatsapp Number, 24/07 BookingVIP Call Girls Ankleshwar 7001035870 Whatsapp Number, 24/07 Booking
VIP Call Girls Ankleshwar 7001035870 Whatsapp Number, 24/07 Booking
 
Navigating Complexity: The Role of Trusted Partners and VIAS3D in Dassault Sy...
Navigating Complexity: The Role of Trusted Partners and VIAS3D in Dassault Sy...Navigating Complexity: The Role of Trusted Partners and VIAS3D in Dassault Sy...
Navigating Complexity: The Role of Trusted Partners and VIAS3D in Dassault Sy...
 
Water Industry Process Automation & Control Monthly - April 2024
Water Industry Process Automation & Control Monthly - April 2024Water Industry Process Automation & Control Monthly - April 2024
Water Industry Process Automation & Control Monthly - April 2024
 
CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record
CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete RecordCCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record
CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record
 
data_management_and _data_science_cheat_sheet.pdf
data_management_and _data_science_cheat_sheet.pdfdata_management_and _data_science_cheat_sheet.pdf
data_management_and _data_science_cheat_sheet.pdf
 
Unleashing the Power of the SORA AI lastest leap
Unleashing the Power of the SORA AI lastest leapUnleashing the Power of the SORA AI lastest leap
Unleashing the Power of the SORA AI lastest leap
 
Double Revolving field theory-how the rotor develops torque
Double Revolving field theory-how the rotor develops torqueDouble Revolving field theory-how the rotor develops torque
Double Revolving field theory-how the rotor develops torque
 
Booking open Available Pune Call Girls Pargaon 6297143586 Call Hot Indian Gi...
Booking open Available Pune Call Girls Pargaon  6297143586 Call Hot Indian Gi...Booking open Available Pune Call Girls Pargaon  6297143586 Call Hot Indian Gi...
Booking open Available Pune Call Girls Pargaon 6297143586 Call Hot Indian Gi...
 
Cara Menggugurkan Sperma Yang Masuk Rahim Biyar Tidak Hamil
Cara Menggugurkan Sperma Yang Masuk Rahim Biyar Tidak HamilCara Menggugurkan Sperma Yang Masuk Rahim Biyar Tidak Hamil
Cara Menggugurkan Sperma Yang Masuk Rahim Biyar Tidak Hamil
 
Call Girls in Ramesh Nagar Delhi 💯 Call Us 🔝9953056974 🔝 Escort Service
Call Girls in Ramesh Nagar Delhi 💯 Call Us 🔝9953056974 🔝 Escort ServiceCall Girls in Ramesh Nagar Delhi 💯 Call Us 🔝9953056974 🔝 Escort Service
Call Girls in Ramesh Nagar Delhi 💯 Call Us 🔝9953056974 🔝 Escort Service
 
FEA Based Level 3 Assessment of Deformed Tanks with Fluid Induced Loads
FEA Based Level 3 Assessment of Deformed Tanks with Fluid Induced LoadsFEA Based Level 3 Assessment of Deformed Tanks with Fluid Induced Loads
FEA Based Level 3 Assessment of Deformed Tanks with Fluid Induced Loads
 
Booking open Available Pune Call Girls Koregaon Park 6297143586 Call Hot Ind...
Booking open Available Pune Call Girls Koregaon Park  6297143586 Call Hot Ind...Booking open Available Pune Call Girls Koregaon Park  6297143586 Call Hot Ind...
Booking open Available Pune Call Girls Koregaon Park 6297143586 Call Hot Ind...
 
VIP Call Girls Palanpur 7001035870 Whatsapp Number, 24/07 Booking
VIP Call Girls Palanpur 7001035870 Whatsapp Number, 24/07 BookingVIP Call Girls Palanpur 7001035870 Whatsapp Number, 24/07 Booking
VIP Call Girls Palanpur 7001035870 Whatsapp Number, 24/07 Booking
 

Code instrumentation in Py with Prometheus and Grafana

  • 1. Code instrumentation in Py with Prometheus & Grafana Francois SCHMIDTS Vlad ZLOTEANU DOLEAD
  • 2. Contents - Prometheus & Grafana - Code instrumentation example - 3 use cases - (Dolead’s) push client
  • 3. Prometheus + Grafana = ❤ Metrics retrieval Target 2 Target 1 Target N Querying PromQL TimeSeriesDB - Multidimensional data model Exporter Grafana Pulls Queries Alert Manager Instrum entation Pulls Dashboard
  • 4. Prometheus - TSDB - Open Source - Incubated by CNCF (After Kubernetes) - Adapted to VM/containers monitoring - Autodiscovery - Pull model - Multidimensional data - Includes alerting
  • 5. Grafana - OS metric analytics / visualisation - multiple providers: CloudWatch, Prometheus, InfluxDb, ES, .. - multiple dashboards already available - in coop with Prometheus exporters
  • 6. Node exporter + Grafana dashboard
  • 7. MongoDB exporter + Grafana dashboard
  • 8. Case study: RR Stats import ● Metric: Duration of execution Labels ● Result ○ success/failure ● Source ○ Google Ads, Fb Ads, Bing Ads, Taboola, etc. ● Category ○ Account, Campaign, Keyword, .. ○ Today vs Past ● Node
  • 10. 1. Debugging / Gain insight "Where does the problem come from / What is going on?" ● Segment by sources (Google Ads, Fb Ads, Bing Ads, Taboola, etc.) ○ Did they slow down? Error rate gone up? Are they unavailable? ● Segment by category ○ Did we introduce a bug on that code? ● Segment by node ○ do I have a problem on that node?
  • 11. All successful stats downloads
  • 12. All successful stats downloads - vs Bing
  • 13. 1. Debugging / Gain insight Combination with external data / corroboration - deployments - CPU/Ram/Load on the node - “can we corroborate with a slow query increase in Mongodb?”
  • 14. Example: Sync activity vs machine load
  • 15. 2. Alerting - Grafana alerts: - alerts based on configured data sources - Prometheus AlertManager: - can alert based on PromQL query - Infrastructure as Code Instrument now, decide later
  • 16. 2. Alerting - Example
  • 17. 2. Alerting - Graph
  • 18. 3. Trends / Scale ● Trends over time, drive scale (technical) / business decisions ○ Capacity planning ○ "Will I (when will I) have a problem in the future?" ● SLA / QoS
  • 19. And all this is available thanks to this code:
  • 20. Push (vs pull) - Async, short-lived processes - The prometheus way => send metrics to a push gateway - One push gateway per process ! - More infrastructure to setup - Our way, the prometheus-distributed-client => send metrics to a database - Available from everywhere - Consistent in case of concurrent calls - Use either
  • 21. Conclusion - Try to always instrument your code - Limite the cardinality of the metrics you use - Make nice graphs ! - Use Our lib : https://github.com/dolead/prometheus-distributed-client