SlideShare une entreprise Scribd logo
1  sur  32
Télécharger pour lire hors ligne
The Power of Metrics, Logs &
Traces with Open Source
● Loves running
● Baking sourdough bread
https: //www.linkedin.com/in/emil-andreas-siemes-a793926/
Emil Siemes
Principal Solutions Engineer
1000+ Employees across
40+ countries
(50+ in Germany)
1M+
Instances across
LGTM Cloud and
OSS
13M+ Users across OSS
and Cloud Free tier
Founded in 2014
We help our customers Build and Operate
resilient systems,
to provide reliable services to their
customers.
Investing in the core of observability
Prometheus
Largest contributor - employs
44% of Prometheus
maintainers
Open source is at the of what we do
GRAFANA
Vizualize, Correlate & Alert
on Multi-sourced Data
Grafana LOKI
Highly Scalable Log
Aggregation
Grafana MIMIR
Scalable & Performant
backend for all
time-series metrics
Graphite
Leading contributor on
Graphite
Grafana TEMPO
Highly Scalable &
Cost Effective Tracing
OpenTelemetry
Large contributor - employs
maintainters; first class
support in Grafana
K6
Modern load testing
built for developers
Grafana On-Call
Oncall management
system with simple
workflow integrated with
your telemetry
Prometheus
Largest contributor - employs
44% of Prometheus
maintainers
Open source is at the of what we do
GRAFANA
Vizualize, Correlate & Alert
on Multi-sourced Data
Grafana LOKI
Highly Scalable Log
Aggregation
Grafana MIMIR
Scalable & Performant
backend for all
time-series metrics
Graphite
Leading contributor on
Graphite
Grafana TEMPO
Highly Scalable &
Cost Effective Tracing
OpenTelemetry
Large contributor - employs
maintainters; first class
support in Grafana
K6
Modern load testing
built for developers
Grafana On-Call
Oncall management
system with simple
workflow integrated with
your telemetry
Deployment Flexibility
Open-source | Enterprise | Grafana Cloud
Big tent
Flexibility and openness without limits to integrate 3rd
party data
Observability by open source
A modern, complete and highly modulable platform fully
built on OSS
A Unique Philosophy : Regain Control Over Your
Observability Strategy
LGTM: A prometheus inspired O11y stack
Tempo
Backend compatible
with OpenTelemtry,
inexpensive at scale,
100% sampling is
possible
Loki
Low TCO, Highly
scalable and easy to
operate logging
system
Mimir
Highly scalable, highly
available multi-tenant,
durable storage
drop-in replacement
for Prometheus
Correlation by design to reduce MTTR
Exemplars
Labels & service discovery
{service="foo", region="eu-west",
instance="node-123"}
Auto-generated
metrics
Logs for trace
TraceID in logs
Logs to metrics
extraction
S3
Frontend application
observability
Frontend
Backend
infrastructure
An open source web SDK for
frontend application observability
Frontend Application Observability
in Grafana Cloud
Fully managed offering for capturing, visualizing,
and analyzing frontend application telemetry data within
Grafana Cloud, powered by Grafana Faro
PRIVATE BETA available today for Grafana Cloud customers
Getting started with Grafana Faro is super simple
Install the Grafana Faro Web SDK:
Import it into your app:
npm install @grafana/faro-web-sdk
import { initializeFaro } from '@grafana/faro-web-sdk'
initializeFaro({
url: '<http://my-endpoint>',
apiKey: '<api_key>',
app: { name: 'my-app' },
});
Loki
Out-of-order ingestion
Simplified deployment
Promtail Kafka support
Regex performance improvements
Parallelized binary ops in queries
Improved tail latencies with request hedging
Promtail Cloudflare, Docker, and Graylog support
Query multiple tenants at once
Faster instant queries
Filter & delete certain log lines
July 2022
April 2022
Nov. 2021
Loki v2.4.0
Loki v2.5.0
Loki v2.6.0
Loki v2.7.0
Coming soon
50%
Less CPU
4x
Faster queries
COMING SOON
Node #1
Node #2 Loki
Promtail
Promtail
Grafana
logcli
AlertManager
REST
API
Object
Storage
Overview
Node #n
Promtail
@160014567432
Timestamp Value
22346
Labels/Selectors
key-value pairs
{app=”nginx”, env=”dev”}
Metric name
http_request
The Prometheus storage model
2019-12-11T10:01:02.123456789Z {app=”nginx”, env=”dev”}
Timestamp
with nanosecond precision Content log line
Labels/Selectors
key-value pairs
indexed unindexed
GET /about 1034 Debug “page not found”
Object
Storage
Object
Storage
Aka “the index” Aka “chunks”
The Loki storage model
● Log any and all formats
● Smaller indexes
● Cheaper to run
● Fast queries
● Cut and slice your logs in dynamic ways
Grafana Loki
Query time processing
VS
● Decide on log formats
● Larger indexes
● More expensive to run
● Faster queries
● Restricted to format chosen at ingestion time
Traditional solutions
Ingest time processing
The better tradeoff
10TB 200MB
Log Data Index
Think of it more like a table of contents than an index.
Loki does not index the contents of the logs. Instead, entries are
grouped into streams and indexed with Prometheus-style labels.
Efficient log storage
1PB 80TB
Raw Logs Label match Brute force search -
heavily parallelized
Time frame
1TB 120GB+/s
Fast queries
Think of it as massively distributed grep.
Per Second A day A month Active tenants
700MB 43TB 1.25PB 4500
170Cores 1TB RAM
27
Large cluster in Grafana Cloud
{cluster=”us-central1”, job=~”dev/loki-.*”} |= “trace_id=7ca877dbda” |~ "SeriesStore.*"
Label matchers
● = contains string.
● != does not contain string.
● =~ matches regular expression.
● !~ does not match regular expression.
Filter expressions
● |= contains string.
● != does not contain string.
● |~ matches regular expression.
● !~ does not match regular expression.
LogQL: Log queries
Parser expressions
Formatter expressions
sum by (host) (rate({job="mysql"} |= "error" != "timeout" | json | duration > 10s [1m]))
Range vector aggregations
● rate() / bytes_rate()
● count_over_time() / bytes_over_time() /
absent_over_time()
● sum_over_time() / avg_over_time() /
min_over_time() / max_over_time () *
Instant vector aggregations
● sum() / min() / max() / avg()
● stddev() / stdvar()
● count()
● topk() / bottomk()
* Unwrapped range aggregations
LogQL: Metric queries
Grafana Agent
Promtail
● Targets discovery for Kubernetes, Syslog, files and more
● Automatically attach labels to your log lines
● Advanced pipeline mechanism for parsing, transforming and
filtering your logs
● Build and expose custom metrics from your logs data
But Loki is open.
logstash Lambda
Loki makes log collection easy
Single Binary
- Testing
- Small installations
without HA or
horizontal scaling
needs
SSD or
Microservices
- Horizontal scalability
- Separate Read/Write
paths
- Large installations
Grafana Cloud
● Batteries included:
○ Multi-tenant
○ Automatic scaling
and upgrades
○ RBAC
○ Per tenant QoS
Enterprise Logs
● Self hosted
● Supported by Grafana
Labs
● Security out of the box
● Part of the GES stack
31
How to run Loki
Thank you!

Contenu connexe

Tendances

Observability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetryObservability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetry
Eric D. Schabell
 
Monitoring_with_Prometheus_Grafana_Tutorial
Monitoring_with_Prometheus_Grafana_TutorialMonitoring_with_Prometheus_Grafana_Tutorial
Monitoring_with_Prometheus_Grafana_Tutorial
Tim Vaillancourt
 

Tendances (20)

stackconf 2022: Open Source for Better Observability
stackconf 2022: Open Source for Better Observabilitystackconf 2022: Open Source for Better Observability
stackconf 2022: Open Source for Better Observability
 
Grafana Loki: like Prometheus, but for Logs
Grafana Loki: like Prometheus, but for LogsGrafana Loki: like Prometheus, but for Logs
Grafana Loki: like Prometheus, but for Logs
 
Log management with ELK
Log management with ELKLog management with ELK
Log management with ELK
 
Elk
Elk Elk
Elk
 
Grafana
GrafanaGrafana
Grafana
 
Cloud Monitoring tool Grafana
Cloud Monitoring  tool Grafana Cloud Monitoring  tool Grafana
Cloud Monitoring tool Grafana
 
Understand your system like never before with OpenTelemetry, Grafana, and Pro...
Understand your system like never before with OpenTelemetry, Grafana, and Pro...Understand your system like never before with OpenTelemetry, Grafana, and Pro...
Understand your system like never before with OpenTelemetry, Grafana, and Pro...
 
Distributed tracing using open tracing &amp; jaeger 2
Distributed tracing using open tracing &amp; jaeger 2Distributed tracing using open tracing &amp; jaeger 2
Distributed tracing using open tracing &amp; jaeger 2
 
ELK Stack
ELK StackELK Stack
ELK Stack
 
Observability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetryObservability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetry
 
Apache Spark Streaming in K8s with ArgoCD & Spark Operator
Apache Spark Streaming in K8s with ArgoCD & Spark OperatorApache Spark Streaming in K8s with ArgoCD & Spark Operator
Apache Spark Streaming in K8s with ArgoCD & Spark Operator
 
Monitoring_with_Prometheus_Grafana_Tutorial
Monitoring_with_Prometheus_Grafana_TutorialMonitoring_with_Prometheus_Grafana_Tutorial
Monitoring_with_Prometheus_Grafana_Tutorial
 
OSMC 2023 | What’s new with Grafana Labs’s Open Source Observability stack by...
OSMC 2023 | What’s new with Grafana Labs’s Open Source Observability stack by...OSMC 2023 | What’s new with Grafana Labs’s Open Source Observability stack by...
OSMC 2023 | What’s new with Grafana Labs’s Open Source Observability stack by...
 
Apache Knox setup and hive and hdfs Access using KNOX
Apache Knox setup and hive and hdfs Access using KNOXApache Knox setup and hive and hdfs Access using KNOX
Apache Knox setup and hive and hdfs Access using KNOX
 
Monitoring using Prometheus and Grafana
Monitoring using Prometheus and GrafanaMonitoring using Prometheus and Grafana
Monitoring using Prometheus and Grafana
 
OSMC 2022 | VictoriaMetrics: scaling to 100 million metrics per second by Ali...
OSMC 2022 | VictoriaMetrics: scaling to 100 million metrics per second by Ali...OSMC 2022 | VictoriaMetrics: scaling to 100 million metrics per second by Ali...
OSMC 2022 | VictoriaMetrics: scaling to 100 million metrics per second by Ali...
 
OpenTelemetry For Operators
OpenTelemetry For OperatorsOpenTelemetry For Operators
OpenTelemetry For Operators
 
Observability, Distributed Tracing, and Open Source: The Missing Primer
Observability, Distributed Tracing, and Open Source: The Missing PrimerObservability, Distributed Tracing, and Open Source: The Missing Primer
Observability, Distributed Tracing, and Open Source: The Missing Primer
 
MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)
 
Grafana introduction
Grafana introductionGrafana introduction
Grafana introduction
 

Similaire à OSMC 2022 | The Power of Metrics, Logs & Traces with Open Source by Emil-Andreas Siemes

Grafana overview deck - Tech - 2023 May v1.pdf
Grafana overview deck  - Tech - 2023 May v1.pdfGrafana overview deck  - Tech - 2023 May v1.pdf
Grafana overview deck - Tech - 2023 May v1.pdf
BillySin5
 
Spark Development Lifecycle at Workday - ApacheCon 2020
Spark Development Lifecycle at Workday - ApacheCon 2020Spark Development Lifecycle at Workday - ApacheCon 2020
Spark Development Lifecycle at Workday - ApacheCon 2020
Pavel Hardak
 
Apache Spark Development Lifecycle @ Workday - ApacheCon 2020
Apache Spark Development Lifecycle @ Workday - ApacheCon 2020Apache Spark Development Lifecycle @ Workday - ApacheCon 2020
Apache Spark Development Lifecycle @ Workday - ApacheCon 2020
Eren Avşaroğulları
 
Running Presto and Spark on the Netflix Big Data Platform
Running Presto and Spark on the Netflix Big Data PlatformRunning Presto and Spark on the Netflix Big Data Platform
Running Presto and Spark on the Netflix Big Data Platform
Eva Tse
 
Webinar september 2013
Webinar september 2013Webinar september 2013
Webinar september 2013
Marc Gille
 
Service Lifecycle Management with Fuse Service Works
Service Lifecycle Management with Fuse Service WorksService Lifecycle Management with Fuse Service Works
Service Lifecycle Management with Fuse Service Works
Kenneth Peeples
 

Similaire à OSMC 2022 | The Power of Metrics, Logs & Traces with Open Source by Emil-Andreas Siemes (20)

Grafana overview deck - Tech - 2023 May v1.pdf
Grafana overview deck  - Tech - 2023 May v1.pdfGrafana overview deck  - Tech - 2023 May v1.pdf
Grafana overview deck - Tech - 2023 May v1.pdf
 
A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)
A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)
A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)
 
Google Cloud Next '22 Recap: Serverless & Data edition
Google Cloud Next '22 Recap: Serverless & Data editionGoogle Cloud Next '22 Recap: Serverless & Data edition
Google Cloud Next '22 Recap: Serverless & Data edition
 
Spark Development Lifecycle at Workday - ApacheCon 2020
Spark Development Lifecycle at Workday - ApacheCon 2020Spark Development Lifecycle at Workday - ApacheCon 2020
Spark Development Lifecycle at Workday - ApacheCon 2020
 
Apache Spark Development Lifecycle @ Workday - ApacheCon 2020
Apache Spark Development Lifecycle @ Workday - ApacheCon 2020Apache Spark Development Lifecycle @ Workday - ApacheCon 2020
Apache Spark Development Lifecycle @ Workday - ApacheCon 2020
 
Scale Your Load Balancer from 0 to 1 million TPS on Azure
Scale Your Load Balancer from 0 to 1 million TPS on AzureScale Your Load Balancer from 0 to 1 million TPS on Azure
Scale Your Load Balancer from 0 to 1 million TPS on Azure
 
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
 
Running Presto and Spark on the Netflix Big Data Platform
Running Presto and Spark on the Netflix Big Data PlatformRunning Presto and Spark on the Netflix Big Data Platform
Running Presto and Spark on the Netflix Big Data Platform
 
(BDT303) Running Spark and Presto on the Netflix Big Data Platform
(BDT303) Running Spark and Presto on the Netflix Big Data Platform(BDT303) Running Spark and Presto on the Netflix Big Data Platform
(BDT303) Running Spark and Presto on the Netflix Big Data Platform
 
Coherence RoadMap 2018
Coherence RoadMap 2018Coherence RoadMap 2018
Coherence RoadMap 2018
 
Cloud Native Applications on OpenShift
Cloud Native Applications on OpenShiftCloud Native Applications on OpenShift
Cloud Native Applications on OpenShift
 
DevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
DevOpsDays Taipei 2019 - Mastering IaC the DevOps WayDevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
DevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
 
Webinar september 2013
Webinar september 2013Webinar september 2013
Webinar september 2013
 
Rails App performance at the limit - Bogdan Gusiev
Rails App performance at the limit - Bogdan GusievRails App performance at the limit - Bogdan Gusiev
Rails App performance at the limit - Bogdan Gusiev
 
How easy (or hard) it is to monitor your graph ql service performance
How easy (or hard) it is to monitor your graph ql service performanceHow easy (or hard) it is to monitor your graph ql service performance
How easy (or hard) it is to monitor your graph ql service performance
 
PostgreSQL
PostgreSQL PostgreSQL
PostgreSQL
 
Service Lifecycle Management with Fuse Service Works
Service Lifecycle Management with Fuse Service WorksService Lifecycle Management with Fuse Service Works
Service Lifecycle Management with Fuse Service Works
 
PostgreSQL
PostgreSQLPostgreSQL
PostgreSQL
 
Scaling up uber's real time data analytics
Scaling up uber's real time data analyticsScaling up uber's real time data analytics
Scaling up uber's real time data analytics
 
Vijendra_resume
Vijendra_resume Vijendra_resume
Vijendra_resume
 

Dernier

Abortion Pill Prices Jane Furse ](+27832195400*)[🏥Women's Abortion Clinic in ...
Abortion Pill Prices Jane Furse ](+27832195400*)[🏥Women's Abortion Clinic in ...Abortion Pill Prices Jane Furse ](+27832195400*)[🏥Women's Abortion Clinic in ...
Abortion Pill Prices Jane Furse ](+27832195400*)[🏥Women's Abortion Clinic in ...
Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg
 
Abortion Clinic in Midrand [(+27832195400*)]🏥Safe Abortion Pills In Midrand |...
Abortion Clinic in Midrand [(+27832195400*)]🏥Safe Abortion Pills In Midrand |...Abortion Clinic in Midrand [(+27832195400*)]🏥Safe Abortion Pills In Midrand |...
Abortion Clinic in Midrand [(+27832195400*)]🏥Safe Abortion Pills In Midrand |...
Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Lisi Hocke
 

Dernier (20)

Abortion Pill Prices Jane Furse ](+27832195400*)[🏥Women's Abortion Clinic in ...
Abortion Pill Prices Jane Furse ](+27832195400*)[🏥Women's Abortion Clinic in ...Abortion Pill Prices Jane Furse ](+27832195400*)[🏥Women's Abortion Clinic in ...
Abortion Pill Prices Jane Furse ](+27832195400*)[🏥Women's Abortion Clinic in ...
 
微信号购买
微信号购买微信号购买
微信号购买
 
Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea Goulet
 
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jGraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
 
Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
Workshop -  Architecting Innovative Graph Applications- GraphSummit MilanWorkshop -  Architecting Innovative Graph Applications- GraphSummit Milan
Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
 
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
 
Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024
 
What Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the SituationWhat Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the Situation
 
Lessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdfLessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdf
 
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaUNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
 
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
 
Test Automation Design Patterns_ A Comprehensive Guide.pdf
Test Automation Design Patterns_ A Comprehensive Guide.pdfTest Automation Design Patterns_ A Comprehensive Guide.pdf
Test Automation Design Patterns_ A Comprehensive Guide.pdf
 
Your Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | EvmuxYour Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | Evmux
 
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
 
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
 
Abortion Pill Prices Aliwal North ](+27832195400*)[ 🏥 Women's Abortion Clinic...
Abortion Pill Prices Aliwal North ](+27832195400*)[ 🏥 Women's Abortion Clinic...Abortion Pill Prices Aliwal North ](+27832195400*)[ 🏥 Women's Abortion Clinic...
Abortion Pill Prices Aliwal North ](+27832195400*)[ 🏥 Women's Abortion Clinic...
 
Abortion Clinic in Midrand [(+27832195400*)]🏥Safe Abortion Pills In Midrand |...
Abortion Clinic in Midrand [(+27832195400*)]🏥Safe Abortion Pills In Midrand |...Abortion Clinic in Midrand [(+27832195400*)]🏥Safe Abortion Pills In Midrand |...
Abortion Clinic in Midrand [(+27832195400*)]🏥Safe Abortion Pills In Midrand |...
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
 
The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)
 
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
 

OSMC 2022 | The Power of Metrics, Logs & Traces with Open Source by Emil-Andreas Siemes

  • 1. The Power of Metrics, Logs & Traces with Open Source
  • 2. ● Loves running ● Baking sourdough bread https: //www.linkedin.com/in/emil-andreas-siemes-a793926/ Emil Siemes Principal Solutions Engineer
  • 3. 1000+ Employees across 40+ countries (50+ in Germany) 1M+ Instances across LGTM Cloud and OSS 13M+ Users across OSS and Cloud Free tier Founded in 2014
  • 4. We help our customers Build and Operate resilient systems, to provide reliable services to their customers.
  • 5. Investing in the core of observability
  • 6. Prometheus Largest contributor - employs 44% of Prometheus maintainers Open source is at the of what we do GRAFANA Vizualize, Correlate & Alert on Multi-sourced Data Grafana LOKI Highly Scalable Log Aggregation Grafana MIMIR Scalable & Performant backend for all time-series metrics Graphite Leading contributor on Graphite Grafana TEMPO Highly Scalable & Cost Effective Tracing OpenTelemetry Large contributor - employs maintainters; first class support in Grafana K6 Modern load testing built for developers Grafana On-Call Oncall management system with simple workflow integrated with your telemetry
  • 7. Prometheus Largest contributor - employs 44% of Prometheus maintainers Open source is at the of what we do GRAFANA Vizualize, Correlate & Alert on Multi-sourced Data Grafana LOKI Highly Scalable Log Aggregation Grafana MIMIR Scalable & Performant backend for all time-series metrics Graphite Leading contributor on Graphite Grafana TEMPO Highly Scalable & Cost Effective Tracing OpenTelemetry Large contributor - employs maintainters; first class support in Grafana K6 Modern load testing built for developers Grafana On-Call Oncall management system with simple workflow integrated with your telemetry
  • 8. Deployment Flexibility Open-source | Enterprise | Grafana Cloud Big tent Flexibility and openness without limits to integrate 3rd party data Observability by open source A modern, complete and highly modulable platform fully built on OSS A Unique Philosophy : Regain Control Over Your Observability Strategy
  • 9. LGTM: A prometheus inspired O11y stack Tempo Backend compatible with OpenTelemtry, inexpensive at scale, 100% sampling is possible Loki Low TCO, Highly scalable and easy to operate logging system Mimir Highly scalable, highly available multi-tenant, durable storage drop-in replacement for Prometheus
  • 10. Correlation by design to reduce MTTR Exemplars Labels & service discovery {service="foo", region="eu-west", instance="node-123"} Auto-generated metrics Logs for trace TraceID in logs Logs to metrics extraction S3
  • 13. An open source web SDK for frontend application observability
  • 14.
  • 15. Frontend Application Observability in Grafana Cloud Fully managed offering for capturing, visualizing, and analyzing frontend application telemetry data within Grafana Cloud, powered by Grafana Faro PRIVATE BETA available today for Grafana Cloud customers
  • 16. Getting started with Grafana Faro is super simple Install the Grafana Faro Web SDK: Import it into your app: npm install @grafana/faro-web-sdk import { initializeFaro } from '@grafana/faro-web-sdk' initializeFaro({ url: '<http://my-endpoint>', apiKey: '<api_key>', app: { name: 'my-app' }, });
  • 17.
  • 18. Loki
  • 19. Out-of-order ingestion Simplified deployment Promtail Kafka support Regex performance improvements Parallelized binary ops in queries Improved tail latencies with request hedging Promtail Cloudflare, Docker, and Graylog support Query multiple tenants at once Faster instant queries Filter & delete certain log lines July 2022 April 2022 Nov. 2021 Loki v2.4.0 Loki v2.5.0 Loki v2.6.0 Loki v2.7.0 Coming soon
  • 21. Node #1 Node #2 Loki Promtail Promtail Grafana logcli AlertManager REST API Object Storage Overview Node #n Promtail
  • 22. @160014567432 Timestamp Value 22346 Labels/Selectors key-value pairs {app=”nginx”, env=”dev”} Metric name http_request The Prometheus storage model
  • 23. 2019-12-11T10:01:02.123456789Z {app=”nginx”, env=”dev”} Timestamp with nanosecond precision Content log line Labels/Selectors key-value pairs indexed unindexed GET /about 1034 Debug “page not found” Object Storage Object Storage Aka “the index” Aka “chunks” The Loki storage model
  • 24. ● Log any and all formats ● Smaller indexes ● Cheaper to run ● Fast queries ● Cut and slice your logs in dynamic ways Grafana Loki Query time processing VS ● Decide on log formats ● Larger indexes ● More expensive to run ● Faster queries ● Restricted to format chosen at ingestion time Traditional solutions Ingest time processing The better tradeoff
  • 25. 10TB 200MB Log Data Index Think of it more like a table of contents than an index. Loki does not index the contents of the logs. Instead, entries are grouped into streams and indexed with Prometheus-style labels. Efficient log storage
  • 26. 1PB 80TB Raw Logs Label match Brute force search - heavily parallelized Time frame 1TB 120GB+/s Fast queries Think of it as massively distributed grep.
  • 27. Per Second A day A month Active tenants 700MB 43TB 1.25PB 4500 170Cores 1TB RAM 27 Large cluster in Grafana Cloud
  • 28. {cluster=”us-central1”, job=~”dev/loki-.*”} |= “trace_id=7ca877dbda” |~ "SeriesStore.*" Label matchers ● = contains string. ● != does not contain string. ● =~ matches regular expression. ● !~ does not match regular expression. Filter expressions ● |= contains string. ● != does not contain string. ● |~ matches regular expression. ● !~ does not match regular expression. LogQL: Log queries Parser expressions Formatter expressions
  • 29. sum by (host) (rate({job="mysql"} |= "error" != "timeout" | json | duration > 10s [1m])) Range vector aggregations ● rate() / bytes_rate() ● count_over_time() / bytes_over_time() / absent_over_time() ● sum_over_time() / avg_over_time() / min_over_time() / max_over_time () * Instant vector aggregations ● sum() / min() / max() / avg() ● stddev() / stdvar() ● count() ● topk() / bottomk() * Unwrapped range aggregations LogQL: Metric queries
  • 30. Grafana Agent Promtail ● Targets discovery for Kubernetes, Syslog, files and more ● Automatically attach labels to your log lines ● Advanced pipeline mechanism for parsing, transforming and filtering your logs ● Build and expose custom metrics from your logs data But Loki is open. logstash Lambda Loki makes log collection easy
  • 31. Single Binary - Testing - Small installations without HA or horizontal scaling needs SSD or Microservices - Horizontal scalability - Separate Read/Write paths - Large installations Grafana Cloud ● Batteries included: ○ Multi-tenant ○ Automatic scaling and upgrades ○ RBAC ○ Per tenant QoS Enterprise Logs ● Self hosted ● Supported by Grafana Labs ● Security out of the box ● Part of the GES stack 31 How to run Loki