Ce diaporama a bien été signalé.
Le téléchargement de votre SlideShare est en cours. ×

OSMC 2022 | The Power of Metrics, Logs & Traces with Open Source by Emil-Andreas Siemes

Publicité
Publicité
Publicité
Publicité
Publicité
Publicité
Publicité
Publicité
Publicité
Publicité

Consultez-les par la suite

1 sur 32 Publicité

OSMC 2022 | The Power of Metrics, Logs & Traces with Open Source by Emil-Andreas Siemes

Télécharger pour lire hors ligne

The talk will show how organisations can drastically reduce their MTTR (Mean Time To Repair) by using, integrating & correlating the open source tools Mimir, Loki & Tempo. We will then take the next step into open source reliability testing to even avoid problems in the first place. And yes, we will use Grafana 🙂

The talk will show how organisations can drastically reduce their MTTR (Mean Time To Repair) by using, integrating & correlating the open source tools Mimir, Loki & Tempo. We will then take the next step into open source reliability testing to even avoid problems in the first place. And yes, we will use Grafana 🙂

Publicité
Publicité

Plus De Contenu Connexe

Publicité

OSMC 2022 | The Power of Metrics, Logs & Traces with Open Source by Emil-Andreas Siemes

  1. 1. The Power of Metrics, Logs & Traces with Open Source
  2. 2. ● Loves running ● Baking sourdough bread https: //www.linkedin.com/in/emil-andreas-siemes-a793926/ Emil Siemes Principal Solutions Engineer
  3. 3. 1000+ Employees across 40+ countries (50+ in Germany) 1M+ Instances across LGTM Cloud and OSS 13M+ Users across OSS and Cloud Free tier Founded in 2014
  4. 4. We help our customers Build and Operate resilient systems, to provide reliable services to their customers.
  5. 5. Investing in the core of observability
  6. 6. Prometheus Largest contributor - employs 44% of Prometheus maintainers Open source is at the of what we do GRAFANA Vizualize, Correlate & Alert on Multi-sourced Data Grafana LOKI Highly Scalable Log Aggregation Grafana MIMIR Scalable & Performant backend for all time-series metrics Graphite Leading contributor on Graphite Grafana TEMPO Highly Scalable & Cost Effective Tracing OpenTelemetry Large contributor - employs maintainters; first class support in Grafana K6 Modern load testing built for developers Grafana On-Call Oncall management system with simple workflow integrated with your telemetry
  7. 7. Prometheus Largest contributor - employs 44% of Prometheus maintainers Open source is at the of what we do GRAFANA Vizualize, Correlate & Alert on Multi-sourced Data Grafana LOKI Highly Scalable Log Aggregation Grafana MIMIR Scalable & Performant backend for all time-series metrics Graphite Leading contributor on Graphite Grafana TEMPO Highly Scalable & Cost Effective Tracing OpenTelemetry Large contributor - employs maintainters; first class support in Grafana K6 Modern load testing built for developers Grafana On-Call Oncall management system with simple workflow integrated with your telemetry
  8. 8. Deployment Flexibility Open-source | Enterprise | Grafana Cloud Big tent Flexibility and openness without limits to integrate 3rd party data Observability by open source A modern, complete and highly modulable platform fully built on OSS A Unique Philosophy : Regain Control Over Your Observability Strategy
  9. 9. LGTM: A prometheus inspired O11y stack Tempo Backend compatible with OpenTelemtry, inexpensive at scale, 100% sampling is possible Loki Low TCO, Highly scalable and easy to operate logging system Mimir Highly scalable, highly available multi-tenant, durable storage drop-in replacement for Prometheus
  10. 10. Correlation by design to reduce MTTR Exemplars Labels & service discovery {service="foo", region="eu-west", instance="node-123"} Auto-generated metrics Logs for trace TraceID in logs Logs to metrics extraction S3
  11. 11. Frontend application observability
  12. 12. Frontend Backend infrastructure
  13. 13. An open source web SDK for frontend application observability
  14. 14. Frontend Application Observability in Grafana Cloud Fully managed offering for capturing, visualizing, and analyzing frontend application telemetry data within Grafana Cloud, powered by Grafana Faro PRIVATE BETA available today for Grafana Cloud customers
  15. 15. Getting started with Grafana Faro is super simple Install the Grafana Faro Web SDK: Import it into your app: npm install @grafana/faro-web-sdk import { initializeFaro } from '@grafana/faro-web-sdk' initializeFaro({ url: '<http://my-endpoint>', apiKey: '<api_key>', app: { name: 'my-app' }, });
  16. 16. Loki
  17. 17. Out-of-order ingestion Simplified deployment Promtail Kafka support Regex performance improvements Parallelized binary ops in queries Improved tail latencies with request hedging Promtail Cloudflare, Docker, and Graylog support Query multiple tenants at once Faster instant queries Filter & delete certain log lines July 2022 April 2022 Nov. 2021 Loki v2.4.0 Loki v2.5.0 Loki v2.6.0 Loki v2.7.0 Coming soon
  18. 18. 50% Less CPU 4x Faster queries COMING SOON
  19. 19. Node #1 Node #2 Loki Promtail Promtail Grafana logcli AlertManager REST API Object Storage Overview Node #n Promtail
  20. 20. @160014567432 Timestamp Value 22346 Labels/Selectors key-value pairs {app=”nginx”, env=”dev”} Metric name http_request The Prometheus storage model
  21. 21. 2019-12-11T10:01:02.123456789Z {app=”nginx”, env=”dev”} Timestamp with nanosecond precision Content log line Labels/Selectors key-value pairs indexed unindexed GET /about 1034 Debug “page not found” Object Storage Object Storage Aka “the index” Aka “chunks” The Loki storage model
  22. 22. ● Log any and all formats ● Smaller indexes ● Cheaper to run ● Fast queries ● Cut and slice your logs in dynamic ways Grafana Loki Query time processing VS ● Decide on log formats ● Larger indexes ● More expensive to run ● Faster queries ● Restricted to format chosen at ingestion time Traditional solutions Ingest time processing The better tradeoff
  23. 23. 10TB 200MB Log Data Index Think of it more like a table of contents than an index. Loki does not index the contents of the logs. Instead, entries are grouped into streams and indexed with Prometheus-style labels. Efficient log storage
  24. 24. 1PB 80TB Raw Logs Label match Brute force search - heavily parallelized Time frame 1TB 120GB+/s Fast queries Think of it as massively distributed grep.
  25. 25. Per Second A day A month Active tenants 700MB 43TB 1.25PB 4500 170Cores 1TB RAM 27 Large cluster in Grafana Cloud
  26. 26. {cluster=”us-central1”, job=~”dev/loki-.*”} |= “trace_id=7ca877dbda” |~ "SeriesStore.*" Label matchers ● = contains string. ● != does not contain string. ● =~ matches regular expression. ● !~ does not match regular expression. Filter expressions ● |= contains string. ● != does not contain string. ● |~ matches regular expression. ● !~ does not match regular expression. LogQL: Log queries Parser expressions Formatter expressions
  27. 27. sum by (host) (rate({job="mysql"} |= "error" != "timeout" | json | duration > 10s [1m])) Range vector aggregations ● rate() / bytes_rate() ● count_over_time() / bytes_over_time() / absent_over_time() ● sum_over_time() / avg_over_time() / min_over_time() / max_over_time () * Instant vector aggregations ● sum() / min() / max() / avg() ● stddev() / stdvar() ● count() ● topk() / bottomk() * Unwrapped range aggregations LogQL: Metric queries
  28. 28. Grafana Agent Promtail ● Targets discovery for Kubernetes, Syslog, files and more ● Automatically attach labels to your log lines ● Advanced pipeline mechanism for parsing, transforming and filtering your logs ● Build and expose custom metrics from your logs data But Loki is open. logstash Lambda Loki makes log collection easy
  29. 29. Single Binary - Testing - Small installations without HA or horizontal scaling needs SSD or Microservices - Horizontal scalability - Separate Read/Write paths - Large installations Grafana Cloud ● Batteries included: ○ Multi-tenant ○ Automatic scaling and upgrades ○ RBAC ○ Per tenant QoS Enterprise Logs ● Self hosted ● Supported by Grafana Labs ● Security out of the box ● Part of the GES stack 31 How to run Loki
  30. 30. Thank you!

×