Saiba como o Elasticsearch combina com eficiência dados em um único armazenamento e como o Kibana é usado para analisá-los. Além disso, veja como os desenvolvimentos recentes ajudam a identificar e resolver problemas operacionais mais rapidamente.

1. Gabriel Moskovicz
Solution Architect
April 15th
Logs, Metrics, and APM for
Unified Observability

2. Elastic Enterprise Search Elastic SecurityElastic
Observability

3. Logs Metrics APM Uptime
Elastic Observability

4. 4
#ObservaBLT
Observability
=
+
+

6. Higher resource utilization
increases monitoring complexity
• Orchestration/Hypervisor
• Dynamic/ephemeral jobs
• You can no longer "point" to where
that job lives
Shift to cloud-native yields
maintainable code, with costs
• Traditional licensing models don't
scale as well as your applications
• Hurdles with autoscaling
Monitoring Complexity
Hardware & software trends are evolving in tandem
Evolving Architectures ~↑ Monitoring Complexity

8. Applications VMs/Containers
Other DBs,
Services &
Middleware
Orchestration InfrastructureAPM
Metrics
Logs
Uptime
Uptime
APM Metrics
APM Logs
APM
APM
Metrics
Logs
Uptime
Metrics
Logs
Uptime
APM

9. Development
Team
Ops: Log
Monitoring
Uptime
Response Time
Uptime Tool
Ops: Infra
Monitoring
Web Logs
App Logs
Database Logs
Container Logs
Log Tool
Ops: Service
Monitoring
Real User Monitoring
Txn Perf Monitoring
Distributed Tracing
APM Tool
Container Metrics
Host Metrics
Database Metics
Network Metrics
Storage Metrics
Metrics Tool
Status Quo: Siloed Collection of Tools

10. Observability is a search use case

11. APM Data Uptime DataMetrics DataLog Data
Elastic Approach to Observability
Uptime
Response Time
Web Logs
App Logs
Database Logs
Container Logs
Real User Monitoring
Txn Perf Monitoring
Distributed Tracing
Container Metrics
Host Metrics
Database Metics
Network Metrics
Storage Metrics
Dev & Ops Teams

12. Unified User Interface
Same UI for KPI summaries and root-cause analysis

13. Correlate multiple data sources for more intelligent anomaly detection
Unified Machine Learning

14. Trigger off any operational data to provide unified SLA monitoring
Unified Alerting

15. Pricing aligned with business value
Unified Licensing Model
PER
AGENT
$$$$
PER
HOST
$$$$
PER
INGEST
$$$$
PER
MONITOR
$$$$
PER
ADD-ON
$$$$
• Intuitive
Single, unified pricing model. No add-ons.
• Cloud native
No problem using with container workloads and serverless.
• Future proof
You pay for capacity and are not locked into a specific use case.

16. Elastic Stack for logs

17. Logs
64.242.88.10 - - [07/Jan/2019:16:10:02 -0800] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291
64.242.88.10 - - [07/Jan/2019:16:10:02 -0800] "POST /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 404 7352
64.242.88.10 - - [07/Jan/2019:16:10:02 -0800] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200 5253
For each event, print out what happened.
Logs are chronological records of events

18. Ongoing investment in log ingest & long-term retention
2015
2016
Hosted Logging in Elastic Cloud & ECE
Introduction of ECE enabling log clusters with index
curation, hot/warm templates
2018
2017
Cold storage for logging: Frozen Indices & ILM
Curated log-based troubleshooting, improved
cold storage efficiency and index lifecycle
management
2019
Modules: Out-of-the-box log parsers
Simplified ingest architecture with Filebeat
modules & ingest node
ELK Stack is born
Logstash and Kibana released, forming an
OSS logging alternative
2011-12
Logs UI: Integrating Logs with Metrics and APM
Logging libraries support Elastic Common Schema,
trace-id in logs, workflow from Logs to APM
Elastic welcomes Beats to the family,
introducing light-weight data shippers
Filebeat: Lightweight log shipper

19. Elastic Stack for metrics

20. Metrics vs Logs
Metrics are periodic measurement of numeric KPIs
07/Jan/2019 16:10:00 all 2.58 0.00 0.70 1.12 0.05 95.55 server1 containerX regionA
07/Jan/2019 16:20:00 all 2.56 0.00 0.69 1.05 0.04 95.66 server2 containerY regionB
07/Jan/2019 16:30:00 all 2.64 0.00 0.65 1.15 0.05 95.50 server2 containerZ regionC
Every x minutes, measure the CPU load, print it out, and annotate with meta-data.
64.242.88.10 - - [07/Jan/2019:16:10:02 -0800] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291
64.242.88.10 - - [07/Jan/2019:16:10:02 -0800] "POST /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 404 7352
64.242.88.10 - - [07/Jan/2019:16:10:02 -0800] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200 5253
For each event, print out what happened.
Logs are chronological records of events

21. Evolution of Elastic Stack to a Metrics Store
BKD trees
Data structures optimized for numerical time
series analysis.
Columnar storage
Structured data storage, resulting in compact
storage and faster analytics
Rollups
Aggregate older data into bigger time buckets
Aggregations framework
Analytics features to slice and dice data along
various dimensions
2012
2016
2014
2018

22. Elastic as an Infrastructure Metrics Solution
201?
2017
Users start putting metrics in Elastic
Need for high-cardinality aggregations, and
correlating metrics and logs
2016
2018
2019

23. Elastic as an Infrastructure Metrics Solution
201?
2017
Users start putting metrics in Elastic
Need for high-cardinality aggregations, and
correlating metrics and logs
2016
2018
2019
Metricbeat: Turnkey metric collection
Metricbeat is introduced for turnkey metrics
collection

24. Elastic as an Infrastructure Metrics Solution
201?
2017
Users start putting metrics in Elastic
Need for high-cardinality aggregations, and
correlating metrics and logs
2016
2018
2019
Metricbeat: Turnkey metric collection
Metricbeat is introduced for turnkey metrics
collection
Time Series Visual Builder
UI for advanced metrics visualization, working
with pipeline aggregations

25. Elastic as an Infrastructure Metrics Solution
201?
2017
Users start putting metrics in Elastic
Need for high-cardinality aggregations, and
correlating metrics and logs
2016
2018
2019
Metricbeat: Turnkey metric collection
Metricbeat is introduced for turnkey metrics
collection
Time Series Visual Builder
UI for advanced metrics visualization, working
with pipeline aggregations
Prometheus / OpenMetrics integration
Enables turnkey collection in Kubernetes
ecosystem and beyond
Your App
Prometheus
Exporter
Your App
Prometheus
Exporter
Metricbeat +
Elasticsearch
Prometheus
Server
Metricbeat +
Elasticsearch

26. Elastic as an Infrastructure Metrics Solution
201?
2017
Users start putting metrics in Elastic
Need for high-cardinality aggregations, and
correlating metrics and logs
2016
2018
2019
Metricbeat: Turnkey metric collection
Metricbeat is introduced for turnkey metrics
collection
Time Series Visual Builder
UI for advanced metrics visualization, working
with pipeline aggregations
Prometheus / OpenMetrics integration
Enables turnkey collection in Kubernetes
ecosystem and beyond
Infrastructure Metrics UI
Containers, hosts, services, cloud monitoring,
ad-hoc metrics exploration

27. Elastic Stack for APM

28. 28
Evolution of Elastic Stack to Open Source APM
Elastic joins forces with Opbeat
A next-generation APM solution designed for
developers
2017
6.1
Search for APM + more agents
Enabled search & Machine Learning for APM,
Java agents GA, RUM GA
6.4
Elastic APM beta release
Including APM Server and curated APM UI
native to Kibana
6.2
Support for open tracing enabled with
Distributed tracing, added Go Agent,
integrated UI with Logs & Metrics
6.6
Elastic APM GA
Agents for Python, Node.js, Ruby, Javascript;
Real User Monitoring
Beyond

29. APM Agents
• Easy to add to your applications
• Multiple Language Support
• Designed to be lightweight
• Open source
• Support distributed tracing
• OpenTracing compatible
Auto-instrumentation of common programming frameworks

31. 31
Elastic Stack for uptime

32. Heartbeat: Uptime Monitoring

34. • Correlate data from different sources
• Ability to re-use analysis content
• Ability to re-use Elastic-provided content
Benefits
• Published at: github.com/elastic/ecs
• Supported in Beats and APM since 7.0
• Community feedback welcome!
Status
Elastic Common Schema (ECS)
Supports ad-hoc analysis in Kibana Dashboards

35. 35
Demo

36. What now?
Try it yourself!

37. While you monitor, why not protect?
Elastic SIEM

38. Come to the AMA booth!
Questions?