Recommended
More Related Content
What's hot
What's hot (20)
Similar to 2020 07-30 elastic agent + ingest management
Similar to 2020 07-30 elastic agent + ingest management (20)
More from Daliya Spasova
More from Daliya Spasova (17)
Recently uploaded
Recently uploaded (20)
2020 07-30 elastic agent + ingest management
- 1. Nicolas Ruflin (@ruflin) Engineer at Elastic 2020-07-30 Technical deep dive into Elastic Agent + Ingest Manager
- 2. Agenda • Teaser on all the parts that are new • Demo • Technical overview • Questions / Discussions
- 3. About me • Engineer at Elastic for 5 years • Tech Lead Ingest Management project • Switzerland based • @ruflin
- 4. Reimagine the way how to ingest data into the Elastic Stack
- 6. Beats to Elastic Agent
- 7. Heartbeat Winlogbeat Metricbeat Filebeat Security APM Many Beats & agents today Elastic Agent Easier to install Easier to upgrade Easier to scale One unified Agent tomorrow
- 9. Beats modules today Integrations tomorrow ● One click, right in Kibana ● Recommendations ● Share with the community
- 10. Config UI
- 11. Configuration files today Configuration UI tomorrow ● Minimal input ● Out of the box defaults ● Logs & metrics combined
- 13. Config management today You’re on your own... ● Powershell / Bash ● Ansible ● Puppet ● Chef ● ServiceNow Fleet central mgmt tomorrow ● Configuration updates automatically ● Binaries update automatically ● View status of Agents
- 14. Password to API Keys
- 15. User/pass today Keys tomorrow ● No need to input user/pass ● More secure
- 16. Demo
- 17. Demo • Enroll agent • Show collecting out of the box • Collect nginx data • Everything is API based
- 19. Our components
- 22. Current Indexing strategy • filebeat-*, metricbeat-* • Beats load all the assets ‒ Index Template ‒ ILM Policy
- 23. Problems with the current strategy • Too many fields • ILM only for logs or metrics • Query always on all data • Bootstrapping tricky • User modifications break things
- 24. New Indexing Strategy • {type}-{dataset}-{namespace} • type: generic type of data (logs, metrics, traces, ...) • dataset: Set of data with the same structure / mapping (nginx.access) • namespace: Use configurable namespace (prod, testing) • Example: logs-nginx.access-default • Default: logs-generic-default
- 25. Indexing Strategy: Bootstrapping and Templates • Generic templates ‒ logs-*-*, metrics-*-* ‒ ECS Based ‒ Loaded by Elasticsearch • Dataset specific templates ‒ logs-nginx.access-* ‒ Ingest pipeline attached to index ‒ Loaded by Ingest Manager
- 26. Indexing Strategy: Elasticsearch Features • Problem: Could not be built on the existing Elasticsearch features • New features ‒ constant_keywords ‒ component templates ‒ Data streams • More details on these in a separate talk
- 27. Elastic Agent One agent to rule them all
- 28. Elastic Agent
- 29. Elastic Agent • Runs and manages processes ‒ Today: Metricbeat, Filebeat, Endpoint ‒ Potential future: *beat, Apm-Server • Communication through GRPC • Two modes ‒ Managed by Fleet ‒ Standalone • Unified configuration for all inputs • Upgrade
- 31. Agent Config
- 32. Agent Config inputs: - type: nginx/metrics hosts: http://127.0.0.1 streams: - dataset.name: nginx.stub_status metricset: stub_status - type: logs streams: - dataset.name: nginx.access paths: /var/log/nginx/access.log* - dataset.name: nginx.error paths: /var/log/nginx/error.log*
- 33. Elastic Package Registry Central place for all packages
- 34. Package Registry
- 36. Packages
- 37. Packages • Contains assets and configs for the Elastic Stack • Package manager knows how to install, upgrade, remove each asset • Delivered as .tar.gz file • Logs, metrics together in one package
- 38. Package Structure: Supported Assets Today • Elasticsearch ‒ Index Template v2 ‒ ILM Policy ‒ Ingest Pipeline • Kibana ‒ Dashboard ‒ Visualization, Map ‒ Search ‒ Index Pattern • Agent ‒ Stream Template Future • Elasticsearch ‒ ML Job ‒ Data • Kibana ‒ Alert ‒ Action ‒ Canvas Template ‒ SIEM rules Note: Any asset type in the Stack can be added to this list
- 39. Config Builder
- 40. Packages
- 41. Agent Config: You can build your own config UI # Stream template dataset.name:nginx.access {{#each paths}} paths: "{{this}}" {{/each}} exclude_files: [".gz$"] processors: - add_locale: ~ # dataset manifest streams: - input: logs title: ... description: ... template_path: stream.yml.hbs vars: - name: paths required: true default: - /var/../access.log* # Stream config - dataset: nginx.access paths: - /var/log/nginx/access.log* exclude_files: - .gz$ processors: - add_locale: ~ + = Note: In case there is a shared config across streams, the package manifest has vars # package manifest datasources: - name: nginx inputs: - type: logs
- 42. Agent Config Built inputs: - type: nginx/metrics hosts: http://127.0.0.1 streams: - dataset.name: nginx.stub_status metricset: stub_status - type: logs streams: - dataset.name: nginx.access paths: /var/log/nginx/access.log* - dataset.name: nginx.error paths: /var/log/nginx/error.log*
- 43. FAQ
- 44. FAQ • How can I start using this? ‒ Start 7.8 Cloud cluster with xpack.ingestManager.enabled: true ‒ Download 7.8 Elastic Agent ‒ Soon: Run 7.9 • Do you have an API? ‒ Yes, it is all API based • Indexing strategy ‒ Can I use it also without agent? Yes
- 45. Try it out!