Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
ELK
Ansible + Vagrant 10 ELK
rueiancsie@gmail.com
2015/10/17
•
• API
•
• ...
• DAU
•
•
•
• Nagios
Zabbix
• elastic.co
elasticsearch
elasticsearch
• elasticsearch shard inverted
index : http://
www.slideshare.net...
ELK
• ELK = elasticsearch + logstash + kibana
• elastic.co
elasticsearch logstash
kibana
• elasticsaerch logstash kibana
E...
logstash
• logstash
( RSS)
elasticsearch
• logstash Java
Fluentd
https://www.elastic.co/products/logstash
logstash-forwarder
• logstash-forwarder elastic.co logstash
go
logstash
• logstash-forwarder
elastic.co filebeat
https://gi...
kibana
• kibana Hapi Node.js
Web UI elasticsearch
• kibana Grafana Grafana
Graphite v2.5
elasticsearch
https://www.elastic...
kibana
beats
• logstash elastic.co
beats
• packetbeat HTTP
Thrift-RPC Mysql PostgreSQL MongoDB Redis
Memcache protocol
• topbeat ...
Marvel
• Marvel Web UI elasticsearch
elasticsearch cluster
CPU
Index
https://www.elastic.co/products/marvel
Marvel
watcher
•
• watcher elastic.co elasticsearch
email slack webhook
• watcher
https://www.elastic.co/products/watcher
logstash
https://www.elastic.co/guide/en/logstash/2.0/deploying-and-scaling.html
logstash
elasticsearch
elasticsaerch
https://www.elastic.co/guide/en/logstash/2.0/deploying-and-scaling.html
logstash
Redis RabbitMQ
logstash
logstash
packetbeat & topbeat
packetbeat elasticsaerch
https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-logstash...
packetbeat & topbeat
Redis logstash
elasticsearch logstash
https://www.elastic.co/guide/en/beats/packetbeat/current/packet...
Marvel Watcher
• elastic.co Marvel Watcher
elasticsearch
• elasticsearch
https://www.elastic.co/guide/en/watcher/watcher-1...
nginx
nodejs
nodejs
nodejs
postgres
postgres
elastic
search
elastic
search
elastic
search
elastic
search
monitor
logstash
...
elasticsearch
• elasticsearch
• Index shard 5 5 elasticsearch
shard
• ES_HEAP_SIZE elasticsearch
• 64000
1024 elasticsearc...
elasticsearch
•
elasticsearch
• Index Curator
• Index Curator
elasticsearch
repository ( S3)
https://www.elastic.co/guide/...
elasticseach
• elasticsearch
elasticsearch
• logstash
statsD Graphite Graphite
1s 1m
ELK
• Ansible + Vagrant ELK
https://github.com/rueian/ansible-elk-example
• vagrant up
kibana
README
• Vagrant
ELK
nginx +
nodejs
postgres
elastic
search
elastic
search
redis +
logstash
logstash
logstash
elastic
search
kibana
redis +...
• [ ] Fluentd: An data collector for unified logging layer
• [ ] StatsD: Simple daemon for easy stats aggregation
• [ ] Gra...
Thanks
Any Questions?
rueiancsie@gmail.com
Prochain SlideShare
Chargement dans…5
×

Elasitcsearch + Logstash + Kibana 日誌監控

4 766 vues

Publié le

介紹 elasticsearch, logstash, kibana, logstash-forwarder, topbeat, packetbeat 等工具,以及如何部署,進行 log analysis & monitoring。
並且附上一個使用 Ansible + Vagrant 部署 10 個節點的 ELK 系統的完整範例。
https://github.com/rueian/ansible-elk-example

Publié dans : Données & analyses

Elasitcsearch + Logstash + Kibana 日誌監控

  1. 1. ELK Ansible + Vagrant 10 ELK rueiancsie@gmail.com 2015/10/17
  2. 2. • • API • • ...
  3. 3. • DAU • • •
  4. 4. • Nagios Zabbix • elastic.co elasticsearch elasticsearch • elasticsearch shard inverted index : http:// www.slideshare.net/rueian3/elasticsearch-45855699
  5. 5. ELK • ELK = elasticsearch + logstash + kibana • elastic.co elasticsearch logstash kibana • elasticsaerch logstash kibana ELK logstash-forwarder beats watcher
  6. 6. logstash • logstash ( RSS) elasticsearch • logstash Java Fluentd https://www.elastic.co/products/logstash
  7. 7. logstash-forwarder • logstash-forwarder elastic.co logstash go logstash • logstash-forwarder elastic.co filebeat https://github.com/elastic/logstash-forwarder
  8. 8. kibana • kibana Hapi Node.js Web UI elasticsearch • kibana Grafana Grafana Graphite v2.5 elasticsearch https://www.elastic.co/products/kibana
  9. 9. kibana
  10. 10. beats • logstash elastic.co beats • packetbeat HTTP Thrift-RPC Mysql PostgreSQL MongoDB Redis Memcache protocol • topbeat process CPU • filebeat logstash-forwarder https://www.elastic.co/products/beats
  11. 11. Marvel • Marvel Web UI elasticsearch elasticsearch cluster CPU Index https://www.elastic.co/products/marvel
  12. 12. Marvel
  13. 13. watcher • • watcher elastic.co elasticsearch email slack webhook • watcher https://www.elastic.co/products/watcher
  14. 14. logstash https://www.elastic.co/guide/en/logstash/2.0/deploying-and-scaling.html
  15. 15. logstash elasticsearch elasticsaerch https://www.elastic.co/guide/en/logstash/2.0/deploying-and-scaling.html
  16. 16. logstash Redis RabbitMQ
  17. 17. logstash logstash
  18. 18. packetbeat & topbeat packetbeat elasticsaerch https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-logstash.html
  19. 19. packetbeat & topbeat Redis logstash elasticsearch logstash https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-logstash.html
  20. 20. Marvel Watcher • elastic.co Marvel Watcher elasticsearch • elasticsearch https://www.elastic.co/guide/en/watcher/watcher-1.0/installing-watcher.html https://www.elastic.co/guide/en/marvel/current/_installation.html
  21. 21. nginx nodejs nodejs nodejs postgres postgres elastic search elastic search elastic search elastic search monitor logstash redis borker logstash logstash logstash-forwarder logstash redis borker packetbeat & topbeat elastic search monitor
  22. 22. elasticsearch • elasticsearch • Index shard 5 5 elasticsearch shard • ES_HEAP_SIZE elasticsearch • 64000 1024 elasticsearch • JVM swapping elasticsearch elastic.co swapping 1. sudo swapoff -a swapping 2. sysctl vm.swappiness 0 3. elasticsearch bootstrap.mlockall: true https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html
  23. 23. elasticsearch • elasticsearch • Index Curator • Index Curator elasticsearch repository ( S3) https://www.elastic.co/guide/en/elasticsearch/client/curator/current/_features.html
  24. 24. elasticseach • elasticsearch elasticsearch • logstash statsD Graphite Graphite 1s 1m
  25. 25. ELK • Ansible + Vagrant ELK https://github.com/rueian/ansible-elk-example • vagrant up kibana README • Vagrant
  26. 26. ELK nginx + nodejs postgres elastic search elastic search redis + logstash logstash logstash elastic search kibana redis + logstash logstash-forwarder packetbeat ELK redis + logstash topbeat
  27. 27. • [ ] Fluentd: An data collector for unified logging layer • [ ] StatsD: Simple daemon for easy stats aggregation • [ ] Grafana: Gorgeous metric viz, dashboards & editors for Graphite, InfluxDB & OpenTSDB • [ ] Graphite: Scalable Realtime Graphing • [ ] Nagios: The Industry Standard In IT Infrastructure Monitoring • [ ] Zabbix: The Enterprise-class Monitoring Solution for Everyone • [ ] QBox: Optimizing Elasticsearch: How Many Shards per Index? • [ ] QBox: What is Elasticsearch, and How Can I Use It? • [ ] Elasticsearch • [ ] Ansible: Application Deployment + Configuration Management + Continuous Delivery • [ ] Vagrant: Create and configure lightweight, reproducible, and portable development environments
  28. 28. Thanks Any Questions? rueiancsie@gmail.com

×