1.
ELK
Ansible + Vagrant 10 ELK
rueiancsie@gmail.com
2015/10/17

2.
•
• API
•
• ...

3.
• DAU
•
•
•

4.
• Nagios
Zabbix
• elastic.co
elasticsearch
elasticsearch
• elasticsearch shard inverted
index : http://
www.slideshare.net/rueian3/elasticsearch-45855699

5.
ELK
• ELK = elasticsearch + logstash + kibana
• elastic.co
elasticsearch logstash
kibana
• elasticsaerch logstash kibana
ELK
logstash-forwarder beats watcher

6.
logstash
• logstash
( RSS)
elasticsearch
• logstash Java
Fluentd
https://www.elastic.co/products/logstash

7.
logstash-forwarder
• logstash-forwarder elastic.co logstash
go
logstash
• logstash-forwarder
elastic.co filebeat
https://github.com/elastic/logstash-forwarder

8.
kibana
• kibana Hapi Node.js
Web UI elasticsearch
• kibana Grafana Grafana
Graphite v2.5
elasticsearch
https://www.elastic.co/products/kibana

9.
kibana

10.
beats
• logstash elastic.co
beats
• packetbeat HTTP
Thrift-RPC Mysql PostgreSQL MongoDB Redis
Memcache protocol
• topbeat process CPU
• filebeat logstash-forwarder
https://www.elastic.co/products/beats

11.
Marvel
• Marvel Web UI elasticsearch
elasticsearch cluster
CPU
Index
https://www.elastic.co/products/marvel

12.
Marvel

13.
watcher
•
• watcher elastic.co elasticsearch
email slack webhook
• watcher
https://www.elastic.co/products/watcher

14.
logstash
https://www.elastic.co/guide/en/logstash/2.0/deploying-and-scaling.html

15.
logstash
elasticsearch
elasticsaerch
https://www.elastic.co/guide/en/logstash/2.0/deploying-and-scaling.html

16.
logstash
Redis RabbitMQ

17.
logstash
logstash

18.
packetbeat & topbeat
packetbeat elasticsaerch
https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-logstash.html

19.
packetbeat & topbeat
Redis logstash
elasticsearch logstash
https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-logstash.html

20.
Marvel Watcher
• elastic.co Marvel Watcher
elasticsearch
• elasticsearch
https://www.elastic.co/guide/en/watcher/watcher-1.0/installing-watcher.html
https://www.elastic.co/guide/en/marvel/current/_installation.html

21.
nginx
nodejs
nodejs
nodejs
postgres
postgres
elastic
search
elastic
search
elastic
search
elastic
search
monitor
logstash
redis
borker
logstash
logstash
logstash-forwarder
logstash
redis
borker
packetbeat & topbeat
elastic
search
monitor

22.
elasticsearch
• elasticsearch
• Index shard 5 5 elasticsearch
shard
• ES_HEAP_SIZE elasticsearch
• 64000
1024 elasticsearch
• JVM swapping elasticsearch elastic.co
swapping
1. sudo swapoff -a swapping
2. sysctl vm.swappiness 0
3. elasticsearch bootstrap.mlockall: true
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html

23.
elasticsearch
•
elasticsearch
• Index Curator
• Index Curator
elasticsearch
repository ( S3)
https://www.elastic.co/guide/en/elasticsearch/client/curator/current/_features.html

24.
elasticseach
• elasticsearch
elasticsearch
• logstash
statsD Graphite Graphite
1s 1m

25.
ELK
• Ansible + Vagrant ELK
https://github.com/rueian/ansible-elk-example
• vagrant up
kibana
README
• Vagrant

26.
ELK
nginx +
nodejs
postgres
elastic
search
elastic
search
redis +
logstash
logstash
logstash
elastic
search
kibana
redis +
logstash
logstash-forwarder packetbeat
ELK
redis +
logstash
topbeat

27.
• [ ] Fluentd: An data collector for unified logging layer
• [ ] StatsD: Simple daemon for easy stats aggregation
• [ ] Grafana: Gorgeous metric viz, dashboards & editors for Graphite, InfluxDB & OpenTSDB
• [ ] Graphite: Scalable Realtime Graphing
• [ ] Nagios: The Industry Standard In IT Infrastructure Monitoring
• [ ] Zabbix: The Enterprise-class Monitoring Solution for Everyone
• [ ] QBox: Optimizing Elasticsearch: How Many Shards per Index?
• [ ] QBox: What is Elasticsearch, and How Can I Use It?
• [ ] Elasticsearch
• [ ] Ansible: Application Deployment + Configuration Management + Continuous Delivery
• [ ] Vagrant: Create and configure lightweight, reproducible, and portable development environments

28.
Thanks
Any Questions?
rueiancsie@gmail.com