Short presentation about the ELK stack (Elasticsearch, Logstash, Kibana) running on top of Docker / Vagrant.

1. ELK

2. Eberhard Wolff - @ewolff
Docker
• No true virtualization
• Linux Containers (lxc)
• i.e. shared kernel
• i.e. separate file systems

3. Eberhard Wolff - @ewolff
Docker File Systems
• Read only base images
• +read/write image
• Can be stacked

4. Eberhard Wolff - @ewolff
Docker
• Linux only
• So: Run it in VM
• Controlled by Vagrant

5. Docker Demo

6. Eberhard Wolff - @ewolff
Communication Between
Docker Container
• Via ports
• Via data volumes

7. Docker is a
Component
System

8. Eberhard Wolff - @ewolff
Example
• Log file analysis
• ELK Stack
• Elasticsearch: Store
• Logstash: Parse
• Kibana: Web based GUI
• Install three servers
• Automate installation ... L

9. Eberhard Wolff - @ewolff
Dockerized ELK Stack
• 1 Vagrantfile (25 lines)
• 6 Dockerfiles (total 28 lines)
• 1 HTML file (11 lines)
• 1 Config (Logstash) (29 lines)
• Total: 93 lines, 9 files
• Including one RegExp L

10. Eberhard Wolff - @ewolff
Vagrant VM
Container
user-registration
8080
8081
Port
8081
Container
elasticsearch
Container
logstash
9200
9200
Port
9200
9200
link
Container
kibana
Port
8080
8080
8080
Browser
Volume
/log

11. Demo ELK
Stack

12. Eberhard Wolff - @ewolff
date time level
1988-10-12 19:42:07.350 INFO 683 [http-nio-8080-exec-6] --- c.e.u.service.RegistrationService : email=eberhard.wolff@gmail.com geloesc
pid thread class logmessage

13. Eberhard Wolff - @ewolff
Scaling Elasticsearch
• More servers
• Shards
• Easy to set up
• Index per day or week or month
• Smaller dataset

14. Eberhard Wolff - @ewolff
Ideas about Logstash
• No parsing
• Send JSON

15. Eberhard Wolff - @ewolff
Break Logstash apart
• Logstash does parsing and
communication
• Not too sensible in a distributed
sytem
• Separate process per responsibility

16. Eberhard Wolff - @ewolff
Shipper
• Ship data from logging system
• Logstash
• Beaver
https://github.com/josegonzalez/
beaver
• Woodchuck
https://github.com/danryan/
woodchuck

17. Eberhard Wolff - @ewolff
Broker
• Buffer data for later processing
• Redis
• In-Memory NoSQL database
• Small
• Fast
• ...and then Parse (Logstash)

18. Eberhard Wolff - @ewolff
Links
• https://github.com/ewolff/user-
registration

19. Eberhard Wolff - @ewolff
Thank You!!
eberhard.wolff@gmail.com
@ewolff