This document discusses using the ELK stack (Elasticsearch, Logstash, Kibana) to centralize logging and visualize log data. It introduces the components of ELK - Logstash collects and processes logs, Elasticsearch stores the data, and Kibana visualizes it. The document demonstrates using Logstash to input Apache/Nginx logs and output to Elasticsearch, and using Kibana to view the data. It also compares traditional distributed logging to the benefits of centralizing logs with ELK.

1. Shipping &
Visualize Your
Data With ELK
Adam Chen

2. HELLO
Everyone!
I am Adam
I am just an engineer like to share some
experience with others.
Thanks Gentoo let me familiar with Linux.
You can find me at @adaam

3. Let’s Start This Session

4. ELK? What !?

5. Develop by

6. Elastic Family

7. ELK ?
DevOps ?

8. Log Still Play An
Important Role

9. Today’s characters
Logstash/Beat
series
Shipping all of
your log to where
it should go, like
ES, AWS, or just
text.
Elasticsearch
The main part to
store your data
with High
Availability.
Kibana
Visualize will
power your data.
To know more
about its value.

10. Traditional Way to Collect Log
When error happened, administrator or
RD/QA will need to login or write/use tool
to grab log from each machines then
analysis.

11. Hey Bob, Please collect
the error log to analyze.
OK, Boss.
BOSS

13. Hey Bob, Please collect
the error log to analyze.
Traditional Way to Collect Log

15. Old Way to Collect Log

16. How ELK Help ?

17. Centralize Log To
One Place

18. Collect Log using ELK

19. Introduce The E,
The L and The K

20. Logstash
An agent install at where log need to be
collect.
Have much filter to process your log.
Also Input/Output module.
Logstash Module

21. Logstash Filters
Date, geoip, json, kv ...etc
GROK !!
Grok online tester

22. Logstash DEMO
Input apache/nginx log then output to
stdout, using local logstash.

23. Elasticsearch
SImply a data store with near real time search
Store data in index, can by hours, day, week,
month. Setting at Logstash.
Machine spec depends on data size.

24. Elasticsearch Modules
They are improve ES function if you need.
Watcher, Shield, Marvel, Cloud-AWS

25. Elasticsearch DEMO
API Webpage

26. Kibana
Show your data to you understand.
But you need to know your data fields first.

27. Kibana Demo
Create some template to show (data pregen
at ES? )
Real time insert data to ES and see from
template pregen. (apache/nginx?)

28. What else ?

29. Push Metrics to elsewhere

30. THANKS!
Any questions?
You can find me at:
@adaam