2. 2
ELK
• What is ELK?
– Elasticsearch
It provides a distributed, multitenant-capable full-text search engine with an HTTP web
interface and schema-free JSON documents
– Logstash
It is a tool to collect, process, and forward events and log messages
– Kibana
It provides visualization capabilities on top of the content indexed on an Elasticsearch
cluster.
Server Filebeat
Server Filebeat
Logstash Elasticsearch Kibana Nginx User
4. 4
Beats
• The Beats are open source data shippers that you install as agents on your
servers to send different types of operational data to Elasticsearch.
• Beats can send data directly to Elasticsearch or send it to Elasticsearch via
Logstash, which you can use to enrich or archive the data
5. 5
Logstash
• The ingestion workhorse for Elasticsearch and more
– Horizontally scalable data processing pipeline with strong Elasticsearch and
Kibana synergy
• Pluggable pipeline architecture
– Mix, match, and orchestrate different inputs, filters, and outputs to play in
pipeline harmony
• Community-extensible and developer-friendly plugin ecosystem
– Over 200 plugins available, plus the flexibility of creating and contributing your
own
6. 6
Logstash
• Logstash Download Link
– Linux
https://download.elastic.co/logstash/logstash/packages/debian/logstash_2.2.2-1_all.deb
– Windows
https://download.elastic.co/logstash/logstash/logstash-2.2.2.zip
• How to set up Logstash
– http://howtorapeurjob.tumblr.com/post/140724250861/
7. 7
Logstash
• Start up
– Linux:
service logstash restart
/opt/logstash/bin/logstash -f /etc/logstash/conf.d/ -v
– Windows:
{file_path}binlogstash.bat
• Parameter
-h or --help 相關資訊
-t or --configtest 測試config是否正常
-f 指定config檔案
-v or --debug 輸出debug資訊
-w 是以多少thread處理資料，預設4
-l 指定log輸出路徑
16. 16
Elasticsearch
Elasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to
store, search, and analyze big volumes of data quickly and in near real time.
17. 17
Elasticsearch
• Elasticsearch on Ubuntu
– http://howtorapeurjob.tumblr.com/post/140678763346/
• Elasticsearch cluster on Windows
– http://howtorapeurjob.tumblr.com/post/140318838221/
• Elasticsearch Cluster Design
– http://howtorapeurjob.tumblr.com/post/140438605026/
• CRUD in Elasticsearch
– http://howtorapeurjob.tumblr.com/post/140323820711/
• Query DSL in Elasticsearch
– http://howtorapeurjob.tumblr.com/post/140616855246/
28. 28
CRUD in Elasticsearch
• Create Index
預設索引為空的，所以要先建立一個為facebook的index
– Linux
– REST Client
– Result
• Query Indices
– Result
curl -XPOST 'http://localhost:9200/facebook'1
{
"acknowledged": true
}
1
2
3
POST http://localhost:9200/facebook1
GET http://localhost:9200/_cat/indices?v1
health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open facebook 5 1 0 0 795b 795b
1
2
35. 35
CRUD in Elasticsearch
• Import Data
– Syntax
– Example
curl -XPOST localhost:9200/_bulk --data-binary @file.json1
curl -POST "http://yuhsuan_chen_w7p:9200/_bulk" --data-binary @C:UsersYuhsuan_chenDesktopshakespeare.json1
36. 36
Query DSL in Elasticsearch
• 特別注意
如果使用linux的curl套件，可以直接使用GET進行參數傳遞
如果使用windows的curl套件，會對字串的判別上有問題，尤其是單引號與雙引號，因此不建議使用
windows版本的curl。在windows上建議直接使用POSTMAN進行查詢
如果使用套件進行操作，因為大多數的http相關的library都不支援GET帶參數，所以建議使用POST操
作
IETF的相關文件內也沒特別規定GET是否可以帶參數，但是Elasticsearch開發人員覺得GET比較合理
，但實際上查詢時還是使用POST降低問題發生
http://tools.ietf.org/html/rfc7231#section-4.3.1
# Create test data
POST http://localhost:9200/facebook/yuhsuan_chen/_bulk
{"create":{"_id":"1"}}
{"message":"Hello!!!","feel":"Good","location": "Taoyuan City", "datetime": "2016-03-
01T12:00:00","temp":25.6}
{"create":{"_index":"facebook","_type":"Other","_id":"2"}}
{"message": "The weather is so good.","feel": "Comfortable","location": "Taipei City",
"datetime": "2016-03-01T12:01:00","temp":27.3}
{"create":{"_id":"3"}}
{"message": "I like my job.","feel": "bad","location": "Taoyuan City", "datetime":
"2016-03-01T12:02:00"}
{"create":{}}
{"message": "Time for launch.","feel": "hungry","location": "Taoyuan City", "datetime":
"2016-03-01T12:03:00","temp":30.3}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
37. 37
Query DSL in Elasticsearch
• Query All Data
• Query Specific Field Data
# Use match_all to query all data
POST http://localhost:9200/facebook/_search
{
"query" : {
"match_all" : {
}
}
}
1
2
3
4
5
6
7
8
# Use match to query “city” in location field
POST http://localhost:9200/facebook/_search
{
"query" : {
"match" : {
"location" : "city"
}
}
}
1
2
3
4
5
6
7
8
9
38. 38
Query DSL in Elasticsearch
• Query Multi Data
• Query Value
# Find location/message field include hello/good
POST http://localhost:9200/facebook/_search
{
"query" : {
"multi_match" : {
"query": ["hello","good"],
"fields": ["location","message"]
}
}
}
1
2
3
4
5
6
7
8
9
10
# Find the temp >= 27, gt -> 大於 ，gte ->大於等於，lt ->小於，lte ->小於等於
POST http://localhost:9200/facebook/yuhsuan_chen/_search
{
"query" : {
"range" : {
"temp":{
"gte":27
}
}
}
}
1
2
3
4
5
6
7
8
9
10
11
39. 39
Query DSL in Elasticsearch
• Query Data By Regular Expression
• Order Data
# Find the message can be match “fo.*” or “he.*”
POST http://localhost:9200/facebook/yuhsuan_chen/_search
{
"query" : {
"regexp" : {
"message":"fo.*|he.*"
}
}
}
1
2
3
4
5
6
7
8
9
# Query all data, order _type by desc, order _id by asc
POST http://localhost:9200/facebook/yuhsuan_chen/_search
{
"query" : {
"match_all" : {}
},
"sort":[
{"_type" : {"order":"desc"}},
{"_id" : {"order":"asc"}}
]
}
1
2
3
4
5
6
7
8
9
10
11
41. 41
Query DSL in Elasticsearch
• Query by “bool”
包含should，_id:3排名會在前面 不包含should，_id:3排名較低
42. 42
Kibana
Kibana is an open source analytics and visualization platform designed to work with Elasticsearch.
You use Kibana to search, view, and interact with data stored in Elasticsearch indices
• Flexible analytics and visualization platform
• Real-time summary and charting of streaming data
• Intuitive interface for a variety of users
• Instant sharing and embedding of dashboards
• Configure Kibana & Elasticsearch
– http://howtorapeurjob.tumblr.com/post/140733691641/
49. 49
Kibana
• 建立資產年齡分布圓餅圖
選擇Visualize -> Pie chart -> From a new search -> Select an index pattern -> ba*
選擇buckets -> Split Slices -> Aggregation -> Range -> Field -> balance ->輸入範圍數值
50. 50
Kibana
接著新增一個 Split Slices -> Aggregation -> Terms-> Field -> age -> Order By -> metric: Count ->
Order -> Descending -> Size -> 5
最後按下上方的綠色三角已產生我們要的圖表資料
如果要儲存可以選擇右上方的儲存
52. 52
Nginx
• Nginx port forwarding
– http://howtorapeurjob.tumblr.com/post/140665750171/
53. 53
ELK parse system log on Ubuntu
• How to set ELK on Ubuntu
– http://howtorapeurjob.tumblr.com/post/140837053206/
Editor's Notes
取得資料→資料蒐集→分析管理
Packetbeat:
Real time network packet analyzer.
Support protocols: HTTP, MySQL, PostgreSQL, Redis, Thrift-RPC, MongoDB, DNS, Memcache.
Topbeat:
Monitor servers.
System-wide statistics
System load
System wide CPU usage
memory usage
swap usage
Per-process statistics
Process name
Process parent pid
Process state
Process pid
Process CPU usage
Process Memory usage
File system statistics
List of available disks
For each disk, the name, type, and where it is mounted
For each disk, the total, used (and percentage), free, and available space
