Introduction to Kibana by Matt Overstreet at OpenSource Connections http://opensourceconnections.com

1. No one listens to my podcast
a Kibana story
Matt Overstreet ! | opensourceconnections.com "

2. Things I Have:
• A Podcast
• Some Logs
• Kibana
Matt Overstreet ! | opensourceconnections.com "

3. Let’s do this!
Matt Overstreet ! | opensourceconnections.com "

4. Put them all in a folder
$ ls -l
drwxr-xr-x@ 12 matt staff 408B Apr 24 13:07 elasticsearch
drwxr-xr-x@ 16 matt staff 544B Apr 17 12:04 kibana
drwxr-xr-x 4672 matt staff 155K Apr 24 14:19 logs
drwxr-xr-x@ 20 matt staff 680B Apr 24 22:55 logstash
Matt Overstreet ! | opensourceconnections.com "

5. Start Elasticsearch
cd elasticsearch/
bin/elasticsearch
Matt Overstreet ! | opensourceconnections.com "

6. Start Kibana
cd kibana and
bin/kibana
Matt Overstreet ! | opensourceconnections.com "

7. It Works!
Matt Overstreet ! | opensourceconnections.com "

8. Let’s look at the data
time_micros c_ip ciptype cipregion cs_method cs_uri sc_status cs_bytes sc_bytes timetakenmicros cs_host cs_referer csuseragent srequestid cs_operation cs_bucket cs_object
1488343917672
486
173.66.45.75 1 GET search-
discomagic-
bullet.mp3
206 0 2 225000 storage.googlea
pis.com
AppleCoreMedia
/1.0.0.14D27
(iPhone; U; CPU
OS 1021 like Mac
OS X;
enus)AEnB2Uqwr
MFQZw537yrbZH
Z823C8Z0RjigtS
wcunEQloBUpX3
noZow1KxBB0LT
Wt_bDeuBoQm9
ex9dIaQhQZE2A
1xtjujaRRA
GET_Object search-disco magic-bullet.mp3
1488343978304
277
173.66.45.75 1 GET search-
discosolr-6-
release.mp3
206 0 2 313000 storage.googlea
pis.com
itunesstored1.0
iOS/10.2.1
modeliPhone5,1
hwp/s5l8950x
build/14D27 (6;
dt:79)
AEnB2UoemzAN
LH64amvUXetx4
Kj8jyHIOPzDesr0
XOTI4ziC8Rf06d
mbVcDpHu4h8z
gMgLYVUHeggsv
0PeT6vnsNmVK
M5iVvdA
GET_Object search-disco solr-6-
release.mp3
1488344010045
421
173.66.45.75 1 GET search-
discofacets-and-
search-ui.mp3
200 0 65202388 30813000 storage.googlea
pis.com
AppleCoreMedia
/1.0.0.14D27
(iPhone; U; CPU
OS 1021 like Mac
OS X; en_us)
AEnB2UqmhUom
_JGo2yWqGpS-
f8PhjUTLsF7fmlt4
gEbDQovE-
JxchtEeZInvW7D
J8tiVpQyTC0qES
Ljq1SpJm7f8V2c
LrKs6dA
GET_Object search-disco facets-and-
search-ui.mp3
1488344052519
416
173.66.45.75 1 GET search-
discosolr-6-
release.mp3
206 0 2 112000 storage.googlea
pis.com
AppleCoreMedia
/1.0.0.14D27
(iPhone; U; CPU
OS 1021 like Mac
OS X; en_us)
AEnB2UpOEjiiFk
bPSpFkimlQUKb
_POvFr85oceSWf
ZrKiWUsCMkuqe
WlABVzQAkFK6t
0Mixl1RtA0BJXh
EczxleLv5Z6pgTw
GET_Object search-disco solr-6-
release.mp3
Matt Overstreet ! | opensourceconnections.com "

9. Lets build a Logstash config!
input {
file {
path => "/Users/matt/Documents/code/podcast/logs/search_disco_usage*v0"
}
}
filter {
csv {
columns => ["time_micros","c_ip","c_ip_type","c_ip_region","cs_method",
"cs_uri","sc_status","cs_bytes","sc_bytes","time_taken_micros",
"cs_host","cs_referer","cs_user_agent","s_request_id",
"cs_operation","cs_bucket","cs_object"]
separator => ","
}
}
output {
elasticsearch {
hosts => "localhost"
index => "logstash-sd"
}
}
Matt Overstreet ! | opensourceconnections.com "

10. Input
input {
file {
path => "/Users/matt/Documents/code/podcast/logs/search_disco_usage*v0"
}
}
Matt Overstreet ! | opensourceconnections.com "

11. Filter
filter {
csv {
columns => ["time_micros","c_ip","c_ip_type","c_ip_region","cs_method",
"cs_uri","sc_status","cs_bytes","sc_bytes","time_taken_micros",
"cs_host","cs_referer","cs_user_agent","s_request_id",
"cs_operation","cs_bucket","cs_object"]
separator => ","
}
}
Matt Overstreet ! | opensourceconnections.com "

12. Output
output {
elasticsearch {
hosts => "localhost"
index => "logstash-sd"
}
}
Matt Overstreet ! | opensourceconnections.com "

13. Timestamps
We need to specify a timestamp for Kibana, but we have a
problem!
Our time data is in Microseconds!?
time_microseconds 1488343917672486
Matt Overstreet ! | opensourceconnections.com "

14. Logstash doesn't
handle
microseconds
Matt Overstreet ! | opensourceconnections.com "

15. No problem, add ruby
filter {
# prepare our time series data
ruby {
code => "event.set('time_micros',
event.get('time_micros').to_i / 1000)"
}
# use it as a timestamp
date {
match => [ "time_micros", "UNIX_MS" ]
}
}
Matt Overstreet ! | opensourceconnections.com "

16. Stash some logs!
cd logstash
bin/logstash -f config/seachdisco.conf
Matt Overstreet ! | opensourceconnections.com "

17. Back to Kibana
Matt Overstreet ! | opensourceconnections.com "

18. Matt Overstreet ! | opensourceconnections.com "

19. Create a Visualization
Matt Overstreet ! | opensourceconnections.com "

20. Add a Sub-bucket
Matt Overstreet ! | opensourceconnections.com "

21. Improvements
Matt Overstreet ! | opensourceconnections.com "

22. What if Someone Digs Into the Back
Catalog?
Let's build a better Y-axis
filter {
mutate {
add_field => {
'request_id' => "%{c_ip}-%{cs_object}"
}
}
}
Matt Overstreet ! | opensourceconnections.com "

23. Matt Overstreet ! | opensourceconnections.com "

24. Add a Map
filter {
geoip {
source => "c_ip"
target => "geoip"
}
}
Matt Overstreet ! | opensourceconnections.com "

25. Doing more with plugins
Matt Overstreet ! | opensourceconnections.com "

26. Parsing user agents
filter {
useragent {
source => 'cs_user_agent'
target => 'user_agent'
}
}
Matt Overstreet ! | opensourceconnections.com "

27. Anonymizing IP’s
Install a plugin
$ bin/logstash-plugin install logstash-filter-anonymize
Add a filter
filter {
anonymize {
fields => ['request_id']
key => 'arbitrary'
}
mutate {
remove_field => [ 'c_ip' ]
}
}
Matt Overstreet ! | opensourceconnections.com "

28. Final Filter Block & Thanks
filter {
csv {
columns => ["time_micros","c_ip","c_ip_type","c_ip_region","cs_method","cs_uri","sc_status","cs_bytes","sc_bytes",
"time_taken_micros","cs_host","cs_referer","cs_user_agent","s_request_id","cs_operation","cs_bucket","cs_object"]
separator => ","
}
# ignore API calls to upload/manipulate data
if [cs_user_agent] =~ "google-api-python-client" {
drop {}
}
ruby {
# do some calculation
code => "event.set('time_micros', event.get('time_micros').to_i / 1000)"
}
date {
match => [ "time_micros", "UNIX_MS" ]
}
mutate {
add_field => {
'request_id' => "%{c_ip}-%{cs_object}"
}
}
geoip {
source => "c_ip"
target => "geoip"
}
useragent {
source => 'cs_user_agent'
target => 'user_agent'
}
anonymize {
fields => ['request_id']
key => 'arbitrary'
}
mutate {
remove_field => [ 'c_ip' ]
}
}
Matt Overstreet ! | opensourceconnections.com "