Rich Viet, Principal Engineer at Cloud Elements presents 'Scalable Logging and Analytics with LogStash' at All Things API meetup in Denver, CO. Learn more about scalable logging and analytics using LogStash. This will be an overview of logstash components, including getting started, indexing, storing and getting information from logs. Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching).

1. Log Scaling and Analytics With
Logstash
Richard Viet
Principal Engineer
Cloud Elements

2. Problems

Logging to a database or filesystem

Logging has placed a load on the database and
filesystem.

Multiple log formats

No easy way to search logs

No easy method to gather statistics

3. Logstash

Open source, Apache licence

Written in JRuby. Runs on jvm.

Plugins easily written in Ruby.

Part of the Elasticsearch family.

www.logstash.net

4. Logstash

Scalable: Elasticsearch for indexing, search
and retrieval

Process multiple log formats

Receive logs from multiple sources

Output logs to multiple destinations

Kibana provides web interface for search and
analytics

Easily extended with plugins written in Ruby

5. Logstash Architecture
Shipper
Broker Indexer
Search
Storage
Shipper
Shipper
Web
Interface

6. Logstash Pipeline

Input → filters → output

Separate threads

Filters are applied in order of config file

Outputs processed in order of config file

7. Logstash Plugins

Input – read input stream
– File input
– Log4j
– Redis
– Syslog

Codecs – decoding log messages
– Json
– Multiline

8. Logstash Plugins

Filters – processing messages
– Csv – define fields in a csv
– Date – define date field formats
– Mutate – change date type
– Xml – extract xml
– Grok – parses arbitrary text

9. Logstash Plugins

Output
– Elasticsearch
– Elasticsearch_http
– Mongodb
– Email
– Nagios

10. Indexer

Send message to Elasticsearch for indexing

An index is created for each day

Each index split into 5 shards by default

Original message is stored

Each field indexed

11. Elasticsearch

Apache Lucene search engine

An elasticsearch index is made up of multiple
shards

Each shard is a lucene index

Primary shard and at least one replica

Shards are moved between servers when
servers are added or removed

12. Elasticsearch Configuration

Self discovery
– Multicast
• Simplest if all nodes on same network
– Unicast
• Provide a list of servers
– Combination

13. Elasticsearch

Adding more nodes improves indexing and
search time.

Primary node is indexed first then replicas

Number of shards determined when index is
created.

Number of replicas is configurable

14. Elasticsearch

Adding more nodes improves indexing and
search time.

Primary node is indexed first then replicas

Number of shards determined when index is
created.

Number of replicas is configurable

15. Kibana

Browser based analytics for time-stamped data

Included in the logstash jar

Connect to the logstash server port 9292

Sends multiple requests to avoid overloading
the server.

16. Log4j to Logstash
App
Logstash Redis
Elasticsearch
Cluster
App
App Logstash

17. Logstash Log4j Server

Configure logstash as a Log4j server
input {
log4j {
mode => "server"
port => 9501
}
}

18. Send to a broker

Configure broker
output {
stdout {}
redis {
host => "redis1"
data_type => "list"
key => "logstash"
}
}

19. Indexing
input {
redis {
host => “redis”
data_type => “list”
key => “logstash”
}}
output {
elasticsearch {
cluster => “logstash”
host => "elasticsearch"
port => "9200"
}}

20. Scaling
Broker
Indexer
Search
Storage
Shipper
Web
Interface
Broker
Indexer

21. Sending to Broker
output {
stdout {}
redis {
host => ["redis1", “redis2”]
data_type => "list"
shuffle_hosts => true
key => "logstash"
}
}

22. Indexing
input {
redis {
host => “redis1”
data_type => “list”
key => “logstash”
redis {
host => “redis2”
data_type => “list”
key => “logstash”
}
}
output { ...

23. Quick Start

Logstash, elasticsearch and kibana configured
to run from the logstash jar

Download and untar

bin/logstash agent -f config.file

bin/logstash web