These are the slides from a talk I gave at the San Francisco Rails meetup at Crowdflower on June 19, 2014. I talked about Elasticsearch(a little), logstash, load balancing, high availability,processing millions of logs for multiple large-scale productoin environments, and lessons we learned implementing this system.

1. ELASTICSEARCH, LOGGING, AND NINEFOLD

2. Hi, I'm Ryan
Lead Rails Support Engineer at Ninefold

3. WHAT WE'LL COVER
Ninefold's stack (ELK, sorta)
Architecting the system
Getting the logs out
Lessons learned

4. LOGGING:

5. UNTIL IT'S NOT!

6. WHAT IS ELASTICSEARCH?
Highly scalable
Auto-clustering
Auto-sharding
Auto-indexing
Auto-magical!
Apache Lucene under the covers

7. WHY ELASTICSEARCH?
See previous slide
Easy to setup
Easy to scale
Readily available resources on the web
Super simple to understand what's going on
Lots of tools/integrations/plugins

8. INSTALLING AND RUNNING
$ curl -L -O http://download.elasticsearch.org/PATH/TO/LATEST/$VERSION.zip
$ unzip elasticsearch-$VERSION.zip
$ cd elasticsearch-$VERSION
$ bin/elasticsearch #-d to daemonize
$ curl 'http://localhost:9200/?pretty'

9. OUR SUPPORTING ROLES:
Beaver (https://github.com/josegonzalez/beaver)
Logstash
RabbitMQ

10. A LITTLE BIT ABOUT BEAVER:
Written in Python
Runs as a daemon on the client
Can listen to and send as many/few logs as you want
Can do some or all log formatting
Formats and transmits logs one line at a time to the queue in
JSON
Attaches metadata for later

11. MORE ON LOGSTASH:
Pulls logs from the queue
Formats logs again and pushes them to Elasticsearch
Grok to easily format messages

12. LOGSTASH GROK:
...
{
# find checkpoint, turn it into a field and add it as a tag
:grok => {
:tags => ['cheflog'],
:pattern => "#{node[:ninefold_logger][:checkpoint_tag]}:%{WORD:checkpoint_type}",
:tag_on_failure => false,
:add_tag => "#{node[:ninefold_logger][:checkpoint_tag]}%{checkpoint_type}",
:singles => true
},
...

13. ABOUT RABBITMQ:
Fast
AMQP Messaging
High Availability
Ability to add nodes to existing cluster

14. THE IDEA:

15. THE IMPLEMENTATION:

17. Uhh..

18. OK...

19. GIMME MY LOGS!

21. Log Entry Model:

22. Find and sort them:

23. DEMO TIME!!

26. WHAT WE'VE LEARNED
Multi-line logs are a challenge
Multiple instances of Logstash can complicate things
Local storage > network storage
Don't use globs - be specific.
Elasticsearch is awesome!

27. THANK YOU
ryan@ninefold.com
@I_am_Ryo
github: IAMRYO