A presentation I did on what, why, how, and benefits of centralized logging in the Enterprise. This presentation was focused on implementing centralized logging in a environment that is mostly .NET/Windows.

1. Centralized
Logging

2. Logging provides value.
How’s that now?

3. How does logging
provide value?
We can gather information on the system while it’s in
operation. We can use this information to provide more
value to the business.

4. More Value?
Information gathered from logs allows you to
proactively take actions that will keep systems
providing the most value to the business.

5. Talk in circles much?
Centralized logging provides visibility into the
operational efficiency of the system and process.

6. A story on operational sleep
efficiency

7. How is it done?
Logs for most/all systems are
shipped, parsed and stored in a
central location for monitoring and
decision making.

8. Um…sounds like a bunch of
work.
IIS Logs
SQL Logs
Router Logs
Etc.

9. Initial focus is on application logging for in-house applications.

10. Where we are…
System logs
file to
directory
System logs
file to
database
?
System logs
file to
directory
Where many are…we’re not alone.

11. Where we want to go.
Shipper
Systems
Parsing, Indexing, St
orage
Logger
Viewer
App
A system that has the ability to store and analyze log
data from any system that provides it.

12. Logging Issues
to resolve

13. Access To Log
Getting access to production servers is not
always easy.

14. Standardizing Logging
format
Log(“Error Occurred, 45, 64”)
Log({Error : “Doh!”,Exeception : ex})
While enabling extensibility…
Mobile Client Log
Database Log
Web Client(js) Log
Service Log
DNN Log
MVC Application Log

15. Access to all log data
Web Client
L
o
g
s
Mobile
Client
L
o
g
s
Service/Logic
L
o
g
s
Data Tier
DB
L
o
g
s
L
o
g
s

16. Use log data for…
• Troubleshoot errors and design
• Monitoring and notification
• Auditing

17. Upcoming version of Kibana!
Business decisions (Analytics)

18. The Good

19. Windows Event Log
• Low Cost – Built in.
• Built in UI
• Can forward logs to a central server

20. Windows Event Log
• Event entries stored in memory.
• Have to configure logging per server.
• Service interface for other clients to
hook into not available.

21. Log to central SQL database
• Low Cost – SQL in
house, create an
database/table.

22. Log to central SQL database
• Relational (Schema based) format
challenging to use for
unstructured data.
• logging straight to database
could degrade application
performance.

23. Commercial Hosted Tools
• Handles multiple log formats
• Robust charting/searching Features

24. • Turned down previously
• $$$ - pricey for amount of space
allotted

25. And the winner is….
A Hybrid Approach.

26. Kibana
Rest API
Archival
Elastic Search
Log
Indexer
Redis Queue
L
o
g
L
o
g
EL
EL
EL
Application
Application
Application

27. Low(No) Cost
OSS FTW!

28. Extensible
Log
Indexer
(Publisher)
Notifications
Statistics
(Subscriber)
(Subscriber)
Redis Queue
Using a Pub/Sub pattern from the queue will allow
other components to subscribe to it.

29. Runs on Windows
Java/Ruby based tools run on windows with some
setup.

30. Blocking

31. Why ElasticSearch?

32. Inverted Index
“In computer science, an inverted index (also referred to as postings file or inverted file) is anindex data
structure storing a mapping from content, such as words or numbers, to its locations in adatabase file, or in a
document or a set of documents. The purpose of an inverted index is to allow fast full text searches, at a cost of
increased processing when a document is added to the database.” via http://en.wikipedia.org/wiki/Inverted_index
Word
Word
Word
Page
Page
Page
Page
Word
Regular Index
Inverted Index

33. Documented Oriented
• No need to define schema upfront
• Can store, index and quickly search unstructured data.
• Schema can be defined per type for customization of
the indexing process.

34. Big Community
• VIA http://www.elasticsearch.org/community/

35. Easy to scale
• Distributed by nature.
• Indexes broken down by shards with 0 or more
replicas.

36. Easy Level 1 Rest API
• Well documented and straight forward api, which
makes it easy to build a client for it.

37. What’s Needed recap
•
•
•
•
Server for Elastic Search (Windows/Linux)
Redis Server
Windows service to index logs from queue.
Modify Enterprise Library Logger to send to queue.

38. In closing
Gathering the data is one part. Logging enough
information and knowing what questions your are
looking to answer is another part.

39. References