This document describes a framework and prototype for analyzing log files from multiple sources. It aims to address the problems of information overload, difficulty correlating different data sources, and manual analysis when debugging issues. The framework allows storage and parsing of different log file types and formats data as sets of named values. A log viewer prototype allows filtering, querying, and custom formatting of parsed log data. Future plans include completing the prototypes, adding more log parsers, automatic analysis capabilities, and integrating with other tools.
The Ultimate Guide to Choosing WordPress Pros and Cons
Smart Log Analysis
1. SMART LOG ANALYSIS
A General Framework
and SMB Prototype
Windows Serviceability
Tim Burke, Kishore Chintalapati (manager)
Mike Tiberio (coach), Apurva Sharma,
Samarth Shetty Badilaguthu
2. TALK OVERVIEW
Problem Space
Current Approaches
Design Objectives
My Project: Smart Log Analysis and SMB Prototype
Benefits
Future Plans
Demo
3. PROBLEM SPACE
Multiple Data Sources
Multiple Tools (Netmon, Perfmon, Notepad, …)
Difficulty in correlating different source
Information Overload
Manual Analysis
Knowledge Loss
4. CURRENT APPROACHES
Open Notepad
Open NetMon
Repeat
The Nuclear Option
Perl
Grep
Credit: Eric Roode
b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).
(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)b
http://www.regular-expressions.info/examples.html
6. DESIGN OBJECTIVES
A unified way of viewing, searching, and
analyzing data
Easily track and highlight relationships
among data.
Group data into high-level operations
Extensibility and Flexibility
7. DESIGN CONSIDERATIONS
Data is data, independent of the source
Data consists of sets of named values
Modular
Easy rule creation
Performance and Scalability
Developer focused
8. MY PROJECT
Framework
Viewer Prototype
Text Rule Editor
From Logs
From Source
Extensible
Component Agnostic
Scalable
Embeddable
9. THE FRAMEWORK
Storage Plugins
Provider RulesFile Format Plugins
Log Viewer
Query Engine
SQL Server
Parsed
Data
Log Parser
ETW Parser
Windows Events
Etc.
RDR
SRV
Log FIles
Config Files
Custom Storage
Parsed
Data
Storage Manager
Format Engine
CLR Adapter
Formatting
Rules
Saved
Queries
10. LOG VIEWER
Boolean expression filters
Filter based on any tag or value
Similar to Netmon filters
Procedural queries
Data correlation
Complex scenarios
Custom formatting
11. TEXT LOG RULE EDITOR
Easy creation of parsing rules
From text logs
From source code
Preview rule effects
12. BENEFITS
Allows quicker, easier debugging
Automates common analysis tasks
Merges data sources to allow cross-source
analysis.
13. FUTURE PLANS
Complete the prototypes
Implement more log parsers (Netmon, …)
Have component experts create rule sets
Implement automatic analyses on top of the
framework
Integrate with other tools for capturing data
like MSDT