Towards structured log analysis
•Download as PPTX, PDF•
3 likes•3,906 views
This document discusses log analysis automation challenges, existing log management tools, and proposes a structured log analysis framework. It notes that while existing tools address some automation needs, they ignore the importance of log structure. The proposed framework uses a declarative language to express any log file format, provides rule-based automation scripts, and a flexible data management scheme. Areas of future work include adding more log management capabilities, real-time analysis, optimizing data handling, and improved user interfaces.
Recommended
Recommended
More Related Content
Similar to Towards structured log analysis
Similar to Towards structured log analysis (20)
More from Dileepa Jayathilake
More from Dileepa Jayathilake (6)
Recently uploaded
Recently uploaded (20)
Towards structured log analysis
- 2. Functional Troubleshooting Conformance Log Analysis in Use Monitoring Statistical Insight System Health
- 3. Log Analysis Domains Web server logs Network logs Security logs System logs Application logs
- 4. Even with Manual analysis expertise, needs manual log acquaintance analysis is with format laborious Manually Manual analysis dealing with hinders reusing vast amount of recurring log information analysis is difficult Automation patterns will save lot of costs
- 5. Log Analysis Automation Challenges Lack of a standard • “Universal Format for Logger Messages” - Expired without a successor • “Syslog” – Serves only a limited range of system logs Log file corruptions • Erasing parts of a log file, mixing up multiple log entries, presence of log entries in wrong order and garbage in the middle of log files Inappropriate log content • Problem stems from incorrect judgments of developers regarding the importance of log entries Varying log semantics • Format and the content logged can continue to evolve Huge sizes of log files • Log files can easily grow into gigabyte sizes in a commercial environment
- 6. Existing Log Management Tools
- 7. Identifying common constructs Log indexing Handling different log sources Dealing with different log types Rich user interfaces Alerts Intrusion detection Compliance validation Automate recurring analysis procedures Structured Log Analysis
- 8. Why Structured Log Analysis? Many log files manifest a structure Analysis needs contextual correctness Automation requires a structure-aware Example tool
- 9. Structured Log Analysis Framework
- 10. Conclusions Existing tools solve a subset of automated log analysis requirements, but ignore the importance of structure New declarative language is capable of expressing any log file format and is resilient to corruptions The scripting language provides solid infrastructure for rule based automation Data management scheme offers flexibility Current UI generation method is not appropriate
- 11. Future Work Add more log management capabilities Real time analysis Built-in format declarations for common log formats Optimize data management module to handle heterogeneous data efficiently UI generation based on HTML5
Editor's Notes
- Your introductory or title slide should convey the overall “feeling” and focus of your presentation. For instance, I typically present about small-business trends, new business ideas, growth opportunities or other positive trends. In this sample presentation, I’m talking about new business ideas, so I used a sun graphic in this slide template to convey a positive feeling. Personalize this slide template with your company’s logo. To add a logo to all slides, place it on the Slide Master. To access the Slide Master, on the Themes tab of the Ribbon, click Edit Master and then click Slide Master.Disclaimer: You understand that Microsoft does not endorse or control the content provided in the following presentation. Microsoft provides this content to you for informational purposes only; it is not intended to be relied upon as business or financial advice. Microsoft does not guarantee or otherwise warrant the accuracy or validity of this information and encourages you to consult with a business or financial professional as appropriate.RIEVA LESONSKY Founder and President, GrowBiz Media RievaLesonsky is founder and president of GrowBiz Media, a content and consulting company specializing in covering small businesses and entrepreneurship. A nationally known speaker and authority on entrepreneurship, Lesonsky has been covering America’s entrepreneurs for nearly 30 years. Before co-founding GrowBiz Media, Lesonsky was Editorial Director of Entrepreneur Magazine.
- I like to speak spontaneously, so I use PowerPoint as an outline to keep me on track. It’s best to keep your PowerPoint text brief, simply reinforcing key points you will talk about at more length. You can use this slide template to convey a series of steps or related points in a short format.
- I like to use an off-balance layout to keep things from getting too symmetrical. Customize this slide template graphics of your choice, including photos, clip art, your logo or illustrations. Good photos really help cement an idea in the audience’s mind. This slide is animated to display an appropriate image as you introduce each business type.
- A plain old bulleted list can get boring, so use graphics to liven it up. An image that conveys what you’re saying in visual format (like this diagram) can reinforce your ideas in the audience’s mind.
- I like to use an off-balance layout to keep things from getting too symmetrical. Customize this slide template graphics of your choice, including photos, clip art, your logo or illustrations. Good photos really help cement an idea in the audience’s mind. This slide is animated to display an appropriate image as you introduce each business type.
- Splunk – This is one of the most popular commercial log analysis tools [11]. It comes as a native application for each of the popular platforms. It provides strong search capabilities within log files. Log files from many different sources can be integrated into an analysis. Splunk is capable of identifying common constructs appearing in logs such as timestamps. In addition to indexing logs based on automatically detected log entries it provides functionality for users to create custom indexes too. Indexed log files can be saved as templates so that the index can be used for a similar log file later. Analysis results are displayed in a dashboard with many feature-rich user interface controls. Although Splunk can handle any kind of text log file, it is appropriate for analyzing line logs. It comes with a free version (without expiration) with an upper limit to the total size of log files analyzed in a day.LogRhythm – This is another widely used commercial tool for log analysis [12]. Its' important features are the ability to analyze a huge number of logs at once, automatic detection of interesting log entries, risk-based prioritization of log events, customizable rules, alerts, real-time log monitoring, normalization between different time zones, configurable charting, ability to save investigation data and file integrity monitoring. It has built in capabilities to evaluate log compliance with a number of standards. In addition it has strong intrusion detection capabilities too. In summary, LogRhythm is a sophisticated, enterprise solution.ArcSight Logger – This is a tool for event log collection and reporting [13]. Being a commercial tool ArcSight Logger has the capability to handle event log messages from many different client platforms. The messages can be sent in a varietyof protocols. The tool can handle terabytes of log data efficiently. It classifies log events so that different syntax used across platforms for same kind of log data is made transparent to the user. Searching is possible using plain text, regular expressions or indexed text. ArcSight Logger provides strong reporting capabilities too. Reports can be exported to various formats before saving. Alerts can be defined based on reports. The tool comes with a free evaluation version.loggly – This provides a cloud based log management system [14]. Log files from various sources can be collected to a central place in cloud for analysis. Log entries can be searched and be viewed in a dashboard. Historic data can also be viewed. The tool supports alerting. A free trial version is provided.loglogic – This is another log management infrastructure tool with the capability to collect logs from either enterprise or cloud and provide analysis [15]. Main features include ability to handle data in ranges of petabytes, advanced searching capabilities, dynamic dashboard, detailed reports, alerts, forensics engine, log retention management and compliance reporting.AWStats – This is a free tool that can analyze logs generated by web servers like Apache web server, Internet Information Server, WebStar and some other proxy, wap, ftp, streaming and mail servers [16]. It is a command line tool that uses Perl scripts. It provides usage statistics, user origin information, popularity of pages, HTTP errors, number of favorites on the site, worm attacks detection, etc.SecureVue – This is a situational awareness platform that utilizes logs from various types of assets in an organization such as hosts, network and security devices, applications and databases for capturing important security information [17]. It provides compliance with many security standards.
- A plain old bulleted list can get boring, so use graphics to liven it up. An image that conveys what you’re saying in visual format (like this diagram) can reinforce your ideas in the audience’s mind.
- I like to use an off-balance layout to keep things from getting too symmetrical. Customize this slide template graphics of your choice, including photos, clip art, your logo or illustrations. Good photos really help cement an idea in the audience’s mind. This slide is animated to display an appropriate image as you introduce each business type.
- I like to use an off-balance layout to keep things from getting too symmetrical. Customize this slide template graphics of your choice, including photos, clip art, your logo or illustrations. Good photos really help cement an idea in the audience’s mind. This slide is animated to display an appropriate image as you introduce each business type.
- I like to use an off-balance layout to keep things from getting too symmetrical. Customize this slide template graphics of your choice, including photos, clip art, your logo or illustrations. Good photos really help cement an idea in the audience’s mind. This slide is animated to display an appropriate image as you introduce each business type.
- I like to use an off-balance layout to keep things from getting too symmetrical. Customize this slide template graphics of your choice, including photos, clip art, your logo or illustrations. Good photos really help cement an idea in the audience’s mind. This slide is animated to display an appropriate image as you introduce each business type.