Software log file analysis helps immensely in software testing and troubleshooting. The first step in automated log file analysis is extracting log data. This requires decoding the log file syntax and interpreting data semantics. The expected output of this phase is an organization of the extracted data for further processing. Log data extractors can be developed using popular programming languages targeting one or few log file formats. Rather than repeating this process for each log file format, it is desirable to have a generic scheme for interpreting elements of a log file and filling a data structure suitable for further processing. The new log data extraction scheme introduced in this paper is an attempt to provide the advanced features demanded by modern log file analysis procedures. It is a generic scheme which is capable of handling both text and binary log files with complex structures and difficult syntax. Its output is a tree filled with the information of interest for the particular case. My speech in ICSCA 2011 - http://dileepaj.blogspot.com/2011/07/speech-in-icsca-2011.html

1. A Novel Mind Map Based Approach for Log Data Extraction Dileepa Jayathilake Department of Electrical Engineering University of Moratuwa Sri Lanka ICIIS 2011

2. Conclusion Implementation AGENDA Solution Design Solution Overview Problem Identification Background

3. Functional Conformance BACKGROUND Quality Verification Troubleshooting System Administrators Domain Experts Application Logs Developers Monitoring Tool Logs Testers LOG FILE ANALYSIS

4. BACKGROUND Labor Intensive Require Expertise Error-prone Advantage of Recurrence not used PITFALLS IN MANUAL APPROACH

5. Different log formats & structure Lack of a common platform Making rules human & machine readable PROBLEM IDENTIFICATION Challenges Result Proprietary Implementation Automation abandoned Reports not customizable Costly Rules not human readable Less resilient to format changes Difficult to add new rules CHALLENGES

7. Ubiquitous use

8. Many tools available

9. Costly meta data

10. Less human readable

11. Associated languages are complex

13. Regular expression based

14. Assume line logs

15. Fail with complex log file structures

16. Unable to handle difficult syntax

18. Log Files SOLUTION OVERVIEW SOLUTION OVERVIEW Interpretation Processing Presentation Unified mechanism for extracting information of interest from both text and binary log files with arbitrary structure and format Easy mechanism to build and maintain a rule base for inferences Flexible means for generating custom reports from inferences Knowledge Representation Schema

19. Easy to add content SOLUTION DESIGN Easy to visualize Resembles human knowledge organization better Easy to combine MIND MAPS Easily convertible to XML Easy access to computers Tree Can utilize existing tree algorithms Can utilize existing tools MIND MAP AS KNOWLEDGE UNIT

20. GENERIC INTERPRETATION SOLUTION DESIGN Interpretation Unified mechanism for extracting information of interest from both text and binary log files with arbitrary structure and format Log Files

21. LOG FILE GRAMMAR SOLUTION IMPLEMENTATION Assume knowledge on file structure and syntax Able to handle a spectrum of log file types Based on hierarchical log entries Log entries identified by attribute combination Translates a log file into a mind map Resilient for malformed log files

22. SOLUTION IMPLEMENTATION PARSER

23. val = 2.3 SOLUTION IMPLEMENTATION LE ≡ ([A,S,E,S,B], NO); A ≡ ([A1,A2,A3], NO); A1 ≡ (‘v’); A2 ≡ (‘a’); A3 ≡ (‘l’); S ≡ ({SPACE, TAB}, -1, 0, NO); SPACE ≡ (‘ ‘); TAB ≡ (‘’); E ≡ (‘=’); B ≡ ({ZERO, ONE, …, NINE, DECIMAL_POINT}, -1, 1); ZERO ≡ (‘0’); ONE ≡ (‘1’); … ; NINE ≡ (‘9’); DECIMAL_POINT ≡ (‘.’) EXAMPLE

24. Difficult syntax SOLUTION IMPLEMENTATION MICROSOFT SHAREPOINT LOG FILE

25. MICROSOFT APPLICATION VERIFIER LOG SOLUTION IMPLEMENTATION XML

26. SOLUTION IMPLEMENTATION TRADING SYSTEM LOG Corrupted Log

27. CONCLUSION The new scheme Is capable of expressing both text and binary log files with different structures and formats ranging from flat messages to complex hierarchies.

28. REFERENCES [1] J. H. Andrews, “Testing using log file analysis: tools, methods and issues,” Proc. 13th IEEE International Conference on Automated Software Engineering, Oct. 1998, pp. 157-166. [2] D. Jayathilake, “A mind map based framework for automated software log file analysis,” International Conference on Software and Computer Applications., in press. [3] T. Takada and H. Koike, “Mielog: a highly interactive visual web browser using information visualization and statistical analysis,” Proc. USENIX Conf. on System Administration, Nov. 2002, pp. 133-144. [4] L. Destailleur, “AWStats,” [Online]. Available: http://awstats.sourceforge.net [5] J. Valdman, “Log file analysis,” Department of Computer Science and Engineering (FAV UWB)., Tech. Rep. DCSE/TR-2001-04, 2001. [6] J. H. Andrews, “Theory and practice of log file analysis,” Department of Computer Science, University of Western Ontario., Tech. Rep. 524, May 1998. [7] T. Buzan and B. Buzan, The Mind Map Book. New York: Penguin Books, 1994, pp.79-91. [8] J. Cowie and W. Lehnert, “Information extraction,” Comm. ACM 39, 1996, pp. 80–91. [9] J. Abela and T. Debeaupuis, “Universal Format for Logger Messages,” The Internet Engineering Task Force. [Online]. Available: http://tools.ietf.org/html/draft-abela-ulm-05

29. QUESTIONS