SherLog: Error Diagnosis Through Connecting Clues from Run-time Logs

SherLog: Error Diagnosis by Connecting Clues from
Run-time Logs
Ding Yuan1 Haohui Mai1 Weiwei Xiong1 Lin Tan1 Yuanyuan
Zhou2 Shankar Pasupathy3
1University of Illinois at Urbana-Champaign
2University of California, San Diego
3NetApp, Inc.
ASPLOS’10, March 13-17, 2010, Pittsburgh, Pennsylvania, USA.
July 18, 2013
Lisong Guo (LIP6/REGAL) July 18, 2013 1 / 13

Introduction
Scenario — Postmortem In-Production Debugging with Logs
postmortem VS. prediction (model checking, static analysis etc.)
in-production VS. in-house (runtime instrumentation etc.)
log VS. others (bug reports, deployment conﬁguration etc.)

Introduction
Subtasks of Debugging
reproduce the bug (procedure non-related to source code)
infer the failure-inducing execution path
infer the conditions along the failure-inducing execution path

Introduction
Subtasks of Debugging
reproduce the bug (procedure non-related to source code)
infer the conditions along the failure-inducing execution path
Research Question
How can we help a developer to debug in the scenario above ?

Approach
Idea
Manual Inspection −→ Automatic Inference
A tool that takes the run-time logs and source code as inputs, and then
produces some debugging hints for developers (i.e connecting the dots)
all possible and valid failure-inducing execution path
the evolution of value on certain variables along the inferred paths

Approach
Idea
Manual Inspection −→ Automatic Inference
A tool that takes the run-time logs and source code as inputs, and then
produces some debugging hints for developers (i.e connecting the dots)
all possible and valid failure-inducing execution path
the evolution of value on certain variables along the inferred paths
Usage Scenario
run the tool to get a list of interesting paths
examine the values of certain suspicious variables along some path
repeat the previous step until the root cause of the bug is found

Design
Components
Log Parsing: locate the logging statements in the source code
Path Inference: infer the failure execution paths and the constraints
Value Inference: infer the value evaluation along the given paths

Log Parsing
Objectives
Identifying the Logging Points and Variables in the source code.

Log Parsing
Objectives
Simple Logging Statements
Solution: regular-expression matching (i.e. grep)
e.g. error(0, 0, _("removing directory, %s"), path);
rule: {error(), 3, 4}

Log Parsing
Objectives
Simple Logging Statements
Solution: regular-expression matching (i.e. grep)
e.g. error(0, 0, _("removing directory, %s"), path);
rule: {error(), 3, 4}
Complicated Logging Facilities
Hierarchy wrappers of standard printing APIs. (alt: coccinelle)
e.g. error() -> strerrno()
rule 1: ’%s’: %{serrono}
rule 2: [{ "speciﬁer": serrno; "regex": Regex; "val_func": ErrMsgToErrno}]

Path Inference
Objectives
infer the constraints of variables along the path

Path Inference
Objectives
Constrained Sequence Matching Problem (NP-Complete?)
based on Saturn, a static analysis framework for C programs
match the control & data ﬂow with the sequence of log messages
convert the path searching problem into a set of declarative rules

Path Inference
Objectives
Constrained Sequence Matching Problem (NP-Complete?)
based on Saturn, a static analysis framework for C programs
match the control & data ﬂow with the sequence of log messages
convert the path searching problem into a set of declarative rules
Glance of Implementation
customized control-ﬂow: main → log@4 → b1@10 → c@16 → log@25
conjunctive constraints: strchr = NULL ∧ verbose = 0 rmdir()@17 = 0

Path Inference

Path Inference
Technique Summary
SAT-based path searching
constraint programming

Path Inference
Technique Summary
SAT-based path searching
constraint programming
Limitations
skip the analysis on the functions of non-log-generating
therefore it might return incorrect results
no alias analysis for pointers
but the underlining framework Saturn support alias analysis
special treatments on some external routines/functions
abort, exit, setjmp, longjmp etc.

Value Inference
Objective
infer the value-ﬂow of certain variables, given the execution paths

Value Inference
Objective
infer the value-flow of certain variables, given the execution paths
Algorithm
model the assignment relationship among memory locations as
guarded points-to graph
predicate(position, variable, value, constraint)
symbolically execute the inferred failure path forwards
refine the constraint according to the scope of variables
incrementally update the graph at each step
generate the sequence of value evolution (value-flow)

Evaluation
Methodology
manually reproduce and diagnose the real-world bugs
collect path summaries at runtime
compare the result of SherLog with the reproduction information

Evaluation
Methodology
manually reproduce and diagnose the real-world bugs
collect path summaries at runtime
compare the result of SherLog with the reproduction information
Metrics
useful: SherLog infers a subset of bug reproduction information
complete: SherLog infers all the bug reproduction information

Evaluation

Assumptions/Limitations
Assumptions
suﬃcient logging messages
reasonable density distribution of logging statements
reasonable amount of logging statements being activated
well-match between the bug manifestation path and the log trace
sequential and single-threaded log messages
cannot handle multi-thread concurrent program
Technical Limitations
skip the functions that do not involve in log production
do not parse the complex constructs of C programming language, such
as pointer arithmetics

More Pointers...
Ding Yuan, Soyeon Park, Yuanyuan Zhou: Characterizing logging practices
in open-source software. ICSE 2012
Adam J. Oliner, Archana Ganapathi, Wei Xu: Advances and challenges in
log analysis. Commun. ACM 2012
Ding Yuan, Jing Zheng, Soyeon Park, Yuanyuan Zhou, Stefan Savage:
Improving Software Diagnosability via Log Enhancement. ASPLSO 2011
Wei Xu, Ling Huang, Armando Fox, David A. Patterson, Michael I. Jordan:
Detecting Large-Scale System Problems by Mining Console Logs. ICML 2010
Thomas Reidemeister, Mohammad Ahmad Munawar, Miao Jiang, Paul A. S.
Ward: Diagnosis of recurrent faults using log files. CASCON 2009
Trishul M. Chilimbi, Ben Liblit, Krishna Mehra, Aditya V. Nori, and Kapil
Vaswani: HOLMES: Effective statistical debugging via efficient path
profiling. ICSE 2009

SherLog: Error Diagnosis Through Connecting Clues from Run-time Logs

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (8)

Similaire à SherLog: Error Diagnosis Through Connecting Clues from Run-time Logs

Similaire à SherLog: Error Diagnosis Through Connecting Clues from Run-time Logs (20)

Dernier

Dernier (20)

SherLog: Error Diagnosis Through Connecting Clues from Run-time Logs