This document summarizes a survey paper on the development of mutation testing. It outlines the objectives, scope, fundamental hypotheses, related concepts, and future trends of mutation testing. It also discusses classification of research in mutation testing, practical applications using mutation testing tools, and empirical studies that have been conducted.

1. An Analysis and Survey
of the Development of Mutation Testing[JH09]
About 30 minutes Tao He
elfinhe@gmail.com
Software Engineering Laboratory
Department of Computer Science, Sun Yat-Sen University
With thanks to Yue Jia and Mark Harman
These slides are mainly extracted from Jia and Harman’s survey work
May 2011
Sun Yat-Sen University, Guangzhou, China
[JH09] Yue Jia, Mark Harman (September 2009). "An Analysis and Survey of the Development of Mutation
Testing" (PDF). CREST Centre, King's College London, Technical Report TR-09-06.
1/37

2. Outline
 Objectives
 Scope
 Classification of Research
 Fundamental Hypotheses
 Related Concepts
 Future Trend
 Tools

3. Objectives of Mutation Testing
 Provide “mutation adequacy score”
 Measure the effectiveness of a test set in
terms of its ability to detect faults
3/20

4. Scope of Mutation Testing
 The unit level
 The integration level
 The specification level
 As a white-box unit test technique

5. Theoretical work on Mutation Testing
 Hypotheses supporting Mutation Testing
 Optimization techniques
 techniques for reducing computational cost
 techniques for the detection of equivalent mutants

6. Practical work on Mutation Testing
 Applications of Mutation Testing
 Development work on Mutation Testing tools
 Empirical work

7. Fundamental Hypotheses

8. Competent Programmer Hypothesis (CPH)
 Programs can be corrected by a few small
syntactical changes.

9. Coupling Effect
 Test data sets that detect simple types of
faults are sensitive enough to detect more
complex types of faults
 A simple fault is represented by a simple mutant
which is created by making a single syntactical
change
 A complex fault is represented as a complex
mutant which is created by making more than one
change

10. A Mutation Operator
 A transformation rule that generates a mutant
from the original program
 Generate ‘realistic faults’

11. Behaviors of Mutation Operators
 Mutation Objects
 Variables
 Expressions
 Mutation Operation
 Replacement
 Insertion
 Deletion

12. Problems of Mutation Analysis
 Executing the enormous number of mutants
 Human effort
 The human oracle problem
 The equivalent mutant problem

13. Cost Reduction Techniques

14. Classification of
Cost Reduction Techniques
 Reduction of the generated mutants (may
be taken into consideration in our
approach for Fault Localization)
 Reduction of the execution cost

15. Mutants Reduction Techniques
 Mutant Sampling
 Random
 Based on the Bayesian sequential probability ratio
test (SPRT)
 Mutant Clustering
 Based on killable test cases
 Selective Mutation
 By reducing the number of mutation operators applied.
 Ignore operators ASR and SVR - redundant generation
 Only using ABS and ROR

16. Evaluation of Mutants Reduction
 A mean mutation score
 Reduction in the number of mutants
 Number of equivalent mutants

17. Execution Cost Reduction Techniques
 Strong Mutation
 Weak Mutation
 Firm Mutation
 by providing a continuum of intermediate
possibilities - compare state, which lies between
execution (Weak Mutation) and the final output
(Strong Mutation).

18. Equivalent Mutant Detection Techniques
 10% to 40% of mutants which are equivalent

19. Empirical Study
 Compare mutation criteria with data flow
criteria such as 'all-use'
 Compare mutants with real faults

20. Future Trend
 A need for high quality higher order mutants
 A need to reduce the equivalent mutants
 A preference for semantics over syntax
mutation
 An interest in achieving a better balance
between cost and value
 A pressing need to generate test cases to kill
mutants

21. Life circle of Mutation Testing
 Generate mutants based on specified
mutation operations
 Reduce mutants
 Run mutants against a test suite

22. Mutation Testing Tools

23. Goals of Mutation
to Enhance Fault Localization
 Not aim to simulate real faults
 Analyze the impact of mutation on the
suspiciousness of the mutated statement

24. Name Application Year Character Available Suitable
Higher Order Mutation,
MILU C 2008 Search-based technique, Yes No
Test harness embedding
MUFORMAT C 2008 Format String Bugs No No
ESPT C/C++ 2008 Tabular No No
Variable type
CSAW C 2007 Yes No
optimization
ExMAn C, Java 2006 TXL No No
SESAME C, Lustre, Pascal 2006 Assembler Injection No No
Certitude C/C++ 2006 General (Commercial) Commercially No
Plextest C/C++ 2005 General (Commercial) Commercially No
Interface Mutation,
Proteum/IM 2.0 C 2001 Yes Yes
Finite State Machines
Source Code
Insure++ C/C++ 1998 Instrumentation Commercially No
(Commercial)
Mutant Schemata
TUMS C 1995 No No
Generation
Interface Mutation,
Proteum 1.4 C 1993 No No
Finite State Machines
Published C Mutation Testing Tools

25. Proteum
 Environment Variable: PROTEUMIMHOME
 li -P pre-filename [-D directory] source-filename LI-filename
 Call gcc to preprocess the source code file source-filename.c and
produce a file pre-filename.c. Then parse the file pre-filename.c
and generate various info files
 opmuta [-<operator> n m] [-all n m] source-filename Li-
filename
 Apply mutant operators to the source code and LI files. As ouput,
opmuta produces a description file in a format that muta is able to
read and include in the mutant database.

26. strutt .h include space.c source code
li ~/smart_debugger/toolkit/proteum/li -P pre-space space li-space
pre-space.c li-space.nli li-space.fun li-space.cgr li-spaec.gfc
statement info function info call graph info def-use pair info
opmuta ~/smart_debugger/toolkit/proteum/opmuta -O Operators pre-one_statement li one_statement > mutants.txt
-
Proteum.py
include
function name by directory
Mutants line number by mutantdsc
.
mutants.txt GenetrateMutants .py
mutation operator by mutant
.dsc
all mutants info
Proteum – An Example: Space

27. Thoughts of Self-Made Mutants Generator
 I once consider to implement a self-made mutants generator.
 Only mutate one statement
 Not aim to simulate real faults
 IPO
 Input
 A piece of source code
 Line number to mutate
 Mutation operators
 Process
 Preprocess -> Scan -> Parse -> Mutate
 Output
 Mutants

28. Collection of Compiler Front-end Tools
 GCC-XML …failed
 E.g.
 source code:
http://www.gccxml.org/HTML/example1in.html
 Parse-tree:
http://www.gccxml.org/HTML/example1out.html
 LLVM … not try, good for Objective-C?
http://llvm.org/

29. Collection of Compiler Front-end Tools
 GCC … failed, maybe no parse-tree for GCC
 -fdump-tree-fixup cfg-lineno
http://stackoverflow.com/questions/697817/how-to-make-gcc-spi
 -fdump-tree-all -fdump-rtl-all
http://stackoverflow.com/questions/1496497/how-can-i-see-parse
 -fprofile-arcs -ftest-coverage
 gcno gcda: not readable for human beings
 gcov: offer the line number, separated statements

30. Useful links
 http://en.wikipedia.org/wiki/Mutation_testing
 http://cs.gmu.edu/~offutt/rsrch/mut.html
 http://www.dcs.kcl.ac.uk/pg/jiayue/repository/

31. Q&A
31/37

32. Thank you!
Contact me via elfinhe@gmail.com
32/37

33. Abstract
 Fault localization is a technique that aims to pinpoint faults by
analyzing program execution spectrum, while mutation is a testing
technique used by generating faulty programs called mutants.
Most of existing research in fault localization focuses on the
spectrum of the given programs. Yet there are few attempts to
introduce mutation’s impact to enhance fault localization
techniques. In this paper, we propose a strategy that automatically
introduces mutation into fault localization techniques, and present
variations of a heuristic method to compute suspiciousness of
each statement by considering the impact of mutation. To validate
our method, experiments is conducted on benchmark programs,
namely Siemens, grep, gzip, sed, space, flex, make, vim, and
bash. Results indicate that the method can help programmers
locate faults more effectively than methods without mutation.
33/20