Contenu connexe
Similaire à Mutation Testing with PIT (Booster 2014, 2014-MAR-13) (20)
Plus de Filip Van Laenen (18)
Mutation Testing with PIT (Booster 2014, 2014-MAR-13)
- 1. © Computas AS 11.03.14
Mutation Testing with PIT
Filip van Laenen
Booster 2014
2014-03-13
- 2. 2 © Computas AS 11.03.14
Agenda
• Basics of mutation testing
• Relation to other testing techniques
• Example
• Mutation testing techniques
• Mutation testing tools
• Personal experiences and recommendations
- 4. 4 © Computas AS 11.03.14
Mutation Testing in a Nutshell
Seeking The Summoner @ The Daily WTF
http://thedailywtf.com/Articles/Seeking-The-Summoner.aspx
- 5. 5 © Computas AS 11.03.14
Mutation Testing in a Nutshell (cont'd)
- 6. 6 © Computas AS 11.03.14
Mutation Testing in a Nutshell (cont'd)
- 7. 7 © Computas AS 11.03.14
Mutation Testing in a Nutshell (cont'd)
• Unit tests guard the source code
• But who guards the guardians?
• Do the unit tests cover all source code?
• Lines?
• Branches?
• Paths?
• Do the unit tests test the right things?
Mutation testing tests the tests!
- 8. 8 © Computas AS 11.03.14
Mutation Testing in a Nutshell (cont'd)
- 9. 9 © Computas AS 11.03.14
Mutation Testing in a Nutshell (cont'd)
- 10. 10 © Computas AS 11.03.14
Mutation Testing in a Nutshell (cont'd)
- 11. 11 © Computas AS 11.03.14
Mutation Testing in a Nutshell (cont'd)
- 12. 12 © Computas AS 11.03.14
Mutation Testing in a Nutshell (cont'd)
- 13. 13 © Computas AS 11.03.14
Mutation Testing in a Nutshell (cont'd)
- 14. 14 © Computas AS 11.03.14
Mutation Testing in a Nutshell (cont'd)
int max(int a, int b) {
return (a < b) ? b : a;
}
int max(int a, int b) {
return (a <= b) ? b : a;
}
- 15. 15 © Computas AS 11.03.14
Mutation Testing in a Nutshell (cont'd)
- 16. “
16 © Computas AS 11.03.14
Does Mutation Testing Work?
In practice, if the software contains a
fault, there will usually be a set of
mutants that can only be killed by a test
case that also detects that fault.
Geist et. al., “Estimation and Enhancement of Real-time
Software Reliability through Mutation Analysis,” 1992
- 17. “
17 © Computas AS 11.03.14
Does Mutation Testing Work? (cont'd)
Complex faults are coupled to simple
faults in such a way that a test data set
that detects all simple faults in a program
will detect most complex faults.
K. Wah, “Fault Coupling in Finite Bijective Functions,”
1995
- 18. 18 © Computas AS 11.03.14
Does Mutation Testing Work? (cont'd)
• “Generated mutants are similar to real faults.”
• Andrews, Briand, Labiche, ICSE 2005
• “Mutation testing is more powerful than
statement or branch coverage.”
• Walsh, Ph.D. Thesis, State University of New York at
Binghampton, 1985
• “Mutation testing is superior to data flow
coverage criteria.”
• Frankl, Weiss, Hu, Journal of Systems and Software,
1997
- 19. 19 © Computas AS 11.03.14
Relation to Other
Testing Techniques
- 20. 20 © Computas AS 11.03.14
Relation to Other Testing Techniques
• Unit tests
• Test-Driven Development (TDD)
• Test coverage
• Static code analysis
• Fuzz testing
- 21. 21 © Computas AS 11.03.14
Relation to Other Testing Techniques
- 22. 22 © Computas AS 11.03.14
Is Mutation Testing New?
• R. Lipton, “Fault Diagnosis of Computer
Programs,” 1971
• R. Lipton et. al., “Hints on Test Data Selection:
Help for the Practicing Programmer,” 1978
• Historical obstacles:
• No unit testing
• No TDD
• Inefficient implementation
• Hence time-consuming
• No integration with IDEs
- 23. 23 © Computas AS 11.03.14
Practical Example
of Mutation Testing
- 24. 24 © Computas AS 11.03.14
Code Along…
https://github.com/computas-fvl/booster2014
git clone
https://github.com/computas-fvl/booster2014.git
mvn clean site:site
org.pitest:pitest-maven:mutationCoverage
- 26. 26 © Computas AS 11.03.14
Mutation Testing Techniques
• Three aspects:
• Mutation injection
• Mutation types
• Unit test selection per mutant
• Key properties:
• Efficiency
• Performance
- 27. 27 © Computas AS 11.03.14
Mutation Injection
• Source code mutation
• Binary code mutation
• Caveats:
• De-mutation
• Compilation errors
• Invalid binary code
- 28. 28 © Computas AS 11.03.14
Mutation Types
• Some mutations never change behaviour
• Constants reused by unit tests
• Log messages
• Some mutations can change behaviour
• Switching between < and ≠
• Switching between < and ≤
• Some mutations always change behaviour
• Switching between < and ≥
- 29. 29 © Computas AS 11.03.14
Mutation Types (cont'd)
int max(int a, int b) {
return (a < b) ? b : a;
}
int max(int a, int b) {
return (a <= b) ? b : a;
}
int max(int a, int b) {
return (a >= b) ? b : a;
}
- 30. 30 © Computas AS 11.03.14
Mutation Types (cont'd)
for (int i = 0; i < 10; i++) …
for (int i = 0; i != 10; i++) …
for (int i = 0; i >= 10; i++) …
- 31. 31 © Computas AS 11.03.14
Mutation Types Guaranteed to Change
Behaviour
• Negation of the comparison
• Switching between = and ≠
• Switching between < and ≥
• Switching between > and ≤
• Negation of boolean conditions
• Adding a ¬, ! or ~
• Shortcutting boolean conditions
• Replacement with True or False
*
- 32. 32 © Computas AS 11.03.14
Unit Test Selection
• Goal: find the unit test that “kills” the mutant
• Selection aids:
• Hints
• Name/package matching
• Code coverage tools
• Automatic learning
• Other heuristics
- 33. 33 © Computas AS 11.03.14
Unit Test Selection (cont'd)
• System:
• 50 classes
• 20 unit tests per class
• 1 ms per unit test
• Unit testing time: 50 × 20 × 1ms = 1s
• 10 mutants per class:
• Brute-force: 10 × 50 × 1s = 6m 20s
• Educated: 10 × 50 × 20 × 1ms = 10s
- 34. 34 © Computas AS 11.03.14
Unit Test Selection (cont'd)
• System:
• 500 classes
• 20 unit tests per class
• 1 ms per unit test
• Unit testing time: 500 × 20 × 1ms = 10s
• 10 mutants per class:
• Brute-force: 10 × 500 × 10s = 13h 53m 20s
• Educated: 10 × 500 × 20 × 1ms = 1m 40s
- 35. 35 © Computas AS 11.03.14
Complexity
• f: Number of function points
• φ: Number of function points per class, ≥ 1
• τ: Number of unit tests per function point, ≥ 1
• μ: Number of mutants per function point, ≥ 1
• Brute-force: (f × τ) × (f × μ) = τ × μ × f²
• Educated force: τ × μ × φ × f
• Ideal: τ × μ × f
- 36. 36 © Computas AS 11.03.14
Complexity (cont'd)
• c: Number of classes
• φ: Number of function points per class, ≥ 1
• τ: Number of unit tests per function point, ≥ 1
• μ: Number of mutants per function point, ≥ 1
• Brute-force: (f × τ) × (f × μ) = τ × μ × φ² × c²
• Educated force: τ × μ × φ² × c
• Ideal: τ × μ × φ × c
- 37. 37 © Computas AS 11.03.14
Loops
for (int i = 0; i < 10; i++) …
for (int i = 0; i < 10; i--) …
- 38. 38 © Computas AS 11.03.14
Infinite Loops
• Terminate mutants that take too long to run
• What's “too long”?
• Ruins the performance
• Can be hard to predict
- 39. 39 © Computas AS 11.03.14
Other Problems
• Recursion:
• Stack overflows
• Out of memory exceptions
• Syntax errors
• Segmentation faults
- 41. 41 © Computas AS 11.03.14
Mutation Testing Tools
• Java:
• PIT
• Jester (Nester, Pester)
• Jumble
• Ruby:
• Mutant
• Heckle
• C#:
• NinjaTurtles
- 42. 42 © Computas AS 11.03.14
PIT
• Java
• Junit & TestNG
• Maven or command-line
• Operates on byte code
• Large set of mutators
• Also possibly equivalent mutators available
• Highly configurable
• Sensible defaults
• http://pitest.org/
- 43. 43 © Computas AS 11.03.14
PIT Mutators
• Conditionals Boundary
• Negate Conditionals
• Remove Conditionals*
• Math
• Increments
• Invert Negatives
• Inline Constant*
• Return Values
• Void Method Calls
• Non Void Method Calls*
• Constructor Calls*
- 45. 45 © Computas AS 11.03.14
Jester
• Java
• JUnit
• Usually run from the command-line
• Grester for Maven2
• Operates on source code
• Runs all unit tests on all classes
• Reporting and documentation could be better
• http://jester.sourceforge.net/
• http://sourceforge.net/projects/grester/
- 48. 48 © Computas AS 11.03.14
Pester and Nester
• Pester
• Jester for Python
• PyUnit
• Nester
• Port of Jester for C#
• NUnit
• Integrated with Visual Studio
• But outdated…
• http://nester.sourceforge.net/
- 50. 50 © Computas AS 11.03.14
Jumble
• Java
• JUnit
• Run from the command-line
• Operates on byte code
• Runs unit tests on a class
• Reporting and documentation could be better
• Claims to be faster than Jester
• http://jumble.sourceforge.net/index.html
- 51. 51 © Computas AS 11.03.14
Jumble Sample Report
Mutating Foo
Tests: FooTest
Mutation points = 12, unit test time limit 2.02s
..
M FAIL: Foo:31: negated conditional
M FAIL: Foo:33: negated conditional
M FAIL: Foo:34: - -> +
M FAIL: Foo:35: negated conditional
......
Score: 67%
- 52. 52 © Computas AS 11.03.14
Mutant
• Ruby 1.9 and 2.0
• rspec2
• Usually run from the command-line
• Good to-the-point reporting
• Good performance
• OK documentation
• Active project
• https://github.com/mbj/mutant
- 53. 53 © Computas AS 11.03.14
Heckle
• Ruby 1.8
• Doesn't work on Ruby 1.9
• Test::Unit and rSpec
• Usually run from the command-line
• Runs a set of unit tests on a class or a method
• Good to-the-point reporting
• Good performance
• Virtually no documentation
• http://rubyforge.org/projects/seattlerb/
• http://docs.seattlerb.org/heckle/
- 54. 54 © Computas AS 11.03.14
Heckle Mutations
• Booleans
• Numbers
• Strings
• Symbols
• Ranges
• Regexes
• Branches (if, while, unless, until)
- 55. 55 © Computas AS 11.03.14
NinjaTurtles
• .Net:
• C#
• Visual Basic/VB.NET
• Any other .NET language code
• Doesn't seem much alive
• http://www.mutation-testing.net/
- 56. 56 © Computas AS 11.03.14
NinjaTurtles Mutations
• Sequence point deletion
• Arithmetic operator substitution (*, /, +, -, %)
• Bitwise operator substitution (&, |, ^)
• Branch substituion (condition, always and never
branch)
• Conditional boundary substition (< and <=, >
and >=)
• Substitution of reads from variables,
parameters and fields of the same type
• Substitution of writes to variables of the same
type
- 57. 57 © Computas AS 11.03.14
Personal Experiences
and Recommendations
- 58. 58 © Computas AS 11.03.14
Experiences and Recommendations
• Use mutation testing from day 1
• Start on a small code base
• Keep number of unit tests per class low
• Have small classes
• Select a good tool
• Configurable
• Flexible
• One that can output the mutant
- 59. 59 © Computas AS 11.03.14
Experiences and Recommendations
(cont'd)
• Believe the tool
• Or try to proof that the tool is wrong
• Fix the problem
• Don't turn mutation testing off
• Embrace the “more than 100%” test coverage
• Path coverage
• Less code
• More unit tests
• More intelligent unit tests
- 61. 61 © Computas AS 11.03.14
Improvements
• Integration with more unit testing frameworks
• Better unit test–source code mapping
• Better heuristics
• Parallellisation
• Better reporting
• IDE integration
• Building tool integration
- 62. 62 © Computas AS 11.03.14
Questions?
Computas AS
Lysaker Torg 45, pb 482
N-1327 Lysaker
NORWAY
Tel +47-67 83 10 00
Fax +47-67 83 10 01
Org.nr: NO 986 352 325 MVA
www.computas.com
Contact:
fvl@computas.com @filipvanlaenen