Quis custodiet ipsos custodes? Better known as: *Who watches the watchmen?* We are all writing tests, doing TDD, BDD. We measure the quality of the tests with line coverage or (even better) branch coverage. This gives you a false sense of security. I've seen projects with tests which have 100% branch coverage but not a single assertion! This is where mutation testing helps out. By creating broken mutated instances of your codebase (mutants) this should result in failing unit tests. This way we can verify that slight code changes (like real life bugs) actually break your tests. In this talk I'll explain what mutation testing is and how it works. We'll also compare some Java frameworks (PIT, Jester, Jumble) that enable automatic mutation testing in your continuous build and how you can start doing mutation testing *right now*.
9. CODE COVERAGE
• Who has seen (or written?) tests
• without verifications or assertions?
• just to fake and boost coverage?
• 100% branch coverage proves nothing
11. MUTATION TESTING
• Proposed by Richard J. Lipton in 1971 (winner of 2014 Knuth
Prize)
• A better way to measure the quality of your tests
• Surge of interest in the 1980s
• Time to revive this interest!
17. OUTCOME #3: TIMED OUT
• The mutant caused the program loop, get stuck
18. OTHER OUTCOMES
• NON-VIABLE
• JVM could not load the mutant bytecode
• MEMORY ERROR
• JVM ran out of memory during test
• RUN ERROR
• An error but none of the above.
19. FAULT INJECTION?
• With fault injection you test code
• Inject faults/mutations and see how the system reacts
• With mutation testing you test your tests
• Inject faults/mutations and see how the tests react
21. USING PIT
• PIT uses configurable ‘mutators'
• ASM (bytecode manipulation) is used to mutate your code
• No mutated code is stored, it can't interfere with your code
• Generates reports with test results
25. MUTATORS: MATH
+ into -
- into +
* into /
/ into *
% into *
& into |
<< into >>
>> into <<
>>> into <<<
a++ into a--
a-- into a++
26. MUTATORS: MANY MORE
• Replacing return values (return a; becomes return 0;)
• Removal of void invocations (doSomething(); is removed)
• Some enabled by default, others are optional/configurable
27. MUTATION TESTING IS SLOW?
• Speed was unacceptable in the 80's
• Mutation testing is still CPU intensive
• But PIT has a lot of methods to speed it up!
28. WHICH TESTS TO RUN?
• PIT uses code coverage to decide which tests to run:
• A mutation is on a line covered by 3 tests? Only run those.
29. SIMPLE EXAMPLE
• 100 classes
• 10 unit tests per class
• 2 ms per unit test
• Total time (all tests): 100 x 10 x 2ms = 2s
30. SIMPLE EXAMPLE
• Total time (all tests): 100 x 10 x 2ms = 2s
• 8 mutants per class, 100 classes x 8 = 800 mutants
• Brute force: 800 x 2s = 26m40s
• Smart testing: 800 x 10 x 2ms = 16s
31. LONGER EXAMPLE
• Total time (all tests): 1000 x 10 x 2ms = 20s
• 8 mutants per class, 1000 classes x 8 = 8000 mutants
• Brute force: 8000 x 20s = 1d20h26m40s…!!!
• Smart testing: 8000 x 10 x 2ms = 2m40s
32. PERFORMANCE TIPS
• Write fast tests
• Good separation or concerns
• Use small classes, keep amount of unit tests per class low
33. INCREMENTAL ANALYSIS
• Experimental feature
• Incremental analysis keeps track of:
• Changes in the codebase
• Previous results
34. HOW ABOUT MOCKING?
• PIT has support for:
• Mockito, EasyMock, JMock, PowerMock and JMockit
35. HOW TO USE PIT?
• Standalone Java process
• Build: Ant task, Maven plugin
• CI: Sonarqube plugin, Gradle plugin
• IDE: Eclipse plugin (Pitclipse), IntelliJ Plugin
36. STANDALONE JAVA
java -cp <your classpath including pit jar and dependencies>
org.pitest.mutationtest.commandline.MutationCoverageReport
--reportDir /somePath/
--targetClasses com.your.package.tobemutated*
--targetTests com.your.package.*
--sourceDirs /sourcePath/
39. USE CASE
The price of an item is 17 euro
If you buy 20 or more, all items cost 15 euro
If you have a coupon, all items cost 15 euro
40. CODE
public int getPrice(int amountOfThings, boolean coupon) {
if (amountOfThings >= 20 || coupon) {
return amountOfThings * 15;
}
return amountOfThings * 17;
}
41. TEST #1
@Test
public void testNormalPricing() {
//Not enough for discount:
int amount = 1;
Assert.assertEquals(17, businessLogic.getPrice(amount, false));
}
42. BRANCH COVERAGE
public int getPrice(int amountOfThings, boolean coupon) {
if (amountOfThings >= 20 || coupon) {
return amountOfThings * 15;
}
return amountOfThings * 17;
}
43. TEST #2
@Test
public void testDiscountPricingByAmount() {
//Enough for discount:
int amount = 100;
Assert.assertEquals(1500, businessLogic.getPrice(amount, false));
}
44. BRANCH COVERAGE
public int getPrice(int amountOfThings, boolean coupon) {
if (amountOfThings >= 20 || coupon) {
return amountOfThings * 15;
}
return amountOfThings * 17;
}
45. TEST #3
@Test
public void testDiscountWithCoupon() {
//Not enough for discount, but coupon:
int amount = 1;
Assert.assertEquals(15, businessLogic.getPrice(amount, true));
}
46. BRANCH COVERAGE
public int getPrice(int amountOfThings, boolean coupon) {
if (amountOfThings >= 20 || coupon) {
return amountOfThings * 15;
}
return amountOfThings * 17;
}
48. PIT RESULT
> org.pitest.mutationtest…ConditionalsBoundaryMutator
>> Generated 1 Killed 0 (0%)
> KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
PIT tells us: Changing >= into > doesn’t trigger a failing test
49. TEST #4
@Test
public void testDiscountAmountCornerCase() {
//Just enough for discount, mutation into > should fail this test
int amount = 20;
Assert.assertEquals(300, businessLogic.getPrice(amount, true));
}
50. BRANCH COVERAGE
public int getPrice(int amountOfThings, boolean coupon) {
if (amountOfThings >= 20 || coupon) {
return amountOfThings * 15;
}
return amountOfThings * 17;
}
53. DID YOU SPOT THE BUG?
@Test
public void testDiscountAmountCornerCase() {
//Just enough for discount, mutation into > should fail this test
int amount = 20;
Assert.assertEquals(300, businessLogic.getPrice(amount, true));
}
54. SUMMARY
• Mutation testing automatically tests your tests
• Mutation testing can find bugs in your tests
• Code coverage is wrong, gives a false sense of security
• Mutation testing with PIT is easy to implement