Talk at DevoxxFR on April 11, 2019

1. 1
The CI as a partner for test improvement suggestions
Benoit Baudry
Project Coordinator and Scientific Leader
KTH, Sweden
Caroline Landry
Project Technical Manager
INRIA, France
19-Apr-2019

2. This work was partially supported by the EU Project
STAMP ICT-16-10 No.731529
•4 research institutions
•5 companies
•1 open source
consortium
•Automated Testing in
DevOps

3. DevOps
3

4. DevOps
4
• Better quality

5. DevOps
5
• Better quality
• Shorter release cycles

6. DevOps
6
• Better quality
• Shorter release cycles
• Reconnect Ops to Dev

7. DevOps
7
Automation• Better quality
• Shorter release cycles
• Reconnect Ops to Dev

8. DevOps
8
Step acceptance & Feedback
Automation• Better quality
• Shorter release cycles
• Reconnect Ops to Dev

9. DevOps
9

10. DevOps
10
CI

11. DevOps
11
IDEs
CI
completion linter

12. DevOps
12
unit
perf.
UI
IDEs
CI
completion linter

13. DevOps
13
unit
perf.
UI
Config.
IDEs
CI
completion linter

14. DevOps
14
unit
perf.
fuzzing
UI
Config.
IDEs
CI
completion linter
coverage
mutation

15. DevOps
15
unit
perf.
fuzzing
UI
Config.
IDEs
CI
completion linter
coverage
mutation
CD

16. DevOps
16
unit
perf.
fuzzing
UI
chaos
A/B testing
Config.
IDEs
CI
completion linter
coverage
mutation
CD

17. DevOps
17
unit
perf.
fuzzing
UI
chaos
Logs
analysis
A/B testing
Config.
IDEs
CI
completion linter
coverage
mutation Crash
analysis
CD

18. DevOps
18
unit
perf.
fuzzing
UI
chaos
Logs
analysis
A/B testing
Config.
IDEs
Continuous testing
CI
completion linter
coverage
mutation Crash
analysis
CD

19. 19

20. STAMP’s concept: amplification
 Amplify (v.): to increase the size or effect of
something
https://dictionary.cambridge.org/dictionary/english/amplify 20

21. STAMP’s concept: amplification
 Amplify (v.): to increase the size or effect of
something
 Test amplification: Increase the effect of test assets
21https://dictionary.cambridge.org/dictionary/english/amplify

22. STAMP’s concept: amplification
 Amplify (v.): to increase the size or effect of
something
 Test amplification: Increase the effect of test assets
 Test assets: test cases, configuration files, production logs
 Effect metrics: mutation score, feature interactions
 Automatic amplification
22https://dictionary.cambridge.org/dictionary/english/amplify

23. DevOps – STAMP focus
23

24. DevOps – STAMP focus
24

25. DevOps – STAMP focus
25

26. DevOps – STAMP focus
26

27. Unit test amplification
27
Descartes

28. Test Your Tests
•What do you expect from test cases?
28

29. Test Your Tests
•What do you expect from test cases?
• Cover requirements
• Code coverage
• Stress the application
• Reveal bugs
29

30. Test Your Tests
•What do you expect from test cases?
• Cover requirements
• Code coverage
• Stress the application
• Reveal bugs
30

31. Example
31
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}

32. Example
32
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
Coverage

33. Example
33
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(obs > 5);
}
Coverage

34. Example
34
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(obs > 5);
}
@Test
factorialWith0Test() {
assertEqual(fact(0), 1);
}
Coverage

35. Example
35
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
Coverage
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(obs > 5);
}
@Test
factorialWith0Test() {
assertEqual(fact(0), 1);
}
Is these test suite good at
detecting bugs?

36. Example
36
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
Coverage
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(obs > 5);
}
@Test
factorialWith0Test() {
assertEqual(fact(0), 1);
}
Is these test suite good at
detecting bugs?
Let’s mutate our code to see.

37. Mutation Analysis
37

38. Mutation Analysis
38
inject fault
Mutant 1

39. Mutation Analysis
39
inject fault
Mutant 1
Run

40. Mutation Analysis
40
inject fault
Mutant 1
Run ✗ Mutant 1

41. Mutation Analysis
41
inject fault
Mutant 1
Run ✗ Mutant 1
Killed


42. Mutation Analysis
42
inject fault
Mutant 1
Mutant 2
Run ✗ Mutant 1
Killed


43. Mutation Analysis
43
inject fault
Mutant 1
Mutant 2
Run ✗ Mutant 1
Killed


44. Mutation Analysis
44
inject fault
Mutant 1
Mutant 2
Run ✗
✗
Mutant 1
Mutant 2
Killed


45. Mutation Analysis
45
inject fault
Mutant 1
Mutant 2
Run ✗
✗
Mutant 1
Mutant 2
Killed

Killed


46. Mutation Analysis
46
inject fault
Mutant 1
Mutant 2
Mutant 3
Run ✗
✗
Mutant 1
Mutant 2
Killed

Killed


47. Mutation Analysis
47
inject fault
Mutant 1
Mutant 2
Mutant 3
Run ✗
✗
Mutant 1
Mutant 2
Killed

Killed


48. Mutation Analysis
48
inject fault
Mutant 1
Mutant 2
Mutant 3
Run ✗
✓
✗
Mutant 1
Mutant 2
Mutant 3
Killed

Killed


49. Mutation Analysis
49
inject fault
Mutant 1
Mutant 2
Mutant 3
Run ✗
✓
✗
Mutant 1
Mutant 2
Mutant 3
Killed

Alive

Killed


50. Mutation Analysis
50
inject fault
Mutant 1
Mutant 2
Mutant 3
Run ✗
✓
✗
 Mutation score
Mutant 1
Mutant 2
Mutant 3
Killed

Alive

Killed


51. Example
51
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(obs > 5);
}
@Test
factorialWith0Test() {
assertEqual(fact(0), 1);
}

52. long fact(int n) {
if (n != 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
Example
52
✗
✓
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(obs > 5);
}
@Test
factorialWith0Test() {
assertEqual(fact(0), 1);
}

53. long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
Example
53
n!=0
return 1+1
i < n
!(i<=n)
i--
result/i
result+1

54. Example
54
n!=0
return 1+1
i < n
!(i<=n)
i--
result/i
result+1
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(obs > 5);
}

55. Example
55
n!=0
return 1+1
i < n
!(i<=n)
i--
result/i
result+1
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
@Test
factorialWith0Test() {
assertEqual(fact(0), 1);
}

56. Example
56
n!=0
return 1+1
i < n
!(i<=n)
i--
result/i
result+1
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(obs > 5);
}
@Test
factorialWith0Test() {
assertEqual(fact(0), 1);
}

57. Example
57
 Mutation score = 71%
Test suite:
• Weak oracle
• Missing input
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(obs > 5);
}
@Test
factorialWith0Test() {
assertEqual(fact(0), 1);
}

58. Example
58
n!=0
return 1+1
i < n
!(i<=n)
i--
result/i
result+1
@Test
factorialWith5Test() {
assertEqual(fact(5),120);
}
long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(obs > 5);
}
@Test
factorialWith0Test() {
assertEqual(fact(0), 1);
}

59. Mutation Analysis
•Tests are good if they can detect bugs
•Mutation operators
• Based on common faults
•PIT or PITest
• Open source, in active development and production ready
• Integrates with major build systems
• State of the art mutation testing
• Extensible via plugins
• Concurrent execution
• Test selection
59

60. Limitations of mutation testing
•Expensive computation
•Huge number of mutants
•Presence of equivalent mutants

61. Limitations of mutation testing
•Expensive computation
•Huge number of mutants
•Presence of equivalent mutants
 Extreme Mutation

62. Extreme mutation
•Proposed in 2016 by Niedermayr et al.
•Remove the body of the method
•Replace by a single return
•Less mutants
•Most equivalent mutants can be detected
62
R. Niedermayr, E. Juergens, and S. Wagner, “Will my tests tell me if I break this code?,” in Proceedings of the International Workshop on Continuous
Software Evolution and Delivery, 2016, pp. 23–29.

63. long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
Example
63
n!=0
return 1+1
i < n
!(i<=n)
i--
result/i
result+1

64. long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
Example
64
n!=0
return 1+1
i < n
!(i<=n)
i--
result/i
result+1

65. long fact(int n) {
if (n == 0) {
return 1;
}
long result = 1;
for (int i = 2; i <= n; i++) {
result = result * i;
}
return result;
}
long fact(int n) {
return 0;
}
long fact(int n) {
return 1;
}
Example
65

66. Descartes I mutate therefore I am
• A mutation engine for PIT
• Implement extreme mutation
• Compute code coverage & mutation score
• Identify weaknesses in your tests
• Find pseudo-tested methods

67. Pseudo-tested methods
•A method executed by the test suite
•No extreme mutation is detected
•Found in well tested projects

68. class SingletonListIterator
implements Iterator<Node> {
...
void add() {
throw
new
UnsupportedOperationException();
}
...
}
A pseudo-tested method
68
Apache Commons Collections

69. A pseudo-tested method
69
class SingletonListIterator
implements Iterator<Node> {
...
void add() {
throw
new
UnsupportedOperationException();
}
...
}
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
}
Apache Commons Collections

70. class SingletonListIterator
implements Iterator<Node> {
...
void add() {
throw
new
UnsupportedOperationException();
}
...
}
A pseudo-tested method
70
Pseudo-tested
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
}
Apache Commons Collections

71. class SingletonListIterator
implements Iterator<Node> {
...
void add() {
throw
new
UnsupportedOperationException();
}
...
}
A pseudo-tested method
71
No exception is thrown
A fail is needed here
Pseudo-tested
@Test
void testAdd() {
SingletonListIterator it = ...;
try {
it.add(value);
}
catch(Exception ex) {}
}
Apache Commons Collections

72. public class VersionedSet {
private long version = 0;
private ArrayList elements = new ArrayList();
public void add(Object item) {
if (! elements.contains(item)) {
elements.add(item);
incrementVersion();
}
}
private void incrementVersion() { version++; }
}
A pseudo-tested method (2)
72

73. public class VersionedSet {
private long version = 0;
private ArrayList elements = new ArrayList();
public void add(Object item) {
if (! elements.contains(item)) {
elements.add(item);
incrementVersion();
}
}
private void incrementVersion() { version++; }
}
A pseudo-tested method (2)
73
@Test
public void testAdd() {
VersionedSet list = new VersionedSet();
list.add(1);
assertEquals(list.size(), 1);
}

74. public class VersionedSet {
private long version = 0;
private ArrayList elements = new ArrayList();
public void add(Object item) {
if (! elements.contains(item)) {
elements.add(item);
incrementVersion();
}
}
private void incrementVersion() { version++; }
}
A pseudo-tested method (2)
74
@Test
public void testAdd() {
VersionedSet list = new VersionedSet();
list.add(1);
assertEquals(list.size(), 1);
}
Pseudo-tested

75. public class VersionedSet {
private long version = 0;
private ArrayList elements = new ArrayList();
public void add(Object item) {
if (! elements.contains(item)) {
elements.add(item);
incrementVersion();
}
}
private void incrementVersion() { version++; }
}
A pseudo-tested method (2)
75
@Test
public void testAdd() {
VersionedSet list = new VersionedSet();
list.add(1);
assertEquals(list.size(), 1);
}
Pseudo-tested
Testability issue

76. Descartes reports
76

77. Descartes reports
77

78. Descartes reports
78

79. Descartes reports
79

80. Descartes results
80

81. Unit test amplification
81
Descartes

82. Unit test amplification
82
Descartes &
DSpot

83. DSpot
•Automatically enhances existing JUnit test suites
•Generate new assertions or new test cases
83

84. Dspot principle
84

85. x
Dspot principle
85
Test criterion

86. x
Dspot principle
86
Test criterion
(Mutation score)

87. x
Dspot principle
87
Test criterion
(Mutation score)

88. x
Dspot principle
88
Test criterion
(Mutation score)

89. x
Dspot principle
89
Test criterion
(Mutation score)

90. Dspot – How does it work ?
90

91. Dspot – How does it work ?
91
Mutation
Analysis
✗
✓
✗
✗
✓

92. Dspot – How does it work ?
92
Amplified
Test 1
Mutation
Analysis
✗
✓
✗
✗
✓
amplify

93. 93
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
benjamin.eat(kouignAmann);
assertFalse(benjamin.isHungry());
}
Example
Original
test case

94. 94
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
benjamin.eat(kouignAmann);
assertFalse(benjamin.isHungry());
}
Example
Original
test case
Remove a method call

95. 95
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
assertFalse(benjamin.isHungry());
}
Example
Original
test case
Remove existing assertions
Remove a method call

96. 96
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
benjamin.isHungry();
Log.log(benjamin, id : "benjamin");
}
Example
Original
test case
Instrument the test

97. 97
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
benjamin.isHungry();
Log.log(benjamin, id : "benjamin");
}
Example
Original
test case
Instrument the test
Run the
instrumented test

98. benjamin.isHungry()true
benjamin.isHappy()false
98
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
benjamin.isHungry();
Log.log(benjamin, id : "benjamin");
}
Example
Original
test case
Instrument the test
Run the
instrumented test

99. benjamin.isHungry()true
benjamin.isHappy()false
99
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
benjamin.isHungry();
Log.log(benjamin, id : "benjamin");
}
Example
Original
test case
Instrument the test
Run the
instrumented test
assertTrue(benjamin.isHungry());
assertFalse(benjamin.isHappy());Generate assertions

100. @Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
assertTrue(benjamin.isHungry());
assertFalse(benjamin.isHappy());
} 100
Example
Original
test case
Amplified
test case
@Test
public void html() {
Food kouignAmann = new Food("KouignAmann");
PhD benjamin = new PhD("Benjamin");
benjamin.eat(kouignAmann);
assertFalse(benjamin.isHungry());
}

101. Dspot – How does it work ?
101
Amplified
Test 1
Mutation
Analysis
✗
✓
✗
✗
✓
amplify

102. Dspot – How does it work ?
102
Amplified
Test 1
Mutation
Analysis
✗
✓
✗
✗
✓
✗
✓
✗
✓
✓
amplify

103. Dspot – How does it work ?
103
Amplified
Test 1
Mutation
Analysis
✗
✓
✗
✗
✓
✗
✓
✗
✓
✓
amplify

104. Dspot – How does it work ?
104
Amplified
Test 1
Mutation
Analysis
✗
✓
✗
✗
✓
✗
✓
✗
✓
✓
amplify

105. Dspot – How does it work ?
105
Amplified
Test 1
Amplified
Test 2
Mutation
Analysis
✗
✓
✗
✗
✓
✗
✓
✗
✓
✓
amplify

106. Dspot – How does it work ?
106
Amplified
Test 1
Amplified
Test 2
Mutation
Analysis
✗
✓
✗
✗
✓
✗
✓
✗
✓
✓ ✓
✓
✓
✗
✓
amplify

107. Dspot – How does it work ?
107
Amplified
Test 1
Amplified
Test 2
Mutation
Analysis
✗
✓
✗
✗
✓
✗
✓
✗
✓
✓ ✓
✓
✓
✗
✓
amplify

108. Dspot – How does it work ?
108
Amplified
Test 1
Amplified
Test 2
Mutation
Analysis
✗
✓
✗
✗
✓
✗
✓
✗
✓
✓ ✓
✓
✓
✗
✓
amplify

109. Dspot – How does it work ?
109
Amplified
Test 1
Amplified
Test 2
Amplified
Test 3
Mutation
Analysis
✗
✓
✗
✗
✓
✗
✓
✗
✓
✓ ✓
✓
✓
✗
✓
amplify

110. Dspot – How does it work ?
110
Amplified
Test 1
Amplified
Test 2
Amplified
Test 3
Mutation
Analysis
✗
✓
✗
✗
✓
✗
✓
✗
✓
✗
✓
✗
✗
✓
✓ ✓
✓
✓
✗
✓
amplify

111. Dspot – How does it work ?
111
Amplified
Test 1
Amplified
Test 2
Amplified
Test 3
Mutation
Analysis
✗
✓
✗
✗
✓
✗
✓
✗
✓
✗
✓
✗
✗
✓
✓ ✓
✓
✓
✗
✓
amplify

112. Dspot – How does it work ?
112
Amplified
Test 1
Amplified
Test 2
Amplified
Test 3
Mutation
Analysis
✗
✓
✗
✗
✓
✗
✓
✗
✓
✗
✓
✗
✗
✓
✓ ✓
✓
✓
✗
✓
amplify

113. Dspot – How does it work ?
113
Amplified
Test 1
Amplified
Test 2
Amplified
Test 3
Mutation
Analysis
✗
✓
✗
✗
✓
✗
✓
✗
✓
✗
✓
✗
✗
✓
✓ ✓
✓
✓
✗
✓
✗
✗
✗
✗
✗
amplify

114. 114
DSpot
Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects. B. Danglot, O. Luis Vera-Pérez, B. Baudry, M. Monperrus. Submitted to EMSE.

115. The pull request loop
Collaborative
platform
115

116. The pull request loop
Collaborative
platformpull req.
116

117. The pull request loop
Collaborative
platformpull req.
code
117

118. The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
118

119. The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
119

120. The pull request loop
pull req.
code
analysesfeedback
120

121. The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
121

122. The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
122
Novel analyses

123. The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
123
Novel analyses

124. Integration
•Command line
•Eclipse plugins
•Maven plugins
•Gradle plugins

125. The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
125
Novel analyses

126. The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
126
Novel analyses

127. Integration
•Jenkins
• Plugin to monitor score and pseudo-tested methods
• Xwiki
• Same strategy used with the code coverage
• Threshold on the mutation score
• https://massol.myxwiki.org/xwiki/bin/view/Blog/MutationTestingDescar
tes

128. The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
128
Novel analyses

129. The pull request loop
Collaborative
platformpull req.
code
analysesfeedback
129
Novel analyses

130. Integration
•In the CI as a service
• GitHub Application
• Find pseudo-tested methods in pull-requests
• Feedback using GitHub Check Runs API

131. More
•Open Source tools
• https://github.com/STAMP-project
•Public website
• https://www.stamp-project.eu
•Medium on Descartes
• https://medium.com/@almyre/short-circuiting-method-
executions-to-assess-test-quality-2d3fda45bc7f
•Publications
• https://www.stamp-project.eu/view/main/publications
131

132. Contacts
https://github.com/STAMP-project/
http://stamp-project.eu/
baudry@kth.se
caroline.landry@inria.fr
http://www.diverse-team.fr/
barais@irisa.fr
132

133. 133