An important challenge in creating automated tests is how to design test fixtures, i.e., the setup code that initializes the system under test before actual automated testing can start. Test designers have to choose between different approaches for the setup, trading off maintenance overhead with slow test execution. Over time, test code quality can erode and test smells can develop, such as the occurrence of overly general fixtures, obscure inline code and dead fields. In this paper, we investigate how fixture-related test smells evolve over time by analyzing several thousand revisions of five open source systems. Our findings indicate that setup management strategies strongly influence the types of test fixture smells that emerge in code, and that several types of fixture smells often emerge at the same time. Based on this information, we recommend important guidelines for setup strategies, and suggest how tool support can be improved to help in both avoiding the emergence of such smells as well as how to refactor code when test smells do appear.

1. Strategies for Avoiding Test
Fixture Smells during
Software Evolution
Michaela Greiler, Andy Zaidman, Arie van Deursen,
Margaret-Anne Storey

2. Test Fixture Smells
2

3. What is a test smell?
A symptom of a problem
• Can originate from several causes
• Refactoring needed
3

4. Structure of a Test
4

5. Test (method)
Test Structure
1. Statement
2. Statement
3. Statement
4. Statement
5. Statement
6. Statement
Setup of Test Fixture
Verify Outcome
Exercise SUT
Teardown Test Fixture
5

6. Test Fixture
The code that initializes and configures the system under test
6

7. How to structure the test fixture
and where to place it?

8. Test – Inline Fixture
Setup within method
Test Fixture
8

9. Test – Implicit
Setup
Method
9

10. Test – Delegate
Method
createTestAirportOriginTest ()
createTestAirportOriginTest(); explicit invocation
deleteTestAirportOriginTest()
deleteTestAirportOriginTest();
}
10
Delegate

11. “Most of the complexity of writing tests involves
how to write the Test Methods; what to include
inline and what to factor out into Test Utility
Methods, and so on.”
(Meszaros)
11

12. Test Fixture Smells
• General Fixture
• Test Maverick
• Dead Fields
• Lack of Cohesion of Test Methods
• Vague Header
• Obscure In-line setup
ICST 2013: Greiler, van Deursen, Storey
Automatic Detection of Test Fixture
Strategies and Smells
12

13. Developers recognize test fixture smells as a
problem(ICST’13)
13

14. Developers recognize test fixture smells as a
problem(ICST’13)
Resolving these smells after a long time can be problematic.
Immediately awareness of smells density important.
14

15. 1. Investigate the evolution of test fixture smells and
2. Understand which software changes lead to increased test smell densities
to develop strategies to avoid fixture smells and
to determine the best time to alert developers.
15
Resolving these smells after a long time can be problematic.
Immediately awareness of smells density important.

16. Azure
39 KLOCs
30 T-classes
358 T-methods
300 revisions
1 year
JSoup
20 KLOCs
23 T-classes
372 T-methods
973 revisions
2.5 years
Checkstyle
66 KLOCs
156 T-classes
549 T-methods
2251 revisions
9.5 years
PMD
174 KLOCs
118 T-classes
739 T-methods
1900 revisions
5 years
Voldemort
130 KLOCs
132 T-classes
520 T-methods
2900 revisions
1.5 years
Case studies
16

17. TestEvoHound
…is a static analysis tool that analyzes the test
fixture smell evolution.

18. TestEvoHound
…is a static analysis tool that analyzes the test
fixture smell evolution.
1.Revision Checkout
2.Build Process
3.Test Fixture Smell Analysis
4.Trend Analysis

19. Fixture smells among projects
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Azure Checkstyle JSoup PMD Voldemort
Test Mavericks General Fixture Vague Header Dead Fields LCOTM Inline
19

20. Does Test Fixture Smell Density Increase?
A general growth of test fixture smells over time
does not occur.
20
Lehman’s law: when a system evolves, its complexity increases unless work is
done to maintain or reduce it.

21. Increase in smell density
Checkstyle: dead fields
Refactoring and forgotten functionality: “Added a helper method to create a
configuration for a check...”.
21

22. Decrease in smell density
Azure: general fixture
New, non-smelly tests added: “Table Client commit [...]”.
22

23. Smell Dispersion
Over time, smelly
classes get smellier
Smells cluster in a few
classes
23

24. Correlations
?
Number of t-methods per test class correlates with test fixture smell density
Number of fields do not necessarily correlate with smell density
# T-methods
# Fields
24

25. Strategies and Recommendations
Keep test classes small
Keep inheritance structures flat
Limit the scope of super classes
Use composition instead of inheritance
Reconsider the “one test class per class” organization
25

26. Findings
0%
20%
40%
60%
80%
100%
Azure Checkstyle JSoup PMD VoldemortTest Mavericks General Fixture Vague Header
Dead Fields LCOTM Inline
# T-methods# Fields
26

27. 0%
20%
40%
60%
80%
100%
Azure Checkstyle JSoup PMD VoldemortTest Mavericks General Fixture Vague Header
Dead Fields LCOTM Inline
# T-methods# Fields
Thanks! Questions?
Connect with me
Michaela Greiler
michaela.greiler@microsoft.com
mgreiler
27