6. Research Questions
Code Review Experiment
with 92 developers
~150 reviews
9 interviews +
survey with 103 respondents
RQ1: Does the order of presenting test code to the
reviewer influence the code review’s effectiveness?
RQ2: How do developers perceive the practice of
Test-Driven Code Review?
7. Research Questions
RQ1: Does the order of presenting test code to the
reviewer influence the code review’s effectiveness?
RQ2: How do developers perceive the practice of
Test-Driven Code Review?
Code Review Experiment
with 92 developers
~150 reviews
9 interviews +
survey with 103 respondents
8. RQ1: Does the order of presenting test code to the reviewer influence the code review’s
effectiveness? — Method
review
1st review
review
2nd review
(Optional)
9. RQ1: Does the order of presenting test code to the reviewer influence the code review’s
effectiveness? — Method
Dependent Variables
Bugs
• 2 different types:
1. Maintainability Issues
2. Functional Defects
• Both in test and in production code
10. RQ1: Does the order of presenting test code to the reviewer influence the code review’s
effectiveness? — Method
Production First
A.java ATest.javaA.javaATest.java
Test First
Independent Variable:
Treatment
A.java
Only Production
11. RQ1: Does the order of presenting test code to the reviewer influence the code review’s
effectiveness? — Method
review
2nd review
review
1st review
Test First Production First
Participant A
review
1st review
Production First
review
2nd review
Only Production
Participant B
12. RQ1: Does the order of presenting test code to the reviewer influence the code review’s
effectiveness? — Method
Control Variables
Code and Review Details, Participant’s Profile
• Duration of the code review
• Patch
• Role
• Experience:
• Reviewing
• Programming
• …
13. RQ1: Does the order of presenting test code to the reviewer influence the code review’s
effectiveness? — Method
Demographics
confounders
review
1st review
32 participants completed
only the 1st review
60 participants
completed
both reviews
review
2nd review
(Optional)
14. Found Maintainability Issues
0%
20%
40%
60%
80%
100%
18%21%13% 8%18%
RQ1: Does the order of presenting test code to the reviewer influence the code review’s
effectiveness? — Results
Found Functional Defects (Bugs)
0%
20%
40%
60%
80%
100%
28%33%17% 28%40%
Test First Production First Only Production
p 0.01
medium eff. size
p 0.01
small eff. size
p 0.05p 0.05
In Test Code In Production Code In Test Code In Production Code
15. RQ1: Does the order of presenting test code to the reviewer influence the code review’s
effectiveness? — Results
Test Bugs Production Maintainability Issues
Estimate Significance Estimate Significance
Total duration -0.73 ** 0.04 .
Is First Review 0.16 . -0.15
Patch 2 -0.92 *** -2.84 ***
Treatment Prod. First -0.09
Treatment Test First 1.17 ** -1.24 **
Review Practice 0.06 0.25 .
Programming Pract. -0.66 -0.21
Profession Dev Exp. -0.09 -0.29 *
Java Exp. -0.01 -0.03
Worked Hours -0.02 -0.04
significance codes: ’***’p 0.001, ’**’p 0.01, ’*’p 0.05, ’.’p 0.1
16. RQ1: Does the order of presenting test code to the reviewer influence the code review’s
effectiveness? — Results
Test Bugs Production Maintainability Issues
Estimate Significance Estimate Significance
Total duration -0.73 ** 0.04 .
Is First Review 0.16 . -0.15
Patch 2 -0.92 *** -2.84 ***
Treatment Prod. First -0.09
Treatment Test First 1.17 ** -1.24 **
Review Practice 0.06 0.25 .
Programming Pract. -0.66 -0.21
Profession Dev Exp. -0.09 -0.29 *
Java Exp. -0.01 -0.03
Worked Hours -0.02 -0.04
significance codes: ’***’p 0.001, ’**’p 0.01, ’*’p 0.05, ’.’p 0.1
17. RQ1: Does the order of presenting test code to the reviewer influence the code review’s
effectiveness? — Results
Test Bugs Production Maintainability Issues
Estimate Significance Estimate Significance
Total duration -0.73 ** 0.04 .
Is First Review 0.16 . -0.15
Patch 2 -0.92 *** -2.84 ***
Treatment Prod. First -0.09
Treatment Test First 1.17 ** -1.24 **
Review Practice 0.06 0.25 .
Programming Pract. -0.66 -0.21
Profession Dev Exp. -0.09 -0.29 *
Java Exp. -0.01 -0.03
Worked Hours -0.02 -0.04
significance codes: ’***’p 0.001, ’**’p 0.01, ’*’p 0.05, ’.’p 0.1
18. Review Practice,
Progr. Practice,
Java Exp.
With TDR,
participants found
more bugs in test
code and the same
in production code
With TDR,
participants found
less maint. issues
in production code
RQ1: Does the order of presenting test code to the reviewer influence the code review’s
effectiveness? — Results
19. Research Questions
Code Review Experiment
with 92 developers
~150 reviews
9 interviews +
survey with 103 respondents
RQ1: Does the order of presenting test code to the
reviewer influence the code review’s effectiveness?
RQ2: How do developers perceive the practice of
Test-Driven Code Review?
21. Working context:
3 OSS
6 Closed Source
9 interviews
103 survey respondents
44% between 2 to 5 years of Experience
28 % between 6 to 10 years of Experience
RQ2: How do developers perceive Test-Driven Code Review?
22. Adoption: How often respondents use TDR
RQ2: How do developers perceive Test-Driven Code Review?
0
10
20
30
40
50
Never Almost never Sometimes Almost always Always
23. Tests give less
knowledge on the
production
behavior
Tests have low
code quality
Can not choose
a different order
in CR tool
Tests are perceived
as less important
Perceived problems with TDR
RQ2: How do developers perceive Test-Driven Code Review?
24. Black-box view of
production code
Useful when
the developer is
unfamiliar with the
code
improved test
code quality
Perceived advantages of TDR
RQ2: How do developers perceive Test-Driven Code Review?
25. Research Questions
RQ1: Does the order of presenting test code to the reviewer influence the
code review’s effectiveness?
RQ2: How do developers perceive the practice of Test-Driven Code Review?
26. The order matters Context dependent
Educate on test writing
and reviewing
Test code importance
==
Prod. code importance