Jaekwon Lee, Dongsun Kim, Tegawendé F. Bissyandé, Woosung Jung and Yves Le Traon, “Bench4BL: Reproducibility Study on the Performance of IR-Based Bug Localization”, in Proceedings of the 27th International Symposium on Software Testing and Analysis (ISSTA 2018), Amsterdam, Netherlands, July 16 – 21, 2018.
Dev Dives: Streamline document processing with UiPath Studio Web
Bench4BL: Reproducibility Study on the Performance of IR-Based Bug Localization
1. Bench4BL: Reproducibility Study on
the Performance of IR-Based Bug
Localization
Jaekwon Lee1, Dongsun Kim1, Tegawendé F. Bissyandé1,
Woosung Jung2, Yves Le Traon1
1SnT, University of Luxembourg - Luxembourg
2Seoul National University of Education - South Korea
17. Experiment
Data Set
RQ1: To what extent do IRBL techniques
perform on up-to-date subjects?
Research Questions
!17
18. Experiment
Data Set
Experiment
Configuration
RQ1: To what extent do IRBL techniques
perform on up-to-date subject?
RQ2: What is the impact of version
matching on the performance of IRBL
techniques?
RQ3: To what extent are IRBL techniques
sensitive to the inclusion of test code files?
Research Questions
!18
19. Experiment
Data Set
Experiment
Configuration
Potential
Improvement
RQ1: To what extent do IRBL techniques
perform on up-to-date subject?
RQ2: What is the impact of version
matching on the performance of IRBL
techniques?
RQ3: To what extent are IRBL techniques
sensitive to the inclusion of test code files?
RQ4: What potential performance gain can
be reached by leveraging duplicate bug
reports?
Research Questions
!19
20. BugLocator
(ICSE 2012)
BLIA
(APSEC 2015)
Locus
(ICSE 2016)
AmaLgam
(ICPC 2014)
BRTracer
(ICSME 2014)
BLUiR
(ASE 2013)
IRBL Features Sub Modules
Bug report fixing historyFull text
Code
segmentations
Identifiers
Identifiers
Identifiers
Identifiers
Bug report fixing history
Bug report fixing history, Revision history
Revision history
Bug report fixing history
Stack Trace Analysis, Revision history
Bug report fixing history,
Stack Trace Analysis
IRBL Techniques we used
!20
31. ●●
●●
0.359
0.35
0.359
0.365
0.363
0.38Locus
BLIA
AmaLgam
BLUiR
BRTracer
BugLocator
0.00 0.25 0.50 0.75 1.00
Distribution of MAP values of all subjects for each techniques
0.455
0.501
0.43
0.516
0.497
0.506Locus
BLIA
AmaLgam
BLUiR
BRTracer
BugLocator
0.00 0.25 0.50 0.75 1.00
Distribution of MRR values of all subjects for each techniques
Baseline Performance
Bug localization still has much room for improvement.
!31
32. Technique
Old Subjects New Subjects
MAP MRR MAP MRR
BugLocator 0.2692 0.3985 ↗0.3052 ↗0.4223
BRTracer 0.2645 0.3664 ↗0.3330 ↗0.4690
BLUiR 0.3102 0.4556 0.2881 0.3869
AmaLgam 0.2950 0.4072 0.2906 0.3899
BLIA 0.2935 0.4242 ↗0.3014 0.4155
Locus 0.2641 0.3399 ↗0.3289 ↗0.4430
Single version matching
Test files included
Summary of MAP/MRR of IRBL techniques
!32
Configuration
RQ1:
The use of old vs. new subjects
33. Summary of MAP/MRR of IRBL techniques
Not over-fitted to old subjects
Technique
Old Subjects New Subjects
MAP MRR MAP MRR
BugLocator 0.2692 0.3985 ↗0.3052 ↗0.4223
BRTracer 0.2645 0.3664 ↗0.3330 ↗0.4690
BLUiR 0.3102 0.4556 0.2881 0.3869
AmaLgam 0.2950 0.4072 0.2906 0.3899
BLIA 0.2935 0.4242 ↗0.3014 0.4155
Locus 0.2641 0.3399 ↗0.3289 ↗0.4430
!33
Single version matching
Test files included
Configuration
RQ1:
The use of old vs. new subjects
34. !34
Summary of MAP/MRR of IRBL techniques
Technique
Single Version Multiple Version
MAP MRR MAP MRR
BugLocator 0.3052 0.4223 ↗0.3713 ↗0.5075
BRTracer 0.3330 0.4690 ↗0.3992 ↗0.5526
BLUiR 0.2881 0.3869 ↗0.3623 ↗0.4802
AmaLgam 0.2906 0.3899 ↗0.3657 ↗0.4840
BLIA 0.3014 0.4155 ↗0.3777 ↗0.5124
Locus 0.3289 0.4430 ↗0.4217 ↗0.5514
New subjects
Test files included
Configuration
RQ2:
The importance of version matching
35. New subjects
Test files included
Summary of MAP/MRR of IRBL techniques
The evaluation/execution of IRBL techniques should apply
multiple version matching
!35
Technique
Single Version Multiple Version
MAP MRR MAP MRR
BugLocator 0.3052 0.4223 ↗0.3713 ↗0.5075
BRTracer 0.3330 0.4690 ↗0.3992 ↗0.5526
BLUiR 0.2881 0.3869 ↗0.3623 ↗0.4802
AmaLgam 0.2906 0.3899 ↗0.3657 ↗0.4840
BLIA 0.3014 0.4155 ↗0.3777 ↗0.5124
Locus 0.3289 0.4430 ↗0.4217 ↗0.5514
Configuration
RQ2:
The importance of version matching
36. ConfigurationSummary of MAP/MRR of IRBL techniques
!36
Multiple version matching
New subjects
RQ3:
The impact of test file inclusion
Technique
Test files excluded Test files included
MAP MRR MAP MRR
BugLocator 0.3811 0.4647 0.3713 ↗0.5075
BRTracer 0.4141 0.5090 0.3992 ↗0.5526
BLUiR 0.3603 0.4385 ↗0.3623 ↗0.4802
AmaLgam 0.3633 0.4420 0.3657 ↗0.4840
BLIA 0.3902 0.4728 ↗0.3777 ↗0.5124
Locus 0.4146 0.5002 ↗0.4217 ↗0.5514
37. Summary of MAP/MRR of IRBL techniques
Technique
Test files excluded Test files included
MAP MRR MAP MRR
BugLocator 0.3811 0.4647 0.3713 ↗0.5075
BRTracer 0.4141 0.5090 0.3992 ↗0.5526
BLUiR 0.3603 0.4385 ↗0.3623 ↗0.4802
AmaLgam 0.3633 0.4420 0.3657 ↗0.4840
BLIA 0.3902 0.4728 ↗0.3777 ↗0.5124
Locus 0.4146 0.5002 ↗0.4217 ↗0.5514
!37
Including test files does not bring bias or noise
Configuration
Multiple version matching
New subjects
RQ3:
The impact of test file inclusion
43. Bug-Code Linking
Bug Report
CAMEL-12558: Transacted and Policy should not have outputs
M main/java/org/apache/camel/model/PolicyDefinition.java
M main/java/org/apache/camel/model/TransactedDefinition.java
A test/java/org/apache/camel/catalog/CamelCatalog.java
A main/java/org/apache/camel/tools/apt/CoreEipAnnotationPrint.java
Added camel-web3j Spring-boot test
A test/java/org/apache/camel/itest/springboot/CamelWeb3jTest.java
Update GoogleBigQueryProducer.java
M main/java/org/apache/camel/component/GoogleBigQueryProducer.java
Code
Repository
Commit Log
!43
44. Bug-Code Linking
Bug Report
CAMEL-12558: Transacted and Policy should not have outputs
M main/java/org/apache/camel/model/PolicyDefinition.java
M main/java/org/apache/camel/model/TransactedDefinition.java
A test/java/org/apache/camel/catalog/CamelCatalog.java
A main/java/org/apache/camel/tools/apt/CoreEipAnnotationPrint.java
Added camel-web3j Spring-boot test
A test/java/org/apache/camel/itest/springboot/CamelWeb3jTest.java
Update GoogleBigQueryProducer.java
M main/java/org/apache/camel/component/GoogleBigQueryProducer.java
Code
Repository
Commit Log
!44
52. Test File Inclusion
CAMEL-12558: Transacted and Policy should not have outputs
M main/java/org/apache/camel/model/PolicyDefinition.java
M main/java/org/apache/camel/model/TransactedDefinition.java
A test/java/org/apache/camel/catalog/CamelCatalog.java
A main/java/org/apache/camel/tools/apt/CoreEipAnnotationPrint.java
Added camel-web3j Spring-boot test
A test/java/org/apache/camel/itest/springboot/CamelWeb3jTest.java
Update GoogleBigQueryProducer.java
M main/java/org/apache/camel/component/GoogleBigQueryProducer.java
Code
Repository
Commit LogBugLocator
BLIA
Locus
AmaLgam
BRTracer
BLUiR
!52
53. Test File Inclusion
CAMEL-12558: Transacted and Policy should not have outputs
M main/java/org/apache/camel/model/PolicyDefinition.java
M main/java/org/apache/camel/model/TransactedDefinition.java
A test/java/org/apache/camel/catalog/CamelCatalog.java
A main/java/org/apache/camel/tools/apt/CoreEipAnnotationPrint.java
Added camel-web3j Spring-boot test
A test/java/org/apache/camel/itest/springboot/CamelWeb3jTest.java
Update GoogleBigQueryProducer.java
M main/java/org/apache/camel/component/GoogleBigQueryProducer.java
Code
Repository
Commit LogBugLocator
BLIA
Locus
AmaLgam
BRTracer
BLUiR
We remove
including “test” or “Test” in a path or filename
!53