The document discusses using usage expertise versus implementation expertise to recommend experts for code changes. An experiment shows that while implementation expertise recommendations are correct 80% of the time within projects, usage expertise alone is only correct 50% of the time. However, using usage expertise plus context correctly recommends experts as often as implementation expertise. A second experiment finds it is possible to recommend developers across projects 25-75% of the time using usage heuristics. However, the precision of cross-project recommendations and recommendations for new projects requires more investigation.
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Expert Recommendation with Usage Expertise
1. Expert Recommendation
with Usage Expertise
David Ma David Schuler
University of Calgary Saarland University
Tom Zimmermann Jonathan Sillito
Microsoft Research University of Calgary
1
6. AutoBuildJob(Workspace workspace) {
super(Messages.events_building_0);
setRule(workspace.getRoot());
setPriority(BUILD);
isAutoBuilding = workspace.isAutoBuilding();
preferences.addPropertyChangeListener(this);
}
Anna changes AutoBuildJob.
She knows how this method works.
6
7. Implementation
Expertise
AutoBuildJob(Workspace workspace) {
super(Messages.events_building_0);
setRule(workspace.getRoot());
setPriority(BUILD);
isAutoBuilding = workspace.isAutoBuilding();
preferences.addPropertyChangeListener(this);
}
Anna changes AutoBuildJob.
She knows how this method works.
7
8. Implementation
Expertise
Projects unlikely to share code
This concept can’t transfer
expertise across projects
8
9. Developers moving between
multiple projects or that are
new to projects
Implementation
Expertise
New projects
don’t have data
Some projects don’t have
enough data or is of poor
quality
9
11. Usage
Expertise
AutoBuildJob(Workspace workspace) {
super(Messages.events_building_0);
setRule(workspace.getRoot());
setPriority(BUILD);
isAutoBuilding = workspace.isAutoBuilding();
preferences.addPropertyChangeListener(this);
}
Matt adds a call to setPriority().
Matt knows how to use setPriority()
without knowing how it was coded.
11
12. Usage
Expertise and Context
AutoBuildJob(Workspace workspace) {
super(Messages.events_building_0);
setRule(workspace.getRoot());
setPriority(BUILD);
isAutoBuilding = workspace.isAutoBuilding();
preferences.addPropertyChangeListener(this);
}
Matt adds a call to setPriority().
Matt knows how to use setPriority()
and the surrounding method calls.
12
13. Why
Usage Expertise?
Projects unlikely to share code
Projects likely to share calls to libraries
13
18. Ranking Usage Expertise
Frequency of Breadth of
Usage Usage
Frequency of Breadth of
Usage Relative Usage Relative
to Everyone Else to Everyone Else
18
19. Research Questions
Can Usage Expertise correctly
1 recommend experts within projects as
often as Implementation Expertise?
Can Usage Expertise + Context correctly
2 recommend experts within projects as
often as Implementation Expertise?
Is it possible to recommend developers
3 across projects?
19
20. Experiment 1 of 2
Recommending Within Projects
accuracy of implementation-based recommendations
vs. accuracy usage-based recommendations
20
21. Training
Database of Profiles
Bob’s Profile
CVS history Dan’s Profile
Alice’s Profile
21
22. Testing
Database of Changeset Ranked by Joe
Profiles impl. expertise
Matt
Changed Bob (expert)
+
Methods
Ranked by
Added usage expertise
Method
Kim
Calls Bob (expert)
Joe
Evaluation
Did we recommend the actual expert?
22
23. Experiment 2 of 2
Recommending Across Projects
Determining if usage-based recommendations
are possible across projects
23
24. Training
Database of Profiles
Eclipse CVS history Bob’s Profile
Jack’s Profile
AspectJ CVS history Dan’s Profile
Alice’s Profile
Kate’s Profile
24
25. Testing
Database of Changeset
Profiles
Changed Ranked by
usage expertise
+
Methods
Alice (Eclipse)
Added Jake (Eclipse)
Method Kim (AspectJ)
Calls
Evaluation
Did we recommend devs. from other projects?
25
26. Results
(Within Projects)
Impl-based recommendations are
correct ~80% of time time while usage
based are correct ~50% of the time.
26
27. Can Usage Expertise correctly
1 recommend experts within projects as
often as Implementation Expertise?
No
27
28. Results
(Within Projects)
Usage Expertise + Context correctly
recommends experts just as often as
Implementation Expertise
28
29. Can Usage Expertise + Context correctly
2 recommend experts within projects as
often as Implementation Expertise?
YES
29
30. Results
(Across Projects)
Our usage expertise heuristics
recommend developers 25-75% of
the time from other projects.
30
31. Is it possible to recommend developers
3 across projects?
YES
31
32. Is it possible to recommend developers
3 across projects?
YES
BUT
32
33. Open Questions
How precise are recommendations
made across projects?
Can we recommend for projects with
no or little history?
What if we combine both
implementation and usage expertise?
33
34. Research Questions
Can Usage Expertise correctly
recommend experts within projects as
often as Implementation Expertise?
No
Can Usage Expertise + Context correctly
recommend experts within projects as
often as Implementation Expertise?
YES
Is it possible to recommend developers
across projects? YES
34