Summarization Techniques for Code, Change, Testing and User Feedback - Validation, Analysis and Evolution of Software Tests (2018)

1. Sebastiano Panichella
Institut für Informatik
Universität Zürich
panichella@ifi.uzh.ch
Validation, Analysis and Evolution of Software Tests (2018)
Summarization Techniques
for Code, Change, Testing
and User Feedback

2. My Background
University of Sannio
PhD Student
June 2014
University of Salerno
Master Student
December 2010
University of Zurich
Research Associate
Since October 2014
2
University of Molise
Bachelor Student

3. My Background
University of Sannio
PhD Student
June 2014
University of Salerno
Master Student
December 2010
University of Zurich
Research Associate
Since October 2014
3
University of Molise
Bachelor Student
VST 2018

4. My Research Interests
4
software
learning
developers
maintenance
testingchanges
machine
program
comprehension
test
code
study
method
reviews
prediction
number
defects
classes
precision
set
source
information
projects
dataapp
user
cost
project
empirical
summaries
bugs
sentences
models
model
defect
analysis
engineering
modep
methods
work
case
cases
emails
api
surf
users
setup
evolution
releases
class
bug
decision
participants
recall
upgraded
documents
descriptions
specific
automated
timefeedback
chat
generated
review
defectprone
example
communication
training
terms
paragraphs
techniques
release
regression
step
issues
reports
sentence
predictors
logistic
metrics
coverage
multiobjective
research
issue
apps
found
problemlanguage
singleobjective
features
apache
java
categories
inspection
useful summary
tree
http
dependencies
page
sources
topics
lucene
automatically
mining
systems
algorithm
process
words
extracted
classification
size
performed
analyzed
mobile
category
values
system
topic
saved
natural
level
change
crossproject
mailing
context
experiment
tasks
commandsfolders
commandtxt
printed
analyzing
developer
heuristics
nlp
tests
generation
development
traceability
kloc
ecosystem
total
feature
validity
lists
exception
evaluation
links
text
documentation
description
value eclipse
identify
average
quality
reported
instead
similar
perform
approaches
effectiveness
client
consider
identified
comments
summarization
order
traditional
possible
analyze
manually
recovery
constraints
compared
defined
performance
working
authors
large
document
important
extract
future
finally
contain
classifier
open
relevant
aim
name
several
help
content
impact
list
loc
algorithms
percentage
upgrade
Word-cloud from my research papers

5. Why Software
Maintenance and Testing are
Relevant in our Society?
5

6. The Cost of Software Bugs
National Institute of
Standards and Technology
$ 59.5
cost to US economy in 2013
40%-50%
could be saved with proper Software Maintenance and Testing
billion $ 312 billion
worldwide cost in 2013
6

7. Current Release Cycle
Release
Planning
1
Coding
Activity
Testing
Tasks
Deployment
Phase
Source
Code
Release
App
Reviews
User (or
costumer)
Feedback
2 3
5
6
CD/CI
4
Test
Code
7

8. Current Release Cycle
Manual testingAutomated testing
CD/CI
A/B testing
App reviews
Crowdsourcing
Microservices
Team coordination
Code Review Survey
Structured e-mails
Security
Performance Testing
…
……
…
Software documentation
Maintenance activities
Coding activities
8

9. “In modern software companies it is nowadays, crucial to enact a
software development process able to dynamically react to
market requirements (i.e., users requests), delivering at same
time high quality and reliable software”.
Current Release Cycle
9

10. What are the Most Expensive Activities
in Software Maintenance and Testing?
10

11. What are the Most Expensive Activities in
Software Maintenance and Testing?
What are the
Open Challenges?
11
Empirical Studies to
Understand
Developers’ Behaviour
Tools for Automating
Software Maintenance
and Testing
How to Identify
such Activities?
How to Reduce the
Cost of such
Activities?

12. Outline
What are the
Open Challenges?
12
Tools for Automating
Software Maintenance
and Testing
How to Identify
such Activities?
How to Reduce the
Cost Activities?
PART I PART II PART III
Empirical Studies to
Understand
Developers’ Behaviour
How to Reduce the
Cost of such
Activities?

13. Empirical Software Engineering
Empirical Studies To
Understand Developers
Behaviour
13
PART I

14. What is Empirical Software Engineering?
Isaac Newton
14

15. What is Empirical Software Engineering?
Isaac Newton
Empirical Studies To
Understand Developers
Behaviour
Join
a Project
Continuous Delivery
15

16. What is Empirical Software Engineering?
Isaac Newton
Empirical Studies To
Understand Developers
Behaviour
Join
a Project
Program
Comprehension
Tasks
Maintenance
tasks
Continuous Delivery
16

17. What is Empirical Software Engineering?
Isaac Newton
Empirical Studies To
Understand Developers
Behaviour
Development
Team
Join
a Project
Program
Comprehension
Tasks
Maintenance
tasks
Team Work Testing
tasks
Continuous Delivery
17

18. What is Empirical Software Engineering?
Isaac Newton
Empirical Studies To
Understand Developers
Behaviour
Development
Team
Join
a Project
Program
Comprehension
Tasks
Maintenance
tasks
Team Work Testing
tasks
Continuous Delivery
Data Extraction
Recommenders
• Versioning Systems
• Mailing Lists
• Issue trackers
• Q&A site (e.g. StackOverflow)
Empirical Studies

19. Empirical Studies To Understand Developers Behaviour
During Program Comprehension and Maintenance Tasks
Study A
How documentation and code are
browsed by developers to perform
maintenance activities?
Study B:
How does a software ecosystem evolve?
An Empirical Investigation on Documentation Usage Patterns in Maintenance Tasks. The 29th International Conference
on Software Maintenance (ICSM) - 2013
The Evolution of Project Inter-Dependencies in a Software Ecosystem: the Case of Apache.
NOMINATED AS BEST PAPER at the 29th IEEE International Conference on Software Maintenance (ICSM) - 2013
19

20. Study A: Context
• Object: software artifacts from SMOS, a school automation
system developed by professional developers of an IT Company
in Salerno (Italy).
• Subjects:
121 121 667 72
33 Developers
An Empirical Investigation on Documentation Usage Patterns in Maintenance Tasks.
The 29th International Conference on Software Maintenance (ICSM) - 2013
20
…………….
S DUJ

21. Maintenance Tasks
Bug Fixing:
Add a new feature:
Improve existing features:
21

22. How Much Time did Participants Spend on Different
Kinds of Artifacts?
72%
13%10%
3%2%
22
S DUJ

23. Navigation Patterns Followed By Developers Before
Reaching Source Code
23
S = Sequence Diagram
D = Class Diagram
U = Use Case
J = Javadoc
Simple Navigation
Patterns
(SD)+ = Sequence Diagram before Class Diagram
(US)+ = Use Case before Sequence Diagram
(DS)+ = Class Diagram before Sequence Diagram
U(SD)+ = Use Case before (SD)+
Complex Navigation
Patterns

24. S
D
(SD)+
(US)+
U(SD)+
(DS)+
J
U
S(US)+
SU(SD)+
Other 0.0545
0.0121
0.0121
0.0242
0.006
0.0364
0.0364
0.097
0.1212
0.1636
0.1151
0.042
0
0.011
0
0.032
0.011
0.011
0.021
0.021
0.084
0.179
Junior Developers
Professional Developers
S= Sequence Diagram
D= Class Diagram
U= Use Case
J= Javadoc
Most Frequent Navigation Patterns
Before Reaching Source Code
24

25. S
D
(SD)+
(US)+
U(SD)+
(DS)+
J
U
S(US)+
SU(SD)+
Other 0.0545
0.0121
0.0121
0.0242
0.006
0.0364
0.0364
0.097
0.1212
0.1636
0.1151
0.042
0
0.011
0
0.032
0.011
0.011
0.021
0.021
0.084
0.179
Junior Developers
Professional Developers
S= Sequence Diagram
D= Class Diagram
U= Use Case
J= Javadoc
Most Frequent Navigation Patterns
Before Reaching Source Code
25

26. 26
Lessons Learnt from Study A
Lesson 2
Lesson 1

27. 27
Lessons Learnt from Study A
Lesson 2
Lesson 1 During the Maintenance Tasks
Most of the Time Was Spent by
Developers on Source Code.

28. 28
Lessons Learnt from Study A
Lesson 2
Help (Junior) Developers during
Program Comprehension Tasks
Lesson 1
To Reduce the Maintenance
Cost we Need to:
During the Maintenance Tasks
Most of the Time Was Spent by
Developers on Source Code.

29. Effort in Program Comprehension
A. Von Mayrhauser and A. Vans. Program comprehension during
software maintenance and evolution. IEEE Computer, 28(8):44–55, Aug, 1995.
“Developers spend more time
reading than writing code”
29

30. Activities in Software Maintenance
Change
documentation
5%
Change
implementation
10%
Change
planning
10%
Change Testing
25%Source code
comprehension
50%
Source: Principles of Software Engineering and
Design, Zelkovits, Shaw, Gannon 1979
Activities of Software Maintenance
Source: Principle of Software Engineering and Design
Zelkovits, Shan, Gannon 1979
30

31. Activities and Cost of Software Maintenance
Relative Maintenance COST: between 50% and
75% of global cost is spent on maintenance!
Source: Principle of Software Engineering and Design
Zelkovits, Shan, Gannon 1979
Activities in Software Maintenance
Change
documentation
5%
Change
implementation
10%
Change
planning
10%
Change Testing
25%Source code
comprehension
50%
Source: Principles of Software Engineering and
Design, Zelkovits, Shaw, Gannon 1979
Cost of MaintenanceActivities of Maintenance
31

32. 32
Lessons from Study A
Lesson 2
Help (Junior) Developers during
Program Comprehension Tasks
Lesson 1
To Reduce the Maintenance
Cost we Need to:
During the Maintenance Tasks
Most of the Time Was Spent by
Developers on Source Code.
32

33. The Evolution of Project
Inter-Dependencies in a Software
Ecosystem: the Case of Apache
Gabriele Bavota, Gerardo Canfora, Massimiliano Di Penta,
Rocco Oliveto, Sebastiano Panichella
Study B: Context
How the Apache Community Upgrades Dependencies: An Evolutionary Study. Empirical Software Engineering (EMSE 2014).
Evolution of Project Inter-Dependencies in a Software Ecosystem: the Case of Apache.
NOMINATED AS BEST PAPER at the 29th IEEE International Conference on Software Maintenance (ICSM) - 2013
https://www.apache.org
33

34. How the Software Projects Evolve over the Time
34

35. How the Software Projects Evolve over the Time
35

36. View of Projects Dependencies in APACHE
36

37. View of Projects Dependencies in APACHE
37

38. View of Projects Dependencies in APACHE
38

39. View of Projects Dependencies in APACHE
39

40. View of Projects Dependencies in APACHE
40

41. View of Projects Dependencies in APACHE
41

42. 42
Lessons Learnt from Study B
Lesson 2
Lesson 1
To Reduce the Maintenance
Cost we Need:
Mentors or Expert to
Train Junior Developers

43. Motivation
https://community.apache.org/mentoringprogramme.html
https://community.apache.org/mentoringprogramme.html
43
Mentors in Software Projects

44. 44
Lessons from Study B
Lesson 2
Lesson 1
To Reduce the Maintenance
Cost we Need:
Mentors or Expert to
Train Junior Developers

45. 45
Lessons from Study B
Lesson 2 Need of Tools for Collaborative
Maintenance and Testing Tasks
Lesson 1
To Reduce the Maintenance
Cost we Need:
Mentors or Expert to
Train Junior Developers

46. 46
Lessons from Experiment B
Lesson 2
Need of Mentors or Expert to
Train Junior Developers
Need of Tools for Collaborative
Maintenance and Testing Tasks
Lesson 1
Who is Going to Mentor Newcomers in Open Source Projects?
International Symposium on the Foundations of Software Engineering
(SIGSOFT FSE 2012)

47. Suggest Appropriate Mentors to Help
Newcomers in Open Source Projects
Development
Team
Join
a Project
Program
Comprehension
Tasks
Maintenance
tasks
Team Work Testing
tasks

48. Mentoring of
Project Newcomers
is Highly Desirable….
Previous Work
Dagenais et al. ICSE 2010
MENTOR
48

49. • Small Projects: find Mentors
is a trivial problem
• Large Projects: find Mentors
is not a trivial problem
When a Newcomer Joins a Project
.........
49

50. Motivation
https://community.apache.org/mentoringprogramme.html
https://community.apache.org/mentoringprogramme.html
50
Identifying Mentors in Software Projects

51. Characteristics of a Good Mentor
51
Enough ability to help
other people.
Enough expertise
about the topic of interest
for the newcomer.

52. 1) Find Past Successful Mentors
2) Suggest Mentors Having Specific Skills
YODA
(Young and newcOmer Developer Assistant)
52
Approach for Mentors Identification in Open Source Projects
Who is Going to Mentor Newcomers in Open Source Projects?
International Symposium on the Foundations of Software Engineering (SIGSOFT FSE 2012)

53. Score Computed Aggregating the
Factors in a Weighted Sum
Identify Past Successful Mentors
∑=
5
1i
ii fw
F1
F2 >
F3
F4 - 1st
F5

54. When Alice joins
the project
Jim Alice
Is the mentor of
IF
F1
F2 >
F3
F4 - 1st
F5
F5
Time
F5: Commits
YODA

55. Score Computed Aggregating the
Factors in a Weighted Sum
Identify Past Successful Mentors
∑=
5
1i
ii fw
F1
F2 >
F3
F4 - 1st
F5

56. Recommending Mentors
56
Time
Project Developers

57. 57
Time
Project Developers
Newcomer
t
Recommending Mentors

58. 58
Time
Project Developers
Newcomer
t
?
Mentor with
Adequate Skills
Recommending Mentors

59. 59
Time
Past Mentors
Newcomer
t
Inspired to the
Work on Bug
Triaging by J.
Anvik et al.,
TOSEM 2011
Recommending Mentors

60. 60
Time
t
Inspired to the
Work on Bug
Triaging by J.
Anvik et al.,
TOSEM 2011
Recommending Mentors

61. 61
Time
t
Inspired to the
Work on Bug
Triaging by J.
Anvik et al.,
TOSEM 2011
Past Project Developers
Recommending Mentors

62. 62
Time
Past Mentors
Newcomer
t
Inspired to the
Work on Bug
Triaging by J.
Anvik et al.,
TOSEM 2011
Recommending Mentors

63. 63
Time
Past Mentors
Newcomer
t
Inspired to the
Work on Bug
Triaging by J.
Anvik et al.,
TOSEM 2011
Recommending Mentors

64. 64
Mentor RecommendaTons: Precision on Top 1 and Top 2
Precision
0%
25%
50%
75%
100%
Apache FreeBSD PostgreSQL Python Samba
82%
77%
100%
24%
81%
94%
64%
100%
30%
85%
Top 1
Top 2
Is it Possible to Recommend Mentors
to Project Newcomers?

65. 65
Mentor RecommendaTons: Precision on Top 1 and Top 2
Precision
0%
25%
50%
75%
100%
Apache FreeBSD PostgreSQL Python Samba
76%
84%
89%
45%
72%
92%91%90%
67%
80%
Top 1
Top 2
Results when Using both Mails and Issues
(Oracle Derived from Historical Data)

66. 66
Mentor RecommendaTons: Precision on Top 1 and Top 2
Precision
0%
25%
50%
75%
100%
Apache FreeBSD PostgreSQL Python Samba
76%
84%
89%
45%
72%
92%91%90%
67%
80%
Top 1
Top 2
With YODA it Possible
to Recommend Mentors
It is Possible to Recommend Mentors
to Project Newcomers?

67. 67
YODA Tool
YODA: Young and newcOmer Developer Assistant. In Proceedings of the
35th International Conference on Software Engineering (ICSE 2013)

68. Summary of PART I
Development
Team
Join
a Project
Program
Comprehension
Tasks
Maintenance
tasks
Team Work Testing
tasks
2) Suggesting Mentors or Expert to Junior Developers
Continuous Delivery
3) Need of Tools for Collaborative
Software Maintenance and Testing
1) Help (Junior) Developers during Program Comprehension Task
68

69. Development
Team
Join
a Project
Program
Comprehension
Tasks
Maintenance
tasks
Team Work Testing
tasks
1) Help (Junior) Developers during Program Comprehension Task
2) Suggesting Mentors or Expert to Junior Developers
Continuous Delivery
Who is Going to Mentor Newcomers in Open Source Projects? (SIGSOFT FSE 2012)
3) Need of Tools for Collaborative
Software Maintenance and Testing 69
Summary of PART I

70. Development
Team
Join
a Project
Program
Comprehension
Tasks
Maintenance
tasks
Team Work Testing
tasks
2) Suggesting Mentors or Expert to Junior Developers
Continuous Delivery
Who is Going to Mentor Newcomers in Open Source Projects? (SIGSOFT FSE 2012)
3) Need of Tools for Collaborative
Software Maintenance and Testing
1) Help (Junior) Developers during Program Comprehension Task
70
Summary of PART I

71. PART 2
How to Reduce the Cost of
Maintenance and Testing Activities?
71

72. Collaborate with Managers and
Developers of Various IT Companies
ITALYSWITZERLAND NETHERLAND JAPAN
(Credit Suisse,
Google, etc.)
(ING Netherland,
etc.)
(Telecom, Flosslab,
etc.)
(Sony Mobile
Communication, etc.)
Development
Team
Join
a Project
Program
Comprehension
Tasks
Maintenance
tasks
Team Work Testing
tasks
Continuous Delivery
72

73. Many Emerging Maintenance and Testing Problems
ITALYSWITZERLAND NETHERLAND JAPAN
(Credit Suisse,
Google, etc.)
(ING Netherland,
etc.)
(Telecom, Flosslab,
etc.)
(Sony Mobile
Communication, etc.)
Traceability
Recovery
Inconsistencies
between Software
Documents and Code
Need to Integrate User
Feedback in the
Maintenance Process
Need of better Tools
for Software Testing
73

74. Many Emerging Maintenance and Testing Problems
ITALYSWITZERLAND NETHERLAND JAPAN
(Credit Suisse,
Google, etc.)
(ING Netherland,
etc.)
(Telecom, Flosslab,
etc.)
(Sony Mobile
Communication, etc.)
Inconsistencies
between Software
Documents and Code
Need of better Tools
for Software Testing
Need to Integrate User
Feedback in the
Maintenance Process
Traceability
Recovery
74

75. Many Emerging Maintenance and Testing Problems
ITALYSWITZERLAND NETHERLAND JAPAN
(Credit Suisse,
Google, etc.)
(ING Netherland,
etc.)
(Telecom, Flosslab,
etc.)
(Sony Mobile
Communication, etc.)
Need to Integrate User
Feedback in the
Maintenance Process
Traceability
Recovery
Inconsistencies
between Software
Documents and Code
Need of better Tools
for Software Testing
75

76. Summarization Approaches

77. Summarization Approaches
“…have the general capability of automatically extracting or abstracting
key content from one or more sources of information thus, determining
the relevant information in the source being summarized and reducing
its content…”
1) indicative summary: it provides a direct link to the required content
relevant sources to users, so that they can read the provided information
more depth.
2) informative summary it has the goal to substitute the origisource
of information, by mainly assembling the relevant content, presenting it
in a new, more concise and structured form.
3) critical summary (or review) it eports or selects the main opinions
or statements related to a specific discussed topic, thus, it brings the
most relevant feedback, both positives and negatives, about a given
subject discussed in the source document.
77

78. Example of informative summary
MILAN, Italy, April 18. A small airplane crashed into a government
building in heart of Milan, setting the top floors on fire, Italian
police reported. There were no immediate reports on casualties as
rescue workers attempted to clear the area in the city's financial
district. Few details of the crash were available, but news reports
about it immediately set off fears that it might be a terrorist act
akin to the Sept. 11 attacks in the United States. Those fears sent
U.S. stocks tumbling to session lows in late morning trading.
Witnesses reported hearing a loud explosion from the 30-story
office building, which houses the administrative offices of the local
Lombardy region and sits next to the city's central train station.
Italian state television said the crash put a hole in the 25th floor
of the Pirelli building. News reports said smoke poured from the
opening. Police and ambulances rushed to the building in downtown
Milan. No further details were immediately available.

79. MILAN, Italy, April 18. A small airplane crashed into a government
building in heart of Milan, setting the top floors on fire, Italian
police reported. There were no immediate reports on casualties as
rescue workers attempted to clear the area in the city's financial
district. Few details of the crash were available, but news reports
about it immediately set off fears that it might be a terrorist act
akin to the Sept. 11 attacks in the United States. Those fears sent
U.S. stocks tumbling to session lows in late morning trading.
Witnesses reported hearing a loud explosion from the 30-story
office building, which houses the administrative offices of the local
Lombardy region and sits next to the city's central train station.
Italian state television said the crash put a hole in the 25th floor
of the Pirelli building. News reports said smoke poured from the
opening. Police and ambulances rushed to the building in downtown
Milan. No further details were immediately available.
How many victims?
Was it a terrorist act?
What was the target?
What happened?
Says who?
When, where?
Example of informative summary

80. 35
http://textcompactor.com/
Example of informative summary

81. 35
http://textcompactor.com/
Example of informative summary

82. Release Cycle
Maintenance activities
Coding activities
Automated testing
CD/CI
A/B testing
App reviews
Crowdsourcing
Microservices
Team coordination
Code Review Survey
Structured e-mails
Security
Performance Testing
…
…
…
…
?
…
Software documentation
Testing

83. Release Cycle
Maintenance activities
Coding activities
Automated testing
CD/CI
A/B testing
App reviews
Crowdsourcing
Microservices
Team coordination
Code Review Survey
Structured e-mails
Security
Performance Testing
… …
…Software documentation
…
…
Testing
?

84. Activities in Software Maintenance
Change
documentation
5%
Change
implementation
10%
Change
planning
10%
Change Testing
25%Source code
comprehension
50%
Source: Principles of Software Engineering and
Design, Zelkovits, Shaw, Gannon 1979
Source Code
Summaries: Why?
84

85. Source Code
Summaries: Why?
Prevent Maintenance Cost….
85

86. Understanding Code…
Not So Happy Developers
Happy Developers
Absence of Comments in the Code
again !!
Comments in the Code
again !!
86

87. Understanding Code…
Not So Happy Developers
Happy Developers
Absence of Comments in the Code
again !!
Comments in the Code
again !!
87

88. Source Code
Summaries: How?

89. Questions when Generating
Summaries of Java Classes
■ 1) What information to include in the summaries?
■ 2) How to generate and present the summaries?
89

90. Software Words Usage Model: deriving <actions>, <themes>,
and <secondary arguments> from class, methods, attributes
and variable identifiers
E. Hill et al. Automatically capturing
source code context of NL-queries
for software maintenance and reuse.
ICSE 2009
Summary Generator

91. When Navigating Java Classes…
https://github.com/larsb/atunesplus/blob/master/aTunes/src/main/java/net/sourceforge/atunes/kernel/modules/repository/audio/AudioFile.java
we look at
- Name of the Class
- Attributes
- Methods
- Dependencies between Classes
Source Code
Summaries: How?

92. When Navigating Java Classes…
Source Code
Summaries: How?
■ Generic responsibilities (domain independent)
■ Class stereotypes [Dragan et al., ICSM’10]
■ E.g., data class, entity, controller, boundary,
etc.

93. When Navigating Java Classes…
Source Code
Summaries: How?
■ Generic responsibilities (domain independent)
■ Class stereotypes [Dragan et al., ICSM’10]
■ E.g., data class, entity, controller, boundary,
etc.

94. An approach for Summarizing
a Java Class (JSummarizer)
[ Moreno at al. - ICPC 2013 ]
94

95. How to present and generate the summaries?
Other Code Artefacts can
be Summarised as well:
- Packages
- Classes
- Methods
- etc.http://www.cs.wayne.edu/~severe/jsummarizer/

96. [ Moreno at al. - ICPC 2013 ]
Evaluation of the Summaries

97. Potential Useful Code Descriptions can be Found
in Developers’ Discussions…
source'code'descrip+ons'in'external'
ar+facts.!
..................................................
When call the method IndexSplitter.split(File
destDir, String[] segs) from the Lucene cotrib
directory(contrib/misc/src/java/org/apache/
lucene/index) it creates an index with
segments descriptor file with wrong data.
Namely wrong is the number representing the
name of segment that would be created next in
this index.
..................................................
CLASS:'IndexSplitter! METHOD:'split!
■ Others researchers proposed to detect source code
descriptions from sources external to the source code:
■ Mailing list and Issue tracker
■ StackOverflow Discussions
[ Vassallo at al. - ICPC 2014 ][ Panichella at al. - ICPC 2012 ]

98. CODES: mining source code descriptions from developers discussions.
BEST TOOL AWARD at the 22nd International Conference on Program Comprehension (IEEE ICPC 2014) 98
CODES was able to
automatically re-document
25% OF
36% OF
WITH a precision >80%
Mining source code descriptions from developer communications.
International Conference on Program Comprehension (IEEE ICPC 2012)
CODES:
Approach for Mining Method Descriptions

99. Software Changes over the Time…
“…as consequence the original documentation tend to be
incomplete and inconsistent with the source code…”
Insufficient
Information
Source Code
Difficult
to Understand
APIs Documents
Inconsistent withComing back
to the reality...
Inconsistent/
Incomplete
99

100. Source Code
APIs Documents
Inconsistent
API Document Defects are Frequent
API Document JDK-1.8
Class:
InputEvent
Method:
getMaskForButton(int button)
————————————-—————————————-
————————————-—————————————-
https://docs.oracle.com/javase/8/docs/api/java/awt/event/InputEvent.html
“…and tend to be discovered and fixed after long time…”
http://stackoverflow.com/questions/2967303/inconsistency-in-java-util-concurrent-future
100

101. DRONE
DetectoR of dOcumentatioN dEfects
Code
API
Document
Software
Artifacts
AST Parsing
Pre-Process
and POS
Tagging
Defect
Reports
Control Flow-
Based Constraint
Analysis
SMT
Solver
Dependency
Parsing and
Pattern Analysis
Code Constraint
FOL Generating
Doc Constraint
FOL Generating
HeuristicsHeuristics
101

102. DRONE
DetectoR of dOcumentatioN dEfects
Code
API
Document
Software
Artifacts
AST Parsing
Pre-Process
and POS
Tagging
Defect
Reports
Control Flow-
Based Constraint
Analysis
SMT
Solver
Dependency
Parsing and
Pattern Analysis
Code Constraint
FOL Generating
Doc Constraint
FOL Generating
HeuristicsHeuristics
102

103. DRONE
DetectoR of dOcumentatioN dEfects
Code
API
Document
Software
Artifacts
AST Parsing
Pre-Process
and POS
Tagging
Defect
Reports
Control Flow-
Based Constraint
Analysis
SMT
Solver
Dependency
Parsing and
Pattern Analysis
Code Constraint
FOL Generating
Doc Constraint
FOL Generating
HeuristicsHeuristics
103

104. DRONE
DetectoR of dOcumentatioN dEfects
Code
API
Document
Software
Artifacts
AST Parsing
Pre-Process
and POS
Tagging
Defect
Reports
Control Flow-
Based Constraint
Analysis
SMT
Solver
Dependency
Parsing and
Pattern Analysis
Code Constraint
FOL Generating
Doc Constraint
FOL Generating
HeuristicsHeuristics
104

105. Code
API
Document
Software
Artifacts
AST Parsing
Pre-Process
and POS
Tagging
Defect
Reports
Control Flow-
Based Constraint
Analysis
SMT
Solver
Dependency
Parsing and
Pattern Analysis
Code Constraint
FOL Generating
Doc Constraint
FOL Generating
HeuristicsHeuristics
DRONE
Documentation Code
With DRONE we analyzed
over 1 million of LOC and more than
30,000 Javadoc documents
belonging to 8 java libraries
detecting around 2000 of
API documentation defects.
with high precision (values between
0.58 - 0.83) and an high recall (values >
0.81) results.
“Analyzing APIs Documentation and Code to Detect Directive Defects”. ICSE 2017
Evaluation of the Summaries
105

106. II. Code Change Summarization

107. 107
1) Generating Commit Messages via Summarization of Source Code Changes
2) Automatic Generation of Release Notes
[ Binkley at al. - ICSM 2013 ]
To Improve Commits
Quality
To Improve Releases
Note Quality
Task-Driven Software Summarization

108. 108
The what: changes implemented during the incremental change
The why: motivation and context behind the changes
Commit Message Should Describe

109. 109
10% of the messages were
removed:
- they were empty
- had very short strings or lacked
any semantical sense
Maalej and Happel - MSR 10
Commit Message Should Describe

110. 110
Generating Commit Messages via
Summarization of Source Code Changes

111. Original Message
This is a large modifier commit: this is a commit with many methods and combines
multiple roles. This commit includes changes to internationalization, properties or
configuration files (pom.xml). This change set is mainly composed of:
1. Changes to package retrofit.converter:
1.1. Add a Converter implementation for simple XML converter. It allows to:
Instantiate simple XML converter with serializer;
Process simple XML converter simple XML converter from body;
Convert simple XML converter to body
Referenced by:
SimpleXMLConverterTest class
Message Automatically
Generated
52% prefer
for small commits
64% prefer
for large commits
111

112. Release Cycle
Testing
Maintenance Activities
Coding Activities
CD/CI
A/B testing
App reviews
Crowdsourcing
Microservices
Team coordination
Code Review Survey
Structured e-mails
Security
Performance Testing
… …
…
…
… Software documentation
Automated testing

113. Release Cycle
Testing
Maintenance activities
Coding activities
CD/CI
A/B testing
App reviews
Crowdsourcing
Microservices
Team coordination
Code Review Survey
Structured e-mails
Security
Performance Testing
… …
…
…
… Software documentation
Automated testing

114. V.S.!
Integration of User Feedback in the
Maintenance Process

115. 115
Maintenance of Mobile Applications
“About one third of app reviews
contain useful information for developers”
Pagano et. al. RE2013

116. 116
“About one third of app reviews
contain useful information for developers”
Pagano et. al. RE2013
Manual analysis of User Reviews

117. 117
Chen et al – ICSE 2014! Panichella et al – ICSME 2015!
PAST WORK

118. 118
Feature Requests Bug Reports
The Problem

119. 119
SURF (Summarizer of User Reviews Feedback)
Summaries of User Reviews
What Would Users Change in My App? Summarizing App Reviews for Recommending Software Changes.
24th ACM SIGSOFT International Symposium on the Foundations of Software Engineering (FSE) - 2016.
SURF: Summarizer of User Reviews Feedback.
Proceedings of the 39th IEEE International Conference on Software Engineering (ICSE) - 2017

120. 120
User Review Model

121. 121
User Review Model
I love this app but it
crashes my whole iPad and it
has to restart itself
• User intention: Problem Discovery!
!
• Review topics: App, Model!
“…The User Reviews Model proposed by the authors is
impressive in how it analyzes a review sentence by
sentence and is able to characterize a sentence with
multiple labels…” – one of FSE reviewers!
11

122. 122
SUMMARIZER OF USER REVIEW FEEDBACK

123. 123
SUMMARIZER OF USER REVIEW FEEDBACK

124. 124
SUMMARIZER OF USER REVIEW FEEDBACK

125. 125
SUMMARIZER OF USER REVIEW FEEDBACK

126. 126
5. Summary Generation5!
http://www.ifi.uzh.ch/en/seal/people/panichella/tools/SURFTool.html
SURF: Summarizer of User Reviews Feedback.
Proceedings of the 39th IEEE International Conference on Software Engineering (ICSE) - 2017

127. 127
Case Study
Involving 23 Developers
2622
Reviews
Of
12
Apps

128. 128
Case Study
Involving 23 Developers
2622
Reviews
Of
12
Apps
JAPAN
Sony Mobile
Communication

129. 129
Results
V.S.!
2) SURF helps to prevent more than half
of the time required for analyzing users
feedback and planning software changes.

130. 130
Results
V.S.!
2) SURF helps to prevent more than half
of the time required for analyzing users
feedback and planning software changes.
3) 92% of manually extracted
feedback appears also in the
automatic generated summaries.

131. 131
Results
V.S.!
2) SURF helps to prevent more than half
of the time required for analyzing users
feedback and planning software changes.
3) 92% of manually extracted
feedback appears also in the
automatic generated summaries.
4) Summaries generated by SURF
are reasonably correct, adequate,
concise, and expressive.

132. Many Emerging Maintenance and Testing Problems
ITALYSWITZERLAND NETHERLAND JAPAN
(Credit Suisse,
Google, etc.)
(ING Netherland,
etc.)
(Telecom, Flosslab,
etc.)
(Sony Mobile
Communication, etc.)
Need to Integrate User
Feedback in the
Maintenance Process
Traceability
Recovery
Inconsistencies
Between Software
Documents and Code
Need of better Tools
For Software Testing

133. Many Emerging Maintenance and Testing Problems
ITALYSWITZERLAND NETHERLAND JAPAN
(Credit Suisse,
Google, etc.)
(ING Netherland,
etc.)
(Telecom, Flosslab,
etc.)
(Sony Mobile
Communication, etc.)
Need to Integrate User
Feedback in the
Maintenance Process
Traceability
Recovery
Inconsistencies
Between Software
Documents and Code
Need of better Tools
For Software Testing
133

134. Release Cycle
Maintenance activities
Coding activities
Automated testing
CD/CI
A/B testing
App reviews
Crowdsourcing
Microservices
Team coordination
Code Review Survey
Structured e-mails
Security
Performance Testing
…
…Software documentation…
Testing

135. Release Cycle
Maintenance activities
Coding activities
CD/CI
A/B testing
App reviews
Crowdsourcing
Microservices
Team coordination
Code Review Survey
Structured e-mails
Security
Performance Testing
…
…… Software documentation
Automated testing Testing

136. Release Cycle
Testing
CD/CI
A/B testing
App reviews
Crowdsourcing
Microservices
Team coordination
Code Review Survey
Structured e-mails
Security
Performance Testing
…
…… Software documentation
Automated testing
Maintenance activities
Coding activities

137. Test Cases Summarization

138. @Test
public void test0() throws Throwable {
Option option0 = new Option("aaabbb", true, "aaabbb");
Option option1 = new Option("aaabbb", true, "aaabbb");
boolean boolean0 = option1.equals((Object) option0);
assertEquals("arg", option1.getArgName());
assertTrue(option0.hasArg());
assertTrue(boolean0);
}
@Test
public void test1() throws Throwable {
Option option0 = new Option("aaabbb", true, "aaabbb");
Option option1 = new Option("aaabbb", true, "aaabbb");
option0.setLongOpt("adafv");
option1.setLongOpt("adafv");
boolean boolean0 = option1.equals((Object) option0);
assertEquals("arg", option1.getArgName());
assertTrue(option0.hasArg());
assertTrue(boolean0);
}
Test Code Comprehension
Generated Tests
Production Code
public class Options implements Serializable
{
private static final long serialVersionUID = 1L;
/** a map of the options with the character key */
private Map shortOpts = new HashMap();
/** a map of the options with the long key */
private Map longOpts = new HashMap();
/** a map of the required options */
private List requiredOpts = new ArrayList();
Earl T. Barr, et al.
“The Oracle Problem in Software Testing: A
Survey”. IEEE Tran. on Software Engineering,
2015.
138

139. Are Generated Tests Easy to Understand?
G. Fraser et al., Does Automated Unit
Test Generation Really Help Software
Testers? A Controlled Empirical Study,
TOSEM 2015.
Do not lead to detection of
many critical faults.
0%
TestingComprehension
Testing time
75% 100%
Automatically Generated
Test cases allow to achieve
high structural coverage.
139

140. @Test
public void test0() throws Throwable {
Option option0 = new Option("aaa", true, "aaa");
Option option1 = new Option("aaa", true, "aaa");
boolean boolean0 = option1.equals((Object) option0);
assertEquals("arg", option1.getArgName());
assertTrue(option0.hasArg());
assertTrue(boolean0);
}
Tool
Why?
Target Class
Class Name: Option.java
Library: Apache Commons-Cli
140

141. @Test
public void test0() throws Throwable {
Option option0 = new Option("aaa", true, "aaa");
Option option1 = new Option("aaa", true, "aaa");
boolean boolean0 = option1.equals((Object) option0);
assertEquals("arg", option1.getArgName());
assertTrue(option0.hasArg());
assertTrue(boolean0);
}
Tool
Why?
Random string
S.Afshan et al. "Evolving Readable String Test Inputs Using
a Natural Language Model to Reduce Human Oracle Cost".
ICST 2013
Target Class
Class Name: Option.java
Library: Apache Commons-Cli
141

142. @Test
public void test0() throws Throwable {
Option option0 = new Option("aaa", true, "aaa");
Option option1 = new Option("aaa", true, "aaa");
boolean boolean0 = option1.equals((Object) option0);
assertEquals("arg", option1.getArgName());
assertTrue(option0.hasArg());
assertTrue(boolean0);
}
Tool
Why?
Target Class
Class Name: Option.java
Library: Apache Commons-Cli
Meaningless method name
Random string
142

143. @Test
public void test0() throws Throwable {
Option option0 = new Option("aaa", true, "aaa");
Option option1 = new Option("aaa", true, "aaa");
boolean boolean0 = option1.equals((Object) option0);
assertEquals("arg", option1.getArgName());
assertTrue(option0.hasArg());
assertTrue(boolean0);
}
Tool
Why?
Target Class
Class Name: Option.java
Library: Apache Commons-Cli
Meaningless Identifier
Meaningless method name
Random string
143

144. @Test
public void test0() throws Throwable {
Option option0 = new Option("aaa", true, "aaa");
Option option1 = new Option("aaa", true, "aaa");
boolean boolean0 = option1.equals((Object) option0);
assertEquals("arg", option1.getArgName());
assertTrue(option0.hasArg());
assertTrue(boolean0);
}
Tool
Why?
Target Class
Class Name: Option.java
Library: Apache Commons-Cli
Meaningless Identifier
Meaningless method name
Random string
@Test
public void test1() throws Throwable {
Option option0 = new Option("aaa", true, "aaa");
Option option1 = new Option("aaa", true, "aaa");
option0.setLongOpt("adafv");
option1.setLongOpt("adafv");
boolean boolean0 = option1.equals((Object) option0);
assertEquals("arg", option1.getArgName());
assertTrue(option0.hasArg());
assertTrue(boolean0);
}
What are the main differences?
144

145. Combining Test Case Generation with NLP
“The impact of test case Summaries on Bug Fixing Performance: And Empirical Investigation”. ICSE 2016
Generated Unit Test
… with Descriptions
145

146. Covered Code
Summary Generator
Main Steps in TestDescriber: public class Option {
public Option(String opt, String longOpt,
boolean hasArg, String descr)
throws IllegalArgumentException {
OptionValidator.validateOption(opt);
this.opt = opt;
this.longOpt = longOpt;
if (hasArg) {
this.numberOfArgs = 1;
}
this.description = descr;
}
...
}
146

147. Summary Generator
public class Option {
public Option(String opt, String longOpt,
boolean hasArg, String descr)
throws IllegalArgumentException {
OptionValidator.validateOption(opt);
this.opt = opt;
this.longOpt = longOpt;
if (hasArg) {
this.numberOfArgs = 1;
}
this.description = descr;
}
...
}
Main Steps in TestDescriber:
1) Select the covered statements
Covered Code
147

148. public class Option {
public Option(String opt, String longOpt,
boolean hasArg, String descr)
throws IllegalArgumentException {
OptionValidator.validateOption(opt);
this opt = opt;
this longOpt = longOpt;
if (hasArg) {false
}
this description = descr;
}
...
}
Summary Generator
Main Steps in TestDescriber:
1) Select the covered statements
2) Filter out Java keywords, etc.
Covered Code
148

149. public class Option {
public Option(String opt, String long Opt,
boolean has Arg, String descr)
throws IllegalArgumentException {
Option Validator.validate Option(opt);
this opt = opt;
this long Opt = long Opt;
if (has Arg) {false
;
}
this description = descr;
}
...
}
Summary Generator
Covered Code
Main Steps in TestDescriber:
1) Select the covered statements
2) Filter out Java keywords, etc.
3) Identifier Splitting (Camel case)
149

150. public class Option {
public Option(String option, String long Option,
boolean has Argument String description)
throws IllegalArgumentException {
Option Validator.validate Option(option);
this option = option;
this long Option = long Option;
if (has Argument) {false
}
this description = description;
}
...
}
Summary Generator
Covered Code
Main Steps in TestDescriber:
1) Select the covered statements
2) Filter out Java keywords, etc.
3) Identifier Splitting (Camel case)
4) Abbreviation Expansion (using
external vocabularies)
150

151. Main Steps in TestDescriber:
1) Select the covered statements
2) Filter out Java keywords, etc.
3) Identifier Splitting (Camel case)
4) Abbreviation Expansion (using
external vocabularies)
5) Part-of-Speech tagger
Summary Generator
<actions> = Verbs
<themes> = Nouns/Subjects
<secondary arguments> = Nouns / objectes, adjectives, etc
public class Option {
Option(String option, String long Option
,
boolean has Argument String description)
throws IllegalArgumentException
Option Validator.validate Option(option);
this option = option
;
this long Option = long Option;
if (has Argument false
}
this description = description;
}
NOUN NOUN NOUNADJ
NOUNNOUNVERB
NOUN NOUN NOUN
NOUN
VERB NOUN
NOUNADJ
ADJ ADJ ADJ
NOUN
NOUN NOUN
VERB
ADJ
NOUN
CON
NOUN
ADJ
Covered Code
151

152. Summary Generator
NOUN NOUN NOUNADJ
NOUNNOUNVERB
NOUN NOUN NOUN
NOUN
VERB NOUN
NOUNADJ
ADJ ADJ ADJ
NOUN
NOUN NOUN
VERB
ADJ
NOUN
CON
NOUN
The test case instantiates an
"Option" with:
- option equal to “...”
- long option equal to “...”
- it has no argument
- description equal to “…”
An option validator validates it
The test exercises the following
condition:
- "Option" has no argument
public class Option {
Option(String option, String long Option
,
boolean has Argument String description)
throws IllegalArgumentException
Option Validator.validate Option(option);
this option = option
;
this long Option = long Option;
if (has Argument false
}
this description = description;
}
NOUN NOUN NOUNADJ
NOUNNOUNVERB
NOUN NOUN NOUN
NOUN
VERB NOUN
NOUNADJ
ADJ ADJ ADJ
NOUN
NOUN NOUN
VERB
ADJ
NOUN
CON
NOUN
ADJ
Natural Language Sentences Parsed Code
152

153. RQ: How do test case summaries impact the
number of bugs fixed by developers?
153

154. 30 Developers
RQ: How do test case summaries impact the
number of bugs fixed by developers?
154

155. Both the two groups had 45 minutes
to fix each class
30 Developers
RQ: How do test case summaries impact the
number of bugs fixed by developers?
155

156. Participants WITHOUT TestDescriber
summaries fixed 40% of bugs
Both the two groups had 45 minutes
to fix each class
30 Developers
RQ: How do test case summaries impact the
number of bugs fixed by developers?
156

157. 30 Developers
Participants WITHOUT TestDescriber
summaries fixed 40% of bugs
Both the two groups had 45 minutes
to fix each class
Participants WITH TestDescriber summaries,
fixed 60%-80% of bugs
RQ: How do test case summaries impact the
number of bugs fixed by developers?
157

158. Test Cases Summaries and Comprehension
WITHOUT Summaries:
(i) Only 15% of participants
consider the test cases as
“easy to understand”.
(iii) 40% of participants
considered the test cases as
incomprehensible.
WITH Summaries:
(i) 46% of participants consider
the test cases as “easy to
understand”.
(iii) Only 18% of participants
considered the test cases as
incomprehensible.
Without
With 4%
6%
14%
33%
14%
6%
32%
9%
36%
45%
Medium High Very High Low Very Low
Perceived test comprehensibility WITH and
WITHOUT TestDescriber summaries
158

159. Empirical Studies to
understand
developers’ behaviour
Tools for Automating
Software Maintenance
and Testing
How to Identify
such Activities?
How to Reduce the
Cost Activities?
Summary
L1: (Junior) Developers
Need proper Tools
for Program
Comprehension tasks
159

160. Empirical Studies to
understand
developers’ behaviour
Tools for Automating
Software Maintenance
and Testing
How to Identify
such Activities?
How to Reduce the
Cost Activities?
L1: (Junior) Developers
Need proper Tools
for Program
Comprehension tasks
L2: Comprehensibility of
Tests is Important Role
on Bug fixing Tasks.
160
Summary

161. Empirical Studies to
understand
developers’ behaviour
Tools for Automating
Software Maintenance
and Testing
How to Identify
such Activities?
How to Reduce the
Cost Activities?
L1: (Junior) Developers
Need proper Tools
for Program
Comprehension tasks
161
Summary
L2: Comprehensibility of
Tests is Important Role
on Bug Fixing Tasks.
L3: Need of better Tools
for Software Testing

162. Empirical Studies to
understand
developers’ behaviour
Tools for Automating
Software Maintenance
and Testing
How to Identify
such Activities?
How to Reduce the
Cost Activities?
L3: Need of better Tools
for Software Testing
162
Summary
L4: Need to Integrate
User Feedback in the
Maintenance Process
L1: (Junior) Developers
Need proper Tools
for Program
Comprehension tasks
L2: Comprehensibility of
Tests is Important Role
on Bug fixing Tasks.

163. TestingAutomated testing
CD/CI
A/B testing
Crowdsourcing
Microservices
Team coordination
Survey
Structured e-mails
Security
Performance Testing
… …
…
…
…
…
Software documentation
Code Review
Coding Activities
Maintenance Activities
App reviews
Summarization Techniques

164. Open Challenges
Challenges
C1: Automating the
Process of Continuous
Integration (Testing)
C2: Integrating User
Feedback in the
Software Testing Process
C3: Conceive
Visualisation Tools for
Software Testing
C4: Conceive a Tools for
Collaborative Software
Maintenance and Testing
C5: Deliver Tools and
Prototypes for Support
Developers During Code
Inspection Tasks
164

165. Other Technical and
Research Challenges
Challenges
C1: Summarization of
Heterogeneous Data
C3: Scalability and
Integration in the other
Development Contexts
165
C2: Visualization of
Software Summaries

166. TestingAutomated testing
CD/CI
A/B testing
Crowdsourcing
Microservices
Team coordination
Survey
Structured e-mails
Security
Performance Testing
… … …
…
…
Code Review
Coding Activities
Maintenance Activities
App reviews
Potential Application Contexts
…Software documentation

167. TestingAutomated testing
CD/CI
A/B testing
Crowdsourcing
Microservices
Team coordination
Survey
Structured e-mails
Security
Performance Testing
… … …
…
…
Software documentation
Code Review
Coding Activities
Maintenance Activities
App reviews
Potential Application Contexts
…

168. G. Grano, A. Ciurumelea, S. Panichella, F. Palomba, H. Gall
SANER - 2018
Giovanni Grano

169. Summarization Techniques for
Code, Change, Testing and User
Feedback
Sebastiano Panichella
Research Associate
University of Zurich
Thanks for
the Attention!
169