Andrea Di Sorbo, Sebastiano Panichella, Carol Alexandru, Corrado A. Visaggio, Gerardo Canfora, Harald Gall: SURF: Summarizer of User Reviews Feedback. Proceedings of the 39th IEEE International Conference on Software Engineering (ICSE 2017). Buenos Aires, Argentina. RANK: A*

1. SURF: Summarizer of
User Reviews Feedback!
Andrea Sebastiano Carol V. Corrado A. Gerardo !
Di Sorbo Panichella Alexandru Visaggio Canfora !
UNIVERSITÀDEGLI
STUDIDELSANNIO !

2. SURF: Summarizer of
User Reviews Feedback!
Andrea Sebastiano Carol V. Corrado A. Gerardo !
Di Sorbo Panichella Alexandru Visaggio Canfora !
UNIVERSITÀDEGLI
STUDIDELSANNIO !

3. OUTLINE
Context:
Manual v.s. Automated
Analysis of User Reviews
Proposed Solution:
SURF
(Summarizer of User
Reviews Feedback)
Case Study:
Assessment of the Summaries
Involving 12 developers
Conclusion

4. Manual v.s. Automated
Analysis of User Reviews
V.S.!

5. Maintenance of Mobile Applications
“About one third of app reviews!
contain useful information for developers”!
Pagano et. al. RE2013!

6. Manual Analysis of Reviews

7. PAST WORK
Chen et al – ICSE 2014! Panichella et al – ICSME 2015!

8. The Problem
Feature Requests Bug Reports

9. Summaries of User Reviews
SURF (Summarizer of User Reviews Feedback)

10. USER REVIEWS MODEL
Di Sorbo et. al. – FSE 2016!

11. USER REVIEWS MODEL
I love this app but it
crashes my whole iPad and it
has to restart itself
• User intention: Problem Discovery!
!
• Review topics: App, Model!
“…The User Reviews Model proposed by the authors is
impressive in how it analyzes a review sentence by
sentence and is able to characterize a sentence with
multiple labels…” – one of FSE reviewers!

12. USER REVIEWS MODEL
I love this app but it
crashes my whole iPad and it
has to restart itself
• User intention: Problem Discovery!
!
• Review topics: App, Model!
“…The User Reviews Model proposed by the authors is
impressive in how it analyzes a review sentence by
sentence and is able to characterize a sentence with
multiple labels…” – one of FSE reviewers!

13. SUMMARIZER OF USER REVIEW FEEDBACK

14. 1. Data Collection1!

15. 2. Intention Classification2!
machine!
learning!

16. 3. Topics Classification3!
Can't change position of icons on main
screen and can't close bookmarks icon too.!
screen, trajectory, button, white, background, interface,
usability, tap, switch, icon, orientation, position, picture,
show, list, category, cover, scroll, touch, website, swipe,
sensitive, view, roll, side, sort, click, small, colorful, glitch,
page, corner, bookmark…!
GUI-related dictionary!
P (SENTENCE, GUI) = 5/14 = 0.357 !

17. 3. Topics Classification3!
Can't change position of icons on main
screen and can't close bookmarks icon too.!
screen, trajectory, button, white, background, interface,
usability, tap, switch, icon, orientation, position, picture,
show, list, category, cover, scroll, touch, website, swipe,
sensitive, view, roll, side, sort, click, small, colorful, glitch,
page, corner, bookmark…!
GUI-related dictionary!
P (SENTENCE, GUI) = 5/14 = 0.357 !

18. 4. Sentence Scoring4!
The scoring function is able to simultaneously reward: !
!
(i) sentences containing feature requests or bug reports
with respect to other kinds of feedback; !
(ii) sentences that are likely to relate specific topics;!
(iii) longer sentences;!
(iv) sentences concerning frequently discussed features.!
!
Only sentences in the top positions of the ranked list are
selected.!

19. 4. Sentence Scoring4!
The scoring function is able to simultaneously reward: !
!
(i) sentences containing feature requests or bug reports
with respect to other kinds of feedback; !
(ii) sentences that are likely to relate specific topics;!
(iii) longer sentences;!
(iv) sentences concerning frequently discussed features.!
!
Only sentences in the top positions of the ranked list are
selected.!

20. 5. Summary Generation5!
DEMO

21. Case Study
21
Involving 12 Developers

22. Case Study
22
2622
Reviews
Involving 12 Developers

23. Case Study
Involving 12 Developers
2622
Reviews
Of
12
Apps
23

24. Study Procedure
24

25. Experiment Involving 3 Countries
ITALY SWITZERLAND
NETHERLAND

26. 26
ITALY SWITZERLAND
NETHERLAND
Experiment Involving 3 Countries

27. ITALY
SWITZERLAND
NETHERLAND
27
Setting of the Experiment

28. Setting of the Experiment
ITALY
SWITZERLAND
NETHERLAND
1) Summaries
for 12
Apps
28

29. ITALY
SWITZERLAND
NETHERLAND
1) Summaries
for 12
Apps
2) Involving 12 Developers
29
Setting of the Experiment

30. ITALY
SWITZERLAND
NETHERLAND
1) Summaries
for 12
Apps
3) We assigned to each participant an app.
30
Setting of the Experiment
2) Involving 12 Developers

31. To what extent SURF help mobile developers
better understand the users' needs?
RQ31

32. 32
To what extent SURF help mobile developers
better understand the users' needs?

33. 75% of participants considered
the summaries HIGH USEFUL and
COMPREHENSIBLE
33
To what extent SURF help mobile developers
better understand the users' needs?

34. 75% of participants considered
the summaries HIGH USEFUL and
COMPREHENSIBLE
34
To what extent SURF help mobile developers
better understand the users' needs?
92% of participants declared that
without summaries, evaluating
user feedback is tedious and
difficult.

35. How do app review summaries generated
by SURF impact the time required by developers to
analyze user reviews?
35

36. The time saving capability of
SURF perceived by all developers
Is of at least 35%.
66% of participants believe that
the time saving capability of
SURF is of 75%.
36
How do app review summaries generated
by SURF impact the time required by developers to
analyze user reviews?

37. The time saving capability of
SURF perceived by all developers
Is of at least 50%.
94% of participants believe that
the time saving capability of
SURF is of 75%.
37
92% of manually extracted feedback
appears also in the automatic
generated summaries.
How do app review summaries generated
by SURF impact the time required by developers to
analyze user reviews?

38. How do app review summaries generated!
by SURF impact the time required by developers to!
analyze user reviews?!
The time saving capability of
SURF perceived by all developers
Is of at least 50%.
94% of participants believe that
the time saving capability of
SURF is of 75%.
SURF helps to prevent more than 50% of the
time required by developers for analyzing users
feedback and planning software changes.
66% of feedback manually extracted by the
participants also appear in the summaries
automatically generated by SURF.
38

39. How do app review summaries generated!
by SURF impact the time required by developers to!
analyze user reviews?!
The time saving capability of
SURF perceived by all developers
Is of at least 50%.
94% of participants believe that
the time saving capability of
SURF is of 75%.
SURF helps to prevent more than 50% of the
time required by developers for analyzing users
feedback and planning software changes.
66% of feedback manually extracted by the
participants also appear in the summaries
automatically generated by SURF.
39

40. Quality of SURF’ Summaries
40

41. Quality of SURF’ Summaries
41

42. Quality of SURF’ Summaries
42

43. Conclusion
2) SURF helps to prevent more than half
of the time required for analyzing users
feedback and planning software changes.
3) 92% of manually extracted
feedback appears also in the
automatic generated summaries.
V.S.!
4) Summaries generated by SURF
are reasonably correct, adequate,
concise, and expressive.
43

44. Thanks for the Attention!!
Questions?!
SURF (Summarizer of User Review Feedback)