This presentation is about a lecture I gave within the "Software systems and services" immigration course at the Gran Sasso Science Institute, L'Aquila (Italy): http://cs.gssi.it/. http://www.ivanomalavolta.com

1. Ivano Malavolta
RESEARCH
in software engineering
VRIJE
UNIVERSITEIT
AMSTERDAM

2. Hello
Empirical software engineering
+ Software Architecture + MDE
applied to
Mobile-enabled systems
Autonomous robots
Microservices
If you think good architecture is expensive,
try bad architecture.
... Brian Foote and Joseph Yoder

3. Course overview (from my perspective)
Module name #hours Instructor
Introduction to the course 2 Inverardi
Research in software engineering 2 Malavolta
Software architecture 6 Malavolta
Common development paradigms 2 Malavolta
SE principles, Eclipse IDE, MDE 8 Iovino
Code generation, applications of MDE, LAB, etc. 10 Iovino
Principles of software testing 4 Bertolino
Homework 1
Christmas break
Homework 2

4. Homework 1
Tasks
• form a team of 2 members
• create an AADL specification describing the architecture of a chosen system
• write an Acceleo program that takes as input the AADL model and produces
an HTML5 report from it
• write a report describing the performed activities
Goals
• to understand what are the advantages of SE principles like abstraction and
separation of concerns
• to concretely understand what architectural modeling means
• to be exposed to concerns and issues related to dependability
• to understand how to use MDE techniques like model transformations and
code generators
Tentative deadline
22/12/2017

5. Homework 2
Tasks
• pick a set of articles related to a chosen research theme
• carefully read them and explore the state of the art about the research
theme
• write a report about your findings
• make a presentation to the classroom
Goals
• to study a specific area of software engineering that may be of interest to
you
• to design a scientific paper
• to train yourself in reading and writing scientific papers
Tentative deadline
20/01/2018
Research themes are
already available!

6. Fraud
Cases of plagiarism or inter-team collaboration and deliverable
contents exchange will be reported to and managed by the official
fraud committee.
In case of fraud, the consequences of those acts may potentially lead
to: formal warning, inclusion of the formal warning in your GSSI student
file, suspension from giving exams for a given period, expulsion from
the GSSI.
The rules above apply for both homework 1 and homework 2.

7. Roadmap
Software engineering research
Empirical strategies
Writing good research papers

8. Software engineering research

9. The application of engineering to software
We deal with software systems that are:
– large and complex
– built by teams
– exist in many versions
– last many years
– undergo changes
Programming skills are not enough
“Physicist example”
Software engineering
Programmer:
complete program
Software engineer:
software component

10. Software engineering VS
computer science
Computer Science
– Computability, algorithms and complexity, programming
languages, data structures, databases, artificial intelligence,
etc.
Software Engineering
– The APPLICATION of computer science, mathematics,
project management to build high quality software

11. My vision about research in SE (CS)
Research
Theory Industrial projectsProgramming Experimentation

12. Research objective
Key objectives
• Quality àutility + functional correctness
• Cost à development + use
• Timeliness à good-enough result, when it’s needed
Address problems that affect practical software
Real world
practical PROBLEM
Real world
practical SOLUTION
?

13. Research objective: example

14. Example

15. Research product: example

16. Research strategy
Real world
practical PROBLEM
Real world
practical SOLUTION
IDEALIZED
PROBLEM
SOLUTION to
IDEALIZED PROBLEM
Research product
(technique, method,
model, system, …)

17. Validation of the results
Real world
practical PROBLEM
Real world
practical SOLUTION
IDEALIZED
PROBLEM
SOLUTION to
IDEALIZED PROBLEM
Research product
(technique, method,
model, system, …)

18. Validation of the results
Real world
practical PROBLEM
Real world
practical SOLUTION
IDEALIZED
PROBLEM
SOLUTION to
IDEALIZED PROBLEM
Research product
(technique, method,
model, system, …)
Validation task 1
Does the product
solve the idealized problem?

19. Validation of the results
Real world
practical PROBLEM
Real world
practical SOLUTION
IDEALIZED
PROBLEM
SOLUTION to
IDEALIZED PROBLEM
Research product
(technique, method,
model, system, …)
Validation task 1
Does the product
solve the idealized problem?
Validation task 2
Does the product
help to solve the practical problem?

20. Validation of the results: example

21. SE research process
Research
questions
Research
validation
Research
results

22. Types of research questions
FEASIBILITY
CHARACTERIZATION
METHOD/MEANS
GENERALIZATION
DISCRIMINATION
Does X exist, and what is it?
Is it possible to do X at all?
What are the characteristics of X?
What exactly do we mean by X?
What are the varieties of X, and how are
they related?
How can we do X?
What is a better way to do X?
How can we automate doing X?
Is X always true of Y?
Given X, what will Y be?
How do I decide whether X or Y?

23. LAB

24. SE research process
Research
questions
Research
results
Research
validation

25. Research results
Real world
practical PROBLEM
Real world
practical SOLUTION
Research setting
IDEALIZED PROBLEM
Research product
(technique, method,
model, system, …)

26. Types of research results
QUALITATIVE &
DESCRIPTIVE
MODELS
TECHNIQUES
SYSTEM
EMPIRICAL
MODELS
ANALYTIC
MODELS
Report interesting observations
Generalize from (real-life) examples
Structure a problem area; ask good questions
Invent new ways to do some tasks, including
implementation techniques
Develop ways to select from alternatives
Embody result in a system, using the system
both for insight and as carrier of results
Develop empirical predictive models from
observed data
Develop structural models that permit formal
analysis

27. LAB

28. SE research process
Research
questions
Research
results
Research
validation

29. Research validation
Real world
practical PROBLEM
Real world
practical SOLUTION
Research setting
IDEALIZED PROBLEM
Research setting
SOLUTION to
IDEALIZED PROBLEM
Research product
(technique, method,
model, system, …)
Validation task 1
Does the product
solve the idealized problem?
Validation task 2
Does the result
help to solve the practical problem?

30. Types of research validation
PERSUASION I thought hard about this, and I believe…
IMPLEMENTATION Here is a prototype of a system that…
EVALUATION Given these criteria, the object rates as…
ANALYSIS Given the facts, here are consequences…
Formal model Rigorous derivation and proof
Empirical model Data on use in controlled situation
Experiment designed experiment with statistics
EXPERIENCE Report on use in practice
Qualitative model Narrative
Decision criteria Comparison of systems in actual use
Empirical model Data, usually statistical, on practice

31. LAB

32. Building blocks for research
Feasibility
Characterization
Method/means
Generalization
Selection
Qualitative model
Technique
System
Empirical model
Analytic model
Persuasion
Implementation
Evaluation
Analysis
Experience
Question Result Validation

33. Is this a good plan?
Feasibility
Characterization
Can X be
done better?
Generalization
Selection
Qualitative model
Technique
Build Y
Empirical model
Analytic model
Persuasion
Implementation
Measure Y,
compare to X
Analysis
Experience
Question Result Validation

34. A common good plan
Feasibility
Characterization
Can X be
done better?
Generalization
Selection
Qualitative model
Technique
Build Y
Empirical model
Analytic model
Persuasion
Implementation
Measure Y,
compare to X
Analysis
Experience
Question Result Validation

35. Is this a good plan?
Feasibility
Characterization
Can X be
done better?
Generalization
Selection
Qualitative model
Here is a new
way to do X
System
Empirical model
Analytic model
I tried it, and it
works
Implementation
Evaluation
Analysis
Experience
Question Result Validation

36. A common, but bad, plan
Feasibility
Characterization
Can X be
done better?
Generalization
Selection
Qualitative model
Here is a new
way to do X
System
Empirical model
Analytic model
I tried it, and it
works
Implementation
Evaluation
Analysis
Experience
Question Result Validation

37. Two other good plans
Can X be done
at all?
Characterization
Is X always
true of Y?
Selection
Qualitative model
Technique
Build a Y
that does X
Empirical model
Formally model
Y, prove X
“Look it works!”
Implementation
Check proof
Experience
Question Result Validation
Method/means Evaluation

38. LAB (at home – for Wednesday)
Choose a research paper and try to map it into the building
blocks of SE research
Feasibility
Characterization
Method/means
Generalization
Selection
Qualitative model
Technique
System
Empirical model
Analytic model
Persuasion
Implementation
Evaluation
Analysis
Experience
Question Result Validation

39. A couple of words on the abstract of
a paper
• People judge papers by their abstracts
– usually decide whether to read the whole paper based on
abstract
• It's important for the abstract to tell the whole story
– problem
– contributions
– results
– validation
• Often it is the only part freely accessible

40. Coming back to the initial example…
Problem
Overall
contribution
Specific
results
Validation
✓✗ ✓✓✗ ✗

41. Second try…
Problem
Overall
contribution
Specific
results
Validation

42. “NO-NO”s for software engineering
research
• Assume that a result demonstrated fro a 10K-line system
will scale to a 500K-line system
• Expect everyone to do things “my way”
• Believe functional correctness is sufficient
• Assume the existence of a complete, consistent
specification
• Just build things without extracting enduring lessons
• Devise a solution in ignorance of how the world really
works

43. How do you trust a research then?
1. What are the problems from the real world?
– Are they “real” and widespread?
2. Are the solutions general? What are their limits?
Real world
practical PROBLEM
Real world
practical SOLUTION
?
EMPIRICAL SOFTWARE ENGINEERING

44. Empirical strategies*
*We will have an advanced course on this topic

45. Empirical software engineering
Scientific use of quantitative and qualitative data to
– understand and
– improve
software products and software development processes
Data is central to address any research question
Issues related to validity addressed continuously
[Victor Basili]

46. Intuition
• You apply the scientific method
– ask yourself a question
– observe a phenomenon
– formulate an hypothesis
– build a prediction
– setup an experiment
– perform analysis on your results

47. Why empirical studies?
Anecdotal evidence or “common-sense” often not good enough
- Anecdotes often insufficient to support decisions in the industry
- Practitioners need better advice on how and when to use methodologies
Evidence important for successful technology transfer
- Systematic gathering of evidence
- Wide dissemination of evidence

48. Example
End Users’ Perception of Hybrid Mobile Apps
in the Google Play Store
Ivano Malavolta, Stefano Ruberto
Tommaso Soru, Valerio Terragni
Ivano Malavolta
Gran Sasso Science Institute (Italy)
ivano.malavolta@gssi.infn.it
ABSTRACT
Recently, companies like IBM and Adobe and a growing
community of developers advocate hybrid mobile apps
development as a possible solution to mobile platforms
fragmentation. Hybrid mobile apps are consistent across
platforms and built on web standards.
In this study, we present an empirical investigation into mobile
hybrid apps. Our goal is to identify and analyse the traits and
distinctions of publicly available hybrid mobile apps from end
users’ perspective. The study has been conducted by mining
11,917 free apps and 3,041,315 reviews from the Google Play
Store, and analyzing them from the end users’ perception
perspective. The results of this study build an objective and
reproducible snapshot about how hybrid mobile development is
performing “in the wild” in real projects, thus establishing a base
for future methods and techniques for developing hybrid mobile
apps.
FINDINGS
• hybrid development frameworks are perceived as better
suited for data-intensive mobile apps, whereas they
perform poorly when dealing with low-level, platform-
specific features
• end users value hybrid and native apps similarly
• in some categories, end users perceive native apps better
than hybrid apps with respect to performance and the
presence of bugs
RESEARCH QUESTIONS
What is the difference between hybrid and native mobile apps
as perceived by end users?
– RQ1: What is the difference in the perceived value between
hybrid and native mobile apps?
– RQ2: What is the difference in the perceived performance
between hybrid and native mobile apps?
– RQ3: What is the difference in the perceived bugginess
between hybrid and native mobile apps?
– RQ4: What is the difference in the initial download overhead
between hybrid and native mobile apps?

49. Dimensions of empirical studies
“In the lab” versus “in the wild” studies
Qualitative versus quantitative studies
Primary versus secondary studies

50. “In the lab” versus “in the wild”
studies
Common “in the lab” methods
– Controlled experiments
– Literature reviews
– Simulations
Common “in the wild” methods
– Quasi-experiments
– Case studies
– Survey research
– Ethnographies
– Action research

51. Examples

52. Qualitative versus quantitative
studies
Qualitative research
studying objects in their natural setting and letting the
findings emerge from the observations
– inductive process
– the subject is the person
Quantitative research
quantifying a relationship with the aim to identify a cause-
effect relationship
– fixed implied factors
– focus on collected quantitative data à promotes comparison and
statistical analyses
They are
complementary

53. Example
They are
complementary

54. Primary versus secondary studies
Primary studies
empirical studies in which we directly make measurements
or observations about the objects of interest, whether by
surveys, experiments, case studies, etc.
Secondary studies
empirical studies that do not generate any data from direct
measurements, but:
– analyze a set of primary studies
– usually seek to aggregate the results from these in order to
provide stronger forms of evidence about a phenomenon

55. Examples

56. …and what about this?

57. Types of empirical studies
• Survey
• Case study
• Experiment

58. Survey
Def: a system for collecting information from or about
people to describe, compare or explain their knowledge,
attitudes and behavior
Often an investigation performed in retrospect
Interviews and questionnaires are the primary means of
gathering qualitative or quantitative data
These are done through taking a sample which is
representative from the population to be studied

59. Example

60. Case study
Def: an empirical enquiry to investigate one instance (or a
small number of instances) of a contemporary software
engineering phenomenon within its real-life context,
especially when the boundary between phenomenon and
context cannot be clearly specified
Observational study
Data collected to track a specific attribute or establishing
relationships between different attributes
Multivariate statistical analysis is often applied

61. Example

62. Experiment
Def: an empirical enquiry that manipulates one factor or
variable of the studied setting
1. Identify and understand the variables that play a role in software
development, and the connections between variables
2. Learn cause-effect relationships between the development
process and the obtained products
3. Establish laws and best practices about software construction

63. Experiment
process

64. Example
http://dl.acm.org/citation.cfm?id=2491411.2491428

65. Examples of empirical studies in SE
• Nejati, Javad, and Aruna Balasubramanian. "An in-depth study of
mobile browser performance." Proceedings of the 25th International
Conference on World Wide Web. International World Wide Web
Conferences Steering Committee, 2016.
• Hassan, Safwat, Weiyi Shang, and Ahmed E. Hassan. "An empirical
study of emergency updates for top android mobile apps." Empirical
Software Engineering (2016): 1-42.
• Tufano, Michele, et al. "An empirical investigation into the nature of
test smells." Proceedings of the 31st IEEE/ACM International
Conference on Automated Software Engineering. ACM, 2016.
• Linares-Vásquez, Mario, et al. "Mining energy-greedy api usage
patterns in android apps: an empirical study." Proceedings of the 11th
Working Conference on Mining Software Repositories. ACM, 2014.
• Procaccianti, Giuseppe, Héctor Fernández, and Patricia Lago.
"Empirical evaluation of two best practices for energy-efficient
software development." Journal of Systems and Software 117 (2016):
185-198.

66. How to have an impact in reality?
This is called technology transfer

67. What this lecture means to you?
Software engineering != computer science
You now know how to carry on research in SE
When conducting your research, don’t make claims you
cannot eventually measure
Start forming teams for homework 1!

68. Suggested readings about scientific
writing
• Shaw, M. (2003), Writing Good Software Engineering Research
Papers, in Lori A. Clarke; Laurie Dillon & Walter F. Tichy, ed., 'ICSE' ,
IEEE Computer Society, , pp. 726-737 .
• Shaw, M. (2002), 'What makes good research in software
engineering?', STTT 4 (1) , 1-7 .

69. References
http://link.springer.com/book/10.1007%2F978-3-642-29044-2

70. Contact
Ivano Malavolta |
Assistant professor
Vrije Universiteit Amsterdam
iivanoo
i.malavolta@vu.nl
www.ivanomalavolta.com

71. Acknowledgement
Some parts of this lecture are inspired by:
• Ivica Crnkovic’s lecture on software engineering research at
Mälardalen University (Sweden)
• Matthias Galster ‘s tutorial titled “Introduction to Empirical Research
Methodologies” at ECSA 2014