This presentation is about a lecture I gave within the "Green Lab" course of the Computer Science master, Software Engineering and Green IT track of the Vrije Universiteit Amsterdam: http://masters.vu.nl/en/programmes/computer-science-software-engineering-green-it/index.aspx http://www.ivanomalavolta.com

1. 1 Het begint met een idee
Empirical software engineering
Ivano Malavolta

2. Vrije Universiteit Amsterdam
Software engineering
Empirical software engineering
- definition
- dimensions
- types of studies
2 Ivano Malavolta / S2 group / Empirical software engineering
Roadmap

3. Vrije Universiteit Amsterdam
3 Ivano Malavolta / S2 group / Empirical software engineering
Software engineering

4. Vrije Universiteit Amsterdam
Where is software today?
4 Ivano Malavolta / S2 group / Empirical software engineering
Software today

5. Vrije Universiteit Amsterdam
How “big” is software today?
5 Ivano Malavolta / S2 group / Empirical software engineering
Software today
http://www.informationisbeautiful.net/visualizations/million-lines-of-code
http://hbr.org/2010/06/why-dinosaurs-will-keep-ruling-the-auto-industry/ar/1

6. Vrije Universiteit Amsterdam
To DESIGN software
- software development has to be a systematic activity
QUALITY assurance
- we have to verify and validate our SW in order to make it something
people can rely on
- we have to do it as soon as possible
ABSTRACTION
- the principal instrument for managing complexity
6 Ivano Malavolta / S2 group / Empirical software engineering
Needs

7. Vrije Universiteit Amsterdam
The application of engineering to software
Field of computer science dealing with software systems that
are:
– large and complex
– built by teams
– exist in many versions
– last many years
– undergo changes
Programming skills are not enough
“Physicist example”
–
7 Ivano Malavolta / S2 group / Empirical software engineering
Software engineering (SE)
Programmer:
complete program
Software engineer:
software component

8. Vrije Universiteit Amsterdam
8 Ivano Malavolta / S2 group / Empirical software engineering
A more precise definition of SE
"Software Engineering means application of a
systematic, disciplined, quantifiable approach
to development, operation and maintenance of
software [1]"
1. “IEEE Standard Glossary of Software Engineering Terminology,” IEEE std 610.12-1990, 1990.

9. Vrije Universiteit Amsterdam
Computer science
- Computability, algorithms and complexity, programming
languages, data structures, databases, artificial intelligence, etc.
Software engineering
- The APPLICATION of computer science, mathematics, project
management to build high quality software
9 Ivano Malavolta / S2 group / Empirical software engineering
Software engineering VS computer science

10. Vrije Universiteit Amsterdam
10 Ivano Malavolta / S2 group / Empirical software engineering
Empirical software engineering

11. Vrije Universiteit Amsterdam
Scientific use of quantitative and qualitative data to
- understand and
- improve
software products and software development processes
[Victor Basili]
Data is central to address any research question
Issues related to validity addressed continuously
11 Ivano Malavolta / S2 group / Empirical software engineering
Empirical software engineering

12. Vrije Universiteit Amsterdam
12 Ivano Malavolta / S2 group / Empirical software engineering
Intuition
● It is an application of the scientific method
○ ask yourself a question
○ observe a phenomenon
○ formulate an hypothesis
○ build a prediction
○ setup an experiment
○ perform analysis on your results

13. Vrije Universiteit Amsterdam
Anecdotal evidence or “common-sense” often not good
enough
- Anecdotes often insufficient to support decisions in the industry
- Practitioners need better advice on how and when to use
methodologies
Evidence important for successful technology transfer
- systematic gathering of evidence
- wide dissemination of evidence
13 Ivano Malavolta / S2 group / Empirical software engineering
Why empirical studies?

14. Vrije Universiteit Amsterdam
14 Ivano Malavolta / S2 group / Empirical software engineering

15. Vrije Universiteit Amsterdam
● Tested on a single system (lack of generalization)
● Data analysis not statistical
● No indications on how many runs were performed, error rate,
etc.
15 Ivano Malavolta / S2 group / Empirical software engineering
How NOT to do experiments

16. Vrije Universiteit Amsterdam
16 Ivano Malavolta / S2 group / Empirical software engineering
How NOT to do experiments

17. Vrije Universiteit Amsterdam
17
How NOT to do experiments
IE9
Chrome
FireFox
Opera
Safari

18. Vrije Universiteit Amsterdam
“In the lab” versus “in the wild” studies
Qualitative versus quantitative studies
Primary versus secondary studies
18 Ivano Malavolta / S2 group / Empirical software engineering
Dimensions of empirical studies

19. Vrije Universiteit Amsterdam
Common “in the lab” methods
– Controlled experiments
– Literature reviews
– Simulations
Common “in the wild” methods
– Quasi-experiments
– Case studies
– Survey research
– Ethnographies
– Action research
19 Ivano Malavolta / S2 group / Empirical software engineering
“In the lab” VS “in the wild” studies

20. Vrije Universiteit Amsterdam
20 Ivano Malavolta / S2 group / Empirical software engineering
Examples

21. Vrije Universiteit Amsterdam
Qualitative research
studying objects in their natural setting and letting the findings
emerge from the observations
– inductive process
– the subject is the person
Quantitative research
quantifying a relationship or to compare two or more groups with the
aim to identify a cause-effect relationship
– fixed implied factors
– focus on quantitative data → promotes comparison and statistical
analyses
21 Ivano Malavolta / S2 group / Empirical software engineering
Qualitative VS quantitative studies

22. Vrije Universiteit Amsterdam
22 Ivano Malavolta / S2 group / Empirical software engineering
Example
They are
complementary

23. Vrije Universiteit Amsterdam
Primary studies
empirical studies in which we directly make measurements or
observations about the objects of interest, whether by surveys,
experiments, case studies, etc.
Secondary studies
empirical studies that do not generate any data from direct
measurements, but:
– analyze a set of primary studies
– usually seek to aggregate the results from these in order to provide
stronger forms of evidence about a phenomenon
23 Ivano Malavolta / S2 group / Empirical software engineering
Primary VS secondary studies

24. Vrije Universiteit Amsterdam
24 Ivano Malavolta / S2 group / Empirical software engineering
Examples

25. Vrije Universiteit Amsterdam
25 Ivano Malavolta / S2 group / Empirical software engineering
… and what about this?

26. Vrije Universiteit Amsterdam
Survey
Case study
Experiment
26 Ivano Malavolta / S2 group / Empirical software engineering
Types of empirical studies

27. Vrije Universiteit Amsterdam
Def: a system for collecting information from or about people
to describe, compare or explain their knowledge, attitudes and
behavior
Often an investigation performed in retrospect
Interviews and questionnaires are the primary means of
gathering qualitative or quantitative data
Surveys are done by taking a sample which is representative
from the population to be studied
27 Ivano Malavolta / S2 group / Empirical software engineering
Survey

28. Vrije Universiteit Amsterdam
28 Ivano Malavolta / S2 group / Empirical software engineering
Example

29. Vrije Universiteit Amsterdam
1. ADLs Identification
– Identify preliminary set of ADLs
– Systematic search
2. Planning the Survey
3. Designing the survey
4. Analyzing the Data
– vertical analysis (and coding) + horizontal analysis
29 Ivano Malavolta / S2 group / Empirical software engineering
Example

30. Vrije Universiteit Amsterdam
Def: an empirical enquiry to investigate one instance (or a
small number of instances) of a contemporary software
engineering phenomenon within its real-life context, especially
when the boundary between phenomenon and context cannot
be clearly specified
Observational study
Data collected to track a specific attribute or establishing
relationships between different attributes
Multivariate statistical analysis is often applied
30 Ivano Malavolta / S2 group / Empirical software engineering
Case study

31. Vrije Universiteit Amsterdam
31 Ivano Malavolta / S2 group / Empirical software engineering
Example

32. Vrije Universiteit Amsterdam
Def: an empirical enquiry that manipulates one factor or
variable of the studied setting
1. Identify and understand the variables that play a role in software
development, and the connections between variables
2. Learn cause-effect relationships between the development process
and the obtained products
3. Establish laws and theories about software construction that
explain development behaviour
32
Experiment

33. Vrije Universiteit Amsterdam
Experiment
Models key characteristics of a reality in a controlled
environment and manipulating them iteratively to investigate
the impact of such variations and get a better understanding of
a phenomenon
Laboratory
Simplified and controllable reality where the phenomenon
under study can be manipulated
CONTROL - CONTROL - CONTROL
33 Ivano Malavolta / S2 group / Empirical software engineering
A more intuitive definition...

34. Vrije Universiteit Amsterdam
34
Example

35. Vrije Universiteit Amsterdam
35 Ivano Malavolta / S2 group / Empirical software engineering
What to choose?

36. Vrije Universiteit Amsterdam
This is called technology transfer
36 Ivano Malavolta / S2 group / Empirical software engineering
How to have an impact in reality?

37. Vrije Universiteit Amsterdam
Software is pervasive
Software engineering != computer science
You know the basics of empirical SE now
Don’t make claims you cannot eventually measure
Don’t think in black and white only
– don’t divide the world in methods, analyses, case study, etc.
– don’t be afraid to look also at other disciplines
→ we are software engineers in any case
37 Ivano Malavolta / S2 group / Empirical software engineering
What this lecture means to you?

38. Vrije Universiteit Amsterdam
38 Ivano Malavolta / S2 group / Empirical software engineering
Readings
Chapters 1, 2, 3Chapters 1 and 2
+ one article among the ones mentioned in the previous slides