Presentation for BARCO and the EFFECTS Project
---Abstract---
With the rise of agile development, software teams all over the world embrace faster release cycles as *the* way to incorporate customer feedback into product development processes. Yet, faster release cycles imply rethinking the traditional notion of software quality: agile teams must balance reliability (minimize known defects) against agility (maximize ease of change). This talk will explore the state-of-the-art in software test automation and the opportunities this may present for maintaining this balance. We will address questions like: Will our test suite detect critical defects early? If not, how can we improve our test suite? Where should we fix a defect?
4. Ericsson, Bombardier,
Saab, System Veri
fi
cation,
Empear, Verifyter, KTH,
MDH, RISE Comiq, E
fi
Code,
Ponsse, Siili,
Qentinel, Symbio,
Uni.Oulu, VTT
Axini, Testwerk,
TNO, Open Uni. AKKA, Expleo,
EKS, FFT,
Fraunhofer,
IFAK, OFFIS,
Parasoft
Alerion,
Prodevelop,,
Uni.Mandragon Kuveyt Bank,
Saha BT
The TESTOMAT project will allow software teams to increase the
development speed without sacri
fi
cing quality.
To achieve this goal, the project will advance the state-of-the-art in test
automation for software teams moving towards a more agile development
process.
5.
6. Six decades into the computer revolution, four
decades since the invention of the microprocessor,
and two decades into the rise of the modern Internet,
all of the technology required to transform industries
through software
fi
nally works and can be widely
delivered at global scale.
19. Unit tests only !
Segmentation
Percentage
0
20
40
60
80
100
Mutation Coverage
Branch Coverage
20. CI
D
evelop
Bu
ild
Test
W
ay
too
slow
We witnessed 48 hours of mutation testing time on a
test suite comprising 272 unit tests and 5,258 lines of
test code for testing a project with 48,873 lines of
production code.
Sten Vercammen, Serge Demeyer, Markus Borg, and Sigrid Eldh; “Speeding up Mutation Testing via the
Cloud: Lessons Learned for Further Optimisations”. Proceedings ESEM 2018
21. Master
1) Initial test Build
2) ∀ files to mutate:
queue file names
3a) Generate mutants
4a) Execute mutants
3b) Store mutants
4b) Store results
3c) Queue mutant
references
5) Process results
44. Question Cases Precision Recall
Who should
fi
x this bug? Eclipse, Firefox, gcc
eclipse: 57%
fi
refox: 64%
gcc: 6%
—
How long will it take to
fi
x
this bug? (*)
JBoss
depends on the component
many similar reports: off by one hour
few similar reports: off by 7 hours
What is the severity of this
bug? (**)
Mozilla, Eclipse, Gnome
mozilla, eclipse:67% -
73%
gnome:
75%-82%
mozilla, eclipse:50% -
75%
gnome:
68%-84%
(*) In CSMR2012 Proceedings
Who should
fi
x this bug? Eclipse, Firefox, gcc
eclipse: 57%
fi
refox: 64%
gcc: 6%
—
Irrelevant for
Practitioners
(**) In CSMR2011; MSR 2010 Proceedings
Arti
fi
cial
Intelligence
Inside
Internal vs.
External
Bug
Reports