CEC'14 presentation.

1. 1 of 15
Comparative Analysis of Classical Multi-
Objective Evolutionary Algorithms and
Seeding Strategies for Pairwise Testing
of Software Product Lines
Roberto E. Lopez-Herrejon*, Javier Ferrer**, Francisco Chicano**,
Alexander Egyed*, Enrique Alba**
* Johannes Kepler University Linz, Austria
** University of Malaga, Spain

2. 2 of 15
Introduction
 Software Product Lines (SPLs)
 Families of software products
• Each product has different feature combinations
 Have multiple economical and technological advantages
• Increased software reuse, faster time to market, better
customization
 Challenge: How to test a Software Product Line effectively?
 Important factors to consider
 Typical SPLs have a large number of different software products
 Avoiding repeating tests
 Within the economical and technical constraints

3. 3 of 15
Pairwise testing of SPLs
 Existing work (Wang13, Henard13)
Use a linearization approach where each optimization
objective is given a weight and later added
𝑖=1..𝑛
𝑤𝑖 × 𝑂𝑏𝑗𝑖
Optimization objectives: coverage and test suite size
 Our proposal
Formalization of SPL pairwise testing problem for
multiple-objective algorithms
Study 4 classical MOEAs for pairwise testing SPLS
Analyze the impact of three seeding strategies
Evaluate using a large and diverse corpus

4. 4 of 15
Combinatorial Interaction
Testing (CIT) for SPLs
 Combinatorial Interaction Testing (CIT)
Select a test suite, which is a group of products where
faults are more likely to occur
 Based on feature models
De facto standard to model all the products (feature
combinations) of a product line
 Pairwise testing – combinations of two features
4 options: selected both, not selected both, one
selected but not the other, and vice versa

5. 5 of 15
Num requires Search SCC requires DFS
CC requires Undirected Cycle requires DFS
CC requires Search Kruskal requires Undirected Kruskal excludes Prim
SCC requires Directed Kruskal requires Weight Shortest requires Directed
Prim requires Undirected Prim requires Weight Shortest requires Weight
Feature Model Example
Graph Product Line (GPL)
GPL
Driver
Benchmark
GraphType
Directed Undirected
Weight Search
DFS BFS
Algorithms
Num CC SCC Cycle
Prim Kruskal
Shortest
Mandatory Optional
Exclusive-or
Inclusive-or
Root
Cross-Tree Constraints (CTC)

6. 6 of 15
Valid Feature Sets, Pairs &
Examples
 A valid feature set is a combination of features that meets all
the constraints from the feature model
 A valid pair is a combination of two features that meets all the
constraints from the feature model

7. 7 of 15
Pairwise Test Suite
 Pairwise test suite is a set of valid feature sets
that covers all possible valid pairs
 GPL Example
73 feature sets
418 pairs
Pairwise test suite for GPL

8. 8 of 15
MOO from the Software
Engineer’s Perspective
Number of Products
CoveragePercentage
Pareto Front for the GPL Example

9. 9 of 15
Our work in a nutshell
 Uses classical MOO algorithms
NSGA-II – crowding distance and ranking
MOCell – cellular GA, based on neighbourhood
SPEA2 – population and archive
PAES – evolution strategy
 Uses standard comparison MOO metrics
Hypervolume
Generational distance
 Analyses the impact of seeding
Three distinct strategies

10. 10 of 15
Analyzing Impact of Seeding
 Seeding
 Embed domain knowledge into the individuals of the population
 We used 3 seeding strategies for the initial population
 Size-based Random Seeding
• Compute a pairwise test suite with CASA and use its size to
generate the population
 Greedy Seeding
• Greedily computes a pairwise test suite and uses its elements
to generate the population
 Single-Objective Based Seeding
• Creates a population based on a single-objective output CASA

11. 11 of 15
Evaluation Overview
 Selection of 19 realistic case studies from different
application domains
 Feature models and implementation publicly available
 Feature model analysis employed standard tools
 FAMA, SPLAR, SPLCA
 Experimental setting
 Quality indicators employed: Hypervolume(HV) and Generational
Distance(GD)
 Total independent runs 6,840
• 4 algorithms × 3 seeding strat. × 19 models × 30 runs = 6,840
 Standard statistical analysis
Wilcoxon Test and Â12

12. 12 of 15
Results
Algorithms HV GD TIME
NSGA-II 0.6583 0.0396 70,523
MOCell 0.6553 0.0293 74,325
SPEA2 0.6533 0.0289 71,349
PAES 0.6390 0.0351 101,246
Algorithms HV GD TIME
NSGA-II – SPEA2 0.5182 0.5172 0.4904
NSGA-II – MOCell 0.5112 0.5202 0.4816
NSGA-II – PAES 0.5626 0.4560 0.2839
SPEA2 – MOCell 0.4932 0.5039 0.4910
SPEA2 – PAES 0.5447 0.4205 0.3019
MOCell - PAES 0.5521 0.4194 0.3027
Seeding HV GD TIME
Sized-Based 0.6421 0.0427 138,404
Greedy 0.6556 0.0447 76,783
Single Obj. 0.6568 0.0123 25,800
Seeding HV GD TIME
Sized Based - Greedy 0.4568 0.4795 0.6377
Sized Based – Single Obj. 0.4558 0.8562 0.8619
Greedy – Single Obj. 0.4977 0.7839 0.8227
Quality Indicators Results Â12 Statistic Test Results

13. 13 of 15
Summary of Results
 RQ1. What is the best algorithm among the four
studies for multi-objective SPL pairwise testing?
No clear winner between NSGA-II, MoCELL, SPEA2
PAES performs slightly worse overall
 RQ2. How does the seeding impact the quality of
solutions obtained by the four algorithms?
Single-objective Based Seeding clearly yields better
results than other two strategies
• The more knowledge used in the initial population
the better

14. 14 of 15
Future Work
 Extending the feature model corpus
Larger and more diverse case studies
 Analysis of the impact of parameter setting
 Integrate other domain knowledge
Control flow
Structural metrics of feature models

15. 15 of 15
Acknowledgements
Spanish Ministry of Economy
and Competitiveness,
FEDER
Austrian Science Fund