Pitfalls and Countermeasures in Software Quality Measurements and Evaluations

Pitfalls and Countermeasures in Software
Quality Measurements and Evaluations
Hironori Washizaki washizaki@waseda.jp
Prof., Global Software Engineering Laboratory,
Waseda University
Visiting Prof., National Institute of Informatics
Director, SYSTEM INFORMATION CO., LTD.
Vice-Chair, IEEE CS Japan Chapter
Chair, SEMAT Japan Chapter
Convenor, ISO/IEC/JTC1/SC7/WG20
PC Co-Chair, APSEC 2018 Nagoya!
5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ)
Nanjing, Dec 4, 2017
H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017

Metric and Measurement
• Mapping attributes to values/names
on scales
– Quality control by single metric
– Estimating metric values by other metrics
• You cannot control what you cannot
measure!
Req.
Def.
Design Impl. Testing
Func. Spec.
Function
point
Module
design
Coupling
Source
code
Complexity
LOC
Defect
report
Defect
density
Test
case
N. tests
passed
Effort, time
Cyclomatic
complexity
0
0.2
0.4
0.6
0.8
1
0 0.05 0.1 0.15 0.2 0.25 0.3
Reuserate
#Header files included
2

Pitfalls and Countermeasures
3
Pitfall Countermeasure
Negative
Hawthorne effects
Goal-orientation
Multidimensional measurements
Organization
misalignment
Visualization of relationships among
organizational goals, strategies, and
measurements
Exhaustive identification of rationales
Uncertain future
Prediction incorporating uncertainty
Measurement program improvement by
machine learning
Self-certified quality
Standard-based evaluation
Pattern-based evaluation

4
Negative
Hawthorne effects
Goal-orientation
Organization
misalignment
measurements
Uncertain future
machine learning

Hawthorne Effect
5nicolasdsampson.com, Observe And Learn: The Magic Of Paying Attention
http://nicolasdsampson.com/wp-content/uploads/2012/10/2010_12_06_observe-learn-magic-paying-attention.jpg

Goal-Question-Metric (GQM) Paradigm
• Goal-oriented framework for identifying goals and necessary
corresponding metrics
• Goal: measurement goals
• Question: questions for evaluating goal achievement
• Metric: objective or subjective metrics for obtaining
necessary quantitative data to answer questions
Identifying
metrics
Interpreting
measurement
results
Metric
Goal
Question
Collected data
Goal achievement
evaluation
Answer
Measured
value
R. van Solingen, E. Berghout, “The Goal/Question/Metric Method” McGraw-Hill Education, 1999

7
GQM-based Multidimensional Measurements
H. Washizaki, R. Namiki, T. Fukuoka, Y. Harada, H. Watanabe, "A Framework for Measuring and Evaluating Program
Source Code Quality", 8th Int’l Conference on Product Focused Software Development and Process Improvement (Profes2007)
Reliability
Maintain
-ability
Portability
Efficiency
Reusability
Analysability
Changeability
Testability
Goal Metric
Depth of
call graph T
Cyclomatic
Number min
Estimated static
path length min
Measured by
some tools
Measured by
some tools
Are functions
not
complicated?
Question Sub-
Question
Is call-
nesting
not too deep?
Is logic
not too
complex?
IndependentIndependent SpecificSpecific
Are entities
named
properly?
(Scale type: T threshold, min smaller is better, max larger is better)

SEMAT-based Multidimensional measurements
8
Software System
Opportunity Stakeholders
Requirements
Work Team
Way of Working
Customer
Solution
Endeavour
Ivar Jacobson, Pan-Wei Ng, Paul E. McMahon, Ian Spence, Svante Lidman, “The Essence of Software Engineering:
The SEMAT Kernel,” Communications of the ACM, vol.55, no.12, pp.42-49, December 2012.

alpha as Project Measurement
9
 State 1
 XXXXXXXXXXXXXXXXXX
 XXXXXXXXXXX
 XXXXXXXXXXXX
 State 2
 XXXXXXXXXXX
 XXXXXXXXXXXX
 State 3
 XXXXXXXXXXX
 XXXXXXXXXXXX
 ……..
 …….
Checklist
Adopted/modified from I. Jacobson, et al.: Tutorial: Essence - Kernel and Language for Software
Engineering Practices, ICSE'13
alpha
Indicator
1
2
3
４
５

Example from ITA WG on project failuers
10
• An employee in charge of Bank office
inquires "registered customer information
cannot be browsed from the terminal“.
• It was because a batch processing for the
previous day has ended abnormally due to
wrong data input.
• It took about two hours to recover the
data, and employees at each office had to
handle customer inquiry manually.
• For that reason, we had received plenty of
complaints from customers who had been
waiting for a long time!
H. Washizaki, “Analyzing and refining project failure cases from wider viewpoints by using SEMAT Essence,” Essence
Conference in Seoul, 2017.

11
Delay in service
deployment
Delay in service
deployment
Huge data inputs
manually
Huge data inputs
manually
Wrong data inputWrong data input
Defects in
operation manual
Defects in
operation manual
Misunderstanding
of abnormal state
as normal state
Misunderstanding
of abnormal state
as normal state
Problem
Cause
Root cause
Requirement
s: NG
“Coherent”
Software
System: NG
“Demonstrable”
Automate data
input
Automate data
input
Countermeasure
Frequent rule-
violations
Frequent rule-
violationsWork:
NG “Under
Control”
Way of Working:
NG “In Use”
Review and modify
rules
Review and modify
rules
Monitor and
enforce compliance
with rules
Monitor and
enforce compliance
with rules
Opportunity:
NG “Benefit
Accrued”
Team:
NG
“Collaborating”
Stakeholders:
NG “Satisfied
in Use”

12
Negative
Hawthorne effects
Goal-orientation
Organization
misalignment
measurements
Uncertain future
machine learning

Organization misalignment
13
We increase customer
satisfaction by quality and
usability improvement!
Top managementWe reduce customer-
reported defects by
improving testing process
and product maintainability.
Development
team We track defect data and
code quality metrics.
Quality
assurance
team

GQM+Strategies
M1: Customer
satisfaction survey
M1: Customer
satisfaction survey
Measurement DataGoals and Strategies
BusinessLevelSoftwareLevel
Goal: Increase
customer
satisfaction by 10%
Goal: Increase
customer
satisfaction by 10%
Strategy: Improve
product quality
Strategy: Improve
product quality
Strategy: Improve
usability of
product
Strategy: Improve
usability of
product
Goal: Reduce
customer-
reported defects
by 20%
Goal: Reduce
customer-
reported defects
by 20%
No sub-level
goal defined
Strategy: Improve
efficiency of
system testing
Strategy: Improve
efficiency of
system testing
Strategy: Improve
maintainability of
software
Strategy: Improve
maintainability of
software
Isolated
strategy
M2: Field defect
data
M2: Field defect
data
M3: Code quality
metrics (McCabe,
coupling, cohesion)
M3: Code quality
metrics (McCabe,
coupling, cohesion)
Isolated
data 14
• Alignment and tracing among goal, strategy
and data
Context & Assumption
Context & Assumption

Context-Assumption-Matrix [IEICE’16]
15Takanobu Kobori, Hironori Washizaki, et al., “Exhaustive and efficient identification of rationales using GQM+Strategies with
stakeholder relationship analysis,” IEICE Transactions on Information and Systems, Vol.E99-D, No.9, pp.2219-2228, 2016.
Target
View
Hospital
Insurance
company
Health
sensor
maker
・・・
Hospital
Insurance
company
Health sensor
maker
・・・
Insurance company can
offer personalized products
by having medical info…
Context / Assumption
Insurance
company
Medical
info
provider
Personal
medical
info.
Customized
product
Resource
Goal
depends

16
Insurance
company
G
S
A
S
Health
sensor maker
Medical
info.
provider
Expectation from the
view of insurance
company
Expectation from
the view of health
sensor maker
Receive medical
info.
Personalized
products developed
Strategy
Insurance product and
fee can be personalized
based on personal
medical info.
Goal
Sales of new
products
Personal
fitness
Confirm personal info.
Management system of
receiver of info.
Secure management
of medical info.
Strategy
Must ensure
personal info.
guideline
Goal
Insurance
company Medical
info.
provider
Confirming
management
systems
Pati
ent
Getting
consent
Conflict, redundancy,
complement
Conflict, redundancy,
complement

17
Towards clear dependency and alignment
G
S
A
S
Insurance
company
Medical
info.
provider
Personal
medical info.
Realizing
personalized
insurance
Objective of
medical
info. use
Managing
personal info.
Clarify objective
of use of medical
info.
Strategy
Establish personal
info. management
system
Strategy
Medical
info.
provider
Secure management
of medical info.
Strategy
Must ensure
personal info.
guideline
Goal
Health
sensor maker
Insurance
company
Receive medical
info.
Personalized
products developed
Strategy
Goal
Sales of new
products
Personal
fitness
Insurance product and
fee can be personalized
based on personal
medical info.

S1 S2
S3 S5
S6
S4 S7
S1
S2
S3
S4
S6
S5
S7
Interpretive Structural Modeling (ISM)
18
Impact
S1 S2 S3 S4 S5 S6 S7
S1 1 0 0 0 0 0 0
S2 0 1 0 0 0 0 0
S3 1 1* 1 0 1 0 0
S4 1 1* 0 1 0 0 1
S5 1* 1 1 0 1 0 0
S6 1* 1 0 1 0 1 1*
S7 1* 1 0 1 0 0 1
Power.
Hierarchical
structuring
可到達行列
S1 S2 S3 S4 S5 S6 S7
S1 1 0 0 0 0 0 0
S2 1 1 0 0 0 0 0
S3 1 0 1 0 1 0 0
S4 1 0 0 1 0 0 1
S5 0 1 1 0 1 0 0
S6 0 1 0 1 0 1 0
S7 0 1 0 1 0 0 1
Relation matrix Reachability matrix

ISM-based Alignment [HICSS’16]
S1
S2
S3
S4
・
・
S1 S2 S3 S4 ・・
1
1
(1) Decompose into elements
(2)Restru
cturing
(3)Analysis&
alignment
S5
S1 S2
S3
S4
S6 S7
Hierarchical structure
S5
S1 S2 S3
S4 S6 S7
ISM
1
Elements (especially strategies)
1
1
1
1
S5
S1 S2
S3 S4 S6 S7
GQM+Strategies
G
GG G
Reachability matrix
Conflict specified
• Alignment for single GQM+Strategies model
• Future: alignment over areas and stakeholders
Yohei Aoki, Takanobu Kobori, Hironori Washizaki, et al., “Identifying Misalignment of Goals and Strategies across Organizational
Units by Interpretive Structural Modeling,” 49th Hawaii International Conference on System Sciences (HICSS), 2016

20
Negative
Hawthorne effects
Goal-orientation
Organization
misalignment
measurements
Uncertain future
machine learning

Identifying parts
that are hard to
maintain…
How large?
ELOC N. functions
Measurement System Improvement by ML
N. Tsuda, et al. Iterative Process to Improve GQM Models with Metrics Thresholds to Detect High-risk Files, SANER 2015
Review
Quality
measurement
Machine
learning
Goal
Question
Measurement
22
10 25
300
150
N. functions
ELOC
OK
NG
71
N. functions
ELOC
OK
NG
Improvement
X

]]
#Issues
Actual
Predicted
DayKiyoshi Honda, Hironori Washizaki and Yoshiaki Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and
Dynamics: Model and Applications”, International Journal of Software Engineering and Knowledge Engineering (IJSEKE), 2016.
Logistic
Gompertz
Non-homogeneous
Poisson
process(NHPP)
Software reliability model (SRM)
as actionable metric
23

Prediction with uncertainty
0
10
20
30
40
50
60
70
80
90
0 2 4 6 8 10 12 14
#Issues
Time
ActualData
Our model
24
Kiyoshi Honda, Hironori Washizaki and Yoshiaki Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and

Prediction with uncertainty
0
10
20
30
40
50
60
70
80
90
0 2 4 6 8 10 12 14
#Issues
Time
ActualData
Our model
-
+
25
Lower (worst case)
Upper (best case)
T1 T2

Uncertainty patterns and prediction
26
0 0.5 1
ConstantIncrease Decrease

27
Negative
Hawthorne effects
Goal-orientation
Organization
misalignment
measurements
Uncertain future
machine learning

Self-Certified Quality
28
Image: How to Spot a Fake: Avoiding Degree Mill Scams https://wenr.wes.org/2015/06/spot-fake-avoiding-degree-mill-scams

ISO/IEC 25000 SQuaRE-based
Quality Measurement and Benchmark
29
Q1. Any path going through internal
servers only?
Q2. Any path going through outside
servers?
Q3. Any P2P communications?
…
G. The events or actions cannot
be repudiated later through
communication channels (paths).
M. Signed communication path ratio
= #Signed_paths / #Total_paths
E.g. Non-repudiation
Waseda U. Team Vendor
1 Concretize SQuaRE
measurements by
GQM
2 Prepare measurement
methods: data forms,
static analysis,
questionnaire, user-
testing
3 Conduct code
static analysis,
user-testing
Fill data
forms,
questionnaire
4 Measure and
evaluate quality
Scores by using percentile
E.g., Top 30% = 0.7
HighLow
#Products
Measured value
H. Nakai, N. Tsuda, K. Honda, H. Washizaki and Y. Fukazawa, Initial Framework for a Software Quality Evaluation based on
ISO/IEC 25022 and ISO/IEC 25023, IEEE International Conference on Software Quality, Reliability & Security (QRS 2016)

21 Japanese Products Measurement
30
Functional
suitability
Performance
efficiency
Compatibility Usability
Reliability Security Maintainability Portability
Effectiveness Efficiency Satisfaction Freedom from
risk
Context
coverage • Low security and compatibility in some products
• Necessary to address these in IoT era

Relationships among characteristics
31
Internal/External Quality Quality in Use
p-value < 0.1p-value < 0.1
性能効率性互換性使用性信頼性セキュリティ保守性移植性有効性効率性満足性リスク回避性利用状況網羅性
機能適合性 0. 31 0. 19 -0. 72 0. 37 -0. 05 0. 50 0. 31 -0. 14 0. 52 1. 00 1. 00 1. 00
性能効率性 0. 44 0. 24 0. 36 -0. 17 0. 37 0. 32 0. 32 -0. 10 -0. 50 -0. 50 -0. 50
互換性 0. 04 0. 17 -0. 06 0. 36 -0. 04 -0. 14 0. 05 -0. 50 -0. 50 -0. 50
使用性 0. 17 -0. 21 0. 11 0. 44 -0. 09 -0. 20 -1. 00 -1. 00 -1. 00
信頼性 0. 30 0. 41 0. 45 -0. 08 0. 11 1. 00 1. 00 1. 00
セキュリティ -0. 06 0. 19 0. 64 -0. 34 0. 50 0. 50 0. 50
保守性 0. 26 -0. 29 0. 01 1. 00 1. 00 1. 00
移植性 -0. 21 0. 67 0. 50 0. 50 0. 50
有効性 0. 03 -1. 00 -1. 00 -1. 00
効率性 1. 00 1. 00 1. 00
満足性 1. 00 1. 00
リスク回避性 1. 00
Func.
Perf.
Comp.
Usa.
Relia.
Sec.
Main.
Port.
Effe.
Effic.
Sati.
Free.
Perf. Comp. Usa. Relia. Sec. Main. Port. Effe. Effic. Sati. Free. Cont.
• Negative correlation between usability
and functionality.
• Need to adopt user-centered
development

Appropriate
design
Inappropriate
design
Security Patterns and Testing
32
Role-based access control (RBAC) pattern

TESEM: Test Driven Secure Modeling Tool
[ARES’13][ARES’13][IJSSE’14][ICST’15][Information’16]
33
Security
Design
Pattern
Problem
Solution
Context
Test design as
requirement
! create Actor
! create UI :
! create Subject..
Test Script Test case
testing
[ARES’13] Validating Security Design Pattern Applications Using Model Testing, Int’l Conf. Availability, Reliability and Security
[ARES’14] Verification of Implementing Security Design Patterns Using a Test Template, Conf. Availability, Reliability and Security
[IJSSE’14] Validating Security Design Pattern Applications by Testing Design Models, Int’l J. Secure Software Engineering 5(4)
[ICST’15] TESEM: A Tool for Verifying Security Design Pattern Applications by Model Testing, IEEE ICST’15 Tools Track
[Information’16] Implementation Support of Security Design Patterns Using Test Templates, Information 7(2)
testing

34
window.onload = setEventHandler;
function setEventHandler() {
$(“reg_type”).onchange = calcPrice;
・・・
$(“reg_addcart”).onclick = addCart;
};
function calcPrice() { ・・・ };
function addCart() {
if(isValidInput()) {
reqRunTrans();
} else {
alert(“Invalid user inputs”);
}
};
function reqRunTrans() {
new Ajax.Request(“runTrans.php”, {
method: “GET”, parameters: getParams(),
onSuccess: succeeded });
};
function succeeded() { disableAll();
jumpToConfirm();
};
・・・
};
reqRunTrans();
} else {
}
};
onSuccess: succeeded });
};
jumpToConfirm();
};

35
Finite State Machine Extraction
……………………………………………
……………………………………………
$(“addcart”).onclick=addCart;
……………………………………………
……………………………………………
……………………………………………
……………………………………………
$(“addcart”).onclick=addCart;
……………………………………………
……………………………………………
onclick
setEventHandler
addCart
E.g. User Action [Mahamoff’06]
“Prevent multiple calls of
specific user event handler”
Y. Maezawa, K. Nishiura, H. Washizaki, S. Honiden, Validating Ajax Applications Using a Delay-Based Mutation Technique”,
29th IEEE/ACM International Conference on Automated Software Engineering (ASE 2014)
Y. Maezawa, H. Washizaki, Y. Tanabe and S. Honiden , “Automated Verification of Pattern-based Interaction Invariants in Ajax
Applications, 28th IEEE/ACM International Conference on Automated Software Engineering (ASE2013)
Duplicate
order
Duplicate
order
[Mahamoff’06] M. Mahamoff, “Ajax Design Patterns”, O’Reilly Media Inc., 2006.

36
・・・
};
$(“addCart”).disabled = true;
reqRunTrans();
} else {
$(“addCart”).disabled = false;
}
};
onSuccess: succeeded }); };
jumpToConfirm(); };
・・・
};
$(“addCart”).disabled = true;
reqRunTrans();
} else {
$(“addCart”).disabled = false;
}
};
onSuccess: succeeded }); };
jumpToConfirm(); };

37
Negative
Hawthorne effects
Goal-orientation
Organization
misalignment
measurements
Uncertain future
machine learning

SamurAI Coding
IPSJ 6th International AI
Programing Contest
World Final
March 14 2018 Tokyo
http://samuraicoding.info
APSEC 2018
25th Asia-Pacific Software
Engineering Conference
Nara
Dec 4-7 (due: June)
PC Chair: H. Washizaki
Int. Journal of Agile and
Extreme Software
Development
(IJAESD)
Editor-in-Chief: H. Washizaki
COMPSAC 2018
42nd IEEE Computer Society Int’l Conf.
Computers, Software & Applications
Tokyo
July 23-27 (due: Jan 15)

Pitfalls and Countermeasures in Software Quality Measurements and Evaluations

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Pitfalls and Countermeasures in Software Quality Measurements and Evaluations

Similaire à Pitfalls and Countermeasures in Software Quality Measurements and Evaluations (20)

Plus de Hironori Washizaki

Plus de Hironori Washizaki (20)

Dernier

Dernier (20)

Pitfalls and Countermeasures in Software Quality Measurements and Evaluations