Fitman webinar 2015 06 Verification and Validation methodology

116/06/2015 FITMAN Webinars, Online
FITMAN Verification & Validation
Methodology
FITMAN Webinars
June 16th, 2015
Dr. Fenareti Lampathaki
National Technical University of Athens (NTUA)

What is Verification & Validation?
Verification Validation
Definition
[IEEE Std
1012-
2004]
The process of providing objective evidence
that the software and its associated products
conform to requirements (e.g., for
correctness, completeness, consistency,
accuracy) for all life cycle activities during
each life cycle process (acquisition, supply,
development, operation, and maintenance);
satisfy standards, practices, and conventions
during life cycle processes; and successfully
complete each life cycle activity and satisfy all
the criteria for initiating succeeding life cycle
activities (e.g., building the software
correctly).
The process of providing evidence that the
software and its associated products satisfy
system requirements allocated to software at
the end of each life cycle activity, solve the
right problem (e.g., correctly model physical
laws, implement business rules, use the
proper system assumptions), and satisfy
intended use and user needs.
Question Are we building the product right? Are we building the right product?
Objective To ensure that the product is being built
according to the requirements and design
specifications.
To ensure that the product actually meets the
user’s needs, the specifications were correct
in the first place and the product fulfils its
intended use when placed in its intended
environment.

Bringing together agile
and waterfall software
engineering philosophies
FITMAN V&V Method purpose
Infusing a crowd
assessment mentality in
the V&V activities
Including both business and
technical perspectives
Providing initial
practical guidelines
An all-inclusive framework for verifying, validating and evaluating a software product
from its conception to final release and implementation in real-life
Key message:
 A general, holistic method including techniques to be applied
for each step and recommendations for its application by the
trials
 A method reusable beyond FITMAN:
• Each trial and development team can streamline the method
according to their own needs

FITMAN WP2 D2.1 4
FITMAN V&V Methodology – Overall Picture

Trial Specific Perspective: Steps T-1 & T-2
 Business Validation (T-2) to assess whether the overall trial solution
eventually offers sufficient added value to the trial.
 Demonstrating that the software developed has clear benefit to the trials, allowing
them to operate more efficiently (usually in terms of cost, time or quality) than
before, or supporting them to do something they couldn't do before.
User Acceptance Testing
Simplified ECOGRAI Methodology
 Trial Solution Validation (T-1) to guarantee that the overall trial solution
satisfies intended use and user needs.
 Validation of each trial solution from a technical and functional point of view only.
5
5 TIs
BPIs

6
Product Specific Perspective: Steps P-(1-5)
 Product Validation (P-5) to examine whether the
product satisfies intended use and user needs.
User Acceptance Testing
 Release Verification (P-4) to
determine whether the
requirements of the final
product release are met.
 Backlog Verification (P-3) to
determine whether the
requirements of the product
after each sprint are met.
 Model Verification (P-2) to
coordinate the alignment
between design and
requirements, as well as
between design and code.
 Code Verification (P-1) to ensure functionality,
correctness, reliability, and robustness of code.
Alpha Testing
Regression Testing
Traceability Analysis
White Box Testing
Developer Acceptance Survey
7+2 TIs
Test Cases Documentation

Assessing Business Impact
Business Performance Indicators (BPI) are identified for each
trial through a simplified ECOGRAI Method.
Characteristics of a Business Performance Indicator (BPI):
• easy to be interpreted, to put in work, to use or to exploit
• easily measurable, quantifiable
• representative of the objective which it measures
For each Business Performance Indicator, the trials are
required to report:
• The current value (AS-IS)
• The target value they want to achieve (Target) and
• The actual achieved values after the solution
implementation (TO-BE)

Initial Technical Indicators (filled in by trials)
8
Fulfilment of
requirements
“The solution fulfils the Trial
requirements”
Learnability
“It is easy to start to use the
solution and learn functionalities”
Understandability
“The solution is easy and self-
clear to understand and the
concepts and terminology are
understandable”
User’s attraction level
“The solution is attractive to the
user. I feel satisfied and
comfortable when using it”
Efficiency
“The time and resources required
to achieve the objectives of the
solution are reasonable, the
solution is fast enough and does
not require too many steps”
Openness
“Ensuring that specific people
groups may access the software
for free with specified rights”
Interoperability
maturity
“The capability of the software to
interact with other systems”
Ease of application
“A measure of the applicability of
the software in the particular
environment in terms of amount
of work and extra actions or
means”
SpecificEnablersTrialsSolution
• Strongly agree
• Agree
• Neither agree
nor disagree
• Disagree
• Strongly
disagree
DifferentLevelsperTI

Additional Technical Indicators
Technical
Indicator
Definition Value (in 3 levels)
Correctness The degree to which the software is
free from defects in its specification
and implementation
High (no defects detected) / Medium (relatively minor
defects detected) / Low (severe defects detected)
Efficiency The capability of the software to
provide appropriate performance,
relative to the amount of resources
used
High (performance beyond expectations, with reasonable
resources consumption) / Medium (expected performance,
even with small increase of resources utilization) / Low
(Marginally accepted performance and/or overuse of
available resources)
Reliability The capability of the software to
maintain a specified level of
performance when used in the
factory settings
High (totally stable performance) / Medium (marginal
performance divergences) / Low (the performance of the
software varies significantly depending on the conditions)
Sustainability How easy it is to maintain the
software and apply required
modifications for adapting to changes
in the operating environment
High (software maintenance and modification are very
easy) / Medium (software maintenance requires reasonable
effort) / Low (software maintenance is very complex and
modifications difficult to be applied)
Additional calculated indicators (not to be assessed by trials nor inserted in SurveyMonkey):
Versatility: to be estimated according to the formulas provided in WP2
Portability: to be estimated on a cross-trial basis using existing indicators
Existing indicators (already assessed for GEs/SEs): Openness / Ease of application / Interoperability maturity

Trial Journals
10
Unstructured
Information
Unstructured information for the experience of each Trial is collected.
Technical Journal:
Registration of the implementation issues encountered in the implementation of the Trial system
Registration of the operational resilience of the Trial (e.g. major bugs, blocking errors, etc.)
Business Journal:
Collection and analysis of the most important operational issues faced in the implementation of the
system in the Trial, e.g. organizational and business difficulties, degradation of the business system.

Overall V&V in practice
11
Developer Acceptance
Questionnaires
Validation of SEs by anyone who reuses
them
4+

An example of Self-certification for SEs

An Example for “Release Verification” documentation

An example for “Product Validation” documentation

Lessons Learnt so far…
• Experience in FITMAN was collected from trial owners and
technical partners:
– The users mainly considered the method sufficient
– Training, support and the online V&V tool are crucial
– Definition of BPIs was not trivial, but it was worth the effort
• Challenges with the definition of objectives and the BPIs, initially not well
formulated / measurable
• Confidentiality issues
• V&V from technical and business perspective must be an
ongoing and iterative process…

FITMAN V&V Decalogue
I. Have a deep knowledge of the requirements and the ends sought in each
trial.
II. Infuse a V&V culture to your trial.
III. Create a concrete V&V plan to be followed in each step, tailored to your
real needs and competences.
IV. Engage from the very beginning the required stakeholders, assigning
them with appropriate responsibilities and decision power.
V. Tap the power of data to effectively conduct V&V activities.
VI. Proceed with each V&V step on the basis of at least 1 recommended
technique.
VII. Keep the balance between business and technical V&V aspects.
VIII. Invest on crowd-sourcing techniques aligned to the trial philosophy.
IX. Keep a complete V&V log and document in detail the V&V findings.
X. Treat V&V as a continuous, iterative procedure.

Thanks for your attention!
Dr. Fenareti Lampathaki (NTUA)
Join Us!
17
fenareti.lampathaki @fenaretiflamp@epu.ntua.gr

Fitman webinar 2015 06 Verification and Validation methodology

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Fitman webinar 2015 06 Verification and Validation methodology

Similaire à Fitman webinar 2015 06 Verification and Validation methodology (20)

Plus de FITMAN FI

Plus de FITMAN FI (20)

Dernier

Dernier (20)

Fitman webinar 2015 06 Verification and Validation methodology