Kerim Cakmak, Moshe Cohen -- Continuous Verification and Validation

Applying Continuous Verification and Validation
to achieve the right Quality in Systems delivery
Business
Risk
Technical
Risk

Kerim Cakmak, PhD
IBM Rational Tiger Team CEE
Moshe S. Cohen
QM Market Manager and Evangelist

© Copyright IBM Corporation 2013

Software and Systems Engineering | Rational

The V is dead! Long live the new V with continuous Verification
and Validation across the full V!
Abstract:
While we know how to reduce technical risks in Systems delivery through Verification activities,
our business risk is on a steep rise. This is largely due to the fact that "Validation" is performed
at the end of the development lifecycle, rather than throughout, and it is done against customers
requirements rather than end-users needs. In this session we will discuss some new best
practices around Verification and Validation that were observed through recent successful and
even more unsuccessful projects, and show you how to continuously verify and validate your
designs throughout the whole lifecycle and not only at the tail end of it. Not only will this improve
the quality of your products and systems, but it will also allow you to converge earlier on the right
requirements, compressing the time to end users feedback, and improve your time to market
predictability. While it doesn't change the V model, it is a new V.

2

© 2012 IBM Corporation


Bottom Line is…
 While delivering Quality is already a major challenge, the definition of Quality is
changing…
– Low tolerance for bad quality (real and perceived)
– Expected short cycles with continuous improvements

 To deliver Quality, we need a holistic approach to Quality
– Continuously verify and validate our designs, throughout the whole lifecycle.

 Business results:
– Deliver the right system
• Compressing time to customer’s feedback
– Better time to market predictability

• Earlier convergence on the right Requirements
– Lower cost of Quality
• Defect* Avoidance, Lower defect density, Higher defect removal efficiency
* Defect, as defined in ISO 9000 (2008): The non-fulfillment of intended usage requirements.
The term Defect will be used here in the general sense, regardless whether it is a product
defect, design defect, manufacturing defect, an error, a fault or a failure.

3



Agenda




4



In the News… Business Implications of a Failed missile test
Could result in major financial consequences, unless Raytheon succeeds in
demonstrating that it is not its fault.

5



Business Implications of Recalls
Not only recalls are very expensive, but they also drive corporate behaviors.
 Sony recalls: “Sony-related recalls are following one another, and that may ruin
the company’s brand image,” said Keita Wakabayashi, an analyst at Mito
Securities Co. with a “neutral plus” rating on the stock. Bloomberg, Oct 2011.
 Toyota Prius gas pedal recall, 2010: ~$3B (CNNMoney.com)
– Repair cost (quality related expenses):

$1.12B

– Legal cost (class-action lawsuit):

$1.1B (WSJ, Dec 27, 2012)

– Image cost (sales losses):

$770M-$880M

– “Toyota said Thursday that a software problem is causing problems with the brakes”…
CNNMoney, Chris Isidore, senior writer, Feb 4, 2010.
 GM presses suppliers for future recall costs (Automotive News, Aug 5, 2013)
– GM has adopted a new purchasing contract that would allow it to recover the cost of safety
recalls from its suppliers.
– The new GM contract has open-ended implications, stating that the supplier's components
"will not, at any time (including after expiration or termination of this contract), pose an
unreasonable risk to consumer or vehicle safety."
6



So… What is Quality?
Exceeding end users needs, even when they keep changing.
No single definition, most definition are very detailed and
verbose… but the common theme is most definitions is:
 Yesterday: Meeting the Spec.
 Today: Meeting or exceeding customers and endusers needs and expectations
– “End users needs” ≠ “Initial Requirements”

“The test of innovation lies not in its novelty, its
scientific content or its cleverness.
It lies in success in the marketplace.”
Peter Drucker.
(Business Management Philosopher)
7



Agenda







8



Continuous Verification & Validation Process

End users needs, at least initially, are imprecise and incomplete – This is Normal!
Initial Elicitation
of end-users needs
Starts with early elicitation of end-users needs
 Helping the customer/end user to flash out their needs (Requirements Elicitation)
– Imagine you want to build a new home, do you know the exact size of the house?

Notice that this results in Initial understanding of the needs, still partial and incomplete

Continuously
Verify…

Continuously
Validate…

Continuously verify, throughout
Development, that you are building the
system right.
•
•

As the builder checking with the architect.
As the city inspections, making sure the house is
being built in compliance with regulations.

Continuously validate with the
customer, as the design is getting more
concrete, that you are building the right
system.
•

As an architect and/or the builder validating
assumptions they have made as your house
is being built.
Continuously improving understanding of

customer/end-users needs

Delivery



Different perspectives
What all the stakeholders need

What the customer needs

What the customer thinks they need
What the customer says they want

Given this

How do we
get to this?



Verification vs. Validation Examples

While Verification is “Internal”, Validation is “External”
 Verification: The evaluation of whether or not a product, service or system complies
with a regulation requirement, specification, or imposed condition. It is often an internal
process. (IEEE)
• Code should never run thru a divide by zero as it may cause a crash or unexpected results.
• Did you implement the design pattern properly?
• Does this block diagram represent the best architecture for the system?

 Validation: The assurance that a product, service, or system meets the needs of the
customer and other identified stakeholders. It often involves acceptance and suitability
with external customers. (IEEE)
• On a cycling computer… The customer asked for certain data fields to be displayed… but
wouldn't the end user want to customize them?
• Is this {…} the right sequence of voice commands for making a phone call while driving
your car?
• What's the right behavior of the wifi enabled home dialysis machine when the connection is
intermittent?

11



Verification vs. Validation

While Verification is “internal”, Validation is “External”
 Verification: Building the system right
– Testing against requirements
• “Internal” requirements
– Utilizes Test, Simulations, Analysis,
Inspections and Demonstrations

Customers
Needs

– Assuring robust design and implementation

Requirements
Analysis

System
Testing

– Compliance with industry regulations
– We “know” how to do that… often done well!

 Validation: Building the right system

Systems Analysis
Systems Design

Integration
Testing

– Validating against your customers needs
• As expected, imprecise and incomplete
• Often negotiated with several teams
• Often negotiated with people who may not be
the end users

– Do we know how to do that? Do we do it?

12

Detailed Design
&
Implementation

Note that the V does not represent a
development process but rather a flow
of information as we decompose and
recompose a system.


While Verification detects defects, Validation is about avoiding them.
Left
side of the V
Verification

• Early defect
removal

Right
side of the V
• Expensive
defect
removal

Customers
Needs
System
Testing

Requirements
Analysis

Validation

13

• Defect
Avoidance
• Early
detection of
potentially
business or
mission
failures

• Early
Acceptance
tests
• Final
Acceptance
test

Systems Analysis
Systems Design

Integration
Testing

Detailed Design
&
Implementation



While Verification reduces Technical risk, Validation reduces Business or
Mission risk
 Verification: Reduces Technical risks
– It is a Testing process…

Business
Risk

– We “know” how to do that… often done well!

 Validation: Reduces Business risks

Technical
Risk

– Focuses on closing the loop with the end-user as
often as possible
• Can’t just blame the Requirements being wrong…

• “Meeting the spec” is not a guarantee for business
or mission success…
– It is NOT a Testing process…
but rather an on-going Discovery process
• Important to “developers”
• Critical to End Users – Helping users figuring out their own Requirements!
– Do we know how to do that? Do we do it?

14



Continuous Verification and Validation (V&V)
Compressing time to customers feedback!
Very
Common:

Customers
Requirements

Late Validation (Re-Evaluate)

Long/Costly
Process
System
Testing

Requirements
Analysis

System
Testing

Requirements
Analysis

Systems Analysis
Systems Design

Integration
Testing

Systems Analysis
Systems Design

Requirements
Analysis

Integration
Testing

System
Testing

Systems Analysis
Systems Design

Integration
Testing

Detailed Design
&
Implementation

Detailed Design
&
Implementation

Detailed Design
&
Implementation

Example:
 AA and United approached Boeing for a new flight entertainment system
 Job has been sub’ed to Sony Trans Com., who designed & delivered a state-of-the-art system.
 End result:
– Premium paying passengers had trouble operating the system
– Flight attendants had trouble supporting the passengers

 End-End result: Flight attendants shutting down the system before take off, claiming malfunction.


Continuous Verification and Validation (V&V)
Compressing time to customers feedback!
Late Validation (Re-Evaluate)

Very
Common:

Long/Costly
Process

Customers
Requirements

System
Testing

Requirements
Analysis

System
Testing

Requirements
Analysis

Systems Analysis
Systems Design

Integration
Testing

Systems Analysis
Systems Design

Integration
Testing

Requirements
Analysis

System
Testing

Systems Analysis
Systems Design

Integration
Testing

Detailed Design
&
Implementation

Detailed Design
&
Implementation

Detailed Design
&
Implementation

Continuous V&V
per Feature:

Use
Scenarios?
Validation

Operational
Scenarios?
Validation

Prototypes?
Validation

HIL/SIL?
Validation

Final?
Validation



Customers
Needs
Verification

16

Verification

Verification

Verification

Verification



Agenda







19



Collaboration through Reviews
Testing for early feedback from Internal and External stakeholders
 Review process where peers and other stakeholders collaborate in order to identify potential
errors and deviations from the spec.
 Objectives include:
– Detect wrong understanding, bad assumptions and defects close to when introduced (Functional review)
– Identify potential improvements and/or risks (Functional review)
– Verify compliance with regulations and conformance to standards (Quality review)

 Audience:
– External Reviews: With customers and/or end-users,
often focused on Validation
– Internal Reviews: With internal stakeholders (peers, management),
often focused on Verification

 Formal inspections (with defined roles &
process) to ad-hoc team reviews and walkthroughs.

 FDA Example: On Going Reviews  / Late Validation 

20



Reviews are collaboration points
Include the voice of the customer for on-going validation
 Applies to the full development life cycle and to all artifacts

 Reviews are short, and per feature (best practice: focus on end user feature)
 Examples:
Review

Internal/External

Requirements elicitation

Identifying Use
Scenarios

External Review

What are the main
functions/activities?

Requirements Review

Defining Operational
Scenarios

External Review

How does the system
operate?

Architecture Design
Review

Is this the “best”
architecture?

Internal Review

Design documents

Initial Prototype Review

Early feedback based
on feature execution

External review

Low Fidelity prototype,
early feedback.

Advanced Prototype
Review

Production code in
HIL/SIL environment

External Review

Hi Fidelity prototype,
still early feedback.

Test Plan Review

Agreement across all
stakeholders

Internal and/or External
Review

Sunny & Rainy days
scenarios

Test Execution Review
21

Objective

Discuss Quality status
and next actions

Internal and/or External
Review

Focused on high level
operational scenarios


Reviews Impact
Reviews are ~2x more efficient in removing defects than Testing
(Not including Defect Avoidance)

Reviews

Testing

Source: Capers Jones, 2012
22



Reviews Impact
Reviews are ~2x more efficient in removing defects than Testing
(Not including Defect Avoidance)

Reviews

Testing

23



End Users Operational Scenarios
Use them as top level test cases to continuously drive all V&V activities
 Operational Scenarios
– Quasi formal but yet intuitive
– Can be extended with sketches and
Interaction Diagrams

 Operational Scenarios are a Testable version
of the requirements!
– Can be converted into System level test cases

 System level test cases:
– Should be Validated (reviewed and approved)
with the customer and/or end-user
– To be used as a Golden Reference model
throughout design and implementation for
Verification purposes.

Although diagrams shown here are SysML diagrams,
concept applies to any system.
24



To leverage System Level Test Cases, you need…
Use Operational scenarios as Tests throughout the whole development Lifecycle
 Validate your understanding of the needs by reviewing the expected scenarios with the
stakeholders and getting them approved.
 A Modeling environment that support Use Case analysis

 A Quality Management process that support the conversion of Operational scenarios into
test cases
– Test cases for either Manual Testing or Automated Testing
– May require internal review and approval

 Traceability: Needs  Requirements  Operational Scenarios  Test Cases

Operational
Scenarios

System
Level
Test
Cases

Real

Prototype
25

Virtual



Example User Scenario
Boat
loaded

Boat lifted
Boat on car

Ready to
sail

Boat
unloaded

sequential
Boat
rigged
parallel

To have
sailed
and
survived

Mast rigged
Center-plate
rigged
Rudder rigged
Gybed

Subgoals

Sailed
normally

Tacked

alternate

Overal
l
goal

Sailed
periodic

Returned
home

Cruised

Maneuvered
alternate

Ashore

Recover
from
Capsize

Righted

exception

Coast
guard
contacted


Requirements from Scenarios
Goal hierarchy

Boat
loaded

Boat lifted

traceability

Two people shall be able
to lift the boat onto the
roof of the average
saloon car.

Boat on car

Ready to
sail

Boat
unloaded

sequential
Boat
rigged
parallel

To have
sailed
and
survived

Stakeholder
requirements

Mast rigged
Center-plate
rigged

The sailor shall be able to
perform a gybe.

Rudder rigged
Gybed

Subgoals

Sailed
normally

Tacked

alternate
Cruised

Maneuvered

Sailed
Overall
goal

periodic

Returned
home

alternate

Ashore

Recover
from
Capsize

exception

Righted
Coast
guard
contacted

The sailor shall be able to
contact the coastguard
when the boat is
capsized.


Virtual Models

Enabling early Verification and Validation… before building anything real.
 Virtual models that can be used for early analysis and trade studies (functionality, behavior,
architecture, structure, performance, reliability, safety, etc.)
– Models abstract mechanics, electronics and/or software entities, operating independently or integrated
– Different domains:
• Mechanical/Control models (such as Mathworks, NI)
• Graphical languages (such as SysML)
• Textual language (such as SystemC for HW design)

 Can be used for
– The system being designed AND its TestBench (Plant models)

 Use System level test cases to drive analysis
– Execution, Simulation or Prototyping

 Various configurations driving on going system analysis:
• Eg. 1: Virtual sensors driving virtual HW+SW models
• Eg. 2: Mechanical prototype driving virtual HW+SW models
• Eg 3.: Mechanical prototype driving virtual HW+production SW
28



Modeling has low potential of introducing defects
Models are best in removing defects
 Models are Executable
– You can ask Questions and get consistent
answers!

 Models can be Analyzed

Model Based:
Low Defect Potential,
Best Removal Efficiency

– Tradeoff studies
– Performance
– Behaviors
– Etc.

 Leverage correct-by-construction Synthesis
capabilities to guide the design and
implementation process

Source: Capers Jones, 2012

– Eg. 3D printers, C/C++ for SW, SystemC for HW
– Enabling Co-Execution for V&V purposes

29



Effective and Efficient Testing
Single source of truth for all Quality related artifacts and processes
 Central Quality Management Hub, acting as a single source of truth, enabling:
– Collaborative Test Planning
• Development and approval for Verification and Validation test plans
– As well as specific plans as needed, such as Safety.
• Traceability back to Requirements (why we need which test?)
– Managing test execution (manual and automated) according to Test Plan
• Who is running which test, when, why, pre-conditions, test lab/equipment reservation etc.
– Integrated Defect Management

• What went wrong, how to reproduce, defect resolution, reporting, etc.
– Traceability across the full lifecycle
• User Needs  Requirements  Test cases  Test Results  Defects  Resolutions

 Other key capabilities
– Collaboration around Rainy days scenarios

– Test assets ReUsability
– Reconciliation process between Requirements and Test Plans/Test Cases
30

– Actionable Analytics to drive process improvements



Overall Impact on Testing Effectiveness
Collaborative Test Planning, Traceability, Test Execution and Analytics
Impact of Quality Management on Process Efficiency
100%
85%

90%
76%

80%
70%
58%

60%

85% 87%

Detect 3,
release 7

75%

60%

Detect 3,
release 2

50%
40%

32%

30%

30%
20%

15%

10%
0%
1
QM Impact:

20%

2

3

40%

40%

CMMI Levels

4

5

40%

10%

W/O QM

W QM

W/O QM: Percentage of defects detected during Requirements review, design
reviews, unit testing and Functional testing – Current practice

W QM: Percentage of defects detected thru Requirements review, design
reviews, unit testing and Functional testing – Improved practice

31

Source: Analysis of over 850 GBS projects and Industry data



Agenda








32



Key Take-Aways
 Verification in Inbound; Validation is Outbound. Both needed.

 Verification is a Testing process. Validation is a Discovery process, helping the customer
refine their understanding of their own needs.
 Compressing time to feedback allows for change… and therefore reduces surprises.
 Reduce business risk with Early and Frequent feedback from all stakeholders
– Use Scenarios and Operational Scenarios

– Validation Test Plan
– Low Fidelity and Hi Fidelity prototypes

 Operational scenarios = Testable Requirements = Golden reference
– Use them for your Testing throughout the whole development process and not only at the end!

Question:
 Unit Testing and Integration Testing are both very important.
 Which is more important?
 Which one would you do first?
34



Continuous V&V – Reducing Technical and Business Risks
 Quality can be delivered by compressing the time to customer feedback with
continuous Verification and Validation.
 Both Technical and Business risks can be lowered
 Business Implications:
– Lower cost of Quality

Business
Risk

Technical
Risk
• Defect Avoidance, Lower defect density, higher defect removal efficiency

– Deliver the right system
• Compressing time to customer feedback

– Better time to market predictability
• Earlier convergence on the right Requirements

35



36
36



www.ibm.com/software/rational

© Copyright IBM Corporation 2013. All rights reserved.
The information contained in these materials is provided for
informational purposes only, and is provided AS IS without
warranty of any kind, express or implied. IBM shall not be
responsible for any damages arising out of the use of, or
otherwise related to, these materials. Nothing contained in
these materials is intended to, nor shall have the effect
of, creating any warranties or representations from IBM or its
suppliers or licensors, or altering the terms and conditions of
the applicable license agreement governing the use of IBM
software. References in these materials to IBM
products, programs, or services do not imply that they will be
available in all countries in which IBM operates. Product
release dates and/or capabilities referenced in these
materials may change at any time at IBM’s sole discretion
based on market opportunities or other factors, and are not
intended to be a commitment to future product or feature
availability in any way. IBM, the IBM logo, Rational, and other
IBM products and services are trademarks of the International
Business Machines Corporation, in the United States, other
countries or both. Other company, product, or service names
may be trademarks or service marks of others.
37


Kerim Cakmak, Moshe Cohen -- Continuous Verification and Validation

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (12)

Similaire à Kerim Cakmak, Moshe Cohen -- Continuous Verification and Validation

Similaire à Kerim Cakmak, Moshe Cohen -- Continuous Verification and Validation (20)

Plus de Anatoly Levenchuk

Plus de Anatoly Levenchuk (20)

Dernier

Dernier (20)

Kerim Cakmak, Moshe Cohen -- Continuous Verification and Validation

Notes de l'éditeur