2. Course contents
The course consists of two parts:
• Requirements – a description of what we
shall develop: 11 lectures
• Tests – a description of how we will
check that we have developed what the
customer required: 19 lectures
The curriculum consists of
• A set of slides
• Copies of journal articles
3. Course grade
The course grade will be decided based on
• One group exercise on writing
requirements – 10%
• One group exercise on writing a test
strategy for a set of requirements – 10%
• One group exercise on testing a software
package – 30%
• Exam – 50%
4. Lecture plan – 1
Tuesdays 14:15 – 16:00 in F3
Fridays 10:15 – 11:00 in F3
• First lecture – Tuesday, January 11, 2011.
• Last lecture – Friday, April 15, 2011.
• Exercise 1 – week 4.
• Exercise 2 – week 7.
• Exercise 3 – weeks 13 and 14.
There will be no lectures during the exercise
periods.
5. Lecture plan – 2
Week Course content
What you will learn
Presentation of lecture plan and exercises
2 Goal oriented requirements specification
Quality issues in requirements
Requirements elicitation with boilerplates (templates)
On requirement – testability and traceability – 1
3 On requirement – testability and traceability – 2
Intro. to exercise 1
4 Exercise 1 – Write a requirements specification
Testing vs. inspection – strong and weak sides
5 Testing vs. inspection – strong and weak sides
Testing and cost / benefit analysis
Testing strategies – a general introduction
Black box vs. White box
6
Grey box testing
Introduction to exercise 2
7 Exercise 2 – Write a testing strategy for the requirements specification from exercise 1
6. How to prioritize requirements
8 Requirements handling in COTS, Outsourcing, sub-contracting etc.
Requirements in embedded and information systems
Aspects, cross-cutting concerns and non-functional requirements
9 Testing COTS, Outsourcing, sub-contracting etc.
Presentation of the FITNESS tool
Domain testing
10 Coverage measures
Use of test coverage measures for code and requirements
Requirements through user stories and scenarios
11 How to write good user stories and scenarios
Advanced use cases
Test driven development – TDD 1
12 Test driven development – TDD 2
Introduction to exercise 3
13 Exercise 3
14 Exercise 3
Regression testing
15 Non-functional requirements – safety, user friendliness etc.
Testing non-functional requirements
7. Main theme
The main message of this course is that
requirements and tests are two sides of
the same coin.
• Requirements without tests will be
ignored.
• Tests without requirements are
meaningless.
8. The need for traceability
It must be possible to trace
– From test to requirement. Why do we need
this test?
– From requirement to test.
• Where is this requirement tested?
• Staus: How many of the tests for this requirement
has been executed successfully?
9. Challenges in Requirements Engineering
What is a requirement?
•What a system must do (functional):
System requirements
•How well the system will perform its functions (non-functional):
Institutt for datateknikk og System quality attributes
informasjonsvitenskap The RE process:
Inah Omoronyia and Tor Stålhane
Requirements Specification and Testing
An introduction defined satisfy
Ultimately: operational
business
needs
capabilities
TDT 4242 TDT 4242 TDT 4242
Challenges in Requirements Engineering Challenges in Requirements Engineering
Importance of getting requirements right:
1/3 budget to correct errors originate from requirements
Source: Benoy R Nair (IBS software services)
TDT 4242 TDT 4242
10. Challenges in Requirements Engineering Challenges in Requirements Engineering
Factors that make a software project challenging: Why projects are cancelled:
Source: Benoy R Nair (IBS software services) Source: Benoy R Nair (IBS software services)
TDT 4242 TDT 4242
Requirements Development - 1 Requirements Development – 2
Requirements Elicitation: Effects of Inadequate Requirements development – Ariane 5:
The process of discovering the requirements for a system by (An expendable launch system used to deliver payloads into
communication with customers, system users and others who have a geostationary transfer orbit or low Earth orbit)
stake in the system development.
Ariane 5 succeeded Ariane 4. Wrong implicit assumptions about the
Requirements gathering techniques parameters, in particular the horizontal velocity that were safe for
Ariane 4 but not Ariane 5.
• Methodical extraction of concrete requirements from high • horizontal velocity exceeded the maximum value for a 16 bit unsigned integer
level goals when it was converted from it's signed 64 bit representation.
• Requirements quality metrics • Ariane 5: component (requirements) should have been designed for reuse –
but the context of reuse was not specified.
Cost of poor requirements in Ariane 5
• Data overflow on launch
• Self-destruction of the complete system
• Loss > 500 Million EUR
TDT 4242 TDT 4242
11. Requirements Development – 3 Problem world and machine solution
Effects of Inadequate Requirements development – Airbus: The problem to be solved is rooted in a complex organizational, technical or physical
world.
Requirement: Reverse thrust may only be used, when the airplane is • The aim of a software project is to improve the world by building some machine
landed. expected to solve the problem.
• Problem world and machine solution each have their own phenomena while sharing
Translation: Reverse thrust may only be used while the wheels are others.
rotating. • The shared phenomena defines the interface through which the machine interacts
with the world.
Implementation: Reverse thrust may only be used while the wheels
are rotating fast enough.
Situation: Rainstorm – aquaplaning
Result: Crash due to overshooting the runway!
Problem: erroneous modeling in the requirement phase E-commerce world
Requirements engineering is concerned with the machine’s effect on the
surrounding world and the assumption we make about that world.
TDT 4242 TDT 4242
Formulation of requirements statements Two types of requirements statements
• Descriptive statements: state properties about the system
that holds regardless of how the system behaves. E.g. If train
doors are open, they are not closed.
• Prescriptive statements: States desirable properties
about the system that may hold or not depending on how the
system behaves
• Need to be enforced by system components
Statement scope: • E.g. Train doors shall always remain closed when the
• Phenomenon of train physically moving is owned by environment. It train is moving
cannot be directly observed by software phenomenon
• The phenomenon of train measured speed being non-null is shared by
software and environment. It is measured by a speedometer in the
environment and observed by the software.
TDT 4242 TDT 4242
12. Formulation of system requirement Formulation of software requirement
A prescriptive statement enforced by the software-to-be.
A prescriptive statement enforced solely by the software-to-
• Possibly in cooperation with other system components be. Formulated in terms of phenomena shared between the
• Formulated in terms of environment phenomena software and environment.
Example: The software “understand” or “sense” the environment
All train doors shall always remain closed while the train is through input data
moving
Example:
In addition to the software-to-be we also requires the The doorState output variable shall always have the value
cooperation of other components: ‘closed’ when the measuredSpeed input variable has a non-
• Train controller being responsible for the safe control null value
of doors.
• The passenger refraining from opening doors
unsafely
• Door actuators working properly
TDT 4242 TDT 4242
Domain properties Goal orientation in requirements engineering – 1
A domain property:
• Is a descriptive statement about the problem world A goal is an objective that the system under
• Should hold invariably regardless of how the system consideration shall achieve.
behaves
– Ranges from high-level strategic to low-level
• Usually corresponds to some physical laws
technical concerns over a system
– System consist of both the software and its
Example:
environment. Interaction between active
A train is moving if and only if its physical speed is non-null.
components, i.e. devices, humans, software etc
also called Agents
TDT 4242 TDT 4242
13. Goal orientation in requirements engineering – 2 Goal statement typology
Goals can be stated at different levels of granularity:
– High-level goal: A goal that requires the cooperation of
many agents. They are normally stating strategic
objective related to the business, e.g.
The system’s transportation capacity shall be increased
by 50%
– Requirement: A goal under the responsibility of a
single agent in the software-to-be.
– Assumption (expectation): A goal under the
responsibility of a single agent in the environment of
the software-to-be. Assumptions cannot be enforced by
the software-to-be
TDT 4242 TDT 4242
Goal types Behavioral goal specialization
TDT 4242
TDT 4242
14. Goal categorization – 1 Goal categorization – 2
Goal categories are similar to requirements categories: Functional goal: States the intent underpinning a system
service
• Satisfaction: Functional goals concerned with
satisfying agent request
• Information: Functional goals concerned with keeping
agents informed about important system states
• Stimulus-response: Functional goals concerned with
providing appropriate response to specific event
Example: The on-board controller shall update the train’s
acceleration to the commanded one immediately on
receipt of an acceleration command from the station
computer
TDT 4242 TDT 4242
Goal categorization – 3 Goal refinement
A mechanism for structuring complex specifications at different levels of
Non-functional goal: States a quality or constraint on concern.
service provision or development. A goal can be refined in a set of sub-goals that jointly contribute to it.
Accuracy goal: Non-functional goals requiring the state of Each sub-goal is refined into finer-grained goals until we reach a
requirement on the software and expectation (assumption) on the
variables controlled by the software to reflect the state of
environment.
corresponding quantities controlled by environment agent
NB: Requirements on software are associated with a single agent and
E.g: The train’s physical speed and commanded speed they are testable
may never differ by more than X miles per hour
Soft goals are different from non-functional goals. Soft goals
are goals with no clear-cut criteria to determine their
satisfaction.
E.g: The ATM interface should be more user friendly
TDT 4242 TDT 4242
15. Goal refinement tree – 1
Goal refinement: Example
Refinement links are two way links: One showing goal decomposition, the
other showing goal contribution
TDT 4242
TDT 4242
Goal refinement tree – 2
Goal feature annotation
Requirements quality metrics – 1
Qualitative Goal‐Requirements tracing:
An approach to requirements
refinement/abstraction that makes it
less likely to generate trace links that
are ambiguous, inconsistent, opaque,
noisy, incomplete or with forward
referencing items
TDT 4242 TDT 4242
16. Requirements quality metrics – 2 Requirements quality metrics – 3
Forward Referencing: Requirement items that make
use of problem world domain features that are not yet
defined.
E, C and D need to be mapped to a requirement item
TDT 4242 TDT 4242
Requirements quality metrics – 4 Requirements quality metrics – 5
Opacity: Requirement items for which rational or
dependencies are invisible.
Noise: Requirement items that yield no information on
Multiple unrelated concept mapping. A is not related to B problem world features. X refers to a concept
undefined in the domain
TDT 4242 TDT 4242
17. Requirements quality metrics – 6 Requirements quality metrics
Completeness: The needs of a prescribed Quality metrics on a requirements set provides useful
system are fully covered by requirement understanding, tracking and control of requirements
items without any undesirable outcome. improvement process.
No requirement item mentions the goal
concept Z
TDT 4242
Where do the goals come from? Summary
We get goals from:
Goals can be defined at different levels of
• Preliminary analysis of the current system. abstraction
• Systematically by searching intentional keywords
There are two types of goals: Behavioral or soft goal
in documents provided, interview transcripts etc.
E.g. ‘objective’, ‘purpose’, ‘in order to’.
• Iterative refinement and abstraction of high‐level There are several categories of goals, e.g.
goals: By asking the how and why question. • Functional and non-functional
Results in a goal refinement tree • Goal refinement provides a natural mechanism
• Approaches: KAOS – Goal driven requirements for structuring complex specifications at
acquisition. different levels of concern:
• Goal refinement graph
TDT 4242 TDT 4242
18. Requirements
There are three levels of requirements:
• Informal – e.g. Natural language (NL): free text,
no rules apply
Institutt for datateknikk og
informasjonsvitenskap • Semiformal
• Guided Natural Language (GNL): free text but
Inah Omoronyia and Tor Stålhane allowable terms are defined by a vocabulare
• Boilerplates (BP): structured text and an
ontology – vocabulary plus relationships
Guided Natural Language between terms
and
• Formal: e.g. state diagrams or predicate logic
Requirement Boilerplates
TDT 4242 TDT 4242 TDT 4242
Requirements elicitation Humans and machines – 1
Step 1: Step 2: Step 3: Step 4: Given the amount and complexity of RE, we
Capture Transfer Requirements and Refine the requirements model Create a
Requirements in functions into a semi-formal and derive detailed requirements preliminary need to automate as much as possible.
Natural Language requirements model design model
based on the
requirement
model (to be
Humans and machines have different strong
Req.012: The system
shall enable cabin Function 1 Function 2 Function 1 used and and weak points.
temperature regulation refined in
between 15°C and 30°C
…
Req 001
Req 002
Req 011
Req 028 Function 1a Req 001.01
SP3) We want to elicit and analyze requirements in a
Req 012 Req 050
…
Req.124: Cabin
… … Function 1b
Req 001.02
….
way that allows both parties to build on their
Req 124 …
temperature shall not
exceed 35° Function 1c strong sides.
Parallel Steps:
Apply dictionary with common vocabulary; validate and check Requirements consistency and completeness
19. Humans and machines - 2 Why BPs and GNL – 1
Machines are GNL and BPs will reduce variation and thus giving
good at observing quantitative data and being
the machines the opportunity to do what they are
deductive, fast and precise. In addition, they are best at: to be fast, precise and consistent.
good at doing consistent repetition of several
By combining humans and machines and let both do
actions.
what they are best at, we get a better result than we
bad at handling variations in written material and
would get if we left the job of handling requirements
pattern recognition. to just one of them.
Humans are good at handling variations in written
material, being inductive. In addition, they are good
at doing error correction.
Why BPs and GNL - 2 GNL and BPs
Template based textual = Syntax + Semantics + Meta Model
The final goal is to allow the machine to assist the Keywords: RMM
- Refinement
developers in analysing requirements for: Guided RSL Boilerplates Reflects requirement,
system and domain Analysis - Specialization
concepts -Correctness
Consistency -Completeness
Requirements expressed on templates -Consistency
Completeness Uses predefined templates based on concepts,
relations and axioms to guide requirements elicitation
-Safety analysis
Safety implications Example:
The <system function> shall provide <system capability> to achieve <goal>
Requirements expressed using a vocabulary guide
Uses predefined concepts, relations and axioms to
guide requirements elicitation
Example:
The ACC system shall be able to determine the speed of the ego-vehicle.
Ontology: General and SP specific
- Requirements classification
- System attributes
- Domain concepts
20. What is GNL - 1 What is GNL - 2
Aim:
Free text requirement elicitation with the
• Bridge the gap between unconstrained
assistance of prescribed words from a expression and quality checking when
dictionary. This will give us requirements representing requirements as free text. Quality
which use all terms in a uniform way, this measures:
reducing misunderstandings Correctness, consistency, completeness and un‐
No formal constraints ambiguity (reduced variability)
Requires minimal expertise. • Provide the basis for semantic processing and
checking of requirements.
• Dictionary – Simple taxonomy or more formal
ontology
Approach for GNL – 1 Approach for GNL – 2
Ontology = Thesaurus + Inference Rules Required Activity
• Thesaurus – Domain concepts: entities, Knowledge capture: Information embedded in
domain events from domain experts and ontologist
terms and events Implementation: Formal representation of captured
• Inference Rules – Relations, attributes and knowledge. Language: OWL, Support environment:
axioms Protégé.
Verification: Checking that represented ontology is
• Causality, similarity, reflexivity, correct using
transitiveness, symmetric, disjoint • Classifiers/reasoners
• Domain experts (semantic accuracy)
(contradiction) …
• Mapping of requirement segments to ontology concepts
21. Motivation for use of templates - 1 Motivation for use of templates - 1
Text has the advantage of unconstrained Template based textual requirements
expression. There is, however, a need for specification (boilerplates) will introduce some
common limitations when representing requirements
but will also reduce the opportunity to
• Understanding of concepts used to express
introduce ambiguities and inconsistencies.
the requirements and relations between them.
Boilerplates
• Format of presentation
• Provides an initial basis for requirements
Lack of common understanding makes
checking
requirement specifications expressed as free text
prone to ambiguous representations and • Are easy to understand for stakeholders
inconsistencies. compared to more formal representations
TDT 4242 TDT 4242
What is a boilerplate – 1 What is a boilerplate – 2
Boilerplates is a set of structures that can be used to The RE process is as follows:
write requirements. They use high-level concept
classification and attributes
1. Select a boilerplate or a sequence of
boilerplates. The selection is based on the
attributes that need to be included and how
they are organized – fixed terms.
2. If needed, identify and include mode boilerplates
3. Instantiate all attributes
A boilerplate consists of fixed terms and attributes.
It may, or may not, contain one or more modes.
TDT 4242
TDT 4242
22. Fixed Terms
Boilerplate examples - 1
Attributes
BP32
The <user> shall be able to <capability>
Attributes:
• <user> = driver
• <capability> = start the ACC system
Requirement
The driver shall be able to start the ACC system
TDT 4242
23. Boilerplate examples - 2 Boilerplate examples - 3
BP43 While <operational condition>
BP2
BP32 The <user> shall be able to <capability>
The <system> shall be able to <action> <entity>
BP43 is a mode
Attributes:
•<system> = ACC system Attributes
•<action> = determine • <operational condition> = activated
•<entity> = the speed of the ego-vehicle • <user> = driver
• <capability> = override engine power control of
Requirement the ACC system
The ACC system shall be able to determine the Requirement
speed of the ego-vehicle
While activated the driver shall be able to override
engine power control of the ACC-system
TDT 4242 TDT 4242
Functional requirements example Non functional requirement example – 1
Functional requirements from the SafeLoc system
The robot control system shall stop the robot within 10 Non‐functional requirements and soft goals fits into
milliseconds if a gate is opened to the zone where the robot is the same BPs as functional requirements
operating
The robot shall only be allowed to start when all gates are BP61
closed and the reset button is pushed The <system> shall be able to <action> to <entity>
The robot shall stop if it tries to move into a zone already
occupied by an operator Suitability:
The <system > shall be able to <provide an
appropriate set of functions> to <the user>
TDT 4242 TDT 4242
24. Non functional requirement example – 2 Non functional requirement example – 3
Non‐functional requirements and soft goals fits into
the same BPs as functional requirements BP43
While <operational condition>
BP2-1 The <system> shall be able to <capability> BP2
BP12 …for a sustained period of at least The <system> shall be able to <action> <entity>
<number> < unit>
While <normal operational condition> the
Maturity: <system> shall be able to <tolerate>
The <system > shall be able to <operate without <90% of software faults of category...>
failure> for a sustained period of at least <quantity>
<time unit>
TDT 4242
Summing up
The use of boiler plates and ontologies will
• Enforce a uniform use of terms
• Reduce the variability of presentations –
requirements that are similar will look similar
Reduced variation in form and contents simplifies
the use of automatic and semi-automatic tools
for
• Checking requirement quality – e.g
completeness and consistency
• Creating test cases
TDT 4242
25. What is requirements traceability
“Requirements traceability refers to the ability
to describe and follow the life of a
Institutt for datateknikk og requirement, in both a forwards and
informasjonsvitenskap backwards direction, i.e. from its origins,
Inah Omoronyia and Tor Stålhane through its development and specification, to
its subsequent deployment and use, and
Requirements Traceability through periods of on‐going refinement and
iteration in any of these phases.”
Gotel and Finkelstein
TDT 4242 TDT 4242 TDT 4242
Traceability Goals - 1 Traceability Goals – 2
• Project Management • Validation
– Status: “When will we finish?” and “what will it cost?” – finding and removing conflicts between requirements
– Quality: “How close are we to our requirements?” – completeness of requirements
• derived requirements cover higher level requirements
• QA manager • each requirement is covered by part of the product
– Improve Quality: “What can we do better?” • Verification
• Change Management – assure that all requirements are fulfilled
– Versioning, documentation of changes (Why? What? • System Inspection
When?)
– identify alternatives and compromises
– Change Impact analysis
• Certification/Audits
• Reuse
– proof of being compliant to standards
– Variants and product families
– Requirements can be targeted for reuse
TDT 4242 TDT 4242
26. Habitat of Traceability Links – 1 Habitat of Traceability Links – 2
Pre- vs. Post-Requirements Specification
TDT 4242 TDT 4242
Challenges of traceability – 1 Challenges of traceability – 2
– Traces have to be identified and recorded among
numerous, heterogeneous entity instances – A variety of tool support
(document, models, code, . . . ). It is challenging • based on traceability matrix, hyperlink, tags
to create meaningful relationships in such a and identifiers.
complex context. • still manual with little automation
– Traces are in a constant state of flux since they – Incomplete trace information is a reality due to
may change whenever requirements or other complex trace acquisition and maintenance.
development artefacts change.
– Trust is a big issue: lack of quality attribute
• There is no use of the information that 70% of
trace links are accurate without knowing which
of the links forms the 70%
TDT 4242 TDT 4242
27. Challenges of traceability – 3 Challenges of traceability – 4
Different stakeholders usage viewpoint (different questions Different stakeholders usage viewpoint (different questions
asked by different stakeholders): asked by different stakeholders):
• Validation:
• QA management: – Tracability can be used as a pointer to the quality of
– “how close are we to our requirements” and “what can we requirements:
do better” to improve quality. • Completeness, ambiguity, correctness/noise,
• Change management inconsistency, forward referencing, opacity
– Tracking down the effect of each change to each involved – Ensures that every requirement has been targeted by at
component that might require adaptations to the change, least one part of the product
recertification or just retesting to proof functionality. • Verification
• Reuse: – Checking that constraints are not violated (in most cases
– Pointing out those aspects of a reused component that this is an extension of validation context)
need to be adapted to the new system requirements.
• Certification/Audit
– Even the requirements themselves can be targeted for
reuse. • Testing, maintenance (reverse engineering)
TDT 4242 TDT 4242
Traceability meta-models – 1 Traceability meta-models – 2
• A model is an abstraction of phenomena in the real world; a
meta model is yet another abstraction, highlighting properties
of the model itself.
• Meta‐models for traceability are often used as the basis for
the traceability methodologies and frameworks:
– Define what type of artefacts should be traced.
– Define what type of relations could be established between these
artefacts.
Traceability Meta Model
Low-end traceability
TDT 4242
TDT 4242
28. Traceability meta-models – 3
European EMPRESS project: Meta model for
requirements traceability
High-end traceability
TDT 4242
Traceability meta-models – 4 Approaches to traceability
Creating trace links:
– Critical tasks in requirements traceability:
establish links between requirements and
between requirements and other artefacts.
– Manual linking and maintaining of such
links is time consuming and error prone
– Focus is on requirements traceability
through (semi‐)automatic link generation.
PRECISE Meta‐model (SINTEF)
29. Manual trace links – 1 Manual trace links – 2
This is the classical traceability methods and
the simplest form of traceability. In this approach,
we create a requirements traceability matrices
using a hypertext or table cross referencing
scheme, often using Excel
Two problems
• Long-term difficulty of maintaining a large
numbers of links.
• The static nature of the links (lack of
attributes) limit the scope of potential
automation.
Scenario driven traceability – 1 Scenario driven traceability – 2
• Test‐based approach to uncover relations amongst The method to achieve traceablity uses the idea of
requirements, design and code artifacts (Alexander “footprint”.
Egyed ) When we are dealing with traceability, a footprint
• Accomplished by observing the runtime behavior of contains two types of information:
test scenarios. • The set of classes that were executed when we were
• IBM Rational PureCoverage, open source tool testing a specific scenario.
(org.jmonde.debug.Trace) • The number of methods that were executed in each
• Translate this behavior into a graph structure to class.
indicate commonalities among entities
associated with the behavior
30. Footprints – 1 Footprints – 2
E.g. scenario A uses 10 methods in class CAboutDlg Only classes are registered – e.g scenario [s3] uses
and 3 methods in Csettings Dlg classes C, J, R and U
Footprints – 3 Footprints – 4
Some problems: Based on the footprint table, we can make a
requirements-to-class trace table
• There might be scenarios that do not cover any
requirement – e.g. [s3]
• There are scenarios that belong to several
requirements, e.g. [s9]
Such scenarios will get separate rows in the trace
matrix and will be marked with an F (Fixed) or a P
(Probable), depending on how sure we are that a
certain class belongs to this scenario.
31. Footprints – 5 Development footprints - 1
Each test scenario will leave a footprint. If we make A solution that enables the project to construct
one test scenario per requirement, then we get one traceability information during development has
footprint per requirement. been suggested by I. Omoronyia et al.
The method requires that each developer
We can make the footprints more fine grained and • Always identifies which requirement – e.g. use case
thus get more information by using methods or – he is currently working on
code chunks instead for classes. • Only works at one use case at a time
This will require more work but also more – better –
traceability information.
Development footprints - 2 Development footprints - 3
The result will be similar to the scenario testing We can extract more info from the development
footprint table. process in order to understand better what has
The resulting table will show which documents, been going on in the project. The next slides shows
classes etc. have been accessed during work on • Types of access: C – Create, U – Update and V –
this particular requirement – e.g. use case. View
Main problem: “false” accesses – e.g. a developer • Timeline – e.g. date or time
looks at some of the code of another requirement • Person – who did what and, more important, who
for info. will have expertise on what?
Each line in the table will show
32. Development footprints - 4 Scenario driven traceability – 3
Problems:
– Semi‐automated but requires a large amount of time
from system engineers to iteratively identify a subset
of test scenarios and how they related to
requirement artifacts.
– Requirements that are not related due to non
matching execution paths might be related in some
other form (e.g calling, data dependency,
implementation pattern similarity, etc).
Trace by tagging – 1 Trace by tagging – 2
This method is easy to understand and simple to
implement. The problem is that it depends on
heavy human intervention.
The principle is as follows:
• Each requirement is given a tag, either manually or
by the tool.
• Each document, code chunk, etc. are marked with
a tag which tells which requirement it belongs to
33. Trace by tagging – 3 Trace by tagging – 4
There are several ways to create tags, e.g.: The quality of traceability through tagging will depend
• Single level tags – e.g. R4. This gives a standard on that we remember to tag all relevant documents.
trace matrix It is possible to check automatically that all
• Multilevel tags – e.g. R4, R4.1 and R4.2 where R4 documents in the project database is tagged.
is the top level requirement and R4.1 and R4.2 are It is, however, not possible to check that this tagging
sub-requirements. This gives us more detailed is correct.
trace information
Conclusion
• Requirements traceability is an important aspect of
requirements management
• Stakeholders have different traceability information needs
• Traceability can be complex for not trivial projects
• Traceability meta-models provide insight on the type of
traceability information required for a project
• There exist several automated approaches for
requirements traceability. The strength is in a synergy of
different automated approaches
35. Testability definition
According to ISO 9126, testability is defined
as:
Testability: The capability of the software
product to enable modified software to be
validated.
NOTE - Values of this sub-characteristic
may be altered by the modifications under
consideration.
37. Testability
Three basic ways to check that we have achieved our
goals:
• Executing a test. Give input, observe and check
output. A test can be a
– Black box test
– White box test
– Grey box test
• Run experiments
• Inspect the code and other artifacts
Usually, we will include all of these activities in the
term testing
38. When to use what
The diagram on the next slide is a high level
overview of when to use
• T – Tests. Input / output. Involves the
computer system and peripherals.
• E – Experiments. Input / output but
involves also the users.
• I – Inspections. Evaluation based on
documents.
40. Testability
In order to be testable, a requirement needs
to be stated in a precise way. For some
requirements this is in place right from the
start:
When the ACC system is turned on, the
“Active” light on the dashboard shall be
turned on.
In other cases we need to change a
requirement to get a testable version.
The system shall be easy to use.
41. Testability challenges - 1
Some requirements are more difficult to test
than others. Problems might rise due to:
• Volume of tests needed, e.g. response
time or storage capacity.
• Type of event to be tested, e.g. error
handling or safety mechanisms.
• The required state of the system before
testing, e.g. a rare failure state or a certain
transaction history.
42. Testability challenges – 2
We can test the requirements at any level. The
formulation of the test will depend on the level
43. Making a requirement testable – 1
One way to make requirements testable is
the “Design by Objective” method
introduced by Tom Gilb.
The method is simple in principle but is in
some cases difficult to use. There are two
problems
• The resulting tests can in some be rather
extensive and thus quite costly.
• The method requires access to the
system’s end users.
44. Making a requirement testable – 2
1. What do you mean by <requirement>?
This will give us either (a) a testable
requirement or (b) a set of testable and
non-testable sub-requirements.
2. In case (a) we are finished. In case (b) we
will repeat question 1 for each non-
testable sub-requirement
45. Making a requirement testable – 3
Requirement: Reverse thrust may only be
used, when the airplane is landed.
The important questions are
• “How do you define landed?”
• Who should you ask – e.g. pilots, airplane
construction engineers, or airplane
designers?
46. Requirements for testability – 1
First and foremost:
The customer needs to know what he wants
and why he wants it. In some cases it is
easier to test if the user actually has
achieved his goal than to test that the
system implements the requirement.
Unfortunately, the “why”-part is usually not
stated as part of a requirement.
47. Requirements for testability – 2
Each requirement needs to be
• Correct, i.e. without errors
• Complete, i.e. has all possible situations
been covered?
• Consistent, i.e. not in disagreement with
other requirements.
• Clear, i.e. stated in a way that is easy to
read and understand – e.g. using a
commonly known notation.
48. Requirements for testability – 3
Each requirement needs to be
• Relevant, i.e. pertinent to the system’s
purpose and at the right level of
restrictiveness.
• Feasible, i.e. possible to realize. If it is
difficult to implement, is might also be
difficult to test.
• Traceable, i.e. it must be possible to relate
it to one or more
– Software components
– Process steps
49. Completeness
All possible situations must be covered.
“If X then….”, “If Y then….” Must also
consider what will happen “If neither X nor
Y…”
Automatic door opener – what is missing?
If the door is closed and a person is
detected then send signal Open_Door. If
no person is detected after 10 sec., send
signal Close_Door.
50. Consistency
Consistency is a challenge since we, at least
in the general case, need a complete
overview of all requirements.
In most cases, we can make do with
checking all requirements that are related
to the same event, function or parameter.
51. Clear – 1
This is mainly a question of representation
such as choice of
• Diagram notation
• Description language
• Level of details
Who shall understand the requirement?
• Customers
• Developers, including hired-in consultants
• Testers
52. Clear – 2
Simple example:
Print the “accounts ledger” for all accounts
This requirement is perfectly clear for
developers who are working in the banking
business.
Other developers might experience some
problems.
53. Relevant
Two questions are important:
• Do we really need this requirement?
• Is it at the right level of strictness – i.e. not
too strict and not too lax.
Only the second question is important for
the tester.
• Too strict means more work for the
developers
• Too lax means more work for the tester.
54. Feasible
This question is really related to the contract but
we should also consider it here – can we really
do this?
Testers can contribute to the feasibility question by
asking how it should be tested. This will help /
force everybody involved to make the
requirement more clear and thus improve on the
requirement.
Requirements that are difficult to tests are also
usually difficult to implement – mainly because
they are badly defined.
56. Modifying Phrases
Words and phrases that include:
• as appropriate
• if practical
• as required
• to the extent necessary / practical.
Their meaning
• is subject to interpretation
• make the requirement optional
Phrases like "at a minimum" only ensure the minimum,
while "shall be considered" only requires the
contractor to think about it.
57. Vague Words
Vague words inject confusion. Examples of frequently
used vague verbs are:
• manage
• track
• handle
• flag
Information systems receive, store, calculate, report, and
transmit data. Use words that express what the system
must do.
Requirement: The system shall process ABC data to the
extent necessary to store it in an appropriate form for
future access.
Correction: The system shall edit ABC data.
60. Negative Requirements
Everything outside the system is what the system
does not do.
Testing would have to continue forever to prove
that the system does not do something.
State what the system does. Substitute an active
verb that expresses what the system must do.
• Change "the system shall not allow X," to "the
system shall prevent Y."
• Use the prefix "un," such as: The system shall
reject unauthorized users.
62. Assumptions and Comparisons – 2
An example, sometimes found in requests for
proposals, is:
"The system shall address the future needs of
users."
The writer is probably thinking ahead to after the
contract is awarded. The requirement is
meaningless because whenever it is read, it will
point to the future.
A requirement on change management included in
the project management processes, would make
more sense than making it a requirement for the
system.
63. Indefinite Pronouns
Indefinite pronouns are “stand in” for unnamed people or
things, which makes their meaning subject to interpretation.
Some of these may find their way into requirements:
• All • Everybody
• Another • Everyone
• Any • Everything
• Anybody • Few
• Anything • Many
• Each • Most
• Either • Much
• Every
65. Autonomy – 1
How many other systems are needed to test this
requirement?
It is best if it can be tested using only the SUT
and the autonomy and testability is
successively reduced as the number of other,
necessary systems increase.
If the other systems needed are difficult to
include at the present we will have to write
more or less complex stubs.
66. Example – 1
Requirement: “If the door is closed and a
person is detected then send signal
Open_Door”
• Sensors and actuators can be tested in
the lab.
• The system with a simulated actuator:
simulate a “person detected” signal on the
sensor and check if a Open_Door signal is
sent to the actuator.
68. Observability
How easy is it to observe the
• Progress of the test execution?
This is important for tests that do not produce
output – e.g. the requirement is only
concerned with an internal state change or
update of a database.
• Results of the test?
Important for tests where the output is
dependent on an internal state or database
content.
71. Final comments
That a requirement is testable does not
necessarily mean that it is easy to test.
In order to have testable requirements it is
important that
• The testers are involved right from the
start of the project. It is difficult to add
testability later.
• The tests are an integrated part of the
requirement
73. The goals ‐ 1
Consider the following goals for an adaptive
cruse controller – ACC:
When the ACC is active, vehicle speed shall be
controlled automatically to maintain one of
the following:
• time gap to the forward vehicle
• the set speed
74. The goals ‐ 2
In case of several preceding vehicles, the
preceding vehicle will be selected
automatically
The ACC shall support a range of types
• Type 1a: manual clutch operation
• Type 1b: no manual clutch operation, no
active brake control
• Type 2a: manual clutch operation, active
brake control
• Type 2b: active brake control
78. What we will cover
• Part 1
– Introduction
– Inspection processes
– Testing processes
• Part 2
– Tests and inspections – some data
– Inspection as a social process – two experiments
and some conclusions
81. Adams’ data – 2
The main information that you get from the
table on the previous slide is that
• Some defects are important because they will
happen quite often.
• Most defects are not important since they will
happen seldom.
How can we tell the difference?
83. Testing and inspection – 1
The important message here is that testing
cannot always be done.
In the first, important phases, we have nothing
to execute and will thus always have to do
some type of inspection.
This might be considered one of the weaknesses
of traditional software engineering over Agile
development.
85. Area of Man Machine
competence
Understanding Good at handling variations in Bad at handling variations in
written material written material
Observe General observations, Specialized, good at observing
multifunctional quantitative data, bad at
pattern recognition
Reasoning Inductive, slow, imprecise but Deductive, fast, precise but
good at error correction bad error correction
Memory Innovative, several access Copying, formal access
mechanisms
Information Single channel, less than 10 Multi channel, several
handling bits per second Megabits per second
Consistency Unreliable, get tired, depends Consistent repetition of several
on learning actions
Power Low level, maximum ca. 150 High level over long periods
watt of time
Speed Slow – seconds Fast
87. Man vs. machine – 2
In order to do the best job possible we need
processes where we let each part
• Do what they are best at:
– Man is innovative
– Machine handles large amounts of data
• Support the other with their specialties.
– Machine supports man by making large amounts
of information available
– Man support machine by providing it with
innovative input
88. General considerations ‐ documents
Architecture, system, sub‐system and
component design plus pseudo code. Here we
can only use inspections.
Man will use experience and knowledge to
identify possible problems
Machine can support by identifying information
– e.g. find all occurrences of a string.
89. General considerations – code (1)
For executable code, we can use inspection,
testing or a combination of both.
The size and complexity – degree of dynamism –
of the code will, to a large degree, decide our
choice.
Other important factors are the degree of
experience with
• The programming language
• The algorithms used
92. Inspections – 1
The term “inspection” is often used in a rather
imprecise manner. We will look at three types
of inspection:
• Walkthrough
• Informal inspection – also called informal
review
• Formal inspection – also called formal review
or just inspection
The first two types are usually project internal
while the last one is used as a final acceptance
activity for a document.
94. The walkthrough process
Walkthrough is a simple process – mostly used
for early decisions for an activity. The
document owner:
1. Makes a rough sketch of the solution –
architecture, algorithm etc.
2. Presents – explain – the sketch to whoever
shows up.
3. Registers feedback – improvements.
107. Improve the product ‐ 2
• Errors in own document:
Make appropriate corrections
• Errors in someone else’s documents:
Inform the owner of this document.
• Misunderstandings in the inspection team:
Improve document to avoid further
misunderstandings.
111. Testing
We will look at three types of testing:
• Unit testing – does the code behave as
intended. Usually done by the developer
• Function verification testing – also called
systems test. Does the component or system
provide the required functionality?
• System verification testing – also called
acceptance test. Does the hardware and
software work together to give the user the
intended functionality?
114. The system test process
A systems test has the following steps:
1. Based on the requirements, identify
– Test for each requirement, including error handling
– Initial state, expected result and final state
2. Identify dependencies between tests
3. Identify acceptance criteria for test suite
4. Run tests and check results against
– Acceptance criteria for each test
– Acceptance criteria for the test suite
120. Test and inspections – some terms
First we need to understand two important
terms – defect types and triggers.
After this we will look at inspection data and
test data from three activity types, organized
according to type of defect and trigger.
We need the defect categories to compare test
and inspections – where is what best?
123. Inspection triggers
• Design conformance
• Understanding details
– Operation and semantics
– Side effects
– Concurrency
• Backward compatibility – earlier versions of this
system
• Lateral compatibility – other, similar systems
• Rare situations
• Document consistency and completeness
• Language dependencies
124. Test triggers – black box
• Test coverage
• Sequencing – two code chunks in sequence
• Interaction – two code chunks in parallel
• Data variation – variations over a simple test
case
• Side effects – unanticipated effects of a simple
test case
125. Test triggers – white box
• Simple path coverage
• Combinational path coverage – same path
covered several times but with different
inputs
• Side effect ‐ unanticipated effects of a simple
path coverage
128. Test data
We will look at test data from three
development activities:
• Unit testing: testing a small unit like a method
or a class
• Function verification testing: functional
testing of a component, a system or a
subsystem
• System verification testing: testing the total
system, including hardware and users.
This is the right hand side of the V‐model
132. Testing – triggers and defects
Activity Trigger Percentage
Test sequencing 41.90
Implementation
Test coverage 33.20
testing
Side effects 11.07
Activity Defect type Percentage
Interface 39.13
Implementation Assignments 17.79
testing Build / Package /
14.62
Merge
133. Some observations – 1
• Pareto’s rule will apply in most cases – both
for defect types and triggers
• Defects related to documentation and
functions taken together are the most
commonly found defect types in inspection
– HLD: 69.81%
– LLD: 41.44%
– Code: 33.34%
134. Some observations – 2
• The only defect type that is among the top
three both for testing and inspection is
“Interface”
– Inspection ‐ HLD: 14.12%
– Testing: 39.13%
• The only trigger that is among the top three
both for testing and inspection is “Side effects”
– Inspection – LLD: 29.73
– Testing: 11.07
138. Data sources
We will base our discuss on data from two
experiments:
• UNSW – three experiments with 200 students.
Focus was on process gain versus process loss.
• NTNU – two experiments
– NTNU 1 with 20 students. Group size and the use
of checklists.
– NTNU 2 with 40 students. Detection probabilities
for different defect types.
140. Gain and loss ‐ 1
In order to discuss process gain and process
loss, we need two terms:
• Nominal group (NG) – a group of persons that
will later participate in a real group but are
currently working alone.
• Real group (RG) – a group of people in direct
communication, working together.
