Getting Started with Risk-Based Testing

MC
Full-Day Tutorial
9/30/2013 8:30:00 AM

"Getting Started with RiskBased Testing"
Presented by:
Dale Perry
Software Quality Engineering

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Dale Perry
Software Quality Engineering
Dale Perry has more than thirty-six years of experience in information technology as a
programmer/analyst, database administrator, project manager, development manager, tester,
and test manager. Dale’s project experience includes large-system development and
conversions, distributed systems, and both web-based and client/server applications. A
professional instructor for more than twenty years, he has presented at numerous industry
conferences on development and testing.

Dale L. Perry

SQE Training
Dperry@SQE.com

FUNDAMENTALS OF RISKBASED TESTING

Blank Page for Spacing

©2013 SQE Training - STAR West 2013

©2013 SQE Training - STAR East 2013

2

1

Hidden Slide for SQE Copyright


3

Table of Contents - Agenda
1. Understanding risk
2. Risk and the testing process
3. Test planning (overview)
4. Test creation (analysis and design)
5. Test execution, reassessing risk, and
reporting


4

2

1
UNDERSTANDING RISK

Risk – A General Definition

Risk
Riskbased
testing


• A factor that could result in future
negative consequences; usually
expressed as impact and likelihood

• An approach to testing to reduce the level
of product risks and inform stakeholders
on their status, starting in the initial
stages of a project. It involves the
identification of product risks and their
use in guiding the test process

6

3

Categories of Risk

Process
risk

• Focus is on procedural issues,
both managerial and technical

Project
risk

• Typically includes operational,
organizational and contractual
issues

Product
risk

• Typically relates to technical
issues and depends on the type
of product being developed


7

Categories of Risk (cont.)
There are many approaches to
categorizing risk
Capers Jones documented sixty
different risk factors
• This list provides an extensive look at
many different areas that represent risk
within an organization, project and
software (process, project and product)


8

4

This Slide is Hidden


9




10

5


11


Understanding Process Risks
When we look at process risk we focus on both
management and technical procedures
Management processes that
relate to risk include:

Technical processes that relate
to risk include:

Planning and staffing

Requirements analysis

Tracking

Design and code

Quality assurance

Testing

Configuration
management


12

6

Understanding Project Risks
Project risks can originate from
several areas
• External sources, such as supplier issues
• Issues related to the organization
• Of the project
• Of the team
• Of the “company” itself – politics
• Issues related to technical problems
• Documentation

13

Project Risks – Organizational Factors
Skill, training and staff shortages
Personnel issues
Political issues, such as:
• Problems with testers communicating their needs and test results
• Failure by the team to follow up on information found in testing
and reviews (e.g., not improving development and testing
practices)

Improper attitude toward or expectations of
testing
• (e.g., not appreciating the value of finding defects during testing)



14

7

Project Risk – Technical/Supplier Issues
Technical issues
• Problems in defining the right requirements
• The extent to which requirements cannot be
met given existing constraints
• Test environment not ready on time

Supplier issues
• Failure of a third party
• Contractual issues

15

Project Risk ― External Sources
Delivery date
Acquisition plan

Budget
Scope of test
Use of automation


16

8

Product Risks
Potential failure areas (adverse future events
or hazards) in the software or system are
known as product risks

They comprise a risk to the quality of the
product and may include:
• Delivering failure-prone software
• The potential that the software/hardware could
cause harm to an individual or company
• Poor software characteristics (e.g. functionality,
reliability, usability and performance)
• Software that does not meet specifications (perform
its intended functions)


17

Examples of Product Risk Areas
Some areas of product risk
Scalability

Functional complexity

Scope of use

Performance

Environment

Safety

Security

Data selection

Reliability

Data integrity

Usability

Recoverability

Interface complexity

New technology

Technical complexity


18

9

Attitudes Toward Risk
Three basic attitudes toward risk
Risk Averse
• Practical, dependable, focus on facts

Risk Seeker
• Speculative, adventurous, adaptable, looking
for new things

Risk Neutral
• Focus on the future, neither overly secure nor
speculative

19

Stakeholder Viewpoints on Software
Stakeholder viewpoint—along with
attitude toward risk—affects our
assessments
There are at least four differing views of
software and software related risks
Management
Marketing
Developer/technician
Client/customer



20

10

Understanding Risk Management
Risk
management

• Systematic application of procedures
and practices to the tasks of
identifying, analyzing, prioritizing, and
controlling risk

Risk
Identification

Risk Mitigation
(Risk Control)

Risk Analysis &
prioritization


21

Risk Identification
There are many approaches and
methods for identifying and qualifying
risks
The method selected will depend, to a
large degree, on the type of system
or application being tested
• Some types of systems contain greater risks
(safety critical, mission critical, etc.)


22

11

Risk Identification Techniques
Expert interviews
Independent assessments
Use of risk templates
Lessons learned
• Metrics and data from past projects
• Develop a risk database or repository

23

Risk Identification Techniques (cont.)
Risk workshops
Brainstorming

Checklists
Calling on past experience
Formal analysis methods
• Failure modes and effects analysis (FMEA)
• Hazard analysis  Cost of failure  Others


24

12

Applying Risk Management
The attitudes, viewpoints and focus
of the individuals involved will affect
risk management, each process is
affected by the other processes
Risk
Identification

Risk Control,
Mitigation

Risk Analysis


25

2
RISK AND THE TESTING
PROCESS


13

Key Elements of Risk Driven Testing
Using testing to influence and direct
software design and development
Testing starts at the beginning of the
lifecycle
Tests are used as requirements and
usage models
• Testware design leads software design

27

Key Elements of Risk Driven Testing (cont.)
Defects are detected earlier or
prevented altogether
• Defects are systematically analyzed

Testers and developers work together
Implicit concurrent engineering of
testware and software



28

14

Testing and Development
To test effectively, testing and development
must be managed together
• Testing must become an integral part of the
development lifecycle
• There are many models used to develop software

For each model, testing will have to adjust
to the process in order to be successful
• The only difference is where and how to test


29

Lifecycle Models (Examples and Types)
Sequential

Incremental

Iterative


•Waterfall models
•“V” model
•Cleanroom® Engineering
•“W” model

•Spiral models
•RAD (Rapid Application Development)
•RUP (Rational Unified Process)

•Rapid prototyping
•Extreme programming (XP)
•SCRUM (not an acronym)
•DSDM (Dynamic Systems Development Method)

30

15

The Testing Process
Project

Master
Test Plan

Detail/Level
Plans

Analyze
Design/
Implement
Execute
Report

Test
Planning
Test
Creation
Test
Execution


31

Test Planning
Within a project there may be several
levels of testing and test plans
Master
test plan

Acceptance
test plan

System
test plan



Build test
plan

Component
test plan

Others

32

16

Test Creation (Testware)
Test Analysis
Inventories
Test Design
Test Specifications
Test Implementation
Test Data
Procedures

33

Test Execution and Reporting
Tests and incident
logs
• Failures and anomalies

Checking adequacy
• Decision to stop testing

Evaluating processes
• Generate summary report



34

17

An Effective Testing Process
Focuses on preventive measures
Is lifecycle-based
• Preventing a defect/fault is cheaper and more effective
than finding and fixing it later in the process

It can be incrementally implemented in an
organization
• You don’t have to change everything at once


35

3
TEST PLANNING OVERVIEW


18

Test Planning - Overview
Planning is a critical part of the
testing process
• It is introduced here as part of the
overall testing process, however, we
will not go into great detail
• Test planning is covered in greater
detail in the tutorial on Test
Management and Planning


37

Test Planning - Strategic Issues
•

Scope of the testing
– Breakdown of the testing
– Levels or other approaches

•

Software risks
– Both technical and
requirements based

•

Planning risks
– Budget
– Resources
•
•
•
•

Skills
Time
Budget (money)
People

– Dependencies

•

Test environment
– Complexity
– Accuracy



•
•

Testing tools
Test requirements

•
•

Configuration management
Relationships between other
organizations
Measurements/metrics
Software life-cycle in use for the
project
Documentation
Need for regression
Schedule
Staffing

– Objects and goals

•
•

•
•
•
•

– Internal
– Third party
– Consultants
38

19

4
TEST CREATION
(ANALYSIS AND DESIGN)

Focus of Testing
• Does the software do what it’s supposed to do?
• Are there unexpected results or events?

Negative
Positive

Other…

Testing


40

20

Approaches to Testing
• Static testing - prevent problems when applied
early in the process
• Dynamic testing - detect problems once the
software is developed
• Learning - explore and learn about the product

Dynamic
Static

Learning
Testing


41

Complete/Exhaustive Testing Is Impossible
No amount of testing will
Prevent all defects
Detect all defects
Prove a product is defect-free



42

21

Complete/Exhaustive Testing ― Example

There are too many possibilities to
completely test a system
• For example, a single screen (GUI) or data
stream, with thirteen variables, each with
three values
• To test every possible combination is 3(13) possible
combinations or 1,594,323 tests
• Plus testing of interfaces, etc.


43

Focusing the Testing
A primary goal in testing is to have the
critical features developed first
• Focus the testing effort where the potential risks
exist within the software
• Knowledge of risks guides selection of what and
where to test



44

22

The Test Inventory Process
A model designed for test analysis
within a project
• Applicable to virtually all development
lifecycle models

Designed to interact with the
development processes used within a
project focusing on key issues
• Understanding testing issues and risks
• Determining the scope of testing
• Improving estimates and budgets

45

Inventory Activities
Gather the reference
materials
Form a team
Select the test objects and
conditions
Create an inventory
Prioritize the objects


46

23

Inventory Activities (cont.)
These activities can be incorporated
into existing development activities
within the project
• At the beginning of every project there are
activities during which the software to be
developed is outlined or defined
• Requirements activities
• Architectural activities
• Detail design and coding activities

• All of these can incorporate the inventory concept

47

Gather Reference Materials
User
requests
Requirements,
stories, use
cases

Release
information

Inventory

Detail
designs



Architecture,
system
designs

48

24

A Well-Balanced Team
Analysts,
architects

Testers/QA

Developers

Users/clients

Marketing

Inventory
process

Management


49

Determining Objects and Conditions
The goal is to gather as much
information as early as
possible about the testing
goals, objects, and conditions
• This enables earlier test design
decisions and locates potential
problems sooner and possibly
prevents them from occurring


50

25

What Is a Test Object/Condition?
Any aspect of the software, its execution
environment, its usage patterns, etc.,
that should be covered by the testing
Basic functions or methods
Configurations
Usability
Interfaces
etc.

51

Identifying Test Objects/Conditions
The inventory process is an incremental,
iterative process starting at the requirements

Requirements (specifications,
models, etc.)
Architecture (high level
design)
Detail design


52

26

What Is an Inventory?
Example

1. Transactions
•

Add

•

Change

•

An inventory is a list
or index of test
objects

Delete

2. Screens
•

Daily

•

Weekly

•

Monthly

It is a detailed answer
to the question of
what should (must) be
tested

3. Reports
4. Constraints
5. Security
6. etc...

53

Inventory Process ― Flexibility
Inventories can be created easily during the
product risk identification and assessment
process
• They can be expanded or changed as needed

Inventories help identify both functional and
non-functional and structural areas to test
• This improves the quality and effectiveness of the
testing and helps avoid problems later in the process



54

27

Applying the Inventory Process
There are some common aspects of
applications/systems that can be used as a
starting point (model) in determining the
test objects/conditions
• Even if there are no formal requirements specifications

The more detailed the initial information, the
more accurate the inventory of test
objects/conditions, and the better the
understanding of risk

55

Applying the Inventory Process (cont.)
As noted earlier building an inventory is
an incremental process
It can be applied at the project level or at
an individual level (system, unit, etc.)
Starting with a global view
• A broader view can help put elements into proper
perspective
• “You need to see the forest before the trees”



56

28

Building the Inventory – Requirements
While creating an
inventory of test
objects you generate
two key elements
• The test objects
• A list of issues and
questions
• Potential errors and
defects

Requirements,
use cases,
stories, etc.

Issues and
questions

Inventory of
test objects


57

General Requirements-based Objects/Conditions
Features, functions, and methods
Constraints (limits on application, environment,
etc.)

Inputs and outputs
Behavior and business rules
Interfaces
Configurations (hardware, software, etc.)
Scenarios
Other elements…


58

29

Design Analysis ― Inputs
Requirements-based inventories

Questions and issues from requirements
can be used to verify design

Items remaining unresolved may result
in major rework later if left open


59

Building the Inventory – Architecture
Requirements,
use cases,
stories, etc.

Issues and
questions

High level
design
(Architecture)

Detail level
design

Inventory of
test objects

• The process can be continued as long as required to
ensure accurate information


60

30

Design Analysis ― Steps
Study the software design
Request clarification
Investigate with sample test cases
Request design enhancements
Inventory and prioritize design-based
objects
Review and revise the inventory

61

General Design-based Objects/Conditions
Detailed inputs and outputs
System and design states
Process paths
Data and communications paths
Design limits and exceptions
etc.


62

31

Investigating Using Tests
Consider a credit card management
application with the following capabilities
• Add, change, delete and inquire on customers
• Add, change, delete and inquire on credit cards

Consider the following scenarios
• Update customer address, inquire on same customer
to validate change and inquire on card balance
• Inquire on customer, delete customer (customer has
multiple cards on same account), delete credit card


63

Investigating Using Tests (cont.)
Using the two scenarios on the previous
page and the design diagram on the next
page

Do you see any problems in the design?

• If so what are they?
• What might happen if they are not corrected?



64

32

Investigating Using Tests (cont.)
DISPLAY
Main Menu
DISPLAY
Customer Menu

ADD
Customer

CHANGE
Customer

DELETE
Customer

Inquire
/Print

DISPLAY
Card Menu
Inquire/
Print

ADD
Card

CHANGE
Card

DELETE
Card


65

Prioritize the Objects/Conditions
Using our list of test objects and
conditions We must determine
which elements are critical and
which are not

To do this we need to understand
risk analysis



66

33

Risk Analysis ― Purpose
Focus the software design and
test/evaluation activities on those
elements/functions that are
Critical to the mission of the
organization
Essential to the physical or
economic well-being of users and
their environment

67

Using Risk Analysis
Risk analysis is the study of the
identified risks
• Categorizing each risk
• Determining the likelihood and impact

Each identified risk should be
placed into an appropriate class
• Classes can be internally defined or may be drawn
from a standard such as the ISO 9126
• The use of checklists can combine the identification
and classification processes



68

34

Risk Analysis ― Characteristics of Risk

Likelihood

Impact

• High
• Medium
• Low

• High
• Medium
• Low


69

Factors Influencing Likelihood
Many technical factors influence risk
potential and vary from project to project
and team to team within a project
Areas to investigate may include:
Complexity of technology and teams
Personnel and training issues
Conflict within the team or between teams
Contractual problems (vendors, suppliers,
contractors, etc.)


70

35

Factors Influencing Likelihood (cont.)
Areas to investigate (continued)
Geographical separation of the team
Legacy versus new ideas and approaches
Use of tools and new technologies
Lack of managerial or technical leadership
Pressures on time, resources and management
Lack of earlier quality assurance
• No focus on quality at early levels of testing


71

Factors Influencing Likelihood (cont.)
Areas to investigate (continued)
A high frequency of change requests
Poor early quality (high defect rates)
Issues with interfacing and integrating software



72

36

Factors Influencing Impact
Risk impact comes down to the impact on
the customer or user of the software under
test
Influencing factors may include:
Frequently-used features
• If it’s not available or corrupt, it will have adverse impacts

Impact on the corporate or system image
• Visibility of failure may lead to negative publicity


73

Factors Influencing Impact (cont.)
Influencing factors (continued)
Potential loss of business
Potential financial, ecological or even social
losses
Legal liabilities (civil or criminal )
Potential loss of an operational license
Work stoppages due to a lack of reasonable
workarounds


74

37

Approach to Risk Analysis

Quantitative

Qualitative

• Assign a value to each aspect (likelihood,
impact) multiply the two values together to
calculate the cost of exposure. This is the
expected loss associated with a particular
risk

• Use categories such as very high, high,
medium, low, or very low


75

Approach to Risk Analysis (cont.)
Most organization use a combination
of both methods
Qualitative helps express risk in terms a person
can understand

Quantitative is then used in conjunction with
qualitative categories to create a matrix with
each risk given a weighted priority



76

38

Initial Risk Priority ― A Risk Model
Likelihood
High

(3)

Medium

(2)

Low

Impact
High

(1)

Medium

(2)

Low

X

(3)
(1)

Impact and likelihood = Potential loss ($)
(helps set the test priority)


77

Adjusting the Assessment
What if we feel one factor is of
greater importance?

Likelihood
High

(3)

Medium

(2)

Low

(1)

Impact
High

X

(5)

Medium

(3)

Low

(2)

Impact and likelihood = Potential loss ($)
(helps set the test priority)


78

39

Example ― Risk Analysis
Test Object

Likelihood

Impact

Priority

Client update

L

M

LM=2

Client delete

M

L

ML=2

Client add

H

H

HH=9

Client inquiry

L

L

LL=1

Status report

L

L

LL=1

Performance

M

H

MH=6

Monthly report

L

L

LL=1

Sales report

M

L

ML=2

Calc. sales com.

M

M

MM=4

Mkt. report

M

M

MM=4


79

Risk Inventory Prioritized
Test Object

Likelihood

Impact

Priority

Client add

H

H

HH=9

Performance

M

H

MH=6

Calc. sales com.

M

M

MM=4

Mkt. report

M

M

MM=4

Client update

L

M

LM=2

Client delete

M

L

ML=2

Sale report

M

L

ML=2

Client inquiry

L

L

LL=1

Status report

L

L

LL=1

Monthly report

L

L

LL=1



80

40

Planning for Contingencies
Test Object

Test Case Number

Priority

Client add

23, 45, 47, 48, 10, 12…

9

Performance

15, 22, 31, 12…

6

Calc. sales com.

52, 53, 101, 17, 19…

4

Mkt. report

4, 7, 33, 37, 42…

4

Client update

57, 58, 41, 9, 8…

2

Client delete

62, 110, 82, 63…

2

Sale report

13, 16, 77, 68…

2

Client inquiry

19, 3, 5, 102…

1

Status report

73, 64, 56, 7…

1

Monthly report

74, 98, 92…

1


81

Risk Control – Scope of Testing
Once we have prioritized the list of test
objects/conditions, the next issue we have
to address is the scope of testing for each
element, how will we control the risk
• How much testing is required for each object?
• What type of tests can be created?
• This will be affected by the development model
employed and the completeness of the information
available



82

41

Approaches to Test Design
Depending on the goals of the testing
there are several ways to approach the
design process
• Formal and informal approaches

The approach chosen depends on many
factors
• Type of development process
• Types of defects anticipated
• Risk issues to be addressed
• Other factors

83

Formal Test Design
Test design using defined document
formats and analytical methods
Static methods
• (Inspections, walkthroughs, etc.)

Formal test techniques and
methods
• Equivalence partitioning and boundary
testing
• Pair-based methods (orthogonal and
combinatorial)
• Decision tables and state diagrams
• etc.


84

42

Formal Test Techniques and Risk
Different test techniques are good at identifying
and resolving certain types of problems (risks)
Static methods (reviews)
Help eliminate errors and mistakes at their
source
Find things that are unclear, mismatched,
vague, ambiguous, and possibly un-testable

Identify overlooked and missing elements
early in the process

85

Selecting a Technique (cont.)
Equivalence partitions
Help define and clarify requirements
Provide a solid basis for test condition selection

Boundary value testing
Improves the focus of test conditions selected
using equivalence partitions
Helps identify and eliminate some of the most
common mistakes made in coding



86

43

Test Techniques and Risk (cont.)
Decision tables
Help simplify complex business rules
Identify incomplete and contradictory logic in
rules
Help identify elements in each rule that must be
tested

State diagrams
Help identify critical sequences of events and
those events that can have a significant impact on
the system or application


87

Test Techniques and Risk (cont.)
Pair-based methods (orthogonal arrays
and combinatorial analysis)
Handle large amounts of interacting values where there
are limited or no dependencies between elements
• Decision tables are better at complex relationships

Informal methods
Help identify errors and mistakes caused by human
activity
Scientific techniques deal with logic very well but the
real world tends towards chaos


88

44

Informal Test Design Approach
Test analysis using the skills and
knowledge of the test team
Exploratory testing
Error guessing
Fault attacks
Defect based (defect and failure history)


89

Selecting the Test Approach
If the information available is relatively
complete, formal methods are a good option
Sequential models tend to focus on complete
requirements as early as possible
• At least that is the intent

Informal methods can be used to supplement the more
formal techniques
Initial test design can be more rigorous and the
documentation can be more detailed
• Care must be taken not to overdo the documentation aspect of testing



90

45

Selecting the Test Approach (cont.)
If the information is incomplete or
intended to evolve, the use of informal
methods may be the best initial choice
supplemented by formal techniques
In iterative and some incremental models, test
design, like the software itself, must be
evolutionary
• Initial tests will be high level and subject to constant change
• The formality of test design and documentation must be
flexible


91

Selecting the Test Approach (cont.)
The nature of the test
object/condition and its level of
risk will affect the depth and
breadth of the test design
• The more flexible the tests must be due to
partial information, the more time it will take
to complete the testing
• Increases risk that something will go untested



92

46

Test Approaches and Risk
Tests that have to evolve with the
software, require testers with a
greater depth of skill and knowledge
• Methods like exploratory testing rely on the skills and
knowledge of the testers
• An in-depth set of mental models (both formal and
informal) is essential to success

• Documentation and the repeatability of the tests
become more problematic
• Enough information has to be retained to allow someone
other than the original tester to repeat the test


93

Completing the Test Design Process
Organize the test objects
and conditions

Document the test design
decisions
• How much documentation is required
and to what level of detail represent a
type of risk


94

47

Organize the Test Objects
Select which objects and conditions
will be tested as a logical set at
which stage/level of testing
Define the test sets for the grouped
test objects and conditions
• Select a design strategy (formal, informal)
• Document the test architecture and design
decisions


95

Example Test Set Definition
Input

Test set
screen

Output

Test
cases

Text

Fields

Test
cases

Test
cases

Internal

Test
cases

External

Test
cases

Navigation

Security

Test
cases

Other


96

48

Test Creation - Summary
There are many test objects/conditions
to be considered
• Different types of projects have
different sources but all have
requirements, designs, etc.
The inventoried objects provide a
means for maintaining, evaluating and
updating test requirements


97

Implementing The Test Design

• All tests require the
proper data and
environment be in place
prior to test execution



98

49

Implementation Risk Areas
Environment

Data

Software

Implementation


99

Key Data Source Characteristics
Production Generated

Captured

Created

Volume

Too Much

Controllable

Controllable

Too Little

Variety

Mediocre

Varies

Varies

Good

Acquisition

Easy

Varies

Fairly Easy

Difficult

Validation
(Calibration)

Hard

Hard

Fairly Hard

Easy

Varies

Usually Easy

Varies

Easy

Change

• Acquiring the necessary data may be the most
time consuming part of test development
– Requires careful planning and the right tools


100

50

The Test Environment and Software
Application/

Hardware

System

Documentation

Testware

Firmware
(system
software)

Application
software


101

Preparing The Test Environment
Establish, manage, and maintain
the environment identified in the
test plan
• Publications
• Supplies
• Personnel
• Facilities
• Hardware
• Software



102

51

5
TEST EXECUTION,
REASSESSING RISK, AND
REPORTING

Test Execution and Risk
The earlier risk identification, analysis,
and prioritization activities helped
identify potential product risk areas

With test execution we begin to
discover, analyze, assess, and reassess
the potential product risks



104

52

Test Execution – Key Questions
Are we running the critical tests early?
Are our tests achieving the level of
coverage we desire?
Are the tests experiencing a high or low
failure rate?
What does the defect information
suggest?

105

Assessing and Reporting
Three key elements related to risk
Testing
status
Defect
status

Coverage
assessment

Test
Effectiveness
and Risk



106

53

Coverage Assessment
Coverage is relative to a specified
inventory and risk assessment
Effectiveness

Thoroughness

Comprehensiveness


107

Which Coverage Measures Matter?
Requirements
Risk

Design
Code



• Which requirements have been tested?

• Which risk areas have been tested?
• What risks are still unaddressed?

• What parts of the architecture have been
tested
• Have we addressed key design issues?
• What percentage of the code has been
tested?
• Paths, conditions, decisions, and statements

108

54

Testing Status
Which tests are we running?
Which are passing and which are failing?
When executing the tests, what do the
numbers tell us?
• Initially planned 10,000 tests, currently executed
1,500—what does this mean
• Are we 15% finished?
• If so, finished with what?
• On the next two slides, let’s look at variations of this

109

Test Status ― Example 1
Planned

Executed

High risk

5,000

Total

1,500

1465

35

3,000

Low risk

Failed

2,000

Medium risk

Passed

10,000

• What does this example tell us about test
coverage?, about risk?
• Why might this be happening?
• What is the potential impact on the project and the
product?


110

55

Test Status ― Example 2
Planned

Executed

Passed

Failed

High risk

2,000

500

491

9

Medium risk

3,000

250

243

7

Low risk

5,000

750

731

19

Total

10,000

• Now, what does this example potentially
tell us?


111

Testing Status – Context
Implemented
Specified

Planned



Executed

Tests

Passed/
Failed

112

56

Test Effectiveness Issues
Tests planned versus executed
On or off schedule
Time spent on key activities and their value
• Documenting the tests versus executing the tests

Issues preventing execution
• Blocking incidents
• Lack of resources (equipment, personnel, data, etc.)


113

Assessing Defect Status
Failing tests represent risks to
the quality of the software
• What is the severity/impact of occurring
defects?
• What is the failure rate?
• What is the distribution, density, and other
characteristics of the failures?
• Are they grouped or generally distributed?

• How much effort (rework) is being expended
to correct and re-test the failed element?
• Too much rework can imperil the schedule


114

57

Classification of Defects
• Is there a standard documented scale?
– Recognized and used by all projects
Fatal

Critical

Cosmetic

Incident

Minor

Major


115

Defect Analysis – What are the Risks?
Where are the defects occurring?
What is being missed?
Why are the defects occurring?
What is the cost and impact?
Defect estimation—how many defects
are left?


116

58

Analyzing Defect Characteristics
Patterns

Trends

Recurrence

Defects

Density

Distribution


117

Defect Assessment – Example
Component

Risk

Planned

Executed

Passed

Failed

Module1

High

300

100

97

3

Module2

High

350

125

123

2

Module3

High

250

75

75

0

Module4

High

300

150

145

5

Module5

High

175

50

48

2

Module6

High

225

0

Module7

High

400

0

• What does this example tell us about our
defects and risk?



118

59

Assessing Defect Status
Open

Cancelled

Status

Closed

Pending


119

Stopping the Testing
Exit criteria met
Coverage Criteria
Testing status
Defect status
Remaining defects estimation criteria
Diminishing returns criteria
Combination of criteria



120

60

Stopping the Testing
“There is no single, valid, rational
criterion for stopping.
Furthermore, given any set of
applicable criteria, how each is
weighed depends very much on
the product, the environment, the
culture, and the attitude to risk.”

— Boris Beizer

121

The Key to Success
Risk-driven testing provides the critical,
objective information to allow
stakeholders to make informed
decisions
When is risk-driven testing unnecessary?
• The software will never be used
• You have unlimited resources 


122

61

Tutorial Evaluations

Tutorial Evaluation
How did we do?
* Great

* Wonderful

* Fantastic

* Terrific

* Superior

* Excellent


123

Thank You

• On behalf of SQE
Training, we appreciate
your attendance at this
course
• If you have additional
training or consulting
needs, please think of
us



124

62

Bibliography ― Books
Boehm, Barry W. Software Risk
Management, IEEE Press, 1989.
Boehm, Barry W. Software Engineering
Economics, PTR Prentice Hall, 1981.

Brooks Jr., Frederick P. The Mythical Man
Month, Addison Wesley, 1995.
DeMarco, Tom and Timothy Lister.
Waltzing with Bears: Managing Risk on
Software Projects, Dorset House, 2003.


125

Bibliography ― Books (cont.)
Down, Alex, Michael Coleman, and Peter
Absolon. Risk Management for Software
Projects, McGraw Hill, 1994.
Gerrard, Paul and Neil Thompson. Risk Based
E-Business Testing, Artech House, 2002.
Grey, Stephen. Practical Risk Assessment for
Project Management, Wiley, 1995.
Hall, Elaine M. Managing Risk: Methods for
Software Systems Development, Addison
Wesley, 1997.


126

63

Bibliography – Books (cont.)
Jones, Capers. Assessment and Control of
Software Risks, Yourdon Press, 1994.
McManus, John. Risk Management in Software
Development Projects, Elsevier, 2004.
Myerson, Marian. Risk Management Processes
for Software Engineering Models, Artech
House, 1996.
Neumann, Peter G. Computer Related Risks,
Addison Wesley, 1995.


127

Bibliography ― Articles and Papers
Bach, James. Troubleshooting Risk Based
Testing. Better Software magazine, May/June
2003.
Gardiner, Julie. Risk – The Testers Favorite Four
Letter Word. Track Session, STAREAST 2005.

Kaner, Cem. The Context-driven Approach to
Software Testing. Track Session, STAREAST
2002.
Rice, Randy. The Risks of Risk-based Testing,
Track Session. STAREAST 2007.


128

64

Bibliography ― Articles and Papers
Sliger, Michele. ScrumBut: Failure to Deliver.
StickyMinds.com column, June 2009.
Sliger, Michele. Risk Management on an Agile
Project. Track Session, Better Software
Conference 2006.
Sliger, Michele. The Agile-Traditional
Development Cooperative. Track Session,
Better Software Conference 2007.


129

Appendix A
RISK BASED DESIGN
USING FORMAL
TECHNIQUES


65

A Solution to the Exhaustive Example
• 13 variable, 3 value
problem
– 1,594,323 possible
combinations

• Here we have 13
fields and values
ready for PICT


131

Results from PICT

• These results from PICT will be pasted
into Excel and we will add a results
column to create a test case matrix


132

66

Results in Excel with Results Column


133

Limits to All Pairs
• All Pairs methods deal very well with
valid values and with tools like PICT we
can create restrictions
– But what about testing invalid values?

• Next, using the initial data from the
PICT tool, duplicating it in Excel we will
modify the results to create invalid test
cases
– We will use Equivalence and boundaries to
create the invalid tests


134

67

Creating Invalid Test Cases
• On the following two slides the
following method was applied:
– Invalid values are in italics and are red
– The first field “F1” is a set so all I need
is a single non-set member
– Field “F2” is a range
• Using boundaries I went above and below
and also did a numeric and alpha as well

– These steps were repeated for all fields


135

Invalid Test Cases for All Pairs Data
Test
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

INVALID TEST CASES
(Using equivalence partitions/boundary analysis)
F1
F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 F13
3
3
D
6 G 9 L 12 N 13 P 16 S
B
0
F
5 H 9 K 10 M 14 R 18 T
B
4
F
5 H 9 K 10 M 14 R 18 T
B
D
F
5 H 9 K 10 M 14 R 18 T
B
@
F
5 H 9 K 10 M 14 R 18 T
C
1
Z
4
I
8
J 11 N 15 R 17 U
A
2
D
4
I
7
J 12 O 14 Q 16 T
A
2
D
8
I
7
J 12 O 14 Q 16 T
A
2
D A
I
7
J 12 O 14 Q 16 T
A
2
D % I
7
J 12 O 14 Q 16 T
C
3
E
5 7
7 K 12 O 15 P 18 U
B
1
E
6 H 6
L 10 M 13 Q 17 S
B
1
E
6 H 10 L 10 M 13 Q 17 S
B
1
E
6 H X L 10 M 13 Q 17 S
B
1
E
6 H $
L 10 M 13 Q 17 S
B
2
F
4 G 8 5 11 M 15 P 16 T



RESULTS
FIELD MSG1
FIELD MSG2
FIELD MSG2
FIELD MSG2
FIELD MSG2
FIELD MSG3
FIELD MSG4
FIELD MSG4
FIELD MSG4
FIELD MSG4
FIELD MSG5
FIELD MSG6
FIELD MSG6
FIELD MSG6
FIELD MSG6
FIELD MSG7

136

68

Invalid Test Cases for All Pairs Data (cont.)
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

C
C
C
C
A
C
C
C
C
A
B
B
B
B
A

1
1
1
1
3
1
1
1
1
3
2
2
2
2
1

D
D
D
D
F
D
D
D
D
F
F
F
F
F
E

6
6
6
6
6
5
5
5
5
5
4
4
4
4
5

H
H
H
H
I
H
H
H
H
I
H
H
H
H
G

7
7
7
7
9
9
9
9
9
8
7
7
7
7
8

K
K
K
K
J
L
L
L
L
L
K
K
K
K
J

9
13
A
&
10
10
10
10
10
11
12
12
12
12
10

O
O
O
O
9
O
O
O
O
N
N
N
N
N
O

13
13
13
13
14
12
16
Q
#
14
13
13
13
13
14

R
R
R
R
P
Q
Q
Q
Q
3
R
R
R
R
R

18
18
18
18
17
16
16
16
16
18
15
19
R
&
17

S
S
S
S
U
U
U
U
U
S
U
U
U
U
6

FIELD MSG8
FIELD MSG8
FIELD MSG8
FIELD MSG8
FIELD MSG9
FIELD MSG10
FIELD MSG10
FIELD MSG10
FIELD MSG10
FIELD MSG11
FIELD MSG12
FIELD MSG12
FIELD MSG12
FIELD MSG12
FIELD MSG13

• With 51 test cases, using all pairs, equivalence classes
and boundaries and experience based techniques we
now have a reasonable set of test cases for a large
problem

137

A Simple Procedure

Test Procedure
Step
1 Navigate to input screen/page
2 Starting at row 1 in first table
3 Enter data from each column in matching field element on screen/page
4 Verify indicated result
5 Repeat steps 2-4 for all rows in the tables



138

69

APPENDIX B
EXAMPLE TEST PLANS
AND INVENTORY

This Slide Is Hidden



140

70



141


142


142

71


143

143




144

72



145




146

73



147




148

74



149




150

75



151




152

76



153




154

77



155




156

78



157




158

79



159




160

80

Getting Started with Risk-Based Testing

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (15)

Similaire à Getting Started with Risk-Based Testing

Similaire à Getting Started with Risk-Based Testing (20)

Plus de TechWell

Plus de TechWell (20)

Dernier

Dernier (20)

Getting Started with Risk-Based Testing