Implementation of Risk-Based Approach for Quality & Cost Optimization

Ananthakrishnan J
Architect,Sonata Software
SonataSoftware Limited
www.sonata-software.com
Risk-Based Testing:
Implementation of Risk-Based
Approach for
Quality & Cost Optimization
Technical White Paper
Author
Kalyanam Kannan

Technical White Paper www.sonata-software.com
Risk-Based Testing 1 Sonata Software Limited
STATEMENT OF CONFIDENTIALITY
Information included in this document, in its entirety, is considered both confidential and proprietary to
Sonata Software and may not be copied or disclosed to any other party without its prior written
consent.
All logos used in this document are registered trademarks of the respective organizations.

Abstract
As a practiced trend in IT projects, Testing is performed only towards the end of a project. Teams
dedicate hours to test possible risks and flaws after the project is ready to run. As software testing at
this level invites several last minute modifications that can cause discomfort, or sometimes even refute
the very concept of the project, it has become the need of the hour to come up with a way to ensure
detection and reduction of risks, at an early stage of the project. Risk-Based Testing, or RBT as referred
to in this paper, is a procedure in software testing which is used to prioritize the development and
execution of tests based upon the impact and likelihood of failure of the functionality or aspect being
tested based on existing patterns of risk.
Taking a cue from the age-old saying of ‘Precaution is better than cure’, RBT aims to find areas where
possibility of risk or defect is most likely to occur. Through this testing technique, a software test
engineer can now select tests based on risk even before the initiation of the project. Example, through
software testing, one can detect 200 errors by testing 5000 defects. RBT on the other hand, enables the
software tester to pick only 500 probable defects areas and conclude with 190 defects, thereby saving
the effort and time of the software tester.
This paper outlines the Risk-Based Testing approach and describes how Risk-Based Testing can positively
impact the development life-cycle based on business-oriented factors, offering organizations an
actionable plan for starting a Risk-Based Testing approach for projects.
About the Author
Kalyanam Kannan has been in the software industry and testing for the past 14 years and has managed
testing projects using different engagement models, Currently, as a Practice Director in testing, he is
responsible for controlling the quality of releases and delivery with optimum cost. He is also involved in
providing testing solutions using the latest technology, tools and operating models, which enable
projects to minimize cost to quality. His current areas of interest include Risk Based Testing, Test Driven
Development and Open Source Based Testing.
If you would like to interact with the author of this White Paper, feel free to contact us.

Contents
Abstract....................................................................................................................................................2
About the Author......................................................................................................................................2
Risk-Based Testing....................................................................................................................................4
Generic Approach for Risk-Based Testing .................................................................................................5
Statistical Models .....................................................................................................................................7
Illustration................................................................................................................................................9
Workflow for Risk-Based approach: .........................................................................................................10
Results......................................................................................................................................................11
Inferences of the concept .........................................................................................................................13
Open Source Test Management................................................................................................................13
Summary ..................................................................................................................................................14
To read more about our views on technology, do visit www.sonatablogs.com

White_Paper_Rbt 4 Sonata Software Limited
Risk – Based Testing
In today’s scenario, the quality of software is becoming a matter of concern. With this issue creating
conflicting challenges, the industry is testing and trying different measures to tackle it. Innovative
techniques, tools, technologies and ideas are being implemented to ensure availability of standard
software. One of the popular measures adapted is Risk-Based Testing – a technique through which a
certain amount of testing can be done without covering an entire gamut of available test cases.
According to Industry Experts, 80% of applications are either not tested or are manually tested before
being delivered to production. This leaves the quality of such software open for speculation and hence,
several software projects cost high due to the risks related to it.
Although a lot of mandatory regression and end-to-end testing is being done, the earlier the defect is
detected; the lower is the cost of solving the issues. To address these issues, Sonata has developed a
statistical model which would provide us a required methodology for Risk-Based Testing.
Sonata with its unparallel experience of product quality assurance services has understood that Risk-
based testing is vital in today's competitive market. With this Risk-based approach, a reduction in cost
per quality with a faster time to market is achieved.
Diagram 1

Risk-Based Testing
Risk-Based Testing is a methodology which after identification of risks and their possible impact on
system allows you to prioritize and plan your test strategy in accordance to the risk rating and mitigation
plans.
These provide us with faster time-to-market that gives us more time to fix the defects. The defects are
not detected at the end of the release; in fact the defects can be detected in the early stages of
Application Development itself. This is a scientific and data-based approach which results in cost
optimization and enhancement of quality. It can identify and execute high risk data hence providing
more time for defect fixes.
Generic Approach for Risk-Based Testing
Going ahead with our Risk-Based approach, a Risk Analysis is performed before starting the testing
activities. The prime objective is to take control over the problems before problems take over the
situation.
The following figure shows the activities involved in Risk Analysis when a project is performed. The
diagram below discusses this in detail:

Diagram 2
Diagram2: Risk analysis activity model - This model is taken from Karolak’s book “Software Engineering Risk Management”,
1996 [6] with some additions made (the oval boxes) to show how this activity model fits in with the test process.
The first step is the Test Planning. In this phase, the risks need to be identified and a Risk Strategy should
be created. A risk can be of many types. One of the key important types could be the complexity in the
available applications, the type of resources and available tools. A clear Risk Strategy needs to be
defined before getting into the Test Planning activities.
Subsequently, the Risk Mitigation plan must be prepared. This plan clearly states for a particular type of
risk what the risk mitigation is. For example, if it is going to be a very complex application the risk
mitigation plan would be “dissecting the application into several components as smaller modules, and
fill up each of those components with more capable resources”.
Once the Risk Mitigation plan is completed, other important areas like the Risk Reporting can be
focused on. Risk Reporting is very important because it provides complete transparency across the
entire stakeholders of the project to gauge and act on the risk area. With all of the testing and
inspection techniques and capturing all test metrics, the risks that get reported are identified. At this
stage, one can predict the risks.

In the Risk Prediction stage there is an entire set of data from which the risk-prone area is identified. In
this model risk prediction feedback is again fed into the risk identification area and it is a cyclic process.
By undergoing several iterations of this cyclic process the Risk Strategy as well as Risk Prediction model
can be refined. The areas with certain defects or minimum defects or no defects can easily be predicted.
This is the core idea of Risk Based Technology.
Statistical Models
In the statistical model the importance lies on the characterization of numerical data. Also, it is very
important to estimate the probability in terms of the behavior of system.
The entire testing activity is nothing but probability. It is the probability of finding out a particular defect
or a particular section failing on a particular area or on a particular type of environment. These all may
be useful in deciding the type of testing required and to ascertain the focus in areas of testing. With this
focus, extrapolation or interpolation of the existing data can be conducted and the best fit for that can
be identified. The best fit will provide the critical path for the defects. It will clearly provide areas that
require testing in that particular application or system.
There is another model called Spectral Analysis of data or model generated output which is an industry
standard. Here the focus is on the algorithms that have been used for this Risk-Based Testing. The
algorithm is extended to suit the current situations where the best fit for various factors is exercised.
This helps in calculating the risk as well as the probability of failure.
The statistical model is based on the probability of defects and a consequence of defect. These two are
very critical in defining the Risk Exposure of the system. One of the important parameters is quality of
the code. It may be suffering from poor designing or it may have been coded by an inexperienced
programmer, it may be to a complex functionality. The probability of defects is defined as P(f),
consequence of defect pertaining to the customer C(c) and consequence of defect to the vendor as C(v).

Diagram 3
Consequences of defect for a customer (which is a cost to the customer) may be:
o The probability of a legal threat
o Losing a market place
o Not fulfilling regulations or FDI regulations
The consequence of defect towards the vendor gives a negative credibility to the vendor or it would
increase the maintenance cost because of the functions with faults.
The combination of these two factors leads to a formula: Risk exposure [Re(f)] is calculated as a product
of the probability of failure and the consequence effect.
Diagram 4
The probability of failure again characterized. It is a combination of multiple probabilities of failures.
Normally, the consequences value weighted between 1-3, and it would count the production fault loss
of revenue impact and incurs cost change impact also. The probability of failure is always weighed
between 0-1.

The weighted average of the probability of failure is dependent upon following factors:
• Changed functionality
• New functionality
• Design quality
• Size of the project
• Complexity
• Programmers’ experience
Illustration
The following calculation of Risk-Based matrix is with a live example from one of the Sonata's projects.
For the sample calculation 18 factors of probability were taken into account. The customer is Europe’s
largest holiday company, serves more than 23 million guests every year. Operates own resorts, hotels,
airlines, travel agencies and cruise ships.
The front-end was a Web Selling platform and the back-end is a mainframe system which supports the
entire set of operations. The backend system is capable of running 600-1000 batch programs on a daily
basis and transacts 5-6 million records every day. The entire system has around 10 interfacing systems
(e.g. Amadeus, Alamo etc). They have multiple staging areas where they present their data and do a
focused analysis for the next quarter or the next season coming through.
The Risk-Based Testing was conducted for this particular engagement because there were nearly 8000-
10,000 test cases for the entire set of Enterprise Applications. While doing so, around 2 to 3 releases are
gathered on a monthly basis at the Enterprise Level. If a complete set of testing or end-to-end testing is
required, it is important to cover all sets of test cases across the enterprise chain. In such scenarios, over
2000 to 3000 test cases are run per release. This consumes a lot of effort and hence the cost, as well as a
delay to market.
In order to handle this situation several testing techniques are adopted, test automation in terms of tool
level as well as in terms of data level which would also test the testing data integrity. In terms of
approach a Risk Based Approach is followed because only a certain amount of test cases or certain
types of test cases catch errors, rest of the test cases were defect-free. Based on our observation it was
found that out of 2000-3000 test cases only around 200-300 test cases were capturing defects. This is

because only these test cases are clearly attached to the risk prone areas. Hence, the statistical model
was adopted to capture most of the defects and implement Risk-Based approach.
Various kinds of testing methods have been implemented in this regard; starting with System Testing
which involves testing of the Web selling platform, testing of their core system, testing of their business
intelligence areas, Integration Testing, Regression Testing, Performance Testing to Security Testing etc. A
specialized testing on Data Integrity on volume testing was also conducted. In compiling all these
methods of testing several areas with defects were identified.
The inputs required for fitting the statistical model are:
o The number of defects
o Types of defect: Database defect, Staging Area defect, Web area defect
o Classification of defects: Defects originated of database, originated from the application server
or from the functionality
o Effort required for defect identification
o Effort required for fixing defects
o Weightage for the probability functions in terms of failure and the consequence
All inputs in the algorithms and routines for iterative runs were rigorously followed. This has resulted in
the probability of failure as well as the risk exposure co-efficient.
Workflow for Risk-Based approach:
As the first step, all defects are classified. Once the defects have been classified the probabilities of
various factors which affect the quality of the release are obtained, post which the risk exposure co-
efficient is derived. Once the Risk Exposure co-efficient is identified, the co-efficients are fed into the
iterative algorithms for various values of probability of failures. From this the type and number of test
cases that will be utilized for Risk Based Testing are obtained.
Example, in a live environment, if there are 2000-3000 test cases and it is known from an existing
analysis that the 23rd or 33rd test case are going to yield defects in a particular area which comes under
the sampling techniques, and there are other defects from a different area, then it is convenient to
sample them on a common algorithm and feed it into algorithm. This enables the identification of the

Risk exposure coefficient. Wherever the Risk exposure co-efficient is high, those are the areas that need
100% coverage.
In this specific example the probable failures in terms of change requests, test interfaces, inexperienced
developer, field validations, business rules validations, positive and negative scenarios, third party
interfaces, system integration testing, backend verifications, UI elements testing, content verifications,
content validations, error messages verifications & validations, cross browser testing, platform
compatibility testing, functional end-to-end flow testing have been taken into consideration. Certain
weights have been assigned to these areas, to calculate the Risk exposure for various iterations from a
value of 0.5 to 2.0 and to achieve a constant consequence as 2 to arrive at the probability of failure.
Results:
There are 3 different depictions:
Iterations:
Re(f){0.5, 1.1, 1.2…..2.0}
having C(c&v)=2
Note: Open Source TM (Algorithms and IP) was used for this study
Defects in Releases
Sample: 1200 TC - Continuous
Graph 1
Defects in Releases
Sample: 1200 TC - Random
Graph 2

Graph 1: It is a sample of 1200 test cases and it is continuous. A particular release has undergone 10-12
iterations and the test cases have been run 1200 continuously. The test cases have been run by an
automated test suite which has been developed on an Open Source Framework. The defects in Graph 1
found in the different iterations and how the applications are stabilized over a period of iteration. In this
case all the 1200 test cases on all releases in an automated way.
Graph 2: Test cases were selected at random, without any logic or reason. This particular method also
provides the defects but the amount of the defect captured is lesser when compared to the amount of
defect captured while the entire set of test cases is run.
Graph 3: The statistical-based algorithm is run and 400 out of the 1200 test cases are sampled. Only a
minimum number of test cases are run but they capture the maximum number of defects. There is no
variation in results as compared to the number of defects identified when all 1200 test cases are run.
This concept saves a lot of effort and cost that leads to an impressive turnaround time.
Defects in Releases
Sample: 1200 TC - RBT
Graph 3

Inferences of the concept
As a result, a 50 – 60% reduction on testing effort (400 test cases out of 2000 test cases) is achieved,
generating data for multiple set of defects scenario. These defect scenarios or data will be applicable in
subsequent releases. In a similar project or a similar type of release the respective test cases can be
pulled straightaway. There is no impact on the critical factors that is on the database side or in the
performance side or in the quality side by doing this.
This has resulted in the 40-50% of the testing cost reduction. In the enterprise environment when
multiple projects in multiple streams are run, each product needs to be tested on a particular day or a
particular time segment. To do this testing continuously in all these areas by using risk-based approach a
greater bandwidth is required to run these tests in spite of running less amount of test cases and achieve
more amount of coverage.
The other important advantage is Defect Predictability. Number of defects can be predicted for a certain
size of application. This helps us in estimating the time for defect fixes. In a project lifecycle analysis,
requirements, developments, testing and release need to be planned. Often, the element that is missed
out is time and effort required for the defect fixes. If a decent estimate of defect fixes can be identified,
then it is easier to estimate the time required to complete it.
Since the developers or the programmers required for this program are selected right at the beginning,
there can be optimized use of relevant expertise and hence the risks can be handled efficiently.
Open Source Test Management
In this activity of statistical model or the iterative algorithm and selecting the relevant test cases and
then running those test cases for execution, you require a proper test management system. Either it can
be a quality center or a QA director or any other tool which is capable of doing that. Sonata has
developed an Open Source Test Management System which is integrated with defect and test
management areas. It houses the entire Risk-Based Testing algorithm and the data for various values of
probabilities of failure in different areas in terms of classification. It has become easy to do the
automated test cases predictability and organize the test cases according to the functionality and defect

areas. Customized simulation for various resources can be obtained. This reduces around 70% of the
regression test cost and 50% improvement in controlled releases.
Summary
The statistical approach for Risk Based Testing is a proven model. It is capable of simulating error
injection, analyzed impacts associated with failures. More importantly, it is simple and cost effective. As
an added advantage an Open Source System supporting it is also available. The algorithms are scalable
and iterative and these algorithms can be used or extended for any type of testing (Web testing, Data
Testing, testing of ERP systems). Customized reports in terms of the available number of risk prone test
cases are generated which definitely need to run in a particular release. All the data is available in a
report format. The Open Source Test Management System houses all of these activities and functions
together and being provided to the customer as a package.

CORPORATE OFFICE
APS Trust Building
Bull Temple Road, N. R. Colony
Bangalore 560 019, India
Tel: 91-80-3097 1999, Fax: 91-80-2661 0972
Email: info@sonata-software.com
WORLDWIDE OFFICES
Dubai
Office # 507, Thurraya Tower No.1
P O Box 502818, Dubai Internet City
Dubai, United Arab Emirates
Tel: 971-4-375-4355, Fax: 971-4-424-0132
Email: info-me@sonata-software.com
Germany
TUI InfoTec GmbH
Karl-Wiechert-Allee 4
30625 Hannover, Germany
Tel: 49-511-567 5296
Email: sales@tui-infotec.com
India
6, Richmond Road
Bangalore - 560 025, India
Tel: 91-80-3097 3299, Fax: 91-80-2248 4045
193, R.V. Road,
Basavanagudi,
Tel: 91-80-3097 2999, Fax: 91-80-2656 7487
Sonata Towers, Global Village,
Pattenegere & Mylasandra,
RVCE Post, Mysore Road,
Tel: +91-80-3097 1499
1-10-176, Begumpet Main Road
Opp. Hyderabad Public School
Hyderabad - 500 016, India
Tel: 91-40-3981 3899, Fax: 91-40-2776 4831
Singapore
1, North Bridge Road, #19-04/05
High Street Center
Singapore – 179094, Singapore
Tel: 65-633-724-72, Fax: 65-633-740-70
UK
5, Churchill Court
58, Station Road, North Harrow
Middlesex HA2 7SA, UK
Tel: 44-20-8863 8833, Fax: 44-20-8863 5533
Email: info-uk@sonata-software.com
USA
39300 Civic Center Drive,
Suite 270, Fremont, CA 94538, USA
Tel: 510-791-7220, Fax: 510-791-7270
Email: info-uswest@sonata-software.com
2018 156th Ave NE, Suite 100,
Building F, Bellevue, WA 98007, USA
Tel: 425-372-2167, Fax: 425 484 7799
Email: info-usnw@sonata-software.com
1901 North Roselle Road, Suite 800,
Schaumburg, IL 60195, USA
Tel: 847-517-6310, Fax: 847-517-6313
Email: info-uscentral@sonata-software.com
11330 Lakefield Drive, Bldg #2, Suite 200
Duluth, GA 30097, USA
Tel: 770-814-4213, Fax: 678-623-0236
Email: info-usse@sonata-software.com
275 Grove Street, Suite 2-400
Newton, MA 02466, USA
Tel: 617-663-4866, Fax: 617-663-6127
Email: info-usne@sonata-software.com
212, Carnegie Center, Suite 206
Princeton, NJ 08540, USA
Tel: 609-919-6325, Fax: 617-663-6127
Email: info-useast@sonata-software.com
If you have any experiences related to Risk Based Testing that you would like to share with us, please
write in to us on info@sonata-software.com

Implementation of Risk-Based Approach for Quality & Cost Optimization

Recommended

Recommended

More Related Content

What's hot

What's hot (18)

Viewers also liked

Viewers also liked (20)

Similar to Implementation of Risk-Based Approach for Quality & Cost Optimization

Similar to Implementation of Risk-Based Approach for Quality & Cost Optimization (20)

More from Sonata Software

More from Sonata Software (18)

Recently uploaded

Recently uploaded (20)

Implementation of Risk-Based Approach for Quality & Cost Optimization