1. ROLE OF TESTING IN CERTIFICATION PROCESS, JANUARY 2011 1
Role of Testing in Certification Process
Rishu Seth
(Bachelor of Engineering, Computer Science)
University of Applied Sciences - Frankfurt am Main, Germany
Email: rishu_seth@yahoo.com
Abstract—This paper gives an overview of the procedure References 5
of testing during verification and validation which identifies
important defects, flaws or errors in the application code that
must be fixed during software development process and resulting L IST OF F IGURES
in successfull certification process for that software.There are 1 V-Model of Software Testing [2] . . . . . . . . . 3
different types of testings available for different types of processes 2 Software Testing Phases [2] . . . . . . . . . . . . 3
according to the requirement of the software.Then this paper
lists and describes the basic certification problems like Relia- 3 DO178B V & V Process [6] . . . . . . . . . . . 4
bilty Certification Problem and Verification Methods Drawbacks 4 Design Control Model [7] . . . . . . . . . . . . . 5
which are contemporary in today’s world. Also, it states some
basic standards and their requirements for testing. I. I NTRODUCTION [1]
Ertification - A written guarantee that a system or compo-
C ONTENTS C nent complies with its specified requirements and is ac-
ceptable for operational use. Certification is usually carried out
I Introduction [1] 1 by government agencies or other organizations with a national
I-A Certification Process [1] . . . . . . . . . 2 standing. Certification can be applied to either organizations
I-A1 Verification . . . . . . . . . . 2 or individuals, tools or methods, or systems or products.
I-A2 Verification Techniques . . . 2 Certification with regard to organisation aims at making sure
I-A3 Validation . . . . . . . . . . 2 that the organisation is making use of certain standards and
I-A4 Validation Techniques . . . . 2 criterias to achieve a level of expertise. Though it is not
completely applicable to all areas because measuring the
II Role of Testing in Certification Process 2 procedures is easy as compared to measuring the competence
II-A Why to do Software Testing? [2] . . . . 2 with which they are executed. So rather than being applied
II-B Who Does The Testing? . . . . . . . . . 2 to design, certification is applied to areas such as quality
II-C Test Specification Techniques [3] . . . . 2 assurance and testing to make sure that the aim is fullfilled.
II-D The V-Model of Software Testing . . . 2 Apart from organisations, certification can also be applied to
II-D1 Unit testing . . . . . . . . . 3 individuals, enabling them to be in a certain profession like
II-D2 System testing . . . . . . . . 3 doctors, lawyers, accountants and civil engineers or to tools
II-D3 Integration testing . . . . . . 3 for being used in different software development processes by
II-D4 User Acceptance Testing . . 3 defining some basic requirements for both. For example, DO-
II-D5 Production Verification Testing 3 178B/ED-12B does not define any specific tools to be used
but gives certain requirements for tools that are to be used to
III Principal Certification Problems 3 gain certification.
III-A Reliability Certification Problem [4] . . 3 Finally systems or products may also be certified. There is
III-B Verification Methods Drawbacks [5] . . 4 an issue with certification process to whether methodolgy is
to be certified rather than the artifact. This is a basic issue
IV Requirements for Testing based on Standards 4 with products consisting of softwares specially safety critical
IV-A Aerospace: RTCA/DO-178B (EURO- softwares, because software testing is so difficult and also just
CAE ED12B) [6] . . . . . . . . . . . . 4 verifies that software is made following some guidelines and
IV-B Biomedical Engineering: IEC 60601-1- has met some standard specified by certifying agency. This
4 [7] . . . . . . . . . . . . . . . . . . . 4 does not necessarily mean that the system is correct.
IV-B1 Hardware Verification . . . . 4 There are different reasons why a product needs certifica-
IV-B2 Software verification . . . . . 4 tion. Sometimes certification is required for legal reasons.
IV-B3 System validation . . . . . . 5 For example, before an aircraft is allowed to fly, it must
IV-C Generic Standard: IEC 61508 [8] . . . . 5 obtain a license. Being certified would also be important for
IV-C1 Overview of Requirements . 5 commercial reasons like having a sales advantage. One of the
main reasons for certification is to show competence in specific
V Conclusion 5 areas.

2. ROLE OF TESTING IN CERTIFICATION PROCESS, JANUARY 2011 2
A. Certification Process [1] in space due to a data conversion error. Investigators dis-
1) Verification: Verification - The process of evaluating a covered that software on the spacecraft performed certain
system or component to determine whether the product of a calculations in English units (yards) when it should have
given development phase satisfys the conditions imposed at used metric units (meters).
• In June 1996 the first flight of the European Space
the start of that phase. i.e.
Agency’s Ariane 5 rocket failed shortly after launching,
"Are we building the product right?" resulting in an uninsured loss of $500,000,000. The
disaster was traced to the lack of exception handling for
2) Verification Techniques: There are many different veri-
a floating-point error when a 64-bit integer was converted
fication techniques but they all basically fall into two major
to a 16-bit signed integer.
categories -
Software testing answers questions that development testing
• Dynamic testing
and code reviews can’t. They are as follows:
• Static testing
• Does it really work as expected?
But verification simply demonstrates whether the output of a
• Does it meet the user’s requirements?
phase conforms to the input of a phase as opposed to showing
• Is it what the users expect?
that the output is actually correct. Also it does not detect errors
• Is it compatible with our other systems?
resulting from wrong input specification and these errors may
• How does it perform?
effect in later development stages. So verification alone is not
• How does it scale when more users are added?
sufficient and we need validation as well to be more sure that
• Which areas need more work?
the system is operational according to specifications.
• Is it ready for release?
3) Validation: Validation - The process of evaluating a
system or component during or at the end of the development Factors involved in testing are -
process to determine whether it satisfies specified require- • Business requirements
ments. i.e. • Functional design requirements
• Technical design requirements
"Are we building the right product?" • Programmer code
4) Validation Techniques: Validation usually takes place at • Hardware configuration
the end of the development cycle, and looks at the complete
system as opposed to verification, which focuses on smaller B. Who Does The Testing?
sub-systems. Various validation techniques are:
Software testing is not a one person job. It takes a team,
• Formal methods but the team may be larger or smaller depending on the size
• Fault injection and complexity of the application being tested. The actual
• Dependability analysis programmers that wrote the code should have a reduced role
• Hazard analysis in testing their own written code.
• Risk analysis
C. Test Specification Techniques [3]
II. ROLE OF T ESTING IN C ERTIFICATION P ROCESS
Test specification techniques can be split up into two groups
Testing always means comparing. It requires an item to are as follows:
be tested and terms of reference with which the item must • White Box Testing - In this testing, techniques are
comply. It provides an insight into the difference between the based on the program code, the program descriptions or
actual status and the required status of the item. technical design. Knowledge about the internal structure
According to International Standard Organisation testing is of the system plays an important role. Other terms used
described as - for this kind of techniques are Glass-box or Structural
"Technical operation that consists of the determination of one testing.
or more characteristics of a given product, process or service • Black Box Testing - In this testing, techniques are based
according to a specified procedure" [3]. on functional specifications and quality requirements. The
whole system is viewed as it would be in actual case.In
A. Why to do Software Testing? [2] this kind of testing, knowledge about the structure of the
system is not used but the judgement is made merely
The most basic and instinctive response of people is "To
from a functional point of view of the system.
find the bugs!", but programmers and developers know that for
finding bugs ’Debugging’ is there. But actually “bug” is really
a problem in the code, whereas software testing is focused on D. The V-Model of Software Testing
finding defects in the final product. Examples of important It is not advisable to leave Software Testing for the end
defects that better testing would have found: of the project. So the V-Model of testing incorporates testing
• In October 1999 the $125 million NASA Mars Climate into the entire software development life cycle. In a diagram
Orbiter— an interplanetary weather satellite — was lost of the V-Model, the V proceeds down and then up, from left to

3. ROLE OF TESTING IN CERTIFICATION PROCESS, JANUARY 2011 3
identifies unexpected changes to existing processes introduced
by the new application. For mission critical applications the
importance of this testing cannot be overstated.
Figure 2. Software Testing Phases [2]
The V-Model of testing identifies five software testing
phases, each with a certain type of test associated with it. Each
Figure 1. V-Model of Software Testing [2]
testing phase and each individual test should have specific
entry criteria that must be met before testing can begin and
specific exit criteria that must be met before the test or phase
right depicting the basic sequence of development and testing
can be certified as successfull. The entry and exit criteria are
activities.
defined by the Test Coordinators and listed in the Test Plan.
The given V-Model figure 1 description given below.
Then after successfull completion of test phases and testing
1) Unit testing: A series of stand-alone tests are conducted procedures, the certification procedure comes in to existence
during Unit Testing. Each test examines an individual com- which is then quite simple because it depends on successfull
ponent that is new or has been modified. A unit test is also testing processes.
called a module test because it tests the individual units of So we can say that sucessfull testing process lays the founda-
code that comprise the application. tion for successfull certification process.
2) System testing: System Testing tests all components and
modules that are new or changed and are needed to form a
III. P RINCIPAL C ERTIFICATION P ROBLEMS
complete system. The system test may require involvement of
other systems but it should be minimum to reduce the risk of This section highlights some problems that arise during
externally-induced problems. The emphasis in system testing certification process.
is validating and verifying the functional design specification
and seeing how all the modules work together. A. Reliability Certification Problem [4]
3) Integration testing: Like system testing, integration test- There has been a phenomenal growth in the utilization of
ing also tests all the components that are new or changed and object-oriented technology for developing software systems
are needed to form a complete system, but it also requires in last decades. It’s popularity is due to its support for data
involvement of other systems and interfaces with other appli- abstraction, information hiding, extensional programming and
cations including those owned by an outside vendor, external in particular reusable software.
partners, or the customer. It has various subtypes- It’s major merit is its flexibity to produce reusable modules,
• Compatibility Testing - It ensures that application works but a thing to be kept in mind is that components are often
with differently configured systems. not reused if their reliability cannot be guaranteed. Therefore
• Performance Testing - Performance tests are used to eval- it is essential to realize that reliability certification is a must
uate and understand the application’s scalability when, when discussing reusability, but the problem is that no specific
for example, more users are added or the volume of data test methods are prescribed by current object-oriented/based
increases. development methods. This problem has just been of major
• Stress Testing - Stress Testing is performance testing at interest as it has been realized that object orientation in itself
higher than normal simulated loads. Stressing runs the is not sufficient to create high quality software.
system or application beyond the limits of its specified For testing there are two major alternatives:
requirements to determine the load under which it fails • Black Box Testing - It takes an external view of the
and how it fails. system and test cases are generated without knowledge
4) User Acceptance Testing: It is also callad as Beta testing of the interior of the system.
or end user testing. It is where, testing moves from the hands • White box testing - It aims at covering paths in the
of the IT department into those of the business users who then code or all lines in the code or maximising some other
perform the real world testing. coverage measure.
5) Production Verification Testing: Production verification The main objective of most testing techniques is to validate
testing is a final opportunity to determine if the software that the system fulfills the requirements, the focus is mostly
is ready for release. As a sort of full dress rehearsal, it on functional requirements whereas reliability certification

4. ROLE OF TESTING IN CERTIFICATION PROCESS, JANUARY 2011 4
focuses on detecting the faults that cause the most frequent
failures, hence maximising the growth in reliability.
So often there are problems with providing reliability certi-
fication to a product that is intended to be reused even after
performing modern testing techniques.
B. Verification Methods Drawbacks [5]
Software verification approaches fall mainly in two groups:
• Dynamic
• Static
Almost every existing standard uses these two approaches to
verify software as a part of safety analysis. However these
verification techniques are not advanced enough in relation to Figure 3. DO178B V & V Process [6]
the safety integrity levels needed for the software.
Formal verification techniques also have some major draw-
backs.They are not entirely practical.For example, showing life cycle activities as appropriate.
consistency between requirements and code does not ensure Verification of coding and integration process involves review
confidence in safety since most of safety problems originates and testing of the source code implemented as per the Software
from flaws in requirements. Another drawback is the feasi- Design Data. The review comments and errors identified from
bility, since few formal verifications applied to real programs this process are fed back to previous life cycle activities as
require massive effort for relatively small software. appropriate.
Only practical demonstrations can validate the usability of Verification of integration process involves testing of the object
some of the verification methods and we know that these two code on Instruction Set Simulator/ Target Emulator, Target
verification techniques directly lead to certification process, board for compliance. The test results from this process are
so often there are problems with certification due to these fed back to previous life cycle activities as appropriate. In
drawbacks. general all errors that are reported are managed and tracked
to closure.
IV. R EQUIREMENTS FOR T ESTING BASED ON S TANDARDS And at last Software Verification Cases and Procedures as
This section highlights requirements with regard to testing well as the Software Verification Results are verified for
for some basic standards used all over the world in different completeness and correctness in the Verification of Verification
fields. Process Results.
Every software has different requirements for testing depend-
ing on its own field.
B. Biomedical Engineering: IEC 60601-1-4 [7]
A. Aerospace: RTCA/DO-178B (EUROCAE ED12B) [6] It is a safety standard used in the field of Biomedical
It is an international standard relating to the safety and Engineering. Medical devices go through many stages of
airworthiness of software for avionics. During planning stage product testing before they are available to customers.
of this standard various requirements related to testing in this Verification and validation activities are repetitive,
standard are mentioned in Software verification plan (SVP). comprehensive, well planned and documented. Outputs
DO-178B distinguishes between testing and verification. are tested against design specifications.
Testing - Process of testing per segment,does not ensure
absence of errors. The Figure 4 explains the verfication and validation process
Verification - Verification on the other hand is a generic term for Biomedical standards.
for activities like Reviews, Analyses, and Testing. 1) Hardware Verification: Safety and Electromagnetic
The figure 3 represents the Verification Process as required Compatibility (EMC) are two areas of focus during medical
by DO-178B and indicates the verification activities at the end device hardware verification including functional and perfor-
of each of the processes like Requirement Process, Design mance testing, simulations, visual inspections, worst case/fault
Process, Coding and Software Integration Process and the tree analysis. All of the verification tests must be applied to
Hardware Integration Process. all possible product variations.
Verification of requirement process involves review/analysis of 2) Software verification: International requirements defined
Software Requirement Data. The review comments from this in IEC 60601-1-4 specifies the design controls that must be
process are fed back to the previous life cycle activities as in place for medical software development. The software’s
appropriate. actual performance must meet the corresponding design-input
Verification of Design process involves review and analysis of document’s requirements. For most time it is a manual pro-
the design that is provided in the Software Design Data. The cess and software developers maintain spreadsheets that link
review comments from this process are fed back to previous verification test results to the associated requirements.

5. ROLE OF TESTING IN CERTIFICATION PROCESS, JANUARY 2011 5
V. C ONCLUSION
Software Testing has the potential to save time and money
by identifying problems early and to improve customer satis-
faction and safety by delivering a more error free product.
Although Software Testing process is a crucial element in
the development of embedded system, it plays more sub-
stantial role in highly safety critical systems for supporting
certification process. In conclusion, to make optimum use of
software testing, the verification phase should be initialized
from the very first preliminary requirements analysis stage
where requirements analysis and design reviews are done till
the concluding stage where functional testing and environment
modelling is finally done.
Testing according to the new development methodoligies like
object oriented development methods and some other modern
softwares that come in group of those safety critical embedded
systems which are intended to be used in different domains,
have to be improved or we can say that be more intense and
Figure 4. Design Control Model [7] specific to ensure more safety than it ever did.For example,
in case of reliability certification because companies cannot
reuse software components without knowing how reliable
3) System validation: After the software and hardware
they are, verification and validation thus should be performed
designs pass verification activities, validation of the integrated
on different modules or objects separately to assure their
systems begins. To ensure that a medical device complies with
reliability level.
the governing rules and regulations of medical device software
Todays world is revolutionized by modern technology which
validation, validation engineers constantly add, update, and
emphasizes the human life to a great extent. A well planned,
develop more detailed and quality-driven validation procedures
comprehensive and documented requirements for verification
to increase the probability of finding errors.
and validation process is implemented to make the safety
C. Generic Standard: IEC 61508 [8] critical systems like Aviation and Medical more reliable and
trustworthy before they are brought into use.
IEC-61508 is an international standard of rules applied in I presume that with implementing and adhering new technolo-
industry and titles as "Functional safety of programmable gies , the requirements for verification and validation should
electronic safety-related systems". AdaTEST95 and Cantata++ be proactively modernized for advancement of the product and
are used to meet the verification and testing requirements of customer satisfaction.
the standard.
1) Overview of Requirements: This standard has 7 parts
ACKNOWLEDGEMENTS
each identifying different processes and the parts which iden-
tify verification, validation and test requirements relevant to I would like to specially thank Prof. Dr. Matthias Wagner
AdaTEST95 and Cantata++ are: and my fellow colleagues for continuously supporting and
• IEC 61508 Part 3: Software Requirements being continuously a source of motivation for me.
• IEC 61508 Part 7: Over View of Techniques and Mea-
sures R EFERENCES
AdaTEST95 and Cantata++ is centered on a dynamic test [1] Verification/Validation/Certification, Carnegie Mellon University, 18-
harness. The test harness can be used to support testing at all 849b Dependable Embedded Systems, Spring 1999, Author: Eushiuan
Tran
levels from module testing through to full integration testing. [2] Software Testing Fundamentals—Concepts, Roles, and Terminology,
IEC 61508 correctly demands planning of dynamic testing as John E. Bentley, Wachovia Bank, Charlotte NC
part of design process for each software object.The product’s [3] Software Testing, A guide to the TMAP Approach, Martin Pol, Ruud
Teunissen, Erik van Veenendaal
structured test scripts can be used as Software Design Test [4] Reliability Certification of Software Components, Claes Wohlin and
Specifications and Software Module Test Specifications as they Björn Regnel, Department of Communication Systems, Lund Institute
are easily readable by quality assurance staff. of Technology, Lund University, Box 118, SE-221 00 Lund, Sweden.
[5] Software Safety Certification: A Multi-domain Problem, Patricia
AdaTEST 95 and Cantata++ are well suited to the develop- Rodriguez-Dapena, European Space Agency, Copyrighted IEEE
ment of the software and facilitate a high degree of automation [6] Applying DO178B for IV & V of Safety critical Software, White Paper,
of the verification and test techniques required for effective use Sreekumar Panicker, Wipro Technologies
[7] Testing Medical Devices, Written by Gary Powalisz, GE
of the standard. Healthcare Available: http://www.evaluationengineering.com/index.
AdaTEST 95 and Cantata++ have been developed to the php/solutions/instrumentation/testing-medical-devices.html. Last access
highest practical standard for software verifcation tools and on:11/01/2011
[8] IPL Testing Tools and IEC 61508, IPL Information Processing Ltd.,
provide comprehensive functionality and they are the only Eveleigh House, Grove Street, UK
tools developed to such high standards.