The history of testing standards has not always been auspicious. Testing standards documents have been expensive to obtain, limited in scope, inflexible in expectations, and inconsistent. However, they contain important lessons learned from experienced practitioners—if a tester is willing to overcome the obstacles to get to the useful information. A set of new international standards is coming. These new standards are tailorable, consistent, and comprehensive in scope. In addition, they will be freely available (some are already). Claire Lohr provides a complete roadmap to all of the available—or soon-to-be-available—testing-related standards. Learn where to go for testing process guidelines, complete definitions of all test design techniques, full examples of test documentation (for both agile and traditional projects), and free international standards documents. Take away a “start-up guide” for how different types of projects can use the new standards along with valuable tips and practical lessons you can get from these standards.

1. T24
Special Topics
5/2/2013 3:00:00 PM
New Testing Standards Are on the
Horizon: What Will Be Their Impact?
Presented by:
Claire Lohr
Lohr Systems
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Claire Lohr
Claire Lohr has been an active professional in the computer field for thirty years, with the last twenty years
emphasizing software process improvement and testing. Claire provides training - design, authoring, and
instruction - and consulting services for a wide variety of both government and commercial clients. She is
a Lloyd’s Register trained ISO 9000 Lead Auditor and has been trained to perform Software Capability
Evaluations for the SW-CMM. Claire was the chair of the Working Group for the IEEE Std 829-2008
Software and System Test Documentation, and has served on both the IEEE Computer Society’s
Software and Systems Engineering Standards Committee and the IEEE Computer Society’s Standards
Advisory Board.

3. 4/16/2013
New Testing Standards on the
Horizon: What Will Be
Their Impact?
Claire L. Lohr
Lohr Systems
P.O. Box 2998
Reston, VA 20195
703.472.5457
clohr@computer.org
1
Topics
•
•
•
•
•
•
•
•
Why bother with standards?
Traditional objections
What’s available (free first)
Impact of ISO 29119
Examples
How to start
Additional resources
?’s
2
1

4. 4/16/2013
Why bother with standards?
• Common problems have common
solutions
• Standards are reviewed and modified
by groups of very experienced ($$$$)
practitioners
• You can’t search the Internet for
information unless you know what it is
commonly called (or if you have never
heard of it)
3
Traditional Objections
Past
Inconsistent and
incomplete
Overkill
Present
Merging together
Includes high integrity
and lower integrity
choices
Just theoretical
Includes examples
Expensive
Increasingly free
Time consuming
ROI of 1:5-6
4
2

5. 4/16/2013
What’s available: free
1. nist.gov
Special Publication 800-115 Technical
Guide to Information Security Testing
and Assessment
2. open-stand.org
Commitment for the future by IEEE, IETF, IAB,
Internet Society and W3C.
5
What’s available: free
3. Certification bodies of knowledge
• swebok.org
Summary of testing (& rest
of SW engineering)
• pmi.org
The Software Extension to
the PMBOK® Guide
• istqb.org
Glossary of testing terms &
multiple tester certifications
• buildsecurityin.us-cert.gov Software
Assurance (SwA) Competency Model
6
3

6. 4/16/2013
What’s available: free
4. Other free resources
• http://pascal.computer.org/sev_display/ind
ex.action
SEVOCAB definitions
• http://standards.iso.org/ittf/PubliclyAvailabl
eStandards/index.html free ISO standards
7
What’s available: free (example)
SWEBOK testing section
Fundamentals
Levels
Techniques
Measures
Process
Tools
8
4

7. 4/16/2013
What’s available: not free
7925-1
Vocabulary
7925-2
Component
testing
12207 Software life
cycle
15026 Integrity levels
25010 Quality
requirements
25051 Quality reqm’ts
for COTS
829 Doc’s
1008 Unit test
1012 V&V
1028 Reviews
1044 Defect
categories
9
What’s coming
29119 Systems and software
engineering—Software
testing—
• Part 1: Concepts and
definitions
• Part 2: Test processes
• Part 3: Test documentation
• Part 4: Test techniques
10
5

8. 4/16/2013
ISO 29119-1 Concepts and
definitions
4. Definitions
5. Concepts
6. Approaches
7. Automation
8. Defects
11
ISO 29119-2 Test processes
6. Organizational
7. Management
8. Dynamic
8.1 Introduction
8.2 Design & implementation
8.3 Environment setup & maintenance
8.4 Incident reporting
12
6

9. 4/16/2013
ISO 29119-3 Test documentation
5. Organizational
5.2 Policy NEW
5.3 Strategy
6. Management
6.2 Plan
6.3 Status Report
6.4 Completion Report
13
ISO 29119-3 Test documentation
7. Dynamic
7.2 Design
7.3 Case
7.4 Procedure
7.5 Data requirements
7.6 Environment requirements
7.7 Data readiness report
7.8 Environment readiness report
7.9 Actual result
7.10 Test Result
7.11 Execution Log
7.12 Incident report
NEW
NEW
NEW
NEW
14
7

10. 4/16/2013
ISO 29119-4 Test techniques
5.2
•
•
•
•
•
•
•
•
•
•
•
•
Equivalence partitioning
Classification tree
Boundary value analysis
Syntax testing
Combinatorial
Decision table
Cause-effect graphing
State transition
Use case
Scenario
Error guessing
Random
15
ISO 29119-4 Test techniques
5.3
•
•
•
•
•
Statement
Branch
Decision
Condition
Data flow
D
6. Coverage
A
B
Annex B Specification-based
examples
E
Annex C Structure-based
examples
16
8

11. 4/16/2013
Impact of ISO 29119
Past
Future
Inconsistent and
incomplete
Overkill
Just theoretical
Expensive
Time consuming
Consistent and complete
in one place
Includes high integrity
and lower integrity
choices
Includes examples
Increasingly free
ROI of 1:5-6
17
Example #1
I want to improve our
testing processes
29119-3 5.3 Strategy (for focus)
Test
Manager
•
•
•
•
•
•
Risk management
Test selection and prioritization
Test documentation and reporting
Test automation and tools
Configuration management
Incident management
18
9

12. 4/16/2013
Example #1 (test manager)
29119-2 8.3 Incident reporting process
a) Test results are analyzed
b) New incidents are confirmed, if any
c) New incident report details are created
d) The status and details of previouslyraised incidents are determined
e) Previously-raised incident report
details are updated as appropriate
f) New and/or updated incident reports are
communicated to the relevant
stakeholders
19
Example #1 (test manager)
29119-3 7.12 Incident report
1.
2.
3.
4.
5.
6.
7.
8.
Timing information
Add related
Originator
incidents
Context
Description of the incident
Originator’s assessment of severity
Originator’s assessment of priority
Risk
Status of the incident
20
10

13. 4/16/2013
Example #2
I want to document better
so I can get another job
Tester
29119-2 Annex A Example
Test Design*
29119-3 Annex H Example
Test Design
* Chose this one – more
complete and more
summary
21
How to start
Suggested steps for transition
1. Survey your current baseline
2. Choose most valuable “next steps”
(Kaizen)
3. Implement a pilot
4. Share what works
5. Continue to improve
22
11

14. 4/16/2013
Additional Resources
www.iso.org
www.ieee.org
www. global.ihs.com
The Roadmap to Software Engineering
Standards: a Standards Based Guide,
James W. Moore, 2006
23
Questions?
24
12

15. 4/16/2013
The End
25
13