Details approaches to software testing, as presented at a Twilight session at Intergen.

1. Producing Testable Requirements
Piers Chamberlain piersc@intergen.co.nz &
Robert Nugent robertn@intergen.co.nz

2. Producing Testable Requirements
Requirements Inspection
THE PROBLEM
What the project team built wasn’t what I asked for…and I can’t believe that no-one
picked up these issues before I started looking at the system myself!
…but was what you asked for what you meant!
1. Natural language is very poor for communicating precise detail.
2. Human beings are so good at “interpreting” language that we make assumptions
without thinking.

3. Producing Testable Requirements
Quality Requirements
The idea is to get your specification as ship-shape as possible.
Feasible Scope
Necessary
Correct
Complete Function
Consistent
Verifiable Testability
Modifiable
Traceable Maintenance
Prioritized

4. Producing Testable Requirements
What is a Testable Requirement
Testable (in principal) means
Precise
Unambiguous
Absolute state – The requirement is either met or not met
…but it needs not be tested – can be either untested or indirectly tested
Indirectly testable in practice
Cost or risk reasons may prevent “real” testing
Simulations, other tests that can provide some confidence

5. Producing Testable Requirements
Validation through Test Design
Define a set of test cases to prove a requirement in
principal
If you have a large number of tests (> 5 or 6) the requirement is
probably too vague
Especially if the tests are of different types
Test Design Rules
You can assume any other requirements in the spec are met
You cannot assume anything not in the spec

6. Producing Testable Requirements
An Example
Change Password
The system will allow a user to change both their
password and username, but the user needs to have
logged in to do this. On updating, the old credentials will
be discarded and the user's new one activated. The rules
for password / email validation remain the same.

7. Producing Testable Requirements
Ambiguous…
Change Password
The system will allow a user to change both their
password and username, but the user needs to have
logged in to do this. On updating, the old credentials will
be discarded and the user's new one activated. The rules
for password / email validation remain the same.

8. Producing Testable Requirements
Inconsistent…
Change Password
The system will allow a user to change both their
password and username, but the user needs to have
logged in to do this. On updating, the old credentials will
be discarded and the user's new one activated. The rules
for password / email validation remain the same.

9. Producing Testable Requirements
Imprecise…
Change Password
The system will allow a user to change both their
password and username, but the user needs to have
logged in to do this. On updating, the old credentials will
be discarded and the user's new one activated. The rules
for password / email validation remain the same.

10. Producing Testable Requirements
Incomplete…
Change Password
The system will allow a user to change both their
password and username, but the user needs to have
logged in to do this. On updating, the old credentials will
be discarded and the user's new one activated. The rules
for password / email validation remain the same.

11. Producing Testable Requirements
Let‟s assume some validation rules…
The system will validate a user's password
The password must be at least 6 and no more than 40 characters
The password can consist only of standard alphanumeric characters (i.e. a-z, A-Z and
0-9)
Until the password is validated, it cannot be saved or used to authenticate a user.
The system will validate a user's username
The system will validate that the username conforms to the RFC 2822 standard as an
email address
The system will verify that the username is valid by sending an activation code as a
hyperlink to that destination
The username is accepted as valid if the user clicks the activation code sent
Until the username is activated, it cannot be used to authenticate a user.

12. Producing Testable Requirements
Version 2
1. Change Password
1.1 A user must be currently authenticated to change their password
1.1.1 The new password must meet the standard password validation rules*
1.2. A user must be currently authenticated to change their username
1.2.1 The new username is subject to the standard username validation rules*
2. Authentication
2.1 To authenticate a user must supply a valid username and password credentials
at the login screen
2.1.2 A user‟s authentication is expired after 30 minutes
2.1.3 A user‟s authentication is expired immediately if they have elected to logout

13. Producing Testable Requirements
Why it works
It forces a user-centric view of how the system should
hang together
Introduces workflow, temporal elements
Checks for consistency and completeness
Can highlight design considerations where they are either
required (and missing) or unnecessary
Can provide behavioural and psychological insights

14. Producing Testable Requirements
Guidelines for Quality Requirements
Keep sentences short and break up paragraphs
Use the active voice
Supply all required detail
Granularity – The “smaller number of tests” principle
Avoid aggregation (conjunctions like “AND” and “OR” can
suggest combined requirements)
Write at a consistent level of detail throughout the
document

15. Producing Testable Requirements
Requirement in good shape … now
what am I prioritising
Requirements (source of the tests)
At the same level of extraction or type
Need to determine a measure to use (I am using
Risk)

16. Producing Testable Requirements
Why Assess the Risks
Not all requirements are the same
I can choice what to test first
Testing duration is NOT guaranteed
How can I conduct an assessment
Need to determine:
The source of requirements
What method
What criteria to use
Who should be involved

17. Producing Testable Requirements
Example – Methods*
RISK
Usage frequency Quality
Probability of failure
Damage Functional Volume
Damage / Loss of Use
(cost of failure)
Example – Criteria
Changed areas
Critical area New component
Visible area Impacted user base
Usage frequency Deadlines
Complex areas Defect history
*Risk Based Testing, Strategies for Prioritising Tests against Deadlines Hans Schaefer.

18. Producing Testable Requirements
Example – risk calc spreadsheet
Risk calculation schema
Impact factors Probability factors Prob. of defect detection
visibility user impact frequency deadline pressure new people complexity
weights weights Risk
Risks addressed 1 3 10 1 3 3
function A 1 1 2 1 5 5 1 744
performance 0 1 1 1 1 4 1 208
usability of x 5 1 2 3 1 5 1 588
… 2 2 2 2 2 5 1 644
… 1 1 1 1 1 2 1 140
1 2 1 2 1 1 1 136

19. Producing Testable Requirements
Prioritising the Requirements
By Requirement Type
Must Have, Should Have, Nice to Have (I Wish)
Core Discretionary
1 to 100
From two factor risk assessment
Clustering First Helps
By Function
By Functional Area
Core vs. Auxiliary
By sprint, cycle, release
By Architectural layer

20. Producing Testable Requirements
Example –using risk spreadsheet
Risk calculation schema
Impact factors Probability factors Prob. of defect detection
visibility user impact frequency deadline pressure new people complexity
weights weights Risk
Risks addressed 31 3 10 1 3 3
function A 1 1 2 1 5 5 1 744
Performance profile of … 0 1 1 1 1 4 1 208
usability of x 5 2 1 2 3 1 5 1 588
UC213.1 2 2 2 2 2 5 1 644
Password must have … 1 1 1 1 1 2 1
1
140
… 1 2 1 2 1 1 1 136
4

21. Producing Testable Requirements
Prioritising Wrap up
To successfully prioritise requirements you need to:
Determine Risk assessment criteria
Use one or two factor method
Weight the results
Rank by chosen method
And you will have:
Objective view of project deliverables (the Risk if not delivered)
Assistance in test scope analysis
Prioritising of test effort

22. Producing Testable Requirements
Other Benefits
From Requirements Inspection
Higher First Release quality
Lower Cost of software maintenance
Improved team dynamics / education
From RiskPrioritisation
Objective estimates
Defect clustering analysis
Defect resolution order
Business perspective progress reporting

23. Producing Testable Requirements
Some Simple First Steps
Evaluate Requirements
Review for Testability
Conduct a risk assessment
Must have, Should have or Could have
Prioritise the Results
Group the requirements
Test this one first … to test this one last

24. Producing Testable Requirements
Questions & Discussion

25. Producing Testable Requirements
References & Further Reading
Generating Testable Requirements
Methodology for Writing High Quality Requirement Specifications Linda Rosenberg
September 24 1998
Writing Quality Requirements by Karl E. Wiegers
Writing Testable and Code-able Requirements Murat Guvenc / Borland
On-Track Requirements by Rodger Drabick Better Software Magazine May/Jun 1999
(Vol. 1, Issue 3)

26. Producing Testable Requirements
References & Further Reading
Risk-Based Prioritisation
Requirement Ranking – NO, No, No to High, Medium or Low Posted April 25th, 2008 by Steven Davis
Risk-Based Quality Management Posted April 24th, 2008 by Tracy Lynne Dedore
Prioritizing Requirements Posted April 9th, 2008 by Patrick Walsh
Theory and Practice of Risk-based Testing by Felix Redmill
Published in Software Testing, Verification and Reliability, Vol. 15, No. 1, March 2005
*Risk Based Testing, Strategies for Prioritizing Tests against Deadlines by Hans Schaefer,
Published in Methods & Tools Volume 13 number 4, Winter 2005.
Exploring Risk-based Testing and Its Implications by Felix Redmill
Published in Software Testing, Verification and Reliability, Vol. 14, No. 1, March 2004
Risk-based Test Planning During System Development by Felix Redmill
Invited paper at KKIO 2004, Gdansk, Poland, 5-8 October 2004
*Risk Based Testing and Metrics by Stale Amland, presented at EuroSTAR „99, November 8 – 12,
1999 Barcelona, Spain