This document discusses how to make test case results more valuable by assigning standardized attributes to test cases. It proposes assigning attributes for priority based on defect severity, stakeholder importance, and likelihood of finding defects. Additional attributes could specify which software quality areas the test case covers, such as functionality, reliability, efficiency. Assigning these attributes allows test results to be analyzed and aggregated in more meaningful ways, such as showing which areas had failures and which priorities had the most failures. This provides more insight than just a pass rate and better informs stakeholders.

1. Empowering the Test Case

2. Introduction
 A pass rate is an abstraction of the actual result of a set of test cases
 This abstraction is done to facilitate for stakeholders
 A 50 % pass rate says that half the test cases passed – but what value does this
information actually hold?
 I most cases, not much!
 We have gone to far in our abstraction and the information now holds almost no real
value to our stakeholders, even if they might still think so

3. Test Case Attributes
 How can the result of a set of test cases be more valuable?
 By assigning standardized attributes to test cases, the value of the test result will be
much higher
 The test case will provide more information that can be aggregated when generating the
test report to the stakeholders
 What information depends on the attributes – this presentation outlines one set of
attributes

4. Test Case Attributes Overview
Software
Quality
Attributes
Test
Case
Priority

5. Priority
 Each test case is assigned a priority based on several factors
 What severity will a defect found by the test case most likely have?
 How important is the functionality to stakeholders?
 With what frequency will the test case most likely find defects?
 There can be many factors that set the priority
 The priority can be a scale between 1 and a number which is set based on the
granularity that is deemed to be needed by stakeholders

6. Software Quality Attributes
 By listing what sort of software quality attribute the test case covers the test result will
reveal much more detail
 You can use any set of software quality attributes
 IEC 9126-1 is one example of software quality attributes
Functionality
Portability Maintainability
Reliability Efficiency
Usability

7. Test Suite Overview
Test Case A Test Case B Test Case C
Performance Functionality Efficiency
Priority 1 Priority 3 Priority 1
Test Case D Test Case E Test Case F
Performance Functionality Functionality
Priority 2 Priority 2 Priority 1

8. Test Results without Test Case Attributes
Test Results
100%
90%
80%
70%
60%
Blocked
50%
Failed
40%
Passed
30%
20%
10%
0%
Pass Rate

9. Test Results with Software Quality Attributes
100%
90%
80%
70%
60%
50% Blocked
40%
30% Failed
20% Passed
10%
0%

10. Test Results with Priority
100%
90%
80%
70%
60% Blocked
50%
Failed
40%
30% Passed
20%
10%
0%
Priority Priority Priority
1 2 3

11. Test Case Attributes Added Value
 Imagine that the three above charts show the same test results
 Initial indications show that 85 % are passed – Quite good results
 The second chart however shows that almost all of the failed test cases are located in
the reliability area – This can for example indicate that we have severe stability
problems
 The third chart shows that almost all of our failed test cases have high priority – also an
increased risk
 Even if the initial indications were positive, analyzing the test results with test case
attributes further reveals more information and additional risks

12. Conclusion
 By adding test case attributes to the test cases, the test result can hold much more
value to the stakeholder
 It can also facilitate for the tester during scope selection
 The cost is very small, and the potential gain can be very large if stakeholders base
critical decisions on the test results
 What test case attributes to add depends on stakeholder needs – what information can
be valuable to the stakeholders?

13. Reference
IEC 9126
http://en.wikipedia.org/wiki/ISO/IEC_9126