Better Managed Outsourced Testing
by Nathan Shearer
Introduction
This guide focuses on three key failure points:
1) misunderstanding of the cost of quality;
2) misalignment of contractual arrangements to the required testing/delivery outcomes; and
3) lack of effective metrics.
Conclusion
Organisations failing to fully understand their outsourced contracts and not including effective metrics and role accountabilities to monitor these often see cost overruns and project delays. There can also be reduced ROI against the outsourced agreement.
Although the main driver for outsourced testing is often cost reduction, relying on a contractual arrangement to drive or manage this outcome often leads to a unit cost for testing KPI, instead of a metric based on total cost for quality. As KPIs drive behaviour, the organisation will be left with statistics to prove the KPI was met, but a less than optimal quality outcome. The outsourced testing provider’s approach to this situation would be to involve a professional, external organisation to perform a capability review. Recommended improvements or enhancements to the organisation’s governance of the commercial arrangement and the testing outcomes would be suggested, together with changes to the metrics used to report against the testing outcomes.
Outsourced testing can be successful. It can speed up the test execution, improve quality and reduce costs. It is important to ensure the total cost of quality is understood, the quality outcomes are defined and aligned to the commercial agreement. Appropriate metrics should also be created, recorded and measured by both the provider and the client.
2. Introduction
This presentation will highlight three key failure points to outsourced testing:
1. misunderstanding of the cost of quality;
2. misalignment of contractual arrangements to the required testing/delivery
outcomes; and
3. lack of effective metrics.
4. ..focus on the test execution
statistics and not quality as an
objective.
5. Is there a solution?
It is important to understand these issues can be mitigated and rectified by
examining current testing capability and improving the testing processes, not
by just policing the fine print in the outsourcing contract.
7. Symptoms
Along with the typical less than
optimal outcomes (e.g. poor quality,
cost overruns and late delivery), other
potential symptoms of poorly
managed outsourced testing include:
15. Organisations should “Understand that testing is required to verify and validate that a business objective is being
realised... throughout the life cycle, not only at a user acceptance phase. Therefore, clear functional and non-functional
acceptance criteria should be defined for each phase of the life cycle.”
Gartner: ID:G00225707, ID:G00209445
17. Misaligned
contractual
arrangements
Often the commercial agreement is
the only governance provided to
measure, manage, report and maintain
the output of the outsourced testing
provider.
Different Outsourced Engagement Models
18. The true value of an outsource
provider is meeting the client’s
delivery need with the capability
(and ability) to deliver
19. Lack of Effective Metrics
If an outsource testing provider is bound by unit cost measures like total
defects found, the onus is on finding the maximum number of defects
(regardless of quality of defects, analysis, triage, allocation, etc.) in the allotted
period of time.
20. Common Results
Typically, the outcome will be an unquantified list of unqualified defects, test
results with a high failure or blockage rate, and no working solution.
The outsource provider may then be rewarded with more resources to come up
with another outcome (reduce the number of defects or pass tests), giving the
provider more revenue and leaving the client with cost and time overruns.
21. Organisations “have limited internal baseline metrics regarding quality and efficiency of their testing capabilities.”
Gartner: ID:G00225707, ID:G00209445
23. 1: A Capability Review
Assessing the current testing capability may be as simple as reviewing past
test results against estimated and planned outcomes. It could also involve a
professional external organisation performing a capability review and auditing
policies, processes, procedures and skills.
24. What is a Capability Review
This review can be used to recommend
improvements to specific areas. These could
include a new or updated test strategy (or
master test plan), to ensure the alignment to
desired outcomes, the introduction of metrics to
align to KPIs in the outsource contract, and
defining effective (light touch) governance
around roles and responsibilities to reach the
desired outcomes. Capability Review
25. How and What to Measure?
relative test effort;
test coverage of requirements;
percentage of tests automated;
defects remaining upon completion, and
defect analysis.
These metrics should be holistically called the cost of quality.
26. The cost of defects increases
throughout the testing lifecycle
and the longer a defect stays
within the process, the more
costly it is to fix.
27. Cost of Defects
To avoid cost overruns, consider using
defect classification metrics to more
easily identify trends in your test
process
Cost of Fixing Software Defects
28. To avoid cost overruns, consider
using defect classification
metrics to more easily identify
trends in your test process.
29. Defect reporting vs. defect analysis
Clients may already have effective defect reporting which can be invaluable for
providing a clear view of the current status of quality.
The move to defect analysis begins with reporting as an input, combined with
additional quantitative information to allow analysis of what is called phase
effectiveness.
30. There are two key metrics that
assist with understanding trends
in quality:
31. (PCE) Phase Containment Effectiveness
PCE is the ratio of faults captured in a phase and signifies how effective the
testing process is at preventing these defects in the first place.
32. (PCE) Phase Containment Effectiveness
In Figure 4, the PCE is calculated as follows:
Requirements captured 10 of 22 defects = 45%
Design contained seven of 22 defects = 32%
Build contained three of 22 defects = 14%
Test contained two of 22 defects = 9%
Phase Containment Effectiveness
33. (PSE) Phase Screening Effectiveness
PSE is the ratio of previously missed (not
contained) defects being captured in each
phase.
For the same example, the PSE is calculated as:
Requirements missed 12 of 22 defects
Design screened seven of these 12 defects = 58%
Build screened three of these 5 defects = 60%
Test screened two of the remaining two defects = 100%
34. Create or update a testing
framework, which will fully align
these metrics to the total cost of
quality.
35. Organisations failing to fully
understand their outsourced
contracts and not including
effective metrics and role
accountabilities to monitor these
often see cost overruns and
project delays.
37. 3 Simple Steps to Rectify
Outsourced testing can be successful.
It can speed up the test execution, improve quality and reduce costs.
It is important to ensure the total cost of quality is understood, the quality
outcomes are defined and aligned to the commercial agreement.
39. 2: Outsourced Agreement Review
Recommend improvements or enhancements to the organisation’s governance
of the commercial arrangement and the expected testing outcomes
40. 3: Improved Metrics
Together with changes to the metrics for both the outsourced provider and the
client organisation should be used to report against the testing outcomes
41. Nathan Shearer - Want to work with me? rapidexecutive.com/coaching
Like This?
Feel Free to Share it with
someone who could benefit
→
Connect with me on LinkedIn
Connect with me on Twitter