Benefits of having a specialist Tester over a Generalist Tester.

1. Disclaimer: All the documentation and other material contained herein is the property of Thinksoft Global Services and all intellectual
property rights in and to the same are owned by Thinksoft Global Services. You shall not, unless previously authorized by Thinksoft
Global Services in writing, copy, reproduce, market, license, lease or in any other way, dispose of, or utilize for profit, or exercise any
ownership rights over the same. In no event, unless required by applicable law or agreed to in writing, shall Thinksoft Global Services,
or any person be liable for any loss, expense or damage, of any type or nature arising out of the use of, or inability to use any material
contained herein. Any such material is provided “as is”, without warranty of any type or nature, either express or implied. All names,
logos are used for identification purposes only and are trademarks or registered trademarks of their respective companies.
For more details visit, www.thinksoftglobal.com
Is Software Testing a “Zero Sum Game”– Yes or No? Case Studies
Initiative Yea $ Cost % reduction
Initial state 0 38000 0%
RPM repository 1-p1 32300 15%
consolidation
Automation 1-p2 19380 25%
Productivity 2 10659 5%
improvement
Productivity 3 5330 5%
improvement
Total 50%
Using this framework, the specialist testing partner:
• Created Centers of Excellence around the cards business and
related test process
• Built and organized structured test repositories
• Designed Test Automation frameworks custom made to the
cards business
• Implemented Risk Prioritization Matrices to ‘Triage’ the most
business critical functionalities that are most likely to fail
• Developed tools to integrate the repositories and automation
frameworks
Net result - a sustainable savings of 50% on annual regression
test costs - $1.2 m!
Epilogue:
From Dr Robert Harris’ presentation it was clear that
collaborating with a reliable and worthy partner with the right set
of capabilities is very critical to success!
What was not made explicitly clear at his presentation was that
in both cases the reliable and worthy testing partner was none
other than Thinksoft Global Services Ltd.
T r u s t t h e E x p e r t s
IS SOFTWARE TESTING A
“ZERO SUM GAME”– YES OR NO?
Case Studies
The Context
• How does one optimize management decisions in the software testing arena?
• Are business users to entrust ‘requirement planning’, product fitment, process mapping and customization to the product
developer?
• Should the developer also be entrusted with all interfacing and data migration tasks?
• In parallel, should the business user also build in-house teams of testers and test managers to interface with the product
developer, and do the “acceptance testing” themselves?
• Can the expertise required to sync with ever changing customer needs be developed and retained in-house
• Alternatively should the entire gamut of testing functions be off loaded to specialist testing houses that are springing up,
independent of software product developers?
• Would such independent expert services be more reliable and flexible over time in highly competitive sectors based on
“deep domain “knowledge?
• How do you balance on the one hand the high cost of failure and on the other, the need to control costs to retain margins
under competitive pressures?
These were some of the important questions occupying Dr. Robert Harris, who was to make a presentation at an industry
seminar.
Being an academic he knew the answers in theory, but he still needed to validate his assumptions using “field data”.
The following case studies illustrate his findings.
Case 1
Testing center of excellence for the Treasury arm of a global conglomerate
The Program Manager (PM) of the global conglomerate had to execute a multiyear program to roll out a sophisticated
reporting tool and a data ware-housing engine.
The existing globally reputed system integrator who was expected to manage requirement planning and testing was not
delivering satisfactorily due to obvious limitations in domain experience.
Business requirements were ill articulated and poorly documented. They existed only as spreadsheets or only in the minds of
the business user.
There was a severe shortage of business users to carry out even mandatory tests.
Existing test practices being weak, defective software was leaking into production resulting in significant rework.T r u s t t h e E x p e r t s

2. Is Software Testing a “Zero Sum Game”– Yes or No? Case Studies
What were the Product Manager’s (PM’s) options?
At that juncture, a colleague suggested the name of a reliable
specialist testing house that focused on the financial services
industry from end to end. The PM took the plunge and the results
were immediate with efficiencies improving and costs shrinking
significantly.
The benefits across the first 2 years were:
Quicker Knowledge Transfer
Being industry focused, the testing team was 40% faster than the
earlier generic IT partner in getting up to speed on mapping
processes and applications. This enabled a faster ramp-up and a
quicker user buy-in.
Static Testing reduces avoidable rework
By identifying gaps between data mapping specs and design
specs upfront, close to 900 errors were un-earthed. Almost the
same numbers of defects were identified during the testing phase.
Thus, 50% of total defects, related rework and repair costs were
avoided, even before a single piece of code was written.
Compared to earlier times, development rework fell by 12%.
Testing schedules were compressed by 10%.
Right Shoring
As the testing transitioned from business users to the specialist
partner up to 68% and 74% of the work was off-shored in Y1 and
Y2 respectively, resulting in significant cost savings.
Defect Prevention
As’ Functional defect’ leakage to production reduced to less than
1%, the business experienced a near-defect free application
delivery
Domain Value Adds
Given the short windows for daily validation in production, the
domain focused test team identified business critical fields. The
team also identified a representative data sample covering all
instruments such that the shorter production validation strategy
covered all the business critical areas.
Benefits across Years 3 and 4
As the relationship matured, the specialist partner brought in
benefits from re-usable repositories, automation and productivity
improvement initiatives.
Productivity Growth
While the specialist test team size grew only 30% in 3 years, the
output increased by 143% and 224%
Automation
The creation of a robust automation framework for data
warehousing cum business analytics engine started delivering
savings of about $100,000 for the first application stream and
was expected to save $740,000 as it is implemented across
application streams.
Re-usable repositories
Creating structured repositories resulted in over 69% of the test
cases being re-used. The re-use is bringing in annual savings of
$200k.
Cost-out initiatives
The specialist implemented various cost-out initiatives such as
‘prototype testing’, ‘QA batch load ownership’ and ‘offshore
shifts’ for further cost savings to the tune of $400k per annum.
Case 2
Global Regression Test Center for an Industry leader in Cards
The Vice President Product Engineering (VP-PE) of a globally
renowned cards software maker found that the product QA was
becoming overwhelmingly complex due global custom
implementations and frequent product releases.
The top Issues of concern
• Customer demand for high product flexibility and wide
parameterization.
• Intense competition meant no second chance on new product
introductions.
• Spiraling maintenance costs due to field releases of
in-adequately tested products
• Large customers invoking penalty clauses due to defective
deliveries ($6m for every month of delay!)
On enquiry, one of the clients recommended a specialist testing
house that is focused on the financial services industry.
On engaging the firm the VP-PE found striking differences in
performance!
As he attempted to quantify the benefits from the specialist, he
wondered why he didn’t do this earlier.
The Short and medium term benefits across first 2 years were: -
ZERO Knowledge Transfer
The product developer had no bandwidth for any knowledge
transfer. Yet the specialist successfully ramped up, thanks to their
financial industry focus, knowledge and experience.
Avoiding Penalties in Customer Rollouts
Rigorous ‘cards industry focused’ test methodologies helped the
software maker avoid client penalties (which were in some cases
as high as $6 million for every month’s of delay)
Defect Leakage less than 5%
Defect leakage was either not measured and was 20% in some
cases. It was uniformly brought down to less than 5% thus
improving the quality of applications delivered and reducing the
maintenance costs for the product vendor.
88% right shoring
Building the QA organization, the specialist started off with an
offshore leverage of 83% and raised it to 88% within the first year.
Long term benefits over Years 3 and 4 were: -
With the specialist, continuous improvement was not an
afterthought. To deliver a 50% saving on regression test costs,
the specialist implemented a ‘continuous improvement’
framework
Release # of static # of defects % of defects
# Testing identified Total avoided
Issues in execution (A/C*100)
identified
upfront
(A) (B) C = (A+B)
1 427 592 1019 42%
2 164 90 254 64%
3 24 17 41 59%
4 28 8 36 78%
4.2 245 173 418 59%
All 888 880 1768 50%
Initiative Annual Cost Savings
1 Prototype Testing $50,000
2 QA Batch Load Ownership $60,000
3 Offshore Shifts $40,000
4 Improve offshore leverage $50,000
5 Productivity Improvements $200,000
Result Specialist General Root Cause
Area Testing IT service Analysis
Partner provider
Defects in
Customer
Environment
6% 20% Functional
Coverage in
Testing
Test Strategy
Time to
Market
On time 7 days for
P1 (SLA = 2
days)
Delays in Defect
Resolution
turnaround
Requiremen
ts Coverage
in Testing
>95% Not
Measured,
60%
Lack of
Traceability,
Test Strategy
and Inadequate
coverage
130%
Team
100%
123%
130%
Year 1
Productivity
Year 2
Year 3
Test Created
100%
268%
243%
Executed
100%
230%
324%
243%
324%
percentagegrowth
350%
300%
250%
200%
150%
100%
50%
0%
Domain COE
• Business Processes
• Functional Artifacts
• Tracking of Global
Regulatory Changes
• Process Maintenance
• Process Definition
• Metric Program
• Process Training
• Continuous Process
Improvement
• Best Practice
implementation
Process COE
Competency
Building,
Recruitment,
Training
Repositories
Ready-to-use
Artifacts
Cards
Processor
Competency
Center
Resource
Management
Quality
Assurance
Resource
Management
Automation
using Tools
Metrics &
Reporting
Standard Processes &
Methodologies
Knowledge
Repositories Best Practices
Continuous Improvement Framework

3. Is Software Testing a “Zero Sum Game”– Yes or No? Case Studies
What were the Product Manager’s (PM’s) options?
At that juncture, a colleague suggested the name of a reliable
specialist testing house that focused on the financial services
industry from end to end. The PM took the plunge and the results
were immediate with efficiencies improving and costs shrinking
significantly.
The benefits across the first 2 years were:
Quicker Knowledge Transfer
Being industry focused, the testing team was 40% faster than the
earlier generic IT partner in getting up to speed on mapping
processes and applications. This enabled a faster ramp-up and a
quicker user buy-in.
Static Testing reduces avoidable rework
By identifying gaps between data mapping specs and design
specs upfront, close to 900 errors were un-earthed. Almost the
same numbers of defects were identified during the testing phase.
Thus, 50% of total defects, related rework and repair costs were
avoided, even before a single piece of code was written.
Compared to earlier times, development rework fell by 12%.
Testing schedules were compressed by 10%.
Right Shoring
As the testing transitioned from business users to the specialist
partner up to 68% and 74% of the work was off-shored in Y1 and
Y2 respectively, resulting in significant cost savings.
Defect Prevention
As’ Functional defect’ leakage to production reduced to less than
1%, the business experienced a near-defect free application
delivery
Domain Value Adds
Given the short windows for daily validation in production, the
domain focused test team identified business critical fields. The
team also identified a representative data sample covering all
instruments such that the shorter production validation strategy
covered all the business critical areas.
Benefits across Years 3 and 4
As the relationship matured, the specialist partner brought in
benefits from re-usable repositories, automation and productivity
improvement initiatives.
Productivity Growth
While the specialist test team size grew only 30% in 3 years, the
output increased by 143% and 224%
Automation
The creation of a robust automation framework for data
warehousing cum business analytics engine started delivering
savings of about $100,000 for the first application stream and
was expected to save $740,000 as it is implemented across
application streams.
Re-usable repositories
Creating structured repositories resulted in over 69% of the test
cases being re-used. The re-use is bringing in annual savings of
$200k.
Cost-out initiatives
The specialist implemented various cost-out initiatives such as
‘prototype testing’, ‘QA batch load ownership’ and ‘offshore
shifts’ for further cost savings to the tune of $400k per annum.
Case 2
Global Regression Test Center for an Industry leader in Cards
The Vice President Product Engineering (VP-PE) of a globally
renowned cards software maker found that the product QA was
becoming overwhelmingly complex due global custom
implementations and frequent product releases.
The top Issues of concern
• Customer demand for high product flexibility and wide
parameterization.
• Intense competition meant no second chance on new product
introductions.
• Spiraling maintenance costs due to field releases of
in-adequately tested products
• Large customers invoking penalty clauses due to defective
deliveries ($6m for every month of delay!)
On enquiry, one of the clients recommended a specialist testing
house that is focused on the financial services industry.
On engaging the firm the VP-PE found striking differences in
performance!
As he attempted to quantify the benefits from the specialist, he
wondered why he didn’t do this earlier.
The Short and medium term benefits across first 2 years were: -
ZERO Knowledge Transfer
The product developer had no bandwidth for any knowledge
transfer. Yet the specialist successfully ramped up, thanks to their
financial industry focus, knowledge and experience.
Avoiding Penalties in Customer Rollouts
Rigorous ‘cards industry focused’ test methodologies helped the
software maker avoid client penalties (which were in some cases
as high as $6 million for every month’s of delay)
Defect Leakage less than 5%
Defect leakage was either not measured and was 20% in some
cases. It was uniformly brought down to less than 5% thus
improving the quality of applications delivered and reducing the
maintenance costs for the product vendor.
88% right shoring
Building the QA organization, the specialist started off with an
offshore leverage of 83% and raised it to 88% within the first year.
Long term benefits over Years 3 and 4 were: -
With the specialist, continuous improvement was not an
afterthought. To deliver a 50% saving on regression test costs,
the specialist implemented a ‘continuous improvement’
framework
Release # of static # of defects % of defects
# Testing identified Total avoided
Issues in execution (A/C*100)
identified
upfront
(A) (B) C = (A+B)
1 427 592 1019 42%
2 164 90 254 64%
3 24 17 41 59%
4 28 8 36 78%
4.2 245 173 418 59%
All 888 880 1768 50%
Initiative Annual Cost Savings
1 Prototype Testing $50,000
2 QA Batch Load Ownership $60,000
3 Offshore Shifts $40,000
4 Improve offshore leverage $50,000
5 Productivity Improvements $200,000
Result Specialist General Root Cause
Area Testing IT service Analysis
Partner provider
Defects in
Customer
Environment
6% 20% Functional
Coverage in
Testing
Test Strategy
Time to
Market
On time 7 days for
P1 (SLA = 2
days)
Delays in Defect
Resolution
turnaround
Requiremen
ts Coverage
in Testing
>95% Not
Measured,
60%
Lack of
Traceability,
Test Strategy
and Inadequate
coverage
130%
Team
100%
123%
130%
Year 1
Productivity
Year 2
Year 3
Test Created
100%
268%
243%
Executed
100%
230%
324%
243%
324%
percentagegrowth
350%
300%
250%
200%
150%
100%
50%
0%
Domain COE
• Business Processes
• Functional Artifacts
• Tracking of Global
Regulatory Changes
• Process Maintenance
• Process Definition
• Metric Program
• Process Training
• Continuous Process
Improvement
• Best Practice
implementation
Process COE
Competency
Building,
Recruitment,
Training
Repositories
Ready-to-use
Artifacts
Cards
Processor
Competency
Center
Resource
Management
Quality
Assurance
Resource
Management
Automation
using Tools
Metrics &
Reporting
Standard Processes &
Methodologies
Knowledge
Repositories Best Practices
Continuous Improvement Framework

4. Disclaimer: All the documentation and other material contained herein is the property of Thinksoft Global Services and all intellectual
property rights in and to the same are owned by Thinksoft Global Services. You shall not, unless previously authorized by Thinksoft
Global Services in writing, copy, reproduce, market, license, lease or in any other way, dispose of, or utilize for profit, or exercise any
ownership rights over the same. In no event, unless required by applicable law or agreed to in writing, shall Thinksoft Global Services,
or any person be liable for any loss, expense or damage, of any type or nature arising out of the use of, or inability to use any material
contained herein. Any such material is provided “as is”, without warranty of any type or nature, either express or implied. All names,
logos are used for identification purposes only and are trademarks or registered trademarks of their respective companies.
For more details visit, www.thinksoftglobal.com
Is Software Testing a “Zero Sum Game”– Yes or No? Case Studies
Initiative Yea $ Cost % reduction
Initial state 0 38000 0%
RPM repository 1-p1 32300 15%
consolidation
Automation 1-p2 19380 25%
Productivity 2 10659 5%
improvement
Productivity 3 5330 5%
improvement
Total 50%
Using this framework, the specialist testing partner:
• Created Centers of Excellence around the cards business and
related test process
• Built and organized structured test repositories
• Designed Test Automation frameworks custom made to the
cards business
• Implemented Risk Prioritization Matrices to ‘Triage’ the most
business critical functionalities that are most likely to fail
• Developed tools to integrate the repositories and automation
frameworks
Net result - a sustainable savings of 50% on annual regression
test costs - $1.2 m!
Epilogue:
From Dr Robert Harris’ presentation it was clear that
collaborating with a reliable and worthy partner with the right set
of capabilities is very critical to success!
What was not made explicitly clear at his presentation was that
in both cases the reliable and worthy testing partner was none
other than Thinksoft Global Services Ltd.
T r u s t t h e E x p e r t s
IS SOFTWARE TESTING A
“ZERO SUM GAME”– YES OR NO?
Case Studies
The Context
• How does one optimize management decisions in the software testing arena?
• Are business users to entrust ‘requirement planning’, product fitment, process mapping and customization to the product
developer?
• Should the developer also be entrusted with all interfacing and data migration tasks?
• In parallel, should the business user also build in-house teams of testers and test managers to interface with the product
developer, and do the “acceptance testing” themselves?
• Can the expertise required to sync with ever changing customer needs be developed and retained in-house
• Alternatively should the entire gamut of testing functions be off loaded to specialist testing houses that are springing up,
independent of software product developers?
• Would such independent expert services be more reliable and flexible over time in highly competitive sectors based on
“deep domain “knowledge?
• How do you balance on the one hand the high cost of failure and on the other, the need to control costs to retain margins
under competitive pressures?
These were some of the important questions occupying Dr. Robert Harris, who was to make a presentation at an industry
seminar.
Being an academic he knew the answers in theory, but he still needed to validate his assumptions using “field data”.
The following case studies illustrate his findings.
Case 1
Testing center of excellence for the Treasury arm of a global conglomerate
The Program Manager (PM) of the global conglomerate had to execute a multiyear program to roll out a sophisticated
reporting tool and a data ware-housing engine.
The existing globally reputed system integrator who was expected to manage requirement planning and testing was not
delivering satisfactorily due to obvious limitations in domain experience.
Business requirements were ill articulated and poorly documented. They existed only as spreadsheets or only in the minds of
the business user.
There was a severe shortage of business users to carry out even mandatory tests.
Existing test practices being weak, defective software was leaking into production resulting in significant rework.T r u s t t h e E x p e r t s