ROI in automated testing

Proprietary & Confidential
6 Ways to Measure the ROI of
Automated Testing
Building a Business Case for Success
Akshita Deora Puram

Learning Objectives
 Fundamentals behind a compelling automation
business case
 Six ways to measure the ROI of automated
testing
 Key criteria needed for a successful
transformation

Chapter 1: Business Case
ROI Gains and Investment
Chapter 2: Transformation Management
Making the Magic Happen
Deep Dive Topics

Release Cycles Months ->
Weeks -> Days -> Hours
Cost
Proliferation of Devices
Automation / CI-CD /
DevOps
Standardization to
Achieve Velocity
Shifting Team
Dynamics
Managing the Iron Triangle
Product
Complexity

ROI:
/rəˈtərn/ /än,ôn/ /inˈves(t)mənt/
a profitability measure that evaluates the
performance of an initiative by dividing net
benefits by net investment
Is the change worth it.

Automation ROI:
Gains – Investment
Investment
Gains from speed, quality, and cost savings
Investment in tools and resources

Common ROI Pitfalls
• Only accounting for creating,
developing and maintain manual
vs. automated tests
• Not accounting for the percentage
of tests that have to stay manual
• Not syncing your automation tool
stack with organizational
capabilities
• Not accounting for test
maintenance as an ongoing process
• Not accounting for an ROI over a
period of time

Not every organization,
industry, and QA team is
the same. How you
define, execute, and
maintain your tests vary.

Key Variables That
Define Test Scope
1. Requirements to be implemented
2. Test cases for each requirement
3. % of tests that can be automated
4. Test case complexity
5. Test cases of prior requirements
for regression testing
6. # of configurations to be tested
What makes your
organization unique?

#1 The Standard: Your New Test Cases
Metrics that Matter Best Practices
#of TotalTestCases
HourstoDevelop,Execute
andMaintain eachTest
TotalAutomationTime
AvgCostofanAutomation
Engineer perHour
1,000
0.5
500Hrs
$43
TotalCost $43K
#ofAutomationEngineers 2
1. Don’t think of ROI as Manual vs.
Automated. Account for tests that will
or should stay manual or exploratory
2. Factor in members that will conduct
both manual and automated tests
3. Don’t stop here!

#2 The Best Practice: Your Prior Test Cases
#of RegressionTests
HourstoMaintain eachTest
TotalAutomationTime
AvgCostofanAutomation
Engineer perHour
1,000
0.25
250Hrs
$43
TotalCost $21.5K
#ofAutomationEngineers 2
1. Immediately integrate new
automated tests as a part of your
regression testing
2. Medium and high complexity test
cases receive the most value from
automation
3. Account for maintenance of
developed test cases as an ongoing
cost for YoY return

#3 Most Variable: Coverage Across Environments
1. Increase environment coverage from
top four uses cases: cross browser
testing, unit testing, regression testing
and smoke testing
Cost of Lab
• # of Devices
• Average Cost per Device
• Cost for Lab Maintenance
Defect Leakage
• Defects that could go downstream
due to lack of environment coverage
• Hours to fix downstream defects
Tool
• Cost of a cross browser tool

#3 Most Variable: Coverage Across Environments
Testingacrossdifferentbrowsersanddevicesisone
ofthemosttimeconsumingaspectsoftestingthe
frontendofyourwebsiteorwebapplication.Run
moretests,againstmorebrowserconfigurationsby
runningtheminparallel.
Accordingtothetestingpyramid,Unittestsshould
beyourmostabundanttesttypeinyourentire
testingsuite.Becauseofthis,running14,000unit
testsinunderanhourisreallyonlypossiblewitha
massiveparalleltestinginfrastructureinvestment.
Needtogetyourminimumtestingdoneinthenext
20minuteswhileyoupushahotfix?Onlywaytodo
thatistoruntheminparallel,allowingyoutogetthe
mosttestingdoneintheshortestamountoftime.
Becausedeploymentsarehappeningatsucharapid
pace,regressiontestingisonofthebestwaystohave
atypeof“testingversioncontrol”makingsurethe
functionalityofthenewbuild,matchesthatofthe
laststablebuild.Runningthesetestsinparallelallows
moretobetested.
CrossBrowserTesting
UnitTesting
RegressionTesting
SmokeTesting
MosteffectiveteststoseeROIfromforparalleltesting
(cont.)

#4 Most Impactful: Reduction of Defect Leakage
1. Find savings by catching defects
earlier in your SDLC
Defect Leakage
• From lack of functionality coverage
• From missing requirements
• Hours to fix downstream defects
Test early.

#5 Most Underused: Test Redundancy and Reuse
1. Build modular test scripts enabling
test reusability
2. Leverage a test case management tool
to be able to search duplicate test
scripts or similar components
• No. of tests that were recorded twice
or have similar components
• Time spent searching for redundant
test cases
• Time to develop and execute
redundant tests

#6 Most Forgotten: Reduction of Knowledge Leakage
1. Test case naming conventions.
2. Leverage custom fields in test case
properties to personalize search for
your organization.
• Average tenure for a test engineer
• Time to re-engineer lost cases
3. Document. Document. Document.

Six Ways to Measure ROI of
Automation Testing
ThestandardforeveryROIanalysis
AutomationofNewTests
RegressionTesting–Thebestpracticethat
appliestoallROIbusinesscases
AutomationofPriorTests
ThemostvariableROImeasure
CoverageAcrossEnvironments
ThemostimpactfulROImeasure
DefectLeakage
Themostunder-usedand-valued
automationadvantage
TestRedundancyandReuse
ThemostforgottenROImeasure
KnowledgeLeakage
1.
4.
2.
5.
3.
6.

Your Automation Investment
Tools Resources
Investment
• Time required to ramp up to speed
• Investment in automation test
experience
• Organic (training costs)
• Inorganic (head count)
Investment
• Total License Cost
• Annual Maintenance Fees
• Customer Service Fees
Value
• Time to value and ease of use
• Flexibility for all levels of experience
• Ongoing customer support and
resources

Page
22
now 1 year 2 years 3 years 4 years 5 years
Our team can save [INPUT 1st Year $$] after one
year. Over [INPUT ROI PERIOD], we can
save [INPUT ROI $$].
Other Benefits
• Matured testing processand results
• Maximized applicationtest coverage
• Reduced risk fromtest redundancy and defect leakage
• FasterExecution of the tests acrossenvironments
• Increased focuson new features
• Enhanced product quality
SampleResultsOverview
Your ROI for Automated Testing
$
$
$$
$$
$$$
$$$

3 T’s Behind Every Successful Transformation
1 2 3
Talent Test Approach Tools
• Automation knowledge,
with in-depth product
knowledge
• Automation-first mindset
• …
• Quick turnaround of new
test cases to automation
for regression testing
• …
• Leverage tools for easy
tracking for success
metrics
• …
Top Initiatives

Page
25
3-year Transformation Map
2018 2019 2020
FUTURE
TODAY
TalentTestApproach
Tools
New hires for
automation engineers
Real-time reporting
Centralize automation
resources
Automation Training
Standardize and
document test processes
Standard QA metrics
Automation Tool
Assessment
Tools Training
Test Management Tool
Assessment
Environment Coverage
Tool Widespread
Adoption
Continuous
Testing
QA-Dev
Collaboration
Test Case Naming
Conventions
Standardize and
document build
environments

QAAutomation Transformation Charter
Initiative Name Owner Description
Automation
Transformation
TBD The implementation of tools and processes to automate our QA test cases for application 1
Objectives
 To automate common QA testing processes
 To reduce the amount of time QA spends on regression testing to
redirect focus on increased coverage of new features and enhanced
overall application quality
Benefits
 Matured testing process and results
 Maximized application test coverage
 Reduced risk from test redundancy and defect leakage
 Faster Execution of the tests across environments
 Increased focus on new features and enhanced product quality
Success Metrics
 Test coverage
 Defect leakage
 Employee productivity
Timeline Overview
Business Strategic Alignment
 Serving Our End Users
 Operating with Automation
 Growing the Business
 Developing Our People
x
x
ROI Analysis
Year 1 Year 2 Year 3 Year 4 Year 5
10,000 80,000 120,000 300,000 500,000
Total: 100,010,000
2018 2019 2020
Automation Transformation for Application 1

Recap
 Fundamentals behind a compelling automation business case
• Common ROI Pitfalls
• Variables that vary across organization
 Six ways to measure the ROI of automated testing
• Metrics that Matter
• Best Practices
 Key criteria needed for a successful transformation
• 3 T’s: Talent, Test Approach, and Tools
• Transformation Map and Initiative Charters

Easy-to-Use Online
ROI Calculator
coming soon…

Thank you!
6 Ways toMeasure ROIof AutomatedTesting

ROI in automated testing

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à ROI in automated testing

Similaire à ROI in automated testing (20)

Dernier

Dernier (20)

ROI in automated testing