6 Ways to Measure the ROI of Automated Testing
•Télécharger en tant que PPTX, PDF•
2 j'aime•4,073 vues
Interested in automated testing, but unsure whether it is worth the initial costs? Find 6 ways to measure the ROI of automated testing for your business in this presentation.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à 6 Ways to Measure the ROI of Automated Testing
Similaire à 6 Ways to Measure the ROI of Automated Testing (20)
Plus de SmartBear
Plus de SmartBear (20)
Dernier
Dernier (20)
6 Ways to Measure the ROI of Automated Testing
- 1. Proprietary & Confidential 6 Ways to Measure the ROI of Automated Testing Building a Business Case for Success Akshita Deora Puram
- 2. Proprietary & Confidential Learning Objectives Fundamentals behind a compelling automation business case Six ways to measure the ROI of automated testing Key criteria needed for a successful transformation
- 3. Proprietary & Confidential Chapter 1: Business Case ROI Gains and Investment Chapter 2: Transformation Management Making the Magic Happen Deep Dive Topics
- 4. Proprietary & Confidential Release Cycles Months -> Weeks -> Days -> Hours Cost Proliferation of Devices Automation / CI-CD / DevOps Standardization to Achieve Velocity Shifting Team Dynamics Managing the Iron Triangle Product Complexity
- 5. Proprietary & Confidential ROI: /rəˈtərn/ /än,ôn/ /inˈves(t)mənt/ a profitability measure that evaluates the performance of an initiative by dividing net benefits by net investment Is the change worth it.
- 6. Proprietary & Confidential Automation ROI: Gains – Investment Investment Gains from speed, quality, and cost savings Investment in tools and resources
- 7. Proprietary & Confidential Common ROI Pitfalls • Only accounting for creating, developing and maintain manual vs. automated tests • Not accounting for the percentage of tests that have to stay manual • Not syncing your automation tool stack with organizational capabilities • Not accounting for test maintenance as an ongoing process • Not accounting for an ROI over a period of time
- 8. Proprietary & Confidential Not every organization, industry, and QA team is the same. How you define, execute, and maintain your tests vary.
- 9. Proprietary & Confidential Key Variables That Define Test Scope 1. Requirements to be implemented 2. Test cases for each requirement 3. % of tests that can be automated 4. Test case complexity 5. Test cases of prior requirements for regression testing 6. # of configurations to be tested What makes your organization unique?
- 10. Proprietary & Confidential Chapter 1: Business Case ROI Gains and Investment Chapter 2: Transformation Management Making the Magic Happen Deep Dive Topics
- 11. Proprietary & Confidential #1 The Standard: Your New Test Cases Metrics that Matter Best Practices #of TotalTestCases HourstoDevelop,Execute andMaintain eachTest TotalAutomationTime AvgCostofanAutomation Engineer perHour 1,000 0.5 500Hrs $43 TotalCost $43K #ofAutomationEngineers 2 1. Don’t think of ROI as Manual vs. Automated. Account for tests that will or should stay manual or exploratory 2. Factor in members that will conduct both manual and automated tests 3. Don’t stop here!
- 12. Proprietary & Confidential #2 The Best Practice: Your Prior Test Cases Metrics that Matter Best Practices #of RegressionTests HourstoMaintain eachTest TotalAutomationTime AvgCostofanAutomation Engineer perHour 1,000 0.25 250Hrs $43 TotalCost $21.5K #ofAutomationEngineers 2 1. Immediately integrate new automated tests as a part of your regression testing 2. Medium and high complexity test cases receive the most value from automation 3. Account for maintenance of developed test cases as an ongoing cost for YoY return
- 13. Proprietary & Confidential Automation ROI: Gains – Investment Investment Gains from speed, quality, and cost savings Investment in tools and resources
- 14. Proprietary & Confidential #3 Most Variable: Coverage Across Environments Metrics that Matter Best Practices 1. Increase environment coverage from top four uses cases: cross browser testing, unit testing, regression testing and smoke testing Cost of Lab • # of Devices • Average Cost per Device • Cost for Lab Maintenance Defect Leakage • Defects that could go downstream due to lack of environment coverage • Hours to fix downstream defects Tool • Cost of a cross browser tool
- 15. Proprietary & Confidential #3 Most Variable: Coverage Across Environments Testingacrossdifferentbrowsersanddevicesisone ofthemosttimeconsumingaspectsoftestingthe frontendofyourwebsiteorwebapplication.Run moretests,againstmorebrowserconfigurationsby runningtheminparallel. Accordingtothetestingpyramid,Unittestsshould beyourmostabundanttesttypeinyourentire testingsuite.Becauseofthis,running14,000unit testsinunderanhourisreallyonlypossiblewitha massiveparalleltestinginfrastructureinvestment. Needtogetyourminimumtestingdoneinthenext 20minuteswhileyoupushahotfix?Onlywaytodo thatistoruntheminparallel,allowingyoutogetthe mosttestingdoneintheshortestamountoftime. Becausedeploymentsarehappeningatsucharapid pace,regressiontestingisonofthebestwaystohave atypeof“testingversioncontrol”makingsurethe functionalityofthenewbuild,matchesthatofthe laststablebuild.Runningthesetestsinparallelallows moretobetested. CrossBrowserTesting UnitTesting RegressionTesting SmokeTesting MosteffectiveteststoseeROIfromforparalleltesting (cont.)
- 16. Proprietary & Confidential #4 Most Impactful: Reduction of Defect Leakage Metrics that Matter Best Practices 1. Find savings by catching defects earlier in your SDLC Defect Leakage • From lack of functionality coverage • From missing requirements • Hours to fix downstream defects Test early.
- 17. Proprietary & Confidential #5 Most Underused: Test Redundancy and Reuse Metrics that Matter Best Practices 1. Build modular test scripts enabling test reusability 2. Leverage a test case management tool to be able to search duplicate test scripts or similar components • No. of tests that were recorded twice or have similar components • Time spent searching for redundant test cases • Time to develop and execute redundant tests
- 18. Proprietary & Confidential #6 Most Forgotten: Reduction of Knowledge Leakage Metrics that Matter Best Practices 1. Test case naming conventions. 2. Leverage custom fields in test case properties to personalize search for your organization. • Average tenure for a test engineer • Time to re-engineer lost cases 3. Document. Document. Document.
- 19. Proprietary & Confidential Six Ways to Measure ROI of Automation Testing ThestandardforeveryROIanalysis AutomationofNewTests RegressionTesting–Thebestpracticethat appliestoallROIbusinesscases AutomationofPriorTests ThemostvariableROImeasure CoverageAcrossEnvironments ThemostimpactfulROImeasure DefectLeakage Themostunder-usedand-valued automationadvantage TestRedundancyandReuse ThemostforgottenROImeasure KnowledgeLeakage 1. 4. 2. 5. 3. 6.
- 20. Proprietary & Confidential Automation ROI: Gains – Investment Investment Gains from speed, quality, and cost savings Investment in tools and resources
- 21. Proprietary & Confidential Your Automation Investment Tools Resources Investment • Time required to ramp up to speed • Investment in automation test experience • Organic (training costs) • Inorganic (head count) Investment • Total License Cost • Annual Maintenance Fees • Customer Service Fees Value • Time to value and ease of use • Flexibility for all levels of experience • Ongoing customer support and resources
- 22. Page Proprietary & Confidential 22 now 1 year 2 years 3 years 4 years 5 years Our team can save [INPUT 1st Year $$] after one year. Over [INPUT ROI PERIOD], we can save [INPUT ROI $$]. Other Benefits • Matured testing processand results • Maximized applicationtest coverage • Reduced risk fromtest redundancy and defect leakage • FasterExecution of the tests acrossenvironments • Increased focuson new features • Enhanced product quality SampleResultsOverview Your ROI for Automated Testing $ $ $$ $$ $$$ $$$
- 23. Proprietary & Confidential Chapter 1: Business Case ROI Gains and Investment Chapter 2: Transformation Management Making the Magic Happen Deep Dive Topics
- 24. Proprietary & Confidential 3 T’s Behind Every Successful Transformation 1 2 3 Talent Test Approach Tools • Automation knowledge, with in-depth product knowledge • Automation-first mindset • … • Quick turnaround of new test cases to automation for regression testing • … • Leverage tools for easy tracking for success metrics • … Top Initiatives
- 25. Page Proprietary & Confidential 25 3-year Transformation Map 2018 2019 2020 FUTURE TODAY TalentTestApproach Tools New hires for automation engineers Real-time reporting Centralize automation resources Automation Training Standardize and document test processes Standard QA metrics Automation Tool Assessment Tools Training Test Management Tool Assessment Environment Coverage Tool Widespread Adoption Continuous Testing QA-Dev Collaboration Test Case Naming Conventions Standardize and document build environments
- 26. Proprietary & Confidential QAAutomation Transformation Charter Initiative Name Owner Description Automation Transformation TBD The implementation of tools and processes to automate our QA test cases for application 1 Objectives To automate common QA testing processes To reduce the amount of time QA spends on regression testing to redirect focus on increased coverage of new features and enhanced overall application quality Benefits Matured testing process and results Maximized application test coverage Reduced risk from test redundancy and defect leakage Faster Execution of the tests across environments Increased focus on new features and enhanced product quality Success Metrics Test coverage Defect leakage Employee productivity Timeline Overview Business Strategic Alignment Serving Our End Users Operating with Automation Growing the Business Developing Our People x x ROI Analysis Year 1 Year 2 Year 3 Year 4 Year 5 10,000 80,000 120,000 300,000 500,000 Total: 100,010,000 2018 2019 2020 Automation Transformation for Application 1
- 27. Proprietary & Confidential Recap Fundamentals behind a compelling automation business case • Common ROI Pitfalls • Variables that vary across organization Six ways to measure the ROI of automated testing • Metrics that Matter • Best Practices Key criteria needed for a successful transformation • 3 T’s: Talent, Test Approach, and Tools • Transformation Map and Initiative Charters
- 28. Proprietary & Confidential Easy-to-Use Online ROI Calculator coming soon…
- 29. Proprietary & Confidential Thank you! 6 Ways toMeasure ROIof AutomatedTesting
Notes de l'éditeur
- Many of you are familiar with the iron triangle. Demands for frequent release cycles, managing infinite combinations of devices and creating standardization as put pressure on delivery time. Trends such automation, continuous integration, devops and overall, new technologies are forcing increases in quality standards by being able to do more with less. And meeting all the new demands, while still keeping costs low and optimized, is a third important element for software delivery.
- This has put a lot of pressure on businesses to always deliver the latest feature as quick as possible and with no issues. More and more companies are exploring new ways of working and assessing the ROI to see if the change is worth it.
- The return on investment or ROI for automated testing evaluates its performance by dividing the net benefit from gains in speed, quality and cost savings from automation by the net investment your team makes in tools and resources.
- We wish calculating the ROI was as simple as this equation, but there are common pitfalls to be aware of.
- In addition, not every organization is the same. Each industry and qa team is unique.
- After speaking to a number of customers and looking at their organizations, there were key variables that stood out that made each organization unique.
- To fully understand how to build an ROI for automated test, we have divided the rest of the webinar into two chapters. 1/ the business case to look at all the ROI gains and investment to build a comprehensive assessment and 2/ how to make the change happen successful with the right transformation management techniques.
- When starting out with an ROI assessment, the most common input is looking at the time it takes to develop, execute and maintain an automated test versus a manual one for new cases to run each year.
- There are more to gains than just automated versus manual though.
- To truly understand the software development life cycle, I have broken this training into two chapters: Chapter 1 focuses on the tactical step-by-step process on how to build software And Chapter 2 focuses on the people and tools behind the scenes that make it happen