In this SauceCon 2019 talk, Fernando and Amir use observations from running 2 billion tests for thousands of Sauce Labs customers to outline the the key drivers of success in continuous testing and an provide an actionable way to see how organizations are doing relative to other successful testers. They discuss a new index which can be used by any organization to evaluate their progression towards continuous testing, and cover the components of the new continuous testing index, what makes these components such strong maturity indicators, how organizations can improve on each of the metrics, and what happens organizationally when those metrics are improved

1. Continuous Testing Maturity
Measure & Optimize Your Way to Deliver Faster, Better

2. Hello
Amir Rozenberg
Director, Product
Fernando Vidal
Sr. Data Scientist
Phil Wells
Software Engineer

3. Agenda
2/11/19
Introduction to CT Maturity Benchmark
The Composition of the CT Maturity Benchmark
From the Practitioner Perspective
Q&A

4. Digital Transformation: Leadership or Extinction
Source: GitLab 2018 Developer Survey
Where in the development cycles you
encounter most delays?
Source: GitLab 2018 Developer Survey

5. Continuous Testing Maturity
Measuring and Optimizing
The Continuous
Testing Journey
LEGACY
TESTING
CONTINUOUS
TESTING
PROOF OF
AUTOMATION VALUE

6. Continuous Testing Maturity
Measuring and Optimizing
The Continuous
Testing Journey
LEGACY
TESTING
CONTINUOUS
TESTING
PROOF OF
AUTOMATION VALUE

7. 2 Billion + Tests Run to Date
Sauce Labs: The Only Vendor with the Data and Expertise to Create this Report
● Millions of tests run every day on Sauce Labs
● Thousands of users run tests every day
● Unique position to use this data to understand what drives success

8. A Simple 0-100 Score for Automated Testing Success

9. Test Quality
Test Suite Maintenance and Battling Flaky Tests

10. Test Quality: Why It Matters
GOAL: Consider a build green if every test passes
● “Quarantine” tests that are flaky
● Test Fix Rate > Test Break Rate
● Quickly repair tests that break due to application changes
High “ambient” levels of test failure due to flaky tests or an ever increasing number
of quarantine tests decrease functional coverage and reduce confidence in the
automated testing process
Test Failure Should Lead to High Confidence that Application is Broken

11. Test Run Time
Key to Test Quality and Maintainability

12. Test Run Time: Why It Matters
Long Tests Bottleneck a Test Suite
Long jobs slow everything down

13. Test Run Time: Short Tests Fail Less Often
Failure Rate for Faster Tests are Dramatically Lower

14. Test Platform Coverage
Quality Means Confidence Across Multiple Platforms

15. Platform Coverage: Why it Matters
DESKTOP
MOBILE
BROWSERS

16. Test Concurrency
Run as Many Tests as Quickly as Possible for Rapid Feedback

17. Test Concurrency: Why it Matters
Accelerate the Feedback Loop and Know Quickly if you can Release
30 Minute Build 6 Minute Build
The Importance of Leveraging Concurrency
A Sample Test Suite with 50 Tests

18. CT Benchmark Score: Average of 4 Scores

19. Why this is Important to Get Right
Develop
Apps Faster
Improve
Developer
Productivity
Delight
Users
19
Continuous Testing Unlocks Your Ability to Achieve Digital Excellence

20. Questions for Phil- (hidden slide)
1. [Fernando] What are some best practices to maintain hygiene at the test suite level and keep the maintenance effort
as low as possible?
2. [Amir] How are you communicating improvement in continuous testing to your management and do you see a buy in
from them to continuous investment in this area?
Backup
3. [Amir] Have you been trying to measure progress in continuous testing, if so what have you been using and how well
is it accepted by the team as metrics to guide improvement?
4. [Fernando] What are some best practices maintaining a large test suite at a high quality?
Phil says: Of these, I think numbers 1 and 4 will give me the most to discuss.

21. Thank you.