Organisations turn to Agile and DevOps to improve customer experience by maximising the speed of delivery without sacrificing quality. As the champions of quality, testers achieve this goal through continuous testing. Test Automation plays a major role in continuous testing; it is the backbone of the continuous test process. To achieve continuous testing, automation must be applied at every stage of the development process. Developing a smart automation strategy and using the right tools is critical in achieving continuous testing since test scripts must be scalable and easy to maintain.

1. Achieving Continuous Testing through Test
Optimization and Automation
June 28th, 2018

2. Gerie Owen
VP, Knowledge and Innovation US
Giovanni Rago
Solutions Engineer, EMEA
Webinar Presenters

3. Agenda
Continuous Testing Test Optimization
Introduction
SummaryTest Automation Q&A

4. Agenda
Continuous Testing Test Optimization
Introduction
SummaryTest Automation Q&A

5. About
QualiTest
QA and testing is
all that we do!
World’s largest
pure play
testing
company
Over 3,000 testing
professionals
worldwide
Business
Assurance
QA &
Testing
Focus
Outcome
Based
Testing
(OBT)
Contractual guarantees for
quality improvement &
cost reduction
Managed
Testing
Solutions
(MTS)
Long term, SLA-based testing
outsourcing
Industry & Technology
Expert
Global
Service
Delivery
Model
On-Shore, Near-Shore,
Off-Shore and Crowd
Testing

6. Sauce Labs
Analytics
Comprehensive
Coverage
Secure
Most
Reliable
Experience
DebuggingHighly Scalable

7. Agenda
Continuous Testing Test Optimization
Introduction
SummaryTest Automation Q & A

8. ContinuousTesting
| ContinuousTesting is the basis for Continuous Delivery
| Continuous Delivery requires not only increased velocity, but more importantly
increased quality.
| Testing Cycles must be shortened yet quality must be increased.
| Increases in microservices architectures require high-quality functional testing including
automated API testing.
| Builds, integrations and deployments are automated; testing must be integrated into the
pipeline.

9. ContinuousTesting
| A core capability in Agile and DevOps where all testing activities run continuously in
an integrated fashion with development and delivery. Immediate bug fixing is
enabled, test environments are provisioned instantly, and unit, functional, and
nonfunctional tests are run in an automated way, orchestrated by Continuous
Integration and Continuous Delivery tools. The ForresterWave™: ModernApplication
FunctionalTest AutomationTools, Q4 2016

10. ContinuousTesting
| ContinuousTesting requires risk analysis and process improvement as well as
implementation of automation throughout the entire software development process.
| “Shift Left” i.e., moving automated testing to the lowest levels of the testing pyramid
is an important part of the approach.

11. ContinuousTesting vsTest Automation
| ContinuousTesting requires automation, however, it encompasses much more.
| ContinuousTesting is an approach to managing risk by focusing not only on improving
testing efficiency but more importantly, increasing the effectiveness of our test
processes.

12. | Assesses business risk coverage as its primary goal
| Establishes a safety net that helps the team protect the user experience
| Requires a stable test environment to be available on demand
| Integrates seamlessly into the software delivery pipeline and DevOps toolchain
| Delivers actionable feedback appropriate for each stage of the delivery pipeline
Attributes of ContinuousTesting

13. ContinuousTesting
| ContinuousTesting Minimizes Business RiskThrough:
| Test Optimization
| Automation throughout the Continuous Delivery Pipeline

14. Agenda
Continuous Testing Test Optimization
Introduction
SummaryTest Automation Q & A

15. Test Optimization
| Test Optimization encompasses risk assessment of the entire test scope:
| Analyze test case for quality, effectiveness, automation potential
| Evaluating the test suites to determine if they are providing the maximum amount of coverage
with the fewest number of test cases.

16. OptimizeTest Scope
| Select end-to-end tests that focus on the critical and high-risk features of the
application
| Understand all integrations involved in critical user workflows including the
technologies employed in those applications (web, mobile, message/API-layer etc.)
| Implement service virtualization for integrations and components that aren’t readily
available in order to run end-to-end test continuously
| Use the most realistic data possible for each test, include synthetic data generation
where needed
| Include exploratory testing to find user-experience and other bugs not easily found by
automated tests

17. OptimizingTest Cases
| Tests should be focused on the functionality they are validating.
| Tests should be as simple as possible; the more complex the test, the more validation
is required for the test code itself.
| Even though service virtualization can be used to fill in for missing components in
integration testing, tests should as independent as possible.

18. OptimizingTest Suites
| Include test cases that validate high risk areas of code
| Include test cases that validate customer-centric features
| Include test cases that validate critical business workflows
| Use FunctionalAutomation tools that create multi-purpose scripts
| Cross-PlatformTesting
• Scripts should run on Desktop,Web, Mobile & Server
| Cross-BrowserTesting
• Scripts should run on multiple browsers including Chrome, Edge, Safari and any others required
| PerformanceTesting
• Scripts should collect performance statistics including page loads etc.

19. Agenda
Continuous Testing Test Optimization
Summary
Introduction
Q & ATest Automation

20. Test Automation
| Test automation is critical for Continuous Integration and Continuous Delivery
| Fast feedback is not just a nice-to-have
| Automated tests require a strategy
| Which cases to automate?
| How to automate them?
| When to run automated tests? (At which point in the delivery pipeline?)
| An automation framework can be a software project of its own
| Specific technical know-how is needed
| Maintenance needs to be accounted for
| Good software engineering practices apply

21. Dependable, Consistent, ReproducibleTests
| Test environment should mimic production to the highest degree possible
| Having a system free of artifacts or stale data is paramount for reproducible results
| Every session should be “fresh”: clean system cache, browser cache, cookies, etc.
| Make sure your execution workstation and system under test have enough bandwidth
and memory to support the number of simultaneous testing threads

22. Beware the Maintenance Overhead
| …big deal in UI testing!
| Can explode and needs to be managed carefully
| Can be kept under control by following best practices
| PageObject model
| In general: loose coupling, good software development practices.

23. Enabling BetterTesting
| Strive to build up speed
| Monitor test performance over time
| Leverage parallelism whenever possible
| Fast test suites mean more testing, better QA
| Trust in your framework is precious
| Flaky tests produce noise, noisy builds reduce user compliance
| An automation friendly SUT makes a big difference
| Get your developers to help out!

24. Enabling BetterTesting
| Use the right tool for the job
| Keep functional tests separate from performance tests
| If reliable testing can be achieved faster with a unit test, use a unit test
| Place functional tests in the appropriate place in the build
| Create shortest and most succinct tests possible by creating a state closest to the
single feature being tested
| API Calls to insert data
| Deep linking
| Setup andTeardown Methods
| …enable atomic and autonomous tests!

25. KeepingTests Atomic
| Atomic
| Test a single feature
| Short and succinct

26. KeepingTests Atomic
| Example:

27. KeepingTests Autonomous
| Autonomous
| Tests should run independently from other tests
| One test’s actions should not drive another test’s acceptance criteria

28. KeepingTests Autonomous
| (Bad) Example:

29. KeepingTests Autonomous
| (Good) Example:

30. Benefits of Atomic and AutonomousTests
| Easier to debug and maintain
| Atomic -> Concise tests tend to be less complex
| Autonomous -> Low coupling ensures tests are “sandboxed”
| Easier to parallelize
| Atomic + Autonomous -> Independent tests testing independent features

31. Why Run in Parallel
| High test coverage across multiple key platforms is highly beneficial
| …but the resulting number of tests can be high, and execution time can explode
| Running tests sequentially is not an option anymore: parallelization is needed

32. Agenda
Continuous Testing Test Optimization
Summary
Introduction
Test Automation Q & A

33. Summary
|ContinuousTesting Focuses on Risk
|Optimize theTest Process
|AutomatedCheckpoints
|Automated feedback loops
|Integrate testing through the delivery pipeline including deployments at
each stage
|MatureTest Automation Enables ContinuousTesting
|Continued maintenance must be accounted for
|Sticking to best practices pays big dividends
|Parallelism enables high speed at high coverage

34. Agenda
Continuous Testing Test Optimization
Introduction
SummaryTest Automation Q & A

36. Thank you