1.
HOW AI IS CHANGING
THE FUTURE OF
SOFTWARE TESTING

2.
2
Vui Nguyen
Software Test Engineer at Axon Active Viet Nam
vuint.it@gmail.com

3.
Agenda
3
What is Software Testing?
1
What is AI?
2
What challenges can AI solve?
3
Advantages and Disadvantages of AI
4
AI Tools for Software Testing
5

4.
What is Software Testing?
International Software Testing Qualifications Board:
“Software testing is a process of executing a program or
application with intent of finding the software bugs.
It can also be stated as process of validating and verifying that a
software program or application or product meets the business
and technical requirements that guided its design and
development”
4

5.
Software Testing Overview
5
Requirement
Analysis
Implement
Test Script
Write test cases
Execute test cases
Compare
Test result == expected value
Expected Result
Test result != expected value
Expected
Results or properties

6.
Software Testing Challenges
6
Years
Months
Months
Weeks
Weeks
Days
Days
Minutes

7.
Software Testing Challenges
7

8.
What is AI?
8
It is the science and engineering of making
intelligent machines, especially intelligent
computers to understand human intelligence,
but AI does not have to confine itself to
methods that are biologically observable
STANFORD
“Artificial intelligence (AI) is a self improving
enable horizontal layer that is solving problems
that were in the realm of science fiction for the
past several decades”
Jeff Bezos, Amazon.com Inc
Artificial intelligence (AI) makes it possible for
machines to learn from experience, adjust to
new inputs and perform human like tasks
SAS
Artificial intelligence is the boarder concept of
machines being able to carry out tasks in a way
that we would consider “smart”
FORBES

9.
Structure of AI
9

10.
Applications of AI
10
Fraud Detection
• AI has the ability to identify
fraudulent behavior, as well as
identify what next pattern of
suspicious behavior will be
• AI algorithms can preempt such
fraudulent transactions and can
lead to huge costs savings for
banks and its customers
Meeting regulatory
requirements
• AI is used to ensure that
regulatory requirements are
met and that data is kept with
monitoring done on a real-
time basis. This allows issues
to be flagged a lot sooner
Boost customer
engagement
• AI will assist in the creation of
customized and intelligent
products and services, with
new features, more intuitive
interactions (e.g. speech) and
advisory skills (e.g. personal
financial management)
BANKING
Computer
aided
diagnosis
• AI is being used
extensively to read
and interpret
complex radiology,
pathology reports
to help doctors
arrive at early
High risk groups
identification
• AI is being fed huge volumes
of data historical medical
records that helps in
identifying whether a
patients is in a high risk
group for any particular
disease say stroke
cardiovascular diseases or
cancer
Epidemic out break prediction
• ML and AI technologies are also being applied
to monitoring and predicting epidemic
outbreaks around the world, based on data
collected from satellites, historical information
on the web, real- time social media updates,
and other sources
• Support vector machines and artificial neural
networks have been used, for example, to
predict malaria outbreaks, talking into account
data such as temperature, average monthly
rainfall, total number of positive cases, and
other data points
HEALTHCARE
10

11.
Applications of AI
11

12.
Processes of software testing are similar to the processes used to
train AI
AI for Software Testing
AI applications use the output of the AI training process and
apply that to specific problems, such as recognizing a stop
sign and stopping a car, traffic lights … which consist of
inputs and comparing the outputs to expected results
12

13.
AI Training data Process
 Training AI systems is very similar to testing
 AI training process assigned into categories: processing, sensing,
learning
13

14.
What challenges can AI solve?
14
Ease of Authoring
and Executing
Tests
Releasing at the
Speed of
Development
Reducing
Maintenance and
Eliminating Flaky
Tests
Faster and More
Stable UI Tests
Continuous
Learning from
Production Data
Removing
Dependencies

15.
Hundreds of attributes used to
identify elements
 A few changes don’t break the test
 Automatic get best location
strategy to successfully identify
15
Faster and More Stable UI Tests

16.
 AI’s self-healing mechanism can detect problems in the
failed tests before they even occur, fix tests instead of us
reacting to them
 AI can figure out which tests are stable or flaky, analysis
what tests need to be modified to ensure test runs are
stable
 Based on large numbers of test runs, AI can optimize the
wait times used in tests to wait for the pages to load and
also can handle tests running on different resolutions
Reducing Maintenance and Eliminating Flaky Tests
16

17.
 AI will start observing and learning how our customers are using the
product and can start creating tests based on real user data
 AI will identify commonly used actions such as logging in/out of the
application and cluster them into reusable component
17
Continuous Learning from Production Data
LEARN BY OBSERVATION
(PRODUCTION)
AGGREGATE USER
ACTIONS INTO FLOWS
TEST PRODUCED
FROM FLOWS

18.
 Once we have authored some tests
and have run them consistently for a
period of time, the AI can start
recording all the server responses
 When run the tests again, instead of
talking to a server or database, the test
will access the stored responses and
will continue to run without any
obstacles
18
Removing Dependencies

19.
Author and execute tests can be done in a matter of hours
Use dynamic locators and the ability to easily create reusable
components
Integrate CI/CD systems easily with public and private grids
Nontechnical people can get involved in test
Increase collaboration within teams and encourages everyone to
own the test automation effort
Ease of Authoring and Executing Tests
19

20.
 With AI powering the transition to autonomous testing, reducing the
maintenance to a minimum, and creating more reliable tests, the ability
for teams to release faster is better than
 Testers an maximize user coverage by connecting authoring of tests with
production apps mapping to real user flows
 We have the ability to take a risk-based approach and base our
decisions on real data
 We are now able to create more user scenarios in short period of time.
This means you can find bugs fast and release faster
20
Releasing at the Speed of Development

21.
 Improved accuracy and efficiency
 Overcoming the limitations of manual testing
 Benefits both testers and developers
 Improving overall test coverage
 Time & cost-saving
Advantages
21
Advantages and Disadvantages of AI

22.
Advantages and Disadvantages of AI
 Artificial intelligence software testers use the
concept of GIGO (Garbage in Garbage Out)
 High costs
 Can’t think outside the box
Disadvantages
22

23.
AI Tools
Visual Testing
Regression Test
Monitoring / Report
23

24.
Visual Testing
24
Baseline
<empty>
Image1
Image2
Image3
Result:
Image1
Image2
Image3
new
new
new
visualTest () {
simulate UI state 1
check (“check 1”)
simulate UI state 2
check (“check 2”)
simulate UI state 3
check (“check 3”)
}
1) Write app & test code 2) Run test 1st time 4) Baseline created3) Review results
Baseline:
Image1
Image2
Image3
visualTest () {
simulate UI state 1
check (“check 1”)
simulate UI state 2
check (“check 2”)
simulate UI state 3
check (“check 3”)
}
Image1F
Image2
Image3B
Result:
Image1
Image2
Image3
diff
==
diff
5) Update app & test code 6) Differences detected 8) Save baseline7) Review results
Baseline:
Image1F
Image2
Image3
Baseline:
Image1
Image2
Image3
3B:
Bug
1F – New feature

25.
 Segmentation algorithm, Convolutional Neural Networks and a
combination of algorithms
 Can be directly incorporated into testing frameworks
 Test results available in Test manager
25
Visual Testing

26.
 Flaky tests
 Maintenance
 Huge data
 Learning curve
26
Regression Testing

27.
Regression Test
 Focuses on reducing flakiness
 Data driven testing
 Automated test case generation
 Reads production user access
27

28.
Monitoring / Reporting
 Different tools, frameworks and test for functional,
performance and security testing
 Numerous test cases
 Non prioritized test suite
 Get meaningful data out of logs
28

29.
 One tool for code coverage for
different technologies
 Prioritizes test cases
 Functional, Performance, Security
 Segregates test into smoke and regression
 Failure prediction though log monitoring
 Logs with reason for test failure
29
Monitoring / Reporting

30.
30
References
 https://techbeacon.com/app-dev-testing/state-test-automation-7-
key-trends-watch
 https://blog.testim.io/summary-blog-how-ai-is-transforming-
software-testing/
 https://www.infoq.com/articles/test-automation-ai-ml
 https://www.akiraanalytics.com/blog/cognitive-solutions/ai-in-
software-testing/
 https://moolya.com/important-testers-learn-ai/
 https://sdtimes.com/ai/whats-beyond-continuous-testing-ai/

31.
Thank you for
listening!