7. Possible objectives of test automation:
- find more bugs
- nightly regression tests
- reduce testing staff
- shorter test periods
- test more
Frequent problems of test automation: [1]
1. unrealistic expectations (of the management)
2. poor testing processes („Automating chaos just gives faster chaos“)
... etc.
Let us examine our goals more exactly!
1: Software Test Automation Effective use of test execution tools – Dorothy Graham & Mark Fewster
7
8. find more bugs
the idea:
manual testing finds bugs
automated tests are faster than manual
tests
so automated tests find more bugs
8
9. find more bugs
the practice:
- automated tests are repeated tests
- looks for unexpected side effects
- so automated tests can find
more regression bugs
- testers have more time for manual
tests
- poor tests are not getting better by
automating them
9
10. nightly regression tests
the idea:
There are unused ressources
(like test systems e.g.)
We could run automated tests „while
we sleep“
10
11. nightly regression tests
the practice:
the correct tests have to be selected
test results need to be analyzed
11
12. reduce testing staff
the idea:
The automation tool costs money, so we
should be able to save somewhere
else
We want to reduce costs and staff
is expensive
12
13. reduce testing staff
the practice:
tests have to be automated and maintained
test results must be analysed
test automation can make work easier
for testers
13
14. shorter test periods
the idea:
reduce deadline pressure
testing is a „bottleneck“, so we will save
time there
14
15. shorter test periods
the practice:
the goal can be achieved more easily:
run fewer tests, omit long tests,
cut regression testing
Automated tests take time:
creation and maintenance of test
cases
analysis of test results
software quality has the greatest influence on the time needed
for testing
15
17. test more
the practice:
unimportant tests produce more
Maintenance instead of being gainful
some tests require higher
automation effort,
e.g. for technical reasons
17
18. Possible realistic objectives
- find more regression bugs
- automate the most importants tests
- a useful amount of tests run at night and / or weekend
- relieve testers and support manual testing
It is important to know why you automate, so you know what and how to
automate.
Objectives of testing should not get mixed up with the objectives of
automation.
Test automation is not a tool against bad test practices
That‘s No Reason To Automate! Why Good Objectvies Are Critical to Test Execution Automation – Dorothy Graham and Mark Fewster
18
19. 04 | HOW DO I KNOW, THAT I‘M DOING
IT RIGHT?
19
20. measuring
metrics
Depending on the objective and the initial situation other metrics are
important.
Does this one consume as much petrol
as this?
Will this car fit into my garage?
Yes?
There might be a hole in the tank.
20
21. 1. DDP– Defect Detection Percentage
Is the relation between found and currently known bugs.
Example: A release contains 100 new bugs, the test run finds 70 bugs:
DDP: 70%
Weaknesses in this approach:
1. The absolute amount of bugs is unknown.
2. It is unknown at what time a bug has come into the product.
3. Not all bugs are equally critical.
4. Not all bugs are equally difficult to find.
5. No reflection of the absolute count.
It is easy to determine this measure and it can have a high significance.
21
22. 1. ROI – Return on investment for process optimization
current process optimized process
Costs of test execution 10 000 € 20 000€
DDP 70% 90%
found bugs 700 900
bugfixing (100 € per bug) 70 000 € 90 000 €
bugs found after release 300 100
bugfixing after release (1000 € per bug) 300 000 € 100 000 €
total costs 380 000 € 210 000 €
savings: 170 000 € investment: 10 000€
ROI (savings/investment): 1700%
22
23. 1. ROI – Return on investment in test automation
manual testing automated testing
Costs for test case design 6 000 € 6 000 €
costs for automation 0 16 000 €
total one-time expenses 6 000 € 22 000€
costs for test execution 5 000 € 1 000 €
amount of test execution cycles 3 3
costs per release 15 000 € 3 000 €
total costs of 4 releases a year 66 000 € 34 000 €
savings per year: 32 000 € investment: 16 000€
ROI (savings/investment): 200%
23
25. How do you get that idea?
- I have a lot of test cases
- which run every night for several
hours
- and provide lots of test results
25
26. - Well, I have:
- saved more hours of manual testing
(effectiveness)
- spent less time on test case
maintenance (maintainability)
- more bugtickets per fault report
(reliability)
- less training time for new employees
(usability)
26
27. Hm…
- I need to edit many tests for each
release, so that they can be
executed
- I need 4 hours a day to analyse
my logs files
- if I find a bug, it is most likely
caused by my test environment
- my new colleagues need 2 weeks
of training before they can help me
- and after every release we have
to fix many bugs that were missed
during testing
27
28. And then I have:
- less effort to find the test case that
tests a specific range (flexibility)
- fewer test cases blocked by the same
software bug (robustness)
- less effort to run test cases on
different test environments
(portability)
28
31. We have 18 test suites for:
- conversion path tests, submit poker accounts, support ticket system,
fraud check, upload images, buy points packages, video section…
- 865 performed tests
- 9 hours test run every night
However, we have two test systems (integration and system test) and almost
all tests run in Internet Explorer and Firefox.
- about 3400 test executions
- 36 hours of test runs in total
- 5 virtual machines for test execution
31
33. 2. efficiency
- before the introduction of test automation:
12 hours of manual regression testing for each release and test
environment
- today:
4 hours of manual regression testing for each release and test
environment
3. reliability
solved problems with external dependencies (Facebook, Google ...)
- requests are routed via the host file on an internal web server
- answers with 1x1 pixel images or empty responses to scripts
33
34. 4. flexibility
- a test suite per functional section
- each test suite can be started by pressing a button in Jenkins
- free choice of browser and the test environment
5. usability
- graphical abstraction of test scripts in QF-Test
- using Jython or Groovy scripts (database access, HTML)
6. robustness
- 3 test suites are currently on “failed", there are 2 bugtickets
- a test suite is unfortunately affected by a bug in Firefox
34
35. 7. portability
- choice of the test environment and the browser via parameter
- at the moment QF-Test supports only Internet Explorer and Firefox
35
