A presentation for: Northern Ireland Developer Conference October 9-10, 2020 www.nidevconf.com By: Vahid Garousi Bahar Software Engineering Consulting Corporation, UK Queen’s University Belfast, UK www.vgarousi.com Alper Buğra Keleş, Yunus Balaman, Zeynep Özdemir Güler Testinium A.Ş., Istanbul, Turkey https://www.linkedin.com/in/alperbugrakeles/ www.testinium.com

1. 1
Next level of test automation with
Model-based Testing (MBT):
Experience and best practices
Vahid Garousi
Bahar Software Engineering Consulting Corporation, UK
Queen’s University Belfast, UK
v.garousi@qub.ac.uk
www.vgarousi.com
@vgarousi
Alper Buğra Keleş, Yunus Balaman, Zeynep Özdemir Güler
Testinium A.Ş., Istanbul, Turkey
{alper.keles, yunus.balaman, zeynep.ozdemir}@testinium.com
www.testinium.com
A presentation for:
Northern Ireland Developer Conference
October 2020

2. 2
Outline
 A review of our industry context
 The needs for model-based testing (MBT)
 Our approach (Solution): Applying MBT
 Assessing the benefits of MBT
 Current/next steps
 Take-away messages

3. 3
A review of our industry context
 Testinium A.Ş. is a provider of software testing
services and solutions in Turkey, UK and across
Europe
 Founded in year 2010
 Has provided automated testing to 60+ clients in several
countries so far
 As of August 2020, has over 190 employees
 Almost all test engineers have the ISTQB certificates
 In addition to the test service offered to clients, the
company has developed and offers two flagship test
automation tools:
 Testinium (testinium.com): A web-based test management
system with many features, e.g., a web interface to Selenium
and Appium
 Loadium (loadium.com): A web-based load test management
system
And many other
clients….

4. 4
One of the flagship test automation tool:
Testinium (testinium.com)

5. 5
Test automation approaches and tools, in use in
the company since 2010
 Takes Behaviour-driven Development
(BDD) to a higher levelgauge.org
selenium.dev
And many other test automation approaches and tools
testinium.com
• For test-case design, we use established black-box testing approaches,
such as category partitioning and boundary-value testing
• But, as also reported by other testers, making sure that all the black-box
test-case design approaches are followed systematically by all test
engineers, is not always easy
• Thus, we have seen the need for “systematic” test-case design, thus the
need for this project! (more next)

6. 6
Outline
 A review of our industry context
 The needs for model-based testing (MBT)
 Our approach (Solution): Applying MBT
 Assessing the benefits of MBT
 Current/next steps
 Take-away messages

7. 7
The needs (motivations) for MBT
 While test automation tools are effective for automated execution of
tests, tools alone are not enough for a successful test automation
outcome
 If test cases are not designed properly, a test tool will only
automatically run bad-quality (read: useless) tests!
 We need “systematic” test-case design
 Imagine: We can have two testers:
 Tester A designs 10 test case which would detect 20 bugs (defects)
 Tester B designs 100 test case which would detect 2 bugs (defects)
 Which one is a more effective test engineer?
 Model-based testing (MBT) is a suitable approach for this need!
Question: How many of
you think this statement
is true?

8. 8
Outline
 A review of our industry context
 The needs for model-based testing (MBT)
 Our approach (Solution): Applying MBT
 Assessing the benefits of MBT
 Current/next steps
 Take-away messages

9. 9
What is Model-based Testing (MBT)?
 Model-based testing:
 Designing test “models”: GUI flow-diagrams of a given Software Under Test (SUT)
 Nodes, or vertices. Edges or transitions
 A small example:
 Benefits:
 Automating both the test-case design and test-case execution
 Deriving almost all possible test paths is a very efficient way (would be very effort intensive to do in
manual testing, or even when developing automated test scripts manually)
Click
Note: MBT is popular…

10. 10
Model-based Testing (MBT)
 The entire login test model for the SUT: Testinium (testinium.com)…

11. 11
Choosing the “right” MBT tool
 There are literally hundreds of MBT tools
 Which MBT tool is the “right” choice for a given
context?
 Type of SUT
 Testing needs
 Other factors
 A non-trivial question
 After a systematic comparison of several tools, we
selected GraphWalker (graphwalker.github.io)
 It’s free and open-source and has a wide user base

12. 12
Time for a quick demo!
 Several online videos from our MBT projects are on
YouTube: bit.ly/MBTVideosTestinium
 The entire test suite with all the code is open-source:
github.com/vgarousi/MBTofTestinium

13. 13
An overview of the MBT approach
 The test architecture:
Web browser
Front-end of the
web application
System Under
Test (SUT)
Call Return
Exercise
(call)
Return
Test
outputs
Return
Anlayzes
GraphWalker
Test models
Designs
Test enigneer
Run using...
Java Selenium
methods
Implementaion of
nodes/edges behavior
in Java using the
Selenium framework
Call
Return
Back-end of the
web application
SUT
Develops
Requirements
(Use-case
diagram)
Uses

14. 14
Development of nodes/edges’ behavior in Java using the
Selenium framework
public void e_valid_login() {
WebElement userNameElement =
methodsPage.findElement(By.id("username"));
userNameElement.clear();
userNameElement.sendKeys(email);
WebElement passwordElement =
methodsPage.findElement(By.id("password"));
passwordElement.clear();
passwordElement.sendKeys(password);
methodsPage.findElement(By.cssSelector(
"input[class$="login-page__submit-btn"][value="Sign In"]")).click();
}

15. 15
Report from one MBT project
 SUT: Our own test automation tool: Testinium
(testinium.com)
 Testing a testing tool!
 Size measures (metrics) from the MBT project:
 # of test models = 18
 Total # of nodes=170
 Total # of edges=260
 Size of the Selenium Java code for nodes/edges’ behavior = 4,699
LOC
 Each test run takes about: 6 hours, going through ~28,000 test
steps! (nodes and edges, stopping condition=%100 edge coverage)
 Reminder: The entire test suite with all the code is open-
source: github.com/vgarousi/MBTofTestinium

16. 16
Outline
 A review of our industry context
 The needs for model-based testing (MBT)
 Our approach (Solution): Applying MBT
 Assessing the benefits of MBT
 Current/next steps
 Take-away messages

17. 17
Assessing the benefits of MBT
 Tangible benefits of MBT
 Improved test-case design practices, due to MBT
In the old approach (manual test-case design), many deficiencies were identified, e.g.,
duplications among different test suites (could lead to test integrity problems, etc.), and many
missing test paths and test cases
 Detecting real fault during regression testing
Several defects were detected, via “Endurance testing” using MBT (soak testing, longevity
testing)
 Measuring code coverage:
Both in client side (JS) and server side (Java)
 Intangible benefits of MBT
 According to the feedbacks we have received: MBT has made the
work of test engineers “more interesting” and more organized
 Many in the company have told us that, thanks to MBT models, they
can now see the “big picture” much more easily with having the test
models in front of them, and the model being directly executable.

18. 18
Current / next work directions
 Our MBT project has showed us several important ongoing work
directions, e.g.:
 Test visualization (showing the number of times each edge and node has been
covered)
 We are in the process of improving the testing tool to incorporate fault tolerance
(when an assertion fails)
 Assessing effectiveness of MBT in detection of injected faults (by mutation
testing).
 Developing a code coverage tool for clients side (JS ) and server side

19. 19
Take-away messages
 In retrospective, using MBT, we have observed various
benefits, compared to the “before state” of test automation
practices (in which test-cases were designed in ad-hoc
manner)
 MBT has helped us:
 By providing clarity and road-map (what to do next) for test engineers
 Resulting in higher-quality test scripts in term of modularity with no test-
code duplication
 Taking test automation to a higher level of maturity
Pre-MBT MBT
State of test
automation in the
company:

20. 20
This work is a part of a large international EU project
 The TESTOMAT project: The Next Level of Test Automation
 www.testomatproject.eu
 Among different topics of the project:
 Model-based testing
 Test prioritization and selection, etc.
 Project is funded by the European ITEA3 (Information Technology for
European Advancement) program and the government of the six
countries involved in the project
 “ITEA is a transnational and industry-driven R&D&I programme in the
domain of software innovation [in Europe]”
 An R&D collaboration among 34 entities (software companies
and universities) across Europe

21. 21
END of the talk
Questions /
Answers
Vahid Garousi
Queen’s University Belfast, UK
Bahar Software Engineering Consulting Corporation
v.garousi@qub.ac.uk
www.vgarousi.com
@vgarousi
Alper Buğra Keleş, Yunus Balaman, Zeynep Özdemir Güler
Testinium A.Ş., Istanbul, Turkey
{alper.keles, yunus.balaman, zeynep.ozdemir}@testinium.com
www.testinium.com