Regarding the automation of mobile applications, we have two kinds of choices. The first is going with native automation frameworks such as Espresso for Android and XTCUITest for iOS applications, or going with non-native tools such as Selenium-based tooling (Appium, Selendroid, Robot Framework, many others), Calabash or also many other paid frameworks. I want to clarify the differences between native and non-native toolings for mobile UI automation. Both tooling ecosystems have advantages and disadvantages, so people are confused about which is best for them. Non-native tools such as Appium that dominated mobile automation for a long time and people have gained a lot of experiences with it so they want to continue using Appium, also Calabash which is older than Appium and it was also very popular for some time. On the other hand, these tools always need to deal with the latest update coming from the latest versions of the Android and the iOS ecosystem. If not those tools would be in history. For example, Calabash almost became useless for the latest versions after Microsoft acquired it and they stopped supporting it.

1. Mesut Güneş
www.testrisk.com
November 19, 2021
HUSTEF 2021
#hustef21

2. Mesut Güneş
www.testrisk.com
Motto: Be part of the development team by native way!
BENEFITS OF USING NATIVE AUTOMATION
TOOLING FOR MOBILE APPS

3. Mesut Güneş
www.testrisk.com
MESUT GÜNEŞ
Staff SDET in Mobile
in/gunesmes
gunesmes
story/gunesmes
gunesmes
testrisk.com
gunesmes@gmail.com

4. Mesut Güneş
www.testrisk.com
■ Test automation
■ Mobile testing
■ Mobile automation tools
■ Native mobile automation tools
■ Benefits of native tooling
TALK INCLUDE THESE TOPICS

5. Mesut Güneş
www.testrisk.com
TABLE OF
CONTENTS
MOBILE
AUTOMATION
ECOSYSTEM
What we have for automation
NON-NATIVE
SOLUTIONS
How non-native tools work
01
03
02
04
05
06
NATIVE
SOLUTIONS
How native tools work
BENEFITS: HANDLING
OBJECTS
What things are handled differently
BENEFITS:
PERFORMANCE
How we can maximize the
benefits
BENEFITS: TEAM
BUILDING
What kind of benefits we
can have

6. Mesut Güneş
www.testrisk.com
MOBILE AUTOMATION
ECOSYSTEM
01

7. Mesut Güneş
www.testrisk.com
Mobile applications are packages/binaries that can be installed to the users mobile
devices so that you can let them do what you want.
Mobile applications testing is installing the packages to the target devices and
performing test steps through the application in the devices.
Automation means that all the thing required for manual testing are done by
tools and scripts.
So mobile applications automation requires the followings:
● Application packages (apk / app / ipa)
● Mobile devices/emulator/simulator (Android / iOS)
● Networks
● Permissions
● etc.
WHAT IS MOBILE APPS AUTOMATION

8. Mesut Güneş
www.testrisk.com
What tests in what level we should test the apps.
TEST TYPES AND LEVELS
Test Level
Test Types
Functional Nonfunctional
Unit unit tests
load, stress, security, code
coverage (metric)
Integration
unit integration tests,
component
integration tests,
microservices
testing
load, stress, security, contract
test
System end-to-end test
load, stress, security, reliability,
maintainability, scalability
Acceptance
uat test, alpha - beta
testing
load, stress, security, usability,
AB test

9. Mesut Güneş
www.testrisk.com
Identify the purpose of the automation to get a better result
Automate everything that are possible and meaningful
● Classes/Methods
● Integrations
○ Unit
○ Component
○ System
● Requirements
○ Functional
○ Non-functional
● Processes
○ Development
○ Testing
○ Distributing
WHAT TO AUTOMATE

10. Mesut Güneş
www.testrisk.com
Regarding the the end-to-end testing automation, it can be done by the tools that can
interact to the element of the application inside the devices.
Basically we can classify these tooling as follows
● Native toolings
○ Available inside the development framework
● Non-native tooling
○ External tools that make automation possible
HOW TO AUTOMATE

11. Mesut Güneş
www.testrisk.com
NON-NATIVE SOLUTIONS
02

12. Mesut Güneş
www.testrisk.com
Non-native mobile automation tools are external tools and interact with the mobile
application object by a server connected to the mobile apps that are running inside a
mobile device/simulator/emulator.
Non-native tooling requires a server to translate automation commands to actions on
apps for black-box testing
● Depends on server-client communication
● Depends on some native libraries that are always upgraded by manufacturer
● Totally black-box testing technique
WHAT IS NON-NATIVE TOOLING

13. Mesut Güneş
www.testrisk.com
Non-native toolings work with http server to interact with mobile objects
Interaction can be done by two ways
● Interact with instrumentation/UI library
○ Espresso / UIAutomator / UIAutomator2
○ Robotium
○ XCUITests / UIAutomation
● Image recognition
○ Server take screenshot and find the relevant part of the screen
HOW NON-NATIVE TOOLS WORK

14. Mesut Güneş
www.testrisk.com
APPIUM
Appium is one of the most popular mobile automation solution.
Its popularity comes from the following advantages:
● Familiarity with the Selenium Webdriver
● Supports most of the popular languages
● Supports multiple platforms such as Android, iOS, Windows
● Easily extent one automation code base for all projects

15. Mesut Güneş
www.testrisk.com
APPIUM ARCHITECTURE
Appium is a non-native automation tool whose drivers use native automation
libraries in the http server.
Image: https://digital.ai/catalyst-blog/what-is-appium-server-a-complete-end-to-end-guide

16. Mesut Güneş
www.testrisk.com
CALABASH
Calabash is another mobile automation tool for black-box testing. It was very
popular until 2016 but now it almost become history.
Its popularity comes from the following advantages:
● Behaviour Driven test framework for Mobile automation
● Open-Sourced project: Calabash-android and Calabash-ios
● Supports iOS and Android and Hybrid Apps (Webviews) - (Cross-platform)
● Supports simulators / emulators / real devices
● Supports CI / CD and Clouds - (CLI tools)
● Ruby Based
● Integrated Cucumber
● Calabash Console for real time automation development

17. Mesut Güneş
www.testrisk.com
CALABASH ARCHITECTURE
Calabash is also a non-native automation tool. Android version uses Robotium
which is an external automation library and iOS version uses XCUITest that is a
native framework.
Calabash-Android Calabash-iOS

18. Mesut Güneş
www.testrisk.com
NATIVE SOLUTIONS
03

19. Mesut Güneş
www.testrisk.com
Native mobile automation toolings are integrated libraries to the mobile application
development framework so that developers can write UI tests for the products
under development.
Native toolings are part of the development frameworks so it is guarantee that the
tools are always up to date with the development frameworks and upcoming new
features.
Since we are talking about two major mobile device operating system, we are just
focus on the following platforms:
● Android
○ Espresso
● iOS
○ XCUITest
WHAT IS NATIVE TOOLING

20. Mesut Güneş
www.testrisk.com
HOW NATIVE TOOLS WORK
Native tools works natively, which means that they are providing direct methods via
APIs to use inside the development project globally.
Interactions are made by directly calling the UI objects inside project architecture
by the APIs.
So these are the main benefits:
● APIs know the state of the objects
● We are free to use tool as black-box as well as
contributing the development code as white-box
● We are free to reach the UI objects
● If the app is the Matrix, native tools are the agents
inside the matrix

21. Mesut Güneş
www.testrisk.com
ESPRESSO - WHAT
Espresso is a test automation tool developed by Google for Android UI Tests.
Espresso is open-source, very stable and there many frameworks have been
developed around Espresso.
● It has full feature-set of Junit that works with Espresso
● Android studio provides everything you need
● It provides white-box testing

22. Mesut Güneş
www.testrisk.com
ESPRESSO - HOW

23. Mesut Güneş
www.testrisk.com
XCUITEST - WHAT
XCUITest stands for Xcode User Interface Testing and it is UI automation
framework iOS applications. XCUITest is developed on top of XCTest and
Accessibility. It is launch by Apple in 2015. It supports all iOS functionality in
terms of testing them by white-box testing techniques.
● XCUITest is native and inside the app source code with different target
● XCode provides everything you need
● It provides white-box testing
● XCTest provides everything you need for testing capabilities
● Accessibility provides UI interaction for all kind of XCode product

24. Mesut Güneş
www.testrisk.com
XCUITEST - HOW

25. Mesut Güneş
www.testrisk.com
BENEFITS: HANDLING OBJECTS
04

26. Mesut Güneş
www.testrisk.com
Native toolings are inside the projects so no need to worry about the use external tool
to get the related objects
Espresso and XCUITest provides the following benefits
● APIs provide methods for all kind of object
● Methods are always up-to-date with latest improvements/updates
● Freely update the source code for the sake of ui-test
NATIVELY AVAILABLE OBJECTS

27. Mesut Güneş
www.testrisk.com
IDEs ARE FRIENDLY

28. Mesut Güneş
www.testrisk.com
BENEFITS: PERFORMANCE
05

29. Mesut Güneş
www.testrisk.com
The performance is very important when you grow and have to handle hounderents
tests. The main point for the performance is that how much time your tests require for
running them locally and in a cloud services
● Simple UI frameworks interact with the objects so no need to translate with
servers
● Native tooling know the state of the elements so no need to check them
regularly
● Automation code is inside the test package so no need to handle it
differently while testing on the cloud. This reduce the time drastically
SERVERS --- NO

30. Mesut Güneş
www.testrisk.com
Simple test to compare performance of the tools shows that native toolings are always
better than Appium. This was done by TestProject engineers.
Reference: https://blog.testproject.io/2020/04/08/fastest-mobile-automation-testing-tool-appium-vs-
xcuitest-vs-testproject-vs-uiautomator-vs-espresso/
COMPARE THE PERFORMANCE

31. Mesut Güneş
www.testrisk.com
BENEFITS: TEAM BUILDING
06

32. Mesut Güneş
www.testrisk.com
Better performance from an automation project can be only get when the developers
and testers work cooperatively. Developers are are always result oriented people so
they focus on the development of the features, but the QAs/SDETs must be keen on
the quality. This quality not only the product quality but also quality of the tests,
process, and the documentation. This can be achieved by cross-functional-team
building.
Cross-functional-team requires the followings:
● Team spirit
● Shared common knowledge, every team members
should know basics about
○ What is being built
○ How the things are developed
○ How the things are being tested
● Absence of one person from a role can be filled by from
another role
○ QA/SDET can be developer
○ Developer can be QA/SDET
BUILD TEAM AROUND QUALITY

33. Mesut Güneş
www.testrisk.com
How I am lucky to get those feedbacks from experienced mobile developers :)
● DEV provides testable features
● QA/SDET writes test
● When QA/SDET get familiarity with whole project, s/he can update/add missing
parts for deterministic test
● Dev are ready for making test framework better :)
BUILD TEAM AROUND QUALITY

34. Mesut Güneş
www.testrisk.com
CREDITS: This presentation template was created by Slidesgo, including
icons by Flaticon, and infographics & images by Freepik.
Please keep this slide for attribution.
Does anyone have any questions?
THANKS
References:
● https://developer.android.com/training/testing/espresso
● https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual
/testing_with_xcode/chapters/09-ui_testing.html
● https://appium.io/docs/en/about-appium/intro/
● https://www.browserstack.com/guide/getting-started-xcuitest-framework
● https://www.browserstack.com/guide/appium-vs-espresso
● https://blog.testproject.io/2020/04/08/fastest-mobile-automation-testing-tool-appium-
vs-xcuitest-vs-testproject-vs-uiautomator-vs-espresso/
● https://devexperts.com/blog/ios-ui-testing-frameworks-performance-comparison/