The Basics of Mobile App Testing & Test Automation

The Basics of
Mobile App Testing
& Test Automation
26 March 2014
Ville-Veikko Helppi
Technical Product Manager
ville-veikko.helppi@bitbar.com

Agenda
• App Quality and Mobile App Testing
• How To Get Started with Test Automation?
• Test Automation Frameworks
• 10 Tips for Successful App Testing
• Testdroid Update
• Demonstration
• Q&A
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
2

Agenda
• Demonstration
• Q&A
3

Challenges in App Development
4
Fact: Most typical application fails when tested
across different device variants used by consumers
• Failures: Both SW and HW cause failures
• Emulators do not offer realistic platform for dev
• Even app installation fails – not tested at all
• Application crashes, leaks memory, consumes too
much resources, behaves not as meant to behave
• UI scaling/display/resolution – common problem
• Application performance is bad, draining battery

Why Apps Fail?
5

Testdroid Research Available Soon…
6

What Improves the App Quality?
• Automate generic things as much as you can
• During the development your app changes –
the same must apply for testing!
• Carefully select (testing) technology & partner
• Use all 24 hours per day with test automation
• Cloud-based platform is the only solution to
get you quickly covered in the global markets
7

Mobile Apps for Different Verticals
Mobile
Games
Utilities & Tools
Video Streaming & Multimedia
Banking & Payment
Retail & Travel
• Testdroid has helped
thousands of app
developers in these
subverticals!
• What are the critical
elements in each these
verticals?
8

Mobile
Games
Utilities & Tools
Banking & Payment
Retail & Travel
• User Experience!
• Resource consumption &
validation (CPU, Mem…)
• Fully utilizing hardware –
e.g. Touch screen
• Mobile games require
access to graphics APIs
(e.g. OpenGL ES)
9

Mobile
Games
Utilities & Tools
Banking & Payment
Retail & Travel
• Usability!
• Functional and
Behavioral testing
• Metrics analysis for all
captured data (e.g.
logs, screenshots, perf
stats)
• Connection with Social
Media + other apps© Copyrights by Bitbar Technologies Ltd. 2014
10

Mobile
Games
Utilities & Tools
Banking & Payment
Retail & Travel
• Performance!
• Connectivity, robustness
and durability
• Screen orientation
(portrait vs. landscape)
• Graphics
quality, streaming
capabilities
• User profiles© Copyrights by Bitbar Technologies Ltd. 2014
11

Mobile
Games
Utilities & Tools
Banking & Payment
Retail & Travel
• Security!
• Secure connectivity with
back-end systems
• Top Quality – extremely
brand sensitive vertical
• Compliances and
verification between real
devices and infrastructure
12

Mobile
Games
Utilities & Tools
Banking & Payment
Retail & Travel
• Data!
• Connectivity and data
connection with back-end
systems
• Configurability of the app
• Bad quality can hurt the
brand and leave you
without customers
13

Agenda
• Demonstration
• Q&A
14

Prerequisites for Mobile App Testing
• Testing environment as authentic as possible
– Real devices <-> emulators/simulators
• Test coverage needs to be adequate
– Software is not identical in all Android/iOS devices
– Hardware is even more fragmented landscape
• Surrounding testing conditions (e.g. network)
• Try not to build in repetitive, manual processes
• Test automation framework that works for you
15

Trade-offs – Manual vs. Automation
• Costs & Assets
– Time, People, Infrastructure, Tools, Training
• App Development & Testing Cycle – Together
– Testing after Daily Builds (Continuous Integration)
• Tools & Technology – Frameworks
– Myriad of frameworks available to help you
• Test Coverage & Reusability
– Global coverage is CRITICAL to SUCCESS
• Outcome – Efficiency, Effectiveness and Results
16

Manual Automation
Smaller coverage, More
money burnt & time
wasted, Error-prone
Large
coverage, quick
ly
completed, Less
money & time
wasted, Exact
results.
17

What Framework Works You The Best?
• High-level & declarative test automation
frameworks work well when app is under dev
• Easy to integrate with continuous integration
• Distance between use case and actual test
• What do you build: Native, Hybrid or Web app?
• The technical competence in your organization
• Desired outcome from test automation
18

Different Ways of Doing Test Automation
Automatic test exercisersRecord and PlaybackHand written test scripts
Benefits:
Accurate, specific to your testing
needs, plenty of options with
frameworks, tools
Fast to create, accurate, not as
sensitive to human-errors as
hand-written tests, tools avail’ty
Fastest & extremely automated,
excellent for smoke testing/quick
testing, availability
Tradeoffs:
Takes a lot of time, ties resources
to write test cases/scripts, error-
prone (humans)
Compelling Recorder+Playback
tools available for only few test
automation frameworks
Not accurate as real test cases
19

Developers
Source code
Repository
Build
failed
Successful
build
QA & Testers
Error reporting
Test Cases
Application
Users / Consumers
Commit
20
Test Automation During the Dev

Developers
Source code
Repository
Build
failed
Successful
build
QA & Testers
Error reporting
Test Cases
Application
Users / Consumers
Commit
21
Test Automation During the Dev

Why Real Devices are Must-to-Have?
• Emulators/simulators won’t help you to test…
– User Experience and Usability
– Hardware
– Software
– Infrastructure
0 % = the percentage of your app users
that use emulator to run your app!
22

Test Automation Will Deliver You…
• App Compatibility & Global coverage
• Reusability
• Test efficiency & effectiveness
• Faster time-to-market
Test Automation during the Mobile App Development Process
Testrequirements/Planning
Smoke testing
Regression testing Regression testing Regression testing Regression testing
Functional & Interoperability
testing
testing
testing
Performance & Stress testing Performance & Stress testing
Compatibility testing
23

Agenda
• Demonstration
• Q&A
24

Android Instrumentation Framework
• InstrumentationTestRunner –
the primary plumbing for
running tests on Android
• Android Instrumentation
Framework is built on top of
JUnit – a standard test
framework on for any Java
development
• Mock objects – methods for
creating mock system objects
such as content, service and
intent
25

iOS Instruments (with Xcode)
26

Android Example: The Family Tree
of Test Automation Frameworks
JUnit
Android Instrumentation Framework
Robotium Espresso
UI Automator
Calabash
Appium
27

Comparison of Test Automation Frameworks
Supports Calabash Robotium uiautomator Espresso Appium
Android Yes Yes Yes Yes Yes
iOS Yes No No No Yes
Mobile web Hybrid
(webviews)
Yes
(Android)
Limited to x.y
clicks
No Yes
(Android & iOS)
Scripting
Language
Ruby Java Java Java Almost any
Test creation
tools
CLI Testdroid
Recorder
UI Automator
viewer
Hierarchy
Viewer
Appium.app
Supported
API levels
All All 16 => 8, 10, 15-19 All
Community Active Contributors Google Google Active
28

Agenda
• Demonstration
• Q&A
29

Tip #1: Test early, Test often
30

Tip #2: Plan What to Automate
People
Infrastructure
Tools
Training
Time
MANUAL
AUTOMATED
31

Tip #3: Use only Real Devices
32

Tip #4: Use Atomic Test Units
Test iteration #1 Test iteration #2
33

Tip #5: Create Hermetic Tests
34

Tip #6: Use All Possible Devices
Testdroid Cloud’s 300+ Android devices
= 93-95% global Android volumes!
35

Tip #7: Separate Apps and Tests
Your App
36

Tip #8: Output Everything to Logs
37

Tip #9: End-to-End Testing
38

Tip #10: Integrate Dev & Testing
Commit
Developers
Source code
Repository
Build
failed
Successful
build
QA & Testers
Error reporting
Test Cases
Application
Users / Consumers
39

Agenda
• Demonstration
• Q&A
40

41

Agenda
• Demonstration
• Q&A
42

Agenda
• Demonstration
• Q&A
43

44

The Basics of Mobile App Testing & Test Automation

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (16)

Plus de Bitbar

Plus de Bitbar (20)

Dernier

Dernier (20)

The Basics of Mobile App Testing & Test Automation