Maximize the Benefits from Your Test Automation Investment

Maximize the Benefits from
Your Test Automation
Investment

14 November 2013
Ville-Veikko Helppi
Technical Product Manager
ville-veikko.helppi@bitbar.com

1

Agenda
• Common Challenges in Today’s App Lifecycle
• How Test Automation Can Shorten Your App’s
Time-to-Market and Improve the Quality?
• Return on Investment with Test Automation
• Different App Verticals – What is Important?
• 10 Tips for Professional Mobile App Testing
• Testdroid Update
• Q&A
© Copyrights by Bitbar Technologies Ltd. 2013
All rights reserved.

3

Agenda
• Q&A

4

Challenges in Today’s App Dev
Fact: Most typical application fails when tested
across different device variants used by consumers
• Failures: Both SW and HW cause failures
• Emulators do not offer realistic platform for dev
• Already installation of app fails – not tested at all
• Application crashes, leaks memory, consumes too
much resources, behaves not as meant to behave
• UI scaling/display/resolution – common problem
• Application performance is bad, draining battery

5

Why Do They Fail?


6

Agenda
• Q&A

7

How to Test: Manual vs. Automation
• Costs & Assets
– Time, People, Infrastructure, Tools, Training

• App Development & Testing Cycle – Together
– Testing after Daily Builds (Continuous Integration)

• Tools & Technology – Frameworks
– Myriad of frameworks available to help you

• Test Coverage & Reusability
– Global coverage is CRITICAL to SUCCESS

• Outcome – Learning, Improving, Faster

8

Manual

Automation
Large
coverage, quick
ly
completed, Less
money & time
wasted, Exact
results.

Smaller coverage, More
money burnt & time
wasted, Error-prone

9

Improving Time-to-Market

Repository

Successful
build

Commit

Users / Consumers

Build
failed

Test Cases
Application
QA & Testers

Error reporting

Source code
Developers


10

Tips to Improve the App Quality
• Automate generic things as much as you can
• During the development your app changes –
the same must apply for testing!
• Carefully select (testing) technology & partner
• Use all 24 hours per day with test automation
• Cloud-based platform is the only solution to
get you quickly covered in the global markets

11

Test Automation Will Deliver You…
• App Compatibility & Global coverage
Test requirements / Planning

Smoke testing

Regression testing

Regression testing

Functional & Interoperability
testing

Regression testing

testing

testing

Performance & Stress testing

Regression testing

Compatibility testing

Performance & Stress testing

Test Automation during the Mobile App Development Process

• Reusability
• Test efficiency & effectiveness
• Faster time-to-market

12

Agenda
• Q&A

13

Example: Cost of Manual Testing
• Test setup for each device – 5 minutes
• 100 test cases, each take about 15 seconds to
execute – 25 minutes
• 4 different application variants: tablet, phone and
2 localized versions (English & German)
• Avg. 2 bugs (of 100 test cases) are found on each
test run – 20 minutes to report bugs
• 15 devices will be used for testing
TIME FOR TESTING: 3000 min (50 hours)
COST FOR TESTING: $60/h x 50 = $3,000 (x4) = $12K

14

Example: Cost of Test Automation
• Creation of test cases/scripts – 100 test cases running 15
minutes per each = 1500 minutes (25h)
COST FOR TEST CREATION: 25 hours x $60/h = $1,500
• Tools for test automation with the following setup:
– Unlimited devices (250+ Android devices)
– Unlimited execution times

TESTDROID CLOUD: $1,500/month
TOTAL COSTS FOR TEST AUTOMATION: $3,000/month
Direct savings compared to manual: $9,000
ROI ratio: (Gains-Investm.) / Investm.: $9K-$3K/$3K = 2.00

15

Benefits with Testdroid Cloud
• Instant availability of hardware and no need to
buy any device by yourself
• Testing gets done much faster – typically the
same outcome can be achieved in 10% of time
• Test cases are reusable for your next project
• Immediate feedback – all reports as
logs, screenshots, performance stats etc.
• QA can focus on quality – not pressing keys!

16

Example: On-Premise Setup
• Creation of test cases/scripts – 100 test cases running 15
minutes per each = 1500 minutes (25h)
COST FOR TEST CREATION: 25 hours x $60/h = $1,500
TESTDROID ENTERPRISE: $2,000/month (15 devices)

TOTAL COSTS FOR TEST AUTOMATION: $3,500/month
Direct savings compared to manual: $8,500
ROI ratio:
(Gains-Investm.) / Investm.: $8.5K-$3.5K/$3.5K = 1.43

17

Agenda
• Q&A

18

Mobile Apps for Different Verticals
Games
Utilities & Tools
Video Streaming & Multimedia

Banking & Payment
Retail & Travel

• Testdroid has helped
thousands of app
developers in these
subverticals!
• What are the critical
elements in each these
verticals?

Mobile

19

Games
Utilities & Tools

Banking & Payment
Retail & Travel
Mobile

• User Experience!
• Resource consumption &
validation (CPU, Mem…)
• Fully utilizing hardware –
e.g. Touch screen
• Mobile games require
access to graphics APIs
(e.g. OpenGL ES)


20

• Usability!
• Functional and
Utilities & Tools
Behavioral testing
• Metrics analysis for all
captured data (e.g.
Banking & Payment
logs, screenshots, perf
Retail & Travel
stats)
Mobile
• Connection with Social
Media + other apps
Games


21

• Performance!
• Connectivity, robustness
Utilities & Tools
and durability
• Screen orientation
(portrait vs. landscape)
Banking & Payment
• Graphics
Retail & Travel
quality, streaming
Mobile
capabilities
• Technologies Ltd. 2013
© Copyrights by Bitbar User profiles
Games


22

Games
Utilities & Tools

Banking & Payment
Retail & Travel
Mobile

• Security!
• Secure connectivity with
back-end systems
• Top Quality – extremely
brand sensitive vertical
• Compliances and
verification between real
devices and infrastructure


23

Games
Utilities & Tools

Banking & Payment
Retail & Travel
Mobile

• Data!
• Connectivity and data
connection with back-end
systems
• Configurability of the app
• Bad quality can hurt the
brand and leave you
without customers


24

Agenda
• Q&A

25

Tip #1: Test early, Test often


27

Tip #2: Plan What to Automate
Time

MANUAL

Infrastructure
Tools

AUTOMATED

People

Training


28

Tip #3: Use only Real Devices


29

Tip #4: Use Atomic Test Units

Test iteration #1

Test iteration #2


30

Tip #5: Create Hermetic Tests


31

Tip #6: Use All Possible Devices

Testdroid Cloud’s 250+ Android devices
= 93-95% global Android volumes!

32

Tip #7: Separate Apps and Tests

Your App


33

Tip #8: Output Everything to Logs


34

Tip #9: End-to-End Testing


35

Tip #10: Integrate Dev & Testing
Repository

Successful
build

Commit

Users / Consumers

Build
failed

Test Cases
Application
QA & Testers

Error reporting

Source code
Developers


36

Agenda
• Q&A

37

Testdroid Products
Complete Solution for Mobile Apps/Games Testing


38

Testdroid
– Because it is important to know what app testing costs!


39

Agenda
• Q&A

40


41

Maximize the Benefits from Your Test Automation Investment

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (19)

En vedette

En vedette (16)

Similaire à Maximize the Benefits from Your Test Automation Investment

Similaire à Maximize the Benefits from Your Test Automation Investment (20)

Plus de Bitbar

Plus de Bitbar (17)

Dernier

Dernier (20)

Maximize the Benefits from Your Test Automation Investment