Watch a live presentation at http://offer.bitbar.com/best-practices-for-devops-in-mobile-app-testing In essence, the core of DevOps methodology aims to speed up the app development delivery and process by getting devs and operation specialists to collaborate throughout the end-to-end app development and deployment process. Stay tuned and join our upcoming webinars at http://bitbar.com/testing/webinars/

1. 3 March 2016 – 12pm GMT
Ville-Veikko Helppi
Head of Demand Generation
ville-veikko.helppi@bitbar.com
Best Practices for DevOps
in Mobile App Testing
W E B I N A R

2. Agenda
• Deconstructing ‘Mobile DevOps’
• Mobile Product Development
Life-Cycle & Used Tools
• The DevOps Approach in Mobile
App Development & Testing
• Best Practices for DevOps in
Mobile App Testing
• Q&A
http://testdroid.com

3. DECONSTRUCTING “MOBILE DEVOPS”
http://testdroid.com

4. Mobile DevOps
DevOps = Practices that emphasize the collaboration and communication of both
software developers and other information-technology (IT) professionals while
automating the process of software delivery and infrastructure changes. --Wikipedia
• Development focus on
building the actual product
and using relevant tools
• QA doing continuous testing
on robustness, functionality,
performance, and preferably
use test automation
• Operations maintain the
continuous
build/integration/deployment/
delivery environments, and
do releases

5. 3 Things in Mobile DevOps
• 3 DevOps Principles
1. Develop & Test Against Real Environment
2. Deploy Frequently
3. Continuously Validate Quality Characteristics
• 3 Business-Critical DevOps Categories
1. Continuous Integration & Continuous Delivery
2. App and Infrastructure (e.g. Back-End) Testing & Monitoring
3. Entire Dev&Test Process and Mobile App Delivery
http://testdroid.com

6. DevOps Strategy & Typical Goals
• Maximize the Delivered Value
• Customers, Internally between BUs and Teams
• Maximize the Efficiency and Productivity
• Improved Process, The ‘Right’ Tools & Methods, Faster T2M
• Maximize the Quality and Robustness of Apps
• Better Quality with Equal Effort, Investments in Quality
• Maximize the Support, Service & The Delivery
• Alignment and Collaboration between Teams
The highest priority is to satisfy the customer through
early and continuous delivery of valuable software.
Agile Manifesto – Principle #1
“ “http://testdroid.com

7. How DevOps Can Make A Difference
• Based on Testdroid Customer Survey 2015
• Things That Go Wrong with App Testing
1. Too Long Dev-Test-Feedback Cycles
2. Bug Catching/Reporting Too Slow to Developers
3. Non-Functional or Not Well Utilized Test Automation
4. Use of Emulators - Setting Up & Maintaining Real Devices
5. Access to Relevant Mobile Platform / OS version / Devices
6. Use of Manual Testing – The Weakest Link in Agile and
Devops Process
7. No Geographically Relevant Testbed – Devices, Networks,
Back-End Integration
http://testdroid.com

8. Where DevOps Make A Difference
• Devices / Hardware
- Handling of new devices and other new HW used in the infrastructure
• Infrastructure
- Ready-to-Go infrastructure means no delays for development, testing and
result-driven approach speeds things up & saves time and money
• Support
- SW devs, SW devs in test (SDET), QA engineers – all need different type of
support when using the infrastructure and DevOps approach can help a lot
• Operations
- Maintaining and Upgrading infrastructure, devices, networks, additional SW
• Tools Integration
- Support for multiple test tools gives the users/developers freedom of choice.
No vendor, tech, tools lock-ins!
http://testdroid.com

9. MOBILE DEVOPS & DEVELOPMENT
- LIFECYCLE & TOOLS
http://testdroid.com

10. DevOps Challenges *
• Building the DevOps culture across organization
• Overcoming the Dev vs. Ops mentality
• Moving from legacy (and sometimes on-premise setup)
to cloud based tools and environment
• Broader scope than before: SW and HW for mobile dev
• Learning new skills – DevOps practice requires change
and collaboration across teams
• Mobile Challenges:
• Fragmentation, Tools and Platforms, Not Enough Real Devices
• Less Worrying Challenges on Mobile:
• Cross-platform, Using Cloud Efficiently
* = Based on Testdroid Customer Survey 2015http://testdroid.com

11. DevOps ‘Toolchain’
• Code – Code Development and Code Reviews, Continuous Integration tools
• Build – Version control tools, SCM, code merging, Build status
• Test – Test and results determine performance
• Package – Artifact repository, Application
pre-deployment staging
• Release – Change management,
Release approvals, release automation
• Configure – Infrastructure
configuration and management,
Infrastructure as Code tools
• Monitor – App perf.
monitoring, UX

12. DevOps ‘Toolchain’
Code & Build
Continuous Integration,
Continuous Development
Package, Release, Configure &
Monitoring
Continuous Delivery, Continuous
Deployment/Release
GOAL: End-to-End Transparency, Traceability,
Seamless Workflow, Continuous Everything!
2 Valid Options for
Mobile Testing:
Cloud or On-
Premise
http://testdroid.com

13. DevOps – Testing in the ‘Cloud’
1
2
3

14. DevOps – Testing On-Premise
CONTINUOUS
INTEGRATION
http://testdroid.com

15. Benefits of Test Automation
• App Compatibility & Global coverage
• Reusability
• Speed & Efficiency
• Test efficiency
• Test effectiveness
• Time-to-market
http://testdroid.com

16. DevOps Must Rely on Automation
• DevOps is more than just a tool or a process change; it
inherently requires an organizational culture shift
• Agile methodologies back up this thinking
• Increase focus on test automation and CI methods
• Best practices and continuous iteration
Time
Resources
Infrastructure
Training
Tools
http://testdroid.com

17. THE DEVOPS APPROACH IN
MOBILE APP TESTING
http://testdroid.com

18. Testing Philosophy
http://testdroid.com

19. Manual Automation
Testing Philosophy
http://testdroid.com

20. Smaller coverage, More
money burnt & time
wasted, Error-prone
Manual Automation
Large
coverage,
quickly
completed,
Less money &
time wasted,
Exact results.
Testing Philosophy

21. Top Requirement for Testing
• Real Devices and Real Browsers, as
emulators cannot help you to test...
• User Experience
• Usability
• Hardware
• Software
• Infrastructure
0 % = the percentage of your app users
that use emulator to run your app!
http://testdroid.com

22. Devices & Daily Tasks for DevOps
• In order to maintain mobile device farm,
environment and all aspects of SW & HW the
following daily check-ups must be done:
• Are all devices connected and operating?
• Are all devices WiFi connected?
• Do all devices have an active USB connection?
• Are battery levels adequate (at least 50%) for every device?
• Are there any pop-ups, system notification or other system
dialogs open on the screen?
• Are screens unlocked?
http://testdroid.com

23. Troubleshooting Ain’t The Same
http://testdroid.com

24. Automation vs Manual
http://testdroid.com

25. Test Breakdown
FAILED FAILED
http://testdroid.com

26. Mobile Test Automation
Automatic test exercisersRecord and PlaybackHand written test scripts
Benefits:
Accurate, specific to
your testing needs,
plenty of options with
frameworks, tools
Fast to create, accurate,
not as sensitive to
human-errors as hand-
written tests, tools
avail’ty
Fastest & extremely
automated, excellent for
smoke testing/quick
testing, availability
Tradeoffs:
Takes a lot of time, ties
resources to write test
cases/scripts, error-
prone (humans)
Compelling Recorder +
Playback tools available
for only few test
automation frameworks
Not accurate as real
test cases
http://testdroid.com

27. Why Should DevOps Be Interested in …
Robotium uiautomator Espresso Appium Calabash
Android Yes Yes Yes Yes Yes
iOS No No No Yes Yes
Mobile web Yes
(Android)
Limited to x.y
clicks
No Yes
(Android &
iOS)
Yes
(Android)
Scripting
Language
Java Java Java Almost any Ruby
Test creation
tools
Testdroid
Recorder
UI Automator
viewer
Hierarchy
Viewer
Appium.app CLI
Supported
API levels
All 16 => 8, 10, 15- All All
Community Contributors Google Google Active Pretty quiet
http://testdroid.com

28. BEST PRACTICES FOR DEVOPS IN
MOBILE APP TESTING
http://testdroid.com

29. Automate (Everything)
• When it comes to mobile app testing, the more
you can automate repetitive things, the better
• Not always possible but leave the last
‘exploratory testing’ for the final phase
• Smoke, Unit, All sorts of Performance,
Regression testing should be automated
• Run your test patterns against EVERY build
• Start small but make sure you can scale-up
http://testdroid.com

30. Ensure Workflow is Seamless
• Only use tools, frameworks and software that is
fully compatible across used toolchain/workflow
• Ensure all processes, results, tools are entirely
transparent and available for use across teams
• This has a significant impact on productivity and
efficiency
• Open source is always highly recommended
http://testdroid.com

31. Rely Only On Real & Quality HW
• Mobile device farms require lots of hardware –
this is the correct place to invest in quality HW
• Make sure the mobile device coverage is
adequate – globally
• Execute the upgrade and support policies
relentlessly
http://testdroid.com

32. “The Right” Testing Framework
• Variety of choices when it comes mobile test
automation frameworks, ask the right questions:
• App, Game, Web? Native, Hybrid?
• Mobile Back-End Integrated?
• Who needs to write, read and interpret tests?
• Does test run results provide you what you
organization needs / is looking for?
http://testdroid.com

34. THANK YOU!
More information about frameworks, devices
and testing at www.testdroid.com
http://testdroid.com