A high-level overview of test automation on the Android platform, beginning with a brief intro to software testing, how mobile isn't all that different from problems faced in the past, a tour of the most popular free tools available, and some words of advice. The slides accompanied a talk presented at the San Francisco Android User Group on October 29, hosted at Twitter.

1. Testing for Android
When, Where, and How to
Successfully Use Test
Automation
Trent Peterson | @tdpeterson | trent@appthwack.com

2. Who?

3. AGENDA
What is testing?
What’s the problem?
Why should I care?
When: Choosing our battles
Where: Best Practices
How: Tools & Frameworks
Q&A

4. INTRO TO TESTING

5. Types of tests
•
•
•
•
•
•
•
•
Functional
Non-functional
Integration
Unit
Regression
Smoke
Usability
Performance
• Stress
• Acceptance
• Load
• Sanity
• End-to-End
• System
• Security
And on and on…

6. Testing distilled for this discussion
Functional Testing
Non-functional Testing

7. THE PROBLEM

8. Building things is hard.

9. WHY SHOULD I CARE?

10. So what’s the big deal?
1,000,000+ apps in
2 crashes and 84% will uninstall
- Compuware as reported in TechCrunch

11. We’ve been here before.

12. CHOOSING OUR BATTLES

13. First, what’s our goal?
Save time!
BE REALISTIC
Save money!
• Increase regularity of testing
Shrink QA!
• Free up resources for other testing efforts
• Enable previously impossible tests
AUTOMATE
EVERYTHING
!

14. Deciding what to automate
How often will I test this?
Is this something a human is good at?
How much effort will automation require?

15. Some examples
AUTOMATE
MANUAL
• Granular functionality
(unit tests)
• Repetitive tasks
(update paths,
navigation, etc.)
• Performance
• UX
• Responsiveness and
“feel”
• New functionality

16. Our options (Machine)
JVM
Emulators
Real Devices
• FAST
• Unit tests
only*
• Mocks
• Cheap
• Catch layout
issues
• Simulate
calls/SMS
• SLOW
• Illusion of 1:1
with real
devices
• Exactly what end
customers use
• Performance data
• Fast
• Expensive:
Purchase,
maintain, etc.

17. Our options (Human)
QA
Crowd
End Users
• Pros
• Real
feedback
• Know the
product
• Expensive
• Real feedback
• Might know
product
• Expensive
(usually)
• Slow
• Don’t do this.

18. BEST PRACTICES

19. No barriers between Dev & QA

20. Automation is a software project

21. Automation is a tool, not a solution

22. Automate with precision

23. It’s all about data

24. Know your matrix

25. Abstraction is important
…but don’t obsess

26. TOOLS and FRAMEWORKS

27. Tool: Recorders
GOOD
BAD
• Fast
• Non-developers can
create tests
• Fragile
• Difficult to maintain
• Non-developers can
create tests
BEST PRACTICE
• Treat as a fancy spy

28. Tool: UI Exerciser Monkey
$ adb shell monkey
–p my.sweet.app –v 1000

29. Framework: Android Testing Framework
SUMMARY
• JUnit/Java
• Instrumentation provides hooks into Android
SDK to control Android-specific functionality
• Very basic functionality supported
• The foundation for most Android testing. Learn
it, use it, love it.

30. Framework: UI Automator
SUMMARY
• JUnit/Java
• Extends testing framework with UI selectors
and manipulators
• Handle/control system dialogs, etc
• 4.1+ only
• Excellent choice if you have the luxury of
supporting 4.1+.

31. Framework: Robolectric
SUMMARY
• JUnit
• Runs in JVM
• Mocks Android functionality with “shadow”
objects
• Super fast
• Not everything is mocked (also, mocks)
• Great for staying sane while developing

32. Framework: Robotium
SUMMARY
• JUnit extension
• Runs on emulators and real devices
• Oriented towards black-box testing and
ensuring real-world outcomes
• Limited to the app under test
• Great choice for developers going beyond
basic unit testing and testing UI.

33. Framework: Cucumber-based
SUMMARY
• Human-readable tests
• Runs on emulators and real devices
• Cross-platform
• Complex scenarios require development
• Behavior-driven design. Be careful!
• Great choice for simple flows and teams with
limited developer resources.

34. Framework: Appium
SUMMARY
• Supports many language adapters
• Runs on emulators and real devices
• Cross-platform
• New, and not all WebDriver concepts map
intuitively to native apps
• Great choice for teams happy with Selenium
and expanding to native.

35. Frameworks: Commercial
GOOD
BAD
• Support
• Added features and
integrations
• Longevity?
• Proprietary
• Often heavy and
difficult to maintain
• $$$
ADVICE
• Evaluate very carefully and acknowledge lock-in.

36. Tool: Spoon
Spoon
Instrumentation Tests

37. When choosing, ask…
What are we testing?
What are our automation goals?
Who will write the automated tests?

38. SHAMELESS PLUG

39. API
• Continuous
Integration
• IDE
• Command Line
Clients
• Browser-based
web app
AppThwack
•
•
•
•
100s of non-rooted devices
Parallel execution
Built-in compatibility tests
Support for all popular
automation frameworks
Results
• Interactive, realtime report
• JSON data via
API

40. Q&A
Trent Peterson | @tdpeterson | trent@appthwack.com