A very big thank you to Michael Palotas from Grid Fusion & eBay International for taking the time and effort to travel across the globe to present at the Australian Test Managers Forum 2014. If you would like any information on TMF please email tmf@kjross.com.au

1. TEST MANAGERS FORUM 2014
SYDNEY, AUSTRALIA
MOBILE TESTING WORKSHOP
1

2. 2
WHO AM I?
Gridfusion Software Solutions
Contact:
Michael Palotas
Gerbiweg 2
8853 Lachen
SWITZERLAND
Tel.: +41 79 6690708
Email: michael.palotas@gridfusion.net
Head of Productivity & Test Engineering, eBay
Founder / Principal Consultant
Gridfusion Software Solutions

3. SETTING THE STAGE
Tell me about yourself J
What are your expectations for today?
3

4. MOBILE TESTING
SPECIFICALITIES
4

5. MOBILE IS A VAST LAND
5

6. YOU ONLY HAVE ONE SHOT
6

7. MOBILE TESTING CHALLENGES
DEVICE DIVERSITY
7

8. MOBILE TESTING CHALLENGES
NEW OS VERSIONS
New versions usually fix issues
but
May also introduce new ones
8

9. MOBILE TESTING CHALLENGES
NEW OS VERSIONS
9

10. MOBILE TESTING CHALLENGES
NEW OS VERSIONS
10

11. NATIVE APPS AND MOBILE WEB
11

12. MOBILE TESTING CHALLENGES
LOCALIZATION
12

13. MOBILE TESTING CHALLENGES
NETWORK
13

14. MOBILE TESTING CHALLENGES
OPERATORS
14

15. MOBILE TESTING CHALLENGES
MOBILE ENVIRONMENT
15

16. 16

17. 17

18. 18

19. 19

20. MOBILE TESTING CHALLENGES
DEVICE MANAGEMENT
20

21. DEVICE OR EMULATOR?
21

22. RISK ASSESSMENT
AND
STRATEGIES
22

23. RISK & STRATEGIES
Hotfixes are difficult and expensive
Agile?
...yes...but...for mobile the goal should be to get it
right the first time
23

24. RISK & STRATEGIES
May shift back to waterfall
(waterfall is not that bad)
Difficult to get (mobile) experienced people
Need to grow capabilities in-house
24

25. RISK & STRATEGIES
No automation in most places today
No continous integration / continous delivery
25

26. RISK & STRATEGIES
DEVICE SELECTION
26
Be pragmatic
Focus on where your money is coming from
Forget the “Long Tail”

27. RISK & STRATEGIES
Multi code base
Many devices
Device / OS management
Simulator vs. real devices
27

28. CLOUD TESTING
28

29. CLOUD VENDORS - DEVICE ANYWHERE
29

30. CLOUD VENDORS - SAUCELABS
30

31. TESTDROID
31

32. GENERAL EXPERIENCES WITH THE CLOUD
No dedicated devices
Or no devices at all
Device management
Your company firewall...
Trust issues
Slow(er)
No in-house knowledge
Not onsite
It is a bit like outsourcing
$ in the beginning
$$$$$$ when you are scaling
32

33. CROWD TESTING
33

34. REALLY?
34

35. CROWD TESTING – WHAT USUALLY WORKS?
Fresh set of eyes
Access to many regions, devices, OS
combinations
(at least theoretically)
Lots of issues reported
Relatively low cost
35

36. CROWD TESTING – CHALLENGES
Significant ramp up required for testers
Low bug quality
You don’t know who is on the other end...
People in mature markets won’t work for 5$ / bug
Only access to production environment
QA system access very difficult
Legal implications, NDAs etc.
No development resources to fix the bugs
36

37. MOBILE
DEVELOPMENT
APPROACHES
37

38. MOBILE DEVELOPMENT APPROACHES
Mobile First
Responsive Design
38

39. TEAM ORGANIZATION
39

40. AGILE AND MOBILE
CI – OK
CD – OK for QA versions
CD – NOK for prod versions
40

41. SKILLSET - THE PERFECT TESTER
41

42. GENERAL TEST AUTOMATION
42

43. FLIP THE AUTOMATION TRIANGLE
Unit
Tests
E2E Automation
Tests
Manual Tests
Unit Tests
E2E
Tests
Manual
Tests
Integration Tests

44. TEST AUTOMATION
WHY?
44

45. TEST AUTOMATION
WHO?
45

46. TEST AUTOMATION
WHAT?
46

47. TEST AUTOMATION
WHEN?
47

48. INSTANT FEEDBACK
D T D T D T D T D T D T D T D T
Development Test

49. AUTOMATION IS GOOD…BUT
Tools help, but tools don’t test
Manual testing is equally important
Manual (exploratory) and automated testing complement each other
49
Automation
Manual

50. WHY IS E2E TESTING SO IMPORTANT?
Because it is the most difficult way to test
Because our users use our products End2End
Because you can verify that the whole system
works together
50

51. PERSONAL THOUGHTS ON AUTOMATION
Manual testing is still very important
E2E tests are expensive, so keep them to the minimum Automation is software development
Be careful with record & playback tools
Automate the repetitive tasks (regression)
Build testability into your product
ROI...
More at http://gridfusion.net/thoughtsonautomation.html
51

52. MOBILE TEST
AUTOMATION
52

53. MOBILE TEST AUTOMATION
Not as mature as web automation
Immature tools market
Tools are usually platform specific
Multi code base
Need to understand the underlying architecture
much more than for the web
Need people who can code well
53

54. EMULATORS VS. DEVICES
What to test where?
54

55. WHAT TO TEST WHERE - EMULATOR
functional test
55

56. WHAT TO TEST WHERE - DEVICE
performance
look and feel
memory usage
carrier specific tests
56

57. IOS SIMULATOR SPECIFICS
57
Comes from Apple
Works well & reliable

58. ANDROID EMULATOR SPECIFIC
58
SLOW

59. ANDROID VS. IOS
Android: lots of variations, versions, vendors
Android: open source (sort of)
iOS: easy to update
iOS: everything comes from one vendor
iOS: not exactly open source J
59

60. SAMPLE MOBILE AUTOMATION
REQUIREMENTS
Use the same approach and infrastructure as web
Business is borderless à test automation needs to be as well
Aim for REAL E2E test automation
Use the Load Balancer / Resource Management Features of Selenium Grid
Manage multiple applications / versions / languages
No modification of mobile app
No modification of mobile device
Mobile App Runtime Inspection Support
Hybrid app support
Support for emulator + real device
Integration into CI system
60

61. SAME AUTOMATION CODE BETWEEN
PLATFORMS?
Sounds good first
but
Most apps are different between platforms
Different element locator strategy
Do reuse the helper functions
61

62. MOBILE AUTOMATION
IOS / ANDROID DIFFERENCES
iOS has no IDs à harder to automate
iOS relies on xcode
Android is open source
Android: ID support
62

63. TEST INFRASTRUCTURE
63
Driver Browsers
Mobiles
AUT
DB
API

64. TOOLSETS - OVERVIEW
Ranorex
QTP
TOSCA
SeeTest
Selenium/ ios-driver/Selendroid
Selenium / APPIUM
64

65. SeeTest
65

66. WHAT ABOUT BDD / ATDD?
I have seen it working
Required to close the loop
Removed complexity
(while adding it at the same time)
Have you ever seen a product owner test?
66

67. SAMPLE TOOLS
JBehave
Fitnesse
Cucumber
67

68. SELENIUM
68

69. SELENIUM – WHAT IS IT?
Web testing tool
Highly successful open source project Stable
Protocol describing the user interactions
Supports most browser implementations Support
for most common programming languages
69

70. SELENIUM – WHAT IT IS NOT
a drag & drop tool
a network testing / monitoring tool
a performance testing tool
a reporting tool
a testcase management tool
70

71. SELENIUM
i.e. Firefox driver
Server
JSON WIRE
PROTOCOL
Client
Java
C#
Ruby
Python
Server
i.e. Selendroid, iOS-Driver

72. SCALING - SELENIUM GRID
72

73. SELENIUM GRID
73

74. SCALING USING SELENIUM GRID
74
Client
serverGRID HubClient
CI
server
server
server
server
server
server

75. 75
Implements the JSON Wire Protocol (so it is fully compatible with Selenium)
Runs on emulators and devices
Can be run as a node in Selenium Grid
Support for localized native apps (1 test for all locales)
No need to change the AUT
No need to jailbreak the device

76. 76
Hybrid support
Different locator types are supported to find elements
Gestures are supported: Advanced User Interactions API
Hot plugging
It can be integrated as a node into the Selenium Grid for scaling and parallel testing
Android target API support from 2.3.3 to 4.3 (latest)
Built in Inspector to simplify test case development.

77. REPORTING
Invest into proper reporting
Easy analysis of the reports is crucial
Selenium does no reporting
TestNG or JUnit are responsible for reporting
100% customizable reports (i.e. different reporting levels,
PDF, HTML, Email, SMS etc...)
Hooks into test case management tools
77

78. REPORT WITH SCREENSHOT
78

79. SELENIUM
SELENDROID
IOS-DRIVER
79