Trends, Challenges & Solutions for Mobile Application Testing : by Siddharth Shinde, Sr QA Specialist - Ishi Systems

1. Crowd testing framework:
Trends, Challenges and Solutions
for Mobile application Testing
0
Siddharth Shinde
Sr.QA Specialist

2. Agenda
 Software testing evolution
 Mobile Testing Challenge
 Why mobile testing is difficult?
 Crowd Testing as a solution
 Principles & Features of crowd testing
 Success factor
 Summary

3. Evolving IT landscape presents more
challenges than ever!
More
customer
driven
IT
- Testing Goals
- Speed
- Cost
-Quality
More cloud
based
Shorter
production
cycle
More HW
and SW
choices
More open
source
More
users,
more
places

4. What it does?
For achieving high Quality, Speed and Costs
have gets compromised.

5. Software Testing Evolution
SOFTWARE TESTING
No dedicated
Testing team Dedicated
testing team (in
house)
Outsourced
testing
SOFTWARE DEVELOPMENT
(In house) resources
Resources in the
cloud

7. Mobile Testing : Challenge
 Only 31% of
firms surveyed
currently test
mobile
applications.
Why?
 Many report they
do not have
appropriate tools
or expertise and
limited access to
the devices to
test effectively
mobile
applications

9. Mobile Trends
 Transactions in both B2C and B2B
 Products and services
 Account management
 Internal processes (B2E)
 Business intelligence
 Process management
 Social collaboration

10. Business as usual in mobile testing
Functionality
Performance/Load
Security
System

11. ... Mobile Testing additional hot spots
User experience
Functionality
Screen Size
Platform
Performance/Load
Security
System

12. Device breakup creates a need to perform
device level testing & UAT for mobile apps

13. Why Mobile Testing is difficult? (1/2)
Touching the
device
Touch, swipe, pinch, rotate, move
Diversity in
platforms, OSs &
devices
Android is the most fragmented, with 5 major versions corresponding to 9 API sets. In typical markets for
feature phones, there can be more than 100 different device types in the
installed base
Automation
challenges
Sophisticated user experiences involve touch, gestures, GPS location, audio, sensors
(accelerometers) and physical actions (touching the handset to NFC readers). Such interactions
can't be fully scripted or simulated, and may involve manual testing on real devices
Network
performance
Cellular networks are nondeterministic, and exhibit wide variations in bandwidth and
latency.
Users in different locations accessing different operators can experience vastly different
application performance
Application
complexity and
sophistication
Mobile devices and applications are becoming more sophisticated, using techniques such as
context, 3D graphics and gaming. Greater sophistication implies more complex testing
New OS versions
often break
applications
Developers have no control over when new OS versions will appear, and when or whether users
will upgrade. Thus, it's common for new OS releases to break existing native applications

14. Why Mobile Testing is difficult? (2/2)
New
Technology
risk
Mobile is at the leading edge of new technologies, such as HTML5, that are not yet well-understood
in terms of testing
Performance
variations
There is a performance difference of greater than one order of magnitude across devices
in the smartphone installed base. An app or HTML5 website that runs well on a top-end
device may not be acceptable on a low-performance handset
Operator
intervention
Operators may modify mobile Web content to optimize network performance so that desk-based
testing may not show what a real world handset user experiences
Contextual
issues
Mobile applications and websites are used in a wide range of contexts, which raises many
new testing challenges. Applications dealing with critical or regulated data may demand
much more rigorous testing
Peripherals Smartphones are acquiring a growing range of add-on devices, such as Bluetooth
peripherals, that are generally not accessible to testing tools
Testing user
opinions
Many applications will be distributed via public app stores, where user reviews and
attitudes play a large part in determining whether an app is downloaded.
Understanding user reactions to an app and its store collateral before it's published may
be valuable

15. Even <<for free>> app
Must be <<right first time>>
Must get into the market<<first>>

16. Outsourcing
QA
Needs
Mobile
and
Web
CROWD TESTING

17. CROWD – Dictionary meaning
 “The mass or multitude of ordinary people”

18.  Crowdsourcing is the process of getting work
or funding, usually online, from a crowd of
people.
Outsourcing
Crowd
Sourcing
Crowd
Testing
Crowd

19. What is crowd testing?
 “If you are testing software that all kinds of
strangers are going to use, then why not use a
bunch of strangers to test it.” (Paul Herzlich)
 “ Crowtesting puts software to test under real-world
conditions by real users, allowing
companies to gather real insight and feedback
and to identify defects quickly and cheaply”
[CSTE2013]

20. Principles
Diversity/coverage/realistic
Independent and test as a user
Payment system
Immediate results

21. Crowd Testing Features

22. Players and Process

23.  Crowdtesting ensures that a software produces expected output across all
desired platforms & user groups

24. Who are the crowd testers?
 The testing crowd can be a pool of:
 Inter-company testers
 Intra/inter-project testers in a company
 Strategic partners
 Domain knowledge experts
 Technology experts
 Tool experts
 Network of end users
 Social network users
Certified
Specialist
Recreational

25. Crowd Testers expertise and
infrastructure = Crowd Testing types
Crowd Testing
Exploratory
crowd testing
Enterprise
crowd testing

26. Exploratory Crowd Testing
 Effective to cover all the many configurations
 Good for non critical systems
 Simulating real load and testing
- Outcome not predictable
- Discovering basic failures
 To help making it more effective:
 Show them the big picture of what they are doing
 Formal process
 A clear report

27. Enterprise Crowd Testing
Large enterprises utilizing their in-house excess capacity for on-demand
• Secured private cloud infrastructure
• Testing Managers
Organized
• Crowd testers screening
• Project management, tracking, effort
planning
Professionalized
Standardized • Standardized process and reporting
Enterprise
Cloud
Infrastructure
crowd sourced testing

28. Enterprise («Managed») Crowd Testing
 «Enterprise crowd testing, although still in its nascent and emerging stage,
promises to be the next wave in the social, mobility and localization era.»
[28+_TestExp14]
Test
Scheduling
Crowd Testing
Platform
Testers
Profile
Knowledge
center
Comm. And
collaboration
Reporting
Test
management

29. Principles of crowd-testing
Preparation Initiation Execution Evaluation Completion
Identify customers
needs
Fix price
Define report
template
Prepare crowd
testers
Prepare test
environment
Initiate testing
Supervise and
monitor
Validate results
Prepare client
deliverables
Testers Payment
Present results to
client
Crowd testers ratings

30. Crowd Testing … Glossary
Crowd Crowd Sourcing
Managed Crowd
Testing
Outsourcing
Crowd
Testing
Beta
Testing
Exploratory
Crowd Testing
Enterprise
Cloud Testing

31. Opportunities for SE research in crowd
testing: COMMUNITIES
 Crowdsourcing in SE
 Global Software Engineering
 Social Software Engineering
 Empirical studies
 Software Testing

32. Effectiveness of different testing methods
Crowd
Testing
Cloud
Testing
Lab
Testing
«When» shall we test with the crowd, when not to?
Which software
domain
Which type of tests

33. Complementing different testing methods
Crowd
Testing
Cloud
Testing
Lab
Testing
How crowd and Lab testing can be used together?
Need: clients’ want to get the best
from the combination of both
Efficiency

34. Complementing different testing methods..cont.
Systematic
Mobile
Testing
Crowd
Testing
Solution to
?
Dimensions to account for
• Multi OS (Apple, Android, BBMS, MS…)
• Multi Technology (Native, Hybrid,
Browser…)
• Multi manufacturer (Samsung,
Micromax, Acer…)
• Network Quality (Intermittent,
disconnected, bandwidth..)
• Device resource (Memory , CPU,
Battery..)
• Device capability (GPS, Camera…)
• System of record (Don’t forget the
backend)
• Does the user like it? (UX)

35. Complementing different testing methods..cont.
Test
Selection
Systematic
Structural
Testing
Crowd
Testing
Test
Execution
Solution to
?
Coverage issues

36. Complementing different testing methods..cont.
Model Based
Testing
Mobile Testing
Model Based
Mobile
Crowd Testing

37. Social Software Engineering
 How to keep Crowd Testers motivated
 How to keep them involved
 Appropriate reward mechanisms

38. Types of Testing that can be
Crowdsourced
 ALL
 Functional
 Load/Performance
 Usability
 Security
 Localization
 Testing Modes
 Exploratory
 Test Case Based
 Automated

39. Forms of crowd
Fully Private
Crowd
• Utilizing internal
infrastructure
• Leverages
organizational bench.
Public
Crowd
• Utilizing external
freelance capacity
• Community and
partnering
Hybrid
Crowd
• Optimizing internal
and freelance
capacity

40. Application in Industry….Summary

41. Challenges in Crowd Testing
Lack of
standards
Coverage
Measurement No control
Critical feature
coverage
Lack of
ownership
Reward
mechanism
Continuous
improvement
Communication
and
management

42. Concerns & Mitigations …..
 Concerns : Security and Confidentiality
 Mitigation:
 Background check
 NDA

43. Benefits
Diversity

44. Few questions you should ask from
your provider
 Can I specify the criteria for selecting testers?
 How do I know which functionality they covered?
 Will I know which configurations were covered?
 Do the testers know what they need to know?
 Who is accountable for quality and schedule?
 Delivery Assurance?

45. Success Factor
Managed
and
controlled
delivery
Process
Governance
Structurally
developed
community
Efficient
technology
backbone

46. Organization can opt…
Build their own
crowd testing solution
Outsourcing to crowd testing
Organizations/vendors

47. Summary
The reason behind the success of
this framework is that this method
is essentially user-centric.
This means it the user who
determines the adoption and
success quotient of particular
software.

48. In Conclusion
Crowd testing is growing
exponentially
Crowd testing may bring
big benefits to QA
Still, there are number of
challenges to be managed

49. Contact
If you are interested to this topic, please
Stop by after the presentation
Contact me at
siddharth.shinde@ishisystems.com

50. References / Acknowledgements
 [TestExp14] «Testing Experience, 25/2014» issue on Crowd testing
 [RV_TestExp14] Rama Murari & Venkat Ramesh Atigadda, “Panorama of Crowd Testing”, in
Testing Experience, 25/2014
 [Stein_TestExp14] Markus Steinhauser “Crowd Testing – An Addition to Traditional Software
Testing”, in Testing Experience, 25/2014
 https://www.passbrains.com
 [34+_TestExp14] Dieter Speidel, Mayank Mittal & Mithun Sridharan “Enterprise Crowd Testing”, in
Testing Experience, 25/2014, page 34+
 [28+_TestExp14] Saisubramanian Sivasailem & Dilipkarthik Jayamohan, “The Next Wave in the
Social, Mobile and Localization Era
 Enterprise Crowd Testing”, in Testing Experience, 25/2014, page 28+
 https://99tests.com/
 Henry Muccini, DISIM, University of L’Aquila
 www.bugfinders.com