1. |
INTRO TO CROWDTESTING
C O M B I N I N G ‘ I N - T H E - L A B ’ & ‘ I N - T H E - W I L D ’ A P P R O A C H E S
Peter Shih | VP of Channel Development | pshih@applause.com

2. |
• Formerly known as uTest, the leader of in-the-wild software testing
– Founded by pair of Israeli entrepreneurs
– Founded in 2007; GA Launch in 2008
• Offices around the globe
– Boston, San Francisco, NYC, Seattle, Tel Aviv, Berlin & Warsaw
• World’s largest & most mature crowdtesting company
– Raised 5 rounds of VC funding, totaling $80 million
- Latest round led by Goldman Sachs in 2014
– Larger than all other crowdtesting companies combined
2
A L I T T L E A B O U T T H E C O M P A N Y

3. |
3,000+
CUSTOMERS
3
B Y T H E N U M B E R S
70,000+
B U I L D S T E S T E D
175,000+
P R O F E S S I O N A L
T E S T E R S
500 Million+
A P P S T O R E
R E V I E W S A N A LY Z E D
OVER 1 MILLION
I N - T H E - W I L D I S S U E S R E P O RT E D

4. | 4
C O M P A N I E S T H A T U S E C R O W D T E S T I N G
Retail Travel SocialMedia

5. | 5
C O M P A N I E S T H A T U S E C R O W D T E S T I N G
Health/FitnessFinancialSoftware Telco

6. |
The Challenge
PROBLEM WITH
THE STATUS QUO
•••

7. |
Test Labs Are Here Users Are Here
Modern apps & sites require real-world testing that mirrors the many
different combinations of devices, use cases and locations of users
7
T H E P R O B L E M W I T H T H E S T A T U S Q U O

8. |
• ‘Inside-out’ approach is still valuable, but it’s merely the starting point
• 360
ₒ
app quality requires an ‘outside-in’ view, which comes in many forms
– Functional defect reports
– Crash reports
– Security vulnerabilities
– Load and performance issues
– Localization issues
– Usability feedback
– User sentiment analytics
• And across all phases of the SDLC
– Research
– Pre-launch
– Post-launch
8
D E T A I L S O F ‘ O U T S I D E - I N ’ A P P R O A C H

9. |
360
ₒ
APP QUALITY
QUALITY THAT MATCHES YOUR USERS’ EXPECTATIONS
9
A P P L A U S E E N A B L E S …

10. |
DESIGN &
DEVELOP
1
TEST &
RELEASE
2
LISTEN &
LEARN
3
ANALYZE
& PLAN
4
360º
APPQUALITY
5
EG: IBM
EG: JIRA EG: CRITTERCISM
EG: FLURRY
10
“Should I launch?”
“What should I do next?”
S O L V I N G T W O M A S S I V E C I O P R O B L E M S

11. |
• Modern apps have different recipe for success
– Infinitely more variables make up the test coverage matrix
- Devices, OSes, browsers, carriers, locations, use cases
– Traditional test playbooks & methodologies are merely starting point
• Crowdtesting augments traditional QA efforts
– True for web, but even more so for mobile and smart TVs/cars/watches
– Move beyond the QA lab, closer to where users work, live & play
– Close the loop between user POV and internal definition of quality
• 360°app quality isn’t just about testing or dev ops
– Requires modern tools and analytics that focus on users-centric quality
– Necessity due to outsized user expectations + voice in app stores
3 K E Y T A K E A W A Y S
11

12. |
The Challenge
A 360
ₒ
APPROACH
TO APP QUALITY
•••

13. |
I N - T H E - W I L D T E S T I N G S E R V I C E S
FU N C TION A L
TESTIN G
U SA B ILITY
TESTING
LOC A LIZATION
TESTIN G
LOA D
TESTIN G
SEC U R ITY
TESTIN G
Do your apps function flawlessly
in the hands of your users?
Are your apps as intuitive and
easy-to-use as you think?
Are your apps primed to thrive in
different markets & cultures?
Can your apps handle the peaks
& pressures of the real world?
Are your apps protecting your
users data & privacy?

14. |
• Real-world functional testing solutions
– Ensuring apps work well in the hands of users
• Compliments in-house testing
– Team of testers provide you with scale, coverage & expertise
• Testing on real devices, in real locations
– Across OS versions, mobile browsers, wireless carriers
– Exploratory testing & test case execution
– Test around the clock or after major releases
• Integrates with your bug tracking systems
– JIRA, Bugzilla, Mantis, Pivotal, IBM Rational, Microsoft TFS and more
• Fully-managed service to reduce your overhead
– Integrated with your test services from QualiTest
– Handle all tester communications and bug triaging
– Professional testers from any region, on any device or OS
– Integrated with your internal team, processes and systems
F U N C T I O N A L T E S T I N G A P P R O A C H
14

15. | 15
I N - T H E - W I L D F U N C T I O N A L D E F E C T
Learn About The Tester
Tiered
Approval
Chat Directly
With The Tester
Reproducible
Action Steps
U I N OT R EN D ER IN G PR OPER LY

16. |
• Real-world usability testing solutions
– Ensuring your apps are designed intuitively for intended audience
• Expert-led usability studies
– Remote unmoderated testing performed by industry experts with an
average of 12 years of expert usability experience
• Audience matching
– Survey 25+ respondents that match the demographics of your users
• Video and audio experience captured
– Watch and analyze your respondents use your app for the first time
• Comprehensive report & consulting
– Receive a complete report with findings, benchmarks and
recommendations on how to improve user experience
– Leverage your usability expert for advice on how to minimize
potential confusion & frustration for first-time users
U S A B I L I T Y T E S T I N G A P P R O A C H
16

17. |
• Real-world localization testing solutions
– Validate content and app localizations to ensure correct context
across languages and cultures
• Go beyond content
– In addition to translations, verify formatting, currency conversions, taxes,
shipping options and more
• In-market localization testers
– Understand the true implications of your localization choices by leveraging
cultural natives
• Double verification
– All issues and recommendations are verified by an additional cultural native
• Available string reporting
– Output a string document to facilitate easier updating of your apps
17
L O C A L I Z A T I O N T E S T I N G A P P R O A C H

18. |
• Real-world security testing solutions
– Protect your users privacy and data
• Expert-led and managed security testing
– A white hat security expert will work with you to develop a
comprehensive test plan
• White hat security testers
– Trusted team of security testers will attack your site through
a combination of manual penetration and automated tools,
looking for common vulnerabilities and creative new attacks
• Comprehensive report
– Receive a complete report with findings and
recommendations on how to improve your site security
• Remediation consulting
– Leverage your security expert for advice on how to fix
specific vulnerabilities and mitigate different types of risks
S E C U R I T Y T E S T I N G A P P R O A C H
18

19. |
• Real-world load testing solutions
– Understand how your app performs under real world pressures
• Expert-led and managed load testing
– A Performance Engineer will assess current infrastructure,
develop JMeter scripts, execute and analyze tests
• Scalable and flexible load scenarios
– A range of options to test your apps and sites and APIs
- Large number of concurrent users
- Varying amount of time on the app or site
- With various bandwidth constraints to mirror real-world conditions
• Comprehensive report
– Receive a complete report with findings and recommendations on
how to improve your apps’ performance
19
L O A D T E S T I N G A P P R O A C H

20. | 20
A P P Q U A L I T Y T O O L S
APPLAUSE
SD K
Discover crashes in real time
and distribute builds
A PPLA U SE
A N A LYTIC S
Listen to what your users are
saying in the app stores
3 6 0 ° A PP QU A LITY
D A SH B OA R D
Access all of your app quality
results in a single place
MOB ILE B ETA
MA N A GEMEN T
Run & manage beta programs
at a global scale

21. |
3 6 0 ° A P P Q U A L I T Y D A S H B O A R D

22. |
M O B I L E S D K ( I N C L U D E D I N F U N C T I O N A L )

23. | 23
U S E R S E N T I M E N T A N A L Y T I C S

24. | 24
U S E R S E N T I M E N T A N A L Y T I C S

25. | 25
C O M P E T I T I V E A N A L Y S I S

26. | 26
U S E R R E V I E W S E N T I M E N T

27. |
The Challenge
MATURING THE
CROWDTESTING MODEL
•••

28. |
• Anyone can join, but many do not qualify for paid projects
– ‘Test the Tester’ – initial round of validations
- Hard skill vetting: technical testing skills; device knowledge; OS expertise
- Soft skill betting: following policies and processes, communication skills
– Ongoing validations from every project:
- Tester Ratings, based on the quality of reports and impact of defects found
› Value determines how much tester gets paid for their work
› Quality affects tester’s performance ratings
› Judged by customers, project managers, and team leads
- Training and Mentoring programs to sharpen tester skills
› Crowd  Teams  Community of Experts
• Protecting customer IP
– NDA in our standard Terms of Use for every tester
– Custom NDA supported for each customer
– Proof: 70,000+ test cycles with zero breaches of IP
Q U A L I T Y Q U A L I T Y Q U A L I T Y
28

29. |
• Project Managers are highly aligned with internal Dev & QA teams
– Support your expectations, timelines, and priorities
– Team Leads in various time zones to provide around-the-clock coverage
– Recommend new ideas based on experience with many international companies
• Integration with your systems and test methodologies
– Bug Tracking Systems: JIRA, HP Quality Center, IBM Rational, Microsoft TFS,
Github, Bugzilla, Pivotal Tracker, Asana, Mantis, FogBugz, Rally, and more
– Methodologies such as continuous integration, test-driven development, agile,
waterfall, lean software development, behavior driven development, etc.
• Continuous improvement and optimization
– Assess results from every project and deliver monthly management reports
– Identify favorite testers and invite new members to optimize team performance
F U L L Y - M A N A G E D S E R V I C E
29

30. |
The Challenge
FINAL POINTS
•••

31. |
• App quality was traditionally conveyed by companies to end users
– Testing in the lab was sufficient due to smaller coverage/environments
• App quality is increasingly being overruled by end users
– Through app store ratings & reviews, social media, blogs, and more
– Across multiple dimensions: usability, security, interoperability, etc.
– And most importantly, the cost and publicity of failure is higher than ever
• Applause delivers tools & services that address both views
– ‘Inside-out’ point of view
- Collaborating with internal development & QA on strategy and methodology
– ‘Outside-in’ point of view
- Complementary ‘in-the-wild’ testing on real devices across location & OS
- Reveal important feedback from end users before launch
31
K E Y T A K E A W A Y S

32. |
The Challenge
Q&A
•••