Matt Johnston from uTest presented on getting testing help from the crowd. He discussed how app quality is now judged more harshly by users and the need for outside-in quality perspectives. USA Today uses a blended approach of in-house, outsourced, and crowdsourced testing with uTest to test across diverse environments and meet tight deadlines while maintaining a high quality bar. Partnering with uTest helped USA Today reduce bugs, improve usability, and consistently achieve high app store ratings.

1. T18
Concurrent Class
10/3/2013 1:30:00 PM
"Get Testing Help from the
Crowd"
Presented by:
Matt Johnston
uTest Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Matt Johnston
uTest
With more than a decade of marketing experience at companies ranging from early-stage startups to publicly traded enterprises, Matt Johnston leads uTest's marketing and community
management efforts. Since joining uTest as employee #8 in late 2008, Matt has provided a
technical sensibility to uTest’s engagement of testers and engineers. As CMO, he leads uTest’s
strategic initiatives, such as partnership formation and helping to define uTest’s expansion
strategy.

3. 9/19/2013
Getting Testing Help From The Crowd
Matt Johnston | CMO/CSO, uTest | matt@utest.com | @matjohnston
October 2013
|
The Challenge
App Quality
Then vs. Now
|
1

4. 9/19/2013
The Way Things Were
You Tell Them What To Think
|
The Way Things Are TODAY
They Tell You What They Think
|
2

5. 9/19/2013
Mobile App Quality Is Judged More
Harshly
• Users less tolerant of
spotty quality
• Social & app stores
give every user a
megaphone
• Switching costs
lower than ever
• Cost of imperfect app
quality has spiked
|
So What Could Possibly Go Wrong?
|
3

6. 9/19/2013
Oh, And It’s Not Just Mobile
Fragmentation of users, devices, OSes, browsers, locations
Cars
Household Appliances
E-Readers
Connected TVs
Laptops & Desktops
Gaming Consoles
Phones & Tablets
|
No Turning Back Now
• Inside-out quality perspective is valuable but
incomplete
– Regardless of mobile app type or dev & test methodologies
• Replaced by outside-in quality worldview
– Quality signals come in many forms, and from USERS
- Bug reports
- Test cases executed
- Crashes
- User feedback
- Analytics
– And at many times (continuous)
|
4

7. 9/19/2013
What’s That Mean?
• External Signals Inform Internal Decisions:
– What you think matters less than what users think
– How YOU measure app quality matters less
– What users feel and say matters MORE
– Court of public opinion rules
- App stores
- Social media
|
Tools For Outside-In App Quality
In-House
InApps
Test
That
Leaders
Win
|
5

8. 9/19/2013
The Challenge
Ummmm
So What Do We Do?
| 11
Stuff Gets Built; Then It Breaks
1. Marketing identifies a need
2. Product specifies a solution
3. Engineering builds the product
4. QA tests the application
5. Help desk listens to complaints
6. Sales & execs blame everyone
7. Then we do it all over again and
hope for better results
Vicious Cycle
| 12
6

9. 9/19/2013
Lab-Based Testing Innovation
Manual vs.
Automation
Managing vs.
Leading Teams
In-House vs.
Outsourced
Agile vs.
Waterfall
Exploratory vs.
Test Cases
Testers vs.
Engineers
Offshore vs.
Nearshore
| 13
Mob Mentality
BUT crowds often look (and act) like unruly mobs
| 14
7

10. 9/19/2013
From Mob to Community
And mobs don’t work in every category
Example: a skilled service like software testing or
requires an orderly and professional “community”
capable of consistently delivering the desired results
| 15
The challenge is to turn this –
| 16
8

11. 9/19/2013
Into this
| 17
The Challenge
The Crowd:
Testing In-the-Wild
| 18
9

12. 9/19/2013
Community Management 101
• What Good CM Isn’t
– It’s not customer service
– It’s not marketing or social media
• What Good CM Is
–
–
–
–
1st gen
reactive & narrowcast (customer service)
nd gen
2
proactive & broadcast (marketing)
rd gen
3
independent & self-serve (program management)
4th gen
interdependent & vetting, teaching and policing one
another (community management)
| 19
Community Management Tool Kit
•
CM goals – to build relationships, develop loyalty and nurture
desired behavior
– Get beyond 1:1 discussions
– Get beyond reactive corrections
•
CM Challenges – to do the following at scale
•
Tool Kit
– Recruiting
–
–
–
–
–
–
–
–
Vetting
Engaging
Incenting
Managing
Forums & message boards > email responses
Help topics & tutorials > repeatedly answering same questions
Webinars > phone calls
Profiles
Matching
Ratings
Reputations
Payments
| 20
10

13. 9/19/2013
Rating & Reputation System
• Top c-sourcing firms use two forms of compensation
– Monetary
– Reputation
• Performance-based ratings based upon a dozen factors:
– Participation Level:
- Lifetime & recent participation: # active test cycles, # reported issues
– Quality of Participation
- Approval percentage for bugs, test cases and usability surveys
- Accuracy of bug type & severity classifications
- Ability to write test cases & create automated test scripts
| 21
The Challenge
uTest Case Study:
USA Today
|
11

14. 9/19/2013
At the Forefront of Innovation
•
Timeline of USA Today Pioneering
– 1982: 1st nat’l paper to use full color
– 1996: Early adopter of web
– 2008: 1st major news brand with native apps
- iOS, Android, Windows, Kindle, web TVs, gaming consoles
– 2012: Re-designed & re-launched entirely new UI/UX across all
- Print + website + 14 native apps
- First web exceeded print; now apps exceed web
|
Impact on Testing
•
•
•
Great for USA Today’s readers
Great for USA Today’s sales & marketing
Vast pressure on tech execs to deliver quality across:
–
–
–
–
–
Device makers & models
OSes & browsers
Carriers
Languages (localized content)
Locations (geo-based intelligence)
|
12

15. 9/19/2013
USA Today’s Testing Portfolio
In-House
Manual
Testing
Outsourced
Manual
Testing
In-The-Lab
Testing
In-House
Outsourced
Test
Automation
Test
Automation
|
The Challenge
The Goal:
High Bar For App Quality
|
13

16. 9/19/2013
The Goals
•
Maintain Quality
– Launch nothing less than a 4.5 star app
– Each new version, across 14 native apps
•
Increase Test Coverage
– Real-world conditions
– Cost-effective
– Scalable
•
Meet Tight Deadlines
– Must have apps ready to win
- For each new platform that gains traction
- For each new device that gains traction
|
The Challenge
The Approach:
Achieve In-the-Wild Testing
|
14

17. 9/19/2013
Two Distinct Choices
• Traditional Outsourcing
– Lives in a lab
- Further removed from users
- Geo + demo
• Crowdsourcing & expertsourcing
– Lives beyond the lab, in the wild
- Professional testers + real-world conditions
– Mirrors USA Today’s user base
– Cheaper vs. in-house
– Hidden costs
- Time zones & language
- Scaling concerns and costs
- Real-world conditions
- Tech + geo + demo
– Blended team
- Recurring testers (incl on-site) + crowd testers
- Dedicated uTest project manager
– High noise-to-signal ratio
- No transparency
– Scales to match shifting needs
|
Their Choice: Partner with uTest
• In-the-wild testing solution for USA Today
– Provide on-demand complement to in-house QA team
– Reduce escape rate by testing beyond the lab
- Across OS, browsers, devices & carriers
- Across locations, languages, industries & hobbies
– Blended test team built into USA Today’s dev & test lifecycle
- Dedicated uTest project manager
– Tester management
– Triaging services
- Team of dedicated testers (remote or on-site) + large team of crowd testers
|
15

18. 9/19/2013
The Challenge
Results:
Apps That Win
|
Results
•
Reduced escape rate
•
Improved usability
•
Crowd details
– ID’d and documented 1,000+ bugs across 6 apps in first 90 days
– Several iterations of usability testing & analysis
– Testers from US, Canada, Brazil, UK, France, Germany, China & Japan
– Some projects (eg: brand new iPad app) involved 80+ testers
– Others with tighter scope required smaller teams of 10-12 testers
•
Hit the ground running
– Launching Kindle Fire app same day the device launched
– Hand-picked testers pre-purchased various models of device
– Tested their app with all over the world, almost instantly
•
Bottom line: better apps
– Enabled USA Today to keep innovating, while maintaining superior quality
– USA Today able to consistently meet 4.5 star app-store rating goal
– 2012 Mobile Excellence Award – Best User Experience
|
16

19. 9/19/2013
Customer Satisfaction
The Final Word
“uTest gives our organization a much better sense of where we
stand in terms of application stability and performance. For a group
that iterates as often as we do, that type of insight is critical”
-- Tim Carlson,
Director of Mobile Product Development, USA Today
|
The Challenge
Crowdsourced Testing
Specific Takeaways
|
17

20. 9/19/2013
Key Takeaways: Crowd Testing
•
Apps universe has forever changed testing
– Exponentially more diverse user environments
– Devices & configurations
– Locations
– User demographics
•
In-the-wild testing is a necessity
– Some testing must be done beyond the lab
– In-the-wild where app users live, work & play
•
Agility is crucial to app success
– Test team size and tester location
•
enterprise-friendly mobile testing
Partner selection is vital
– Not all created equal
– Alignment of capabilities, needs and philosophy
– Track record of relevant enterprise success
|
The Challenge
Crowdsourcing
OverarchingTakeaways
|
18

21. 9/19/2013
Understand The Job To Be Done
|
The Challenge
Questions?
Answers
Matt Johnston | CMO @ uTest
mattj@utest.com | @matjohnston
| 38
19