1. The SoLoMo Revolution
And The Resulting Tectonic Shift In Testing
|
Matt Johnston | @matjohnston | CMO @ uTest | March 2012

2. The Challenge
SoLoMo
Huh?
|
2

3. SoLoMo Breakdown
So Lo Mo
Social Local Mobile
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 3

4. SoLoMo Coming Fast
In the past six months…
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 4

5. QA Leaders Better Sit Down
• Media & analysts aren’t the only ones talking about it
– Marketing
– Product
– Engineering
– IT
– Sales
– C-suite
• Want a seat at the table?
– Get informed
– Get involved
– Develop an opinion
– Advocate strongly
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 5

6. The Challenge
Trends In
Social Apps
|
6

7. Not Just For B2C Anymore
• The social revolution isn’t coming… it’s here
– Nothing is disconnected
– No industry is immune
• The stats on social media adoption in Fortune 500 firms:
– 77% integrate Twitter
– 84% incorporate LinkedIn
– 93% utilize Facebook
– 57% publish on YouTube
• Gartner: By 2016, social
integrated w/ nearly all apps
– Websites & apps
– Mobile apps & mobile web
– Desktop clients
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 7

8. Goes Soooo Much Further
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 8

9. The Challenge
Trends In
Local Apps
|
9

10. Not Just For Marketing
• Businesses use location for more than just marketing
• Growing trend toward location-based intelligence in apps
– Asset tracking
– Maps & directions
– Location finders
– Check-in services
– Geo-based personalization
– Location-based ads
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 10

11. Increasing Granularity
Good news for users (and marketers)… good luck to testers.
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 11

12. The Challenge
Trends In
Mobile Apps
|
12

13. Mobile Apps Boom Market
• ABI: mobile revenues will hit $133 billion in 2014
• B2B mobility no longer just for email
– Business apps: CRM, ERP, HR systems
– Productivity apps: docs, spreadsheets, presentations
– Collaboration apps: email, IM, publishing
– Medical apps: health records, patient education
• B2C mobility growth even steeper
– Retail: location-intelligent m-commerce
– Media: magazines & newspapers going purely digital
– Travel: mobile bookings, check-ins, maps, deals
– Education: tablets in every classroom
– Healthcare: patient records, physician notes
– Social: no explanation required
– Gaming: ditto
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 13

14. Mobile’s Just Getting Started
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 14

15. Did I Mention…
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 15

16. Psst… It’s Not Just Mobile
• Smartphones & tablets make QA immeasurably tougher
• Now imagine a whole world of apps:
E-Readers
Cars
Household Appliances
Smart Homes
Gaming Consoles
Connected TVs |
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 16

17. The Challenge
SoLoMo Impact
On Testing
|
17

18. Brave New World
• Impact on design, dev & IT well-recognized & resourced
• Sadly, but not at all surprisingly, not true for testing
– Want proof? Specialized designers & devs… but who tests?
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 18

19. Social: Impact On Testing
• Social presents functional test challenges
– Testing code & integrations that aren’t yours…
– That constantly changes…
– And you get the blame
• As well as security & privacy pitfalls
– Internally, a channel for spam, malware and data breaches
– Confidential data may be posted and trigger regulatory penalties
– Externally, increasingly used for user authentication on your products
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 19

20. Social: Impact On Testing
• New challenges include how will you:
– Regression test multiple social integrations for web & mobile apps?
– Write (and execute) test cases with zero visibility into social’s roadmaps?
– Automate anything?
– Test sharing/liking in a QA environment without publishing via social?
– Test your social footprint?
- FB page (and apps)
- LinkedIn page
- Twitter page
– Ensure security/privacy if you enable users to use social sign-on?
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 20

21. Local: Impact On Testing
• Not just a mobile problem
– Localization testing
– Geo-based personalization
• But it is a mobile problem too
– Apps used outside the confines of the
QA lab, under in-the-wild conditions
• Impossible to test effectively from
only inside the lab
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 21

22. Local: Impact On Testing
• New challenges include, how will you:
– Replicate location (answer: you don’t)
– Test in NYC, SF, LA, DFW, CHI?
– Test abroad
- SIM cards
- Carriers
– Test GPS-based intelligence?
– Test IP-based localization?
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 22

23. Mobile: Impact On Testing
• New challenges include how will you:
– Native apps: Test across ever-changing OS, devices and carriers?
– Mobile web: Test across wide range of OS and mobile browsers?
– Either way, how does your testing replicate real-world conditions
- Specific and varied locales
- Tower-to-tower hopping
- Roaming
- Low battery
- Touch screen actions (pinch, swipe, pan, etc)
- Portrait vs. landscape
- Display definition
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 23

24. Mobile: Impact On Testing
• Mobile web vs. native apps
– 2009: 5-horse race
– 2010: 4-horse race
– 2011: 2-horse race (depending upon your industry)
– 2012: 3 horses (including HTML5-driven mobile web)
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 24

25. F|R|A|G|M|E|N|T|E|D
• Android alone (and that’s only by OS version)
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 25

26. Mobile Testing Landscape
• Form factors matter again
– Phones (feature & smart)
– Tablets
– Hybrids
• Prohibitive HW & SW costs
• Here we go again
– Late-90s web, part deux
- Lack of mature tools
- Lack of usability standards
- Lack of security standards
- Extremely fluid landscape
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 26

27. Multiple Launch Schedules
Multiple products, roadmaps & launch schedules
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 27

28. Test Coverage
Web Testing Matrix Is Tough
Features
OS & Browser
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 28

29. Test Coverage
Mobile Testing Matrix Is Insane
Features
Handset Makers
OS & Browser & Models
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 29

30. Test Coverage
Mobile Testing Matrix Is Insane
Features
Wireless
Handset Makers Carriers
OS & Browser & Models
| 30
In-The-Wild Testing for Functional + Security + Load + Usability + Localization

31. Test Coverage
Mobile Testing Matrix Is Insane
Features
Location
Wireless
Handset Makers Carriers
OS & Browser & Models
| 31
In-The-Wild Testing for Functional + Security + Load + Usability + Localization

32. Increasingly Complex Testing
• The laughably simplistic notion of a “test matrix”
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 32

33. The Challenge
Ummmm…
So What Do We Do?
|
33

34. Sufficiently Freaked Out?!?
• Don’t be… the good news is that testing ain’t dead!
– Chance for QA leaders to step up…
– To lead the charge…
– To defend users and app quality
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 34

35. …And The Bad News…
• You should be scared… need to completely re-think testing
– Can’t automate our way out of this
– Can’t hire or train our way out
– Can’t solve this problem from soley within the lab
• More distributed problem than ever before
– Need a new testing mindset
– Need to bridge the growing gap
- Between testing & users
- Between artificial and real
- Between in-the-lab & in-the-wild testing
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 35

36. Lab-Based Testing Innovation
Manual vs.
Automation Managing vs.
Leading Teams
In-House vs.
Outsourced Agile vs.
Waterfall
Exploratory vs. Testers vs.
Test Cases Engineers
Offshore vs.
Nearshore
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 36

37. Users Work & Live In The Wild
I n
The
Wi l
In-The-Wild Testing for Functional + Security + Load + Usability + Localization
|
37

38. To Make Matters Worse
• Users are less tolerant of quality failures
• Social media & app stores give everyone a megaphone
• And their problem… is your problem
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 38

39. The Challenge
A Call To Action For
Test Leaders (Two, Actually)
|
39

40. 1. Look Beyond The Lab
• Yes, keep testing in-the lab
– Keep training in-house
– Keep outsourcing
– Keep automating
– Keep innovating
• Recognize it will never again be sufficient on its own
• Build your company’s plan for testing in-the-wild
– Real-world conditions: Live testers, real devices, imperfect connectivity
– Mirror your user base
- Technologically: OS, browser, anti-virus, device, carrier
- Geographically: Continent, country, city, language
- Demographically: Age, gender, education, employment, industry, hobby
• In-house QA leadership is vitally important
– You have the business and product context
– You know what your firm cares about
– You need to be the orchestrator who leads this symphony
- What to test in the wild, how to test in the wild, holistic test planning
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 40

41. 2. Be The Voice For QA
• SoLoMo is coming – with or without QA’s buy-in
• Want QA to be heard? Want quality to be built in?
– Don’t wait: No one else is coming to the rescue
– Get informed: Develop an thoughtful opinion
– Get involved: Find out when/where decisions are being made
– Be proactive: Build a plan for quality from the ground up
– Be protective: You’re a protector of the user… of the brand
|
In-The-Wild Testing for Functional + Security + Load + Usability + Localization 41

42. The Challenge
SoLoMoWoSto
A Few Stories (if there’s time)
|
42

43. The Challenge
Questions?
Answers.
Matt Johnston | CMO @ uTest
mattj@utest.com | @matjohnston
|
43