Nokia transitioned to a new mobile platform in 6 months and faced challenges validating mobile apps. They took an adaptive approach to quality assurance, using unit tests during prototyping, component tests during setup, and both automated and manual UI tests during execution. This approach provided minimum quality while allowing tests to focus on different levels from controls to the full app stack. Lessons included automating as much as possible, having testers and developers share environments, and estimating story impact during sprints.

1. Mobile Apps Validation:
Facing the Challenge
Virgilio Raniero
QA&TEST 2011
October 2011

2. “Six months. Be on time.”
Nokia World, two days ago
2 © Nokia 2011

3. Story of a transition
Investigation Prototyping Setup Execution
- Transition to a new platform
- Deep change in mind-set and culture
- Very short time to market (6 months)
- Pivotal role of QA
3 © Nokia 2011

4. Early investigation
Investigation Prototyping Setup Execution
- Quick “know-how injection”
- 3 to 4 team members (1 QA)
- Find out what we have and what we need
- Unit Test framework and CM tools evaluation
- Explore, try and adapt solutions
4 © Nokia 2011

5. Proof of concept
Investigation Prototyping Setup Execution
- ~Daily releases
- Unit Test, CM, CI and bug tracking tools in place
- Component tests evaluation
- UI Visual Test Automation evaluation
- Show some solutions are valuable and why
5 © Nokia 2011

6. Team set-up
Investigation Prototyping Setup Execution
- ~14 people (~35% QA), 2 Scrum Teams, 2-week sprints
- Ramp-up and knowledge transfer
- New CI and bug tracking tools in place
- “Feature branching”
- Let QA influence the architecture to increase testability
6 © Nokia 2011

7. Deliver!
Investigation Prototyping Setup Execution
- Fortnightly releases
- New UI Test Automation in place
- Exploratory+Automated testing
- “Master branch focused”
- Keep the master clean and working
7 © Nokia 2011

8. An adaptive strategy
- Different QA solutions across the project phases
- Work both on emulators and devices
Test coverage
120
100
80
60
40
20
0
Prototyping (I) Prototyping(II) Production (I) Production (II)
Unit Tests Component tests UI Automation Manual Testing
8 © Nokia 2011

9. Flexibility: Test granularity
UI Automated • Functional Test
Tests • Stress Tests
Component • UI native controls
Tests • UI custom controls
• Algorithmic failures
Unit Tests • Static libraries
9 © Nokia 2011

10. Adaptive: Test impact
- Low level services
- Single control and control-to-control interaction
- Complete app stack
UI Automated
test
UI layer
Control1 Control2
Component
test
Data Service Service
component Agent 1 Agent 2 Unit test
10 © Nokia 2011

11. Unit tests
- Constant sanity check on prototype
- Continuous low level verification on product
- Focused on the Model
• White box testing
• Use 3rd party (quasi-)stable frameworks
• Handle asynchronous calls scenarios
• Don’t push it too far!
11 © Nokia 2011

12. Component tests
- Efficient vertical testing of the application
- Focused on Model, View, ViewModel of single
Controls
12 © Nokia 2011
Interaction Properties APIs

13. UI automated tests
- Automated functional tests
- Helps stabilizing the product at each release
- Focused on View and ViewModel of the App
• Grey-to-black box testing
• Too easy means useless
• Let developers and testers share the same environment
13 © Nokia 2011

14. Conclusions
Investigation Prototyping Setup Execution
120
100
80
60
40
20
0
Prototyping (I) Prototyping(II) Production (I) Production (II)
Unit Tests Component tests UI Automation Manual Testing
14 © Nokia 2011

15. What we learned
• Guarantee a minimum quality level
−QA can only lead the change through a robust credibility
• Automate as much as you can
−Exploratory testing will cover edge cases and new scenarios
• Coding tests is the best test
−Testers and developers should share the same development
environment
15 © Nokia 2011

16. Improvements
• Component tests partially surrogate back-end mocking
• Create an automated test per each relevant bug
• CI must be continuous and continuously online
• Estimate the impact of a story change during the sprint
16 © Nokia 2011

17. We are hiring!
- Nokia in Berlin has several open positions
- These are exciting times!
www.nokia.com/careers
developer.jobs.berlin@nokia.com
17 © Nokia 2011

18. Q&A
18 © Nokia 2011