Mozilla QA has created a new feature called the Mozmill Crowd Extension that allows any Firefox user to automatically and effortlessly run our set of basic Mozmill functional test cases against their installed Firefox browser and report the results to our central reporting server. Any user in any locale can now participate in giving us valuable feedback on how their browser is operating in their environment. Mainly targeting the thousands of Firefox nightly testers, this extension will give Mozilla unprecedented insights on how a particular nightly build is working across the globe. In this session we will describe and demonstrate how the extension works, what data is collected from the tests and how the results are reported and analyzed.

1. quality.mozilla.org qa automation
Mozmill Crowd
A change in how we test Firefox
Henrik Skupin <hskupin@mozilla.com>
Lead of QA Test Automation

3. Overview
• Release schedule for Firefox
• Testing Firefox releases
• How the Automation helps QA
• Who contributes to QA?
• Crowd-sourced Testing (Mozmill Crowd)
• Demo

4. FIREFOX RELEASES

5. FIREFOX RELEASES
Firefox 4.0 Beta
• Lately approx. 2 weeks beta cycles
• Various bug fixes and improvements
Firefox 3.5 and 3.6
• Security releases each 4 to 6 weeks
• Both releases happen simultaneously
• About 50 - 80 fixed bugs per release

6. FIREFOX RELEASES

7. FIREFOX RELEASES
Testing Firefox Releases
• Exploratory and scripted testing of affected areas
• Execution of functional tests, including:
• approx. 40 Smoketests
• approx. 330 BFT tests (Basic Functional Tests)
• Time to ship to beta audience - only a few days
• Time to ship the release - about 1 week

8. FIREFOX RELEASES

9. FIREFOX RELEASES
Testing Firefox Releases
• Software Update
• Tests from previous versions (partial / complete)
• Testing ‘betatest’ and ‘beta’ channels before beta
• Testing ‘releasetest’ and ‘release’ channels before
release

10. FIREFOX RELEASES

11. FIREFOX RELEASES
How Automation Helps QA
• Usage of Mozmill to run automated functional tests
• approx. 110 (out of 330) functional tests automated
• Execution of localization specific tests for all locales
• (Nearly) fully automated update tests for all locales
• Simultaneously execution on all supported platforms

12. CONTRIBUTING

13. CONTRIBUTING
Who contributes to QA?
• Small QA team (~30) and contractors for Releases
• Community members execute manual tests on Litmus
• 80.000 nightly users for Firefox 4.0 builds
• 15.000 nightly users for older Firefox branches

14. CONTRIBUTING
Who contributes to QA?
• Small QA team (~30) and contractors for Releases
• Community members execute manual tests on Litmus
• 80.000 nightly users for Firefox 4.0 builds
• 15.000 nightly users for older Firefox branches
How can we outreach testing?

15. CONTRIBUTING
How can we outreach testing?

16. CONTRIBUTING
Who contributes to QA?
• Small QA team (~30) and contractors for Releases
• Community members execute manual tests on Litmus
• 80.000 nightly users for Firefox 4.0 builds
• 15.000 nightly users for older Firefox branches
How can we outreach testing?

17. CROWD SOURCED TESTING
Crowdsourcing relies on people, and thus works when
participants are treated as people

18. CROWD SOURCED TESTING
Crowd-sourced Testing
Crowdsourcing relies on people, and thus works when
participants are treated as people

19. CROWD SOURCED TESTING

20. CROWD SOURCED TESTING
What do we expect?
• Improved quality of localized builds
• Knowledge how Firefox behaves in different locations
• Results from machines with unknown configurations
• Performance related information and memory usage
• ...

21. CROWD SOURCED TESTING

22. CROWD SOURCED TESTING
What’s necessary?
• No global installation of additional software
• Automatic setup of the Mozmill-CLI environment
• Wide range of different test scenarios (BFT, L10n, ...)
• Reporting and analysis of collected test results
• Promoting tests even outside of our community

23. CROWD SOURCED TESTING

24. CROWD SOURCED TESTING
How it works
• Mozmill-Crowd add-on for Firefox 4.0 and 3.6
• Automatic setup of the prepared test environment
• Simplified UI for the configuration of the test-run
• Automatic download of the latest tests
• Test execution and reporting to our database

25. CROWD SOURCED TESTING
User Profile Testing Profile
3.
1.
Repositories
2. (Tests and Scripts)
MozMill
CLI 4.
Database
Virtual Test Environment

26. CROWD SOURCED TESTING
How it works
User Profile Testing Profile
3.
1.
Repositories
2. (Tests and Scripts)
MozMill
CLI 4.
Database
Virtual Test Environment

27. CROWD SOURCED TESTING

28. CROWD SOURCED TESTING
Analyzing test results
L10n test results
• Multiple access keys
• Cropped elements
• Screenshots

29. CROWD SOURCED TESTING

30. CROWD SOURCED TESTING
What’s left
• Proof reliability of our Mozmill tests
• Fix focus problems with Mozmill (native events)
• Offer more options in the extension for contribution
• Add more test-runs (i.e. add-ons and endurance tests)
• We need your feedback!

31. REFERENCES
Demo

32. REFERENCES
Mozmill Crowd:
https://addons.mozilla.org/firefox/addon/mozmill-crowd
QA Test Automation team page:
http://quality.mozilla.org/teams/automation/
Mozmill Code:
https://github.com/mozautomation/mozmill
Mozmill Tests:
https://developer.mozilla.org/en/Mozmill_Tests

33. QUESTIONS & ANSWERS
Any Questions?

34. Thanks