Crowd-sourced Automated Firefox UI Testing

Crowd-sourced Automated
Firefox UI Testing
Henrik Skupin <hskupin@mozilla.com>
Lead of QATest Automation
quality.mozilla.org qa automation
Thursday, November 4, 2010

Who is Mozilla QA?

WHO IS MOZILLA QA
Group of about 30 employees / contractors
Are you serious?
No!

WHO IS MOZILLA QA
1000s of community members across the globe
who are partners to the Mozilla QA core team

Mozilla Scale
400+
Developers
10,000+
Nightly Testers
2,500,000
Beta Testers
400,000,000
Users
400,000,000
350,000,000
300,000,000
250,000,000
200,000,000
150,000,000
100,000,000
50,000,0000

What are our Areas of Work?

• Testing new features in beta versions and nightly builds
• i.e. Switch toTab,Add-ons Manager, ...
• Testing release builds before they get shipped
• Smoketests across all platforms
• ~ 300 basic functional tests (BFT)
• Software update tests
WHAT ARE OUR AREAS OF WORK?

A Brief History of Automation in Mozilla QA

A BRIEF HISTORY OF AUTOMATION IN MOZILLA QA
• No automated functional tests existent by early 2009
• 100% manual testing of new Firefox features
• 100% manual testing of release builds
(limited to a single platform)
Huge amount of manual resources necessary!

So what has been changed so far?

• Integration of MozMill (based on WindMill)
for functional tests
• Automation of basic tests (30% done - ~100 tests)
• Automation of update tests (nearly done - Only
1single person needed to run 192 different update
paths across 5 platforms and 4 P1 locales in under 2
hours)
Perfect time to open up testing for everyone

Crowd SourcedTesting

CROWD SOURCEDTESTING
Jeff Howe: "It’s not outsourcing; it’s crowdsourcing."

Crowdsourcing relies on people, and thus works when
participants are treated as people

What can be achieved?
• Support testers, localizers, and add-on authors to
allow them to focus on their main tasks
• Constantly testing builds across platforms and locales
• Assembling test results from a large number of
different environments (settings and performance)
How can this be done?

MozMill Crowd Extension

MOZMILL CROWD EXTENSION
Pre-requisites:
• Trivial setup (“One click install”)
• Uses MozMill to run tests in any version of Firefox
• Support for different test-runs (basic tests, add-ons, ...)
• Reporting test results to a central database
Under Construction

MOZMILL CROWD EXTENSION

What remains:
• How reliable are our tests in the crowd
• Fix focus issues to allow background tests (webdriver)
• Analyzing results based on the environment
• ...

DEMO
What will be shown?
• Setup of the test environment by using MozMill-Crowd
• Test-run for a subset of general Firefox tests
• Analyzing test results
• Test-run for the GoogleToolbar extension

REFERENCES

QATest Automation team page:
http://quality.mozilla.org/teams/automation/
MozmillTests Repository:
https://developer.mozilla.org/en/Mozmill_Tests
Mozmill Code:
https://github.com/mozautomation/mozmill
Mozmill Crowd Extension Project:
https://wiki.mozilla.org/QA/Mozmill_Test_Automation/
Crowd_Testing

QUESTIONS & ANSWERS
Questions?

Thanks

Crowd-sourced Automated Firefox UI Testing

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Crowd-sourced Automated Firefox UI Testing

Similaire à Crowd-sourced Automated Firefox UI Testing (20)

Dernier

Dernier (20)

Crowd-sourced Automated Firefox UI Testing