This document discusses how eBay tests its products and services. It notes that testing is done globally by over 300 test engineers across multiple sites. While manual testing remains important, especially for new features, eBay has invested heavily in automation to help test over 3500 test cases across its European sites in under an hour. This large-scale automation was made possible through the use of a testing grid with over 150 virtual machines and browsers. The document shows that increased automation efforts have correlated with reduced bug rates and higher quality bugs found by the EU testing team.
4. Setup
Mobile
Global Portland Global
Product USA Product
San Jose Chennai
USA India
EU
Quality
Engineering
Global Local
Product
(Zürich, London,
Berlin, Paris)
Product
Shanghai Zürich,
China London
Adjacencies
(i.e. Brands
for Friends)
4
Produced by: Michael Palotas
5. Not only the Web
Windows
Desktop
Web
Mobile
5
Produced by: Michael Palotas
6. Landscape
Teams all around the globe in all possible time zones
90.000 development days / year
ina
India + Ch
our ced to
ece s outs
M ajor pi
300 test engineers around the globe
nt
t environme
tes
Unstable
Continuou 1000+ new bugs found every week
s Integratio
n not cons
istent
eBay Inc Produced by: Michael Palotas
6
8. Testing at eBay in the past
Very manual
Manual regression testing for 22 sites, every 2 weeks
Teams did not rely on automation
Skill set of QA Engineers mainly non-technical
Sources: www.linesofflight.net
www.generallyAwesome.com
8
Produced by: Michael Palotas
9. Testing Today – Ownership of Quality
Manual Test
Consulting
+
Coaching
Automation
9
Produced by: Michael Palotas
10. Manual Test
Still very important, mainly exploratory type
All new features are tested manually first
Edge cases
Not automatable tests
Manual testing still yields the highest number of bugs
10
Produced by: Michael Palotas
11. Automation
Automation is not about eliminating jobs !!!
Save time and reinvest into manual testing
Regression Testing (fully) automated
Programmatic approach (JAVA)
Open source only (WebDriver/Selenium 2 GRID)
3500 end to end testcases running in less than 1 hour
11
Produced by: Michael Palotas
12. GRID
eBay Europe:
- 3500 e2e regression testcases
- 2-3 minutes execution time (each)
executions
Execution time 7000 minutes (~116 hours = ~5 days)
Sequential
Parallel
Execution Test
1
Test
2
Test
3
Test
…
Test
…
Test
3500
Execution Time
Parallel Test Test Test Test Test
Parallel executions
150 Virtual Machines / Browsers in
execution in Test Test
the WebDriver GRID
Test Test Test
Execution time: 60 minutes
Grid
Test Test Test Test Test
Execution Time
12
Produced by: Michael Palotas
13. Coaching / Consulting
EU team conducts project audits
Improvement initiatives
Consulting for dev and test teams
13
Produced by: Michael Palotas
18. eBay checkbox – vs. HTML checkbox
<div align="center”>
<input type="checkbox" name=”myName" value=”eBay"> eBay<br>
</div>
18
Produced by: Michael Palotas
19. Summary
Testing is changing
Great career opportunities in testing
More technical skill sets required
Testautomation can have a big impact
….
or make your life hell
Essential to work with developers
Measure and improve
19
Produced by: Michael Palotas