8 minute lightning talk from the September 23, 2011 NYC Selenium meetup at Etsy labs. http://bit.ly/p1duO3

1. Scaling Selenium
Noah Sussman
Etsy

2. 2006: Test JavaScript and CSS locally

3. 2011: Test the whole Web application stack.

4. Why would we need to scale our Selenium testing?

5. MTBF: Mean Time Between Failures
MTTR: Mean Time To Recover
MTTR > MTBF

6. It’s not whether you get knocked down; it’s
whether you get back up. ~Vince Lombardi

7. Recovery usually requires deploying new code

8. faster deploys = smaller changesets = fewer change-related outages

9. End-to-end tests are slow as hell

11. Scaling communication, concurrency and hardware

12. Most people have a hard time
interpreting the results of one failing test

13. Aggregating and reporting
on test results becomes
increasingly challenging.

15. graphs > emails

17. Use Jenkins to run tests concurrently.

18. There are other ways to do it, but
Jenkins was the simplest (for us).

19. Jenkins Matrix Build plugin

20. ######
Master Build ######
########

22. Most regressions are catastrophic

23. Walk the happy path

24. early 2010:
Headless Firefox on Linux, raw EC2 instances

25. late 2010:
Physical Hardware

26. xxxx
2011: SauceLabs
xxxx

27. hybrid cluster

28. early 2010: about 10 people making 5 deployments a day
~150 Selenium tests run every day
mid 2011: about 20 people making 30+ deployments
~700 Selenium tests a day

29. Lessons Learned
Communication and discoverability remain our biggest challenges.
Jenkins’ concurrent build capacity is a bottleneck.
Even with a vendor in front of it, the cloud is still the cloud.

30. etsy.com/careers

31. Noah Sussman
noah@etsy.com
@noahsussman

32. photo credits
parkour guy: loki_v http://www.flickr.com/photos/loki_v/268938583/
easy button: crd http://www.flickr.com/photos/crd/283156445
sleeping cat: binibix http://www.flickr.com/photos/binibix/6090137450
etsy bike rack: cubicle spy http://mashable.com/2010/10/13/etsy-office-pics
etsy infrastructure diagram: john allspaw
etsy deployment velocity graph: erik kastner
upside down server: chris munns
http://www.flickr.com/photos/teknogeek/5316475923/