If you count the alternatives, there are 50 tools for software testing focused on open source projects - test planning and management, test execution, test reporting, front-end and backend testing, automated mobile testing, security scanners, issue tracking and others

1. www.questers.com
10 Useful Testing
Tools for Open
Source Projects
Peter Sabev
QA Manager
Questers
psabev@gmail.com
@BORIME4KA
www.linkedin.com/in/petersabev
www.facebook.com/peter.sabev

3. АКО ИМАХ ESP8266 ИЛИ ПОНЕ...

4. GOOGLE TRENDS: ARDUINO IN BULGARIA

5. 1 час по-късно…

6. THE POWER OF COMMUNITY!

8. OPEN SOURCE VS. PROPRIETARY

9. OPEN SOURCE VS. PROPRIETARY
• Всичко е публично (код,
тестове, бъгове, data flow и т.
н.)
• Лъсваш пред цял свят (но и ти
помагат да си оправиш
нещата).
• Хората, които тестват и
оправят бъгове, го правят ЗА
УДОВОЛСТВИЕ.
• Рядко нещо е публично. Никой
отвън не знае какво има отдолу,
как работи и кои данни къде
отиват.
• Ако тестовете ти са кофти,
продуктът ти също може да е
кофти и компанията губи пари и
имидж.
• Хората, които тестват и оправят
бъгове, го правят ЗА ПАРИ

10. OPEN SOURCE VS. PROPRIETARY
• Тестовете рядко зависят от
конкретен environment, setup
или конфигурация.
• Сензитивна информация
(пароли, device id, hostnames,
ports) се подава като
параметър, защото всичко е
публично.
• Тестовете са настроени да се
пускат в тестовата среда на
дадената компания.
• Сензитивната информация
обикновено не е проблем,
защото тестовете се пускат в
изолирана среда.

11. OPEN SOURCE VS. PROPRIETARY
• Всеки може да допринесе за
продукта и тестовете
• Нужен е добър механизъм да
отсееш хубавите неща от
боклука.
• При добре развито community,
нещата се тестват много
бързо от много хора, с много
гледни точки.
• Определена група
професионалисти (или не чак
такива) допринасят за
продукта и тестовете.
• Оправянето на проблеми
минава през по-дълъг процес
на одобрения, обикновено
фиксът се тества от един
човек и една гледна точка.

12. OPEN SOURCE VS. PROPRIETARY
• Хората тестват каквото си искат,
колкото си искат => част от
нещата може да са
неизтествани…
• Практически няма
централизирани tools, стратегия
за тестване и поради тази
причина static code analysis,
performance, security и др. се
тестват по-рядко.
Има ясна тестова стратегия и
тестов план

13. КАКВО Е НЕОБХОДИМО
ЗА КАЧЕСТВЕН OPEN SOURCE ПРОЕКТ?
• По-добра стратегия, процеси
и управление на тестовете!
• Continuous Integration и
Continuous Deployment са
изключително важни;
• Задължително Code Review;
• Подходящи интегрирани
testing tools.

14. Благодаря за
вниманието!
Въпроси?
Peter Sabev
QA Manager
www.psabev.info
psabev@gmail.com
www.facebook.com/peter.sabev
www.linkedin.com/in/petersabev
skype: psabev82

15. СИПВАМ С КОФАТА!
10 TESTING TOOLS
50

16. TESTING TOOLS
Requirements
and Test
Management
API Testing
Cross-
Browser
Testing
Mobile
Testing
Performance
Testing
Web Robots
Other
Checks
Test
Reporting
Tools
Security
Testing &
Scanning
Bug
Tracking
Backend
Front-end

17. TOOL NO. 1:
TESTLINK
A GPL open source web-based
requirements and test management and
test execution system that allows
executing test cases and tracking test
results dynamically
http://testlink.org/

18. BASIC FEATURES
• Manage test cases, builds, results, milestones and tester
assignments
• Many people can work on test planning, design and execution
simultaneously
• Traceability between requirements and tests
• Excellent bug tracking system integration (Mantis, JIRA, Bugzilla,
FogBugz, Redmine, and others)
• Custom UI, fields and user roles

20. THE ALTERNATIVES
• Fitnesse (http://www.fitnesse.org/), acceptance testing and
team collaboration
• Tarantula (http://www.tarantula.fi/), modern tool for agile
software projects test management, has requirements, small
but growing community
• qaManager (http://qamanager.sourceforge.net/site/en/),
releases tracking, good reporting, keeps track of testing
cycles

21. TOOL NO. 2:
SoapUI is a free and open source
cross-platform tool that allows you to
easily and rapidly create and execute
automated functional, regression,
compliance,
and load tests
www.soapui.org

22. BASIC FEATURES
• Simple drag-drop test creation
• Complete test coverage for SOAP, WSDL, REST, JMS, db and
others
• Service mocking for SOAP, WSDL and REST
• HTTP recording and replaying
• Good load testing integration with LoadUI
• Plugins support
• Transform functional test into basic security test

24. THE ALTERNATIVES
• Advanced REST client
(https://chromerestclient.appspot.com/), Google Chrome
plugin for creating and testing custom HTTP requests
• Rest Assured (https://code.google.com/p/rest-assured/),
Testing and validating Java REST services in Ruby and Groovy
• Postman (www.getpostman.com), construct, send and save
requests quickly and analyze the responses sent by the API

25. TOOL NO. 3:
A suite of tools to automate web
browsers across many platforms.
www.seleniumhq.org
Selenium

26. BASIC FEATURES
• Consists of basic and advanced part
• Selenium IDE for simple automated scripts (a Firefox add-on)
• Selenium WebDriver for robust, browser-based regression automation
suites and tests
• Very big community, including adoption in many commercial products
• Supports Java, C#, Ruby, Python, Javascript, hundreds of other plugins
• Scalable and portable - same scripts can be run on multiple
environments (different number of machines, OS & browsers)

28. THE ALTERNATIVES
• Watir (http://watir.com/), open source (BSD) family of Ruby
libraries for automating web browsers
• WatiN (http://watin.org/), inspired by Watir but uses C#.
• Sikuli (http://www.sikuli.org/), screenshot based automation
• Canoo WebTest (http://webtest.canoo.com/), open source tool
using Java 5, Groovy 1.6 and HTMLUnit 2.4
• Webrat (https://rubygems.org/gems/webrat/), a Ruby gem

29. TOOL NO. 4:
Appium aims to automate any mobile
app from any language and any test
framework, with full access to back-end
APIs and DBs from test code
http://appium.io/

30. BASIC FEATURES
• Free, open-source, big community
• Supports Selenium (i.e. tests can be written in Python, JS with Perl,
Node.js, Java, Ruby, PHP, C#, RobotFramework, Objective-C,
Clojure)
• Focus on both iOS and Android native, hybrid or mobile web apps
• Can be tested on real devices, simulators or emulators
• Aims full access to mobile backend (e.g. turning wi-fi on/off) and DB
• No need to recompile the apps tested

32. THE ALTERNATIVES
https://code.google.com/p/robotium/ http://calaba.sh/
• Ruby knowledge needed
• Reuse scripts for iOS and
Android
• For iOS, needs adding additional
library
• All code written in Java
• Heavy focus on Android and less
on iOS
• Generally very stable which is
important for mobile world

33. TOOL NO. 5:
Cross browser testing on multiple browsers,
platforms and resolutions
www.equafy.com

34. BASIC FEATURES
• Record/replay tool for different browsers (incl. mobile) and
resolutions
• Execute on one baseline browser and compare to the others
• Screenshot comparison to check what exactly has failed
• Pretty and easy-to-use UI, one-click re-execution
• Automatic scan and Selenium scripts import plus execution
• Made in Bulgaria by experienced QA specialists
• Free for open source projects

36. THE ALTERNATIVES
www.browserstack.com http://saucelabs.com/
• Cloud access to 1000’s of
device/browser combinations
• Developer tools included in the
browsers (such as Firebug, YSlow)
• Separate use / automate accounts
• Combines Selenium with JS unit
tests
• Very good integration with Appium
• Video recordings for all manual test
executions

37. TOOL NO. 6:
Apache JMeter™ application is
open source Java application
designed to load test functional
behavior and measure
performance.
http://jmeter.apache.org/

38. BASIC FEATURES
• 100% open source, Java
• Wide variety of server/protocol types to test with: HTTP, HTTPS,
SOAP, REST, FTP, DB via JDBC, LDAP, JMS, SMTP, POP3,
IMAP, MongoDB, TCP, native commands and shell scripts
• Multithreading scalable for heavy load on a server, group of
servers or network
• Core extensible with pluggable functions
• Caching and offline analysis/replaying of test results

40. THE ALTERNATIVES
• OpenSTA (http://opensta.org/), distributed testing architecture
for HTTP(S) heavy load tests with performance measurements
• Grinder (http://grinder.sourceforge.net/), load testing framework
with flexible scripting in Jython and Clojure
• Pylot (http://www.pylot.org/), has GUI and console modes
• Multi-Mechanize (http://testutils.org/multi-mechanize/), Python
• Gatling (http://gatling.io/) is an open-source load testing
framework based on Scala, Akka and Netty

41. TOOL NO. 7:
The Zed Attack Proxy (ZAP) is an
easy to use integrated
penetration testing tool for
finding vulnerabilities in web
applications.
https://www.owasp.org/index.ph
p/OWASP_Zed_Attack_Proxy_Pr
oject

42. BASIC FEATURES
• The Open Web Application Security Project (OWASP) is a big
community including corporations, universities and individuals
• Works as intercepting proxy, no changes in code needed
• Easy to use for people who are new to pen testing
• Automated scanning plus manual tools for vulnerabilities
• Supports SSL, smart cards, web sockets and wide range of
scripting languages

44. THE ALTERNATIVES
• Nessus (http://www.tenable.com/products/nessus-vulnerability-
scanner), passive scanner for network traffic
• Wireshark (https://www.wireshark.org/), protocol analyzer
• OpenVAS (http://www.openvas.org/), open vulnerability scanner
• Wapiti (http://wapiti.sourceforge.net/), web apps scanner
• Fiddler (http://www.telerik.com/fiddler), web debugging proxy
• GoLismero (http://www.golismero.com/), combined scanners

45. TOOL NO. 8:
GTmetrix tells you a lot about your
website performance and gives
recommendations by combining
PageSpeed and YSlow scores.
www.gtmetrix.com

46. BASIC FEATURES
• Very easy to use – just type an URL
• PageSpeed and YSlow scores
• Page Load Details (time, size, number of requests)
• Various Analysis Options – test from different regions and
browsers
• Waterfall, Video and Report History
• Recommendations how to improve the website

48. THE ALTERNATIVES
• Jenu/Xenu (http://jenu.sourceforge.net/) – Link checker
• Markup Validation Service (https://validator.w3.org/) - HTML
validator, supports HTML, XHTML, SMIL, CSS, MobileOK & other
• Power Mapper (http://try.powermapper.com/Demo/SortSite)
• Website Speed Test (http://tools.pingdom.com/fpt/)
• Spell Checker (http://bit.ly/VhVuyg), limited to 5 uses per day
• Nibbler (http://nibbler.silktide.com/), UX & marketing oriented

49. TOOL NO. 9:
Serenity BDD helps you write better,
more effective automated acceptance
tests, and use these acceptance tests
to produce world-class test reports and
living documentation.
http://www.thucydides.info/

50. BASIC FEATURES
• Based on Behaviour Driven Development (BDD)
• Screenshots for each step in the test
• Very nice-looking test reports including:
• Details on passed/failed execution
• Error messages for test cases
• Execution times
• Functional test coverage
• Integration with popular bug tracking systems like JIRA

52. THE ALTERNATIVES
• Testopia (https://developer.mozilla.org/en-
US/docs/Mozilla/Bugzilla/Testopia), a generic tool for
tracking test cases, test management and test reporting
• Zephyr (http://zephyragile.com/), test management and
reporting platform, integrates with JIRA but can be used
standalone
• Sonar (http://www.sonarqube.org/), an open platform to
manage code quality, works great with Java/Maven/Jenkins

53. TOOL NO. 10:
The flexible and scalable issue tracker
for software teams.
https://www.atlassian.com/software/jira

54. BASIC FEATURES
• The most popular test management platform, free for open-source
• Mobile browser support, email subscriptions and notifications
• Highly customizable – gadgets, dashboards, filters, reports, workflows
• Custom issue types, fields, statuses and resolutions
• Massive plugin ecosystem, remote APIs
• Bulk issue modifications
• Wikipedia plus social network style collaboration

56. THE ALTERNATIVES
• Bugzilla (www.bugzilla.org), the open-source alternative to JIRA
• Mantis (www.mantisbt.org), open source, PHP/MySQL-based
• Trac (http://trac.edgewall.org/), Python based
• Redmine (www.redmine.org) – Ruby-based, GPL, cross-platform
• Trello (https://trello.com/), simple collaboration tool with boards
and cards, very useful for smaller projects

58. SPECIAL THANKS TO
Dimitar Topuzov
Slavy Slavov
Borislav Traykov

60. QA BEER?
https://www.facebook.com/groups/
qabulgaria/

61. Благодаря за
вниманието!
Въпроси?
Peter Sabev
QA Manager
www.psabev.info
psabev@gmail.com
www.facebook.com/peter.sabev
www.linkedin.com/in/petersabev
skype: psabev82