- Mixing open source and commercial tools can provide benefits but also risks that require careful integration. A case study describes blending open source and commercial testing tools for a Java application. Subversion, JIRA, Eclipse, IBM Rational Functional Tester, and Maveryx were combined in the test environment. The strategy was to reuse tests developed in Rational Functional Tester for legacy functionality and develop new tests for new features using Maveryx.

1. Mixing Open and Commercial
Tools
Mauro Garofalo, Maveryx
W22 – Automation

2. Summary
 Costs, Advantages and Risks of mixing open
source and commercial software
 A case study of blending open source and
commercial tools for testing Java applications
 Some practical rules to combine open and
commercial test tools
2
All trademarks referenced herein are the properties of their respective owners.

3. So, what is Open Source?
 First, no official definition exists (http://www.opensource.org/osd.html)
 Open source doesn't just mean access to the source
code
 Open source rules are, simply enough:
 Allow free redistribution
 Allow source code access
 Permit derived works
 Protect the integrity of the author‟s source code
 ….
3

4. Key Predictions
 By 2012, at least 80% of all commercial software solutions will
include elements of open source technology (Gartner Highlights Key
Predictions for IT Organizations and Users in 2008 and Beyond)
 By 2010, Global 2000 IT organizations will use open-source products
in 80% of infrastructure-focused software investments and 25% of
business software investments (Gartner's Positions on the Five Hottest IT
Topics and Trends in 2005)
 …
4

5. Survey: Do you use open source software?
 Do you use…
 Firefox to browse the web?
 OpenOffice to write documents?
 Linux or Ubuntu as server or end user OS?
 Subversion or CVS for the source code?
 MySQL as database?
 Eclipse to write code?
 FileZilla for file transfer?
 …
 http://downloadpedia.org/Open_Source_Alternative_to_Commercial_Software
5

6. Companies use open source because…
 COST is the main reason why companies are turning to open source
[Jeffrey Hammond, Forrester Research]
 But cost is not the only reason for open source‟s growing popularity:
many firms now know that it offers more FLEXIBILITY than
proprietary programs [Matthew Aslett, 451 Group]
 Improving quality of products a/o processes
 Achieving faster time-to-market
 Higher level of security
 …
6

7. OSS vs. Commercial
 Commonly it is framed as
 Linux vs. Microsoft
 Security vs. Innovation
7
 Rewards of OSS
availability of source code
freedom to customize, enhance or
repair
lower price, sometimes free
does not depend on vendor
security
…
 Risks
lack of support, training and docs
frequent updates & forks
backwards compatibility
…
 Rewards of Commercial software
vendor professional services
easier to adopt in organizations
comply with industrial standards
more user friendly
mature products offer more
features
…
 Risks
costs
dependent upon vendor
do not allow users to alter or
customize the product
.…

8. A ‘mixed source’ strategy
 The greatest chance for success with open source software involves a
strategy of mixing open with commercial software
 A mixed-source strategy means identifying and integrating both open
and commercial products that
 are best-of-breed technologies
 fits the requirements
 minimizes integration „headaches‟
rather than using “good enough” products
 included in legacy solutions or
 available (for free…) on SourceForge!
8

9. Integration
 Mixing open & commercial tools requires integration and interoperability
 Integrating any two programs is often challenging…
 Integrating commercial software & OSS can be difficult because:
 variety of technologies
 (commercial) tools use proprietary „approaches‟
 data integration is not friendly (OpenOffice vs. Microsoft Office file formats)
 no design for interoperability
 different products have varying requirements for integration
 many products are intended for use with specific combinations of tools or designated platforms
 …
 The result is that integration of OSS with commercial offerings is often partial
or missing completely, or must be provided by a third-party vendor…
9

10. Test Tools
 Application Test Tools for:
 Source testing
 Functional & GUI testing
 Load & Performance testing
 Embedded testing
 Database testing
 …
 Supporting Tools for:
 Test Management
 Bug Tracking
 Requirements Management
 Configuration / version control
 …
10

11. Can they cooperate?
11
 Open Source  Commercial

12. Case Study
 Testing a Java™ stand-alone, GUI-based application for the
Telecom market
 The application-under-test has
 „legacy‟ functionalities (taken from past project / application)
 new features continuously changing (specially at early stage)
 Reusing of many automated functional tests built with IBM Rational
Functional Tester for legacy functionality testing
12

13. The Test Environment
13

14. Subversion
 Subversion is a free & open source version control
system
 Industry standard
 Easy to use
 Local baseline
 Atomic commits
 One repository to maintain (single url for all projects)
 …
 There are several plug-ins implementing SVN support into
Eclipse (→ Subversive)
14

15. JIRA
 JIRA is a commercial issue, bug and project tracking tool
 Easy to use
 Highly configurable (e.g. to manage a complex workflow)
 Flexible for different kind of issues (e.g. for test reporting)
 Accessible from tests scripts through Java API
 Extensible and customizable (source code is available to customers)
 Eclipse integration
 Subversion integration
 Fully supported
 Quite cheap
 …
15

16. Eclipse
 Eclipse is a free & open source development environment
comprising an IDE and an extensible plug-in system
 Industry standard
 Fully supported
 Extensible and configurable more than 1200 plug-in available at the
Marketplace (most free!)
 Simple and powerful Workbench → Views, Editors and Perspectives
 …
16

17. IBM Rational Functional Tester
 Rational Functional Tester is a commercial tool
for automated functional and regression testing
 Built-in support for Java applications
 Java scripting language
 Eclipse rich client application
 ScriptAssure technology to accommodate UI changes
 Data-driven & Keyword-driven testing
 …
17

18. Maveryx
 Maveryx is a free & open source tool for automated
functional, regression, GUI and data-driven testing
 Built-in support for Java applications
 No „GUI Map‟ needed to create and run tests
 Intelligent GUI objects recognition (including fuzzy matching)
 Available as Eclipse plug-in
 Extensible and configurable through plug-ins
 Integrated with all Java IDEs and testing frameworks (Eclipse,
NetBeans, JUnit, IBM Rational Functional Tester, …)
 …
18

19. Test Strategy
 Legacy functionalities tested by reusing test scripts
developed with IBM Rational Functional Tester
 Needed to update / re-capture the GUI Maps for the
current application
 New functionalities tested by using Maveryx
 Maximizing the Rational Test Manager usage
 Refactoring of some tests by mixing, where
possible, Rational code with Maveryx code
19

20. Mixing open & commercial tools
 You must decide which components be open vs. closed source
 A possible strategy:
 closed source for “commodity” components
 open source for “differentiating” components
 Following such a strategy:
 classify features as commodity or differentiating
 determine for each feature whether using open or closed SW based on:
 budget
 technology
 know-how
 requested quality
20

21. Evaluation Process
21
Identify candidates
Read existing reviews
Compare attributes to your needs
Analyze top candidates
Wrap-up

22. Identify Candidates
 First step : find out your options
 Search for open and commercial tools:
 ask friends and co-workers
 search using specialized sites
 use search engines
 thumb specialized magazines
 …
22

23. Read existing reviews
 After you've identified your options, read existing evaluations about the
alternatives:
 use search engines
 search for specialized web sites
 read blogs, forum
 thumb specialized & independent magazines
 look at social networks
 …
23

24. Evaluation Criteria
24
TOOL #1 TOOL #2 TOOL #3
Functionality
Cost
Support
Market Share
Maintenance
/ Longevity
Reliability
Performance
Scalability
Usability
Security
Flexibility /
Customizability
Interoperability
…

25. Analyze top candidates
 After the initial evaluation, pick the top candidates, and perform a more
in-depth analysis
 Try any contender in non-critical situations first, by evaluating the
functionality that you‟re interested in using (use a checklist)
 Examine also the program‟s documentation, source code (for OSS),
and other related materials
 If the tool had some but all the functions you need, examine what it
would take to add those functions
25

26. Evaluating Software
 In general, a mature platform will meet the following
criteria:
 Project extensions are available
 The project has reached a one-year maturity mark
 Security patches, bug fixes and new features/enhancements are delivered
separately
 The software has reasonable automated unit and functional tests with code
coverage in the 30 percent - 80 percent range
 The software easily integrates with external tools
 The component‟s bug database is kept up-to-date with revision numbers for each
product enhancement
 The solution has been ported across multiple platforms (Linux, Windows, Solaris
and Mac)
 Large-scale adoption, including both public and well-known large-scale
organizational deployments exists
 There is documentation purpose separation: User guide (or Getting Started
Guide), Installation guide, Admin guide and Development guide
26

27. Important factors influencing testing tools
 Factors influencing the choice of Software Testing Tools
 Programming Language
 Support
 Long feedback
 Evaluation Period
 Selling is driven by Marketing
 Vendor locking
 Specialization & Generalization
 …
27

28. Wrap-up
 Review and present to your manager the results of your evaluation
 Identify the recommended solution
 Identify the main alternatives, first of all the recommended alternative
 Once a decision has been made, get the software & go!
28

29. Conclusion
 A “mixed-source” approach combines open source & proprietary software,
by taking the best of both worlds
 The right mixed solution offers advantages in flexibility, functionality,
performance, and management that aren't available when either open
source or commercial technologies are used alone
 But, integrating commercial software and OSS can be difficult…
 For testing purposes, evaluate open source and commercial tools by making
decisions based on cost, technical and integration factors
29

30. 30
info@maveryx.com