From open source to commercial tools - how we selected the right tool for our automation framework in several projects and products in HPE Software.
Selecting the right automation framework is hard. Today, there are so many possibilities. From open source solutions (Selenium Appium, etc.), to our own developed commercial tools (QTP, UFT, Mobile Center) What was the process that we do in order to select the right automation framework. Our approach was to 1st define our needs from such a framework and we ended up with a list of 10 must have requirements for such a framework and then went on to pick the right tool. We ended up with different frameworks for different projects with a wide combinations of open source (yes, we at HP use open source) to our own tools (UFT, LeanFT, etc.)
2. About me - @bendet_ori
5.5y in HPE Software in various managerial QA roles
Michael’s father | Naomi’s husband
For small talk later:
Today – Inbound PM for UFT & a new cloud
offering
Largest Pharm.
Customer in
Europe
Critical Defect in
Diaper
14. Analytic Platform for IT Executives (v 1.0)
– AUT technology Stack
– Glassfish Server
– Flex + GWT
– SAP BODS for ETLs
– SAP BOE for BI
– MSSQL
– Automation
– People doing automation: 0.5 / 5
– Focusing mainly on APIs
– Java Beans (EJBs)
– Internal tool called FIST
– ROI: LOL
14
3.5 hours to install
21. Lessons Learned
Standalone Tool developed internally
– Cons
– Nobody knew about the tool, wasn’t cool, no buzz around it
– External tool: unable to get DEV to cooperate with automation (or even install the tool)
– Manually triggered (no part of the CI process)
– Almost every new test required changes to be done by DEV (expose new API methods)
– No direct access to source code
– Pros
– Small investment
– QTTV
– Easily extendable
– Stability
21
22. Analytic Platform for IT Executives (v 1.0 2.0)
– AUT technology Stack
– Glassfish Server
– Flex + GWT
– SAP BODS for ETLs
– SAP BOE for BI
– MSSQL
– Automation
– People doing automation: 4 / 10, dedicated developer to assist
– Focusing mainly on APIs Automation installation, E2E flows
– Java Beans (EJBs) REST APIs
– Internal tool called FIST Internally built framework REST client, Selenium, Flex Monkey
– Automatic Deployment Solution: internally developed tool called Slick
– ROI: 3 MD each sprint 1 MM per release
22
Automatic Deployment Solution
24. Why?
– High demand for automation coverage
– Developers selected the automation framework
– Wanted to work together with developers
– Invested in automated installation as well
24
25. Lessons Learned
Internally built Automation Framework (inside the IDE)
– Cons
– Required more coding skills
– Large effort of getting things started
– Harder to be used by less-technical testers
– UI automation stability
– Pros
– Harness developers into the automation
– Developers re-used testing assets for their own benefits
– Testers have access to source code
– Part of the CI process
25
34. TruClient?
TruClient is a tool for recording Web-based applications. It is used inside LoadRunner for
performance testing on the browser level
34
1. TruClient Sidebar
2. TruClient Toolbox
3. Firefox browser
4. Application Browser
Window
5. TruClient Sidebar
Status Pane
35. TruClient
– AUT technology Stack
– Firefox Plugin
– Pure Web
– WPF
– Windows app
– Automation
– People doing automation: 2 / 6
– Did not have an automation suite as they could not find a tool which has automation abilities for the full flow –
including the three technologies.
– Supports the three main browsers and needs an automation tool that can identify and test the objects in all the
supported browsers
– Selected LeanFT as the tool
35
36. The Automation Suite
36
LeanFT Application
Model
Containing the full AUT
hierarchy, the App
Model displays a
Modular view of all the
objects implemented in
the tested application
37. The Automation Suite
37
LeanFT test
- Everything is written in the IDE, Dev have
access to tests (and used them for sanity)
- Test code is completely reusable among the
whole test suite
- The test is authored once, and can be run on
all three browsers
- In the test setup, the TruClient launcher –
written in WPF - is started and being used,
and during the test, web and standard
windows technologies are tested.
38. Why?
– Cross Technology support (Desktop & Web)
– Script once – run of all browsers
– Re-use, share testing assets with Dev
38
39. Want to buy a new automation framework for 1 shekel??
39
54. FIST Selenium built FWK LeanFT UFT
IDE X √ √ X
Cross
Browsers/Technology
X Mobile/Web √ √√√
Model the UI X X* √ √
REST API testing √ X* X* √
DB Layer √ X* X* √
Combine UI/API/DB X X* X* √
Parsers √ X* X* √
Parameterization √ √ √ √
CI/CD X √ √ √
Modularity for less-tech X X √ √
Cross Platforms X √ X X
Lightweight X √ √ X
54= can be added by user
General
REST framework written in the same language as the developers (preferable as part of a UI framework)
Infrastructure that supports all REST functionalities
Validators for REST responses
Logs
Error handling
JSON handling
Support different browsers (UI framework)
Modular framework
Parameterization of tests
Backend tests:
• Grunt module to build the entire framework – according to CI/CD request
• REST Wrapper, extending functionality
• DB Layer – Postgresql
• Payload templates
• Logs
• JSON parser
• Expectation module
• Parameter mechanism
• Support for E2E Flows
Client testing:
Grunt module to build the entire framework – according to CI/CD request
Protractor wrapper – add more functionality to the driver API’s
DB Layer – Postgresql – to help in DB commination and functionality
UI Controls – map all the product elements and controls
Parameter mechanism
Reports
Multi browsers / OS
JS help function (XML, JSON, etc..)
General
REST framework written in the same language as the developers (preferable as part of a UI framework)
Infrastructure that supports all REST functionalities
Validators for REST responses
Logs
Error handling
JSON handling
Support different browsers (UI framework)
Modular framework
Parameterization of tests
Backend tests:
• Grunt module to build the entire framework – according to CI/CD request
• REST Wrapper, extending functionality
• DB Layer – Postgresql
• Payload templates
• Logs
• JSON parser
• Expectation module
• Parameter mechanism
• Support for E2E Flows
Client testing:
Grunt module to build the entire framework – according to CI/CD request
Protractor wrapper – add more functionality to the driver API’s
DB Layer – Postgresql – to help in DB commination and functionality
UI Controls – map all the product elements and controls
Parameter mechanism
Reports
Multi browsers / OS
JS help function (XML, JSON, etc..)
A new product
Startup mode
no UI automating investment
No DEV involvement in automation (asked for new public APIs from DEV for automation purposes)
“Why do I want to learn another tool?? I have my IDE”
A new product
Startup mode
no UI automating investment
No DEV involvement in automation (asked for new public APIs from DEV for automation purposes)
Extending unit/integration to include more cases
Enlrage the left and map to right side
The TruClient user interface is made up of the following sections:
TruClient Sidebar. The heart of the interface, the sidebar contains all the tools you need to develop your TruClient scripts.
TruClient Toolbox. The toolbox contains all of the steps that you can add to a TruClient script. The toolbox opens and closes by clicking on the tab, and moves by dragging it up or down.
Browser Navigation Bar. Enter the URL of the application for which you are developing a script.
Application Browser Window.The window containing the browser where you develop and replay your script interactively.
TruClient Sidebar Status Pane. A pane that displays status details about the active action in the TruClient Sidebar.
Extending unit/integration to include more cases
Came ACoE and wanted us to use their own automation framework
General
REST framework written in the same language as the developers (preferable as part of a UI framework)
Infrastructure that supports all REST functionalities
Validators for REST responses
Logs
Error handling
JSON handling
Support different browsers (UI framework)
Modular framework
Parameterization of tests
Backend tests:
• Grunt module to build the entire framework – according to CI/CD request
• REST Wrapper, extending functionality
• DB Layer – Postgresql
• Payload templates
• Logs
• JSON parser
• Expectation module
• Parameter mechanism
• Support for E2E Flows
Client testing:
Grunt module to build the entire framework – according to CI/CD request
Protractor wrapper – add more functionality to the driver API’s
DB Layer – Postgresql – to help in DB commination and functionality
UI Controls – map all the product elements and controls
Parameter mechanism
Reports
Multi browsers / OS
JS help function (XML, JSON, etc..)
General
REST framework written in the same language as the developers (preferable as part of a UI framework)
Infrastructure that supports all REST functionalities
Validators for REST responses
Logs
Error handling
JSON handling
Support different browsers (UI framework)
Modular framework
Parameterization of tests
Backend tests:
• Grunt module to build the entire framework – according to CI/CD request
• REST Wrapper, extending functionality
• DB Layer – Postgresql
• Payload templates
• Logs
• JSON parser
• Expectation module
• Parameter mechanism
• Support for E2E Flows
Client testing:
Grunt module to build the entire framework – according to CI/CD request
Protractor wrapper – add more functionality to the driver API’s
DB Layer – Postgresql – to help in DB commination and functionality
UI Controls – map all the product elements and controls
Parameter mechanism
Reports
Multi browsers / OS
JS help function (XML, JSON, etc..)
General
REST framework written in the same language as the developers (preferable as part of a UI framework)
Infrastructure that supports all REST functionalities
Validators for REST responses
Logs
Error handling
JSON handling
Support different browsers (UI framework)
Modular framework
Parameterization of tests
Backend tests:
• Grunt module to build the entire framework – according to CI/CD request
• REST Wrapper, extending functionality
• DB Layer – Postgresql
• Payload templates
• Logs
• JSON parser
• Expectation module
• Parameter mechanism
• Support for E2E Flows
Client testing:
Grunt module to build the entire framework – according to CI/CD request
Protractor wrapper – add more functionality to the driver API’s
DB Layer – Postgresql – to help in DB commination and functionality
UI Controls – map all the product elements and controls
Parameter mechanism
Reports
Multi browsers / OS
JS help function (XML, JSON, etc..)
General
REST framework written in the same language as the developers (preferable as part of a UI framework)
Infrastructure that supports all REST functionalities
Validators for REST responses
Logs
Error handling
JSON handling
Support different browsers (UI framework)
Modular framework
Parameterization of tests
Backend tests:
• Grunt module to build the entire framework – according to CI/CD request
• REST Wrapper, extending functionality
• DB Layer – Postgresql
• Payload templates
• Logs
• JSON parser
• Expectation module
• Parameter mechanism
• Support for E2E Flows
Client testing:
Grunt module to build the entire framework – according to CI/CD request
Protractor wrapper – add more functionality to the driver API’s
DB Layer – Postgresql – to help in DB commination and functionality
UI Controls – map all the product elements and controls
Parameter mechanism
Reports
Multi browsers / OS
JS help function (XML, JSON, etc..)
General
REST framework written in the same language as the developers (preferable as part of a UI framework)
Infrastructure that supports all REST functionalities
Validators for REST responses
Logs
Error handling
JSON handling
Support different browsers (UI framework)
Modular framework
Parameterization of tests
Backend tests:
• Grunt module to build the entire framework – according to CI/CD request
• REST Wrapper, extending functionality
• DB Layer – Postgresql
• Payload templates
• Logs
• JSON parser
• Expectation module
• Parameter mechanism
• Support for E2E Flows
Client testing:
Grunt module to build the entire framework – according to CI/CD request
Protractor wrapper – add more functionality to the driver API’s
DB Layer – Postgresql – to help in DB commination and functionality
UI Controls – map all the product elements and controls
Parameter mechanism
Reports
Multi browsers / OS
JS help function (XML, JSON, etc..)
General
REST framework written in the same language as the developers (preferable as part of a UI framework)
Infrastructure that supports all REST functionalities
Validators for REST responses
Logs
Error handling
JSON handling
Support different browsers (UI framework)
Modular framework
Parameterization of tests
Backend tests:
• Grunt module to build the entire framework – according to CI/CD request
• REST Wrapper, extending functionality
• DB Layer – Postgresql
• Payload templates
• Logs
• JSON parser
• Expectation module
• Parameter mechanism
• Support for E2E Flows
Client testing:
Grunt module to build the entire framework – according to CI/CD request
Protractor wrapper – add more functionality to the driver API’s
DB Layer – Postgresql – to help in DB commination and functionality
UI Controls – map all the product elements and controls
Parameter mechanism
Reports
Multi browsers / OS
JS help function (XML, JSON, etc..)
General
REST framework written in the same language as the developers (preferable as part of a UI framework)
Infrastructure that supports all REST functionalities
Validators for REST responses
Logs
Error handling
JSON handling
Support different browsers (UI framework)
Modular framework
Parameterization of tests
Backend tests:
• Grunt module to build the entire framework – according to CI/CD request
• REST Wrapper, extending functionality
• DB Layer – Postgresql
• Payload templates
• Logs
• JSON parser
• Expectation module
• Parameter mechanism
• Support for E2E Flows
Client testing:
Grunt module to build the entire framework – according to CI/CD request
Protractor wrapper – add more functionality to the driver API’s
DB Layer – Postgresql – to help in DB commination and functionality
UI Controls – map all the product elements and controls
Parameter mechanism
Reports
Multi browsers / OS
JS help function (XML, JSON, etc..)
General
REST framework written in the same language as the developers (preferable as part of a UI framework)
Infrastructure that supports all REST functionalities
Validators for REST responses
Logs
Error handling
JSON handling
Support different browsers (UI framework)
Modular framework
Parameterization of tests
Backend tests:
• Grunt module to build the entire framework – according to CI/CD request
• REST Wrapper, extending functionality
• DB Layer – Postgresql
• Payload templates
• Logs
• JSON parser
• Expectation module
• Parameter mechanism
• Support for E2E Flows
Client testing:
Grunt module to build the entire framework – according to CI/CD request
Protractor wrapper – add more functionality to the driver API’s
DB Layer – Postgresql – to help in DB commination and functionality
UI Controls – map all the product elements and controls
Parameter mechanism
Reports
Multi browsers / OS
JS help function (XML, JSON, etc..)
General
REST framework written in the same language as the developers (preferable as part of a UI framework)
Infrastructure that supports all REST functionalities
Validators for REST responses
Logs
Error handling
JSON handling
Support different browsers (UI framework)
Modular framework
Parameterization of tests
Backend tests:
• Grunt module to build the entire framework – according to CI/CD request
• REST Wrapper, extending functionality
• DB Layer – Postgresql
• Payload templates
• Logs
• JSON parser
• Expectation module
• Parameter mechanism
• Support for E2E Flows
Client testing:
Grunt module to build the entire framework – according to CI/CD request
Protractor wrapper – add more functionality to the driver API’s
DB Layer – Postgresql – to help in DB commination and functionality
UI Controls – map all the product elements and controls
Parameter mechanism
Reports
Multi browsers / OS
JS help function (XML, JSON, etc..)
General
REST framework written in the same language as the developers (preferable as part of a UI framework)
Infrastructure that supports all REST functionalities
Validators for REST responses
Logs
Error handling
JSON handling
Support different browsers (UI framework)
Modular framework
Parameterization of tests
Backend tests:
• Grunt module to build the entire framework – according to CI/CD request
• REST Wrapper, extending functionality
• DB Layer – Postgresql
• Payload templates
• Logs
• JSON parser
• Expectation module
• Parameter mechanism
• Support for E2E Flows
Client testing:
Grunt module to build the entire framework – according to CI/CD request
Protractor wrapper – add more functionality to the driver API’s
DB Layer – Postgresql – to help in DB commination and functionality
UI Controls – map all the product elements and controls
Parameter mechanism
Reports
Multi browsers / OS
JS help function (XML, JSON, etc..)
General
REST framework written in the same language as the developers (preferable as part of a UI framework)
Infrastructure that supports all REST functionalities
Validators for REST responses
Logs
Error handling
JSON handling
Support different browsers (UI framework)
Modular framework
Parameterization of tests
Backend tests:
• Grunt module to build the entire framework – according to CI/CD request
• REST Wrapper, extending functionality
• DB Layer – Postgresql
• Payload templates
• Logs
• JSON parser
• Expectation module
• Parameter mechanism
• Support for E2E Flows
Client testing:
Grunt module to build the entire framework – according to CI/CD request
Protractor wrapper – add more functionality to the driver API’s
DB Layer – Postgresql – to help in DB commination and functionality
UI Controls – map all the product elements and controls
Parameter mechanism
Reports
Multi browsers / OS
JS help function (XML, JSON, etc..)
General
REST framework written in the same language as the developers (preferable as part of a UI framework)
Infrastructure that supports all REST functionalities
Validators for REST responses
Logs
Error handling
JSON handling
Support different browsers (UI framework)
Modular framework
Parameterization of tests
Backend tests:
• Grunt module to build the entire framework – according to CI/CD request
• REST Wrapper, extending functionality
• DB Layer – Postgresql
• Payload templates
• Logs
• JSON parser
• Expectation module
• Parameter mechanism
• Support for E2E Flows
Client testing:
Grunt module to build the entire framework – according to CI/CD request
Protractor wrapper – add more functionality to the driver API’s
DB Layer – Postgresql – to help in DB commination and functionality
UI Controls – map all the product elements and controls
Parameter mechanism
Reports
Multi browsers / OS
JS help function (XML, JSON, etc..)
General
REST framework written in the same language as the developers (preferable as part of a UI framework)
Infrastructure that supports all REST functionalities
Validators for REST responses
Logs
Error handling
JSON handling
Support different browsers (UI framework)
Modular framework
Parameterization of tests
Backend tests:
• Grunt module to build the entire framework – according to CI/CD request
• REST Wrapper, extending functionality
• DB Layer – Postgresql
• Payload templates
• Logs
• JSON parser
• Expectation module
• Parameter mechanism
• Support for E2E Flows
Client testing:
Grunt module to build the entire framework – according to CI/CD request
Protractor wrapper – add more functionality to the driver API’s
DB Layer – Postgresql – to help in DB commination and functionality
UI Controls – map all the product elements and controls
Parameter mechanism
Reports
Multi browsers / OS
JS help function (XML, JSON, etc..)
General
REST framework written in the same language as the developers (preferable as part of a UI framework)
Infrastructure that supports all REST functionalities
Validators for REST responses
Logs
Error handling
JSON handling
Support different browsers (UI framework)
Modular framework
Parameterization of tests
Backend tests:
• Grunt module to build the entire framework – according to CI/CD request
• REST Wrapper, extending functionality
• DB Layer – Postgresql
• Payload templates
• Logs
• JSON parser
• Expectation module
• Parameter mechanism
• Support for E2E Flows
Client testing:
Grunt module to build the entire framework – according to CI/CD request
Protractor wrapper – add more functionality to the driver API’s
DB Layer – Postgresql – to help in DB commination and functionality
UI Controls – map all the product elements and controls
Parameter mechanism
Reports
Multi browsers / OS
JS help function (XML, JSON, etc..)
General
REST framework written in the same language as the developers (preferable as part of a UI framework)
Infrastructure that supports all REST functionalities
Validators for REST responses
Logs
Error handling
JSON handling
Support different browsers (UI framework)
Modular framework
Parameterization of tests
Backend tests:
• Grunt module to build the entire framework – according to CI/CD request
• REST Wrapper, extending functionality
• DB Layer – Postgresql
• Payload templates
• Logs
• JSON parser
• Expectation module
• Parameter mechanism
• Support for E2E Flows
Client testing:
Grunt module to build the entire framework – according to CI/CD request
Protractor wrapper – add more functionality to the driver API’s
DB Layer – Postgresql – to help in DB commination and functionality
UI Controls – map all the product elements and controls
Parameter mechanism
Reports
Multi browsers / OS
JS help function (XML, JSON, etc..)