Software Testing Tools Overview

Software Testing Tools
An overview
As per OU Syllabus
Dr Ganesh Neelakanta Iyer
Anad Adike

© 2015 Progress Software Corporation. All rights reserved.3
Manual vs Automated testing

Manual Testing vs Automated Testing
 Manual software testing
• performed by a human sitting in front of a computer carefully going through application
screens, trying various usage and input combinations, comparing the results to the expected
behavior and recording their observations
• Manual tests are repeated often during development cycles for source code changes and
other situations like multiple operating environments and hardware configurations
 Automation Testing (using a tool)
• An automated testing tool is able to playback pre-recorded and predefined actions, compare
the results to the expected behavior and report the success or failure of these manual tests
to a test engineer
• Once automated tests are created they can easily be repeated and they can be extended to
perform tasks impossible with manual testing
• Because of this, savvy managers have found that automated software testing is an essential
component of successful development projects

Automation
 Functional Automation (Black box testing)
• Windows applications – Winrunner, QTP
• Web applications – Silk Test, Selenium, Telerik Test Studio
• Regression Testing – Same set of tools
 Performance/Load Testing
• Load Runner, Jmeter
 Security Testing (black box and White box)
• Veracode, ZAP

Functional Testing
Overview of WinRunner, TSL, SilkTest
Demo of Selenium

WinRunner

WinRunner
 Designed to help customers save testing time and effort by automating the manual
testing process
 Manual process
• Perform operations by hand,
• visually check results, and
• log results by hand
 Automated process:
• Create a test script that will perform the same operations as a Human operator
• check the same information, and
• create a summary report showing the test status

Recording Modes
• Two Modes
• Context-sensitive mode
• Analog mode
• Tests can combine both
recording modes
• Context-Sensitive is the default
mode
• Switch between modes using
same record key (F2)

Context-Sensitive Mode
• Object-based
• Unaffected by minor UI changes
• Maintainable (readable/editable)
• Generally used with GUI applications
• Portable script

Context-Sensitive Mode
set_window ("Save As");
edit_set(“File Name”,“output14”);
output14
Set focus to the window
Set the edit field content
Button press
button_press(“OK”);

Analog Mode
• Position-dependent
• Works with any application
• UI changes force test script changes
• Usually drives tests with
• mouse,
• keyboard and
• other such manual user inputs
• Less maintainable
y
x

Analog Mode
move_locator_track (1);
mouse drag
mtype (" <T55> <kLeft>-<kLeft>+"); mouse click
output14
type (" <t3>output14" );
keyboard
move_locator_track (2);
mtype (" <T35><kLeft>-<kLeft>+ ");
timing

Recording Modes
 Context-Sensitive mode statements can be recorded or
programmed
• record: button_press, win_activate
• program: list_get_num_items, edit_get_text
• recommended for most situations due to greater robustness
 Analog mode statements are rarely programmed, mostly
recorded and edited
• record: move_locator, type, mtype
• program: move_locator_abs, move_locator_rel, click
• Analog statements are useful for literally describing the keyboard, mouse, and mouse button
input of the user

Run Modes
 Debug
• debug is good to use while the test script is being debugged
• these test results are overwritten with each new run
 Verify
• Corresponds to actual results
• Generally used when executing testing sessions where results need to be stored
 Update
• Corresponds to “expected” results. Expected results are the benchmarks used to
verify test results
• Test runs in “Update” mode generate the expected results for future runs to
compare back against
• These test results become the expected results for subsequent test runs in “Verify”
mode

Synchronization
 Enhances a test script to ensure reliable replay
• accounts for delays in order to prevent the automated script from running faster than the
tested application
• critical for successful test automation implementation
• among the main reasons why record-n-playback is not reliable
 In Context-Sensitive mode
• Examples: (operations)
– wait for a window to appear
– wait for a bitmap to refresh
– wait for an object property
– wait for a specific amount of time
 In Analog mode
• Examples: (operations)
– wait for a window bitmap to appear / refresh
– wait for a specific amount of time

Window Synchronization
invoke_application(“Notepad”,””, “c:temp”, SW_SHOW);
set_window (”Login”, 10);
edit_set(“User ID:”, “guest”);
edit_set(“Password:”, “mercury”);
button_press(“OK”);set_window
Waits for the specified window to appear onscreen. If the
window appears before the timeout, the script immediately
proceeds to the next line.

Synchronization Controls

Test Script Language

TSL (Test Script Language)
 TSL is a C-like language
• High-level proprietary programming language designed for test automation
• procedural language
 Full programming support
• variables, arrays, functions
• regular expressions
• control flow, decision logic, looping

Built-in TSL Functions
 TSL provides a comprehensive library of hundreds of built-in functions to
simplify test creation
• window/object functions
• environment functions
• reporting functions
• database query functions
• file/spreadsheet functions
• Win32 functions
 WinRunner provides a graphical function browser to assist you
• Function Generator

Language Syntax
*** Same syntax as in standard C ***

Variables
 Basic Rules
• do not need to be declared / defined
• specific data types are not explicitly
defined
• case sensitive
• first letter must be a character or
underscore
• cannot be a reserved word
• by default all variables are local
(static)
• can also be public and/or const
 Arrays
• single dimension: cust[1], cust[2],
cust[3]
• multi-dimension: address[1,1],
address[1,2]
• Can be indexed with number
– address[1], address[2]
• Can be indexed with strings
(associative)
– address[“John”], address[“Mary”]

Operators
 Math
+ - * / ^ % ++ --
 Logical
&& || !
 Relational
== != >= <= > <
 Assignment
= += -= *= /= ^= %=
 Concatenation
&

Test Verification
 Enhancing a test script to verify data onscreen
• check objects’ values / states
• check images
• check text
• check the database
 Context-Sensitive verification
 Analog verification

Exception/Recovery Handling (*)
 Unexpected errors during replay
• unlike error-handling, these can appear at any time when running a script
 WinRunner provides a mechanism to trap and handle
exceptions
• popup exceptions
– popup windows
• object exceptions
– object property value changes
• TSL exceptions
– TSL error codes

Functions and Libraries
 simplifies building test frameworks
• application-specific functions
• general-purpose functions
• greater modularity
 can be stored in a script
• compiled module (function library)
• can be loaded as part of startup or initialization script and available globally
 facilitates data-driven testing
• data-driven testing is where data retrieved externally from the test being executed drives
the test rather than using hard-coded data within each test case. Using application
specific custom functions and scripts helps further the benefits of data-driven testing.

Architecture and use of Silk Test

What is SilkTest ?
 A powerful tool for running automated test cases on the front end
• A tool for testing Web based applications across different browsers
• Very very stupid – you have to tell it everything in its own language
(4test)
• Inflexible when it comes to interpreting your commands
• It cannot guess at what you mean
• It requires a certain syntax with certain words
– This can be quite maddening

What can you do with it ?
 Mostly anything you can do in a Web browser, you can do with
SilkTest
• Enter text in text fields
• Click buttons
• Check boxes
• Get files
• Clear your cache
• etc.

Using a browser
 Though SilkTest is its own product, you still have to use a browser to access your
application
 You have to tell SilkTest what kind of browser you are going to use for your testing so
that it knows what it is looking at.
 However, you can only run SilkTest on one instance of a browser at a time
• One IE browser going at a time

Architecture
 Project .vtp (for maintenance)
 Testplan .pln (organizing the tests)
 Test Frame* .inc (AUT Window details)
 Scripts .t (test scripts)
 Data driven tests .g.t (scripts with many data)
 Include file .inc
 Suite .s (for batch runs)
 Result .res

Test Process
 Manage requirements using SilkCentral Test Manager
 Manage test plans and cases using Silk test
 Associate test plans/cases to requirements using test
manager
 Declare GUI declarations using SilkTest
 Develop scripts for atomic activities first so that they can be
reused later
 Develop scripts using SilkTest and run them
 Track defects using Defect Tracker

Selenium

Overview of Selenium
 Selenium is a robust set of tools that supports rapid development of test automation for
web-based applications.
 Selenium provides a rich set of testing functions specifically geared to the needs of
testing of a web application.
 Selenium operations are highly flexible, allowing many options for locating UI elements
and comparing expected test results against actual application behavior.
 Supports Cross Browser Testing. The Selenium tests can be run on multiple browsers.
 Allows scripting in several languages like Java, C#, PHP and Python.
 Assertion statements provide an efficient way of comparing expected and actual
results.
 Inbuilt reporting mechanism.

Demo

Performance and Load Testing
Overview of LoadRunner
Demo of JMeter

Why Load Test An Application
 Does the application respond quickly enough for the intended users?
 Will the application handle the expected user load and beyond?
 Will the application handle the number of transactions required by the
business?
 Is the application stable under expected and unexpected user loads?

Functional vs. Load Web Testing
Functional test
Load
test
Do business processes
function properly after
implementation?
Functionality
EXAMPLEOBJECTIVE
Will 2,000 concurrent hits
crash the server?
Stability
Is response time acceptable
according to specifications?
Performance
Do business processes
function properly under
heavy load?
Functionality
under load
EXAMPLEOBJECTIVE

Types of Performance Testing
Find out whether the system can handle the expected load upon
deployment under real-world conditions.
Load
Testing
Find the application's breaking point. Apply testing that measures whether
the application's environment is properly configured to handle expected or
potentially unexpected high transaction volumes.
Stress
Testing
Find the stability of the system with respect to handling large amounts
of data over extended time periods.
Volume Testing
Find the behavior and performance of each tier.
Component
Testing

Objectives of Performance Testing
How long does it take to complete a task?Application Response Time
Reliability
Configuration Sizing
Capacity Planning
Acceptance
Bottleneck Identification
Regression
Product Evaluation
How Stable is the system under a heavy work load?
Which configuration provides the best performance
level?
At what point does degradation in performance
occur?
Is the system stable enough to go into Production?
What is the cause of degradation in performance?
Does the new version of Software adversely affect
response time?
What is the best server for 100 users?

Testers
Load Generation System Under Test
Do you have the testing resources?
• Testing personnel
• Client machines
How do you synchronize users?
How do you collect and analyze results?
How do you achieve test repeatability?
Analysis?
All of you, click
the GO button
again
Manual Testing Is Problematic
Web server Database
server
Coordinator

Use of LoadRunner

System Under Test
The LoadRunner Solution
Web server Database
server
Load Generation
Vuser
host
Overcomes resource limitations
• Meaningful results with analysis tools
• Repeats tests with scripted actions
AnalysisController
• Runs many Vusers on few machines
• Replaces testers with “Virtual Users”
• Controller manages the Vusers

Load Testing Tools Available
 LoadRunner : HP (Formerly Mercury Interactive)
 e-Load :Emprix
 Silk Performer : Borland (Seague)
 QALoad : Compuware
 Rational Performance Tester : IBM Rational
 Web Load : Radview
 Neo Load : Neotys
 Open STA : Open Source
 Jmeter
 Telerik Test Studio

Introduction to Load Runner
• Load Runner is a Mercury Interactive Tool that predicts performance and
behavior of the system
• By creating lots of load, you can see how the system reacts at peak
levels or with simultaneous Users
• To test the application, LoadRunner emulates an environment where
multiple users work concurrently. While the application is under load,
LoadRunner accurately measures and analyzes the system
performance, and its functionality

Supporting Environments
• Application Deployment Solution - The Citrix protocol.
• Client/Server - MS SQL, ODBC, Oracle Web Applications 11i and
DNS protocols.
• Custom - C templates, Visual Basic templates, Java templates,
Javascript, and VBScript
• Distributed Components - COM/DCOM, Corba-Java, and Rmi-Java
protocols.
• E-Business - FTP, LDAP, Palm, Web (HTTP/HTML), Web Services
• Enerprise Java Beans -EJB Testing and RMI-Java protocols.
• ERP/CRM - Baan, Oracle NCA, Peoplesoft 8, SAP-Web, SAPGUI

• Legacy
Terminal Emulation (RTE).
• Mailing Services
Internet Messaging (IMAP), MS Exchange (MAPI), POP3, and SMTP.
• Streaming
MediaPlayer and RealPlayer protocols.
• Wireless
i-Mode, VoiceXML, and WAP protocols.
Internet/
Intranet
Web ServersClients App. Servers Database
Server

50
• Platforms
• NT, 2000, XP
• Sun
• HP
• IBM
• Linux

The LoadRunner Solution
Virtual User Generator
Generates load
and collects test results
Creates Scripts as one Single User.
Compiles and displays
test results with
graphical and statistical tools
LoadRunner Controller
LoadRunner Analysis

LoadRunner Terminology
• Scenarios
• Using LoadRunner, you divide your application performance testing requirements into
scenarios.
• A scenario defines the events that occur during each testing sessions.
• For example, a scenario defines and controls the number of users to emulate, the
actions that they perform, and the machines on which they run their emulations.
• Vusers
• In a scenario, LoadRunner replaces human users with virtual users or Vusers.
• When you run a scenario, Vusers emulate the actions of human users—submitting
input to the server.
• A scenario can contain tens, hundreds, or even thousands of Vusers.

• Vuser Scripts
• The actions that a Vuser performs during the scenario are described in a Vuser
script.
• When you run a scenario, each Vuser executes a Vuser script. Vuser scripts
include functions that measure and record the performance of the server during
the scenario.
• Transactions
• To measure the performance of the server, you define transactions.
• Transactions measure the time that it takes for the server to respond to tasks
submitted by Vusers.

• Rendezvous Points
• You insert rendezvous points into Vuser scripts to emulate heavy user load on the server.
• Rendezvous points instruct multiple Vusers to perform tasks at exactly the same time.
• For example, to emulate peak load on the bank server, you insert a rendezvous point to
instruct 100 Vusers to simultaneously deposit cash into their accounts.
• Controller
• You use the LoadRunner Controller to manage and maintain your scenarios.
• Using the Controller, you control all the Vusers in a scenario from a single workstation.

• Hosts
• When you execute a scenario, the LoadRunner Controller distributes each Vuser in the
scenario to a host.
• The host is the machine that executes the Vuser script, enabling the Vuser to emulate
the actions of a human user.
• Performance Analysis
• Vuser scripts include functions that measure and record system performance during
load-testing sessions.
• During a scenario run, you can monitor the network and server resources.
• Following a scenario run, you can view performance analysis data in reports and
graphs.

LoadRunner Components
ControllerVuGen
Analysis
LoadRunner
Tuning

57
How LoadRunner Works ?

Tune
System Based
on Analysis
LoadRunner Expert Workflow
“The Big Picture”
Analyze
System Under Load
Phase 5
LoadRunner
V U G E N
LoadRunner
C O N T R O L L E R &
A N A L Y S I S
Run
Scenarios
Phase 4
Create
Scenarios
Phase 3
Create Web Virtual
Users
Phase 2Phase 1
Plan Load Test

Use of JMeter

Introduction to JMeter
 Definition
• JMeter is an Apache Jakarta project that can be used as a load testing tool for analyzing and
measuring the performance of a variety of services, with a focus on web applications.
 Why ?
• Free 
• JMeter can be used as a unit test tool for JDBC database connection, FTP, LDAP,
WebServices,J MS, HTTP and generic TCP connections.
• JMeter can also be configured as a monitor, although this is typically considered an ad-hoc
solution in lieu of advanced monitoring solutions.

Load test Process
 System Analysis
 Creating Virtual User Scripts
 Defining User Behavior
 Creating a Load Test Scenario
 Running Load Test Scenario
 Analyzing Results

System Analysis
 Enable customers to convert their goals and requirements into a
successful test script.
 you can:
-Create your own test Script consistently
- Careful UI design allows you to create your own test Easily

Creating Virtual User Scripts
- Tester must emulate the real user by driving the real application as a
client.
- JMeter support this by adding thread group element,
this tell JMeter:
- the number of users you want to simulate,
- how often the users send requests
- how many requests they send.
- What request (FTP Request, HTTP Request, and JDBC request).
- validate that your application is returning the results you expect.

Defining User Behavior
 JMeter allows you to simulate human actions more closely
- By controlling how long JMeter engine delays between each sample
- So, you can define the way that the script runs

Creating a Load Test Scenario
JMeter offers support for this step by:
 Assign scripts to individual virtual user
 Tester can define any number of virtual users needed to run the tests
 Allows user to simulate concurrent connections to server application.By
enables user to create multiple threads ( virtual user) executing different
test plan.
 Increasing the number of virtual users in a controlled fashion.

Running Load Test Scenario
 With JMeter you can run your tests in a very easy way

Analyzing Results
JMeter offers support for this step by:
 displaying the data visually (Graph Results) .
 save data in file.
 allows user to see one multiple views of the data
 displays the response from the server.
 shows the URL of each sample taken .
 listeners will show different sets of data.
 it can send email based on test results.

Analyzing Results
 Showing detailed performance results that can be easily understood and
analyzed to quickly pinpoint the root cause of problems.

Demo of JMeter

Souce code Testing Utilities
http://sqa.fyicenter.com/FAQ/Testing-
Tools/Other_Source_Code_Testing_Tools.html

Why source code testing?
 Code coverage
• To determine what proportion of your project's code is actually being tested by coded tests
such as unit tests
• code coverage is a measure used to describe the degree to which the source code of a
program is tested by a particular test suite.
• A program with high code coverage has been more thoroughly tested and has a lower
chance of containing software bugs than a program with low code coverage
 Enables in white box testing
• method of testing software that tests internal structures or workings of an application, as
opposed to its functionality (i.e. black-box testing)
• also known as clear box testing, glass box testing, transparent box testing, and
structural testing

Advantages of source code testing
 Enables to test better
 Enables to understand uncovered test scenarios
 Ensuring messages for various scenarios are correct
• Error messages
• Notifications

Source Code Testing Utilities
 Splint / LCLint (Lint - static source code analysis tool)Splint (formerly LCLint) is a
rewrite of the famous Unix lint utility. It is a static source code analyser, that checks
your C program source code for bugs and potential problems. This static source code
analysis tool requires a certain amount of configuration before you can use it
productively. There are versions for a number of operating systems, including Win32,
Linux and other Unix systems.
 Ftncheck (Lint utility / Static Source Code Analysis Utility for Fortran)ftncheck, a static
source code analysis utility, checks your Fortran programs for semantic errors, leaks
and potential bugs. For example, you can use ftncheck to scan your program for the
use of uninitialized variables and undeclared variables.
 Jlint: Java Program Checker (Lint)Like the lint utility which checks C programs, jlint is a
static source code analysis tool that scrutinizes your Java source code for suspicious
constructs that might indicate a program bug. Although the program is shareware, it is
distributed with source code without any restrictions (and hence is listed here).
 http://sqa.fyicenter.com/FAQ/Testing-Tools/Other_Source_Code_Testing_Tools.html

Future of Software Testing
and
Test Engineer

PAST

What does this mean from a Testing perspective?
 Seamless Integration
 Sky is the limit for a Software Test Engineer
• Internet of Things
• Big Data and Analytics
• Social
• Security and Privacy
• Open Hardware

Dimensions of a Quality Analyst
 Technical QA
 Consultant
 Analyst
 Coaching
 DevOps

Career Opportunities as Software Quality (Test) Engineer
 Every software needs to be QAed
 Familiarize with Popular tools
• Web UI Testing  Selenium
• Performance Testing  JMeter
• Security Testing  ZAP

ACM Membership Benefits
 Computing Information Resources:
 Learning Center with resources for lifelong learning
• Online courses targeted toward essential IT skills and popular certifications
• Online books from Books24x7®, Safari®, Morgan Kaufmann and Syngress
• Webinars on hot topics, presented by today's innovators
 The option to subscribe to the full ACM Digital Library, which includes over 2 million pages of text
 A full year (12 issues) electronic subscription to Communications of the ACM
 A full-year (4 issues) subscription to XRDS: Crossroads, ACM's Student Magazine
 Complimentary e-mentoring services from MentorNet
 A full-year electronic subscription to monthly MemberNet and quarterly Student Quick Takes, ACM's
member newsletters.
 A full-year subscription to TechNews, ACM's tri-weekly email IT news digest
 A full-year subscription to CareerNews, ACM's twice monthly email career news digest
 acmqueue website featuring articles, commentary, blogs, roundtables, case studies, mutlimedia and
more
 Access to Ubiquity, ACM's IT opinion magazine and forum, and eLearn, ACM's magazine dedicated
to distance education

Software Testing Tools Overview

Software Testing Tools Overview

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Software Testing Tools Overview

Similaire à Software Testing Tools Overview (20)

Plus de Dr Ganesh Iyer

Plus de Dr Ganesh Iyer (20)

Dernier

Dernier (20)

Software Testing Tools Overview