This document discusses moving test automation beyond just the user interface layer. Conventional UI-based automation is limited because it cannot test components early in the development process and is vulnerable to frequent changes. The document recommends exploring service-based and API-level testing to allow for more holistic and efficient testing. This enables testing business logic independently of the UI for better coverage and faster feedback in agile development cycles. Case studies show how non-UI automation approaches improved test coverage and reduced defect detection time.

1. Rethinking Test Automation:
The Case for Moving Beyond the
User Interface
Rapid development models are forcing quality teams to
balance speed with coverage. To enable both effective
and efficient testing in this environment, businesses need
to replace conventional UI-based automation techniques
with more holistic approaches.
Executive Summary
With digital disruption and a fiercely competi-
tive business landscape, many organizations are
forcedtobalancefrequentsoftwareupgradesand
new feature additions with the ability to support
all possible platforms and accelerate time-to-
market.1
As pressures mount, many businesses
are adopting lean development methods that
offer complete lifecycle automation, from envi-
ronment setup through build and deployment.
These efforts make systems more complex,
spawning a higher incidence rate of product
quality issues due to insufficient testing.2
Conventional user interface (UI)-based
automation approaches cannot handle the needs
of these rapid development models, as they are
unable to test components or subsystems from
the get-go that contain business logic beyond
the UI. For speed and coverage, it is important
to look beyond automating the UI layer. This
whitepaper focuses on unconventional methods
and techniques that enable more effective and
efficient test automation, early on in the process.
Limitations of UI-based Automation
Focusing test automation on functional behavior
at the application UI is an easier option for test
strategists, given the abundance of commercial
and open source tools and implementation
frameworks. In our experience, however, limiting
testing to the UI layer may be inadequate due to
the following factors:
• Dependence on UI readiness: In most applica-
tions, the UI is delivered later in the lifecycle.
Waiting for the UI to be ready would mean
introducing automation at a much later point
in the application lifecycle. Minor UI changes
almost always prompt object definitions to be
updated frequently, leading to limited reuse
and defeating the very purpose of automation.
• Incomplete strategy: For complex applica-
tions, in which business logic resides in multiple
tiers, database objects and other components,
it can be a challenge to cover all application
controls and subsequent code interactions.
cognizant 20-20 insights | august 2016
• Cognizant 20-20 Insights

2. cognizant 20-20 insights 2
• Lack of focus on the business layer: Some
components, or a subsystems, containing
business logic may need to be tested indepen-
dently from the UI-based tests.
• The need for faster turnaround: In projects
based on Agile practices and projects that
follow weekly/biweekly release cycles, fixing
and turning around automated tests is chal-
lenging due to shorter testing windows. It may
be time-consuming to fix multiple test failures
due to UI changes.
Automation beyond the UI is immune to UI
changes, resulting in reduced script maintenance
efforts. This approach also enables testing of
complicated business rules or validating compo-
nent-level behavior independent of the UI, thus
ensuring a complete testing strategy. Usually,
testing at the middle tier or data level is faster
than UI tests and provides faster feedback on
application behavior. This also acts as an enabler
for early testing in upstream environments.
Exploring Alternative Test Automation
Approaches
The conventional strategy is to test end-to-end
(i.e., as an end-user would do when using the
application UI manually) or, more efficiently,
with UI automation. Applications with multi-
tiered architectures communicate using APIs,
Web services and messaging middleware via the
enterprise messaging bus. The UI (Web browsers,
client applications) takes care of end-user inter-
actions and presentation.
However, most business rules and functional com-
plexities typically reside in the middle tier (see
Figure 1). This creates an opportunity to validate
business rules and functionality, enhancing
coverage at the middle tier. Since communication
between the interacting components, or interfac-
ing layers, happens through message transac-
tions, this creates an opportunity to:
• Simulate component or application behav-
ior: Input messages directly into a system,
thereby simulating end-user action and reduc-
ing UI dependency.
• Launch processes: Make a call to trigger API
processing data or rules or, in turn, calling
another API.
• Validate business functionality: Make a call
to a service or API, parse messages and verify
message content and/or querying status in the
database for additional confirmation.
We propose the following solution:
• Analyze the technical stack: Study the appli-
cation platform, services and message types,
protocols and message formats, API available
for interactions and database available for
conducting additional validations.
A Typical Multi-Tiered Application Architecture
Message
Transactions
Message
Transactions
Message
Transactions
Message
Transactions
Interfaces Interfaces Interfaces Interfaces Interfaces
Database Servers
Legacy Applications
& Mainframes
External
Applications
E-mail Servers JMS. IBM MQ, etc.
Message
Transactions
DatabaseUser Interface
Client Systems,
Devices & Users
Message
Transactions
Middle
Tier
Business Rules Layer
Logic Layer
Load Balancer
ENTERPRISE SERVICE LAYER
Figure 1

3. • Identify tests for non-UI automation: Ana-
lyze the testing flow to identify the functional-
ities, components or flows that can be validat-
ed using the API or services. Also, identify the
inputs for the tests and output to be validated.
Any dependencies, such as test data and other
third-party services, should be marked for con-
sideration in the design phase.
• Evaluate design and tool fitment: Commer-
cial tools, such as CA Technologies’ DevTest,
IBM’s Rational Integration Tester, SmartBear
Software’s SoapUI NG Pro, etc., may be evalu-
ated for fit from a technology and design per-
spective. This approach should be governed
by the costs involved; while these tools are
feature-rich, it can be more economical to de-
velop a client in Java or native technology. The
objective should be to build a scalable solution
to support all service or API testing needs
within the portfolio.
• Develop solution constituents: Using a
framework with reusable components that has
multi-protocol and communication support
can help reduce redundancies, as well as en-
able socialization and know-how. Building an
architectural prototype will help establish the
concept and encourage stakeholder feedback.
The solution design should provide for
segregated test data to facilitate data changes
during releases. The solution should run tests
individually or in a batch, with single-command
execution, potentially run in a continuous inte-
gration3
pipeline, to ensure DevOps4
readiness
in the development environment (duly
supported by stubs or virtualization of services
or third-party application behavior, in certain
cases). This way, testing is automated using the
interface layer rather than the UI tier, which is
much slower.5
cognizant 20-20 insights 3
Quick Take
We helped a large investment bank improve
its approach to testing and speed defect
management. The bank’s cash equities portfolio
consisted of regional order entry applications, a
fix engine, order crossing and other applications
that communicated with order management
systems and trade reporting applications using
a custom protocol based on TCP/IP. The appli-
cations followed weekly or fortnightly release
cycles.
Although the bank employed test automation
through the UI, a tighter maintenance window for
automation scripts resulted in the need for more
timely feedback. A custom server-side automation
solution was developed to simulate trade entry,
replay fix messages and check trade status in the
order management system, as well as verify trade
reporting status across various suites. A single
test case could be executed in a few seconds that
enabled the execution of thousands of tests within
a few hours. This resulted in a drastic reduction
of cycle time across the portfolio, as well as the
ability to run thousands of tests on demand.
More than 60,000 tests a year could be run to
certify the quality of software through server-
side and UI automation. For applications with
mature development processes, the automated
suites were integrated in a continuous integra-
tion pipeline, providing continuous feedback on
quality and reducing the defect detection and
defect fixing lifecycle.
Automating Testing at a Large Investment Bank
3cognizant 20-20 insights

4. cognizant 20-20 insights 4
Potential benefits of our approach include:
• Greater requirement stability and traceabil-
ity: Tests break less frequently as interfaces
are well-defined and business process inputs
and outputs undergo less change and deliver
stable responses, hence bringing stability to
the automation bed. These characteristics
make the automation robust and ready to shift
left6
(test execution) from the testing phase to
the development phase of the SDLC.
• Early and faster automation via parallel
and multiple testing: Techniques such as or-
thogonal array and combinatorial testing can
be achieved only if tests can be performed be-
yond the UI. This is because UI-level validation
prevents the code coverage from being exer-
cised on edge conditions.
• Rapid development support: The feedback
obtained from the API or service tests can be
used to fine-tune the functional behavior at
the UI layer, ensuring a better customer expe-
rience. These tests are well-suited for Agile/
DevOps orchestration because each test runs
in a few seconds, and is not susceptible to UI
changes. Hence, it works well in continuous
integration execution mode and can be made
ready for DevOps.
Quick Take
A large commercial bank in the U.S. sought
to automate its trade order management and
settlement systems.
The upstream applications book and manage
trades throughout the lifecycle. The trades are
then routed using RESTful Web services to an
application server, and then to a Temenos banking
application. In addition to existing UI-based
automation with HP Unified Functional Testing, we
recommended using service-based automation
for faster feedback and setup of a continuous
feedback mechanism using a continuous integra-
tion methodology (see Figure 2).
A Java-based solution was used in the services
layer to inject trades into the booking applica-
tion and further verify the state of the trade
throughout the lifecycle and its final status in
the trade database. Approximately 100 tests
are run in roughly 45 minutes and provide quick
feedback for different trade types. The entire
test cycle span was reduced to one day for both
service and UI tests. The automated test could be
readily integrated with TeamCity for continuous
integration.
Saving Test Time and Effort for a Large Commercial Bank
Beyond UI: Continuous Testing Across the Lifecycle
CYCLE TIME
UI-based automation
using HP Unified
Functional Testing
API/SERVICES/MESSAGES
Automation introduced
late in lifecycle
Service-based automation introduced early in lifecycle
Downstream
Communication
Upstream
Communication
Upstream
Communication
Downstream
Communication
Validation of Transactions &
Message Communication
USER INTERFACE
Trade Order
Management System
Trade Settlement
System
Database
DOWNSTREAM APPLICATIONS
Figure 2

5. cognizant 20-20 insights 5
Looking Forward
Testers usually employ black box testing
techniques, with a range of input conditions,
using the UI and verifying outputs. This leaves
the core business logic residing in the API layer
to be tested by developers, many of whom
are not well schooled in the art and science of
quality assurance.
The right approach for maximizing test coverage
is to create a balance between UI and services
or API automation. This approach provides more
timely feedback and an ability to test early in the
development process, whereas UI automation
aims to validate logic in the UI layer and integra-
tion layer and mimics the actual end user of the
system.
Tests at the non-UI layer will also increase code
covered at the branch level, providing an option
to test more edge conditions. Our experience has
been that automation coverage can comprise
service and server-side/API automation to a range
of 60% to 70%, and UI automation to a range of
30% to 40%. However, a measured approach is
to implement impact-based testing techniques
that measure and map tests that cover all appli-
cation code and suggest gaps where lines of code
are not covered.7
With the adoption of cloud and the advent of
microservices architecture,8
API and services-
based testing becomes inevitable. QA organiza-
tions need to ascertain the quality of components
early and independently in the development
lifecycle before releasing them for further con-
sumption by related components or subsystems.
Footnotes
1 “Modernize Application Development to Succeed as a Digital Business,” Gartner, Inc, March 2016, https://
www.gartner.com/doc/3270018/modernize-application-development-succeed-digital.
2 Donald Firesmith, “Common Testing Problems: Pitfalls to Prevent and Mitigate,” Carnegie Mellon University,
April 2013, https://insights.sei.cmu.edu/sei_blog/2013/04/common-testing-problems-pitfalls-to-prevent-
and-mitigate.html.
3 “Continuous Integration,” Wikipedia, https://en.wikipedia.org/wiki/Continuous_integration.
4 “DevOps,” Gartner, http://www.gartner.com/it-glossary/devops/.
5 Brandon Getty, “4 Advantages of API Testing,” QASource, August, 2014, https://blog.qasource.com/4-
advantages-of-api-testing/. The author elaborates the time effectiveness of automated API tests over con-
ventional graphical user interface-based automation.
6 “Shift left” is an approach to software testing in which testing occurs early (to find defects early) in the
SDLC lifecycle as opposed to limiting testing to just the testing phase of the SDLC. For more information,
see https://insights.sei.cmu.edu/sei_blog/2015/03/four-types-of-shift-left-testing.html.
7 Steve Cornett, “What Is Wrong with Statement Coverage,” Bullseye Testing Technology, http://www.
bullseye.com/statementCoverage.html. The author explains the risks and misconceptions of a commonly
used code coverage metric. Also see “A Brief Discussion of Code Coverage Types,” Jason Rudolph’s blog,
June 10, 2008, http://jasonrudolph.com/blog/2008/06/10/a-brief-discussion-of-code-coverage-types/.
8 Microservices architecture is an approach to developing software and enterprise applications with the
ability to scale, covering a range of applications (Web, mobile) on multiple devices and platforms running in
a cloud or hybrid ecosystem, and moving away from simple, monolithic Web applications. For more infor-
mation, see https://en.wikipedia.org/wiki/Microservices and Mark Russinovich, “Microservices: An Appli-
cation Revolution Powered by the Cloud,” Microsoft Azure, March 17, 2016, https://azure.microsoft.com/
en-us/blog/microservices-an-application-revolution-powered-by-the-cloud/.

6. About Cognizant
Cognizant (NASDAQ: CTSH) is a leading provider of information technology, consulting, and business
process services, dedicated to helping the world’s leading companies build stronger businesses. Head-
quartered in Teaneck, New Jersey (U.S.), Cognizant combines a passion for client satisfaction, technol-
ogy innovation, deep industry and business process expertise, and a global, collaborative workforce that
embodies the future of work. With over 100 development and delivery centers worldwide and approxi-
mately 233,000 employees as of March 31, 2016, Cognizant is a member of the NASDAQ-100, the S&P
500, the Forbes Global 2000, and the Fortune 500 and is ranked among the top performing and fastest
growing companies in the world. Visit us online at www.cognizant.com or follow us on Twitter: Cognizant.
World Headquarters
500 Frank W. Burr Blvd.
Teaneck, NJ 07666 USA
Phone: +1 201 801 0233
Fax: +1 201 801 0243
Toll Free: +1 888 937 3277
Email: inquiry@cognizant.com
European Headquarters
1 Kingdom Street
Paddington Central
London W2 6BD
Phone: +44 (0) 20 7297 7600
Fax: +44 (0) 20 7121 0102
Email: infouk@cognizant.com
India Operations Headquarters
#5/535, Old Mahabalipuram Road
Okkiyam Pettai, Thoraipakkam
Chennai, 600 096 India
Phone: +91 (0) 44 4209 6000
Fax: +91 (0) 44 4209 6060
Email: inquiryindia@cognizant.com
­­© Copyright 2016, Cognizant. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by any
means, electronic, mechanical, photocopying, recording, or otherwise, without the express written permission from Cognizant. The information contained herein is
subject to change without notice. All other trademarks mentioned herein are the property of their respective owners.
About the Author
Nandan Shinde is Associate Director of Projects with Cognizant’s Quality Engineering and Assurance
business unit and is part of the company’s Technology Center of Excellence. Currently a test automation
architect, he previously worked as a test automation expert, project lead, automation strategist and
program manager over a period of 11 years with Cognizant’s banking and finance services clients. His
responsibilities pivot around building and socializing automation frameworks, as well as implementing
solutions – from front-end, middle-tier and data automation strategies – to leading automation program
for large banking customers. He handles automation consulting engagements across the vertical
industries that Cognizant serves. He can be reached at Nandan.Shinde@cognizant.com.
Codex 2192