2. CONTENTS
Introduction .......................................................................................................................... 2
Why is Automated Testing Such a Challenge for SAP ERP? ................................................................... 2
SAP Manual Testing is Underserved .............................................................................................. 3
Deciding What to TEST: Cost vs. Risk ......................................................................................... 4
Creating Test Scenarios for Scripted Testing ............................................................................... 4
Unscripted “Exploratory” Testing ............................................................................................. 5
Capacity of Business Key Users ................................................................................................ 5
Documenting Tests Performed ................................................................................................. 5
Managing the Manual Testing Process ........................................................................................ 5
Optimizing the Manual Testing Process.......................................................................................... 6
Automated Test Scoping of SAP-Driven Changes ........................................................................... 6
Accelerated Creation of Test Scenarios ...................................................................................... 6
Accelerated Execution of Manual Testing ................................................................................... 6
Accelerated documentation of test runs..................................................................................... 7
Designed and Built for Business Users ........................................................................................ 7
Test Management ................................................................................................................ 7
-1-
3. INTRODUCTION
SAP ERP systems must evolve at the speed of business change. It is therefore imperative for SAP ERP managers
to adapt their SAP landscapes in a timely manner to address business needs such as process changes, rollouts to
new sites, and regulatory updates. In addition, they must cater to the maintenance needs of their SAP systems
and implement SAP support packs and enhancement package upgrades on a regular basis.
SAP ERP systems are highly complex, built on hundreds of millions of lines of standard code and customizations.
This means that even a small change to the system could trigger a series of inadvertent consequences. To
prevent production failures, companies regularly perform a series of functional test cycles – which can be
either manually or automatically executed.
Industry papers discussing approaches to test automation have been available for well over a decade. Little has
been discussed, however, about the applicability of test automation to SAP ERP. This document describes the
specific challenges of SAP ERP functional testing, when test automation is inappropriate, and how to perform
manual testing faster, better, and cheaper in these cases.
WHY IS AUTOMATED TESTING SUCH A CHALLENGE FOR SAP ERP?
A survey performed by Panaya in 2012 of 100 SAP-run enterprises revealed that even the most mature testing
organizations perform only 15% to 25% of their functional regression testing using test automation tools; the
remainder is manual. There are three main reasons, inherent to SAP ERP, which explain why test automation
coverage is relatively low:
(i)
(ii)
(iii)
SAP ERP functionality is heavily data driven. SAP business logic is dependent on and functions
differently with change in the input data; this translates to changes in screen flow, screen content
and screen format that must be factored into the test automation script.
For business processes that change frequently, the test automation scripts must be continually
kept up to date. A challenging and costly proposition.
Preparing the required test data for test automation is also challenging due to SAP’s complex data
structure and the various dependencies that exist in an SAP business process. Think about an
automatic test script for a purchase-to-pay process that requires five different material types for
full testing coverage. Maintaining integrity of the data required to support the end-to-end process
for the five material types is a daunting task.
-2-
4. SAP MANUAL TESTING IS UNDERSERVED
Since it’s virtually impossible for a company to develop automation test scripts to cover every possible SAP
business scenario, manual testing is heavily relied upon to ensure that all the needed functionality is covered.
Yet manual testing is severely underserved in terms of tools that increase productivity, enforce processes, and
provide control.
Performing manual tests requires an intimate understanding of the business processes to be tested, and so, it
is typically the functional experts in IT or key business users (sometimes referred to as super users) that are
tasked with performing manual testing. They do their best to test, report defects, report progress, and
document test runs, on top of their other daily duties. This is all performed manually, with tools such as Word
and Excel.
In organizations where a dedicated test manager is allocated to the SAP team, management of the testing
cycle is performed by the test manager. In the majority of SAP shops, however, there isn’t a dedicated test
manager; the test manager’s role is performed by project managers, in the scope of their respective projects,
as an added duty.
Test management includes defining the testing strategy, identifying the tests to be performed, assigning tests
to the manual testers, chasing the testers for their individual status reports and consolidating them to a “bigpicture” testing progress and coverage status, managing defects, ensuring that testers perform the testing on
time, and more! Approximately two thirds of SAP shops use general productivity software like spreadsheets in
support of these test management activities. It is little surprise, therefore, that they lack visibility and control
over the testing cycle, and put too much time and effort into managing the testing activities.
-3-
5. DECIDING WHAT TO TEST: COST VS. RISK
The first pain point comes early in the testing cycle: determining what, exactly, to test. Following a technical
or functional change in a massive application like SAP ERP, it’s a challenge to determine where exactly in the
system testing should be focused.
Customers have two options. One is to try to test every possible SAP transaction that may be affected to
minimize risk of production failures. That, of course, increases testing time and cost. The other alternative is
to essentially ‘guess’ where the greatest impact will be and test those areas. The second option comes with
lower costs but much more risk. In practice, most organizations seek a happy medium between the two
extremes, but the costly nature of manual testing makes many lean toward less testing.
Finding a method to test more functionality within a strict timeline is one of the biggest challenges associated
with implementing changes in the ERP application, so finding a cost effective resolution is extremely valuable.
CREATING TEST SCENARIOS FOR SCRIPTED TESTING
Test scenarios describe the steps needed to test a given function or process and the anticipated results of every
step. In theory, this should work well: each manual tester has a written script telling him or her what to do,
how to do it, and what results to expect. But in practice, this is far from the case.
Typically written in Word or Excel, test scripts contain varying levels of detail, with no standardization as to
what should and should not be included. It’s a time-consuming process, as each script can take from 30 minutes
to several hours to create. And there’s no guarantee that others will be able to reliably follow the script;
what’s clear to whoever wrote the script may not be to those charged with following it.
-4-
6. UNSCRIPTED “EXPLORATORY” TESTING
In some cases, test scenarios don’t exist at all. Instead, organizations practice a form of ad-hoc testing, known
as exploratory testing, where manual testers are tasked to perform a test as they see fit and report results.
There are two main use cases for exploratory testing:
1. The company doesn’t have pre-defined scripts and there is no time to create them. Therefore, manual
testers run a test cycle based on their knowledge of the way the application is used. While this reduces
test preparation time, it adds risk that the business process will not be tested adequately.
2. The QA team or functional testers create test scripts and conduct tests based on these scripts. At the
User Acceptance Test phase the test manager doesn’t want the key user to repeat the exact same
scripts, because they were successfully tested already. Furthermore, the test manager needs to ensure
the application is tested exactly as the business users will use it in their daily work, so he asks the user
to simply do their day-to-day tasks on the QA system and check that everything works properly. This
method provides an additional assurance for the quality of the application.
Problems may arise when exploratory testing uncovers issues that developers must address. With no record of
what, exactly, the tester did, it’s difficult for the developers to accurately recreate problems that need fixing.
What’s more, there’s no knowledge transfer of the testing process from the business user to IT to enable IT to
create test scenarios for future use.
CAPACITY OF BUSINESS KEY USERS
Testing takes the business key users away from their pressing day jobs and limits their ability to fulfil their
other duties, such as supporting end-users, delivering training, defining processes, or liaising with IT regarding
functional needs. These key users are typically among the most experienced and highest paid in the user
organization and have very little time to spare. Since the key users are forced to fit test activities around their
normal workload, testing often has to wait.
DOCUMENTING TESTS PERFORMED
At the same time, these time-strapped key users are often required to collect evidence to prove whether or
not these tests were successful. This is especially true for organizations subject to government regulations,
such as Sarbanes Oxley (SOX), U.S. Food and Drug Administration (FDA) rules, and others. Users must collect
detailed proof of what was tested, by whom, and how. It can be a painstaking, time-consuming process for
testers to constantly take screen shots, then copy and paste them into a document to show which tests were
performed. This process can take at least 20 to 30 minutes per test, which translates into several days of nonproductive work per test cycle per key user
MANAGING THE MANUAL TESTING PROCESS
Finally, when testing is manual, the testing manager has no real-time visibility into the process. He can
distribute detailed instructions to testers but has no way of knowing if, when, and how they are conducting the
tests. As a result, he can’t unequivocally know the level of progress being made and the precise issues which
have cropped up. When testers are located at multiple sites, the challenge becomes all the greater, forcing the
manager to resort to emails, spreadsheets and phone calls to keep tabs on testers.
-5-
7. OPTIMIZING THE MANUAL TESTING PROCESS
So it’s clear that manual testing will always be required; there’s simply no way around having experienced
users verify that the SAP system functions as it should. However, many aspects of the testing process can and
should be accelerated and managed. These include:
•
•
•
•
•
•
Determining what to test (test scoping)
Creating detailed test scenarios
Executing manual tests
Reporting defects that can be easily reproduced
Documenting test run evidence
Tracking test run progress
In short, manual testers require better tools to make the process more efficient, accurate – and less painful.
Testing managers need visibility and control over manual testing cycles. This visibility will enable organizations
to conduct more testing in the same time window and hence, find more problems before the code goes into
production. Increasing testing efficiency also has the effect of driving down the cost of testing and potentially
speeding up project implementation, thus increasing business agility.
SAP IT teams seeking a solution to accelerate the different facets of the manual testing process should
evaluate manual testing solutions on the basis of the following attributes:
1. AUTOMATED TEST SCOPING OF SAP-DRIVEN CHANGES
Determining which functionality of the SAP ERP to test following a major change – especially when
implementing SAP-driven changes, such as support packages, enhancement packages or business functions – is
crucial to minimizing the risk and cost of change. Analyzing the production usage of your ERP transactions
combined with an impact analysis of the changes on your system can yield a risk-based approach to test
prioritization that can reduce testing efforts by as much as 70%.
2. ACCELERATED CREATION OF TEST SCENARIOS
Writing test scenarios is a painstaking process but it’s also one that’s well suited to automation. Your testing
tool should capture the business process to be tested while a functional expert or key user navigates the ERP
application. The testing tool should translate the steps into structured human-readable test scenarios,
including documentation of expected results for every step of the scenario. You’re essentially capturing the
knowledge of that key user and making it available for all others to emulate in their own testing. At the same
time, you ensure the scripts are highly accurate – but take far less time to create.
3. ACCELERATED EXECUTION OF MANUAL TESTING
Similarly, your testing solution should ease the burden on testers by offloading many manual tasks associated
with testing, such as keying the right data into the transaction fields and comparing the results to those
expected, to ensure accuracy.
-6-
8. 4. ACCELERATED DOCUMENTATION OF TEST RUNS
Another aspect of test execution is the process of capturing and documenting test evidence for every test run.
Seek a testing solution that enables you to determine exactly which process the tester followed, what the
results were and how those results compare to what was expected. This is a crucial step in terms of providing
proof of testing for auditors, as well as tracing production failures to the gaps in your test scenarios. It also
relieves testers of the tedious process of manually documenting test results.
5. ACCELERATED REPORTING OF DEFECTS
The ability to effectively report functional failure is crucial in enabling IT address issues uncovered by testing.
Developers should get an accurate, screen-by-screen and keystroke-by-keystroke depiction of how the tester
reached a problem point, so they can recreate and fix the problem. This is exactly the kind of data that testers
struggle to capture and communicate accurately.
6. DESIGNED AND BUILT FOR BUSINESS USERS
Remember your key business users will ultimately have to use the testing tool as part of their daily routine.
They are already pressed for time; the last thing they want is to spend time learning how to use a testing tool
that they never wanted to use in the first place. So they need seamless access to the testing tool from the ERP
application. It should require minimal, if any, training to perform such testing activities as: viewing the
assigned test tasks, executing the test tasks, testing in collaboration with other testers to complete a full
business process, and reporting defects and progress. The tool should not require complex installation of clientside software, and must be intuitive and simple to use, without interfering with the non-testing tasks of the key
user.
7. TEST MANAGEMENT
Finally, your manual testing solution should provide you with ongoing visibility and governance over the end-toend testing process. You should be able to plan test projects, assign tasks to all manual testers, determine how
close the testers are to completing their assigned tasks, and receive alerts when problems arise or steps are
completed. The solution should also enable you to track a distributed multi-site testing operation.
8. SIMPLE SET-UP, MINIMAL ONGOING MAINTENANCE
You need a solution that doesn’t require cumbersome software installations, frequent patching or painful
upgrade processes on the manual testers’ client machines. It should take IT less than a day to set up the full
solution.
KEEP UP THE PACE
The pace of business change isn’t likely to slow down, so you can’t expect the SAP ERP testing burden to
diminish either. Organizations that take the initiative to optimize their manual testing processes will ease the
burden on manual testers, enabling them to spend less time testing while allowing IT to complete more
projects. This ultimately empowers the company to become more agile and competitive.
Learn how Panaya can help you optimize your SAP testing processes. Visit us at: www.panaya.com.
-7-
