1. Trading & Clearing Systems
Test Automation
Exactpro Systems
21st December 2013

2. Approach to Test Automation Recommended by Exactpro
Given the relative difficulty of achieving efficiencies through automation of GUI testing, Exactpro has evolved a progressive strategy towards test automation that we recommend be
followed:
STEP 1: Test Server Functionality by
FIX or other standard protocol
STEP 2: ‘GUI Bypass’ Testing
STEP 3: Connect
to the GUI
GUI Object API
Web Tools
Windows Controls
Screen Scraping
• Wherever possible exercise functionality by message injection into FIX or
other standard interfaces first - it’s far easier to automate & debug
• If the system under test has documented interfaces between the GUI and
the server emulate the GUI with a test harness to exercise more of the
server functionality that is available through external interfaces
• Connect to the GUI using the best technical interface possible – the
technology available makes a big difference in the payback on the
automation investment
STEP 4: Semi Automated GUI Testing
• Partial automation can also be effective, e.g. enter data with automation
script and verify manually on screen or script checks database
STEP 5: Fully Automated GUI Testing
• This builds upon the other levels or automation, now it’s about testing the
GUI itself not the underlying system
STEP 6: ‘Big Button’ Framework
• Progressively link all of the automated tests into a framework that allows
highly productive control over test execution & reporting across of the
entire test set
2

3. Sample of Test Automation Achieved Using the Approach
STEP 1: Test Server Functionality by
FIX or other standard protocol
3

4. Exactpro’s Bespoke Test Automation Tools Suite
Sailfish:
• Can test Order Entry, Market Data and Post
Trade connections in one test scenario
• Each test scenario is independent
• Allows running test scripts in any sequence
• Simulation of multiple user connections
• Server simulators
• All messages are stored into a data base
• Generates test reports
Shsha:
• Post-transactional tool
• Analyzes clients' activity and forecasts system response
• Parses and displays logs in a user-friendly way
• Parses messages and then puts each to a data base
table where each column corresponds to each message
field
• Allows making summarized reports, etc
• Easy to understand GUI
ClearTH:
MiniRobots:
• Executes multithreaded java
code
• Complexity of test algorithms is
defined by the test developer
• Supports multiple client fix
connections, order entry and
market data via FIX
• Can use GUI to iterate through
sent and received messages
Dolphin:
• Model-based testing
of market surveillance
systems
• Post-Trade testing tool
• Verifies each stage of the
DLC
• Integrated schedule
• Automated matrices
• Can create multiple days
test scenarios
• Concurrent multiple tests
• Integrated simulators
• SWIFT ISO protocol support
Load Injector:
• Simulates multiple client
connections with a
specified load shape for
each connection or a group
of connections
• Up to 75K messages /
second from a single CPU
core
• Measures latencies in
microsecond range
• Performance test reports
4

5. Explanation of Sailfish
Item
Description
Exactpro Test Strategy
Step(s)
STEP 1: Test Server Functionality by FIX or other standard protocol; STEP 2: ‘GUI Bypass’ Testing; STEP
4: Semi Automated GUI Testing
Testing Type
Active Real-Time
Target SUT
Trading Platforms and Market Data Delivery Systems
SUT Interface
Back-end (typically connected to message gateways / APIs, and DBs); GUI Testing Capabilities
supported via plug-ins to other tools (e.g., Selenium)
SUT Interaction
Method
Message injection and capture for testing of real-time low-latency bi-directional message flows; DB
queries for data verification
Protocols
Extant plug-ins for Industry-standard (FIX and dialects, FAST, SWIFT, ITCH, HTTP, SOAP, etc.) and
Proprietary (MIT, SAIL, HSVF, RTF, RV, Reuters, Fidessa OA, Quant House, etc.) protocols. New plug-ins
for additional protocols developed by request (codecs are shared between Sailfish and Shsha)
Test Scripts
Human-readable CSV files; scripts generated manually by test analysts or automatically by test script
generator using results of passive testing performed by other tool (e.g., Shsha)
Test Management,
Execution and
Reporting
Integrated (Web front-end), allows for multiple simultaneous heterogeneous connections,
consecutive execution of multiple planned scripts, test results summary and detailed test reports.
REST API supports remote control of Sailfish instances. Optional Big Button framework supported
Platform requirements
Low footprint cross-platform application, MySQL or other RDBMS
Primary Competitor
FIX Testing Tools
5

6. Explanation of Shsha
Item
Description
Exactpro Test Strategy
Step(s)
STEP 1: Test Server Functionality by FIX or other standard protocol; STEP 2: ‘GUI Bypass’ Testing; STEP
4: Semi Automated GUI Testing
Testing Type
Passive Real-Time/Batch
Target SUT
Trading Platforms and Market Data Delivery Systems
SUT Interface
Back-end (typically connected to message gateways / APIs, and DBs); GUI Testing Capabilities not
supported
SUT Interaction
Method
Inputs and outputs monitored by means of message capture and log parsing to analyze client activity
and forecast system response; DB queries for data verification; files transfer, upload, export and
comparison. Captured messages can be viewed and analyzed in real-time or post-factum
Protocols
Extant plug-ins for Industry-standard (FIX and dialects, FAST, SWIFT, ITCH, HTTP, SOAP, etc.) and
Proprietary (MIT, SAIL, HSVF, RTF, RV, Reuters, Fidessa OA, Quant House, etc.) protocols. New plug-ins
for additional protocols developed by request (codecs are shared between Sailfish and Shsha)
Test Scripts
Certification tests and data reconciliation may be performed by using ordinary SQL queries. Test
message traffic generated in real-time or replayed from log files by other tool (e.g., Sailfish)
Test Management,
Execution and
Reporting
Integrated (Desktop front-end), allows for multiple simultaneous heterogeneous connections,
consecutive execution of multiple planned scripts, test results summary and detailed test reports.
Optional Big Button framework supported
Platform requirements
Low footprint cross-platform application, MySQL
Primary Competitor
FIX Onboarding Tools
6

7. Explanation of ClearTH
Item
Description
Exactpro Test Strategy
Step(s)
STEP 1: Test Server Functionality by FIX or other standard protocol; STEP 2: ‘GUI Bypass’ Testing; STEP
4: Semi Automated GUI Testing
Testing Type
Active Batch
Target SUT
Post-Trade Platforms and Middle Office
SUT Interface
Back-end (typically connected to message gateways / APIs, and DBs); GUI Testing Capabilities
supported via plug-ins to other tools (e.g., Selenium)
SUT Interaction
Method
Schedule-driven batch processing (often spanning several business days). Internal scheduler aligned
with the SUT scheduler. DB queries for data verification
Protocols
Extant plug-ins for Industry-standard (FIX and dialects, FAST, SWIFT, ITCH, HTTP, SOAP, FTP, etc.) and
proprietary protocols. New plug-ins for additional protocols developed by request
Test Scripts
Human-readable CSV files; scripts generated manually by test analysts or automatically by test script
generator using transactional and reference databases
Test Management,
Execution and
Reporting
Integrated (Web front-end), allows for multiple simultaneous heterogeneous connections, concurrent
execution of multiple planned scripts linked to particular global steps, test results summary and
detailed test reports. Based on Big Button framework principles
Platform requirements
Low footprint cross-platform application, MySQL or other RDBMS
Primary Competitor
Custom BPM solutions
7

8. Explanation of MiniRobots
Item
Description
Exactpro Test Strategy
Step(s)
STEP 1: Test Server Functionality by FIX or other standard protocol; STEP 2: ‘GUI Bypass’ Testing; STEP
4: Semi Automated GUI Testing
Testing Type
Active Multi-Participants (applicable for testing at the confluence of functional and non-functional
testing)
Target SUT
Trading Platforms and Market Data Delivery Systems
SUT Interface
Back-end (typically connected to message gateways / APIs); GUI Testing Capabilities not supported
SUT Interaction
Method
Message injection and capture to emulate multiple participants’ activity in electronic markets
(essential when there is a need to reproduce complex scenarios that can be created by trading
algorithms)
Protocols
Extant plug-ins for Industry-standard (FIX and dialects, etc.) and proprietary protocols. New plug-ins
for additional protocols developed by request
Test Scripts
Multi-threaded Java code specifying different liquidity profiles
Test Management,
Execution and
Reporting
Integrated (Web front-end), allows for multiple simultaneous heterogeneous connections, concurrent
emulation of multiple participants, detailed test reports. Optional Big Button framework supported
Platform requirements
Low footprint cross-platform application, MySQL or other RDBMS
Primary Competitor
Custom market and algo trading simulation solutions
8

9. Test Automation Essential Ingredients
Principles
Essential Ingredients
Type of Tests
Low Start
Passive & Active
Testing
Codecs, Drivers
and Stubs
The “Big Button”
Framework
High Volume
Automated
Testing (HiVAT)
Bespoke automation tools built up as an investment
over several projects, in combination with open
source tools
Highest calibre development staff on the team
Control of test environment back-end by QA and the
ability to emulate external systems in order to set
the required pre-conditions and increase test
automation coverage
Transactional
Scheduling
Reconciliation
Participant
Pragmatism and determination
Experience
9

10. Design of the Automation Will Embody These Principles
Principle
Description
1. Low Start
•
•
•
Don’t wait for the full comprehensive test harness to be available; use simple tools as soon as they are ready
Start with getting tools to execute with partial validation & progressively add more validations
Capture logs and executed files. Even semi-manual test execution provides better evidence and serves as the
foundation for future scripting
2. Passive &
Active Testing
•
Passive testing is a good way to collect test evidence from other tests and can be used to perform client
certification
Messages stored during passive testing can be used to generate active test cases and repeat previously executed
activities
3. Codecs,
Drivers and
Stubs
•
•
•
•
•
•
Codecs are test code capable of parsing a particular protocol or file format; using these allows increasing testing
depth, localizing defects and validating the components earlier
A Driver is a test program that can initiate the injection of messages into the system
A Stub is a test program that simulates an absent component called by the system under test
Drivers and Stubs both rely on Codecs
A Test Harness is a set of drivers and stubs for every interface link in the system
An automated test library relies on the Test Harness to execute the test scenarios
4. The “Big
Button”
Framework
•
•
•
•
“Big Button” organizes large automated test libraries, so that they can be executed in an unattended mode
It is necessary to carefully design reference/test data sets and plan operational schedule of the test environment
Reduces effort required for regression test execution, but also allows having a better test library structure
The Big Button approach requires more stability from the test tools and test scripts
5. High
Volume
Automated
Testing (HiVAT)
•
Like other complex distributed real-time systems, Trading and Clearing platforms have intermittent defects that
manifest themselves only if load is applied
They can be attributed to race conditions or to rare internal integrity failures
Such defects might not be spotted in the course of ordinary functional test execution due to their low statistical
probability under low volumes
HiVAT techniques are based on automated test case generation
•
•
•
•
10

11. Automated Tests can Address Four Areas
Areas
Description
1. Transactional
(Active Real-Time)
•
•
•
•
Processes incoming requests in real-time
Set up controlled transaction data set
Execute volume of independent test rapidly
Independent test cases and is easily supportable
2. Scheduling
(Active Batch)
•
•
•
•
Tests aggregation and batches functionality
Execute scenarios lasting multiple business days
Execute many test scenarios in parallel
Internal scheduler aligned with scheduler of the system
under the test
3. Reconciliation
(Passive Real-Time
/ Batch)
•
Verify system consistency by monitoring inputs and
outputs
Agnostic to test input source
Complements active test approaches
4. Participants
(Active
Multi-Participants)
•
•
•
Sample of Exactpro Tool
•
•
•
Emulation of participants activity in electronic markets
Reproduce complex scenarios
Testing at the confluence of functional and non
functional testing
Verify system from participant's interfaces
11