Difficulties in coping with functional complexity
experienced by the project team:
• Aligning tests correctly in batches that correspond to events in
the daily life cycle, including multiple days
• Accelerating the daily life cycle to speed up test execution and to cover
Settlement Failure processes in testing within a reasonable timeframe
• Accurately simulating and controlling test inputs from upstream systems
and outputs to downstream systems
• Management of complex Reference Data setup
• Involvement of complex components such as Risk Management
and Collateral Management systems
• Test cycle turnaround time, client certification and test automation
Keynote Speakers:
Alyona Lamash FRM Head of Risk Management Practice, Exactpro Systems
Mark Ryland Financial Markets Technology Consultant, Independent Consulting, United Kingdom
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
Delivering Large Post-Trade Initiatives: Quality Assurance and Key Challenges
1. How Highly Effective Quality Assurance
Techniques Can Make A Positive Difference
In Delivering Large Post Trade Systems
Initiatives
Post Trade Forum, Vienna 11 September 2014
2. Exactpro Company & Experience
Exactpro is:
2
• A specialist firm focused on functional
and non functional testing of wholesale
financial products systems
• An independent company incorporated
in 2009 now employing over 240
specialists
Our locations in Russia:
• A US company with four QA & development centres in Russia and sales
support in the UK
• Mostly engaged in trading platforms for exchanges and brokers, but also
significant post trade experience
3. Introducing Your Speakers
3
Mark Ryland
Strategic Adviser to Exactpro
• Financial Consultancy and QA of complex
Risk Management solutions, including
calculation algorithms
• BA for Collateral & Risk Management for
investment banking;
• QA Delivery Manager for large Post Trade
projects
Alyona Lamash
Head of Exactpro Risk Management Practice
• IT Manager/CTO
• Banking & Market Infrastructure
• Trading and Post Trade
• Insight and experience of QA
4. Talk Content
PART A
Challenges that Affect Large Projects - in General
How Great QA Techniques Can Help – Generically
PART B
Challenges that Affect Post Trade Initiatives – Specifically
QA Techniques that Help Post Trade Initiatives - Specifically
4
5. Testing Making a
Positive Contribution
to the Whole
Responding to
Issues/ Unknowns
Detailed Functional
Understanding
Achieving & then
Maintaining Test
Automation
Speed of Test
Execution
Clarity & Accuracy of Test
Results and Defects Reporting
5
Testing a Large System Poses ChallengesPART A:
Full Test
Coverage
6. How Can Effective QA Be Part of The Solution, Not Part of The Problem
• GENERIC
– Highly capable people with the right behaviours
– Efficient and effective test process
– Right use of technology
– Independent & objective measurement of progress
– RESULTS IN GREAT TEAM DYNAMICS ACROSS THE WHOLE PROJECT TEAM
6
7. Make a Positive Contribution to the Overall Team
7
Conception Initiation Development Test Implement
Project Sponsor
Project Manager
Business Analyst
Architect/Designer
Development Team
QA Team
Production Team
9. Make a Positive Contribution to the Overall Team
9
Pacemaker(s)
10. 1. High
Quality Staff
The Elements of Highly Effective QA - 1
10
3. Energy,
Enthusiasm &
Proactivity
2. Deep
Domain
Knowledge
4. Creativity &
Flexibility in Problem
Solving
Highly capable
people with the
right behaviours
11. 8. Test
Execution &
Reporting
1. High
Quality Staff
3. Energy,
Enthusiasm &
Proactivity
5. Planning &
Delivering Ability
2. Deep
Domain
Knowledge
4. Creativity &
Flexibility in Problem
Solving
The Elements of Highly Effective QA - 2
11
7. Defect
Reporting &
Evidence
Highly capable
people with the
right
behaviours
Efficient and effective test process
6. Test Case
Quality
12. 6. Test Case
Quality
8. Test
Execution &
Reporting
10. Range of Excellent
Automation Tools
9. Detailed
Management of
Test Environment
The Elements of Highly Effective QA - 3
12
7. Defect
Reporting &
Evidence
Right use of technology
1. High
Quality Staff
3. Energy,
Enthusiasm &
Proactivity
2. Deep
Domain
Knowledge
4. Creativity &
Flexibility in Problem
Solving
Highly capable
people with the
right behaviours
5. Planning &
Delivering Ability
Efficient and effective test process
14. Independence & Objectivity
14
1. Understanding of Requirements1. Understanding of Requirements
2. Delivery of Solution2. Delivery of Solution
3A. Independent
Measurement by
QA Team
3A. Independent
Measurement by
QA Team
3B. Independent Measurement by High Quality QA Team3B. Independent Measurement by High Quality QA Team
DeveloperDesigner
Business
Analyst
Business
Manager
End
Users
15. Team Dynamics When the QA Team is Excellent
15
DEVELOPERSDEVELOPERS
Try harder to get it right in the first place
Get rapid/timely feedback on what’s wrong
Work harder to solve defects to keep the project
on track
Try harder to get it right in the first place
Get rapid/timely feedback on what’s wrong
Work harder to solve defects to keep the project
on track
ANALYSTS AND
DESIGNERS
ANALYSTS AND
DESIGNERS
Have clear problem statements to work from –
with detailed supporting evidence
Have clear problem statements to work from –
with detailed supporting evidence
SPONSORS &
PROJECT
MANAGEMENT
SPONSORS &
PROJECT
MANAGEMENT
Have (justified) confidence in the system
Fewer problems are going to be found when Live
Project will cost less
Have (justified) confidence in the system
Fewer problems are going to be found when Live
Project will cost less
19. 6 Resulting Challenges From a QA Perspective
1) Testing of complex multistep scenarios
2) Limited test availability of downstream and upstream
systems
3) Most of endpoints are accessible via API or File
exchange (no GUI)
4) Reference Data setup or migration
5) Complexity of Risk Calculation Algorithms
6) Regression Cycle for a substantial number of test
scenarios
19
28. Solving: 6) Regression Cycle for a substantial
number of test scenarios
28
Test Library
(100K Test cases)
Test Tools
Automated
Test Scripts
Controlled
Test Environment
Automated
Regression Run
(“The Big Button”)
Post Trade Experience
Newclear initiative with MIT & LSEG
OEKB Project – Cash Equities and Fixed Income
Tools for T2S
LME Clear – Listed Derivs
Singapore Project
All large projects that are realised are the result of a team effort
Never seen a successful project that did not rely on stong contributions from several parts of the project team
Challenging projects call out something extra and it’s usual to see some individuals make outstanding contributions to the whole
Observing and being part of that is one of the most rewarding aspects of large project work = buzz, camaraderie, satisfaction of doing an excellent job
Proposition for the Talk
Thesis is that certain parts of the project can provide an exceptional contribution that raises the game of the others
Look for image of a race with a pace maker
The pace maker carries everyone else forward, others need to make an effort to keep up otherwise they can fall behind
In the analogy the paced could come from any part of the project team
You might expect it to come from the sponsor or the development manager of project manager or maybe the architect or lead analyst – in fact in a well performing team its likely to come from more that one place, but my thesis is that one of the places it can come from if the QA team
Almost no-one would think it can come from there
Independence and Objectivity
Subject to human interpretation & relationships - someone may say something is fine or not because they like or are influenced by the presentation
Some project team members may ‘sell’ the results of the work, be arrogant or fail to understand what is really needed
Benefits in independence of QA
Gives a ‘Measurement’ on status of delivery which is quite distinct from how anybody might ‘spin’ the progress
A great QA team can bring insight beyond the brief and specs they have been given and pick up things that have been missed by others
The most useful feedback is objective, but that’s not easy; Challenges of objectivity
Really need to understand the detail of how is supposed to work
Defects articulated clearly with necessary supporting evidence
Communicate based on facts not supposition
Reserve judgement
Consistency
All of the above leads to earning respect
Developers are going to try harder to get it right in the first place
Developers are going to get rapid/timely feedback on what’s wrong
Developers are going to work harder to solve defects to keep the project on track
Project sponsors are going to have clear visibility or quality level and progress
Project designers and analysts are going to have clear problem statements to work from
When the system goes live the project team is going to have (justified) confidence in the system – that will feed through to the users
Fewer problems are going to found in live (where they are much more expensive to fix) & hence the project will cost less
BIG BUTTON – Automated execution of complete regression test library with minimum human effort
This is a software development and infrastructure project
Supporting and maintaining requires additional effort (separate from Dev and QA)
Test Environment must be completely controlled by QA team
Where possible Test Environment should be isolated from external instability (such as Market data or downstream system)
In order to run many scenarios in parallel there QA need to maintain substantial number of different types of participants accounts in Test environments
Test Environment must have enough HW capacity to execute many tests within limited timeframe
Interfaces between Test Harness and Test environment must be stable. Relying on manipulation of unstable UI may lead on effort exceeding achievements due to test automation
Conclusion
A great QA Team can make a Disproportionate Contribution to Project Success