The next generation of testing tools for exchanges
•Télécharger en tant que PPT, PDF•
0 j'aime•1,498 vues
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (15)
En vedette
En vedette (17)
Similaire à The next generation of testing tools for exchanges
Similaire à The next generation of testing tools for exchanges (20)
Plus de Iosif Itkin
Plus de Iosif Itkin (20)
Dernier
Dernier (20)
The next generation of testing tools for exchanges
- 1. The Next Generation of Testing Tools for Exchanges Iosif Itkin Exactpro Systems 24 March 2015
- 2. Test Tools Typology • Connectivity Testing • Test Automation • Load Testing 2
- 3. Generic Test Tools • No Connectivity Layer • No Session Layer • No Framework for Messages 3
- 5. Test Tools Evolution 5 Passive Testing – investigating the behavior of a system without affecting its operations Generic Trading +Passive
- 7. Test Tools Evolution 7 Passive Testing – invaluable for Client onboarding and certification Generic Trading +Passive
- 8. Ideal Passive Testing Tool 8
- 10. Ideal Active Testing Tool 10
- 11. Production Algo Trading Engine 11
- 12. Test Tools Evolution 12 Testing at the Confluence of Functional and Non-Functional Generic Trading +Passive Production
- 13. Test Tools Evolution Generic 13 Trading +Passive Production Mutated and Deformed Next Generation
- 14. 14 Human Intelligence is Still Required
- 15. Build Software to Test Software 15 ClearTH: • Post-Trade testing tool • Verifies each stage of the DLC • Integrated schedule • Automated matrices • Can create multiple days test scenarios • Concurrent multiple tests • Integrated simulators • SWIFT ISO protocol support Minirobots: • Executes multithreaded java code • Complexity of test algorithms is defined by the test developer • Supports multiple client fix connections, order entry and market data via FIX • Can use GUI to iterate through sent and received messages Dolphin: • Model-based testing of market surveillance systems • Production-scale capacity and throughput • Interactive real-time alerts and reports Shsha: • Post-transactional tool • Analyzes clients' activity and forecasts system response • Parses and displays logs in a user-friendly way • Parses messages and then puts each to a data base table where each column corresponds to each message field • Allows making summarized reports, etc • Easy to understand GUI Load Injector: • Simulates multiple client connections with a specified load shape for each connection or a group of connections • Up to 75K messages / second from a single CPU core • Measures latencies in microsecond range • Performance test reports Sailfish: • Can test Order Entry, Market Data and Post Trade connections in one test scenario • Each test scenario is independent • Allows running test scripts in any sequence • Simulation of multiple user connections • Server simulators • All messages are stored into a data base • Generates test reports
- 16. Thank You 16
Notes de l'éditeur
- Hello Everyone. My name is Iosif Itkin and I work for Exactpro Systems. We have 300 engineers focused on Quality Assurance for Trading 80% of our projects are related to execution venues and clearing – equities, derivatives, swaps and other asset classes. Over the last five years, we tested extraordinary exchange platforms to the edge and we’ve seen things you people wouldn’t believe. I would like to use this opportunity to share some thoughts on the test tools used in exchange area.
- Test tools were created by man to make life easier. Machines can simulate other machines. Programs can repeat the same tasks over and over again. Tools can amplify the human power.
- Lets start with generic testing tools. No one got fired for using standard commercial solutions. The problem with many of them is that they are targeted at desktop and web interfaces. The integration with trading APIs usually requires developing connectivity and session libraries. You will also need a framework to uniformly define message contents and expected results. Essentially you will build one testing tool on top of another.
- So, the next step is a specialized testing tool. The are a few of them available that already support FIX, FAST, ITCH, binary exchange protocols and have expandable architectures. You specify what to send into the market and what should the expected result from every gateway be. This process is called active testing. If the tool is OK and your QA team is capable enough you’ll be able to prepare and maintain thousands of test cases to validate most of your exchange platform’s functionality.
- The execution of huge regression libraries produces a lot of data. Additional insights can be obtained from analyzing these big data sets.
- The next evolutionary step in test tools is passive testing where you use non-intrusive test tools and capture every message going back and forth, normalize them and store them for processing. You can obtain interesting results about your system and your test library coverage.
- We also have discovered that passive test tools are invaluable in supporting client onboarding and certification. Clients connect to the exchange platform and perform the necessary steps, while the tool captures everything and can produce the self-certification report. You can have a good functional testing tool. You can also create a scalable load testing tool like we did to generate hundreds of thousands of messages per second and stress your platform to its limits. But there is another area in between. Defects that are intermittent, the ones that are not reproduced every time you run the test, potential black swans. You need to repeat the tests many times and under load to trigger them. On the other hand these defects might not have severe technical consequences and they can be easily missed in the enormously large load testing logs. These defects are at the confluence of the functional and non-functional testing.
- What do we need from a passive testing tool to capture such defects? It should be scalable, with minimal impact on the system under test, it should collect all events from all components in the system, it should capture all the data efficiently for analysis, it should make me aware when something goes wrong, I should be able to configure the scenarios for such a notification and when something indeed goes wrong, I would like to restore the transactions around the event.
- Such test tool looks exactly like a very good market surveillance or production monitoring system. So, the ideal passive testing tool for exchanges looks exactly like a production monitoring/market surveillance solution. What about the active testing tool?
- Again, it should be scalable, connected to many test environments, easily configurable, validating data consistency across various feeds, storing the audit trail and capable of executing complex scenarios.
- Well it is clearly an algo trading system. Instead of running Excel in getting Excel out, connect something that behaves like a market participant with an algo engine. Is that it? Can the next generation of the test tools be just algo-trading and market-surveillance platforms? Not really.
- For two distinct reasons. Imagine trying to fit several algo engines and an additional monitoring solution into your budget. It will substantially increase the cost of testing. And using ordinary real-life systems has the limitation in that their intention is to do what is right. But, the purpose of testing is to find what is wrong. A good tester knows that he or she must not trust anything and always look into the ways how to break the system… That is why Russian testers are so good probably.
- The next generation of test tools will be very close to production systems, but deformed and mutated to decrease the hardware footprint and to enable all possible permutations. In a more distant future there might be test tools based on machine-learning techniques. However, they can be enabled only by ensuring the scalability and mutability for the next generation of the test tools.
- As singularity is not here yet, human intelligence is still required to create and operate test tools and large regression libraries. The best part of any testing solution are brains, hands and hearts of the people who created it.
- We know how to build software to test our client’s software. Exactpro has developed tools for trading, clearing and surveillance systems testing. We constantly work on evolving our test tools. These tools have been proven on many projects for our clients worldwide.
- If you ever need a next generation testing tool or a comprehensive quality assurance process, please do not hesitate to talk to us. Thank you