QA Financial Singapore: Deliberate Practice of Software Testing in Agile World
•Télécharger en tant que PPT, PDF•
4 j'aime•852 vues
QA Financial Forum Singapore 2017. Event is fully dedicated to quality assurance in financial services software. 25 May 2017 Iosif Itkin, Exactpro CEO London Stock Exchange Group
Recommandé
Recommandé
Contenu connexe
Similaire à QA Financial Singapore: Deliberate Practice of Software Testing in Agile World
Similaire à QA Financial Singapore: Deliberate Practice of Software Testing in Agile World (20)
Plus de Iosif Itkin
Plus de Iosif Itkin (20)
Dernier
Dernier (20)
QA Financial Singapore: Deliberate Practice of Software Testing in Agile World
- 1. 1 Deliberate Practice of Software Testing in Agile World 25 May 2017 Iosif Itkin, Exactpro CEO & Co-Founder London Stock Exchange Group Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 2460, +1 415 830 38 49 www.exactpro.com
- 2. 2 London Stock Exchange Group is a leading diversified international exchange and infrastructure group with assets across the entire exchange value chain. Real time data Primary Markets Cash Equities Derivatives Fixed Income IDEM RNS MOT X2MTechnology Monte TitoliCC&G ORB ExtraMOT LSEDMIDEX GlobeSettle Unavista Hosting and Connectivity Information Services Post Trade Capital Markets SEDOL Business Services Academy Events &Studios
- 3. 3 Deliberate Practice of Software Testing
- 4. 4
- 5. 5
- 6. 6
- 7. 7
- 8. 8 No Trust No Fear No Begging Nobel Prize in literature in 1970 Author “The Gulag Archipelago”
- 9. 9 NO TRUST
- 11. 11 • A specialist firm focused on functional and non functional testing of systems that process wholesale financial products, particularly market infrastructures • A UK company with operations in the US and four QA & software development centres in Russia • Part of London Stock Exchange Group as of May 29, 2015 • Incorporated in 2009 with 10 people, our company has experienced significant growth as satisfied clients require more services; now employing 515 specialists Exactpro is: Front Office Listed Derivatives CRM Dark Pool T2S Clearing FX Fixed Income SEFReconciliation Onboarding Ticker Plant Commodities Smart Order Router Trading DesktopsIndex Dissemination FPGA Equities Connectivity Exchange MTF Swaptions Now employing 515 specialists
- 12. 12 Automated testing Compares intended and received results Automated end 2 end clearing system testing Automated monitoring, analysis and reporting 75K messages / second from a single CPU core Measures latencies in microsecond range A variety of algoes simulating end-clients will run and see how system performs Build Software to Test Software
- 13. 13
- 18. 18 Post-Trade
- 20. 20 New Platform
- 21. 21 Broken Links
- 25. 25
- 26. 26
- 27. 27 Things Can Go Wrong
- 28. 28 1/8 of employees working in QA 128m of transactions per day near Brexit and US elections 84 microseconds equity markets roundtrip times
- 30. 30 1. Random load “Pace Maker” 2. Variety of Passive Liquidity 3. “Aggressors” try to interact with the market to increase the number of test cases happening in the environment 4. Analyze the situation in retrospect. Make sure that what happened is correct. Tick checkboxes against tests that were actually executed Updated test library Minirobots
- 31. 31 1. 3. Simulate real trading participants 2.
- 32. 32 Have No Fear
- 33. 33 • Unrealistic Scenarios • Annihilate the System Under Test • Defects Leaking out of a Sprint Have No Fear
- 34. 34
- 37. 37 • Confirmation that PROD is Configured as TEST • Tested Drops out of Development • Things that You Do Not Trust NO BEGGING
- 39. 39 https://ru.linkedin.com/in/iosifitkin The seventh EXTENT conference will take place in London, UK. Find out more: www.extentconf.com – LSEG Technology and Quality Assurance – Risk controls and FPGA – MiFID2 Software Testing Challenges – Blockchain and Trading Technology Trends – Learning from Other Industries
Notes de l'éditeur
- Good Afternoon. Thanks a lot for the introduction. My name is Iosif Itkin, I am co-founder and CEO of Exactpro.
- Exactpro is an open access software testing subsidiary of the London Stock Exchange Group. We do functional and non-functional testing for the Group’s own markets. We also provide service to other exchanges, market infrastructures, banks and financial firms worldwide. One in every eight of LSE employees works in our business fully dedicated to quality assurance of software used in the financial services industry.
- Our focus is Deliberate Practice of Software Testing. As many other organizations, London Stock Exchange and most of our clients are going through agile transformation. When people discuss software testing and agile they sometimes wonder: is there any room left for independent QA nowadays?
- As a result, we often receive requests to confine QA into scrum teams.
- At our clients’ scale, it is usually not a single scrum team. It is more like a train of teams or a set of trains.
- A prison for software testers.
- So, we have had to formulate and follow some principles of surviving and keeping our internal freedom in such harsh environments.
- In one of his books, Alexander Solzhenitsyn (a famous Russian author who won the Nobel Prize in literature in 1970) had quoted a saying among the prisoners of GULAG, which helped them survive during their daily interaction with the prison authorities. It goes like this: "Don't believe them, don't fear them, don't ask anything of them". It is also known that a similar saying is attributed to an ancient Greek philosopher Epicurus ("Noli credere, noli timere, noli petere"). So, we decided to go with a shorter version of the same as our principles: “No Trust, No Fear, No Begging”.
- There are hundreds of papers stating that agile requires trust. However, complex and stable systems are not built on trust.
- They are built on the absence of trust. Think about it: you do not trust that you’ll be able to elect an ideal president every time. Instead, you put a system of checks and balances in place, and that gives you a stable society. If you put your blind trust into anything, even if the idea looks great, you may end up living in a dystopian nightmare.
- Seven year ago, Exactpro had a very small team. We employ over five hundred specialists now. In addition to luck and hard work, there is another factor which has helped us grow as a company. People born in the former Soviet Union generally grew up in the atmosphere of profound doubts about everything, thus developing a mindset of mistrust. It is sad, but it is an ideal mentality for a software tester.
- We Build Software to Test Software and we use our own gateways to connect to the systems.
- This allows doing independent automated testing. For example, our tool Sailfish has helped us a lot in doing connectivity testing for many brokers and exchanges.
- We often receive requests to automate everything. Sometimes, that contradicts our concept of the Deliberate Practice of Software Testing.
- Deliberate Practice is not just repeating something for 10,000 hours. Rather, deliberate practice means constantly applying your brainpower to stretch and improve yourself. Many people drive their car to and from work for many hours, but never achieve true mastery. It’s because in most cases we switch our brain off while doing repeatable things. Automating every user story without taking context into account is the closest thing to switching your brain off.
- For our markets, we automate test libraries containing tens of thousands of scenarios. In doing so, though we are not wasting resources on automating user interfaces that will change in the next Sprint.
- Same people who insist on automating everything frequently ask to collocate the QA team with software development one. Think about a firefighting robot. It was created to help firefighting teams to put out fires. Do you really need the robot’s operator to sit in the burning building where the robot is working?
- On the other hand, creating such a robot is a sophisticated technical process. It might require a separate dedicated development team not embedded in any existing scrum teams. We’ve managed to create testing tools, such as ClearTH, reflecting complex operational life-cycles and reference data setup of the post-trade systems and executing huge regression libraries in an unattended mode
- It makes more sense to collocate QA with Business, and we therefore are trying to maintain some presence at our client’s locations. It is important, though, not to underestimate the value of information that can be obtained from the systems themselves.
- Everyone these days talks about big data and machine learning. Frequently people forget to look at their own systems already in production as the source of the requirements.
- We have seen it as an anti-pattern. A company migrates to a new system and it is necessary to implement connectivity with legacy platforms. Instead of looking at the actual production logs and data setup, projects rely on written specifications describing how it is supposed to work. You do not need to be collocated to obtain valuable insight from the data.
- Listening is sometimes more important than talking. So, we have created a passive testing tool capable of reading the wire and log files and extracting data from them. It stores the data for processing and runs many checks against it to find patters and problems.
- We also use the passive testing approach for latency measurement across all of our markets. Passive testing can help to onboard clients and even run crowdsource testing
- Let’s talk a bit about cross-functional teams. We believe that excellence is achieved from focus. When you call 911 both the police and paramedics arrive. A policeman can offer immediate medical assistance, but any complex case requires a medic. Policemen and medics have the same purpose – saving and protecting human lives. However, they have different mentality, which allows each of them to do their work better.
- Software developers focus on building software. Software testers focus on finding failure. These are different attitudes. Both are required. One is not ok without the other. It is said however that the presence of a software testing team removes the ownership of quality from software developers.
- I would like to recommend a book. Its name is “Extreme Ownership: How U.S. Navy SEALs Lead and Win”. You can’t always expect that your scrum team consists of individuals with super human abilities – extremely professional, tireless, dedicated, selected through an elaborate process out of dozens or hundreds of others. If there is a representation of an ideal Agile team – it is of course a squad of Navy SEALs. They truly take extreme ownership of the quality in everything they do. This superb cross-functional team can accomplish many missions autonomously. You can only dream of having something like this as a software development team. Read the book and you’ll find out something unexpected about such a self-sufficient “Ultimate Agile” team.
- While they plan their operations, they always consider that the things are likely to go wrong and additional firepower support from the Tanks will be required. This is the way to run operations efficiently and with minimal casualties. Independent software testing is like those tanks.
- Speaking about firepower. We have a lot of it. LSEG operates some of the most advanced and well-performing systems. Round-trip latency is 84 microseconds on average. Last year the number of transactions on a single platform exceeded 128 million per day.
- To stress-test such systems, we use our dedicated testing tools. To reduce the required hardware footprint for load generation software, we make it very simple, but powerful. It’s like a bag of nails.
- In addition, we use tools that mimic real market participants and look at the system as real traders do. Minirobots.
- They can spot problems missed by the load generators or data analysis tools as they look from an entirely different perspective.
- One of the common fears is to have unrealistic performance testing. Our strong view is that you should not be afraid. Have no fear. The tests can be unrealistic. By limiting your validation to only production-like scenarios you will not be able to prepare yourself for the things to come.
- People are often afraid to break the system and therefore go only to certain pre-specified volumes and throughputs considered to be realistic. We always try to test until the system breaks down. It is important to observe what will happen when the system is no longer able to stand the load. Micro-bursts do happen in real world. Any throughput level considered to be unrealistic will be observed on the micro-scale sooner or later. Don’t be afraid to stretch the limits.
- Let’s talk about defects. If they leaked out of a sprint, should we be afraid? Software Testing is relentless learning. Defects are the best source of information about the system. They help everyone to understand the system better. They help develop better software testing tools and libraries. Defects serve as a great collaboration tool between business, developers and QA.
- Frequently, the absence of defects is not the evidence of ideal quality, but just a consequence of confirmation bias and groupthink. The presence of defects within software development life cycle is what makes your projects Antifragile.
- Let’s talk a bit about critical defects leaking into Production. Isn’t it the one thing you should be afraid of? Our view is that Production outages are like death. They are inevitable in complex technology systems. It is not a question of if. It is the question of what you will do when it happens. Always assume that the system will break. It is the only way to get a truly resilient system and avoid a situation when problems turn into disasters. To have a good life one should always think about death. If you fear death, you’ll die.
- Asking questions is crucial for software testing. Begging is not asking. Begging is requesting things that you will not get. You do not need things that you will not get. Your plan should not be dependent on the things that you do not trust. Examples – ask for a confirmation that Production has the same configuration as the test environment. You can't rely on a confirmation unless you can validate it. Request that only tested drops come to QA from development. We do not ask for it, as we do not need a quality gate here. Early testing is a better way to push forward and defects are one of the best learning tools.
- There is no need to confine QA into a scrum team or process. Instead the focus should be on individuals and interactions capable of finding defects. There is no need to request all test scenarios upfront for test first development. Instead focus on delivering working software. Defects should be used as a collaboration tool instead of tracking the SLAs between Sprints and Gates. Deliberate Practice of Software Testing enables you to respond to change in a truly agile manner.
- Thank you