10 signs your software testing may not be enough
•Download as PPTX, PDF•
0 likes•689 views
Доклад Mitko Mitev на конференции SQA Days-20. 24-26 ноября 2016. Минск www.sqadays.com
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to 10 signs your software testing may not be enough
Similar to 10 signs your software testing may not be enough (20)
More from SQALab
More from SQALab (20)
Recently uploaded
Recently uploaded (20)
10 signs your software testing may not be enough
- 1. 10 signs your testing is not enough
- 2. How do we know? How do we know if we tested enough? • Frequently asked question • Fiascos • Assuring the minimum acceptable level of quality is a must • Need to analyze if you should be investing more resources, time, or money in your software testing • “Not enough” vs “Not Good Enough”
- 3. Background
- 4. Background
- 5. Background
- 6. Another horror story Knight Capital • Only one defect in the trading algorithm for Knight Capital caused a loss of $ 440 million in about 45 minutes. • That loss is 3 times the company’s annual earnings. • The shock that followed caused Knight Capital’s stock to lose 75% of its value in 1 business day. Infinium shut their algorithm down after just three seconds of trading, but it had already lost more than $1 million.
- 7. Lack of understanding Management misunderstanding • We don’t have bugs! • Testing is expensive! • It is an expense • What is the alternative? • Preventive by nature • We don’t need testers! • We do DevOps! • We do Agile! • Testing and Dev – 2 different skill sets
- 8. Positive trend Budget for software testing grows
- 9. How good is your testing?
- 10. 1: Glitches Excessive amount of Production bugs: • Unknown • How long have they been there? 1, 2 or more Test cycles? • Why not discovered earlier? • Not only investing more in testing but also considering changing Testing Strategy • Known – Why? • If found by Customers - Worst from business point of view • Bad PR • May have disastrous effects • Most expensive to fix
- 11. 2: Bugs in UAT Too much bugs discovered in Acceptance Testing: • UAT supposed to build confidence in product • Last stage before releasing • goal is to find deviations from intended design • Goal is NOT to find bugs
- 12. 3: Bug counts grow Are you aware of such trends? • If bug counts grow you should analyze asap to find reasons • If you don’t take measures your project/product might go in a Death Spiral But do you have the Metrics to see trends? • Basic Metrics are: • easy to apply • Great for analyzing trends
- 13. Quality Metrics Quality Metrics Project – Level Metrics 1. Requirements and Requirement Coverage – How to measure the coverage? 2. Defect Open and Close Rates
- 14. Quality Metrics Quality Metrics Department – Level Metrics 1. Mean Time To Detect (MTTD) and Mean Time To Repair – MTTD measures how long it takes QA professionals to find a problem, while the MTTR demonstrates the amount of time needed to effectively address it.
- 15. Quality Metrics Quality Metrics Company – Level Metrics 1. Number of Issues reported by Customers
- 16. 4: Not investing enough Investment in Testing is below industry average • Always good to know what competition is doing • You are investing less than similar companies
- 17. 5: No clear criteria No definition of what “enough” testing is: • Coverage • Execution • Acceptable levels of defects • Major • Minor • Plan for fixing known defects • etc
- 18. 6: Listen to Testers’ NO You receive negative advice from Testers to go ahead: • Always listen to employees • Value of good tester • Analyze the legitimated issues raised
- 19. 7: Weak Prevention Get the Team involved in: • Requirements assessments • Definition of coding standards/ good practices • Reviews/Inspections – Static Testing
- 20. Prevention
- 21. 8: No Dev involvement You don’t have Unit testing in place? • Software quality is a team responsibility • Unit testing is natural to developers • Find bugs earlier
- 22. Picture so far Choices and effects
- 23. 9: Time problems
- 24. 9: Time problems Testing periods are regularly reduced
- 25. 9: Time problems Deadlines problems regularly? • How often does it happen that testing is squeezed • Lack of time for testing is a symptom of deeper project, process and organizational issues. • Make sure you make enough time for Testing
- 26. 10: Testers keep quitting Dissatisfaction of Test Team: • Ultimate sign! • Happy employees – better performance • Can be financial • Can be also professional • Not enough time for testing • Not enough resources • Frustration levels grow up
- 27. Testing is enough! Make it even better! 1. Use testing as a high – level risk management 2. Test reusability 3. Automate
- 29. Questions?
Editor's Notes
- Each year companies spend more than $ 300 billion debugging their software – according to a research from Cambridge University’s Judge Business School. And another 60 Billion for fixing of Production bugs
- The stock price for companies that have experienced software failure fell about 4% on average in 2015 – according to Parasoft research. And 5.7% when they suffered multiple failures.
- That equaled $ 2.7 billion in lost market capitalization.
- http://www.cio.com/article/2393212/agile-development/software-testing-lessons-learned-from-knight-capital-fiasco.html
- Let’s face it – during the development of a product, software will have bugs. As much as we hate to admit, mistakes will be made, and it can be difficult for a developer to foresee an issue in how they have coded their work. If you aren’t doing a lot of testing because you think that your software doesn’t have bugs, you are probably incorrect, and you should consider more in testing. Software testing is expensive - yes, software testing is an expense, but it is wrong to think of it as expensive – just think for a minute about the alternatives. Software testing is by its nature a preventative approach!!! Software testers are not necessary – developers can do the testing – never forget: developing and testing – two different skills; also developers tend to perform only positive tests and the problems are usually found by negative tests;
- https://www.capgemini.com/thought-leadership/world-quality-report-2015-16
- An obvious sign that you should be investing more in software testing is if a lot of bugs are discovered in your released software. An even larger issue than just bugs that are found in production are bugs that have been in the software from early in development and have survived multiple test cycles. If bugs are not being discovered over a course of several test cycles, this implies that your testing is inefficient. It has been a proven fact in software testing that the earlier that bugs are discovered, the cheaper it is to fix them. If your software that is in Production contains bugs that have been in the code for a long time, you should consider not only investing more in testing, but also changing your testing methodology. It’s never fun for others to find faults in your work, but it is even worse when those who find faults are customers who have paid for your product. If your customers are finding excessive bugs, you should consider investing more in testing, as bugs do not reflect well on your company, employees, or product.
- User Acceptance Testing (UAT) acts as a provision of confidence that the delivered software will meet the business requirements of both the developing company and the prospective users. UAT is one of the final stages of the development process before software is released to the customer, and is designed to merely deviations from the intended design. When bugs are discovered during UAT that are not just design flaws, you should consider investing more in testing.
- http://www.getzephyr.com/resources/whitepapers/qa-metrics-value-testing-metrics-within-software-development
- http://www.getzephyr.com/resources/whitepapers/qa-metrics-value-testing-metrics-within-software-development
- http://www.getzephyr.com/resources/whitepapers/qa-metrics-value-testing-metrics-within-software-development
- It is always a good idea to know what your competitors are doing. The same concept applies to the investment of software testing. If you discover that you are investing less in software testing than similar companies, you should probably consider investing more.
- Always listen to your employees if they are raising legitimate issues. The value of a good software tester partially lies in comprehending their project and understanding the values of a well-tested application. If you are considering releasing your software and your software testers are telling you that they have not had enough time to test, you should consider investing more in testing. 2. Fighting a lose – lose situation – you report the software is not ready for production – you are blamed for the delays; you certify it is OK, the software goes live, many bugs are found by the end users – you are blamed again; 1. Having to say “NO” - as testers usually bring the bad news, so they become the enemy. So, set clear roles, responsibilities and procedures;
- It costs four – to – five times as much to fix a software bug after release, rather than during the design process according to the Systems Sciences Institute at IBM. These bugs may cause people to lose their lives. Always have in mind that fiascos bring LOSS of REPUTATION.
- Elisabeth Hendrickson, 2000
- Happy employees tend to output better work, particularly if they have sufficient time and resources to complete their project. If your testers keep quitting because they are overworked and their time is stretched too thin, you should invest more in testing.
- Happy employees tend to output better work, particularly if they have sufficient time and resources to complete their project. If your testers keep quitting because they are overworked and their time is stretched too thin, you should invest more in testing.
- http://www.cio.com/article/2393212/agile-development/software-testing-lessons-learned-from-knight-capital-fiasco.html; http://www.satisfice.com/repeatable.shtml