Software Quality Metrics for Testers - StarWest 2013
•
8 j'aime•2,954 vues
Presentation by Phil Lew at StarWest 2013. When implementing software quality metrics, we need to first understand the purpose of the metrics and who will be using them. Will the metric be used to measure people or the process, to illustrate the level of quality in software products, or to drive toward a specific objective? QA managers typically want to deliver productivity metrics to management but management may want to see metrics that describe customer or user satisfaction. Philip Lew believes that software quality metrics without actionable objectives toward increasing customer satisfaction are a waste of time. Learn how to connect each metric with potential actions based on evaluating the metric. Metrics for the sake of information may be helpful but often just end up in spreadsheets of interest to no one. Take home methods to identify metrics that support actionable objectives. Once the metrics and their objectives have been established, learn how to define and use metrics for real improvement.
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (20)
Similaire à Software Quality Metrics for Testers - StarWest 2013
Similaire à Software Quality Metrics for Testers - StarWest 2013 (20)
Plus de XBOSoft
Plus de XBOSoft (20)
Dernier
Dernier (20)
Software Quality Metrics for Testers - StarWest 2013
- 1. Software Quality Metrics For Testers Why and What to Measure
- 2. Meet Your Instructor • Phil Lew – Telecommunications consultant and network designer – Architect, Data warehousing product development – Software product manager, BI product – COO, large IT services company – CEO, XBOSoft, software qa and testing services • Relevant specialties – Software quality process improvement – Software quality evaluation and measurement – Software quality in use / UX design STARWEST 2013 2
- 3. Session Spirit and Expectations • Interactive • I won’t read the slides… • Slides for you as a take-away • Different than your handouts – Always thinking of new examples and ideas, you can write me and I’ll send these to you STARWEST 2013 3
- 4. Understand, Evaluate and Improve • If our end goal is improvement in what? • To improve, we need to evaluate • In order to evaluate, we must understand what we are evaluating • To do this… We need metrics Can you think of other examples in our lives where this applies? Where do you use metrics to evaluate and improve? STARWEST 2013 4
- 5. Metrics in real life Food Eaten Weight Performance Race Results STARWEST 2013 5 • Calories • Fat • Carbohydrates • Protein • Time of day • Vitamins • … • Blood pressure • Cholesterol • Blood glucose • Red cell count • White cell count • Hematocrit • Hemoglobin • Body fat % • … • Placing • … • Effort/Power • Heart rate/Watts • Speed • Time Intelligence Finesse Context • Training • Sleep
- 6. Metrics - Benefits • Understand how QA, testing, and its processes and where the problems are • Evaluate the process and the product in the right context • Predict and control process and product qualities • Reuse successful experiences – Feed back experience to current and future projects • Monitor how something is performing • Analyze the information provided to drive improvements STARWEST 2013 6
- 7. How can measurement help us (YOU) • Create a organizational memory – baselines of current practices-situation • Determine strengths and weaknesses of the current process and product – What types of errors are most common? • Develop reasoning for adopting/refining techniques – What techniques will minimize the problems? • Assess the impact of techniques – Does more/less functional testing reduce defects? • Evaluate the quality of the process/product – Are we applying inspections appropriately? – What is the reliability of the product before/after delivery? STARWEST 2013 7
- 8. All this is good But in reality…
- 9. Why we need to measure ? • Our bosses want us to… • They want someone to point fingers at • They want to fire some people and save money • They need to report to their managers • They want some basis on which to evaluate us and give us a raise! • We need to figure out a way to do beHer! • We want to improve our work and improve soKware quality STARWEST 2013 9
- 10. The Metric Conundrum • QA and Testing Language – Defects – Execution status – Test cases – Pass/fail rates – DRE… • Business Language – Cost effecRve – ROI – Cost of ownership – Cost of poor quality – ProducRvity – Calls to help desk – Customer saRsfacRon – Customer retenRon STARWEST 2013 10
- 11. In your organization…? • What measurements do you take in your organization and why? • Who uses them and for what? STARWEST 2013 11
- 12. The Metric Reality • Measurement is like dinner. It takes 2-3 hours to make dinner, and 15 minutes to consume… • But… many metrics are never reviewed or analyzed (consumed) • WHY? STARWEST 2013 12
- 13. The Metric Conundrum (cont.) • Test leads and test managers rarely have the right metrics to show or quantify value • Metric collection and reporting are a drag • QA metrics usually focus only on test execution • Test tools don’t have most of the metrics we want • Reports generated by QA are only rarely reviewed • Metrics are not connected to anything of value/ meaningful for ________. STARWEST 2013 13
- 15. Outline • Choose the right metrics • Design a measurement framework • Report what is meaningful • Use metrics not only to manage testing, but as a Quality barometer • Show value STARWEST 2013 15
- 16. Guiding principles about metrics • Metrics can represent a snapshot • Metrics can change over time • Metrics can be simple – don’t have to be complicated • Metrics can assist us in making decisions today • Metrics can assist us in predicting the future • Some metrics are critical for some while useless for others STARWEST 2013 16
- 17. Metrics can represent a snapshot • Not only at a particular moment, like this week, but for a process STARWEST 2013 17 Code quality Internal Quality Tested in a lab External Quality • What the end user sees • What the end user experiences In-‐Use Quality At what stage in the process do you take measurements?
- 18. Metric Basics • What is a metric? • What is a measurement? • What is an indicator? STARWEST 2013 18
- 19. Some Metrics – Let’s get started • How many of these do you know/use? • What others do you use? • What are they used for? • By who? Schedule variance Effort variance Cost variance Defect removal effecRveness ProducRvity Defect aging CriRcal Defect rate Test cases executed Test coverage Pass/Fail Rate ROI AutomaRon coverage STARWEST 2013 19
- 20. Metrics Exercise Work in teams of 2-3 List the metrics you use, what you use them for, who wants them, what decisions they make based on them STARWEST 2013 20
- 21. Developing a Measurement Framework Stakeholders Measurement IdenRficaRon Measurement Report Design Measurement Framework Data CollecRon Methods and Sources STARWEST 2013 21
- 22. Stakeholders • Who are they? • What are their objectives? • What questions do they have? • What answers can metrics provide? STARWEST 2013 22
- 23. Possible Stakeholders Stakeholder Objec0ve Ques0on Answer CIO CEO VP – QA – Development Director -‐ Development Owner Auditor Test Lead Partner ? STARWEST 2013 23
- 24. Metric Q&A Worksheet Category Ques0on Metric Tool to Use Needed Customer SaRsfacRon Are we meeRng schedule? Schedule Variance Project Management Tool None SoKware TesRng Efficiency SoKware Quality What kinds of defects are most common? Usability Are end users gedng lost, understand our UI ? ? • QuesRons are the fundamental building block • Answering quesRons that lead to decisions à VALUE 24
- 25. Stakeholder Workshop (15 minutes) • Divide up into teams of 2-3 • List out stakeholders • Brainstorm stakeholder questions (2 per stakeholder) • Determine answers and metrics to that question • Group your metrics into categories • Present and discuss to the group STARWEST 2013 25
- 26. Organizing our thoughts • Putting metrics into a framework • Helps us to make sure we have all functions in our organization and process covered • Helps us to develop relationships between different elements in our process STARWEST 2013 26
- 27. Identify Metrics to Answer Questions Customer and Management Metrics OperaRons OpRmizaRon Metric CalculaRon Departmental Metrics Project/Build Metrics Planning Quality Status Data CollecRon Metric Identification Req’ts. & Questions Reporting Efficiency and EffecRveness Schedule and Budget Defect types and quanRRes Resource uRlizaRon Snapshots and trends STARWEST 2013 27
- 28. Measurement and Reporting • Measure, build metrics AND develop reports that answer the questions of your stakeholders Planning • Requirements coverage % • Test case prioriRes and complexity/atomicity • Requirements traceability • Test ExecuRon prioriRzaRon • Requirements aging • Resource and budget variance STARWEST 2013 28
- 29. Test Measurement and Reporting • Measure, build metrics AND develop reports that answer the questions of your stakeholders Quality • Test pass/fail rate • # open defects • # priority 1 defects open • # priority 1 defects found • Re-‐opened defects • Defect density • Defects found in operaRon • Defects found/test case/script • Test execuRon progress STARWEST 2013 29 • Customer saRsfacRon • Help desk calls
- 30. Test Measurement and Reporting • Measure, build metrics AND develop reports that answer the questions of your stakeholders OperaRons • Time spent gedng environment ready • Time to close defects – aging • Effort per test case • TesRng schedule variance • Defect arrival and closure rate • Test schedule variance STARWEST 2013 30
- 31. Test Measurement and Reporting • Measure, build metrics AND develop reports that answer the questions of your stakeholders OpRmizaRon • AutomaRon script execuRon Rme • Regression Rme • AutomaRon coverage • Resource efficiency • Resource effecRveness • Test cycle Rme STARWEST 2013 31
- 32. Test Measurement and Reporting • Measure, build metrics AND develop reports that answer the questions of your stakeholders Status • Test execuRon % complete • Test case prioriRes and complexity/ atomicity • Requirements status/trending • Test ExecuRon EffecRveness • Over Rme (trending) versus snapshot STARWEST 2013 32
- 33. What’s Missing? STARWEST 2013 33 Requirem ents stability Internal Quality Defects Automa Ron coverage External Quality Customer retenRon Help desk call rate In-‐Use Quality Can you think of where other metrics we have listed out before, and where they would fit?
- 34. Data and Reporting OperaRons OpRmizaRon Metric CalculaRon Customer and Management Metrics Departmental Metrics Project/Build Metrics Planning Quality Status Automated Data CollecRon All for nothing unless you have supporting data Config mgmt Rqmt mgmt Time tracking Test mgmt Project planning Other? STARWEST 2013 34
- 35. Exercise: Develop Your Own Framework STARWEST 2013 35 Work backwards or forwards Work in small groups 2-3
- 36. Reporting • Work with stakeholders to report information so that they can make decisions • Metric is meaningless if not understood or misunderstood • Develop a report catalog – Metric library – Canned reports usually have limited impact STARWEST 2013 36
- 37. Reporting Root Causes and Trends • The most important metrics you can report – Root causes • Sources of defects • Sources of mistakes • Sources of delay or inefficiency • Sources of improvement – Trending • Showing metrics over time, pre-set intervals • Showing change • Make sure to differentiate Snapshot versus trend STARWEST 2013 37
- 38. Reporting – Show Value • Answer Questions – Are we delivering a quality product? – Are our customers satisfied? – Are we meeting timelines? – Did we understand requirements correctly and timely? – How much is QA costing? – Are we meeting budget? – How have we cut costs? – How have we improved efficiency? – What are the primary causes of delay? – What are the causes of poor quality? – Is quality improving? – What is the status? – Current versus goal? – Have we gotten better? STARWEST 2013 38
- 39. Show Value – Connect the dots – Did we understand requirements correctly and timely? – What are the primary causes of delay? – What are the causes of poor quality? STARWEST 2013 39 Requirem ents stability Internal Quality Defects Automa Ron coverage External Quality Customer retenRon Help desk call rate In-‐Use Quality QA and Testing Language Business Language – Is quality improving? – What is the status? – Current versus goal? – Are we delivering a quality product? – Are our customers satisfied? – Are we meeting timelines? – How much is QA costing? – Are we meeting budget and schedule? – How have we improved efficiency? – What are the primary causes of delay? – What are the causes of poor quality?
- 40. Getting Started with Measurement • What questions-answers are important to your organization? – Develop a model/determine what to measure – Define improvement objectives • Based on intuition/available data • Look at high payoff areas, problem areas • What data can you collect/Which technique can you use – Maybe some elements of the model drop out- can’t be measured that easily STARWEST 2013 40
- 41. Set up an action plan • Define a metrics improvement plan – What are your goals? • Conduct stakeholder workshops – Involve all • Build your measurement framework with phases in the process and categories – Q&A catalog – Metric catalog – Report catalog • Set up the infrastructure and operationalize • Start collecting and developing benchmarks STARWEST 2013 41
- 42. What’s Next • Choose a project – start small and build up for early wins • Organize to support the change – Committed people – Recognize the new processes – Assign roles and resources STARWEST 2013 42
- 43. Software Testing Metrics Take Aways • Quality is different for each organization – No one right answer for what metrics – Different metrics have different meanings for different people – SET UP FRAMEWORK AND PLAN FOR YOUR Company • Management wants metrics – Know what key questions need to be answered – REPORTING - Metrics programs often fail due to poor articulation • Measurement and improvement is a long term effort, just like a diet. STARWEST 2013 43
- 44. Putting It All Together Measurement Framework Improvement Decisions Stakeholders STARWEST 2013 44 Questions and Answers
- 45. Thanks Please fill out an evaluation form and drop it in the collection basket located at the back of the room. www.xbosoft.com @xbosoft 408-350-0508 Philip Lew @philiplew philip.lew@xbosoft.com Some resources: http://www.xbosoft.com/knowledge_center/ www.facebook.com/xbosoft