This presentation explains the metrics based management for Software Quality, its workflow, expectations of management and customers

1. 0
Metrics Based
Management

2. 1
Contents
Key AD / AM Metrics
Who are the Key personnel
Why Metrics
When and how to use Metrics
How to identify Data Quality issues
Some Enablers for Improvement

3. METRICS

4. Effort Metrics Formula
Effort Variance ((Actual Effort – Estimated Effort) / Estimated
Effort)*100
Load Factor (Actual Effort / Effort available))
% Review Effort (Total Effort expended on Reviews across all
stages)/ (Actual Overall Project Effort) *100
% Cost of Quality (Effort spent on Prevention + Effort spent on
Appraisal + Effort spent on Failure) / (Effort
spent on Prevention + Effort spent on Appraisal
+ Effort spent on Failure + Effort spent on
Production)) *100
Schedule Metrics Formula
Schedule Variation ((Actual End date – Planned End date) /
(Planned End date - Planned Start
date))*100
Duration Variation ((Actual End Date – Actual Start Date) –
(Planned End Date – Planned Start Date))
/ (Planned End Date – Planned Start
Date) * 100
AD - Basic Process Metrics and Formula
Schedule Effort

5. AD - Basic Process Metrics and Formula – Cont.
Defect
Metrics Formula
Defect Removal
Efficiency
(Total number of Pre-shipment Defects)/ (Total
number of Pre-shipment Defects + Total number of
post-shipment Defects + Total number of Post
production Defects) *100
Defect Detection
Efficiency
(Number of Pre-shipment defects / Appraisal Effort)
Defect Density by
Effort
Total no of Defects Detected/Total overall actual
effort spent.
Defect Leakage Sum((Number of defects attributed to a stage but
only captured in subsequent stages) / (Total number
of defects captured in that stage + Total Number of
defects attributed to a stage but only captured in
subsequent stages)) *100
Metrics Formula
Size Variation ((Actual Overall Size – Planned Overall
Size) / (Planned Overall Size))* 100
Productivity Overall Productivity = (Overall Project
Size) / (Total Effort for the Project)
Size

6. AVM - Basic Process Metrics, Definition and Formula
Metrics Formula
Acknowledging
Severity 1….5 Incident
(No. of Sev 1/2/3/4/5 incidents
acknowledged within the applicable
Acknowledgement Time / Total No. of
Sev 1/2/3/4/5 Incidents) * 100
Severity 1…..5
Incidents resolved
within the allotted time
(Number of Sev 1/2/3/4/5 incidents
resolved within the allotted time /
number of Sev 1/2/3/4/5 incidents
resolved) * 100.
Avoidable Problems /
Unforced Errors for
Severity level 1 and 2
(No. of incidents/problems caused by
the Supplier's Actions / Total No. of
incidents/problems) * 100
Metrics Formula
Function Points per
$1K Spent
FP count / Total amount spend in ($1K )
Defect Injection Rate -
Release or Project
(Total number of defects injected in the
Release or Project / size of product)
% of SLAs met % of fixes without
escalation

7. Other set of Product Quality Metrics
Metric /
UOM
Formula Operational Definition
Tools Usage
.Net Java
Code Review
Coverage
Number of impacted programs
reviewed / Total number of
programs * 100
A measure of the review coverage on the
number of programs
This is higher the better metric
VSTS-Code
Analysis,
FxCop, Sonar
SONAR, JCAP, PMD,
Checkstyle, FindBugs
Unit Test
Coverage
Based on Unit Test Coverage
Tools such as Junits/JCoverage
The Code coverage metric identifies the
sections of the source code that were either
tested/ not tested as part of white box testing
This is higher the better metric
NUnit, VSTS-
Unit Testing
NCover
Junit, Test NG, Code
pro analytix,
Cobertura, EMMA
Code Quality -
Cyclomatic
complexity
Cyclomatic Complexity at class
level
(Highest method CC)
This metric estimates the complexity of the
individual functions, modules, methods or
classes within a program so as to measure the
program's structural complexity.
Lower the better metric
IDE,Sonar,
VSTS-Code
Analysis,
FxCop
SONAR, JCAP, PMD,
Checkstyle, FindBugs
Requirements
to Test Case
coverage
% of requirements linked to
Test cases
An indication of how extensively the
requirements are covered by Test Cases
This is higher the better metric
VSTS- Unit
Testing,
NCover
Junit, Test NG, Code
pro analytix,
Cobertura, EMMA

8. Metrics – Testing
Project
Metrics
Intent Definition
Reporting
Frequency
Test
Effectiveness
Indicates ability to unearth and fix defects
before they reach UAT and Production
(Number of accepted defects in SIT / (Number of accepted
defects in SIT + UAT + Post UAT)) * 100
Monthly
Test Design
coverage
How much requirements are covered by
test cases ?
(Total number of baselined testable requirements mapped to
test cases / Total number of baselined testable
requirements)*100
Monthly
Test Case
Preparation
Productivity
Test case creation productivity of the team
((No of Test Cases or Test Case points (TCP) prepared)/ (Effort
spent for Test Case Preparation)
Monthly
Test Case
Execution
Productivity
Indicates test execution productivity of
the team
((No of Test Cases or TCP Executed)/ (Effort spent for Test
Execution)
Monthly

9. METRICS

10. 9
Metrics Work Flow
Developer / TL
enters/updates
the data
PL Reviews the
data
PM Approves the
Metrics
Metrics review
by DD / DM

11. Some proactive approaches for reviewing the Metrics data
 Metrics Submission date to be done by end of every month.
 Review of Metrics by PM/Delivery Manger to be completed subsequently.
 Monthly Metrics Review scheduled with Delivery Director by 1st week of subsequent
month for critical projects.

12. METRICS

13. 12
Customer’s Expectations (i.e. Why Metrics)
Improved
Business Value
&
New Revenue
Generation
On time delivery &
Improvement in
Time to Market
Zero defects
High
Quality
Business /
Technology
Solutions
Reduction in IT
Operating Cost
Am I getting
more work for
lesser $ spent
over a period

14. 13
Senior Management Expectations (i.e Why Metrics)
Are we fixing
more Tickets
over a period
Are we building
more LoC over a
period
Are we
delivering
Zero
Defects
Software
Are we making
expected
Profitability
Are we
adopting Best
Practices and
Reusable
Are we getting
accurate
Productivity
while
submitting RFPs
Are we getting repeat
business from this
engagement

15. 14
Where are we - Now
Follow-up
on Project
planning &
tracking
Data Quality Issues
No Data porting
from external tool
Expending our energy in
process compliance Follow up
mailers for
Metrics data
submission
Project Health
Scorecards
JUST THINK
Are these parameters helping you to meet
Customer Expectations ?
Senior Management expectations ??
Delivery Managers Expectations ???
Are we using these data in a true sense for the success of the project ????

16. 15
Metrics Based Project Management – Work flow
Process Performance Objective ; Process
Performance models
Benchmarking
Guidelines for Metrics and QPM, Statistical
Techniques
Data Validation, Analysis and reporting ; Facilitation
for usage of statistical tools like control charts
Facilitation
Data Trend analysis and validation of trends using
hypothesis testing
Metrics Based Project Management

17. METRICS

18. Scenario - 1
Effort Variation Effort Over Run
Schedule Variation On Schedule
Defects (Internal)
Defects (Customer)
Inference
• Is there a problem in Estimation
• Is the team over burdened?
• Is the team possess right skills to carry out the tasks in a stipulated
time period? Or takes longer time to find a solution & fix
• Is there any scope scale down?
Bring back to Track
• Revisit the Estimation
• Additional Trainings to Team

19. Scenario - 2
Effort Variation Effort Over Run
Schedule Variation Schedule Over Run
Defects (Internal)
Defects (Customer)
Inference
• Is there a problem in Estimation
• Is the team over burdened?
• Is the team possess right skills to carry out the tasks in a stipulated
time period? Or takes longer time to find a solution & fix
• Is there any scope creep ?
• Are requirements changed frequently (stability of requirement is low)
Bring back to Track
• Revisit the Estimation
• Additional Trainings to Team
• Reach out to customer, if Scope creep observed, Requirements
changed

20. METRICS

21. Data Quality Issues- What is wrong here
A Project with a –ve Effort variation cannot have a +ve Schedule overrun and a high
LF

22. Data Quality Issues- What is wrong here
Failure Cost is zero in spite of Defects recorded and having a very High
Appraisal cost. Production Cost shows zero

23. Thank you
22