Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Metrics
1. Metrics & Measurements
A Product Ship Perspective
Vijayan Reddy
Quality Manager
Adobe Systems
22/March/2007
2. Metrics & Measurements : Why
Objective Indicators for Product
Quality
Check Points at various phase
Watch Quality of Testing
Avoiding Subjective Decisions
3. Metrics : Why
Informative
Track the Progress of Quality
Improvement
Aiding Decisions
Are We Good to Enter / Exit a phase
Triggering Action Items
Bad Metrics : Fix Product Fast !
Are We Slipping? Then look where to
do better.
4. Essential Minimal Metrics…
K.I.S.S
Less Overhead in Collection of
Metrics
Discussed
Product Health Parameters
Ship Readiness Metrics
Not Discussed
Project Management Metrics
Process Efficiency Metrics
Applicable for Me?
No One Size Fits All Theory
Apply due judgment
5. Dev Phase : Code Metrics
Code Cyclomatic Complexity
Coding Violations DPMO
Automated Code Review
Guidelines & Practices
Naming, Comments, Indent
guidelines
Hard coded values, Strings
Uncaught Exceptions
Un-freed Memory
High Logical Nesting
Possible Bugs
Fall through Switch Cases
History Of Violations & Baselining
6. Dev Phase : Test Addition Metrics
Traceability Matrix : Tracks Metrics
Requirements
Features % Tests Added as per Planned Matrix
Sub Features Status of Test Addition
Test Scenario % Tests Per Feature
Test Case Reinforce Weightages of features on
Test Data Criticality.
Input for Planning Test Cycles for
Place Holder for Bugs / Changelist Ids
estimation of test execution tasks.
% New Tests Vs Legacy Tests
Test Bugs Fixed Vs Known Test Failures
7. Test Phase : Execution Metrics
Test Passes Covered
Functional
Non Functional
Tests Executed
Pass Ratio Progress
Regressions Rate
MTBC / MTBF Trends
Builds / Fixes Covered
Platforms / Languages Covered
8. Test Phase : Test Coverage Metrics
Code Coverage
Packages / Classes Coverage
Method Coverage
Statement Coverage
Condition / Path Coverage
Coverage Driven Test Development
Quality Check on Completeness of
Testing
9. Test Phase : Stability Trends
Stability T rends : Features
Most Stable / Buggy Features
70
Prioritize Testing on Buggy Features 60
Feature wise Regression Incidence 50
Most Fixed / Changed Features 40
Failures
30
Most Stable / Buggy Platform 20
Prioritize Testing on Platform Builds 10
Stability 0
1xxx20 1xxx21 1xxx22 1xxx23 1xxx24 1xxx25
Builds
Fe ature Wise Bugs
A
B
C
D
E
F
G
H
I
J
10. Test Phase : Prediction Metrics
Bug Glide Path
Prediction based on Find Rate, Fix Rate, Deferral Rate
Daily Update on Actuals
Predict Hitting Zero Bug Count Date
Open Bug Status & Prediction
(Includes actual open, new and needs Targeting)
70
60
50
40
30
20
10
0
12/15/06
12/22/06
12/29/06
01/05/07
01/12/07
01/19/07
01/26/07
02/02/07
02/09/07
02/16/07
02/23/07
03/02/07
03/09/07
03/16/07
03/23/07
03/30/07
04/06/07
Predicted Plan Combine Actual
11. Test Phase : Bug Metrics
Daily Auto Reminders
Bugs To Target
Bugs To Fix
Bugs To Test
Deferred Bugs
Bugs Incidence / Fix Trends
Average Life Time of Bugs
% of Re-Opened Bugs
% of False Alarms
12. Ready to Ship? : ZBC
Bug Glide Path
Is ZBC achieved from Release Candidate build
Any Spike in Open Bugs recently?
Any Crash Landing of the curve recently?
Open Bug Status & Prediction
(Includes actual open, new and needs Targeting)
70
60
50
40
30
20
10
0
12/15/06
12/22/06
12/29/06
01/05/07
01/12/07
01/19/07
01/26/07
02/02/07
02/09/07
02/16/07
02/23/07
03/02/07
03/09/07
03/16/07
03/23/07
03/30/07
04/06/07
Predicted Plan Combine Actual
13. Ready To Ship ? : Tested Enough?
Number of Test Passes
Functional Stability
Along the Builds
Along the Configurations
Along the Features
All Regressions Cleared?
All Failures Tracked?
Code Coverage
Have we Tested Enough?
14. Ready to Ship? : Bug Metrics
Are We Finding Less Severe Bugs ? Are We Finding Less Priority Bugs?
Bugs Incidence : Priority Based Bugs Incide nce : Se v e rity
25 20
18
20 16
14
15 P4 12 S4
P3 10 S3
10 P2 S2
8
P1 S1
6
5 4
2
0 0
15/02/2007 22/02/2007 01/03/2007 08/03/2007 15/03/2007 22/03/2007 15/02/2007 22/02/2007 01/03/2007 08/03/2007 15/03/2007 22/03/2007
Are We Still Fixing High Priority Bugs ? Are We Still Fixing Highly Critical
Bugs?
Fixing T re nd : Priority Fixing T re nd : Se v e rity Base d
20 20
18 18
16 16
14 14
12 P4 12 S4
10 P3 10 S3
P2 8 S2
8
P1 S1
6 6
4 4
2 2
0
0
15/02/2007 22/02/2007 01/03/2007 08/03/2007 15/03/2007 22/03/2007
15/02/2007 22/02/2007 01/03/2007 08/03/2007 15/03/2007 22/03/2007
15. Ready to Ship? : Bug Metrics
Found / Fixed / Deferred : Priority Based
250
Bugs Found vs Fixed vs Deferred
Are We Fixing the Right Bugs? 200
Are We Deferring the Wrong Bugs? 150 Found
Fixed
False Alarm
Deferral Rate : Are We Deferring Our Way 100
Deferred
to the Release? 50
0
P1 P2 P3 P4
Ov e rall Distribution of Found Bugs Found / Fixe d / De fe rre d : Se v e rity Base d
250
46
9%
44 200
9%
150 Found
Fixed
Fixed
False Alarm
False Alarm
Deferred 100
Deferred
50
411
82%
0
S1 S2 S3 S4
16. Ready to Ship ? : Non Functional Metrics
Performance Metrics (Product Dependent
Let us assume Web Application for Eg.)
Launch / Startup Time
Http Response Time
Concurrent Http Requests Handled
Users Serviced
Successful / Failed Requests
Average Transaction Time
Baseline / Goal Results Comparison
17. Ready to Ship ? : Non Functional Metrics
Reliability Metrics
MTBC / MTBF
Scalability
Memory Profiling / Foot print
Footprint Size for various platforms
Memory Usage : Startup, Under Load,
Under No Load
18. Ready to Ship ? : Other Checks
Security
List of Risks & Mitigated Risks
Memory Leaks
Found / Fixed
Size of Existing Leaks
Beta / Customer Feedback
Issues Reported / Fixed
Open Issues
Incidences Trend Since Beta
