1. Verification Metrics
Dave Williamson
CPU Verification and Modeling Manager
Austin Design Center
June 2006
1
2. Verification Metrics: Why do we care?
Predicting functional closure of a design is hard
Design verification is typically the critical path
CPU design projects rarely complete on schedule
Cost of failure to predict design closure is significant
2
3. Two key types of metrics
Verification test plan based metrics
Amount of direct tests completed
Amount of random testing completed
Number of assertions written
Amount of functional coverage written and hit
Verification reviews completed
Health of the design metrics
Simulation passing rates
Bug rate
Code stability
Design reviews completed
3
4. Challenges and limitations
Limitations of test plan based metrics
Will give a best case answer for completion date
The plan will grow as testing continues
Limitations of health of the design based metrics
Can give false impressions if used independent from test plan metrics
Requires good historical data on similar project for proper interpretation
General concerns to be aware of for all metrics
What you measure will affect what you do
Gathering metrics is not free
Historical data can be misleading
Don’t be a slave to the metrics:
they are a great tool, but not the complete answer
4
5. Bug rate example
Bug History
1200 20
Knee in curve
18
1000
16
Bug Rate Rolling Average
14
800
Total Bug Count
12
600 10
8
400
6
4
200
2
0 0
1
5
9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
101
105
109
113
Week number
Total Bug Count Weekly Bug Count (4wk rolling average)
5
