Topics in Verification: Reuse, Coverage, Regression Engineering, Planning, Qualification
1. Welcome to RTPWelcome to RTP
DVClubDVClub!!
Topics in VerificationTopics in Verification
Pete LaFauciPete LaFauci
October 18, 2006October 18, 2006
2. AgendaAgenda
Brief Discussion/Overview aboutBrief Discussion/Overview about ““the Clubthe Club””
Historical Perspective of Languages & SimulatorsHistorical Perspective of Languages & Simulators
Topics in VerificationTopics in Verification
ReuseReuse
CoverageCoverage
Regression EngineeringRegression Engineering
PlanningPlanning
QualificationQualification
SummarySummary
3. DVClubDVClub OverviewOverview
Voluntary, Community Based Verification GroupVoluntary, Community Based Verification Group
Discuss Present Day ChallengesDiscuss Present Day Challenges
EDA neutralEDA neutral
Presentation ofPresentation of ““sharablesharable”” ideas, experiences, and resultsideas, experiences, and results
Advisory BoardAdvisory Board
Currently about 10 members, spanning 6 companies (looking forCurrently about 10 members, spanning 6 companies (looking for
additional participants)additional participants)
Planning, Logistics, Speakers, TopicsPlanning, Logistics, Speakers, Topics
Topics should be Chip Verification related, but can spanTopics should be Chip Verification related, but can span
adjacent areas:adjacent areas:
Design for Verification, IT, Project ManagementDesign for Verification, IT, Project Management
Please Contact Pete LaFauci or Justin Sprague regarding interestPlease Contact Pete LaFauci or Justin Sprague regarding interest
in the Advisory Board or Presenting atin the Advisory Board or Presenting at DVClubDVClub!!
4. Historical Perspective: WhatHistorical Perspective: What’’s dones done
Produce some test vector stimulus, and simulateProduce some test vector stimulus, and simulate
Look at the output and waveforms on workstation, debug the restLook at the output and waveforms on workstation, debug the rest in the labin the lab
Produce a test plan and lots of directed, procedural test casesProduce a test plan and lots of directed, procedural test cases
Run each test case in the test bucketRun each test case in the test bucket
Completion: each individual test case passesCompletion: each individual test case passes
Produce a verification plan, which includes modelProduce a verification plan, which includes model
requirements, functions & coverage goalsrequirements, functions & coverage goals
Combine generation capabilities, checkers, randomness, and coverCombine generation capabilities, checkers, randomness, and coverage monitors toage monitors to
gauge the simulationsgauge the simulations
Run tests with multipleRun tests with multiple ““seedsseeds””
Completion: all tests pass, and coverage output isCompletion: all tests pass, and coverage output is ““analyzedanalyzed””
Produce a verification methodology,Produce a verification methodology, ““architectarchitect”” aa
verification environment & plan, and leverage powerful toolsverification environment & plan, and leverage powerful tools
Automate both efficiency and thoroughness, through prediction, sAutomate both efficiency and thoroughness, through prediction, steering, and proofsteering, and proofs
VIP, Environment, Library, and TestVIP, Environment, Library, and Test ““ReuseReuse””
““RankRank”” the test suite efficiency, directed tests for coverage closurethe test suite efficiency, directed tests for coverage closure
Completion: all tests pass, coverage goals are met using measuraCompletion: all tests pass, coverage goals are met using measurable metricsble metrics
Then
Now
5. Historical Prevalence:Historical Prevalence:
Verification LanguagesVerification Languages
VHDL
Records / Dynamic
Variable Indices & Slicing
2-d ports
HVL – ‘e’ & Vera
Constraint Solvers
Easy RTL Access
Functional Coverage
Built-in libraries
C, C++. Perl
Complex Data Structs
Superior String Functions
Verilog
Concurrency
Tasks & Functions
PLI
Early ’90s
Late ’90s –
Early 00’s
Mid ’90s
Today / Tomorrow / Future?
Modeling – SystemC
Higher Abstraction, TLM, HW/SW
Algorithmic Modeling & Prototype
HVDL – SystemVerilog
RTL/HVL Integration/Unification
SystemC & TLM Interfaces
Assertion Based Verification
DPI
7. Reuse: Going Beyond Traditional VIPReuse: Going Beyond Traditional VIP
““ComponentsComponents””: traditional VIP: traditional VIP
Common bus and data protocolsCommon bus and data protocols
Monitors, Predictors, DriversMonitors, Predictors, Drivers
Environments:Environments: new(ernew(er) with OOP/HVL) with OOP/HVL
TestbenchesTestbenches && ““HarnessesHarnesses””
Constraints & Test CasesConstraints & Test Cases
Configuration ComponentsConfiguration Components
Addresses ScalabilityAddresses Scalability
“instance & program”
“inherit, derive, build, configure”
8. Reuse: Inheritance and AbstractionReuse: Inheritance and Abstraction
Base Class Packages
DUT templateTest Harness
Globals
MemoriesRegistersData Objects
Interface & Protocol
templates
Harness
Drivers
Scoreboard
User Test Case & Constraints
Environment Test Methods and User Interfaces
Test Benches/Harnesses
DUT VIP
Tool Packages: any_unit, any_env, any_sequence
Application
Specific
Verification
Environment
Company
Library
Inheritance
And
Instantiations
Base
Library
Protocol VIP
Interface VIP
Note : “Horizontal (Interface/Protocol) and Vertical (Base Classes and Methods) Reuse Strategies are
Deployable using OO Design Techniques”
11. Coverage: AchievingCoverage: Achieving
ThoroughnessThoroughness
Identify which Design Functions have not been exercisedIdentify which Design Functions have not been exercised
Identify What Test Code has not been simulatedIdentify What Test Code has not been simulated
Complements the Checking Code (traffic types, latency,Complements the Checking Code (traffic types, latency,
performance, etc)performance, etc)
Structural
Coverage
Assertions
Code
Coverage
Functional
Coverage
12. Coverage: Collection, Merging,Coverage: Collection, Merging,
and Reportingand Reporting
Project Level Coverage
Progress Reports
Regression 1
Test Cases,
Procedural
Code
Test Cases,
Procedural
Code
Test Cases,
Procedural
Code
Test Cases,
Procedural
Code
Simulation
Coverage
Output
(Cover DB)
Coverage Data
Merging
“Process”
Test Cases,
Procedural
Code
Test Cases,
Procedural
Code
Test Cases,
Procedural
Code
Test Cases,
Procedural
Code
Simulation
Coverage
Output
Regression n “Total Coverage”
Reports & HTML views
Test Cases,
Procedural
Code
Test Cases,
Procedural
Code
Test Cases,
Procedural
Code
Test Cases,
Procedural
Code
Simulation
Coverage
Output
(Cover DB)
13. Coverage: Challenges in CoverageCoverage: Challenges in Coverage
Driven MethodologyDriven Methodology
Goals are manually created from the engineeringGoals are manually created from the engineering
SpecsSpecs --> Labor Intensive> Labor Intensive
Random generation can be wasteful if overallRandom generation can be wasteful if overall
coverage is not increasingcoverage is not increasing
Simulation cycles can be slow and/or expensive,Simulation cycles can be slow and/or expensive,
especially if they are being wastedespecially if they are being wasted
Execution Predictability with Schedules &Execution Predictability with Schedules &
ResourcesResources
16. RegressionRegression ““EngineeringEngineering””::
ChallengesChallenges
Manage thousands of simulation jobs & resultsManage thousands of simulation jobs & results
Multiple environmentsMultiple environments
Many Servers, Lots of output!Many Servers, Lots of output!
Team environment, Individual OwnersTeam environment, Individual Owners
AnalysisAnalysis
Reproducing test fails easily & accuratelyReproducing test fails easily & accurately --> test/bench, machine, seed, tools,> test/bench, machine, seed, tools,
versionsversions
More debug output necessary?More debug output necessary?
Correctly assigned or delegated to appropriateCorrectly assigned or delegated to appropriate owner(sowner(s))
EfficiencyEfficiency
Dynamically Allocating Right Machines for the Right Job TypesDynamically Allocating Right Machines for the Right Job Types
Maximize Utilization of both Hardware and SoftwareMaximize Utilization of both Hardware and Software
Coverage Closure: Hole Analysis, RankingCoverage Closure: Hole Analysis, Ranking
24/7 rebalancing, including interactive session priority managem24/7 rebalancing, including interactive session priority managementent
18. Planning: Improvements in PlanPlanning: Improvements in Plan
Creation and Change ManagementCreation and Change Management
More Tightly Couple the Design Specification with theMore Tightly Couple the Design Specification with the
Verification PlanVerification Plan
Change Management SystemChange Management System
More Automatic Synchronization and NotificationMore Automatic Synchronization and Notification
Improve the Efficiency between Coverage Plan andImprove the Efficiency between Coverage Plan and
ImplementationImplementation
Reduce or Remove the Disconnect between the FunctionalReduce or Remove the Disconnect between the Functional
Coverage Code and the Coverage PlanCoverage Code and the Coverage Plan
Automate Functional Coverage CodeAutomate Functional Coverage Code ““GenerationGeneration””
20. Qualification: WhatQualification: What’’s this?s this?
What is Qualification?What is Qualification?
WhoWho’’s checking the verification, and when?s checking the verification, and when?
Error insertion to prove the accuracy and effectiveness of yourError insertion to prove the accuracy and effectiveness of your
verification systemsverification systems
Error insertion is placed in the DUT (not verification code)Error insertion is placed in the DUT (not verification code)
Challenges in adoption:Challenges in adoption: ““how, what, whenhow, what, when””, effort levels, effort levels
Benefits for SimulationBenefits for Simulation
Qualify the Test BenchQualify the Test Bench
Qualify the Models & VIPQualify the Models & VIP
Qualify the Test Cases & ConstraintsQualify the Test Cases & Constraints
Benefits for Formal ProofsBenefits for Formal Proofs
Qualify the AssertionsQualify the Assertions
Qualify the ConstraintsQualify the Constraints
*Qualify the Answer the Tool is Giving you**Qualify the Answer the Tool is Giving you*
21. SummarySummary
Combining innovative techniques, tools, processes canCombining innovative techniques, tools, processes can
yield significant results!yield significant results!
NotNot ““uncommonuncommon”” to achieve 10x productivity andto achieve 10x productivity and
thoroughness improvements through methodology andthoroughness improvements through methodology and
tuningtuning
Verification processes will continue to changeVerification processes will continue to change
considerably to keep up with managing the exponentialconsiderably to keep up with managing the exponential
growth in design complexitygrowth in design complexity
