Tracking Verification : are we done?

Growing design complexity has turned verification into an NP-hard problem. Achieving 100% verification is unrealistic. The approach followed finds an interesting corollary here. Verification methodologies are continuously evolving to answer the most important questions before the verification team i.e. where are we now and are we done? Deployment of different metrics limit the verification space for a project and help in measuring the progress. Some of these include –

- Functional coverage

- Code coverage

- Formal/Assertion based coverage

As design size grew, the divide and rule policy entered. Divide the system into IP or sub systems so as to verify them thoroughly as a separate entity. Further plug-in these entities into the system and verify integration. The coverage metrics can track progress from the entity to system level.

Are these coverage metrics good enough indicators to measure progress or should the verification teams look out for more such pointers?

Other useful indicators include –

- Tracking Bug rate on - RTL, VIP/Golden model and Test bench

- Tracking version control update frequency

- Regression failure rate

- Correlating the above with similar complexity projects in the past and predicting accordingly.

What else could point to the progress?

- Status on Use cases scenarios in functional simulation.

- For hardware assisted acceleration, results of simulations with real life data.

- For prototyping, results of executing drivers/APIs with real life data.

- Gate sims passing with post layout SDF on different delay parameters.

- Server load and license usage. 

More ….

:) Designer’s tasks - blessing coverage waivers instead of fixing bugs

:) Email frequency and subject/content  of emails between team members.

:) Engineers demanding compensatory off or planning vacations

Verification dashboards available in market use some of the above pointers to indicate the progress. However, there is a need for next level, something like a cockpit that provides the control to the user to extract the progress with all possible indicators and mitigate the risk on the schedule and quality.