The market window of opportunity for electronic products is gradually shrinking. This directly affects the design cycle of SOCs going into these products. Aggressive schedules leave no margin of error. Since verification claims most part of the design schedule it becomes all the more important to improve efficiency and weed out redundancies wherever possible. Mini regression, ubiquitous to all ASIC projects is a subset of stable tests to ensure sanity when –
- RTL was fixed for a bug.
- TB was fixed for issues.
- Before main regression to confirm that the database is clean.
- Precedes check-in (configuration management).
Considering the above list, mini regression assumes an important role throughout the project. If not defined logically, this can invite inefficiencies in the process and affect schedule. Imagine if the tests in mini regression don’t exercise the common functionality of the design that gets affected due to a bug fix. The next regression run is a complete waste of resources - both simulation cycles and debugging efforts which otherwise could have been avoided.
So, how to define a mini regression?
Coverage driven verification has been religiously adopted by all new project starts. An essential part of this approach is to run coverage frequently as compared to legacy approach of directed scenario based verification where coverage was run in the later part of the projects. If you are working on a legacy test suite then you already have enough tests in place to run the coverage at the start itself. Verification tools have supported ‘coverage grading’ for a long time. Coverage grading helps you identify the tests contributing maximum to the coverage goals. If you are working at the block/IP level the focus of the coverage would be on the functionality of the block and once you move to subsystem or SOC level, the integration verification becomes the focus. To analyze the coverage grading, it is important to make sure that the coverage goals are in line with the scope of verification. With this list in hand, engineers can easily identify the list of tests that have the maximum reach to the functionality of the design thereby improving the effectiveness of mini regression.
But, hold on; is this criterion good enough to select the tests? Remember, mini regression is used quite frequently. Another important aspect to be considered is the total turn-around time for mini regression. A test running for a long time can contribute maximum to coverage. However this test may not be the best choice for the mini regression as the total time to execute this test will be quite long and waste resources otherwise. So now we have two parameters, ‘coverage grading’ and ‘simulation time grading’ and we need to identify a compromise between the two to define a mini regression suite. Finally, since the main regression keeps on growing as the verification progresses, it is important to update the mini regression test list periodically.
With design complexity increasing, verification is becoming even more complex. Efficiency is vital to accomplish the given task in short time. A well planned mini regression is one of the keys to improve that efficiency.
Hi Gaurav,
ReplyDeleteI disagree quit a bit with the rational that coverage grading and simulation time are the only factor for choosing the mini/sanity regression test suites.
some cases can even have lower coverage grading and long simulation time compartively, even these have to be part of the mini/sanity regression.
Hi Udit,
ReplyDeleteThank you for your comment.
Let's analyze the conditions when one would need a test with lower coverage grading and long simulation time is -
1. The feature(s) covered by this long test is not covered by any other shorter tests in the suite.
2. The feature(s) indicated above are critical and affect functionality of the verification scope grossly.
3. It is a critical bug fix that is under consideration.
For #2 if the features are critical it would be good to spend some time and add a couple of redundant tests that hit the feature early enough.
For #3 it should be temporary in the mini regression.
However if you still see an exception to the above condition I would be interested in getting more details on that and discuss further.
Hi Guarav, Udit:
ReplyDeleteGuarav, you've presented a great summary, and it is no doubt a useful technique to grade on coverage and runtime. But I think Udit has a point. There are lots of conditions for which you may want to grade tests. I co-founded a startup, and our tool addresses this problem among others. Our philosophy is collect everything and then data mine at-will. In our customer visits we've seen requests for:
1) Test grading by bugs (not just critical bugs)
2) Test grading by line based and toggle based coverage (not just functional coverage)
3) Architectural coverage (think instruction level vs. transaction level)
4) Legacy directed tests that you don't necessarily consider critical, but your father's father's father who did verification ran those tests, and by God, you should too. (aka risk mitigation)
5) Regressing according to files changed (picking the right test for the task at hand, not for the task in general).
We also have customers whose regressions get rather large. Test grading (at any level) is useful, but segmenting that regression up - especially in an software-agile like fashion - may return larger benefits than spending time on grading. This is distinct from #5 above in that you may have individual teams picking the regression rather than
Udit - I'm interested - what other conditions have you seen for picking tests?
Eric Peers
Co-Founder
www.missinglinktools.com
Hi Eric,
ReplyDeleteThank you for sharing the insight.
Yes the summary above is more generalized approach and includes the solutions readily available with engineers. As you pointed out there can be 100s of temporal needs during the project life cycle where the your tool could be a savior at multiple levels.
Let's discuss more about the possibilities.
Thanks,
Gaurav Jalan