Communicating BUGS or BUGgy Communication

A few decades back when the designs had limited gate count, designers used to verify their code themselves. With design complexity increasing, the verification engineers were introduced to the ASIC teams. As Moore’s law continues to drive the complexity further, IP reuse picked up and this ensured that the engineers are spread all around the globe and communication across geographies is inevitable.

 

The reason for introducing the verification engineer was BUGS. A lot has been written and discussed on verification but reference to bugs is limited. With IPs getting sourced from different parts of the world and companies having extended teams everywhere, communicating the BUGs effectively becomes all the more important. “Wrong assumptions are costly” and in semiconductor industry this can throw a team out of business completely. Recently, in a mid-program verification audit for a startup with teams sitting between US & India, I realized that putting a well defined structure on communicating bugs could improve the turnaround time tremendously. Due to different time zones and working style, there was a lot of back & forth communication between the team members. Having a well defined format for communicating bugs helped a lot.

 

BUG COMMUNICATION CYCLE

 

BUG reported à BUG reproduced by designer à BUG fixed à FIX validated à BUG closed à revisit later for review, reproducing when required or data mining.

 

APPROACH

 

Before the advent of CRV, directed verification approach was common. Communicating bugs was simple and required limited information.  Introduction of CRV helped in the finding the bugs faster but sharing and preserving information around bugs became complicated. With a well defined approach, this problem can be moderated. Here is a sample format on what should be taken care of at each stage of the above cycle –

 

BUG reporting

 

Defining mnemonics for blocks or data path or scenarios etc. that can be appended along with the bug title helps. This enables categorizing the bugs at any time during project execution.

 

While the tool enforces adding the project related information, severity, prioritization and contact details of all concerned to broadcast information, the detailed description section should include –

 

- Brief description of the issue

- Information on diagnosis based on the debug

- Test case(s) used to discover this bug

- Seed value(s) for which the bug is surfaced

- Command line option to simulate the scenario and reproduce the issue

- Testbench Changelist/TAG on which the issue was diagnosed

- RTL Changelist/ TAG on which this issue was diagnosed

- Assertion/Cover point that covers this bug

- Link to available logs & dump that has the failure

 

BUG fixing

 

After the designer has root caused and fixed the bug, the bug report needs to be updated with –

 

- Root cause analysis and the fix

- Files affected during this fix

- RTL changelist/TAG that has the fix

 

FIX Validation

 

After the BUG moves to fixed stage, the verification engineer needs to update the TB if required and rerun the test. The test should pass with the seed value(s) required and then with multiple random seeds. The assertion/cover point should be covered in these runs multiple times. With this, the report can be updated further with –

 

- RTL changelist/TAG used to validate the bug

- Testbench changelist/TAG on which the issue was validated

- Pointer to the logs & waveforms of the validated test (if required to be preserved)

 

BUG Closed

 

After validating the fix, the bug can be moved to closed state. The verification engineer needs to check if there is a need to move this test to the smoke test list/mini regression and update the list accordingly.

 

Following the above approach would really help to communicate, reproduce and preserve the information around BUGs. The rising intricacies of the design demand a disciplined approach to attain efficiency in the outcome. Communicating bugs is the first step towards it!

 

HAPPY BUG HUNTING!!!

Laws and Verification

It’s Independence Day in India and I decided to use the freedom of applying some of the common laws to “Sid’dha-karana” aka “Verification" :)

LAW means “a generalization that describes recurring facts or events in nature” and engineering is full of laws. Though laws are formulated for a particular aspect of engineering, most of these are widely applicable to parallel areas too. The idea here is to salute the heroes who articulated these postulates and derive conclusions applicable to our field of interest.

The Laws

In semiconductor industry the most widely referenced one is Moore’s Law formulated by Intel co-founder Gordon E. Moore, who described the trend in his paper (1965) later generalized as – The number of transistors on a chip will double approximately every two years.

Another common adage quite popular is Murphy’s Law named after Edward A Murphy based on his conclusion after experiments in 1949 stating "If there's more than one way to do a job, and one of those ways will result in disaster, then somebody will do it that way." The law is more generalized since then as – Anything that can go wrong, will go wrong.

Next set of laws form the basis for classical mechanics popularly known as Newton’s law of motion compiled by Sir Isaac Newton in 1687. The laws are –

First Law - Every object in a state of uniform motion tends to remain in that state of motion unless an external force is applied to it.

Second Law – The relationship between an object's mass m, its acceleration a, and the applied force F is F = ma. Acceleration and force are vectors and the direction of the force vector is the same as the direction of the acceleration vector.

Third Law – For every action there is an equal and opposite reaction.

Another law commonly discussed with reference to multiprocessing is Amdahl's law named after computer architect Gene Amdahl. It states that the performance improvement to be gained from using some faster mode of execution is limited by the fraction of the time the faster mode can be used.

Finally the law of Natural Selection formulated by Charles Darwin in support of the theory of evolution stating that, some individuals survive and reproduce more often than others, and as a consequence their heritable characteristics become more common over time.

Applying to Verification

Moore’s law – Amount of code to be verified doubles every 2 years.

Murphy’s law – Any code that isn’t verified will not work.

Based on the above two laws, we can further derive a conclusion that total Verification effort doubles every 2 years where effort = Engg time + Tools + Flows + CPU cycles.

Newton laws –

First Law - Once the bug rate stabilizes, every code continues to be stable unless there is a change in external force. This force can be exerted by modifying constraints, introducing another methodology like assertions, formal etc.

Second Law - The improvement in rate of finding bugs is directly proportional to the net force applied and is inversely proportional to the total lines of code. This means, the effort, approach and outcome all depend on the F=ma relationship wherein, force is same as described in First law, m = lines of code and a is improvement in bug rate. In comparison to IPs, for large designs like SoCs, the effort is more while modifying the approach and the rate of finding bugs slows down for a given time.

Third Law - Every attempt to break the design results into more stable and mature design.

Amdahl’s law – When multiple verification techniques are applied, the overall gain (i.e. stability of code) is limited by the fraction of the time these techniques can be used e.g. CRV over directed, Formal over CRV, Manual tests vs auto generation of tests etc.

Law of Natural Selection – Change is inevitable and during the process of verification evolution, multiple approaches show up but only few find their footprint across design tape outs.

Drop in a comment with other laws you find applying to verification!

Happy Independence Day!!!

Verification claims 70% of the chip design schedule!

Human psychology points to the fact that constant repetition of any statement registers the same into sub-conscious mind and we start believing into it. The statement, “Verification claims 70% of the schedule” has been floating around in articles, keynotes and discussions for almost 2 decades so much that even in absence of any data validating it, we believed it as a fact for a long time now. However, the progress in verification domain indicate that this number might actually by a "FACT".

20 years back, the designs were few K gates and the design team verified the RTL themselves. The test benches and tests were all developed in HDLs and sophisticated verification environment was not even part of the discussions. It was assumed that the verification accounted for roughly 50% of the effort.

Since then, the design complexity has grown exponentially and state of the art test benches with lot of metrics have pre empted legacy verification. Instead of designers, a team of verification engineers is deployed on each project to overcome the cumbersome task. Verification still continues to be an endless task demanding aggressive adoption of new techniques quite frequently.

A quick glance at the task list of verification team shows following items –

- Development of metric driven verification plan based on the specifications.

- Development of HVL+Methodology based constrained random test benches.

- Development of directed test benches for verifying processor integration in SoC.

- Power aware simulations.

- Analog mixed signal simulations.

- Debugging failures and regressing the design.

- Add tests to meet coverage goals (code, functional & assertions).

- Formal verification.

- Emulation/Hardware acceleration to speed up the turnaround time.

- Performance testing and usecases.

- Gate level simulations with different corners.

- Test vector development for post silicon validation.

The above list doesn’t include modeling for virtual platforms as it is still in early adopter stage. Along with the verification team, significant quanta of cycles are added by the design team towards debugging. If we try to quantify the CPU cycles required for verification on any project, the figures would easily over shadow any other task of the ASIC design cycle.

Excerpts from the Wilson Research study (commissioned by Mentor) indicate interesting data (approximated) –

- The industry adoption of code coverage has increased to 72 percent by 2010.

- The industry adoption of assertions had increased to 72 percent by 2010.

- Functional coverage adoption grew from 40% to 72% from 2007 to 2010.

- Constrained-random simulation techniques grew from 41% in 2007 to 69% in 2010.

- The industry adoption of formal property checking has increased by 53% from 2007 to 2010.

- Adoption of HW assisted acceleration/emulation increased by 75% from 2007 to 2010.

- Mean time a designer spends in verification has increased from an average of 46% in 2007 to 50% in 2010.

- Average verification team size grew by a whopping 58% during this period.

- 52% of chip failures were still due to functional problems.

- 66% of projects conitnue to be behind schedule. 45% of chips require two silicon passes and 25% require more than two passes.

While the biggies of the EDA industry are evolving the tools incessantly, a brigade of startups has surfaced with each trying to check this exorbitant problem of verification. The solutions are attacking the problem from multiple perspectives. Some of them are trying to shorten the regressions cycle, some moving the task from engineers to tools, some providing data mining while others providing guidance to reduce the overall efforts.

The semiconductor industry is continuously defining ways to control the volume of verification not only by adding new tools or techniques but redefining the ecosystem and collaborating at various levels. The steep rise in the usage of IPs (e.g. ARM’s market valuation reaching $12 billion, and Semico reporting the third-party IP market grew by close to 22 percent) and VIPs (read posts 1, 2, 3) is a clear indicative of this fact.

So much has been added to the arsenal of verification teams and their ownership in the ASIC design cycle that one can safely assume the verification efforts having moved from 50% in early 90s to 70% now. And since the process is still ON, it would be interesting to see if this magic figure of 70% still persist or moves up further!!!

Choosing the right VIP

In past few months, while interacting with customers, I came across a couple of cases where the VIP played a spoilsport. In one case, the IP & VIP were procured from the same vendor during the early phase of standard protocol evolution. One of the key USPs of the product was this IP and the demonstration of the complete reference design was anticipated at a global computing event. The limitations of the VIP were revealed quite late during SoC verification. The team struggled hard to manage the bug rate in the IP itself during the SoC verification phase. Finally the product got delayed to an extent where the company (~startup) couldn’t manage to sustain and went for an asset sale. The problem wasn’t selecting the IP & VIP from the same vendor but lack of home work and maybe a wrong business decision during the selection process.

In another case, an in house developed IP having seen multiple tapeouts was getting validated on a new SoC architecture targeting a different application. The VIP was third party which had been used to verify the IP throughout. After integration testing at SoC level, while developing usecase scenarios, the limitation of the VIP came forward. Since the schedule didn’t allow the liberty to add features to the VIP and validate this scenario, the team went ahead with the tapeout. Unfortunately the application didn’t work on silicon and the root cause analysis revealed a bug in the IP itself. Result, re spin required.

Showstopper bugs, Project cancellations, Tapeout delays etc. all point to missing the market window and the post mortem may point figure to the VIP!!!

Selecting the right VIP demands a detailed evaluation process along with a comprehensive checklist to compare the solution from different vendors. To find available solutions in the market, www.chipestimate.com and www.design-reuse.com are quite useful. After identifying the set of applicable vendors, it is important to  collect relevant information for initial analysis –

- Detail documentation i.e. reference manual (architecture of VIP) and user manual (how to use it). These documents should list out all aspects of the VIP functionality including parameterizable sequences, component configurations, APIs, parameters, coverage, error messages etc.

- Understanding the process of VIP development from vendor i.e. how is the VIP validated, the maturity process, how many and what kind of design it has verified. The release process, bug reporting mechanism, turn around time for fixing bugs, commitment and plans to support the evolving standards.

- Vendor overall VIP portfolio, languages & methodology used, membership & participation to the relevant standards, development/support staff availability and domain expertise, level of onsite support possible.

- Standard compliance, compliance coverage report, items not supported in the current release, list of assertions, code & functional coverage report.

- Support across methodologies, simulators, accelerators, formal verification engines and ESL verification.

- Exiting customer references.

A detailed comparative study of the above data will help in narrowing down the search to final 1 or 2 vendors. The next phase involves qualitative analysis to validate the data provided by the vendor and the assumptions made during first phase by the evaluator. Here the engineer(s) needs to play around with the VIP using the examples provided by the vendor / test in target env / create a short circuit env with master-slave agents of VIP and try different scenarios to evaluate –

- Ease of use in terms of configuring the components, using the sequences, portability from block to system level, how would it work when simulated in parallel with other VIPs in the SoC verification environment.

- Does it meet the requirements of the metrics planned for determining verification completion and to what extent does the VIP inherently reports it by itself.

- APIs provided to develop tests for customized functionality of the IP, developing real life scenarios, directed tests, stress testing, errors injection & detection capability, debug and observability.

Interestingly the experience with the vendor during the evaluation phase indicates the level of support that can be expected further. The key to this two phase evaluation process is the checklist. VSIA provides a comprehensive QIP metrics (Final VSI edition) that provides a jump start to this evaluation process.

As it is said - "the devil is in the details", an additional effort during the start without overlooking the finer points can prevent from serious problems later. 

It would be interesting to learn your VIP evaluation experience. Drop a comment now!

Related posts -
Verification IP : Changing landscape - Part 1
Verification IP : Changing landscape - Part 2
 

Verification IP : Changing landscape - Part 2

In the part one of this document we discussed about VIP and the changing landscape. While the increasing demand of VIPs would drive more participants to enter this space, there are significant challenges to come up with a decent VIP portfolio.

Challenges for new entrants

- Which protocol(s)/standard(s)? SoC design houses are well aware of their requirements, but any other entity wanting to invest in developing the VIPs to license further needs to engage with potential customer(s) and build the team with right skill set accordingly.

- Which HVL and Methodology? Starting with an early customer helps in resolving this. However the risk is that the other potential customers may want a different HVL & Methodology. The current trends indicate that development with SV (IEEE 1800) and UVM is a safe bet.

- Expertise with domain/protocol and VIP development? Building a team without a design partner would mean bringing in domain experts along with VIP developers. Further the team needs to be around after implementation so as to work with customers for VIP deployment, fix bugs, modify based on customer requirement and upgrade as the standards evolve.

- How to validate the VIP? Initial clean up may involve short circuiting the Master & Slave agents of the VIP. To prove it golden, either it has to be validated with an already available golden VIP in the market (added cost) or working with a strategic customer who is willing to try out this solution on the already available golden IP or IP under development. Note that it takes significant cycles for the VIP to mature.

- Inter operability with tools? Once the VIP is working with one set of tools, it is important to evaluate it with other simulators. Porting of the VIPs to work on the emulators and formal tools (if applicable) should be planned next. Note that all tool vendors have alliance programs to facilitate such requirements.

- Additional cost? Apart from engineering & tool cost, membership to the standard bodies contributes to additional cost. Members are required to pay yearly fees for adopting the standard to develop a commercial product.

If new comers have a challenge to enter this arena, what should the existing partners of this ecosystem do?

SoC Integrators

When the focus is on rolling out SoCs, investing into in-house development of a VIP for standard protocol(s) doesn’t help in adding any differentiation to the end product. VIP development would incur cost towards tool licenses and engineers during and after development as resources would be required to support multiple projects deploying the VIP, fix bugs and upgrade the VIP as standards evolve at a fast pace. For big design houses getting a good deal with vendors would clearly direct ‘buy’ decision. For startups, utilizing resources to get the product out faster is more important than investing into VIP development. Getting into strategic engagement with VIP providers is a better solution.

EDA Vendors

Investing into VIP makes perfect sense for the EDA vendors for multiple reasons.

- The cost of development reduces with the tool sets available in house.

- The licensing model for the VIP is same as that of tools and complements the product portfolio quite well.

- The sales channel can be reused since the customer base is same.

- If the solution has limited competition it can open up sockets with new customers while making sure that the design houses sticking to vendor specific flow are maintained.

- The VIP(s) can be used as a test vehicle for validating the tools.

- With cloud computing picking up, a strong VIP portfolio will help in offering cloud based verification services quite easily.

Design Service Providers

For design service providers, it is good to invest in VIP development as it complements the services offered. Since they work with both SoC integrators and EDA vendors it makes sense for them to get into partnerships with either or both of these parties instead of developing a solution standalone. Sub system as an IP shall pick up soon. This is where the design services companies should invest by building a complete verification environment and test suite to verify these sub systems while partnering with the IP & VIP providers.

Changing landscape ....

Verification has been the forte for India. Whether to get the right skill set or reduce cost, the verification engineers from India have been involved at all levels i.e. VIP development, IP verification or SoC verification. Geographically, the changing landscape on the VIPs point the efforts to be converging from SoC integrators, EDA vendors and Design service providers in India.

Related posts -
Verification IP : Changing landscape - Part 1
Choosing the right VIP

Verification IP : Changing landscape - Part 1

For decades, EDA industry has been working out options to improve their offerings and ensure silicon success for the semiconductor industry. A few decades back, while the EDA giants were unknown, design automation was exercised individually in every organization developing a product. Gradually these tools moved out of the design houses and the ‘make vs buy’ decision carved a new league with EDA front runners. Talking specifically about verification, while the simulation tools were procured from outside, the in house CAD groups were still required to develop a flow for automation, self checking, regression management and data analysis. In the last decade, all of this came up as a package from EDA vendors thereby further squeezing the CAD teams in the design houses. So, what next?  Well, the recent acquisitions in this space i.e. Cadence acquiring Denali, Synopsys acquiring nSys and ExpertIO indicate where we are heading. Initially the VIP providers were able to sustain with licensing and basic verification support model but now the pressure is surmounting as VIPs get commoditized. The VIP only vendors will have to identify themselves either with the EDA vendors or with Design service providers wherein the VIPs complement other offerings. Before moving ahead let’s discuss what is a VIP?

What is a VIP?

A Verification IP is a standalone ‘plug and play’ verification component that enables the verification engineer in verifying the relevant DUT (design under test) module at block, sub system and SoC level. Based on the need, the VIP can act as a BFM to drive DUT signals or MONITOR the signals and VALIDATE them for correctness and data integrity. It may have a set of protocol CHECKERS and test scenarios to confirm compliance with the standards (if any) or COVER groups identifying corner cases and test completeness. VIPs are developed using HDLs or HVLs with standard methodology or as a set of assertions. It needs to have enough APIs providing flexibility for modifying the internal units as per DUT. User should be able to extend the available monitors, scoreboard and checker hooks while developing a verification environment around this VIP. Ease of developing scenarios at different levels using available functions and sequences is important in minimizing the verification turn around cycle. Quick integration with other tools like coverage, simulate, debug and transaction analysis is critical in successfully deploying the VIP.  Finally the most crucial is the support provided by the VIP vendor to the user in responding to issues and upgrading the VIP as the standards evolve.

The common VIPs available include –

- MIPI protocols like DSI, CSI, HSI, SlimBus, Unipro, DigRF & RFFE.

- Bus protocols like AXI, AHB, APB, OCP & AMBA4.

- Interfaces like PCIexpress, USB2.0, USB3.0, Interlaken, RapidIO, JTAG, CAN, I2C, I2S, UART & SPI.

- Memory models & protocol checkers for SD/SDIO, SATA, SAS, ATAPI, DDR2/DDR3, LPDDR etc.

Companies providing the above VIPs fall into one of the below categories -

- EDA vendors

- VIP only providers (some of them offering IPs also)

- Design services companies

Although, there is a lot of IP development still continuing with the design houses, the motivation to procure VIP from outside is much higher for obvious reasons. 

In the next post we will discuss the challenges for new entrants that raise barriers to entry and role of Soc Integrators, EDA companies and design services organizations in this changing landscape.

Related posts -
Verification IP : Changing landscape - Part 2
Choosing the right VIP

 

Constrained Random Verification and Simulators

In response to my earlier post on CRV, Daniel Payne raised an interesting question on Semiwiki – “Is there any difference in choosing a VHDL, Verilog, System C or System Verilog simulator from Cadence, Synopsys or Mentor and then using the CRV approach”?

Before answering this, let’s quickly check the CRV approach. Constrained Random Verification includes choosing an HVL, defining the test bench architecture and developing constraints to generate legal random stimuli. When this test bench is used to simulate the DUT, a random seed value and a simulator become part of the verification environment. The seed helps in reproducing the failure only if the other inputs i.e. the test bench architecture (components hierarchy) and the set of constraints are constant. Any change to these inputs may lead to different results with the same seed value also. The random seed value and the constraints are fed to the constraint solver integrated in the simulator to generate random values.

Let’s assume the HVL used is System Verilog (IEEE 1800). The standard provides details on the data types, how to randomize, how to define constraints and output of expressions. It does not specify the algorithms involved and the order in which the constraint solver should generate the outcome. The implementation of the solver is left to the sole discretion of the simulator R&D teams. The available simulators from different vendors may also support multiple solvers that tradeoff between memory usage and performance. Therefore, a given input may not give the same results when run on –

- Simulators from different sources OR

- Different versions of the same simulator OR

- Using different solver from the same simulator.  

All the constraint solvers claim to solve standard puzzles like Sudoku, N-Queen problem etc. Some of them even claim to solve these standard problems in limited or minimum iterations. Although these claims are correct but its reproducibility entirely depends upon how the constraints are defined. The capability of any constraint solver to return results in a given time primarily depends upon the representation of constraints. For a given equation, the solver may sometimes time out, but if the constraints typifying the equation are restructured it gives results instantly.

So, what to look for in the simulator when implementing CRV approach?

All the industry standard simulators have stable solvers varying on performance and there is no standard way to benchmark them. However one should consider other important aspects too -

- Support of the data types planned for implementing the test bench.

- Ease of debugging the constraint failures.

Next important part is how to define clean and clear constraints that would aid the solver in giving out results quickly. Following the KISS (Keep It Simple Silly) principle sets the right ground. Split the constraints to smaller equations, particularly if the expressions involve multiple mathematical operators and data types. This will provide better control on the constraints blocks and help in solving these constraints easily. Defining easy to solve constraint would improve the predictability of the outcome i.e. the generated value to be in a legal range and how fast it can be generated.

Coming back to the main question, the answer is YES, different simulators will give different results (values and fastness) while using CRV approach. This will affect the turnaround time of regressions and coverage closure too. However, there is no standard benchmarking available on the constraint solvers from these simulators. It is up to the verification engineer how best he/she can define the constraints and turn the output to be solver independent.

NOTE – This doesn’t mean that solver will never hit a bottleneck. Even with well defined constraints, the constraint solvers fail to deliver the results sometimes.

2011 : Bidding Adieu

The year 2011 had its own set of twists and turns. A plethora of applications riding over interesting set of electronic gadgets became part of our lives. Consumption of electronics devices continued to rise and market dynamics were unpredictable more than ever. New process nodes stabilized thereby promising to shrink designs  and reduce power consumption further. The tsunami added to the woes but Japan defied the odds and showcased its ability to bounce back quickly. The sad demise of Steve Jobs was a big loss to the electronics world. Amongst all of this we experienced that market window for products is constantly tightening which means the turnaround time for designs needs to reduce incessantly. Since verification claims most of the cycles in ASIC design, the community took one step further in multiple areas to bridge this gap. Some of the highlights include -

- UVM : Final release happened with quick adoption.
- Steps taken to bring together the differences in UPF & CPF.
- Tools for generating assertions automatically adopted widely.
- Cloud computing discussions moved from closed rooms to open forums.
- Hardware acceleration jumped to mainstream.

 

Some interesting facts also came up. While we keep talking about next generation of verification technologies, the primitive techniques continue to dominate in parallel. For example, at ‘Siddhakarana’ the most popular post was “Gate Level Simulations” with more than thousand hits and still getting visitors on daily basis. This essentially means there are areas that need attention before they become a bottleneck in getting the designs shipped faster.

On this note, I thank you for your valuable suggestions and readership throughout 2011. I look forward for the same in 2012.

Wish you and your family "HAPPY & PROSPEROUS 2012".

:)

Constrained Random Verification : A case study

Continuing the discussion further on CRV, here is a case study where a performance intense core was verified from scratch.

GIVEN

- Architecture documents for each block and integrated core.

- Reference model for each block developed for architecture validation and early software development.

- Periodic deliveries of core tests from architecture validation team, reusable at block and core level.

- Enough compute cycles, tool licenses and access to hardware accelerator to speed up verification at core level.

- Timeline for Architecture to RTL freeze = 13 months.

APPROACH

Develop block level test benches targeting CRV and capable of reusing system level tests to ensure staged bring up of functionalities in parallel. Top level test bench would reuse monitors and assertions from block level and simulate system level tests (no CRV) only. Sign off criteria would be 100% functional coverage, Code coverage (Line, expression & FSM), assertion coverage at block level and toggle coverage at core level.

Why limit CRV at block level –

- Test simulation time at core level was expected to be 12-48 hours, throttling iterations and affecting schedule.

- Low probability of finding an RTL bug using CRV with focus on integration verification.

- Test cases coming from architecture validation team would cover most of the use case scenarios.

- Bring up of constrained random TB for core would require staged approach i.e. append 1 block at a time and stabilize that. Verification closure in given time was unpredictable.

EXECUTION

The estimated effort was 200 man months and the team started off with 1 engineer per block taking CDV approach. After test plan, block level test benches were developed using standard methodology and code shared for similar functionalities. The test benches became functional using system level tests executing at block level. While RTL debugging started, coverage models were developed to realize the test plan and hooked onto the TB. Next the CRV support was added to the test bench. The random tests were initially directed towards areas yet to be explored by system tests to exercise DUT from all fronts. When the system tests stagnated for a data path within the block, the random tests continued to weed out bugs and hit scenarios for coverage closure. By the time system tests started passing for individual blocks, the team was ready with the core level TB. Every engineer finishing block level started contributing to top level. Tests at core level consumed a lot of simulation cycles (i.e. idle time for engineers) so a couple of engineers focused on CR TB for adjacent blocks (choking point from performance perspective) to regress the RTL further. No specific coverage goals were planned at this level and the emphasis was to hit random scenarios. Finally the team was able to deliver quality RTL with acceptable slippage in schedule.

CONCLUSIONS

Following CDV and standard methodology for CRV helped in defining, tracking and achieving goals in an organized manner. Maintenance and rework claimed limited overhead. Reusing components from block to top level was smooth with minimum efforts.

With CRV TB, the bug rate increased drastically, averaging to 7 bugs per week per block for at least 6 weeks. The scenario generation increased exponentially while tests delivered by architecture validation team continued to be incrementing linearly. The team was able to uncover hidden bugs and hit corner cases difficult to cover otherwise. Performance bottleneck scenario generation was easily achievable through CRV.

The CRV TB for adjacent blocks revealed interesting outcome. Converging on the valid constraints was challenging. The set of valid constraints at block level required additional constraints guided by introduction of a new block in the DUT data path. This TB was able to uncover 3 bugs, 2 with software work around and 1 critical where the two blocks would hang in a particular scenario. The latter was difficult to catch through system tests.

Lesson learnt – If you are unable to introduce CRV at top level, try to deploy it to the level possible!

Constrained Random Verification : + and -

In the last decade, adherence to Moore’s law demanded ‘divide and conquer approach’ for developing SoC/ASIC. The design cycle now requires develop/procure IPs; build sub systems using them and integrate these sub systems further to realize the final product. Some IPs (Networking protocols, Graphics, Video, DSP… etc) are complex enough that further division into blocks during design and verification is unavoidable. Amidst all this, verification still is the biggest challenge in meeting schedules and taping out bug-free products. The ever increasing design complexity problem has enabled rapid development in ASIC verification. From traditional directed verification approach to Constrained Random Verification (CRV), it has been a long way. CRV brought a paradigm shift in the way we verify our designs and enabled development of Coverage Driven Verification (CDV) and Standard methodologies (eRM, RVM, AVM, VMM, OVM & UVM).

ADVANTAGES

+ Enables the test bench to hit corner cases & hidden (unplanned) bugs

+ When coupled with CDV leads to faster convergence of verification schedule  

+ Number of test cases to be developed / maintained is fewer as compared to directed test

+ Updating verification environment for changes in specifications during the design cycle is better organized

+ Methodical approach during development enables reusability from module to higher levels

+ Improves Portability across projects

+ Test bench reviews are more structured and probability of finding issues increases

+ Following a standard methodology helps in knowledge transfer across engineers & teams

+ Increases compute resource & license utilization

LIMITATIONS

- Requires a reference/Golden model to predict the output

- Requires lot of planning and structured code development (Methodology is key)

- In absence of CDV it would hard to define ‘are we done yet’

- During TB development, RTL bug rate is 0 and initially TB bug rate may be > RTL bug rate

- Converging on valid constraints when moving from IP to higher level involves iterations

- With increased design scope debugging random scenarios is challenging

- Limited randomness possible while verifying at SoC level with processor(s) in place.

- Gate level simulations require a set of directed test cases where CRV doesn’t contribute much

- In absence of adequate licenses and compute resources it stagnates to deliver desired results

Undoubtedly CRV has been instrumental in improving the quality of verification and getting organized verification closure. Most of the above limitations can be nullified with proper planning during the initial phase of any project. For IP level, CRV is a de-facto standard. However, as we move one level up, the limitations start influencing. Teams are forced to think if CRV will be utilized to its true potential? Should we follow CRV for sub-system and top level? What are the bottlenecks in extending CRV to increased scope of the design & is it worth the investment?

Coming next is a quick case study. Stay tuned…

Mythology & Verification

It’s festive season in India! During this period one can experience the rich & diversified culture across the country. Apart from the fun and celebrations, this is the period when Indian mythological stories come across very frequently before everyone. While these stories convey subtle facts, rules and maxims to guide our daily lives, they also add meaning to these celebrations. All these stories related to Gods and demons are based on certain basic premises and are usually filled with some common concepts and ideas. It is considered that in every epoch as time passes the demons become prominent disturbing the regular functionality of the world. While these demons are on a rise, God himself arrives on this earth in different avatars (incarnations). He would master the art of war, build his arsenal with all sorts of weapons and finally start his quest to hunt all the demons and bring back the lost balance to this world.  

While pondering over these various stories a thought struck my mind. Isn’t this whole scheme the same in ASIC world? In every ASIC design, as the time passes and the RTL design gets ready, the bugs become prominent disturbing the regular functionality of the ASIC. While these bugs are on a rise, the verification engineer comes into picture in different roles (IP verification, SOC verification, Formal…). He would have mastered the art of verification and ready with his arsenal equipped with various techniques (constraint random verification, coverage, assertions, power aware verification, AMS, GLS…) to hunt down the bugs hiding in the design. At the end of this cycle he would restore the balance i.e. desired functionality of the ASIC and move on to the next one.

Verification surely has come a long way. I still remember the initial days of my career when every fresher (engg. graduate) wanted to be a designer and verification was considered to be a 2^nd option. However in the past decade the rising complexity of the designs have improved this role and placed it in the centre. During the recession (2009), my ASIC team had to face layoffs. Even during that period the verification engineers were able to manage more than 1 job offers in limited time.  The demand in verification is continuously on a rise. Now the verification engineer probably develops more lines of code (in comparison to RTL) to build a state of art test bench and uses multiple techniques to get the job done. The verification cycle directly affects both the schedule and quality (first Si success) of the product. The involvement has expanded to TLM modeling, Virtual platform development, functional verification, GLS, ATE vector generation and bring up keeping the engineer busy throughout the ASIC design process.   

So, to the members of the verification fraternity - while you are busy bringing balance to the ASIC functionality, enjoying the festive season and fascinated by the stories, make sure that your arsenal is always well equipped with the tricks and techniques in your pursuit towards Bug Hunting. As Albert Einstein rightly said –

“No amount of experimentation can ever prove me right; a single experiment can prove me wrong.”

Happy Dussehra! Happy Bug Hunting!

PS : This post is an experiment towards bridging technical and off topic subjects. Please provide your feedback by posting a comment or voting (Just 2 clicks).

Scoreboard

Recently while reviewing a test bench architecture with a group of verification engineers, an inveterate though interesting debate started. The engineer proposing the verification plan had ‘scoreboard’ as well as ‘checkers’ as two different components in his test bench. The debate was on the clear definition of these 2 terminologies as every engineer had his own understanding of these words.  Though various verification methodologies have termed the test bench components differently, still names like driver, bfm, generator and sequencer etc. reveal a lot about their functionality with the exception of scoreboard. Engineers across the globe have varied opinion on the history of this term and its origin as a verification component is untraceable.

Here is a quick preview of the various definitions of ‘what is a scoreboard’?

A scoreboard is a large board in a ballpark, sports arena, or the like for publicly displaying the score in a game or match and often other statistics, such as the count of the balls and strikes etc.

A scoreboard is a centralized hardware mechanism (initially used in CDC 6600 – 1964) used to keep a track on the activity (instructions being fetched, issued, executed etc. and the resources they use) inside a system that helps in determining certain decisions (dynamically scheduling instructions). Here the activities are tracked by flipping bits in scoreboard that keep a score of what is going on in the system.

These definitions are relevant to the scoreboard usage in verification. There are examples of primitive test environment where testing as an activity was maintained using an array of registers (scoreboard) representing tests to be executed. The registers would maintain if a test ran and the status of the result. On completion of the test suite the scoreboard would help generate the statistics and determine next steps. Scoreboard however has evolved from tracking & decision making into multiple avatars wherein it can be as simple as a data structure or as complicated as having the transfer function (predictor) and compare function included. So far in my experience I haven’t seen scoreboards performing protocol compliance checks so I believe my colleague was right in proposing scoreboard as an entity, predicting and comparing while the protocol checks to be handled by a separate component he referred to as checker.

Later that day while reading an intriguing article on context aware applications I came across this definition of context which states that –

“Context is any information (the parts of a written or spoken statement that precede or follow a specific word or passage, usually influencing its meaning or effect) that can be used to characterize the situation of an entity. An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and applications themselves.”

With the dilemma on the real intention of the user who coined this word (scoreboard) in verification domain or where it was first used I could only conclude that the definition of scoreboard is relative to the context it is used in. The context could be a methodology that defines this term clearly or the test bench architecture where it is used. If you know the functionality of the other components in the test bench (which you can fairly conclude by their names) the rest can be assumed to be taken care by scoreboard J

Check out what Scoreboard means in UVM @ verification academy.

Drop in a comment to share your experiments with the scoreboard definitions!

Gate Level Simulations : A Necessary Evil - Part 3

The response to the initial 2 posts (Part 1, Part 2) clearly show that GLS touches the career profile of almost all verification engineers. I am wrapping up this 3 part series on GLS with some recommendations that might be of some help if you aren’t already deploying them. Please do drop in an email or comment to share your tips & tricks.

To start, it is important to identify features targeted for GLS in the test plan upfront. Adding functional coverage/assertions to these features is a good practice. To set up GLS, develop the Verilog/VHDL TB (with DUT instantiation) from scratch so as to avoid any exceptions/force statements that might just pass from RTL simulations to GLS. If there is a provision, setup a process to develop/tune test cases for GLS such that all HDL code (DUT, TB & test cases) is compiled once and using ‘plusargs’ different tests can be simulated. The SDF should be compiled once and used for all tests. For initial netlist, whether you use zero delay/unit delay/pre layout SDF, disable timing check and have few smoke tests for sanity checks on the netlist. In case of large gate count, implementation team would be using hierarchical synthesis. If the hierarchy is logical you may want to try segregating GLS test points that are local to the components in hierarchy and try out GLS at that level. For designs that take some initial time to synchronize before the actual feature testing starts (particularly in networking domain), the tests can be developed such that multiple features are clubbed in 1 test. Some level of test case rewriting will save a lot of time during simulations.

Following is a basic common list of tests that most of the teams have in the GLS test suite –
1. Initialization tests i.e. chip coming out of reset and functional in different boot modes
2. Clock, Reset generation and clock dividers
3. Power aware simulations – covering features based on the power format specification
4. Test structures
5. Digital - Analog boundaries
6. Dedicated tests for timing exceptions in the STA
7. Stimulus to each block of the design atleast once

In order to have a controlled rework on the hierachical paths for force statements or monitors, pull out all paths in 1 HDL file under compile directives (e.g. `define in Verilog). This file can be included in the TB and modified/reviewed for every netlist release.

To control the SDF timing on synchronizers –
Enforce naming rules for such modules so that with a perl script you can turn off timing in SDF or through tcl interface to the tool turn off timing checks on selective instances. Where naming convention is not followed, there are some interesting ideas in the following papers –
- My favorite DC/PT TCL tricks, SNUG San Jose 2003
- SoC Gate Level Simulations (GLS) cycle time reduction – Simulation flow enablers, SNUG India 2008.
- Identification of Non-resettable flops for faster Gate Level Simulation, SNUG India 2010.

For debugging, simulation tools have some interesting features that can help narrow down the problem to some extent. A couple of them are – tracing the source of X propagation and expanding time step to highlight the order of execution. With some basic perl scripting you can triage the timing violations report into bins and then chase it accordingly.

Most of the above recommendations have worked well across multiple projects. It would be interesting to know if they hit any limitations in the design space that you work on so that we can discuss further on those specific corner cases for these recommendations.

Hope some of these would help you in your next tryst with GLS.
Happy BUG hunting!!!

Gate Level Simulations : A Necessary Evil - Part 1
Gate Level Simulations : A Necessary Evil - Part 2