Negative Testing in functional verification!!!

Imagine someone on an important call and the mobile device reboots suddenly! The call was to inform that the devices installed at the smart home seems to be behaving erratically with only elderly parents & kids to provide any further details. On booting up, the smartphone flashes that there has been a security breach and data privacy has been compromised. Amidst this chaos, the car’s cruise control didn’t respond to pressing of the pedals!!! Whew!!!.... nothing but one of the worst nightmares in the age of technology we live in! But what if some of it could be true someday? What if the user has little or no idea about that technology?

The mobile revolution has enabled a common man to access technology and use it for different applications. The data from Internet world statistics suggest that internet adoption worldwide has increased from 0.4% of world population in 1995 to 54.4% in 2017. Related data also indicate that a sizable portion of the users are aged & illiterate. The ease of use has potentially driven this adoption further with the basic assumption that devices would be functioning correctly 24x7 even if used incorrectly out of ignorance. The same assumptions are seamlessly getting extended to safety critical domains such as Medical & Auto introducing several unknown risks for the user.

So how does this impact the way we verify our designs?

Traditionally, verification is assumed to be ensuring that the RTL is an exact representation of the specifications. Given that the state space based on the design elements is so very huge, a targeted verification approach covering positive verification has been in practice all throughout. Here, Proof of no bug is assumed to be equal to No proof of bug! The only traces of anything beyond this approach include –

- Introducing asynchronous reset during the test execution to check that the design boots up correctly again.

- Introducing stimulus triggering exceptions in the design.

- Simulating architecture or design deadlock scenarios.

- Playing around with key signals per clock for low power scenarios and reviewing the corresponding design response.

But as we move forward with security and safety becoming key requirements of the design, is this good enough? There is a clear need to redefine the existing approach and bring Negative testing to mainstream! Negative testing ensures that the design can gracefully handle invalid inputs, unexpected user behavior, potential security threats or defects such as structural faults introduced while the device is operational. Amidst shrinking design schedules, negative testing really requires creative thinking coupled with focused effort. 

To start with, it is important to question the assumptions used while defining the verification plan for the design. Validating those assumptions itself can lead to a set of scenarios to be verified under this category. Next, review the constraints applied while generating stimulus to list out potential illegal inputs of interest. Caution should be taken in defining this list as the state space would be large. Reviewing it in the context (Context Aware Verification) of end application would surely help in narrowing down this illegal stimulus set. Further to this, faults need to be injected at critical points inside the DUT using EDA tools or innovative testbench techniques. This is important for safety critical applications where the design needs to respond to random faults and exit properly while notifying about the fault or even correct it. Of course not to forget that appropriate coverage needs to be applied to measure the reach of this additional effort.

As we step into an era of billions of devices empowering humans further, it is crucial that this system of systems is defect free especially when it touches safety critical part of our life. Negative testing is a potential way forward ensuring reliability of designs for such applications. As is always said – 

Better safe than sorry!

Portable Stimulus : redefining verification play field yet again!

In the last 3+ decades, verification has come a long way! Right from quick testing by designers to dedicated verification teams moving from directed testing to constrained random and adding elements of formal apps at times, it has been an eventful journey! Standardization of UVM enabled a common framework with which the fraternity marched forward in sync with each other. Horizontal reuse viz. at the IP level experienced maximum benefits of UVM while vertical resume viz. from IP to SoC level observed limited returns. Apart from the methodology, verification has also proliferated beyond simulation or emulation into virtual prototyping, FPGA validation, post silicon functional validation & HW SW co-verification. Today, the reuse aspects are not limited from IP to SoC or across projects but between platforms too. It is extremely important to realize reuse of efforts at any level across different vehicles enabling first silicon success. The challenge however is that each of these vehicles involve multiple stakeholders like architects (Chip , SW, system), SystemC modelling engineers, RTL designers, verification engineers, prototyping experts, post silicon debuggers and SW developers, each defining & driving the stages of SoC design cycle. Different goals focusing on a specific stage, different approaches in solving these problems and different means of representing solutions has made the task of reuse across the platforms – a convoluted puzzle!!!

To solve this problem, Accellera initiated a task force called Portable Stimulus Working Group (PSWG) that reviewed the concern & potential solutions. After long & regular sessions of intense activity in last couple of years, the group has come up with a proposal in form of a standard.  Beta release of the preliminary version of the Portable Test and Stimulus Standard is now opened for public review.

The basis of the solution relies upon taking the stimulus definition across platforms to an abstraction layer where the focus is on what is needed instead of how it shall be implemented? The idea is to understand & represent the action/outcome i.e. the behavior of the DUT when the test runs. While representing these actions, the focus would be on what would be the inputs and outputs, what resources are required in the DUT, and their relationship with each other? The how part i.e. the implementation will be left to a hidden layer (read EDA tool) to generate the required stimulus for the target platform based on the scenario definition. The actions referred above are all declarative and not procedural or executable by themselves. A set of these static actions can be used to construct a scenario, analyzed for coverage to determine the paths to be traversed & dumped into a test format using the hidden layer.  

To represent these actions, the PSWG has proposed 2 formats - Domain Specific Language (DSL) that is close to System Verilog and a restricted C++ library. Both these formats have equivalent semantics that are completely interchangeable such that for each line in the DSL there is an exactly equivalent C++ library entry. If one defines the actions of an IP in DSL and another IP in C++ library, both can be read together to generate a SoC level scenario for the target platform.

While the road of moving from developing each testcase to a testbench that generates tests has been long, it’s time to take another step in the direction of standardizing the stimuli! This would feed the testbenches for IP/SoC verification & be reused by other workhorses of verification & validation in the SoC design cycle. Remember, reuse is one of the keys to keep up with Moore's law!!!

The public review period for this proposal is open until Friday, March 30, 2018. Download & review NOW!!!