This article is reposted from the Cadence blog.

Can verification engineers gain control over the verification process, and stop being full-time firefighters? With proper planning, communication, and organization, the answer is “yes,” according to Allison Goodman, validation program manager at Intel for client and enterprise solid state hard drives.

Goodman spoke at a Silicon Valley DVClub lunch meeting January 26 at Dave and Buster’s restaurant in Milpitas, California. DVClub is an interesting organization. With chapters in Austin, Bangalore, Boston, Dallas, Research Triangle Park, San Diego, and Silicon Valley, the club’s stated purpose is “to have fun while helping build the verification community through quarterly educational and networking events.” IC engineers can join for free, and events are free. Costs are picked up by sponsors, including Cadence.

The January 26 event brought together around 120 attendees. There were a few EDA folks, but as far as I could tell, most attendees were verification engineers. Goodman’s speech was entitled “Tales from the trenches – validation missteps making us full time firefighters.” Goodman started her speech by noting that “it’s not technical problems that cause bad things to happen. It’s usually on the people side.” She identified four “missteps” that force engineers to put out fires rather than proactively validate a product’s quality.

Misstep #1: Insufficient planning

Insufficient planning occurs when you don’t have what you need to do testing, and your test coverage falls short. It’s caused by undocumented assumptions, the increasing scope of projects, and “missed dependencies” (you need 10 prototypes but only get 5). “If you don’t plan for it, it will surprise you, and every surprise will end up as a fire.”

The solution? Put your plan in writing – including who does what, how features work, what it means to be “done,” what checkpoints will monitor progress, and criteria for success. Keeping track of assumptions may be the biggest part of the solution. Write them down!

Misstep #2: Not designing for test

Designers often think their designs won’t have any mistakes, so there’s no plan for testing and no communication with validators. This makes it difficult to find and replicate bugs, to figure out what you need to monitor, and to know when you’re done. Interpreting test results as “pass” or “failure” may be very difficult. The antidote is for validators to get involved in the earliest stages of the design process. “Ask how you’re going to test it and how you’re going to tell if it’s working.”

DVClub provides an opportunity for networking as well as speakers and lunches.

Misstep #3: Not creating and integrating feedback loops

All too often, the marketing team or the design engineers make changes to a product, and don’t communicate those changes to the verification team. Further, many companies place engineers in “silos” with little or no communication – for example, there are software engineers, hardware engineers, and firmware engineers who don’t talk to each other.

What’s needed is continuous feedback about any changes in the product, as well as problems found with the product. Tests should be monitored for effectiveness and continually improved.

Misstep #4: Lack of transparency

Lack of transparency happens when you tell your boss (or team) that everything is well when it really isn’t. Or, you skimp on tests and coverage as schedule pressure rises, and don’t let managers know. As a result, risks and coverage gaps increase. “Tell the real story, and encourage others to do the same. Don’t declare that it’s done until it’s really done.”

My takeaway

While there are tools that can help with verification planning and monitoring – such as Cadence Incisive Enterprise Manager – quality verification depends on “people” factors such as whether and how verification teams plan, how early they’re involved with the design process, how well and how honestly people communicate, and how adaptable teams are to feedback and change. Pay attention to these issues and perhaps you can put the fire extinguishers away.

Richard Goering

10th International Workshop on Microprocessor Test and Verification (MTV 2009)
December 7-8, 2009, Hyatt Regency On Town Lake, Austin, Texas, USA.

Website: http://mtv.ece.ucsb.edu/MTV/

This is the 10th edition of the MTV Workshop, a testament to its success in providing an ideal environment for cross- examination of test and verification experiences and innovative solutions. MTV has been held in Austin for the last 8 years, so please plan on participating in order to make this another successful forum.

Purpose

The purpose of this workshop is to bring researchers and practitioners from the fields of verification and test together to exchange innovative ideas and to develop new methodologies to solve the difficult challenges facing us today in various processor and SOC design environments. In the past few years, some work has been done on exploiting techniques from test to solve problems in verification and vice versa.

Topics

AREAS OF INTEREST include, but not limited to:

• Validation of microprocessors and SOCs
• Test/Verification of multimedia processors
• Performance testing
• High-level test generation for functional verification
• Emulation techniques
• Silicon debugging
• Formal techniques and their applications
• Verification coverage
• Test Generation at the transistor level
• Equivalence checking of custom circuits
• ESL Methodology
• Virtual Platforms
• Software verification
• Circuit level verification
• Switch-level circuit modeling
• Timing validation techniques
• Path analysis for verification or test
• Design error models
• Design error diagnosis
• Design for Testability or Verifiability
• Optimizing SAT procedures with applications to testing and formal verification

Important dates

Submission: Sept 1, 2009
Notification: Oct 1, 2009
Final version due: Nov 1, 2009

August drew our largest DVClub event yet in Silicon Valley with over 140 attendees coming out to listen to David Whipp of NVIDIA talk about his ideas on redefining how verification gets accomplished. If you missed his presentation, be sure to have a look his paper entitled “Stop Writing Assertions! Creating Efficient Verification Methodologies”.

In his presentation, Whipp makes some clever assertions and points out that “Verification is 70% of the problem” when it comes to chip design. Although this is widely accepted to be true, few verification engineers have done much to change this over the years. Of course, there are companies that offer various products, services, and methodologies to aid in the daunting challenge of verification, but few have sought to break down the verification process into smaller, more manageable components such as Whipp has done here.

Statement of the Problem

In the figure below, Whipp explains the typical workflow of the verification process. That is, a “big paper spec” is first written. Although it quickly becomes outdated, and hopelessly remains so, the spec is deemed vitally important as the team attempts to make it match the actual design. Furthermore, the verification department is tasked with solving all problems shown below in the red boxes. They must write the testbench, checkers, derive useful tests and so on.

Figure 1 – Bad (Traditional) Hardware Development Flow

A Possible Solution

Whipp’s proposed solution for this problem is to reconsider the way in which we approach verification. Rather than writing an enormously bloated paper spec that will be inevitably deemed obsolete, he states that it may be more beneficial to use an executable spec. My first thought is that this is cheating, but the more that I consider the details of doing this, I think that he may have something here. As you can see below, the paper spec does still exist, but it has been drastically reduced in size for manageability.

Figure 2 – Revised Flow

The red boxes above remain the responsibility of the verification team; however, the design team now steps in to handle the green boxes on the left. The remaining white boxes in the middle become shared tasks that both teams must work together on.

Going from 70% to 30%

How can we then reasonably expect to get all of the work done by doing less? The first step is to drop the detailed spec. This frees up the design team to do other things. In the traditional verification model, the natural language spec is never really maintained, and  DV engineers struggle to write checkers based on outdated specs, which is inherently problematic. By contrast, Whipp’s new model uses a very short spec.  Under this system, architects build a set of models at different levels of abstraction, creating and using them as executable models.

Overworking the Design Team?

The new executable spec will include multiple models at different levels of abstractions including ISSs, thread-based models, and structural models. 

The architecture team must then:

• build models
• make sure that they are correct
• connect them to standard interface

But will dropping the Big-Spec even out the workflow given these added tasks? It’s difficult to say and may depend on many other factors, but my first impression is that the architecture team should be more productive at these tasks. If not, all of this added work may create a backlash within some departments, and this could possibly become a difficult challenge to overcome. Design departments may or may not be equipped to undertake the added workload, and it is possible that personnel may need to be shifted between teams to match the newly distributed workflow. But if all of these things can be satisfied, this plan possesses the potential to streamline the verification  process and reduce time-to-market figures for the entire project.

Conclusion: ESL, Triage, and Debug

When comparing figures 1 and 2 above, it seems that a lot of the traditional verification effort is shifted to the architecture team and labeled “ESL”.  I have to admit, I’ve been hot and cold on ESL. It has the potential to be a really great idea with tremendous vision if implemented correctly, but the engineer in me has a hard time pinning down what exactly ESL is, and ambiguity has a way of making people nervous.

On the typical “BAD” flow, 70% of the boxes fall under DV.  On the new and improved flow, it is only 30%.  The first major change is ESL.  ESL is one of those great things that you can stick anything into. In this case it means that the architects build a useful high level model of the design that can be used by the entire team.  At Nvidia, this takes the form of the de-emphasizing the English specification and creating a transaction level specification that IS the spec.  In my experience it’s usually the case that most groups end up using the C model as the specification as the design documentation wilts throughout the project. The difference in this case is that using the C model is the goal from the beginning. Assertions are then added to the transaction model as part of the QA cycle.  Next, a structural C model is created that models the implementation and allows the assertions to be reused by adoption of common interface points.  The end result is a C transaction model and a C implementation model that can be reused in DV with all the architects’ assertions. This is the most interesting implementation of ESL that I’ve heard in a while.