Greg Smith, Sr. Verification Manager at Oracle

Abstract:
So much of today’s metrics used to gauge the progress of a verification project are backwards looking – telling us what ground we have covered. In addition, many metrics commonly in use are subjective and prone to human errors of omission.  I would like to present a different approach to DV project metrics using bug arrivals to actually provide some predictive capability as well as aid in overall project planning.

Download the Presentation Here
Download the “Sample Metrics” File Here

High Performance Collection of Coverage Metrics Using a Relational Database Backend

James Roberts, Sr. Verification Engineer at Oracle

Abstract:
A database is an ideal medium for collecting and analyzing coverage. At Oracle, we marry our Oracle database with coverage collection of our verification, and then use SQL to extract coverage metrics on-demand. This presentation outlines an intuitive scheme for database collection of coverage, and presents data showing the scalability and the high bandwidth this scheme is able to handle.

Download the Presentation Here

Written by Brian Bailey for DVClub

At DAC this year, one of the main themes was ESL but not in the usual sense of it having a lot of promise but little to deliver. This year it had a lot to say in two main categories, the first being high-level synthesis and the second being virtual platforms. Given the main focus of the DVClub, I will only talk about the virtual platforms. Quite a few companies were showing their platforms, including Mentor, Synopsys, CoFluent, CoWare and I am sure there were others. These platforms are at two main levels of abstraction.

At the higher end are platforms typified by the Synopsys Innovator which are primarily intended for software development, verification and debug. These are loosely timed platforms where speed is one of the primary factors. Then there are the more accurately timed platforms such as the Mentor Vista product which is intended for architectural exploration of the hardware system. Other companies such as Imperas also provide high performance processor models that fit into these platforms. The one thing common to most of them, and the main reason why they were such a force at DAC this year was the introduction of the OSCI TLM 2.0 specification at last years DAC. These platforms can now exchange models (although there are still some minor issues) and that is huge. A lack of models was perhaps the biggest reason why these platforms have not taken off. That roadblock has now essentially been removed.

Some new companies such as Docea were touting high-level power estimation platforms, and just for completeness, Mentor, Cadence, AutoESL, BlueSpec, Synfora, Forte and I am sure others were showing high level synthesis tools.

There was a panel session on Tuesday about virtual platforms that was one of the worst DAC panels I have ever sat through. It was supposed to address the issue of if platforms should be virtual, physical or hybrid. Ron Wilson tried hard to make it sound fun and interesting, but this is not a debate topic – we all want models in any form that we can get them in and we want them to play together nicely! End of debate – end of panel – nothing to discuss, just some solid engineering that has to happen.

On Wednesday, there was a much better conceived workshop on virtual platforms that I had been asked to speak at. The workshop was organized by Soha Hassoun and Larry Lapidas and included lots of interesting talks about platforms at many levels of abstraction and intended for many uses. Over lunch was a panel session that also had some much more interesting discussions. Sadly, I had to leave in order moderate a panel entitled “The Holy Grail of Verification – Coverage Closure”. Any of you who have listened to my DVClub talks will know that I have strong views on that issue, but unfortunately I was moderating so had to keep my mouth shut. Ouch that was difficult!

TLM 2.0 was finally ratified at DAC this year – I wonder if that will have a similar impact on next years DAC. I am hoping to see many more platforms which are extensible – add timing as a layer, add power as a layer, add X as a layer. Then we will have something that will play through multiple levels of abstraction and start to tie together the whole ESL flow.

Brian Bailey – keeping you covered
brian_bailey at acm.org

10th International Workshop on Microprocessor Test and Verification (MTV 2009)
December 7-8, 2009, Hyatt Regency On Town Lake, Austin, Texas, USA.

Website: http://mtv.ece.ucsb.edu/MTV/

This is the 10th edition of the MTV Workshop, a testament to its success in providing an ideal environment for cross- examination of test and verification experiences and innovative solutions. MTV has been held in Austin for the last 8 years, so please plan on participating in order to make this another successful forum.

Purpose

The purpose of this workshop is to bring researchers and practitioners from the fields of verification and test together to exchange innovative ideas and to develop new methodologies to solve the difficult challenges facing us today in various processor and SOC design environments. In the past few years, some work has been done on exploiting techniques from test to solve problems in verification and vice versa.

Topics

AREAS OF INTEREST include, but not limited to:

• Validation of microprocessors and SOCs
• Test/Verification of multimedia processors
• Performance testing
• High-level test generation for functional verification
• Emulation techniques
• Silicon debugging
• Formal techniques and their applications
• Verification coverage
• Test Generation at the transistor level
• Equivalence checking of custom circuits
• ESL Methodology
• Virtual Platforms
• Software verification
• Circuit level verification
• Switch-level circuit modeling
• Timing validation techniques
• Path analysis for verification or test
• Design error models
• Design error diagnosis
• Design for Testability or Verifiability
• Optimizing SAT procedures with applications to testing and formal verification

Important dates

Submission: Sept 1, 2009
Notification: Oct 1, 2009
Final version due: Nov 1, 2009

August drew our largest DVClub event yet in Silicon Valley with over 140 attendees coming out to listen to David Whipp of NVIDIA talk about his ideas on redefining how verification gets accomplished. If you missed his presentation, be sure to have a look his paper entitled “Stop Writing Assertions! Creating Efficient Verification Methodologies”.

In his presentation, Whipp makes some clever assertions and points out that “Verification is 70% of the problem” when it comes to chip design. Although this is widely accepted to be true, few verification engineers have done much to change this over the years. Of course, there are companies that offer various products, services, and methodologies to aid in the daunting challenge of verification, but few have sought to break down the verification process into smaller, more manageable components such as Whipp has done here.

Statement of the Problem

In the figure below, Whipp explains the typical workflow of the verification process. That is, a “big paper spec” is first written. Although it quickly becomes outdated, and hopelessly remains so, the spec is deemed vitally important as the team attempts to make it match the actual design. Furthermore, the verification department is tasked with solving all problems shown below in the red boxes. They must write the testbench, checkers, derive useful tests and so on.

Figure 1 – Bad (Traditional) Hardware Development Flow

A Possible Solution

Whipp’s proposed solution for this problem is to reconsider the way in which we approach verification. Rather than writing an enormously bloated paper spec that will be inevitably deemed obsolete, he states that it may be more beneficial to use an executable spec. My first thought is that this is cheating, but the more that I consider the details of doing this, I think that he may have something here. As you can see below, the paper spec does still exist, but it has been drastically reduced in size for manageability.

Figure 2 – Revised Flow

The red boxes above remain the responsibility of the verification team; however, the design team now steps in to handle the green boxes on the left. The remaining white boxes in the middle become shared tasks that both teams must work together on.

Going from 70% to 30%

How can we then reasonably expect to get all of the work done by doing less? The first step is to drop the detailed spec. This frees up the design team to do other things. In the traditional verification model, the natural language spec is never really maintained, and  DV engineers struggle to write checkers based on outdated specs, which is inherently problematic. By contrast, Whipp’s new model uses a very short spec.  Under this system, architects build a set of models at different levels of abstraction, creating and using them as executable models.

Overworking the Design Team?

The new executable spec will include multiple models at different levels of abstractions including ISSs, thread-based models, and structural models. 

The architecture team must then:

• build models
• make sure that they are correct
• connect them to standard interface

But will dropping the Big-Spec even out the workflow given these added tasks? It’s difficult to say and may depend on many other factors, but my first impression is that the architecture team should be more productive at these tasks. If not, all of this added work may create a backlash within some departments, and this could possibly become a difficult challenge to overcome. Design departments may or may not be equipped to undertake the added workload, and it is possible that personnel may need to be shifted between teams to match the newly distributed workflow. But if all of these things can be satisfied, this plan possesses the potential to streamline the verification  process and reduce time-to-market figures for the entire project.

Conclusion: ESL, Triage, and Debug

When comparing figures 1 and 2 above, it seems that a lot of the traditional verification effort is shifted to the architecture team and labeled “ESL”.  I have to admit, I’ve been hot and cold on ESL. It has the potential to be a really great idea with tremendous vision if implemented correctly, but the engineer in me has a hard time pinning down what exactly ESL is, and ambiguity has a way of making people nervous.

On the typical “BAD” flow, 70% of the boxes fall under DV.  On the new and improved flow, it is only 30%.  The first major change is ESL.  ESL is one of those great things that you can stick anything into. In this case it means that the architects build a useful high level model of the design that can be used by the entire team.  At Nvidia, this takes the form of the de-emphasizing the English specification and creating a transaction level specification that IS the spec.  In my experience it’s usually the case that most groups end up using the C model as the specification as the design documentation wilts throughout the project. The difference in this case is that using the C model is the goal from the beginning. Assertions are then added to the transaction model as part of the QA cycle.  Next, a structural C model is created that models the implementation and allows the assertions to be reused by adoption of common interface points.  The end result is a C transaction model and a C implementation model that can be reused in DV with all the architects’ assertions. This is the most interesting implementation of ESL that I’ve heard in a while.