Assertion-based verification has many advantages, but is not particularly easy to use. At Silicon Valley DVClub April 26, two engineers discussed the benefits and challenges of assertions, and described their experience with two tools that help answer the question, “who’s going to write all those assertions?”

DVClub (Design Verification Club), co-sponsored by Cadence, presents free educational and networking events at various locations in the U.S., Europe, and India. Presenters at the Silicon Valley DVClub luncheon were Jing Li, verification engineer at Broadcom, and Eric Deal, president of silicon IP provider Cyclic Design.

Li described Broadcom’s experience with BugScope, an “assertion synthesis” tool from NextOp Software, while Deal described his experience with Zazz, a tool from Zocalo that helps users create and debug SystemVerilog assertions. (Both companies are Cadence partners and both tools are closely integrated with the Cadence Incisive simulation environment. Last year I ran Industry Insights Q&A interviews with Yunshan Zhu, CEO of NextOp, and Howard Martin, president of Zocalo).

Broadcom: Challenges of Assertions

Li described a “traditional” verification flow at Broadcom that includes block-level testing, coverage signoff, subsystem testing, chip-level testing, and emulation. While this flow has been quite successful, she noted that “as design complexity increases, we’re finding bugs later than what we’d like to see. It’s an indication we need to improve the methodology so that at each level of verification, we have more visibility into what is being tested.”

Assertion-based verification (ABV) can provide that visibility, but has not been part of the Broadcom flow because “we have some issues that couldn’t be solved.” Li identified the following problems:

• Learning the SystemVerilog assertion (SVA) language and mastering assertion coding is difficult for engineers
• Assertions are time-consuming to debug
• Assertions may not directly match designer intent, resulting in false failures in simulation
• There’s no good way to measure the quality of hand-generated assertions
• It’s unclear how many assertions one needs to write
• Assertion reuse is a problem, with new assertions often needed even for small design changes

These challenges led Broadcom to evaluate BugScope. Li described how it automatically generates assertions based on regressions, and how designers then evaluate assertions to determine which are “true” assertions and which are functional coverage properties.

“We found that using this assertion synthesis technology helps improve the quality of block-level verification,” Li said. “For almost every block for which we tried BugScope, we were able to find bugs, and most of those bugs could not be found with the old flow. And we were able to find bugs even during the property review process.” All this is possible with very little change to the existing verification flow, she said.

Li provided four examples of bugs found with BugScope that would not have been detected without assertion synthesis. She described a bug that was found without running any tests at all, a bug hiding in a functional coverage hole, a bug that was not detected with manually generated assertions, and a bug that appeared only in emulation and could not be replicated with simulation or formal verification.

However, she also listed some improvements Broadcom would like to see, including generation of assertions for cross-module bugs, a GUI for the assertion classification process, and better performance with large numbers of instances. BugScope, she concluded, is “now officially part of our signoff criteria and is really increasing our verification confidence.”

Cyclic Design: Assertions for IP Verification

Eric Deal brought a different perspective to the DVClub meeting – he’s a designer, and he’s president of a company that specializes in error correction (ECC) IP for NAND flash. He’s long been a believer in ABV, and he noted a number of advantages of assertions. He said they can cut debug time, improve designer-to-verification engineer communications, document design behavior, detect unobservable faults, and ease integration of IP modules. On this last point, he said that assertions “really provide a lot of added value to my customers.”

Deal started using the Open Verification Library (OVL) some years ago when it was being standardized by Accellera. While easy to use, the assertions are simple and inflexible, and result in “messy” code when they get instantiated into modules. Then he learned SVA, and found that it provided more power and flexibility. However, he noted, it’s difficult to construct “anything beyond relatively simple assertions” with SVA.

Approached by a founder of Zocalo, Deal evaluated an early version of Zazz. The product has two big advantages, he said. First, its graphical Visual SVA environment makes it possible to create complex assertions without becoming an expert in SVA syntax. Secondly, and perhaps most importantly, Zazz provides a way to debug assertions at the time of creation. It does this by effectively creating a constrained-random testbench around each assertion, and generating a pass or fail waveform.

The impact on Cyclic Design? “It improved my internal verification and debug time by quickly identifying both the time and location of errors in simulation,” Deal said. Today the company ships assertions with its IP. The assertions help customers find problems in ports and interfaces, and provide insights not covered in the user’s guide. But customers must be educated to turn the assertions on.

Conclusion

Assertions are a powerful tool for designers and verification engineers, but writing assertions is a pain. For this reason tools from NextOp and Zocalo have attracted a good deal of interest. There’s no better way to learn about them than to hear directly from the users. Thus, I think this DVClub presentation was very timely. See the DVClub web site for information about upcoming presentations in various cities.

Richard Goering

The goal of DVClub events is to help build the verification community through quarterly educational and networking functions. We currently have active branches in Austin, Boston, RTP and Silicon Valley as well as international branches in Europe and India.

Events generally involve a free lunch followed by a technical program
and time allocated for networking. Topics vary, but the core focus is
end user verification stories, verification technology, and speculation
on our chosen career paths.

For more information on becoming a presenter, please contact us at: admin@dvclub.org

We think that the topic is super-interesting, so we’ve provided some preliminary reading material on the subject.

If you haven’t already registered to attend the presentation, we recommend doing so while there are still spots available.

• Hierarchical Simulation-Based Verification of Anton, a Special-Purpose Parallel Machine (PDF)
• A Conversation with David E. Shaw (PDF)
• Grand Challenge: Millisecond-scale Molecular Dynamics Simulations (PDF)
• Slashdot – Supercomputer Sets Protein Folding Record
• Wikipedia – Anton (Computer)

JOB POSTINGS UPDATED – OCT 29, 2010

The application deadline for the MTV Scholarship Program is now less than 30 days away. This scholarship will provide round-trip airfare, conference registration fees, and hotel accommodations for several students across the nation to attend the 11th annual workshop on Microprocessor Test and Verification in Austin, TX.

“It’s always difficult to have to choose applicants”, said Eric Hennenhoefer, President of Obsidian Software. “There are so many talented young engineers out there, and we like to help really determined and motivated individuals whenever we come across them.”

2010 marks the first year of this scholarship, which is open to applicants until October 1st. More information can be found by visiting the scholarship program page:

2010 Microprocessor Test and Verification Scholarship

The notoriety of DVClub is spreading organically by word of mouth – something that’s not easy to achieve among a community of engineers. “We continually see new people at each event” said DVClub founder and organizer Eric Hennenhoefer. “Once they’ve experienced a DVClub event, they tend to come back to future events and bring more co-workers with them”. The average quarterly growth rate for DVClub in 2010 has been around 7% per quarter and climbing.

About DVClub:

The principal goal of DVClub is to have fun while helping build the verification community through quarterly educational and networking events. Currently over 50 advisors actively keep the club organized by brainstorming on topics, tracking down speakers, promoting events, and helping out on event day.

This article is reposted from the Cadence blog.

Can verification engineers gain control over the verification process, and stop being full-time firefighters? With proper planning, communication, and organization, the answer is “yes,” according to Allison Goodman, validation program manager at Intel for client and enterprise solid state hard drives.

Goodman spoke at a Silicon Valley DVClub lunch meeting January 26 at Dave and Buster’s restaurant in Milpitas, California. DVClub is an interesting organization. With chapters in Austin, Bangalore, Boston, Dallas, Research Triangle Park, San Diego, and Silicon Valley, the club’s stated purpose is “to have fun while helping build the verification community through quarterly educational and networking events.” IC engineers can join for free, and events are free. Costs are picked up by sponsors, including Cadence.

The January 26 event brought together around 120 attendees. There were a few EDA folks, but as far as I could tell, most attendees were verification engineers. Goodman’s speech was entitled “Tales from the trenches – validation missteps making us full time firefighters.” Goodman started her speech by noting that “it’s not technical problems that cause bad things to happen. It’s usually on the people side.” She identified four “missteps” that force engineers to put out fires rather than proactively validate a product’s quality.

Misstep #1: Insufficient planning

Insufficient planning occurs when you don’t have what you need to do testing, and your test coverage falls short. It’s caused by undocumented assumptions, the increasing scope of projects, and “missed dependencies” (you need 10 prototypes but only get 5). “If you don’t plan for it, it will surprise you, and every surprise will end up as a fire.”

The solution? Put your plan in writing – including who does what, how features work, what it means to be “done,” what checkpoints will monitor progress, and criteria for success. Keeping track of assumptions may be the biggest part of the solution. Write them down!

Misstep #2: Not designing for test

Designers often think their designs won’t have any mistakes, so there’s no plan for testing and no communication with validators. This makes it difficult to find and replicate bugs, to figure out what you need to monitor, and to know when you’re done. Interpreting test results as “pass” or “failure” may be very difficult. The antidote is for validators to get involved in the earliest stages of the design process. “Ask how you’re going to test it and how you’re going to tell if it’s working.”

DVClub provides an opportunity for networking as well as speakers and lunches.

Misstep #3: Not creating and integrating feedback loops

All too often, the marketing team or the design engineers make changes to a product, and don’t communicate those changes to the verification team. Further, many companies place engineers in “silos” with little or no communication – for example, there are software engineers, hardware engineers, and firmware engineers who don’t talk to each other.

What’s needed is continuous feedback about any changes in the product, as well as problems found with the product. Tests should be monitored for effectiveness and continually improved.

Misstep #4: Lack of transparency

Lack of transparency happens when you tell your boss (or team) that everything is well when it really isn’t. Or, you skimp on tests and coverage as schedule pressure rises, and don’t let managers know. As a result, risks and coverage gaps increase. “Tell the real story, and encourage others to do the same. Don’t declare that it’s done until it’s really done.”

My takeaway

While there are tools that can help with verification planning and monitoring – such as Cadence Incisive Enterprise Manager – quality verification depends on “people” factors such as whether and how verification teams plan, how early they’re involved with the design process, how well and how honestly people communicate, and how adaptable teams are to feedback and change. Pay attention to these issues and perhaps you can put the fire extinguishers away.

Richard Goering

If you receive our newsletters, then you’re already pre-approved.

Today I attended the latest meeting of the Silicon Valley branch of the DVClub. For those not familiar with the DVClub (DV = Design Verification), it was started by Eric Hennenhoefer in Austin a few years ago. It now has branches in Austin, Bangalore, Boston, Bristol, Dallas, RTP, San Diego and Silicon Valley. In Silicon Valley it meets about once a quarter for a talk on some aspect of verification. I first heard of this about 1.5 years ago when we were invited from Tensilica to give a talk about verifying our video subsystem. The club has all the right ingredients to attract a crowd of engineers:

1. a free lunch
2. interesting speakers
3. did I mention a free lunch?
4. a chance to meet new colleagues and old friends
5. and of course, a free lunch

(Sponsors such as Cadence, Doulos, Denali, Silicon Elite and Obsidian pick up the tab for the venue and lunch (updated after original post, on Friday 20 March 2009, to correct list of sponsors)).

Today’s speaker was Brian Bailey, a friend and co-author of mine, speaking on “Is it time to declare a verification war?” The place was packed out with about 130 people, filling the room to capacity (Eric said this was the largest Silicon Valley DVClub crowd to date).

Brian spoke about his philosophy of verification, drew some analogies to Sun-Tzu’s Art of War, and also spoke about three technologies that he felt had potential to change verification significantly:

1. Functional Qualification – as exemplified by Certess (now SpringSoft) Certitude
2. Raising abstraction – as exemplified by Calypto’s sequential equivalence checking
3. “Intelligent testbenches” – as exemplified by Jasper’s Behavioural Indexing

Brian’s slides are available here.

IF you live or work anywhere any of these branches of the DVClub, and have an interest in verification, I recommend that you check them out. Sign up for their newsletter and get notified of meetings in advance.