Assertion-based verification has many advantages, but is not particularly easy to use. At Silicon Valley DVClub April 26, two engineers discussed the benefits and challenges of assertions, and described their experience with two tools that help answer the question, “who’s going to write all those assertions?”

DVClub (Design Verification Club), co-sponsored by Cadence, presents free educational and networking events at various locations in the U.S., Europe, and India. Presenters at the Silicon Valley DVClub luncheon were Jing Li, verification engineer at Broadcom, and Eric Deal, president of silicon IP provider Cyclic Design.

Li described Broadcom’s experience with BugScope, an “assertion synthesis” tool from NextOp Software, while Deal described his experience with Zazz, a tool from Zocalo that helps users create and debug SystemVerilog assertions. (Both companies are Cadence partners and both tools are closely integrated with the Cadence Incisive simulation environment. Last year I ran Industry Insights Q&A interviews with Yunshan Zhu, CEO of NextOp, and Howard Martin, president of Zocalo).

Broadcom: Challenges of Assertions

Li described a “traditional” verification flow at Broadcom that includes block-level testing, coverage signoff, subsystem testing, chip-level testing, and emulation. While this flow has been quite successful, she noted that “as design complexity increases, we’re finding bugs later than what we’d like to see. It’s an indication we need to improve the methodology so that at each level of verification, we have more visibility into what is being tested.”

Assertion-based verification (ABV) can provide that visibility, but has not been part of the Broadcom flow because “we have some issues that couldn’t be solved.” Li identified the following problems:

• Learning the SystemVerilog assertion (SVA) language and mastering assertion coding is difficult for engineers
• Assertions are time-consuming to debug
• Assertions may not directly match designer intent, resulting in false failures in simulation
• There’s no good way to measure the quality of hand-generated assertions
• It’s unclear how many assertions one needs to write
• Assertion reuse is a problem, with new assertions often needed even for small design changes

These challenges led Broadcom to evaluate BugScope. Li described how it automatically generates assertions based on regressions, and how designers then evaluate assertions to determine which are “true” assertions and which are functional coverage properties.

“We found that using this assertion synthesis technology helps improve the quality of block-level verification,” Li said. “For almost every block for which we tried BugScope, we were able to find bugs, and most of those bugs could not be found with the old flow. And we were able to find bugs even during the property review process.” All this is possible with very little change to the existing verification flow, she said.

Li provided four examples of bugs found with BugScope that would not have been detected without assertion synthesis. She described a bug that was found without running any tests at all, a bug hiding in a functional coverage hole, a bug that was not detected with manually generated assertions, and a bug that appeared only in emulation and could not be replicated with simulation or formal verification.

However, she also listed some improvements Broadcom would like to see, including generation of assertions for cross-module bugs, a GUI for the assertion classification process, and better performance with large numbers of instances. BugScope, she concluded, is “now officially part of our signoff criteria and is really increasing our verification confidence.”

Cyclic Design: Assertions for IP Verification

Eric Deal brought a different perspective to the DVClub meeting – he’s a designer, and he’s president of a company that specializes in error correction (ECC) IP for NAND flash. He’s long been a believer in ABV, and he noted a number of advantages of assertions. He said they can cut debug time, improve designer-to-verification engineer communications, document design behavior, detect unobservable faults, and ease integration of IP modules. On this last point, he said that assertions “really provide a lot of added value to my customers.”

Deal started using the Open Verification Library (OVL) some years ago when it was being standardized by Accellera. While easy to use, the assertions are simple and inflexible, and result in “messy” code when they get instantiated into modules. Then he learned SVA, and found that it provided more power and flexibility. However, he noted, it’s difficult to construct “anything beyond relatively simple assertions” with SVA.

Approached by a founder of Zocalo, Deal evaluated an early version of Zazz. The product has two big advantages, he said. First, its graphical Visual SVA environment makes it possible to create complex assertions without becoming an expert in SVA syntax. Secondly, and perhaps most importantly, Zazz provides a way to debug assertions at the time of creation. It does this by effectively creating a constrained-random testbench around each assertion, and generating a pass or fail waveform.

The impact on Cyclic Design? “It improved my internal verification and debug time by quickly identifying both the time and location of errors in simulation,” Deal said. Today the company ships assertions with its IP. The assertions help customers find problems in ports and interfaces, and provide insights not covered in the user’s guide. But customers must be educated to turn the assertions on.

Conclusion

Assertions are a powerful tool for designers and verification engineers, but writing assertions is a pain. For this reason tools from NextOp and Zocalo have attracted a good deal of interest. There’s no better way to learn about them than to hear directly from the users. Thus, I think this DVClub presentation was very timely. See the DVClub web site for information about upcoming presentations in various cities.

Richard Goering

For those of you unfamiliar with DVClub, membership is free and is open to all non-service provider semiconductor professionals. Most members work in verification, but there are also plenty of entrepreneurs, professors, students, managers, investors, and even design engineers who attend. If you’re interested and would like to learn more, why not join the club?

Chuck Alley, IBM

Using PSL and FoCs for Functional Coverage Verification

Bob Colwell, Intel (Retired)

The Validation Attitude

Raj Dayal, Qualcomm

Managing Deployment of SVAs in Your Project

Ish Kumar Dham, Texas Instruments

Design Verification to Application Validation of a Multiprocessor SoC

Sanjay Gupta, IBM

Cell Verification Metrics

Narasimha Karunakar, AMD

Low-Power Verification Challenges

Mark A Firstenberg, IBM

Experience with Formal Methods, Especially Sequential Equivalence Checking

Jai Kumar, Sun

Leveraging Low-Cost FPGA Prototyping for Validation of Highly Threaded Server-on-Chip

John Ludden, IBM

Mainline Functional Verification of IBM’s POWER7 Processor Core

Milind Padhye, Freescale

Wireless Low Power and Verification Challenges

Somdipta Basu Roy, Texas Instruments

OMAP Verification

Scott Runner, Qualcomm

Verification of Wireless SoCs: No Longer in the Dark Ages

Sakar Jain, Freescale

Verification of the QorIQ Communication Platform’s CoreNet Fabric with SystemVerilog

Shahram Salamian, Intel

Intel Atom Processor Pre-Silicon Verification Experience

CPU Verification Metrics

Jason Stinson, Intel

Pre-Si Verification for Post-Si Validation

Paul Tobin, AMD

Verification in a Global Design Community

Durgam Vahia, Sun

Mapping Server-Class Multi-Threaded OpenSPARC T1 Processor Core on FPGAs

David Williamson, ARM

Verification Metrics

Paul Zehr, Intel

Intel Xeon Pre-Silicon Validation

Complexity

In his recent presentation discussing verification of the Power7 processor, John Ludden of IBM opened with a quote from an IBM exec more than a decade ago. “it’s not rocket science”- a perception held by some members of the management and design communities at that time.

However, designs have become a whole lot more complex over time. The Power7 processor at 45nm has 1.2B transistors on a 567 sq. mm die, supporting 8 cores with 4 threads each, an on-chip eDRAM, 3 levels of caches and 2 DDR memory controllers. Yet as verification complexity multiplies in this multi-threaded design, it’s very helpful to have some of the more advanced tools and methodology at your disposal.

Tools and Methodology

Fortunately for Ludden and the Power7 team, IBM has invested in verification technology for years (in spite the quote from the exec). The company continues to develop and rely on in-house tools for many of the advanced verification technologies for processor-specific testing. These include the test-bench, multi-thread test generators, hardware accelerators, formal and semi-formal tools, micro-architecture checkers (API based), cache coherency checkers and coverage tools. Exercisers
originally developed for post-silicon validation were used to exploit the hardware acceleration platform. Forty-five thousand coverage points were organized to assist with big picture and were used to re-direct the test generator and exercisers for accelerators.

To support corner case testing for events that occur rarely, especially in multi-threaded scenarios, software irritator threads were used. These irritators are capable of creating the worst possible contentions. Through their application, twenty-three high quality bugs were revealed hiding in the corners.

A methodical application of these tools and technology clearly captured and advanced the industry best practices.

Designing for Verification

Designing for Verification was an important element in managing the overall risk to verification time line. IBM minimized the risks by maintaining a tight interaction between the specification and verification teams during the design phase and allowing the verification team to maintain architectural changes. “Chicken switches” were placed in silicon that allowed verification team to back-off an area considered risky or possible of otherwise compromising the verification effort. These switches provide workarounds, with some small impact on performance but no functional change, for accessing difficult to verify micro architectural features. Hardware irritators were also used to enable stress testing of corner cases in both pre-silicon and post-silicon testing.

Conclusion

The Power7 draws many architectural features from the Power5 and 6 designs, although it is a much more complex and powerful processor with a much shorter verification cycle. Ludden and the Power7 team accomplished this remarkable feat with a lot of foresight in planning, metrics collection and careful execution. Tight interlocking between metrics collected and verification plan was key part of tracking mechanism and functional closure. This project should serve as an example of how to plan for and manage risks in a complex verification project.

Kudos to John and the IBM team. His full presentation can be downloaded here.

Normal Accidents : Living with High-Risk Technologies
Charles Perrow

A History of Murphy’s Law
Nick T. Spark

Inviting Disaster: Lessons From the Edge of Technology
James R. Chiles

Fluid Concepts And Creative Analogies: Computer Models Of The Fundamental Mechanisms Of Thought
Douglas R. Hofstadter

The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA
Diane Vaughn

Conferences aren’t my favorite events to attend. They tend to be dominated by the big three EDA companies, and the messages are usually just a variation on what was said last year. However, there is always something useful to glean if you listen hard enough, and I think DVCon this year is no exception.

While DVCon is generally more of a verification conference, I found design related topics surprisingly absent. Cliff Cummings presented a good paper on using SystemVerilog’s unique, priority, and 1800-2009′s unique0 constructs, but other than that, everything centered on verification except for some brief discussion on C synthesis at a panel and the SystemC synthesizable subset at the OSCI tutorial session. Verification continues to dominate the industry’s focus as well as high-level modeling.

In fact, I felt that the major topics at DVCon this year were verification methodologies (VMM & OVM), TLM 2.0, and SystemVerilog. I’ll just say a brief word on each.

Both VMM and OVM have recently been updated. Synopsys has added significant features to VMM in their 1.2 release. Doulos sponsored a VMM 1.2 tutorial along with other VMM Central partners highlighting the new features like TLM 2.0 support, implicit phasing, and enhanced testbench structure and configuration as well as explaining how to exploit the RAL register package. In conjunction, Doulos gave away their new VMM 1.2 Golden Reference Guide and has made available a VMM 1.2 tutorial on their website. OVM is also recently updated (version 2.1), but it hasn’t majorly changed so the story is still much the same.

The SystemC NASCUG meeting was co-located with DVCon and there seemed to be a lot of interest around TLM 2.0. OSCI also hosted a TLM 2.0 tutorial session and there was a user paper session centering on TLM. VMM’s TLM 2.0 implementation generated a bit of interest as well. While I don’t use TLM for SystemC modeling, given all the buzz about it I have to conclude that it’s being well-embraced by the industry and it looks like it’s here to stay. I certainly find TLM connections quite useful in an OVM/VMM testbench.

Personally, I found the most interesting papers were those discussing SystemVerilog. Dave Rich from Mentor proposed a multiple class inheritance enhancement, which seems to have great potential. Cliff Cummings talked about enhancing the language to handle X optimism and pessimism. Eduard Cerny discussed new SV-2009 checker and assertion features. But I have to admit, the nagging question I have is, “Will this language everstop exploding?” If I may say, SystemVerilog is like an ever-expanding patchwork, where piece after piece is added but none of it ever seems to truly fit together. And every year, more and more ideas are proposed to enhance it. Oh well, I guess it’s what we have to live with. For those not converted yet to SystemVerilog, my colleague, Alan Fitch, wrote in his DVCon paper, “How to Achieve Sample-Based Coverage Using VHDL” — quite a unique topic among all the other presented papers. Keep an eye out on the Doulos website for the upload of his paper if you’re interested. I usually write papers that show how to work with or around what we already have. That’s why I presented a paper on matching asynchronous behaviors using SystemVerilog assertions (soon to be uploaded to the Doulos website), and likewise, my colleague, John Aynsley, presented a great paper on using the DPI to interface with C/C++ models.

I think the most exciting news at DVCon this year came from Accellera. Accellera’s Verification IP (VIP) technical subcommittee has announced that a universal verification methodology (UVM) is planned for release mid-March. UVM will be based on OVM 2.0.3 and have features of VMM incorporated into it. The amazing thing is that Synopsys, Cadence, and Mentor are all unanimously behind UVM. I think this will definitely reshape the verification methodology story in the industry over the coming year. I was also pleased to hear that the unified coverage interoperability standard (UCIS) is due out in October. This should give us a common way to access and merge all of our coverage data. Lastly, I was rather surprised by the take-away message from Brian Bailey’s panel on minimizing verification time and effort—engineers need more training!! As a trainer, I couldn’t agree more!

Doug Smith

This article is reposted from the Cadence blog.

Can verification engineers gain control over the verification process, and stop being full-time firefighters? With proper planning, communication, and organization, the answer is “yes,” according to Allison Goodman, validation program manager at Intel for client and enterprise solid state hard drives.

Goodman spoke at a Silicon Valley DVClub lunch meeting January 26 at Dave and Buster’s restaurant in Milpitas, California. DVClub is an interesting organization. With chapters in Austin, Bangalore, Boston, Dallas, Research Triangle Park, San Diego, and Silicon Valley, the club’s stated purpose is “to have fun while helping build the verification community through quarterly educational and networking events.” IC engineers can join for free, and events are free. Costs are picked up by sponsors, including Cadence.

The January 26 event brought together around 120 attendees. There were a few EDA folks, but as far as I could tell, most attendees were verification engineers. Goodman’s speech was entitled “Tales from the trenches – validation missteps making us full time firefighters.” Goodman started her speech by noting that “it’s not technical problems that cause bad things to happen. It’s usually on the people side.” She identified four “missteps” that force engineers to put out fires rather than proactively validate a product’s quality.

Misstep #1: Insufficient planning

Insufficient planning occurs when you don’t have what you need to do testing, and your test coverage falls short. It’s caused by undocumented assumptions, the increasing scope of projects, and “missed dependencies” (you need 10 prototypes but only get 5). “If you don’t plan for it, it will surprise you, and every surprise will end up as a fire.”

The solution? Put your plan in writing – including who does what, how features work, what it means to be “done,” what checkpoints will monitor progress, and criteria for success. Keeping track of assumptions may be the biggest part of the solution. Write them down!

Misstep #2: Not designing for test

Designers often think their designs won’t have any mistakes, so there’s no plan for testing and no communication with validators. This makes it difficult to find and replicate bugs, to figure out what you need to monitor, and to know when you’re done. Interpreting test results as “pass” or “failure” may be very difficult. The antidote is for validators to get involved in the earliest stages of the design process. “Ask how you’re going to test it and how you’re going to tell if it’s working.”

DVClub provides an opportunity for networking as well as speakers and lunches.

Misstep #3: Not creating and integrating feedback loops

All too often, the marketing team or the design engineers make changes to a product, and don’t communicate those changes to the verification team. Further, many companies place engineers in “silos” with little or no communication – for example, there are software engineers, hardware engineers, and firmware engineers who don’t talk to each other.

What’s needed is continuous feedback about any changes in the product, as well as problems found with the product. Tests should be monitored for effectiveness and continually improved.

Misstep #4: Lack of transparency

Lack of transparency happens when you tell your boss (or team) that everything is well when it really isn’t. Or, you skimp on tests and coverage as schedule pressure rises, and don’t let managers know. As a result, risks and coverage gaps increase. “Tell the real story, and encourage others to do the same. Don’t declare that it’s done until it’s really done.”

My takeaway

While there are tools that can help with verification planning and monitoring – such as Cadence Incisive Enterprise Manager – quality verification depends on “people” factors such as whether and how verification teams plan, how early they’re involved with the design process, how well and how honestly people communicate, and how adaptable teams are to feedback and change. Pay attention to these issues and perhaps you can put the fire extinguishers away.

Richard Goering

This article presents an overview of functional design verification using a coverage driven methodology while attempting to answer the question of how much testing is enough. The part being verified in this case will be a general purpose microprocessor, such as those found in mobile computing devices. Note that an approach of this magnitude is not always required. Designs with very limited instruction sets or highly restricted functionalities may be satisfied by simply writing directed assembly code tests to verify their intended functionalities.

Comparison of Simple and Complex Architectures

Figure 1 depicts a simple architecture as compared to a complex one. Note that the number of corner cases and unpredictability of the verification space increases as the architecture gains complexity. Thus, the complexity of the architecture determines how much testing will need to be accomplished to properly verify the component’s function.

Figure 1. Comparison of Verification Spaces

Measuring Verification Progress

Coverage metrics are the dominant method for measuring verification progress in the industry today. Coverage points are normally designated by the design engineers looking at the logic of their block and by verification or system engineers looking at the functional definition of the part. Both of these are critical insights into the required verification coverage of the design.

Coverage points, indicated by the red dots in Figure 2, are deliberately chosen with respect to placement and density according to design knowledge and risk assessment.

Figure 2. Distribution of Coverage Points

Directed Testing

In the past, directed tests were typically written to hit coverage points. Because directed tests are by their very nature highly targeted and relatively inflexible, this resulted in much of the design not being tested as is shown by the ratio of red to gray in Figure 5. In addition to the low overall coverage that results from this approach, creation of directed tests is time consuming and requires highly skilled engineers. In this approach, testbench checkers that detect hits to coverage points are often overlooked with the assumptions that the engineers writing the tests know how to hit the required coverage points and that human errors will not be significantly problematic. In addition, as the design changes over the course of development, the directed test may lose track of its target coverage point. Without coverage monitors, these types of errors will not be detected and the design will not be as thoroughly verified as it appears to be on paper.

Using a Random Test Generator to Close Coverage

As processor designs became more complex, the need to hit more coverage points became apparent. Once the grid has been established, large numbers of purely random tests may be incorporated to begin closing coverage. Some of these tests may hit points on the coverage grid while others will not.

Figure 3. Intersection of Coverage Grid and Pure Random

Approaching the problem of hitting coverage points from a random test generator viewpoint, a single engineer begins by writing a few generator templates and then generates tests using those templates. The generated tests are then run on a testbench which incorporates coverage monitors. The coverage monitors report all coverage points that are hit by the tests. As long as tests generated from the templates continue to hit new coverage points, the templates are kept in the nightly suite. As the rate of hitting new coverage points declines, new generator templates are created to target coverage holes. This approach requires skilled engineers to write checkers for the testbench but less skilled engineers to run the test generator.

Directed-random templates are created around points not hit by the purely random templates. We now begin to see the coverage grid closing more tightly (around 95%), and the verification process comes closer to completion.

Figure 4. Coverage Grid, Directed Random and Pure Random

Hitting Corner Cases

Not all coverage points will be hit by fully random or directed random templates. Some coverage points require a long series of events before the targeted behavior takes place. In this case, there are two possible approaches: write directed tests and write directed templates. Probably both of these approaches should be used. Directed tests can get to these most difficult coverage points more quickly but prove only one or a few cases around that point. Directed templates take more time to create but can be written to allow as much random behavior around the coverage point as possible.

Figure 5. Review Templates and Relax Restrictions

Finally, existing tests are reviewed, and as much directed behavior as possible is removed before the tests are run again. Coverage then reaches full closure, and these tests are run until the schedule no longer permits.

Today I attended the latest meeting of the Silicon Valley branch of the DVClub. For those not familiar with the DVClub (DV = Design Verification), it was started by Eric Hennenhoefer in Austin a few years ago. It now has branches in Austin, Bangalore, Boston, Bristol, Dallas, RTP, San Diego and Silicon Valley. In Silicon Valley it meets about once a quarter for a talk on some aspect of verification. I first heard of this about 1.5 years ago when we were invited from Tensilica to give a talk about verifying our video subsystem. The club has all the right ingredients to attract a crowd of engineers:

1. a free lunch
2. interesting speakers
3. did I mention a free lunch?
4. a chance to meet new colleagues and old friends
5. and of course, a free lunch

(Sponsors such as Cadence, Doulos, Denali, Silicon Elite and Obsidian pick up the tab for the venue and lunch (updated after original post, on Friday 20 March 2009, to correct list of sponsors)).

Today’s speaker was Brian Bailey, a friend and co-author of mine, speaking on “Is it time to declare a verification war?” The place was packed out with about 130 people, filling the room to capacity (Eric said this was the largest Silicon Valley DVClub crowd to date).

Brian spoke about his philosophy of verification, drew some analogies to Sun-Tzu’s Art of War, and also spoke about three technologies that he felt had potential to change verification significantly:

1. Functional Qualification – as exemplified by Certess (now SpringSoft) Certitude
2. Raising abstraction – as exemplified by Calypto’s sequential equivalence checking
3. “Intelligent testbenches” – as exemplified by Jasper’s Behavioural Indexing

Brian’s slides are available here.

IF you live or work anywhere any of these branches of the DVClub, and have an interest in verification, I recommend that you check them out. Sign up for their newsletter and get notified of meetings in advance.

In many cases, a processor design team will select a simple test generator for the first project and gradually evolve it into a more advanced form as the architecture matures. This continual evolution of test generator technology stems from several causes:

• Earlier designs tend to be simpler with later revisions adding more features and complexity.
• Later designs may prove complex enough to require a new approach.
• The verification effort may initially be underestimated.
• Estimates become more realistic over time as they become based on knowledge gained in earlier revisions.
• Products that go through several revisions and enhancements are likely to be those that have proven successful in the market and these tend to have better funding for both design and verification.

Table Based Generators

Table based test generators are the simplest generators available. Creation of such generators can be accomplished relatively quickly, and maintenance requirements are often low. These generators work by capturing ISA knowledge and storing it in a central table for later use. Because of their simplistic nature, table based generators may be used by less skilled personnel to create interesting tests. There is a drawback to these generators however, as their implementation is generally restricted to simple architectures. Usage on more complex ISAs may result in an inability to reach corner-cases or create complex scenarios. Table based generators may also generate invalid tests at times.

Static Generators

Static generators are similar to table based generators with the exception that the majority of the instruction, operand and data selection reside in complex procedural code. Static generators are capable of producing more random behavior than table based generators, but still have trouble
hitting many corner-cases. In addition, the skill level required to create and maintain such a tool rises sharply once this level of sophistication is reached.

Dynamic Generators

Dynamic generators incorporate significant knowledge about the architecture being tested. They enhance the ability of less-skilled users to generate complex tests that can hit hard-to-reach corner cases without stumbling on subtle programming pitfalls. This added knowledge, flexibility and ease-of-use is reflected in a more complex generator and consequently the cost of creating and maintaining the generator are greater than for table-based or static generators.