a multi-paradigm verification flow dave whipp. copyright © nvidia corporation 2005 multiple...

A Multi-Paradigm Verification Flow

Dave Whipp

Copyright © NVIDIA Corporation 2005

Multiple Paradigms

Analog

Mixed Signal

Digital

Formal

Semi-formal

Simulation

Emulation

Directed tests

Directed random

Constrained random

Self checking

Model based

Cycle based

Transaction based

Unit level

System level

Plan driven

Coverage driven

Pure Verilog

SystemVerilog

VHDL

SystemC

E / Vera

HDCaml / Confluence


A Multi-Paradigm Verification Flow

What do I mean by Multiple Paradigms?

Why would we want one?

Engineering a [ Multiple Paradigm ] Flow

Was it worth the effort?


Two Philosophies

ConformityEmphasize Standardization

DiversityEmphasize Innovation

Synergy or Conflict?


Context

Large Project

Multiple Teams

Long Legacy

People move between projectsbenefit of familiar environment

exposure to alternative approaches


Understanding Variation

To understand what to standardize:you need to understand what not to standardize

Personal Preferences

Technical Aspects of the Designs

Supporting Legacy

Seeking the Next Big Thing



Emacs!

vi



Choice of editor doesn’t affect othersAt least, not much

Choice of scripting language has greater impactBut is encapsulated

A script’s users don’t see the implementation language

Choice of HVL affects whole teamCan’t write “E” tests for a “Vera” testbench!

But a unit testbench isn’t seen by other units

A good flow will allow encapsulation of preferencesI can go to any unit and build & run its tests

Enables rapid localization of infrastructure issues


Technological Differences


Technical Characteristics

PCIE Video Frame

Buffer

Off-chip Memory

Graphics Pipe


Unit Level System Testing

Stage N-1

Stage N+1

Stage N

Graphics Pipeline (Transaction Model)

RTL Unit DIFFDIFF


Transactions Vs Cycles

Data min_val (Addr a1, Addr a2){ Data d1 = mem_read(a1); Data d2 = mem_read(a2);

if (d1 < d2)return d1;

elsereturn d2;

}

Address

Data

a1

d1

Pipelined Bus

a2

d2

t1 t2 t3 t4 t5 t6


Reuse Vs Stagnation

Reuse considered GoodAvoid reinventing the wheel

Build on the shoulders of giants

Reuse invites InertiaReuse can propagate dependencies

Dependencies make things harder to change

Resistance to change is known as inertia

Inertia can lead to StagnationImproper reuse accumulates dependencies

Reused code that is not understood will bit-rot

To avoid stagnation, inject agitation


New Challenges

New Tools

New Platforms

New People

New Ideas

Refactoring

Testability

D.R.Y.

Avoiding Stagnation


Are Single Paradigm Projects Possible?

Project 1

Unit A

Unit B

Unit C

Project 2 Project 3

Unit D

Unit B

Unit C

Unit B

Paradigm 1

Paradigm 2

Paradigm 3

time


1999

GeForce 25622 Million Transistors

2002

GeForce463 MillionTransistors

2003

GeForce FX130 Million

Transistors

2004

GeForce 6 Series222 Million

Transistors1993

NV11 Million Transistors

2005

GeForce 7 Series302 Million

Transistors

A Rich Legacy

A project must support at least three approachesLegacy

Mainstream

Future

A Paradigm-Agnostic Verification Flow


What is a Verification Flow?

The User Interface to VerificationCommand Line Interface

Graphical Interface

Web (browser) Interface

Programmers Interface

Used ByVerification Engineers

Designers

Managers

Robots


Purpose of a Flow

Conceptual FrameworkWhere do things live?

How do things work?

Keep out of the wayDon’t make life difficult for people

Define MetaphorsSubtly influence future directions

Anchor for VariationGive people something to complain about


Concepts

A well defined interface behaves as expectedObvious how it works, just by looking (“affordances”)

As consistent as possible, but no more so

Options are decision points

Users don’t read the manual


Designing a Flow (Interface)

Invent some Users

Figure out the important activities

Create the user model

Sketch a first draft

Iterate

Watch some real users; and iterate again


Invent some Users

Tim Nalu

Dusk Mike


Figure out the important Activities

TimburyNew Hire

Needs to get started

Checkout Tree

Run Sanity Regression

Simple Debug

DuskLead Designer

Intensive Debug

Modify RTLQuick Regressions

Analyze Coverage

Uses Emacs

NaluTesting Guru

Creates Testbenches

Hacks the Flow

Intensive Debug

Setup Regressions

Uses vi

MikeManager

View ProgressTests Written

Pass/fail counts

Coverage

Windows® User


Task Summary

build testbenchesrun quick regression tests (sanity tests)

run specific testrerun specific test with debug options

rerun specific phase(s) of a test

Add a testAdd a testbench

overnight regression testingSummarize results (web page)


User Model for Running Tests

Build Testbench

Generate Tests

Run Predictor

Run DUT

Compare Behaviors


The Implementation

“Make” driven flowMakefiles define the build

Automated testbench creation

Compile/generate tests where appropriate

“makepp” is a better “make”

Perl scripting for the gutsEach test is a Perl object

Small number of standard classes

Individual units define additional classes

rdistnvmk run_unit

GEN DIFF

VSIM

CSIM

tbgen

fmtbgen

fmgen

run_testgen

compare

“grok”

randomizetestlist

rand_regress

mod

stgen

ttgen

build_testlist

tjob

LSF

vcs

An Early Sketch


Command Line Interface

nvmknvmk sanity

nvmk regress

nvmk all

nvmk open_bugs TESTNAME=bug42

nvmk open_bugs TESTNAME=bug42 VSIM=1 DUMP=1

nvmk open_bugs TESTNAME=bug42 CSIM=gdb


Detailed Triage and Debug

cd rundir/open_bugs/bug42

ls *.loggen.log csim.log

vsim.log diff.log

replay vsim.log +foo +bar

replay –gdb csim.log


Testing the Flow

Considered a hard problem“If it works, then it works”

Testability is a featureStart testing early

Some tests are better than none

Create a multi purpose “Example Project”A stable testing environment

Acts as documentation

Use the “Invented Users” to create activity scripts

Keep it up-to-date


Watch some Real Users

Image courtesy of DAZ Productions


How to “Watch” Users

Meetings and Discussions

Coffee-Break Grousing

Bug Reports

Keep Track of Support Requestscreate FAQs

VNC (Remote Desktop)

Instrumentation


Biggest Problems

Performancemake-driven user interface was subjectively slow

too much dependency checking

users know what file’s they’ve changed

how to communicate this to the tools?

Too much flexibilityOptions imply decisions

Lack of sufficient understanding

So pervasive cut & paste reuse


Build Time Distribution: 10,000 per sample

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

> 1 hour

< 1 hour

< 30 min

< 15 min

< 10 min

< 5 min

< 2 min

< 1 min

< 30 sec

< 20 sec


Build Time Distribution: 1000 per sample

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

> 1 hour

< 1 hour

< 30 min

< 15 min

< 10 min

< 5 min

< 2 min

< 1 min

< 30 sec

< 20 sec


Summary

Verification Engineers do more than write tests

A flow is a user interfaceEncourage user feedback

Add instrumentation

A flow is an enabler, not disablerencourage innovation

don’t tell users what not to do

Without diversity, a flow stagnates

Without standardization, a flow flounders

a multi-paradigm verification flow dave whipp. copyright © nvidia corporation 2005 multiple...

Documents