11

ece156B, Slide 1

Functional Verification and TestBench Generation – Direct and Random Testing

Lecture 7

Slide # 2

Functional Verification Demand

• Functional verification cost grows faster than design complexity

100K 1M 10M

1M

100M

10B

(gates)

(cycles)

22

Slide # 3

A simple example of why

• If you have a design with 10-bit flipflops, the maximum number of states is 1024

• If it is 11-bit, the number of states is 2048

• So in theory, the verification space grows exponentially while the design grows linearly

Slide # 4

Approaches

• Exhaustive simulation is impossibleYou can simulate every instruction but not every instruction sequence

• Formal verification on everything is impossible

Formal verification can handle up to a certain number of states

• So, the only feasible solution is to simulate under certain guidance

33

Slide # 5

Pre-silicon Validation cyclesnot that we don’t try

0

1000

2000

3000

4000

5000

600040

'98

43'9

8

46'9

8

49'9

8

52'9

8

03'9

9

06'9

9

09'9

9

12'9

9

15'9

9

18'9

9

21'9

9

24'9

9

27'9

9

30'9

9

33'9

9

36'9

9

39'9

9

42'9

9

45'9

9

48'9

9

51'9

9

(Millions)

Pentium 4

Full-chip

~1/4 secof real time execution

Slide # 6

Verification Crisis

• More than 50% of the project budget already goes to verification

• Simulation and testbench preparation time already drags the time-to-market

• Design complexity grows tremendously with the use of IP cores

• Cost of chip re-spin is high> $100K for ASIC> $1M for a complex SOC

44

Slide # 7

Functional Verification ≈ Testbench Simulation• In practice, testbench simulation remains the only

effective method for full-chip functional verificationIssues of simulation

Design specification and abstraction– Hand books

Test stimuli– Manual construction– Direct and random

Definition of correctness– Manual observation– Simulate another behavior model and compare– Construction of monitors – assertion-based methodology

Effectiveness of tests– Guided by coverage metrics

Length of simulationbillions of cycles (for μ-processors and complex SOCs)

Slide # 8

IBM RTPG

• RTPG – Biased Pseudo-Random Test Program Generation

• IBM Journal of R&D, 19921st to talk about RTPG for verification of R6000

• Design Automation Conference, 19951st public talk on RTPG for verifying PowerPC

• The RTPG evolves into constrained random verification (Synposys Vera, System Verilog)

Synopsys’s Vera verification environmentCheck Vera web page for informationVery important piece of technology

55

Slide # 9

How to verify a microprocessor• Tests for a microprocessor is nothing but an

assembly program• You would try a set of assembly programs• In the old days, people had tables of

important programs to be appliedThese tables were accumulated over yearsBased on previous experience

• DisadvantagesYou can only apply them when design is readyIf you are developing a new design, then what?

Slide # 10

Random Test Program Generation - RTPG• Basic Ideas

Allow users to specify test program biasing templates instead of the test programs themselves

Hide detail from usersThe tool ensures the validity of the test programs randomly generated

Automatic guarantee to satisfy all architectural constraints (among instructions)

Use biasing seeds and constraints to guide the random generation process

66

Slide # 11

Architectural constraint• For example, a special register has to be set first

before using a branch instructionEx.

CMP R0, A, B (if A>B, R0=0 otherwise R0=1)JNE addr, R0 (if R0 <> 0, goto addr)Moreover, addr should be a 6-bit constant

• Constraints can be defined for one instructionThe instruction format limit what you can use

• Or defined among instructions• Architectural constraints (for examples)

32 registers (20 general purpose, 12 special purpose)24-bit addressingIndirect addressing

Slide # 12

Biasing structure• Random bias can be supplied along each pre-

defined dimensionFor example, you may want to define

Bias to select an instructionBias to select the next instruction based on the current oneBias to select an operandBias to use branch and jumpBias to cause overflow, underflowBias to interrupt and to cause exception

• A test template is a skeleton of an assembly program with some random selection along the pre-defined dimension

77

Slide # 13

Example

• You may have a test program template look like the following flow

Add <random R1-R4> <random R4-R8> <random R8-R20><0.9 prob to add/ 0.1 prob to sub> R3 R5 <random R4-R7><0.1 prob> Jump addrAdd R1, R2, <0.9 to have 0xffffffff/ 0.1 to randomly generate a number>

Slide # 14

RTPG Methodology for Microprocessors

Billions of cycles are simulatedVerification coverages are systematically measured based on a collection of known eventsA subset of tests are selected for post-silicon validation

MicroprocessorArchitecturalReference

Model

Random TestPattern

Generator

Biasing and constrainedtest program templates

Tests

ExpectedResults

RTPG

RTL Full-ChipModel

Logic Simulator

Pass?

88

Slide # 15

Test Program• A test program consists of three parts• Initialization

Set up all relevant initial states in registers, flags, tables, caches, and memory locations

• Instruction sequenceThe actual instructions to be tested

• Expected resultThe final states that were changed during the tests (not necessarily at the end of the tests)

Slide # 16

Initialization and Observation• The initialization sequence is important

It allows a test program to be run without the interference from other test programs

• You may also want to add a suffix program to ensure observability of the results

Move register contents to fixed memory locations for inspection (like a dump operation)

• The initialization and observation parts may have constraints associated with the actual program content

99

Slide # 17

RTPG design – biasing structure• From architecture, you need to decide what

dimensions can vary randomlyTo decide the structure of biasing

• For more examplesRandom selection of an instruction subsetRandom addition of branch or call instructionsRandom selection of operand among “immediate” “register” or “memory” modeRandom selection of operand’s contentRandom selection among a group of pre-determine instruction sequencesRandom enforcement of exceptions or interrupts

Slide # 18

RTPG design – expected results

• It is important to decide how and where expected results should be obtained

• For examplesDo or do not have an explicit behavior modelUse or do not use assertionsCan or cannot observe at internal registersCan or cannot observe at internal cachesCan or cannot observe at internal signal lines

1010

Slide # 19

Reference Model of a Processor• ISA – Instruction Set Architecture

An abstract model viewed by assembly programmer

• Program status wordsInstruction addressConditionsKey machine states

• Registers32 general purpose registersSpecial purpose registers

• Memory4K sized page block

Slide # 20

Observability – Consistency Checking• Logical registers may not be the same as

physical registersRegister renaming depends on the implementation

• Logical cache may not be the same as physical cache structure

1 level cache vs. 3 level cache

• Memory is the only “safe” observation point

Memory

μ-processorinst

ruct

ion

sequ

ence

s Safeobservationpoint

registers

cache

potentialobservation points

1111

Slide # 21

RTPG

• User supplies test templates for test generation• User supplies biases for randomized tests• RTPG ensure

tests are correctexpected results are computed based on the logical model (or the higher-level model)

processorfeatures

pseudo random test generatorparse refine dispatch simulate

Tests

symbolictemplates

manual

inputbiasing

manual

Slide # 22

An Example (DAC 95)• Add op1 op2 base• op1 is one of the 16 registers, 0… 15• op2 is a 8-bit displacement 00 …. FF• base is one of the 16 32-bit base register, 0….

15• What the instruction does

[op1] + [op2+[base]] -> op1• [op1] is a 32-bit signed number• [op2]+base point to a virtual memory address• The content of the memory location is a 32-bit

signed number

1212

Slide # 23

Instruction tree – to help test generation

op1 op2base[op2]+base

L: lengthA: addressD: the data content

Slide # 24

Use the tree to constrain tests

1313

Slide # 25

An actual test case

Add 7, 0100, 9

virtual address

data

(syntax)

(semantic)

Slide # 26

Syntax Vs. Semantic• You need to define the (syntactic and semantic) model

for each instructionThis serves as the basis for biasing

• Then, you need to define the model for a pair or more instructions

Only if they can interact with the given architectureFor example,

One instruction set the overflow bit and the next instruction test for the overflow bitOne instruction set the jump address and the next instruction perform the actual jump

To begin with, you can provide a set of instruction sequences where each is considered as a unitThen, you provide a model for each sequence to specify its syntax and semantic

Also define where the biases can be supplied and take effect

1414

Slide # 27

Unexpected Events• Test programs are generated randomly

May result in unexpected events like exceptions

Test templateInstantiate: t1,t2,t3,t4

Repeat 100 timesAdd t1, t2, R1Div t3, R1, t4

Test program

InitializationR4 = 12, R5 = -1, R3 = 2

Add R3, 10, R1Div R4, R1, R4Add R4, R5, R1Div R3, R1, R3<exception>

Slide # 28

Unexpected Events – Cont.

• Unexpected events may be avoided by constraint specification

• If happens, some actions may be required to

Continue the simulationStop and correct the template

• Some unexpected events may be useful for exploring design bugs

1515

Slide # 29

Event Handler

Test templateInstantiate: t1,t2,t3,t4

Repeat 100 timesAdd t1, t2, R1Div t3, R1, t4

Test program

InitializationR4 = 12, R5 = -1, R3 = 2

Add R3, 10, R1Div R4, R1, R4Add R4, R5, R1Div R3, R1, R3Move R1, 100Store 1, R1Add R7, R1, R1

Event: R1 =0Action:

Move R1, 100Store 1, R1

Slide # 30

Basic Principles• Mandatory sequences

Single instruction set; Instruction pairs; 3-instruction set; Selected sequences (experience + new ideas); Random sequences

• Coverage Measurement Statement coverage; Toggle coverage; State transition coverage; Assertion coverageerror discovery rate – tapeout

Weeks

# of

erro

rs fo

und

Stop!

1616

Slide # 31

More about coverage• The basic coverage can be measured by line

coverage in the codeA line is actually simulated in the event driven simulation, i.e. an event occurs at the line

• Toggle coverageEach signal line can be toggled (between 0 and 1)

• Finite state machine (FSM) state coverageEach state has been reached

• Design error coverageRandomly inject an errorCheck to see if the error can be detected

ece156B, Slide 32

Design Error Modeling

1717

Slide # 33

Quality Measurement

• Structure-based logic error models provide a good way (independent of the function) for measuring the quality of verification

• May implicitly helps to verify the functional correctness of a design

Equivalence

ErrorModels

Design model 2Design model 1

Slide # 34

Design Error Models

Logic design error modelsSimple structural mutations

Extra inverter, Gate substitution, Extra wire, Extra gate, Missing gate, Wrong wire, etc.Have high correlations with manufacturing fault models

Recent studies show that many realistic design errors can be captured by capturing those errors

1818

Slide # 35

Reference

• Original design error paper (TCAD1988)• Date 1998 best paper (to evaluate various

verification approaches)Also describe how to inject error and simulate errors using a logic simulator

Slide # 36

Error Injection (Gate)

• N extra lines allows injection of 2^n –1 errors• Can use logic simulator to collect results

May be time-consumingInject to sensitive areas only

decoder

MUXgood

faulty

selectextrainputs

1919

Slide # 37

Consistent Behavior – Gate vs. Transistor

3.23%

A

D 2.35%

71.76%B

22.97%

Redundant

0.098%

C 0.196%

11.53%

A

D 1.66%

69.1%B

16.72%

Redundant

C 0.59%

0.098%

0.29

3%

Gate-Level Results Transistor-Level Results

A,B,C,D are 4 different verification methods

Slide # 38

Divide and Conquer

• As design becomes extremely complex, fixing a bug become more tedious

• If we start RTPG directly on the whole chip,

We spend a lot of time to fix bugs in some unitsYet some other units receive less attention

2020

Slide # 39

You want to start small

• Pick a block of interest• Try to verify this block only• You can inject and simulate 1000 errors

on this block and measure coverage• Then, you do it block by block

Slide # 40

Pre-RTPG Simulation – UniSim

A chip is usually divided into 6-7 unitsGoal: detect 95% of the design errors at unit level and the remaining 5% at the full-chip levelUniSim allows individual unit to be verified independently from the completion of other unitsAt this level, verification does not need to be complete, nor 100% correct.

Tests

ExpectedResults

RTPG

A C++ system toemulate the I/O

behaviors of otherparts of the chip

New Tests

NewExpectedResults

Logic simulator

Unit underverification Pass?

2121

Slide # 41

In our case

• You need to dump the internal results at the boundary of a block and record them as inputs and expected outputs for the block

This should be recorded as cycle-based simulation

Capture inputs here

Record resultsTest programssupplied here

Slide # 42

An Example (Taylor&Quinn α-μp DAC98)

• Direct and pseudo random testing• Start recording after all units are integrated• After initial tests to setup the system

Time

# of bugs found

2222

Slide # 43

Excitation of Bugs

• More than 3 quarters % of bugs can be excited via pseudo random tests

• Direct tests are designed to compensate misses from pseudo random tests

• There are bugs that can be excited by multiple means

0% 20% 40% 60% 80% 100%

Pseudo Random Tests

Direct Tests

Other

% of Total Bugs

Slide # 44

Observation of Bugs

• Assertion checkers are manually placed by designers to monitor design properties

Very effective in practice• Roughly half of the bugs are found by comparing to the reference model

Register comparisonPC comparisonMemory comparison

25%

22%

15%

14%

8%

6%

5%

3%

2%

0% 5% 10% 15% 20% 25% 30%

Assertion Checker

Register Compare

Simulation Hang

PC Compare

Memory Compare

Manual Inspection

Self-Check Tests

Cache Coherency

Saves Check

% of Total Bugs

2323

Slide # 45

Types of Bugs

• Recall: Unit level verification has been carried out• Current verification is to catch bugs during the

implementation of architectural SPEC

78%

9%

5%

3%

5%

0% 20% 40% 60% 80% 100%

ImplementationBugs

ProgrammingMistakes

RTL/SchematicMismatch

ArchitecturalConception

Other

% of Total Bugs

Slide # 46

8 Bugs missed into 1st Silicon (Taylor&Quinn α-μp DAC98)

• 2: RTL vs. schematics mismatch1997 results – Logic EC may have solved these problems

• 1: electrical issuesWe expect more in the future – no good solution so far

• 1: insufficient randomization of initial state in RTL• 2: logic bugs that should be found by more

sophisticated checkers• 1: logic bug that is hard to catch by either pseudo

random tests or direct testsSomething is going to be missed