dv club europe - test and verification safety · pdf filedv club europe certitude tm on c/c++...
TRANSCRIPT
© 2014 Synopsys, Inc. All rights reserved. 1
DV Club Europe
CertitudeTM on C/C++
JM Forey
2014, April the 28th
© 2014 Synopsys, Inc. All rights reserved. 2
Verification Goals
• Checks whether something complies with its specification
and intended usage
– Using the lowest possible amount of resources
© 2014 Synopsys, Inc. All rights reserved. 3
But Verification Is Difficult…
• Not specific to electronic or software
– One way to do something right, billions to do it wrong
– Wrong assumptions about what to verify
• False positive issues are silent
– Does the stimulus cover all scenarios effectively?
– Are the checkers able to catch all possible misbehaviors?
– Are infrastructure or process issues masking bugs?
• Inadequate verification affects productivity, quality, reputation, revenues
Must assess verification effectiveness
© 2014 Synopsys, Inc. All rights reserved. 4
Effective Verification Exercise, propagate, and detect bugs
To detect a bug…
• The test must activate the bug
• An effect of the bug must propagate to an “output” of the software
• The testing infrastructure must detect the behavior difference due to the bug
C/C++ under
Verification
Testing Infrastructure
Compare
Bug
Test
Cases
Expect Results
Detection Propagation Activation
© 2014 Synopsys, Inc. All rights reserved. 5
Assessing Verification Effectiveness
C/C++ under
Verification
Testing Infrastructure
Compare
Bug?
Expect Results
Code coverage tells about what is (not) exercised,
but nothing about propagation nor detection
??? ???
Test
Cases
Activation
© 2014 Synopsys, Inc. All rights reserved. 6
Introducing Certitude
• Automatically inserts “artificial bugs” called faults into the design
• Measures the ability of the verification environment to activate, propagate, and detect faults
• Identifies missing / broken / incomplete test scenarios
• Identifies missing / broken checkers
• Provides objective measure of overall effectiveness and robustness
C/C++ under
Verification
Testing Infrastructure
Compare
Fault
Expect Results
Functional Qualification
Detection Propagation
Test
Cases
Activation
© 2014 Synopsys, Inc. All rights reserved. 7
How Fault Injection Works?
• Modifies code to insert faults
o1 = f(i1) o1 // variable doesn’t receive value
if (a) if (TRUE) // fault forces execution of “true” branch
f1(); f1();
else else
f2(); f2();
a = b | c a = b & c // fault changes operator
• Run broken code
– Does at least one test fail? Great!
– Environment is robust enough to detect the software / model is broken
– Do all tests pass? Help!
– Original and broken software both compliant with environment
© 2014 Synopsys, Inc. All rights reserved. 8
Certitude Flow with C/C++
• Leverages existing regression
environment
• Certitude steps
– Model
– Activate
– Detect
– Report analysis
• DUT can be
– Compiled as part of executable
– A separate linked shared object
Executable Executable .so
Test
Case
IDs
C
C++
Compile
Script Execute
Script
Certitude HTML
Report
Config
File
Tcl DB
Access
Verification Environment
Executable
© 2014 Synopsys, Inc. All rights reserved. 9
Certitude: Beyond Traditional Coverage
C++ example:
cppcheck software
© 2014 Synopsys, Inc. All rights reserved. 10
Highlighted locations:
Inserted faults
Click to get details on
fault
Certitude: Beyond Traditional Coverage
© 2014 Synopsys, Inc. All rights reserved. 11
Certitude: Beyond Traditional Coverage
After regression run
Not exercise code
Conditions were
always false
© 2014 Synopsys, Inc. All rights reserved. 12
Certitude: Beyond Traditional Coverage
After detection run
Uncaught systematic faults:
Anything not green
© 2014 Synopsys, Inc. All rights reserved. 13
Certitude: Beyond Traditional Coverage
After detection run
© 2014 Synopsys, Inc. All rights reserved. 14
Certitude Applications Today
• Fully automated fault insertion
– Software, reference models (ex: ISS), high level synthesis models (HLS)
– C, C++, SystemC
– Hardware models (RTL, netlist)
– vhdl, verilog, system verilog
– Static and / or dynamic verification
• Effectiveness of the verification
– Test selection
– Consistency between high level model and RTL implementation
• Robustness and effectiveness of safety mechanism
– In presence of systematic or random faults
– For ISO26262 can the model recover, return to a safe state or report an undesired fault
© 2014 Synopsys, Inc. All rights reserved. 15