cool fpv tricks: reaching deep bounds with not-quite-formal methods

Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods

Erik SeligmanCS 510, Lecture 13, February 2009

Agenda Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments

Agenda Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments

Bounded FPV Majority of FPV work done at low bound

• Around 50-100 cycles• Many modules testable this way• Limits of engines: exponential blowup

Good in many (most?) cases• Verification of isolated modules• Tricky bugs often don’t need much depth

– with right input & constraints

• Cases where deep queues/counters not critical Double-check with coverage points!

Problem: FPV Bounds Bounded FPV is ignoring possibilities

• Known industry cases of “deep” errors Deep bugs suspected in some cases

• Example: 2 independent 13-bit ctrs– How many combinations possible?– How many cycles needed?

• Bugs may be lurking beyond FPV bounds• Maybe too much logic to solve with pruning

Strategies desired for hi-bound FPV

Compromise: Abstraction Counters, state machines: free outputs

• As we saw in memory controller exampleAll values possible

• Useful in many cases• But at significant cost

• Realism of cases: values jump around• No coverage for abstracted logic

Compromise: Initial State Set initial state to something ‘interesting’

• Fill queues, set high counters, etc. FPV no longer exhaustive

• Errors from different initial state not covered• Focus is bug hunting in suspicious area

Can we do better?

Limits of Pruning Pruning often helps complexity issues

• In best case, bounded full proof• Or can deepen bounds

But it can’t help in worst cases• Maybe long, complex interactions of parts

– Think about 2-counter example again

• Lots of logic may be involved

Key Insight: Leverage Simulation Simulation env has design knowledge

• Usually set up to create interesting states• Wide variety of states possible in sim

Can we leverage simulation for FPV?• Simple example: Snapshot a simulation

state, use as FPV input• Are there bigger opportunities?

Agenda Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments

0in Dynamic Formal FPV Monitor simulation runs

• Tool runs as plugin to simulator Identify “interesting” states

• Hit cover point• Transition counter, state machine, etc.

Launch many low-bound FPV runs

Ordinary FPV & State Space

• Small proof radius covered after reset

Dynamic Formal Approach

• Many mini-FPV runs launched from sim• Smaller proof radius for each FPV run

Advantages/Disadvantages Advantages

• Based on real tests likely good examples• Produce realistic counterexamples since

most cycles are from simulation• Finds deep bugs “almost” tested in sim

Disadvantages?

Advantages/Disadvantages Advantages

• Based on real tests likely good examples• Produce realistic counterexamples since most

cycles are from simulation• Finds deep bugs “almost” tested in sim

Disadvantages?• Depends on setting up simulation env

– Not usable really early or arbitrary hierarchy• Can’t find counterexamples distant from tests

– Misses major selling point of FPV: corner cases not conceived by designer

• Tool pain: FPV & sim tools interact?– Harder than it sounds!

Agenda Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments

Synopsys Magellan Approach Similar motivations to 0in Recognize problem: simulation env

• Few real designers use ‘raw’ sim tool• Many layers of scripts & wrappers• Really painful to enable 0in-like method

Solution: Random Simulation• Still uses simulation engine• Randomly tries to simulate to get cover pts

– No use of real tests like 0in

Hybrid Formal Approach

• Lots of random simulation paths• Many mini-FPV runs launched from sim• Smaller proof radius for each FPV run

Advantages/Disadvantages Advantages

• Not limited by tests from validation team• Can theoretically find very deep bugs• Independent of simulation environment

Disadvantages?

Advantages/Disadvantages Advantages

• Not limited by tests from validation team• Can theoretically find very deep bugs• Independent of simulation environment

Disadvantages?• Very random: need to get lucky for bug

– Loses both comprehensiveness & test-guidance– Good cover points may mitigate

• Still have sim/fpv tool integration issues– Many “simulation-synthesis mismatches”: time delays,

system functions, non-det behavior, X/Z vals, etc.

Agenda Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments

Can We Use A Non-Simulation Approach? FPV user suspects deep error

• But proof bounds not enough• Standard complexity techniques not working

What does FPV user know?• Probably has likely scenarios in mind• But FPV tool isn’t getting there

So why not use user-guided FPV?• Try to manually get FPV tool close to error• Then launch bounded, comprehensive run

User-Guided FPV Create cover points for reset state

• Suspected intermediate state on way to err Ask FPV tool to reach cover point

• CEX at cover == reset state for next• Save state manually

– Nicer if tool help, but not seen in current tools

After cover pt, use real bounded FPV• Run isn’t full formal proof of properties• But is comprehensive bug hunt from there

User Guided Approach

• Use FPV engine to visit cover point• Launch bounded FPV from targeted pt

Advantages/Disadvantages Advantages

• Completely eliminates sim engine issues• Can theoretically find very deep bugs• Takes advantage of user knowledge

Disadvantages?

Advantages/Disadvantages Advantages

• Completely eliminates sim engine issues• Can theoretically find very deep bugs• Takes advantage of user knowledge

Disadvantages?• Highly dependent on user intuition

– Loses ‘automatic search’ aspect of 0in/Synopsys

• Problem areas must be specifically targeted

Agenda Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments

Deep Bounds: Why ‘Sexy’? Much effort in industry/academia Recognition of limits of bounded FPV Potential for fame and fortune

• Really cool to claim find of 10000-cycle-deep error nobody noticed!

• Common view: One such error justifies cost of full deployment + BMW for the validator

But is this really the key FPV problem?

Where Do FPV users Spend Their Time? Intuitive explanation, take with grain of salt 70% “Wiggling”

• Early attempts with quick counterexamples• Root-causing CEXs, adding assumptions, & finding occasional

shallow bugs• Many FPV efforts never get beyond this stage

20% “Solidifying”• Have bounded proofs on most assertions• Incrementally deepening bounds, checking covers

10% “Heroics”• Extra, expert efforts to get full proofs or deep proofs in

suspected problem areas • Trying exotic FPV technologies

Where Should FPV Research Be Concentrated? Vast majority of FPV effort is “wiggling”

• Assuming previous slide roughly accurate! Then next largest chunk is “solidifying”

• Understaning, improving low bounds• Basic complexity issues

What are FPVers actually doing?• Running low-bound proof attempts• Root-causing failures to add assumptions• Attacking basic complexity issues

(IMHO) The real problem is Usability

What Would An FPV Breakthrough Look Like? Need quantum leaps in productivity of

“wiggling” stage• Secondary priority: addressing complexity

Key functionality I want to see• Viewing and understanding counterexamples• More & better live what-if experiments• Intuitive user interaction• Really fast incremental responsiveness • Support for interactive complexity analysis

My Advice To Academia Better engines & deeper proofs are nice

• But only leveraged by small group of engineers who progress to “heroic” FPV stage

Best impact: where engineers spend time• Wiggling stage: quickly understanding cex• Solidifying stage: root-causing & fixing complexity

Focus on usability, interactivity, and incremental resposiveness!

References / Further Reading http://www.edacafe.com/Vision/200208/d

esign3.html http://drona.csa.iisc.ernet.in/~deepakd/tal

ks/formal-iisc-0306.pdf http://www.te.rl.ac.uk/europractice/vendo

rs/magellan_ds.pdf