cool fpv tricks: reaching deep bounds with not-quite-formal methods
DESCRIPTION
Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods. Erik Seligman CS 510, Lecture 13, February 2009. Agenda. Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments. Agenda. - PowerPoint PPT PresentationTRANSCRIPT
Cool FPV Tricks: Reaching Deep Bounds With Not-Quite-Formal Methods
Erik SeligmanCS 510, Lecture 13, February 2009
Agenda Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments
Agenda Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments
Bounded FPV Majority of FPV work done at low bound
• Around 50-100 cycles• Many modules testable this way• Limits of engines: exponential blowup
Good in many (most?) cases• Verification of isolated modules• Tricky bugs often don’t need much depth
– with right input & constraints
• Cases where deep queues/counters not critical Double-check with coverage points!
Problem: FPV Bounds Bounded FPV is ignoring possibilities
• Known industry cases of “deep” errors Deep bugs suspected in some cases
• Example: 2 independent 13-bit ctrs– How many combinations possible?– How many cycles needed?
• Bugs may be lurking beyond FPV bounds• Maybe too much logic to solve with pruning
Strategies desired for hi-bound FPV
Compromise: Abstraction Counters, state machines: free outputs
• As we saw in memory controller exampleAll values possible
• Useful in many cases• But at significant cost
• Realism of cases: values jump around• No coverage for abstracted logic
Compromise: Initial State Set initial state to something ‘interesting’
• Fill queues, set high counters, etc. FPV no longer exhaustive
• Errors from different initial state not covered• Focus is bug hunting in suspicious area
Can we do better?
Limits of Pruning Pruning often helps complexity issues
• In best case, bounded full proof• Or can deepen bounds
But it can’t help in worst cases• Maybe long, complex interactions of parts
– Think about 2-counter example again
• Lots of logic may be involved
Key Insight: Leverage Simulation Simulation env has design knowledge
• Usually set up to create interesting states• Wide variety of states possible in sim
Can we leverage simulation for FPV?• Simple example: Snapshot a simulation
state, use as FPV input• Are there bigger opportunities?
Agenda Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments
0in Dynamic Formal FPV Monitor simulation runs
• Tool runs as plugin to simulator Identify “interesting” states
• Hit cover point• Transition counter, state machine, etc.
Launch many low-bound FPV runs
Ordinary FPV & State Space
• Small proof radius covered after reset
Dynamic Formal Approach
• Many mini-FPV runs launched from sim• Smaller proof radius for each FPV run
Advantages/Disadvantages Advantages
• Based on real tests likely good examples• Produce realistic counterexamples since
most cycles are from simulation• Finds deep bugs “almost” tested in sim
Disadvantages?
Advantages/Disadvantages Advantages
• Based on real tests likely good examples• Produce realistic counterexamples since most
cycles are from simulation• Finds deep bugs “almost” tested in sim
Disadvantages?• Depends on setting up simulation env
– Not usable really early or arbitrary hierarchy• Can’t find counterexamples distant from tests
– Misses major selling point of FPV: corner cases not conceived by designer
• Tool pain: FPV & sim tools interact?– Harder than it sounds!
Agenda Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments
Synopsys Magellan Approach Similar motivations to 0in Recognize problem: simulation env
• Few real designers use ‘raw’ sim tool• Many layers of scripts & wrappers• Really painful to enable 0in-like method
Solution: Random Simulation• Still uses simulation engine• Randomly tries to simulate to get cover pts
– No use of real tests like 0in
Hybrid Formal Approach
• Lots of random simulation paths• Many mini-FPV runs launched from sim• Smaller proof radius for each FPV run
Advantages/Disadvantages Advantages
• Not limited by tests from validation team• Can theoretically find very deep bugs• Independent of simulation environment
Disadvantages?
Advantages/Disadvantages Advantages
• Not limited by tests from validation team• Can theoretically find very deep bugs• Independent of simulation environment
Disadvantages?• Very random: need to get lucky for bug
– Loses both comprehensiveness & test-guidance– Good cover points may mitigate
• Still have sim/fpv tool integration issues– Many “simulation-synthesis mismatches”: time delays,
system functions, non-det behavior, X/Z vals, etc.
Agenda Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments
Can We Use A Non-Simulation Approach? FPV user suspects deep error
• But proof bounds not enough• Standard complexity techniques not working
What does FPV user know?• Probably has likely scenarios in mind• But FPV tool isn’t getting there
So why not use user-guided FPV?• Try to manually get FPV tool close to error• Then launch bounded, comprehensive run
User-Guided FPV Create cover points for reset state
• Suspected intermediate state on way to err Ask FPV tool to reach cover point
• CEX at cover == reset state for next• Save state manually
– Nicer if tool help, but not seen in current tools
After cover pt, use real bounded FPV• Run isn’t full formal proof of properties• But is comprehensive bug hunt from there
User Guided Approach
• Use FPV engine to visit cover point• Launch bounded FPV from targeted pt
Advantages/Disadvantages Advantages
• Completely eliminates sim engine issues• Can theoretically find very deep bugs• Takes advantage of user knowledge
Disadvantages?
Advantages/Disadvantages Advantages
• Completely eliminates sim engine issues• Can theoretically find very deep bugs• Takes advantage of user knowledge
Disadvantages?• Highly dependent on user intuition
– Loses ‘automatic search’ aspect of 0in/Synopsys
• Problem areas must be specifically targeted
Agenda Motivation: FPV and Deep Bounds 0in’s ‘Dynamic Formal’ FPV Synopsys ‘Hybrid Formal’ FPV When FPV Is Not FPV Some Meta-Comments
Deep Bounds: Why ‘Sexy’? Much effort in industry/academia Recognition of limits of bounded FPV Potential for fame and fortune
• Really cool to claim find of 10000-cycle-deep error nobody noticed!
• Common view: One such error justifies cost of full deployment + BMW for the validator
But is this really the key FPV problem?
Where Do FPV users Spend Their Time? Intuitive explanation, take with grain of salt 70% “Wiggling”
• Early attempts with quick counterexamples• Root-causing CEXs, adding assumptions, & finding occasional
shallow bugs• Many FPV efforts never get beyond this stage
20% “Solidifying”• Have bounded proofs on most assertions• Incrementally deepening bounds, checking covers
10% “Heroics”• Extra, expert efforts to get full proofs or deep proofs in
suspected problem areas • Trying exotic FPV technologies
Where Should FPV Research Be Concentrated? Vast majority of FPV effort is “wiggling”
• Assuming previous slide roughly accurate! Then next largest chunk is “solidifying”
• Understaning, improving low bounds• Basic complexity issues
What are FPVers actually doing?• Running low-bound proof attempts• Root-causing failures to add assumptions• Attacking basic complexity issues
(IMHO) The real problem is Usability
What Would An FPV Breakthrough Look Like? Need quantum leaps in productivity of
“wiggling” stage• Secondary priority: addressing complexity
Key functionality I want to see• Viewing and understanding counterexamples• More & better live what-if experiments• Intuitive user interaction• Really fast incremental responsiveness • Support for interactive complexity analysis
My Advice To Academia Better engines & deeper proofs are nice
• But only leveraged by small group of engineers who progress to “heroic” FPV stage
Best impact: where engineers spend time• Wiggling stage: quickly understanding cex• Solidifying stage: root-causing & fixing complexity
Focus on usability, interactivity, and incremental resposiveness!
References / Further Reading http://www.edacafe.com/Vision/200208/d
esign3.html http://drona.csa.iisc.ernet.in/~deepakd/tal
ks/formal-iisc-0306.pdf http://www.te.rl.ac.uk/europractice/vendo
rs/magellan_ds.pdf