model checking java programs using structural heuristics
Post on 02-Feb-2016
61 Views
Preview:
DESCRIPTION
TRANSCRIPT
Model Checking Java Programs using
Structural Heuristics
Alex GroceCarnegie Mellon University
Willem VisserNASA Ames Research Center
Model Checking
• Explores graph of reachable system states– Checking for local assertions, invariants and
general temporal (logic) properties
• Symbolic model checking
• Explicit-state model checking
Java PathFinder
void add(Object o) { buffer[head] = o; head = (head+1)%size;}
Object take() { … tail=(tail+1)%size; return buffer[tail];}
Java Code
JAVAC JVM
0: iconst_01: istore_22: goto #395: getstatic 8: aload_09: iload_210: aaload
Bytecode
Special JVM
Model Checker
Depth-first Search
push initial state on Stackwhile (Stack not empty)
s = top(Stack)if s has no more successors
pop the Stackelse
s’ = next successor of sif s’ not already visited
mark s’ visitedif s’ is a goal state
then terminatepush s’ on Stack
Problems with DFS
• Produces lengthy counterexamples
• If state-space is too large to fully explore– May expend all resources on a single path when
shallow counterexamples exist– Failed runs give little information because
states explored may be very “similar”
Directed Model Checking
• Model checking as a search in a state space
• Why not use heuristics to guide the search?– Need to know what we’re looking for
• Can we find good heuristics for model checking?
• Bug-finding rather than verification
Best-first Search
priority queue Q = {initial state}while (Q not empty)
s = state in Q with lowest fremove s from Qfor each successor state s’ of s
if s’ not already visitedmark s’ visitedif s’ is a goal state
then terminatef = h(s’)store (s’, f) in Q
Two Kinds of Heuristics
• Property-specific heuristics
– Directed at a specific error• Number of unblocked threads as a measure of
distance to deadlock
• Static analysis for distance to an assertion check
– Focus of most previous work in field
Two Kinds of Heuristics
• Structural heuristics
– Designed to explore the structure of a program in a systematic fashion
– But what do we mean by structure?
Structural Heuristics
• One obvious kind of structure in a program:
– Control flow
• Reachable control flow rather than just CFG
• Motivation for branch coverage metrics used in software testing
Branch Coverage
• Instrument model checker to calculate branch coverage
• Using a simple coverage measure as a heuristic doesn’t work well
– Easily falls into local minima (once any branches are taken, every state on that path has “better” coverage)
– Doesn’t distinguish between branches explored once and branches explored many times
The Branch Counting Heuristic
• Count the number of times each branch has been taken
• Heuristic value is then:– Branches never before taken get lowest value– Non-branching transitions are next lowest– Otherwise, score is equal to the count
(lower values are explored first)
Three Searches
CFGBranch Counting
DFS
BFS
Each CFG state is a basic block that incrementssome variable x.
ERROR
Three Searches
CFGBranch Counting
DFS
BFS
Three Searches
CFGBranch Counting
DFS
BFS
Three Searches
CFGBranch Counting
DFS
BFS
Three Searches
CFGBranch Counting
DFS
BFS
Heuristic avoids taking
Three Searches
CFGBranch Counting
DFS
BFS
Three Searches
CFGBranch Counting
DFS
BFS
Expands 15 states
Expands 25 states
Terminatesonly with
depth limit
Experimental Results
• DEOS real-time operating system example
• This version uses an integer valued counter, without abstraction
Results for DEOS
All experiments performed on a 1.4GHz Athlon, limiting Java heap size to 512MB, all times are in seconds
Search Strategy States Time Memory Length Max DepthBranch-count 2,701 60 91MB 136 139%-coverage 20,215 FAIL FAIL FAIL 334Random heuristic 8,057 162 240MB 334 360BFS 18,054 FAIL FAIL FAIL 135DFS 14,678 FAIL FAIL FAIL 14,678DFS depth 500 392,470 6,782 383MB 455 500DFS depth 1000 146,949 2,222 196MB 987 1,000DFS depth 4000 8,481 171 270MB 3,997 4,000
The Interleaving Heuristic
• An important (and very hard to find) class of errors in Java is concurrency errors
• What kind of structure could we explore to catch these?
– Thread-interdependency
The Interleaving Heuristic
• Not clear how to heuristically define actual thread-interdependence
• So we use an approximation:– Executions in which context is switched more
often are given better heuristic values– Explores executions unlikely to appear in
testing (JVM/JITs schedule quite differently)
The Interleaving Heuristic
• Keep track on each path of which threads are executed at each transition
• Give lower (better) heuristic score to paths in which the most recently executed thread has been run less frequently
• Slightly more complicated in practice, counting live threads
Limiting the Queue
• With heuristics we are more interested in finding bugs than in verification
• So, we apply a technique from heuristic search literature:– Limit the size of the priority queue!
• When queue has more than k states in it, remove all but k states with best heuristic values
Experimental Results
• Dining Philosophers– Comparison to other results:
• Godefroid and Khurshid in TACAS ’02 paper apply genetic algorithms to dining philosophers
– Best result reported is 17 philosophers, 177 seconds, 50% success rate (on a slower machine)
• HSF-SPIN– Not clear how to compare (times not given)
– Best result they show is 16 philosophers, and SPIN (using partial order reduction) itself fails with 14 philosophers
Experimental ResultsSearch Strategy Threads States Time Memory Length Max DepthRandom heuristic 8 218,500 FAIL FAIL FAIL 86BFS 8 436,068 FAIL FAIL FAIL 13DFS 8 398,906 FAIL FAIL FAIL 384,286DFS depth 500 8 1,354,747 FAIL FAIL FAIL 500DFS depth 1000 8 1,345,289 FAIL FAIL FAIL 1,000DFS depth 4000 8 1,348,398 FAIL FAIL FAIL 4,000Interleaving 8 487,942 FAIL FAIL FAIL 16Most-blocked 8 310,317 FAIL FAIL FAIL 285Interleaving (k = 1000) 8 354,552 60 137MB 67 67Most-blocked (k = 5) 8 891,177 17,259 378MB 78,353 78,353Most-blocked (k = 160) 8 25,023 10 12MB 172 172Most-blocked (k = 1000) 8 123,640 46 59MB 254 278Interleaving (k = 40) 16 69,987 16 45MB 131 131Interleaving (k = 160) 16 290,637 60 207MB 131 132Most-blocked (k = 40) 16 101,576 38 69MB 1,008 1,008Interleaving (k = 5) 64 101,196 59 206MB 514 514
One Last Heuristic
• The choose-free heuristic:– Works only for abstracted Java programs– Rewards transitions that do not involve
nondeterminism introduced by the abstraction– Prefers counterexamples that do not result from
loss of precision introduced by the abstraction
• Structure of abstraction, not program
Previous Work
• Edelkamp, Lafuente, and Leue– HSF-SPIN: SPIN + heuristic search framework
• Bloem, Ravi, and Somenzi– Symbolic Guided Search: BDDs + heuristics– With BDDs heuristics can aid verification
• Cobleigh, Clarke, and Osterweil– FLAVERS verification work
Conclusions
• Structural heuristics: a useful class of heuristics– When model checking is used for debugging, we may
not know what kinds of bugs we are hunting
• Property-specific heuristics are also useful; approach is complementary, not replacement– Most-blocked can perform as well or better than
interleaving in the Remote Agent example, depending on the k limit and search method
Future Work
• Experiment with other, larger examples• Static analysis for property-specific heuristics• Language for properties/search/heuristics• Discover how heuristics work when symbolic
execution is introduced into JPF• Counterexample analysis for “bug causality”• What other kinds of structure can be exploited
with heuristics?– Counting occurrences of data values, perhaps
top related