![Page 1: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/1.jpg)
A Performance Study of BDD-Based Model Checking
Bwolen Yang
Randal E. Bryant, David R. O’Hallaron,Armin Biere, Olivier Coudert, Geert Janssen
Rajeev K. Ranjan, Fabio Somenzi
![Page 2: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/2.jpg)
2
Motivation for Studying Model Checking (MC)An important application of BDD Representations
Not well studied
Packages are tuned using combinational circuits (CC)
Qualitative differences between CC and MC computations CC: build outputs, constant time equivalence checking MC: build model, many fixed-points to verify the specs CC: BDD algorithms are polynomial MC: key BDD algorithms are exponential
![Page 3: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/3.jpg)
3
Outline
BDD Overview
Organization of this Study participants, benchmarks, metrics, evaluation process
Experimental Results with and without dynamic variable reordering
BDD Evaluation Methodology evaluation platform
» various BDD packages» real workload
metrics
![Page 4: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/4.jpg)
4
BDD Overview
BDD DAG representation for Boolean functions fixed order on Boolean variables
BDD Algorithms dynamic programming sub-problems (operations): Shannon decomposition memoization: computed cache
![Page 5: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/5.jpg)
5
BDD Overview (Cont’d)
Garbage Collection recycle unreachable (dead) nodes
Dynamic Variable Reordering BDD graph size depends on the variable order sifting based
» nodes in adjacent levels are swapped
![Page 6: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/6.jpg)
6
Organization of this Study:ParticipantsArmin Biere: ABCD
Carnegie Mellon / Universität Karlsruhe
Olivier Coudert: TiGeRSynopsys / Monterey Design Systems
Geert Janssen: EHVEindhoven University of Technology
Rajeev K. Ranjan: CALSynopsys
Fabio Somenzi: CUDDUniversity of Colorado
Bwolen Yang: PBFCarnegie Mellon
![Page 7: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/7.jpg)
7
Organization of this Study:SetupMetrics: 17 statistics
Benchmark: 16 SMV execution traces traces of BDD-calls from verification of
» cache coherence, Tomasulo, phone, reactor, TCAS… size
» 6 million - 10 billion sub-operations» 1 - 600 MB of memory
Evaluation platform: trace driver “drives” BDD packages based on execution trace
![Page 8: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/8.jpg)
8
Organization of this Study:Evaluation ProcessPhase 1: no dynamic variable reorderingPhase 2: with dynamic variable reordering
hypothesize design experiments validate
collect stats
identify issues
suggest improvements validate
Iterative Process
![Page 9: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/9.jpg)
9
Phase 1 Results: Initial / Final Time Comparison
Conclusion: collaborative efforts have led to significant performance improvements
16
13
76
1e+1
1e+2
1e+3
1e+4
1e+5
1e+6
1e+1 1e+2 1e+3 1e+4 1e+5 1e+6
final results (sec)
initi
al re
sults
(sec
)1x10x100x
n/a
n/a
s
![Page 10: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/10.jpg)
10
Computed Cache effects of computed cache size amounts of repeat sub-problems across time
Garbage Collection death rate rebirth rate
Complement Edge Representation work space
Memory Locality for Breadth-First Algorithms
Computed Cache effects of computed cache size amounts of repeat sub-problems across time
Garbage Collection death rate rebirth rate
Complement Edge Representation work space
Memory Locality for Breadth-First Algorithms
Phase 1:Hypotheses / Experiments
![Page 11: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/11.jpg)
11
Phase 1:Hypotheses / Experiments (Cont’d)For Comparison
ISCAS85 circuits (> 5 sec, < 1GB) c2670, c3540 13-bit, 14-bit multipliers based on c6288
Metrics depends only on the trace and BDD algorithms machine-independent implementation-independent
![Page 12: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/12.jpg)
12
Computed Cache:Repeated Sub-problems Across TimeSource of Speedup
increase computed cache size
Possible Cause many repeated sub-problems are far apart in time
Validation study the number of repeated sub-problems across user issued operations (top level operations).
![Page 13: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/13.jpg)
13
Hypothesis:Top-Level Sharing
HypothesisMC computations have a large number of repeated sub-problems across the top-level operations.
Experiment measure the minimum number of operations with GC disabled and complete cache. compare this with the same setup, but cache is flushed between top-level operations.
![Page 14: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/14.jpg)
14
Results on Top-Level Sharing
flush: cache flushed between top-level operations
Conclusion: large cache is more important for MC
1
10
100
Model Checking Traces ISCAS85
# of
ops
(flus
h / m
in)
![Page 15: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/15.jpg)
15
Garbage Collection:Rebirth RateSource of Speedup
reduce GC frequency
Possible Cause many dead nodes become reachable again (rebirth)
» GC is delayed till the number of dead nodes reaches a threshold» dead nodes are reborn when they are part of the result of new sub-problems
![Page 16: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/16.jpg)
16
Hypothesis:Rebirth Rate
HypothesisMC computations have very high rebirth rate.
Experimentmeasure the number of deaths and the number of rebirths
![Page 17: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/17.jpg)
17
Results on Rebirth Rate
0
0.2
0.4
0.6
0.8
1
Model Checking Traces ISCAS85
rebi
rths
/ dea
ths
Conclusions delay garbage collection triggering GC should not base only on # of dead nodes delay updating reference counts
![Page 18: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/18.jpg)
18
BF BDD Construction
Breadth-first based BDD construction is not a demonstrated advantage over traditional
depth-first based techniques.
Two packages (CAL and PBF) are BF based.
![Page 19: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/19.jpg)
19
BF BDD Construction Overview
Level-by-Level Access operations on same level (variable) are processed together one queue per level
Localitygroup nodes of the same level together in memory
Good memory locality due to BF ==> # of ops processed per queue visit must be high
![Page 20: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/20.jpg)
20
Average BF Locality
Conclusion: MC traces generally have less BF locality
0
1000
2000
3000
4000
5000
Model Checking Traces ISCAS85
# of
ops
pro
cess
edpe
r lev
el-v
isit
![Page 21: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/21.jpg)
21
Average BF Locality / Work
Conclusion: For comparable BF locality,MC computations do much more work.
0
50
100
150
200
250
Model Checking Traces ISCAS85
avg.
loca
lity
/ # o
f ops
(x 1
e-6)
![Page 22: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/22.jpg)
22
Phase 1:Some Issues / Open QuestionsMemory Management
space-time tradeoff» computed cache size / GC frequency
resource awareness» available physical memory, memory limit, page fault rate
Top-Level Sharing
![Page 23: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/23.jpg)
23
Phase 2:Dynamic Variable ReorderingBDD Packages Used
CAL, CUDD, EHV, TiGeR improvements from phase 1 incorporated
Variable Reordering is hard to study ==> limited experimental results
![Page 24: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/24.jpg)
24
Why is Variable Reordering Hard to StudyTime-space tradeoff
how much time to spent to reduce graph sizes
Chaotic behaviore.g., small changes to triggering / termination algorithm
can have significant performance impact
Resource intensive reordering is expensive space of possible ordering is combinatorial
Different variable order ==> different computatione.g., many “don’t-care space” optimization algorithms
![Page 25: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/25.jpg)
25
Quality of Variable Order Generated
Variable Grouping Heuristic keep strongly related variables adjacent
Reorder Transition Relation BDDs for the transition relation are used repeatedly
Effects of Initial Variable Order for both with and without variable reordering
Quality of Variable Order Generated
Variable Grouping Heuristic keep strongly related variables adjacent
Reorder Transition Relation BDDs for the transition relation are used repeatedly
Effects of Initial Variable Ordering for both with and without variable reordering
Phase 2:Experiments
Only CUDD is used
![Page 26: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/26.jpg)
26
Variable Grouping Heuristic:Group Current / Next Variables
Current / Next State Variables for transition relation, split variable into two copies:
» current state and next state
HypothesisGrouping the corresponding current- and next-state variables is a good heuristic.
Experiment compare results between with and without grouping
» work (# of operations)» space (max # of live BDD nodes)» reorder cost (# of nodes swapped with their children)
![Page 27: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/27.jpg)
27
Results onGrouping Current / Next Variables
Space
0
1
2
3
MC traces
max
live
nod
es
Reorder Cost
0
1
2
3
MC traces
node
s sw
appe
d
Work
0
1
2
3
4
MC traces
# of
ops
All results are normalized against no variable grouping.Conclusion: grouping is generally effective
![Page 28: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/28.jpg)
28
Effects of Initial Variable Order:Experimental SetupFor each trace,
find a good variable ordering O perturb O to generate new variable orderings
» fraction of variables perturbed» distance moved
use these new orderings to study the results with and without dynamic variable reordering
![Page 29: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/29.jpg)
29
Effects of Initial Variable Order:Perturbation AlgorithmPerturbation Parameters (p, d)
p: probability that a variable will be perturbed d: perturbation distance
Properties in average, p fraction of variables is perturbed max distance moved is 2d
Note (p = 1, d = infinity) ==> completely random variable order
![Page 30: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/30.jpg)
30
Effects of Initial Variable Order:ParametersFor each perturbation level (p, d)
generate a number (k) of random variable orders
Parameter Values p: (0.1, 0.2, …, 1.0) d: (10, 20, …, 100, infinity) k: 10
==> for each trace, 1100 orderings 2200 runs (w/ and w/o dynamic reordering)
![Page 31: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/31.jpg)
31
Effects of Initial Variable Order:Smallest Test CaseBase Case (best ordering)
time: 13 sec memory: 127 MB
Limits on Generated Orders time: 128x base case memory: 500 MB
![Page 32: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/32.jpg)
32
Effects of Initial Variable Order: Result # of unfinished cases
At 128x/500MB limit, “no reorder” finished 33%, “reorder” finished 90%.
Conclusion: reordering alg. is effective
0
400
800
1200
>1x >2x >4x >8x >16x >32x >64x >128x
time limit
# of
cas
esno reorderreorder
![Page 33: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/33.jpg)
33
Phase 2:Some Issues / Open QuestionsComputed Cache Flushing
cost
Effects of Initial Variable Order determine sample size k
Need New Better Experimental Design
![Page 34: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/34.jpg)
34
BDD Evaluation Methodology
Trace-Driven Evaluation Platform real workload (BDD-call traces) study various BDD packages focus on key operations
Evaluation Metrics more rigorous quantitative analysis
![Page 35: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/35.jpg)
35
BDD Evaluation MethodologyMetrics: Time
elapsed time(performance)
CPU time page fault rate
BDD Alg timeGC time reorderingtime
# of GCs # of node swaps(reorder cost)
memoryusage
# of ops(work)
# of reorderings
computed cache size
![Page 36: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/36.jpg)
36
BDD Evaluation MethodologyMetrics: Space
memoryusage
# of GCs # of reorderings
computed cache size
max # ofBDD nodes
![Page 37: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/37.jpg)
37
Importance of Better MetricsExample: Memory Locality
# of GCs # of reorderings
computed cache size
elapsed time(performance)
CPU time page fault rate
cache miss rate TLB miss rate
memory locality# of ops(work)
without accountingfor work, may draw
premature conclusions
![Page 38: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/38.jpg)
38
Summary
Collaboration + Evaluation Methodology significant performance improvements
» up to 2 orders of magnitude characterization of MC computation
» computed cache size» garbage collection frequency» effects of complement edge» BF locality » reordering heuristic
– current / next state variable grouping» effects of reordering the transition relation» effects of initial variable orderings
other general results (not mentioned in this talk) issues and open questions for future research
![Page 39: A Performance Study of BDD-Based Model Checking](https://reader035.vdocument.in/reader035/viewer/2022070520/58f15a151a28abca408b45d9/html5/thumbnails/39.jpg)
39
Conclusions
Rigorous BDD evaluation can lead to dramatic results
Adopt the Evaluation Methodology more benchmark traces
» for IP issue, BDD-call trace is like assembly language use / improve on the metrics proposed for future evaluation
For data, traces, trace-driver used in this study,http://www.cs.cmu.edu/~bwolen/fmcad98/