sampling techniques for boolean satisfiability
DESCRIPTION
Kuldeep S Meel 1 (Joint work with Supratik Chakraborty 2 , Moshe Y Vardi 1 ). Sampling Techniques for Boolean Satisfiability. 1 Department of Computer Science, Rice University 2 Indian Institute of Technology Bombay, India. COMP 600. Sept 9, 2013. Life in The 21 st Century!. - PowerPoint PPT PresentationTRANSCRIPT
1
Sampling Techniques for Boolean Satisfiability
Kuldeep S Meel1
(Joint work with Supratik Chakraborty2, Moshe Y Vardi1)
1Department of Computer Science, Rice University2Indian Institute of Technology Bombay, India
Sept 9, 2013 COMP 600
2
Life in The 21st Century!
Was my “I Love You” message/email to my girlfriend delivered to her or her roommate?
3
Life in The 21st Century!
How do we guarantee that the systems work correctly ?
4
Motivating Example
a b
c
64 bit 64 bit
64 bit
c = ab
How do we verify that this circuit works ?• Try for all values of a and b• 2128 possibilities (1022 years)• Not scalable
• Randomly sample some a’s and b’s• Wait! None of the circuits in
the past faulted when 10 < b < 40
• Finite resources! • Let’s sample from regions where
it is likely to fault
5
Constraints DesignDesigning Constraints
• Designers: 1. 100 < b < 2002. 300 < a < 4513. 40 < a < 50 and 30 < b <
40• Past Experience:
1. 400 < a < 20002. 120 < b < 230
• Users:1. 1000<a < 11002. 20000 < b < a < 22000
Problem: How can we uniformly sample the values of a and b satisfying the above constraints?
a b
c
64 bit
64 bit
c = ab
64 bit
6
Set of Constraints
SAT Formula
Uniform Generation of SAT-Witnesses
{(0,1), (1,0), (1,1)}
SAT is NP-complete (Cook 1971)
(a V b)
7
Set of Constraints
Given a SAT formula, can one uniformly sample solutions without enumerating all solutions
SAT Formula
Uniform Generation of SAT-Witnesses
Uniform Generation of SAT-Witnesses
8
Set of Constraints
Given a SAT formula, can one uniformly sample solutions without enumerating all solutions while scaling to real world problems?
SAT Formula
Scalable Uniform Generation of SAT-Witnesses
Uniform Generation of SAT-Witnesses
9
Overview
Uniform Generation of SAT-witnesses
Approximate Model Counting
Future Directions
10
Overview
Uniform Generation of SAT-witnesses
Approximate Model Counting
Future Directions
11
Prior Work
Heuristic WorkGuarantees: weakPerformance: strong
BGP Algorithm XORSample’
Theoretical WorkGuarantees: strongPerformance: weak
BDD-basedGuarantees: strongPerformance: weak
SAT-based heuristics
Guarantees: weakPerformance: strong
INDUSTRY
ACADEMIA
12
Our Contribution
Heuristic WorkGuarantees: weakPerformance: strong
BGP Algorithm XORSample’
Theoretical WorkGuarantees: strongPerformance: weak
BDD-basedGuarantees: strongPerformance: weak
SAT-based heuristics
Guarantees: weakPerformance: strong
INDUSTRY
ACADEMIA
UniWitGuarantees : strongPerformance: strong
13
Central Idea
14
Partitioning into equal “small” cells
15
How to Partition?
How to partition into roughly equal small cells of solutions without knowing the distribution of solutions?
Universal Hashing[Carter-Wegman 1979, Sipser 1983]
16
Universal Hashing Hash functions from mapping {0,1}n to {0,1}m
(2n elements to 2m cells)
Random data => All cells are roughly small
Universal hash functions: Adversarial data => All cells are roughly small
Need stronger bounds on range of the size of cells
17
Lower Universality Lower Complexity
H(n,m,r): Family of r-universal hash functions mapping {0,1}n to {0,1}m (2n
elements to 2m cells)
Higher the r => Stricter guarantees on range of size of cells
r-wise universality => Polynomials of degree r-1
Lower universality => lower complexity
18
Hashing-Based Approaches
n-universal hashing
Uniform Generation
All cells should be small
BGP Algorithm
Solution space
19
Scaling to Thousands of Variables
n-universal hashing 2-universal hashing
Uniform Generation
Random
All cells should be small
Only a randomly chosen cells needs to be “small”
BGP Algorithm
Near Uniform Generation
UniWit
Solution space
20
n-universal hashing 2-independent hashing
Uniform Generation
Random
All cells should be small
Only a randomly chosen cells needs to be “small”
BGP Algorithm
Near Uniform Generation
UniWit
Solution space
From tens of variables to thousands of variables!
Scaling to Thousands of Variables
21
Highlights Employs XOR-based hash functions
instead of computationally infeasible algebraic hash functions
Uses off-the-shelf SAT solver CryptoMiniSAT (MiniSAT+XOR support)
22
Strong Theoretical Guarantees Uniformity
For every solution y of RF Pr [y is output] = 1/|RF|
23
Strong Theoretical Guarantees Near Uniformity
Success Probability
Polynomial calls to SAT Solver
For every solution y of RF Pr [y is output] >= 1/8 x 1/|RF|
Algorithm UniWit succeeds with probability at least 1/8
24
Experimental Methodology Benchmarks (over 200)
Bit-blasted versions of word-level constraints from VHDL designs
Bit-blasted versions from SMTLib version and ISCAS’85
Objectives Comparison with algorithms BGP &
XORSample’
Uniformity Performance
25
Better Uniformity than State-of-art Generators
XORSample’ UniWit
• Benchmark: case110.cnf; #var: 287; #clauses: 1263
• Total Runs: 1.08x108; Total Solutions : 16384• XORSample’ could not find 772 solutions and
more than 250 solutions were generated only once
26
Results : Performance
case
47
case
105
case
203
case
61
case
15
case
_2_b
14_1
squa
ring1
4
case
_2_p
tb_1
case
_2_b
14_2
0.11
10100
100010000
100000
UniWitXORSample'
Benchmarks
Time(s)
27
Results : Performance
case
47
case
105
case
203
case
61
case
15
case
_2_b
14_1
squa
ring1
4
case
_2_p
tb_1
case
_2_b
14_2
0.11
10100
100010000
100000
UniWitXORSample'
Benchmarks
Time(s)
• UniWit is is 2-3 orders of magnitude faster than XORSample’
• Observed success probability = 0.6 ( >> theoretical guarantee of 0.125)
28
The Story So Far Theoretical guarantees of near
uniformity Major improvements in running time and
uniformity compared to existing generators
But………. How many samples should I test my system to achieve desired coverage? Are 105 samples enough?
Case A: Total solutions -106
Case B: Total solutions - 1060
29
The missing link
What is the total number of satisfying assignments to system of constraints?
30
Overview
Uniform Generation of SAT-witnesses
Approximate Model Counting
Future Directions
31
What is Model Counting? Given a SAT formula F RF: Set of all solutions of F Problem (#SAT): Estimate the number of
solutions of F (#F) i.e., what is the cardinality of RF?
E.g., F = (a v b) RF = {(0,1), (1,0), (1,1)} The number of solutions (#F) = 3#P: The class of counting problems
for decision problems in NP!
32
Practical ApplicationsExciting range of applications!
Probabilistic reasoning/Bayesian inference
Planning with uncertainty
Multi-agent/ adversarial reasoning [Roth 96, Sang 04, Bacchus 04, Domshlak 07]
33
But it is hard! #SAT is #P-complete
Even for counting solutions of 2-CNF SAT
#P is really hard! Believed to be much harder than NP-
complete problems PH P#P
34
The Hardness of Model Counting
35
The Hardness of Model Counting
Can we do better?Approximate counting (with guarantees) suffices for most of the applications
36
Prior Work
Counters Guarantee Confidence
Remarks
Exact counter (e.g. sharpSAT, Cachet)
C = #F 1 Poor Scalability
Lower bound counters (e.g. MBound, SampleCount)
C ≤ #F d Very weak guarantees
Upper bound counters(e.g. MiniCount)
C ≥ #F d Very weak guarantees
Input Formula: F; Total Solutions: #F; Return Value: C
37
Approximate Model CountingDesign an approximate model counter G: inputs:
CNF formula F tolerance e confidence d
the count returned by it is within e of the #F with confidence at least d
Approximate Model Counting
38
Approximate Model CountingDesign an approximate model counter G: inputs:
CNF formula F tolerance e confidence d
the count returned by it is within e of the #F with confidence at least d and scales to real world problems
Scalable Approximate Model CountingLies in the 2nd level of Polynomial hierarchy: S2
P
39
Our ContributionInput Formula: F; Total Solutions: #FCounters Guarantee Confidenc
eRemarks
Exact counter (e.g. sharpSAT, Cachet)
C = #F 1 Poor Scalability
ApproxMC #F/(1+e)≤ C ≤ (1+ e) #F
d Scalability + Strong guarantees
Lower bound counters (e.g. MBound, SampleCount)
C ≤ #F d Very weak guarantees
Upper bound counters(e.g. MiniCount
C ≥ #F d Very weak guarantees
The First Scalable Approximate Model
Counter
40
How do we count?
41
Naïve Enumeration: Not Scalable
Not Scalable! (Think of enumerating 2100 solutions)
• Enumerate all solutions
• Exact Counting!• Cachet, Relsat,
sharpSAT
42
Counting through Partitioning
43
Counting through Partitioning
Pick a random cell
Total # of solutions= #solutions in the cell* total # of cells
44
Algorithm in Action
690 710 730 730 731 831 834………….…
t
Algorithm
45
Algorithm in Action
Algorithm
690 710 730 730 731 831 834………….…
t
Median
46
Partitioning
Linear hash functions (3-wise independent)
How to partition into roughly equal cells of solutions without knowing the distribution of solutions?
47
Theoretical ResultsApproxMC (CNF: F, tolerance: e, confidence:d)Suppose ApproxMC(F,e,d) returns C. Then,
Pr [ #F/(1+e)≤ C ≤ (1+ e) #F ] ≥ d
ApproxMC runs in time polynomial in log (1-d)-1,|F|, e-1 relative to SAT oracle
48
Experimental Methodology Benchmarks (over 200)
Grid networks, DQMR networks, Bayesian networks
Plan recognition, logistics problems Circuit synthesis
Tolerance: e= 0.75, Confidence: d = 0.9 Objectives
Comparison with exact counters (Cachet) & bounding counters (MiniCount, Hybrid-MBound, SampleCount) Performance Quality of bounds
49
Results: Performance Comparison
0 6 12 18 24 30 36 42 48 54 60 66 72 78 84 90 96 1021081141201261321381441501561621681741801860
10000
20000
30000
40000
50000
60000
70000
ApproxMCCachet
50
Results: Performance Comparison
0 6 12 18 24 30 36 42 48 54 60 66 72 78 84 90 96 1021081141201261321381441501561621681741801860
10000
20000
30000
40000
50000
60000
70000
ApproxMCCachet
51
Can Solve a Large Class of Problems
Large class of problems that lie beyond the exact counters but can be computed by ApproxMC
0 6 12 18 24 30 36 42 48 54 60 66 72 78 84 90 96 1021081141201261321381441501561621681741801860
10000
20000
30000
40000
50000
60000
70000
ApproxMCCachet
52
Mean Error: Only 4% (allowed: 75%)
Mean error: 4% – much smaller than the theoretical guarantee of 75%
0 10 20 30 40 50 60 70 80 903.2E+01
1.0E+03
3.3E+04
1.0E+06
3.4E+07
1.1E+09
3.4E+10
1.1E+12
3.5E+13
1.1E+15
3.6E+16
Cachet*1.75Cachet/1.75ApproxMC
53
Results: Bounding Counters Range of count from bounding counters = C2-C1
C1: From lower bound counters(MBound/SampleSAT) C2: From upper bound counters (MiniCount)
Range from ApproxMC: [C/(1+e), (1+e)C]
Smaller the range, better the algorithm!
54
Better Bounds Than Existing Counters
ApproxMC improved the upper bounds significantly while also improving the lower bounds
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 622.6E+02
8.2E+03
2.6E+05
8.4E+06
2.7E+08
8.6E+09
2.7E+11
8.8E+12
2.8E+14
9.0E+15
2.9E+17
ApproMCMBound/SampleCount/Mini-Count
55
Overview
Uniform Generation of SAT-witnesses
Approximate Model Counting
Future Directions
56
Future Directions Extension to more expressive constraints (SMT)
Design of independent hash functions SMT solver – efficiently handles input + hash
function
Strong guarantees for uniform generation
Handling user bias (weighted uniform generation and weighted counting)
Scaling beyond thousands of variables
57
Collaboration Opportunities MOOC: Automatic Problem Generation
Use of templates to design questions Uniform generation of template parameters
Applications in Program Synthesis: e.g., Autotuning
Probabilistic reasoning/Bayesian inference Planning with uncertainty/adversarial
reasoning Extension to general set of constraints
Counting points in polyhedra (Loop optimization) Discrete Integration
58
Key Takeaways Uniform sampling and approximate counting are
important problems Prior work didn’t scale or offered weak guarantees UniWit: Scales and theoretical guarantees of near
uniformity ApproxMC: The first scalable approximate model counter Uses easy-to-implement linear hash functions Only a randomly chosen cell has to be small Tools are available online! Go and Try them out!
59
Publications S. Chakraborty, K.S. Meel, M.Y. Vardi “A
Scalable and Nearly-Uniform Generator of SAT-Witnesses” In Proc. of CAV 2013
S. Chakraborty, K.S. Meel, M.Y. Vardi “A Scalable Approximate Model Counter” In Proc. of CP 2013
60
Discussion
Thank You for your attention!
Acknowledgments• NSF• ExCAPE• Intel• BRNS, India• Sun Microsystems• Sigma Solutions,Inc
61
Key Takeaways Uniform sampling and approximate counting
are important problems Prior work either didn’t scale or offered weak
guarantees UniWit: Scales and provides theoretical
gurantees of near uniformity ApproxMC: The first scalable approximate
model counter Uses easy-to-implement linear hash functions Only a randomly chosen cell has to be small Tools are available online! Go and Try them
out!
62
UniWitRF
63
UniWit
IsSmall?
RF
NO
64
UniWit
?
65
UniWit
? NOIsSmall?
66
UniWit
?
??
???
67
UniWit
?
??
???
IsSmall?
YES
68
UniWit
IsSmall?
YES Select a solution randomly with probability “c” from the partition. If no solution is picked, return Failure