hierarchical hardness models for sat
DESCRIPTION
Hierarchical Hardness Models for SAT. Lin Xu, Holger H. Hoos and Kevin Leyton-Brown University of British Columbia {xulin730, hoos, kevinlb}@cs.ubc.ca. One Application of Hierarchical Hardness Models --- SATzilla-07. Old SATzilla [ Nudelman, Devkar, et. al, 2003 ] 2nd Random - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/1.jpg)
1
Hierarchical Hardness Models Hierarchical Hardness Models for SATfor SAT
Lin Xu, Holger H. Hoos Lin Xu, Holger H. Hoos and Kevin Leyton-Brown and Kevin Leyton-Brown
University of British ColumbiaUniversity of British Columbia
{xulin730, hoos, kevinlb}@cs.ubc.ca{xulin730, hoos, kevinlb}@cs.ubc.ca
![Page 2: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/2.jpg)
22
One Application of Hierarchical Hardness Hierarchical Hardness Models --- SATzilla-07Models --- SATzilla-07 Old SATzilla [Nudelman, Devkar, et. al, 2003 ]
2nd Random 2nd Handmade (SAT) 3rd Handmade
SATzilla-07 [Xu, et.al, 2007]
1st Handmade 1st Handmade (UNSAT) 1st Random 2nd Handmade (SAT) 3rd Random (UNSAT)
![Page 3: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/3.jpg)
3
Outline Introduction Motivation Predicting the Satisfiability of SAT
Instances Hierarchical Hardness Models Conclusions and Futures Work
![Page 4: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/4.jpg)
4
Introduction
![Page 5: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/5.jpg)
5
Introduction NP-hardness and solving big problems Using empirical hardness model to predict
algorithm’s runtime Identification of factors underlying algorithm’s
performance Induce hard problem distribution Automated algorithm tuning [Hutter, et al. 2006]
Automatic algorithm selection [Xu, et al. 2007]
…
![Page 6: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/6.jpg)
6
Empirical Hardness Model (EHM) Predicted algorithm’s runtime based on
poly-time computable features Features: anything can characterize the problem
instance and can be presented by a real number 9 category features [Nudelman, et al, 2004]
Prediction: any machine learning technique can return prediction of a continuous value
Linear basis function regression [Leyton-Brown et al, 2002; Nudelman, et al, 2004]
![Page 7: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/7.jpg)
7
Linear Basis Function Regression
Features () Runtime (y)
fw() = wT • 23.34
• 7.21
• …
• …
( | ) ( | , )TP y N y w
![Page 8: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/8.jpg)
8
Motivation
![Page 9: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/9.jpg)
9
Motivation If we train EHM for SAT/UNSAT separately (Msat, Munsat),
Models are more accurate and much simpler
[Nudelman, et al, 2004]Moracular: always use good modelMuncond.: Trained on mixture of instance
Solver: satelite; Dataset: Quasi-group completion problem
![Page 10: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/10.jpg)
10
Motivation If we train EHM for SAT/UNSAT separately (Msat, Munsat),
Models are more accurate and much simpler Using wrong model can cause huge error
Munsat: only use UNSAT modelMsat: only use SAT model
Solver: satelite; Dataset: Quasi-group completion problem
![Page 11: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/11.jpg)
11
Motivation
Question: How to select the right model for a given problem?
Attempt to guess Satisfiability of given instance!
![Page 12: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/12.jpg)
12
Predicting Satisfiability of SAT Instances
![Page 13: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/13.jpg)
13
Predict Satisfiability of SAT Instances NP-complete No poly-time method
has 100% accuracy of classification
BUT For many types of SAT instances, classification
with poly-time features is possible Classification accuracies are very high
![Page 14: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/14.jpg)
14
Performance of Classification Classifier: SMLR Features: same as
regression
Datasets rand3sat-var rand3sat-fix QCP SW-GCP
Dataset Classification Accuracy
sat. unsat. overall
rand3sat-var 0.979 0.989 0.984
rand3sat-fix 0.848 0.881 0.865
QCP 0.980 0.932 0.960
SW-QCP 0.752 0.711 0.734
[Krishnapuram, et al. 2005]
![Page 15: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/15.jpg)
1515
Performance of Classification (QCP)
![Page 16: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/16.jpg)
1616
Performance of Classification
For all datasets: Much higher accuracies than random
guessing
Classifier output correlates with classification accuracies
High accuracies with few features
![Page 17: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/17.jpg)
17
Hierarchical Hardness Models
![Page 18: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/18.jpg)
1818
Back to QuestionQuestion: How to select the right model for a
given problem?
Can we just use the output of classifier?NO, need to consider the error distribution
Note: best performance would be achieved by model selection Oracle!
Question: How to approximate model selection oracle based on features
![Page 19: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/19.jpg)
1919
Hierarchical Hardness Models
, s Features &
Classifier output s: P(sat)
Z Model selection
Oracle
y
Runtime
Mixture of experts problem with FIXED experts,
use EM to find the parameters for z [Murphy, 2001]
{ , }
( | , ) ( | , ) ( | , )Tz z
z sat unsat
P y s P z s N y w
![Page 20: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/20.jpg)
20
B
A
Importance of Classifier’s Output
Two ways:A: Using
classifier’s output as a feature
B: Using classifier’s output for EM initialization
A+B
![Page 21: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/21.jpg)
21
Big Picture of HHM Performance
Oracular Uncond. Hier. Oracular Uncond. Hier.
Solver RMSE (rand3-var) RMSE (rand3-fix)
satz 0.329 0.358 0.344 0.343 0.420 0.413
march_dl 0.283 0.396 0.306 0.444 0.542 0.533
kcnfs 0.294 0.373 0.312 0.397 0.491 0.486
Oksolver 0.356 0.443 0.378 0.479 0.596 0.587
Solver RMSE (QCP) RMSE (SW-GCP)
Zchaff 0.303 0.675 0.577 0.657 0.993 0.983
Minisat 0.305 0.574 0.500 0.682 1.022 1.024
Satzoo 0.240 0.397 0.334 0.384 0.581 0.581
Satelite 0.247 0.426 0.372 0.618 0.970 0.978
Sato 0.375 0.711 0.635 0.723 1.352 1.345
oksolver 0.427 0.548 0.506 0.601 1.337 1.331
![Page 22: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/22.jpg)
22
Example for rand3-var (satz)
Left: unconditional model Right: hierarchical model
![Page 23: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/23.jpg)
23
Example for rand3-fix (satz)
Left: unconditional model Right: hierarchical model
![Page 24: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/24.jpg)
24
Example for QCP (satelite)
Left: unconditional model Right: hierarchical model
![Page 25: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/25.jpg)
25
Example for SW-GCP (zchaff)
Left: unconditional model Right: hierarchical model
![Page 26: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/26.jpg)
26
Correlation Between Prediction Error and Classifier’s Confidence
Left: Classifier’s output vs runtime prediction error
Right: Classifier’s output vs RMSE
Solver: satelite; Dataset: QCP
![Page 27: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/27.jpg)
27
Conclusions and Futures Work
![Page 28: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/28.jpg)
2828
Conclusions Models conditioned on SAT/UNSAT have much
better runtime prediction accuracies.
A classifier can be used to distinguish SAT/UNSAT with high accuracy.
Conditional models can be combined into hierarchical model with better performance.
Classifier’s confidence correlates with prediction error.
![Page 29: Hierarchical Hardness Models for SAT](https://reader036.vdocument.in/reader036/viewer/2022062500/56815972550346895dc6b7cb/html5/thumbnails/29.jpg)
29
Future Work Better features for SW-GCP
Test on more real world problems
Extend underlying experts beyond satisfiability