satzilla-07: the design and analysis of an algorithm portfolio for sat lin xu, frank hutter, holger...
Post on 18-Dec-2015
215 views
TRANSCRIPT
![Page 1: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/1.jpg)
SATzilla-07: The Design and SATzilla-07: The Design and Analysis of an Algorithm Analysis of an Algorithm
Portfolio for SATPortfolio for SAT
Lin Xu, Frank Hutter,Lin Xu, Frank Hutter,Holger H. Hoos and Kevin Leyton-Brown Holger H. Hoos and Kevin Leyton-Brown
University of British ColumbiaUniversity of British Columbia
{xulin730, hutter, hoos, kevinlb}@cs.ubc.ca{xulin730, hutter, hoos, kevinlb}@cs.ubc.ca
![Page 2: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/2.jpg)
2
Outline
Motivation
History of SATzilla and related work
SATzilla methodology
Example Problem
SATzilla for the SAT Competition
Conclusions and ongoing research
![Page 3: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/3.jpg)
Motivation
Lots of high performance solvers, but …. No single SAT solver dominates all others on
all types of instances
Question: How to select the best solver for a given SAT instance?
![Page 4: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/4.jpg)
4
Algorithm Selection Problem [Rice, 1976]
Reference: Select solvers based on previous experience or research papers
“Winner-Take-All”: Test solvers on some samples from target distribution;
select the solver with best performance.
SATzilla: Automatically based on instance characteristics’
![Page 5: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/5.jpg)
Related work: Portfolio of stochastic algorithm [Gomes & Selman,1997]
Running multiple algorithms at the same time
Reinforcement learning [Lagoudakis & Littman, 2001]
Select branching rule at each decision point
Branch & bound algorithm selection [Lobjois & LemaÎter, 1998]
Based on an estimation of search tree size
![Page 6: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/6.jpg)
6
History of SATzilla Old SATzilla [Nudelman, Devkar, et. al, 2003 ]
2nd Random 2nd Handmade (SAT) 3rd Handmade
SATzilla-07 1st Handmade 1st Handmade (UNSAT) 1st Random 2nd Handmade (SAT) 3rd Random (UNSAT)
![Page 7: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/7.jpg)
7
SATzilla-07 Methodology (offline)
Target
Distribution
Solvers
Features
Final solver
selection
Build Model
Winner-
Take-All
Collect Data
Pre-Solvers
![Page 8: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/8.jpg)
8
SATzilla-07 Methodology (online)
Run
Pre-solver
Compute
Features
Predict
Runtime
Run Winner-
Take-All
Run Best
If error
If error AND time left; run second best
![Page 9: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/9.jpg)
9
Solvers Used Eureka [Nadel, Gordon, Palti & Hanna,
2006]
Kcnfs2006 [Dubois & Dequen, 2006]
March_dl2004 [Heule & Maaren, 2006]
Minisat2.0 [Eén & Sörensson, 2006]
OKsolver [Kullmann, 2002]
Rsat [Pipatsrisawat & Darwiche, 2006]
Vallst [Vallstrom, 2005]
Zchaff_Rand [Mahajan, Fu & Malik, 2005]
![Page 10: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/10.jpg)
SATzilla-07 Example
Using quasi-group completion Problems (QCP) to validate our
general approach
![Page 11: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/11.jpg)
11
SATzilla-07 Example Problem Problem distribution
QCP problems generated near phase transition [Gomes & Selman,
1997]
Solvers Eureka, OKsolver, Zchaff_Rand
Features Same as in previous work [Nudelman, et al. 2004]
Collect Data Compute instances’ features and determine solvers’ runtime
Pre-Solver & “Winner take all” Build Models Final solver selection
![Page 12: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/12.jpg)
12
Empirical Hardness Model (EHM) The Core of SATzilla --- EHM
Accurately predict algorithm’s runtime based on cheaply computable features
Linear basis function regression
Features () Runtime (y)
fw() = wT • 23.34
• 7.21
• …
• …
![Page 13: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/13.jpg)
13
Improve EHM (deal with censoring) Heavy-tailed behavior and censoring
Three ways for censored data Drop them Keep them as if finished at cutoff Censored sampling
Schmee & Hahn ‘s approach [1979] REPEAT
1. Estimate runtime conditional on EHM and real runtime bigger than cutoff runtime
2. Build new EHM with estimated runtimeUNTIL no more changes in EHM
![Page 14: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/14.jpg)
14
How to deal with censored data
A: Drop themB: Finished at cutoff
C: Censor sampling Censor point
![Page 15: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/15.jpg)
15
How to deal with censored data
A: Drop them
B: Finished at cutoffC: Censor sampling
Censor pointCensor point
![Page 16: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/16.jpg)
16
How to deal with censored data
A: Drop them
B: Finished at cutoff
C: Censor sampling Censor point
Censor point
![Page 17: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/17.jpg)
17
Improve EHM (using Hierarchal Hardness Models )
EHM often more accurate, much simpler when trained with SAT/UNSAT samples only [Nudelman, et al. 2004]
Building hierarchal hardness models by
approximate model selection Oracle Mixture of experts problem with fixed experts
![Page 18: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/18.jpg)
SATzilla-07 for QCP
Average Runtime
![Page 19: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/19.jpg)
SATzilla-07 for QCP
Empirical CDF
![Page 20: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/20.jpg)
2007 SAT Competition
Three submissions for 2007 SAT Competition
BIG_MIX for all three category (demo)RANDOM
HANDMADE
![Page 21: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/21.jpg)
21
SATzilla-07 for SAT Competition Target Distribution
Previous SAT competition and SAT Race Solver (with/without preprocessing, Hyper)
Eureka, Kcnfs2006, March_dl2004, Minisat2.0 Vallsat, Rsat, Zchaff_Rand
Features Reduce probing time to 1 second Only cheap features, total about 3 seconds
Pre-Solvers March_dl 5 seconds, SAPS 2 seconds
![Page 22: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/22.jpg)
22
SATzilla-07 for SAT Competition “ Winner take all” solver
March_dl2004 Final candidates
BIG_MIX Eureka, Kcnfs2006, March_dl2004, Rsat RANDOM March_dl2004, Kcnfs2006, Minisat2.0+ HANDMADE March_dl2004, Vallst, March_dl2004+,
Minisat2.0+, Zchaff_Random+
![Page 23: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/23.jpg)
23
SATzilla-07 for BIG_MIX
![Page 24: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/24.jpg)
24
SATzilla-07 for BIG_MIX
Pre-solvers
Feature time
![Page 25: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/25.jpg)
25
SATzilla-07 for RANDOM
![Page 26: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/26.jpg)
26
SATzilla-07 for RANDOM
Pre-solvers
Feature time
![Page 27: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/27.jpg)
27
SATzilla-07 for HANDMADE
![Page 28: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/28.jpg)
28
SATzilla-07 for HANDMADE
Pre-solvers
Feature time
![Page 29: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/29.jpg)
Conclusions
![Page 30: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/30.jpg)
30
Conclusions Can combine algorithms into portfolios,
improving performance and robustness SATzilla approach has been proven to be
successful in real world competition
With more training data and more solvers, SATzilla can be even better
![Page 31: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/31.jpg)
Ongoing research SATzilla for industrial category
Use the same approach, SATzilla is 25% faster and solves 5% more instances
Score function Optimize objective function other than runtime
Local search Improve SATzilla performance by using local
search solvers as component
![Page 32: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/32.jpg)
32
Special Thanks Creators of solvers
Alexander Nadel, Moran Gordon, Amit Palti and Ziyad Hanna (Eureka)
Marijn Heule, Hans van Maaren (March_dl2004) Niklas Eén, Niklas Sörensson (Minisat2.0) Oliver Kullmann (OKsolver) Knot Pipatsrisawat and Adnan Darwiche (Rsat 1.04) Daniel Vallstrom (Vallst) Yogesh S. Mahajan, Zhaohui Fu and Sharad Malik
(Zchaff_Rand)
![Page 33: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/33.jpg)
SATzilla Pick for BIG_MIX
![Page 34: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/34.jpg)
SATzilla Pick for RANDOM
![Page 35: SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT Lin Xu, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown University of British](https://reader036.vdocument.in/reader036/viewer/2022062515/56649d245503460f949fae4a/html5/thumbnails/35.jpg)
SATzilla Pick for HANDMADE