satzilla: portfolio-based algorithm selection for sat
DESCRIPTION
SATzilla: Portfolio-based Algorithm Selection for SAT. Lin Xu , Frank Hutter , Holger H. Hoos , and Kevin Leyton -Brown Department of Computer Science University of British Columbia. Mainly based on [ Xu , Hutter , Hoos , Leyton -Brown, JAIR, 2008]. SATzilla. - PowerPoint PPT PresentationTRANSCRIPT
SATzilla: Portfolio-based Algorithm Selection for SAT
Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown
Department of Computer ScienceUniversity of British Columbia
Mainly based on [Xu, Hutter, Hoos, Leyton-Brown, JAIR, 2008]
SATzilla
• Excellent performance in SAT competitions [http://www.satcompetition.org/]
– 2 second places, one third place in 2003 SAT competition [Leyton-Brown, Nudelman, Andrew, McFadden & Shoham, 2003]
– 3 Gold, 1 Silver, 1 Bronze in 2007 SAT competition [Xu, Hutter, Hoos, & Leyton-Brown, 2007]
– 3 Gold, 2 Silver in 2009 SAT competition [Xu, Hutter, Hoos & Leyton-Brown, 2009]
• One gold in every category: Industrial/Handmade/Random
• Domain independent– MIPzilla, CSP, Planning …
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla 2
Solvers
InstanceInstance Feature extractor
Algorithm selector
Impact on the SAT community
Encouraged research on portfolio-based solvers for SAT
Final 2011 SAT competition results[http://www.satcompetition.org/]
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla
Category Random Handmade Industrial
Medalists
3Sppfolio (2) ppfolio seq
sparrow2011 sattime2011
eagleup march_rw
mphasesat_m
3S (3)Ppfolio (2)
ppfolio seq (2) clasp
glucose
glucose (2)glueminisat (2)
lingeling contrasat hack cirminisat hack mphasesat64
qutersat
Total medals 4/9 7/9 0/9
3
Outline
• Introduction• Related Work: – Algorithm selection
• SATzilla overview:– Building SATzilla – Running SATzilla
• SATzilla07 and beyond• Results • SATzilla since 2008• Conclusion
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla 4
Related work: algorithm selection
Select the best solver for each instance [Rice, 76]
– Regression-based• Linear regression (SATzilla)
[Leyton-Brown, Nudelman, Andrew, McFadden & Shoham, 2003; Xu, Hutter, Hoos, Leyton-Brown, 2007+]
• Matchbox [Stern et al., 2010]
– Classification-based• Logistic regression [Samulowitz & Memisevic, 2007]
• K-NN [O’Mahony & Hebrard, 2008]
• Decision tree [Guerri & Milano, 2004]
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla 5
What is SATzilla?
• Portfolio-based algorithm selection based on empirical hardness models [Leyton-Brown et al., 2002;2009]
– Use regression models to predict each algorithm’s performance
– Select the predicted best
• Many new techniques for improving robustness– Pre-solvers – Backup solver– Solver subset selection
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla 6
SATzilla: prerequisites
Identify a target distribution of instances
Select a set of candidate solvers, and measure runtime on a training set
Identify a set of features based on domain knowledge and collect feature values– Cheap to compute– Informative for prediction
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla 7
SATzilla core: predictive modelsGoal: accurately predict solver performance based on cheaply
computable features
Supervised machine learning: learn function f: features → performance
– Input: <features, performance> pairs– Output: f
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla
fw() = wT
Performance(y)Features ()
• 23.34• 7.21• …• …
Algorithm selector
8
SATzilla: selection based on models
• Predict each solver’s performance• Pick solver based on predictions
Feature extractor
Algorithm selector
InstanceInstanceInstanceInstance
Compute features
Make prediction
Run on selected solver
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla 9
Improving robustness
• Solver subset selection– More candidate solvers can hurt– select subset of solvers that achieves best overall performance
• Pre-solvers– Solve some instances quickly without need for feature computation– Increase model robustness
• Backup solver – In case something goes wrong
Feature extractor
Algorithm selector
Dealing with imperfect predictions:
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla
Algorithm selector
10
Time to run SATzilla!
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla 11
Step 1: pre-solving
Run pre-solvers for a fixed short amount of time– If solved, done
InstanceInstance
Feature extractor
Algorithm selector
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla 12
Step 2: compute features
Run feature extractor to get a set of feature values– If solved, done– If anything fails, simply run backup solver
InstanceInstance
Feature extractor
Algorithm selector
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla 13
Step 3: performance prediction
Predict performance of solvers and order them according to runtime predictions
Feature extractor
Algorithm selector
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla 14
Step 4: run top-ranked solver
Run the top-ranked solver. If a run fails, run the next-ranked solver.
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla 15
Feature extractor
Algorithm selector
InstanceInstance
SATzilla07: construction
Pre-solvers: manually selected
Backup solver: overall best
Model building:– Predict runtime– Hierarchical hardness models [Xu, Hoos, Leyton-Brown, 2007]
– Models handle “censored data” [Schmee & Hahn, 1979]
Solver subset selection: exhaustive searchXu, Hutter, Hoos, and Leyton-Brown: SATzilla 16
Improving SATzilla07 (1.)
Problem: manual pre-solver selection is not idealSolution: auto pre-solver selection based on validation
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla 17
Feature extractor
Algorithm selector
Improving SATzilla07 (2.-5.)
Problem: overall best solver best backup solverSolution: choose the best solver for instances not solved by pre-solvers and with problematic features
Problem: objective is not always runtimeSolution: predict and optimize true objective directly also enables cleaner use of incomplete solvers
Problem: exhaustive search for solver subset selection is infeasible for large portfolios Solution: local search for solver subset selection
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla 18
SATzilla vs candidate solvers
HANDMADE
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla 19
SATzilla vs candidate solvers
HANDMADE
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla
Oracle
20
SATzilla vs candidate solvers
HANDMADE
Pre-solving
Feat. Comp.
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla
Oracle
SATzilla07
21
SATzilla vs candidate solvers
HANDMADE
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla
Oracle
SATzilla07
SATzilla07(improved)
22
SATzilla09 [Xu et al., 2009]
New techniques– New features: clause learning, …– Prediction of feature computation cost
Feature time predictor
InstanceInstance
Minimal cost feature
extractor
Results: 3 Gold, 2 Silver in 2009 SAT competition - Gold for Industrial (SAT) category - Gold for Handmade (UNSAT) category - Gold for Random (SAT+UNSAT) category
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla 23
Feature extractor
Algorithm selector
New Results: Cost sensitive classification Goal: directly optimize the objective [Xu et al., 2011]
– Cost sensitive classification for every pair of candidates– Vote to decide the order of candidates
Improved performance over regression models, particularly with heterogeneous data sets
New SATzilla on SAT competition 2011
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla
Random Gap closed: 80%
55%33%
OracleHandmade Industrial
24
Recent work: Combining algorithm selection and algorithm configuration
Problem: what if you only have one strong solver but it is highly parameterized?
“Hydra”: combine algorithm selection and algorithm configuration [Xu et al., 2010; 2011]
– Automatically find strong, uncorrelated parameter settings– Construct portfolio from these
Results: clear improvement for both SAT and MIP– 2 × to 31 × faster for MIP– 1.2 × to 73 × faster for SAT
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla 25
Conclusion
• Portfolio-based algorithm selection can improve performance and robustness
• SATzilla-like portfolio approaches are the state of the art for many problems
• New directions since our JAIR paper:– better predictive models – automatic portfolio construction (“Hydra”)
Xu, Hutter, Hoos, and Leyton-Brown: SATzilla 26