a hands-on introduction to automatic machine learninglarsko/slides/automl18.pdf · automatic...
TRANSCRIPT
![Page 1: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/1.jpg)
A Hands-On Introduction toAutomatic Machine Learning
Lars KotthoffUniversity of Wyoming
AutoML Workshop, 28 August 2018, Nanjing
1
![Page 2: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/2.jpg)
Machine Learning
Data Machine Learning Predictions
2
![Page 3: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/3.jpg)
Automatic Machine Learning
HyperparameterTuning
Data Machine Learning Predictions
3
![Page 4: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/4.jpg)
Grid and Random Search▷ evaluate certain points in parameter space
Bergstra, James, and Yoshua Bengio. “Random Search forHyper-Parameter Optimization.” J. Mach. Learn. Res. 13, no. 1 (February2012): 281–305.
4
![Page 5: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/5.jpg)
Local Search
▷ start with random configuration▷ change a single parameter (local search step)▷ if better, keep the change, else revert▷ repeat, stop when resources exhausted or desired solution
quality achieved▷ restart occasionally with new random configurations
5
![Page 6: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/6.jpg)
Local Search ExampleGoing Beyond Local Optima: Iterated Local Search
(Initialisation)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
graphics by Holger Hoos
6
![Page 7: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/7.jpg)
Local Search ExampleGoing Beyond Local Optima: Iterated Local Search
(Initialisation)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
graphics by Holger Hoos
7
![Page 8: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/8.jpg)
Local Search ExampleGoing Beyond Local Optima: Iterated Local Search
(Local Search)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
graphics by Holger Hoos
8
![Page 9: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/9.jpg)
Local Search ExampleGoing Beyond Local Optima: Iterated Local Search
(Local Search)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
graphics by Holger Hoos
9
![Page 10: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/10.jpg)
Local Search ExampleGoing Beyond Local Optima: Iterated Local Search
(Perturbation)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
graphics by Holger Hoos
10
![Page 11: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/11.jpg)
Local Search ExampleGoing Beyond Local Optima: Iterated Local Search
(Local Search)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
graphics by Holger Hoos
11
![Page 12: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/12.jpg)
Local Search ExampleGoing Beyond Local Optima: Iterated Local Search
(Local Search)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
graphics by Holger Hoos
12
![Page 13: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/13.jpg)
Local Search ExampleGoing Beyond Local Optima: Iterated Local Search
(Local Search)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
graphics by Holger Hoos
13
![Page 14: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/14.jpg)
Local Search ExampleGoing Beyond Local Optima: Iterated Local Search
?
Selection (using Acceptance Criterion)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
graphics by Holger Hoos
14
![Page 15: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/15.jpg)
Model-Based Search
▷ evaluate small number of configurations▷ build model of parameter-performance surface based on the
results▷ use model to predict where to evaluate next▷ repeat, stop when resources exhausted or desired solution
quality achieved▷ allows targeted exploration of promising configurations
15
![Page 16: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/16.jpg)
Model-Based Search Example
●
●
●
●
●
yei
−1.0 −0.5 0.0 0.5 1.0
0.0
0.4
0.8
0.000
0.005
0.010
0.015
0.020
0.025
x
type
● init
prop
type
y
yhat
ei
Iter = 1, Gap = 1.9909e−01
Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas,and Michel Lang. “MlrMBO: A Modular Framework for Model-BasedOptimization of Expensive Black-Box Functions,” March 9, 2017.http://arxiv.org/abs/1703.03373. 16
![Page 17: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/17.jpg)
Model-Based Search Example
●
●
●
●
●
yei
−1.0 −0.5 0.0 0.5 1.0
0.0
0.4
0.8
0.00
0.01
0.02
0.03
x
type
● init
prop
seq
type
y
yhat
ei
Iter = 2, Gap = 1.9909e−01
Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas,and Michel Lang. “MlrMBO: A Modular Framework for Model-BasedOptimization of Expensive Black-Box Functions,” March 9, 2017.http://arxiv.org/abs/1703.03373. 17
![Page 18: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/18.jpg)
Model-Based Search Example
●
●
●
●
●
yei
−1.0 −0.5 0.0 0.5 1.0
0.0
0.4
0.8
0.000
0.002
0.004
0.006
x
type
● init
prop
seq
type
y
yhat
ei
Iter = 3, Gap = 1.9909e−01
Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas,and Michel Lang. “MlrMBO: A Modular Framework for Model-BasedOptimization of Expensive Black-Box Functions,” March 9, 2017.http://arxiv.org/abs/1703.03373. 18
![Page 19: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/19.jpg)
Model-Based Search Example
●
●
●
●
●
yei
−1.0 −0.5 0.0 0.5 1.0
0.0
0.4
0.8
0e+00
2e−04
4e−04
6e−04
8e−04
x
type
● init
prop
seq
type
y
yhat
ei
Iter = 4, Gap = 1.9992e−01
Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas,and Michel Lang. “MlrMBO: A Modular Framework for Model-BasedOptimization of Expensive Black-Box Functions,” March 9, 2017.http://arxiv.org/abs/1703.03373. 19
![Page 20: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/20.jpg)
Model-Based Search Example
●
●
●
●
●
yei
−1.0 −0.5 0.0 0.5 1.0
0.0
0.4
0.8
0e+00
1e−04
2e−04
x
type
● init
prop
seq
type
y
yhat
ei
Iter = 5, Gap = 1.9992e−01
Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas,and Michel Lang. “MlrMBO: A Modular Framework for Model-BasedOptimization of Expensive Black-Box Functions,” March 9, 2017.http://arxiv.org/abs/1703.03373. 20
![Page 21: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/21.jpg)
Model-Based Search Example
●
●
●
●
●
yei
−1.0 −0.5 0.0 0.5 1.0
0.0
0.4
0.8
0.00000
0.00003
0.00006
0.00009
0.00012
x
type
● init
prop
seq
type
y
yhat
ei
Iter = 6, Gap = 1.9996e−01
Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas,and Michel Lang. “MlrMBO: A Modular Framework for Model-BasedOptimization of Expensive Black-Box Functions,” March 9, 2017.http://arxiv.org/abs/1703.03373. 21
![Page 22: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/22.jpg)
Model-Based Search Example
●
●
●
●
●
yei
−1.0 −0.5 0.0 0.5 1.0
0.0
0.4
0.8
0e+00
1e−05
2e−05
3e−05
4e−05
5e−05
x
type
● init
prop
seq
type
y
yhat
ei
Iter = 7, Gap = 2.0000e−01
Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas,and Michel Lang. “MlrMBO: A Modular Framework for Model-BasedOptimization of Expensive Black-Box Functions,” March 9, 2017.http://arxiv.org/abs/1703.03373. 22
![Page 23: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/23.jpg)
Model-Based Search Example
●
●
●
●
●
yei
−1.0 −0.5 0.0 0.5 1.0
0.0
0.4
0.8
0.0e+00
5.0e−06
1.0e−05
1.5e−05
2.0e−05
x
type
● init
prop
seq
type
y
yhat
ei
Iter = 8, Gap = 2.0000e−01
Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas,and Michel Lang. “MlrMBO: A Modular Framework for Model-BasedOptimization of Expensive Black-Box Functions,” March 9, 2017.http://arxiv.org/abs/1703.03373. 23
![Page 24: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/24.jpg)
Model-Based Search Example
●
●
●
●
●
yei
−1.0 −0.5 0.0 0.5 1.0
0.0
0.4
0.8
0.0e+00
2.5e−06
5.0e−06
7.5e−06
1.0e−05
x
type
● init
prop
seq
type
y
yhat
ei
Iter = 9, Gap = 2.0000e−01
Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas,and Michel Lang. “MlrMBO: A Modular Framework for Model-BasedOptimization of Expensive Black-Box Functions,” March 9, 2017.http://arxiv.org/abs/1703.03373. 24
![Page 25: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/25.jpg)
Model-Based Search Example
●
●
●
●
●
yei
−1.0 −0.5 0.0 0.5 1.0
0.0
0.4
0.8
0e+00
1e−07
2e−07
3e−07
4e−07
x
type
● init
prop
seq
type
y
yhat
ei
Iter = 10, Gap = 2.0000e−01
Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas,and Michel Lang. “MlrMBO: A Modular Framework for Model-BasedOptimization of Expensive Black-Box Functions,” March 9, 2017.http://arxiv.org/abs/1703.03373. 25
![Page 26: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/26.jpg)
Problems
▷ How good are we really?▷ How much of it is just random chance?▷ Can we do better?
26
![Page 27: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/27.jpg)
Underlying Issues
▷ true performance landscape unknown▷ resources allow to explore only tiny part of hyperparameter
space▷ results inherently stochastic
27
![Page 28: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/28.jpg)
Potential Solutions
▷ better-understood benchmarks▷ more comparisons▷ more runs with different random seed
28
![Page 29: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/29.jpg)
Two-Slide MBO ML# http://www.cs.uwyo.edu/~larsko/mbo.pyparams = { 'C': np.logspace(-2, 10, 13),
'gamma': np.logspace(-9, 3, 13) }param_grid = [ { 'C': x, 'gamma': y } for x in params['C']
for y in params['gamma'] ]# [{'C': 0.01, 'gamma': 1e-09}, {'C': 0.01, 'gamma': 1e-08}...]
initial_samples = 3evals = 10random.seed(1)
def est_acc(pars):clf = svm.SVC(**pars)return np.median(cross_val_score(clf, iris.data, iris.target, cv = 10))
data = []for pars in random.sample(param_grid, initial_samples):
acc = est_acc(pars)data += [ list(pars.values()) + [ acc ] ]
# [[1.0, 0.1, 1.0],# [1000000000.0, 1e-07, 1.0],# [0. 1, 1e-06,0.9333333333333333]]
29
![Page 30: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/30.jpg)
Two-Slide MBO ML
regr = RandomForestRegressor(random_state = 0)for evals in range(0, evals):
df = np.array(data)regr.fit(df[:,0:2], df[:,2])
preds = regr.predict([ list(pars.values()) for pars in param_grid ])i = preds.argmax()
acc = est_acc(param_grid[i])data += [ list(param_grid[i].values()) + [ acc ] ]print("{}: best predicted {} for {}, actual {}"
.format(evals, round(preds[i], 2), param_grid[i], round(acc, 2)))
i = np.array(data)[:,2].argmax()print("Best accuracy ({}) for parameters {}".format(data[i][2], data[i][0:2]))
30
![Page 31: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/31.jpg)
Two-Slide MBO ML
0: best predicted 0.99 for {'C': 1.0, 'gamma': 1e-09}, actual 0.931: best predicted 0.99 for {'C': 1000000000.0, 'gamma': 1e-09}, actual 0.932: best predicted 0.99 for {'C': 1000000000.0, 'gamma': 0.1}, actual 0.933: best predicted 0.97 for {'C': 1.0, 'gamma': 0.1}, actual 1.04: best predicted 0.99 for {'C': 1.0, 'gamma': 0.1}, actual 1.05: best predicted 1.0 for {'C': 1.0, 'gamma': 0.1}, actual 1.06: best predicted 1.0 for {'C': 1.0, 'gamma': 0.1}, actual 1.07: best predicted 1.0 for {'C': 1.0, 'gamma': 0.1}, actual 1.08: best predicted 1.0 for {'C': 0.01, 'gamma': 0.1}, actual 0.939: best predicted 1.0 for {'C': 1.0, 'gamma': 0.1}, actual 1.0Best accuracy (1.0) for parameters [1.0, 0.1]
31
![Page 32: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/32.jpg)
Tools and Resources
iRace http://iridia.ulb.ac.be/irace/TPOT https://github.com/EpistasisLab/tpot
mlrMBO https://github.com/mlr-org/mlrMBOSMAC http://www.cs.ubc.ca/labs/beta/Projects/SMAC/
Spearmint https://github.com/HIPS/SpearmintTPE https://jaberg.github.io/hyperopt/
Auto-WEKA http://www.cs.ubc.ca/labs/beta/Projects/autoweka/
Auto-sklearn https://github.com/automl/auto-sklearn
Available soon: edited book on automatic machine learninghttps://www.automl.org/book/ (Frank Hutter, Lars Kotthoff,Joaquin Vanschoren)
32
![Page 33: A Hands-On Introduction to Automatic Machine Learninglarsko/slides/automl18.pdf · Automatic Machine Learning Lars Kotthoff University of Wyoming larsko@uwyo.edu AutoML Workshop,](https://reader030.vdocument.in/reader030/viewer/2022041015/5ec60880700f064af2337fd1/html5/thumbnails/33.jpg)
I’m hiring!
Several funded graduate/postdoc positions available.
33