plant a (decision) tree and save the world*!vivekaxl.github.io/assets/advancedanalytics.pdf ·...
TRANSCRIPT
Advanced Analytics:
Plant a (decision) TREE and save the world*!
Vivek NairNorth Carolina State University
* Configure software using less resources
Most Valuable Point
“Information is a source of learning. But unless it is organized, processed, and available to the right people in a format for decision making, it is a burden, not a benefit” -- Dr. William Pollard
Decision Trees - Use Cases
TAR(ZAN)2 [1]
2002
[1] Menzies, Tim, and Ying Hu. "Just enough learning (of association rules): the TAR2 “Treatment” learner." Artificial Intelligence Review 25.3 (2006): 211-229.
TAR(ZAN)2 Learns Small Theories
Problem: Find picture on a page from 11 features
TAR(ZAN)2 Learns Small Theories
Problem: Find picture on a page from 11 features
TAR(ZAN)2 Learns Small Theories
Find picture on a page from 11 features
TAR(ZAN)2 Learns Small Theories
Find picture on a page from 11 features
(34 ≤ height ≤ 86) ⋀ (3.9 ≤ mean_tr < 9.5)
TAR(ZAN)2 Learns Small Theories
Find picture on a page from 11 features
(34 ≤ height ≤ 86) ⋀ (3.9 ≤ mean_tr < 9.5)
Decision Trees - Use Cases
TAR(ZAN)2 [1]
2002
[1] Menzies, Tim, and Ying Hu. "Just enough learning (of association rules): the TAR2 “Treatment” learner." Artificial Intelligence Review 25.3 (2006): 211-229.
Decision Trees - Use Cases
TAR(ZAN)2 [1]
2002SWAY[2]
2016
XTREE[4]
2017Performance Optimization[3]
2017
[1] Menzies, Tim, and Ying Hu. "Just enough learning (of association rules): the TAR2 “Treatment” learner." Artificial Intelligence Review 25.3 (2006): 211-229.[2] Nair et al. "An (accidental) exploration of alternatives to evolutionary algorithms for sbse." SSBSE- 2016.[3] Guo et al. “Variability-aware performance prediction: A statistical learning approach." ASE-2013.[4] Krishna et al.. "Less is more: Minimizing code reorganization using XTREE." IST-2017
Decision Trees - Use Cases
Optimization
PlanningSoftware Variability
[1] Menzies, Tim, and Ying Hu. "Just enough learning (of association rules): the TAR2 “Treatment” learner." Artificial Intelligence Review 25.3 (2006): 211-229.[2] Nair et al. "An (accidental) exploration of alternatives to evolutionary algorithms for sbse." SSBSE- 2016.[3] Guo et al. “Variability-aware performance prediction: A statistical learning approach." ASE-2013.[4] Krishna et al.. "Less is more: Minimizing code reorganization using XTREE." IST-2017
TAR(ZAN)2 [1]
2002SWAY[2]
2016
XTREE[4]
2017Performance Optimization[3]
2017
Decision Trees - Use Cases
Optimization
PlanningSoftware Variability
TAR(ZAN)2 [1]
2002SWAY[2]
2016
XTREE[4]
2017Performance Optimization[3]
2017
[1] Menzies, Tim, and Ying Hu. "Just enough learning (of association rules): the TAR2 “Treatment” learner." Artificial Intelligence Review 25.3 (2006): 211-229.[2] Nair et al. "An (accidental) exploration of alternatives to evolutionary algorithms for sbse." SSBSE- 2016.[3] Guo et al. “Variability-aware performance prediction: A statistical learning approach." ASE-2013.[4] Krishna et al.. "Less is more: Minimizing code reorganization using XTREE." IST-2017
Configurable Systems and Variability
System
Configurable Systems and Variability
Systeminput.y4m
Configurable Systems and Variability
System output.x264input.y4m
Configurable Systems and Variability
System output.x264
c1 c2 c3 c4 cn. . .
Configuration options
input.y4m
Configurable Systems and Variability
System output.x264
Non-functional behavior: response time, throughput, etc.
c1 c2 c3 c4 cn. . .
Configuration options
input.y4m
Configurable Systems and Variability
System output.x264
c1 c2 c3 c4 cn. . .
Non-functional behavior: response time, throughput, etc.
Configuration options
input.y4m
Performance Optimization is Necessary!
Performance Optimization is getting more Complex!
[1] Xu et. al. 2015. Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software.FSE 2015[2] Van Aken, Dana, et al. "Automatic Database Management System Tuning Through Large-scale Machine Learning." International Conference on Management of Data. ACM, 2017.
● Necessary
Performance Optimization is required since Default Configuration is Bad!
[1] Van Aken, Dana, et al. "Automatic Database Management System Tuning Through Large-scale Machine Learning." International Conference on Management of Data. ACM, 2017.[2] Herodotou, Herodotos, et al. "Starfish: A Self-tuning System for Big Data Analytics." CIDR
● Necessary
● Complex
Performance Optimization can be Expensive!
• Evaluation of single instance of software/hardware co-design problem can take weeks[1]
• Rolling Sort use-case required 21 days, within a total experimental time of about 2.5 months[2]
• Test suite generation using Evolutionary Algorithm can take weeks[3]
• Image recognition workload and speech recognition workload, jobs ran for many hours or days[4]
[1] Zuluaga, Marcela, et al. "Active learning for multi-objective optimization." International Conference on Machine Learning. 2013.[2] Jamshidi, Pooyan, and Giuliano Casale. "An uncertainty-aware approach to optimal configuration of stream processing systems."MASCOTS-2016[3] Wang, Tiantian, et al. "Searching for better configurations: a rigorous approach to clone evaluation." FSE-2013[4] Venkataraman, Shivaram, et al. "Ernest: Efficient Performance Prediction for Large-Scale Advanced Analytics." NSDI. 2016.
● Necessary
● Complex
● Default is bad
Database- otter-tune- ituned
Is it pervasive?
Machine Learning- Hyperparameter Tuning- Random search - SMBO- Fabolas
Software Engineering- Tuning or Default Values?- Tuning for Software Analytics- Tuning for Defect Prediction- Topic Modelling
Cloud Computing- Ernest- Cherrypick- PARIS
● Necessary
● Complex
● Default is bad
Expensive
Performance Optimization!
● Necessary
● Complex
● Default is bad
Expensive
Pervasive
Performance Optimization!
● Necessary
● Complex
● Default is bad
Expensive
Pervasive
Road Map
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Road Map
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Residual-based Methods
Road Map
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Rank-based Method
Road Map
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Bayesian-based Method
Road Map
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Surrogate is a cheap(er) version of the actual system
APACHEhttp server
Request
Surrogate APACHE
http server
Response Time
Request
Response Time
Progressive Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Guo, Jianmei, et al. "Variability-aware performance prediction: A statistical learning approach." ASE-2013.
Residual-based MethodsProgressive Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
How to find the ‘best performing’ configuration for any given system?
Residual-based MethodsProgressive Sampling
Configuration Space
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Residual-based MethodsProgressive Sampling
Configuration Space
Training Pool Testing Pool Validation Pool
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
40%40% 20%
Residual-based MethodsProgressive Sampling
Configuration Space
Training Pool Testing Pool Validation Pool
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Residual-based MethodsProgressive Sampling
Configuration Space
Training Pool Testing Pool Validation Pool
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Residual-based MethodsProgressive Sampling
Configuration Space
Training Pool Testing Pool Validation Pool
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Train
Residual-based MethodsProgressive Sampling
Configuration Space
Training Pool Testing Pool Validation Pool
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
TrainIF MRE < τ: ExitELSE: continue
Check Performance
Residual-based MethodsProgressive Sampling
Configuration Space
Training Pool Testing Pool Validation Pool
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Train Check Performance
Mean Relative Error
MRE = |actual - predicted|actual
IF MRE < τ: ExitELSE: continue
Residual-based MethodsProgressive Sampling
Configuration Space
Training Pool Testing Pool Validation Pool
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
TrainIF MRE < τ: ExitELSE: continue
Check Performance
Residual-based MethodsProgressive Sampling
Configuration Space
Training Pool Testing Pool Validation Pool
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
TrainIF MRE < τ: ExitELSE: continue
Check Performance
Residual-based MethodsProgressive Sampling
Configuration Space
Training Pool Testing Pool Validation Pool
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
TrainIF MRE < τ: ExitELSE: continue
Check Performance
Residual-based MethodsProgressive Sampling
Configuration Space
Training Pool Testing Pool Validation Pool
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
TrainIF MRE < τ: ExitELSE: continue
Check Performance
Residual-based MethodsProgressive Sampling
Configuration Space
Training Pool Testing Pool Validation Pool
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Residual-based MethodsProgressive Sampling
Configuration Space
Training Pool Testing Pool Validation Pool
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Use the model to find the best configuration
Residual-based MethodsProgressive Sampling - Limitation
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
• The stopping condition is arbitrary
• Cannot estimate cost required to build a surrogate
Projective Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Sarkar, Atri, et al. "Cost-efficient sampling for performance prediction of configurable systems." ASE 2015.
Residual-based MethodsProjective Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Given an acceptable accuracy estimate (τ), how many samples is required to build a ‘quality’ surrogate?
Residual-based MethodsProjective Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Learning Curve
Residual-based MethodsProjective Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Learning Curve
Residual-based MethodsProjective Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Learning Curve
Residual-based MethodsProjective Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Learning Curve
Steep Incline
Residual-based MethodsProjective Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Steep Incline
Gradual Incline
Learning Curve
Residual-based MethodsProjective Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Steep Incline
Gradual Incline
Pleatue
Learning Curve
Residual-based MethodsProjective Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Steep Incline
Gradual Incline
Plateau
Optimal Sample SizeLearning Curve
Estimates the Learning Curve
Residual-based MethodsProjective Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Generate initial data points for the learning curve
Find best fit projective function
Calculate optimal training set
Build surrogate
Estimates the Learning Curve
Residual-based MethodsProjective Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Generate initial data points for the learning curve
Find best fit projective function
Calculate optimal training set
Build surrogate
Estimates the Learning Curve
Residual-based MethodsProjective Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Requirement: Initial samples should reflect relationship between all configuration options
Intuition: Performance depends if configuration option is selected or deselected
Heuristic: Feature Frequency - initial samples have each option selected or deselected, at least, δ times
Generate initial data points for the learning curve
Find best fit projective function
Calculate optimal training set
Build surrogate
Residual-based MethodsProjective Sampling
Configuration Space
Training Pool Testing Pool
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Residual-based MethodsProjective Sampling
Configuration Space
Training Pool Testing Pool
Selected 0 0 0 0
Deselected 0 0 0 0
c1 c2 c3 c4
Feature frequency table (δ=2)
#Samples Accuracy
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Residual-based MethodsProjective Sampling
Configuration Space
Training Pool Testing Pool
Selected 0 0 0 0
Deselected 0 0 0 0
c1 c2 c3 c4
Train
c1 c2 c3 c4
1 0 1 1
#Samples Accuracy
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Feature frequency table (δ=2)
Residual-based MethodsProjective Sampling
Configuration Space
Training Pool Testing Pool
Selected 1 0 1 1
Deselected 0 1 0 0
c1 c2 c3 c4
Train
c1 c2 c3 c4
1 0 1 1
1 5%
#Samples Accuracy
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Feature frequency table (δ=2)
Residual-based MethodsProjective Sampling
Configuration Space
Training Pool Testing Pool
Selected 1 0 1 1
Deselected 0 1 0 0
c1 c2 c3 c4
Train
c1 c2 c3 c4
1 0 1 1
0 1 0 0
1 5%
#Samples Accuracy
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Feature frequency table (δ=2)
Residual-based MethodsProjective Sampling
Configuration Space
Training Pool Testing Pool
Selected 1 1 2 1
Deselected 1 1 0 1
c1 c2 c3 c4
Train
c1 c2 c3 c4
1 0 1 1
0 1 1 0
1 5%
2 17%
#Samples Accuracy
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Feature frequency table (δ=2)
Residual-based MethodsProjective Sampling
Configuration Space
Training Pool Testing Pool
Selected 1 1 2 1
Deselected 1 1 0 1
c1 c2 c3 c4
Train
c1 c2 c3 c4
1 0 1 1
0 1 1 0
1 1 0 0
1 5%
2 17%
#Samples Accuracy
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Feature frequency table (δ=2)
Residual-based MethodsProjective Sampling
Configuration Space
Training Pool Testing Pool
Selected 2 2 2 1
Deselected 1 1 1 2
c1 c2 c3 c4
Train
c1 c2 c3 c4
1 0 1 1
0 1 1 0
1 1 0 0
1 5%
2 17%
3 29%
#Samples Accuracy
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Feature frequency table (δ=2)
Residual-based MethodsProjective Sampling
Configuration Space
Training Pool Testing Pool
Selected 2 2 2 1
Deselected 1 1 1 2
c1 c2 c3 c4
Train
c1 c2 c3 c4
1 0 1 1
0 1 1 0
1 1 0 0
0 0 0 1
1 5%
2 17%
3 29%
#Samples Accuracy
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Feature frequency table (δ=2)
Residual-based MethodsProjective Sampling
Configuration Space
Training Pool Testing Pool
Selected 2 2 2 2
Deselected 2 2 2 2
c1 c2 c3 c4
Train
c1 c2 c3 c4
1 0 1 1
0 1 1 0
1 1 0 0
0 0 0 1
1 5%
2 17%
3 29%
4 35%
#Samples Accuracy
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Feature frequency table (δ=2)
Estimates the Learning Curve
Residual-based MethodsProjective Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Generate initial data points for the learning curve
Find best fit projective function
Calculate optimal training set
Build surrogate
Residual-based MethodsProjective Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
1 5%
2 17%
3 29%
4 35%
#Samples Accuracy
Residual-based MethodsProjective Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Exponential
Power
Log
Estimates the Learning Curve
Residual-based MethodsProjective Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Generate initial data points for the learning curve
Find best fit projective function
Calculate optimal training set
Build surrogate
Residual-based MethodsProjective Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Coefficients of projective functionNumber of configurations whose performance value will be predicted
Penalty factor
Taken from Sarkar et al.
Estimates the Learning Curve
Residual-based MethodsProjective Sampling
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Generate initial data points for the learning curve
Find best fit projective function
Calculate optimal training set
Build surrogate
Residual-based MethodsProjective Sampling
Configuration Space
Training Pool Testing Pool Validation Pool
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Use the model to find the best configuration
Optimal Sample Size
Assumes an accurate model can be built
Residual-based MethodsProjective Sampling - Limitation
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Rank-based Method
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Nair, Vivek, et al. "Using Bad Learners to find Good Configurations." FSE 2017
Rank-based Method
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Rank-based Method
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Rank-based Method
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Rank-based Method
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Rank-based Method
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
What happens when an accurate model cannot be built?
Rank-based MethodCore Insights
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Rank Preserving Model
Rank Preserving Model
Rank Preserving Model
Rank Preserving Model
Rank Preserving Model
Rank Preserving Model
Rank Preserving Model
Rank Preserving Model
Configuration Space
Training Pool Testing Pool Validation Pool
TrainIF accuracy < τ: ExitELSE: continue
Check Performance
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Rank Preserving Model
Configuration Space
Training Pool Testing Pool Validation Pool
TrainIF accuracy < τ: ExitELSE: continue
Check Performance
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Rank Preserving Model - Limitation
Requires Testing Pool - 20% of configuration space
Configuration Space
Training Pool Testing Pool Validation Pool
40%40% 20%
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Future - Bayesian based Method
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Nair, Vivek et al. "FLASH: A Faster Optimizer for SBSE Tasks." preprint
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Bayesian-based Method
Can we avoid measurement of configurations in the testing pool?
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Bayesian-based MethodBayesian Optimization
Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Bayesian-based MethodBayesian Optimization
True Performance Distribution
Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Bayesian-based MethodBayesian Optimization
True Performance Distribution
Predicted Performance Distribution
Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Bayesian-based MethodBayesian Optimization
True Performance Distribution
Predicted Performance Distribution
Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
True Performance Distribution
Bayesian-based MethodBayesian Optimization
Predicted Performance Distribution
Acquisition Function
Which configuration should I evaluate next?
Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Bayesian-based MethodBayesian Optimization
True Performance Distribution
Predicted Performance Distribution
Acquisition Function
Which configuration should I evaluate next?
Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Bayesian-based MethodBayesian Optimization
Tradeoff between Exploration vs Exploitation
True Performance Distribution
Predicted Performance Distribution
Acquisition Function
Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Bayesian-based MethodBayesian Optimization
Surrogate of choice: Gaussian Processes (GP)
True Performance Distribution
Predicted Performance Distribution
Acquisition Function
Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Bayesian-based MethodBayesian Optimization
Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Bayesian-based MethodBayesian Optimization
Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)
GP lose efficiency in high dimensional spaces i.e. number of features exceeds a dozen
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Bayesian-based MethodBayesian Optimization
Bayesian-based Method - FLASH
GP
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Bayesian-based Method - FLASH
GP
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Bayesian-based Method - FLASH
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Bayesian-based Method - FLASH
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Bayesian-based Method - FLASH
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Bayesian-based Method - FLASH
2013 2015
[Sarkar’15][Guo’13]
2017
[Nair’17]
Future
Bayesian-based Method - FLASH
Bayesian-based Method - FLASH
● Fast
● Effective
● Comprehensible
Bayesian-based Method - FLASH
Conclusion• ML Algorithms are not a black box
– How to use Decision Tree in Planning?– Can I explain the results to the Decision
Makers?
• Lazy is good– Only do what is required– Optimization does not require an accurate
model
• Easy over Hard – Try simplest first– Tuning SVM outperforms DL
Rank-based Method: http://tiny.cc/wnhenyFlash: http://tiny.cc/hohenyePAL: http://www.spiral.net/software/pal.htmlBayesian Optimization: https://youtu.be/vz3D36VXefI
"Look for me...beneath the tree...North!"
―Three-eyed raven