![Page 1: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/1.jpg)
Bayesian Optimization for Experiment Design
Barnabas Poczos
Carnegie Mellon University
Machine Learning Department
![Page 2: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/2.jpg)
www.autonlab.org
Computer vision,
Robotics
Cosmology
2
AI in games
High Energy
Physics
Art
Barnabas Poczos
Quantum
ChemistryML in Agriculture
Material Science
EEG, fMRI, MEG, …
![Page 3: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/3.jpg)
Contents
ML on Sets / Point Clouds / Distributions
Adversarial Neural Networks
Bayesian Optimization
3
![Page 4: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/4.jpg)
ML on Point Cloud Data
4
![Page 5: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/5.jpg)
www.juhokim.com/projects.phpCristiano Ronaldo
Rio FerdinandOwen Hargreaves
Manchester United 07/08
Distributional Data
5
![Page 6: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/6.jpg)
Distribution Regression / Classification
Y1=1
P1
Y2=0
P2
?
Pm+1
Y3=1
P3
Ym=0
Pm… The inputs are distributions, density functions (not vectors) We don’t know these distributions, only sample sets are available The sizes of the sets can be arbitrary and all different
Differences compared to standard methods on vectors
6
![Page 7: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/7.jpg)
Using
Estimate divergence
RÉNYI DIVERGENCE ESTIMATION
without density estimation
Distances / Divergences between Distributions
7
![Page 8: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/8.jpg)
The Estimator
8
![Page 9: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/9.jpg)
Applications
![Page 10: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/10.jpg)
High Energy Physics
End-to-End Event Classification
10
![Page 11: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/11.jpg)
Find new scientific “laws” in physics
Goal: Estimate dynamical mass of galaxy clusters.
Importance: Galaxy clusters are being the largest gravitationally bound systems in the Universe. Dynamical mass measurements are important to understand the behavior of dark matter and normal matter.
Difficulty: We can only measure the velocity of galaxies not the mass of their cluster. Physicists estimate dynamical cluster mass from single velocity dispersion.
Our method: Estimate the cluster mass from the whole distribution of velocities rather than just a simple velocity distribution. 11
![Page 12: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/12.jpg)
Find new scientific laws in physics
Michelle Ntampaka et al, A Machine Learning Approach for Dynamical Mass Measurements of Galaxy Clusters, APJ 2015
12
![Page 13: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/13.jpg)
Find the parameters of UniverseGiven a distribution of particles, our goal is to predict the parameters of the simulated universe
13
![Page 14: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/14.jpg)
B. Póczos, L. Xiong & J. Schneider, UAI, 2011.
What are the most anomalous galaxy clusters?
The most anomalous galaxy cluster contains mostly star forming blue galaxies irregular galaxies
Sloan Digital Sky Survey (SDSS) continuum spectrum 505 galaxy clusters
(10-50 galaxies in each) 7530 galaxies
Find interesting Galaxy Clusters
Blue galaxy Red galaxy
Credits: ESA, NASA 14
![Page 15: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/15.jpg)
Finding Vortices in Simulated Fluid Flow
Classification probabilities 15
![Page 16: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/16.jpg)
Find Interesting Phenomena in Turbulence Data
Anomaly scores
Anomaly detection
16
![Page 17: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/17.jpg)
Lidar Point Cloud
![Page 18: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/18.jpg)
Agriculture Recommend experiments (which plants to cross) to sorghum breeders.
18
![Page 19: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/19.jpg)
Surrogate robotic system in the field
19
![Page 20: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/20.jpg)
Surrogate robotic system in the field
The surrogate system collecting data at the TAMU field site. The carriage supports two boom assemblies each
one of which carries a sensor pod. The carriage slides up and down on the column allowing full scanning of a
plant.
20
![Page 21: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/21.jpg)
Name Range RMSE error
Leaf angle* 75.94 3.30 (4.35%)
Leaf radiation angle* 120.66 4.34 (3.60%)
Leaf length* 35.00 0.87 (2.49%)
Leaf width [max] 3.61 0.27 (7.48%)
Leaf width [average] 2.99 0.21 (7.o2%)
Leaf area* 133.45 8.11 (6.08%)21
![Page 22: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/22.jpg)
Emerging images
Niloy J. Mitra, 2009 22
![Page 23: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/23.jpg)
Emerging images
Niloy J. Mitra, 200923
![Page 24: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/24.jpg)
Generative Adversarial Networks
![Page 25: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/25.jpg)
Generating Adversarial Networks
25
Generated fake celebrity images
Tero Karras, Timo Aila, Samuli Laine, ICLR 2018
Generated fake living room images
![Page 26: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/26.jpg)
![Page 27: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/27.jpg)
Machine Learning and Art
![Page 28: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/28.jpg)
Ravanbakhsh, M., Lanusse, F., Mandelbaum, R., Schneider, J., and Poczos, B., AAAI 2017
28
![Page 29: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/29.jpg)
Point Cloud GAN
For each object, from left to right is training data, Auto Encoder competitor, and PC-GAN. PC-GAN is better in capturing fine details like wheels of the airplanes or proper chair legs.
29
![Page 30: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/30.jpg)
Theory of Generative Adversarial Networks
with the goal of solving:
30
![Page 31: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/31.jpg)
Statistical Results
Then the GAN is consistent and minimax optimal
Under some conditions
31
![Page 32: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/32.jpg)
Open Questions
o Statistical properties under less restrictive conditions?o Results for convolutional neural nets?o Best way for training GANs (i.e best way for solving the minmax optimization)?o GANs on manifolds?o Rare event generation?o Generate uniform distribution on the support of the data?
32
![Page 33: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/33.jpg)
Experiment Design
![Page 34: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/34.jpg)
Experiment Design
Given a collection of results from past experiments, recommend new experiments to run.
some experiments are expensive (time, money) and high-quality, some are
cheap and low-quality.
consider costs and expected information gain
Code available: https://github.com/dragonfly/dragonfly
34
![Page 35: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/35.jpg)
Gaussian Processes
35
![Page 36: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/36.jpg)
Why GPs for Regression?
Motivation:
All the above regression methods give point estimates only. We would like a method that could
also provide “confidence” during the estimation.
Regression methods:
Linear regression, support vector regression, kNN regression, deep neural networks, etc…
36
Sampling from GP: We can also use GP the generate samples from the posterior (red, blue, green)
![Page 37: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/37.jpg)
37
Results: Sin function
![Page 38: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/38.jpg)
38
Properties of Multivariate Gaussian Distributions
![Page 39: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/39.jpg)
39
Marginal and Conditional distributions of Gaussians are Gaussian
![Page 40: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/40.jpg)
40
Gaussian ProcessesDefinition: (Gaussian Processes)
GP is an (uncountably infinite) collection of random variables, s.t. any finite number of them have a joint Gaussian distribution
Notations:
![Page 41: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/41.jpg)
41
Gaussian Process Pseudo CodeInputs:
![Page 42: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/42.jpg)
42
Outputs:
Gaussian Process Pseudo Code (Continued)
![Page 43: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/43.jpg)
43
Results: Sinc function
![Page 44: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/44.jpg)
44
Take me home!
GPs can be used for nonlinear regression can provide “confidence” of the estimate and can sample from the posterior distribution of regression
functions
![Page 45: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/45.jpg)
45
GP Application: Bayesian Optimization
![Page 46: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/46.jpg)
46
Bayesian optimization (BO) is a popular recipe for optimizing expensive black-box functions where the goal is to find a global maximizer of the black-box objective function We don’t even need to know the gradient of the objective function
Bayesian optimization has been used for a variety of practical optimization tasks such as hyperparameter tuning for machine learning algorithms, experiment design, online advertising, …
Often heuristics only Big gaps in our theoretical understanding …
Convergence rate? Dependence on dimension?
Bayesian Optimization
![Page 47: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/47.jpg)
Bayesian Optimization with Thompson SamplingGoal: maximize function f using the results from previous experiments.
47
![Page 48: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/48.jpg)
Bayesian Optimization with Thompson SamplingGoal: maximize function f using the results from previous experiments.
48
![Page 49: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/49.jpg)
Bayesian Optimization with Thompson SamplingGoal: maximize function f using the results from previous experiments.
49
![Page 50: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/50.jpg)
Bayesian Optimization with Thompson SamplingGoal: maximize function f using the results from previous experiments.
50
![Page 51: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/51.jpg)
Bayesian Optimization with Thompson SamplingGoal: maximize function f using the results from previous experiments.
51
![Page 52: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/52.jpg)
Bayesian Optimization with Thompson SamplingGoal: maximize function f using the results from previous experiments.
52
![Page 53: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/53.jpg)
Bayesian Optimization with Thompson SamplingGoal: maximize function f using the results from previous experiments.
53
![Page 54: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/54.jpg)
Bayesian Optimization with Thompson SamplingGoal: maximize function f using the results from previous experiments.
54
![Page 55: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/55.jpg)
Bayesian Optimization with Thompson SamplingGoal: maximize function f using the results from previous experiments.
55
![Page 56: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/56.jpg)
Bayesian Optimization with Thompson SamplingGoal: maximize function f using the results from previous experiments.
56
![Page 57: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/57.jpg)
Bayesian Optimization
Other criteria for selecting xt :Upper Confidence Bounds (Srinivas et al. 2010)Expected improvement (Jones et al. 1998)Probability of improvement (Kushner et al. 1964)Entropy search (Hernandez-Lobato et al. 2014, Wang et al. 2017). . . and a few more.
Bayesian models for f :Gaussian Processes (most popular)Neural networks (Snoek et al. 2015)Random forests (Hutter 2009)
57
![Page 58: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/58.jpg)
Bayesian Optimization with Upper Confidence Bounds
58
Goal: maximize function f using the results from previous experiments.
![Page 59: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/59.jpg)
Gaussian Processes with Upper Confidence Bounds
59
![Page 60: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/60.jpg)
Gaussian Processes with Upper Confidence Bounds
60
![Page 61: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/61.jpg)
Gaussian Processes with Upper Confidence Bounds
61
![Page 62: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/62.jpg)
Gaussian Processes with Upper Confidence Bounds
62
![Page 63: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/63.jpg)
Gaussian Processes with Upper Confidence Bounds
63
![Page 64: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/64.jpg)
Gaussian Processes with Upper Confidence Bounds
64
![Page 65: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/65.jpg)
Gaussian Processes with Upper Confidence Bounds
65
![Page 66: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/66.jpg)
Gaussian Processes with Upper Confidence Bounds
66
![Page 67: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/67.jpg)
Take me home!
Bayesian Optimization can be used for function optimization When function evaluation is expensive Gradient is not available
67
![Page 68: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/68.jpg)
Recommend experiments for creating better batteries
Goal: perform autonomous experimentation on electrolyte/electrochemically functional material
combinations based on feedback from previous experiments.
Machine Learning Unified Synchronous Experimentation (MUSE): Rapid Autonomous Discovery/Optimization
of Electrode and Electrolyte Materials (joint project with Jay Whitacre & Venkat Viswanathan)
68
![Page 69: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/69.jpg)
![Page 70: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/70.jpg)
Myopic Bayesian Design of Experiments via Posterior Sampling (MPS)
70
Solvation energy, viscosity, max conductivity
Setting
Example:
![Page 71: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/71.jpg)
Myopic Bayesian Design of Experiments via Posterior Sampling (MPS)
71
(Experiment parameters, results of experiments)
(predicted error after trying the experiment with parameters x)
![Page 72: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/72.jpg)
Myopic Bayesian Design of Experiments via Posterior Sampling (MPS)
Action space X: proportion of two solvents EC and EMC, molarity of the salt LiPF6, and T temperature
72
![Page 73: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/73.jpg)
Goal: Recommend settings for wire cutting machines
Manufacturing: Recommend parameter settings for machines
73
![Page 74: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/74.jpg)
parameter space
Find the True Parameters of the Universe
hypothetical
parameters,
simulated
observations
Hypothesis tests, MSE
likelihood:
true
parameters
real
universe
noisy
observations
NASA
NA
SA
/ E
SA
mathematical
model
Question: How to search the parameter space?
g()
surrogate function
Solution: Learn a surrogate function and make
experiment decisions using it 74
![Page 75: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/75.jpg)
Multi-fidelity optimization
75
![Page 76: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/76.jpg)
Multi-fidelity optimization
Example:f: expensive target functionf1:cheap computer simulation f2:cheap lab experimentf3:expensive lab experiment
76
![Page 77: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/77.jpg)
Likelihood function: Robertson-Walker metric (Requires numerical integration).
Multi-fidelity:
We construct a p = 2 dimensional multi-fidelity problem where we can choose between data set size N ∈ [50, 192] and perform the integration on grids of size G ∈ [10^2, 10^6] via the trapezoidal rule.
As the cost function for fidelity selection, we used λ(N, G) = NG as the computation time is linear in both parameters
Multi-fidelity application:Astrophysical Maximum Likelihood Inference
Data: We use Type Ia supernova data for maximum likelihood inference on 3 cosmological parameters: the Hubble constant H0 ∈ (60, 80), the dark matter fraction ΩM ∈ (0, 1) and dark energy fraction ΩΛ ∈ (0, 1), hence d = 3.
77
![Page 78: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/78.jpg)
We plot the maximum average log likelihood against wall clock time as that is the cost in this experiment.
The plot includes the time taken by each method to tune the GPs and determine the next points/fidelities for evaluation.
Bayesian Optimization with Continuous Approximations (BOCA)
We compare BOCA to the following four baselines: I. GP-UCB, II. the GP-EI criterion in BO (Jones et al., 1998), III. MF-GP-UCB (Kandasamy et al., 2016a) IV. and MF-SKO, the multi-fidelity sequential kriging
optimisation method from Huang et al. (2006).
78
Multi-fidelity application:Astrophysical Maximum Likelihood Inference
![Page 79: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/79.jpg)
Neural Architecture Search with Bayesian Optimization and Optimal Transport
Multi-fidelity learning: Training a neural network on large dataset is very expensive, but sometimes cheap approximations are available.
Optimized neural network architecture on CIFAR 10 dataset79
![Page 80: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/80.jpg)
Neural Architecture Search
The first row gives the number of samples N and the dimensionality D of each dataset in the form (N, D).
The subsequent rows show the regression MSE or classification error (lower is better) on the test set for each method.
The last column is for Cifar10 where we took the best models found by each method in 24K iterations and trained it for 120K iterations. When we trained the VGG-19 architecture using our training procedure, we got test errors 0.1718 (60K iterations) and 0.1018 (150K iterations)
RAND: random search; EA (Evolutionary algorithm): TreeBO: a BO method which only searches over feed forward structures.
80
![Page 81: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/81.jpg)
Take me home!
Methods for Multi-fidelity optimization Applications in Cosmology, Neural Network Search Code available
81
![Page 82: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/82.jpg)
Many real world applications can be framed as multi-objective optimization problems, where we wish to simultaneously optimize for multiple criteria (multi-objective optimization).
Classification: max True positive rate, max True negative rate, min False positive rate, min False negative rate, min computation cost, fast decision, min energy, min memory usage, min hard drive storage…
Drug discovery: each evaluation of the functions is an in-vitro experiment measure the solubility, toxicity and potency of a candidate example. Goal: max solubility, max potency, min toxicity, min experiment cost.
Multi-objective optimization
82
![Page 83: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/83.jpg)
Multi-objective optimization
Goal: recover (some part of the) Pareto front of these objectives with minimum number of experiments.
Goal: Maximize Objective 1 and Objective 2
83
![Page 84: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/84.jpg)
Brighter colors were sampled in the later iterations
Multi-objective optimization. Results:
84
![Page 85: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/85.jpg)
Take me home!
Bayesian Optimization based algorithm for multi-objective optimization Code is available!
85
![Page 86: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/86.jpg)
Experiment Recommendationfor Efficient Posterior Estimation
86
Query Efficient Posterior Estimation in Scientific Experiments via Bayesian Active LearningKirthevasan Kandasamy, Jeff Schneider, and Barnabas PoczosArtificial Intelligence Journal, 2017
![Page 87: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/87.jpg)
Our goal is an efficient way to estimate posterior densities as functions when calls to this black box / simulators are expensive.
Several cosmological phenomena are characterized by the cosmological parameters (e.g. Hubble constant, dark energy fraction).
Given observations, we wish to make inferences about the parameters.
Physicists have developed simulation-based probability models (= black box) of the Universe which can be used to compute the likelihood function of cosmological parameters for a given observation.
Posterior Estimation
87
![Page 88: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/88.jpg)
Likelihood of the Parameters of the Universe
parameter space
hypothetical
parameters, simulated
observations
Hypothesis tests, MSE
likelihood:
true
parameters
real
universe
noisy
observations
NASA
NA
SA
/ E
SA
mathematical
model
88
![Page 89: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/89.jpg)
Goal:
Active Posterior Estimation
Posterior distribution:
89
![Page 90: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/90.jpg)
Goal:
Ideal Greedy Parameter Selection:
Training data:
Unknown!
Active Posterior Estimation
90
![Page 91: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/91.jpg)
91
Utility Function Based Active Posterior Estimation
![Page 92: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/92.jpg)
Training data:
Negative Expected Divergence (NED)
92
Ideally we should choose the point that results in the highest reduction in divergence if we knew the likelihood and the true posterior at that point.
In NED, we choose the point with the highest expected reduction in the divergence.
Ideal greedy experiment recommendation:
Negative Expected Divergence
![Page 93: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/93.jpg)
Training data:
Negative Expected Divergence
Negative Expected Divergence
93
![Page 94: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/94.jpg)
Negative Expected Divergence
94
Negative Expected Divergence
![Page 95: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/95.jpg)
95
Experiments on Synthetic Data
100 points chosen in order by NED for a synthetic 2D experiment. The green contours are the true posterior. Initially the algorithm explores the space before focusing on high probability regions.
![Page 96: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/96.jpg)
96
Experiments on Synthetic Data
Comparison of NED/EV against MCMC-DE, ABC, MCMC-R and RAND for a 1D synthetic example
KL divergence between the estimated posterior and the true posterior
![Page 97: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/97.jpg)
We use supernovae data for inference on 3 cosmological parameters:
• Hubble Constant (H0 ∈ (60, 80), • Dark Matter Fraction ΩM ∈ (0, 1)• Dark Energy Fraction ΩΛ ∈ (0, 1).
97
Estimating Cosmological Parameters using Type-1 Supernovae data
The likelihood for the experiment is given by the Robertson– Walker metric which models the distance to a supernova given the parameters and the observed red-shift.
KL was approximated via numeric integration.
Dataset is taken from: T. M. Davis et al. Scrutinizing Exotic Cosmological Models Using ESSENCE Supernova Data Combined with Other Cosmological Probes. The Astrophysical Journal, pages 716–725, 2007.
![Page 98: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/98.jpg)
98
Estimated Posterior Distributions
Projections of the points selected by EV (bottom row) and the marginal distributions (top row).
![Page 99: Bayesian Optimization for Experiment Design · Find new scientific “laws” in physics Goal: Estimate dynamical mass of galaxy clusters. Importance: Galaxy clusters are being the](https://reader033.vdocument.in/reader033/viewer/2022050300/5f69df75609ad1643d04a8d1/html5/thumbnails/99.jpg)
Thanks for your Attention!
99