Forest Learning from Data

Joe Suzuki

July 17, 2017

Road Map

PART-II: July 24, 2017 (based on PART-I)

1. Estimating Mutual Information (15 mins)

2. Learning Forests from Data (25 mins)

3. Learning Bayesian Networks from Data (5 mins)

4. Exercise (45 mins)

PART-I: July 17, 2017

A Bayesian Approach to Data Compression

Entropy

Mutual Information (MI)

Correlation may not detect independence!

ML Estimator of MI

Bayesian Testing of Independence

Bayesian Estimation of MI

From Stirling’s formula

For large n

Experiments 500 trials for binary seq. of length n=200

BNSL: a CRAN package (J. Suzuki and J. Kawahara, 2017)

Bayesian Network Learning Structure

https://cran.r-project.org/web/packages/BNSL/index.html

collects research results by Joe Suzuki.

install(“BNSL”)

library(BNSL)

n=200; p=0.5; x=rbinom(n,1,p); y=rbinom(n,1,p) # seqs are generated

mi(x,y, proc=9) # I_n

mi(x,y) # J_n

Tree Approximation

Factorization w.r.t. A Tree

Find E s.t. D(P||P’) is minimized

Kruskal’s Algorithm

Chow-Liu Algorithm

Experiments using Asia data set

• library(BNSL)

• mm=mi_matrix(asia, proc=9) # I_n is used

• edge.list=kruskal(mm)

• g=graph_from_edgelist(edge.list, directed=FALSE)

• plot(g)

• mm=mi_matrix(asia) # J_n is used

• edge.list=kruskal(mm)

• g=graph_from_edgelist(edge.list, directed=FALSE)

• plot(g)

Asia (8 variables)

S. Lauritzen, D. Spiegelhalter. Local

Computation with Probabilities on

Graphical Structures and their

Application to Expert Systems (with

discussion). Journal of the Royal

Statistical Society: Series B

(Statistical Methodology), 50(2):157-224, 1988

Asia Data Set

I. A. Beinlich, H. J. Suermondt, R. M. Chavez, and G. F. Cooper. The ALARM Monitoring System: A Case Study with Two Probabilistic Inference Techniques for Belief Networks. In Proceedings of the 2nd European Conference on Artificial Intelligence in Medicine, pages 247-256. Springer-Verlag, 1989.

Alarm (37 varibles)

Alarm Data Set

Learning Bayesian Networks from Data

The # of candidate structures with p nodes is more than exponential with p

25 DAGs exist for p=3 but only 11 BNs are considered

7 local scores and 11 global scores

• Estimating Mutual Information

• Learning Forests from Data

• Learning Bayesian Networks from Data

Summary

Problem Set #2