conistency of random forests

Consistency of Random ForestsHoang N.V.

hoangnvvnua@gmail.comDepartment of Computer Science

FITA – Viet Nam Institute of Agriculture

Seminar IT R&D, HANUHa Noi, December 2015

Machine Learning, what is?

“true”

Parametric

Non-parametric

Supervised problems: not too difficult

Unsupervised problems: is very difficult

Find a parameter which minimize the loss function

Supervised Learning

ℒ𝑛

L is a loss function

Classification: zero-one loss function

Regression: 𝕃1, 𝕃2

Bias-variance tradeoff

If the model is too simple, the solution is biased and does not fit the data.

If the model is too complex then it is very sensitive to small changes in the data.

[Hastie et all., 2005]

Ensemble Methods

Bagging[Random Forest]

Tree Predictor

ℒ𝑛

ℒ𝑛

Pick an internal node to split

Pick the best split in

Split A into two child nodes ( and )

Set

A splitting scheme induces a partition Λ of the feature space into non-overlapping rectangles 𝑃1, … , 𝑃ℓ.

Tree Predictor

ℒ𝑛

ℒ𝑛

Select an internal node to split

Select the best split in

Split A into two child nodes ( and )

Set

A splitting scheme induces a partition Λ of the feature space into non-overlapping rectangles 𝑃1, … , 𝑃ℓ.

Predicting Rule

Λ

Λ ℒ𝑛

Λ ℒ𝑛

Tree Predictor

ℒ𝑛

ℒ𝑛

Select an internal node to split

Select the best split in

Split A into two child nodes ( and )

Set

A splitting scheme induces a partition Λ of the feature space into non-overlapping rectangles 𝑃1, … , 𝑃ℓ.

Training Methods

ID3 (Iterative Dichotomiser 3)

C4.5

CART (Classification and Regression Tree)

CHAID

MARS

Conditional Inference Tree

Predicting Rule

Λ

Λ ℒ𝑛

Λ ℒ𝑛

Forest = Aggregation of trees

Aggregating Rule

ℒ𝑛 ℒ𝑛

ℒ𝑛 ℒ𝑛 ℒ𝑛

Grow different trees from same learning set ℒ𝑛

Sampling with replacement [Breiman, 1994]

Random subspace sampling [Ho, 1995 & 1998]

Random output sampling [Breiman, 1998]

Randomized C4.5 [Dietterich, 1998]

Purely random forest [Breiman, 2000]

Extremely random trees [Guerts, 2006]

Grow different trees from same learning set ℒ𝑛

Sampling with replacement - random subspace [Breiman, 2001]

Sampling with replacement - weighted subspace [Amaratunga, 2008; Xu, 2008; Wu, 2012]

Sampling with replacement - random subspace and regularized [Deng, 2012]

Sampling with replacement - random subspace and guided-regularized [Deng, 2013]

Sampling with replacement - random subspace and random split position selection [Saïp Ciss, 2014]

Some RF extensions

quantile estimation Meinshausen, 2006

survival analysis Ishwaran et al., 2008

ranking Clemencon et al., 2013

online learning Denil et al., 2013;

Lakshminarayanan et al., 2014

GWA problems Yang et al., 2013; Botta et al., 2014

What is a good learner?[What is friendly with my data?]

What is good in high-dimensional settings

Breiman, 2001

Wu et al., 2012

Deng, 2012

Deng., 2013

Saïp Ciss, 2014

Simulation Experiment

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

y1 y2 y3 y4

AA ABB B

Random Forest [Breiman, 2001]

WSRF [Wu, 2012]

Random Uniform Forest [Saïp Ciss, 2014]

RRF [Deng, 2012]

GRRF [Deng, 2013]

Simulation Experiment

Random Forest [Breiman, 2001]

WSRF [Wu, 2012]

Random Uniform Forest [Saïp Ciss, 2014]

RRF [Deng, 2012]

GRRF [Deng, 2013]

GRRF with AUC [Deng, 2013]

GRRF with ER [Deng, 2013]

Simulation Experiment

B

A

C

D

B E

A

E D

Multiple Class Tree

Random Forest [Breiman, 2001]

WSRF [Wu, 2012]

Random Uniform Forest [Saïp Ciss, 2014]

RRF [Deng, 2012]

GRRF [Deng, 2013]

GRRF with ER [Deng, 2013]

What is a good learner?[Nothing you do will convince me]

[I need rigorous theoretical guarantees]

Asymptotic statistics and learning theory[go beyond experiment results]

Machine Learning, what is?

Parametric

Non-parametric

Supervised problems: not too difficult

Unsupervised problems: is very difficult

Find a parameter which minimize the loss function

“right”

Pattern (which learnt from ℒ𝑛 ) is “true”, isn’t it? How much do I believe?

Is this procedure friendly with my data?

What is the best possible procedure for my problem?

What is if our assumptions are wrong?

“efficient”

How many observations do I need in order to achieve a “believed” pattern?

How many computations do I need?

Assumption: There are some patterns

Learning Theory [Vapnik, 1999]

asymptotic theory

necessary and sufficient conditions

the best

possible

Supervised Learning

ℒ𝑛

L is a loss function

Classification: zero-one loss function

Regression: 𝕃1, 𝕃2

Supervised Learning

Generator 𝒙

𝒙

𝑦

𝑦 𝑦′

Supervisor

Machine Learning

Two different goals

imitate (prediction accuracy)

identify (interpretability)

What is the best predictor?

What is the best predictor

Bayes model

residual error

A model built from any learning set ℒ, Err( ) ≤ Err( )

In theory, when 𝑃(𝑋𝑌) is known

What is the best predictor

If is zero-one loss function, the Bayes model is

In classification, the best possible classifier consists in systematically predicting the most likely class 𝑦 ∈ {𝑐1, … , 𝑐𝐽} given 𝑋 = 𝒙

What is the best predictor

If is the squared error loss function, the Bayes model is

In regression, the best possible regressor consists in systematically predicting the average value of 𝑌 given 𝑋 = 𝒙

Given a learning algorithm 𝒜 and a loss function

𝜑𝑛 = 𝒜(ℒ𝑛)

ℒ𝑛

𝒜 ℒ𝑛 }

Learning algorithm 𝒜 is consistent in L if and only if

𝐸𝑟𝑟( ) ⟶𝑛→∞𝑃 𝐸𝑟𝑟(𝜑𝐵)

𝐸𝑟𝑟(𝜑𝑛 ) ⟶𝑛→∞𝑃 𝐸𝑟𝑟(𝜑𝐵)

Random Forests are consistent, aren’t they?

𝒜

Θ 𝒜

Θ is used to sample the training set or to select the candidate directionsor positions for splitting

Θ is independent of the dataset and thus unrelated to the particularproblem

In some new variants of RF, Θ is depend on the dataset

Generalized Random Forest

Bagging Procedure

Θ Θ1, … , Θm

Θi ℒ𝑛 𝒜(Θ𝑖 ℒ𝑛)

ℒ𝑛 Θ𝑖 ℒ𝑛

ℒ𝑛 Θ1 ℒ𝑛 Θ𝑚 ℒ𝑛

Generalized Random Forest

Consistency of Random Forests

lim𝑛→∞

Err(𝐻𝑚 . ; ℒ𝑛 ) = Err(𝜑𝐵)Problem

- m is finite ⇒ predictor depend on trees that formed forest

- structure of a tree depend on Θi and learning set⟹ finite forest is actually a subtle combination of randomness and

depending-on-data structures⟹ finite forests predictions can be difficult to interpret (randomprediction or not)

- Non-asymptotic rate of convergence

Challenges

𝐻∞(𝑥; ℒ𝑛) = 𝔼Θ{ Θ ℒ𝑛 }

lim𝑚→∞

𝐻𝑚(𝑥; ℒ𝑛) = 𝐻∞(𝑥; ℒ𝑛)

Problem

- infinite forest is good than finite forest, isn’t it?- What is good m? (rate of convergence)

Challenge

Consistency of Random Forests

Review some recent results

Strength, Correlation and Err [Breiman, 2001]

Θi ℒ𝑛 Θi ℒ𝑛

Θ ℒ𝑛 Θ ℒ𝑛

Theorem 2.3 An upper bound for the generalization error is given by:𝑃𝐸∗ ≤ 𝜌(1 − 𝑠2)/𝑠2

where 𝜌 is the mean value of the correlation, s is the strength of theset of classifiers.

RF and Adaptive Nearest Neighbors [Lin et al, 2006]

Θi ℒ𝑛 𝒜(Θ𝑖 ℒ𝑛)

Θ

Θi ℒ𝑛=

1

𝒙𝐣:𝒙

𝐣∈ 𝐿 Θ𝑖,𝒙

𝑗:𝒙𝐣∈ 𝐿 Θ𝑖,𝒙

𝑦j = 𝑗=1𝑛 𝑤

𝑗Λ𝑖

𝑦𝑗

ℒ𝑛 Θ𝑖 ℒ𝑛 𝑗=1𝑛 𝑤𝑗𝑦𝑗

𝑤𝑗 𝑤𝑗Λ𝑖

𝐻∞(𝒙; ℒ𝑛) ΘΘ

Θ

Non-adaptive if 𝑤𝑗 not depend on 𝑦𝑖’s of the learning set

RF and Adaptive Nearest Neighbors [Lin et al, 2006]

Θi ℒ𝑛 𝒜(Θ𝑖 ℒ𝑛)

Θ

Θi ℒ𝑛=

1

𝒙𝐣:𝒙

𝐣∈ 𝐿 Θ𝑖,𝒙

𝑗:𝒙𝐣∈ 𝐿 Θ𝑖,𝒙

𝑦j = 𝑗=1𝑛 𝑤

𝑗Λ𝑖

𝑦𝑗

ℒ𝑛 Θ𝑖 ℒ𝑛 𝑗=1𝑛 𝑤𝑗𝑦𝑗

𝑤𝑗 𝑤𝑗Λ𝑖

𝐻∞(𝒙; ℒ𝑛) ΘΘ

Θ

Non-adaptive if 𝑤𝑗 not depend on 𝑦𝑖’s of the learning set

The terminal node size k should be made to increase with the sample size 𝑛. Therefore, growing large trees (k being a small constant) does not always

give the best performance.

Biau et al, 2008Given a learning set ℒ𝑛 = 𝒙1, 𝑦1 , … , 𝒙𝑛, 𝑦𝑛 of ℝ𝑑 ∗ {0, 1}

Binary classifier 𝜑𝑛 which trained from ℒ𝑛: ℝ𝑑 ⟶ {0, 1}

𝜑𝑛 𝜑𝑛(𝑋) ≠ 𝑌

𝜑𝐵 𝑥 = 𝕀{ } 𝜑𝐵

A sequence {𝜑𝑛} of classifiers is consistent for a certain distribution of (𝑋, 𝑌) if 𝐸𝑟𝑟(𝜑𝑛) ⟶ in probability

Assume that the sequence {𝑇𝑛} of randomized classifiers is consistent for a certain distribution of 𝑋, 𝑌 . Then the voting classifier 𝐻𝑚(for any value of m) and the averaged classifier 𝐻∞ are also consistent.

Biau et al, 2008

Growing Trees

Node 𝐴 is randomly selected

The split feature j is selected uniformly at random from [1, … , 𝑝]

Finally, the selected node is split along the randomly chosen feature at a random location

* Recursive node splits do not depend on the labels 𝑦1, … , 𝑦𝑛

Theorem 2 Assume that the distribution of 𝑋 is supported on [0, 1]𝑑.Then purely random forest classifier 𝐻∞ is consistent whenever 𝑘 ⟶∞ and 𝑘

𝑛 ⟶ 0 as n ⟶ ∞.

Biau et al, 2008

Growing Trees

ℒ𝑛 𝒜( )

Theorem 6 Let {𝑇Λ} be a sequence of classifiers that is consistent for thedistribution of 𝑋𝑌. Consider the bagging classifier 𝐻𝑚 and 𝐻∞, using parameter𝑞𝑛. If 𝑛𝑞𝑛 ⟶ ∞ as n ⟶ ∞ then both classifier are consistent.

Biau et al, 2012

Growing Trees

At each node, a coordinate is selected with 𝑝𝑛𝑗 ∈ (0, 1) is the probability j-th feature is selected

the split is at the midpoint of the chosen side

Theorem 1 Assume that the distribution of 𝑋 has support on [0, 1]𝑑.Then the random forests estimate 𝐻∞(𝒙; ℒ𝑛) is consistent whenever

𝑝𝑛𝑗𝑙𝑜𝑔𝑘𝑛 ⟶ ∞ for all j=1, …, p and 𝑘𝑛𝑛 ⟶ 0 as 𝑛 ⟶ ∞.

Biau et al, 2012

Assume that X is uniformly distributed on [0,1]𝑝

𝒑𝒏𝒋 = (𝟏/𝑺)(𝟏 + 𝝃𝒏𝒋) 𝒇𝒐𝒓 𝒋 ∈ 𝓢

In sparse settings

Estimation Error (variance)

𝔼{[𝐻∞ 𝒙; ℒ𝑛 − 𝐻∞ 𝒙; ℒ𝑛 ]2} ≤ 𝐶𝜎2S2

S − 1

𝑆2𝑝

(1 + 𝜉𝑛)𝑘𝑛

𝑛(𝑙𝑜𝑔𝑘𝑛)𝑆/2𝑝

If 𝑎 < 𝑝𝑛𝑗 < 𝑏 form some constants 𝑎, 𝑏 ∈ 0,1 then

1 + 𝜉𝑛 ≤𝑆 − 1

𝑆2𝑎 1 − 𝑏

𝑆2𝑝

Biau et al, 2012

Assume that X is uniformly distributed on [0,1]𝑝 and 𝜑𝐵 𝒙𝒮 is 𝐿 − 𝐿𝑖𝑝𝑠𝑐ℎ𝑖𝑡𝑧 on [0,1]𝑠

𝒑𝒏𝒋 = (𝟏/𝑺)(𝟏 + 𝝃𝒏𝒋) 𝒇𝒐𝒓 𝒋 ∈ 𝓢

In sparse settings

Approximation Error (bias2)

𝔼 𝐻∞ 𝒙; ℒ𝑛 − 𝜑𝐵 𝒙2

≤ 2𝑆𝐿2𝑘𝑛−

0.75𝑆𝑙𝑜𝑔2 1+𝛾𝑛 + [ sup

𝑥∈[0,1]𝑝𝜑𝐵

2(𝒙)]𝑒−𝑛/2𝑘𝑛

where 𝛾𝑛 = min𝑗∈ 𝒮

𝜉𝑛𝑗 tends to 0 as n tends to infinity.

Finite and infinite RFs [Scornet, 2014]

ℒ𝑛 Θ𝑖 ℒ𝑛

𝐻∞(𝒙; ℒ𝑛) = 𝔼Θ{ Θ ℒ𝑛 }

ℒ𝑛

ℒ𝑛 𝐻∞(𝒙; ℒ𝑛)

Theorem 3.1 Conditionally on ℒ𝑛, almost surely, for all 𝑥 ∈ 0, 1 𝑝, wehave: 𝐻𝑚 𝒙; ℒ𝑛

𝑀→∞𝐻∞(𝒙; ℒ𝑛).

Finite and infinite RFs [Scornet, 2014]

One has 𝑌 = 𝑚 𝑋 + 𝜀 where 𝜀 is a centered Gaussian noise withfinite variance 𝜎2, independent of 𝑋,

and 𝑚 ∞ = sup𝑥∈[0,1]𝑝

|𝑚(𝑥)| < ∞.

Assumption H

Theorem 3.3 Assume H is satisfied. Then, for all m, 𝑛 ∈ ℕ∗,

𝐸𝑟𝑟 𝐻𝑚 𝒙; ℒ𝑛 = 𝐸𝑟𝑟 𝐻∞(𝒙; ℒ𝑛) +1

𝑚𝔼𝑋,ℒ

𝑛[𝕍Θ[𝑇Λ (𝒙; Θ, ℒ𝑛)]]

⇒ 𝑚 ≥8 𝑚 ∞

2 + 𝜎2

𝜀+

32𝜎2𝑙𝑜𝑔𝑛

𝜀𝑡ℎ𝑒𝑛 𝐸𝑟𝑟 Hm − 𝐸𝑟𝑟 H∞ ≤ 𝜀

0 ≤ 𝐸𝑟𝑟 Hm − 𝐸𝑟𝑟 H∞ ≤8

m( 𝑚 ∞

2 + 𝜎2(1 + 4𝑙𝑜𝑔𝑛))

RF and Additive regression model [Scornet et al., 2015]

Growing Treeswithout replacement

Assume that 𝐴 is selected node and 𝐴 > 1

Select uniformly, without replacement, a subset ℳ𝑡𝑟𝑦 ⊂ 1, … , 𝑝 , |ℳ𝑡𝑟𝑦| = 𝑚𝑡𝑟𝑦

Select the best split in A by optimizing the CART-split criterion along the coordinates in ℳ𝑡𝑟𝑦

Cut the cell 𝐴 according to the best split. Call 𝐴𝐿 and 𝐴𝑅 the true resulting cell

Set 𝐴 𝐴𝐿 𝐴𝑅

RF and Additive regression model [Scornet et al., 2015]

𝑌 = 𝑗=1𝑝

𝑚𝑗(𝑋(𝑗)) + 𝜀

Assumption H1

Theorem 3.1 Assume that (H1) is satisfied. Then, provided 𝑛 ⟶ ∞ and𝑡𝑛(𝑙𝑜𝑔𝑎𝑛)9/𝑎𝑛 ⟶ 0, is consistent.

Theorem 3.2 Assume that (H1) and (H2) are satisfied and let 𝑡𝑛 = 𝑎𝑛.Then, provided 𝑎𝑛 ⟶ ∞, 𝑡𝑛 ⟶ ∞ and 𝑎𝑛𝑙𝑜𝑔𝑛/𝑛 ⟶ 0, is consistent.

RF and Additive regression model [Scornet et al., 2015]

, 𝑗1,𝑛 𝑋 , … , 𝑗𝑘,𝑛 𝑋 the first cut directions used to construct

the cell containing 𝑋. 𝑗𝑞,𝑛 𝑋 = ∞ if the cell has been cut strictly less than

q times.

Theorem 3.2 Assume that (H1) is satisfied. let k ∈ ℕ∗ and 𝜉 > 0.Assume that there is no interval [𝑎, 𝑏] and no 𝑗 ∈ {1, … , 𝑆} such that𝑚𝑗 is constant on [𝑎, 𝑏]. Then, with probability 1 − 𝜉, for all 𝑛 large

enough, we have, for all 1 ≤ 𝑞 ≤ 𝑘, 𝑗𝑞,𝑛 𝑋 ∈ {1, … , 𝑆}.

[Wager, 2015]

A partition Λ is 𝛼, 𝑘 − 𝑣𝑎𝑙𝑖𝑑 if can generated by a recursive partitioning scheme inwhich each child node contains at least a fraction 𝛼 of the data points in its parentnode for some 0 < 𝛼 < 0.5, and each terminal node contains at least 𝑘 trainingexamples for some k ∈ N.

Given a dataset 𝑋, 𝒱𝛼,𝑘(𝑋) denote the set of 𝛼, 𝑘 − 𝑣𝑎𝑙𝑖𝑑 partitions

𝑇Λ: [0,1]𝑝→ ℝ, 𝑇Λ 𝒙 =1

|{𝒙𝑖: 𝒙𝑖 ∈ 𝐿(𝒙)}| {𝑖: 𝒙𝑖∈𝐿(𝒙)} 𝑦𝑖 (is called valid tree)

𝑇Λ∗: [0,1]𝑝→ ℝ, 𝑇Λ

∗ 𝒙 = 𝔼[𝑌|𝑋 ∈ 𝐿(𝒙)] (is called partition-optimal tree)

Whether we can treat 𝑇Λ as a good approximation to 𝑇Λ∗ the

supported on the partition Λ

Given a learning set ℒ𝑛 of [0,1]𝑝∗ −𝑀

2,𝑀

2with 𝑋~𝑈([0,1]𝑝)

[Wager, 2015]

Theorem 1Given parameters n, p, k such that

lim𝑛→∞

log 𝑛 log 𝑝

𝑘= 0 𝑎𝑛𝑑 𝑝 = Ω(𝑛)

then

lim𝑛,𝑑,𝑘→∞

ℙ sup𝑥∈ 0,1 𝑝,Λ∈𝒱𝛼,𝑘

|𝑇Λ − 𝑇Λ∗| ≤ 6𝑀

log 𝑛 log(𝑝)

klog((1 − 𝛼)−1)= 1

[Wager, 2015]

Growing Trees (Guest-and-check)

Select a currently un-split node 𝐴 containing at least 2k training examples

Pick a candidate splitting variable 𝑗 ∈ {1, … , 𝑝} uniformly at random

Pick the minimum squared error (ℓ( 𝜃)) splitting point 𝜃

If either there has already been a successful split along variable j for some other nod or

ℓ( 𝜃) ≥ 36𝑀2log 𝑛 log(𝑑)

𝑘𝑙𝑜𝑔((1 − 𝛼)−1)

The split succeeds and we cut the node 𝐴 at 𝜃 along the j-th variable; if not we do not split the node 𝐴 this time.

[Wager, 2015]In sparse settings

and a set of sign variables 𝜎𝑗 ∈ ±1 such that, for all

𝑗 ∈ and all 𝑥 ∈ [0,1]𝑝,

𝔼 𝑌 𝑋 −𝑗 = 𝑥 −𝑗 , 𝑋 𝑗 >12

− 𝔼 𝑌 𝑋 −𝑗 = 𝑥 −𝑗 , 𝑋 𝑗 ≤12

≥ 𝛽𝜎𝑗

Assumption H1

is Lipschitz-continuous in

Assumption H2

[Wager, 2015]In sparse settings

and a set of sign variables 𝜎𝑗 ∈ ±1 such that, for all

𝑗 ∈ and all 𝑥 ∈ [0,1]𝑝,

𝔼 𝑌 𝑋 −𝑗 = 𝑥 −𝑗 , 𝑋 𝑗 >12

− 𝔼 𝑌 𝑋 −𝑗 = 𝑥 −𝑗 , 𝑋 𝑗 ≤12

≥ 𝛽𝜎𝑗

Assumption H1

is Lipschitz-continuous in

Assumption H2

Theorem 2 Under the conditions of theorem 1, suppose thatassumptions in the sparse setting hold, then guest-and-check forest isconsistent.

[Wager, 2015]

𝐻{Λ}1𝐵: [0,1]𝑝→ ℝ, 𝐻 Λ 1

𝐵 𝒙 =1

𝐵 𝑏=1

𝐵 𝑇Λ𝑏(𝒙) (is called valid forest)

𝐻{Λ}1

𝐵∗ : [0,1]𝑝→ ℝ, 𝐻

{Λ}1𝐵

∗ 𝒙 =1

𝐵 𝑏=1

𝐵 𝑇Λ𝑏

∗ (𝒙) (is called partition-optimal forest)

Theorem 4

lim𝑛,𝑑,𝑘→∞

ℙ sup𝐻∈ℋ𝛼,𝑘

1

𝑛

𝑖=1

𝑛

(𝑦𝑖 − 𝐻(𝑥𝑖))2 − 𝔼[(𝑌 − 𝐻(𝑋))2] ≤ 11𝑀2log 𝑛 log(𝑝)

klog((1 − 𝛼)−1)= 1

ReferencesB. Efron. Estimation and accuracy after model selection. Journal of the American Statistical Association, 2013.

B. Lakshminarayanan et al. Mondrian forests: Efficient online random forests. arXiv:1406.2673, 2014.

B. Xu et al. Classifying very high-dimensional data with random forests built from small subspaces. International Journal of Data Warehousing and Mining (IJDWM) 8(2), 37:44–63, 2012.

D. Amaratunga et al. Enriched random forests. Bioinformatics (Oxford, England), 24(18): 2010–2014, 2008 doi:10.1093/bioinformatics/btn356

E. Scornet. On the asymptotics of random forests. arXiv:1409.2090, 2014

E. Scornet et al. Consistency of random forests. The Annals of Statistics. 43 (2015), no. 4, 1716--1741. doi:10.1214/15-AOS1321. http://projecteuclid.org/euclid.aos/1434546220.

G. Biau et al. Consistency of random forests and other averaging classifiers. Journal of Machine Learning Research, 9:2015–2033, 2008.

G. Biau. Analysis of a random forests model. Journal of Machine Learning Research, 13:1063–1095, 2012.

H. Deng et al. Feature Selection via Regularized Trees, The 2012 International Joint Conference on Neural Networks (IJCNN), IEEE, 2012.

H. Deng et al. Gene Selection with Guided Regularized Random Forest , Pattern Recognition, 46.12 (2013): 3483-3489

H. Ishwaran et al. Random survival forest. The Annals of Applied Statistics, 2:841–860, 2008.

L. Breiman. Bagging predictors. Technical Report No. 421, Statistics Department, UC Berkeley, 1994.

ReferencesL.Breiman. Randomizing outputs to increase prediction accuracy, Technical Report 518, Statistics Department, UC Berkeley, 1998.

L. Breiman. Some infinite theory for predictor ensembles. Technical Report 577, Statistics Department, UC Berkeley, 2000.

L. Breiman. Random forests. Machine Learning, 45:5–32, 2001.

M. Denil et al. Consistency of online random forests. International Conference on Machine Learning (ICML) 2013.

N. Meinshausen. Quantile regression forests. Journal of Machine Learning Research, 7:983–999, 2006.

P. Geurts et al. Extremely randomized trees. Machine Learning, 63(1):3-42, 2006.

Q. Wu et al. Snp selection and classification of genome-wide snp data using stratified sampling random forests. NanoBioscience, IEEE Transactions on 11(3): 216–227, 2012.

T. Dietterich. An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting and Randomization, Machine Learning 1-22, 1998

T.K. Ho. Random decision forests, Document Analysis and Recognition, 1995., Proceedings of the Third International Conference on , 1:278-282, 14-16 Aug 1995, doi: 10.1109/ICDAR.1995.598994

T.K. Ho. The random subspace method for constructing decision forests, IEEE Trans. on Pattern Analysis and Machine Intelligence, 20(8):832-844, 1998

Saïp Ciss. Random Uniform Forests. 2015. <hal-01104340v2>

S. Cl´emen¸con et al. Ranking forests. Journal of Machine Learning Research, 14:39–73, 2013.

ReferencesS. Wager. Asymptotic theory for random forests. arXiv:1405.0352, 2014

S. Wager. Uniform convergence of random forests via adaptive concentration. arXiv:1503.06388, 2015

Y. Lin and Y. Jeon. Random forests and adaptive nearest neighbors. Journal of the American Statistical Association, 101:578–590, 2006.

V. N. Vapnik. An overview of Statistical Learning Theory. IEEE Trans. on Neural Networks, 10(5):988-999, 1999.

RF and Additive regression model [Scornet et al., 2015]

the indicator that falls in the same cell as in the

random tree designed with ℒn and the random parameter

where ′is an independent copy of .

′, 𝑋1, … , 𝑋𝑛,

′, 𝑋1, … , 𝑋𝑛

RF and Additive regression model [Scornet et al., 2015]