stick-breaking constructions

STICK-BREAKING CONSTRUCTIONSPatrick DallaireJune 10th, 2011

Outline Introduction of the Stick-Breaking

process

Outline Introduction of the Stick-Breaking

process Presentation of fundamental

representation

Outline Introduction of the Stick-Breaking

process Presentation of fundamental

representation The Dirichlet process The Pitman-Yor process The Indian buffet process

Outline Introduction of the Stick-Breaking

process Presentation of fundamental

representation The Dirichlet process The Pitman-Yor process The Indian buffet process

Definition of the Beta process

Outline Introduction of the Stick-Breaking

process Presentation of fundamental

representation The Dirichlet process The Pitman-Yor process The Indian buffet process

Definition of the Beta process A Stick-Breaking construction of Beta

process

Outline Introduction of the Stick-Breaking

process Presentation of fundamental

representation The Dirichlet process The Pitman-Yor process The Indian buffet process

Definition of the Beta process A Stick-Breaking construction of Beta

process Conclusion and current work

The Stick-Breaking process

The Stick-Breaking process

Assume a stick of unit length

The Stick-Breaking process

Assume a stick of unit length

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining

stick is broken by sampling the proportion to cut

The Stick-Breaking process

Assume a stick of unit length At each iteration, a part of the remaining stick

is broken by sampling the proportion to cut How should we sample these proportions?

Beta random proportions Let be the proportion to cut at

iteration

Beta random proportions Let be the proportion to cut at

iteration The remaining length can be expressed

as

Beta random proportions Let be the proportion to cut at

iteration The remaining length can be expressed

as

Thus, the broken part is defined by

Beta random proportions Let be the proportion to cut at

iteration The remaining length can be expressed

as

Thus, the broken part is defined by

We first consider the case where

Beta distribution The Beta distribution is a density

function on

Parameters and control its shape

The Dirichlet process

The Dirichlet process Dirichlet processes are often used to

produce infinite mixture models

The Dirichlet process Dirichlet processes are often used to

produce infinite mixture models Each observation belongs to one of the

infinitely many components

The Dirichlet process Dirichlet processes are often used to

produce infinite mixture models Each observation belongs to one of the

infinitely many components The model ensures that only a finite

number of components have appreciable weight

The Dirichlet process A Dirichlet process, , can be constructed

according to a Stick-Breaking process

Where is the base distribution and is a unit mass at

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

Construction demo

The Pitman-Yor process

The Pitman-Yor process A Pitman-Yor process, , can be

constructed according to a Stick-Breaking process

Where and

Evolution of the Beta cuts The parameter controls the speed at

which the Beta distribution changes

Evolution of the Beta cuts The parameter controls the speed at

which the Beta distribution changes The parameter determines initial

shapes of the Beta distribution

Evolution of the Beta cuts The parameter controls the speed at

which the Beta distribution changes The parameter determines initial

shapes of the Beta distribution When , there is no changes over

time and its called a Dirichlet process

Evolution of the Beta cuts The parameter controls the speed at

which the Beta distribution changes The parameter determines initial

shapes of the Beta distribution When , there is no changes over

time and its called a Dirichlet process

MATLAB DEMO

The Indian Buffet process

The Indian Buffet process The Indian Buffet process was initially

used to represent latent features

The Indian Buffet process The Indian Buffet process was initially

used to represent latent features Observations are generated according to

a set of unknown hidden features

The Indian Buffet process The Indian Buffet process was initially

used to represent latent features Observations are generated according to

a set of unknown hidden features The model ensure that only a finite

number of features have appreciable probability

The Indian Buffet process

Recall the basic Stick-Breaking process

The Indian Buffet process

Recall the basic Stick-Breaking process

The Indian Buffet process

Recall the basic Stick-Breaking process Here, we only consider the remaining

parts

The Indian Buffet process

Recall the basic Stick-Breaking process Here, we only consider the remaining

parts

The Indian Buffet process

Recall the basic Stick-Breaking process Here, we only consider the remaining

parts Each value corresponds to a feature

probability of appearance

Summary

Summary The Dirichlet process induces a

probability over infinitely many classes

Summary The Dirichlet process induces a

probability over infinitely many classes This is the underlying de Finetti mixing

distribution of the Chinese restaurant process

De Finetti theorem It states that the distribution of any

infinitely exchangeable sequence can be written

where is the de Finetti mixing distribution

Summary The Dirichlet process induces a

probability over infinitely many classes This is the underlying de Finetti mixing

distribution of the Chinese restaurant process

The Indian Buffet process induces a probability over infinitely many features

Summary The Dirichlet process induces a

probability over infinitely many classes This is the underlying de Finetti mixing

distribution of the Chinese restaurant process

The Indian Buffet process induces a probability over infinitely many features

Its underlying de Finetti mixing distribution is the Beta process

The Beta process

The Beta process This process

Beta with Stick-Breaking The Beta distribution has a Stick-

Breaking representation which allows to sample from

Beta with Stick-Breaking The Beta distribution has a Stick-

Breaking representation which allows to sample from

The construction is

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking

Beta with Stick-Breaking The Beta distribution has a Stick-

Breaking representation which allows to sample from

The construction is

The Beta process A Beta process is defined as

as , and is a Beta process

Stick-Breaking the Beta process

The Stick-Breaking construction of the Beta process is such that

Stick-Breaking the Beta process

Expending the first terms

Conclusion We briefly described various Stick-Breaking

constructions for Bayesian nonparametric priors

These constructions help to understand the properties of each process

It also unveils connections among existing priors

The Stick-Breaking process might help to construct new priors

Current work Applying a Stick-Breaking process to

select the number of support points in a Gaussian process

Defining a stochastic process for unbounded random directed acyclic graph

Finding its underlying Stick-Breaking representation