Nonparametric Bayesian Statistics: Part II

Tamara BroderickITT Career Development Assistant Professor Electrical Engineering & Computer Science

MIT

Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:

…

⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)

[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]

Vkiid⇠ Beta(1,↵) ⇢k =

2

4k�1Y

j=1

(1� Vj)

3

5Vk

1 2 3 4 ……

• Part of: DP mixture model

1

Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:

…

⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)

[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]

Vkiid⇠ Beta(1,↵) ⇢k =

2

4k�1Y

j=1

(1� Vj)

3

5Vk

1 2 3 4 ……

• Part of: DP mixture model

1

Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:

…

⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)

[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]

Vkiid⇠ Beta(1,↵) ⇢k =

2

4k�1Y

j=1

(1� Vj)

3

5Vk

1 2 3 4 ……

• Part of: DP mixture model

1

Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:

…

⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)

[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]

Vkiid⇠ Beta(1,↵) ⇢k =

2

4k�1Y

j=1

(1� Vj)

3

5Vk

1 2 3 4 ……

• Part of: DP mixture model

1

Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:

…

⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)

[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]

Vkiid⇠ Beta(1,↵) ⇢k =

2

4k�1Y

j=1

(1� Vj)

3

5Vk

1 2 3 4 ……

• Part of: DP mixture model

1

Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:

⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)

[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]

1 2 3 4 ……

• Part of: DP mixture model

1

Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:

⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)

[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]

1 2 3 4 ……

• Part of: DP mixture model

1

Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:

⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)

[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]

1 2 3 4 ……

• Part of: DP mixture model

1

Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:

⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)

[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]

1 2 3 4 ……

• Part of: DP mixture model

1

Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:

…

⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)

[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]

1 2 3 4 ……

• Part of: DP mixture model

1

Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:

…

⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)

[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]

1 2 3 4 ……

• Part of: DP mixture model

1

Vkiid⇠ Beta(1,↵)

Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:

…

⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)

[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]

1 2 3 4 ……

• Part of: DP mixture model

1

Vkiid⇠ Beta(1,↵) ⇢k =

2

4k�1Y

j=1

(1� Vj)

3

5Vk

Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:

…

⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)

[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]

1 2 3 4 ……

• Part of: DP mixture model

1

Vkiid⇠ Beta(1,↵) ⇢k =

2

4k�1Y

j=1

(1� Vj)

3

5Vk

Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:

…

⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)

[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]1

Vkiid⇠ Beta(1,↵) ⇢k =

2

4k�1Y

j=1

(1� Vj)

3

5Vk

1 2 3 4 ……

• Part of: DP mixture model

DP or not DP, that is the question• GEM: • Compare to:

• Finite (small K) mixture model !

!

• Finite (large K) mixture model !

!

• Time series

…

…

…

2

DP or not DP, that is the question• GEM: • Compare to:

• Finite (small K) mixture model !

!

• Finite (large K) mixture model !

!

• Time series

…

…

…

2

DP or not DP, that is the question• GEM: • Compare to:

• Finite (small K) mixture model !

!

• Finite (large K) mixture model !

!

• Time series

…

…

…

2

DP or not DP, that is the question• GEM: • Compare to:

• Finite (small K) mixture model !

!

• Finite (large K) mixture model !

!

• Time series

…

…

…

2

DP or not DP, that is the question• GEM: • Compare to:

• Finite (small K) mixture model !

!

• Finite (large K) mixture model !

!

• Time series

…

…

…

2

DP or not DP, that is the question• GEM: • Compare to:

• Finite (small K) mixture model !

!

• Finite (large K) mixture model !

!

• Time series

…

…

…

2

Nonparametric Bayes: Part II• Last time:

• Understand what it means to have an infinite/growing number of parameters

• Finite representation allows use of infinite model • www.tamarabroderick.com/tutorials.html

• This time: • Avoid the infinity of parameters for inference • e.g. Chinese restaurant process

3

Nonparametric Bayes: Part II• Last time:

• Understand what it means to have an infinite/growing number of parameters

• Finite representation allows use of infinite model • www.tamarabroderick.com/tutorials.html

• This time: • Avoid the infinity of parameters for inference • e.g. Chinese restaurant process

3

Nonparametric Bayes: Part II• Last time:

• Understand what it means to have an infinite/growing number of parameters

• Finite representation allows use of infinite model • www.tamarabroderick.com/tutorials.html

• This time: • Avoid the infinity of parameters for inference • e.g. Chinese restaurant process

3

Nonparametric Bayes: Part II• Last time:

• Understand what it means to have an infinite/growing number of parameters

• Finite representation allows use of infinite model • www.tamarabroderick.com/tutorials.html

• This time: • Avoid the infinity of parameters for inference • e.g. Chinese restaurant process

3

Nonparametric Bayes: Part II• Last time:

• Understand what it means to have an infinite/growing number of parameters

• Finite representation allows use of infinite model • www.tamarabroderick.com/tutorials.html

• This time: • Avoid the infinity of parameters for inference • e.g. Chinese restaurant process

3

Nonparametric Bayes: Part II• Last time:

• Understand what it means to have an infinite/growing number of parameters

• Finite representation allows use of infinite model • www.tamarabroderick.com/tutorials.html

• This time: • Avoid the infinity of parameters for inference • e.g. Chinese restaurant process

3

Nonparametric Bayes: Part II• Last time:

• Understand what it means to have an infinite/growing number of parameters

• Finite representation allows use of infinite model • www.tamarabroderick.com/tutorials.html

• This time: • Avoid the infinity of parameters for inference • e.g. Chinese restaurant process

3

Marginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2), zniid⇠ Cat(⇢1, ⇢2)

4

• Integrate out the frequenciesMarginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2), zniid⇠ Cat(⇢1, ⇢2)

4

• Integrate out the frequenciesMarginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2)

p(zn = 1|z1, . . . , zn�1), zn

iid⇠ Cat(⇢1, ⇢2)

4

• Integrate out the frequenciesMarginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2)

p(zn = 1|z1, . . . , zn�1), zn

iid⇠ Cat(⇢1, ⇢2)

=

Zp(zn = 1|⇢1)p(⇢1|z1, . . . , zn�1)d⇢1

4

• Integrate out the frequenciesMarginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2)

p(zn = 1|z1, . . . , zn�1), zn

iid⇠ Cat(⇢1, ⇢2)

=

Zp(zn = 1|⇢1)p(⇢1|z1, . . . , zn�1)d⇢1

=

Z⇢1Beta(⇢1|a1,n, a2,n)d⇢1

4

• Integrate out the frequenciesMarginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2)

p(zn = 1|z1, . . . , zn�1), zn

iid⇠ Cat(⇢1, ⇢2)

=

Zp(zn = 1|⇢1)p(⇢1|z1, . . . , zn�1)d⇢1

=

Z⇢1Beta(⇢1|a1,n, a2,n)d⇢1

a1,n := a1 +n�1X

m=1

1{zm = 1}, a2,n = a2 +n�1X

m=1

1{zm = 2}

4

• Integrate out the frequenciesMarginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2)

p(zn = 1|z1, . . . , zn�1), zn

iid⇠ Cat(⇢1, ⇢2)

=

Zp(zn = 1|⇢1)p(⇢1|z1, . . . , zn�1)d⇢1

=

Z⇢1Beta(⇢1|a1,n, a2,n)d⇢1

=

Z⇢1

�(a1,n + a2,n)

�(a1,n)�(a2,n)⇢a1,n�11 (1� ⇢1)

a2,n�1d⇢1

a1,n := a1 +n�1X

m=1

1{zm = 1}, a2,n = a2 +n�1X

m=1

1{zm = 2}

4

• Integrate out the frequenciesMarginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2)

p(zn = 1|z1, . . . , zn�1), zn

iid⇠ Cat(⇢1, ⇢2)

=

Zp(zn = 1|⇢1)p(⇢1|z1, . . . , zn�1)d⇢1

=

Z⇢1Beta(⇢1|a1,n, a2,n)d⇢1

=

Z⇢1

�(a1,n + a2,n)

�(a1,n)�(a2,n)⇢a1,n�11 (1� ⇢1)

a2,n�1d⇢1

=�(a1,n + a2,n)

�(a1,n)�(a2,n)

�(a1,n + 1)�(a2,n)

�(a1,n + a2,n + 1)

a1,n := a1 +n�1X

m=1

1{zm = 1}, a2,n = a2 +n�1X

m=1

1{zm = 2}

4

• Integrate out the frequenciesMarginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2)

p(zn = 1|z1, . . . , zn�1), zn

iid⇠ Cat(⇢1, ⇢2)

=

Zp(zn = 1|⇢1)p(⇢1|z1, . . . , zn�1)d⇢1

=

Z⇢1Beta(⇢1|a1,n, a2,n)d⇢1

=

Z⇢1

�(a1,n + a2,n)

�(a1,n)�(a2,n)⇢a1,n�11 (1� ⇢1)

a2,n�1d⇢1

=�(a1,n + a2,n)

�(a1,n)�(a2,n)

�(a1,n + 1)�(a2,n)

�(a1,n + a2,n + 1)

=a1,n

a1,n + a2,n

a1,n := a1 +n�1X

m=1

1{zm = 1}, a2,n = a2 +n�1X

m=1

1{zm = 2}

4

• Integrate out the frequencies !

!

!

!

• Pólya urn

Marginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2)

p(zn = 1|z1, . . . , zn�1), zn

iid⇠ Cat(⇢1, ⇢2)

=a1,n

a1,n + a2,n

a1,n := a1 +n�1X

m=1

1{zm = 1}, a2,n = a2 +n�1X

m=1

1{zm = 2}

• Choose any ball with equal probability • Replace and add ball of same color

lim

n!1

# orange

# total

= ⇢orange

d= Beta(a

orange

, agreen

)

5

• Integrate out the frequencies !

!

!

!

• Pólya urn

Marginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2)

p(zn = 1|z1, . . . , zn�1), zn

iid⇠ Cat(⇢1, ⇢2)

=a1,n

a1,n + a2,n

a1,n := a1 +n�1X

m=1

1{zm = 1}, a2,n = a2 +n�1X

m=1

1{zm = 2}

• Choose any ball with equal probability • Replace and add ball of same color

lim

n!1

# orange

# total

= ⇢orange

d= Beta(a

orange

, agreen

)

5

• Integrate out the frequencies !

!

!

!

• Pólya urn

Marginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2)

p(zn = 1|z1, . . . , zn�1), zn

iid⇠ Cat(⇢1, ⇢2)

=a1,n

a1,n + a2,n

a1,n := a1 +n�1X

m=1

1{zm = 1}, a2,n = a2 +n�1X

m=1

1{zm = 2}

• Choose any ball with equal probability • Replace and add ball of same color

lim

n!1

# orange

# total

= ⇢orange

d= Beta(a

orange

, agreen

)

5

• Integrate out the frequencies !

!

!

!

• Pólya urn

Marginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2)

p(zn = 1|z1, . . . , zn�1), zn

iid⇠ Cat(⇢1, ⇢2)

=a1,n

a1,n + a2,n

a1,n := a1 +n�1X

m=1

1{zm = 1}, a2,n = a2 +n�1X

m=1

1{zm = 2}

• Choose any ball with equal probability • Replace and add ball of same color

lim

n!1

# orange

# total

= ⇢orange

d= Beta(a

orange

, agreen

)

5

• Integrate out the frequencies !

!

!

!

• Pólya urn

Marginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2)

p(zn = 1|z1, . . . , zn�1), zn

iid⇠ Cat(⇢1, ⇢2)

=a1,n

a1,n + a2,n

a1,n := a1 +n�1X

m=1

1{zm = 1}, a2,n = a2 +n�1X

m=1

1{zm = 2}

• Choose any ball with equal probability • Replace and add ball of same color

lim

n!1

# orange

# total

= ⇢orange

d= Beta(a

orange

, agreen

)

5

• Integrate out the frequencies !

!

!

!

• Pólya urn

Marginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2)

p(zn = 1|z1, . . . , zn�1), zn

iid⇠ Cat(⇢1, ⇢2)

=a1,n

a1,n + a2,n

a1,n := a1 +n�1X

m=1

1{zm = 1}, a2,n = a2 +n�1X

m=1

1{zm = 2}

• Choose any ball with equal probability • Replace and add ball of same color

lim

n!1

# orange

# total

= ⇢orange

d= Beta(a

orange

, agreen

)

5

• Integrate out the frequencies !

!

!

!

• Pólya urn

Marginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2)

p(zn = 1|z1, . . . , zn�1), zn

iid⇠ Cat(⇢1, ⇢2)

=a1,n

a1,n + a2,n

a1,n := a1 +n�1X

m=1

1{zm = 1}, a2,n = a2 +n�1X

m=1

1{zm = 2}

• Choose any ball with equal probability • Replace and add ball of same color

lim

n!1

# orange

# total

= ⇢orange

d= Beta(a

orange

, agreen

)

5

• Integrate out the frequencies !

!

!

!

• Pólya urn

Marginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2)

p(zn = 1|z1, . . . , zn�1), zn

iid⇠ Cat(⇢1, ⇢2)

=a1,n

a1,n + a2,n

a1,n := a1 +n�1X

m=1

1{zm = 1}, a2,n = a2 +n�1X

m=1

1{zm = 2}

• Choose any ball with equal probability • Replace and add ball of same color

lim

n!1

# orange

# total

= ⇢orange

d= Beta(a

orange

, agreen

)

5

• Integrate out the frequencies !

!

!

!

• Pólya urn

Marginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2)

p(zn = 1|z1, . . . , zn�1), zn

iid⇠ Cat(⇢1, ⇢2)

=a1,n

a1,n + a2,n

a1,n := a1 +n�1X

m=1

1{zm = 1}, a2,n = a2 +n�1X

m=1

1{zm = 2}

• Choose any ball with equal probability • Replace and add ball of same color

lim

n!1

# orange

# total

= ⇢orange

d= Beta(a

orange

, agreen

)

5

• Integrate out the frequencies !

!

!

!

• Pólya urn

Marginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2)

p(zn = 1|z1, . . . , zn�1), zn

iid⇠ Cat(⇢1, ⇢2)

=a1,n

a1,n + a2,n

a1,n := a1 +n�1X

m=1

1{zm = 1}, a2,n = a2 +n�1X

m=1

1{zm = 2}

• Choose any ball with equal probability • Replace and add ball of same color

lim

n!1

# orange

# total

= ⇢orange

d= Beta(a

orange

, agreen

)

5

• Integrate out the frequencies !

!

!

!

• Pólya urn

Marginal cluster assignments

1 2

⇢1 ⇠ Beta(a1, a2)

p(zn = 1|z1, . . . , zn�1), zn

iid⇠ Cat(⇢1, ⇢2)

=a1,n

a1,n + a2,n

a1,n := a1 +n�1X

m=1

1{zm = 1}, a2,n = a2 +n�1X

m=1

1{zm = 2}

• Choose any ball with prob proportional to its mass • Replace and add ball of same color

lim

n!1

# orange

# total

= ⇢orange

d= Beta(a

orange

, agreen

)

5

• Integrate out the frequencies !

!

!

!

• multivariate Pólya urn

Marginal cluster assignments

• Choose any ball with prob proportional to its mass • Replace and add ball of same color

1 2 3 4

6

• Integrate out the frequencies !

!

!

!

• multivariate Pólya urn

Marginal cluster assignments

• Choose any ball with prob proportional to its mass • Replace and add ball of same color

1 2 3 4

⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)

6

• Integrate out the frequencies !

!

!

!

• multivariate Pólya urn

Marginal cluster assignments

• Choose any ball with prob proportional to its mass • Replace and add ball of same color

1 2 3 4

⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)

p(zn = k|z1, . . . , zn�1) =ak,nPKj=1 aj,n

6

• Integrate out the frequencies !

!

!

!

• multivariate Pólya urn

Marginal cluster assignments

• Choose any ball with prob proportional to its mass • Replace and add ball of same color

1 2 3 4

⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)

p(zn = k|z1, . . . , zn�1) =ak,nPKj=1 aj,n

ak,n := ak +n�1X

m=1

1{zm = k}

6

• Integrate out the frequencies !

!

!

!

• multivariate Pólya urn

Marginal cluster assignments

• Choose any ball with prob proportional to its mass • Replace and add ball of same color

1 2 3 4

⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)

p(zn = k|z1, . . . , zn�1) =ak,nPKj=1 aj,n

ak,n := ak +n�1X

m=1

1{zm = k}

6

• Integrate out the frequencies !

!

!

!

• multivariate Pólya urn

Marginal cluster assignments

• Choose any ball with prob proportional to its mass • Replace and add ball of same color

1 2 3 4

⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)

p(zn = k|z1, . . . , zn�1) =ak,nPKj=1 aj,n

ak,n := ak +n�1X

m=1

1{zm = k}

6

• Integrate out the frequencies !

!

!

!

• multivariate Pólya urn

Marginal cluster assignments

• Choose any ball with prob proportional to its mass • Replace and add ball of same color

1 2 3 4

⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)

p(zn = k|z1, . . . , zn�1) =ak,nPKj=1 aj,n

ak,n := ak +n�1X

m=1

1{zm = k}

6

• Integrate out the frequencies !

!

!

!

• multivariate Pólya urn

Marginal cluster assignments

• Choose any ball with prob proportional to its mass • Replace and add ball of same color

1 2 3 4

⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)

p(zn = k|z1, . . . , zn�1) =ak,nPKj=1 aj,n

ak,n := ak +n�1X

m=1

1{zm = k}

6

• Integrate out the frequencies !

!

!

!

• multivariate Pólya urn

Marginal cluster assignments

• Choose any ball with prob proportional to its mass • Replace and add ball of same color

1 2 3 4

⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)

p(zn = k|z1, . . . , zn�1) =ak,nPKj=1 aj,n

ak,n := ak +n�1X

m=1

1{zm = k}

lim

n!1

(# orange, # green, # red, # yellow)

# total

6

• Integrate out the frequencies !

!

!

!

• multivariate Pólya urn

Marginal cluster assignments

• Choose any ball with prob proportional to its mass • Replace and add ball of same color

1 2 3 4

⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)

p(zn = k|z1, . . . , zn�1) =ak,nPKj=1 aj,n

ak,n := ak +n�1X

m=1

1{zm = k}

lim

n!1

(# orange, # green, # red, # yellow)

# total

! (⇢orange

, ⇢green

, ⇢red

, ⇢yellow

)

6

• Integrate out the frequencies !

!

!

!

• multivariate Pólya urn

Marginal cluster assignments

• Choose any ball with prob proportional to its mass • Replace and add ball of same color

1 2 3 4

⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)

p(zn = k|z1, . . . , zn�1) =ak,nPKj=1 aj,n

ak,n := ak +n�1X

m=1

1{zm = k}

lim

n!1

(# orange, # green, # red, # yellow)

# total

! (⇢orange

, ⇢green

, ⇢red

, ⇢yellow

)d= Dirichlet(a

orange

, agreen

, ared

, ayellow

)6

Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn

[Blackwell, MacQueen 1973; Hoppe 1984]7

Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn

[Blackwell, MacQueen 1973; Hoppe 1984]7

Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn

[Blackwell, MacQueen 1973; Hoppe 1984]7

Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn

• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color

[Blackwell, MacQueen 1973; Hoppe 1984]7

Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn

• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color

[Blackwell, MacQueen 1973; Hoppe 1984]7

Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn

• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color

[Blackwell, MacQueen 1973; Hoppe 1984]7

Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn

• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color

Step 0

[Blackwell, MacQueen 1973; Hoppe 1984]7

Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn

• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color

Step 0 Step 1

[Blackwell, MacQueen 1973; Hoppe 1984]7

Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn

• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color

Step 0 Step 1 Step 2

[Blackwell, MacQueen 1973; Hoppe 1984]7

Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn

• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color

Step 0 Step 1 Step 2 Step 3

[Blackwell, MacQueen 1973; Hoppe 1984]7

Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn

• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color

Step 0 Step 1 Step 2 Step 3 Step 4

[Blackwell, MacQueen 1973; Hoppe 1984]7

Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn

• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color

Step 0 Step 1 Step 2 Step 3 Step 4

(#orange,#other) = PolyaUrn(1,↵)(#green,#other) = PolyaUrn(1,↵)

(#red,#other) = PolyaUrn(1,↵)• not orange:• not orange, green:

7

Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn

• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color

Step 0 Step 1 Step 2 Step 3 Step 4

(#orange,#other) = PolyaUrn(1,↵)(#green,#other) = PolyaUrn(1,↵)

(#red,#other) = PolyaUrn(1,↵)• not orange:• not orange, green:

7

Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn

• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color

Step 0 Step 1 Step 2 Step 3 Step 4

(#orange,#other) = PolyaUrn(1,↵)(#green,#other) = PolyaUrn(1,↵)

(#red,#other) = PolyaUrn(1,↵)• not orange:• not orange, green:

7

Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn

• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color

Step 0 Step 1 Step 2 Step 3 Step 4

(#orange,#other) = PolyaUrn(1,↵)(#green,#other) = PolyaUrn(1,↵)

(#red,#other) = PolyaUrn(1,↵)• not orange:• not orange, green:

Vkiid⇠ Beta(1,↵)

7

Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn

• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color

Step 0 Step 1 Step 2 Step 3 Step 4

(#orange,#other) = PolyaUrn(1,↵)(#green,#other) = PolyaUrn(1,↵)

(#red,#other) = PolyaUrn(1,↵)• not orange:• not orange, green:

Vkiid⇠ Beta(1,↵)

⇢1 = V1

7

Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn

• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color

Step 0 Step 1 Step 2 Step 3 Step 4

(#orange,#other) = PolyaUrn(1,↵)(#green,#other) = PolyaUrn(1,↵)

(#red,#other) = PolyaUrn(1,↵)• not orange:• not orange, green:

Vkiid⇠ Beta(1,↵)

⇢1 = V1

⇢2 = (1� V1)V2

7

Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn

• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color

Step 0 Step 1 Step 2 Step 3 Step 4

(#orange,#other) = PolyaUrn(1,↵)(#green,#other) = PolyaUrn(1,↵)

(#red,#other) = PolyaUrn(1,↵)• not orange:• not orange, green:

Vkiid⇠ Beta(1,↵)

⇢1 = V1

⇢2 = (1� V1)V2

⇢3 = [Y2

k=1(1� Vk)]V3

7

Chinese restaurant process

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1

�1

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1

�1

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1

2�1

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1 3

2�1

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1 3

2�1

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1 3

2�1 �2

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1 3

2�1 �2

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1 3

2 4�1 �2

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1 3

2 4�1 �2

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1 3

2 4�1 �2 �3

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1 3

2 4�1 �2 �3

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1 3

2 45�1 �2 �3

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1 3

2 45

6

�1 �2 �3

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1 3

2 45

67

�1 �2 �3

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}[Aldous 1983]8

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

• Same thing we just did • Each customer walks into the restaurant

• Sits at existing table with prob proportional to # people there

• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !

• Partition of [8]: set of mutually exclusive & exhaustive sets of

z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}

[8] = {1, . . . , 8}8

• Probability of this seating: !

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

KN nk

p(v1, v2, v3)

9

↵

↵· 1

↵+ 1· ↵

↵+ 2· ↵

↵+ 3· 1

↵+ 4· 2

↵+ 5· 2

↵+ 6· 3

↵+ 7

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

KN nk

p(v1, v2, v3)

• Probability of this seating: !

9

↵

↵· 1

↵+ 1· ↵

↵+ 2· ↵

↵+ 3· 1

↵+ 4· 2

↵+ 5· 2

↵+ 6· 3

↵+ 7

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

KN nk

p(v1, v2, v3)

• Probability of this seating: !

9

↵

↵· 1

↵+ 1· ↵

↵+ 2· ↵

↵+ 3· 1

↵+ 4· 2

↵+ 5· 2

↵+ 6· 3

↵+ 7

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

KN nk

p(v1, v2, v3)

• Probability of this seating: !

9

↵

↵· 1

↵+ 1· ↵

↵+ 2· ↵

↵+ 3· 1

↵+ 4· 2

↵+ 5· 2

↵+ 6· 3

↵+ 7

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

KN nk

p(v1, v2, v3)

• Probability of this seating: !

9

↵

↵· 1

↵+ 1· ↵

↵+ 2· ↵

↵+ 3· 1

↵+ 4· 2

↵+ 5· 2

↵+ 6· 3

↵+ 7

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

• Probability of this seating: !

• Probability of N customers ( tables, at table k):

!

• Prob doesn’t depend on customer order: exchangeable • Gibbs sampling review: target distribution

• Start: • t th step:

KN nk

p(v1, v2, v3)

9

↵

↵· 1

↵+ 1· ↵

↵+ 2· ↵

↵+ 3· 1

↵+ 4· 2

↵+ 5· 2

↵+ 6· 3

↵+ 7

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

• Probability of this seating: !

• Probability of N customers ( tables, at table k):

!

• Prob doesn’t depend on customer order: exchangeable • Gibbs sampling review: target distribution

• Start: • t th step:

KN nk

p(v1, v2, v3)

9

↵

↵· 1

↵+ 1· ↵

↵+ 2· ↵

↵+ 3· 1

↵+ 4· 2

↵+ 5· 2

↵+ 6· 3

↵+ 7

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

• Probability of this seating: !

• Probability of N customers ( tables, at table k):

!

• Prob doesn’t depend on customer order: exchangeable • Gibbs sampling review: target distribution

• Start: • t th step:

KN nk

p(v1, v2, v3)

9

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

• Probability of this seating: !

• Probability of N customers ( tables, at table k):

!

• Prob doesn’t depend on customer order: exchangeable • Gibbs sampling review: target distribution

• Start: • t th step:

KN nk

p(v1, v2, v3)

↵

↵· 1

↵+ 1· ↵

↵+ 2· ↵

↵+ 3· 1

↵+ 4· 2

↵+ 5· 2

↵+ 6· 3

↵+ 7

9

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

• Probability of this seating: !

• Probability of N customers ( tables, at table k):

!

• Prob doesn’t depend on customer order: exchangeable • Gibbs sampling review: target distribution

• Start: • t th step:

KN nk

p(v1, v2, v3)

↵

↵· 1

↵+ 1· ↵

↵+ 2· ↵

↵+ 3· 1

↵+ 4· 2

↵+ 5· 2

↵+ 6· 3

↵+ 7

9

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

• Probability of this seating: !

• Probability of N customers ( tables, at table k):

!

• Prob doesn’t depend on customer order: exchangeable • Gibbs sampling review: target distribution

• Start: • t th step:

KN nk

p(v1, v2, v3)

↵KNQKN

k=1(nk � 1)!

↵ · · · (↵+N � 1)= P(⇧N = ⇡N )

↵

↵· 1

↵+ 1· ↵

↵+ 2· ↵

↵+ 3· 1

↵+ 4· 2

↵+ 5· 2

↵+ 6· 3

↵+ 7

9

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

• Probability of this seating: !

• Probability of N customers ( tables, at table k):

!

• Prob doesn’t depend on customer order: exchangeable • Gibbs sampling review: target distribution

• Start: • t th step:

KN nk

p(v1, v2, v3)

↵KNQKN

k=1(nk � 1)!

↵ · · · (↵+N � 1)= P(⇧N = ⇡N )

↵

↵· 1

↵+ 1· ↵

↵+ 2· ↵

↵+ 3· 1

↵+ 4· 2

↵+ 5· 2

↵+ 6· 3

↵+ 7

9

Chinese restaurant process1 3

2 45

67

8

�1 �2 �3

• Probability of this seating: !

• Probability of N customers ( tables, at table k):

!

• Prob doesn’t depend on customer order: exchangeable • Gibbs sampling review: target distribution

• Start: • t th step:

KN nk

p(v1, v2, v3)

↵KNQKN

k=1(nk � 1)!

↵ · · · (↵+N � 1)= P(⇧N = ⇡N )

↵

↵· 1

↵+ 1· ↵

↵+ 2· ↵

↵+ 3· 1

↵+ 4· 2

↵+ 5· 2

↵+ 6· 3

↵+ 7

9

References (Part II)

10

DJ Aldous. Exchangeability and related topics. Springer, 1983.

D Blackwell and JB MacQueen. Ferguson distributions via Pólya urn schemes. The Annals of Statistics, 1973.

S Engen. A note on the geometric series as a species frequency model. Biometrika, 1975.

W Ewens. Population genetics theory -- the past and the future. Mathematical and Statistical Developments of Evolutionary Theory, 1987.

FM Hoppe. Pólya-like urns and the Ewens' sampling formula. Journal of Mathematical Biology, 1984.

H Ishwaran and LF James. Gibbs sampling methods for stick-breaking priors. Journal of the American Statistical Association, 2001.

JW McCloskey. A model for the distribution of individuals by species in an environment. Ph.D. thesis, Michigan State University, 1965.

GP Patil and C Taillie. Diversity as a concept and its implications for random communities. Bulletin of the International Statistical Institute, 1977.

J Sethuraman. A constructive definition of Dirichlet priors. Statistica Sinica, 1994.