Smoothing

• Problem with MLE

Beijing Shanghai … MengCheng CLASS = ‘china’

1 1 … 0 +

1 0 … 0 +

0 1 … 0 +

0 0 … 0 -

A small city in the Anhui

province

A small city in the Anhui

province

Beijing Shanghai … MengCheng CLASS = ‘china’

1 1 … 1 ?

€

p(+ | x)∝ p(+) ⋅ p(Beijing =1 | +) ⋅ p(Shanghai =1 | +)

⋅...⋅ p(MengCheng =1 | +)

€

=0

Common reasons: data sparseness, rare features, …Common reasons: data sparseness, rare features, …

Smoothing

• Add-one smoothing (Laplace smoothing)– Essentially, every possible value for a variable

have non-zero count in any classBeijing Shanghai … MengCheng CLASS = ‘china’

1 1 … 0 +

€

p(MengCheng =1 | +) =count(MengCheng =1 | +) +1

count(+) + B

B = # of possible values for the variable in question.B = # of possible values for the variable in question.

€

p(MengCheng = 0 | +) =count(MengCheng = 0 | +) +1

count(+) + B

€

p(MengCheng = 0 | +) + p(MengCheng =1 | +) =1.0

€

B = 2

Bernoulli

• TrainingChinese Beijing Chinese +

Chinese Chinese Shanghai +

Chinese Macao +

Tokyo Japan Chinese -

Chinese Beijing Shanghai Macao Tokyo Japan CLASS

1 1 0 0 0 0 +

1 0 1 0 0 0 +

1 0 0 1 0 0 +

1 0 0 0 1 1 -

Bernoulli

• Training

Chinese Beijing Shanghai Macao Tokyo Japan CLASS

1 1 0 0 0 0 +

1 0 1 0 0 0 +

1 0 0 1 0 0 +

1 0 0 0 1 1 -

€

p(Chinese | +) = 3/3

€

p(Chinese | +) = (3 +1) /(3+ 2)

Chinese Beijing Shanghai Macao Tokyo Japan CLASS

3 1 1 1 0 0 + * 3

1 0 0 0 1 1 - * 1

B = # of possible values for the variable in question = 2B = # of possible values for the variable in question = 2

Bernoulli

• TestingChinese Chinese Chinese Tokyo Japan ?

€

p(+ | x)∝ p(+) ⋅4

5⋅(1−

2

5) ⋅(1−

2

5) ⋅(1−

2

5) ⋅

1

5⋅1

5≈ 0.005

Chinese Beijing Shanghai Macao Tokyo Japan CLASS

3 1 1 1 0 0 + * 3

1 0 0 0 1 1 - * 1

€

p(− | x)∝ p(−) ⋅2

3⋅(1−

1

3) ⋅(1−

1

3) ⋅(1−

1

3) ⋅

2

3⋅

2

3≈ 0.022

Chinese Beijing Shanghai Macao Tokyo Japan CLASS

1 0 0 0 1 1 ?

Multinomial

• TrainingChinese Beijing Chinese +

Chinese Chinese Shanghai +

Chinese Macao +

Tokyo Japan Chinese -

W1 W2 W3 Wi CLASS

Chinese Beijing Chinese +

Chinese Chinese Shanghai +

Chinese Macao +

Tokyo Japan Chinese -

Multinomial

• TrainingW1 W2 W3 CLASS

Chinese Beijing Chinese +

Chinese Chinese Shanghai +

Chinese Macao +

Tokyo Japan Chinese -

Wi CLASS

Chinese +

Beijing +

Chinese +

Chinese +

Chinese +

Shanghai +

Chinese +

Macao +

Tokyo -

Japan -

Chinese -€

p(W i = Chinese | +) = 5 /8

€

p(W i = Chinese | +) = (5 +1) /(8 + 6)

B = # of possible values for the variable in question = 6B = # of possible values for the variable in question = 6

€

assume p(W i | +) = p(W j | +)

Multinomial

• TestingChinese Chinese Chinese Tokyo Japan ?

€

p(+ | x)∝ p(+) ⋅6

14⋅

6

14⋅

6

14⋅

1

14⋅

1

14≈ 0.003

€

p(− | x)∝ p(−) ⋅2

9⋅2

9⋅2

9⋅2

9⋅

2

9≈ 0.0001

W1 W2 W3 W4 W5 CLASS

Chinese Chinese Chinese Tokyo Japan ?

W CLASS

Chinese +

Beijing +

Chinese +

Chinese +

Chinese +

Shanghai +

Chinese +

Macao +

Tokyo -

Japan -

Chinese -