smoothing problem with mle beijingshanghai…mengchengclass = ‘china’ 11…0+ 10…0+ 01…0+...
Post on 31-Mar-2015
219 Views
Preview:
TRANSCRIPT
Smoothing
• Problem with MLE
Beijing Shanghai … MengCheng CLASS = ‘china’
1 1 … 0 +
1 0 … 0 +
0 1 … 0 +
0 0 … 0 -
A small city in the Anhui
province
A small city in the Anhui
province
Beijing Shanghai … MengCheng CLASS = ‘china’
1 1 … 1 ?
€
p(+ | x)∝ p(+) ⋅ p(Beijing =1 | +) ⋅ p(Shanghai =1 | +)
⋅...⋅ p(MengCheng =1 | +)
€
=0
Common reasons: data sparseness, rare features, …Common reasons: data sparseness, rare features, …
Smoothing
• Add-one smoothing (Laplace smoothing)– Essentially, every possible value for a variable
have non-zero count in any classBeijing Shanghai … MengCheng CLASS = ‘china’
1 1 … 0 +
€
p(MengCheng =1 | +) =count(MengCheng =1 | +) +1
count(+) + B
B = # of possible values for the variable in question.B = # of possible values for the variable in question.
€
p(MengCheng = 0 | +) =count(MengCheng = 0 | +) +1
count(+) + B
€
p(MengCheng = 0 | +) + p(MengCheng =1 | +) =1.0
€
B = 2
Bernoulli
• TrainingChinese Beijing Chinese +
Chinese Chinese Shanghai +
Chinese Macao +
Tokyo Japan Chinese -
Chinese Beijing Shanghai Macao Tokyo Japan CLASS
1 1 0 0 0 0 +
1 0 1 0 0 0 +
1 0 0 1 0 0 +
1 0 0 0 1 1 -
Bernoulli
• Training
Chinese Beijing Shanghai Macao Tokyo Japan CLASS
1 1 0 0 0 0 +
1 0 1 0 0 0 +
1 0 0 1 0 0 +
1 0 0 0 1 1 -
€
p(Chinese | +) = 3/3
€
p(Chinese | +) = (3 +1) /(3+ 2)
Chinese Beijing Shanghai Macao Tokyo Japan CLASS
3 1 1 1 0 0 + * 3
1 0 0 0 1 1 - * 1
B = # of possible values for the variable in question = 2B = # of possible values for the variable in question = 2
Bernoulli
• TestingChinese Chinese Chinese Tokyo Japan ?
€
p(+ | x)∝ p(+) ⋅4
5⋅(1−
2
5) ⋅(1−
2
5) ⋅(1−
2
5) ⋅
1
5⋅1
5≈ 0.005
Chinese Beijing Shanghai Macao Tokyo Japan CLASS
3 1 1 1 0 0 + * 3
1 0 0 0 1 1 - * 1
€
p(− | x)∝ p(−) ⋅2
3⋅(1−
1
3) ⋅(1−
1
3) ⋅(1−
1
3) ⋅
2
3⋅
2
3≈ 0.022
Chinese Beijing Shanghai Macao Tokyo Japan CLASS
1 0 0 0 1 1 ?
Multinomial
• TrainingChinese Beijing Chinese +
Chinese Chinese Shanghai +
Chinese Macao +
Tokyo Japan Chinese -
W1 W2 W3 Wi CLASS
Chinese Beijing Chinese +
Chinese Chinese Shanghai +
Chinese Macao +
Tokyo Japan Chinese -
Multinomial
• TrainingW1 W2 W3 CLASS
Chinese Beijing Chinese +
Chinese Chinese Shanghai +
Chinese Macao +
Tokyo Japan Chinese -
Wi CLASS
Chinese +
Beijing +
Chinese +
Chinese +
Chinese +
Shanghai +
Chinese +
Macao +
Tokyo -
Japan -
Chinese -€
p(W i = Chinese | +) = 5 /8
€
p(W i = Chinese | +) = (5 +1) /(8 + 6)
B = # of possible values for the variable in question = 6B = # of possible values for the variable in question = 6
€
assume p(W i | +) = p(W j | +)
Multinomial
• TestingChinese Chinese Chinese Tokyo Japan ?
€
p(+ | x)∝ p(+) ⋅6
14⋅
6
14⋅
6
14⋅
1
14⋅
1
14≈ 0.003
€
p(− | x)∝ p(−) ⋅2
9⋅2
9⋅2
9⋅2
9⋅
2
9≈ 0.0001
W1 W2 W3 W4 W5 CLASS
Chinese Chinese Chinese Tokyo Japan ?
W CLASS
Chinese +
Beijing +
Chinese +
Chinese +
Chinese +
Shanghai +
Chinese +
Macao +
Tokyo -
Japan -
Chinese -
top related