lecture 11: viterbi and forward algorithms · 2019-04-14 · this lecture vtwo important algorithms...
TRANSCRIPT
Lecture 11: Viterbi and Forward
Algorithms
Kai-Wei ChangCS @ University of Virginia
Couse webpage: http://kwchang.net/teaching/NLP16
1CS6501 Natural Language Processing
Quiz 1
vMax: 24; Mean: 18.1; Median: 18; SD: 3.36
CS6501 Natural Language Processing 2
0
5
10
15
20
25
30
[0-5] [6-10] [11-15] [16-20] [21-25]
Quiz1
This lecture
vTwo important algorithms for inferencevForward algorithmvViterbi algorithm
3CS6501 Natural Language Processing
CS6501 Natural Language Processing ‹#›
Three basic problems for HMMs
vLikelihood of the input:vForward algorithm
vDecoding (tagging) the input:vViterbi algorithm
vEstimation (learning):vFind the best model parameters
v Case 1: supervised – tags are annotatedvMaximum likelihood estimation (MLE)
v Case 2: unsupervised -- only unannotated textvForward-backward algorithm
CS6501 Natural Language Processing 5
Howlikelythesentence”Ilovecat”occurs
POStagsof”Ilovecat”occurs
Howtolearnthemodel?
Likelihood of the input
vHow likely a sentence “I love cat” occurvCompute 𝑃(𝒘 ∣ 𝝀) for the input 𝒘 and
HMM 𝜆
vRemember, we model 𝑃 𝒕,𝒘 ∣ 𝝀
v𝑃 𝒘 𝝀 = ∑ 𝑃 𝒕,𝒘 𝝀 𝒕
CS6501 Natural Language Processing 6
Marginalprobability:Sumoverallpossibletagsequences
Likelihood of the input
v How likely a sentence “I love cat” occurv 𝑃 𝒘 𝝀 = ∑ 𝑃 𝒕,𝒘 𝝀 𝒕
= ∑ Π./01 𝑃 𝑤. 𝑡. 𝑃 𝑡. ∣ 𝑡.40 𝒕 v Assume we have 2 tags N, Vv 𝑃 “𝐼𝑙𝑜𝑣𝑒𝑐𝑎𝑡” 𝝀
= 𝑃 “𝐼𝑙𝑜𝑣𝑒𝑐𝑎𝑡”, ”𝑁𝑁𝑁” 𝝀 + 𝑃 “𝐼𝑙𝑜𝑣𝑒𝑐𝑎𝑡”, ”𝑁𝑁𝑉” 𝝀+𝑃 “𝐼𝑙𝑜𝑣𝑒𝑐𝑎𝑡”, ”𝑁𝑉𝑁” 𝝀 + 𝑃 “𝐼𝑙𝑜𝑣𝑒𝑐𝑎𝑡”, ”𝑁𝑉𝑉” 𝝀+𝑃 “𝐼𝑙𝑜𝑣𝑒𝑐𝑎𝑡”, ”𝑉𝑁𝑁” 𝝀 + 𝑃 “𝐼𝑙𝑜𝑣𝑒𝑐𝑎𝑡”, ”𝑉𝑁𝑉” 𝝀
+𝑃 “𝐼𝑙𝑜𝑣𝑒𝑐𝑎𝑡”, ”𝑉𝑉𝑁” 𝝀 + 𝑃 “𝐼𝑙𝑜𝑣𝑒𝑐𝑎𝑡”, ”𝑉𝑉𝑉” 𝝀
v Now, let’s write down 𝑃(“𝐼𝑙𝑜𝑣𝑒𝑐𝑎𝑡” ∣ 𝝀) with 45 tags…
CS6501 Natural Language Processing 7
Trellis diagram
vGoal: P(𝐰 ∣ 𝝀) = ∑ Π./01 𝑃 𝑤. 𝑡. 𝑃 𝑡. ∣ 𝑡.40 𝒕
CS6501 Natural Language Processing 8
𝑃(𝑡C = 2|𝑡0 = 1)
𝑖 = 1𝑖 = 2𝑖 = 3𝑖 = 4
𝑃(𝑡J = 1|𝑡C = 1) 𝑃(𝑤J|𝑡J = 1)
⋯
𝝀 istheparametersetofHMM.Let’signore itinsomeslidesforsimplicity’ssake
Trellis diagram
v P(“I eat a fish”, NVVA )
CS6501 Natural Language Processing 9
𝑖 = 1𝑖 = 2𝑖 = 3𝑖 = 4
N
V
A
N
V
A
N
V
A
N
V
A
𝑃(𝐼|𝑁)
𝑃(𝑒𝑎𝑡|𝑉) 𝑃(𝑎|𝑉)
𝑃(𝑓𝑖𝑠ℎ|A)
𝑃(𝑉|𝑁) 𝑃(𝑉|𝑉)
𝑃(𝑁| < 𝑆 >)
𝑃(𝐴|𝑉)
⋯
Trellis diagram
v ∑ Π./01 𝑃 𝑤. 𝑡. 𝑃 𝑡. ∣ 𝑡.40 𝒕 : sum over all paths
CS6501 Natural Language Processing 10
𝑖 = 1𝑖 = 2𝑖 = 3𝑖 = 4
N
V
A
N
V
A
N
V
A
N
V
A
⋯
Dynamic programming
vRecursively decompose a problem into smaller sub-problems
vSimilar to mathematical inductionvBase step: initial values for 𝑖 = 1v Inductive step: assume we know the values for 𝑖 = 𝑘, let’s compute 𝑖 = 𝑘 + 1
CS6501 Natural Language Processing 11
Forward algorithm
v Inductive step: from 𝑖 = 𝑘 toi =k+1v 𝒕Y :tagsequencewithlength𝑘,𝒘Y = 𝑤0, 𝑤C …𝑤Yv ∑ 𝑃(𝒕Y,𝒘𝒌)𝒕g = ∑ ∑ 𝑃(𝒕Y40, 𝒘𝒌, 𝑡Y = 𝑞)𝒕gij k
CS6501 Natural Language Processing 12𝑖 = 1𝑖 = 2𝑖 = 3𝑖 = 4
N
V
A
N
V
A
N
V
A
N
V
A
⋯
tagsequences tag@i=k
P(𝒘𝒌, 𝑡Y = 𝑞)
tagsequences
Forward algorithm
v Inductive step: from 𝑖 = 𝑘 toi =k+1v ∑ 𝑃(𝒕Y,𝒘)𝒕g = ∑ P(𝒘𝒌, 𝑡Y = 𝑞) k
v P(𝒘𝒌, 𝑡Y = 𝑞)= ∑ 𝑃(kl 𝒘𝒌, 𝑡Y40 = 𝑞m, 𝑡Y = 𝑞)
=∑ 𝑃(km 𝒘𝒌40, 𝑡Y40 = 𝑞m)𝑃(𝑡Y = 𝑞 ∣ 𝑡Y40 = 𝑞′)𝑃(𝑤Y ∣ 𝑡Y = 𝑞)
CS6501 Natural Language Processing13𝑖 = 𝑘 − 1𝑖 = 𝑘
N
V
A
N
V
A
N
V
A
N
V
A
⋯
Forward algorithm
v Inductive step: from 𝑖 = 𝑘 toi =k+1v ∑ 𝑃(𝒕Y,𝒘)𝒕g = ∑ P(𝒘𝒌, 𝑡Y = 𝑞) k
v P(𝒘𝒌, 𝑡Y = 𝑞)= ∑ 𝑃(kl 𝒘𝒌, 𝑡Y40 = 𝑞m, 𝑡Y = 𝑞)
=∑ 𝑃(km 𝒘𝒌40, 𝑡Y40 = 𝑞m)𝑃(𝑡Y = 𝑞 ∣ 𝑡Y40 = 𝑞′)𝑃(𝑤Y ∣ 𝑡Y = 𝑞)
CS6501 Natural Language Processing14𝑖 = 𝑘 − 1𝑖 = 𝑘
N
V
A
N
V
A
N
V
A
N
V
A
⋯
Let’scallit𝛼Y(𝑞)
Thisis𝛼Y40(𝑞′)
Forward algorithm
v Inductive step: from 𝑖 = 𝑘 toi =k+1v 𝛼Y(𝑞)=∑ 𝛼Y40(𝑞′)km 𝑃(𝑡Y = 𝑞 ∣ 𝑡Y40 = 𝑞′)𝑃(𝑤Y ∣ 𝑡Y = 𝑞)
CS6501 Natural Language Processing15𝑖 = 𝑘 − 1𝑖 = 𝑘
N
V
A
N
V
A
N
V
A
N
V
A
⋯
Forward algorithm
v Inductive step: from 𝑖 = 𝑘 toi =k+1v 𝛼Y(𝑞)= ∑ 𝛼Y40(𝑞′)km 𝑃(𝑡Y = 𝑞 ∣ 𝑡Y40 = 𝑞′)𝑃(𝑤Y ∣ 𝑡Y = 𝑞)
=𝑃(𝑤Y ∣ 𝑡Y = 𝑞)∑ 𝛼Y40(𝑞′)km 𝑃(𝑡Y = 𝑞 ∣ 𝑡Y40 = 𝑞′)
CS6501 Natural Language Processing16𝑖 = 𝑘 − 1𝑖 = 𝑘
N
V
A
N
V
A
N
V
A
N
V
A
⋯
Forward algorithm
v Base step: i=0 v 𝛼0 𝑞 = 𝑃 𝑤0 𝑡0 = 𝑞 𝑃(𝑡0 = 𝑞 ∣ 𝑡q)
CS6501 Natural Language Processing17𝑖 = 𝑘 − 1𝑖 = 𝑘
N
V
A
N
V
A
N
V
A
N
V
A
⋯
initialprobability𝑝(𝑡0 = 𝑞)
Implementation using an array
vUse a 𝑛×𝑇 table to keep 𝛼Y(𝑞)
CS6501 Natural Language Processing 18
FromJuliaHockenmaier, IntrotoNLP
Implementation using an array
CS6501 Natural Language Processing 19
Initial:Trellis[1][q]= 𝑃 𝑤0 𝑡0 = 𝑞 𝑃(𝑡0 = 𝑞 ∣ 𝑡q)
Implementation using an array
CS6501 Natural Language Processing 20
Induction:𝛼Y(𝑞)=𝑃(𝑤Y ∣ 𝑡Y = 𝑞)∑ 𝛼Y40(𝑞′)km 𝑃(𝑡Y = 𝑞 ∣ 𝑡Y40 = 𝑞′)
ii
The forward algorithm (Pseudo Code)
CS6501 Natural Language Processing 21
.fwd=0
Jason’s ice cream
vP(”1,2,1”)?
CS6501 Natural Language Processing 22
p(…|C) p(…|H) p(…|START)p(1|…) 0.5 0.1p(2|…) 0.4 0.2p(3|…) 0.1 0.7p(C|…) 0.8 0.2 0.5p(H|…) 0.2 0.8 0.5
Scroll to the bottom to see a graph of what states and transitions the model thinks are likely on each day. Those likely states and transitions can be used to reestimate the red probabilities (this is the "forward-backward" or Baum-Welch algorithm), incr
#cones
C
H
C
H
C
H
0.5
0.5
0.8
0.80.8
0.8
0.2
0.2
0.2
0.2
0.5 0.50.4
0.1 0.10.2
Three basic problems for HMMs
vLikelihood of the input:vForward algorithm
vDecoding (tagging) the input:vViterbi algorithm
vEstimation (learning):vFind the best model parameters
v Case 1: supervised – tags are annotatedvMaximum likelihood estimation (MLE)
v Case 2: unsupervised -- only unannotated textvForward-backward algorithm
CS6501 Natural Language Processing 23
Howlikelythesentence”Ilovecat”occurs
POStagsof”Ilovecat”occurs
Howtolearnthemodel?
Prediction in generative model
v Inference: What is the most likely sequence of tags for the given sequence of words w
vWhat are the latent states that most likely generate the sequence of word w
CS6501 Natural Language Processing 24
initialprobability𝑝(𝑡0)
Tagging the input
vFind best tag sequence of “I love cat”vRemember, we model 𝑃 𝒕,𝒘 ∣ 𝝀
v 𝒕∗ = argmax𝒕𝑃 𝒕,𝒘 𝝀
CS6501 Natural Language Processing 25
Findthebestonefromallpossibletagsequences
Tagging the input
v Assume we have 2 tags N, Vv Which one is the best?𝑃 “𝐼𝑙𝑜𝑣𝑒𝑐𝑎𝑡”, ”𝑁𝑁𝑁” 𝝀 , 𝑃 “𝐼𝑙𝑜𝑣𝑒𝑐𝑎𝑡”, ”𝑁𝑁𝑉” 𝝀 ,𝑃 “𝐼𝑙𝑜𝑣𝑒𝑐𝑎𝑡”, ”𝑁𝑉𝑁” 𝝀 , 𝑃 “𝐼𝑙𝑜𝑣𝑒𝑐𝑎𝑡”, ”𝑁𝑉𝑉” 𝝀 ,𝑃 “𝐼𝑙𝑜𝑣𝑒𝑐𝑎𝑡”, ”𝑉𝑁𝑁” 𝝀 , 𝑃 “𝐼𝑙𝑜𝑣𝑒𝑐𝑎𝑡”, ”𝑉𝑁𝑉” 𝝀 ,
𝑃 “𝐼𝑙𝑜𝑣𝑒𝑐𝑎𝑡”, ”𝑉𝑉𝑁” 𝝀 , 𝑃 “𝐼𝑙𝑜𝑣𝑒𝑐𝑎𝑡”, ”𝑉𝑉𝑉” 𝝀
v Again! We need an efficient algorithm
CS6501 Natural Language Processing 26
Trellis diagram
vGoal: argmax𝒕Π./01 𝑃 𝑤. 𝑡. 𝑃 𝑡. ∣ 𝑡.40
CS6501 Natural Language Processing 27
𝑃(𝑡C = 2|𝑡0 = 1)
𝑖 = 1𝑖 = 2𝑖 = 3𝑖 = 4
𝑃(𝑡J = 1|𝑡C = 1) 𝑃(𝑤J|𝑡J = 1)
⋯
Trellis diagram
v Goal: argmax𝒕Π./01 𝑃 𝑤. 𝑡. 𝑃 𝑡. ∣ 𝑡.40
v Find the best path!
CS6501 Natural Language Processing 28
𝑖 = 1𝑖 = 2𝑖 = 3𝑖 = 4
N
V
A
N
V
A
N
V
A
N
V
A
⋯
Dynamic programming again!
vRecursively decompose a problem into smaller sub-problems
vSimilar to mathematical inductionvBase step: initial values for 𝑖 = 1v Inductive step: assume we know the values for 𝑖 = 𝑘, let’s compute 𝑖 = 𝑘 + 1
CS6501 Natural Language Processing 29
Viterbi algorithm
v Inductive step: from 𝑖 = 𝑘 toi =k+1v 𝒕Y :tagsequencewithlength𝑘,𝒘Y = 𝑤0, 𝑤C …𝑤Yv max
𝒕𝒌𝑃(𝒕Y,𝒘𝒌) = 𝑚𝑎𝑥kmax𝒕𝒌i𝟏
𝑃(𝒕Y40, 𝑡Y = 𝑞,𝒘𝒌)
CS6501 Natural Language Processing 30𝑖 = 1𝑖 = 2𝑖 = 3𝑖 = 4
N
V
A
N
V
A
N
V
A
N
V
A
⋯
tagsequencestag@i=k tagsequences
Viterbi algorithm
v Inductive step: from 𝑖 = 𝑘 toi =k+1v max
𝒕𝒌i𝟏𝑃(𝒕Y40, 𝑡Y = 𝑞,𝒘𝒌)
= 𝑚𝑎𝑥kl max𝒕𝒌i𝟏𝑃(𝒕Y4𝟐, 𝑡Y = 𝑞, 𝑡Y40 = 𝑞m,𝒘𝒌)
= 𝑚𝑎𝑥kl max𝒕𝒌i𝟏𝑃 𝒕Y4𝟐, 𝑡Y40 = 𝑞m,𝒘𝒌4𝟏 𝑃 𝑡Y = 𝑞, 𝑡Y40 = 𝑞m 𝑃 𝑤Y 𝑡Y = 𝑞
CS6501 Natural Language Processing31𝑖 = 𝑘 − 1𝑖 = 𝑘
N
V
A
N
V
A
N
V
A
N
V
A
⋯
Let’scallit𝛿Y(𝑞)
Thisis𝛿Y40(𝑞′)
Viterbi algorithm
v Inductive step: from 𝑖 = 𝑘 toi =k+1v 𝛿Y 𝑞 =max
km𝛿Y40(𝑞m)𝑃(𝑡Y = 𝑞 ∣ 𝑡Y40 = 𝑞′)𝑃(𝑤Y ∣ 𝑡Y = 𝑞)
CS6501 Natural Language Processing32𝑖 = 𝑘 − 1𝑖 = 𝑘
N
V
A
N
V
A
N
V
A
N
V
A
⋯
Viterbi algorithm
v Inductive step: from 𝑖 = 𝑘 toi =k+1v 𝛿Y 𝑞 =max
kl𝛿Y40 𝑞m 𝑃 𝑡Y = 𝑞 ∣ 𝑡Y40 = 𝑞m 𝑃 𝑤Y ∣ 𝑡Y = 𝑞
=𝑃 𝑤Y ∣ 𝑡Y = 𝑞 maxkl
𝛿Y40 𝑞m 𝑃 𝑡Y = 𝑞 ∣ 𝑡Y40 = 𝑞m
CS6501 Natural Language Processing33𝑖 = 𝑘 − 1𝑖 = 𝑘
N
V
A
N
V
A
N
V
A
N
V
A
⋯
Viterbi algorithm
v Base step: i=0 v 𝛿0 𝑞 = 𝑃 𝑤0 𝑡0 = 𝑞 𝑃(𝑡0 = 𝑞 ∣ 𝑡q)
CS6501 Natural Language Processing34𝑖 = 𝑘 − 1𝑖 = 𝑘
N
V
A
N
V
A
N
V
A
N
V
A
⋯
initialprobability𝑝(𝑡0 = 𝑞)
Implementation using an array
CS6501 Natural Language Processing 35
Initial:Trellis[1][q]= 𝑃 𝑤0 𝑡0 = 𝑞 𝑃(𝑡0 = 𝑞 ∣ 𝑡q)
Implementation using an array
CS6501 Natural Language Processing 36
Induction:𝛿Y 𝑞 = 𝑃 𝑤Y ∣ 𝑡Y = 𝑞 max
kl𝛿Y40 𝑞m 𝑃 𝑡Y = 𝑞 ∣ 𝑡Y40 = 𝑞m
Retrieving the best sequence
vKeep one backpointer
CS6501 Natural Language Processing 37
The Viterbi algorithm (Pseudo Code)
CS6501 Natural Language Processing 38
.fwd=0
Maxinsteadofsum
CS6501 Natural Language Processing 39
Jason’s ice cream
vBest tag sequence for P(”1,2,1”)?
CS6501 Natural Language Processing 40
p(…|C) p(…|H) p(…|START)p(1|…) 0.5 0.1p(2|…) 0.4 0.2p(3|…) 0.1 0.7p(C|…) 0.8 0.2 0.5p(H|…) 0.2 0.8 0.5
Scroll to the bottom to see a graph of what states and transitions the model thinks are likely on each day. Those likely states and transitions can be used to reestimate the red probabilities (this is the "forward-backward" or Baum-Welch algorithm), incr
#cones
C
H
C
H
C
H
0.5
0.5
0.8
0.80.8
0.8
0.2
0.2
0.2
0.2
0.5 0.50.4
0.1 0.10.2
Trick: computing everything in log space
vHomework:vWrite forward and Viterbi algorithm in log-space
vHint: you need a function to compute log(a+b)
CS6501 Natural Language Processing 41
Three basic problems for HMMs
vLikelihood of the input:vForward algorithm
vDecoding (tagging) the input:vViterbi algorithm
vEstimation (learning):vFind the best model parameters
v Case 1: supervised – tags are annotatedvMaximum likelihood estimation (MLE)
v Case 2: unsupervised -- only unannotated textvForward-backward algorithm
CS6501 Natural Language Processing 42
Howlikelythesentence”Ilovecat”occurs
POStagsof”Ilovecat”occurs
Howtolearnthemodel?