edinburgh mt lecture 7: phrase-based mt
TRANSCRIPT
![Page 1: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/1.jpg)
Phrase-Based Translation
![Page 2: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/2.jpg)
The IBM Models
![Page 3: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/3.jpg)
The IBM Models
•Fertility probabilities.
![Page 4: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/4.jpg)
The IBM Models
•Fertility probabilities.
•Word translation probabilities.
![Page 5: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/5.jpg)
The IBM Models
•Fertility probabilities.
•Word translation probabilities.
•Distortion probabilities.
![Page 6: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/6.jpg)
The IBM Models
•Fertility probabilities.
•Word translation probabilities.
•Distortion probabilities.
•Some problems:
•Weak reordering model -- output is not fluent.
•Many decisions -- many things can go wrong.
![Page 7: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/7.jpg)
The IBM Models
•Fertility probabilities.
•Word translation probabilities.
•Distortion probabilities.
•Some problems:
•Weak reordering model -- output is not fluent.
•Many decisions -- many things can go wrong.
![Page 8: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/8.jpg)
The IBM Models
•Fertility probabilities.
•Word translation probabilities.
•Distortion probabilities.
•Some problems:
•Weak reordering model -- output is not fluent.
•Many decisions -- many things can go wrong.
![Page 9: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/9.jpg)
Although north wind howls , but sky still very clear .虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
However , the sky remained clear under the strong north wind .
虽然
�
� �北 风 呼啸 , 天空 天空 依然 清澈 。
north wind strong , the sky remained clear . under theHowever
IBM Model 4
![Page 10: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/10.jpg)
Tradeoffs: Modeling v. Learning
IBM Model 1 ✔ ✘ ✘ ✔ ✔
HMM ✔ ✔ ✘ ✘ ✔
IBM Model 4 ✔ ✔ ✔ ✘ ✘
Lexical
Tran
slatio
n
Local orderi
ng depen
dency
Fertilit
y
Convex
Tracta
ble Exa
ct
Inferen
ce
![Page 11: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/11.jpg)
Tradeoffs: Modeling v. Learning
IBM Model 1 ✔ ✘ ✘ ✔ ✔
HMM ✔ ✔ ✘ ✘ ✔
IBM Model 4 ✔ ✔ ✔ ✘ ✘
Lexical
Tran
slatio
n
Local orderi
ng depen
dency
Fertilit
y
Convex
Tracta
ble Exa
ct
Inferen
ce
Lesson:Trade exactnessfor expressivity
![Page 12: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/12.jpg)
Although north wind howls , but sky still very clear .虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
However , the sky remained clear under the strong north wind .
虽然
�
� �北 风 呼啸 , 天空 天空 依然 清澈 。
north wind strong , the sky remained clear . under theHowever
IBM Model 4
What are some things this model doesn’t account for?
![Page 13: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/13.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。Although north wind howls , but sky still very clear .
However , the sky remained clear under the strong north wind .
虽然 北 风 呼啸 , 天空 天空 依然 清澈 。
north wind strong , the sky remained clear . under theHowever
�
�
�
What are some things this model doesn’t account for?
![Page 14: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/14.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。Although north wind howls , but sky still very clear .
Phrase-based Models
![Page 15: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/15.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。Although north wind howls , but sky still very clear .
Phrase-based Models
![Page 16: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/16.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。Although north wind howls , but sky still very clear .
However
Phrase-based Models
![Page 17: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/17.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。Although north wind howls , but sky still very clear .
However the strong north wind , the sky remained clear under .
Phrase-based Models
![Page 18: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/18.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。Although north wind howls , but sky still very clear .
However the strong north wind , the sky remained clear under .
However
Phrase-based Models
![Page 19: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/19.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。Although north wind howls , but sky still very clear .
However the strong north wind , the sky remained clear under .
However ,
Phrase-based Models
![Page 20: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/20.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。Although north wind howls , but sky still very clear .
However the strong north wind , the sky remained clear under .
However , the sky remained clear under the strong north wind .
Phrase-based Models
![Page 21: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/21.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。Although north wind howls , but sky still very clear .
However the strong north wind , the sky remained clear under .
However , the sky remained clear under the strong north wind .
p(English, alignment|Chinese) =p(segmentation) · p(translations) · p(reorderings)
Phrase-based Models
![Page 22: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/22.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。Although north wind howls , but sky still very clear .
However the strong north wind , the sky remained clear under .
However , the sky remained clear under the strong north wind .
p(English, alignment|Chinese) =p(segmentation) · p(translations) · p(reorderings)
Phrase-based Models
![Page 23: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/23.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。Although north wind howls , but sky still very clear .
However the strong north wind , the sky remained clear under .
However , the sky remained clear under the strong north wind .
p(English, alignment|Chinese) =p(segmentation) · p(translations) · p(reorderings)
Phrase-based Models
distortion = 6
![Page 24: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/24.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。Although north wind howls , but sky still very clear .
However the strong north wind , the sky remained clear under .
However , the sky remained clear under the strong north wind .
p(English, alignment|Chinese) =p(segmentation) · p(translations) · p(reorderings)
Phrase-based Models
distortion = 6
![Page 25: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/25.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。Although north wind howls , but sky still very clear .
However the strong north wind , the sky remained clear under .
However , the sky remained clear under the strong north wind .
p(English, alignment|Chinese) =p(segmentation) · p(translations) · p(reorderings)
Phrase-based Models
distortion = 6
![Page 26: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/26.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。Although north wind howls , but sky still very clear .
However the strong north wind , the sky remained clear under .
However , the sky remained clear under the strong north wind .
p(English, alignment|Chinese) =p(segmentation) · p(translations) · p(reorderings)
Phrase-based Models
distortion = 6
![Page 27: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/27.jpg)
Phrase-based Models
![Page 28: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/28.jpg)
Phrase-based Models
•Segmentation probabilities.
![Page 29: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/29.jpg)
Phrase-based Models
•Segmentation probabilities.
•Phrase translation probabilities.
![Page 30: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/30.jpg)
Phrase-based Models
•Segmentation probabilities.
•Phrase translation probabilities.
•Distortion probabilities.
![Page 31: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/31.jpg)
Phrase-based Models
•Segmentation probabilities.
•Phrase translation probabilities.
•Distortion probabilities.
•Some problems:
•Weak reordering model -- output is not fluent.
•Many decisions -- many things can go wrong.
![Page 32: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/32.jpg)
Phrase-based Models
•Segmentation probabilities.
•Phrase translation probabilities.
•Distortion probabilities.
•Some problems:
•Weak reordering model -- output is not fluent.
•Many decisions -- many things can go wrong.
![Page 33: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/33.jpg)
Phrase-based Models
•Segmentation probabilities.
•Phrase translation probabilities.
•Distortion probabilities.
•Some problems:
•Weak reordering model -- output is not fluent.
•Many decisions -- many things can go wrong.
![Page 34: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/34.jpg)
Phrase-based Models
•Segmentation probabilities: fixed (uniform)
•Phrase translation probabilities.
•Distortion probabilities: fixed (decaying)
![Page 35: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/35.jpg)
Learning p(Chinese|English)
•Reminder: (nearly) every problem comes down to computing either:
•Sums: MLE or EM (learning)
•Maximum: most probable (decoding)
![Page 36: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/36.jpg)
Recap: Expectation Maximization
•Arbitrarily select a set of parameters (say, uniform).
•Calculate expected counts of the unseen events.
•Choose new parameters to maximize likelihood, using expected counts as proxy for observed counts.
•Iterate.
•Guaranteed that likelihood is monotonically nondecreasing.
![Page 37: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/37.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
However , the sky remained clear under the strong north wind .
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
However , the sky remained clear under the strong north wind .
p(
p(
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
However , the sky remained clear under the strong north wind .
p( )
) +
) +
Marginalize: sum all alignments containing the link
![Page 38: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/38.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
However , the sky remained clear under the strong north wind .
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
However , the sky remained clear under the strong north wind .
p(
p(
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
However , the sky remained clear under the strong north wind .
p(
) +
) +
)
Divide by sum of all possible alignments
![Page 39: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/39.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
However , the sky remained clear under the strong north wind .
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
However , the sky remained clear under the strong north wind .
p(
p(
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
However , the sky remained clear under the strong north wind .
p(
) +
) +
)
Divide by sum of all possible alignments
We have to sum over exponentially many alignments!
![Page 40: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/40.jpg)
EM for Model 1
probability of an alignment.
p(F,A|E) = p(I|J)Y
ai
p(ai = j)p(fi|ej)
![Page 41: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/41.jpg)
EM for Model 1
probability of an alignment.
observed uniform
p(F,A|E) = p(I|J)Y
ai
p(ai = j)p(fi|ej)
![Page 42: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/42.jpg)
factors across words.
EM for Model 1
probability of an alignment.
observed uniform
p(F,A|E) = p(I|J)Y
ai
p(ai = j)p(fi|ej)
![Page 43: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/43.jpg)
EM for Model 1
p(ai = j|F,E) =p(ai = j, F |E)
p(F,E)=
![Page 44: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/44.jpg)
EM for Model 1
北北
.�
a⇥A: �north
p(north| ) · p(rest of a)
p(ai = j|F,E) =p(ai = j, F |E)
p(F,E)=
![Page 45: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/45.jpg)
EM for Model 1
北北
.�
a⇥A: �north
p(north| ) · p(rest of a)
marginal probability of alignments containing link
p(ai = j|F,E) =p(ai = j, F |E)
p(F,E)=
![Page 46: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/46.jpg)
EM for Model 1
p(north| ).�
a⇥A: �north
p(rest of a)北北
marginal probability of alignments containing link
![Page 47: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/47.jpg)
EM for Model 1
p(north| ).�
a⇥A: �north
p(rest of a)北北
marginal probability of alignments containing link
�
c⇥Chinese words
p(north|c).�
a⇥A: �north
p(rest of a)
marginal probability of all alignments
![Page 48: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/48.jpg)
EM for Model 1
p(north| ).�
a⇥A: �north
p(rest of a)北北
marginal probability of alignments containing link
�
c⇥Chinese words
p(north|c).�
a⇥A: �north
p(rest of a)
marginal probability of all alignments
c
![Page 49: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/49.jpg)
EM for Model 1
p(north| ).�
a⇥A: �north
p(rest of a)北北
marginal probability of alignments containing link
�
c⇥Chinese words
p(north|c).�
a⇥A: �north
p(rest of a)
marginal probability of all alignments
c
identical!
![Page 50: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/50.jpg)
EM for Model 1
北p(north| ).�
c�Chinese words p(north|c)
![Page 51: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/51.jpg)
EM for Phrase-Based
•Model parameters: p(E phrase|F phrase)
•All we need to do is compute expectations:
p(ai = j|F,E) =p(ai,i0 = hj, j0i, F |E)
p(F,E)
![Page 52: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/52.jpg)
EM for Phrase-Based
•Model parameters: p(E phrase|F phrase)
•All we need to do is compute expectations:
p(F,E) sums over all possible phrase alignments
p(ai = j|F,E) =p(ai,i0 = hj, j0i, F |E)
p(F,E)
![Page 53: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/53.jpg)
EM for Phrase-Based
•Model parameters: p(E phrase|F phrase)
•All we need to do is compute expectations:
p(F,E) sums over all possible phrase alignments...which are one-to-one by definition.
p(ai = j|F,E) =p(ai,i0 = hj, j0i, F |E)
p(F,E)
![Page 54: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/54.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。Although north wind howls , but sky still very clear .
However
EM for Phrase-Based
p(ai = j|F,E) =p(ai,i0 = hj, j0i, F |E)
p(F,E)
, the sky remained clear under the strong north wind .
![Page 55: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/55.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。Although north wind howls , but sky still very clear .
However
EM for Phrase-Based
p(ai = j|F,E) =p(ai,i0 = hj, j0i, F |E)
p(F,E)
Can we compute this quantity?
, the sky remained clear under the strong north wind .
![Page 56: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/56.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。Although north wind howls , but sky still very clear .
However
EM for Phrase-Based
p(ai = j|F,E) =p(ai,i0 = hj, j0i, F |E)
p(F,E)
Can we compute this quantity?
, the sky remained clear under the strong north wind .
How many 1-to-1 alignments are there ofthe remaing 8 Chinese and 8 English words?
![Page 57: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/57.jpg)
Recap: Expectation Maximization
•Arbitrarily select a set of parameters (say, uniform).
•Calculate expected counts of the unseen events.
•Choose new parameters to maximize likelihood, using expected counts as proxy for observed counts.
•Iterate.
•Guaranteed that likelihood is monotonically nondecreasing.
![Page 58: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/58.jpg)
Recap: Expectation Maximization
•Arbitrarily select a set of parameters (say, uniform).
•Calculate expected counts of the unseen events.
•Choose new parameters to maximize likelihood, using expected counts as proxy for observed counts.
•Iterate.
•Guaranteed that likelihood is monotonically nondecreasing.
Computing expectations from a phrase-based model, given a sentence pair, is #P-Complete(by reduction to counting perfect matchings;
DeNero & Klein, 2008)
![Page 59: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/59.jpg)
argmaxa p(a|f,e) is also hard
![Page 60: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/60.jpg)
argmaxa p(a|f,e) is also hard
![Page 61: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/61.jpg)
argmaxa p(a|f,e) is also hard
![Page 62: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/62.jpg)
Now What?
•Option #1: approximate expectations
•Restrict computation to some tractable subset of the alignment space (arbitrarily biased).
•Markov chain Monte Carlo (slow).
![Page 63: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/63.jpg)
Now What?•Change the problem definition
•We already know how to learn word-to-word translation models efficiently.
•Idea: learn word-to-word alignments, extract most probable alignment, then treat it as observed.
•Learn phrase translations consistent with word alignments.
•Decouples alignment from model learning -- is this a good thing?
![Page 64: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/64.jpg)
Phrase Extraction
I open the box
watashi
wa
hako
wo
akemasu
![Page 65: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/65.jpg)
Phrase Extraction
I open the box
watashi
wa
hako
wo
akemasu
akemasu / open
![Page 66: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/66.jpg)
Phrase Extraction
I open the box
watashi
wa
hako
wo
akemasu
watashi wa / I
![Page 67: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/67.jpg)
Phrase Extraction
I open the box
watashi
wa
hako
wo
akemasu
watashi / I
![Page 68: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/68.jpg)
Phrase Extraction
I open the box
watashi
wa
hako
wo
akemasu
watashi / I ✘
![Page 69: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/69.jpg)
Phrase Extraction
I open the box
watashi
wa
hako
wo
akemasu
hako wo / box
![Page 70: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/70.jpg)
Phrase Extraction
I open the box
watashi
wa
hako
wo
akemasu
hako wo / the box
![Page 71: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/71.jpg)
Phrase Extraction
I open the box
watashi
wa
hako
wo
akemasu
hako wo / open the box
![Page 72: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/72.jpg)
Phrase Extraction
I open the box
watashi
wa
hako
wo
akemasu
hako wo / open the box✘
![Page 73: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/73.jpg)
Phrase Extraction
I open the box
watashi
wa
hako
wo
akemasu
hako wo akemasu / open the box
![Page 74: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/74.jpg)
Phrasal Translation Estimation
![Page 75: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/75.jpg)
Phrasal Translation Estimation
•Option #1 (EM over restricted space)
•Align with a word-based model.
•Compute expectations only over alignments consistent with the alignment grid.
![Page 76: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/76.jpg)
Phrasal Translation Estimation
•Option #1 (EM over restricted space)
•Align with a word-based model.
•Compute expectations only over alignments consistent with the alignment grid.
•Option #2 (Non-global estimation)
•View phrase pairs as observed, irrespective of context or overlap.
![Page 77: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/77.jpg)
Decoding
We want to solve this problem:
e⇤ = arg max
ep(e|f)
![Page 78: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/78.jpg)
Decoding
We want to solve this problem:
e⇤ = arg max
ep(e|f)
Q: how many English sentences are there?
![Page 79: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/79.jpg)
北 风 呼啸 。
![Page 80: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/80.jpg)
北 风 呼啸 。
segmentationssubstitutionspermutations
![Page 81: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/81.jpg)
北 风 呼啸 。
O(2n)segmentationssubstitutionspermutations
![Page 82: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/82.jpg)
北 风 呼啸 。
O(2n)O(5n)
segmentationssubstitutionspermutations
![Page 83: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/83.jpg)
北 风 呼啸 。
O(2n)O(5n)O(n!)
segmentationssubstitutionspermutations
![Page 84: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/84.jpg)
Key Idea
![Page 85: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/85.jpg)
Key Idea
![Page 86: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/86.jpg)
Key Idea
![Page 87: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/87.jpg)
Key Idea
![Page 88: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/88.jpg)
Key Idea
![Page 89: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/89.jpg)
Key Idea
Dynamic Programming
![Page 90: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/90.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
![Page 91: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/91.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
![Page 92: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/92.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
START Although
crystal clear
START However
![Page 93: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/93.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
START Although
crystal clear
START However
![Page 94: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/94.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
wind shrieked
wind screamed
north wind
![Page 95: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/95.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
wind shrieked
wind screamed
north wind
![Page 96: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/96.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
the sky
shrieked ,
, yet
![Page 97: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/97.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
the sky
shrieked ,
, yet
![Page 98: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/98.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
sky ,
![Page 99: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/99.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
sky ,
![Page 100: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/100.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。clear .
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。still quite
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。blue .
![Page 101: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/101.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。clear .
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。still quite
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。blue .
![Page 102: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/102.jpg)
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
Although the northern wind shrieked across the sky, but was still very clear.
![Page 103: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/103.jpg)
Approximation: Pruning
![Page 104: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/104.jpg)
Approximation: Pruning
Idea: prune states by accumulated path length
![Page 105: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/105.jpg)
Approximation: Pruning
![Page 106: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/106.jpg)
Approximation: Pruning
Solution: Group states by number of covered words.
![Page 107: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/107.jpg)
![Page 108: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/108.jpg)
•Some (not all) key ingredients in Google Translate:
![Page 109: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/109.jpg)
•Some (not all) key ingredients in Google Translate:
•Phrase-based translation models
![Page 110: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/110.jpg)
•Some (not all) key ingredients in Google Translate:
•Phrase-based translation models
•... Learned heuristically from word alignments
![Page 111: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/111.jpg)
•Some (not all) key ingredients in Google Translate:
•Phrase-based translation models
•... Learned heuristically from word alignments
•... Coupled with a huge language model
![Page 112: Edinburgh MT lecture 7: phrase-based MT](https://reader031.vdocument.in/reader031/viewer/2022020106/55b3bf49bb61ebd0088b477e/html5/thumbnails/112.jpg)
•Some (not all) key ingredients in Google Translate:
•Phrase-based translation models
•... Learned heuristically from word alignments
•... Coupled with a huge language model
•... And decoding w/ severe pruning heuristics