modified distortion matrices for phrase-based smt arianna bisazza & marcello federico – fbk...

Modified Distortion Matrices for Phrase-Based SMT

Arianna Bisazza & Marcello Federico – FBK (Italy)

w0 w1 w2 w3 w4 w5 w6 w7 w8 w9w1

0

<s> 0 1 2 3 4 5 6 7 8 9 10w0 0 1 2 3 4 5 6 7 8 9w1 2 0 1 2 3 4 5 6 7 8w2 3 2 0 1 2 3 4 5 6 7w3 4 3 2 0 1 2 3 4 5 6w4 5 4 3 2 0 1 2 3 0 5w5 6 5 4 3 2 0 1 2 3 4w6 7 6 5 4 3 2 0 1 2 3w7 8 7 6 5 4 3 2 0 1 0w8 9 8 7 6 5 4 3 2 0 0w9 10 9 8 7 6 2 2 3 2 0w10 11 10 9 8 7 6 5 4 3 2

2 A. Bisazza & M. Federico – Modified Distortion Matrices for PSMT

A. Bisazza & M. Federico – Modified Distortion Matrices for PSMT

2

PSMT decoding overview

E' necessario incoraggiare tale mobilità garantendo la sicurezza dei percorsi professionali

3

Freedom of movement

must be encouraged

LM scores



3



LM scores

TM

scores

TM scores

ReoM scores

ReoM scores

4

career paths …

while ensuring that

Freedom of movement

must be encouraged

LM scoresLM scoresLM scores



4



LM scores

TM scoresTM

scores TM

scor

esTM

scores

ReoM scores

ReoM scores

ReoM scores

ReoM scores

5

LM scoresLM scoresLM scores



5



Freedom of movement must be encouraged while ensuring that career paths …

LM scores

TM scoresTM

scores TM

scor

esTM

scores

ReoM scores

ReoM scores

ReoM scores

ReoM scores



6

Reordering Models


ReoM scores

ReoM scores

ReoM scores

ReoM scores

Many solutions have been proposed

with different reo. classes, features,

train modes etc.

Tillman 04, Zens & Ney 06AlOnaizan & Papineni 06Galley & Manning 08Green & al.10, Feng & al.10…



7

Reordering Models


ReoM scores

ReoM scores

ReoM scores

ReoM scores

No matter what reordering model is used, permutation search space must be limited!

The power of all reordering models is bound to the reordering constraints in use

Tillman04, Zens&Ney06AlOnaizan & Papineni06Galley & Manning08Green &al.10, Feng &al.10…

Many solutions have been proposed

with different reo. classes, features,

train modes etc.

Tillman 04, Zens & Ney 06AlOnaizan & Papineni 06Galley & Manning 08Green & al.10, Feng & al.10…



8


ReoM scores

ReoM scores

ReoM scores

ReoM scores



9

Reordering Constraints


#perm.=11!≈40,000,000

ReoM scores

ReoM scores

ReoM scores

ReoM scores



10


Source-to-Source distortion

#perm.=11!≈40,000,000

D(x,y)=|y-x-1|

w0 w1 w2 w3 w4 w5 w6 w7 w8 w9 w10

<s>

0 1 2 3 4 5 6 7 8 9 10

w0 0 1 2 3 4 5 6 7 8 9w1 2 0 1 2 3 4 5 6 7 8w2 3 2 0 1 2 3 4 5 6 7w3 4 3 2 0 1 2 3 4 5 6w4 5 4 3 2 0 1 2 3 4 5w5 6 5 4 3 2 0 1 2 3 4w6 7 6 5 4 3 2 0 1 2 3w7 8 7 6 5 4 3 2 0 1 2w8 9 8 7 6 5 4 3 2 0 1w9 10 9 8 7 6 5 4 3 2 0w10 11 10 9 8 7 6 5 4 3 2




11



#perm.=11!≈40,000,000

D(x,y)=|y-x-1|

DL=3 #perm.≈7,000

w0 w1 w2 w3 w4 w5 w6 w7 w8 w9 w10

<s>

0 1 2 3 4 5 6 7 8 9 10

w0 0 1 2 3 4 5 6 7 8 9w1 2 0 1 2 3 4 5 6 7 8w2 3 2 0 1 2 3 4 5 6 7w3 4 3 2 0 1 2 3 4 5 6w4 5 4 3 2 0 1 2 3 4 5w5 6 5 4 3 2 0 1 2 3 4w6 7 6 5 4 3 2 0 1 2 3w7 8 7 6 5 4 3 2 0 1 2w8 9 8 7 6 5 4 3 2 0 1w9 10 9 8 7 6 5 4 3 2 0w10 11 10 9 8 7 6 5 4 3 2

DL: distortion limit




12

The problem with DL…

Arabic-English

AR

EN

AR

EN

w0 w1 w2 w3 w4 w5 w6 w7 w8 w9 w1

0

<s>

0 1 2 3 4 5 6 7 8 9 10

w0 0 1 2 3 4 5 6 7 8 9w1 2 0 1 2 3 4 5 6 7 8w2 3 2 0 1 2 3 4 5 6 7w3 4 3 2 0 1 2 3 4 5 6w4 5 4 3 2 0 1 2 3 4 5w5 6 5 4 3 2 0 1 2 3 4w6 7 6 5 4 3 2 0 1 2 3w7 8 7 6 5 4 3 2 0 1 2w8 9 8 7 6 5 4 3 2 0 1w9 10 9 8 7 6 5 4 3 2 0w10 1110 9 8 7 6 5 4 3 2



13

German-English

DE

EN

DE

EN

w0 w1 w2 w3 w4 w5 w6 w7 w8 w9 w1

0

<s>

0 1 2 3 4 5 6 7 8 9 10

w0 0 1 2 3 4 5 6 7 8 9w1 2 0 1 2 3 4 5 6 7 8w2 3 2 0 1 2 3 4 5 6 7w3 4 3 2 0 1 2 3 4 5 6w4 5 4 3 2 0 1 2 3 4 5w5 6 5 4 3 2 0 1 2 3 4w6 7 6 5 4 3 2 0 1 2 3w7 8 7 6 5 4 3 2 0 1 2w8 9 8 7 6 5 4 3 2 0 1w9 10 9 8 7 6 5 4 3 2 0w10 1110 9 8 7 6 5 4 3 2

The problem with DL…



14


w0 w1 w2 w3 w4 w5 w6 w7 w8 w9 w10

<s>

0 1 2 3 4 5 6 7 8 9 10

w0 0 1 2 3 4 5 6 7 8 9w1 2 0 1 2 3 4 5 6 7 8w2 3 2 0 1 2 3 4 5 6 7w3 4 3 2 0 1 2 3 4 5 6w4 5 4 3 2 0 1 2 3 4 5w5 6 5 4 3 2 0 1 2 3 4w6 7 6 5 4 3 2 0 1 2 3w7 8 7 6 5 4 3 2 0 1 2w8 9 8 7 6 5 4 3 2 0 1w9 10 9 8 7 6 5 4 3 2 0w10 11 10 9 8 7 6 5 4 3 2

Current solution:

increase the DLimit

#perm.=11! ≈40,000,000

D(x,y)=|y-x-1|

DL=3 #perm.≈7,000



15


w0 w1 w2 w3 w4 w5 w6 w7 w8 w9 w10

<s>

0 1 2 3 4 5 6 7 8 9 10

w0 0 1 2 3 4 5 6 7 8 9w1 2 0 1 2 3 4 5 6 7 8w2 3 2 0 1 2 3 4 5 6 7w3 4 3 2 0 1 2 3 4 5 6w4 5 4 3 2 0 1 2 3 4 5w5 6 5 4 3 2 0 1 2 3 4w6 7 6 5 4 3 2 0 1 2 3w7 8 7 6 5 4 3 2 0 1 2w8 9 8 7 6 5 4 3 2 0 1w9 10 9 8 7 6 5 4 3 2 0w10 11 10 9 8 7 6 5 4 3 2

Current solution:

increase the DLimit

Generally leads to worse translations!

#perm.=11! ≈40,000,000

D(x,y)=|y-x-1|

DL=3 #perm.≈7,000

DL=7 #perm.≈7,000,000



16


#perm.=11! ≈40,000,000

D(x,y)=|y-x-1|

DL=3 #perm.≈7,000

DL=7 #perm.≈7,000,000

w0 w1 w2 w3 w4 w5 w6 w7 w8 w9 w10

<s>

0 1 2 3 4 5 6 7 8 9 10

w0 0 1 2 3 4 5 6 7 8 9w1 2 0 1 2 3 4 5 6 7 8w2 3 2 0 1 2 3 4 5 6 7w3 4 3 2 0 1 2 3 4 5 6w4 5 4 3 2 0 1 2 3 4 5w5 6 5 4 3 2 0 1 2 3 4w6 7 6 5 4 3 2 0 1 2 3w7 8 7 6 5 4 3 2 0 1 2w8 9 8 7 6 5 4 3 2 0 1w9 10 9 8 7 6 5 4 3 2 0w10 11 10 9 8 7 6 5 4 3 2

Our solution:



17


#perm.=11! ≈40,000,000

D(x,y)=|y-x-1|

DL=3 #perm.≈7,000

DL=7 #perm.≈7,000,000

DL=3 & modif(D)

#perm.≈20,000

w0 w1 w2 w3 w4 w5 w6 w7 w8 w9 w10

<s>

0 1 2 3 4 5 6 7 8 9 10

w0 0 1 2 3 4 5 6 7 8 9w1 2 0 1 2 3 4 0 0 7 8w2 3 2 0 1 2 3 0 0 6 7w3 4 3 2 0 1 2 3 4 5 6w4 5 4 3 2 0 1 2 3 4 5w5 6 5 4 3 2 0 1 2 3 0w6 7 6 5 4 3 2 0 1 2 3w7 8 7 6 5 4 3 2 0 1 2w8 9 8 7 6 5 4 3 2 0 1w9 10 9 8 2 2 5 4 3 2 0w10 11 10 9 8 7 6 5 4 3 2

Our solution:

modify distortion for each test

sentence

Simplifies the task of reordering models!

18

Rest of the talk:

How to modify the distortion matrix?

What effect on translation quality?

What effect on baseline runtimes?




19

Chunk-basedfuzzy reordering

rulesShallow syntax chunking:

• cheaper and easier than deep parsing

• constrains reorderings in a softer way

Fuzzy (non-determinisic) reordering rules:

• generate N permutations for each matching sequence

• final reordering decision is taken during translation, guided by all SMT models (reoM, LM...)

Few rules for language pair, to only capture long reordering



20

Arabic-English

“Move verb chunk (and following chunk) to the right by 1 to N

chunks”


rules

CC1 VC2 PC3 NC4 PC5 Pct6

w- $Ark fy AltZAhrp E$rAt AlmslHyn mn AlktA}b . and took part in the march dozens of militants from the Brigades



21

Arabic-English


chunks”


CC1 VC2PC3 NC4 PC5

VC2PC3 NC4

VC2PC3 NC4 PC5

CC1

CC1

PC5

Pct6

Pct6

Pct6



rules



22

Arabic-English


chunks”


CC1 VC2PC3 NC4 PC5

VC2 PC3NC4

VC2PC3 NC4

VC2 PC3NC4 PC5

VC2PC3 NC4 PC5

CC1

CC1

CC1

CC1

PC5

PC5

Pct6

Pct6

Pct6

Pct6

Pct6



rules



23


CC1 VC2PC3 NC4 PC5

VC2 PC3NC4

VC2PC3 NC4

VC2 PC3NC4 PC5

VC2PC3 NC4 PC5

CC1

CC1

CC1

CC1

PC5

PC5

Pct6

Pct6

Pct6

Pct6

Pct6



rulesReordering selection

Reordered source LM

0.9

0.4

0.10.1

0.7



24


CC1 VC2PC3 NC4 PC5

VC2 PC3

Pct6

Pct6



rulesReordering selection

Reordered source LM

0.9

0.7

0.4

0.10.1

Reorderings to encode in the distortion matrix

NC4 PC5 CC1



25

Modifying the distortion

matrix


w0 w1 w2 w3 w4 w5 w6 w7 w8

<s>

0 1 2 3 4 5 6 7 8

CC1 w0 0 1 2 3 4 5 6 7VC2 w1 2 0 1 2 3 4 5 6

PC3

w2 3 2 0 1 2 3 4 5w3 4 3 2 0 1 2 3 4

NC4

w4 5 4 3 2 0 1 2 3w5 6 5 4 3 2 0 1 2

PC5

w6 7 6 5 4 3 2 0 1w7 8 7 6 5 4 3 2 0

Pct6 w8 9 8 7 6 5 4 3 2

CC1 VC2PC3 NC4 PC5

VC2 PC3


NC4 PC5 CC1

Pct6

Pct6

26

CC1 VC2PC3 NC4 PC5

VC2 PC3


NC4 PC5 CC1



26


matrix


w0 w1 w2 w3 w4 w5 w6 w7 w8

<s>

0 1 2 3 4 5 6 7 8

CC1 w0 0 0 0 3 4 5 6 7VC2 w1 2 0 1 2 3 4 5 6

PC3

w2 3 2 0 1 2 3 4 5w3 4 3 2 0 1 2 3 4

NC4

w4 5 4 3 2 0 1 2 3w5 6 5 4 3 2 0 1 2

PC5

w6 7 6 5 4 3 2 0 1w7 8 7 6 5 4 3 2 0

Pct6 w8 9 8 7 6 5 4 3 2

Pct6

Pct6



27


matrix


w0 w1 w2 w3 w4 w5 w6 w7 w8

<s>

0 1 2 3 4 5 6 7 8

CC1 w0 0 0 0 3 4 5 6 7VC2 w1 2 0 1 2 3 4 5 6

PC3

w2 3 2 0 1 2 3 4 5w3 4 2 2 0 1 2 3 4

NC4

w4 5 4 3 2 0 1 2 3w5 6 5 4 3 2 0 1 2

PC5

w6 7 6 5 4 3 2 0 1w7 8 7 6 5 4 3 2 0

Pct6 w8 9 8 7 6 5 4 3 2

CC1 VC2PC3 NC4 PC5

VC2 PC3


NC4 PC5 CC1

Pct6

Pct6



28


matrix


w0 w1 w2 w3 w4 w5 w6 w7 w8

<s>

0 1 2 3 4 5 6 7 8

CC1 w0 0 0 0 3 4 5 6 7VC2 w1 2 0 1 0 0 4 5 6

PC3

w2 3 2 0 1 2 3 4 5w3 4 2 2 0 1 2 3 4

NC4

w4 5 4 3 2 0 1 2 3w5 6 5 4 3 2 0 1 2

PC5

w6 7 6 5 4 3 2 0 1w7 8 7 6 5 4 3 2 0

Pct6 w8 9 8 7 6 5 4 3 2

CC1 VC2PC3 NC4 PC5

VC2 PC3


NC4 PC5 CC1

Pct6

Pct6



29


matrix


w0 w1 w2 w3 w4 w5 w6 w7 w8

<s>

0 1 2 3 4 5 6 7 8

CC1 w0 0 0 0 0 0 5 6 7VC2 w1 2 0 1 0 0 4 5 6

PC3

w2 3 2 0 1 2 3 4 5w3 4 2 2 0 1 2 3 4

NC4

w4 5 4 3 2 0 1 2 3w5 6 5 4 3 2 0 1 2

PC5

w6 7 6 5 4 3 2 0 1w7 8 7 6 5 4 3 2 0

Pct6 w8 9 8 7 6 5 4 3 2

CC1 VC2PC3 NC4 PC5

VC2 PC3


NC4 PC5 CC1

Pct6

Pct6



30


matrix


w0 w1 w2 w3 w4 w5 w6 w7 w8

<s>

0 1 2 3 4 5 6 7 8

CC1 w0 0 0 0 0 0 5 6 7VC2 w1 2 0 1 0 0 4 5 6

PC3

w2 3 2 0 1 2 3 4 5w3 4 2 2 0 1 2 3 4

NC4

w4 5 4 3 2 0 1 2 3w5 6 5 4 3 2 0 1 2

PC5

w6 7 2 5 4 3 2 0 1w7 8 2 6 5 4 3 2 0

Pct6 w8 9 8 7 6 5 4 3 2

CC1 VC2PC3 NC4 PC5

VC2 PC3


NC4 PC5 CC1

Pct6

Pct6



31


matrix


w0 w1 w2 w3 w4 w5 w6 w7 w8

<s>

0 1 2 3 4 5 6 7 8

CC1 w0 0 0 0 0 0 5 6 7VC2 w1 2 0 1 0 0 4 5 6

PC3

w2 3 2 0 1 2 3 4 0w3 4 2 2 0 1 2 3 0

NC4

w4 5 4 3 2 0 1 2 3w5 6 5 4 3 2 0 1 2

PC5

w6 7 2 5 4 3 2 0 1w7 8 2 6 5 4 3 2 0

Pct6 w8 9 8 7 6 5 4 3 2

CC1 VC2PC3 NC4 PC5

VC2 PC3


NC4 PC5 CC1

Pct6

Pct6



32


matrix


w0 w1 w2 w3 w4 w5 w6 w7 w8

<s>

0 1 2 3 4 5 6 7 8

CC1 w0 0 0 0 0 0 5 6 7VC2 w1 2 0 1 0 0 4 5 6

PC3

w2 3 2 0 1 2 3 4 0w3 4 2 2 0 1 2 3 0

NC4

w4 5 4 3 2 0 1 2 3w5 6 5 4 3 2 0 1 2

PC5

w6 7 2 5 4 3 2 0 1w7 8 2 6 5 4 3 2 0

Pct6 w8 9 8 7 6 5 4 3 2

CC1 VC2PC3 NC4 PC5

VC2 PC3


NC4 PC5 CC1

Pct6

Pct6



33

Experiments

• Tasks: NIST-MT09 for Ar-En, WMT10 for De-En

• Systems based on Moses, include state-of-the-art hierarchical lexicalized reordering models [Tillmann 04; Koehn & al 05; Galley & Manning 08]

• Baseline Distortion Limits: 5 in Ar-En, 10 in De-En

• Evaluation by: - BLEU for lexical match & local order - KRS for global order



35

Arabic-English:

Test set: eval09-NW

Distortion modified with 3-best reorderings per rule-matching sequence

Translation QualityTranslation Time

+0.9 BLEU+0.6 KRS(signif.)



37

German-English:

Test set: newstest10

Distortion modified with 3-best reorderings per rule-matching sequence

Translation QualityTranslation Time

+0.4 BLEU+0.7 KRS(signif.)



38

Conclusions

• Modified distortion allows for finer & linguistically motivated definition of search space

• We achieve better translation & faster decoding in language pairs where long reordering concentrates on few patterns

• Our method is complementary to reordering modeling

• For now, few reordering rules are needed to modify distortion

• We are currently working on a fully data-driven approach to replace the rules



39

w0 w1 w2 w3 w4 w5 w6 w7 w8 w9 w1

0

<s>

0 1 2 3 4 5 6 7 8 9 10

w0 0 1 2 3 4 5 6 7 8 9w1 2 0 1 2 3 4 5 6 7 8w2 3 T 0 1 2 3 4 5 6 7w3 4 H 2 0 1 2 3 Y 5 6w4 5 A T T E N T I O N !w5 6 N 4 3 2 0 1 U 3 4w6 7 K 5 4 3 2 F O R 2 3w7 8 S 6 5 4 3 2 0 1 2w8 9 8 7 6 5 4 3 2 0 1w9 10 9 8 7 6 5 4 3 2 0w10 11 10 9 8 7 6 5 4 3 2



40

w0 w1 w2 w3 w4 w5 w6 w7 w8 w9 w1

0

<s>

0 1 2 3 4 5 6 7 8 9 10

w0 0 1 2 3 4 5 6 7 8 9w1 2 0 1 2 3 4 5 6 7 8w2 3 T 0 1 2 3 4 5 6 7w3 4 H 2 0 1 2 3 Y 5 6w4 5 A T T E N T I O N !w5 6 N 4 3 2 0 1 U 3 4w6 7 K 5 4 3 2 F O R 2 3w7 8 S 6 5 4 3 2 0 1 2w8 9 8 7 6 5 4 3 2 0 1w9 10 9 8 7 6 5 4 3 2 0w10 11 10 9 8 7 6 5 4 3 2

modified distortion matrices for phrase-based smt arianna bisazza & marcello federico – fbk...

Documents