Microsoft Research Faculty Summit 2008Microsoft Research Faculty Summit 2008

Statistical Machine Translation Research at MSR

Robert MoorePrincipal ResearcherMicrosoft Research

A Brief History of MT Research in the U.S.

1950s – mid 1960sMT based on ad-hoc, hand-written rules and lexicons

1966ALPAC report kills MT research funding in U.S. for 20 years

Late 1980s – early 1990sDARPA restarts US funding for MT researchIBM speech recognition group lays foundation for statistical MT

2000sDARPA greatly expands funding for MT research (TIDES, GALE)Statistical MT becomes the hottest research topic in NLP

Sketch of Current Baseline Method for SMT

Sentence-align and word-align a parallel bilingual corpusExtract translation pairs of contiguous phrasesOptimize weights of a linear model for scoring a source-string, target-string, phrase-alignment triple, combining

Sum of logarithms of phrase-translation probabilitiesLogarithm of target language string probabilityWord reordering penaltyOther features

Translate a source language string by trying to find the corresponding target language string and phrase alignment that score the highest according to the linear model

Approaches to Sentence and Word Alignment

Tendencies that can be leveraged in statistical alignment

Lengths of sentences that are translations are highly correlatedWords that are translations co-occur in sentence translation pairs far more often than chanceN-to-M word or sentence translations are possible, but less probable the more N and M differWord reordering is possible, but less probable the more extreme the reordering

These tendencies can be reflected in the structure of statistical alignment models, whose parameters can be trained by machine learning methods

Some Issues with Baseline Approach

Exact search for the best translation is NP-hard, and even approximate search can be slowString-based reordering models are very weak, and work poorly for language pairs that have substantially different word ordersTaking words as minimal units is problematic with heavily inflected languages, especially translation into such languagesVariations in models that make some translations better usually make others worse; can we combine systems to get advantages of multiple approaches?

MSR Work Addressing These Problems

Translation speed – Moore and Quirk developed improved pruning methods for a widely-used beam search algorithm, producing up to order-of-magnitude speed-upsWord order – Quirk, Menezes, and Cherry developed a tree-to-string translation model that leverages a dependency parse of the source sentence to make more intelligent ordering decisionsHeavily inflected languages – Toutanova et al. developed methods for explicitly modeling word inflection that significantly improve translation into such languages

MSR Work on MT System Combination

Overall approachTake N-best translations from several systemsPick one translation as a “backbone”Word-align all other translations to the backboneUse alignments to determine a set of alternative words for each position in the combined output (in backbone order)Choose a word from each set of alternatives by weighted vote of system outputs, plus a statistical target language model

Our main innovationHe et al. developed a probabilistic alignment method that consistently out-performs previous edit-distance based methods

System Combination Results

A combination of 8 systems fromMSR RedmondMSR AsiaSRI InternationalCanadian National Research Council

obtained the highest score in the NIST 2008 Open MT Evaluation on the constrained training track for Chinese-to-English (BLEU = 31.0 vs. 26.2 for best single system)Adding 7 more systems from ISI/LW and BBN substantially improved the system combination (BLEU = 34.9 vs. 29.9 for best single system)

Sample Chinese-to-English Translations

Reference translationRelief officials said today that in total it had caused the deaths of 43 people

15-system combinationRescue officials said today that a total of 43 people were killed

Single system 1Rescue officials said today that killed 43 people

Single system 2Rescue officials said today that a total of 43 people were killed

Sample Chinese-to-English Translations

Reference translationwhite house pushes for nuclear inspectors to be sent as soon as possible to monitor north korea's closure of its nuclear reactors

15-system combinationthe white house urges nuclear inspection and supervision north korea shut down the nuclear reaction as soon as possible to

Single system 1white house urges nuclear north korea as soon as possible to close the furnace

Single system 2the white house urged to send nuclear inspection to monitor north korea nuclear reactor shut down as soon as possible

Sample Chinese-to-English Translations

Reference translationfrom june to september every year , strong winds and heavy rains brought by the monsoon often cause widespread flood disasters and even human casualties in india , a country with a population of 1.1 billion .

15-system combinationfrom june to september every year , strong winds brought about by monsoon rains often lead to india which has a population of 1.1 billion , serious floods in the country and even causing casualties .

Single system 1every monsoon winds and rains often leads to india from june to september , causing serious floods in the country with a population of 1.1 billion , and even causing casualties .

Single system 2the flood swamped , causing casualties every year from june to september , strong winds and heavy rains brought by the monsoon often lead to india which has a population of 1.1 billion .

Future Directions

Continuing to work in most of the areas mentioned aboveExtending other approaches to syntax-based translation based on synchronous probabilistic context-free grammars, seeking

Faster translation algorithmsBetter translation models

Exploring “minimum Bayes risk” approaches to choosing the best translationDeveloping discriminative, syntax-based target language models for MT