Proceedings of the First Conference on Machine Translation, Volume 1: Research Papers, pages 74–82,Berlin, Germany, August 11-12, 2016. c©2016 Association for Computational Linguistics

Using Factored Word Representationin Neural Network Language Models

Jan Niehues, Thanh-Le Ha, Eunah Cho and Alex WaibelInstitute for Anthropomatics

Karlsruhe Institute of Technology, Germanyfirstname.secondname@kit.edu

Abstract

Neural network language and translationmodels have recently shown their great po-tentials in improving the performance ofphrase-based machine translation. At thesame time, word representations using dif-ferent word factors have been translationquality and are part of many state-of-the-art machine translation systems. used inmany state-of-the-art machine translationsystems, in order to support better transla-tion quality.

In this work, we combined these two ideasby investigating the combination of bothtechniques. By representing words in neu-ral network language models using differ-ent factors, we were able to improve themodels themselves as well as their impacton the overall machine translation perfor-mance. This is especially helpful for mor-phologically rich languages due to theirlarge vocabulary size. Furthermore, it iseasy to add additional knowledge, such assource side information, to the model.

Using this model we improved the trans-lation quality of a state-of-the-art phrase-based machine translation system by 0.7BLEU points. We performed experimentson three language pairs for the news trans-lation task of the WMT 2016 evaluation.

1 Introduction

Recently, neural network models are deployed ex-tensively for better translation quality of statisti-cal machine translation (Le et al., 2011; Devlin etal., 2014). For the language model as well as forthe translation model, neural network-based mod-els showed improvements when used during de-coding as well as when used in re-scoring.

In phrase-based machine translation (PBMT),word representation using different factors (Koehnand Hoang, 2007) are commonly used in state-of-the-art systems. Using Part-of-Speech (POS)information or automatic word clusters is es-pecially important for morphologically rich lan-guages which often have a large vocabulary size.Language models based on these factors are ableto consider longer context and therefore improvethe modelling of the overall structure. Further-more, the POS information can be used to improvethe modelling of word agreement, which is often adifficult task when handling morphologically richlanguages.

Until now, word factors have been used rela-tively limited in neural network models. Auto-matic word classes have been used to structure theoutput layer (Le et al., 2011) and as input in feedforward neural network language models (Niehuesand Waibel, 2012).

In this work, we propose a multi-factor recur-rent neural network (RNN)-based language modelthat is able to facilitate all available informationabout the word in the input as well as in the out-put. We evaluated the technique using the surfaceform, POS-tag and automatic word clusters usingdifferent cluster sizes.

Using this model, it is also possible to integratesource side information into the model. By usingthe model as a bilingual model, the probability ofthe translation can be modelled and not only theone of target sentence. As for the target side, weuse a factored representation for the words on thesource side.

The remaining of the paper is structured as fol-lowing: In the following section, we first reviewthe related work. Afterwards, we will shortly de-scribe the RNN-based language model used in ourexperiments. In Section 4, we will introduce thefactored RNN-based language model. In the next

74

section, we will describe the experiments on theWMT 2016 data. Finally, we will end the paperwith a conclusion of the work.

2 Related Work

Additional information about words, encoded asword factors, e.g. the lemma of word, POStags, etc., is employed in state-of-the-art phrase-based systems. (Koehn and Hoang, 2007) decom-poses the translation of factored representationsto smaller mapping steps, which are modelled bytranslation probabilities from input factor to out-put factor or by generating probabilities of addi-tional output factors from existing output factors.Then those pre-computed probabilities are jointlycombined in the decoding process as a standardtranslation feature scores. In addition, languagemodels using these word factors have shown tobe very helpful to improve the translation qual-ity. In particular, the aligned-words, POS or wordclasses are used in the framework of modern lan-guage models (Mediani et al., 2011; Wuebker etal., 2013).

Recently, neural network language models havebeen considered to perform better than standardn-gram language models (Schwenk, 2007; Le etal., 2011). Especially the neural language modelsconstructed in recurrent architectures have showna great performance by allowing them to take alonger context into account (Mikolov et al., 2010;Sundermeyer et al., 2013).

In a different direction, there has been a greatdeal of research on bringing not only target wordsbut also source words into the prediction process,instead of predicting the next target word based onthe previous target words (Le et al., 2012; Devlinet al., 2014; Ha et al., 2014).

However, to the best of our knowledge, wordfactors have been exploited in a relatively limitedscope of neural network research. (Le et al., 2011;Le et al., 2012) use word classes to reduce theoutput layer’s complexity of such networks, bothin language and translation models. In the workof (Niehues and Waibel, 2012), their RestrictedBoltzmann Machines language models also en-code word classes as an additional input feature inpredicting the next target word. (Tran et al., 2014)use two separate feed forward networks to predictthe target word and its corresponding suffixes withthe source words and target stem as input features.

Our work exhibits several essential differences

from theirs. Firstly, we leverage not only the targetmorphological information but also word factorsfrom both source and target sides in our models.Furthermore, we could use as many types of wordfactors as we can provide. Thus, we are able tomake the most of the information encoded in thosefactors for more accurate prediction.

3 Recurrent Neural Network-basedLanguage Models

In contrast to feed forward neural network-basedlanguage models, recurrent neural network-basedlanguage models are able to store arbitrary longword sequences. Thereby, they are able to directlymodel P (w|h) and no approximations by limitingthe history size are necessary. Recently, severalauthors showed that RNN-based language modelscould perform very well in phrase-based machinetranslation. (Mikolov et al., 2010; Sundermeyer etal., 2013)

In this work, we used the torch71 implementa-tion of an RNN-based language model (Leonardet al., 2015). First, the words were mapped totheir word embeddings. We used an input embed-ding size of 100. Afterwards, we used two LSTM-based layers. The first has the size of the wordembeddings and for the second we used a hiddensize of 200. Finally, the word probabilities werecalculated using a softmax layer.

The models were trained using stochastic gra-dient descent. The weights were updated usingmini-batches with a batch size of 128. We useda maximum epoch size of 1 million examples andselected the model with the lowest perplexity onthe development data.

4 Factored Language Model

When using factored representation of words,words are no longer represented as indices in theneural network. Instead, they are represented a tu-ples of indices w = (f1, . . . , fD), where D is thenumber of different factors used to describe theword. These factors can be the word itself, as wellas the POS, automatic learned classes (Och, 1999)or other information about the word. Furthermore,we can use different types of factors for the inputand the output of the neural network.

1http://torch.ch/

75

Figure 1: Factored RNN Layout

4.1 Input Representation

In a first step, we obtained a factored representa-tion for the input of the neural network. In theexperiments, we represented a word by its surfaceform, POS-tags and automatic word class, but theframework can be used for any number of wordfactors. Although there are factored approachesfor n-gram based language models (Bilmes andKirchhoff, 2003), most n-gram language modelsonly use one factor. In contrast, in neural networkbased language models, it is very easy to add ad-ditional information as word factors. We can learndifferent embeddings for each factor and representthe word by concatenating the embeddings of sev-eral factors. As shown in the bottom of Figure 1,we first project the different factors to the contin-uous factor embeddings. Afterwards, we concate-nate these embeddings into a word embedding.

The advantage of using several word factors isthat we can use different knowledge sources torepresent a word. When a word occurs very rarely,the learned embedding from its surface form mightnot be helpful. The additional POS information,however, is very helpful. While using POS-basedlanguage models in PBMT may lead to losing theinformation about high frequent words, in this ap-proach we can have access to all information byconcatenating the factor embeddings.

4.2 Output Representation

In addition to use different factors in the input ofthe neural network, we can also use different fac-tors on the output. In phrase-based machine trans-lation, n-gram language models based on POS-tags have been shown to be very successful formorphologically rich languages.

Porting this idea to neural network lan-guage models, we can not only train amodel to predict the original word f1 giventhe previous words in factor representationh = (f1,1, . . . , f1,D), . . . , (fi,1, . . . , fi,D), butalso train a model to predict he POS-tags (e.g. f2)given the history h.

In a first step, we proposed to train individualmodels for all factors 1, . . . , D generating proba-bilities P1, . . . , PD for every sentence. Then theseprobabilities can be used as features for examplein re-scoring of the phrase-based MT system.

Considering that it can be helpful to considerall factors of the word in the input, it can be alsohelpful to jointly train the models for predictingthe different output factors. This is motivated bythe fact that multi-task learning has shown to bebeneficial in several NLP tasks (Collobert et al.,2011). Predicting all output features jointly re-quires a modification of the output layer of theRNN model. As shown in Figure 1, we replace thesingle mapping from the LSTM-layer to the soft-max layer, by D mappings. Each mapping thenlearns to project the LSTM-layer output to the fac-tored output probabilities. In the last layer, we useD different softmax units. In a similar way as theconventional network, the error between the out-put of the network and the reference is calculatedduring training.

Using this network, we will no longer pre-dict the probability of one word factor Pd, d ∈{1, . . . D}, but D different probability distribu-tions P1, . . . , PD. In order to integrate this modelinto the machine translation system we exploredtwo different probabilities. First, we used only thejoint probability P =

∏Dd=1 Pd as a feature in the

log-linear combination. In addition, we also usedthe joint probability as well as all individual prob-abilities Pd as features.

4.3 Bilingual Model

Using the model presented before, it is possible toadd additional information to the model as well.One example we explored in this work is to use

76

Figure 2: Bilingual Model

the model as a bilingual model (BM). Instead ofusing only monolingual information by consider-ing the previous target factors as input, we usedsource factors additionally. Thereby, we can nowmodel the probability of a word given the previ-ous target words and information about the sourcesentence. So in this case we model the transla-tion probability and no longer the language modelprobability.

When predicting the target word wi+1 with itsfactors fi+1,1, . . . , fi+1,D, the input to the RNNis the previous target word wi = fi,1, . . . , fi,D.Using the alignment, we can find the source wordsa(i+1), which is aligned to the target word wi+1.When we add the features of source word

sa(i+1) = (fsa(i+1),1, . . . , f

sa(i+1),Ds

)

to the ones of the target word wi and create a newbilingual token

bi = (fi+1,1, . . . , fi+1,D, fsa(i+1),1, . . . , f

sa(i+1),Ds

)

, we now can predict the target word given the pre-vious target word and the aligned source word.

In the example in Figure 2, we wouldinsert (completed,VVD,87,ein,ART) to predict(a,DT,37).

In this case the number of input factors and out-put factors are no longer the same. In the input,we have D+Ds input factors, while we have onlyD factors on the output of the network.

5 Experiments

We evaluated the factored RNNLM on three dif-ferent language pairs of the WMT 2016 NewsTranslation Task. In each language pair, we cre-ated an n-best list using our phrase-based MT sys-tem and used the factored RNNLM as an addi-tional feature in rescoring. It is worth noting that

the POS and word class information are alreadypresent during decoding of the baseline system byn-gram-based language models based on each ofthese factors. First, we performed a detailed analy-sis on the English-Romanian task. In addition, weused the model in a German-English and English-German translation system. In all tasks, we usedthe model in re-scoring of a PBMT system.

5.1 System Description

The baseline system is an in-house implementa-tion of the phrase-based approach. The systemused to generate n-best lists for the news tasks istrained on all the available training corpora of theWMT 2015 Shared Translation task. The systemuses a pre-reordering technique (Rottmann andVogel, 2007; Niehues and Kolss, 2009; Herrmannet al., 2013) and facilitates several translation andlanguage models. As shown in Table 1, we usetwo to three word-based language models and oneto two cluster-based models using 50, 100 or 1,000clusters. The custers were trained as describedin (Och, 1999). In addition, we used a POS-based language model in the English-Romainiansystem and a bilingual language model (Niehueset al., 2011) in English to German and Germanto English systems. The POS tags for English-Romanian were generated by the tagger describedin (Ion et al., 2012) and the ones for German byRFTagger (Schmid and Laws, 2008).

Table 1: Features

EN-RO EN-DE DE-ENwordLM 2 3 3POSLM 1 0 0clusterLM 2 1 2BiLM 0 1 1#features 22-23 20 22

In addition, we used discriminative word lex-ica (Niehues and Waibel, 2013) during decodingand source discriminative word lexica in rescoring(Herrman et al., 2015).

A full system description can be found in (Ha etal., 2016).

The German to English baseline system uses 20features and the English to German systems uses22 features.

The English-Romanian system was optimizedon the first part of news-dev2016 and the rescor-ing was optimized on this set and a subset of 2,000

77

sentences from the SETimes corpus. This part ofthe corpus was of course excluded for training themodel. The system was tested on the second halfof news-dev2016.

The English-German and German-English sys-tems were optimized on news-test2014 and alsothe re-scoring was optimized on this data. Wetested the system on news-test2015.

For English to Romanian and English to Ger-man we used an n-best List of 300 entries andfor German to English we used an n-best list with3,000 entries.

For decoding, for all language directions, theweights of the system were optimized using mini-mum error rate training (Och, 2003). The weightsin the rescoring were optimized using the List-Net algorithm (Cao et al., 2007) as described in(Niehues et al., 2015).

The RNN-based language models for English toRomanian and German to English were trained onthe target side of the parallel training data. For En-glish to German, we trained the model and the Eu-roparl corpus and the News commentary corpus.

5.2 English - Romanian

In the first experiment on the English to Roma-nian task, we only used the scores of the RNN lan-guage models. The baseline system has a BLEUscore (Papineni et al., 2002) of 29.67. Using onlythe language model instead of the 22 features, ofcourse, leads to a lower performance, but we cansee clear difference between the different languagemodels. All systems use a word vocabulary of 5Kwords and we used four different factors. We usedthe word surface form, the POS tags and wordclusters using 100 and 1,000 classes.

The baseline model using words as input andwords as output reaches a BLEU score of 27.88.If we instead represent the input words by factors,we select entries from the n-best list that gener-ates a BLEU score of 28.46. As done with then-gram language models, we can also predict theother factors instead of the words themselves. Inall cases, we use all four factors as input factors.As shown in Table 2, all models except for theone with 100 classes perform similarly, reachingup between 28.46 and 28.49. The language modelpredicting only 100 classes only reaches a BLEUscore of 28.23. It suggests that this number ofclasses is too low to disambiguate the entries inthe n-best list.

Table 2: English - Romanian Single Score

Input Prediction SingleWord Word 27.88All factors Word 28.46All factors POS 28.48All factors 100 Cl. 28.23All factors 1,000 Cl. 28.49All factors All factors 28.54

If we predict all factors together and use thenthe joint probability, we can reach the best BLEUscore of 28.54 as shown in the last line of the ta-ble. This is 0.7 BLEU points better than the initialword based model.

After evaluating the model as the only knowl-edge source, we also performed experiments usingthe model in combination with the other models.We evaluated the baseline and the best model inthree different configuration in Table 3 using onlythe joint probability. The three baseline configu-ration differ in the models used during decoding.Thereby, we are able to generate different n-bestlists and test the models on different conditions.

Table 3: English - Romanian Language Models

Model Conf1 Conf2 Conf3Baseline 29.86 30.00 29.75LM 5K 29.79 29.84 29.73LM 50K 29.64 29.84 29.83Factored LM 5K 29.94 30.01 30.01Factored LM 50K 30.05 30.27 30.29

In Table 3, we tested the word-based and thefactored language model using a vocabulary of 5Kand 50K words. Features from each model areused in addition to the features of the baseline sys-tem. As shown in the table, the word-based RNNlanguage models perform similarly, but both couldnot improve over the baseline system. One possi-ble reason for this is that we already use severallanguage models in the baseline model and theyare partly trained on much larger data. While theRNN models are trained using only the target lan-guage model, one word-based language model istrained on the Romanian common crawl corpus.Furthermore, the POS-based and word cluster lan-guage models use a 9-gram history and therefore,can already model quite long dependencies.

But if we use a factored language model, we are

78

able to improve over the baseline system. Usingthe additional information of the other word fac-tors, we are able to improve the bilingual model inall situations. The model using a surface word vo-cabulary of 5,000 words can improve by 0.1 to 0.3BLEU points. The model using a 50K vocabularycan even improve by up to 0.6 BLEU points.

Table 4: English - Romanian Bilingual Models

Model Dev TestBaseline 40.12 29.75+ Factored LM 50K 40.87 30.17+ Factored BM 5K 41.11 30.44+ Factored BM 50K 41.16 30.57

After analyzing the different language models,we also evaluate how we can use the factoredrepresentation to include source side information.The results are summarized in Table 4. In theseexperiments, we used not only the the joint proba-bility, but also the four individual probabilities asfeatures. Therefore, we will add five scores forevery model, since each model is added to its pre-vious configuration in this experiment.

Exploiting all five probabilities of the lan-guage model brought us the similar improvementwe achieved using the joint probability from themodel. On the test set, the improvements areslightly worse. When adding the model usingsource side information based on a vocabulary of5K and 50K words, however, we get additional im-provements. Adopting the both bilingual models(BM) along with a factored LM, we improved theBLEU score further leading up to the best score of30.57 for the test set.

5.3 English - GermanIn addition to the experiments on English to Ro-manian, we also evaluated the models on the taskof translating English News to German. For theEnglish to German system, we use three factorson the source side and four factors on the tar-get side. In English, we used the surface formsas well as automatic word cluster based on 100and 1,000 classes. On the target side, we usedfine-graind POS-tags generated by the RFTagger(Schmid and Laws, 2008), in addition to the fac-tors for the source side.

The experiments using only the scores of themodel are summarized in Table 5. In this exper-iment, we analyzed a word based- and a factored

Table 5: English - German Single Score

Model SingleLM 5K 20.92Factored LM 5K 21.69BM 5K 21.33Factored BM 5K 21.92

language models as well as bilingual models. Asdescribed in section 4.3, the difference betweenthe language model and the bilingual model is thatthe latter uses the source side information as addi-tional factor.

Using only the word-based language model weachieved a BLEU score of 20.92. Deploying a fac-tored language model instead, we can improve theBLEU score by 0.7 BLEU points to 21.69. Whilewe achieved a score of 21.33 BLEU points by us-ing a proposed bilingual model, we improved thescore up to 21.92 BLEU points by adopting all fac-tors for the bilingual model.

Table 6: English-German Language Model

Model Conf1 Conf2Baseline 23.25 23.40Factored LM 5K 23.63 23.77Factored BM 5K 23.43 23.48

In addition to the analysis on the single model,we also evaluated the model’s influence by com-bining the model with the baseline features. Wetested the language model as well as the bilingualmodel on two different configurations. Adoptingthe factored language model on top of the base-line features improved the translation quality byaround 0.4 BLEU points for both configurations,as shown in Table 6. Although the bilingual modelcould also improve the translation quality, it couldnot outperform the factored language model. Thecombination of the two models, LM and BM, didnot lead to further improvements. In summary,the factored language model improved the BLEUscore by 0.4 points.

5.4 German - English

Similar experiments were conducted on the Ger-man to English translation task. For this languagepair, we built models using a vocabulary size of5,000 words. The models cover word surfaceforms and two automatic word clusters, which are

79

based on 100 and 1,000 word classes respectively.First, we will evaluate the performance of the sys-tem using only this model in rescoring. The resultsare summarized in Table 7.

Table 7: German - English Single Score

Model SingleLM 5K 26.11Factored LM 5K 26.96BM 5K 26.77Factored BM 5K 26.81

The word based language model achieves aBLEU score 26.11. Extending the model to in-clude factors improves the BLEU score by 0.8BLEU points to 26.96. If we use a bilingualmodel, a word based model achieves a BLEUscore of 26.77 and the factored one a BLEU scoreof 26.81. Although the factored model performedbetter than the word-based models, in this case thebilingual model cannot outperform the languagemodel.

Table 8: German - English Language Model

Model SingleBaseline 29.33+ Factored BM 5K 29.51+ Factored LM 5K 29.66

In a last series of experiments, we used thescores combined with the baseline scores. The re-sults are shown in Table 8. In this language pair,we can improve over the baseline system by usingboth models. The final BLEU score is 0.3 BLEUpoints better than the initial system.

6 Conclusion

In this paper, we presented a new approach tointegrate additional word information into a neu-ral network language model. This model is es-pecially promising for morphologically rich lan-guages. Due to their large vocabulary size, addi-tional information such as POS-tags are expectedto model rare words effectively.

Representing words using factors has been suc-cessfully deployed in many phrase-based machinetranslation systems. Inspired by this, we repre-sented each word in our neural network languagemodel using factors, facilitating all available in-formation of the word. We showed that using the

factored neural network language models can im-prove the quality of a phrase-based machine trans-lation system, which already uses several factoredlanguage models.

In addition, the presented framework allows aneasy integration of source side information. Byincorporating the alignment information to thesource side, we were able to model the translationprocess. In this model, the source words as well asthe target words can be represented by word fac-tors.

Using these techniques, we are able to im-prove the translation system on three different lan-guage pairs of the WMT 2016 evaluation. Weperformed experiments on the English-Romanian,English-German and German-English translationtask. The suggested technique yielded up to 0.7BLEU points of improvement on all three tasks.

Acknowledgments

The project leading to this application has receivedfunding from the European Union’s Horizon 2020research and innovation programme under grantagreement n◦ 645452. The research by Thanh-Le Ha was supported by Ministry of Science, Re-search and the Arts Baden-Wurttemberg.

ReferencesJeff A. Bilmes and Katrin Kirchhoff. 2003. Factored

language models and generalized parallel backoff.In Proceedings of the 2003 Conference of the NorthAmerican Chapter of the Association for Compu-tational Linguistics on Human Language Technol-ogy: Companion Volume of the Proceedings of HLT-NAACL 2003–short Papers - Volume 2, NAACL-Short ’03, pages 4–6, Stroudsburg, PA, USA. As-sociation for Computational Linguistics.

Z. Cao, T. Qin, T.-Y. Liu, M.-F. Tsai, and H. Li. 2007.Learning to rank: from pairwise approach to listwiseapproach. In Proceedings of the 24th internationalconference on Machine learning, Icml a07, pages129–136, New York, NY, USA. Acm.

Ronan Collobert, Jason Weston, Leon Bottou, MichaelKarlen, Koray Kavukcuoglu, and Pavel P. Kuksa.2011. Natural language processing (almost) fromscratch. CoRR, abs/1103.0398.

J. Devlin, R. Zbib, Z. Huang, T. Lamar, R. Schwartz,and J. Makhoul. 2014. Fast and robust neuralnetwork joint models for statistical machine trans-lation. In Proceedings of the 52st Annual Meet-ing of the Association for Computational Linguis-tics (ACL 2014), volume 1, pages 1370–1380, Balti-more, Maryland, USA.

80

Thanh-Le Ha, Jan Niehues, and Alex Waibel. 2014.Lexical Translation Model Using a Deep NeuralNetwork Architecture. In Proceedings of the 11thInternational Workshop on Spoken Language Trans-lation (IWSLT14), Lake Tahoe, CA, USA.

Thanh-Le Ha, Eunah Cho, Jan Niehues, MohammedMediani, Matthias Sperber, Alexandre Allauzen,and yAlex Waibel. 2016. The karlsruhe instituteof technology systems for the news translation taskin wmt 2016. In Proceedings of the ACL 2016 FirstConference on Machine Translation (WMT2016).

Teresa Herrman, Jan Niehues, and Alex Waibel. 2015.Source Discriminative Word Lexicon for Transla-tion Disambiguation. In Proceedings of the 12th In-ternational Workshop on Spoken Language Transla-tion (IWSLT15), Danang, Vietnam.

Teresa Herrmann, Jan Niehues, and Alex Waibel.2013. Combining Word Reordering Methods ondifferent Linguistic Abstraction Levels for Statisti-cal Machine Translation. In Proceedings of the Sev-enth Workshop on Syntax, Semantics and Structurein Statistical Translation, Altanta, Georgia, USA.

Radu Ion, Elena Irimia, Dan Stefanescu, and Dan Tufis.2012. Rombac: The romanian balanced annotatedcorpus. In Nicoletta Calzolari (Conference Chair),Khalid Choukri, Thierry Declerck, Mehmet U?urDo?an, Bente Maegaard, Joseph Mariani, Asun-cion Moreno, Jan Odijk, and Stelios Piperidis, ed-itors, Proceedings of the Eight International Con-ference on Language Resources and Evaluation(LREC’12), Istanbul, Turkey, may. European Lan-guage Resources Association (ELRA).

Philipp Koehn and Hieu Hoang. 2007. Factored trans-lation models. In In Proceedings of the 2007 JointConference on Empirical Methods in Natural Lan-guage Processing and Computational Natural Lan-guage Learning (EMNLP-CoNLL, pages 868–876.

Hai-Son Le, Ilya Oparin, Alexandre Allauzen, Jean-Luc Gauvain, and Francois Yvon. 2011. Structuredoutput layer neural network language model. In Pro-ceedings of the International Conference on Audio,Speech and Signal Processing, pages 5524–5527.

Hai-Son Le, Alexandre Allauzen, and Francois Yvon.2012. Continuous Space Translation Models withNeural Networks. In Proceedings of the 2012 con-ference of the north american chapter of the asso-ciation for computational linguistics: Human lan-guage technologies (NAACL), pages 39–48. Associ-ation for Computational Linguistics.

Nicholas Leonard, Sagar Waghmare, Yang Wang, andJin-Hwa Kim. 2015. rnn : Recurrent library fortorch. CoRR, abs/1511.07889.

Mohammed Mediani, Eunah Cho, Jan Niehues, TeresaHerrmann, and Alex Waibel. 2011. The KITenglish-french translation systems for IWSLT 2011.

In 2011 International Workshop on Spoken Lan-guage Translation, IWSLT 2011, San Francisco, CA,USA, December 8-9, 2011, pages 73–78.

Tomas Mikolov, Martin Karafiat, Lukas Burget, JanCernocky, and Sanjeev Khudanpur. 2010. Recur-rent neural network based language model. In IN-TERSPEECH, volume 2, page 3.

Jan Niehues and Muntsin Kolss. 2009. A POS-BasedModel for Long-Range Reorderings in SMT. InProceedings of the Fourth Workshop on StatisticalMachine Translation (WMT 2009), Athens, Greece.

J. Niehues and A. Waibel. 2012. Continuous spacelanguage models using restricted boltzmann ma-chines. In Proceedings of the Ninth InternationalWorkshop on Spoken Language Translation (IWSLT2012), Hong Kong.

Jan Niehues and Alex Waibel. 2013. An MT Error-Driven Discriminative Word Lexicon using Sen-tence Structure Features. In Proceedings of theEighth Workshop on Statistical Machine Transla-tion, Sofia, Bulgaria.

Jan Niehues, Teresa Herrmann, Stephan Vogel, andAlex Waibel. 2011. Wider Context by Using Bilin-gual Language Models in Machine Translation. InSixth Workshop on Statistical Machine Translation(WMT 2011), Edinburgh, Scotland, United King-dom.

Jan Niehues, Quoc Khanh Do, Alexandre Allauzen,and Alex Waibel. 2015. Listnet-based MT Rescor-ing. EMNLP 2015, page 248.

Franz Josef Och. 1999. An Efficient Method for Deter-mining Bilingual Word Classes. In Proceedings ofthe Ninth Conference of the European Chapter of theAssociation for Computational Linguistics (EACL1999), Bergen, Norway.

F.J. Och. 2003. Minimum error rate training in sta-tistical machine translation. In Proceedings of the41th Annual Meeting of the Association for Compu-tational Linguistics (ACL 2003), Sapporo, Japa.

K. Papineni, S. Roukos, T. Ward, and W.-jing Zhu.2002. Bleu: a method for automatic evaluation ofmachine translation. In Proceedings of the 40th An-nual Meeting of the Association for ComputationalLinguistics (ACL 2002), pages 311–318, Philadel-phia, Pennsylvania.

Kay Rottmann and Stephan Vogel. 2007. Word Re-ordering in Statistical Machine Translation with aPOS-Based Distortion Model. In Proceedings of the11th International Conference on Theoretical andMethodological Issues in Machine Translation (TMI2007), Skovde, Sweden.

Helmut Schmid and Florian Laws. 2008. Estimationof Conditional Probabilities with Decision Trees andan Application to Fine-Grained POS Tagging. InProceedings of the 22nd International Conference

81

on Computational Linguistics, Manchester, UnitedKingdom.

Holger Schwenk. 2007. Continuous space languagemodels. Computer Speech & Language, 21(3):492–518.

Martin Sundermeyer, Ilya Oparin, Jean-Luc Gauvain,Ben Freiberg, Ralf Schluter, and Hermann Ney.2013. Comparison of feedforward and recur-rent neural network language models. In Acous-tics, Speech and Signal Processing (ICASSP), 2013IEEE International Conference on, pages 8430–8434. IEEE.

Ke Tran, Arianna Bisazza, Christof Monz, et al. 2014.Word translation prediction for morphologically richlanguages with bilingual neural networks. Associa-tion for Computational Linguistics.

Joern Wuebker, Stephan Peitz, Felix Rietig, and Her-mann Ney. 2013. Improving statistical machinetranslation with word class models. In Conferenceon Empirical Methods in Natural Language Pro-cessing, pages 1377–1381, Seattle, WA, USA, Oc-tober.

82