Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processingand the 9th International Joint Conference on Natural Language Processing, pages 877–886,Hong Kong, China, November 3–7, 2019. c©2019 Association for Computational Linguistics

877

Context-Aware Monolingual Repair for Neural Machine Translation

Elena Voita1,2 Rico Sennrich4,3 Ivan Titov3,2

1Yandex, Russia 2University of Amsterdam, Netherlands3University of Edinburgh, Scotland 4University of Zurich, Switzerland

lena-voita@yandex-team.rusennrich@cl.uzh.ch ititov@inf.ed.ac.uk

Abstract

Modern sentence-level NMT systems oftenproduce plausible translations of isolated sen-tences. However, when put in context, thesetranslations may end up being inconsistentwith each other. We propose a monolingualDocRepair model to correct inconsistenciesbetween sentence-level translations. DocRe-pair performs automatic post-editing on a se-quence of sentence-level translations, refiningtranslations of sentences in context of eachother. For training, the DocRepair model re-quires only monolingual document-level datain the target language. It is trained as amonolingual sequence-to-sequence model thatmaps inconsistent groups of sentences intoconsistent ones. The consistent groups comefrom the original training data; the inconsis-tent groups are obtained by sampling round-trip translations for each isolated sentence.We show that this approach successfully imi-tates inconsistencies we aim to fix: using con-trastive evaluation, we show large improve-ments in the translation of several contex-tual phenomena in an English→Russian trans-lation task, as well as improvements in theBLEU score. We also conduct a human eval-uation and show a strong preference of theannotators to corrected translations over thebaseline ones. Moreover, we analyze whichdiscourse phenomena are hard to capture us-ing monolingual data only.1

1 Introduction

Machine translation has made remarkableprogress, and studies claiming it to reach a humanparity are starting to appear (Hassan et al., 2018).However, when evaluating translations of thewhole documents rather than isolated sentences,human raters show a stronger preference for

1The code and data sets (including round-trip translations)are available at https://github.com/lena-voita/good-translation-wrong-in-context.

human over machine translation (Läubli et al.,2018). These findings emphasize the need to shifttowards context-aware machine translation bothfrom modeling and evaluation perspective.

Most previous work on context-aware NMT as-sumed that either all the bilingual data is avail-able at the document level (Jean et al., 2017; Wanget al., 2017; Tiedemann and Scherrer, 2017; Baw-den et al., 2018; Voita et al., 2018; Maruf andHaffari, 2018; Agrawal et al., 2018; Kuang et al.,2018; Miculicich et al., 2018) or at least its frac-tion (Voita et al., 2019). But in practical scenarios,document-level parallel data is often scarce, whichis one of the challenges when building a context-aware system.

We introduce an approach to context-awaremachine translation using only monolingualdocument-level data. In our setting, a sep-arate monolingual sequence-to-sequence model(DocRepair) is used to correct sentence-leveltranslations of adjacent sentences. The key idea isto use monolingual data to imitate typical incon-sistencies between context-agnostic translationsof isolated sentences. The DocRepair model istrained to map inconsistent groups of sentencesinto consistent ones. The consistent groups comefrom the original training data; the inconsistentgroups are obtained by sampling round-trip trans-lations for each isolated sentence.

To validate the performance of our model, weuse three kinds of evaluation: the BLEU score,contrastive evaluation of translation of several dis-course phenomena (Voita et al., 2019), and humanevaluation. We show strong improvements for allmetrics.

We analyze which discourse phenomena arehard to capture using monolingual data only. Us-ing contrastive test sets for targeted evaluation ofseveral contextual phenomena, we compare theperformance of the models trained on round-trip

878

translations and genuine document-level paralleldata. Among the four phenomena in the test setswe use (deixis, lexical cohesion, VP ellipsis andellipsis which affects NP inflection) we find VPellipsis to be the hardest phenomenon to be cap-tured using round-trip translations.

Our key contributions are as follows:

• we introduce the first approach to context-aware machine translation using only mono-lingual document-level data;

• our approach shows substantial improve-ments in translation quality as measured byBLEU, targeted contrastive evaluation of sev-eral discourse phenomena and human evalu-ation;

• we show which discourse phenomena arehard to capture using monolingual data only.

2 Our Approach: Document-level Repair

We propose a monolingual DocRepair modelto correct inconsistencies between sentence-leveltranslations of a context-agnostic MT system. Itdoes not use any states of a trained MT modelwhose outputs it corrects and therefore can in prin-ciple be trained to correct translations from anyblack-box MT system.

The DocRepair model requires only monolin-gual document-level data in the target language. Itis a monolingual sequence-to-sequence model thatmaps inconsistent groups of sentences into consis-tent ones. Consistent groups come from mono-lingual document-level data. To obtain inconsis-tent groups, each sentence in a group is replacedwith its round-trip translation produced in isola-tion from context. More formally, forming a train-ing minibatch for the DocRepair model involvesthe following steps (see also Figure 1):

1. sample several groups of sentences from themonolingual data;

2. for each sentence in a group, (i) translate itusing a target-to-source MT model, (ii) sam-ple a translation of this back-translated sen-tence in the source language using a source-to-target MT model;

3. using these round-trip translations of isolatedsentences, form an inconsistent version of theinitial groups;

Figure 1: Training procedure of DocRepair. First,round-trip translations of individual sentences are pro-duced to form an inconsistent text fragment (in the ex-ample, both genders of the speaker and the cat becameinconsistent). Then, a repair model is trained to pro-duce an original text from the inconsistent one.

Figure 2: The process of producing document-leveltranslations at test time is two-step: (1) sentences aretranslated independently using a sentence-level model,(2) DocRepair model corrects translation of the result-ing text fragment.

4. use inconsistent groups as input for theDocRepair model, consistent ones as output.

At test time, the process of getting document-level translations is two-step (Figure 2):

1. produce translations of isolated sentences us-ing a context-agnostic MT model;

2. apply the DocRepair model to a sequenceof context-agnostic translations to correct in-consistencies between translations.

In the scope of the current work, the DocRe-pair model is the standard sequence-to-sequenceTransformer. Sentences in a group are concate-nated using a reserved token-separator betweensentences.2 The Transformer is trained to correctthese long inconsistent pseudo-sentences into con-sistent ones. The token-separator is then removedfrom corrected translations.

3 Evaluation of Contextual Phenomena

We use contrastive test sets for evaluation of dis-course phenomena for English-Russian by Voitaet al. (2019). These test sets allow for testing dif-ferent kinds of phenomena which, as we show, can

2In preliminary experiments, we observed that this per-forms better than concatenating sentences without a separa-tor.

879

distancetotal 1 2 3

deixis 3000 1000 1000 1000lex. cohesion 2000 855 630 515ellipsis (infl.) 500ellipsis (VP) 500

Table 1: Size of test sets: total number of test instancesand with regard to the distance between sentences re-quiring consistency (in the number of sentences). Forellipsis, the two sets correspond to whether a model hasto predict correct NP inflection, or correct verb sense(VP ellipsis).

be captured from monolingual data with varyingsuccess. In this section, we provide test sets statis-tics and briefly describe the tested phenomena. Formore details, the reader is referred to Voita et al.(2019).

3.1 Test sets

There are four test sets in the suite. Each testset contains contrastive examples. It is specifi-cally designed to test the ability of a system toadapt to contextual information and handle thephenomenon under consideration. Each test in-stance consists of a true example (a sequence ofsentences and their reference translation from thedata) and several contrastive translations whichdiffer from the true one only in one specific aspect.All contrastive translations are correct and plau-sible translations at the sentence level, and onlycontext reveals the inconsistencies between them.The system is asked to score each candidate trans-lation, and we compute the system accuracy asthe proportion of times the true translation is pre-ferred to the contrastive ones. Test set statistics areshown in Table 1. The suites for deixis and lexicalcohesion are split into development and test sets,with 500 examples from each used for validationpurposes and the rest for testing. Convergence ofboth consistency scores on these development setsand BLEU score on a general development set areused as early stopping criteria in models training.For ellipsis, there is no dedicated development set,so we evaluate on all the ellipsis data and do notuse it for development.

3.2 Phenomena overview

Deixis Deictic words or phrases, are referential ex-pressions whose denotation depends on context.

This includes personal deixis (“I”, “you”), placedeixis (“here”, “there”), and discourse deixis,where parts of the discourse are referenced (“that’sa good question”). The test set examples are all re-lated to person deixis, specifically the T-V distinc-tion between informal and formal you (Latin “tu”and “vos”) in the Russian translations, and test forconsistency in this respect.

Ellipsis Ellipsis is the omission from a clauseof one or more words that are nevertheless under-stood in the context of the remaining elements.In machine translation, elliptical constructions inthe source language pose a problem in two situa-tions. First, if the target language does not allowthe same types of ellipsis, requiring the elided ma-terial to be predicted from context. Second, if theelided material affects the syntax of the sentence.For example, in Russian the grammatical functionof a noun phrase, and thus its inflection, may de-pend on the elided verb, or, conversely, the verbinflection may depend on the elided subject.

There are two different test sets for ellipsis. Onecontains examples where a morphological form ofa noun group in the last sentence can not be under-stood without context beyond the sentence level(“ellipsis (infl.)” in Table 1). Another includescases of verb phrase ellipsis in English, whichdoes not exist in Russian, thus requires predictingthe verb when translating into Russian (“ellipsis(VP)” in Table 1).

Lexical cohesion The test set focuses on reiter-ation of named entities. Where several translationsof a named entity are possible, a model has to pre-fer consistent translations over inconsistent ones.

4 Experimental Setup

4.1 Data preprocessing

We use the publicly available OpenSubtitles2018corpus (Lison et al., 2018) for English and Rus-sian. For a fair comparison with previous work,we train the baseline MT system on the data re-leased by Voita et al. (2019). Namely, our MT sys-tem is trained on 6m instances. These are sentencepairs with a relative time overlap of subtitle framesbetween source and target language subtitles of atleast 0.9.

We gathered 30m groups of 4 consecutive sen-tences as our monolingual data. We used only doc-uments not containing groups of sentences fromgeneral development and test sets as well as fromcontrastive test sets. The main results we report

880

are for the model trained on all 30m fragments.We use the tokenization provided by the cor-

pus and use multi-bleu.perl3 on lowercaseddata to compute BLEU score. We use beam searchwith a beam of 4.

Sentences were encoded using byte-pair encod-ing (Sennrich et al., 2016b), with source and tar-get vocabularies of about 32000 tokens. Trans-lation pairs were batched together by approxi-mate sequence length. Each training batch con-tained a set of translation pairs containing approx-imately 150004 source tokens. It has been shownthat Transformer’s performance depends heavilyon batch size (Popel and Bojar, 2018), and wechose a large batch size to ensure the best perfor-mance. In training context-aware models, for earlystopping we use both convergence in BLEU scoreon the general development set and scores on theconsistency development sets. After training, weaverage the 5 latest checkpoints.

4.2 Models

The baseline model, the model used for back-translation, and the DocRepair model are allTransformer base models (Vaswani et al., 2017).More precisely, the number of layers is N = 6with h = 8 parallel attention layers, or heads. Thedimensionality of input and output is dmodel =512, and the inner-layer of a feed-forward net-works has dimensionality dff = 2048. Weuse regularization as described in Vaswani et al.(2017).

As a second baseline, we use the two-passCADec model (Voita et al., 2019). The first passproduces sentence-level translations. The secondpass takes both the first-pass translation and rep-resentations of the context sentences as input andreturns contextualized translations. CADec re-quires document-level parallel training data, whileDocRepair only needs monolingual training data.

4.3 Generating round-trip translations

On the selected 6m instances we train sentence-level translation models in both directions. To cre-ate training data for DocRepair, we proceed as fol-lows. The Russian monolingual data is first trans-lated into English, using the Russian→English

3https://github.com/moses-smt/mosesdecoder/tree/master/scripts/generic

4This can be reached by using several of GPUs or by ac-cumulating the gradients for several batches and then makingan update.

model BLEU

baseline 33.91CADec 33.86sentence-level repair 34.12DocRepair 34.60

Table 2: BLEU scores. For CADec, the original imple-mentation was used.

model and beam search with beam size of 4. Then,we use the English→Russian model to sampletranslations with temperature of 0.5. For eachsentence, we precompute 20 sampled translationsand randomly choose one of them when forming atraining minibatch for DocRepair. Also, in train-ing, we replace each token in the input with a ran-dom one with the probability of 10%.

4.4 OptimizerAs in Vaswani et al. (2017), we use the Adam opti-mizer (Kingma and Ba, 2015), the parameters areβ1 = 0.9, β2 = 0.98 and ε = 109. We vary thelearning rate over the course of training using theformula:

lrate = scale ·min(step_num0.5,

step_num · warmup_steps1.5),

where warmup_steps = 16000 and scale = 4.

5 Results

5.1 General resultsThe BLEU scores are provided in Table 2 (weevaluate translations of 4-sentence fragments). Tosee which part of the improvement is due to fixingagreement between sentences rather than simplysentence-level post-editing, we train the same re-pair model at the sentence level. Each sentencein a group is now corrected separately, then theyare put back together in a group. One can see thatmost of the improvement comes from accountingfor extra-sentential dependencies. DocRepair out-performs the baseline and CADec by 0.7 BLEU,and its sentence-level repair version by 0.5 BLEU.

5.2 Consistency resultsScores on the phenomena test sets are provided inTable 3. For deixis, lexical cohesion and ellip-sis (infl.) we see substantial improvements overboth the baseline and CADec. The largest im-provement over CADec (22.5 percentage points)

881

model deixis lex. c. ell. infl. ell. VP

baseline 50.0 45.9 53.0 28.4CADec 81.6 58.1 72.2 80.0DocRepair 91.8 80.6 86.4 75.2

+10.2 +22.5 +14.4 -4.8

Table 3: Results on contrastive test sets for specificcontextual phenomena (deixis, lexical consistency, el-lipsis (inflection), and VP ellipsis).

Figure 3: (a) Example of a discrepancy caused by VPellipsis: correct meaning is “believe”, but MT producesсказала (“say”). (b) Example of producing round-triptranslations. From top to bottom: target, first trans-lation, round-trip translation. When translating fromRussian, main verbs are unlikely to be translated intoauxiliary ones in English, and VP ellipsis is not present.

is for lexical cohesion. However, there is a dropof almost 5 percentage points for VP ellipsis. Wehypothesize that this is because it is hard to learnto correct inconsistencies in translations causedby VP ellipsis relying on monolingual data alone.Figure 3(a) shows an example of inconsistencycaused by VP ellipsis in English. There is noVP ellipsis in Russian, and when translating aux-iliary “did” the model has to guess the main verb.Figure 3(b) shows steps of generating round-triptranslations for the target side of the previousexample. When translating from Russian, mainverbs are unlikely to be translated as the auxil-iary “do” in English, and hence the VP ellipsis israrely present on the English side. This impliesthe model trained using the round-trip translationswill not be exposed to many VP ellipsis examplesin training. We discuss this further in Section 6.2.

Table 4 provides scores for deixis and lexi-cal cohesion separately for different distances be-tween sentences requiring consistency. It can beseen, that the performance of DocRepair degradesless than that of CADec when the distance be-tween sentences requiring consistency gets larger.

distancetotal 1 2 3

deixisbaseline 50.0 50.0 50.0 50.0CADec 81.6 84.6 84.4 75.9DocRepair 91.8 94.8 93.1 87.7

+ 10.2 +10.2 +8.7 +11.8

lexical cohesionbaseline 45.9 46.1 45.9 45.4CADec 58.1 63.2 52.0 56.7DocRepair 80.6 83.0 78.5 79.4

+ 22.5 +20.2 +26.5 +22.3

Table 4: Detailed accuracy on deixis and lexical cohe-sion test sets.

all equal better worse700 367 242 90

100% 52% 35% 13%

Table 5: Human evaluation results, comparing DocRe-pair with baseline.

5.3 Human evaluationWe conduct a human evaluation on random 700examples from our general test set. We pickedonly examples where a DocRepair translation isnot a full copy of the baseline one.5

The annotators were provided an original groupof sentences in English and two translations: base-line context-agnostic one and the one correctedby the DocRepair model. Translations were pre-sented in random order with no indication whichmodel they came from. The task is to pick one ofthe three options: (1) the first translation is better,(2) the second translation is better, (3) the trans-lations are of equal quality. The annotators wereasked to avoid the third answer if they are able togive preference to one of the translations. No otherguidelines were given.

The results are provided in Table 5. In about52% of the cases annotators marked translationsas having equal quality. Among the cases whereone of the translations was marked better than theother, the DocRepair translation was marked bet-ter in 73% of the cases. This shows a strong pref-erence of the annotators for corrected translationsover the baseline ones.

5As we further discuss in Section 7, DocRepair does notchange the base translation at all in about 20% of the cases.

882

BLEU deixis lex. c. ellipsis

2.5m 34.15 89.2 75.5 81.8 / 71.65m 34.44 90.3 77.7 83.6 / 74.030m 34.60 91.8 80.6 86.4 / 75.2

Table 6: Results for DocRepair trained on differentamount of data. For ellipsis, we show inflection/VPscores.

6 Varying Training Data

In this section, we discuss the influence of thetraining data chosen for document-level models.In all experiments, we used the DocRepair model.

6.1 The amount of training data

Table 6 provides BLEU and consistency scores forthe DocRepair model trained on different amountof data. We see that even when using a dataset ofmoderate size (e.g., 5m fragments) we can achieveperformance comparable to the model trained on alarge amount of data (30m fragments). Moreover,we notice that deixis scores are less sensitive to theamount of training data than lexical cohesion andellipsis scores. The reason might be that, as we ob-served in our previous work (Voita et al., 2019), in-consistencies in translations due to the presence ofdeictic words and phrases are more frequent in thisdataset than other types of inconsistencies. Also,as we show in Section 7, this is the phenomenonthe model learns faster in training.

6.2 One-way vs round-trip translations

In this section, we discuss the limitations of us-ing only monolingual data to model inconsisten-cies between sentence-level translations. In Sec-tion 5.2 we observed a drop in performance onVP ellipsis for DocRepair compared to CADec,which was trained on parallel data. We hypoth-esized that this is due to the differences betweenone-way and round-trip translations, and now wetest this hypothesis. To do so, we fix the datasetand vary the way in which the input for DocRe-pair is generated: round-trip or one-way transla-tions. The latter assumes that document-level datais parallel, and translations are sampled from thesource side of the sentences in a group rather thanfrom their back-translations. For parallel data, wetake 1.5m parallel instances which were used forCADec training and add 1m instances from ourmonolingual data. For segments in the parallel

data deixis lex. c. ell. infl. ell. VP

one-way 85.4 63.4 79.8 73.4round-trip 84.0 61.7 78.4 67.8

Table 7: Consistency scores for the DocRepair modeltrained on 2.5m instances, among which 1.5m are par-allel instances. Compare round-trip and one-way trans-lations of the parallel part.

data BLEU deixis lex. c. ellipsis

from mon. 34.15 89.2 75.5 81.8 / 71.6from par. 33.70 84.0 61.7 78.4 / 67.8

Table 8: DocRepair trained on 2.5m instances, eitherrandomly chosen from monolingual data or from thepart where each utterance in a group has a translation.

part, we either sample translations from the sourceside or use round-trip translations. The results areprovided in Table 7.

The model trained on one-way translations isslightly better than the one trained on round-triptranslations. As expected, VP ellipsis is the hard-est phenomena to be captured using round-triptranslations, and the DocRepair model trained onone-way translated data gains 6% accuracy on thistest set. This shows that the DocRepair model ben-efits from having access to non-synthetic Englishdata. This results in exposing DocRepair at train-ing time to Russian translations which suffer fromthe same inconsistencies as the ones it will have tocorrect at test time.

6.3 Filtering: monolingual (no filtering) orparallel

Note that the scores of the DocRepair modeltrained on 2.5m instances randomly chosen frommonolingual data (Table 6) are different from theones for the model trained on 2.5m instances com-bined from parallel and monolingual data (Ta-ble 7). For convenience, we show these two inTable 8.

The domain, the dataset these two data sam-ples were gathered from, and the way we gener-ated training data for DocRepair (round-trip trans-lations) are all the same. The only difference liesin how the data was filtered. For parallel data, as inthe previous work (Voita et al., 2018), we pickedonly sentence pairs with large relative time over-lap of subtitle frames between source-languageand target-language subtitles. This is necessary to

883

ensure the quality of translation data: one needsgroups of consecutive sentences in the target lan-guage where every sentence has a reliable transla-tion.

Table 8 shows that the quality of the modeltrained on data which came from the parallel partis worse than the one trained on monolingual data.This indicates that requiring each sentence in agroup to have a reliable translation changes thedistribution of the data, which might be not benefi-cial for translation quality and provides extra mo-tivation for using monolingual data.

7 Learning Dynamics

Let us now look into how the process of DocRe-pair training progresses. Figure 4a shows how theBLEU scores with the reference translation andwith the baseline context-agnostic translation (i.e.the input for the DocRepair model) are changingduring training. First, the model quickly learns tocopy baseline translations: the BLEU score withthe baseline is very high. Then it gradually learnsto change them, which leads to an improvement inBLEU with the reference translation and a drop inBLEU with the baseline. Importantly, the modelis reluctant to make changes: the BLEU score be-tween translations of the converged model and thebaseline is 82.5. We count the number of changedsentences in every 4-sentence fragment in the testset and plot the histogram in Figure 4b. In overthan 20% of the cases the model has not changedbase translations at all. In almost 40%, it mod-ified only one sentence and left the remaining 3sentences unchanged. The model changed morethan half sentences in a group in only 14% of thecases. Several examples of the DocRepair transla-tions are shown in Figure 6.

Figure 5 shows how consistency scores arechanging in training.6 For deixis, the modelachieves the final quality quite quickly; for therest, it needs a large number of training steps toconverge.

8 Related Work

Our work is most closely related to two linesof research: automatic post-editing (APE) anddocument-level machine translation.

6Deixis and lexical cohesion scores are evaluated on thedevelopment sets which were used in training for the stoppingcriteria. Ellipsis test sets were not used at training time; thescores are shown here only for visualization purposes.

(a) (b)

Figure 4: (a) BLEU scores progression in train-ing. BLEU evaluated with the target translations andwith the context-agnostic baseline translations (whichDocRepair learns to correct). (b) Distribution in the testset of the number of changed sentences in 4-sentencefragments.

Figure 5: Consistency scores progression in training.

8.1 Automatic post-editing

Our model can be regarded as an automatic post-editing system – a system designed to fix system-atic MT errors that is decoupled from the main MTsystem. Automatic post-editing has a long history,including rule-based (Knight and Chander, 1994),statistical (Simard et al., 2007) and neural ap-proaches (Junczys-Dowmunt and Grundkiewicz,2016; Pal et al., 2016; Freitag et al., 2019).

In terms of architectures, modern approachesuse neural sequence-to-sequence models, ei-ther multi-source architectures that consider boththe original source and the baseline translation(Junczys-Dowmunt and Grundkiewicz, 2016; Palet al., 2016), or monolingual repair systems, asin Freitag et al. (2019), which is concurrent workto ours. True post-editing datasets are typi-cally small and expensive to create (Specia et al.,2017), hence synthetic training data has been cre-ated that uses original monolingual data as outputfor the sequence-to-sequence model, paired withan automatic back-translation (Sennrich et al.,2016a) and/or round-trip translation as its input(s)(Junczys-Dowmunt and Grundkiewicz, 2016; Fre-

884

Figure 6: Examples of the DocRepair translations. First is the baseline translation, then – corrected by the DocRe-pair. The differences between translations are underlined. (a) corrected wrong translation of “it”: violated genderagreement with the antecedent. (b) corrected wrong gender (marked on a verb): from the third sentence it’s clearthat the speaker in the second one is feminine (Rayna), but the baseline translation was masculine. (c) correctedwrong morphological form of the pronoun, which was not understood with the elided verb in the third sentence.

itag et al., 2019).

While previous work on automatic post-editingoperated on the sentence level, the main noveltyof this work is that our DocRepair model oper-ates on groups of sentences and is thus able to fixconsistency errors caused by the context-agnosticbaseline MT system. We consider this strategyof sentence-level baseline translation and context-aware monolingual repair attractive when paralleldocument-level data is scarce.

For training, the DocRepair model only requiresmonolingual document-level data. While we cre-ate synthetic training data via round-trip transla-tion similarly to earlier work (Junczys-Dowmuntand Grundkiewicz, 2016; Freitag et al., 2019),note that we purposefully use sentence-level MTsystems for this to create the types of consistencyerrors that we aim to fix with the context-awareDocRepair model. Not all types of consistency er-rors that we want to fix emerge from a round-triptranslation, so access to parallel document-leveldata can be useful (Section 6.2).

8.2 Document-level NMT

Neural models of MT that go beyond the sentence-level are an active research area (Jean et al., 2017;Wang et al., 2017; Tiedemann and Scherrer, 2017;Bawden et al., 2018; Voita et al., 2018; Maruf andHaffari, 2018; Agrawal et al., 2018; Miculicichet al., 2018; Kuang et al., 2018; Voita et al., 2019).Typically, the main MT system is modified to takeadditional context as its input. One limitation ofthese approaches is that they assume that paralleldocument-level training data is available.

Closest to our work are two-pass models fordocument-level NMT (Xiong et al., 2019; Voitaet al., 2019), where a second, context-aware modeltakes the translation and hidden representationsof the sentence-level first-pass model as its input.The second-pass model can in principle be trainedon a subset of the parallel training data (Voitaet al., 2019), somewhat relaxing the assumptionthat all training data is at the document level.

Our work is different from this previous workin two main respects. Firstly, we show thatconsistency can be improved with only monolin-gual document-level training data. Secondly, the

885

DocRepair model is decoupled from the first-passMT system, which improves its portability.

9 Conclusions

We introduce the first approach to context-aware machine translation using only monolin-gual document-level data. We propose a monolin-gual DocRepair model to correct inconsistenciesbetween sentence-level translations. The modelperforms automatic post-editing on a sequence ofsentence-level translations, refining translations ofsentences in context of each other. Our approachresults in substantial improvements in translationquality as measured by BLEU, targeted contrastiveevaluation of several discourse phenomena andhuman evaluation. Moreover, we perform er-ror analysis and detect which discourse phenom-ena are hard to capture using only monolingualdocument-level data. While in the current workwe used text fragments of 4 sentences, in futurework we would like to consider longer contexts.

Acknowledgments

We would like to thank the anonymous reviewersfor their comments. The authors also thank DavidTalbot and Yandex Machine Translation team forhelpful discussions and inspiration. Ivan Titovacknowledges support of the European ResearchCouncil (ERC StG BroadSem 678254) and theDutch National Science Foundation (NWO VIDI639.022.518). Rico Sennrich acknowledges sup-port from the Swiss National Science Foundation(105212_169888), the European Union’s Horizon2020 research and innovation programme (grantagreement no 825460), and the Royal Society(NAF\R1\180122).

ReferencesRuchit Agrawal, Turchi Marco, and Negri Matteo.

2018. Contextual Handling in Neural MachineTranslation: Look Behind, Ahead and on BothSides.

Rachel Bawden, Rico Sennrich, Alexandra Birch, andBarry Haddow. 2018. Evaluating Discourse Phe-nomena in Neural Machine Translation. In Proceed-ings of the 2018 Conference of the North AmericanChapter of the Association for Computational Lin-guistics: Human Language Technologies, Volume1 (Long Papers), pages 1304–1313, New Orleans,USA. Association for Computational Linguistics.

Markus Freitag, Isaac Caswell, and Scott Roy. 2019.Ape at scale and its implications on mt evaluation

biases. In Proceedings of the Fourth Conference onMachine Translation (Volume 1: Research Papers),pages 34–44, Florence, Italy. Association for Com-putational Linguistics.

Hany Hassan, Anthony Aue, Chang Chen, VishalChowdhary, Jonathan Clark, Christian Feder-mann, Xuedong Huang, Marcin Junczys-Dowmunt,William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu,Renqian Luo, Arul Menezes, Tao Qin, Frank Seide,Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, YingceXia, Dongdong Zhang, Zhirui Zhang, and MingZhou. 2018. Achieving human parity on auto-matic chinese to english news translation. InarXiv:1803.05567. ArXiv: 1803.05567.

Sebastien Jean, Stanislas Lauly, Orhan Firat, andKyunghyun Cho. 2017. Does Neural MachineTranslation Benefit from Larger Context? InarXiv:1704.05135. ArXiv: 1704.05135.

Marcin Junczys-Dowmunt and Roman Grundkiewicz.2016. Log-linear combinations of monolingual andbilingual neural machine translation models for au-tomatic post-editing. In Proceedings of the FirstConference on Machine Translation, pages 751–758, Berlin, Germany. Association for Computa-tional Linguistics.

Diederik Kingma and Jimmy Ba. 2015. Adam: Amethod for stochastic optimization. In Proceedingsof the International Conference on Learning Repre-sentation (ICLR 2015).

Kevin Knight and Ishwar Chander. 1994. Automatedpostediting of documents. In Proceedings of theTwelfth National Conference on Artificial Intelli-gence (Vol. 1), AAAI ’94, pages 779–784, MenloPark, CA, USA. American Association for ArtificialIntelligence.

Shaohui Kuang, Deyi Xiong, Weihua Luo, andGuodong Zhou. 2018. Modeling coherence forneural machine translation with dynamic and topiccaches. In Proceedings of the 27th InternationalConference on Computational Linguistics, pages596–606, Santa Fe, New Mexico, USA. Associationfor Computational Linguistics.

Samuel Läubli, Rico Sennrich, and Martin Volk. 2018.Has Machine Translation Achieved Human Parity?A Case for Document-level Evaluation. In Proceed-ings of the 2018 Conference on Empirical Methodsin Natural Language Processing, pages 4791–4796.Association for Computational Linguistics.

Pierre Lison, Jörg Tiedemann, and Milen Kouylekov.2018. Opensubtitles2018: Statistical rescoring ofsentence alignments in large, noisy parallel corpora.In Proceedings of the Eleventh International Confer-ence on Language Resources and Evaluation (LREC2018), Miyazaki, Japan.

Sameen Maruf and Gholamreza Haffari. 2018. Docu-ment context neural machine translation with mem-ory networks. In Proceedings of the 56th Annual

886

Meeting of the Association for Computational Lin-guistics (Volume 1: Long Papers), pages 1275–1284, Melbourne, Australia. Association for Com-putational Linguistics.

Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas,and James Henderson. 2018. Document-level neuralmachine translation with hierarchical attention net-works. In Proceedings of the 2018 Conference onEmpirical Methods in Natural Language Process-ing, pages 2947–2954, Brussels, Belgium. Associ-ation for Computational Linguistics.

Santanu Pal, Sudip Kumar Naskar, Mihaela Vela, andJosef van Genabith. 2016. A neural network basedapproach to automatic post-editing. In Proceed-ings of the 54th Annual Meeting of the Associationfor Computational Linguistics (Volume 2: Short Pa-pers), pages 281–286, Berlin, Germany. Associationfor Computational Linguistics.

Martin Popel and Ondrej Bojar. 2018. Training Tipsfor the Transformer Model. pages 43–70.

Rico Sennrich, Barry Haddow, and Alexandra Birch.2016a. Improving neural machine translation mod-els with monolingual data. In Proceedings of the54th Annual Meeting of the Association for Compu-tational Linguistics (Volume 1: Long Papers), pages86–96, Berlin, Germany. Association for Computa-tional Linguistics.

Rico Sennrich, Barry Haddow, and Alexandra Birch.2016b. Neural machine translation of rare wordswith subword units. In Proceedings of the 54th An-nual Meeting of the Association for ComputationalLinguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computa-tional Linguistics.

Michel Simard, Cyril Goutte, and Pierre Isabelle. 2007.Statistical phrase-based post-editing. In HumanLanguage Technologies 2007: The Conference ofthe North American Chapter of the Association forComputational Linguistics; Proceedings of the MainConference, pages 508–515, Rochester, New York.Association for Computational Linguistics.

Lucia Specia, Kim Harris, Frédéric Blain, AljoschaBurchardt, Vivien Macketanz, Inguna Skadin, a, Mat-teo Negri, , and Marco Turchi. 2017. Translationquality and productivity: A study on rich morphol-ogy languages. In Machine Translation Summit XVI,pages 55–71. Asia-Pacific Association for MachineTranslation.

Jörg Tiedemann and Yves Scherrer. 2017. Neural Ma-chine Translation with Extended Context. In Pro-ceedings of the Third Workshop on Discourse inMachine Translation, DISCOMT’17, pages 82–92,Copenhagen, Denmark. Association for Computa-tional Linguistics.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N Gomez, Lukasz

Kaiser, and Illia Polosukhin. 2017. Attention is allyou need. In NeurIPS, Los Angeles.

Elena Voita, Rico Sennrich, and Ivan Titov. 2019.When a good translation is wrong in context:Context-aware machine translation improves ondeixis, ellipsis, and lexical cohesion. In Proceedingsof the 57th Annual Meeting of the Association forComputational Linguistics, pages 1198–1212, Flo-rence, Italy. Association for Computational Linguis-tics.

Elena Voita, Pavel Serdyukov, Rico Sennrich, and IvanTitov. 2018. Context-aware neural machine trans-lation learns anaphora resolution. In Proceedings ofthe 56th Annual Meeting of the Association for Com-putational Linguistics (Volume 1: Long Papers),pages 1264–1274, Melbourne, Australia. Associa-tion for Computational Linguistics.

Longyue Wang, Zhaopeng Tu, Andy Way, and QunLiu. 2017. Exploiting Cross-Sentence Context forNeural Machine Translation. In Proceedings of the2017 Conference on Empirical Methods in Natu-ral Language Processing, EMNLP’17, pages 2816–2821, Denmark, Copenhagen. Association for Com-putational Linguistics.

Hao Xiong, Zhongjun He, Hua Wu, and Haifeng Wang.2019. Modeling coherence for discourse neural ma-chine translation. Proceedings of the AAAI Confer-ence on Artificial Intelligence, 33(01):7338–7345.