Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 190–199Brussels, Belgium, October 31 - November 1, 2018. c©2018 Association for Computational Linguistics

190

Bidirectional Generative Adversarial Networks for NeuralMachine Translation

Zhirui Zhang†∗, Shujie Liu§, Mu Li¶, Ming Zhou§, Enhong Chen††University of Science and Technology of China, Hefei, China

§Microsoft Research Asia†zrustc11@gmail.com †cheneh@ustc.edu.cn

§{shujliu,mingzhou}@microsoft.com ¶limugx@outlook.com

Abstract

Generative Adversarial Network (GAN) hasbeen proposed to tackle the exposure biasproblem of Neural Machine Translation(NMT). However, the discriminator typicallyresults in the instability of the GAN train-ing due to the inadequate training problem:the search space is so huge that sampledtranslations are not sufficient for discrimina-tor training. To address this issue and stabi-lize the GAN training, in this paper, we pro-pose a novel Bidirectional Generative Adver-sarial Network for Neural Machine Transla-tion (BGAN-NMT), which aims to introducea generator model to act as the discriminator,whereby the discriminator naturally considersthe entire translation space so that the inade-quate training problem can be alleviated. Tosatisfy this property, generator and discrimina-tor are both designed to model the joint prob-ability of sentence pairs, with the differencethat, the generator decomposes the joint prob-ability with a source language model and asource-to-target translation model, while thediscriminator is formulated as a target lan-guage model and a target-to-source transla-tion model. To further leverage the symme-try of them, an auxiliary GAN is introducedand adopts generator and discriminator mod-els of original one as its own discriminator andgenerator respectively. Two GANs are alter-nately trained to update the parameters. Exper-iment results on German-English and Chinese-English translation tasks demonstrate that ourmethod not only stabilizes GAN training butalso achieves significant improvements overbaseline systems.

1 Introduction

The past several years have witnessed the rapid de-velopment of Neural Machine Translation (NMT)

∗This work was done when the first author was the internat Microsoft Research Asia.

(Cho et al., 2014; Sutskever et al., 2014; Bahdanauet al., 2014), from catching up with Statistical Ma-chine Translation (SMT) (Koehn et al., 2003; Chi-ang, 2007) to outperforming it by significant mar-gins on many languages (Sennrich et al., 2016;Gehring et al., 2017; Vaswani et al., 2017; Has-san et al., 2018). The most common approachto training NMT is to maximize the conditionallog-probability of the correct translation given thesource sentence. However, as argued in Bengioet al. (2015), the Maximum Likelihood Estimation(MLE) principle suffers from so-called exposurebias in the inference stage: the model predicts nexttoken conditional on its previously predicted onesthat may be never observed in the training data. Toaddress this problem, much recent work attemptsto reduce the inconsistency between training andinference, such as adopting sequence-level objec-tives and directly maximizing BLEU scores (Ben-gio et al., 2015; Ranzato et al., 2015; Shen et al.,2016; Wiseman and Rush, 2016).

Generative Adversarial Network (GAN) (Good-fellow et al., 2014) is another promising frame-work for alleviating exposure bias problem and re-cently shows remarkable promise in NMT (Yanget al., 2017; Wu et al., 2017). Formally, GANconsists of two ”adversarial” models: a generatorand a discriminator. In machine translation, NMTmodel is used as the generator that produces trans-lation candidates given a source sentence, and an-other neural network is introduced to serve as thediscriminator, which takes sentence pairs as inputand distinguishes whether a given sentence pair isreal or generated. Adversarial training betweenthe two models involves optimizing a min-max ob-jective, in which, the discriminator learns to dis-tinguish whether a given data instance is real orfake, and the generator learns to confuse the dis-criminator by generating high-quality translationcandidates. Since the generated data is based on

191

discrete symbols (words), we usually adopt policygradient method (Yu et al., 2017) to update modelparameters of the generator. Specifically, given abunch of translation candidates sampled from thegenerator, confidence scores calculated by the dis-criminator are employed as rewards to update thegenerator.

However, in this training process, the discrim-inator typically suffers from inadequate trainingproblem, leading to the instability of GAN train-ing. In practice, sampling large translation candi-dates is time-consuming for NMT system, so weonly use a few samples to train the discriminator.For a given source sentence, there is usually onlyone positive example (real target sentence). If thesampled negative examples are also few, the dis-criminator will easily overfit to the data. This isthe inadequate training problem for the discrimi-nator. In such a case, rewards calculated by thediscriminator could be biased, especially for unob-served samples. These biased rewards will providea wrong signal to the generator and make it up-date incorrectly, resulting in performance degra-dation of the generator. Since such issue can oc-cur repeatedly throughout the entire training pro-cess, GAN training becomes unstable and the per-formance of generator will drop drastically.

On the other hand, the generator has well-designed properties that benefit the discriminator,since it models the probability distribution overthe entire translation space so that the genera-tor does not overfit to observed samples, whileprior knowledge for unobserved samples is natu-rally considered. At the same time, the generatoralso exhibits a certain ability to identify whethera given data instance is good enough. For ex-ample, target-to-source translation model servesas the discriminator to improve source-to-targettranslation model (He et al., 2016; Tu et al., 2017).Inspired by this, we propose a novel BidirectionalGenerative Adversarial Network for Neural Ma-chine Translation (aka BGAN-NMT), which em-ploys a generator model to perform the role ofthe discriminator so as to handle inadequate train-ing problem and stabilize GAN training. To sat-isfy this property, both generator and discriminatorof original GAN are designed to model the jointprobability of sentence pairs, with the differencethat, the generator model A is decomposed intoa source language model and a source-to-targettranslation model, while the discriminator model

B is formulated as a target language model and atarget-to-source translation model. Intuitively, wecan also leverage A to act as the discriminator toimprove B, and then improved B reversely servesas a better discriminator to guide the training ofA.To make use of this symmetry, we bring in an aux-iliary GAN that adopts generator and discrimina-tor models of original one as its own discriminatorand generator respectively. Then we design a jointtraining algorithm to alternately utilize these twoGANs to update the source-to-target and target-to-source translation models.

Our experiments are conducted on German-English and Chinese-English translation data sets.Experimental results demonstrate that our BGAN-NMT not only achieves the stability of GAN train-ing but also significantly improves translation per-formance over baseline systems.

2 Background

2.1 Neural Machine Translation

Attention-based NMT model (Bahdanau et al.,2014) is adopted as the source-to-target and target-to-source translation models used in our BGAN-NMT. The attention-based NMT system is im-plemented as an encoder-decoder framework withrecurrent neural networks (RNN), which can beGated Recurrent Unit (GRU) (Cho et al., 2014)or Long Short-Term Memory (LSTM) (Hochreiterand Schmidhuber, 1997) networks in practice.

2.1.1 Encoder-Decoder FrameworkThe encoder reads the source sentence x =(x1, x2, ... , xT ) and transforms it into a sequenceof hidden states h = (h1, h2, ... , hT ) using a bi-directional RNN. The decoder uses another RNNto generate the translation y = (y1, y2, ... , yT ′)based on the hidden states h. At each time stampi, the conditional probability of each word yi froma target vocabulary Vy is computed with

p(yi|y<i, h) = g(yi−1, zi, ci), (1)

where zi is the ith hidden state of the decoder,which is calculated conditioned on the previoushidden state zi−1, previous word yi−1 and thesource context vector ci:

zi = RNN(zi−1, yi−1, ci), (2)

The source context vector ci is a weighted sum ofthe hidden states (h1, h2, ... , hT ) with the coeffi-

192

cients α1, α2, ... , αT calculated with

αt =exp (a(ht, zi−1))∑k exp (a(hk, zi−1))

(3)

where a is a feed-forward neural network with asingle hidden layer.

2.1.2 MLE TrainingNMT systems are usually trained to maximize theconditional log-probability of the correct transla-tion given a source sentence with respect to theparameters θ of the model:

θ∗ = argmaxθ

N∑n=1

|yn|∑i=1

log p(yni |yn<i, xn) (4)

where N is size of the training corpus, and |yn|is the length of the target sentence yn. However,MLE training suffers from exposure bias prob-lem: in training stage, the history of any targetword is correct and has been observed in the train-ing data, while during testing, the model predictsnext token conditioned on its previously predictedones that may be never observed in the trainingdata. To solve this problem, reinforcement learn-ing methods are used to sample translation candi-dates, based on which, rewards are calculated andutilized to update the parameters. GAN followsthe same way to solve exposure bias problem andrewards are computed by the discriminator.

2.2 Generative Adversarial Network

As a new paradigm of training generative models,GAN (Goodfellow et al., 2014) has been success-fully applied in computer vision tasks (Radfordet al., 2015; Arjovsky et al., 2017). Conceptu-ally, GAN consists of two “adversarial” models:a generator G that captures the data distribution,and a discriminator D that estimates the probabil-ity that a sample is sampled from the training datarather than from G. When GAN is used for NMT,NMT model is employed as G, and CNN-basedor RNN-based neural networks serve as D (Yanget al., 2017; Wu et al., 2017). During adversarialtraining,G andD play a two-player minmax gamewith the following value function V (D,G):

minG

maxD

V (D,G) = E(x,y)∼Pd(x,y) [logD(x, y)]

+ E(x,y′)∼PG(x,y)

[log(1−D(x, y′))

](5)

where (x, y) is a sentence pair, Pd represents thedata distribution and PG denotes the generator dis-tribution. With this objective function, the dis-criminator learns to distinguish whether sentencepair is real (sampled from bilingual corpus) or fake(generated by G), and the generator tries to con-fuse the discriminator by generating high-qualitytranslation samples.

In practice, policy gradient method (Yu et al.,2017) is usually used to calculate gradients for thegenerator due to discrete symbols (words). To up-date the generator model, translation candidatesare firstly sampled, for which rewards are cal-culated using the discriminator. With these re-wards, we can compute gradients and run back-propagation to update the generator. In such atraining process, real target sentence and sampledtranslation candidates are used as positive and neg-ative examples of discriminator training respec-tively. Due to the computation cost, we cannotgenerate many negative examples, so that the dis-criminator is easy to overfit. The overfitted dis-criminator will give biased signals to the generatorand make it update incorrectly, leading to the in-stability of the generator training. Wu et al. (2017)found that combining adversarial training objec-tive with MLE can significantly improve the sta-bility of generator training, which is also reportedin language model and neural dialogue generation(Lamb et al., 2016; Li et al., 2017). Actually, al-though this method leverages real translation sig-nal to guide the generator and alleviate the effectof overfitted discriminator, it cannot deal with theinadequate training problem of the discriminator,which essentially plays a more important role inGAN training.

3 Bidirectional Generative AdversarialNetwork

In GAN for NMT, the generator does not sufferfrom the inadequate training problem, because thegenerator is proposed to model probability distri-bution over the entire translation space (maximiz-ing probability of one translation candidate meansminimizing probabilities of the others). At thesame time, the generator exhibits a certain abilityto discriminate good sentence pairs, for example,target-to-source translation model is used to scoresamples generated from source-to-target transla-tion model. Thus, introducing a generator modelto perform the role of the discriminator is expected

193

𝑃(𝑥|𝑦)

(𝑥, 𝑦1)

(𝑥, 𝑦𝑚)…

NMT

LM𝑃(𝑦)

𝑃(𝑦|𝑥) NMT

LM𝑃(𝑥)

SamplesData

Source 𝑥

(𝑥, 𝑦)

𝑃(𝑥|𝑦)NMT

LM 𝑃(𝑦)

Target 𝑦

Samples(𝑥1, 𝑦)

(𝑥𝑚, 𝑦)… Data

(𝑥, 𝑦)

𝑃(𝑦|𝑥)NMT

LM 𝑃(𝑥)

Discriminator 𝐷

Generator 𝐺

GAN1

Generator 𝐺′

Discriminator 𝐷′GAN2

rewards rewards

Figure 1: The architecture of BGAN-NMT consisting of two GANs. The dotted line represents thatGAN2 adopts both generator and discriminator models of GAN1 but interchanges their roles.

to address the inadequate training problem and sta-bilize GAN training. Based on these observations,we design a Bidirectional Generative AdversarialNetwork for Neural Machine Translation, namedas BGAN-NMT.

As illustrated in Figure 1, the overall archi-tecture of BGAN-NMT consists of an originalGAN (GAN1) and an auxiliary GAN (GAN2).Both generator and discriminator of original GANare defined to model the joint probability of sen-tence pairs P (x, y). Formally, P (x, y) can bedecomposed in two equivalent ways: P (x, y) =P (x)P (y|x) and P (x, y) = P (y)P (x|y), andthey are used as generator G and discriminatorD for GAN1 respectively. Further, the generatormodel can be decomposed into a source languagemodel and a source-to-target translation model,while the discriminator can be formulated as a tar-get language model and a target-to-source trans-lation model. Auxiliary GAN (GAN2) employsG and D of GAN1 as its own discriminator D′

and generator G′ to better exploit the symmetrybetween G and D. The following of this sectiondetails the objective function and joint training al-gorithm for BGAN-NMT.

3.1 Training Objective

As G and D are defined as P (x)P (y|x) andP (y)P (x|y) respectively, the adversarial trainingobjective V (D,G) ofGAN1 in Equation 5 can be

rewritten as

minG

maxD

V (D,G) = E(x,y)∼Pd(x,y) [logP (x|y)P (y)]

+ Ex∼Pd(x),y′∼P (y|x)

[log(1− P (x|y′)P (y′))

](6)

which means, given a source sentence x, source-to-target translation model P (y|x) tries to gener-ate high quality translation y′ to fool the discrimi-nator P (x|y)P (y), while target-to-source transla-tion model P (x|y) and language model P (y) learnto distinguish translation candidates from real sen-tence pairs. In our implementations, two languagemodels P (x) and P (y) are fixed to reduce trainingcomplexity.

For discriminator D, D is trained with theground-truth sentence pair (x, y) and the gener-ated sample (x, y′) from G, respectively as posi-tive and negative examples. Formally, the objec-tive function of D is to maximize V (D,G):

LD = E(x,y)∼Pd(x,y) [logP (x|y)P (y)]

+ Ex∼Pd(x),y′∼P (y|x)

[log

(1− P (x|y′)P (y′)

)] (7)

Since P (y) is fixed, the gradient of parameter θDfor the target-to-source translation model P (x|y)is calculated as:

∂LD

∂θD= E(x,y)∼Pd(x,y)

[∂ logP (x|y)

∂θD

]+ Ex∼Pd(x),y

′∼P (y|x)[(1− 1

1− P (x|y′)P (y′)

)∂ logP (x|y′)

∂θD

](8)

194

in which ∂ logP (x|y)∂θD

is the gradient specified withstandard sequence-to-sequence NMT network.

For generator G, following Goodfellow (2016),the objective of training G is to maximize the ex-pected rewards (probability of D) instead of di-rectly minimizing V (D,G):

LG = Ex∼Pd(x),y′∼P (y|x)

[P (x|y′)P (y′)

](9)

Since the output of the generator G is a sequenceof discrete symbols (words), policy gradient isused to update the parameters, and then the gra-dient of parameter θG for source-to-target transla-tion model P (y|x) can be calculated as:

∂LG

∂θG= Ex∼Pd(x),y

′∼P (y|x)

[P (x|y′)P (y′)

∂ logP (y′|x)∂θG

](10)

By exchanging generator and discriminatormodels of GAN1, we introduce GAN2, in whichthe original G is used as the discriminator D′

and original D serves as the generator G′. Sim-ilarly, the adversarial training objective V (D′, G′)of GAN2 is defined as:

minG′

maxD′

V (D′, G′) = E(x,y)∼Pd(x,y) [logP (y|x)P (x)]

+ Ey∼Pd(y),x′∼P (x|y)

[log(1− P (y|x′)P (x′))

](11)

According to this adversarial training objective,the objective functions of D′ and G′ are definedas following:

LD′ = E(x,y)∼Pd(x,y) [logP (y|x)P (x)]

+ Ey∼Pd(y),x′∼P (x|y)

[log(1− P (y|x′)P (x′))

](12)

LG′ = Ey∼Pd(y),x′∼P (x|y)

[P (y|x′)P (x′)

](13)

where the gradients of parameters θD′ = θG forP (y|x) and θG′ = θD for P (x|y) can be respec-tively calculated as:

∂LD′

∂θG= E(x,y)∼Pd(x,y)

[∂ logP (y|x)

∂θG

]+ Ey∼Pd(y),x

′∼P (x|y)

[(1− 1

1− P (y|x′)P (x′))∂ logP (y|x′)

∂θG]

(14)

∂LG′

∂θD= Ey∼Pd(y),x

′∼P (x|y)[P (y|x′)P (x′)∂ logP (x′|y)

∂θD]

(15)

3.2 Joint Training AlgorithmIn our approach, G and D actually act as discrim-inator systems for each other in a joint trainingprocess: the generator G can be improved withthe discriminator D in GAN1, and then the en-hanced G serves as a better discriminator to guide

Algorithm 1: Joint Training Algorithm forBGAN-NMT

Input : Bilingual corpus T = {(x, y)}Nn=1;Pre-trained source-side language model P (x);Pre-trained target-side language model P (y);

Output: Well-trained models P (y|x) and P (x|y)1 Pre-train P (y|x) and P (x|y) on T with MLE principle ;2 for number of training iterations do3 for k steps do4 Get m samples {(x, y)}mi=1 from T ;5 Generate m samples {(x, y′)}mi=1 using

P (y|x) given source sentences of{(x, y)}mi=1;

6 Update the parameter θD with Equation 8 ;7 Generate m samples {(x′, y)}mi=1 using

P (x|y) given target sentences of{(x, y)}mi=1;

8 Update the parameter θG with Equation 14 ;9 end

10 Get m samples {(x, y)}mi=1 from T ;11 Generate m samples {(x, y′)}mi=1 using P (y|x)

given source sentences of {(x, y)}mi=1;12 Update the parameter θG with Equation 10 ;13 Generate m samples {(x′, y)}mi=1 using P (x|y)

given target sentences of {(x, y)}mi=1;14 Update the parameter θD with Equation 15 ;15 end

the training of D in GAN2. This training pro-cess can be iteratively carried out to obtain furtherimprovements because after each iteration both Gand D are expected to be improved with adver-sarial training. To simultaneously optimize thesetwo models, we design a joint training algorithmto learn the parameters (θG and θD) shared in twoGANs of BGAN-NMT (GAN1 and GAN2).

As shown in Algorithm 1, the whole algorithmis mainly divided into two steps. Firstly, givenparallel corpora T = {(x, y)}Nn=1, we pre-trainP (y|x) and P (x|y) with MLE principle, whilesource and target language models P (x) and P (y)are pre-trained with corresponding sentences ofbilingual data. Next, based on these pre-trainedmodels, we implement the two player minmaxgame using an iterative approach, in which, wealternate between k (equals to 5 in our experi-ments) steps of optimizing all discriminators (Dand D′) and one step of optimizing all generators(G and G′). The iterative training continues untilthe performance of a development data set is notincreased.

4 Experiments

4.1 SetupTo examine the effectiveness of our proposed ap-proach, we conduct experiments on translation

195

tasks with two language pairs: German-English(De-En for in short) and Chinese-English (Zh-Enfor in short). In all experiments, BLEU (Papineniet al., 2002) is adopted as the automatic metric fortranslation quality evaluation and computed usingMoses multi-bleu.perl script.

4.1.1 DatasetFor German-English translation task, followingprevious work (Ranzato et al., 2015; Bahdanauet al., 2016), we select data from German-Englishmachine translation track of IWSLT2014 evalua-tion tasks, which consists of sentence-aligned sub-titles of TED and TEDx talks. We closely followthe pre-processing as described in Ranzato et al.(2015). The training corpus contains 153k sen-tence pairs with 2.83M English words and 2.68MGerman words. The validation set comprises of6,969 sentence pairs taken from the training data,and the test set is a combination of dev2010,dev2012, tst2010, tst2011 and tst2012 with totalnumber of 6,750 sentence pairs.

For Chinese-English translation task, trainingdata consists of a set of LDC datasets1, which hasaround 2.6M sentence pairs with 65.1M Chinesewords and 67.1M English words respectively. Anysentence longer than 80 words is removed fromtraining data. NIST OpenMT 2006 evaluation setis used as the validation set, and NIST 2005, 2008,2012 datasets as test sets. We limit the vocabu-lary to contain up to 50K most frequent words onboth source and target sides, and convert remain-ing words into the <unk> token.

4.1.2 Model ArchitectureRNNSearch model proposed by Bahdanau et al.(2014) is leveraged to be the translation model,but it should be noted that our BGAN-NMT isindependent of the NMT network structure. Weuse a single layer GRU for encoder and decoder.For Zh-En, the size of word embedding (for bothsource and target words) is 256 and the size of hid-den layer is set to 1024. For De-En, in order tocompare with previous work (Ranzato et al., 2015;Bahdanau et al., 2016), the size of word embed-ding and GRU hidden state are both set to 256. Inaddition, P (x) and P (y) are designed as a single-layer GRU language model, which is pre-trained

1 LDC2002E17, LDC2002E18, LDC2003E07,LDC2003E14, LDC2005E83, LDC2005T06, LDC2005T10,LDC2006E17, LDC2006E26, LDC2006E34, LDC2006E85,LDC2006E92, LDC2006T06, LDC2004T08, LDC2005T10

Methods Baseline ModelMIXER (Ranzato et al., 2015) 20.10 21.81MRT (Shen et al., 2016) - 25.84BSO (Wiseman and Rush, 2016) 24.03 26.36Adversarial-NMT (Wu et al., 2017) - 27.94A-C (Bahdanau et al., 2016) 27.56 28.53Softmax-Q (Ma et al., 2017) 27.66 28.77Adversarial-NMT* 27.63 28.03BGAN-NMT 27.63 29.17

Table 1: Comparison with previous work onIWSLT2014 German-English translation task.The “Baseline” means the performance of pre-trained model used to warmly start training.

to compute the marginal probability of a sentence,and the size of word embedding and GRU hiddenstate are the same as RNNSearch model.

4.1.3 Training DetailsFor the training of BGAN-NMT, parameters arefirstly initialized using a normal distribution witha mean of 0 and a variance of

√6/(drow + dcol),

where drow and dcol are the number of rowsand columns in the structure (Glorot and Ben-gio, 2010). Then we pre-train NMT and lan-guage models with MLE principle to convergence,and select the best model according to the per-formances on the validation set, where BLEUscores and the perplexity are adopted as evaluationmetrics for NMT and language models respec-tively. Both generator and discriminator models inBGAN-NMT are warmly started with those pre-trained models, and optimized using the vanillaSGD algorithm with mini-batch 32 for De-En and128 for Zh-En. We re-normalize gradients if theirnorm exceeds 2.0. The initial learning rate is setas 0.2 for De-En and 0.02 for Zh-En, and it ishalved when BLEU scores on the validation set donot increase for 20,000 batches. To generate thesynthetic bilingual data, beam search strategy withbeam size 4 is adopted for both De-En and Zh-En.At test time, beam search is employed to find thebest translation with beam size 8 and translationprobabilities normalized by the length of the can-didate translations. Follow Luong et al. (2015),<unk> is replaced with the corresponding targetword in a post processing step.

4.2 Results on German-English Translation

For German-English translation task, in additionto the baseline system which is used to warmlystart our BGAN-NMT training, we also include

196

System NIST2006 NIST2005 NIST2008 NIST2012 AverageHPSMT 32.46 32.42 25.23 26.20 29.08

RNNSearch 38.61 38.31 30.04 28.48 33.86Adversarial-NMT* 39.79 38.81 31.86 30.19 35.16

BGAN-NMT 40.74 39.20 33.55 31.30 36.19

Table 2: Case-insensitive BLEU scores (%) on Chinese-English translation. The “Average” denotes theaverage results of all datasets.

22

23

24

25

26

27

28

29

30

31

32

10000 20000 30000 40000 50000 60000 70000 80000 90000

BLEU

Mini-Batches

RNNSearch Adversarial-NMT* BGAN-NMT

Figure 2: The BLEU score changes on IWSLT2014German-English validation set for RNNSearch,Adversarial-NMT* and BGAN-NMT as trainingprogresses.

results of other existing NMT systems, includ-ing MIXER (Ranzato et al., 2015), MRT (Shenet al., 2016)2, BSO (Wiseman and Rush, 2016),Adversarial-NMT (Wu et al., 2017), A-C (Bah-danau et al., 2016) and Softmax-Q (Ma et al.,2017). Besides, following Wu et al. (2017),we also implement Adversarial-NMT* systemwhich combines adversarial training objectivewith MLE. All the results are reported based oncase-sensitive BLEU.

From Table 1, we can see that our BGAN-NMT achieves significant improvements over thebaseline RNNSearch system. It demonstrates thatGAN framework can alleviate exposure bias prob-lem and improve the robustness of NMT sys-tems. Our BGAN-NMT also obtains satisfac-tory translation quality against other existing NMTsystems. In particular, our BGAN-NMT outper-forms Adversarial-NMT* by 1.14 BLEU points.Adversarial-NMT* adopts MLE to stabilize thetraining of generator but gains limited improve-ment due to the inadequate training problem of thediscriminator, while our BGAN-NMT can effec-tively handle this issue and obtain significant im-provement.

2The result of MRT method is taken from Wu et al. (2017)

To better analyze training process of the differ-ent methods, we compare the BLEU score changeson IWSLT2014 German-English validation setfor RNNSearch, Adversarial-NMT* and BGAN-NMT during the entire training. As illustratedin Figure 2, initialized with the best RNNSearchmodel, Adversarial-NMT* and BGAN-NMT cancontinually improve the translation performance.In addition, our BGAN-NMT steadily performsmuch better than Adversarial-NMT* in the wholetraining process. It confirms that our proposed ap-proach not only stabilizes GAN training but alsoachieves better results.

4.3 Results on Chinese-English Translation

We also conduct experiments on Chinese-Englishtranslation task with strong SMT and NMT base-lines: HPSMT, RNNSearch and Adversarial-NMT*. HPSMT is an in-house implementation ofthe hierarchical phrase-based MT system (Chiang,2007), where a 4-gram language model is trainedusing the modified Kneser-Ney smoothing algo-rithm over the target data from bilingual data.

Table 2 shows the evaluation results of differ-ent models on NIST datasets. All the resultsare reported based on case-insensitive BLEU. Wecan observe that RNNSearch significantly outper-forms HPSMT by 4.78 BLEU points on average,and BGAN-NMT can further improve the perfor-mances, with 2.33 BLEU points on average. Ad-ditionally, our BGAN-NMT gains better perfor-mances than Adversarial-NMT* with 1.03 BLEUpoints on average. These experimental results con-firm the effectiveness of our proposed approach,similar as shown in the German-English transla-tion task.

4.4 Effect on Long Sentences

Longer source sentence implies longer translationthat more easily suffers from exposure bias prob-lem. In this subsection, we group source sen-tence of similar length together and calculate the

197

20

22

24

26

28

30

32

34

36

38

40

[0,10) [10,20) [20,30) [30,40) [40,50) [50,60)

BLEU

Source sentence length

RNNSearch Adversarial-NMT* BGAN-NMT

(a) German-English Translation

20

22

24

26

28

30

32

34

[0,10) [10,20) [20,30) [30,40) [40,50) [50,60)

BLEU

Source sentence length

RNNSearch Adversarial-NMT* BGAN-NMT

(b) Chinese-English Translation

Figure 3: Performance of the generated translations with respect to the length of source sentences ondifferent datasets. For Chinese-English, we merge all NIST datasets in this experiment. For German-English, we only use test datasets.

BLEU score for each group. As shown in Fig-ure 3, we can view that our approach outperformsRNNSearch and Adversarial-NMT* in all lengthsegments, especially achieving notable improve-ments on long sentences. These results furtherdemonstrate that our approach can better handlethis problem and yield higher quality translations.

4.5 Effect of Discriminative LossWe also perform an ablation experiment in orderto quantify the effect of the discriminative loss onour models. As shown in Table 3, the discrimi-native loss can bring 0.58 and 0.73 BLEU scoreimprovements on English-German and Chinese-English dataset respectively. This result provesthat the discriminative loss can improve the dis-criminative ability of bidirectional NMT models,which can give more accurate rewards for the gen-erator training in GAN framework.

5 Related Work

As a new paradigm of machine translation, NMTtypically suffers from the exposure bias problemdue to MLE training. To handle this issue, manymethods have been proposed, including designingnew training objectives (Shen et al., 2016; Wise-man and Rush, 2016) and adopting reinforcementlearning approaches (Ranzato et al., 2015; Bah-danau et al., 2016). Shen et al. (2016) proposedto directly minimize expected loss (maximize theexpected BLEU) with Minimum Risk Training(MRT). Wiseman and Rush (2016) adopted abeam-search optimization algorithm to reduce in-consistency between training and inference. Be-sides, Ranzato et al. (2015) proposed a mixturetraining method to perform a gradual transition

Model DE-EN ZH-ENBGAN-NMT 29.17 36.19-Discriminative Loss 28.59 35.46

Table 3: Translation performance of BGAN-NMTwithout discriminative loss on German-English(DE-EN) and Chinese-English (ZH-EN) transla-tions. The BLEU score for Chinese-English trans-lation is the average results of all datasets we usedin the experiment.

from maximum likelihood learning into optimiz-ing BLEU scores using reinforcement algorithm.Bahdanau et al. (2016) designed an actor-criticalgorithm for sequence prediction, in which theNMT system is the actor, and a critic network isproposed to predict the value of output tokens. Re-cently, Yang et al. (2017) and Wu et al. (2017)proposed to leverage GAN framework to deal withthe exposure bias problem, in which NMT modelis employed as the generator, and CNN-based orRNN-based model is used as the discriminator.Different from their work, both generator and dis-criminator in our approach are designed to modelthe joint probability of sentence pairs and then wedesign an auxiliary GAN to take advantage of thesymmetry of them.

Another similar research in NMT is to leveragebidirectional dependency to improve translationquality. Tu et al. (2017) designed a re-constructormodule for NMT in order to make the target repre-sentation contain the complete source informationwhich can reconstruct back to the source sentence.Cheng et al. (2016) and He et al. (2016) proposedto reconstruct monolingual data by auto-encoder,

198

in which bidirectional translation models form aclosed loop and are jointly updated. Recently, thissimilar idea is used in unsupervised machine trans-lation tasks(Artetxe et al., 2017; Lample et al.,2018).

6 Conclusion

In this paper, we have presented a BidirectionalGenerative Adversarial Network for Neural Ma-chine Translation, consisting of an original GANand an auxiliary GAN. Both generator and dis-criminator in original GAN are designed to modelthe joint probability of sentence pairs. AuxiliaryGAN adopts generator and discriminator modelsof original one but exchanges their roles to full uti-lize the symmetry of them. Then these two GANsare alternately updated using joint training algo-rithm. Experimental results on German-Englishand Chinese-English translation tasks demonstratethat our proposed approach not only stabilizesGAN training but also leads to significant im-provements. In the future, we plan to extend thismethod to other sequence-to-sequence NLP tasks.

Acknowledgments

We appreciate Dongdong Zhang, Shuangzhi Wu,Wenhu Chen and Guanlin Li for the fruitful dis-cussions. We also thank the anonymous reviewersfor their careful reading of our paper and insightfulcomments.

ReferencesMartın Arjovsky, Soumith Chintala, and Leon Bottou.

2017. Wasserstein gan. CoRR, abs/1701.07875.

Mikel Artetxe, Gorka Labaka, Eneko Agirre, andKyunghyun Cho. 2017. Unsupervised neural ma-chine translation. CoRR, abs/1710.11041.

Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu,Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron C.Courville, and Yoshua Bengio. 2016. An actor-critic algorithm for sequence prediction. CoRR,abs/1607.07086.

Dzmitry Bahdanau, Kyunghyun Cho, and YoshuaBengio. 2014. Neural machine translation byjointly learning to align and translate. CoRR,abs/1409.0473.

Samy Bengio, Oriol Vinyals, Navdeep Jaitly, andNoam Shazeer. 2015. Scheduled sampling for se-quence prediction with recurrent neural networks.In NIPS.

Yong Cheng, Wei Xu, Zhongjun He, Wei He, HuaWu, Maosong Sun, and Yang Liu. 2016. Semi-supervised learning for neural machine translation.In ACL.

David Chiang. 2007. Hierarchical phrase-based trans-lation. Computational Linguistics, 33:201–228.

Kyunghyun Cho, Bart van Merrienboer, CaglarGulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol-ger Schwenk, and Yoshua Bengio. 2014. Learningphrase representations using rnn encoder-decoderfor statistical machine translation. In EMNLP.

Jonas Gehring, Michael Auli, David Grangier, DenisYarats, and Yann Dauphin. 2017. Convolutional se-quence to sequence learning. In ICML.

Xavier Glorot and Yoshua Bengio. 2010. Understand-ing the difficulty of training deep feedforward neuralnetworks. In AISTATS.

Ian J. Goodfellow. 2016. Nips 2016 tutorial: Genera-tive adversarial networks. CoRR, abs/1701.00160.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza,Bing Xu, David Warde-Farley, Sherjil Ozair,Aaron C. Courville, and Yoshua Bengio. 2014. Gen-erative adversarial nets. In NIPS.

Hany Hassan, Anthony Aue, Chang Chen, VishalChowdhary, Jonathan Clark, Christian Feder-mann, Xuedong Huang, Marcin Junczys-Dowmunt,William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu,Renqian Luo, Arul Menezes, Tao Qin, Frank Seide,Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, YingceXia, Dongdong Zhang, Zhirui Zhang, and MingZhou. 2018. Achieving human parity on auto-matic chinese to english news translation. CoRR,abs/1803.05567.

Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu,Tie-Yan Liu, and Wei-Ying Ma. 2016. Dual learningfor machine translation. In NIPS.

Sepp Hochreiter and Jurgen Schmidhuber. 1997. Longshort-term memory. Neural computation.

Philipp Koehn, Franz Josef Och, and Daniel Marcu.2003. Statistical phrase-based translation. In HLT-NAACL.

Alex Lamb, Anirudh Goyal, Ying Zhang, SaizhengZhang, Aaron C. Courville, and Yoshua Bengio.2016. Professor forcing: A new algorithm for train-ing recurrent networks. In NIPS.

Guillaume Lample, Myle Ott, Alexis Conneau, Lu-dovic Denoyer, and Marc’Aurelio Ranzato. 2018.Phrase-based & neural unsupervised machine trans-lation. CoRR, abs/1804.07755.

Jiwei Li, Will Monroe, Tianlin Shi, Alan Ritter, andDaniel Jurafsky. 2017. Adversarial learning for neu-ral dialogue generation. In EMNLP.

199

Thang Luong, Ilya Sutskever, Quoc V. Le, OriolVinyals, and Wojciech Zaremba. 2015. Addressingthe rare word problem in neural machine translation.In ACL.

Xuezhe Ma, Pengcheng Yin, Jingzhou Liu, Gra-ham Neubig, and Eduard H. Hovy. 2017. Soft-max q-distribution estimation for structured predic-tion: A theoretical interpretation for raml. CoRR,abs/1705.07136.

Kishore Papineni, Salim E. Roucos, Todd Ward, andWei-Jing Zhu. 2002. Bleu: a method for automaticevaluation of machine translation. In ACL.

Alec Radford, Luke Metz, and Soumith Chintala.2015. Unsupervised representation learning withdeep convolutional generative adversarial networks.arXiv preprint arXiv:1511.06434.

Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli,and Wojciech Zaremba. 2015. Sequence leveltraining with recurrent neural networks. CoRR,abs/1511.06732.

Rico Sennrich, Barry Haddow, and Alexandra Birch.2016. Improving neural machine translation modelswith monolingual data. In ACL.

Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, HuaWu, Maosong Sun, and Yang Liu. 2016. Minimumrisk training for neural machine translation. In ACL.

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.Sequence to sequence learning with neural net-works. In NIPS.

Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu,and Hang Li. 2017. Neural machine translation withreconstruction. In AAAI.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N. Gomez, LukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In NIPS.

Sam Wiseman and Alexander M. Rush. 2016.Sequence-to-sequence learning as beam-search op-timization. In EMNLP.

Lijun Wu, Yingce Xia, Li Zhao, Fei Tian, Tao Qin,Jian-Huang Lai, and Tie-Yan Liu. 2017. Adversarialneural machine translation. CoRR, abs/1704.06933.

Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. 2017.Improving neural machine translation with condi-tional sequence generative adversarial nets. CoRR,abs/1703.04887.

Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu.2017. Seqgan: Sequence generative adversarial netswith policy gradient. In AAAI.