pretraining methods for dialog context representation learning · 2019-07-15 · stream tasks that...

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3836–3845Florence, Italy, July 28 - August 2, 2019. c©2019 Association for Computational Linguistics

3836

Pretraining Methods for Dialog Context Representation Learning

Shikib Mehri∗, Evgeniia Razumovskaia∗, Tiancheng Zhao and Maxine EskenaziLanguage Technologies Institute, Carnegie Mellon University{amehri,erazumov,tianchez,max+}@cs.cmu.edu

Abstract

This paper examines various unsupervisedpretraining objectives for learning dialog con-text representations. Two novel methods ofpretraining dialog context encoders are pro-posed, and a total of four methods are exam-ined. Each pretraining objective is fine-tunedand evaluated on a set of downstream dialogtasks using the MultiWoz dataset and strongperformance improvement is observed. Fur-ther evaluation shows that our pretraining ob-jectives result in not only better performance,but also better convergence, models that areless data hungry and have better domaingeneralizability.

1 Introduction

Learning meaningful representations of multi-turndialog contexts is the cornerstone of dialog sys-tems. In order to generate an appropriate response,a system must be able to aggregate informationover multiple turns, such as estimating a beliefstate over user goals (Williams et al., 2013) andresolving anaphora co–references (Mitkov, 2014).In the past, significant effort has gone into de-veloping better neural dialog architectures to im-prove context modeling given the same in-domaintraining data (Dhingra et al., 2017; Zhou et al.,2016). Recent advances in pretraining on mas-sive amounts of text data have led to state-of-the-art results on a range of natural language pro-cessing (NLP) tasks (Peters et al., 2018; Radfordet al., 2018; Devlin et al., 2018) including naturallanguage inference, question answering and textclassification. These promising results suggest anew direction for improving context modeling bycreating general purpose natural language repre-sentations that are useful for many different down-stream tasks.

∗* Equal contribution.

Yet pretraining methods are still in their infancy.We do not yet fully understand their properties.For example, many pretraining methods are vari-ants of language modeling (Howard and Ruder,2018; Radford et al., 2018; Devlin et al., 2018),e.g. predicting the previous word, next word orthe masked word, given the sentence context. Thisapproach treats natural language as a simplestream of word tokens. It relies on a com-plex model to discover high-level dependencies,through the use of massive corpora and expen-sive computation. Recently the BERT model (De-vlin et al., 2018) achieved state-of-the-art per-formance on several NLP benchmarks. It intro-duces a sentence-pair level pretraining objective,i.e. predicting whether two sentences should comeafter one another. This is a step towards havingpretraining objectives that explicitly consider andleverage discourse-level relationships. However,it is still unclear whether language modeling is themost effective method of pretrained language rep-resentation, especially for tasks that need to ex-ploit multi-turn dependencies, e.g. dialog contextmodeling. Thornbury and Slade (2006) underlineseveral discourse-level features which distinguishdialog from other types of text. Dialog must becoherent across utterance and a sequence of turnsshould achieve a communicative purpose. Fur-ther, dialog is interactive in nature, with feedbackand back-channelling between speakers, and turn-taking. These unique features of dialog suggestthat modelling dialog contexts requires pretrainingmethods specifically designed for dialog.

Building on this prior research, the goal of thispaper is to study various methods of pretrain-ing discourse-level language representations, i.e.modeling the relationship amongst multiple utter-ances. This paper takes a first step in the cre-ation of a systematic analysis framework of pre-training methods for dialog systems. Concretely,

3837

we pretrain a hierarchical dialog encoder (Serbanet al., 2016) with four different unsupervised pre-training objectives. Two of the objectives, next-utterance generation (Vinyals and Le, 2015) andretrieval (Lowe et al., 2016), have been exploredin previous work. The other two pretraining ob-jectives, masked-utterance retrieval and inconsis-tency identification, are novel. The pretraineddialog encoder is then evaluated on several down-stream tasks that probe the quality of the learnedcontext representation by following the typicalpretrain & fine-tune procedure.

Pretraining and downstream evaluation use theMultiWoz dialog dataset (Budzianowski et al.,2018), which contains over 10,000 dialogs span-ning 6 different domains. The downstream tasksinclude next-utterance generation (NUG), next-utterance retrieval (NUR), dialog act prediction(DAP), and belief state prediction (BSP). The pre-training objectives are assessed under four differ-ent hypotheses: (1) that pretraining will improvedownstream tasks with fine-tuning on the entireavailable data, (2) that pretraining will result inbetter convergence, (3) that pretraining will per-form strongly with limited data and (4) that pre-training facilitates domain generalizability. Theresults here show that pretraining achieves sig-nificant performance gains with respect to thesehypotheses. Furthermore, the novel objectivesachieve performance that is on-par with or bet-ter than the pre-existing methods. The contribu-tions of this paper are: (1) a study of four differ-ent pretraining objectives for dialog context rep-resentation, including two novel objectives. (2) acomprehensive analysis of the effects of pretrain-ing on dialog context representations, assessed onfour different downstream tasks.

2 Related Work

This work is closely related to research in auxi-liary multi-task learning and transfer learning withpretraining for NLP systems.

Training with Auxiliary Tasks

Incorporating a useful auxiliary loss function tocomplement the primary objective has been shownto improve the performance of deep neural net-work models, including, but not limited to, errordetection (Rei and Yannakoudakis, 2017), cross-lingual speech tagging (Plank et al., 2016), do-main independent sentiment classification (Yu and

Jiang, 2016), latent variable inference for dialoggeneration (Zhao et al., 2017) and opinion ex-traction (Ding et al., 2017). Some auxiliary lossfunctions are designed to improve performanceon a specific task. For instance, Yu and Jiang(2016) pretrained a model for sentiment classifica-tion with the auxiliary task of identifying whethera negative or positive word occurred in the sen-tence. In some cases, auxiliary loss is createdto encourage a model’s general representationalpower. Trinh et al. (2018) found that a model cancapture far longer dependencies when pretrainedwith a suitable auxiliary task. This paper falls inline with the second goal by creating learning ob-jectives that improve a representation to capturegeneral-purpose information.

Transfer Learning with Pretraining

The second line of related research concerns thecreation of transferable language representationvia pretraining. The basic procedure is typicallyto first pretrain a powerful neural encoder on mas-sive text data with unsupervised objectives. Thesecond step is to fine-tune this pretrained model ona specific downstream task using a much smallerin-domain dataset (Howard and Ruder, 2018).Recently, several papers that use this approachhave achieved significant results. ELMo (Peterset al., 2018) trained a two-way language modelwith Bidirectional Long Short-Term Memory Net-works (biLSTM) (Huang et al., 2015) to pre-dict both the next and previous word. OpenAI’sGPT created a unidirectional language modelusing transformer networks (Radford et al., 2018)and BERT was trained with two simultaneous ob-jectives: the masked language model and next sen-tence prediction (Devlin et al., 2018). Each ofthe models has demonstrated state-of-the-art re-sults on the GLUE benchmark (Wang et al., 2018).The GPT model has also been adapted to improvethe performance of end-to-end dialog models. Inthe 2nd ConvAI challenge (Dinan et al., 2019), thebest models on both human and automated eval-uations were generative transformers (Wolf et al.,2019), which were initialized with the weights ofthe GPT model and fine-tuned on in-domain di-alog data. These models, which leveraged large-scale pretraining, outperformed the systems whichonly used in-domain data.

There has been little work on pretraining meth-ods that learn to extract discourse level informa-tion from the input text. Next sentence predic-

3838

tion loss in BERT (Devlin et al., 2018) is a stepin this direction. While these pretraining meth-ods excel at modelling sequential text, they do notexplicitly consider the unique discourse-level fea-tures of dialog. We therefore take the first steps inthe study of pretraining objectives that extract bet-ter discourse-level representations of dialog con-texts.

3 Pretraining Objectives

This section discusses the unsupervised pretrain-ing objectives, including two novel approachesaimed at capturing better representations of di-alog context. When considering a specific pre-training method, both the pretraining objective andthe model architecture must facilitate the learn-ing of strong and general representations. We de-fine a strong representation as one that capturesthe discourse-level information within the entiredialog history as well as utterance-level informa-tion in the utterances that constitute that history.By our definition, a representation is sufficientlygeneral when it allows the model to perform bet-ter on a variety of downstream tasks. The nextsection describes the pretraining objectives withinthe context of the strength and generality of thelearned representations.

For clarity of discussion, the following notationis used: an arbitrary T -turn dialog segment is rep-resented by a list of utterances c = [u1, ...uT ],where ui is an utterance. Further, we denote theset of all observed dialog responses in the data byR = {r1, ...rM}.

The pretraining objectives, discussed below,are next-utterance retrieval (NUR), next-utterancegeneration (NUG), masked-utterance retrieval(MUR), and inconsistency identification (InI).

3.1 Next-Utterance Retrieval

NUR has been extensively explored both as an in-dependent task (Lowe et al., 2015, 2016) and as anauxiliary loss in a multi-tasking setup (Wolf et al.,2019). Given a dialog context, the aim of NURis to select the correct next utterance from a setof k candidate responses. NUR can be thoughtof as being analogous to language modelling, ex-cept that the utterances, rather than the words, arethe indivisible atomic units. Language modellingpretraining has produced strong representations oflanguage (Radford et al., 2018; Peters et al., 2018),thereby motivating the choice of NUR as a pre-

training objective.For this task we use a hierarchical encoder to

produce a representation of the dialog context byfirst running each utterance independently througha Bidirectional Long-short Term Memory Net-work (biLSTM) and then using the resulting ut-terance representations to produce a representa-tion of the entire dialog context. We use a sin-gle biLSTM to encode candidate responses. Given[u1, ...uT−1], the task of NUR is to select the cor-rect next utterance uT from R. Note that forlarge dialog corpora, R is usually very large andit is more computationally feasible to sample asubset of R and as such we retrieve K negativesamples for each training example, according tosome distribution pn(r), e.g. uniform distribu-tion (Mikolov et al., 2013). Concretely, we min-imize the cross entropy loss of the next utteranceby:

ui = fu(ui) i ∈ [1, T − 1] (1)

[h1, ...hT−1] = fc(u1, ...uT−1) (2)

rgt = fr(uT ) (3)

rj = fr(rj) rj ∼ pn(r) (4)

αgt = hT−1T rgt (5)

αj = hT−1T rj (6)

where fu, fc and fr are three distinct biLSTMmodels that are to be trained. The final loss func-tion is:

L = − log p(uT |u1, ...uT−1) (7)

= − log

(exp(αgt)

exp(αgt) +∑K

j=1 exp(αj)

)3.2 Next-Utterance GenerationNUG is the task of generating the next utteranceconditioned on the past dialog context. Sequence-to-sequence models (Sutskever et al., 2014; Bah-danau et al., 2015) have been used for pretraining(Dai and Le, 2015; McCann et al., 2017), and havebeen shown to learn representations that are usefulfor downstream tasks (Adi et al., 2016; Belinkovet al., 2017).

The hierarchical recurrent encoder-decoder ar-chitecture (Serban et al., 2016) was used duringNUG pretraining. Although the decoder is used inpretraining, only the hierarchical context encoderis transferred to the downstream tasks. Similarlyto NUR, the optimization goal of NUG is to maxi-mize the log-likelihood of the next utterance given

3839

the previous utterances. However, it differs in thatit factors the conditional distribution to word-levelin an auto-regressive manner. Specifically, let theword tokens in uT be [w1, ...wN ]. The dialog con-text is encoded as in Eq 8 with an utterance anda context biLSTM. Then the loss function to beminimized is shown in Eq 9:

L = − log p(uT |u1, ...uT−1) (8)

= −N∑k

log p(wk|w<k,hT−1) (9)

3.3 Masked-Utterance Retrieval

MUR is similar to NUR: the input contains a di-alog context and a set of K candidate responses.The objective is to select the correct response. Thedifference between the two is twofold. First, oneof the utterances in the dialog context has been re-placed by a randomly chosen utterance. Secondly,rather than use the final context representation toselect the response that should immediately fol-low, the goal here is to use the representation ofthe replacement utterance to retrieve the correctutterance. The replacement index t is randomlysampled from the dialog segment:

t ∼ Uniform[1, T ] (10)

Then ut is randomly replaced by a replacement ut-terance q that is sampled from the negative dis-tribution pn(r) defined in NUR. Finally, the goalis to minimize the negative log-likelihood of theoriginal ut given the context hidden state at time-stamp t, i.e. − log p(ugt|u1, ...q, ...uT ), where ugtis the original utterance at index t.

ui = fu(ui) i ∈ [1, T ] (11)

[h1, ...hT] = fc(u1, ...uT) (12)

rgt = fr(ugt) (13)

rj = fr(rj) rj ∼ pn(r) (14)

αgt = htT rgt (15)

αj = htT rj (16)

The final loss function is:

L = − log p(ut|u1, ...q, ...uT ) (17)

= − log

(exp(αgt)

exp(αgt) +∑K

j=1 exp(αj)

)

MUR is analogous to the MLM objective of De-vlin et al. (2018), which forces model to keep adistributional contextual representation of each in-put token. By masking entire utterances, instead ofinput tokens, MUR learns to produce strong repre-sentations of each utterance.

3.4 Inconsistency Identification

InI is the task of finding inconsistent utteranceswithin a dialog history. Given a dialog contextwith one utterance replaced randomly, just likeMUR, InI finds the inconsistent utterance. Thereplacement procedure is the same as the one de-scribed for MUR, where a uniform random index tis selected in the dialog context and ut is replacedby a negative sample q.

While MUR strives to create a model that findsthe original utterance, given the replacement indext, InI aims to train a model that can identify thereplacement position t. Specifically, this is donevia:

ui = fu(ui) i ∈ [1, T ] (18)

[h1, ...hT] = fc(u1, ...uT) (19)

αi = hTThi i ∈ [1, T ] (20)

Finally, the loss function is to minimize the crossentropy of the replaced index:

L = − log p(t|u1, ...q, ...uT ) (21)

= − log

(exp(αt)∑Tj=1 exp(αi)

)

This pretraining objective aims to explicitlymodel the coherence of the dialog, which encour-ages both local representations of each individualutterance and a global representation of the dialogcontext. We believe that this will improve the gen-erality of the pretrained representations.

4 Downstream Tasks

This section describes the downstream tasks cho-sen to test the strength and generality of the repre-sentations produced by the various pretraining ob-jectives. The downstream evaluation is carried outon a lexicalized version of the MultiWoz dataset(Budzianowski et al., 2018). MultiWoz containsmulti-domain conversations between a Wizard-of-Oz and a human. There are 8422 dialogs for train-ing, 1000 for validation and 1000 for testing.

3840

4.1 Belief State Prediction

Given a dialog context, the task is to predict a1784-dimensional belief state vector. Belief stateprediction (BSP) is a multi-class classificationtask, highly dependant on strong dialog contextrepresentations. The belief state vector representsthe values of 27 entities, all of which can be in-ferred from the dialog context. To obtain the 1784-dimensional label, the entity values are encoded asa one-hot encoded vector and concatenated. Theentities are shown in Appendix ??. Performance ismeasured using the F-1 score for entities with non-empty values. This approach is analogous to theone used in the evaluation of Dialog State Track-ing Challenge 2 (Henderson et al., 2014).

This task measures the ability of a system tomaintain a complete and accurate state repre-sentation of the dialog context. With a 1784-dimensional output, the hidden representation forthis task must be sufficiently general. Therefore,any pretrained representations that lack generalitywill struggle on belief state prediction.

4.2 Dialog Act Prediction

Dialog act prediction (DAP), much like belief stateprediction, is a multi-label task aimed at producinga 32-dimensional dialog act vector for the systemutterances. The set of dialog acts for a system ut-terance describes the actions that may be taken bythe system. This might include: informing the userabout an attraction, requesting information abouta hotel query, or informing them about specifictrains. There are often multiple actions taken in asingle utterance, and thus this is a multi-label task.To evaluate performance on dialog act prediction,we use the F-1 score.

4.3 Next-Utterance Generation

NUG is the task of producing the next utteranceconditioned on the dialog history. We evaluate theability of our models to generate system utterancesusing BLEU-4 (Papineni et al., 2002). This taskrequires both a strong global context representa-tion to initialize the decoder’s hidden state andstrong local utterance representations.

4.4 Next-Utterance Retrieval

Given a dialog context, NUR selects the correctnext utterance from a set of k candidate responses.Though this task was not originally part of theMultiWoz dataset, we construct the necessary data

for this task by randomly sampling negative ex-amples. This task is underlined by Lowe et al.(2016)’s suggestion that using NUR for evaluationis extremely indicative of performance and is oneof the best forms of evaluation. Hits@1 (H@1) isused to evaluate our retrieval models. The latter isequivalent to accuracy.

Although some of these pretraining models hada response encoder, which would have been usefulto transfer to this task, to ensure a fair comparisonof all of the methods, we only transfer the weightsof the context encoder.

5 Experiments and Results

This section presents the experiments and resultsaimed at capturing the capabilities and proper-ties of the above pretraining objectives by evalu-ating on a variety of downstream tasks. All unsu-pervised pretraining objectives are trained on thefull MultiWoz dataset (Budzianowski et al., 2018).Data usage for downstream fine-tuning differs, de-pending on the property being measured.

5.1 Experimental Setup

Each model was trained for 15 epochs, with thevalidation performance computed at each epoch.The model achieving the highest validation setperformance was used for the results on the testdata. The hyperparameters and experimental set-tings are shown in the Appendix ??. The sourcecode will be open-sourced when this paper is re-leased.

In the experiments, the performance on eachdownstream task was measured for each pretrain-ing objective. Combinations where the pretrainingobjective is the same as the downstream task wereexcluded.

The pretraining and finetuning is carried out onthe same dataset. This evaluates the pretrainingobjectives as a means of extracting additional in-formation from the same data, in contrast to eval-uating their ability to benefit from additional data.Though pretraining on external data may proveto be effective, identifying a suitable pretrainingdataset is challenging and this approach more di-rectly evaluates the pretraining objectives.

5.2 Performance on Full Data

To first examine whether the pretraining objectivesfacilitate improved performance on downstreamtasks a baseline model was trained for each down-

3841

BSP DAP NUR NUGF-1 F-1 H@1 BLEU

None 18.48 40.33 63.72 14.21NUR 17.80 43.25 – 15.39NUG 17.96 42.31 67.34 –MUR 16.76 44.87 62.38 15.27InI 16.61 44.84 62.62 15.52

Table 1: Results of evaluating the chosen pretrainingobjectives, preceded by the baseline, on the four down-stream tasks. This evaluation used all of the trainingdata for the downstream tasks as described in Section5.2.

stream task, using the entire set of MultiWoz data.The first row of Table 1 shows the performance ofrandomly initialized models for each downstreamtask. To evaluate the full capabilities of the pre-training objectives above, the pretrained modelswere used to initialize the models for the down-stream tasks.

Results are shown on Table 1. This experimen-tal setup speaks to the strength and the general-ity of the pretrained representations. Using un-supervised pretraining, the models produce dia-log representations that are strong enough to im-prove downstream tasks. The learned represen-tations demonstrate generality because the multi-ple downstream tasks benefit from the same pre-training. Rather than learning representations thatare useful for just the pretraining objective, or fora single downstream task, the learned representa-tions are general and beneficial for multiple tasks.

For the DAP and NUG downstream tasks, thepretrained models consistently outperformed thebaseline. InI has the highest BLEU score forNUG. This may be a consequence of the impor-tance of both global context representations andlocal utterance representations in sequence gen-eration models. Both InI and MUR score muchhigher than the baseline and the other methods forDAP, which may be due to the fact that these twoapproaches are trained to learn a representation ofeach utterance rather than just an overall contextrepresentation. NUR has significant gains whenpretraining with NUG, possibly because the infor-mation that must be captured to generate the nextutterance is similar to the information needed toretrieve the next utterance. Unlike the other down-stream tasks, BSP did not benefit from pretraining.A potential justification of this result is that due tothe difficulty of the task, the model needs to resort

to word-level pattern matching. The generality ofthe pretrained representations precludes this.

5.3 Convergence Analysis

This experimental setup measures the impact ofpretraining on the convergence of the downstreamtraining. Sufficiently general pretraining objec-tives should learn to extract useful representationsof the dialog context. Thus when fine-tuning on agiven downstream task, the model should be ableto use the representations it has already learnedrather than having to learn to extract relevant fea-tures from scratch. The performance on all down-stream tasks with the different pretraining objec-tives is evaluated at every epoch. The results arepresented on Figure 1.

These figures show faster convergence acrossall downstream tasks with significant improve-ment over a random initialization baseline. The re-sults show that performance on the initial epochsis considerably better with pretraining than with-out. In most cases, performance evens out duringtraining, thus attaining results that are comparableto the pretraining methods on the full dataset. It isimportant to note that performance of the modelsafter just a single epoch of training is significantlyhigher on all downstream tasks when the encoderhas been pretrained. This underlines the useful-ness of the features learned in pretraining.

The convergence of BSP shown in Figure 1is very interesting. Though the baseline ulti-mately outperforms all other methods, the pre-trained models attain their highest performance inthe early epochs. This suggests that the represen-tations learned in pretraining are indeed useful forthis task despite the fact that they do not show im-provement over the baseline.

5.4 Performance on Limited Data

Sufficiently strong and general pretrained repre-sentations, should continue to succeed in down-stream evaluation even when fine-tuned on signifi-cantly less data. The performance on downstreamtasks is evaluated with various amounts of fine-tuning data (1%, 2%, 5%, 10% and 50%).

The effect of the training data size for eachdownstream task is also evaluated. The perfor-mance of NUR with different amounts of train-ing data is shown on Figure 2. With 5% of thefine-tuning data, the NUG pretrained model out-performs the baseline that used 10%. With 10%

3842

Figure 1: The performance of (from left to right) BSP, DAP, NUR, NUG across epochs with different pretrainingobjectives. For the BLEU-4 score in NUG, the results are noisy due to the metric being the BLEU score, howeverthe general trend is still apparent.

Figure 2: NUR Hits@1 at different training set sizes.The blue horizontal line is the baseline performancewith 50% of the data. The red horizontal line is thebaseline performance with 10% of the data.

of the fine-tuning data, this model outperforms thebaseline that used 50% of the data.

Table 2 shows all of the results with 1% of thefine-tuning data, while Table 3 shows the resultswith 10% of the fine-tuning data. More resultsmay be found in the Appendix ??.



Table 2: Performance using 1% of the data; therows correspond to the pretraining objectives and thecolumns correspond to the downstream tasks.

The results shown here strongly highlight theeffectiveness of pretraining. With a small fraction



Table 3: Results with 10% of the data; the rows cor-respond to the pretraining objectives and the columnscorrespond to the downstream tasks.

of the data, unsupervised pretraining shows com-petitive performance on downstream tasks.

When the amount of data is very limited, thebest results were obtained by models pretrainedwith NUG. This may be indicative of the general-ity of NUG pretraining. Since the generation taskis difficult, it is likely that the pretrained modellearns to capture the most general context repre-sentation that it can. This makes the represen-tations especially suitable for low resource con-ditions since NUG pretrained representations aregeneral enough to adapt to different tasks giveneven very small amounts of data,

5.5 Domain Generalizability

Sufficiently general pretrained representationsshould facilitate domain generalizability on thedownstream tasks, just as pretraining should en-courage the downstream models to use domain ag-nostic representations and identify domain agnos-tic relationships in the data.

This experimental setup is designed to mimicthe scenario of adding a new domain as the down-stream task. It assumes that there are large quanti-ties of unlabeled data for unsupervised pretrainingin all domains but that there is a limited set of la-beled data for the downstream tasks. More specif-

3843



Table 4: Results of evaluating pretrained objectiveson their capacity to generalize to the restaurant do-main using only 50 in-domain samples and 2000 out-of-domain samples during training. The evaluation iscarried out only on the in-domain test samples.

ically, for each downstream task there are 1000 la-beled out-of-domain examples (2% of the dataset)and only 50 labeled in-domain examples (0.1%of the dataset). The performance of the down-stream models is computed only on the in-domaintest samples, thereby evaluating the ability of ourmodels to learn the downstream task on the limitedin-domain data. The results on Table 4 show thatpretraining produces more general representationsand facilitates domain generalizability.

6 Discussion

The results with different experimental setupsdemonstrate the effectiveness of the pretrainingobjectives. Pretraining improves performance,leads to faster convergence, works well in low-data scenarios and facilitates domain generaliz-ability. We now consider the respective strengthsof the different pretraining objectives.

NUR and NUG are complementary tasks.Over all of the results, we can see that pretrain-ing with either NUG or NUR, gives strong re-sults when fine-tuning on the other one. This pro-perty, which has also been observed by Wolf et al.(2019), is a consequence of the similarity of thetwo tasks. Both for retrieval and generation, con-text encoding must contain all of the informationthat is necessary to produce the next utterance.

NUG learns representations that are verygeneral. We see that NUG, especially in low dataexperiments, effectively transfers to many down-stream tasks. This speaks to the generality of itsrepresentations. To auto-regressively generate thenext utterance, the context encoder in NUG mustcapture a strong and expressive representation ofthe dialog context. This representation is all thatthe decoder uses to generate its response at wordlevel so it must contain all of the relevant infor-

< 3 ≥ 3 & < 7 ≥ 7

None 11.02 14.17 15.30NUR 13.95 15.08 15.88MUR 12.21 15.36 16.10InI 11.52 15.40 16.63

Table 5: Results on the downstream task of NUG, withdifferent dialog context lengths (< 3 utterances, 3-7utterances, and > 7 utterances.

mation. Despite the similarity of NUG and NUR,generation is a more difficult task, due to the po-tential output space of the model. As such, therepresentations learned by NUG are more generaland expressive. The representative capabilities ofthe encoder in a generation model are also demon-strated by the work of Adi et al. (2016).

InI and MUR learn strong local representa-tions of each utterance. The two novel pretrain-ing objectives, InI and MUR, consistently showstrong improvement for the downstream NUGtask. Both of these objectives learn local repre-sentations of each utterance in the dialog contextsince both of their respective loss functions use therepresentation of each utterance instead of just thefinal hidden state. In an effort to better understandthe properties of the different objectives, Table 5shows performance on the NUG task for differentdialog context lengths.

Generating a response to a longer dialog contextrequires a strong local representation of each in-dividual utterance. A model that does not capturestrong representations of each utterance will likelyperform poorly on longer contexts. For example,for a dialog in which the user requests a restau-rant recommendation, in order to generate the sys-tem utterance that recommends a restaurant, themodel must consider all of the past utterances inorder to effectively generate the recommendation.If the local representations of each utterance arenot strong, it would be difficult to generate the sys-tem output.

The results in Table 5 demonstrate that bothInI and MUR strongly outperform other methodson long contexts, suggesting that these methodsare effective for capturing strong representationsof each utterance. Both MUR and InI performpoorly on shorter contexts. This further demon-strates that fine-tuned NUG models learn to relyon strong utterance representations, and thereforestruggle when there are few utterances.

Using the same dataset for pretraining and

3844

finetuning. The pretraining objectives demon-strate large improvements over directly trainingfor the downstream task. No additional data isused for pretraining, which suggests that the pro-posed objective allow the model to extract strongerand more general context representations from thesame data. The reduced data experiments showthat pretraining on a larger corpora (i.e., the fulldata), results in strong performance on smallertask-specific datasets (i.e., the reduced data). Assuch, it is likely that pretraining on larger exter-nal data will result in further performance gains,however, it is challenging to identify a sufficientcorpus.

7 Conclusion and Future Work

This paper proposes several methods of unsuper-vised pretraining for learning strong and generaldialog context representations, and demonstratestheir effectiveness in improving performance ondownstream tasks with limited fine-tuning data aswell as out-of-domain data. It proposes two novelpretraining objectives: masked-utterance retrievaland inconsistency identification which better cap-ture both the utterance-level and context-level in-formation. Evaluation of the learned represen-tations on four downstream dialog tasks showsstrong performance improvement over randomlyinitialized baselines.

In this paper, unsupervised pretraining has beenshown to learn effective representations of dialogcontext, making this an important research direc-tion for future dialog systems. These results openthree future research directions. First, the modelsproposed here should be pretrained on larger exter-nal dialog datasets. Second, it would be interestingto test the representations learned using unsuper-vised pretraining on less-related downstream taskssuch as sentiment analysis. Finally, the addition ofword-level pretraining methods to improve the di-alog context representations should be explored.

ReferencesYossi Adi, Einat Kermany, Yonatan Belinkov, Ofer

Lavi, and Yoav Goldberg. 2016. Fine-grained anal-ysis of sentence embeddings using auxiliary predic-tion tasks. arXiv preprint arXiv:1608.04207.

Dzmitry Bahdanau, Kyunghyun Cho, and YoshuaBengio. 2015. Neural machine translation byjointly learning to align and translate. CoRR,abs/1409.0473.

Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, HassanSajjad, and James Glass. 2017. What do neural ma-chine translation models learn about morphology?In Proceedings of the 55th Annual Meeting of theAssociation for Computational Linguistics (Volume1: Long Papers), volume 1, pages 861–872.

Paweł Budzianowski, Tsung-Hsien Wen, Bo-HsiangTseng, Inigo Casanueva, Stefan Ultes, Osman Ra-madan, and Milica Gasic. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the2018 Conference on Empirical Methods in NaturalLanguage Processing, pages 5016–5026.

Andrew M Dai and Quoc V Le. 2015. Semi-supervisedsequence learning. In Advances in neural informa-tion processing systems, pages 3079–3087.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2018. Bert: Pre-training of deepbidirectional transformers for language understand-ing. arXiv preprint arXiv:1810.04805.

Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, WilliamCohen, and Ruslan Salakhutdinov. 2017. Gated-attention readers for text comprehension. In Pro-ceedings of the 55th Annual Meeting of the Associa-tion for Computational Linguistics (Volume 1: LongPapers), volume 1, pages 1832–1846.

Emily Dinan, Varvara Logacheva, Valentin Malykh,Alexander Miller, Kurt Shuster, Jack Urbanek,Douwe Kiela, Arthur Szlam, Iulian Serban, RyanLowe, et al. 2019. The second conversationalintelligence challenge (convai2). arXiv preprintarXiv:1902.00098.

Ying Ding, Jianfei Yu, and Jing Jiang. 2017. Recur-rent neural networks with auxiliary labels for cross-domain opinion target extraction. In Thirty-FirstAAAI Conference on Artificial Intelligence.

Matthew Henderson, Blaise Thomson, and Jason DWilliams. 2014. The second dialog state trackingchallenge. In Proceedings of the 15th Annual Meet-ing of the Special Interest Group on Discourse andDialogue (SIGDIAL), pages 263–272.

Jeremy Howard and Sebastian Ruder. 2018. Universallanguage model fine-tuning for text classification. InProceedings of the 56th Annual Meeting of the As-sociation for Computational Linguistics (Volume 1:Long Papers), volume 1, pages 328–339.

Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirec-tional lstm-crf models for sequence tagging. arXivpreprint arXiv:1508.01991.

Ryan Lowe, Nissan Pow, Iulian V Serban, and JoellePineau. 2015. The ubuntu dialogue corpus: A largedataset for research in unstructured multi-turn di-alogue systems. In 16th Annual Meeting of theSpecial Interest Group on Discourse and Dialogue,page 285.

3845

Ryan Lowe, Iulian V Serban, Mike Noseworthy, Lau-rent Charlin, and Joelle Pineau. 2016. On the evalu-ation of dialogue systems with next utterance classi-fication.

Bryan McCann, James Bradbury, Caiming Xiong, andRichard Socher. 2017. Learned in translation: Con-textualized word vectors. In Advances in Neural In-formation Processing Systems, pages 6294–6305.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor-rado, and Jeff Dean. 2013. Distributed representa-tions of words and phrases and their compositional-ity. In Advances in neural information processingsystems, pages 3111–3119.

Ruslan Mitkov. 2014. Anaphora resolution. Rout-ledge.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic eval-uation of machine translation. In Proceedings ofthe 40th annual meeting on association for compu-tational linguistics, pages 311–318. Association forComputational Linguistics.

Matthew Peters, Mark Neumann, Mohit Iyyer, MattGardner, Christopher Clark, Kenton Lee, and LukeZettlemoyer. 2018. Deep contextualized word repre-sentations. In Proceedings of the 2018 Conferenceof the North American Chapter of the Associationfor Computational Linguistics: Human LanguageTechnologies, Volume 1 (Long Papers), volume 1,pages 2227–2237.

Barbara Plank, Anders Søgaard, and Yoav Goldberg.2016. Multilingual part-of-speech tagging withbidirectional long short-term memory models andauxiliary loss. In Proceedings of the 54th AnnualMeeting of the Association for Computational Lin-guistics (Volume 2: Short Papers), volume 2, pages412–418.

Alec Radford, Karthik Narasimhan, Tim Salimans, andIlya Sutskever. 2018. Improving language under-standing by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/languageunsupervised/language under-standing paper. pdf.

Marek Rei and Helen Yannakoudakis. 2017. Auxiliaryobjectives for neural error detection models. In Pro-ceedings of the 12th Workshop on Innovative Use ofNLP for Building Educational Applications, pages33–43.

Iulian Vlad Serban, Alessandro Sordoni, Yoshua Ben-gio, Aaron C Courville, and Joelle Pineau. 2016.Building end-to-end dialogue systems using gener-ative hierarchical neural network models. In AAAI,volume 16, pages 3776–3784.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.Sequence to sequence learning with neural net-works. In Advances in neural information process-ing systems, pages 3104–3112.

Scott Thornbury and Diana Slade. 2006. Conversation:From description to pedagogy. Cambridge Univer-sity Press.

Trieu Trinh, Andrew Dai, Thang Luong, and Quoc Le.2018. Learning longer-term dependencies in rnnswith auxiliary losses. In International Conferenceon Machine Learning, pages 4972–4981.

Oriol Vinyals and Quoc Le. 2015. A neural conversa-tional model. arXiv preprint arXiv:1506.05869.

Alex Wang, Amapreet Singh, Julian Michael, FelixHill, Omer Levy, and Samuel R Bowman. 2018.Glue: A multi-task benchmark and analysis platformfor natural language understanding. arXiv preprintarXiv:1804.07461.

Jason Williams, Antoine Raux, Deepak Ramachan-dran, and Alan Black. 2013. The dialog state track-ing challenge. In Proceedings of the SIGDIAL 2013Conference, pages 404–413.

Thomas Wolf, Victor Sanh, Julien Chaumond, andClement Delangue. 2019. Transfertransfo: Atransfer learning approach for neural networkbased conversational agents. arXiv preprintarXiv:1901.08149.

Jianfei Yu and Jing Jiang. 2016. Learning sentence em-beddings with auxiliary tasks for cross-domain sen-timent classification. In Proceedings of the 2016Conference on Empirical Methods in Natural Lan-guage Processing, pages 236–246.

Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi.2017. Learning discourse-level diversity for neuraldialog models using conditional variational autoen-coders. In Proceedings of the 55th Annual Meet-ing of the Association for Computational Linguistics(Volume 1: Long Papers), volume 1, pages 654–664.

Xiangyang Zhou, Daxiang Dong, Hua Wu, Shiqi Zhao,Dianhai Yu, Hao Tian, Xuan Liu, and Rui Yan. 2016.Multi-view response selection for human-computerconversation. In Proceedings of the 2016 Confer-ence on Empirical Methods in Natural LanguageProcessing, pages 372–381.

pretraining methods for dialog context representation learning · 2019-07-15 · stream tasks that...

Documents