Proceedings of NAACL-HLT 2013, pages 808–813,Atlanta, Georgia, 9–14 June 2013. c©2013 Association for Computational Linguistics

Discourse Connectors for Latent Subjectivity in Sentiment Analysis

Rakshit TrivediCollege of Computing

Georgia Institute of TechnologyAtlanta, GA 30308, USA

rtrivedi6@gatech.edu

Jacob EisensteinSchool of Interactive ComputingGeorgia Institute of Technology

Atlanta, GA 30308, USAjacobe@gatech.edu

AbstractDocument-level sentiment analysis can bene-fit from fine-grained subjectivity, so that sen-timent polarity judgments are based on therelevant parts of the document. While fine-grained subjectivity annotations are rarelyavailable, encouraging results have been ob-tained by modeling subjectivity as a latentvariable. However, latent variable modelsfail to capitalize on our linguistic knowledgeabout discourse structure. We present a newmethod for injecting linguistic knowledge intolatent variable subjectivity modeling, usingdiscourse connectors. Connector-augmentedtransition features allow the latent variablemodel to learn the relevance of discourse con-nectors for subjectivity transitions, withoutsubjectivity annotations. This yields signif-icantly improved performance on document-level sentiment analysis in English and Span-ish. We also describe a simple heuristic forautomatically identifying connectors when nopredefined list is available.

1 Introduction

Document-level sentiment analysis can benefit fromconsideration of discourse structure. Voll andTaboada (2007) show that adjective-based sentimentclassification is improved by examining topicality(whether each sentence is central to the overallpoint); Yessenalina et al. (2010b) show that bag-of-ngrams sentiment classification is improved by ex-amining subjectivity (whether a sentence expressesa subjective opinion or objective fact). However, itis unclear how best to obtain the appropriate dis-course analyses. Voll and Taboada (2007) find that

domain-independent discourse parsing (Soricut andMarcu, 2003) offers little improvement for senti-ment analysis, so they resort to training a domain-specific model for identifying topic sentences in re-views. But this requires a labeled dataset of topicsentences, imposing a substantial additional cost.

Yessenalina et al. (2010b) treat sentence levelsubjectivity as a latent variable, automatically in-ducing the “annotator rationale” (Zaidan et al., 2007;Yessenalina et al., 2010a) for each training sen-tence so as to focus sentiment learning on the sub-jective parts of the document. This yields sig-nificant improvements over bag-of-ngrams super-vised sentiment classification. Latent variable sub-jectivity analysis is attractive because it requiresneither subjectivity annotations nor an accuratedomain-independent discourse parser. But while the“knowledge-free” nature of this approach is appeal-ing, it is unsatisfying that it fails to exploit decadesof research on discourse structure.

In this paper, we explore a lightweight approachto injecting linguistic knowledge into latent variablemodels of subjectivity. The entry point is a set ofdiscourse connectors: words and phrases that signala shift or continuation in the discourse structure.Such connectors have been the subject of exten-sive study in the creation of the Penn DiscourseTreebank (PDTB: Prasad et al. 2008). The roleof discourse connectors in sentiment analysis canbe clearly seen in examples, such as “It’s hard toimagine the studios hiring another manic Germanmaverick to helm a cop thriller. But that’s exactlywhy the movie is unmissable.” (Huddleston, 2010)

808

We present a new approach to incorporatediscourse connectors in a latent subjectivitymodel (Yessenalina et al., 2010b). This approachrequires no manually-specified information aboutthe meaning of the connectors, just the connectorsthemselves. Our approach builds on proximityfeatures, which give the latent variable model a wayto prefer or disprefer subjectivity and sentimenttransitions, usually with the goal of encouragingsmoothness across the document. By takingthe cross-product of these features with a set ofdiscourse connectors, we obtain a new set ofconnector-augmented transition features, whichcapture the way discourse connectors are used toindicate subjectivity and sentiment transitions. Themodel is thus able to learn that subjectivity shiftsare likely to be accompanied by connectors such ashowever or nonetheless.

We present experiments in both English and Span-ish showing that this method of incorporating dis-course connectors yields significant improvementsin document-level sentiment analysis. In case nolist of connectors is available, we describe a sim-ple heuristic for automatically identifying candidateconnector words. The automatically identified con-nectors do not perform as well as the expert-definedlists, but they still outperform a baseline methodthat ignores discourse connectors (in English). Thisdemonstrates both the robustness of the approachand the value of linguistic knowledge.

2 Model

Given accurate labels of the subjectivity of eachsentence, a document-level sentiment analyzercould safely ignore the sentences marked as non-subjective.1 This would be beneficial for training aswell as prediction, because the learning algorithmwould not be confused by sentences that contradictthe document label. But in general we cannot rely onhaving access to sentence-level subjectivity annota-tions. Instead, we treat subjectivity as a latent vari-able, and ask the learner to impute its value. Givendocument-level sentiment annotations and an initial

1Discourse parsing often focuses on sub-sentence elemen-tary discourse units (Mann and Thompson, 1988). For sim-plicity, we consider units at the sentence level only, and leavefiner-grained analysis for future work.

model, the learner can mark as non-subjective thosesentences whose analysis disagrees with the docu-ment label.

More formally, each document has a label y ∈{−1, 1}, a set of sentences x, and a set of per-sentence subjectivity judgments h ∈ {0, 1}S , whereS is the number of sentences. We compute a setof features on these variables, and score each in-stance by a weighted combination of the features,wTf(y,x,h). At prediction time, we seek a labely which achieves a high score given the observed xand the ideal h.

y = arg maxy

(max

hwTf(y,x,h)

). (1)

At training time, we seek weights w whichachieve a high score given all training examples{x, y}t,

w = arg maxw

∑t

maxh

wTf(yt,xt,h). (2)

We can decompose the feature vector into twoparts: polarity features fpol(y,x,h), and subjectiv-ity features fsubj(x,h). The basic feature set decom-poses across sentences, though the polarity featuresinvolve the document-level polarity. For sentence i,we have fpol(y,xi, hi) = yhixi: the bag-of-wordsfeatures for sentence i are multiplied by the docu-ment polarity y ∈ {−1, 1} and the sentence sub-jectivity hi ∈ {0, 1}. The weights wpol capture thesentiment polarity of each possible word. As for thesubjectivity features, we simply have fsubj(xi, hi) =hixi. The weights wsubj capture the subjectivity ofeach word, with large values indicate positive sub-jectivity.

However, these features do not capture transi-tions between the subjectivity and sentiment of ad-jacent sentences. For this reason, Yessenalina et al.(2010b) introduce an additional set of proximity fea-tures, fprox(hi, hi−1), which are parametrized by thesubjectivity of both the current sentence i and theprevious sentence i− 1. The effect of these featureswill be to learn a preference for consistency in thesubjectivity of adjacent sentences.

By augmenting the transition features with thetext xi, we allow this preference for consistencyto be modulated by discourse connectors. We de-sign the transition feature vector ftrans(xi, hi, hi−1)

809

to contain two elements for every discourse connec-tor, one for hi = hi−1, and one for hi 6= hi−1. Forexample, the feature 〈moreover, CONTINUE〉 fireswhen sentence i starts with moreover and hi−1 =hi,i. We would expect to learn a positive weight forthis feature, and negative weights for features suchas 〈moreover, SHIFT〉 and 〈however, CONTINUE〉.

3 Experiments

To evaluate the utility of adding discourse connec-tors to latent subjectivity sentiment analysis, wecompare several models on movie review datasetsin English and Spanish.

3.1 DataWe use two movie review datasets:

• 50,000 English-language movie reviews (Maaset al., 2011). Each review has a rating from1-10; we marked ratings of 5 or greater as pos-itive. Half the dataset is used for test and halffor training. Parameter tuning is performed bycross-validation.

• 5,000 Spanish-language movie reviews (Cruzet al., 2008). Each review has a rating from1-5; we marked 3-5 as positive. We randomlycreated a 60/20/20 split for training, validation,and test.

3.2 ConnectorsWe first consider single-word discourse connectors:in English, we use a list of all 57 one-word con-nectors from the Penn Discourse Tree Bank (Prasadet al., 2008); in Spanish, we selected 25 one-wordconnectors from a Spanish language education web-site.2 We also consider multi-word connectors. Us-ing the same sources, this expands the English set to93 connectors, and Spanish set to 80 connectors.

In case no list of discourse connectors is avail-able, we propose a simple technique for automati-cally identifying potential connectors. We use a χ2

test to select words which are especially likely to ini-tiate sentences. The top K words (with the lowest pvalues) were added as potential connectors, whereK is equal to the number of “true” connectors pro-vided by the gold-standard resource.

2russell.famaf.unc.edu.ar/˜laura/shallowdisc4summ/discmar/

Finally, we consider a model with connector-augmented transition features for all words in thevocabulary. Thus, there are four connector sets:

• true-unigram-connectors: unigram connec-tors from the Penn Discourse Treebank and theSpanish language education website

• true-multiword-connectors: unigram andmultiword connectors from these same re-sources

• auto-unigram-connectors: automatically-selected connectors using the χ2 test

• all-unigram-connectors: all words are poten-tial connectors

3.3 SystemsThe connector-augmented transition features are in-corporated into a latent variable support vector ma-chine (SVM). We also consider two baselines:

• no-connectors: the same latent variable SVM,but without the connector features. This isidentical to the prior work of Yessenalina et al.(2010b).

• SVM: a standard SVM binary classifier

The latent variable models require an initial guessfor the subjectivity of each sentence. Yessenalina etal. (2010b) compare several initializations and findthe best results using OpinionFinder (Wilson et al.,2005). For the Spanish data, we performed initialsubjectivity analysis by matching against a publicly-available full-strength Spanish lexicon set (Rosas etal., 2012).

3.4 Implementation detailsBoth our implementation and the baselines arebuilt on the latent structural SVM (Yu andJoachims, 2009; http://www.cs.cornell.edu/˜cnyu/latentssvm/), which is in turnbuilt on the SVM-Light distribution (http://svmlight.joachims.org/). The regulariza-tion parameter C was chosen by cross-validation.

4 Results

Table 1 shows the sentiment analysis accuracy witheach system and feature set. The best overall re-sults in both language are given by the models with

810

system English Spanishtrue-multiword-connectors 91.25 79.80

true-unigram-connectors 91.36 77.50

auto-connectors 90.22 76.90all-unigram-connectors 87.60 74.30

No-connectors 88.21 76.42SVM 84.79 69.44

0.84 0.85 0.86 0.87 0.88 0.89 0.90 0.91 0.92sentiment analysis accuracy

SVM

no-connectors

all-unigram

auto-unigram

true-unigram

true-multiword

English

0.70 0.75 0.80sentiment analysis accuracy

SVM

no-connectors

all-unigram

auto-unigram

true-unigram

true-multiword

Spanish

Figure 1: Document-level sentiment analysis accuracy.The 95% confidence intervals are estimated from the cu-mulative density function of the binomial distribution.

connector-augmented transition features. In En-glish, the multiword and unigram connectors per-form equally well, and significantly outperform allalternatives at p < .05. The connector-based fea-tures reduce the error rate of the latent subjectivitySVM by 25%. In Spanish, the picture is less clearbecause the smaller test set yields larger confidenceintervals, so that only the comparison with the SVMclassifier is significant at p < .05. Nonetheless,the connector-augmented transition features give thebest accuracy, with an especially large improvementobtained by the multiword connectors.

Next, we investigated the quality of theautomatically-induced discourse connectors.The χ2 heuristic for selecting candidate connectorsgave results that were significantly better than theno-connector baseline in English, though the

Figure 2: Precision-Recall curve for top-K discoveredconnectors when compared with PDTB connector set

difference in Spanish was minimal. However, whenevery word is included as a potential connectors, theperformance suffers, dropping below the accuracyof the no-connector baseline. This shows that theimprovement in accuracy offered by the connectorfeatures is not simply due to the increased flexibilityof the model, but depends on identifying a small setof likely discourse connectors.

For a qualitative evalatuation, we ranked allEnglish-language unigram connectors by their fea-ture weights, and list the top ten for each subjectivitytransition:

• SHIFT: however; though; but; if; unlike; al-though; while; overall; nevertheless; still

• CONTINUATION: as; there; now; even; in; af-ter; once; almost; because; so

Overall these word lists cohere with our intu-itions, particularly the words associated with SHIFT

transitions: however, but, and nevertheless. As oneof the reviewers noted, some of the words associ-ated with CONTINUATION transitions are better seenas discourse cues rather than connectors, such asnow. Other words seem to connect two subsequentclauses, e.g., if Nicholas Cage had played every role,the film might have reached its potential. Incorporat-ing such connectors must be left for future work.

Finally, in learning weights for each connectorfeature, our model can be seen as discovering dis-course connectors. We compare the highly weighteddiscovered connectors from the all-unigram andauto-unigram settings with the one-word connec-tors from the Penn Discourse Tree Bank. The results

811

of this comparison are shown in Figure 2, whichtraces a precision-recall curve by taking the top Kconnectors for various values of K. The auto-unigram model is able to identify many true con-nectors from the Penn Discourse Treebank, whilethe all-unigram model achieves low precision. Thisgraph helps to explain the large performance gapbetween the auto-unigram and all-unigram fea-ture sets; the all-unigram set includes too manyweak features, and the learning algorithm is not ableto distinguish the true discourse connectors. TheSpanish discourse connectors identified by this ap-proach were extremely poor, possibly because somany more of the Spanish connectors include mul-tiple words.

5 Related Work

Polanyi and Zaenen (2006) noted the importance ofaccounting for valence shifters in sentiment analy-sis, identifying relevant connectors at the sentenceand discourse levels. They propose a heuristic ap-proach to use shifters to modify the contributionsof sentiment words. There have been several sub-sequent efforts to model within-sentence valenceshifts, including compositional grammar (Moilanenand Pulman, 2007), matrix-vector products acrossthe sentence (Yessenalina and Cardie, 2011), andmethods that reason about polarity shifters withinthe parse tree (Socher et al., 2012; Sayeed et al.,2012). The value of discourse structure towards pre-dicting opinion polarity has also demonstrated in thecontext of multi-party dialogues (Somasundaran etal., 2009). Our approach functions at the discourselevel within single-author documents, so it is com-plementary to this prior work.

Voll and Taboada (2007) investigate various tech-niques for focusing sentiment analysis on sentencesthat are central to the main topic. They obtainnegative results with the general-purpose SPADEdiscourse parser (Soricut and Marcu, 2003), butfind that training a decision tree classifier to iden-tify topic-central sentences yields positive results.Wiebe (1994) argues that in coherent narratives, ob-jectivity and subjectivity are usually consistent be-tween adjacent sentences, an insight exploited byPang and Lee (2004) in a supervised system forsubjectivity analysis. Later work employed struc-

tured graphical models to model the flow of sub-jectivity and sentiment over the course of the doc-ument (Mao and Lebanon, 2006; McDonald et al.,2007). All of these approaches depend on labeledtraining examples of subjective and objective sen-tences, but Yessenalina et al. (2010b) show that sub-jectivity can be modeled as a latent variable, using alatent variable version of the structured support vec-tor machine (Yu and Joachims, 2009).

Our work can be seen as a combination of themachine learning approach of Yessenalina et al.(2010b) with the insight of Polanyi and Zaenen(2006) that connectors play a key role in transitionsbetween subjectivity and sentiment. Eisenstein andBarzilay (2008) incorporated discourse connectorsinto an unsupervised model of topic segmentation,but this work only considered the role of such mark-ers to differentiate adjoining segments of text, andnot to identify their roles with respect to one an-other. That work was also not capable of learningfrom supervised annotations in a downstream task.In contrast, our approach uses document-level senti-ment annotations to learn about the role of discourseconnectors in sentence-level subjectivity.

6 Conclusion

Latent variable machine learning is a powerfultool for inducing linguistic structure directly fromdata. However, adding a small amount of linguisticknowledge can substantially improve performance.We have presented a simple technique for combin-ing a latent variable support vector machine witha list of discourse connectors, by creating an aug-mented feature set that combines the connectorswith pairwise subjectivity transition features. Thisimproves accuracy, even with a noisy list of connec-tors that has been identified automatically. Possibledirections for future work include richer representa-tions of discourse structure, and the combination ofdiscourse-level and sentence-level valence and sub-jectivity shifters.

Acknowledgments

Thanks to the anonymous reviewers for their help-ful feedback. This work was supported by a GoogleFaculty Research Award.

812

ReferencesFermin L. Cruz, Jose A. Troyano, Fernando Enriquez,

and Javier Ortega. 2008. Clasificacion de documen-tos basada en la opinion: experimentos con un cor-pus de crıticas de cine en espanol. Procesamiento deLenguaje Natural, 41.

Jacob Eisenstein and Regina Barzilay. 2008. Bayesianunsupervised topic segmentation. In Proceedings ofEMNLP.

Tom Huddleston. 2010. Review of The Bad Lieutenant:Port of Call New Orleans. Time Out, May 18.

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, DanHuang, Andrew Y. Ng, and Christopher Potts. 2011.Learning word vectors for sentiment analysis. In Pro-ceedings of ACL.

William C Mann and Sandra A Thompson. 1988.Rhetorical structure theory: Toward a functional the-ory of text organization. Text, 8(3).

Yi Mao and Guy Lebanon. 2006. Isotonic condi-tional random fields and local sentiment flow. InB. Scholkopf, J. Platt, and T. Hoffman, editors, Ad-vances in Neural Information Processing Systems 19.

Ryan McDonald, Kerry Hannan, Tyler Neylon, MikeWells, and Jeff Reynar. 2007. Structured models forfine-to-coarse sentiment analysis. In Proceedings ofACL.

Karo Moilanen and Stephen Pulman. 2007. Sentimentcomposition. In Proceedings of RANLP.

Bo Pang and Lillian Lee. 2004. A sentimental education:Sentiment analysis using subjectivity summarizationbased on minimum cuts. In Proceedings of ACL.

Livia Polanyi and Annie Zaenen. 2006. Contextual va-lence shifters. Computing attitude and affect in text:Theory and applications.

Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and BonnieWebber. 2008. The penn discourse treebank 2.0. InProceedings of LREC.

Veronica Perez Rosas, Carmen Banea, and Rada Mihal-cea. 2012. Learning sentiment lexicons in spanish. InProceedings of LREC.

Asad B. Sayeed, Jordan Boyd-Graber, Bryan Rusk, andAmy Weinberg. 2012. Grammatical structures forword-level sentiment detection. In Proceedings ofNAACL.

Richard Socher, Brody Huval, Christopher D. Manning,and Andrew Y. Ng. 2012. Semantic compositionalitythrough recursive matrix-vector spaces. In Proceed-ings of EMNLP-CoNLL.

Swapna Somasundaran, Galileo Namata, Janyce Wiebe,and Lise Getoor. 2009. Supervised and unsupervisedmethods in employing discourse relations for improv-ing opinion polarity classification. In Proceedings ofEMNLP.

Radu Soricut and Daniel Marcu. 2003. Sentence leveldiscourse parsing using syntactic and lexical informa-tion. In Proceedings of NAACL.

Kimberly Voll and Maite Taboada. 2007. Not all wordsare created equal: Extracting semantic orientation asa function of adjective relevance. In Proceedings ofAustralian Conference on Artificial Intelligence.

Janyce M. Wiebe. 1994. Tracking point of view in nar-rative. Computational Linguistics, 20(2).

Theresa Wilson, Paul Hoffmann, Swapna Somasun-daran, Jason Kessler, Janyce Wiebe, Yejin Choi, ClaireCardie, Ellen Riloff, and Siddharth Patwardhan. 2005.Opinionfinder: A system for subjectivity analysis. InProceedings of HLT-EMNLP: Interactive Demonstra-tions.

Ainur Yessenalina and Claire Cardie. 2011. Composi-tional matrix-space models for sentiment analysis. InProceedings of EMNLP.

Ainur Yessenalina, Yejin Choi, and Claire Cardie. 2010a.Automatically generating annotator rationales to im-prove sentiment classification. In Proceedings of ACL:Short Papers.

Ainur Yessenalina, Yisong Yue, and Claire Cardie.2010b. Multi-Level structured models for Document-Level sentiment classification. In Proceedings ofEMNLP.

Chun-Nam John Yu and Thorsten Joachims. 2009.Learning structural svms with latent variables. In Pro-ceedings of ICML.

Omar F. Zaidan, Jason Eisner, and Christine Piatko.2007. Using ”annotator rationales” to improve ma-chine learning for text categorization. In Proceedingsof HLT-NAACL.

813