Hidden Topic-Emotion Transition Model forMulti-Level Social Emotion Detection

Donglei Tanga, Zhikai Zhangb, Yulan Hec, Chao Linb, Deyu Zhoub,∗

aSchool of Mathematics and Statistics, Nanjing Audit University, Nanjing, ChinabSchool of Computer Science and Engineering, Key Laboratory of Computer Network and

Information Integration, Southeast University, Nanjing, ChinacDepartment of Computer Science, University of Warwick, UK

Abstract

With the fast development of online social platforms, social emotion detection,focusing on predicting readers’ emotions evoked by news articles, has been in-tensively investigated. Considering emotions as latent variables, various prob-abilistic graphical models have been proposed for emotion detection. However,the bag-of-words assumption prohibits those models from capturing the inter-relations between sentences in a document. Moreover, existing models can onlydetect emotions at either the document-level or the sentence-level. In this paper,we propose an effective Bayesian model, called hidden Topic-Emotion Transitionmodel, by assuming that words in the same sentence share the same emotionand topic and modelling the emotions and topics in successive sentences as aMarkov chain. By doing so, not only the document-level emotion but also thesentence-level emotion can be detected simultaneously. Experimental results onthe two public corpora show that the proposed model outperforms state-of-the-art approaches on both document-level and sentence-level emotion detection.

Keywords: Social Emotion Detection, Sentiment Analysis, Topic Model,Hidden Topic-Emotion Transition Model

1. Introduction

With the fast development of social web, more and more users tend to sharetheir opinions about social events, post discussions of political movements andexpress their preferences over products and services on online platforms [1, 2].Some news portals, for example, the Sina news, allows readers to vote for theiremotions after reading a news article. Essentially, each news article could be

∗Corresponding author.Email addresses: tdonglei@nau.edu.cn (Donglei Tang), 220151517@seu.edu.cn (Zhikai

Zhang), y.he@cantab.net (Yulan He), 220171708@seu.edu.cn (Chao Lin),d.zhou@seu.edu.cn (Deyu Zhou)

Preprint submitted to Knowledge-based Systems November 9, 2018

associated with multiple emotions of its readers over a predefined set of emo-tion categories. An example of a news article with the readers’ emotion votes isshown in Figure 1. Here, the readers’ emotion votes form the document-level e-motion labels which are observed. We assume that such document-level emotionlabels can be inferred from the unobserved sentence-level topics and emotions.In Figure 1, we manually annotate the sentence-level topics and emotions anddisplay them against a shaded background to denote that they are unobserved.

EmotionSentence

Topic

妈妈突然车祸去世11岁男孩作文看哭老师，忍不住把它发到了网上。The teacher read the 11-year-old boy's essay with tears in her eyes whose mother

passed away in a car accident suddenly and couldn't help posting it on the Internet.

Touching

“家里人一直瞒着我,可我却早已知道了,那是我睡觉时没睡着听到的。”把一切都藏在心底的冬冬，在后来的一次考试中，将所有对妈妈的思念和不舍

写在了作文里。"My family have been kept it from me but I knew this long time ago, I heard it

when I am sleeping." Hiding everything in the heart, Dongdong expressed his

thoughts and memories of his mother in the essay in a later exam.

“妈妈去世了。”“我大概猜到了……”两句简单的对话后，冬冬就回房写

作业了，虽然爸爸带来的噩耗让他瞬间红了眼眶，但他并没有立即哭出来。"Your mom passed away.", "I propably guessed it", after the conversation,

Dongdong went to bedroom to do homework, though bad news from his dad made

his eyes turn red, he didn't cry immediately.

Readers’emotion votes

Touching

Sadness

Moved by

essay

Missing someone

Death of

family

Figure 1: Excerpt of a news article from Sina and its corresponding reader votes over prede-fined emotion classes. Sentence-level topics and emotions in the left shaded pane are manuallyannotated and unobserved.

Approaches for social emotion detection can be categorized into two types:discriminative-model based and topic-model based. By casting emotion detec-tion into a classification problem, many discriminative-model based methodshave been proposed [3, 4, 5], such as the logistic regression model with emo-tion dependency [6] and the social opinion mining model based on K-NearestNeighbour (KNN) [7]. However, such approaches are unable to reveal the latenttopic information in order to understand how the emotions are evoked. To ad-dress this problem, topic-model based approaches have been largely applied forsocial emotion detection [8, 9]. Generally, the hidden topics and emotions aremined simultaneously from the documents by adding an emotion layer into topicmodels such as Latent Dirichlet Allocation (LDA) [10]. However, most of themmake the “bag-of-words” assumption, i.e., the order of words is ignored and thetopic or emotion assignment of each word in the document is independent fromeach other. The simplified assumption ignores the structural information in adocument, which might be critical for social emotion detection. A news articlecontains a title typically summarizing an event, followed by several structured

2

and coherent groups of sentences describing the detailed information of the even-t. As can be seen in Figure 1 that (1) one sentence typically only describes asingle event/topic; (2) the event or topic directly triggers reader’s emotions; (3)successive sentences tend to share the same emotion or topic.

Based on the above observations, we propose a novel probabilistic graphi-cal model, called hidden Topic-Emotion Transition (TET) model to model theemotion and topic transitions simultaneously. In specific, TET considers eachsentence as a basic unit and all the words in the same sentence are assumedto share the same topic label and the same emotion label. The topic transi-tion and emotion transition in successive sentences are modelled via a hiddenMarkov model and learned from data, in a way avoiding the document-levelbag-of-words assumption. By doing that, TET can detect both the document-level and the sentence-level emotions though it only requires data with thedocument-level emotion labels for training. We conduct experiments on the twopublic corpora, a news dataset and a blog dataset. Experimental results showthat TET outperforms several state-of-the-art approaches for both document-level and sentence-level emotion detection and also detects underlying topicswhich evoke the corresponding emotions.

2. Related Work

Our work is related to two lines of research: social emotion detection andtopic models for emotion/sentiment analysis [11].

2.1. Social Emotion Detection

Approaches for social emotion detection can be mainly divided into twocategories: discriminative-model based and topic-model based.

For the first category, social emotion detection is often casted as a classifi-cation problem. If only choosing the strongest emotion as the label for a giventext, emotion detection is essentially a single-label classification problem. Lin etal. [12] studied the classification of news articles into different categories basedon readers’ emotions with various combinations of feature sets. Strapparava andMihalcea [13] proposed several knowledge-based and corpus-based methods foremotion classification. Quan et al. [6] proposed a logistic regression model withemotion dependency for emotion detection. Latent variables were introducedto model the latent structure of input text. Li et al. [14] combined biterm top-ic model and conventional neural network to detect single social emotion fromshort texts. Li et al. [15] incorporated domain-specific and universal knowledgeinto a unified sentiment classification framework to build a domain-adaptive sen-timent classification model. Label propagation was employed to unify universaland domain-specific sentiment lexicons.

To predict multiple emotions simultaneously, emotion detection can be solvedusing multi-label classification. Bhowmick [16] presented a method for classi-fying news sentences into multiple emotion categories using an ensemble basedmulti-label classification technique. Wang et al. [17] output multiple emotions

3

with intensities using non-negative matrix factorization with several novel con-straints such as topic correlation and emotion bindings. To predict multipleemotions with different intensities in a single sentence, Zhou et al. [18] proposeda novel approach based on emotion distribution learning. In the same vein, arelevant label ranking framework for emotion detection is proposed for predictmultiple relevant emotions as well as the ranking of emotional intensity [19, 20].A KNN-like approach called social opinion mining model (SOMM) was proposedin [7] where word embeddings were used to calculate word mover’s distance ofall the documents via solving an optimizing problem. The average emotiondistribution of the k nearest neighbors of a test document was then used foremotion prediction.

As for aspect-level sentiment classification, it can be seen as a fine-grainedtask in emotion detection. Wang et al.[21] employed an attention-based longshort-term memory network for aspect-level sentiment classification. It can con-centrate on different parts of a sentence when different aspects are consideredas input. As word embedding has been proven to be a useful model for senti-ment analysis and emotion detection, Li et al. [22] incorporated prior sentimentinformation into the embedding process to investigate the influence each wordhas on the sentiment label of both target word and context words. It can learnbetter representations for sentiment analysis.

However, discriminative-model based approaches are unable to detect andincorporate latent topic information to understand how the emotions are evoked.To address this problem, topic-model based approaches have been largely ap-plied for social emotion detection [8, 9]. For example, the Emotion-Topic Model(ETM) [23] or the Sentiment Latent Topic Model (SLTM) [8] are similar tothe Joint Sentiment-Topic (JST) model [24] in which an additional emotion orsentiment layer is inserted between the topic layer and the observed words inLDA so as to detect emotion- or polarity-bearing topics. Contextual SentimentTopic Model (CSTM) [9] assumes each word is either drawn from a backgroundtheme, a contextual theme or a topic. Emotion classification is based on context-independent topics.

Our proposed approach belongs to the second category. However, while mostexisting topic model approaches are based on the bag-of-words assumption,our TET model takes the sentence sequential information into considerationand models the topic/emotion transitions explicitly which is crucial for socialemotion detection.

2.2. Topic Models for Emotion/Sentiment Analysis

Emotion detection, classifying text into one or more emotion categories suchas ‘Happy ’, ‘Sad ’ and ‘Surprise’, is closely related to sentiment analysis whichtypically classifying text into one of the three polarity categories, ‘Positive’,‘Negative’ or ‘Neutral ’. As such, we review topic modeling approaches to emo-tion or sentiment detection here.

Many topic-model based approaches have been proposed for detecting bothsentiments and topics [25]. For example, the topic sentiment mixture mod-el [26] jointly modelled topics and sentiments, and sampled a word either from

4

the background model or topical themes where the latter are further catego-rized into three sub categories, i.e., neutral, positive and negative sentimentmodels. The Joint Sentiment-Topic (JST) [24] inserted an additional sentimen-t layer into LDA for polarity-bearing topic detection. Aspect and sentimentunification model (ASUM) [27] extended JST by enforcing words in a singlesentence to share the same topic and sentiment label. Emotion-Topic Model(ETM) [23] first generated a set of latent topics from emotions, followed bygenerating affective terms from each topic. Contextual Sentiment Topic Model(CSTM) [9] distinguished context-independent topics from both a backgroundtheme, which characterizes non-discriminative information, and a contextualtheme, which characterizes context-dependent information across different col-lections. Two models have been proposed in [8]. One is Multi-Label SupervisedTopic Model (MSTM), which generated a set of topics from words firstly, fol-lowed by sampling emotions from each topic. The other is Sentiment LatentTopic Model (SLTM), which generated topics from social emotions directly.

However, these models make the bag-of-words assumption and ignore wordordering. To address this problem, some approaches have explicitly consid-ered the order of words and the structure of sentences for topic generation.For example, [28] assumed that words are either generated from the syntacticclasses which are drawn from the previous syntactic class or from topics thatare drawn randomly. To model topic transition, Hidden Topic Markov Model(HTMM) [29] modelled the topics of words in a document as a Markov chainand assumed that topic transitions can only occur between sentences. In a sim-ilar vein, structural topic model [30] assumed that words in a sentence share thesame topic. Hidden Topic Sentiment Model (HTSM) [31] extended HTMM andmodels the combination of aspect and sentiment label of sentences as a Markovchain. It assumed that words within a sentence share the same aspect label andsentiment label and constrained the transitions with the assumption that onlyone sentiment polarity can be associated with a specific aspect in a document.

Our proposed TET model is partly inspired by HTSM, but differs fromHTSM in several aspects: (1) HTSM only considers binary sentiment polar-ities and models the sentiment transitions simply by a switch variable, whileTET models multi-class emotion transitions and takes emotion correlations in-to consideration; (2) Different from the assumption of HTSM that one topic ina document can only be associated with one polarity, in TET, one topic canbe associated with different emotions in a document since the same topic mightevoke different emotions of different readers. (3) HTSM utilizes the predefinedaspects and is trained on documents annotated with aspect labels, while no suchinformation is available in our data.

3. Methodology

In this section, we propose the hidden Topic-Emotion Transition (TET)model. In specific, TET considers each sentence as a whole and assumes allthe words in the same sentence share the same topic label and the same emo-tion label. It models the topic/emotion transition in successive sentences via a

5

hidden Markov model, essentially avoiding the bag-of-words assumption at thedocument-level.

3.1. Hidden Topic-Emotion Transition Model

Assuming that we have a set of documents denoted as D = {d1, d2, ..., d|D|},where each document d consists of md sentences denoted as S = {s1, s2, ..., smd

},and each word in the sentences is an item from a vocabulary with V distinctterms denoted as {1, 2, ..., V }. Let E be the number of distinct emotion labels,and an emotion label is denoted as e ∈ {1, 2, ..., E}. Also, T is the total numberof topics and a topic label is denoted as t ∈ {1, 2, ..., T}. For a given documentd, its document-specific topic-emotion proportion θd is assumed to be drawnfrom a shared Dirichlet distribution, i.e., θd∼Dir(α). Each sentence si in d hasNsi words and is associated with an emotion label ei and a topic label ti, whichis sequentially drawn from a document-specific Markov chain. As the topic andemotion labels of sentences are unobservable, we introduce a latent variable ψto control the topic transition and τ to control the emotion transition. Theplate diagram of TET is given in Figure 2. In what follows, we will describehow topic and emotion transitions are modelled before presenting the overallgenerative process for TET.

α

θ

e1 e2 • • • emd

t1 t2 • • • tmd

w w • • • w

• • •

Ns1 Ns2Nsmd

ϕ

E×Tβ

D

τ2 ψ2 τmd ψmd

λ γ ε

Figure 2: Graphical model for the proposed hidden Topic-Emotion Transition (TET) model.

3.1.1. Topic Transition

Since topics are inferred in a totally unsupervised manner without any priorknowledge, it is difficult to model the probabilities of topic transitions explicitlyas a transition matrix. Therefore, we employ ψi as a switch variable to indicatewhether there is a topic transition between si−1 and si. As observed in Figure 1,

6

two successive sentences are less likely to share similar topics if their contentsare very different. Therefore, it is reasonable to assume that the more similartwo sentences are in terms of their content, the less likely there will be a topictransition between them. Several linguistic features in the adjacent sentences areemployed to guide the topic transition. The probability of the switch variableψi is defined as follows:

p(ψi|si, si−1, si+1, ε) =1

1 + exp(−εTft(si, si−1, si+1)), (1)

where ε is the feature weights and ft(si, si−1, si+1) is the topic transition featurefunction which takes the current sentence si, the previous sentence si−1 and thenext sentence si+1 as input and outputs a feature vector. It includes: (1)content-based cosine similarity between si and si−1; (2) sentence length ratioof si to si−1; (3) the relative position of si in d; (4) the content-based cosinesimilarity difference between (si, si−1) and (si, si+1). The first three featuresare inspired by [31], while the last one assumes that there is no topic transitionfrom the previous sentence to the current sentence if the current sentence ismore similar to the previous sentence than the next one.

3.1.2. Emotion Transition

Assuming that an emotion lexicon consisting of words with their correspond-ing emotion distributions is available, i.e., each word is associated with multipleemotion labels with different intensities. We use ES(w, e) to denote the emotionscore of a word w associated with emotion e. We will show later in this sectionhow to generate the emotion lexicon from the training data.

As some emotions co-occur more often that the others, we believe that suchinformation would be useful for the derivation of emotion transitions. The Pear-son coefficient between two emotions ej and ek, denoted as λej ,ek is calculatedfrom the training set in the following way: First, each emotion ej is representedas a vector EVj with D dimensions where D is the number of documents in thetraining set and each element of the vector, EVj [i], is the number of votes foremotion ej in the ith document. Then, the Pearson coefficients between twoemotions ej and ek is calculated as:

λej ,ek =cov(EVj , EVk)

σEVjσEVk

. (2)

where cov(·) denotes the covariance and σ denotes standard derivation. Thevalue of Pearson coefficient ranges from −1 to 1, with a value of 1 indicatingpositive correlation and −1 indicating negative correlation. A value of 0 showsthat the two emotions are independent from each other. We convert the Pearson

coefficient values to the range of [0, 1] usingλej,ek

+1

2 so that they can be usedas the probabilities measuring how strongly two emotions ej and ek are relatedto each other.

After defining ES(w, e) and λej ,ek , we can construct τi, the E×E emotiontransition matrix for sentence si. τi,ej ,ek indicates the probability of emotion

7

transition from ej to ek for si. The transition probability from ej to ek insentence si is determined by their Pearson coefficient weighted by the sum ofthe normalized emotion scores of ej in si−1 and ek in si, which is defined asfollows:

p(τi,ej ,ek |si, si−1, λ, γ)∝λej ,ek + 1

2× (

Nsi−1∑p=1

ES(wp, ej)

Nsi−1

+

Nsi∑q=1

ES(wq, ek)

Nsi) + γ

(3)A smoothing parameter γ = 0.001 is incorporated to avoid zero probabilities.

3.1.3. Generative Process

Based on the ψi and τi,ej ,ek defined above, the topic-emotion transitionprobability is designed in the following ways: (1) If ψi = 0 and j = k, thereis no change in either topic or emotion label for si; (2) If ψi = 1 and j = k,the emotion label for si remains the same, but a new topic will be drawn underthe same emotion label from θd. Therefore, ti 6= ti−1 and ei = ei−1; (3) Ifψi = 1 and j 6= k, a new topic label and a new emotion label will be drawnfrom θd. As we expect that if the topic label stays the same, the emotion labelwould also stay the same, there is no definition for ψi = 0 and j 6= k. Therefore,the transition probability p(ei, ti|τi,ei−1,ei , ψi, ei−1, ti−1, θ, si−1, si, ε, λ, γ) can beformally specified as,

p(ψi = 0)p(τi,ei−1,ei), if ei = ei−1, ti = ti−1, i≥2

p(ψi = 1)p(τi,ei−1,ei)θei,ti , if ei = ei−1, ti 6= ti−1, i≥2

p(ψi = 1)p(τi,ei−1,ei)θei,ti , if ei 6= ei−1, ti 6= ti−1, i≥2

θei,ti , if i = 1

0, otherwise (4)

The overall generative process for TET is presented below:

• For each possible topic and emotion label combination (e, t), draw ϕe,t∼Dir(β).

• For each document d ∈ D,

– Draw topic-emotion proportion θd ∼ Dir(α).

– For each sentence si ∈ d,

∗ If i = 1, set ψi = 1.

∗ Else

· Calculate topic transition probability p(ψi|si, si−1, si+1, ε).

· Calculate emotion transition matrix p(τi,ej ,ek |si, si−1, λ, γ) ,1≤j, k≤E.

∗ Perform block sampling (ei, ti) by (ei, ti)∼

p(ei, ti|τi,ei−1,ei , ψi, ei−1, ti−1, θ, si−1, si, ε, λ, γ).

∗ Sample each word wn in si, wn∼Mul(ϕei,ti).

8

3.2. Parameter Estimation

In TET, the parameters can be estimated efficiently using the ExpectationMaximization (EM) algorithm. As TET can be seen as a special type of HMM,the customized forward-backward algorithm and Viterbi algorithm in E-step ofeach iteration to accumulate the sufficient statistics and then update parametersin M-step are employed.

The latent variables in TET are the emotion label assignments e, the topiclabel assignments t, the emotion transition indicator τ and the topic transitionindicator ψ. The combinations of (ei, ti, ψi, τi) at sentence si is considered ashidden states in our Markov chain of document d.

The parameters of TET needed to be updated iteratively are θ, ϕ and ε.The hyper-parameters α and β are set empirically and λ is pre-calculated in-corporating the emotion correlations. Let CDd,e,t denote the total number ofwords in d associated with emotion label e and topic label t which are drawnfrom θd, CWw,e,t denote the number of times that word w is associated withthe emotion label e together with the topic label t.

In E-step, the forward-backward algorithm along with Viterbi algorithm isexecuted and the following sufficient statistics are accumulated.

E[CDd,e,t] =

md∑i=1

p(ei = e, ti = t, ψi = 1|d)Nsi (5)

E[CWw,e,t] =

|D|∑d=1

md∑i=1

Nsi∑j=1

δ(wj = w)p(ei = e, ti = t|d) (6)

E[ψi] = p(ψi = 1|d), s.t. i≥2 (7)

In M-step, the parameters θ, ϕ and ε are updated as follows,

θd,e,t∝E[CDd,e,t] + α (8)

ϕe,t,w∝E[CWw,e,t] + β (9)

ε = arg maxε

|D|∑d

md∑i=1

E[ψi]log p(ψi = 1|d, ε)

+ (1− E[ψi])log (1− p(ψi = 1|d, ε)),(10)

where ε is updated by optimizing a cross-entropy loss function via a gradient-based optimizer.

3.3. Emotion Lexicon Generation from the Training Data

In TET, an emotion lexicon is needed to calculate ES and initialize ϕ. Weexplored two ways of constructing the emotion lexicon.

9

One is based on an existing emotion lexicon, cev, provided in the Chineseemotional vocabulary ontology library1. It contains seven top-level emotion cat-egories including “happy”,“good”,“angry”, “sad”, “fear”, “disgust” and “sur-prise”. As the emotion categories in cev are not exactly the same as the emotionlabels annotated in our data, they are aligned manually. Moreover, to accom-modate uncertainties in emotion alignment, each term in cev is assigned withthe score of 0.9 for its corresponding emotion category in cev and the remaining0.1 is equally spread among all the other emotions categories. Emotion lexiconconstructed in this way is referred to as lex cev.

The other way is to construct the emotion lexicon from the training data,which is inspired by the lexicon generation method [32]. The emotion lexicon isrepresented as the term-by-emotion matrix. It can be derived by matrix multi-plication between the term-by-document matrix and the document-by-emotionmatrix, which can be easily constructed from the training data. For example,each document’s emotion labels in the training data corresponds to one rowof the document-by-emotion matrix. As there are many ways of representingdocument such as raw frequencies, normalized frequencies, and term frequency-inverse document frequency (TF-IDF), three different emotion lexicons denotedby lex f, lex nf and lex tfidf are constructed respectively.

4. Experiments

4.1. Setup

To evaluate the performance of the proposed approach for social emotiondetection, we conduct experiments on the News Dataset 2016 (ND16) [7], whichwas collected from the social channel of Sina between January and December of2016, consisting of 5,258 news documents with 6 reader emotions (i.e., Touching,Anger, Amusement, Sadness, Shock and Curiosity). The statistics are shownin Table 1. Following the same setup as in [7], the first 3,109 documents inchronological order are used for training and the rest 2,149 are for testing. Pre-processing is conducted on ND16 by first splitting each document into sentencesthrough sentence boundary detection, then removing stop words, numbers, per-son names, punctuations and words appeared in less than 3 documents.

As mentioned before, λej ,ek is calculated based on Pearson correlation coeffi-cient of emotions in the training set describing the degree of correlation betweentwo emotion, which is shown in Figure 3. Dark color indicates that two emotionsare positively correlated, e.g., Shock and Curiosity, while light color indicatesthat two emotions are negatively correlated, e.g., Anger and Touching. It is clearthat the results are in line with our intuition, which justifies the incorporationof such information into our model.

Our TET model can output both document-level and sentence-level emo-tion labels. ND16 only contains news articles annotated with document-level

1http://ir.dlut.edu.cn/EmotionOntologyDownload/

10

Table 1: Statistics of the News Dataset 2016 (ND16).

Emotion label# of documents # of votesTrain Test Train Test

Touching 464 365 313,998 175,022Anger 1077 831 835,644 583,826Amusement 977 533 613,334 313,455Sadness 393 293 331,450 230,035Shock 123 93 292,523 160,670Curiosity 75 33 133,995 66,682

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1Tou

ching

Shock

Amusement

SadnessCuriosity

Anger

Touchin

g

Shock

Amusement

Sadness

Curiosity

Anger

Figure 3: Gray-scale image of emotion correlations in the News Dataset 2016.

11

emotions from readers’ perspective. It is thus infeasible to use it to evaluatethe accuracy of the sentence-level emotion detection of TET. Therefore, we useanother corpus, RenCECps [33], which contains both the document-level andsentence-level emotion annotations. An example of an excerpt of a blog withits emotion annotations is presented in Table 2. It is worth mentioning that forboth ND16 and RenCECps, only the document-level emotion annotations areused for training TET. The sentence-level emotion annotations of RenCECpsare only used for performance evaluation of sentence-level emotion detection onthe test set.

Table 2: An example of a part of one blog with its emotion annotation.

Document level Joy:0.0 Hate:0.0 Love:0.0Sorrow: 0.7 Anxiety: 0.7Surprise: 0.0 Anger: 0.0Expect: 0.0

Sentence 1: ù�¯Ký�éJ£�§·�,¬3�³��\�«����g¥"(Thisquestion is hard to answer, and I will suddenlyfall in deep meditation in dusk.)

Joy:0.0 Hate:0.0 Love:0.0Sorrow: 0.0 Anxiety: 0.9Surprise: 0.0 Anger: 0.0Expect: 0.0

Sentence 2: Ø��gC´3Z�o§� %päO�«�Õa§½Nù«�ÕÒ´á(j"(Do not know what I am doing, but feellonely. Maybe such loneliness is desolation.)

Joy:0.0 Hate:0.0 Love:0.0Sorrow: 0.8 Anxiety: 0.7Surprise: 0.0 Anger: 0.0Expect: 0.0

The RenCECps corpus contains 1,487 blog documents with 35,381 sentences.Each sentence is annotated with 8 basic emotions (i.e., Joy, Hate, Love, Sorrow,Anxiety, Surprise, Anger and Expectation), together with their emotion inten-sities as shown in Table 2. The statistics of RenCECps are presented in Table 3.The first 1,187 documents consisting of 27,528 sentences by chronological orderare used for training and the rest 300 documents consisting of 7853 sentencesare for testing. The emotion correlations in RenCECps are shown in Figure 4,which indicates Hate and Anger is highly positively related, and that Love andAnxiety is highly negatively related.

4.2. Baselines and Evaluation Metrics

The baselines can be roughly divided into two categories, classification mod-els and topic-model based. Classification models can be further classified intothree sub-categories, word-level based, optimization based, neural network basedmethods. Word-level based approaches include Emotion-Term model (ET) [23]and Supervised UnigrAm Model (SWAT) [34]. ET follows the Naıve Bayesmethod by assuming words are independently generated from social emotionclasses. SWAT is a supervised approach using an uigram model trained toannotate emotion content. SOMM [7] is one of the optimization approacheswhich employs word embeddings to calculate word mover’s distance of all thedocuments via solving an optimization problem. ConstraintOp [17] employs

12

−0.5

0

0.5

1Joy Hate Love

SorrowAnxiety

SurpriseAnger

Expect

Joy

Hate

Love

Sorrow

Anxiety

Surprise

Anger

Expect

Figure 4: Gray-scale image of emotion correlations in RenCECps.

Table 3: Statistics of RenCECps.

Emotion labelnumber of documents number of sentencesTrain Test Train Test

Joy 275 26 5766 899Hate 110 0 2237 118Love 345 108 6506 2636Sorrow 145 91 4188 1754Anxiety 121 44 4731 1918Surprise 4 0 468 60Anger 34 1 989 133Expectation 153 30 2643 335

constraints such as emotion bindings and topic correlations for emotion detec-tion from social media. The input features are generated using the traditionalTF-IDF features or long short term memory (LSTM) network. Neural networkbased approaches include CNN [3, 4], CNN-SVM [5] and LSTM with atten-tion [35]. The CNN uses CNN directly while CNN-SVM uses CNN to constructfeatures first and then trains Support Vector Machine (SVM) based on the CNN-extracted features for emotion classification. LSTM with attention employs twokinds of attention mechanisms on top of word embeddings.

Topic models based approaches include JST, ASUM, ETM [23], CSTM [9],MSTM [8], SLTM [8] and the Affective Topic Model (ATM) [36]. JST as-sumes that each word is associated with a sentiment label and a topic label.ASUM assumes that each sentence is associated with a sentiment label and atopic label instead. ETM first generates a set of latent topics from emotions,followed by generating affective terms from each topic. CSTM distinguishescontext-independent topics from both a background theme, which characterizesnon-discriminative information, and a contextual theme, which characterizescontext-dependent information across different collections. MSTM generates a

13

set of topics from words firstly, followed by sampling emotions from each topic.SLTM generates topics from social emotions directly. ATM is a multi-labeledtopic model which is homologous to JST in generative process.

Two evaluation metrics are employed: accuracy at the top one predictionAcc@1 and averaged Pearson correlation coefficient over all documents AP . Fora document d in the test set D, given the top one predicted emotion edp, the top

one ground truth emotion edg, Accd@1 is defined as:

Accd@1 =

{1, if edpp = edg

0, otherwise(11)

Acc@1 can then be calculated by averaging Accd@1 over all documents.For a document d in the test set D, given the predicted emotion distribution

rdp and the ground truth emotion distribution rdg , AP is defined as

AP =1

|D|

|D|∑d=1

cov(rdp, rdg)

σrdpσrdg, (12)

where cov is the covariance and σ is the standard deviation.

4.3. Results

The overall performance of the proposed approach and the baselines on ND16is shown in Table 4. For TET, the number of topics is set to 10 and the emotionlexicon lex nf is used. For the baseline results reported here, the results ofConstraintOp and LSTM with attention were obtained by running the authorpublished source code on our data.p JST and ASUM results were generatedbased our adaptation of the published source code since both models were notfor social emotion detection originally. For TET, JST and ASUM, the hyperpa-rameter values, α = 0.01, β = 0.001, γ = 0.001, were set empirically. JST andASUM also used the same emotion lexicon as in TET for the incorporation ofword prior emotion information. The results of other baselines are cited from [7]since we used the same dataset and the same experimental setup here.

It can be observed that TET outperforms all the other approaches on bothAcc@1 and AP metrics. In specific, the word-level models assume words areindependent from each other, with global document context ignored, whiletopic-model based approaches achieve better results with latent topics encodingdocument-level global context. However, none of them consider the associationsof successive sentences. Neural network based methods perform better thanword-level models but poorer than topic-model based approaches. However, theLSTM based method performs the worst. It might attribute to the complexi-ty of the documents. LSTM based approaches, usually working on sentences,are unable to handle such long sequence. SOMM utilizes word embeddings toconstruct a network of documents, which take into consideration semantic cor-relations between words to some extent. Therefore, it performs slightly betterthan the best result in the category of topic-model based approaches. Still, it

14

Table 4: Overall performance of the proposed approach and the baselines on ND16.

Category Approach Acc@1(%) APD

iscr

imin

ati

ve Word-level modelET 48.04 0.43SWAT 38.97 0.40

Neural network basedCNN-SVM 52.63 -CNN 51.23 -LSTM with attention 21.29 -

Optimization basedConstraintOp using TF-IDF features 28.21 0.21ConstraintOp using LSTM features 24.81 0.31SOMM 58.59 0.64

Top

ic-m

odel

base

d JST 58.33 0.63ETM 54.19 0.49ASUM 43.16 0.39CSTM 40.74 0.43MSTM 32.17 0.34ATM 29.20 0.28SLTM 28.95 0.26

Ours TET 61.83 0.66

dose not model the topic/emotion transitions directly. Instead, our proposedTET encodes topic/emotion transitions into the generative process of the topicmodel and thus achieves the best results overall.

To further investigate the performance of TET in comparison with JST andASUM, experiments are conducted on ND16 using the same lexicon lex nf withdifference topic number settings. Figure 5 shows the performance of TET, JSTand ASUM under topic numbers in [1, 5, 10, 15, 20, 50]. It can be observed thatTET achieves better performance compared to JST and ASUM regardless ofthe topic number setting. It can also be observed that the performance of TETunder different topic numbers is relatively stable while the results of JST andASUM fluctuate greatly.

1 5 10 15 20 503537394143454749515355575961

Number of topics

Acc

@1 TET

JSTASUM

(a) Acc@1 results.

1 5 10 15 20 500.280.310.340.37

0.40.430.460.490.520.550.580.610.640.67

Number of topics

AP

TETJSTASUM

(b) AP results.

Figure 5: Performance comparison of TET, JST and ASUM with different number of topicson ND16.

15

4.3.1. Impact of Emotion Lexicon and Topic Number

We also investigate the performance of TET with different emotion lexiconsand different topic numbers. Figure 6 shows the performance of TET on ND16using different lexicons, such as, lex nf, lex f, lex tfidf and lex cev underthe topic number in [1, 5, 10, 15, 20, 50].

1 5 10 15 20 504244464850525456586062

Number of topics

Acc

@1

lex_nflex_flex_tfidflex_cev

(a) Acc@1 results.

1 5 10 15 20 500.4

0.420.440.460.48

0.50.520.540.560.58

0.60.620.640.660.68

Number of topicsA

P

lex_nflex_flex_tfidflex_cev

(b) AP results.

Figure 6: Performance of TET on ND16 using different emotion lexicons under differentnumber of topics.

It can be observed that using the generated emotion lexicons from the train-ing data, such as lex nf, lex f and lex tfidf, TET achieves significantlybetter results compared to lex cev. The worst results using lex cev might beattributed to the indirect mapping of emotion categories and the differences ofthe data corpora used. As there is no no one-to-one correspondence betweenthe emotion categories in the emotion lexicon, cev, and the readers’ emotionsin ND16, manual alignment of emotion categories might introduce some noises.Moreover, the data from which cev was derived and the social emotion datasetwe used here are different. Therefore, it is more effective to automatically gen-erate an emotion lexicon from our data directly. It can also be observed that thenumber of topics has less impact on the performance of TET although overallthe best Acc@1 and AP results are obtained when the number of topics is setto 50.

4.3.2. Extracted topics

Apart from social emotion detection, TET can also extract topics from da-ta. To evaluate the effectiveness of topics and emotions captured by TET, thetop 10 words associated with the combination of one topic and one emotionextracted using Equation 9 are shown in Table 5. Under each of the six pre-defined emotions, two example topics are listed. It can be observed that mosttopics correspond to some events in social news. For example, topic 1 underthe “Touching” emotion is about organ donation; topic 2 under the “Amuse-ment” emotion is about live broadcasting on an online platform; topic 2 underthe “Shock” emotion is about breast implant surgery. The topic results clearly

16

show the triggering events associated with certain social emotions.

Table 5: An example of topics extracted by TET under different emotions on ND16 dataset.Touching Anger Amusement

Topic 1 Topic 2 Topic 1 Topic 2 Topic 1 Topic 2��(hospital) P�(teacher) v¦<(suspect) /c(metro) ç¦(lottery) �Â(live broadcasting)�)(doctor) Æ)(student) ��(commit crimes) À�(truck) [�(parents) ²�(platform)Ãâ(surgery) Í<(rescue) y|(scene) ~¬(villager) ì¡(photo) �ä(network)ì((organ) �Ï(help) Y�(case) ¦�(passenger) ù�(red packet) ÌÂ(anchor)ì¡(photo) ÆS(learn) úSÛ(public security bureau) {�(court) *l(friend) úi(company)%9(heart) y|(scene) �´(call the police) y|(scene) &E(message) å¯(girl)�ö(patient) �(run) �Ѥ(police station) iÅ(driver) ¥øö(winner) ®j(fan)íz(donate) �Â]�(samaritan) rr(rape) k�(die) ø7(bonus) �Ѥ(police station)£�(therapy) O%(love) 8¼(arrest) �w<(defendant) Õ1(bank) åÌÂ(camgirl))·(life) �~(village) �´(traffic police) ��(carriage) ó�<(staff) �Õ(website)

Sadness Shock CuriosityTopic 1 Topic 2 Topic 1 Topic 2 Topic 1 Topic 2

�N(dead body) y|(scene) ~¬(villager) Ãâ(surgery) S�(Spring Festival Gala) ~(fish)a¢(jump from a building) >F(elevator) ��(boa) �)(doctor) ̱<(host) �~(big fish)r�(get lost) ¯u(incident) ÄÔ(animal) �9(breast implant) �ò(smiling face) �¬(fisherman)O(love) �«(community) ;[(specialist) ��(hospital) ²((star) �7(Phoebe zhennan)U�(like) k�(die) �(boa) 9Ü(breast) ̱(preside over) Ó¼(catch)F"(hope) Í�(rescue) Ã<«(depopulated zone) 5�(injection) *¯(audience) ~¬(villager)��(doggie) »/(fire) �o(protect) �Þ(ophicephalous) À(CCTV) �)(doctor)�l(missing) ËA(hotel) �(snake) JÈ(jelly) ü(monkey) ¿Ú(crow)P[(hometown) �Ѥ(police station) M(turtle) ;¥(basketball) áN(stereoscopic) �o(protect)y|(scene) �(run) äÚ(on foot) \<�(Canada) S!(Spring Festival) 7l~(tuna)

For comparison purpose, we also show the most frequent words under eachemotion found by [17] in Table 6. Compared to the topics extracted by TET,they are less meaningful. For example, under the “Anger” emotion, there arewords such as hospital, school and driver. It is not obvious to find the relation-ship between the topic and the emotion, therefore can not really explain why the“Anger” emotion is evoked. On the contrary, words in the topics extracted byTET make it more easier to understand which events evoke the correspondingemotion.

Table 6: Most frequent words under different emotion labels found by [17] on ND16 dataset.Touching Shock Amusement Sadness Curiosity Anger

úi(company) {�(court) ��(hospital) ��(hospital) �(grandson) ��(hospital)y|(scene) �Â(live broadcasting) úi(company) y|(scene) ûû(dandan) a(money))��(hospital) ²�(platform) y|(scene) �)(doctor) �,(Pan) Æ�(school)�)(doctor) a(money) å¯(girl) å¯(girl) åÖ(girl) y|(scene)ó(employee) ��(hospital) {�(court) >{(phone) �å¯(little girl) >{(phone)k�(die) k�(die) a(money) a(money) Su(security check) Æ)(student)ÅO(couple) �w<(defendant) >{(phone) {�(court) êý (Ma Yanling) �´(call the police)�?(WangXiu) �³<(victim) ó�<(staff) £[(go home) �dË(Chen Junbo) �Ѥ(police station)��(die) ç¦(lottery) ¯u(incident) �½(authenticate) Í<(rescue) iÅ(driver)v¦<(suspect) Y�(case) [á(family) F"(hope) ~¬(villager) å¯(girl)

To further evaluate the effectiveness of emotion and topic transitions cap-tured by TET, an example of a news article with the predicted topic-emotiontransition results and its ground truth readers’ emotion votes are shown in Fig-ure 7. It can be observed that not only the document-level emotions but also thesentence-level emotions can be detected simultaneously by the proposed TETwith high accuracy.

4.4. Sentence-Level Emotion Classification

Since TET can also detect sentence-level emotions, we compare the perfor-mance of TET with JST and ASUM on the RenCECps corpus for sentence-levelemotion detection. As JST does not assign emotion labels to sentences directly,the sentence-level emotion label of JST is obtained by aggregating the emotion-topic probabilities of words within each sentence. The detailed results of TET,

17

15 154 45 50 9 533

Touching Shock Amusement Sadness Curiosity Anger 0.0206

0.1031 0.0928

0.1546

0.0206

0.6082

Touching Shock Amusement Sadness Curiosity Anger

probability

Shock

Sad

Angry

Sentence

难道他是凶手？很快，鉴定报告确定了这个答案 。

Is he a perpetrator？Soon, the identification report identifies

this answer.

伤者身中10余刀，刀伤造成其心脏破裂、右颈外静脉破

裂，最终，其经抢救无效死亡 。There were more than ten knives in the injured body. The

knife wound caused the heart to rupture and the right external

jugular vein to rupture. Eventually, it died after being

rescued.

仅一个多小时的时间，两个年轻人的人生就被彻底改

变。Just over one hour, the lives of two young man had been

completely changed.

Emotion Topic

Accidental offence

The wounded died

Life destruction

Ground truth Predicted results

Figure 7: An example of a news article from Sina, its corresponding readers votes over prede-fined emotion classes, predicted document level and sentence level emotions.

JST and ASUM under different number of topics are shown in Figure 8. Theemotion lexicon used is lex nf. Moreover, we also conducted an experimentusing LSTM with attention [35]. It adopts LSTM networks augmented withattention mechanisms for short-text emotion detection. The results of LSTMwith attention were achieved by running the author published source code onour data set.

Table 7: Performance comparison on RenCECps.

Model Acc@1 APJST 38.404 0.3138ASUM 35.741 -0.1112TET 41.908 0.3463LSTM with attention 43.072 -

It can be seen from Table 7 and Figure 8 that TET achieves better per-formance compared with JST and ASUM on both metrics for sentence-levelemotion classification. On Acc@1, JST and ASUM give similar results for topicnumbers between 5 and 20. But ASUM gives the worst results on AP . LSTMwith attention achieves the best performance on Acc@1. However, it should bementioned that it is unfair to compare the proposed approach with the LSTMwith attention approach since only document level annotations are employed inthe proposed approach while the more grained sentence level annotations areneeded by the LSTM with attention approach. It is also observed that the per-

18

formances of all the approaches on RenCECps are worse than those on ND16,except LSTM with attention. It is perhaps not surprising since sentence-level e-motion classification is performed on sentences with limited content information.On the contrary, document-level emotion classification operates on the wholedocument and thus likely gives higher performance compared to classificationon the finer-grained sentence level.

1 5 10 15 20 5030313233343536373839404142

Number of topics

Acc

@1

TETJSTASUM

(a) Acc@1 result.

1 5 10 15 20 50−0.12−0.09−0.06−0.03

00.030.060.090.120.150.180.210.240.270.3

0.33

Number of topicsA

P TETJSTASUM

(b) AP result.

Figure 8: Performance comparison between TET, JST and ASUM with different number oftopics on RenCECps.

5. Conclusions

In this paper, we have proposed the Topic-Emotion Transition (TET) modelwhich models the transition of emotions and topics of successive sentences as aMarkov chain. In TET, we used linguistic features of sentences to guide topictransition. Emotion correlations calculated from emotion lexicons automaticallyconstructed from data is employed to guide emotion transitions. A customizedforward-backward algorithm along with a Viterbi algorithm are used for infer-ence and parameter estimation. Experiments show that our model outperformsthe state-of-the-art methods in both document-level and sentence-level emotionclassification.

Acknowledgments

We would like to thank the anonymous reviewers for their valuable com-ments and suggestions. This work was funded by the National Natural ScienceFoundation of China (61772132, 61528302), the National Key Research and De-velopment Program of China (2016YFC1306704), the Jiangsu Natural ScienceFunds (BK20161430), the 333 Project of Jiangsu Province and Innovate UK(103652).

19

References

[1] X. Li, H. Xie, L. Chen, J. Wang, X. Deng, News impact on stock pricereturn via sentiment analysis, Knowledge-Based Systems 69 (2014) 14 –23.

[2] D. Zhou, T. Gao, Y. He, Jointly event extraction and visualization ontwitter via probabilistic modelling, in: Proceedings of the 54th AnnualMeeting of the Association for Computational Linguistics, ACL, 2016, pp.269–278.

[3] Y. Kim, Convolutional neural networks for sentence classification, in: Pro-ceedings of the 2014 Conference on Empirical Methods in Natural LanguageProcessing, pp. 1746–1751.

[4] S. Poria, E. Cambria, A. Gelbukh, Deep convolutional neural network tex-tual features and multiple kernel learning for utterance-level multimodalsentiment analysis, in: Proceedings of the 2015 Conference on EmpiricalMethods in Natural Language Processing, 2015, pp. 2539–2544.

[5] S. Poria, E. Cambria, D. Hazarika, P. Vij, A deeper look into sarcas-tic tweets using deep convolutional neural networks, in: Proceedings ofthe 26th International Conference on Computational Linguistics, 2016, pp.1601–1612.

[6] X. Quan, Q. Wang, Y. Zhang, L. Si, L. Wenyin, Latent discriminative mod-els for social emotion detection with emotional dependency, ACM Trans-actions on Information Systems 34 (2) (2015) 1–19.

[7] X. Li, Q. Peng, Z. Sun, L. Chai, Y. Wang, Predicting social emotions fromreaders’ perspective, IEEE Transactions on Affective Computing.

[8] Y. Rao, Q. Li, X. Mao, L. Wenyin, Sentiment topic models for social emo-tion mining, Information Sciences 266 (2014) 90–100.

[9] Y. Rao, Contextual sentiment topic model for adaptive social emotion clas-sification, IEEE Intelligent Systems 31 (1) (2016) 41–47.

[10] D. M. Blei, A. Y. Ng, M. I. Jordan, Latent dirichlet allocation, Journal ofMachine Learning Research 3 (Jan) (2003) 993–1022.

[11] E. Cambria, Affective computing and sentiment analysis, IEEE IntelligentSystems 31 (2) (2016) 102–107.

[12] K. H.-Y. Lin, C. Yang, H.-H. Chen, Emotion classification of onlinenews articles from the reader’s perspective, in: Proceedings of the 2008IEEE/WIC/ACM International Conference on Web Intelligence and Intel-ligent Agent Technology, 2008, pp. 220–226.

20

[13] C. Strapparava, R. Mihalcea, Learning to identify emotions in text, in:Proceedings of the 2008 ACM symposium on Applied computing, 2008,pp. 1556–1560.

[14] X. Li, J. Pang, B. Mo, Y. Rao, Hybrid neural networks for social emo-tion detection over short text, in: Proceedings of the International JointConference on Neural Networks, 2016, pp. 537–544.

[15] Y. Li, H. Guo, Q. Zhang, J. Gu Mingyunand Yang, Imbalanced tex-t sentiment classification using universal and domain-specific knowledge,Knowledge-Based Systems 160 (2018) 1–15.

[16] P. K. Bhowmick, Reader perspective emotion analysis in text through en-semble based multi-label classification framework, Computer and Informa-tion Science 2 (4) (2009) 64–74.

[17] Y. Wang, A. Pal, Detecting emotions in social media: A constrained op-timization approach, in: Proceedings of the Twenty-Fourth InternationalJoint Conference on Artificial Intelligence, 2015, pp. 996–1002.

[18] D. Zhou, X. Zhang, Y. Zhou, Q. Zhao, X. Geng, Emotion distributionlearning from texts, in: Proceedings of the 2016 Conference on EmpiricalMethods in Natural Language Processing, EMNLP, 2016, pp. 638–647.

[19] D. Zhou, Y. Yang, Y. He, Relevant emotion ranking from text constrainedwith emotion relationships, in: Proceedings of the 2018 Conference of theNorth American Chapter of the Association for Computational Linguistics:Human Language Technologies NAACL-HLT, 2018, pp. 561–571.

[20] Y. Yang, D. Zhou, Y. He, An interpretable neural network with topicalinformation for relevant emotion ranking, in: Proceedings of the 2018 Con-ference on Empirical Methods in Natural Language Processing, 2018, pp.3423–3432.

[21] Y. Wang, M. Huang, X. Zhu, L. Zhao, Attention-based lstm for aspect-level sentiment classification, in: Proceedings of the 2017 Conference onEmpirical Methods in Natural Language Processing, 2017, pp. 606–615.

[22] Y. Li, Q. Pan, T. Yang, S. Wang, J. Tang, E. Cambria, Learning wordrepresentations for sentiment analysis, Cognitive Computation (6) (2017)1–9.

[23] S. Bao, S. Xu, L. Zhang, R. Yan, Z. Su, D. Han, Y. Yu, Joint emotion-topic modeling for social affective text mining, in: Proceedings of the NinthIEEE International Conference on Data Mining, 2009, pp. 699–704.

[24] C. Lin, Y. He, Joint sentiment/topic model for sentiment analysis, in: Pro-ceedings of the 18th ACM conference on Information and knowledge man-agement, 2009, pp. 375–384.

21

[25] L. Zhu, Y. He, D. Zhou, Hierarchical viewpoint discovery from tweets usingbayesian modelling, Expert Systems with Applications 116 (2019) 430–438.

[26] Q. Mei, X. Ling, M. Wondra, H. Su, C. Zhai, Topic sentiment mixture:modeling facets and opinions in weblogs, in: Proceedings of the 16th inter-national conference on World Wide Web, 2007, pp. 171–180.

[27] Y. Jo, A. H. Oh, Aspect and sentiment unification model for online reviewanalysis, in: Proceedings of the fourth ACM international conference onWeb search and data mining, 2011, pp. 815–824.

[28] T. L. Griffiths, M. Steyvers, D. M. Blei, J. B. Tenenbaum, Integratingtopics and syntax, in: Advances in Neural Information Processing Systems17, 2005, pp. 537–544.

[29] A. Gruber, Y. Weiss, M. Rosen-Zvi, Hidden topic markov models, in: Pro-ceedings of the Eleventh International Conference on Artificial Intelligenceand Statistics, 2007, pp. 163–170.

[30] H. Wang, D. Zhang, C. Zhai, Structural topic model for latent topicalstructure analysis, in: Proceedings of the 49th Annual Meeting of the As-sociation for Computational Linguistics: Human Language Technologies-Volume 1, 2011, pp. 1526–1535.

[31] M. M. Rahman, H. Wang, Hidden topic sentiment model, in: Proceedingsof the 25th International Conference on World Wide Web, 2016, pp. 155–165.

[32] J. Staiano, M. Guerini, Depechemood: a lexicon for emotion analysis fromcrowd-annotated news, in: Proceedings of the 52nd Annual Meeting of theAssociation for Computational Linguistics, 2014, pp. 427–433.

[33] C. Quan, F. Ren, A blog emotion corpus for emotional expression analysisin chinese, Computer Speech & Language 24 (4) (2010) 726–749.

[34] P. Katz, M. Singleton, R. Wicentowski, Swat-mp: the semeval-2007 systemsfor task 5 and task 14, in: Proceedings of the International Workshop onSemantic Evaluations, 2007, pp. 308–313.

[35] C. Baziotis, N. Pelekis, C. Doulkeridis, Datastories at semeval-2017 task4: Deep lstm with attention for message-level and topic-based sentimentanalysis, in: Proceedings of the International Workshop on Semantic Eval-uation, 2017, pp. 747–754.

[36] Y. Rao, Q. Li, L. Wenyin, Q. Wu, X. Quan, Affective topic model for socialemotion detection, Neural Networks 58 (2014) 29–37.

22