Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4678–4683Florence, Italy, July 28 - August 2, 2019. c©2019 Association for Computational Linguistics

4678

Context-aware Embedding for Targeted Aspect-based Sentiment Analysis

Bin Liang1, Jiachen Du1, Ruifeng Xu1∗, Binyang Li2, Hejiao Huang1

1Department of Computer Science, Harbin Institute of Technology (Shenzhen), China2 School of Information Science and Technology,

University of International Relations, Beijing, China18b951033@stu.hit.edu.cn, dujiachen@stmail.hitsz.edu.cn

xuruifeng@hit.edu.cn, byli@uir.edu.cnhuanghejiao@hit.edu.cn

Abstract

Attention-based neural models were employedto detect the different aspects and sentimentpolarities of the same target in targeted aspect-based sentiment analysis (TABSA). However,existing methods do not specifically pre-trainreasonable embeddings for targets and aspectsin TABSA. This may result in targets or as-pects having the same vector representationsin different contexts and losing the context-dependent information. To address this prob-lem, we propose a novel method to refine theembeddings of targets and aspects. Such piv-otal embedding refinement utilizes a sparse co-efficient vector to adjust the embeddings oftarget and aspect from the context. Hencethe embeddings of targets and aspects canbe refined from the highly correlative wordsinstead of using context-independent or ran-domly initialized vectors. Experiment resultson two benchmark datasets show that our ap-proach yields the state-of-the-art performancein TABSA task.

1 Introduction

Targeted aspect-based sentiment analysis(TABSA) aims at detecting aspects accord-ing to the specific target and inferring sentimentpolarities corresponding to different target-aspectpairs simultaneously (Saeidi et al., 2016). Forexample, in sentence “location1 is your best betfor secure although expensive and location2 istoo far.”, for target “location1”, the sentimentpolarity is positive towards aspect “SAFETY”but is negative towards aspect “PRICE”. While“location2” only express negative polarity aboutaspect “TRANSIT-LOCATION”. This can be seenin Figure 1, e.g., where opinions on the aspects“SAFETY” and “PRICE” are expressed for target“location1” but not for target “location2”, whose

∗∗ Corresponding Author

corresponding aspect is “TRANSIT-LOCATION”. Here, an interesting phenomenon is that, theopinion “Positive” towards aspect “SAFETY” isexpressed for target “location1” will be changeif “location1” and “location2” are exchanged.That is to say, the representation of target andaspect should take full account of context in-formation rather than use context-independentrepresentation.

location1 is your best bet for secure although expensive

and location2 is too far

Target

location1

Aspect

SAFETY

Sentiment

Positive

location1 PRICE Negative

location2 TRANSIT Negative

Figure 1: Example of TABSA task. Highly correlativewords and corresponding aspects are in the same color.Entity names are masked by location1 and location2.

Aspect-based sentiment analysis (ABSA) is abasic subtask of TABSA, which aims at inferringthe sentiment polarities of different aspects in thesentence (Ruder et al., 2016; Chen et al., 2017; Guiet al., 2017; Peng et al., 2018; Ma et al., 2018a).Recently, attention-based neural models achieveremarkable success in ABSA (Fan et al., 2018;Wang et al., 2016; Tang et al., 2016). In TABSAtask, the attention-based sentiment LSTM (Maet al., 2018b) is proposed to tackle the challengesof both aspect-based sentiment analysis and tar-geted sentiment analysis by incorporating exter-nal knowledge. For neural model improvement,a delayed memory is proposed to track and updatethe states of targets at the right time with externalmemory (Liu et al., 2018).

Despite the remarkable progress made by theprevious works, they usually utilize context-independent or randomly initialized vectors to rep-

4679

resent targets and aspects, which losses the seman-tic information and ignores the interdependenceamong specific target, corresponding aspects andthe context. Because the targets themselves haveno expression of sentiment, and the opinions of thegiven sentence are generally composed of wordshighly correlative to the targets. For example, ifthe words “price” and “expensive” are in the sen-tence, it probably expresses the “Negative” senti-ment polarity about aspect “PRICE”.

To alleviate these problems above, we proposea novel embedding refinement method to obtaincontext-aware embedding for TABSA. Specifi-cally, we use a sparse coefficient vector to selecthighly correlated words from the sentence, andthen adjust the representations of target and aspectto make them more valuable. The main contribu-tions of our work can be summarized as follows:

• We reconstruct the vector representation fortarget from the context by means of a sparsecoefficient vector, hence the representationof target is generated from highly correlativewords rather than using context-independentor randomly initialized embedding.

• The aspect embedding is fine-tuned to beclose to the highly correlated target and beaway from the irrelevant targets.

• Experiment results on SentiHood and Se-meval 2015 show that our proposed methodcan be directly incorporated into embedding-based TABSA models and achieve state-of-the-art performance.

2 Methodology

In this section, we describe the proposed methodin detail. The framework of our proposed methodis demonstrated in Figure 2.

We assume a words sequence of a given sen-tence as an embedding matrix X ∈ Rm×n, wheren is the length of sentence, m is the dimension ofembedding, and each word can be represented asan m-dimensional embedding x ∈ Rm includingthe embedding of target t ∈ Rm via random ini-tialization and the embedding of aspect a ∈ Rm

which is an average of its constituting word em-beddings or single word embedding. The sentenceembedding matrix X is fed as input into our modelto achieve the sparse coefficient vector u′ via thefully connected layer and the step function succes-sively. The hidden output u′ is utilized to compute

the refined representation of target t ∈ Rm and as-pect a ∈ Rm. Afterwards, the squared Euclideanfunction d(t, t) and d(a, t, t′) are used to itera-tively minimize the distance to get the refined em-beddings of target and aspect.

...

...

···

...

...

...

...

...

...

...

...

...

Figure 2: The framework of our refinement model. ⊗is element-wise product, ⊕ is vector addition, Φ is stepfunction.

2.1 Task DefinitionGiven a sentence consisting of a sequence ofwords s = {w1, w2, . . . , LOC, . . . , wn}, whereLOC is a target in the sentence, there will be 1or 2 targets in the sentence corresponding to sev-eral aspects. There are a pre-identified set of tar-gets T and a fixed set of aspects A. The goalof TABSA can be regarded as a fine-grained sen-timent expression as a tuple (t, a, p), where prefers to the polarity which is associated with as-pect a, and the aspect a belongs to a target t.The objective of TABSA task is to detect the as-pect a ∈ A and classify the sentiment polarityp ∈ {Positive,Negative,None} according toa specific target t and the sentence s.

2.2 Target RepresentationThe idea of target representation is to reconstructthe target embedding from a given sentence ac-cording to the highly correlated words in the con-text. By this means we can extract the correlationbetween target and context, the target representa-tion is computed as:

t = X ∗ u′ (1)

4680

where t is the representation of target, u′ is asparse coefficient vector indicating the importanceof different words in the context, defined as:

u′ = Φ(u) (2)

where Φ is a step function given a real value:

Φ(ui) =

{ui ui > mean(u)

0 ui < mean(u)(3)

where mean(·) is an average function, and thevector u can be computed by a non-linear functionof basic embedding matrix X:

u = f(Xᵀ ·W + b) (4)

where f is a non-linear operation function like sig-moid, W ∈ Rm and b ∈ Rn denote the weightmatrix and bias respectively. The target represen-tation is inspired by the recent success of embed-ding refinement (Yu et al., 2017). For each target,our reconstruction operation aims to get a contex-tual relevant embedding by iteratively minimizesthe squared Euclidean between the target and thehighly correlative words in the sentence. The ob-jective function is defined as:

d(t, t) =

n∑i=1

( m∑j=1

(tji − tji )

2 + λu′i

)(5)

where λu′i aims to control the sparseness of vec-tor u′. Through the iterative procedure, the vectorrepresentation of the target will be iteratively up-dated until the number of the non-zero elementsof vector u′ less than the threshold value: k 6 c.Where k is the number of the non-zero elementsof vector u′ and c is a threshold to control the non-zero number of vector u′.

2.3 Aspect RepresentationGenerally, the words of aspects contain the mostimportant semantic information. Coordinate withthe aspect itself, the context information can alsoreflect the aspect, such as the word “price” in thesentence probably has relevant to aspect “PRICE”.To this end, we refine the aspect representation ac-cording to target representation. By incorporatinghighly correlated words into the representation ofaspect, every element in the fine-tuned aspect em-bedding a is calculated as:

ai = ai + αXi ∗ u′i (6)

where α is a parameter to control the influence be-tween aspect and the context.

For each aspect, the fine-tuning method aims tomove closer to the homologous target and furtheraway from the irrelevant one by iteratively mini-mizes the squared Euclidean. The objective func-tion is thus divided into two parts:

d(a, t, t′) =

n∑i=1

[ m∑j=1

((aji − tji )

2 − β(aji − t′ji )

2)+ λu′i

](7)

where t is the the homologous target and t′ is theirrelevant one. β is a parameter that controls thedistance from the irrelevant target.

3 Experiments

This section evaluates several deep neural mod-els based on our proposed embedding refinementmethod for TABSA.

Dataset. Two benchmark datasets: Senti-Hood (Saeidi et al., 2016) and Task 12 of Semeval2015 (Pontiki et al., 2015) are used to evaluate ourproposed method. SentiHood contains annotatedsentences containing one or two location targetmentions. The whole dataset contains 5215sentences with 3862 sentences containing a singlelocation and 1353 sentences containing multiple(two) locations. Location target names aremasked by LOCATION1 and LOCATION2in the whole dataset. Following (Saeidi et al.,2016), we only consider the top 4 aspects (“GEN-ERAL”, “PRICE”, “TRANSIT-LOCATION”and “SAFETY”) when evaluate aspect detec-tion and sentiment classification. To show thegeneralizability of our method, we evaluate ourworks in another dataset: restaurants domain inTask 12 for TABSA from Semeval 2015. Weremove sentences containing no targets as well asNULL targets like the work of (Ma et al., 2018b).The whole dataset contains 1,197 targets in thetraining set and 542 targets in the testing set.

Experiment setting. We use Glove (Penningtonet al., 2014)1 to initialize the word embeddingsin our experiments, and target embeddings (loca-tion1 and location2) are randomly initialized. Weinitialize W and b with random initialization. Theparameters of c, α and β in our experiment are 4,1 and 0.5 respectively. Given a unit of text s, a listof labels (t, a, p) is provided correspondingly, the

1http://nlp.stanford.edu/projects/glove/

4681

ModelAspect Detection Sentiment Classification

Acc. (%) F1 (%) AUC (%) Acc. (%) AUC (%)

LSTM-Final — 68.9 89.8 82.0 85.4LSTM-Loc — 69.3 89.7 81.9 83.9

SenticLSTM 67.4 78.2 — 89.3 —RE+SenticLSTM† (ours) 73.8 79.3 — 93.0 —

Delayed-memory 73.5 78.5 94.4 91.0 94.8RE+Delayed-memory† (ours) 76.4 81.0 96.8 92.8 96.2

Table 1: Experimental results on SentiHood. † denotes average score over 10 runs, and best scores are in bold.

ModelAspect Detection Sentiment Classification

Acc. (%) F1 (%) AUC (%) Acc. (%) AUC (%)

SenticLSTM 67.3 76.4 — 76.5 —RE+SenticLSTM†(ours) 71.2 78.6 89.5 76.8 82.3

Delayed-memory 70.3 77.4 90.8 76.4 83.6RE+Delayed-memory†(ours) 71.6 79.1 91.8 77.2 84.6

Table 2: Experimental results on Semeval 2015.

overall task of TABSA can be defined as a three-class classification task for each (t, a) pair withlabels Positive, Negative, None. We use macro-average F1, Strict accuracy (Acc.) and AUC foraspect detection, and Acc. and AUC for sentimentclassification ignoring the class of None, which in-dicates that a sentence does not contain an opinionfor the target-aspect pair (t, a).

Comparison methods. We compare our methodwith several typical baseline systems (Saeidi et al.,2016) and remarkable models (Ma et al., 2018b;Liu et al., 2018) proposed for the task of TABSA.

(1) LSTM-Final (Saeidi et al., 2016): A bidi-rectional LSTM model takes the final states to rep-resent the information.

(2) LSTM-Loc (Saeidi et al., 2016): A bidirec-tional LSTM model takes the output representa-tion at the index corresponding to the location tar-get.

(3) SenticLSTM (Ma et al., 2018b): A bidirec-tional LSTM model incorporates external Sentic-Net knowledge.

(4) Delayed-memory (Liu et al., 2018): Amemory-based model utilizes a delayed memorymechanism.

(5) RE+SenticLSTM: Incorporating our pro-posed method into SenticLSTM.

(6) RE+Delayed-memory: Incorporating our

proposed method into Delayed-memory.

3.1 Comparative Results of SentiHood

The experimental results are shown in Table1. The classifiers based on our proposed meth-ods (RE+Delayed-memory, RE+SenticLSTM)achieve better performance than competitor mod-els for both aspect detection and sentiment clas-sification. In comparison with the previous best-performing model (Delayed-memory), our bestmodel (RE+Delayed-memory) significantly im-proves aspect detection (by 2.9% in strict accu-racy, 2.5% in macro-average F1 and 2.4% in AUC)and sentiment classification (by 1.8% in strict ac-curacy and 1.4% in AUC) on SentiHood.

The comprehensive results show that by incor-porating refined context-aware embeddings of tar-gets and aspects into the neural models can sub-stantially improve the performance of aspect de-tection. This indicates that the refined represen-tation is more learnable and is able to extractthe interdependence between aspect and the corre-sponding target in the context. On the other hand,the performance of sentiment classification is im-proved certainly in comparison with the remark-able models (Delayed-memory and SenticLSTM).It indicates that our context-aware embeddingscan capture sentiment information better than themodels using traditional embeddings even incor-

4682

porating external knowledge.

3.2 Comparative Results of Semeval 2015

To illustrate the robustness of our proposedmethod, a comparative experiment was conductedon Semeval 2015. As shown in Table 2, our em-bedding refinement method achieves better per-formance for both aspect detection and sentimentclassification than two original embedding-basedmodels, for aspect detection in particular. Conse-quently, our method is capable of achieving state-of-the-art performance on different datasets.

(a) RE+Delayed-memory. (b) Delayed-memory.

(c) RE+SenticLSTM. (d) SenticLSTM.

Figure 3: The visualization of intermediate embed-dings learned by embedding-based models. Differentcolors represent different aspects.

3.3 Visualization

To qualitatively demonstrate how the proposedembedding refinement improves the performancefor both aspect detection and sentiment classi-fication in TABSA, we visualize the proposedcontext-aware aspect embeddings a and origi-nal aspect embeddings a which are learned withDelayed-memory and SenticLSTM models via t-SNE (Maaten and Hinton, 2008). As shown inFigure 3, compared with randomly initialized em-bedding, it is observed a significantly clearer sep-aration between different aspects represented byour proposed context-aware embedding. This in-

dicates that different representations of aspects canbe distinguished from the context, and that thecommonality of a specific aspect can also be ef-fectively preserved. Hence the model can extractdifferent semantic information according to differ-ent aspects, when detecting multiple aspects in thesame sentence in particular. The results verify thatencoding aspect by leveraging context informationis more effective for aspect detection and senti-ment classification in TABSA task.

4 Conclusion

In this paper, we proposed a novel method for re-fining representations of targets and aspects. Theproposed method is able to select a set of highlycorrelated words from the context via a sparse co-efficient vector and then adjust the representationsof targets and aspects. Hence, the interdependenceamong specific target, corresponding aspect, andthe context can be extracted to generate superiorembedding. Experimental results demonstratedthe effectiveness and robustness of the proposedmethod on two benchmark datasets over the taskof TABSA. In future works, we will explore theextension of this approach for other tasks.

Acknowledgments

This work was supported by National Natu-ral Science Foundation of China U1636103,61632011, 61876053, U1536207, Key Tech-nologies Research and Development Pro-gram of Shenzhen JSGG20170817140856618,Shenzhen Foundational Research FundingJCYJ20180507183527919, Joint Research Pro-gram of Shenzhen Securities Information Co.,Ltd. No. JRPSSIC2018001.

ReferencesPeng Chen, Zhongqian Sun, Lidong Bing, and Wei

Yang. 2017. Recurrent attention network on mem-ory for aspect sentiment analysis. In Proceedings ofthe 2017 Conference on Empirical Methods in Nat-ural Language Processing, pages 452–461.

Chuang Fan, Qinghong Gao, Jiachen Du, LinGui, Ruifeng Xu, and Kam-Fai Wong. 2018.Convolution-based memory network for aspect-based sentiment analysis. In The 41st InternationalACM SIGIR Conference on Research & Develop-ment in Information Retrieval, SIGIR 2018, Ann Ar-bor, MI, USA, July 08-12, 2018, pages 1161–1164.

Lin Gui, Yu Zhou, Ruifeng Xu, Yulan He, and Qin Lu.2017. Learning representations from heterogeneous

4683

network for sentiment classification of product re-views. Knowledge-Based Systems, 124(C):34–45.

Fei Liu, Trevor Cohn, and Timothy Baldwin. 2018. Re-current entity networks with delayed memory updatefor targeted aspect-based sentiment analysis. In Pro-ceedings of the 2018 Conference of the North Amer-ican Chapter of the Association for ComputationalLinguistics: Human Language Technologies, pages278–283.

Dehong Ma, Sujian Li, and Houfeng Wang. 2018a.Joint learning for targeted sentiment analysis. InProceedings of the 2018 Conference on EmpiricalMethods in Natural Language Processing, pages4737–4742.

Yukun Ma, Haiyun Peng, and Erik Cambria. 2018b.Targeted aspect-based sentiment analysis via em-bedding commonsense knowledge into an attentiveLSTM. In Proceedings of the Thirty-Second Con-ference on Artificial Intelligence,(AAAI-18), pages5876–5883.

Laurens van der Maaten and Geoffrey Hinton. 2008.Visualizing data using t-sne. Journal of machinelearning research, 9(Nov):2579–2605.

Minlong Peng, Qi Zhang, Yu-gang Jiang, and Xuan-jing Huang. 2018. Cross-domain sentiment classi-fication with target domain specific information. InProceedings of the 56th Annual Meeting of the As-sociation for Computational Linguistics (Volume 1:Long Papers), pages 2505–2513.

Jeffrey Pennington, Richard Socher, and ChristopherManning. 2014. Glove: Global vectors for wordrepresentation. In Proceedings of the 2014 Con-ference on Empirical Methods in Natural LanguageProcessing (EMNLP), pages 1532–1543.

Maria Pontiki, Dimitris Galanis, Haris Papageorgiou,Suresh Manandhar, and Ion Androutsopoulos. 2015.Semeval-2015 task 12: Aspect based sentiment anal-ysis. In Proceedings of the 9th International Work-shop on Semantic Evaluation (SemEval 2015), pages486–495.

Sebastian Ruder, Parsa Ghaffari, and John G. Breslin.2016. A hierarchical model of reviews for aspect-based sentiment analysis. In Proceedings of the2016 Conference on Empirical Methods in NaturalLanguage Processing, pages 999–1005.

Marzieh Saeidi, Guillaume Bouchard, Maria Liakata,and Sebastian Riedel. 2016. Sentihood: Targetedaspect based sentiment analysis dataset for urbanneighbourhoods. In COLING 2016, 26th Inter-national Conference on Computational Linguistics,Proceedings of the Conference: Technical Papers,December 11-16, 2016, pages 1546–1556.

Duyu Tang, Bing Qin, and Ting Liu. 2016. Aspectlevel sentiment classification with deep memory net-work. In Proceedings of the 2016 Conference onEmpirical Methods in Natural Language Process-ing, pages 214–224.

Yequan Wang, Minlie Huang, xiaoyan zhu, andLi Zhao. 2016. Attention-based lstm for aspect-levelsentiment classification. In Proceedings of the 2016Conference on Empirical Methods in Natural Lan-guage Processing, pages 606–615.

Liang-Chih Yu, Jin Wang, K. Robert Lai, and XuejieZhang. 2017. Refining word embeddings for sen-timent analysis. In Proceedings of the 2017 Con-ference on Empirical Methods in Natural LanguageProcessing, pages 534–539.