recursive deep models for semantic compositionality over a ...ddekok/dl4nlp/patricia-neele.pdf ·...

Philosophische FakultätSeminar für Sprachwissenschaft

Recursive Deep Models for SemanticCompositionality Over a Sentiment Treebanktext06 July 2017, Patricia Fischer & Neele Witte

OverviewSentiment Analysis

Sentiment Treebank

Neural Network Architecture

Recursive Neural NetworkMatrix Vector RNNRecursive Neural Tensor Network

ExperimentsFine-grained Sentiment for All PhrasesFull Sentence Binary SentimentContrastive ConjunctionHigh Level NegationMost Positive/Negative Phrases

2 | Patricia Fischer & Neele Witte: RNNs for Semantic Compositionality c© 2017 Universität Tübingen

Sentiment Analysis


Sentiment Analysis

• “Sentiment analysis is the measurement of positive and nega-tive language.”

• “Sentiment Analysis is the process of determining whether apiece of writing is positive, negative or neutral. It’s also knownas opinion mining, deriving the opinion or attitude of a speaker.”

• “Using NLP, statistics, or machine learning methods to extract,identify, or otherwise characterize the sentiment content of atext unit.”


Sentiment Analysis

• Classification of users, texts, phrases, words

• Ratings

- Binary: or or

- Scales:

- Open category:


Sentiment Analysis

Challenges

• Opinions expressed in complex ways

• Stylistic devices such as sarcasm, irony etc.


Sentiment Analysis

Examples


From

To


Motivation for New Model and Database

• Not only want to represent sentiment by the sum of the senti-ments of their components, but by the composition of them

• Word order is important, especially for detecting negation

• No database with annotated single sentences (usually docu-ments)→ good results for long texts but not for short texts (e.g.Twitter Data), phrases, segments

• Accuracy for three classes on short texts: below 60%

• Aim: construct a database to train and evaluate compositionalmodels


Sentiment Treebank


Sentiment Treebank

Normalized histogram of sentiment annotations at each n-gram length.

• 11,855 single sentences, 215,154 uniquephrases

• Movie review excerpts from rottentomato-es.com

• Stanford parser• Labeling: amazon mechanical turk→ Fine-grained sentiment classification


Semantic Representation of Words

• Map words into vector space to represent their meaning (se-mantic)• Similar words are close to each other• How can we represent meaning of longer phrases?


Semantic Representation of Sentences

• Can we find a semantic representation for sentences (of arbi-trary length) as well?• Map phrases into the same vector space as well• How?


Semantic Representation of Sentences

• Bag of Words: represent sentence as Bag of words and createone vector per sentence

→ Problem: word order ignored

• Sentence embeddings: create embeddings for n-grams (e.g.7-gram represents a sentence embedding)

→ Problem: cannot create so many embeddings, sentences canbe very long


Recursive Neural Network

Principle of CompositionalityThe meaning (vector) of a sentence is defined by1. the meaning of its words2. the rules that combine them

Recursive Neural Nets can jointly learn compositional vectorrepresentations and parse trees


Recursive Structure

1. Extract a binary syntactic tree

2. Recursively merge smaller segments to get representation ofbigger segments / the whole sentence


Building Blocks for Neural Network

Composition function for merging two children

f (W [c1; c2] + b) (1)

Classification function for assigning a label to each node

ya = softmax(Wsa) (2)

Loss function: the cross-entropy error between the predicteddistribution and the target distribution


Recursive NN Structure


MV-RNN Structure


Recursive Neural Tensor Network

p2

a

not

p1

b

very

c

good

p1 = f

([bc

]T

V [1:d ][bc

]+ W

[bc

])

p2 = f

([ap1

]T

V [1:d ][

ap1

]+ W

[ap1

])


Experiments


Fine-grained Sentiment for All Phrases

ModelFine-grained Positive/Negative

All Root All RootNB 67.2 41.0 82.6 81.8SVM 64.3 40.7 84.6 79.4BiNB 71.0 41.9 82.7 83.1VecAvg 73.3 32.7 85.1 80.1RNN 79.0 43.2 86.1 82.4MV-RNN 78.7 44.4 86.8 82.9RNTN 80.7 45.7 87.6 85.4


Full Sentence Binary Sentiment

ModelFine-grained Positive/Negative

All Root All RootNB 67.2 41.0 82.6 81.8SVM 64.3 40.7 84.6 79.4BiNB 71.0 41.9 82.7 83.1VecAvg 73.3 32.7 85.1 80.1RNN 79.0 43.2 86.1 82.4MV-RNN 78.7 44.4 86.8 82.9RNTN 80.7 45.7 87.6 85.4


Contrastive Conjunction

“There are slow and repetitive partsbut it has just enough spice to keep it interesting.”


High Level Negation

Can the Model correctly classify the reversal from positive to nega-tive sentiment?


Negating Sentence with Negative Sentiment

How often did the model increase positive activation in the senti-ment?


Negating Sentence with Negative Sentiment

→ Sentiment of the sentence will become less negative (not neces-sarily positive)


Most Positive/Negative Phrases


ReferencesBo Pang and Lillian Lee. (2008)Opinion mining and sentiment analysis.In Foundations and Trends in Information Retrieval.

R. Socher, C. D. Manning, and A. Y. Ng. (2010)Learning continuous phrase representations and syntactic parsing with recursive neuralnetworks.In Proceedings of the NIPS-2010 Deep Learning and Unsupervised Fea- ture LearningWorkshop.

R. Socher, C. Lin, A. Y. Ng, and C.D. Manning. (2011a)Parsing Natural Scenes and Natural Language with Recursive Neural Networks.In ICML.

R. Socher, B. Huval, C. D. Manning, and A. Y. Ng. (2012)Semantic compositionality through recursive matrix-vector spaces.In EMNLP.


Thank you!Contact:

Philosophische FakultätSeminar für SprachwissenschaftWilhelmstraße 19, 72074 TübingenPhone: +49 (0)7071 29-75927Fax: +49 (0)7071 [email protected]


recursive deep models for semantic compositionality over a ...ddekok/dl4nlp/patricia-neele.pdf ·...

Documents