annotating topics of opinions veselin stoyanov claire cardie

Annotating Topics of Opinions

Veselin Stoyanov

Claire Cardie

May 29, 2008 LREC 2008, Marakech, Morocco

Talk Overview

Fine-grained sentiment analysis Definitions Examples

Opinion topic annotation Definitions Issues Approach and Corpus IA agreement


Background

Sentiment Analysis:

Extraction and representation of attitudes, evaluations, opinions, and sentiment in text.

Fine-grained Sentiment Analysis:

At the level of individual expressions of opinions.


The Australian press has launched a bitter attack on Italy after seeing their beloved Socceroos eliminated on a controversial late penalty. Italian coach Lippi has been blasted for his favorable comments toward the penalty.

Lippi is preparing his side for the upcoming clash with Ukraine. He hailed 10-man Italy's determination to beat Australia and reiterated that the penalty was rightly given.

Fine-grained vs. Coarse-grained Sentiment Analysis

Coarse-grained Sentiment classification Useful in the product

review domain

Fine-grained Individual expressions of

opinions Multiple opinions per

document (even sentence)

Review 1

Review 2

Positive

Negative

[SThe Australian press] has launched a bitter attack on [TItaly] after seeing their beloved [TSocceroos] eliminated on a controversial late [Tpenalty]. [S+TItalian coach Lippi] has also been blasted for his favorable comments toward [Tthe penalty].

Lippi is preparing his side for the upcoming clash with Ukraine. [SHe] hailed 10-man [TItaly]'s determination to beat Australia and reiterated that the [Tpenalty] was rightly given.


Fine-grained opinions: ExampleThe Australian press has launched a bitter attack on Italy.


Fine-grained opinions: Example

Opinion trigger (opinion words)

Source (opinion holder) Polarity – positive/negative Strength Topic (target)

The Australian press has launched a bitter attack on Italy.

Definitions differ, but five main components:





The Australian press has launched a bitter attack on Italy.


launched a bitter attack





[SThe Australian press] has launched a bitter attack on Italy.



The Australian press









negative









negative

high





[SThe Australian press] has launched a bitter attack on [TItaly]




negative

high

Italy


Fine-grained opinions

Five components Source (opinion holder)

e.g. [Bethard et al., 2004] [Choi et al., 2005] [Kim and Hovy, 2006] Opinion trigger (opinion words)

e.g. [Yu and Hatzivassiloglou, 2003] [Riloff and Wiebe, 2003] Polarity – positive/negative

As above Strength

e.g. [Wilson et al. 2004] Topic (target)

????


Annotating Topics of Fine-grained Opinions

Definitions Issues Approach and Corpus IA agreement


Examples

(1)[OH John] likes Marseille for its weather and cultural diversity.

(2)[OH Al] thinks that the government should tax gas more in

order to curb CO2 emissions.


Definitions(1)[OH John] likes Marseille for its weather and cultural diversity.


Definitions(1)[OH John] likes Marseille for its weather and cultural diversity.

Topic: city of Marseille

Topic - the real-world object, event or abstract entity that is the subject of the opinion as intended by the opinion holder


Definitions(1)[OH John] likes [TOPIC SPAN Marseille] for its weather and cultural

diversity.



Topic span - the closest, minimal span of text that mentions the topic


Definitions(1)[OH John] likes [TARGET+TOPIC SPAN Marseille] for its weather and

cultural diversity.



Topic span - the closest, minimal span of text that mentions the topic

Target span - the span of text that covers the syntactic surface form comprising the contents of the opinion


Definitions

(2)[OH Al] thinks that the government should tax gas

more in order to curb CO2 emissions.


Definitions

(2)[OH Al] thinks that [TARGET SPAN the government should

tax gas more in order to curb CO2 emissions].


Definitions

(2)[OH Al] thinks that [TARGET SPAN [TOPIC SPAN? the

government] should [TOPIC SPAN? tax gas] more in order

to [TOPIC SPAN? curb [TOPIC SPAN? CO2 emissions]]].


Definitions


tax gas more in order to curb CO2 emissions].

Context:

(3) Although he doesn’t like government imposed taxes, he thinks that a fuel tax is the only effective solution.


Definitions


[TOPIC SPAN tax gas] more in order to curb CO2

emissions].

Context:

(3) Although he doesn’t like government imposed taxes, he thinks that a fuel tax is the only effective solution.


Related Work

Product reviews E.g. Kobayashi et al. (2004), Yi et al. (2003), Popescu and Etzioni

(2005), Hu and Liu (2004 Limit “topics” to mentions of product names, components, and

their attributes Lexicon look-up Focused on methods for lexicon acquisition

MPQA corpus (Wiebe, Wilson, Cardie, 2004) Fine-grained opinions Topic annotation deemed too difficult Target span annotation is underway

Kim & Hovy (2006) Target span extraction using semantic frames Limited evaluation


Issues in Opinion Topic Identification Multiple potential topics mentioned within a single target span

(2)[OH Al] thinks that [TARGET SPAN [TOPIC SPAN? the government] should [TOPIC SPAN?

tax gas] more in order to [TOPIC SPAN? curb [TOPIC SPAN? CO2 emissions]]].

Requires context

Topic of an opinion is the entity that comprises the main information goal of the opinion based on the discourse context.


Issues in Opinion Topic Identification Opinion topics are not always explicitly

mentioned

(4) [OH John] believes the violation of Palestinian

human rights is one of the main factors.

Topic: ISRAELI-PALESTINIAN CONFLICT

(5) [OH I] disagree entirely!


A Coreference Approach

Hypothesize that the notion of topic coreference will facilitate identification of opinion topics

Easier than specifying the topic of each opinion in isolation

Two opinions are topic-coreferent if they share the same opinion topic.


Opinion Topic Corpus

Build on the MPQA corpus: 535 Documents manually annotated for fine-

grained opinions No opinion topic annotation Our goal: Add the opinion topic information on

top of the existing opinion annotations Created and used a GUI

(www.cs.pitt.edu/mpqa)


Annotation Process

List of opinions to be processed

Set of current clusters

Document text


Annotation Process


Annotation Process

fuel tax


Interannotator Agreement

Annotator 1 150 of the 535 MPQA documents

Annotator 2 20 of these 150

IAG measures from noun phrase coreference resolution

B3 CEAF

all opinions .64 .55 .69

sentiment-bearing opinions

.72 .73 .80

strong opinions .74 .77 .82


Baselines all-in-one

assigns all opinions to the same cluster 1 opinion per cluster

assigns each opinion to its own cluster same paragraph

opinions in the same paragraph are assigned to the same cluster


Results

Baselines

vs. Interannotator agreement

B3 CEAF

all-in-one .37 -.10 .30

1 opinion per cluster .29 .22 .27

same paragraph .55 .31 .50

all opinions .64 .55 .69

sentiment-bearing .72 .73 .80

strong opinions .74 .77 .82


Thank you

Questions?

Annotation instructions + more information available at:www.cs.cornell.edu/~ves


Example

The Australian press has launched a bitter attack on Italy after seeing their beloved Socceroos eliminated on a controversial late penalty. Italian coach Lippi has been blasted for his favorable comments toward the penalty.

Lippi is preparing his side for the upcoming clash with Ukraine. He hailed 10-man Italy's determination to beat Australia and reiterated that the penalty was rightly given.


Example – fine-grained opinions[SThe Australian press] has launched a bitter attack on [TItaly] after seeing [Stheir] beloved [TSocceroos] eliminated on a controversial late [Tpenalty]. [S+TItalian coach Lippi] has also been blasted for his favorable comments toward [Tthe penalty].

Lippi is preparing his side for the upcoming clash with Ukraine. [SHe] hailed 10-man [TItaly]'s determination to beat Australia and reiterated that [Tthe penalty] was rightly given.


Motivation

Sentiment analysis: Useful as stand-alone application Product reviews Tracking opinions in the press Flame detection, etc.

Opinion information can benefit many NLP applications Multi-Perspective Question Answering

[Stoyanov, Cardie, Litman and Wiebe. AAAI WS 2004] and

[Stoyanov, Cardie and Wiebe. HLT-EMNLP 2005] Opinion-Oriented Information Retrieval Clustering, etc.


Annotation Process


`

annotating topics of opinions veselin stoyanov claire cardie

Documents

morocco finegrained

bitter attack slide

australian press slide

controversial late t

s t italian coach lippi

main components

beloved t socceroos

controversial late penalty