From Words to Senses: A Case Study of Subjectivity

Recognition

Author: Fangzhong Su & Katja Markert (University of Leeds, UK)

Source: COLING 2008 Reporter: Yong-Xiang Chen

Background & Motivation• Subjectivity analysis focuses on determining

whether a language unit expresses subjectivity– private state, opinion or attitude– and, if so, what polarity is expressed

• Many words being subjectivity-ambiguous– Having both subjective and objective senses– Example: two sense of the word “positive”

• having a positive electric charge (objective)• involving advantage or good (subjective)

– The annotation of words independent of sense or domain does not capture such distinctions

Goal & Advantage

• Determine the subjectivity of word sense– Avoid costly annotation during training step– Evaluate how useful of existing resources

• Which are not tailored towards word sense

• Increase the lexica’s usability– Allow to group fine-grained senses into higher-level cl

asses based on subjectivity/objectivity• Improve WSD task

– For subjectivity-ambiguous words

Related work• Esuli and Sebastiani (2006)

– Determine the polarity of word senses in WordNet– Training set: Expand a small, manually determined se

ed set of WordNet senses – Use the resulting larger training set for supervised cla

ssification• Wiebe and Mihalcea (2006)

– Label word senses in WordNet as subjective or objective

– The method relying on • an independent, large manually annotated opinion corpus (M

PQA)• distributional similarity

Subjectivity VS. Polarity

• In this study, do not see polarity as a indicator to the subjectivity of sense– Most subjective senses have a relatively clear

polarity– But polarity can be attached to objective word

s/senses as well• Tuberculosis 結核病 (objective)(negative)

Annotation for subjectivity and polarity of word senses

• Annotate the Micro-WNOp corpus as test set – containing 1,105 WordNet synsets

• Subjectivity– subjective (S), objective (O), both (B)– (B): a WordNet synset contains both opinionated and objective e

xpressions• Polarity

– positive (P), negative (N), varied (V)– (V): a sense’s polarity varies strongly with the context

• Uncompromising(不妥協 ) will be positive or negative depending on what a person is uncompromising

• 7 sub categories– O:NoPol, O:P, O:N, S:P, S:N, S:V, and B

Annotation scheme

• Manually annotate polarity for subjective senses, as well as objective senses that carry a strong association – Annotate subjectivity for finding and analysing

directly expressed opinions– Annotate polarity for either classifying these fu

rther or extracting objective words

High Agreement

• The overall agreement using all 7 categories is 84.6%, with a kappa of 0.77– Between two annotators

• High agreement is due to– annotation of senses instead of words– sense descriptions providing more information– split of subjectivity and polarity annotation

made the task clearer

Gold Standard

• The purpose is focus on subjectivity, so integrate labels into: S, O, B

• The Micro-WNOp corpus includes 298 different words– 97 (32.5%) are subjectivity-ambiguous

• Excluded all senses with the label B from Micro-WNOp for testing the automatic algorithms– resulting in a final 1061 senses

• 703 objective• 358 subjective

Algorithms

1. Standard Supervised Approach2. Sentence Collections: Movie 3. Sentence Collections: MPQA4. Word Lists: General Inquirer 5. Word Lists: Subjectivity List

Standard Supervised Approach• 10-fold cross validation for training and testing on t

he annotated Micro-WNOp corpus• Applied a Naive Bayes classifier• Three types of features:

– Lexical Features• unigrams in the glosses as bag-of-words• WordNet synsets

– Part-of-Speech Features– Relation Features

• Employ 8 relations– antonym, similar-to, derived from, attribute, also-see, direct-hypony

m, direct hypernym, and extended-antonym• Each relation R leads to 2 features

– describe for a sense A how many links of that type it has to synsets in the subjective or the objective training set

Sentence Collections Approach• Cast word sense subjectivity classification as a sentence

classification task• Take the glosses that WordNet provides for each sense

as the sentences to be classified• Can in theory feed any collection of annotated sentences

as training data1. Movie-domain Subjectivity Data Set (Movie)

• 5000 subjective sentences and 5000 objective sentences2. MPQA Corpus

• contains news articles manually annotated at the phrase level• 6127 subjective and 4985 objective sentences

• Use a Naive Bayes algorithm with lexical unigram features

Word Lists Approach• General Inquirer (GI)

– concentrates on word polarity– assume that both positive and negative words in the

GI list are subjective clues– 1915 positive, 2291 negative and 7582 no-polarity

words• Subjectivity clues list (SL)

– centers on subjectivity and provides part-of-speech, subjectivity strength, and prior polarity

– 8,000 subjective words• Both are not include word senses information

and cannot be used directly

Unsupervised algorithm• Consider occurrence of subjective words in gloss to indica

te a subjective sense overall• Adopt rule-based unsupervised algorithm• Compute a subjectivity score S for each WN synset

– summing up the weight values of all subjectivity clues in its gloss• GI:all subjectivity clues weighted 1• SL:2 to strongly subjective clues and 1 to weakly subjective clues

1. If S is equal or higher than an agreed threshold T, then the synset is classified as subjective• Best thresholds: 2 for SL and 4 for the GI

2. Set two thresholds as rule to divide all synsets into subjective/objective training set• Best thresholds

– SL: T1=4 and T2=2– GI: T1=3 and T2=1

Experiments and Evaluation

Discussion• To three star methods, small but consistent

improvement when we use additional features• Why using SL always greatly outperforms GI?

– the GI lexicon is annotated for polarity, not subjectivity• It includes words that we see as objective but with a strong

positive or negative– GI lexicon does not operate with a clearly expressed

polarity definition and leading to conflicting annotations

– GI contains fewer features– GI contains many fewer subjective clues

Discussion

• The results of using sentence dataset are not satisfactory– the subjectivity definition in the Movie corpus

does not seem to match ours• we define a word sense or a sentence as subjectiv

e if it expresses a private state (i.e., emotion, opinion, sentiment, etc.)

• in Movie dataset, its “objective” data set rarely contain opinions about the “movie”, but contain other opinionated content

• for example: about the “characters”

Comparison to Prior Approaches

• VS. SentiWordNet– If the sum of positive and negative scores of a sense in SentiWo

rdNet is more than or equal to 0.5, then it is subjective and otherwise objective

– SentiWordNet achieves 75.3% accuracy on the Micro-WNOp– The CV* and SL* perform slightly better than SentiWordNet

• Test data of Wiebe and Mihalcea (2006) is not publically available– Precision = 48.9%, Recall = 60% for subjective senses– our best SL* method has a precision = 66% at about the same re

call

Conclusion

• Proposed different ways of extracting training data and clue sets

• The effectiveness of the resulting algorithms depends on the different definitions of subjectivity

• At least one of purpose methods performed on a par with a supervised classifier– it is possible to avoid any manual annotation for the

subjectivity classification of word senses