opinions in question answering
DESCRIPTION
Opinions in Question Answering. Jan Wiebe University of Pittsburgh Claire Cardie Cornell University Ellen Riloff University of Utah. Overview. Techniques and tools to support multi-perspective question answering (MPQA) Goals: produce high-level summaries of opinions - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/1.jpg)
Jan Wiebe University of Pittsburgh
Claire Cardie Cornell University
Ellen Riloff University of Utah
Opinions in Question Answering
![Page 2: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/2.jpg)
Overview
Techniques and tools to support multi-perspective question answering (MPQA)
Goals: produce high-level summaries of opinions incorporate rich information about
opinions extracted from text
![Page 3: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/3.jpg)
Overview
Opinion-oriented information extraction Extract opinion frames for individual
expressions Combine to create opinion-oriented
“scenario” templates
OpinionSummaryTemplate
![Page 4: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/4.jpg)
MPQA Corpus
Grew out of the 2002 ARDA NRRC Workshop on Multi-Perspective Question Answering
Detailed annotations of opinions Freely available (thanks to David Day):
nrrc.mitre.org/NRRC/publications.htm
![Page 5: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/5.jpg)
Collaborations
Interactions with end-to-end system teams
Integrated corpus annotation Pilot opinion evaluation
![Page 6: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/6.jpg)
Outline
Recent activities Subjective sentence identifier Clause intensity identifier Extended annotation scheme Version 1 Q&A corpus Nested opinions Opinion summaries
What’s next
![Page 7: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/7.jpg)
Subjective Sentence Identifier
Input is unlabeled data Evaluated on manual annotations of
the MPQA corpus Accuracy as good as supervised
systems which classify all sentences
![Page 8: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/8.jpg)
Subjective Sentence Identifier
Bootstraps from a known subjective vocabulary, labeling the sentences it can with confidence
Extraction pattern learner finds clues of subjectivity in that corpus
Incorporated into a statistical model trained on the automatically labeled data
Multiple classification strategies 76% accuracy with 54% baseline 80% subj. precision and 66% subj. recall 80% obj. precision: and 51% obj. recall
![Page 9: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/9.jpg)
Clause-level intensity (strength) identification
Maximum intensity of the opinions in a clause
Neutral, low, medium, high Evaluated on manual annotations of
the MPQA corpus
![Page 10: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/10.jpg)
I am furious that my landlord refused to return my security deposit until I sued them.
Example
return
my
that
am
them
sued
I
to
refused
landlord
furious
I
until
deposit
securitymy
High Strength
Medium Strength
Neutral
Opinionated Sentence
![Page 11: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/11.jpg)
Clause-level intensity (strength) identification
Classification and regression learners Accuracy: how many clauses are assigned
exactly the correct class? Mean Squared Error: how close are the
answers to the right ones? Accuracy: classification > regression
23-79% over baseline MSE: regression > classification
57-64% over baseline
![Page 12: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/12.jpg)
Opinion Frames
direct subjective annotation Span: “strongly criticized and condemned” Source: <writer,many-countries> Strength (intensity): high Attitudes: negative toward the report Target: report
The report has been strongly criticized and condemned by many countries.
![Page 13: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/13.jpg)
Major Attitude Types
Positive Negative Arguing for ones world view Intention
![Page 14: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/14.jpg)
Negative and Positive Example
People are happy because Chavez has fallen, she said.
direct subjective annotation span: are happy source: <writer, she, People> attitude:
attitude annotation span: are happy because Chavez has fallen type: positive and negative positive target: negative target:
target annotation span: Chavez has fallen
target annotation span: Chavez
![Page 15: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/15.jpg)
Arguing for World View Example
Putin remarked that events in Chechnia “could be interpreted only in the context of the struggle against international terrorism.”
direct subjective annotation span: remarked source: <writer, Putin> attitude:
attitude annotation span: could be interpreted only in the context of the struggle against international terrorism type: argue for world view target:
target annotation span: events in Chechnia
![Page 16: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/16.jpg)
Characteristics
Sarcastic "Great, keep on buying dollars so there'll be
more and more poor people in the country," shouted one.
Speculative Leaders probably held their breath…
Characteristics of the linguistic realization
![Page 17: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/17.jpg)
Q&A Corpus
Includes 98 documents from the NRRC corpus, split into four topics:
Kyoto Protocol 2002 elections in Zimbabwe U.S. annual human rights report 2002 coup in Venezuela
![Page 18: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/18.jpg)
Q&A Corpus
Includes 30 questions 15 questions classified as fact
What is the Kyoto Protocol about? What is the Kiko Network? Where did Mugabe vote in the 2002 presidential
election? 15 questions classified as opinion
How do European Union countries feel about the US opposition the Kyoto protocol?
Are the Japanese unanimous in their opinion of Bush’s position on the Kyoto Protocol?
What was the American and British reaction to the reelection of Mugabe?
![Page 19: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/19.jpg)
Q&A Corpus
Answers annotations added by two annotators Minimal spans that constituted or
contributed to an answer Confidence Partial?
![Page 20: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/20.jpg)
Difficulties in Corpus Creation
Annotating answers Difficult to decide what constitutes an answer:
Q: “Did most Venezuelans support the 2002 coup?” A: “Protesters…failed to gain the support of the army.” ???
Not clear what sources to attribute to collective entities
European Union: The EU Parliament? Great Britain? GB government? Tony Blair?
The Japanese: The Japanese government? Emperor Akihito? Empress Michiko? The Kiko Network?
![Page 21: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/21.jpg)
Q&A Corpus
Interannotator agreement 85% on average using Wiebe et. al’s agr(a||b) measure 78% and 93%, respectively for each
annotator
![Page 22: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/22.jpg)
Evaluating MPQA Opinion Annotations
Answer probability: estimate P(opinion answer | opinion question) P(fact answer | fact question)
Low-level opinion information reliable predictor facts: 78% opinions: 93%
Answer rank Sentence-based retrieval Filter based on opinion annotations Examine rank of first sentence w/answer Filtering improves answer rank
![Page 23: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/23.jpg)
Summary Representations of Opinions
Direct subjective annotationSource:Attitude:
OpinionSummaryTemplate
![Page 24: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/24.jpg)
Reporting in text
Clapp sums up the environmental movement’s reaction: “The polluters are unreasonable’’
Charlie was angry at Alice’s claim that Bob was unhappy
![Page 25: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/25.jpg)
Hierarchy of Perspective & Speech Expressions
Charlie was angry at Alice’s claim that Bob was unhappy
angry
claim
implicit speech event
unhappy
sums up
implicit speech event
reaction
Clapp sums up the environmental movement’s reaction: “The polluters are unreasonable’’
![Page 26: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/26.jpg)
Baseline 1: Only filter through writer
66% correct
angry
claim
implicit
unhappy
unhappy
unhappy
![Page 27: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/27.jpg)
Baseline 2: Dependency Tree
72% correct
angry
implicit
claim
unhappy
claim
unhappy
claim
unhappy
![Page 28: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/28.jpg)
ML Approach
Features Parse-based Positional Lexical Genre-specific
IND decision trees (mml criterion)
78% correct
![Page 29: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/29.jpg)
Summary Representations of Opinions
Direct subjective annotationSource:Attitude:
OpinionSummaryTemplate
![Page 30: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/30.jpg)
Opinion Summaries
Summaries based on manual annotations Single-document summaries Opinion annotations grouped by source and target Sources characterized by degree of
subjectivity/objectivity Simple graph-based graphical interface
Overview of entire graph Focus on portion of the graph Drill-down to opinion annotations (highlighted) Grouping/deleting of sources/targets JGRAPH package
![Page 31: Opinions in Question Answering](https://reader034.vdocument.in/reader034/viewer/2022042718/56814654550346895db369bf/html5/thumbnails/31.jpg)
The next 6 months
Identify individual expressions of subjectivity
Perform manual annotations Extract Sources Opinion summaries with automatic
annotations