aquaint a hybrid approach to answering biographical/definitional questions ana licuanan, scott...

AQUAINT

A Hybrid Approach to Answering A Hybrid Approach to Answering Biographical/Definitional QuestionsBiographical/Definitional Questions

Ana Licuanan, Scott Miller, Ralph Weischedel, Jinxi Xu

10 June 2003

2

AQUAINTTechniquesTechniques

• Sentence selection using

– Information Retrieval (IR)

– Linguistic features • Appositives • Copulas • Propositions

– Semantic processing using Information Extraction • Co-reference within document• Relations

• Sentence compression

• Redundancy removal

3

AQUAINTBaseline for Comparison PurposesBaseline for Comparison Purposes

Question Classification

Question

Sentence Ranking by IR

List of Sentences

Training Data

4

AQUAINTSentence Selection by IRSentence Selection by IR

• Hypothesis: – Good sentences tend to contain words that are common in

human created biographies• Method:

– Candidate sentences that contain the question subject are ranked according to their similarity to a set of training biographies

• Similarity is computed using a IR engine (BBN's IR system)• Each sentence is treated as a query• The training biographies are treated as a single vector of words• The similarity scores are normalized to make them comparable

– Top N sentences containing the question term returned as the answer

• Training Data:– 17,000 short biographies from www.s9.com– 6,000 online encyclopedia biographies from www.wikipedia

.org

5

AQUAINTSystem DiagramSystem Diagram

Question Classification

Question

Document Retrieval

Linguistic Processing

Compression

Phrase Ranking using Linguistic & Semantic

Features

Redundancy Removal

Treebank

Name Annotation

List of Responses

Parsing

Name Tagging

Proposition Finding

Co-reference

Relation Extraction

Semantic Processing

Training Data

Linguistically Motivated

6

AQUAINT

Sentence Selection using Linguistic Sentence Selection using Linguistic FeaturesFeatures

• Hypothesis: – Good sentences tend to contain linguistic features that are common

in human generated answers• Method:

– Good features include the target (QTERM) as an argument in• Propositions • Appositives• Copulas extracted from parse trees

– Headword is used to represent argument value to reduce data sparseness

– Sentence with a feature f is ranked by • P(R|f)=P(R) P(f|R) / P(f) (Bayes rule)• P(f|R) estimated from training data (i.e, mined biographies)• P(f) estimated from a background corpus (a sample of the AQUAINT

corpus with 10M words)

• Example: “Blobel, a biologist at Rockefeller University, won the Nobel Prize in Medicine.”– Proposition: <sub>=QTERM <verb>=“won” <obj>=“prize”– Appositive: <appositive>=“biologist”

7

AQUAINT

Sentence Selection based on Sentence Selection based on Information ExtractionInformation Extraction

• Hypothesis: – Good sentences tend to contain semantic relations that are

common in human generated answers• Method:

– SERIF, a state of art Information Extraction Engine used– Co-reference used for name comparison, e.g.,

• Depending on context, “he” and “Bush” may be the same person

– Relations used as additional features for sentence selection. Types of relations include:

• Spouse-of (e.g. “Clinton”, “Hillary”)

• Founder-of (e.g. “Gates”, “Microsoft”)

• Management-of (e.g. “Welch”, “GE”)

• Residence-of (e.g. “John Doe”, “Boston”)

• Citizenship-of (e.g. “John Doe”, “American”)

• Staff-of (e.g. “Weischedel”, “BBN”)

8

AQUAINTSentence CompressionSentence Compression

• Motivation:

– A good sentence may contain portions irrelevant to the question

– Goal: extract only the pertinent parts of a sentence

• Method:

– Operations are performed on parse trees

– Find the smallest phrase that contains all the arguments of an important fact (i.e. proposition/appositive/copula/relation)

– Relative clauses not attached to the question term are trimmed from phrase

9

AQUAINTRedundancy RemovalRedundancy Removal

• Hypothesis: – Challenge: variations of surface strings – Challenge mitigated by the use of linguistic and semantic

features• Method:

– Each response item (a sentence or phrase) is decomposed into a set of features

• Appositives• Copulas• Propositions • Relations

– All candidate items are ranked based on features PRf) = P(R) . P (f|R) /P(f)

– For each response item, • Output it if it contains new features• Skip it if it does not

10

AQUAINT

Case study: “Who is Guenter Blobel?”

11

AQUAINT

Reference (Human) Answer: Reference (Human) Answer: Full SentencesFull Sentences

Dr. Gunter Blobel is a cellular and molecular biologist at Rockefeller University in New York City. He won the 1999 Nobel Prize in medicine for discovering that proteins carry signals that act as ZIP codes, helping them find their correct locations within the cell. The research that Blobel has conducted for 30 years helps explain the molecular mechanisms behind diseases like cystic fibrosis.

Blobel said he was donating most of the prize money to the Friends of Dresden, an independent American group that supports the restoration and preservation of Dresden's artistic and architectural legacy. He witnessed thebombing of Dresden as a child. Blobel was born in 1936 in Waltersdorf, Silesia, Germany, now part of Poland.Blobel graduated from high school with high grades, but couldn't continue to study because he didn't want to join the Communist youth parties. Blobel was born in 1936 in Waltersdorf, Silesia, Germany, now part of Poland.

After earning a medical degree from the University of Tubingen, he interned at a small hospital where he said he "realized that treatment of disease was irrational and not based on profound knowledge." His interest then became research. In the early 1950s, Blobel escaped to the West through Berlin and became a U.S. citizen in the 1980s. Blobel earned a Ph.D. from the University of Wisconsin in 1967. That same year, Blobel moved to Rockefeller University as a post-doctoral fellow largely because he wanted to work with Dr. George E. Palade, a pioneering cell biologist.

12

AQUAINTReference (Human) Answer: PhrasesReference (Human) Answer: Phrases

• A cellular and molecular biologist at Rockefeller University

• Won the 1999 Nobel Prize in medicine for discovering that proteins carry signals that act as ZIP codes, helping them find their correct locations within the cell

• Has conducted research for 30 years • Research helps explain the molecular mechanisms

behind diseases like cystic fibrosis • Born in 1936 in Waltersdorf, Silesia, Germany, now part

of Poland • Earned a medical degree from the University of Tubingen • Escaped to the West through Berlin and became a U.S.

citizen in the 1980s• Earned a Ph.D. from the University of Wisconsin in 1967.

13

AQUAINTSentences Selected by IRSentences Selected by IR

•Blobel was born in 1936 in Waltersdorf , Silesia , Germany , now part of Poland

•The Nobel Prize was awarded not for a single major discovery , but for the series of findings in the decades that it took to prove the hypothesis , Blobel said

•`` My work was a direct extension of the work Dr. Palade had started at Rockefeller , '' Blobel said of Palade , who went on to win a Nobel Prize in 1974

•In the early 1950s , Blobel escaped to the West through Berlin

•In 1971 , Blobel and Dr. David D. Sabatini , who now heads cell biology at New York University School of Medicine , proposed a bold idea known as the signal hypothesis

•Blobel , a cellular and molecular biologist , discovered in the early 1970s that newly synthesized proteins have an intrinsic signal that is essential for directing them

•Blobel , who recalled witnessing the bombing of Dresden as a child , said he was donating most of the $ 960,000 in prize money to the Friends of Dresden , an independent American group that supports the reconstruction , restoration and preservation of Dresden 's artistic and architectural legacy

•Guenter Blobel , a cell biologist at Rockefeller University in New York , was awarded the Nobel Prize in Physiology or Medicine Monday for discovering how proteins get shipped to their proper destinations within the body after being manufactured by tiny molecular factories inside cells

•Blobel was awarded the Nobel Prize in Physiology or Medicine on Monday for discovering that proteins carry signals that act as ZIP codes , helping them find their correct locations within the cell

Useful/not redundant Useful/but redundant Not usefulHuman Judgements:

14

AQUAINT

Sentence Selected Using Linguistic Sentence Selected Using Linguistic Features Features

Dr. Gunter Blobel , a cellular and molecular biologist at Rockefeller University in New York City , won the 1999 Nobel Prize in medicine on Monday for discovering that proteins carry signals that act as ZIP codes , helping them find their correct locations within the cell . (*APP*=biologist)

-LRB- Angel Franco/New York Times Photo -RRB- -LRB- NYT10 -RRB- NEW YORK -- Oct. 11 , 1999 -- SCI - NOBEL - MEDICINE , 10-11 -- Dr. Guenter Blobel , a cellular and molecular biologist at Rockefeller University on Monday . (*APP*=biologist)

-- Guenter Blobel , a cell biologist at Rockefeller University in New York , was awarded the Nobel Prize in Physiology or Medicine Monday for discovering how proteins get shipped to their proper destinations within the body after being manufactured by tiny molecular factories inside cells . (*APP*=biologist)

Reefers : SCI - NOBEL - MEDICINE -LRB- Undated -RRB- -- Dr. Guenter Blobel , a cellular and molecular biologist at Rockefeller University in New York City , is awarded the 1999 Nobel Prize in Physiology of Medicine for discovering that proteins carry signals that act as ZIP codes , helping them find their correct locations within the cell . (*APP*=biologist)

Blobel was born in 1936 in Waltersdorf , Silesia , Germany , now part of Poland . (verb=born obj=*TERM* in=Germany)

Young Gunter graduated from high school `` with highest grades , although he never studied very much , '' his oldest brother , Dr. Hans Blobel , said in a telephone interview from his home in Geissen , Germany . (verb=graduated, sub=*TERM*)

Blobel reasoned that proteins have to be transported either out of the cell , or to the different organelles . (verb=reasoned, sub=*TERM*)


15

AQUAINTSentences Selected Using RelationsSentences Selected Using Relations





LRB- AP -RRB- -- Dr. Guenter Blobel of The Rockefeller University in New York won the Nobel Prize for medicine today for protein research that shed new light on diseases including cystic fibrosis and early development of kidney stones . (rname=ROLE/GENERAL-STAFF arg1=*TERM* arg2=University)



Relation


16

AQUAINTSentence CompressionSentence Compression








17

AQUAINTRedundancy RemovalRedundancy Removal








18

AQUAINTSystem OutputSystem Output

• A cellular and molecular biologist at Rockefeller University in New York City

• Dr. Guenter Blobel of The Rockefeller University in New York• Blobel was born in 1936 in Waltersdorf , Silesia , Germany , now part

of Poland• Young Gunter graduated from high school `` with highest grades ,

although he never studied very much , ''

19

AQUAINTEvaluationEvaluation

• Goal: – A repeatable, automatic scorer to allow frequent

experiments• Test bed: 26 biographical questions with human created

answers (1 human answer/question)– ¼ from pilot corpus– ¾ from BBN creation

• For each question, system produces the top N response items that are less or equal to the size of the manual answer– BLEU metric from machine translation evaluations used– Answer brevity, which should be rewarded for bio/def QA, is

penalized by BLEU

20

AQUAINT

BLEU Scores with Phrase-based BLEU Scores with Phrase-based Answer KeysAnswer Keys

• Sentence selection via IR and name finding surprisingly good according to BLEU

• Relation extraction and redundancy removal promising

• Sentence compression improves scores on phrasal reference answers

Doc

Ret

.

IR S

ent.

Ling

. Fea

ts.

Rel

atio

ns

Red

unca

ncy

Com

pres

s

4gram

2gram00.050.1

0.150.2

0.250.3

0.350.4

4gram3gram2gram1gram

21

AQUAINT

BLEU Scores with Sentence-based BLEU Scores with Sentence-based Answer KeysAnswer Keys

• BLEU scores consistently higher for longer (human generated) answer keys

• Sentence selection via IR and name finding surprisingly good according to BLEU

• Relation extraction and redundancy removal promising

• Sentence compression hurts scores on sentential reference answers

– May be due to omission of entity name in compressed answersD

oc R

et.

IR S

ent.

Ling

. Fea

ts.

Rel

atio

ns

Red

unca

ncy

Com

pres

s

4gram

1gram0

0.050.1

0.150.2

0.250.3

0.350.4

0.45

4gram3gram2gram1gram

22

AQUAINTBLEU vs. Human JudgmentsBLEU vs. Human Judgments

• Too few human judgments and too little data to draw firm conclusions

• BLEU may not be sufficiently sensitive

– Does not fully agree with human rankings

– But generally does D

oc

Ret

.

IR S

ent.

Lin

g. F

eats

.

Rel

atio

ns

Red

un

can

cy

Co

mp

ress

3gram

Subjective0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

3gram

2gram

Subjective

1gram

23

AQUAINTLessons LearnedLessons Learned

• IR model of biographies improves sentence selection

– Select sentences like those seen in human-generated biographies

• Relation extraction modestly improves sentence selection

• Bleu measure is stable with respect to length of n-grams

• BLEU vs. subjective evaluation

– BLEU promising for automatic evaluation of progress in system development

– May not be accurate enough for cross-system evaluation

24

AQUAINTSummarySummary

• Approach to biography generation combines– Information retrieval– Linguistic analysis– Information extraction– Redundancy detection– Sentence compression

• Automatic evaluation by BLEU metric• Much work remains

– Demonstration tomorrow– Improvements in quality– Extension to

• Cross-document entity tracking• Organization profiles• Definitions of things

– Participation in TREC QA evaluation

aquaint a hybrid approach to answering biographical/definitional questions ana licuanan, scott...

Documents