1

Another Application of STS: GALE Distillation

Marjorie Freedman, Scott Miller, Sameer Pradhan, Ralph Weischedel

March 2012

2

STS and GALE Distillation/BOLT IR

Given an open ended query, find relevant non-redundant passages that answer the query in a corpus, e.g.,

– List facts about [Saudi Arabia's decision to invest in Egypt]– List facts about [Hugo Chavez calls Vicente Fox a “lap dog” of President Bush”]

STS as a trigger of relevance and to identify redundancy

BBN STS components: synonyms (without WSD use first sense in WordNet), syntactic normalization, co-reference (limited to classes covered by ACE), name variation

arrest

his

possessive

Baghdad

in

captured

Abbas

object

Baghdad

in

Match?

‘Abbas was captured in Baghdad’ arrest (capture, apprehend,

apprehension, detain)

his (Abu Abbas, the PLF leader,

Mohammed Abbas, Abul

Abbas)

possessive

Baghdad (Baghhdad, the Iraqi capital,

Bagadad, Baghdag, Bagdad)

in

Semantic

Augments

~80% Matchdespite minimal token overlap

‘His arrest in Baghdad’

‘His arrest in Baghdad’

3

STS and IR: Challenges

Related (but not identical) text is important– Question: List facts about [Saudi Arabia's decision to invest in

Egypt]• Ahead of the meeting of the joint Egyptian-Saudi Committee on

April 2 [2006-04-02] , investors made it clear that the period taken to finish the procedures of investment dealings reached 93 days

• They [Saudi and Egyptian businessmen] also agreed on establishing a Saudi-Egyptian company for transporting goods between the markets of the two countries

– Question: List facts about [Hugo Chavez calls Vicente Fox a “lap dog” of President Bush]

• “Don't mess with me, sir, or you will get stung”, snarled Chavez, hurling an additional insult at his former friend, Vicente Fox

• But there is a political dimension at work, too, in his[Chavez] tirade aimed at the U.S.-friendly Fox.

4

Impact of Semantic Components on STS

Measure precision of the top ranked answer to a query– 233 query-document pairs with at least 1 relevant answer in each

document

3 versions of STS:– Node Matching: Bag-of-words overlap with stop words removed

• In the case of ties, select the earliest snippet in the document

– +WordNet: Bag-of-words expanded to include synonyms (from the first WordNet Synset)

– +Predicate Argument Connections: Penalize matches where predicate argument structure differs

English

Node 79.0%

Node+WordNet 84.1%

Predicate Argument Connections 88.0%

5

STS over Machine Translation: Challenges

Errors in the machine translation degrade component performance (e.g. parsing, co-reference)– Reference: Last year Russia presented a plate of gold to Iran

• MT: Russia and US - European blessing submitted last year dish went to Iran ,

– Reference: The National Authority had conducted international communications to warn about the danger of expanding the settlements and building the dividing wall

• MT: The National Authority held contacts international warning against the expansion of settlements and building the separating wall

– Reference: Reporters Without Borders from the International Press Freedom Organization criticized the Yahoo Internet Company for providing the Chinese Communist Party with information and cooperating with Chinese Communist Party police

• MT: International press freedom organization Reporters Without Borders accused Yahoo Internet company to provide information to the Chinese authorities and police cooperation with China.

6

STS in Arabic and Chinese

STS in Arabic and Chinese– Compare node-only, syntactic parses, and normalized predicate arguments– Gains from semantic

information are smaller in Arabic and Chinese

Measure impact of translation errors for STS in Arabic and Chinese– Translate the question, use STS to find relevant answers in Arabic/Chinese

documents, compare• Reference translation• Web English Arabic, English Chinese (EA, EC not a priority for GALE)• Pre-GALE English Arabic, English Chinese

– Errors in translation degrade STS• More than lack of semantic information

– Low precision numbers suggest that when translation quality is low, STS preferred non-similar text over similar text!

Chinese Arabic

Reference 86.0% 91.3%

Web 83.9% 84.8%

Pre-GALE 65.0% 79.1%

English Chinese Arabic

Node 84.1% 85.3% 90.9%

Parse-Edge 85.4% 86.0% 91.3%

Proposition-Edge 88.0% 86.0% NA