textual entailment | learning lexical entailment | wikipedia | extraction types | results &...


Entailment - What is it and what is it good for?

• Question Answering:

• Information Retrieval: “The Beatles”

“Which are produced in ?”luxury cars Britain


Lexical Entailment

• Lexical Entailment rules model such lexical relations

• Part of the Textual Entailment paradigm – a generic

framework for semantic inference

• Encompasses a variety of relations:– Synonymy: Hypertension Elevated blood-pressure

– IS-A: Jim Carrey actor

– Predicates: Crime and Punishment Fyodor Dostoyevsky

– Reference: Abbey Road The Beatles


What was done so far?

• Lexical database, made for computational consumption, NLP resource - WordNet– Costly, need experts, many years of development (since 1985)

• Distributional similarity– Country and State share similar contexts

– But also Nurse and Doctor, Bear and Tiger - Low precision

• Patterns:– NP1 such as NP2 luxury car such as Jaguar

– NP1 and other NP2 dogs and other domestic pets

– Low coverage, mainly IS-A patterns


Our approach – Utilize Definitions

• Pen: an instrument for writing or drawing with ink.– pen is-an instrument– pen used for writing / drawing– ink is part of pen

• Source of definitions: – Dictionary: describes language terms, slow growth – Encyclopedia: contains knowledge, proper names, events, concepts,

rapidly grow

• We chose Wikipedia– Very dynamic, constantly growing and updating– Covers a vast range of domains– Gaining popularity in research - AAAI 2008 workshop


Extraction Types

•Be-complimentnoun in the position of a compliment of a verb ‘be’

•All-Nounsall nouns in the definition

different likelihood to be entailed


• The likelihood of entailment depends greatly on the syntactic path connecting the title and the noun.– Path in a parsed tree

• An unsupervised entailment likelihood score for a syntactic path p within a definition:

• Split Def-N into Def-Ntop and Def-Nbot– Indicative for rule reliability - Def-Ntop rules’ precision is much higher

than Def-Nbot’s.

Ranking All-Nouns Rules

)(

)(

pC

pCT

film

title directed

by

noun

subj vrel

by-subj

pcomp-n


Extraction Types

•Redirectnoun in the position of a

•Parenthesisall nouns in the definition

•Linkall nouns in the definition


Ranking Rules by Supervised Learning


Ranking Rules by Supervised Learning

Extraction Types

• An alternative approach for deciding which rules to select out of all extracted rules.

• Each rule is represented by:– 6 binary features: one for each extraction type

– 2 binary features: one for each side of the rule indicating whether it is NE

– 2 numerical features: rule sides’ co-occurrence & count extracted

– 1 numeric feature: the score of the path for Def-N extraction type

• Manually annotated set used to train SVMlight

– Varied the J parameter in order to obtain different recall-precision tradeoffs


Results and Evaluation

• The obtained knowledge base include:– About 10 million rules

• For comparison: Snow’s extension to WordNet includes 400,000 relations.

– More than 2.4 million distinct RHSs

– 18% of the rules extracted by more than one extraction type

– Mostly named entities and specific concepts, as expected from encyclopedia

• Two Evaluation types:– Rule-based: rule correctness relative to human judgment

– Inside real application: the utility of the extracted rules for lexical expansion in keyword-based text categorization

Results & Evaluations


Rule-base Evaluation• Randomly sampled 830 rules and annotated them for correctness

– inter annotators agreement achieved Kappa of 0.7

• Precision: the percentage of correct rules

• Est. # of correct rules: number of rules annotated as correct multiply by the

sampling proportion.

Extraction Type

Per TypeAccumulated

PEst. # RulesPR

Redirect0.872,232,8770.870.31

Be-Comp0.82,740,9570.820.6

Def-Ntop0.722,179,3950.770.71

Parenthesis0.7166,8530.770.72

Link0.7708,6380.760.8

Def-Nbot0.471,657,9440.661



Supervised Learning Evaluation

• 5-fold cross validation on the annotated sample:

• Although considering additional information, performance is almost identical to considering only extraction types.

• Further research is needed to improve our current feature set and classification performance.


J0.30.40.50.91.11.3

R0.860.820.750.750.70.66

P0.320.590.730.810.911


Text Categorization Evaluation

• Represent a category by a feature vector of characteristic terms for it.– The characteristic terms should entail the category name.

• Compare the term-based feature vector of a classified document with the feature vectors of all categories. – Assign the document to the category which yields the highest cosine

similarity score (single-class classification).

• 20-News Groups collection

• 3 baselines: No expansions, WordNet, WikiBL, [Snow]

• Also evaluated the union of Wikipedia and WordNet



Text Categorization Evaluation


Rule BasePRF1

Baselines

No Expansion0.530.190.28

WordNet0.460.290.36

WikiBL0.530.190.28

Extraction

Types

Redirect only0.540.210.3

+ Be-comp0.550.210.3

+ Parenthesis and Link0.410.30.35

+ Def-Ntop0.420.30.35

+ Def-Nbot (all rules)0.390.320.35

SVMJ = 0.30.550.210.31

J = 1.10.310.280.3

UnionWN + Wiki (all)0.40.340.37

WN + Wiki (redir + Be-comp)0.50.330.39


Promising Directions for Future Work

• Learning semantic relations in addition to Taxonomical relations (hyponym, synonyms) :

• Fine-grained relations of LE is important for inference

RelationRulePath Pattern

LocationLovek CambodiaLovek city in Cambodia

OccupationGeorge Bogdan Kistiakowsky chemistryGeorge Bogdan Kistiakowsky chemistry professor

CreationCrime and Punishment Fyodor Dostoyevsky

Crime and Punishment is a novel by Fyodor Dostoyevsky

OriginWillem van Aelst DutchWillem van Aelst Dutch artist

AliasDean Moriarty Benjamin Linus Dean Moriarty is an alias of Benjamin Linus on Lost

SpellingEgushawa AgushawayEgushawa, also spelled Agushaway...

Conclusions & Future Work


Promising Directions for Future Work

• Natural Types, naturally phrased entities:– 56,000 terms entail Album

– 31,000 terms entail Politician

– 11,000 terms entail Footballer

– 20,000 terms entail Actor

– 15,000 terms entail Actress

– 4,000 terms entail American Actor



Conclusions


• First large-scale rule base directed to cover LE.

• Learning ontology which is a very important knowledge for reasoning systems (one of the conclusions of the first 3 RTE benchmarks).

• Automatically extracting lexical entailment rules from an unstructured source

• Comparable results, on a real NLP task, to a costly manually crafted resource such as WordNet.


Inference System

t: Strong sales were shown for Abbey Road in 1969.grammar rule: passive to active

Abbey Road showed strong sales in 1969.

lexical entailment rule: Abbey Road The Beatles

The Beatles showed strong sales in 1969.

lexico-syntactic rule: show strong sales gain commercial success

h: The Beatles gained commercial success in 1969.

Textual Entailment

textual entailment | learning lexical entailment | wikipedia | extraction types | results &...

Documents