1

Recognizing Partial Textual Entailment

Torsten Zesch Iryna GurevychUbiquitous Knowledge Processing Lab

Technische Universität Darmstadt

Omer Levy Ido DaganNatural Language Processing Lab

Bar-Ilan University

2

Textual Entailment

3

T: muscles move bones

4

T: muscles move bones

H: muscles generate movement

5

T: muscles move bones

H: muscles generate movement

6

T: muscles move bones

H: the muscles’ main job is to move bones

?

7

T: muscles move bones

H: the muscles’ main job is to move bones

8

Complete Textual Entailment

9

Complete Textual Entailment

10

Complete Textual Entailment. Partial .

11

Complete Textual Entailment. Partial .

(Nielsen et al, 2009)

12

T: muscles move bones

H: the muscles’ main job is to move bones

13

T: muscles move bones

(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones

14

T: muscles move bones

(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones

15

T: muscles move bones

(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones

Student Response Analysis

16

T: muscles move bones

(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones

Student Response Analysis

Summarization

Information Extraction

17

T: muscles move bones

(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones

Facets

18

T: muscles move bones

(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones (main job, muscles) (muscles, move) (move, bones)

Facets

19

T: muscles move bones

(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones (main job, muscles) (muscles, move) (move, bones)

Facets

20

T: muscles move bones

(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones (main job, muscles) (muscles, move) (move, bones)

Facets

21

Facet: a pair of terms and their implied semantic relation.

22

Facet: a pair of terms and their implied semantic relation.

(move, bones)

23

Facet: a pair of terms and their implied semantic relation.

(move, bones)

… is to move bones

theme

24

Recognizing Faceted Entailment

25

T: muscles move bones

H: the muscles’ main job is to move bones

Recognizing Faceted Entailment

26

T: muscles move bones

H: the muscles’ main job is to move bones

Recognizing Faceted Entailment

27

T: muscles move bones

H: the muscles’ main job is to move bones

Recognizing Faceted Entailment

28

T: muscles move bones

H: the muscles’ main job is to move bones

Recognizing Faceted Entailment

29

What did we do?

30

Approach: Leverage Existing Methods

31

Approach: Leverage Existing Methods

32

Exact Match P: 96%, R: 30%

penicillin cures pneumonia (penicillin, cure) Lexical Inference (penicillin, cure) pneumonia is treated by penicillin

Lexical-Syntactic Inference

Approach: Leverage Existing Methods

33

Exact Match P: 96%, R: 30%

penicillin cures pneumonia (penicillin, cure) Lexical Inference (penicillin, cure) pneumonia is treated by penicillin

Lexical-Syntactic Inference

Approach: Leverage Existing Methods

34

Exact Matchpenicillin cures pneumonia (penicillin, cure) Lexical Inference WordNet

pneumonia is treated by penicillin (penicillin, cure) Lexical-Syntactic Inference

Approach: Leverage Existing Methods

35

Exact Matchpenicillin cures pneumonia (penicillin, cure) Lexical Inference WordNet

pneumonia is treated by penicillin (penicillin, cure) Lexical-Syntactic Inference

Approach: Leverage Existing Methods

36

Exact Matchpenicillin cures pneumonia (penicillin, cure) Lexical Inferencepneumonia is treated by penicillin (penicillin, cure) Lexical-Syntactic Inference

Approach: Leverage Existing Methods

penicillin cures pneumonia

pneumonia is treated by penicillin

37

Bar-Ilan University Textual Entailment Engine

(Stern and Dagan, 2011)

38

BIUTEE

39

BIUTEE

pneumonia

cures

penicillin

T H

40

BIUTEE

pneumonia

cures

penicillin

penicillin

treated

pneumonia is by

T H

41

BIUTEE

pneumonia

cures

penicillin

penicillin

treated

pneumonia is by

T H

42

pneumonia

cures

penicillin

43

pneumonia

cures

penicillin

pneumonia

treats

penicillin

cure treat(lexical)

44

pneumonia

cures

penicillin

penicillin

treated

pneumonia is by

pneumonia

treats

penicillin

cure treat(lexical)

active passive(syntactic)

45

Facets to Dependencies

pneumonia is by

treated

penicillin

46

Facets to Dependencies

penicillin

pneumonia is by

treated

47

Baseline

BaseLex

BaseSyn

Majority

Decision Mechanisms

Exact Match

Exact Match OR Lexical Inference

Exact Match OR Syntactic Inference

Lexical Inference

Exact Match OR AND

Syntactic Inference

48

Baseline

BaseLex

BaseSyn

Majority

Decision Mechanisms

Exact Match

Exact Match OR Lexical Inference

Exact Match OR Syntactic Inference

Lexical Inference

Exact Match OR AND

Syntactic Inference

49

Baseline

BaseLex

BaseSyn

Majority

Decision Mechanisms

Exact Match

Exact Match OR WordNet

Exact Match OR BIUTEE

Lexical Inference

Exact Match OR AND

Syntactic Inference

50

Baseline

BaseLex

BaseSyn

Majority

Decision Mechanisms

Exact Match

Exact Match OR WordNet

Exact Match OR BIUTEE

WordNet

Exact Match OR AND

BIUTEE

51

SemEval 2013 Task 7

58

SciEntsBank (Djikovska et al, 2012)

15 Domains

5,106 Student Responses

16,263 Facets

Dataset

59

SciEntsBank (Djikovska et al, 2012)

15 Domains

5,106 Student Responses

16,263 Facets

Dataset

60

SciEntsBank (Djikovska et al, 2012)

15 Domains

5,106 Student Responses

16,263 Facets

Dataset

61

SciEntsBank (Djikovska et al, 2012)

15 Domains

13,145 Train Facets

16,263 Test Facets

Dataset

62

SciEntsBank (Djikovska et al, 2012)

15 Domains

13,145 Train Facets

16,263 Test Facets

Dataset

63

Unseen Answers 9%

Unseen Questions 12%

Unseen Domains 79%

Scenarios

64

Unseen Answers 9%

Unseen Questions 12%

Unseen Domains 79%

Scenarios

65

Unseen Answers 9%

Unseen Questions 12%

Unseen Domains 79%

Scenarios

66

Unseen Answers 9%

Unseen Questions 12%

Unseen Domains 79%

Scenarios

67

Results

68

Performance (Weighted F1)

Unseen Answers Unseen Questions Unseen Domains0

0.2

0.4

0.6

0.8

1

BaselineBaseLexBaseSynMajority

69

Summary

70

Problem: Need fine-grained entailment

Solution: Partial Textual Entailment

Approach: Leverage existing methods

Results: Significant improvement

71

Problem: Need fine-grained entailment

Solution: Partial Textual Entailment

Approach: Leverage existing methods

Results: Significant improvement

72

Problem: Need fine-grained entailment

Solution: Partial Textual Entailment

Approach: Leverage existing methods

Results: Significant improvement

73

Problem: Need fine-grained entailment

Solution: Partial Textual Entailment

Approach: Leverage existing methods

Results: Significant improvement

76

Partial Textual Entailment is interesting

77

Partial Textual Entailment is useful

78

Partial Textual Entailment is practical

79

Partial Textual Entailment is interesting

80

Questions?