the relevance of a cognitive model of the mental lexicon to automatic word sense disambiguation
Post on 03-Feb-2016
39 Views
Preview:
DESCRIPTION
TRANSCRIPT
The Relevance of a Cognitive Model of the Mental Lexicon to Automatic Word SenseDisambiguationMartha Palmer and Susan Brown
University of Colorado
August 23, 2008
HJCL /Coling, Manchester, UK1
2
Outline Sense Distinctions Annotation
Sense Inventories created by human judgments WordNet PropBank and VerbNet Mappings to VerbNet and FrameNet Groupings
An hierarchical model of sense distinctions OntoNotes
Empirical evidence of replicable sense distinctions Reading response experiments based on the
hierarchical modelCLEAR – Colorado HJCL2008
Word sense in Machine Translation
Different syntactic frames John left the room
Juan saiu do quarto. (Portuguese) John left the book on the table.
Juan deizou o livro na mesa. Same syntactic frame? Same sense?
John left a fortune.
Juan deixou uma fortuna.
3CLEAR – Colorado HJCL2008
Word sense in Machine Translation – not just syntax
Different syntactic frames John left the room
Juan saiu do quarto. (Portuguese) John left the book on the table.
Juan deizou o livro na mesa. Same syntactic frame? Same sense?
John left a fortune to the SPCA.
Juan deixou uma fortuna.
HJCL2008--4CLEAR – Colorado HJCL2008
5
WordNet – Princeton (Miller 1985, Fellbaum 1998)
On-line lexical reference (dictionary) Nouns, verbs, adjectives, and adverbs grouped into
synonym sets Other relations include hypernyms (ISA), antonyms,
meronyms Typical top nodes - 5 out of 25
(act, action, activity) (animal, fauna) (artifact) (attribute, property) (body, corpus)
CLEAR – Colorado HJCL2008
6
WordNet – Princeton – leave, n.4, v.14 (Miller 1985, Fellbaum 1998) Limitations as a computational lexicon
Contains little syntactic information No explicit lists of participants Sense distinctions very fine-grained, Definitions often vague
Causes problems with creating training data for supervised Machine Learning – SENSEVAL2
Verbs > 16 senses (including call) Inter-annotator Agreement ITA 71%, Automatic Word Sense Disambiguation, WSD 64%
Dang & Palmer, SIGLEX02
CLEAR – Colorado HJCL2008
7
PropBank – WSJ Penn Treebank
a GM-Jaguar pact
that would give
*T*-1
the US car maker
an eventual 30% stake in the British company
Arg0
Arg2
Arg1
expect(Analysts, GM-J pact)give(GM-J pact, US car maker, 30% stake)
Analysts have been expecting a GM-Jaguar pact that would give the U.S. car maker an eventual 30% stake in the British company.a GM-Jaguar
pact
Arg0 Arg1
have been expecting
Analysts
Palmer, Gildea, Kingsbury., CLJ 2005
CLEAR – Colorado HJCL2008 PropBank - Palmer
8
Lexical Resource - Frames Files: giveRoles: Arg0: giver Arg1: thing given Arg2: entity given to
Example: double object The executives gave the chefs a standing ovation. Arg0: The executives REL: gave Arg2: the chefs Arg1: a standing ovation
PropBank - PalmerCLEAR – Colorado HJCL2008
9
Word Senses in PropBank Orders to ignore word sense not feasible for 700+
verbs Mary left the room Mary left her daughter-in-law her pearls in her will
Frameset leave.01 "move away from":Arg0: entity leavingArg1: place left
Frameset leave.02 "give":Arg0: giver Arg1: thing givenArg2: beneficiary
How do these relate to traditional word senses in VerbNet and WordNet?
PropBank - PalmerCLEAR – Colorado HJCL2008
10
Limitations to PropBank as a Sense Inventory
Sense distinctions are very coarse-grained – only 700 verbs
High ITA, > 94%, High WSD,> 90%
PropBank - PalmerCLEAR – Colorado HJCL2008
11
Limitations to Levin Classes as a Sense Inventory
Coverage of only half of the verbs (types) in the Penn Treebank (1M words,WSJ)
Usually only one or two basic senses are covered for each verb
Confusing sets of alternations Different classes have almost identical
“syntactic signatures” or worse, contradictory signatures
Dang, Kipper & Palmer, ACL98
VerbNet – Kipper, Dang, PalmerCLEAR – Colorado HJCL2008
12
Intersective Levin Classes
More syntactically and semantically coherent sets of syntactic patterns explicit semantic components relations between senses
Multiple class memberships viewed as base classes and regular sense extensions
VERBNET
Kipper, Dang & Palmer, IJCAI00, Coling00
CLEAR – Colorado HJCL2008
13
VerbNet – Karin Kipper Class entries:
Capture generalizations about verb behavior Organized hierarchically Members have common semantic elements,
semantic roles and syntactic frames Verb entries:
Refer to a set of classes (different senses) each class member linked to WN synset(s) (not
all WN senses are covered)
VerbNetCLEAR – Colorado HJCL2008
14 HJCL2008
15
Mapping from PropBank to VerbNet(similar mapping for PB-FrameNet)
Frameset id = leave.02
Sense =
give
VerbNet class =
future-having 13.3
Arg0 Giver Agent/Donor*
Arg1 Thing given Theme
Arg2 Benefactive Recipient
VerbNet CLEAR – Colorado
*FrameNet Label Baker, Fillmore, & Lowe, COLING/ACL-98Fillmore & Baker, WordNetWKSHP, 2001
HJCL2008
16
Mapping from PB to VerbNetverbs.colorado.edu/~semlink
VerbNetCLEAR – Colorado HJCL2008
17
Mapping PropBank/VerbNet http://verbs.colorado.edu/~mpalmer/verbnet
Extended VerbNet now covers 80% of PropBank tokens. Kipper, et. al., LREC-04, LREC-06
(added Korhonen and Briscoe classes) Semi-automatic mapping of PropBank
instances to VerbNet classes and thematic roles, hand-corrected.
VerbNet class tagging as automatic WSD Run SRL, map Arg2 to VerbNet roles, Brown
performance improvesVerbNet
Yi, Loper, Palmer, NAACL07
CLEAR – Colorado HJCL2008
Limitations to VN/FN as sense inventories Concrete criteria for sense distinctions
Distinct semantic roles Distinct frames Distinct entailments
But…. Limited coverage of lemmas For each lemma, limited coverage of senses
CLEAR – Colorado 18HJCL2008
Sense inventory desiderata
Coverage of WordNet Sense distinctions captured by concrete
differences in underlying representations as in VerbNet and FrameNet Distinct semantic roles Distinct frames Distinct entailments
Start with WordNet and be more explicit Groupings
CLEAR – Colorado HJCL2008 19
20
WordNet: - leave, 14 senses, grouped
WN1, WN5,WN8
WN6 WN10 WN2 WN 4 WN9 WN11 WN12
WN14 Wnleave_off2,3 WNleave_behind1,2,3
WNleave_alone1 WN13
WN3 WN7
WNleave_off1
WNleave_out1, Wnleave_out2
Depart, a job, a room, a dock, a country
Leave behind, leave alone
“leave off” stop, terminate
exclude
Create a State
CLEAR – Colorado HJCL2008 Groupings
21
WordNet: - leave, 14 senses, groups, PB
WN1, WN5,WN8
WN6 WN10 WN2 WN 4 WN9 WN11 WN12
WN14 WNleave_off2,3 WNleave_behind1,2,3
WNleave_alone1 WN13
WN3 WN7
WNleave_off1
WNleave_out1, WNleave_out2
Depart, a job, a room, a dock, a country (for X)
Leave behind, leave alone
stop, terminate: the road leaves off, not leave off your jacket, the resultsexclude
Create a State /cause an effect:Left us speechless, leave a stain
CLEAR – Colorado
4
2
1
3
5
HJCL2008 Groupings
22
Overlap between Groups and PropBank Framesets – 95%
WN1 WN2 WN3 WN4
WN6 WN7 WN8 WN5 WN 9 WN10
WN11 WN12 WN13 WN 14
WN19 WN20
Frameset1
Frameset2
develop
Palmer, Dang & Fellbaum, NLE 2007
CLEAR – Colorado HJCL2008 Groupings
23
Sense Hierarchy (Palmer, et al, SNLU04 - NAACL04, NLE07, Chen, et. al, NAACL06)
PropBank Framesets – ITA >90% coarse grained distinctions
20 Senseval2 verbs w/ > 1 FramesetMaxent WSD system, 73.5% baseline, 90%
Sense Groups (Senseval-2) - ITA 82% Intermediate level (includes Levin classes) – 71.7%
WordNet – ITA 73% fine grained distinctions, 64%
Tagging w/groups, ITA 90%, 200@hr,Taggers - 86.9% Semeval07
Chen, Dligach & Palmer, ICSC 2007
CLEAR – Colorado HJCL2008 Groupings
Groupings Methodology – Human Judges(w/ Dang and Fellbaum) Double blind groupings, adjudication
Syntactic Criteria (VerbNet was useful) Distinct subcategorization frames
call him a bastard call him a taxi
Recognizable alternations – regular sense extensions: play an instrument play a song play a melody on an instrument SIGLEX01, SIGLEX02, JNLE07
24CLEAR – Colorado HJCL2008 Groupings
Groupings Methodology (cont.)
Semantic Criteria Differences in semantic classes of arguments
Abstract/concrete, human/animal, animate/inanimate, different instrument types,…
Differences in the number and type of arguments Often reflected in subcategorization frames John left the room. I left my pearls to my daughter-in-law in my will.
Differences in entailments Change of prior entity or creation of a new entity?
Differences in types of events Abstract/concrete/mental/emotional/….
Specialized subject domains
25CLEAR – Colorado HJCL2008 Groupings
Creation of coarse-grained resources Unsupervised clustering using rules (Mihalcea &
Moldovan, 2001)
Clustering by mapping WN senses to OED (Navigli, 2006).
OntoNotes - Manually grouping WN senses and annotating a corpus (Hovy et al., 2006)
Supervised clustering WN senses using OntoNotes and another set of manually tagged data (Snow et al., 2007) .
26CLEAR – Colorado HJCL2008 Groupings
OntoNotes Goal: Modeling Shallow Semantics DARPA-GALE AGILE Team: BBN, Colorado, ISI,
Penn Skeletal representation of literal
meaning Synergistic combination of:
Syntactic structure Propositional structure Word sense Coreference
Text
Co-referenceWord Sense wrt Ontology
Treebank
PropBank
OntoNotesAnnotated Text
27CLEAR – Colorado HJCL2008
Empirical Validation – Human Judges the 90% solution (1700 verbs)
28CLEAR – Colorado
Leave 49% -> 86%
HJCL2008 Groupings
Question remains: Is this the “right” level of granularity? “[Research] has not directly addressed the
problem of identifying senses that are distinct enough to warrant, in psychological terms, a separate representation in the mental lexicon.” (Ide and Wilks, 2006)
Can we determine what type of distinctions are represented in people’s minds?
Will this help us in deciding on sense distinctions for WSD?
29CLEAR – Colorado HJCL2008
30
Sense Hierarchy PropBank Framesets – ITA >90% coarse grained distinctions
20 Senseval2 verbs w/ > 1 FramesetMaxent WSD system, 73.5% baseline, 90%
Sense Groups (Senseval-2) - ITA 82% Intermediate level (includes Levin classes) – 71.7%
WordNet – ITA 73% fine grained distinctions, 64%
CLEAR – Colorado HJCL2008 Groupings
Computational model of the lexicon based on annotation Hypothesis: Syntactic structure overtly marks
very coarse-grained senses Subsequently subdivided into more and more
fine-grained distinctions. A measure of distance between the senses
The senses in a particular subdivision share certain elements of meaning.
CLEAR – Colorado 31HJCL2008
Psycholinguistic theories on semantic representations – Susan Brown Discrete-senses theory (Klein and Murphy, 2001)
Each sense has its own separate representation. Related senses and unrelated senses are stored
and processed in the same way Shared-representation theory (Rodd et al., 2002)
Related senses share a portion of their meaning representation
32Brown, ACL08CLEAR – Colorado HJCL2008
Hypotheses
33
Broke the radio (WN 3)
Broke the vase (WN 5)
Discrete-Senses
Brown, ACL08CLEAR – Colorado HJCL2008
Hypotheses
34
Broke the radio (WN 3)
Broke the vase (WN 5)
Discrete-Senses
Slow access
Brown, ACL08CLEAR – Colorado HJCL2008
Broke the radio (WN 3)
Broke the vase (WN 5)
Broke the horse (WN 12)
Discrete-Senses
Slow access
35Brown, ACL08
Hypotheses
CLEAR – Colorado HJCL2008
Broke the radio (WN 3)
Broke the vase (WN 5)
Broke the horse (WN 12)
Discrete-Senses
Slow access Slow access
36Brown, ACL08
Hypotheses
CLEAR – Colorado HJCL2008
Broke the radio (WN 3)
Broke the vase (WN 5)
Broke the horse (WN 12)
Discrete-Senses
Slow access Slow access
Shared-Representn
37Brown, ACL08
Hypotheses
CLEAR – Colorado HJCL2008
Broke the radio (WN 3)
Broke the vase (WN 5)
Broke the horse (WN 12)
Discrete-Senses
Slow access Slow access
Shared-Representn
Fast access
38Brown, ACL08
Hypotheses
CLEAR – Colorado HJCL2008
Broke the radio (WN 3)
Broke the vase (WN 5)
Broke the horse (WN 12)
Discrete-Senses
Slow access Slow access
Shared-Representn
Fast access Slow access
39Brown, ACL08
Hypotheses
CLEAR – Colorado HJCL2008
Procedure
Semantic decision task Judging semantic coherence of short phrases
banked the plane “makes sense” hugged the juice doesn’t “make sense”
Pairs of phrases with the same verb Primed with a sense in the first phrase Sense in the second phrase was one of 4
degrees of relatedness to the first
40Brown, ACL08CLEAR – Colorado HJCL2008
Materials – stimuli based on groupings
Prime Target
Unrelated banked the plane banked the money
Distantly related ran the track ran the shop
Closely related broke the glass broke the radio
Same sense cleaned the cup cleaned the shirt
41Brown, ACL08CLEAR – Colorado HJCL2008
Mean response time (in ms)
700800900
100011001200130014001500
Same Close Distant Unrelated
42Brown, ACL08
CLEAR – Colorado HJCL2008
Mean accuracy (% correct)
40
50
60
70
80
90
100
Same Close Distant Unrelated
43Brown, ACL08CLEAR – Colorado HJCL2008
Broke the radio
Broke the vase
Broke the horse
Discrete-Senses
Slow access Slow access
Shared-Representn
Fast access Slow access
44Brown, ACL08CLEAR – Colorado HJCL2008
Closely related senses were processed more quickly (t32=5.85; p<.0005)
Broke the radio
Broke the vase
Broke the horse
Discrete-Senses
Slow access Slow access
Shared-Representn
Fast access1157 ms
Slow access1330 ms
45Brown, ACL08CLEAR – Colorado HJCL2008
Shared-representation theory
Highly significant linear progression for response time (F1,32=95.8; p<.0001) and accuracy
(F1,32=100.1; p<.0001). Overlapping portions of meaning
representation Closely related senses share a great deal Distantly related senses share only a little Unrelated senses share nothing
46Brown, ACL08CLEAR – Colorado HJCL2008
Implications for WSD
Enumerating discrete senses may be a convenient (necessary?) construct for computers or lexicographers
Little information loss when combining closely related senses
Distantly related senses are more like homonyms, so they are more important to keep separate
47CLEAR – Colorado HJCL2008
Computational model of the lexicon based on annotation Hypothesis: Syntactic structure overtly marks
very coarse-grained senses Subsequently subdivided into more and more
fine-grained distinctions. A measure of distance between the senses
The senses in a particular subdivision share certain elements of meaning.
Computational model provided us with the stimuli and the factors for the experiments.
CLEAR – Colorado 48HJCL2008
Synergy between cognitive models and computational models of the lexicon Hierarchical computational model provides
stimuli and factors for response time experiment
Confirmation of shared representation theory provides support for computational model and suggestions for richer representations
All based on Human Judgments!
CLEAR – Colorado 49HJCL2008
Future work in psycholinguistics Further analysis of categories of relatedness
Closely related pairs had 2 literal senses Distantly related pairs often had 1 literal and 1
metaphoric sense Corpus study of literal and metaphoric uses of
verbs and their frequency Use for further experiments on how people store
and process sense distinctions Annotation study looking at how well people can
make fine-grained sense distinctions in context
50CLEAR – Colorado HJCL2008
Future work in NLP
Improvements to WSD Dynamic dependency neighbors
(Dligach & Palmer, ACL08, ICSC 08) Using language models for sentence selection
Semi-supervised clustering of word senses Hierarchical clustering? Soft clustering? Against PB, VN, FN, Groupings, WN, etc.
Clustering of verb classes…..
CLEAR – Colorado 51HJCL2008
Acknowledgments
We gratefully acknowledge the support of the National Science Foundation Grant NSF-0415923, Consistent Criteria for Word Sense Disambiguation and DARPA-GALE via a subcontract from BBN.
We thank Walter Kintsch and Al Kim for their advice on the psycholinguistic experiments.
52CLEAR – Colorado HJCL2008
Leave behind, leave alone…
John left his keys at the restaurant.We left behind all our cares during our vacation.They were told to leave off their coats.Leave the young fawn alone.Leave the nature park just as you found it.I left my shoes on when I entered their house.When she put away the food she left out the pie. Let's leave enough time to visit the museum.He'll leave the decision to his wife.When he died he left the farm to his wife.I'm leaving our telephone and address with you.
CLEAR – Colorado 53HJCL2008
Category criteria
WordNet OED Relatedness Rating (0-3)
Unrelated pairs Different Unrelated 0-0.3
Distantly related different Related 0.7-1.4
Closely related Different 1.7-2.4
Same sense same 2.7-3.0
54CLEAR – Colorado HJCL2008 Brown, ACL08
Closely related senses were processed more quickly (t32=5.85; p<.0005) and accurately (t32=8.65;
p<.0001) than distantly related and unrelated senses.
Broke the radio
Broke the vase
Broke the horse
Discrete-Senses
Slow access Slow access
Shared-Repr Fast access1157 ms91%
Slow access1330 ms71%
55Brown, ACL08CLEAR – Colorado HJCL2008
Closely related senses were processed more quickly (t32=5.85; p<.0005) and accurately (t32=8.65;
p<.0001) than distantly related and unrelated senses.
Distantly related senses were processed more quickly (t32=2.38; p<.0001) and accurately (t32=5.66; p<.0001) than unrelated senses.
Broke the radio
Broke the vase
Broke the horse
Discrete-Senses
Slow access Slow access
Shared-Repr Fast access1157 ms91%
Slow access1330 ms71%
56Brown, ACL08CLEAR – Colorado HJCL2008
top related