1

Discriminating Among Word Meanings by Identifying Similar Contexts

Amruta PurandareAdvisor: Dr. Ted Pedersen

July 27, 2004

Research Supported by National Science FoundationFaculty Early Career Development Award (#0092784)

2

Word Sense Discrimination

shells exploded in a US diplomatic complex in Liberiashell scripts are user interactive

artillery guns were used to fire highly explosive shellsthe biggest shop on the shore for serious shell collectors

shell script is a series of commands written into a file that Unix executesshe sells sea shells by the sea shore

sherry enjoys walking along the beach and collecting shellsfirework shells exploded onto usually dark screens in a variety of colors

shells automate system administrative taskswe specialize in low priced corals, starfish and shells

we help people in identifying wonderful sea shells along the coastlinesshop at the biggest shell store by the shoreshell script is much like the ms dos batch file

3

sherry enjoys walking along the beach and collecting shellswe specialize in low priced corals, starfish and shells

we help people in identifying wonderful sea shells along the coastlinesshop at the biggest shell store by the shore

she sells sea shells by the sea shorethe biggest shop on the shore for serious shell collectors

shell script is much like the ms dos batch fileshell script is a series of commands written into a file that Unix executes

shell scripts are user interactiveshells automate system administrative tasks

shells exploded in a US diplomatic complex in Liberiafirework shells exploded onto usually dark screens in a variety of colors

artillery guns were used to fire highly explosive shells

4

Our Approach

Strong Contextual HypothesisSea Shells => (sea, beach, ocean, water, corals)Bomb Shells => (kill, attack, fire, guns, explode)Unix Shells => (machine, OS, computer, system)

Corpus—Based Unsupervised Clustering

Knowledge—LeanPortable – Other languages, domainsScalable – Large Raw TextAdaptable – Fluid Word Meanings

5

FeatureSelection

ContextRepresentation

MeasuringSimilarities

ClusteringEvaluation

Methodology

6

Surface Lexical Features

Unigrams in today’s world the scallop is a popular design in architecture and iswell known as the shell gasoline logo

Bigramsshe sells sea shells on the sea shore

Co-occurrencesbivalve shells are mollusks with two valves joined by a hingeshells can decorate an aquarium

7

Local TrainingPectens or Scallops are one of the few bivalve shells that actually swim. This is accomplished by rapidly opening & closing their valves, sending the shell backward.

Fire marshals hauled out something that looked like a rifle with tubes attached to it, along with several bags of bullets and shells.

If you hear a snapping sound when you’re in the water, chances are it is the sound of the valves hitting together as it opens and shuts its shell.

Teenagers tried to make a bomb or some kind of homemade fireworks by taking the bullets and shotgun shells apart and collecting the black powder.

Bivalve shells are mollusks with two valves joined by a hinge. Most of the 20,000 species are marine including clams, mussels, oysters and scallops.

There was an explosion in one of the shells, it flamed over the top of the other shells and sealed in the fireworks, so when they ignited, it made it react like a pipe bomb."

These edible oysters are the most commonly known throughout the world as a popular source of seafood. The shell is porcelaneous and the pearls produced from these edible oysters have little value.

8

Global Training

U.S. researchers said sea shells may be the product of a geological accident that flooded ancient oceans with calcium, thereby diversifying marine life. Researchers at the U.S. Geological Survey have found the amount of calcium in sea water shot up between the end of the Proterozoic era (about 544 million years ago) and the early Cambrian period (515 million years ago). This increase, they suggested, allowed soft-bodied marine organisms to create hard shells or body parts from the calcium minerals. The researchers studied the chemical composition of liquids trapped in the cavities of salty rocks called halites, which provide samples of prehistoric oceans.

John Kerry is a man who knows how to keep a secret. The Democratic White House hopeful was so obsessed with making sure the name of John Edwards, his vice presidential running mate, remained under wraps until the announcement that he had vendors who printed up placards and T-shirts sign a non-disclosure agreement. Kerry himself telephoned his plane charter company at 6 p.m. on Monday night to let them in on his decision in time to have the red, white and blue aircraft's decal changed to read "Kerry-Edwards A Stronger America." Edwards did not travel to Pittsburgh to attend the rally at which his name was announced, which also might have alerted the media. After months of speculation, first reports began emerging less than 90 minutes before Kerry made his public announcement at 9 a.m.

9

1st Order Context Vectors

C1: if she sells shells by the sea shore, then the shells she sells must be sea shore shells and not firework shells

C2: store the system commands in a unix shell and invoke csh to execute these commands

2

0

commands

101100C2

010022C1

unixfireworkexecutesystemshoresea

10

2nd Order Context VectorsThe largest shell store by the sea shore

06272.852.913362.608420.0321176.8451.021O2context

018818.55000205.5469134.5102Store

000136.044129.57600Shore

008.739951.781230.5203324.9818.5533Sea

ArtillerySalesBombsSandyNorth-West

WaterSells

11

2nd Order Context Vectors

12

052.2700.9204.210

28.7203.2401.2802.53

WeaponMissileShootFireDestroyMurderKill

17.77014.646.222.1034.2

19.232.36072.701.282.56

ExecuteCommandBombPipeFireCDBurn

13

Latent Semantic Analysis

Singular Value Decomposition

Resolves Polysemy and Synonymy

Captures Conceptual Similarities

Converts Word Space to Semantic Space

14

Before SVD …

0

0

0

0

2

0

0

Shoot

3020003312

2030002202

0222210000

0210120010

3002322030

2231321020

3020000102

MachineFireExecuteMurderGunsBombPipeUnixKillComputer

15

After SVD …

0.054

0.02

0.471

0.386

0.761

0.75

0.028

Shoot

3.23-0.112.6750.0050.075-0.012.2622.450.592.292

2.67-0.152.2-0.05-0.03-0.081.872.070.421.934

0.8131.030.8121.0131.841.350.45-0.281.25-0.27

0.5520.850.57120.8341.5121.110.289-0.321-0.306

2.041.611.9061.6082.932.161.240.112.110.102

2.1811.582.0221.5872.8992.111.340.242.120.222

2.073-0.091.713-0.0120.021-0.031.451.590.361.48

MachineFireExecuteMurderGunsBombPipeUnixKillComputer

16

Clustering

UPGMAHierarchical : Agglomerative

Repeated BisectionsHybrid : Divisive + Partitional

17

Similarity Coefficients

(X П Y)_____________________

√ |X| * |Y|

Cosine

(X П Y)_______________________

min (|X|,|Y|)

Overlap

(X П Y)_______________

|X U Y|

Jaccard

2 (X П Y)___________________

|X| + |Y|

Dice

(X П Y)Matching

18

Evaluation

2015212C417

110

55111215

10612C310171C2152310C1

19

DataLine, Hard, Serve

4000+ Instances / Word60:40 training:test Split3-5 Senses / Word

SENSEVAL-272 words = 28 V + 29 N + 15 AApprox. 50-100 Test, 100-200 Training8-12 Senses / Word

Associated Press Worldstream NewswireNov 1994-June 2002 by LDC, U Penn539,665,000 words

20

Experiments

TrainingLocal versus Global

FeaturesUnigrams, Bigrams, Co-occurrences

Context Representations1st order versus 2nd order

ClusteringRBR versus UPGMA

21

Conclusions

Smaller Data2nd Order + RBR

Larger Local Data1st Order + UPGMA

Global TrainingHelps 1st Order Bigrams, UPGMAOverall Local Training Better

Corpus + DictionaryHelps only with smaller training data

22

Applications

Word Sense DisambiguationSynonymy IdentificationEmail/News ClassificationName DiscriminationOntology AcquisitionText Summarization

23

Contributions

Systematic ComparisonPedersen & Bruce (1997)Schütze (1998)

Discrimination ParametersFeaturesContext RepresentationsClustering Approaches

24

Contributions

Training VariationsLocal Global

Relative ComparisonRaw CorpusCorpus + Dictionary

Softwarehttp://senseclusters.sourceforge.net

25