looking at word meaning - uni-konstanz.de · [email protected]...

Overview SVS types&tokens Data Visualization Conclusion References

Looking at Word MeaningAn interactive visualization of Semantic Vector Spaces

for Dutch synsets

Kris Heylen, Dirk Speelman & Dirk Geeraerts

KULeuvenQuantitative Lexicology and Variational Linguistics


Purpose of the talk

• Peak inside the black box of Vector Space Models of lexicalsemantics

• through an interactive visualization of word uses

• Allow Computational Linguists to do a direct, intrinsicevaluation of their models and the semantics they capture

• Provide Lexicologists and Lexicographers with an explorativetool for analyzing word meaning in large corpora


Overview

1. Semantic Vector Spaces as models of word meaning

2. Type vs token-level vector spaces

3. Case study: Data and set-up

4. Visualization

5. Conclusion and future work


Semantic Vector Spaces as models of word meaning

Semantic Vector Spaces in Computational Linguistics

• standard technique in statistical NLP for the large-scaleautomatic modeling of (lexical) semantics

• aka Vector Spaces Models, Distributional Semantic Models,Word Spaces,... (see Turney & Pantel (2010) for overview)

• intuitive rationale, but largely black-box statistical technique

Linguistic origin: Distributional Hypothesis

• ”You shall know a word by the company it keeps” (Firth, 1957)

• a word’s meaning can be induced from its co-occurring words

• words appearing in similar contexts will have similar meanings



PracticalWhich two words out of a set of three have the same meaning?

ongeval, koffie, accident

Occurrences in context from a corpus

Op de Brusselse ring deed zich een ongeval met een vrachtwagen voor’s Morgens drinkt hij een kop koffie met melk en suiker2 bestuurders raakten gekwetst bij een ongeval met een vrachtwagenin de avondspits veroorzaakte een accident een kilometerslange fileals vieruurtje serveert het hotel koffie en gebak voor de gastende auto was betrokken in een accident met een dodelijke afloopMet winterbanden is het risico op een ongeval bij vriesweer veel kleiner



word by context co-occurrence matrixau

to

slac

htoff

er

vrac

htw

agen

file

gekw

etst

suik

er

mel

k

kop

ongeval 120 424 388 82 270 11 3 1accident 154 401 376 99 305 20 1 5koffie 5 8 18 4 1 72 102 93



word by word similarity matrix

ongeval accident koffieongeval 1 .91 .08accident .91 1 .17koffie .08 .17 1


Vector Space Models of lexical semantics

Many different parameter settings

• context definition (document, window, dependency relations)

• weighting and similarity measures (PMI, cosine, jaccard,...)

• dimensionality reduction (SVD, LDA, NNMF, RI...)

• type vs token level; words vs relations

Wide variety of applications

• Psycholinguistic modeling of semantic memory

• Thesaurus extraction (WordNet)

• Lexical entailment, Query expansion

• Word sense disambiguation/induction

• Lexical variation between language varieties

• Historical studies of change in word meaning



Unclear relation between parameters and semantics

• Which semantic structure do SVS models capture and how?

• Task-based evaluations only assess a-priori relations

• actual lexical-semantic structure is richer (Geeraerts (2010))

• Appeal for an intrinsic evaluation (Baroni & Lenci (2011))

SVSs have found little application in Linguistics proper

• Theoretical linguistics is becoming more data-driven

• Lexicologists (and lexicographers) try to describe semanticstructure based on a large number of corpus occurrences

• SVSs can provide such a (preliminary) structure but it needsto be accessible for linguists



Potential win-win solution for both problems

⇒An intuitive visualization of SVS output matrix

Benefits:

• For computational linguist: Making SVS accessible forevaluation by lexical semantic experts that goes beyond thepre-defined semantic relations of task-based evaluation

• For Lexicology: Tool for exploring and analyzing wordmeaning in large amounts of corpus data that unliketraditional concordances have some preliminary structure


Overview




4. Visualization



Type vs token-level SVS

SVSs can model lexical semantics on two levels:

1. the type level: aggregating over all occurrences of a word,giving a representation of a word’s general semantics. (e.g.Thesaurus extraction)

2. the token level: representing the semantics of each individualoccurrence of a word.(e.g. WSD)

Lexicological studies typically take a set of types and analyze howthey ’carve up’ semantic space by looking at their tokens

We use a type-level SVS for finding synsets and a token-level spacefor modeling the tokens within each synset.


Type vs token-level vector spaces

Token vector approach of Schutze (1998):

Token vector = average of context words’ type vectorWhile walking to work, the teacher saw a barking dog chasing a cat

foot

office

nigh

t

hare

pet

pupi

l

milk

purr

walk 4.7 2.3 2.4 0.2 1.9 0.1 0 0work 1.2 4.9 3.2 0 0.1 2.3 0.1 0teacher 0.3 1.3 0.8 0 1.2 4.3 0.5 0.1see 0.2 0.4 1.2 0.7 0.9 0.8 0.7 0.1bark 0.3 0.2 1.9 1.8 2.1 1.8 0.7 2.1chase 2.8 1 2.1 3.1 2.2 1.1 0.9 0.8cat 1.1 0.9 2.3 1.9 3.9 0.5 2.8 4.6

AVERAGE 1.51 1.57 1.99 1.10 1.76 1.56 0.81 1.10


Type vs token-level vector spaces

Our modified approach:

Token vector = weighted average of context words’ type vector,with as weights the PMI values between type and context words

WE

IGH

T

foot

office

nigh

t

hare

pet

pupi

l

milk

purr

walk 1.1 4.7 2.3 2.4 0.2 1.9 0.1 0 0work 0.2 1.2 4.9 3.2 0 0.1 2.3 0.1 0see 0.1 0.2 0.4 1.2 0.7 0.9 0.8 0.7 0.1bark 3.1 0.3 0.2 1.9 1.8 2.1 1.8 0.7 2.1chase 2.7 2.8 1 2.1 3.1 2.2 1.1 0.9 0.8cat 2.1 1.1 0.9 2.3 1.9 3.9 0.5 2.8 4.6

w.Av. 1.73 0.95 2.11 1.94 2.44 1.14 1.13 1.95


Overview




4. Visualization



Case study: Data and set-up

Corpus

• Dutch newspaper materials from 1999 to 2005

• stratified for Netherlandic (500M) and Belgian Dutch(1.3G)

• automatically lemmatized, POS tagged and parsed withAlpino (van Noord (2006)).

Dutch synsets

• 218 synsets containing 476 nouns (Ruette et al. (2012))

• dependency-based type-level SVS (Pado & Lapata (2007))

• clustered with Clustering by Committee ( Pantel & Lin(2002))



Concept nouns in synsetInfringement inbreuk, overtreding

Genocide volkerenmoord, genocidePoll peiling, opiniepeiling, rondvraag

Marihuana cannabis, marihuanaCoup staatsgreep, coup

Meningitis hersenvliesontsteking, meningitisDemonstrator demonstrant, betoger

Airport vliegveld, luchthavenCollision aanrijding, botsing

Computer screen computerschem, beeldscherm, monitor

Table: Dutch synsets (sample)



Token model: second order contexts (Schutze (1998))

STEP 1: type-level SVS for context words

• 1 order context words: 573,127 words with frequency > 2

• 2 order context words: window of 4 left/right; 5430 wordsamong the 7000 most frequent (minus stoplist of 34high-frequent function words) AND that occurred at least 50times in both the Netherlandic and Belgian part of the corpus.

• weighting: positive PMI

STEP 2 token vectors

• sample: 100 Netherlandic and 100 Belgian newspaper issues

• window: 5 context words left and right of token

• token vector=(weighted) average of context word vectors



STEP 3 token by token similarity matrix

• similarity measure: cosine

• final output: similarity matrix for each of 218 synsets

• reflect how the different synonyms carve up the “semanticspace” of the concept among themselves


Overview




4. Visualization



Visualization

HighD to 2D

• token similarity matrix is high dimensional

• faithful rendering in 2D: Kruskal’s non-metricMultidimensional Scaling

• aim is not (yet) to find/impose latent structure

Integrated and interactive chart

• intergrate MDS plots with different types of meta-data

• let researcher choose which data to visualize in plot

• Motion Charts from Google Chart Tools

• also open source implementation with Python Image library

• Demo for Infringement , Computer Screen

https://perswww.kuleuven.be/~u0038536/googleVis/wSOCC/inbreukOvertreding.html

https://perswww.kuleuven.be/~u0038536/googleVis/wSOCC/beeldscherm.html


Traditional KWIC concordance


Overview




4. Visualization



Conclusion and future work

Double benefit of visualizing SVSs

• For CompLx: Making SVS accessible for evaluation by lexicalsemantic experts

• For Lexicology: Tool for exploring lexical semantics in largeamounts of corpus data

Desiderata (due to rather opportunistic use of GMC)

• larger stretches of text in bubbles

• applications to historic data (cf. Sagi et al. (2009))

• provide more structure in plots. (cf. Rohrdantz et al. (2011))

• show context features that make tokens similar

• allow input from users (e.g. additional coding)

• track feed-back from users (e.g. misplaced tokens)


For more information:http://wwwling.arts.kuleuven.be/qlvl

[email protected]

[email protected]


References I

Baroni, Marco, & Lenci, Alessandro. 2011. How we BLESSeddistributional semantic evaluation. Pages 1–10 of: Proceedingsof the GEMS 2011 Workshop on GEometrical Models of NaturalLanguage Semantics. Edinburgh, UK: Association forComputational Linguistics.

Firth, J. 1957. A synopsis of linguistic theory 1930-1955. In:Palmer, F R (ed), Selected papers of J.R. Firth. Longman.

Geeraerts, Dirk. 2010. Theories of Lexical Semantics. Oxford:Oxford University Press.

Pado, Sebastian, & Lapata, Mirella. 2007. Dependency-basedconstruction of semantic space models. ComputationalLinguistics, 33(2), 161–199.


References II

Pantel, Patrick, & Lin, Dekang. 2002. Document clustering withcommittees. Pages 199–206 of: Proceedings of the 25th annualinternational ACM SIGIR conference on Research anddevelopment in information retrieval. SIGIR ’02. New York, NY,USA: ACM.

Rohrdantz, Christian, Hautli, Annette, Mayer, Thomas, Butt,Miriam, Keim, Daniel A, & Plank, Frans. 2011. TowardsTracking Semantic Change by Visual Analytics. Pages 305–310of: Proceedings of the 49th Annual Meeting of the Associationfor Computational Linguistics: Human Language Technologies.Portland, Oregon, USA: Association for ComputationalLinguistics.


References III

Ruette, Tom, Geeraerts, Dirk, Peirsman, Yves, & Speelman, Dirk.2012. Semantic weighting mechanisms in scalable lexicalsociolectometry. In: Szmrecsanyi, Benedikt, & Walchli,Bernhard (eds), Aggregating dialectology and typology:linguistic variation in text and speech, within and acrosslanguages. Berlin: Mouton de Gruyter.

Sagi, Eyal, Kaufmann, Stefan, & Clark, Brady. 2009. SemanticDensity Analysis: Comparing Word Meaning across Time andPhonetic Space. Pages 104–111 of: Proceedings of theWorkshop on Geometrical Models of Natural LanguageSemantics. Athens, Greece: Association for ComputationalLinguistics.

Schutze, Hinrich. 1998. Automatic word sense discrimination.Computational Linguistics, 24(1), 97–124.


References IV

Turney, Peter D., & Pantel, Patrick. 2010. From Frequency toMeaning: Vector Space Models of Semantics. Journal ofArtificial Intelligence Research, 37(1), 141–188.

van Noord, Gertjan. 2006. At Last Parsing Is Now Operational.Pages 20–42 of: Verbum Ex Machina. Actes de la 13econference sur le traitement automatique des langues naturelles(TALN06). Leuven, Belgium: Presses universitaires de Louvain.

looking at word meaning - uni-konstanz.de · [email protected]...

Documents