Automatic Discovery of Shared Automatic Discovery of Shared Interest Interest
Minimum Spanning Trees Displaying Minimum Spanning Trees Displaying Semantic SimilaritySemantic Similarity
Włodzisław Duch & Co
Department of Informatics, Nicolaus Copernicus University, Torun, Poland, & School of Computer Engineering, Nanyang
Technological University, Singapore
Google: Duch
The Vision Vannevar Bush imagined in 1945 linked text/film information, a
kind of Wikipedia, his Memex was first hypertext systems.
Ted Nelson in “Computer Lib and Dream Machines” (1974) extended this vision to all kinds of information integrated in project Xanadu, a project founded in 1960. In essence:unbreakable two-way links, connected to origin of info, facilitating incremental publishing, deep version management & comparison.
WWW is not yet Xanadu, no links to origin of information, little maintenance, searches are frequently tedious, and linking information about any given subject is done in manual way.
Xanadu vision is not sufficient; all knowledge should be organized in form of ideas supported by evidence, with links between related pieces of information automatically created (QED project).
If only computes could analyze and present it in coherent way, linking major ideas to papers, data, software, experiments ...
The Problem Finding all people who share similar interests in large
organizations or worldwide is difficult (NTU experience). Find who is related to me and in which way?
Each individual may have many different interests so the search process should be topic-oriented, not people-oriented.
The process should be automatic – use info on people’s homepages and their lists of publications.
Visualize relations using graphs with individuals as nodes and different type of relations as edges.
The structure of the graphical representation depends strongly on the selection of key entities of the nodes – text should be projected first on domain ontology.
Steps WWW spiders used to collect documents from some domain
(NTU home pages have been used for tests).
Convert html documents to text, clean using stop-words, apply stemming etc.
Final filtering & dimensionality reduction to obtain vector representation of the term-document matrix.
Cluster info in some way (try Clusty, Vivisimo or Carrot2).
Visualize related nodes that represent individual homepages, link them by estimates of similarity of shared interest: see Websom and its applications in digital lib, astro VizieR etc.
This goes beyond visualization of Google link analysis or “the brain interface” use in Britannica BrainStormer.
Implementation and Design
Document-word matrix
Document1: word1 word2 word3. word4 word3 word5.
Document2: word1 word3 word5. word1 word3 word6.
The matrix: documents x word frequencies
1 1 2 1 1 0
2 0 2 0 1 1
F
Document 1
Document 2
W1 W2 W3 W4 W5 W6
First shot: methods used
Inverse document frequency and term weighting.
Simple selection of relevant terms or
Latent Semantic Analysis (LSA) for dimensionality reduction – standard method in info retrieval.
Minimum Spanning Trees for visual representation.
TouchGraph XML visualization of MST trees.
Data Preparation Normalize columns of F dividing by highest word
frequencies:
Among n documents, term j occurs dj times; inverse document frequency idfj measures uniqueness of term j:
2log / 1, 0j j jidf n d d
tf x idf term weights:
ij ij jw tf idf
/ maxij ij iji
tf f f
Simple selection
Take wij weights above certain threshold, binarize and remove zero rows:
1
22
1 1
ik jkk
ij
ik jkk k
h hs
h h
Calculate similarity using cosine measure (takes care of the vector length normalization):
ij ij jh w
Similarity using cosine measure
1V 3 1 2 0 18 0 0
2V 2 1 5 4 4 0 0
3V 6 0 4 2 0 2 1
Using the same vectors, V1 ,V2 ,V3
Similarity of vector 1 and vector 2 is S12=0.615, and S13=0.1811. Document 1 and Document 2 are more likely to be related.
For visualization a threshold value (ex. Sij > 0.3) which will determine which links to show is used.
1
22
1 1
ik jkk
ij
ik jkk k
h hs
h h
Dimensionality reduction
Latent Semantic Analysis (LSA): use Singular Value Decomposition on weight matrix W
i j
iji j
s
W W
W W
with U = eigenvectors of WWT and V of WTW.Remove small eigenvalues from , recreate reduced W and calculate similarity:
TW UΛV
Kruskal’s MST Algorithm and Top - Down Kruskal’s MST Algorithm and Top - Down ClusterizationClusterizationMinimum spanning tree = weighted graph with minimum total cost, created by a simple greedy algorithm.
Cluster identification during MST construction.
Some experiments:
Reuters-215785 datasets, with 5 categories and 1 – 176 elements per category, 600 documents: can we see categories?
124 Personal Web Pages of the School of Electrical and Electronic Engineering (EEE) of the Nanyang Technological University (NTU) in Singapore.
5 department names may be used as categories: control, microelectronics, information, circuit, power, with 14 – 41 documents per category.Can we discover department structure?
Reuters results
For 600 documents W rank in SVD is Wrank= 595
Method topics clusters accuracy
No dim red. 41 129 78.2%LSA dim red. 0.8 (476) 41 124 76.2%LSA dim red. 0.6 (357) 41 127 75.2%Simple Selection 41 130 78.5%
0.8 means 0.8*Wrank eigenvectors retained
Results for EEE NTU Web pages
Method topics clusters accuracy
No dim red. 10 142 84.7%LSA dim red. 0.8 (467) 10 129 84.7%LSA dim red. 0.6 (350) 10 137 82.8%Simple Selection 10 145 85.5%
Examples
Live demo http://www.neuron.m4u.pl/search
EEE full EEE Selected EEE Selected small clusters
Limitations
Keywords have been derived from what we find on web pages only, too many, too sparse matrices.
Synonymous concepts should be treated as a single feature, producing larger frequency counts.
People working on “architecture in mechanical design” who are interested in “computer art” are associated with someone in “computer architecture”.
Web pages contain many irrelevant information.
Abbreviations of all sorts are used.
No topics, therefore only a single category used.
Adding ontologies
Select relevant terms using engineering ontologies (from keywords used in library classification)
Add medical concepts (ULMS) and use MetaMap to discover these concepts in text.
Processing: Term weighting, stemming etc
Simple selection of relevant terms.
TouchGraph XML visualization
EEE: Simple Word-Doc Vector Space
EEE: Transformed Concept Vector Space
Med: Simple Word-Doc Vector Space
Med: Meta-Map Concept Vector Space
Med: after Metamap transformation
Results for Summary Discharges
New experiments on medical texts.Short (~ half page) hospital summary
discharges.10 classes and 10 documents per class = main
disease treated.
Plain Doc-Word matrix ≈ 23% Stop-List, TW-IDF, S.S. ≈ 64% Metamap Transformation ≈ 93%
Summary In real application knowledge-based approach is needed
to select relevant concepts and to parse web pages but problems with acronyms, abbreviations, synonyms etc should be solved.
Other visualization and clusterization methods should be explored.
People have many interests and thus may belong to several topic groups – topics are related to concepts that should be high in ontology, but have no simple description.
Could be a very useful tool to create new shared interest groups for social networks in the Internet.
Could point out to potential collaborators or interesting research from individual point of view.
Similar attempts
Flink is presentation of the scientific work and social connectivity of Semantic Web reseachers, displaying homepages of experts who have contributed to the International Semantic Web Conference (ISWC) series. http://flink.semanticweb.org
Kartoo is a metasearch engine that displays topic maps: http://www.kartoo.com
Related work in my group
Neural basis of language: creation of network of concepts instead of vector models.
Medical text analysis using UMLS ontologies.
Instead of clustering formulate minimum number of questions to define more precise search.
Creativity – inventing new names.
Words in the brainWords in the brainWords in the brainWords in the brainThe cell assembly model of language has strong experimental support; F. Pulvermuller (2003) The Neuroscience of Language. On Brain Circuits of Words and Serial Order. Cambridge University Press.
Acoustic signal => phonemes => words => semantic concepts.Semantic activations are seen 90 ms after phonological in N200 ERPs.
Phonological density of words = # words that sound similar to a given word, that is create similar activations in phonological areas.
Semantic density of words = # words that have similar meaning, or similar extended activation network.
Perception/action networks, results from ERP & fMRI.
Words: simple modelWords: simple modelWords: simple modelWords: simple modelGoals: • make the simplest testable model of creativity; • create interesting novel words that capture some features of products;• understand new words that cannot be found in the dictionary.
Model inspired by the putative brain processes when new words are being invented. Start from keywords priming auditory cortex.
Phonemes (allophones) are resonances, ordered activation of phonemes will activate both known words as well as their combinations; context + inhibition in the winner-takes-most leaves one or a few words.
Creativity = imagination (fluctuations) + filtering (competition)
Imagination: many chains of phonemes activate in parallel both words and non-words reps, depending on the strength of synaptic connections. Filtering: associations, emotions, phonological/semantic density.
Beyond ontologiesBeyond ontologiesBeyond ontologiesBeyond ontologies
Neurocognitive approach to language understanding: use recognition, semantic and episodic memory models, create graphs of consistent concepts for interpretation, use spreading activation and inhibition to simulate effect of semantic priming, annotate and disambiguate text.
For medical texts ULMS has >2M concepts, 15M relations … See: Unambiguous Concept Mapping in a Medical Domain, Thursday 11:45 (Matykiewicz, Duch, Pestian).
Humanized interface
Store
Applications, eg. 20 questions game
Query
Semantic memory
Parser
Part of speech tagger& phrase extractor
On line dictionaries
Manual
verification
DREAM modules DREAM modules
Natural input
modules Cognitive functions
Affectivefunctions
Web/text/databases interface
Behavior control
Control of devices
Talking head
Text to speechNLP
functions
Specializedagents
DREAM project is focused on perception (visual, auditory, text inputs), cognitive functions (reasoning based on perceptions), natural language communication in well defined contexts, real time control of the simulated/physical head.
Thank Thank youyoufor for
lending lending your your ears ears
......
Google: Duch => Papers