random indexing: on space and meaning simon belak
TRANSCRIPT
![Page 1: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/1.jpg)
Random indexing:On space and meaning
Simon Belak
![Page 2: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/2.jpg)
Order of the day
• Meaning– Philosophy– Neuroscience– Computer science
• Space– Words as points in space– On dimensionality
• Random indexing
![Page 3: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/3.jpg)
What’s the meaning of meaning?
![Page 4: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/4.jpg)
Philosophers say:
“Meaning just is use.”– Wittgenstein
![Page 5: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/5.jpg)
Neuroscientists say:
• Episodic memory semantic memory(concrete event abstract concept)
• Hebbian process
![Page 6: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/6.jpg)
Computer scientists say:
LSA semantic networks
HALTLC
SAMACT-R
ontology
![Page 7: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/7.jpg)
Projecting meaning into space
![Page 8: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/8.jpg)
Adjacent words closely related
![Page 9: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/9.jpg)
Movement
• Co-occurrences
• Hebbian process– Self-organisation– Clustering
• Evolution of language– Coach (Kocs carriage train car)
![Page 10: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/10.jpg)
Problem: homonymsTable1. a. An article of furniture supported by one or more vertical legs and having a flat horizontal
surface.b. The objects laid out for a meal on this article of furniture.
2. The food and drink served at meals; fare: kept an excellent table.3. The company of people assembled around a table, as for a meal.4 A plateau or tableland.5. a. A flat facet cut across the top of a precious stone.
b. A stone or gem cut in this fashion.6. Music
a. The front part of the body of a stringed instrument.b. The sounding board of a harp.
7. Architecture a. A raised or sunken rectangular panel on a wall.b. A raised horizontal surface or continuous band on an exterior wall; a stringcourse.
8. A part of the human palm framed by four lines, analyzed in palmistry.9. An orderly arrangement of data, especially one in which the data are arranged in columns and rows
in an essentially rectangular form.10. An abbreviated list, as of contents; a synopsis.11. An engraved slab or tablet bearing an inscription or a device.12. Anatomy The inner or outer flat layer of bones of the skull separated by the dipole.
![Page 11: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/11.jpg)
Solution: high dimensionality
• One dimension per word • Table extends into food, furniture, music,... dimensions
![Page 12: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/12.jpg)
Problem: synonyms
amazing, stupefying, staggering, awesome, awful, awe-inspiring, awing, astonishing, astounding
![Page 13: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/13.jpg)
Solution: latent meaning
• Reduced dimensionality
• Closely related words fold into one
• “Higher-order” meaning
![Page 14: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/14.jpg)
Random indexing
![Page 15: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/15.jpg)
The idea
• Word is the sum of it’s contexts
• Context is the sum of it’s words
• Grounding?
![Page 16: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/16.jpg)
The algorithm
1) Take a context of words
2) Generate a context index vector
3) Add index to all the word vectors
4) Go to 1)
Episodic memory (2) + Hebbian process (3)
![Page 17: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/17.jpg)
Dimensionality reduction
• Sparse high-dimensional ternary index
(a small number of randomly distributed +1s and -1s)
• Nearly orthogonal– Distances approximately preserved
![Page 18: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/18.jpg)
The good
• Fast, scalable
• Trivially parallelised– Per word– Addition is associative, commutative
• Stable– Words are independent– Integer arithmetics
• Incremental
![Page 19: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/19.jpg)
The bad
• Memory hungry– Caching (Zipf’s law)
![Page 20: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/20.jpg)
Uses
• Comparing words to words– Query expnasion
• Comparing documents to documents – Clustering– Search– Recomendations
• Comparing documents to words– Keyword extraction
![Page 21: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/21.jpg)
Key points
![Page 22: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/22.jpg)
1. Meaning is use
![Page 23: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/23.jpg)
2. Words in space
![Page 24: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/24.jpg)
3. Many meanings, many dimensions
![Page 25: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/25.jpg)
4. Random indexing
• Cognitive rationale
• Simple
• Fast, scalable
![Page 26: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/26.jpg)
Key points
1. Meaning is use
2. Words in space
3. Many meanings, many dimensions
4. Random indexing– Cognitive rationale– Simple– Fast, scalable
![Page 27: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/27.jpg)
Questions?
![Page 28: Random indexing: On space and meaning Simon Belak](https://reader036.vdocument.in/reader036/viewer/2022062407/56649cf75503460f949c7fc5/html5/thumbnails/28.jpg)
References• http://www.sics.se/~mange/papers/KarlgrenSahlgren2001.pdf• http://www.kfs.org/~jonathan/witt/tlph.html• http://www.mtsu.edu/~sschmidt/Cognitive/semantic/semantic.html• http://memory.syr.edu/marc/papers/HowaAddiJingKaha-LSAChap-doc.pdf• http://memory.psych.upenn.edu/research/research_episodic_memory.php• http://www.rni.org/kanerva/cogsci2k-poster.txt• http://www.sics.se/~mange/papers/RI_intro.pdf
• http://code.google.com/p/cl-random-indexing