60-538: information retrieval - university of...
TRANSCRIPT
![Page 1: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/1.jpg)
what is IRcourse schedulegrading scheme
60-538: Information Retrieval
September 7, 2017
1 / 48
![Page 2: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/2.jpg)
what is IRcourse schedulegrading scheme
Outline
1 what is IR
2 course schedule
3 grading scheme
2 / 48
![Page 3: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/3.jpg)
what is IRcourse schedulegrading scheme
Outline
1 what is IR
2 course schedule
3 grading scheme
3 / 48
![Page 4: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/4.jpg)
what is IRcourse schedulegrading scheme
IR not long time ago
4 / 48
![Page 5: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/5.jpg)
what is IRcourse schedulegrading scheme
5 / 48
![Page 6: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/6.jpg)
what is IRcourse schedulegrading scheme
now IR is mostly about search engines
there are many search engines ...
6 / 48
![Page 7: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/7.jpg)
what is IRcourse schedulegrading scheme
7 / 48
![Page 8: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/8.jpg)
what is IRcourse schedulegrading scheme
8 / 48
![Page 9: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/9.jpg)
what is IRcourse schedulegrading scheme
9 / 48
![Page 10: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/10.jpg)
what is IRcourse schedulegrading scheme
10 / 48
![Page 11: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/11.jpg)
what is IRcourse schedulegrading scheme
11 / 48
![Page 12: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/12.jpg)
what is IRcourse schedulegrading scheme
12 / 48
![Page 13: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/13.jpg)
what is IRcourse schedulegrading scheme
IR is more than web search
These days we frequently think first of web search, but there aremany other cases:
digital library search
E-mail search, Searching your desktop and laptop computers
Corporate knowledge bases, local business search, expertsearch
Legal information retrieval, patent search
news search
image and video search
(micro-)blog search
product search, federated search
social search, community Q&A, question-answering
recommender systems
opinion mining
13 / 48
![Page 14: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/14.jpg)
what is IRcourse schedulegrading scheme
definition of information retrieval
Information retrieval (IR) is finding material (usually documents) ofan unstructured nature (usually text) that satisfies an informationneed from within large collections (usually stored on computers).
–from IIR book.
Introduction to Information Retrieval, by C. Manning, P.Raghavan, and H. Schutze. Cambridge University Press
book website https://nlp.stanford.edu/IR-book/
14 / 48
![Page 15: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/15.jpg)
what is IRcourse schedulegrading scheme
definition of information retrieval
Information retrieval (IR) is finding material (usually documents) ofan unstructured nature (usually text) that satisfies an informationneed from within large collections (usually stored on computers).
–from IIR book.
Introduction to Information Retrieval, by C. Manning, P.Raghavan, and H. Schutze. Cambridge University Press
book website https://nlp.stanford.edu/IR-book/
14 / 48
![Page 16: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/16.jpg)
what is IRcourse schedulegrading scheme
definition of information retrieval
Information retrieval (IR) is finding material (usually documents) ofan unstructured nature (usually text) that satisfies an informationneed from within large collections (usually stored on computers).
–from IIR book.
Introduction to Information Retrieval, by C. Manning, P.Raghavan, and H. Schutze. Cambridge University Press
book website https://nlp.stanford.edu/IR-book/
14 / 48
![Page 17: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/17.jpg)
what is IRcourse schedulegrading scheme
definition of information retrieval
Information retrieval (IR) is finding material (usually documents) ofan unstructured nature (usually text) that satisfies an informationneed from within large collections (usually stored on computers).
–from IIR book.
Introduction to Information Retrieval, by C. Manning, P.Raghavan, and H. Schutze. Cambridge University Press
book website https://nlp.stanford.edu/IR-book/
14 / 48
![Page 18: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/18.jpg)
what is IRcourse schedulegrading scheme
definition of information retrieval
Information retrieval (IR) is finding material (usually documents) ofan unstructured nature (usually text) that satisfies an informationneed from within large collections (usually stored on computers).
–from IIR book.
Introduction to Information Retrieval, by C. Manning, P.Raghavan, and H. Schutze. Cambridge University Press
book website https://nlp.stanford.edu/IR-book/
14 / 48
![Page 19: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/19.jpg)
what is IRcourse schedulegrading scheme
definition of information retrieval
Information retrieval (IR) is finding material (usually documents) ofan unstructured nature (usually text) that satisfies an informationneed from within large collections (usually stored on computers).
–from IIR book.
Introduction to Information Retrieval, by C. Manning, P.Raghavan, and H. Schutze. Cambridge University Press
book website https://nlp.stanford.edu/IR-book/
14 / 48
![Page 20: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/20.jpg)
what is IRcourse schedulegrading scheme
definition of information retrieval
Information retrieval (IR) is finding material (usually documents) ofan unstructured nature (usually text) that satisfies an informationneed from within large collections (usually stored on computers).
–from IIR book.
Introduction to Information Retrieval, by C. Manning, P.Raghavan, and H. Schutze. Cambridge University Press
book website https://nlp.stanford.edu/IR-book/
14 / 48
![Page 21: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/21.jpg)
what is IRcourse schedulegrading scheme
Structured vs. unstructured data
in the 90’s. todayInformation retrieval is finding material of an unstructured naturethat satisfies an information need from within large collections
15 / 48
![Page 22: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/22.jpg)
what is IRcourse schedulegrading scheme
other definitions
Jaime Arguello
Information retrieval (IR) is the science and practice ofdesigning, developing, and evaluating systems that matchinformation seekers with the information they seek.
Gerard Salton, 1968:
Information retrieval is a field concerned with the structure,analysis, organization, storage, and retrieval ofinformation.
16 / 48
![Page 23: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/23.jpg)
what is IRcourse schedulegrading scheme
The search task
Given a query and a corpus, find relevant items
query: user’s expression of their information need
corpus: a repository of retrievable items
relevance: satisfaction of the user’s information need
17 / 48
![Page 24: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/24.jpg)
what is IRcourse schedulegrading scheme
Why is IR fascinating?
Information retrieval is an uncertain process
Query
users don’t know what they wantusers don’t know how to convey what they wantcomputers can’t elicit information like a librariancomputers can’t understand natural language text
Relevance
the search engine can only guess what is relevantthe search engine can only guess if a user is satisfiedover time, we can only guess how users adjust their short- andlong-term behavior
18 / 48
![Page 25: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/25.jpg)
what is IRcourse schedulegrading scheme
classic search model
19 / 48
![Page 26: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/26.jpg)
what is IRcourse schedulegrading scheme
A query is an impoverished description of the user’sinformation need
Highly ambiguous to anyone other than the user
Retrieval Model
A formal method that predicts the degree of relevance of adocument to a query
20 / 48
![Page 27: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/27.jpg)
what is IRcourse schedulegrading scheme
taxonomy of IR models
Document Property
text
links
multimedia
IR models
Boolean
vector
probalistic
Semistructured text
proximal nodes
xml based
web
page rank
hubs and authorities (HITs)
Multimedia
image retrieval
audio
video
Set theoretic
fuzzy
extended boolean
set-based
algebraic
generalized vector
LSI
NN
probablistic
BM25
language models
Bayersian networks
21 / 48
![Page 28: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/28.jpg)
what is IRcourse schedulegrading scheme
Boolean Retrieval Model
The user describes their information need using booleanconstraints (e.g., AND, OR, and AND NOT)
The burden is on the user to formulate a good boolean query
22 / 48
![Page 29: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/29.jpg)
what is IRcourse schedulegrading scheme
Example
Which plays of Shakespeare contain the words Brutus ANDCaesar but NOT Calpurnia?
One choice: use grep command in unix.
grep all of Shakespeare’s plays for Brutus and Caesar,strip out lines containing Calpurnia
Why is that not the answer?
Slow (for large corpora)NOT Calpurnia is non-trivialOther operations (e.g., find the word Romans nearcountrymen) not feasibleRanked retrieval (best documents to return)
so we need to index the text
23 / 48
![Page 30: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/30.jpg)
what is IRcourse schedulegrading scheme
what is an index
24 / 48
![Page 31: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/31.jpg)
what is IRcourse schedulegrading scheme
index construction process
25 / 48
![Page 32: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/32.jpg)
what is IRcourse schedulegrading scheme
Initial stages of text processing
Tokenization
Cut character sequence into word tokens
NormalizationMap text and query term to same form
You want U.S.A. and USA to match
StemmingWe may wish different forms of a root to match
authorize, authorization
Stop wordsWe may want to omit very common words (modern methodsmay not)
the, a, to, of
26 / 48
![Page 33: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/33.jpg)
what is IRcourse schedulegrading scheme
postings
Multiple term entriesin a single documentare merged.
Split into Dictionaryand Postings
Doc. frequencyinformation is added.
27 / 48
![Page 34: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/34.jpg)
what is IRcourse schedulegrading scheme
28 / 48
![Page 35: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/35.jpg)
what is IRcourse schedulegrading scheme
query processing
Consider processing the query:
Brutus AND Caesar
Locate Brutus in the Dictionary;
Retrieve its postings.
Locate Caesar in the Dictionary;
Retrieve its postings.
Merge the two postings (intersect the document sets):
brutus 1 2 4 11 31 45 173 174
caesar 1 2 4 5 6 16 57 132
29 / 48
![Page 36: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/36.jpg)
what is IRcourse schedulegrading scheme
Outline
1 what is IR
2 course schedule
3 grading scheme
30 / 48
![Page 37: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/37.jpg)
what is IRcourse schedulegrading scheme
tentative schedule
boolean model
text transformation
web crawling. Data cleaning (e.g. near-duplicate detection)
build a search engine using Lucene
vector space model
evaluation methods in information retrieval
link analysis and PageRank
document classification
document clustering
representation learning
31 / 48
![Page 38: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/38.jpg)
what is IRcourse schedulegrading scheme
Text Book
[IIR] Introduction to Information Retrieval, by C. Manning, P.Raghavan, and H. Schutze. Cambridge University Press, 2008.
32 / 48
![Page 39: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/39.jpg)
what is IRcourse schedulegrading scheme
Other reference books
SE Search Engines: Information Retrieval in Practice, by BruceCroft, Donald Metzler and Trevor Strohman.
MIR Modern Information Retrieval, by R. Baeza-Yates and B.Ribeiro-Neto. 2-nd edition 2010.
MMD Anand Rajaraman and Jeff Ullman, Mining of massivedatasets , 2013.
33 / 48
![Page 40: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/40.jpg)
what is IRcourse schedulegrading scheme
IIR 02: The term vocabulary and postings lists
Phrase queries: “Stanford University”
Proximity queries: Gates near Microsoft
We need an index that captures position information forphrase queries and proximity queries.
34 / 48
![Page 41: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/41.jpg)
what is IRcourse schedulegrading scheme
IIR 04: Index construction
masterassign
mapphase
reducephase
assign
parser
splits
parser
parser
inverter
postings
inverter
inverter
a-f
g-p
q-z
a-f g-p q-z
a-f g-p q-z
a-f
segmentfiles
g-p q-z
35 / 48
![Page 42: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/42.jpg)
what is IRcourse schedulegrading scheme
statistic properties of text
0 1 2 3 4 5 6 7
01
23
45
67
log10 rank
log1
0 cf
Zipf’s law, heaps’ law, power law.
the mechanism: Yule process, Preferential attachment
36 / 48
![Page 43: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/43.jpg)
what is IRcourse schedulegrading scheme
IIR 06: Scoring, term weighting and the vector spacemodel
Ranking search results
Boolean queries only give inclusion or exclusion of documents.For ranked retrieval, we measure the proximity between the query andeach document.One formalism for doing this: the vector space model
Key challenge in ranked retrieval: evidence accumulation for a term ina document
1 vs. 0 occurence of a query term in the document3 vs. 2 occurences of a query term in the documentUsually: more is betterBut by how much?Need a scoring function that translates frequency into score or weight
37 / 48
![Page 44: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/44.jpg)
what is IRcourse schedulegrading scheme
Language models
assign a probability to a sequence of m words by means of aprobability distribution.
How to compute this joint probability:
P(its,water , is, so, transparent, that) (1)
P(w1w2 . . .wn) = ΠP(wi )? (2)
38 / 48
![Page 45: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/45.jpg)
what is IRcourse schedulegrading scheme
Text classification & Naive Bayes
Text classification = assigning documents automatically topredefined classes
Examples:
CS vs. Non-CS paperspositive/negative reviewsSpams
Naive Bayes (Multinomial and Bernoulli model), Supportvector machine, feature selection, representation learning,neural networks.
39 / 48
![Page 46: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/46.jpg)
what is IRcourse schedulegrading scheme
clustering
Flat clustering
Hierarchical agglomerative clustering (HAC)
Single-link and complete-link clustering
Centroid and group-average agglomerative clustering (GAAC)
Bisecting K-means
How to label clusters automatically
40 / 48
![Page 47: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/47.jpg)
what is IRcourse schedulegrading scheme
HAC
41 / 48
![Page 48: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/48.jpg)
what is IRcourse schedulegrading scheme
Latent Semantic Indexing
how to find semantically related documents?matrix decompositionSVD
42 / 48
![Page 49: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/49.jpg)
what is IRcourse schedulegrading scheme
Crawling
43 / 48
![Page 50: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/50.jpg)
what is IRcourse schedulegrading scheme
Link analysis / PageRank
which web page is more important?
who are in a community?
PageRank algorithm
graph analysis and mining. Modularity maximizationalgorithms.
44 / 48
![Page 51: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/51.jpg)
what is IRcourse schedulegrading scheme
Outline
1 what is IR
2 course schedule
3 grading scheme
45 / 48
![Page 52: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/52.jpg)
what is IRcourse schedulegrading scheme
marking scheme
exam 40%
project 50%
class participation 10%
46 / 48
![Page 53: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/53.jpg)
what is IRcourse schedulegrading scheme
project
indexing and searching engine
CS papers
enhance the search engine by adding one or more features,such as:
semantic searchclassificationclusteringrankingpersonalizationrecommendation...
47 / 48
![Page 54: 60-538: Information Retrieval - University of Windsorjlu.myweb.cs.uwindsor.ca/538/538intro2017.pdf · 2017-09-07 · opinion mining 13 / 48. what is IR course schedule grading scheme](https://reader034.vdocument.in/reader034/viewer/2022042106/5e849d73f12a9a37a81e7d8c/html5/thumbnails/54.jpg)
what is IRcourse schedulegrading scheme
open source search engines
Lemur
C++
used by IR researchers
compare advanced IR techniques
Lucene
Java-based
relatively simple IR techniques
Galago
Java-based
used by the book [SE] Search Engines: Information Retrievalin Practice, by Bruce Croft et al.
48 / 48