introduction to information retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf ·...
TRANSCRIPT
![Page 1: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/1.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Introduction to Information Retrievalhttp://informationretrieval.org
IIR 1: Boolean Retrieval
Hinrich Schutze
Center for Information and Language Processing, University of Munich
2014-04-09
Schutze: Boolean Retrieval 1 / 60
![Page 2: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/2.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Take-away
Schutze: Boolean Retrieval 2 / 60
![Page 3: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/3.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Take-away
Boolean Retrieval: Design and data structures of a simpleinformation retrieval system
Schutze: Boolean Retrieval 2 / 60
![Page 4: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/4.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Take-away
Boolean Retrieval: Design and data structures of a simpleinformation retrieval system
What topics will be covered in this class?
Schutze: Boolean Retrieval 2 / 60
![Page 5: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/5.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Outline
1 Introduction
2 Inverted index
3 Processing Boolean queries
4 Query optimization
5 Course overview
Schutze: Boolean Retrieval 3 / 60
![Page 6: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/6.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Definition of information retrieval
Information retrieval (IR) is finding material (usually documents) ofan unstructured nature (usually text) that satisfies an informationneed from within large collections (usually stored on computers).
Schutze: Boolean Retrieval 4 / 60
![Page 7: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/7.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Definition of information retrieval
Information retrieval (IR) is finding material (usually documents) ofan unstructured nature (usually text) that satisfies an informationneed from within large collections (usually stored on computers).
Schutze: Boolean Retrieval 4 / 60
![Page 8: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/8.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Definition of information retrieval
Information retrieval (IR) is finding material (usually documents) ofan unstructured nature (usually text) that satisfies an informationneed from within large collections (usually stored on computers).
Schutze: Boolean Retrieval 4 / 60
![Page 9: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/9.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Definition of information retrieval
Information retrieval (IR) is finding material (usually documents) ofan unstructured nature (usually text) that satisfies an informationneed from within large collections (usually stored on computers).
Schutze: Boolean Retrieval 4 / 60
![Page 10: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/10.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Definition of information retrieval
Information retrieval (IR) is finding material (usually documents) ofan unstructured nature (usually text) that satisfies an informationneed from within large collections (usually stored on computers).
Schutze: Boolean Retrieval 4 / 60
![Page 11: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/11.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Definition of information retrieval
Information retrieval (IR) is finding material (usually documents) ofan unstructured nature (usually text) that satisfies an informationneed from within large collections (usually stored on computers).
Schutze: Boolean Retrieval 4 / 60
![Page 12: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/12.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Definition of information retrieval
Information retrieval (IR) is finding material (usually documents) ofan unstructured nature (usually text) that satisfies an informationneed from within large collections (usually stored on computers).
Schutze: Boolean Retrieval 4 / 60
![Page 13: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/13.jpg)
![Page 14: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/14.jpg)
![Page 15: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/15.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Boolean retrieval
The Boolean model is arguably the simplest model to base aninformation retrieval system on.
Schutze: Boolean Retrieval 7 / 60
![Page 16: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/16.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Boolean retrieval
The Boolean model is arguably the simplest model to base aninformation retrieval system on.
Queries are Boolean expressions, e.g., Caesar and Brutus
Schutze: Boolean Retrieval 7 / 60
![Page 17: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/17.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Boolean retrieval
The Boolean model is arguably the simplest model to base aninformation retrieval system on.
Queries are Boolean expressions, e.g., Caesar and Brutus
The seach engine returns all documents that satisfy theBoolean expression.
Schutze: Boolean Retrieval 7 / 60
![Page 18: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/18.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Boolean retrieval
The Boolean model is arguably the simplest model to base aninformation retrieval system on.
Queries are Boolean expressions, e.g., Caesar and Brutus
The seach engine returns all documents that satisfy theBoolean expression.
Does Google use the Boolean model?
Schutze: Boolean Retrieval 7 / 60
![Page 19: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/19.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Does Google use the Boolean model?
Schutze: Boolean Retrieval 8 / 60
![Page 20: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/20.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Does Google use the Boolean model?
On Google, the default interpretation of a query [w1 w2
. . .wn] is w1 AND w2 AND . . . AND wn
Schutze: Boolean Retrieval 8 / 60
![Page 21: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/21.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Does Google use the Boolean model?
On Google, the default interpretation of a query [w1 w2
. . .wn] is w1 AND w2 AND . . . AND wn
Cases where you get hits that do not contain one of the wi :
Schutze: Boolean Retrieval 8 / 60
![Page 22: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/22.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Does Google use the Boolean model?
On Google, the default interpretation of a query [w1 w2
. . .wn] is w1 AND w2 AND . . . AND wn
Cases where you get hits that do not contain one of the wi :
anchor text
Schutze: Boolean Retrieval 8 / 60
![Page 23: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/23.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Does Google use the Boolean model?
On Google, the default interpretation of a query [w1 w2
. . .wn] is w1 AND w2 AND . . . AND wn
Cases where you get hits that do not contain one of the wi :
anchor textpage contains variant of wi (morphology, spelling correction,synonym)
Schutze: Boolean Retrieval 8 / 60
![Page 24: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/24.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Does Google use the Boolean model?
On Google, the default interpretation of a query [w1 w2
. . .wn] is w1 AND w2 AND . . . AND wn
Cases where you get hits that do not contain one of the wi :
anchor textpage contains variant of wi (morphology, spelling correction,synonym)long queries (n large)
Schutze: Boolean Retrieval 8 / 60
![Page 25: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/25.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Does Google use the Boolean model?
On Google, the default interpretation of a query [w1 w2
. . .wn] is w1 AND w2 AND . . . AND wn
Cases where you get hits that do not contain one of the wi :
anchor textpage contains variant of wi (morphology, spelling correction,synonym)long queries (n large)boolean expression generates very few hits
Schutze: Boolean Retrieval 8 / 60
![Page 26: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/26.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Does Google use the Boolean model?
On Google, the default interpretation of a query [w1 w2
. . .wn] is w1 AND w2 AND . . . AND wn
Cases where you get hits that do not contain one of the wi :
anchor textpage contains variant of wi (morphology, spelling correction,synonym)long queries (n large)boolean expression generates very few hits
Simple Boolean vs. Ranking of result set
Schutze: Boolean Retrieval 8 / 60
![Page 27: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/27.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Does Google use the Boolean model?
On Google, the default interpretation of a query [w1 w2
. . .wn] is w1 AND w2 AND . . . AND wn
Cases where you get hits that do not contain one of the wi :
anchor textpage contains variant of wi (morphology, spelling correction,synonym)long queries (n large)boolean expression generates very few hits
Simple Boolean vs. Ranking of result set
Simple Boolean retrieval returns matching documents in noparticular order.
Schutze: Boolean Retrieval 8 / 60
![Page 28: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/28.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Does Google use the Boolean model?
On Google, the default interpretation of a query [w1 w2
. . .wn] is w1 AND w2 AND . . . AND wn
Cases where you get hits that do not contain one of the wi :
anchor textpage contains variant of wi (morphology, spelling correction,synonym)long queries (n large)boolean expression generates very few hits
Simple Boolean vs. Ranking of result set
Simple Boolean retrieval returns matching documents in noparticular order.Google (and most well designed Boolean engines) rank theresult set – they rank good hits (according to some estimatorof relevance) higher than bad hits.
Schutze: Boolean Retrieval 8 / 60
![Page 29: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/29.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Outline
1 Introduction
2 Inverted index
3 Processing Boolean queries
4 Query optimization
5 Course overview
Schutze: Boolean Retrieval 9 / 60
![Page 30: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/30.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Unstructured data in 1650: Shakespeare
Schutze: Boolean Retrieval 10 / 60
![Page 31: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/31.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Unstructured data in 1650
Which plays of Shakespeare contain the words Brutus and
Caesar, but not Calpurnia?
Schutze: Boolean Retrieval 11 / 60
![Page 32: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/32.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Unstructured data in 1650
Which plays of Shakespeare contain the words Brutus and
Caesar, but not Calpurnia?
One could grep all of Shakespeare’s plays for Brutus andCaesar, then strip out lines containing Calpurnia.
Schutze: Boolean Retrieval 11 / 60
![Page 33: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/33.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Unstructured data in 1650
Which plays of Shakespeare contain the words Brutus and
Caesar, but not Calpurnia?
One could grep all of Shakespeare’s plays for Brutus andCaesar, then strip out lines containing Calpurnia.
Why is grep not the solution?
Schutze: Boolean Retrieval 11 / 60
![Page 34: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/34.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Unstructured data in 1650
Which plays of Shakespeare contain the words Brutus and
Caesar, but not Calpurnia?
One could grep all of Shakespeare’s plays for Brutus andCaesar, then strip out lines containing Calpurnia.
Why is grep not the solution?
Slow (for large collections)
Schutze: Boolean Retrieval 11 / 60
![Page 35: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/35.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Unstructured data in 1650
Which plays of Shakespeare contain the words Brutus and
Caesar, but not Calpurnia?
One could grep all of Shakespeare’s plays for Brutus andCaesar, then strip out lines containing Calpurnia.
Why is grep not the solution?
Slow (for large collections)grep is line-oriented, IR is document-oriented
Schutze: Boolean Retrieval 11 / 60
![Page 36: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/36.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Unstructured data in 1650
Which plays of Shakespeare contain the words Brutus and
Caesar, but not Calpurnia?
One could grep all of Shakespeare’s plays for Brutus andCaesar, then strip out lines containing Calpurnia.
Why is grep not the solution?
Slow (for large collections)grep is line-oriented, IR is document-oriented“not Calpurnia” is non-trivial
Schutze: Boolean Retrieval 11 / 60
![Page 37: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/37.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Unstructured data in 1650
Which plays of Shakespeare contain the words Brutus and
Caesar, but not Calpurnia?
One could grep all of Shakespeare’s plays for Brutus andCaesar, then strip out lines containing Calpurnia.
Why is grep not the solution?
Slow (for large collections)grep is line-oriented, IR is document-oriented“not Calpurnia” is non-trivialOther operations (e.g., find the word Romans nearcountryman) not feasible
Schutze: Boolean Retrieval 11 / 60
![Page 38: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/38.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Term-document incidence matrix
Anthony Julius The Hamlet Othello Macbeth . . .and Caesar Tempest
CleopatraAnthony 1 1 0 0 0 1Brutus 1 1 0 1 0 0Caesar 1 1 0 1 1 1Calpurnia 0 1 0 0 0 0Cleopatra 1 0 0 0 0 0mercy 1 0 1 1 1 1worser 1 0 1 1 1 0. . .Entry is 1 if term occurs. Example: Calpurnia occurs in Julius Caesar.Entry is 0 if term doesn’t occur. Example: Calpurnia doesn’t occur in The
tempest.
Schutze: Boolean Retrieval 12 / 60
![Page 39: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/39.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Term-document incidence matrix
Anthony Julius The Hamlet Othello Macbeth . . .and Caesar Tempest
CleopatraAnthony 1 1 0 0 0 1Brutus 1 1 0 1 0 0Caesar 1 1 0 1 1 1Calpurnia 0 1 0 0 0 0Cleopatra 1 0 0 0 0 0mercy 1 0 1 1 1 1worser 1 0 1 1 1 0. . .Entry is 1 if term occurs. Example: Calpurnia occurs in Julius Caesar.Entry is 0 if term doesn’t occur. Example: Calpurnia doesn’t occur in The
tempest.
Schutze: Boolean Retrieval 12 / 60
![Page 40: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/40.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Term-document incidence matrix
Anthony Julius The Hamlet Othello Macbeth . . .and Caesar Tempest
CleopatraAnthony 1 1 0 0 0 1Brutus 1 1 0 1 0 0Caesar 1 1 0 1 1 1Calpurnia 0 1 0 0 0 0Cleopatra 1 0 0 0 0 0mercy 1 0 1 1 1 1worser 1 0 1 1 1 0. . .Entry is 1 if term occurs. Example: Calpurnia occurs in Julius Caesar.Entry is 0 if term doesn’t occur. Example: Calpurnia doesn’t occur in The
tempest.
Schutze: Boolean Retrieval 12 / 60
![Page 41: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/41.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Incidence vectors
So we have a 0/1 vector for each term.
To answer the query Brutus and Caesar and not
Calpurnia:
Schutze: Boolean Retrieval 13 / 60
![Page 42: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/42.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Incidence vectors
So we have a 0/1 vector for each term.
To answer the query Brutus and Caesar and not
Calpurnia:
Take the vectors for Brutus, Caesar, and Calpurnia
Complement the vector of Calpurnia
Do a (bitwise) and on the three vectors110100 and 110111 and 101111 = 100100
Schutze: Boolean Retrieval 13 / 60
![Page 43: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/43.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
0/1 vectors and result of bitwise operations
Anthony Julius The Hamlet Othello Macbeth . . .and Caesar Tempest
CleopatraAnthony 1 1 0 0 0 1Brutus 1 1 0 1 0 0Caesar 1 1 0 1 1 1Calpurnia 0 1 0 0 0 0Cleopatra 1 0 0 0 0 0mercy 1 0 1 1 1 1worser 1 0 1 1 1 0. . .
result: 1 0 0 1 0 0
Schutze: Boolean Retrieval 14 / 60
![Page 44: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/44.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Answers to query
Anthony and Cleopatra, Act III, Scene ii
Agrippa [Aside to Domitius Enobarbus]: Why, Enobarbus,When Antony found Julius Caesar dead,He cried almost to roaring; and he weptWhen at Philippi he found Brutus slain.
Hamlet, Act III, Scene iiLord Polonius: I did enact Julius Caesar: I was killed i’ the
Capitol; Brutus killed me.
Schutze: Boolean Retrieval 15 / 60
![Page 45: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/45.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Bigger collections
Consider N = 106 documents, each with about 1000 tokens
Schutze: Boolean Retrieval 16 / 60
![Page 46: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/46.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Bigger collections
Consider N = 106 documents, each with about 1000 tokens
⇒ total of 109 tokens
Schutze: Boolean Retrieval 16 / 60
![Page 47: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/47.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Bigger collections
Consider N = 106 documents, each with about 1000 tokens
⇒ total of 109 tokens
On average 6 bytes per token, including spaces andpunctuation ⇒ size of document collection is about 6 · 109 =6 GB
Schutze: Boolean Retrieval 16 / 60
![Page 48: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/48.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Bigger collections
Consider N = 106 documents, each with about 1000 tokens
⇒ total of 109 tokens
On average 6 bytes per token, including spaces andpunctuation ⇒ size of document collection is about 6 · 109 =6 GB
Assume there are M = 500,000 distinct terms in the collection
Schutze: Boolean Retrieval 16 / 60
![Page 49: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/49.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Bigger collections
Consider N = 106 documents, each with about 1000 tokens
⇒ total of 109 tokens
On average 6 bytes per token, including spaces andpunctuation ⇒ size of document collection is about 6 · 109 =6 GB
Assume there are M = 500,000 distinct terms in the collection
(Notice that we are making a term/token distinction.)
Schutze: Boolean Retrieval 16 / 60
![Page 50: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/50.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Can’t build the incidence matrix
M = 500,000× 106 = half a trillion 0s and 1s.
Schutze: Boolean Retrieval 17 / 60
![Page 51: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/51.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Can’t build the incidence matrix
M = 500,000× 106 = half a trillion 0s and 1s.
But the matrix has no more than one billion 1s.
Schutze: Boolean Retrieval 17 / 60
![Page 52: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/52.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Can’t build the incidence matrix
M = 500,000× 106 = half a trillion 0s and 1s.
But the matrix has no more than one billion 1s.
Matrix is extremely sparse.
Schutze: Boolean Retrieval 17 / 60
![Page 53: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/53.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Can’t build the incidence matrix
M = 500,000× 106 = half a trillion 0s and 1s.
But the matrix has no more than one billion 1s.
Matrix is extremely sparse.
What is a better representations?
Schutze: Boolean Retrieval 17 / 60
![Page 54: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/54.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Can’t build the incidence matrix
M = 500,000× 106 = half a trillion 0s and 1s.
But the matrix has no more than one billion 1s.
Matrix is extremely sparse.
What is a better representations?
We only record the 1s.
Schutze: Boolean Retrieval 17 / 60
![Page 55: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/55.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Inverted Index
For each term t, we store a list of all documents that contain t.
Brutus −→ 1 2 4 11 31 45 173 174
Caesar −→ 1 2 4 5 6 16 57 132 . . .
Calpurnia −→ 2 31 54 101
...
︸ ︷︷ ︸ ︸ ︷︷ ︸
dictionary postings
Schutze: Boolean Retrieval 18 / 60
![Page 56: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/56.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Inverted Index
For each term t, we store a list of all documents that contain t.
Brutus −→ 1 2 4 11 31 45 173 174
Caesar −→ 1 2 4 5 6 16 57 132 . . .
Calpurnia −→ 2 31 54 101
...
︸ ︷︷ ︸ ︸ ︷︷ ︸
dictionary postings
Schutze: Boolean Retrieval 18 / 60
![Page 57: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/57.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Inverted Index
For each term t, we store a list of all documents that contain t.
Brutus −→ 1 2 4 11 31 45 173 174
Caesar −→ 1 2 4 5 6 16 57 132 . . .
Calpurnia −→ 2 31 54 101
...
︸ ︷︷ ︸ ︸ ︷︷ ︸
dictionary postings
Schutze: Boolean Retrieval 18 / 60
![Page 58: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/58.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Inverted index construction
1 Collect the documents to be indexed:
Friends, Romans, countrymen. So let it be with Caesar . . .
2 Tokenize the text, turning each document into a list of tokens:
Friends Romans countrymen So . . .
3 Do linguistic preprocessing, producing a list of normalized
tokens, which are the indexing terms: friend roman
countryman so . . .
4 Index the documents that each term occurs in by creating aninverted index, consisting of a dictionary and postings.
Schutze: Boolean Retrieval 19 / 60
![Page 59: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/59.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Tokenization and preprocessingDoc 1. I did enact Julius Caesar: Iwas killed i’ the Capitol; Brutus killedme.Doc 2. So let it be with Caesar. Thenoble Brutus hath told you Caesarwas ambitious:
=⇒
Doc 1. i did enact julius caesar i waskilled i’ the capitol brutus killed meDoc 2. so let it be with caesar thenoble brutus hath told you caesar wasambitious
Schutze: Boolean Retrieval 20 / 60
![Page 60: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/60.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Generate postings
Doc 1. i did enact julius caesar i waskilled i’ the capitol brutus killed meDoc 2. so let it be with caesar thenoble brutus hath told you caesar wasambitious
=⇒
term docID
i 1did 1enact 1julius 1caesar 1i 1was 1killed 1i’ 1the 1capitol 1brutus 1killed 1me 1so 2let 2it 2be 2with 2caesar 2the 2noble 2brutus 2hath 2told 2you 2caesar 2was 2ambitious 2
Schutze: Boolean Retrieval 21 / 60
![Page 61: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/61.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Sort postingsterm docID
i 1did 1enact 1julius 1caesar 1i 1was 1killed 1i’ 1the 1capitol 1brutus 1killed 1me 1so 2let 2it 2be 2with 2caesar 2the 2noble 2brutus 2hath 2told 2you 2caesar 2was 2ambitious 2
=⇒
term docID
ambitious 2be 2brutus 1brutus 2capitol 1caesar 1caesar 2caesar 2did 1enact 1hath 1i 1i 1i’ 1it 2julius 1killed 1killed 1let 2me 1noble 2so 2the 1the 2told 2you 2was 1was 2with 2
Schutze: Boolean Retrieval 22 / 60
![Page 62: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/62.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Create postings lists, determine document frequencyterm docID
ambitious 2be 2brutus 1brutus 2capitol 1caesar 1caesar 2caesar 2did 1enact 1hath 1i 1i 1i’ 1it 2julius 1killed 1killed 1let 2me 1noble 2so 2the 1the 2told 2you 2was 1was 2with 2
=⇒
term doc. freq. → postings lists
ambitious 1 → 2
be 1 → 2
brutus 2 → 1 → 2
capitol 1 → 1
caesar 2 → 1 → 2
did 1 → 1
enact 1 → 1
hath 1 → 2
i 1 → 1
i’ 1 → 1
it 1 → 2
julius 1 → 1
killed 1 → 1
let 1 → 2
me 1 → 1
noble 1 → 2
so 1 → 2
the 2 → 1 → 2
told 1 → 2
you 1 → 2
was 2 → 1 → 2
with 1 → 2
Schutze: Boolean Retrieval 23 / 60
![Page 63: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/63.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Split the result into dictionary and postings file
Brutus −→ 1 2 4 11 31 45 173 174
Caesar −→ 1 2 4 5 6 16 57 132 . . .
Calpurnia −→ 2 31 54 101
...
︸ ︷︷ ︸ ︸ ︷︷ ︸
dictionary postings file
Schutze: Boolean Retrieval 24 / 60
![Page 64: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/64.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Later in this course
Index construction: how can we create inverted indexes forlarge collections?
Schutze: Boolean Retrieval 25 / 60
![Page 65: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/65.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Later in this course
Index construction: how can we create inverted indexes forlarge collections?
How much space do we need for dictionary and index?
Schutze: Boolean Retrieval 25 / 60
![Page 66: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/66.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Later in this course
Index construction: how can we create inverted indexes forlarge collections?
How much space do we need for dictionary and index?
Index compression: how can we efficiently store and processindexes for large collections?
Schutze: Boolean Retrieval 25 / 60
![Page 67: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/67.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Later in this course
Index construction: how can we create inverted indexes forlarge collections?
How much space do we need for dictionary and index?
Index compression: how can we efficiently store and processindexes for large collections?
Ranked retrieval: what does the inverted index look like whenwe want the “best” answer?
Schutze: Boolean Retrieval 25 / 60
![Page 68: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/68.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Outline
1 Introduction
2 Inverted index
3 Processing Boolean queries
4 Query optimization
5 Course overview
Schutze: Boolean Retrieval 26 / 60
![Page 69: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/69.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Simple conjunctive query (two terms)
Consider the query: Brutus AND Calpurnia
Schutze: Boolean Retrieval 27 / 60
![Page 70: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/70.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Simple conjunctive query (two terms)
Consider the query: Brutus AND Calpurnia
To find all matching documents using inverted index:
Schutze: Boolean Retrieval 27 / 60
![Page 71: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/71.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Simple conjunctive query (two terms)
Consider the query: Brutus AND Calpurnia
To find all matching documents using inverted index:1 Locate Brutus in the dictionary
Schutze: Boolean Retrieval 27 / 60
![Page 72: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/72.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Simple conjunctive query (two terms)
Consider the query: Brutus AND Calpurnia
To find all matching documents using inverted index:1 Locate Brutus in the dictionary2 Retrieve its postings list from the postings file
Schutze: Boolean Retrieval 27 / 60
![Page 73: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/73.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Simple conjunctive query (two terms)
Consider the query: Brutus AND Calpurnia
To find all matching documents using inverted index:1 Locate Brutus in the dictionary2 Retrieve its postings list from the postings file3 Locate Calpurnia in the dictionary
Schutze: Boolean Retrieval 27 / 60
![Page 74: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/74.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Simple conjunctive query (two terms)
Consider the query: Brutus AND Calpurnia
To find all matching documents using inverted index:1 Locate Brutus in the dictionary2 Retrieve its postings list from the postings file3 Locate Calpurnia in the dictionary4 Retrieve its postings list from the postings file
Schutze: Boolean Retrieval 27 / 60
![Page 75: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/75.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Simple conjunctive query (two terms)
Consider the query: Brutus AND Calpurnia
To find all matching documents using inverted index:1 Locate Brutus in the dictionary2 Retrieve its postings list from the postings file3 Locate Calpurnia in the dictionary4 Retrieve its postings list from the postings file5 Intersect the two postings lists
Schutze: Boolean Retrieval 27 / 60
![Page 76: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/76.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Simple conjunctive query (two terms)
Consider the query: Brutus AND Calpurnia
To find all matching documents using inverted index:1 Locate Brutus in the dictionary2 Retrieve its postings list from the postings file3 Locate Calpurnia in the dictionary4 Retrieve its postings list from the postings file5 Intersect the two postings lists6 Return intersection to user
Schutze: Boolean Retrieval 27 / 60
![Page 77: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/77.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Intersecting two postings lists
Brutus −→ 1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
Calpurnia −→ 2 → 31 → 54 → 101
Intersection =⇒
Schutze: Boolean Retrieval 28 / 60
![Page 78: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/78.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Intersecting two postings lists
Brutus −→ 1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
Calpurnia −→ 2 → 31 → 54 → 101
Intersection =⇒
Schutze: Boolean Retrieval 28 / 60
![Page 79: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/79.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Intersecting two postings lists
Brutus −→ 1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
Calpurnia −→ 2 → 31 → 54 → 101
Intersection =⇒
Schutze: Boolean Retrieval 28 / 60
![Page 80: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/80.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Intersecting two postings lists
Brutus −→ 1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
Calpurnia −→ 2 → 31 → 54 → 101
Intersection =⇒ 2
Schutze: Boolean Retrieval 28 / 60
![Page 81: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/81.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Intersecting two postings lists
Brutus −→ 1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
Calpurnia −→ 2 → 31 → 54 → 101
Intersection =⇒ 2
Schutze: Boolean Retrieval 28 / 60
![Page 82: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/82.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Intersecting two postings lists
Brutus −→ 1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
Calpurnia −→ 2 → 31 → 54 → 101
Intersection =⇒ 2
Schutze: Boolean Retrieval 28 / 60
![Page 83: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/83.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Intersecting two postings lists
Brutus −→ 1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
Calpurnia −→ 2 → 31 → 54 → 101
Intersection =⇒ 2
Schutze: Boolean Retrieval 28 / 60
![Page 84: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/84.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Intersecting two postings lists
Brutus −→ 1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
Calpurnia −→ 2 → 31 → 54 → 101
Intersection =⇒ 2 → 31
Schutze: Boolean Retrieval 28 / 60
![Page 85: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/85.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Intersecting two postings lists
Brutus −→ 1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
Calpurnia −→ 2 → 31 → 54 → 101
Intersection =⇒ 2 → 31
Schutze: Boolean Retrieval 28 / 60
![Page 86: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/86.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Intersecting two postings lists
Brutus −→ 1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
Calpurnia −→ 2 → 31 → 54 → 101
Intersection =⇒ 2 → 31
Schutze: Boolean Retrieval 28 / 60
![Page 87: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/87.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Intersecting two postings lists
Brutus −→ 1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
Calpurnia −→ 2 → 31 → 54 → 101
Intersection =⇒ 2 → 31
Schutze: Boolean Retrieval 28 / 60
![Page 88: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/88.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Intersecting two postings lists
Brutus −→ 1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
Calpurnia −→ 2 → 31 → 54 → 101
Intersection =⇒ 2 → 31
Schutze: Boolean Retrieval 28 / 60
![Page 89: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/89.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Intersecting two postings lists
Brutus −→ 1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
Calpurnia −→ 2 → 31 → 54 → 101
Intersection =⇒ 2 → 31
This is linear in the length of the postings lists.
Schutze: Boolean Retrieval 28 / 60
![Page 90: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/90.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Intersecting two postings lists
Brutus −→ 1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
Calpurnia −→ 2 → 31 → 54 → 101
Intersection =⇒ 2 → 31
This is linear in the length of the postings lists.
Note: This only works if postings lists are sorted.
Schutze: Boolean Retrieval 28 / 60
![Page 91: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/91.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Intersecting two postings lists
Intersect(p1, p2)1 answer ← 〈 〉2 while p1 6= nil and p2 6= nil
3 do if docID(p1) = docID(p2)4 then Add(answer , docID(p1))5 p1 ← next(p1)6 p2 ← next(p2)7 else if docID(p1) < docID(p2)8 then p1 ← next(p1)9 else p2 ← next(p2)10 return answer
Schutze: Boolean Retrieval 29 / 60
![Page 92: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/92.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Query processing: Exercise
france −→ 1 → 2 → 3 → 4 → 5 → 7 → 8 → 9 → 11 → 12 → 13 → 14 → 15
paris −→ 2 → 6 → 10 → 12 → 14
lear −→ 12 → 15
Compute hit list for ((paris AND NOT france) OR lear)
Schutze: Boolean Retrieval 30 / 60
![Page 93: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/93.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Boolean retrieval model: Assessment
The Boolean retrieval model can answer any query that is aBoolean expression.
Schutze: Boolean Retrieval 31 / 60
![Page 94: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/94.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Boolean retrieval model: Assessment
The Boolean retrieval model can answer any query that is aBoolean expression.
Boolean queries are queries that use and, or and not to joinquery terms.
Schutze: Boolean Retrieval 31 / 60
![Page 95: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/95.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Boolean retrieval model: Assessment
The Boolean retrieval model can answer any query that is aBoolean expression.
Boolean queries are queries that use and, or and not to joinquery terms.Views each document as a set of terms.
Schutze: Boolean Retrieval 31 / 60
![Page 96: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/96.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Boolean retrieval model: Assessment
The Boolean retrieval model can answer any query that is aBoolean expression.
Boolean queries are queries that use and, or and not to joinquery terms.Views each document as a set of terms.Is precise: Document matches condition or not.
Schutze: Boolean Retrieval 31 / 60
![Page 97: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/97.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Boolean retrieval model: Assessment
The Boolean retrieval model can answer any query that is aBoolean expression.
Boolean queries are queries that use and, or and not to joinquery terms.Views each document as a set of terms.Is precise: Document matches condition or not.
Primary commercial retrieval tool for 3 decades
Schutze: Boolean Retrieval 31 / 60
![Page 98: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/98.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Boolean retrieval model: Assessment
The Boolean retrieval model can answer any query that is aBoolean expression.
Boolean queries are queries that use and, or and not to joinquery terms.Views each document as a set of terms.Is precise: Document matches condition or not.
Primary commercial retrieval tool for 3 decades
Many professional searchers (e.g., lawyers) still like Booleanqueries.
Schutze: Boolean Retrieval 31 / 60
![Page 99: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/99.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Boolean retrieval model: Assessment
The Boolean retrieval model can answer any query that is aBoolean expression.
Boolean queries are queries that use and, or and not to joinquery terms.Views each document as a set of terms.Is precise: Document matches condition or not.
Primary commercial retrieval tool for 3 decades
Many professional searchers (e.g., lawyers) still like Booleanqueries.
You know exactly what you are getting.
Schutze: Boolean Retrieval 31 / 60
![Page 100: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/100.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Boolean retrieval model: Assessment
The Boolean retrieval model can answer any query that is aBoolean expression.
Boolean queries are queries that use and, or and not to joinquery terms.Views each document as a set of terms.Is precise: Document matches condition or not.
Primary commercial retrieval tool for 3 decades
Many professional searchers (e.g., lawyers) still like Booleanqueries.
You know exactly what you are getting.
Many search systems you use are also Boolean: spotlight,email, intranet etc.
Schutze: Boolean Retrieval 31 / 60
![Page 101: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/101.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Commercially successful Boolean retrieval: Westlaw
Largest commercial legal search service in terms of thenumber of paying subscribers
Schutze: Boolean Retrieval 32 / 60
![Page 102: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/102.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Commercially successful Boolean retrieval: Westlaw
Largest commercial legal search service in terms of thenumber of paying subscribers
Over half a million subscribers performing millions of searchesa day over tens of terabytes of text data
Schutze: Boolean Retrieval 32 / 60
![Page 103: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/103.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Commercially successful Boolean retrieval: Westlaw
Largest commercial legal search service in terms of thenumber of paying subscribers
Over half a million subscribers performing millions of searchesa day over tens of terabytes of text data
The service was started in 1975.
Schutze: Boolean Retrieval 32 / 60
![Page 104: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/104.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Commercially successful Boolean retrieval: Westlaw
Largest commercial legal search service in terms of thenumber of paying subscribers
Over half a million subscribers performing millions of searchesa day over tens of terabytes of text data
The service was started in 1975.
In 2005, Boolean search (called “Terms and Connectors” byWestlaw) was still the default, and used by a large percentageof users . . .
Schutze: Boolean Retrieval 32 / 60
![Page 105: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/105.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Commercially successful Boolean retrieval: Westlaw
Largest commercial legal search service in terms of thenumber of paying subscribers
Over half a million subscribers performing millions of searchesa day over tens of terabytes of text data
The service was started in 1975.
In 2005, Boolean search (called “Terms and Connectors” byWestlaw) was still the default, and used by a large percentageof users . . .
. . . although ranked retrieval has been available since 1992.
Schutze: Boolean Retrieval 32 / 60
![Page 106: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/106.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Westlaw: Example queries
Information need: Information on the legal theories involved inpreventing the disclosure of trade secrets by employees formerlyemployed by a competing company
Query: “trade secret” /s disclos! /s prevent /s employe!
Schutze: Boolean Retrieval 33 / 60
![Page 107: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/107.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Westlaw: Example queries
Information need: Requirements for disabled people to be able toaccess a workplace
Query: disab! /p access! /s work-site work-place (employment /3place)
Schutze: Boolean Retrieval 33 / 60
![Page 108: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/108.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Westlaw: Example queries
Information need: Cases about a host’s responsibility for drunkguests
Query: host! /p (responsib! liab!) /p (intoxicat! drunk!) /p guest
Schutze: Boolean Retrieval 33 / 60
![Page 109: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/109.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Westlaw: Comments
Proximity operators: /3 = within 3 words, /s = within asentence, /p = within a paragraph
Schutze: Boolean Retrieval 34 / 60
![Page 110: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/110.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Westlaw: Comments
Proximity operators: /3 = within 3 words, /s = within asentence, /p = within a paragraph
Space is disjunction, not conjunction! (This was the default insearch pre-Google.)
Schutze: Boolean Retrieval 34 / 60
![Page 111: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/111.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Westlaw: Comments
Proximity operators: /3 = within 3 words, /s = within asentence, /p = within a paragraph
Space is disjunction, not conjunction! (This was the default insearch pre-Google.)
Long, precise queries: incrementally developed, not like websearch
Schutze: Boolean Retrieval 34 / 60
![Page 112: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/112.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Westlaw: Comments
Proximity operators: /3 = within 3 words, /s = within asentence, /p = within a paragraph
Space is disjunction, not conjunction! (This was the default insearch pre-Google.)
Long, precise queries: incrementally developed, not like websearch
Why professional searchers often like Boolean search:precision, transparency, control
Schutze: Boolean Retrieval 34 / 60
![Page 113: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/113.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Westlaw: Comments
Proximity operators: /3 = within 3 words, /s = within asentence, /p = within a paragraph
Space is disjunction, not conjunction! (This was the default insearch pre-Google.)
Long, precise queries: incrementally developed, not like websearch
Why professional searchers often like Boolean search:precision, transparency, control
When are Boolean queries the best way of searching? Dependson: information need, searcher, document collection, . . .
Schutze: Boolean Retrieval 34 / 60
![Page 114: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/114.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Outline
1 Introduction
2 Inverted index
3 Processing Boolean queries
4 Query optimization
5 Course overview
Schutze: Boolean Retrieval 35 / 60
![Page 115: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/115.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Query optimization
Consider a query that is an and of n terms, n > 2
Schutze: Boolean Retrieval 36 / 60
![Page 116: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/116.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Query optimization
Consider a query that is an and of n terms, n > 2
For each of the terms, get its postings list, then and themtogether
Schutze: Boolean Retrieval 36 / 60
![Page 117: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/117.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Query optimization
Consider a query that is an and of n terms, n > 2
For each of the terms, get its postings list, then and themtogether
Example query: Brutus AND Calpurnia AND Caesar
Schutze: Boolean Retrieval 36 / 60
![Page 118: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/118.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Query optimization
Consider a query that is an and of n terms, n > 2
For each of the terms, get its postings list, then and themtogether
Example query: Brutus AND Calpurnia AND Caesar
What is the best order for processing this query?
Schutze: Boolean Retrieval 36 / 60
![Page 119: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/119.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Query optimization
Example query: Brutus AND Calpurnia AND Caesar
Schutze: Boolean Retrieval 37 / 60
![Page 120: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/120.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Query optimization
Example query: Brutus AND Calpurnia AND Caesar
Simple and effective optimization: Process in order ofincreasing frequency
Schutze: Boolean Retrieval 37 / 60
![Page 121: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/121.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Query optimization
Example query: Brutus AND Calpurnia AND Caesar
Simple and effective optimization: Process in order ofincreasing frequency
Start with the shortest postings list, then keep cutting further
Schutze: Boolean Retrieval 37 / 60
![Page 122: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/122.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Query optimization
Example query: Brutus AND Calpurnia AND Caesar
Simple and effective optimization: Process in order ofincreasing frequency
Start with the shortest postings list, then keep cutting further
Brutus −→ 1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
Calpurnia −→ 2 → 31 → 54 → 101
Caesar −→ 5 → 31
Schutze: Boolean Retrieval 37 / 60
![Page 123: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/123.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Query optimization
Example query: Brutus AND Calpurnia AND Caesar
Simple and effective optimization: Process in order ofincreasing frequency
Start with the shortest postings list, then keep cutting further
In this example, first Caesar, then Calpurnia, thenBrutus
Brutus −→ 1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
Calpurnia −→ 2 → 31 → 54 → 101
Caesar −→ 5 → 31
Schutze: Boolean Retrieval 37 / 60
![Page 124: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/124.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Optimized intersection algorithm for conjunctive queries
Intersect(〈t1, . . . , tn〉)1 terms ← SortByIncreasingFrequency(〈t1, . . . , tn〉)2 result ← postings(first(terms))3 terms ← rest(terms)4 while terms 6= nil and result 6= nil
5 do result ← Intersect(result, postings(first(terms)))6 terms ← rest(terms)7 return result
Schutze: Boolean Retrieval 38 / 60
![Page 125: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/125.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
More general optimization
Example query: (madding or crowd) and (ignoble or
strife)
Schutze: Boolean Retrieval 39 / 60
![Page 126: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/126.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
More general optimization
Example query: (madding or crowd) and (ignoble or
strife)
Get frequencies for all terms
Schutze: Boolean Retrieval 39 / 60
![Page 127: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/127.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
More general optimization
Example query: (madding or crowd) and (ignoble or
strife)
Get frequencies for all terms
Estimate the size of each or by the sum of its frequencies(conservative)
Schutze: Boolean Retrieval 39 / 60
![Page 128: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/128.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
More general optimization
Example query: (madding or crowd) and (ignoble or
strife)
Get frequencies for all terms
Estimate the size of each or by the sum of its frequencies(conservative)
Process in increasing order of or sizes
Schutze: Boolean Retrieval 39 / 60
![Page 129: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/129.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Outline
1 Introduction
2 Inverted index
3 Processing Boolean queries
4 Query optimization
5 Course overview
Schutze: Boolean Retrieval 40 / 60
![Page 130: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/130.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Course overview
We are done with Chapter 1 of IIR (IIR 01).
Schutze: Boolean Retrieval 41 / 60
![Page 131: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/131.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Course overview
We are done with Chapter 1 of IIR (IIR 01).
Plan for the rest of the semester: 18–20 of the 21 chapters ofIIR
Schutze: Boolean Retrieval 41 / 60
![Page 132: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/132.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Course overview
We are done with Chapter 1 of IIR (IIR 01).
Plan for the rest of the semester: 18–20 of the 21 chapters ofIIR
In what follows: teasers for most chapters – to give you asense of what will be covered.
Schutze: Boolean Retrieval 41 / 60
![Page 133: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/133.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 02: The term vocabulary and postings lists
Phrase queries: “Stanford University”
Proximity queries: Gates near Microsoft
We need an index that captures position information forphrase queries and proximity queries.
Schutze: Boolean Retrieval 42 / 60
![Page 134: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/134.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 03: Dictionaries and tolerant retrieval
rd aboard ardent boardroom border
or border lord morbid sordid
bo aboard about boardroom border
✲ ✲ ✲ ✲
✲ ✲ ✲ ✲
✲ ✲ ✲ ✲
Schutze: Boolean Retrieval 43 / 60
![Page 135: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/135.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 04: Index construction
masterassign
mapphase
reducephase
assign
parser
splits
parser
parser
inverter
postings
inverter
inverter
a-f
g-p
q-z
a-f g-p q-z
a-f g-p q-z
a-f
segmentfiles
g-p q-z
Schutze: Boolean Retrieval 44 / 60
![Page 136: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/136.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 05: Index compression
0 1 2 3 4 5 6
01
23
45
67
log10 rank
7
log
10
cf
Zipf’s law
Schutze: Boolean Retrieval 45 / 60
![Page 137: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/137.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 06: Scoring, term weighting and the vector space
model
Ranking search results
Boolean queries only give inclusion or exclusion of documents.For ranked retrieval, we measure the proximity between the query andeach document.One formalism for doing this: the vector space model
Key challenge in ranked retrieval: evidence accumulation for a term ina document
1 vs. 0 occurence of a query term in the document3 vs. 2 occurences of a query term in the documentUsually: more is betterBut by how much?Need a scoring function that translates frequency into score or weight
Schutze: Boolean Retrieval 46 / 60
![Page 138: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/138.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 07: Scoring in a complete search system
Documents
Document cache
Indexes
k-gramScoring
parameters
MLR
training set
Results page
Indexers
Parsing Linguistics
user query
Free text query parser
Spell correction Scoring and ranking
Tiered inverted positional index
Inexact top K
retrieval
Metadata in zone and
field indexes
Schutze: Boolean Retrieval 47 / 60
![Page 139: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/139.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 08: Evaluation and dynamic summaries
Schutze: Boolean Retrieval 48 / 60
![Page 140: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/140.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 09: Relevance feedback & query expansion
Schutze: Boolean Retrieval 49 / 60
![Page 141: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/141.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 12: Language models
q1
w P(w |q1) w P(w |q1)
STOP 0.2 toad 0.01the 0.2 said 0.03a 0.1 likes 0.02frog 0.01 that 0.04
. . . . . .
This is a one-state probabilistic finite-state automaton – a unigramlanguage model – and the state emission distribution for its onestate q1.
Schutze: Boolean Retrieval 50 / 60
![Page 142: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/142.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 12: Language models
q1
w P(w |q1) w P(w |q1)
STOP 0.2 toad 0.01the 0.2 said 0.03a 0.1 likes 0.02frog 0.01 that 0.04
. . . . . .
This is a one-state probabilistic finite-state automaton – a unigramlanguage model – and the state emission distribution for its onestate q1.
STOP is not a word, but a special symbol indicating that theautomaton stops.
Schutze: Boolean Retrieval 50 / 60
![Page 143: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/143.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 12: Language models
q1
w P(w |q1) w P(w |q1)
STOP 0.2 toad 0.01the 0.2 said 0.03a 0.1 likes 0.02frog 0.01 that 0.04
. . . . . .
This is a one-state probabilistic finite-state automaton – a unigramlanguage model – and the state emission distribution for its onestate q1.
STOP is not a word, but a special symbol indicating that theautomaton stops.
frog
Schutze: Boolean Retrieval 50 / 60
![Page 144: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/144.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 12: Language models
q1
w P(w |q1) w P(w |q1)
STOP 0.2 toad 0.01the 0.2 said 0.03a 0.1 likes 0.02frog 0.01 that 0.04
. . . . . .
This is a one-state probabilistic finite-state automaton – a unigramlanguage model – and the state emission distribution for its onestate q1.
STOP is not a word, but a special symbol indicating that theautomaton stops.
frog
P(string) = 0.01
Schutze: Boolean Retrieval 50 / 60
![Page 145: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/145.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 12: Language models
q1
w P(w |q1) w P(w |q1)
STOP 0.2 toad 0.01the 0.2 said 0.03a 0.1 likes 0.02frog 0.01 that 0.04
. . . . . .
This is a one-state probabilistic finite-state automaton – a unigramlanguage model – and the state emission distribution for its onestate q1.
STOP is not a word, but a special symbol indicating that theautomaton stops.
frog said
P(string) = 0.01
Schutze: Boolean Retrieval 50 / 60
![Page 146: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/146.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 12: Language models
q1
w P(w |q1) w P(w |q1)
STOP 0.2 toad 0.01the 0.2 said 0.03a 0.1 likes 0.02frog 0.01 that 0.04
. . . . . .
This is a one-state probabilistic finite-state automaton – a unigramlanguage model – and the state emission distribution for its onestate q1.
STOP is not a word, but a special symbol indicating that theautomaton stops.
frog said
P(string) = 0.01 ·0.03
Schutze: Boolean Retrieval 50 / 60
![Page 147: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/147.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 12: Language models
q1
w P(w |q1) w P(w |q1)
STOP 0.2 toad 0.01the 0.2 said 0.03a 0.1 likes 0.02frog 0.01 that 0.04
. . . . . .
This is a one-state probabilistic finite-state automaton – a unigramlanguage model – and the state emission distribution for its onestate q1.
STOP is not a word, but a special symbol indicating that theautomaton stops.
frog said that
P(string) = 0.01 ·0.03
Schutze: Boolean Retrieval 50 / 60
![Page 148: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/148.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 12: Language models
q1
w P(w |q1) w P(w |q1)
STOP 0.2 toad 0.01the 0.2 said 0.03a 0.1 likes 0.02frog 0.01 that 0.04
. . . . . .
This is a one-state probabilistic finite-state automaton – a unigramlanguage model – and the state emission distribution for its onestate q1.
STOP is not a word, but a special symbol indicating that theautomaton stops.
frog said that
P(string) = 0.01 ·0.03 ·0.04
Schutze: Boolean Retrieval 50 / 60
![Page 149: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/149.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 12: Language models
q1
w P(w |q1) w P(w |q1)
STOP 0.2 toad 0.01the 0.2 said 0.03a 0.1 likes 0.02frog 0.01 that 0.04
. . . . . .
This is a one-state probabilistic finite-state automaton – a unigramlanguage model – and the state emission distribution for its onestate q1.
STOP is not a word, but a special symbol indicating that theautomaton stops.
frog said that toad
P(string) = 0.01 ·0.03 ·0.04
Schutze: Boolean Retrieval 50 / 60
![Page 150: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/150.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 12: Language models
q1
w P(w |q1) w P(w |q1)
STOP 0.2 toad 0.01the 0.2 said 0.03a 0.1 likes 0.02frog 0.01 that 0.04
. . . . . .
This is a one-state probabilistic finite-state automaton – a unigramlanguage model – and the state emission distribution for its onestate q1.
STOP is not a word, but a special symbol indicating that theautomaton stops.
frog said that toad
P(string) = 0.01 ·0.03 ·0.04 ·0.01
Schutze: Boolean Retrieval 50 / 60
![Page 151: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/151.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 12: Language models
q1
w P(w |q1) w P(w |q1)
STOP 0.2 toad 0.01the 0.2 said 0.03a 0.1 likes 0.02frog 0.01 that 0.04
. . . . . .
This is a one-state probabilistic finite-state automaton – a unigramlanguage model – and the state emission distribution for its onestate q1.
STOP is not a word, but a special symbol indicating that theautomaton stops.
frog said that toad likes
P(string) = 0.01 ·0.03 ·0.04 ·0.01
Schutze: Boolean Retrieval 50 / 60
![Page 152: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/152.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 12: Language models
q1
w P(w |q1) w P(w |q1)
STOP 0.2 toad 0.01the 0.2 said 0.03a 0.1 likes 0.02frog 0.01 that 0.04
. . . . . .
This is a one-state probabilistic finite-state automaton – a unigramlanguage model – and the state emission distribution for its onestate q1.
STOP is not a word, but a special symbol indicating that theautomaton stops.
frog said that toad likes
P(string) = 0.01 ·0.03 ·0.04 ·0.01 ·0.02
Schutze: Boolean Retrieval 50 / 60
![Page 153: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/153.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 12: Language models
q1
w P(w |q1) w P(w |q1)
STOP 0.2 toad 0.01the 0.2 said 0.03a 0.1 likes 0.02frog 0.01 that 0.04
. . . . . .
This is a one-state probabilistic finite-state automaton – a unigramlanguage model – and the state emission distribution for its onestate q1.
STOP is not a word, but a special symbol indicating that theautomaton stops.
frog said that toad likes frog
P(string) = 0.01 ·0.03 ·0.04 ·0.01 ·0.02
Schutze: Boolean Retrieval 50 / 60
![Page 154: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/154.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 12: Language models
q1
w P(w |q1) w P(w |q1)
STOP 0.2 toad 0.01the 0.2 said 0.03a 0.1 likes 0.02frog 0.01 that 0.04
. . . . . .
This is a one-state probabilistic finite-state automaton – a unigramlanguage model – and the state emission distribution for its onestate q1.
STOP is not a word, but a special symbol indicating that theautomaton stops.
frog said that toad likes frog
P(string) = 0.01 ·0.03 ·0.04 ·0.01 ·0.02 ·0.01
Schutze: Boolean Retrieval 50 / 60
![Page 155: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/155.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 12: Language models
q1
w P(w |q1) w P(w |q1)
STOP 0.2 toad 0.01the 0.2 said 0.03a 0.1 likes 0.02frog 0.01 that 0.04
. . . . . .
This is a one-state probabilistic finite-state automaton – a unigramlanguage model – and the state emission distribution for its onestate q1.
STOP is not a word, but a special symbol indicating that theautomaton stops.
frog said that toad likes frog STOP
P(string) = 0.01 ·0.03 ·0.04 ·0.01 ·0.02 ·0.01
Schutze: Boolean Retrieval 50 / 60
![Page 156: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/156.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 12: Language models
q1
w P(w |q1) w P(w |q1)
STOP 0.2 toad 0.01the 0.2 said 0.03a 0.1 likes 0.02frog 0.01 that 0.04
. . . . . .
This is a one-state probabilistic finite-state automaton – a unigramlanguage model – and the state emission distribution for its onestate q1.
STOP is not a word, but a special symbol indicating that theautomaton stops.
frog said that toad likes frog STOP
P(string) = 0.01 ·0.03 ·0.04 ·0.01 ·0.02 ·0.01 ·0.2
Schutze: Boolean Retrieval 50 / 60
![Page 157: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/157.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 12: Language models
q1
w P(w |q1) w P(w |q1)
STOP 0.2 toad 0.01the 0.2 said 0.03a 0.1 likes 0.02frog 0.01 that 0.04
. . . . . .
This is a one-state probabilistic finite-state automaton – a unigramlanguage model – and the state emission distribution for its onestate q1.
STOP is not a word, but a special symbol indicating that theautomaton stops.
frog said that toad likes frog STOP
P(string) = 0.01 ·0.03 ·0.04 ·0.01 ·0.02 ·0.01 ·0.2= 0.0000000000048
Schutze: Boolean Retrieval 50 / 60
![Page 158: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/158.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 13: Text classification & Naive Bayes
Text classification = assigning documents automatically topredefined classes
Examples:
Language (English vs. French)Adult contentRegion
Schutze: Boolean Retrieval 51 / 60
![Page 159: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/159.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 14: Vector classification
X
X
XX
X
X
X
X
X
X
X
∗
Schutze: Boolean Retrieval 52 / 60
![Page 160: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/160.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 15: Support vector machines
Schutze: Boolean Retrieval 53 / 60
![Page 161: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/161.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 16: Flat clustering
Schutze: Boolean Retrieval 54 / 60
![Page 162: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/162.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 17: Hierarchical clustering
http://news.google.com
Schutze: Boolean Retrieval 55 / 60
![Page 163: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/163.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 18: Latent Semantic Indexing
Schutze: Boolean Retrieval 56 / 60
![Page 164: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/164.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 19: The web and its challenges
Unusual and diverse documents
Unusual and diverse users and information needs
Beyond terms and text: exploit link analysis, user data
How do web search engines work?
How can we make them better?
Schutze: Boolean Retrieval 57 / 60
![Page 165: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/165.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
IIR 21: Link analysis / PageRank
Schutze: Boolean Retrieval 58 / 60
![Page 166: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/166.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Take-away
Boolean Retrieval: Design and data structures of a simpleinformation retrieval system
What topics will be covered in this class?
Schutze: Boolean Retrieval 59 / 60
![Page 167: Introduction to Information Retrieval ` `%%%`#`&12 ...hs/teach/14s/ir/pdf/01intro.pdf · Information retrieval (IR) is finding material (usually documents) of an unstructured nature](https://reader034.vdocument.in/reader034/viewer/2022042403/5f17deade32b795a5a0da879/html5/thumbnails/167.jpg)
Introduction Inverted index Processing Boolean queries Query optimization Course overview
Resources
Chapter 1 of IIR
http://cislmu.org
course scheduleinformation retrieval linksShakespeare search engine
Schutze: Boolean Retrieval 60 / 60