Experiments with the Negotiated Boolean Queries

of theTREC 2007

Legal Discovery Track

Stephen Tomlinson

Open Text Corporation

2007 Nov 8

Overview

• who won the boolean query “negotiations” ?• can dropping the boolean operators improve on

the boolean run’s Recall@B ?• did the boolean keywords (synonyms) improve

on the natural language request text ?• can just relaxing the proximity constraints

improve Recall@B ?• can blind feedback improve Recall@B ?• can a fusion of vector and boolean approaches

improve Recall@B ?

3 Boolean Queries

• Defendant – initial boolean query proposed by the

defendant

• Plaintiff– rejoinder boolean query from the plaintiff

• Final– final negotiated boolean query

Topic 74: “All scientific studies expressly referencing health effects

tied to indoor air quality.”

Defendant:"health effect!" w/10 "air quality"

Plaintiff:(scien! OR stud! OR research) AND ("air quality" OR health)

Final:(scien! OR stud! OR research) AND ("air quality" w/15 health)

Topic 74 Boolean Results

Defendant:"health effect!" w/10 "air quality"– 2691 matches, 82% precision, 3% recall

Plaintiff:(scien! OR stud! OR research) AND ("air quality" OR health)

– 858,700 matches, 64% precision@25000 (ranked), 25% recall@25000 (ranked)

Final:(scien! OR stud! OR research) AND ("air quality" w/15 health)

– 20,516 matches, 77% precision, 22% recall

Topic 74: Missed Relevant Documents

Final Boolean:(scien! OR stud! OR research) AND ("air quality" w/15 health)

Passages in Missed Relevant Documents:• “… Lowrey A.H. (1980). Indoor air pollution …”• “assessment … entitled “Respiratory Health

Effects of Passive Smoking …”• “study … funded by the Center for Indoor Air

Research”

Defendant vs. Final Boolean: Precision

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43

Prec

• Def. Boolean won 20• Boolean won 22• (1 tied)

Mean in (-0.09, 0.15)

Topic 63: 1.00 vs. 0.02 (sugar contract)

Topic 69: 0.00 vs. 0.97 (indoor smoke ventilation)

Defendant vs. Final Boolean: Recall

-1

-0.9

-0.8

-0.7

-0.6

-0.5

-0.4

-0.3

-0.2

-0.1

0

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43

R@B

• Def. Boolean won 0• Boolean won 42• (1 tied)

Mean in (-0.27, -0.11)

Topic 77: 0.00 vs. 0.00 (smoke NOT tobacco)

Topic 52: 0.00 vs. 0.98 (boosting crop yields)

Plaintiff vs. Final Boolean: Recall@25000

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43

R25000

• Pl. Boolean won 35• Boolean won 6• (2 tied)

Mean in (0.03, 0.19)

Topic 59: 0.76 vs. 0.01 (limestone treatment)

Topic 58: 0.24 vs. 0.94 (phosphates and health)

Plaintiff vs. Final Boolean: Recall@B

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43

R@B

• Pl. Boolean won 15• Boolean won 27• (1 tied)

Mean in (-0.09, 0.04)

Topic 63: 0.73 vs. 0.27 (sugar contract)

Topic 58: 0.18 vs. 0.94 (phosphates and health)

Vector vs. Boolean (Example)

Boolean: (scien! OR stud! OR research) AND ("air quality" w/15 health)

Vector: scien! OR stud! OR research OR air OR quality OR health

Relevance Ranking

• term frequency dampening (BM25)– wildcard variants treated as same term– for boolean proximity constraints, only count

term occurrences satisfying proximity– metadata + ocr included in document length

• inverse document frequency (log)– based on most common variant for wildcards

Vector vs. Boolean: Recall@B

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43

R@B

• Vector won 16• Boolean won 26• (1 tied)

Mean in (-0.13, 0.02)

Topic 63: 0.79 vs. 0.27 (sugar contract)

Topic 58: 0.08 vs. 0.94 (phosphates and health)

Topic 58: “… health problems caused by HPF …”

Vector R@B=0.08, Boolean R@B=0.94 • (B=8183, estRel = 1151)

Phosphat! w/75 (caus! OR relat! OR assoc! OR derive! OR correlat!) w/75 (health OR disorder! OR toxic! OR "chronic fatigue" OR dysfunction! OR irregular OR memor! OR immun! OR myopath! OR liver! OR kidney! OR heart! OR depress! OR loss OR lost)

• vector matches often didn’t mention “Phospat!”

Topic 72: “… chemical process(es) which result in onions … making persons cry”

Vector R@B=0.03, Boolean R@B=0.78 • (B=119, estRel = 98)

((scien! OR research! OR chemical)

w/25 onion!)

AND (cries OR cry! OR tear!)

• proximity clause found some long documents with just one reference to onions’ effects

Topic 63: “… exclusivity clause in a sugar contract …”

Vector R@B=0.79, Boolean R@B=0.27

• (B=294, estRel = 18)

(Sugar w/20

(contract! OR agreement! OR deal!))

AND exclusiv!

• boolean missed “U.S. sugar quota law”

Request vs. Vector: R@25000

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43

R25000

• Req. Vector won 21• Vector won 22• (0 tied)

Mean in (0.00, 0.13)

Topic 87: 1.00 vs. 0.13 (SEC reporting)

Topic 84: 0.64 vs. 0.91 (1960s films)

Impact of Doubling Proximity Distances: Recall@B

-0.4

-0.35

-0.3

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43

R@B

• 2x-Prox Boolean won 14• Boolean won 8• (21 tied)

Mean in (-0.03, 0.02)

Topic 61: 0.49 vs. 0.44 (waste treatment)

Topic 72: 0.39 vs. 0.78 (onions effect)

Impact of Blind Feedback: Recall@B

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43

R@B

• Boolean+BF won 16• Boolean won 21• (6 tied)

Mean in (-0.12, 0.03)

Topic 90: 0.64 vs. 0.10 (sales in England)

Topic 58: 0.01 vs. 0.94 (phosphates and health)

Fusion of Boolean, Request and Vector: Recall@B

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43

R@B

• Fusion won 20• Boolean won 20• (3 tied)

Mean in (-0.08, 0.03)

Topic 65: 0.88 vs. 0.67 (candy packaging)

Topic 58: 0.10 vs. 0.94 (phosphates and health)

Conclusions

• final negotiated boolean query often had substantially lower recall than the plaintiff boolean query

• boolean operators (AND, proximity) often have value

• blind feedback and fusion did not improve the boolean run’s Recall@B (on average)