query anchoring using discriminative query...
Post on 23-Apr-2020
19 Views
Preview:
TRANSCRIPT
Query Anchoring Using Discriminative
Query ModelsSaar Kuzi Anna Shtok Oren Kurland
Technion – Israel Institute Of Technology
We thank SIGIR for the conference travel grant
Pseudo-feedback-basedQuery ExpansionHighly ranked documents are used to induce a query model
Relying on pseudo feedback may result in query drift:
Documents in the result list could be non relevant
Relevant documents can contain non query-pertaining information(Harman ’92, He&Ounis ’09, Lv&Zhai ’09)
2
Initial Result List
Pseudo Feedback
𝐷𝑖𝑛𝑖𝑡
Query
Query AnchoringTechniques for mitigating the risk in relying on pseudo feedback
Direct:
Interpolation with a model of the original query (e.g., Zhai&Lafferty ’01, Abdul-Jaleel et al. ’04, Lv&Zhai ’09)
Using the original query model as a prior (Tao&Zhai ’04, Tao&Zhai ’06)
Indirect:
Term clipping (e.g., Zhai&Lafferty ’01, Abdul-Jaleel et al. ’04, Ye et al. ’10)
Differential impact of documents on the query model (Lavrenko&Croft ‘01, Abdul-Jaleel et al. ’04, Lv&Zhai ‘14)
3
Our Approach A novel indirect query anchoring approach using a new discriminativeterm-based model
An accurate term-based representation of the initial ranking
Initial Result List𝐷𝑖𝑛𝑖𝑡
Query Model
Method
4
Discriminative Query Model
Anchored Query Model
Learning-to-rank
Language Model Notation Unigram language models are used
Given text 𝑥,
𝑝𝑀𝐿𝐸 𝑡 𝑥 ≝𝑡𝑓 𝑡 ∈ x
𝑥
𝑝𝐷𝑖𝑟 𝑡 𝑥 ≝𝑡𝑓 𝑡 ∈ 𝑑 + 𝜇𝑝MLE 𝑡 𝐶
𝜇 + 𝑥
Two language models, 𝜃1 and 𝜃2, are compared using cross entropy:
𝑡 is a term, 𝑥 is the length of 𝑥, and 𝐶 is the collection of documents
5
𝐶𝐸(𝑝(∙ |𝜃1)||𝑝 ∙ 𝜃2 ) = −
𝑡
𝑝 𝑡 𝜃1 log 𝑝(𝑡|𝜃2)
Mixture Model (Zhai&Lafferty ’01)
6
C
𝒅𝟏
𝜽𝑻
𝒅𝟐 𝒅𝒏…
𝑑∈𝐷𝑖𝑛𝑖𝑡
𝑡∈𝑑
𝑡𝑓 𝑡 ∈ 𝑑 log( 1 − 𝛾 𝑝 𝑡 𝜃𝑇 + 𝛾𝑝 𝑡 𝐶 ))
𝑝 𝑡 𝑀𝑀 ≝ 𝜆𝑝𝑀𝐿𝐸 𝑡 𝑞 + 1 − 𝜆 𝑝(𝑡|𝜃𝑇𝑐𝑙𝑖𝑝𝑝𝑒𝑑
)
R
𝒅𝟏q 𝒅𝒏…
𝑝 𝑡 𝑅𝑀1 ≝
𝑑∈𝐷𝑖𝑛𝑖𝑡
𝑝𝐷𝑖𝑟 𝑡 𝑑 𝑝(𝑑|𝑞)
𝑝 𝑡 𝑅𝑀3 ≝ 𝜆𝑝𝑀𝐿𝐸 𝑡 𝑞 + 1 − 𝜆 𝑝 𝑡 𝑅𝑀1𝑐𝑙𝑖𝑝𝑝𝑒𝑑
Relevance Model (Lavrenko&Croft ’01)
(Abdul-Jaleel et al. ’04)
The log-likelihood of documents in 𝐷𝑖𝑛𝑖𝑡:
Generative Query
Models
Discriminative Model
The pseudo feedback assumption: the higher a document is
ranked in the initial result list, the higher its relevance likelihood
∀𝑑𝑖 ,𝑑𝑗 ∈ 𝐷𝑖𝑛𝑖𝑡, if 𝑟(𝑑𝑖) > 𝑟(𝑑𝑗), then 𝑑𝑖 is more likely to
be relevant than 𝑑𝑗
7
SVM-rank (Joachims ’02)
min1
2𝑤 ∙ 𝑤 + 𝐶
𝑖,𝑗
𝜉𝑖,𝑗
∀𝑖∀𝑗. 𝑟 𝑑𝑖 > 𝑟 𝑑𝑗 𝑤(𝜙 𝑑𝑖 − 𝜙 𝑑𝑗 ) ≥ 1 − 𝜉𝑖,𝑗
∀𝑖∀𝑗. 𝑟 𝑑𝑖 > 𝑟 𝑑𝑗 𝜉𝑖 ,𝑗 ≥ 0
𝜙 𝑑 = (log 𝑝𝐷𝑖𝑟 𝑡1 𝑑 , . . . , 𝑙𝑜𝑔 𝑝𝐷𝑖𝑟(𝑡 𝑉 |𝑑))
𝑉 is the vocabulary used in the initial result list
8
Model Derivation
9
𝑤
𝑤+ 𝑤−
𝜃𝑤+ 𝜃𝑤−
𝐿1 Norm
Positive Anchor Model
Negative Anchor Model
NegativeComponents
Positive Components
AnchorPos Method
∀𝑡. 𝑠(𝑡) = 𝜆2𝑝(𝑡|𝜃) + 𝜆3𝑝(𝑡|𝜃𝑤+)
𝑝(𝑡|𝜗+)
𝑝(𝑡|𝜃𝐴𝑛𝑐ℎ𝑜𝑟𝑃𝑜𝑠) = 𝜆1𝑝𝑀𝐿𝐸 (𝑡|𝑞) + (1 − 𝜆1)𝑝(𝑡|𝜗+)
Anchoring a generative model, 𝜃, using the positive anchor model, 𝜃𝑤+
1. Term clipping
2. Sum normalization
Interpolation with the original query
model
𝜆1 + 𝜆2 + 𝜆3 = 1, 𝜆𝑖 ≥ 0
10
ClipNeg MethodClipping negative anchor terms
1. Setting to 0 the probabilities of negative anchor terms
2. Term clipping
3. Sum normalization
𝑝(𝑡|𝜗−)
𝑝(𝑡|𝜃𝐶𝑙𝑖𝑝𝑁𝑒𝑔) = 𝜆𝑝𝑀𝐿𝐸(𝑡|𝑞) + (1 − 𝜆)𝑝(𝑡|𝜗−)
Interpolation with the original query
model
𝑝(𝑡|𝜃)
11
Related Work Existing query anchoring techniques (direct query anchoring, term
clipping and differential weighting)
Applying our approach on top of these yields further improvements
Methods for improving the quality of the pseudo feedback result list
(e.g., Mitra et al. ’98, Lee et al. ’08)
Our model can be induced from any ranked list
12
Related Work A supervised term classification approach (Cao et al. ’08)
Our approach is unsupervised and focuses on unigram query models
Clustering of terms in a query model (Udupa et al. ’09)
A method for cluster selection was not proposed
Fusing the result lists: the initial and the expansion-based
(Zighelnic&Kurland ’08)
Our methods operate on the model level and post better performance
13
Evaluation TREC datasets: TREC123, ROBUST, and WT10G
The initial result list is retrieved using a standard language model
method (Lafferty&Zhai ’01): −𝐶𝐸(𝑝𝑀𝐿𝐸(∙ |𝑞)||𝑝𝐷𝑖𝑟 ∙ 𝑑 )
Baselines:
◦ Generative Models (RM3, MM)
◦ Fusion (Zighelnic&Kurland ’08)
Values of free parameters are set using leave-one-out cross validation
14
The Discriminative Model
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Ke
nd
all's
tau
ROBUST
v=100
v=500
v=ALL
α
−𝛼𝐶𝐸(𝑝(∙ |𝜃𝑤+)||𝑝𝐷𝑖𝑟 ∙ 𝑑 ) + 1 − 𝛼 𝐶𝐸(𝑝(∙ |𝜃𝑤−)||𝑝𝐷𝑖𝑟 ∙ 𝑑 )
• Re-ranking an initial result list of 100 documents according to:
• The positive and negative anchor models are clipped to use ν terms
16
𝐑𝐌𝟏 (𝐀𝐏 = 𝟐𝟒. 𝟖)
𝛉𝐰+ (𝐀𝐏 = 𝟒𝟗. 𝟑)
Query: Airport Security, ROBUST, QL(AP=24.8) Discriminative vs.
Generative
16
• The discriminative model assigns high probabilities to terms with high IDF values
• The generative models are much more similar to each other, with respect to the terms they promote, than they are to the discriminative model
20
22
24
26
28
30
RM3 MM
MA
P
TREC123
An
cho
rPo
s
An
cho
rPo
s
Fusi
on
Fusi
on
20
22
24
26
28
30
RM3 MMM
AP
ROBUST
Fusi
on
18
19
20
21
22
RM3 MM
MA
P
WT10G
Fusi
on
17
𝜓∗∗
𝜓∗ ∗
𝜓∗ 𝜓∗
𝜓
∗∗
AnchorPos
Statistically significant differences with:
Generative Model
Fusion
Anchoring a generative model using the positive anchor
model
𝜓
∗
Fusi
on
An
cho
rPo
s
An
cho
rPo
s
Fusi
on
Fusi
on
An
cho
rPo
s
An
cho
rPo
s
Fusi
on
Fusi
on
20
22
24
26
28
30
RM3 MM
MA
P
ROBUST
20
22
24
26
28
30
RM3 MM
MA
P
TREC123
ClipNeg
18
19
20
21
22
RM3 MM
MA
P
WT10G
Clip
Ne
g
Clip
Ne
g
Clip
Ne
gC
lipN
eg
Clipping negative
anchor terms
Statistically significant differences with the generative model
18
∗
∗
Clip
Ne
g
Clip
Ne
g
Clip
Ne
g
Clip
Ne
g
Clip
Ne
g
∗
Summary We presented a novel unsupervised pseudo-feedback-based
discriminative query model that is based on a learning-to-rank-
approach
We devised a few methods that use the discriminative model to
perform (indirect) query anchoring of existing query models
Empirical evaluation showed that using our methods can improve the
performance of highly effective generative query models
19
top related