implementing an advanced click model · implementing an advanced click model practical session 2...

Implementing an Advanced Click ModelPractical Session 2

Aleksandr Chuklin§,¶ Ilya Markov§ Maarten de Rijke§

a.chuklin@uva.nl i.markov@uva.nl derijke@uva.nl

§University of Amsterdam¶Google Research Europe

AC–IM–MdR Implementing an Advanced Click Model 1

Challenge

1 Download code and slides fromclickmodels.weebly.com/russir-2016-course.html

No further internet access required

2 Work in teams up to 3 people

3 Submit your solution to i.markov@uva.nl and a.chuklin@uva.nl

4 Deadline is Thursday, August 25, 14:00

5 The winners will be announced at RuSSIR party

AC–IM–MdR Implementing an Advanced Click Model 2

Data and tools

Relevance prediction challenge dataset (WSCD 2012)

PyClick – an open-source Python library of click models forweb search

AC–IM–MdR Implementing an Advanced Click Model 3

Setup

Download files fromclickmodels.weebly.com/russir-2016-course.html

Install PyPy

Install enum34 package

Install PyClick:

git clone https://github.com/markovi/PyClick.gitcd PyClick && pypy setup.py install –user

http://clickmodels.weebly.com/russir-2016-setup.html

AC–IM–MdR Implementing an Advanced Click Model 4

Starting point

TCM class copied from PBM

Basic experiment

Load the train and test dataTrain TCMTest TCM

Additional classes explained during the session

AC–IM–MdR Implementing an Advanced Click Model 5

Preliminaries

AC–IM–MdR Implementing an Advanced Click Model 6

Preliminaries

1 Initial run(a) Load data and train the model(b) Get the same output as for PBM, save the output

2 Switch from single query sessions to search tasks(a) Extend SearchSession into TaskCentricSearchSession

(b) Create SearchTask

(c) Extend the data loading procedure, re-load data(d) Output the number of search tasks

3 Train on search tasks(a) Pass search tasks to TCM.train()

(b) Get an error4 Implement new inference

(a) Create TaskCentricInference as a subclass of EMInference(b) Pass a search task into Param.update()

(c) Use the new inference in TCM, re-run training(d) Get the same output as for PBM (compare with the previously

saved output)

AC–IM–MdR Implementing an Advanced Click Model 7

Implementing TCM

AC–IM–MdR Implementing an Advanced Click Model 8

Implementing TCM

1 Adding TCM-specific parameters

(a) Create TCMMatch, TCMNew, TCMFresh as subclasses of ParamEM(b) Add these parameters to TCM, re-run training(c) Still get the same output as in PBM

2 Update the EM formulas for attractiveness and examination

(a) Add matching and freshness to the EM formulas,re-run training

(b) Get the updated output

3 Implement EM formulas for other parameters

(a) Plug the EM updating formulas into Param.update(),re-run training

(b) Output the trained TCM parameters

AC–IM–MdR Implementing an Advanced Click Model 9

Session parameters

Rank Document

1 u12 u23 u34 u45 u5

Session Parameters

{αu1q, ε1, µ, ν, φ}{αu2q, ε2, µ, ν, φ}{αu3q, ε3, µ, ν, φ}{αu4q, ε4, µ, ν, φ}{αu5q, ε5, µ, ν, φ}

AC–IM–MdR Implementing an Advanced Click Model 10

EM update rules for TCM parameters

Attractiveness

α(t+1)uq =

1

|Suq|∑s∈Suq

(c(s)u +

(1− c

(s)u

) (1− εrµφ)αuq

1− εrαuqµφ

)

Examination

ε(t+1)r =

1

|Sr |∑s∈Sr

(c(s)u +

(1− c

(s)u

) (1− αuqµφ) εr1− εrαuqµφ

)Freshness

φ(t+1) =1

|S|∑s∈S

1

|s|∑

u∈s:H(s)u

(c(s)u +

(1− c

(s)u

) (1− εrαuqµ)φ

1− εrαuqµφ

)

AC–IM–MdR Implementing an Advanced Click Model 11

EM update rules for TCM parameters (cont’d)

Query matching the user’s information need

µ(t+1) =1

|S|∑s∈S

P(M(s) = 1 | C

)Submitting a new query after a matching query

ν(t+1) =

∑s∈S I ((s) 6= (l))P

(M(s) = 1 | C

)∑s∈S P

(M(s) = 1 | C

)Use TCMUtils to calculate P

(M(s) = 1 | C

)

AC–IM–MdR Implementing an Advanced Click Model 12

Summary

Implementing an advanced click model is relativelystraightforward if you have EM update formulae

Requires minimal changes in the infrastructure

TCM does not perform as well as PBM

Possible reasons

Simplification (previous occurrence instead of examination)Evaluation is not performed properly (discussed next)Problems with the model

AC–IM–MdR Implementing an Advanced Click Model 13

Next steps: Testing

Extend LogLikelihood to work with search tasks

Derive formulas for TCM click probabilities(full and conditional)

Plug the formulas into TCM.get full click probs()

and TCM.get conditional click probs()

AC–IM–MdR Implementing an Advanced Click Model 14

Next lecture

Basic Click Models

Parameter Estimation Evaluation

Data and ToolsResultsApplications

Advanced Models

Recent Studies

Future Research

AC–IM–MdR Implementing an Advanced Click Model 15

Acknowledgments

All content represents the opinion of the authors which is not necessarily shared orendorsed by their respective employers and/or sponsors.

AC–IM–MdR Implementing an Advanced Click Model 16