an interactive learning approach to optimizing information retrieval systems yahoo! august 24 th,...

An Interactive Learning Approach to Optimizing Information Retrieval Systems

Yahoo!August 24th, 2010

Yisong Yue Cornell University

InformationSystems

Designer

How to collect feedback?•What to ask human judges?•What results to show users?

How to interpret feedback?•What is the utility function?•What are useful signals?

Optimization

Optimize Utility Function•Tune parameters (signals)

Collecting Feedback

Explicit Judgments•Is this document good?

Usage Logs (clicks)•Clicks on search results.

Designer



Optimization


Collecting Feedback


Usage Logs•Clicks on search results.

Collecting Feedback


Usage Logs•Clicks on search results.

Optimization


Designer



Interactive Learning Setting

• Find the best ranking function (out of 1000)– Show results, evaluate using click logs– Clicks biased (users only click on what they see)– Explore / exploit problem

Interactive Learning Setting

• Find the best ranking function (out of 1000)– Show results, evaluate using click logs– Clicks biased (users only click on what they see)– Explore / exploit problem

• Technical issues– How to interpret clicks?– What is the utility function?– What results to show users?

Team-Game Interleaving

1. Kernel Machines http://svm.first.gmd.de/

2. Support Vector Machinehttp://jbolivar.freeservers.com/

3. An Introduction to Support Vector Machineshttp://www.support-vector.net/

4. Archives of SUPPORT-VECTOR-MACHINES ...http://www.jiscmail.ac.uk/lists/SUPPORT...

5. SVM-Light Support Vector Machine http://ais.gmd.de/~thorsten/svm light/

1. Kernel Machines http://svm.first.gmd.de/

2. SVM-Light Support Vector Machine http://ais.gmd.de/~thorsten/svm light/

3. Support Vector Machine and Kernel ... Referenceshttp://svm.research.bell-labs.com/SVMrefs.html

4. Lucent Technologies: SVM demo applet http://svm.research.bell-labs.com/SVT/SVMsvt.html

5. Royal Holloway Support Vector Machine http://svm.dcs.rhbnc.ac.uk

1. Kernel Machines T2http://svm.first.gmd.de/

2. Support Vector Machine T1http://jbolivar.freeservers.com/

3. SVM-Light Support Vector Machine T2http://ais.gmd.de/~thorsten/svm light/

4. An Introduction to Support Vector Machines T1http://www.support-vector.net/

5. Support Vector Machine and Kernel ... References T2http://svm.research.bell-labs.com/SVMrefs.html

6. Archives of SUPPORT-VECTOR-MACHINES ... T1http://www.jiscmail.ac.uk/lists/SUPPORT...

7. Lucent Technologies: SVM demo applet T2http://svm.research.bell-labs.com/SVT/SVMsvt.html

f1(u,q) r1 f2(u,q) r2

Interleaving(r1,r2)

(u=thorsten, q=“svm”)

Interpretation: (r1 Â r2) ↔ clicks(T1) > clicks(T2)

Invariant: For all k, in expectation same number of team members in top k from each team.

NEXTPICK

[Radlinski, Kurup, Joachims; CIKM 2008]

Setting

• Find the best ranking function (out of 1000)– Evaluate using click logs– Clicks biased (users only click on what they see)– Explore / exploit problem

• Technical issues– How to interpret clicks?– What is the utility function?– What results to show users?

Dueling Bandits Problem

• Given K bandits b1, …, bK

• Each iteration: compare (duel) two bandits– E.g., interleaving two retrieval functions

[Yue & Joachims, ICML 2009] [Yue, Broder, Kleinberg, Joachims, COLT 2009]

Dueling Bandits Problem

• Given K bandits b1, …, bK

• Each iteration: compare (duel) two bandits– E.g., interleaving two retrieval functions

• Cost function (regret):

• (bt, bt’) are the two bandits chosen

• b* is the overall best one• (% users who prefer best bandit over chosen ones)

T

tttT bbPbbPR

1

1)'*()*(

[Yue & Joachims, ICML 2009] [Yue, Broder, Kleinberg, Joachims, COLT 2009]

T

tttT bbPbbPR

1

1)'*()*(

•Example 1•P(f* > f) = 0.9•P(f* > f’) = 0.8•Incurred Regret = 0.7

•Example 2 •P(f* > f) = 0.7•P(f* > f’) = 0.6•Incurred Regret = 0.3

•Example 3•P(f* > f) = 0.51•P(f* > f) = 0.55•Incurred Regret = 0.06

Assumptions

• P(bi > bj) = ½ + εij (distinguishability)

• Strong Stochastic Transitivity– For three bandits bi > bj > bk :

– Monotonicity property

• Stochastic Triangle Inequality– For three bandits bi > bj > bk :

– Diminishing returns property

• Satisfied by many standard models– E.g., Logistic / Bradley-Terry

jkijik , max

jkijik

Explore then Exploit

• First explore– Try to gather as much information as possible– Accumulates regret based on which bandits we decide to

compare

• Then exploit– We have a (good) guess as to which bandit best– Repeatedly compare that bandit with itself

• (i.e., interleave that ranking with itself)

T

tttT bbPbbPR

1

1)'*()*(

Naïve Approach

• In deterministic case, O(K) comparisons to find max

• Extend to noisy case:– Repeatedly compare until confident one is better

• Problem: comparing two awful (but similar) bandits

• Example:– P(A > B) = 0.85 – P(A > C) = 0.85– P(B > C) = 0.51– Comparing B and C requires many comparisons!

Interleaved Filter • Choose candidate bandit at random ►

[Yue, Broder, Kleinberg, Joachims, COLT 2009]

Interleaved Filter• Choose candidate bandit at random

• Make noisy comparisons (Bernoulli trial)against all other bandits simultaneously– Maintain mean and confidence interval for each pair of bandits being compared

►




• …until another bandit is better – With confidence 1 – δ

►





• Repeat process with new candidate– (Remove all empirically worse bandits)

►





• Repeat process with new candidate– (Remove all empirically worse bandits)

• Continue until 1 candidate left►


Intuition

• Simulate comparing candidate with all remaining bandits simultaneously


Intuition

• Simulate comparing candidate with all remaining bandits simultaneously

• Example:– P(A > B) = 0.85 – P(A > C) = 0.85– P(B > C) = 0.51– B is candidate– # comparisons between B vs C bounded

by comparisons between B vs A!


Regret Analysis• Round: all the time steps for a particular

candidate bandit– Halts when better bandit found …– … with 1- δ confidence– Choose δ = 1/(TK2)

• Match: all the comparisons between two bandits in a round– At most K matches in each round– Candidate plays one match against each

remaining bandit


Regret Analysis• O(log K) total rounds• O(K) total matches

• Each match incurs regret– Depends on δ = K-2T-1

• Finds best bandit w.p. 1-1/T• Expected regret:

TO log1

TK

ORE

TOT

TK

OT

RE

T

T

log

)(1

log1

1


Removing Inferior Bandits• At conclusion of each round

– Remove any empirically worse bandits

• Intuition:– High confidence that winner is better

than incumbent candidate

– Empirically worse bandits cannot be “much better” thanincumbent candidate

– Can show via Hoeffding bound that winner is also better than empirically worsebandits with high confidence

– Preserves 1-1/T confidence overall that we’ll find the best bandit

Summary

• Provably efficient online algorithm – (In a regret sense)– Also results for continuous (convex) setting

Summary

• Provably efficient online algorithm – (In a regret sense)– Also results for continuous (convex) setting

• Requires comparison oracle– Informative (not too noisy)– Independence / Unbiased– Strong transitivity– Triangle Inequality

Revisiting Assumptions

• P(bi > bj) = ½ + εij (distinguishability)

• Strong Stochastic Transitivity– For three bandits bi > bj > bk :

– Monotonicity property

• Stochastic Triangle Inequality– For three bandits bi > bj > bk :

– Diminishing returns property

• Satisfied by many standard models– E.g., Logistic / Bradley-Terry

jkijik , max

jkijik

Directions to Explore

• Relaxing assumptions– E.g., strong transitivity & triangle inequality

• Integrating context

• Dealing with large K– Assume additional structure on for retrieval functions?

• Other cost models– PAC setting (fixed budget, find the best possible)

• Dynamic or changing user interests / environment

Improving Comparison Oracle

Improving Comparison Oracle

• Dueling Bandits Problem– Interactive learning framework– Provably minimizes regret– Assumes ideal comparison oracle

• Can we improve the comparison oracle?– Can we improve how we interpret results?

Determining Statistical Significance

• Each q, interleave A(q) and B(q), log clicks

• t-Test– For each q, score: % clicks on A(q)

• E.g., 3/4 = 0.75

– Sample mean score (e.g., 0.6)– Compute confidence (p value)

• E.g., want p = 0.05 (i.e., 95% confidence)

– More data, more confident

Limitation

• Example: query session with 2 clicks – One click at rank 1 (from A)– Later click at rank 4 (from B)– Normally would count this query session as a tie

Limitation

• Example: query session with 2 clicks– One click at rank 1 (from A)– Later click at rank 4 (from B)– Normally would count this query session as a tie– But second click is probably more informative…– …so B should get more credit for this query

Linear Model

• Feature vector φ(q,c):

• Weight of click is wTφ(q,c)

click previousrank than higher if 1

clicklast if 1

download toledclick if 1

always 1

),( cq

[Yue, Gao, Chapelle, Zhang, Joachims, SIGIR 2010]

Example

• wTφ(q,c) differentiates last clicks and other clicks

else 0 click;last not is c if 1

else 0 click;last is c if 1),( cq


Example


• Interleave A vs B– 3 clicks per session– Last click 60% on result from A– Other 2 clicks random




Example


• Interleave A vs B– 3 clicks per session– Last click 60% on result from A– Other 2 clicks random

• Conventional w = (1,1) – has significant variance• Only count last click w = (1,0) – minimizes variance




Scoring Query Sessions

• Feature representation for query session:

)()(

),(),(1

qBcqAcq cqcq

q


Scoring Query Sessions

• Feature representation for query session:

• Weighted score for query:

• Positive score favors A, negative favors B

)()(

),(),(1

qBc

T

qAc

Tq

T cqwcqwq

w

)()(

),(),(1

qBcqAcq cqcq

q


Supervised Learning

• Will optimize for z-Test: Inverse z-Test– Approximately equal t-Test for large samples– z-Score = mean / standard deviation

1

1

2

)(

)(maxarg*

)(1

)(

1)(

Tw

qq

T

qq

T

wstd

wmeanw

wmeanwn

wstd

wn

wmean

(Assumes A > B)


Experiment Setup

• Data collection – Pool of retrieval functions– Hash users into partitions– Run interleaving of different pairs in parallel

• Collected on arXiv.org– 2 pools of retrieval functions– Training Pool: (6 pairs) know A > B – New Pool: (12 pairs)

Training Pool – Cross Validation

Experimental Results

• Inverse z-Test works well– Beats baseline on most of new interleaving pairs– Direction of tests all in agreement– In 6/12 pairs, for p=0.1, reduces sample size by 10%– In 4/12 pairs, achieves p=0.05, but not baseline

• 400 to 650 queries per interleaving experiment

• Weights hard to interpret (features correlated)• Largest weight: “1 if single click & rank > 1”

Interactive Learning

• System dynamically decides how to service users– Learn about user preferences– Maximize total user utility over time– Exploration / exploitation tradeoff

Interactive Learning

• System dynamically decides how to service users– Learn about user preferences– Maximize total user utility over time– Exploration / exploitation tradeoff

• Existing mechanism (interleaving) to collect feedback– Can also improve how we interpret feedback

• Simple yet practical cost model– Compatible with interleaving oracle– Yields efficient algorithms

Thank You!

Work supported by NSF IIS-0905467, Yahoo! KSC Award, and

Microsoft Graduate Fellowship.

Slides, papers & software available at www.yisongyue.com

http://www.yisongyue.com/

an interactive learning approach to optimizing information retrieval systems yahoo! august 24 th,...

Documents

support vector machines

support vector machinest1

support vector machinet1

bandits b i b j b

kernel machines http

referencest2 http

references http

exploit problem slide