rank aggregation methods for the web
Post on 31-Dec-2015
97 Views
Preview:
DESCRIPTION
TRANSCRIPT
Rank Aggregation Methods for the Web
CS728
Lecture 11
Web Page Ranking Methods Reviewed
• PageRank – global link analysis
• Indegree – local link analysis
• HITS- topic-based link analysis
• Voting –NNN and Correlation
• Graph distance from seed
• URL length and depth
• Text-based methods (e.g., tf*idf)
Rank Aggregation
BDCA
ABDCFE
BCDAFE
“Consensus” ranking of all BDCAFE
Notations for Ranking
• Given a universe U, and ordered list τ of a subset of S of U
τ=[x1≥ x2≥… ≥xd] , xi in S τ(i) : position of rank of i
|τ|: number of elements
• full list : τ which contains all the elements in U• partial list : rank only some of elements in U• top d list : all d ranked elements are above all unra
nked elements• Question: when are two orderings similar? Can you
give a distance measure?
Measuring Distance Between Orderings
• Spearman’s Footrule Distance– σ , τ : two full list.– σ( i ) :rank of candidate i
• Kendall tau distance– Count the number of pairwise
disagreements between the two lists
Example of Ordered-List Distance
• Example
– S = {A,B,C,D,E}
– σ , τ : two full list
• Spearman’s Footrule Distance
– F(σ , τ ) = 1 + 2 + 1 + 0 + 2 = 6
• Kendall tau distance
– K(σ , τ ) = |{(A,C), (B.D), (B,E), (D,E)}| = 4
ACEDB
CABDE
12345
τσ
Optimal ranking aggregation
• Optimality depends on the distance measure we use.
• Optimizing with Kendall tau distance, we obtain Kemeny optimal aggregation
• Can show satisfies neutrality and consistency – important properties of rank aggregation
functions.• Useful but computationally hard. Kemeny
optimal aggregation is NP-hard.• Will show that footrule-optimal is in P.
Two properties relate K and F
• For any full lists σ,τK(σ,τ) ≤ F(σ,τ) ≤ 2 K(σ,τ)So we get a 2-approximation to Kemeny-optimality
• Since, if σ is the Kemeny optimal aggregation of full lists τ1 ,…, τk and σ’ optimizes the footrule aggregation then,
K(σ’, τ1 ,…, τk ) ≤ 2 K(σ, τ1 ,…, τk )
• Condorcet Criterion– An element of S which wins every other in pairwise sim
ple majority voting should be ranked first.• Extended Condorcet Criterion (XCC):
– If most voters prefer candidate a to candidate b (i.e., # of i s.t. i(a) < i(b) is at least n/2), then also should prefer a to b (i.e., (a) < (b)).
• XCC is effective in ‘spam-fighting’ and thus good to use in meta-search.
Condorcet Criteria and SPAM Filters
XCC: Not always realizable
a b c
b a a
c c b
a) < (b) < (c)
a b c
b c a
c a b
Not realizable
Voting Theory: Desired Properties
• Given set of candidates and voter preferences: seek an algorithm that ranks candidates which satisfies a set of desired properties
• Which combination of properties are realizable?
• 1) Independence from Irrelevant Alternatives:Relative order of a and b in should depend only on relative order of a and b in 1,…,n.– Ex: if i = (a b c) changes to (a c b), relative order of
a,b in should not change.
Desired Properties:
• 2) NeutralityNo candidate should be favored to others.– If two candidates switch positions in 1,…,n, they
should switch positions also in .
• 3) AnonymityNo voter should be favored to others.– If two voters switch their orderings, should
remain the same.
Desired Properties:
• 4) MonotonicityIf the ranking of a candidate is improved by a voter, its ranking in can only improve.
• 5) ConsistencyIf voters are split into two disjoint sets, S and T, and both the aggregation of voters in S and the aggregation of voters in T prefer a to b, then also the aggregation of all voters should prefer a to b.
Desired Properties
• 6) No Dictatorship: f(1,…,n) != I
• 7) Unanimity (a.k.a. Pareto optimality):
If all voters prefer candidate a to candidate b (i.e., i(a) < i(b) for all i), then also should prefer a to b (i.e., (a) < (b)).
Desired Properties
• 8) Democracy: satisfies extended Condorcet Criterion XCC.– Always works for m = 2.– Not always realizable for m ≥ 3.
• Theorem [May, 1952]: For m = 2, Democracy is the only rank aggregation function which is monotone, neutral, and anonymous.
Arrow’s Impossibility Theorem [Arrow, 1951]
• Theorem: If m ≥ 3, then the only rank aggregation function that is unanimous and independent from irrelevant alternatives is dictatorship.– Won Nobel prize (1972)
Borda’s method
• Easy and intuitive - Several “score-based”variants; 1781
• Violates independence from irrelevant alternatives
C3C1...
C7C8
C10
C7C1...
C8C3
C10
C3C2...
C7C10C9
C3C8...
C1C15C10
1 2 3 4
Bi(c)=the number of candidates ranked below c in i
B(c)=iBi(c)Sorted in
decreasing order
Bi(C8) = 1 2 0 13
Partial lists
• Handle partial lists by giving all the excess scores equally among all unranked candidates,
Example: Candidates number = 100 Ranked candidates number =70
(score: 31~100)
=>Assign score 31/30 to each 30 unranked candidates
Footrule optimal aggregation
• Footrule optimal aggregation can be computed in polynomial time. is a good approximation of Kemeny optimal aggregation.
• Proof : Via minimum cost perfect matching
Markov Chain method for rank aggregation.
• States=candidates• Transitions depend on the preference orders
given by voters• Basic idea: probabilistically switch to a
“better candidate”• Rank candidates based on stationary
probabilities!
Markov chain advantages
• Handling partial list and top d list by using available comparisons to infer new ones
• Handling uneven comparison and list length
• Computation efficiency– O(NK) preprocessing,O(K) per step for about O(N) steps
Four ways to build transition Matrix• Current state is candidate a.
• MC1: Choose uniformly from multiset of all candidates that were ranked at least as high as a by some voter.
– Probability to stay at a: ~ average rank of a.
• MC2: Choose a voter i uniformly at random and pick uniformly at random from among the candidates that the i-th voter ranked at least as high as a.
• MC3: Choose a voter i uniformly at random and pick uniformly at random a candidate b. If i-th voter ranked b higher than a, go to b. Otherwise, stay in a.
• MC4: Choose a candidate b uniformly at random If most voters ranked b higher than a, go to b. Otherwise, stay in a.
– Rank of a ~ # of “pairwise contests” a wins.
A locally Kemeny optimal aggregation is a relaxation of Kemeny Optimality
• A locally Kemeny optimal aggregation satisfies the extended Condorcet property and can be computed in “kO(nlogn)” worst case, O(n2)
• Many of existing aggregation methods do not satisfy ECC.
=>Given τ1 , … ,τk use your favorite aggregation method to obtain a full list μ. And Apply local kemenization to μ with respect to τ1 , … ,τk .
• A local Kemenization of a full list with respect to Compute a locally Kemeny optimal aggregation of that is maximally consistent with
This approach:
(1) preserves the strengths of the initial aggregation .
(2) ranks non-spam above spam.
(3) gives a result that disagrees with on any pair ( i, j ) only if a majority of the τ’s endorse this disagreement.
(4) for every d, 1 ≤ d ≤ | μ |, the restriction of the output is a local Kemenization of the top d elements of μ
Local Kemenization is a procedure to get locally Kemeny optimal aggregation.
1 2, ,..., K 1 2, ,..., K
How do we perform local kemenization?
ABFECD
BCAEFD
ACFDEB
BFDCAE
CABFED
BADCEF
BBAABABD
ABDC
ABCD
ABCFED
• Local Kemenization Example!
disagree
A>B: 3A<B: 2
B>D: 4B<D: 1
Experiments: meta-search
K = Kendall distance SF = scaled footrule distance
IF = induced footrule distance LK = Local Kemenization
top related