rank aggregation

1. Comparison-based evaluation
Jose Antonio Martinez Torres

2. Problem

3. Law of Large number
1 spin
-$500
1000 spin
$100,000

5. 2
5
4

6. 4
3
2

7. Experiments
• Movies Dataset
– 100,000 ratings
– 1000 users
– 1700 movies
• http://www.grouplens.org/node/73

8. True rating
• Based on the law of large numbers, average
rating using this movies dataset can work very
well.
• However, by using a small sample, average
rating would significantly differ from the true
rating.

9. = { 2, 4, 5,5,4,4,5,3,3,4,5,4,
5,4,5,4,5,4,3,2,3,3,4,5,4,4,5,5,3,4}
True rating = 4
= { 2, 4 }
Rating = 3

10. Kendall tau
• Statistic used to measure the degree of
similarity between two rankings.
• Practical use:
– Compare how close the top-10 results produced
by Google and Bing are.

11. A
B
C
D
E
C
D
A
B
E
Rank 1 Rank 2
T = 6 - 4 / ½ (5) (5-1) = 0.2
Dissimilarity goes from 1 to -1 where 1 means the two rankings are
the same and -1 means one ranking is the reverse of the other

12. Movie1 = {3, 4, 5,4,5,3,2,4,4,5,4,3,4,5,4,3,2,3,4,4,4,5,4,3,3,2}
Movie2 = {4, 3, 2,3,2,3,4,3,2,1,2,3,4,3,4,3,2,3,2,3,4,3,4,3,4,3}
Movie3 = {3, 4, 2,4,5,3,2,4,,4,5,4,3,4,5,4,3,2,3,4,4,4,5,4,3,3,2}
Movie4 = {2, 3, 4,5,3,4,4,4,5,4,3,3,2}
Movie5 = {3, 4, 5,2,4,5,4,3,3,2}
Movie6 = {5, 3, 4,3,3,5,4,3,3,2}
Movie7 = {1, 3, 4,3,2,3,43,2,3,4,2,3,4,2,3,4,3,3,2}
.
.
Movie1700= {3, 4, 5,4,2,3,3,2,3,2,1,1,2,3}
Ranking 1 = 3 , 4 , 3, 2, 3, 5, 1
Ranking 2 = 3.5 ,3.5 , 3.5, 2.5, 3.5, 4, 2
.
.
.
Calculate 1 to n ranking n and true rating Kendall tau correlation

13. 0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
Kendalltvalue
Error rate k-distance
Serie…