social book search: a combination of personalized recommendations and retrieval

Social Book Search:A Combination of Personalized Recommendations and Retrieval

Author:Justin van WeesSupervisor:Marijn KoolenSecond assessor:Frank Nack

Master ThesisInformation Science

Human Centered MultimediaAugust 23, 2012

Outline

1. Background

2. Research questions

3. Data collection

4. Experiments and results

5. Conclusions

6. Discussion and future work

7. Questions

Current situation

• Traditional information retrieval (IR) models:

• developed for use on small collections

• contain only officially published documents, annotated by professionals

• Many modern web (2.0) applications still use traditional models for search

• Millions of documents

• Combination of user-generated content (UDG) and professional metadata

Current situation

• User uses IR system to find those documents that are topically relevant to her information need

• Queries can lead to thousands of relevant documents

• Evaluating large number of results expensive for user

• Other notions of relevance, i.e. how well-written, popular, recent, fun is the document

• Combination of professional and user-generated metadata

Social Book Search Track

• Evaluate relative value of controlled book metadata versus social metadata (Koolen et al., 2012)

• Amazon.com and LibraryThing (LT) corpus

• ~2.8 million book records, both social and professional metadata

• book search requests from LT discussion forums as topics, suggestions by other users as relevance judgements

Recommender Systems

• Recommender Systems (RSs) suggest items of interest to individuals or groups of users (Resnick and Varian, 1997)

• Assumes that individual’s taste or interest in a particular item can be explained by features recorded by the RS (demographics, previous interactions, etcetera)

• Different strategies: collaborative filtering (CF), content-, community-, knowledge-based, hybrid (Burke, 2007)

• Differs from traditional retrieval in terms of query formulation, source of relevance feedback and personalization (Furner, 2002)

Research Questions

• What data are we able to collect?

• Can we automatically make accurate predictions of a user’s preference for an unknown book?

• How do we combine results from IR system with RSs?

• Social Book Search scenario and data

Does a combination of techniques from the field of IR with those from RSs improve retrieval performance when searching for works in a large scale on-line collaborative media catalogue?

Crawling LibraryThing

• Perform four different crawls of user profiles and personal catalogues

• For each crawl, also crawl links to other profiles

• Compare crawls to determine representativeness for entire LT user-base

• All crawl combined approximately 6% of LT userbase

Crawl seed list profiles unique works profile overlap

Forum users 1,104 60,131 4,354,387

Random – 211 works 1,306 8,040 2,537,065 7,048

Random – 1,000 works 5,577 18,381 3,580,296 14,262

Random – 10,000 works 35,671 64,379 5,122,848 37,300

Total - 89,693 5,299,399 -

Crawling LibraryThing

Crawl min. max. median mean std. dev.

Forum users

Friends 0 172 3.0 8.47 16.31

Groups 0 10 9.0 6.79 3.74

Interesting Libraries 0 510 2.0 11.19 26.46

Random – 211 works

Friends 0 79 0.0 2.61 7.46

Groups 0 10 0.0 1.70 3.05

Interesting Libraries 0 394 0.0 3.30 17.80

Random – 1,000 works

Friends 0 84 0.0 2.18 6.07

Groups 0 10 0.0 1.64 3.02

Interesting Libraries 0 574 0.0 2.74 14.41

Random – 10,000 works

Friends 0 2,858 0.0 1.73 17.49

Groups 0 10 0.0 1.24 2.61

Interesting Libraries 0 855 0.0 1.69 10,40

Total

Friends 0 2,858 1.0 2.14 12.77

Groups 0 10 0.0 1.18 2.44

Interesting Libraries 0 855 0.0 1.27 8.00

Crawling LibraryThing

Crawl min. max. median mean std. dev. sum

Forum users

Unrated 0 28,402 84.0 397.22 929.70 23,885,23

Rated 0 12,190 3.0 78.80 238.53 4,738,018

Total 0 28,402 148.00 476.02 980.88 28,623,249

Random – 211 works

Unrated 0 28,402 458.00 1,112.81 1,835.81 8,946,997

Rated 0 12,190 10.00 182.77 472.08 1,469,531

Total 0 28,402 657.00 1,295.58 1,908.65 10,416,528

Random – 1,000 works

Unrated 0 28,402 331.00 864.32 1,480.98 15,887,025

Rated 0 12,190 3.00 130.20 369.06 2,393,233

Total 0 28,402 475.00 994.52 1539.15 18,280,258

Random – 10,000 works

Unrated 0 28,402 163.00 486.63 955.86 31,328,971

Rated 0 12,190 1.00 74.04 237.01 4,766,750

Total 0 28,402 201.00 560.68 1,000.50 36,095,721

Total

Unrated 0 28,402 102.00 378.18 834.94 33,920,353

Rated 0 12,190 1.00 62.85 206.40 5,637,097

Total 0 28,402 156.00 441.03 876.76 39,557,450

Generating Recommendations

• Collaborative filtering approach

• Unary and rated transactions

• Memory- and model-based recommenders

• Randomly split transactions (80% train/20% test) for performance evaluation

Generating Recommendations

• Neighbourhood (Desrosiers and Karypis, 2011):

• Directly use user-item ratings to predict ratings for ‘unseen’ items

• Find n most similar neighbours (Pearson correlation)

• Use the weighted average rating given by the user’s neighbours

• Let neighbours ‘vote’ on unary transactions

Generating Recommendations

• Singular Value Decomposition (SVD) (Schafer et al., 2007):

• Reduce domain complexity by mapping item space to k dimensions

• Remaining dimensions represent the latent topics: preferences classes of users, categorical classes of items

• Currently considered ‘state of the art’

Recommender Performance

Method MAE RMSE P@5 P@10 P@50

Neighbourhood (N=25) 0.7813 1.0286 0.0712 0.0661 0.0614

Neighbourhood (N=50) 0.7721 1.0105 0.0376 0.0371 0.0339

Neighbourhood (N=100) 0.7633 0.9927 0.0246 0.0239 0.0232

SVD (K=50) 0.6210 0.8139 0.0021 0.0019 0.0026

SVD (K=100) 0.6203 0.8131 0.0025 0.0022 0.0028

SVD (K=150) 0.6192 0.8122 0.0281 0.0107 0.0030

Method Accuracy P@5 P@10 P@50

Neighbourhood (N=25) 0.2430 0.3711 0.2425 0.1829

Neighbourhood (N=50) 0.3014 0.3824 0.2561 0.1861

Neighbourhood (N=100) 0.3621 0.3640 0.2422 0.1812

SVD (K=50) 0.2240 0.0214 0.0198 0.0216

SVD (K=100) 0.2601 0.0219 0.0203 0.0229

SVD (K=150) 0.2676 0.0424 0.0212 0.0234

Retrieving Works

• Setup used for INEX 2012; top performing run

• Index consists of user-generated content

• Removed stopwords

• Stemming with Krovetz

• Topic titles as queries

• Language model

• Pseudo relevance feedback, 50 terms of top 10 results

Combining IR and RS

• Retrieval system: ranked list, probability score between 0 and 1 per work

• Recommendations: estimated preference of user for work between 0.5 and 5.0 or 0 or 1 (unary)

• Normalise ratings

• ‘Boost’ works with estimated preference, CombSUM (Fox and Shaw, 19994)

• Use average rating when no prediction can be made

• Introduce weight (λ) between systems

Results

Method � nDCG@10 P@10 R@10

Baseline - 0.1437 0.1219 0.1494

Neighbourhood

Rated (n=25) 0.0001700 0.1709 (18.93%) 0.1490 (22.23%) 0.1899 (27.11%)

Rated (n=50) 0.0001855 0.1778 (23.73%) 0.1500 (23.05%) 0.1913 (28.05%)

Rated (n=100) 0.0001800 0.1669 (16.14%) 0.1490 (22.23%) 0.1878 (25.70%)

Unary (n=25) 0.0001500 0.1446 (0.63%) 0.1229 (0.82%) 0.1520 (1.74%)

Unary (n=50) 0.0001500 0.1441 (0.28%) 0.1229 (0.82%) 0.152 (1.74%)

Unary (n=100) 0.0001500 0.1441 (0.28%) 0.1229 (0.82%) 0.152 (1.74%)

SVD

Rated (K=50) 0.0001800 0.1718 (19.55%) 0.149 (22.23%) 0.1866 (24.9%)

Rated (K=100) 0.0001850 0.1721 (19.76%) 0.149 (22.23%) 0.1866 (24.9%)

Rated (K=150) 0.0001850 0.172 (19.69%) 0.149 (22.23%) 0.1866 (24.90%)

Unary (K=50) 0.0001500 0.1449 (0.84%) 0.124 (1.72%) 0.1541 (3.15%)

Unary (K=100) 0.0001550 0.1441 (0.28%) 0.1229 (0.82%) 0.1520 (1.74%)

Unary (K=150) 0.0001550 0.1424 (-0.9%) 0.1250 (2.54%) 0.1561 (4.48%)

Conclusions

• Collected representative sample of user profiles

• Collaborative filtering obvious choice

• SVD best at estimating rated preference

• Poor performance on unary transactions

• Successfully combined retrieval with personalized recommendations

• Rated transactions most useful

• Personal preference is relevance evidence that can highly improve retrieval performance in SBS

Discussion and Future Work

• Popularity as relevance evidence

• Value of λ depending on IR score distribution

• Other (mixtures of) RS setups

• Scaling, cold-start problems

• Trust and transparency of the system

Questions?

References• R. Burke. Hybrid web recommender systems. In The adaptive web, pages 377–408. Springer-Verlag, 2007.

• C. Desrosiers and G. Karypis. A comprehensive survey of neighborhood-based recommendation methods. Recommender

Systems Handbook, pages 107–144, 2011.

• E. Fox and J. Shaw. Combination of multiple searches. NIST SPECIAL PUBLICATION SP, pages 243–243, 1994.

• J. Furner. On recommending. Journal of the American Society for Information Science and Technology, 53(9):747–763, 2002.

• M. Koolen, G. Kazai, J. Kamps, A. Doucet, and M. Landoni. Overview of the INEX 2011 books and social search track. In S. Geva, J.

Kamps, and R. Schenkel, editors, Focused Retrieval of Content and Structure: 10th International Workshop of the Initiative for the

Evaluation of XML Retrieval (INEX 2011), volume 7424 of LNCS. Springer, 2012.

• P. Resnick and H. Varian. Recommender systems. Communi- cations of the ACM, 40(3):56–58, 1997.

• J. B. Schafer, D. Frankowski, J. Herlocker, and S. Sen. Collaborative Filtering Recommender Systems. Inter- national Journal of Electronic

Business, 2(1):77, 2007. ISSN 14706067. doi: 10.1504/IJEB.2004.004560. URL http://www.springerlink.com/index/

t87386742n752843.pdf.

Number of Books in Catalogue

(a) Unrated works (b) Rated works

Document scoring

S(d) = (1� �)PRet(d|q) + �PCF (d)

• PRet(d|q): work’s score obtained through IR system

• PCF : estimated rating of current user for work obtained through RS

• �: weight between systems

Estimating preference (rated)

r̂ui =

Pv2Ni(u)

wuvrvi

Pv2Ni(u)

|wuv|

• r̂ui: estimated preference of user u for item i

• wuv: preference similarity between users v and u

• Ni(u): k-NN of u that rated item i

Desrosiers and Karypis, 2011

Estimating preference (unary)

vir =X

v2Ni(u)

�(rvi = r)Wuv

Desrosiers and Karypis, 2011