foxtrot seminar 18.1.2002 capturing knowledge of user preferences with recommender systems stuart e....
TRANSCRIPT
Foxtrot seminar 18.1.2002
Capturing knowledge of user preferenceswith recommender systems
Stuart E. MiddletonDavid C. De Roure, Nigel R. Shadbolt
Intelligence, Agents and Multimedia Research GroupDept of Electronics and Computer Science
University of SouthamptonUnited Kingdom
Email: [email protected]
• Introduction to recommender systems• Knowledge capture of user profiles• Quickstep architecture and approach• Issues arising from Quickstep evaluation• Foxtrot architecture and approach• Future work
Capturing knowledge of user preferenceswith recommender systems
Foxtrot seminar 18.1.2002
• Introduction to recommender systems
Capturing knowledge of user preferenceswith recommender systems
Recommender systemsCollaborative filters (several commercial examples)Content-based filtersHybrid filters
A real world problem domainOn-line research paper recommendation for researchersEvaluation of users in a real work setting
Knowledge acquisition must be unobtrusiveSystem must not interfere with normal work practiceMonitoring should be unobtrusiveFeedback requested only when recommendations checked
WWW information overload
Foxtrot seminar 18.1.2002
• Knowledge capture of user profiles
Capturing knowledge of user preferenceswith recommender systems
Binary class profile representation‘Interesting’ and ‘not interesting’ examplesTime-decay function favours recent examplesMachine learning classifies new information (e.g. TF-IDF)
Foxtrot seminar 18.1.2002
• Knowledge capture of user profiles
Capturing knowledge of user preferenceswith recommender systems
Binary class profile representation
Foxtrot seminar 18.1.2002
User A Interesting Not Interesting
Doc Doc
User B Interesting Not Interesting
Doc Doc
• Knowledge capture of user profiles
Capturing knowledge of user preferenceswith recommender systems
Collaborative similarityBehaviour correlation finds similar users (e.g. Pearson r)New information comes from similar users
Binary class profile representation‘Interesting’ and ‘not interesting’ examplesTime-decay function favours recent examplesMachine learning classifies new information (e.g. TF-IDF)
Foxtrot seminar 18.1.2002
• Knowledge capture of user profiles
Capturing knowledge of user preferenceswith recommender systems
Collaborative similarity
Foxtrot seminar 18.1.2002
Groups ofsimilar users
B
A
F
E
D
C
User
User ratings
Ratings vector space
• Knowledge capture of user profiles
Capturing knowledge of user preferenceswith recommender systems
Collaborative similarityBehaviour correlation finds similar users (e.g. Pearson r)New information comes from similar users
Our approach - Multi-class profileClasses explicitly represent using domain ontologyDomain knowledge can enhance profilingExamples of classes can be sharedAccuracy decreases with number of classes
Binary class profile representation‘Interesting’ and ‘not interesting’ examplesTime-decay function favours recent examplesMachine learning classifies new information (e.g. TF-IDF)
Foxtrot seminar 18.1.2002
• Knowledge capture of user profiles
Capturing knowledge of user preferenceswith recommender systems
Multi-class profile representation
Foxtrot seminar 18.1.2002
Topic A Topic B
Doc Doc
User AInteresting Topic A,BNot interesting Topic C
Topic C
Doc
User BInteresting Topic B,CNot interesting Topic A
• Quickstep architecture and approach
Capturing knowledge of user preferenceswith recommender systems
Classifierk-nearest neighbourUsers can add examples
World WideWeb
Classified papers
Classifier
ProfileUsers
Recommender
Research papersTF vector representation
Foxtrot seminar 18.1.2002
• Quickstep architecture and approach
Capturing knowledge of user preferenceswith recommender systems
K-Nearest Neighbour - kNNTF vector representationExamples exist in an n dimensional spaceNew papers are added to this spaceClassification is a function of its ‘closeness’ to examples
n-dimensional space(n = number of terms)
Example paper (class1)
Unclassified paper
Example paper (class2)
Foxtrot seminar 18.1.2002
• Quickstep architecture and approach
Capturing knowledge of user preferenceswith recommender systems
Research papersTF vector representation
Classifierk-nearest neighbourUsers can add examples
World WideWeb
Classified papers
Classifier
ProfileUsers
Recommender
ProfilerFeedback and browsed papers give time/interest profileTime decay function computes current interests
Classified paper databaseGrows as users browse
Foxtrot seminar 18.1.2002
• Quickstep architecture and approach
Capturing knowledge of user preferenceswith recommender systems
ProfilingTime/Interest profileIs-a hierarchy infers topic interest in super-classesTime decay function biases towards recent interests
Time
Interest
Current interests
Subclass(multi-agent
systems)
Subclass(recommender
systems)
Super-class(agents)
Foxtrot seminar 18.1.2002
• Quickstep architecture and approach
Capturing knowledge of user preferenceswith recommender systems
Research papersTF vector representation
Classifierk-nearest neighbourUsers can add examples
Classified paper databaseGrows as users browse
ProfilerFeedback and browsed papers give time/interest profileTime decay function computes current interests
RecommenderRecommends new papers on current topics of interest
World WideWeb
Classified papers
Classifier
ProfileUsers
Recommender
Foxtrot seminar 18.1.2002
• Issues arising from our empirical evaluation
Capturing knowledge of user preferenceswith recommender systems
What advantages does an ontology bring to the system?Adding super-classes ‘rounded’ out profilesOntology gave a consistent conceptual model to usersOntology users had more interesting recommendations
Does using domain knowledge compensate for the reduced accuracy of the multi-class classifier?
Classifier accuracy was lower than a typical binary classifierWhen wrong, k-NN chose a topic in a related areaRecommendations best for reading around an area
Experimental evaluationTwo trials, 24 and 14 users, 1.5 months each trial Evaluate use of an is-a hierarchy and dynamic flat-list
Foxtrot seminar 18.1.2002
• Issues arising from our empirical evaluation
Capturing knowledge of user preferenceswith recommender systems
How does Quickstep compare to other recommender systems?There is a lack of trials with real usersThere is no standard metric to measure ‘usefulness’Performance compared reasonably with other systems
Work published in the K-CAP2001 conferencehttp://sern.ucalgary.ca/ksi/K-CAP/K-CAP2001/
Is the recommender system useful as a workplace tool?About 10% of recommendations led to good jumpsUsers felt system was moderately usefulTopic classes were too broad for some users
Foxtrot seminar 18.1.2002
• Foxtrot architecture and approach
Capturing knowledge of user preferenceswith recommender systems
Ontology and training set96 classes, based on CORA paper database hierarchy5-10 example papers per class (714 training examples)
Searchable database of papersTitle, content, topic, quality and date search supportedHTML support in addition to PS,PDF and zip,gz,Z
Profile visualizationUsers can provide explicit feedback on their interest profile
More collaborative recommendationQuality feedback used to rank recommendationsPearson r correlation to find similar users
Foxtrot seminar 18.1.2002
Experiment currently runningRun over this academic yearAll 3rd and 4th year UG’s, staff and PG’s can use Foxtrot70+ registered users15,000+ research papersTwo groups, random subject selection
One group can provide explicit profile feedbackOne group cannot (just relevance feedback)
• Foxtrot empirical evaluation
Capturing knowledge of user preferenceswith recommender systems
Sign up!Just email me with your username and I will register [email protected]
Foxtrot seminar 18.1.2002
• Future work
Capturing knowledge of user preferenceswith recommender systems
Foxtrot experimentFull results in July, written up in a journal articleWill also appear in my Thesis
Profile algorithm analysis on log dataRun profile algorithms on 1 year’s worth of URL logsLog data could become an IAM resource
Short paper for WWW conference with HarithLooking at synergies between Quickstep and COPCould result in a full paper
Foxtrot seminar 18.1.2002