collaborative filtering in map/reduce

Collaborative Filteringin

Map/Reduce

Ole-Martin Mørk - Open AdExchange

tirsdag 14. september 2010

Vision

• Learn that Map/Reduce is simple

• Learn that Map/Reduce may be powerful

• Collaborative Filtering is fun!


Agenda

• Map/Reduce

• Collaborative Filtering

• Collaborative Filtering with Map/Reduce

• Amazon Elastic MapReduce


Map/Reduce


Map/Reduce

• Very scalable algorithm

• Inspirered by map and reduce from functional programming.

• Everything is based on key/value


6 phases

• Reader

• Map

• Partition

• Comparison

• Reduce

• Writer


6 phases

• Reader

•Map

• Partition

• Comparison

•Reduce

• Writer


Map


List(“hello”,“dude”).map{x=>x.substring(0,1)}

functional map


Map/Reduce map

• Input is key/value

• Output is key/value


Simple Example, Map

• Count occurences of words in a document

• Input is: <linenumber>, <content of line>

• For each word on the line, the output is <word>, <count>


Map


Reducetirsdag 14. september 2010

functional reduce

val sum=List(32,40,23).reduceLeft{_+_}


Map/Reduce reduce

• Input is key/list of values

• Output is key/value


Simple Example, Reduce

• Reduce input is <word, counts>

• For each value we increase the count

• Output is <word>, <sum of counts>


Reduce


CollaborativeFiltering


Amazon


Last.fm


Sceneami.com


User based

• Useful when we have

• Small number of users

• High correlation between users

• Data that changes often


Item based

• Useful for big sites like Amazon etc..

• Small overlap between users

• Mostly static data


Min

drø

mm

eapp

likas

jon

Pattern Matching in Scala

Euclidean Distance

Rating

Rating

Match

Match


Euclidean Distance

• Alf‘s presentations:1,25,56,57,58,98 (6)

• Kari’s presentations: 2,25,98,99 (4)

• Equal presentations: 25 and 98 (2)

• Unmatched presentations: 6-2 + 4-2 = 6

• Distance score: 1/1+sqr(6)= 0.29


Recommended sessions

• Me:1,2,5,6,7

• Kate (0.31): 5,6,8,9

• Paul (0.41): 1,2,4,5,6

• Mary(0.31):1,5,8,9



• Me:1,2,5,6,7

• Kate (0.31): 5,6,8,9

• Paul (0.41): 1,2,4,5,6

• Mary(0.31):1,5,8,9

• Recommended: 8 (0.62)



• Me:1,2,5,6,7

• Kate (0.31): 5,6,8,9

• Paul (0.41): 1,2,4,5,6

• Mary(0.31):1,5,8,9

• Recommended: 8 (0.62), 9 (0.62)



• Me:1,2,5,6,7

• Kate (0.31): 5,6,8,9

• Paul (0.41): 1,2,4,5,6

• Mary(0.31):1,5,8,9

• Recommended: 8 (0.62), 9 (0.62), 4 (0.41)


Demo


More Map/Reduce


Several iterations

Iteration 1

Iteration 2

Iteration 3


Several iterations

Iteration 3

Iteration 1 Iteration 2


Partitioning

Reducer Reducer

Jeff

Kate

Mary

Ali

Lea

Paul

Paul Mary Kate Lea Jeff Ali


Comparison

Reducer Reducer

Pres 2

Kate

Pres 2 JeffPres 2

Mary

Pres 1

Paul

Pres 1 AliPres 1

Lea

Pres 1Pres 1Pres 1 Pres 2Pres 2Pres 2Paul Lea Ali Jeff Mary Kate


Guidelines

• Never access external sources during computation.

• Your functions should be small and fast

• You might not have all the data available


Hadoop

• Hadoop is reusing objects, so remember to clone if you plan to keep them.

• You can read and write all objects implementing hadoop.WritableComparable

• write(DataOutput)

• readFields(DataInput)

• compareTo(Object)


Collaborative Filtering, the Map/Reduce way


Overview

• Create an application that recommends JavaZone presentations.

• Overall goal: Scalable performance

• 4 iterations

• Reading input from text file


Iteration 1

• Map input: <user>, <presentations>

• Map output: <presentation>, <user>

• Reduce output: <presentation>, <userList>


Iteration 2

• Map input: <presentation>, <userList>

• Map output: <user>, <userList>

• Reduce input: <user>, <list of userList>

• Reduce output: <userTuplet>, <match count>


Iteration 3

• Map input: <userTuplet>, <match count>

• Map output: <userTuplet>, <diff>

• Map output: <userTuplet reversed>, <diff>

• Reduce output: <user>, <similaruser>


Iteration 4

• Map input: <user>, <similaruser>

• Map output: <user>, <presentation with score>

• Reduce output: <user>, <presentations>


Demo


Map/Reduce on EC2


Elastic Map/Reduce

• Same code

• Same input

• Different configuration


Upload files

s3cmd put oax-jz10:jar/oax-jz10.jar target/oax.jz10.jar

s3cmd.rb put oax-jz10:input/data.txt data.txt


Create job flow

elastic-mapreduce --create --alive --log-uri s3n://oax-jz10/log


Register iterations

elastic-mapreduce --jobflow j-1NLAIW45QUN4B --jar s3n://oax-jz10/jar/oax-jz10.jar --arg com.openadex.pres.iterations.Iteration1 --arg s3n://oax-jz10/input --arg s3n://oax-jz10/output1


Download output

s3cmd.rb get oax-jz10:output4/part-00000 out


Demo


Summary

• Map/Reduce may be simple

• Map/Reduce can be really powerful

• Collaborative filtering is fun :-)


Thank you

Ole-Martin Mø[email protected]/olemartin

del.icio.us/olemartin/jz10

All images are licensed with Creative Commons. See http://bit.ly/mr-photos for details,


http://bit.ly/mr-photos

http://bit.ly/mr-photos

collaborative filtering in map/reduce

Technology

Collaborative Filtering Recommendation System

eecs349 collaborative filtering - GitHub Pages · Collaborative Filtering EECS 349 Machine Learning Bongjun Kim Fall, 2015. What is Collaborative Filtering? •Recommendation system

Collaborative Filtering

Recommender Systems. Collaborative Filtering Process

Collaborative Filtering in iCAMP

Hypothesis-Based Collaborative Filtering

Collaborative Filtering & Recommender Systems

Collaborative filtering intro - Full

Personalized Recommender System Using Entropy Based Collaborative Filtering … · 2013-11-19 · 2. Literature Review of Collaborative Filtering Recommender Systems Collaborative

Socially Collaborative Filtering

Recommender systems using collaborative filtering

Collaborative Filtering Recommender Systemsfaculty.chas.uni.edu/~schafer/publications/CF_AdaptiveWeb_2006.pdf · 1 Introduction Collaborative Filtering is the process of filtering

Collaborative Filtering Introduction Search or Content based Method User-Based Collaborative Filtering Item-to-Item Collaborative Filtering Using

Collaborative Filtering Recommender Systems

Collaborative Filtering - Practical Machine Learning, CS ... › ... › lectures › collaborative › slides.… · What is Collaborative Filtering? Insight: Personal preferences

ITEM BASED COLLABORATIVE FILTERING USING CLOUD COMPUTING ... · item based collaborative filtering using cloud computing ... social portal, ... item based collaborative filtering

Introduction to Collaborative Filtering

DISTRIBUTED COLLABORATIVE FILTERING FOR ROBUST RECOMMENDATION AGAINST SHILLING ATTACKS DISTRIBUTED COLLABORATIVE FILTERING FOR ROBUST RECOMMENDATION AGAINST

H2E: A Privacy Provisioning Framework for Collaborative Filtering … · 2019-09-10 · collaborative filtering, content-based filtering, and hybrid filtering [3]. Content-based filtering,

Tutorial 14 (collaborative filtering)

Comparison of Collaborative Filtering Algorithms

Collaborative Filtering using KNN

Vertical Recommendation Using Collaborative Filtering

User Modeling and Personalization 6: Collaborative ... · Recommender Systems Collaborative Filtering and Personalization Applications of Collaborative Filtering Functionality of

Social network collaborative filtering

CoBaFi : Collaborative Bayesian Filtering

Collaborative Filtering Recommendation

Case Study 4: Collaborative Filtering · Case Study 4: Collaborative Filtering Collaborative Filtering ! Goal: Find movies of interest to a user based on movies watched by the user

Collaborative Filtering with Spark

Boston.rb: Collaborative Filtering

1 Information Filtering Rong Jin. 2 Outline Brief introduction to information filtering Collaborative filtering Adaptive filtering

Product Quantized Collaborative Filtering

Collaborative filtering with CCAM

Collaborative filtering with privacy

Recommendations via Collaborative Filtering