what's wrong with recruiter-john? a non-trivial recommender challenge
TRANSCRIPT
![Page 1: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/1.jpg)
What's wrong with you,
Recruiter-John? A non-
trivial recommender
challenge.Budapest, June 2016
@fabianabel
http://recsyschallenge.com
![Page 2: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/2.jpg)
Challenge
Given a user, the goal is to recommend job postings…
1. that the user may be interested in and
2. for which the user is an appropriate candidate.
2
Scala Dev(m/w)
ScalaEngineer
Scala Dev, Hamburg
user
job postings
Job
recommende
r
companies
![Page 3: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/3.jpg)
Job recommendations
![Page 4: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/4.jpg)
Job recommendations
![Page 5: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/5.jpg)
5
Title
Company
Employment type
and career level
Full-text
description
Key properties of a job posting
![Page 6: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/6.jpg)
6
Key sources for understanding user demands
Social Network
explicit and
implicit
connections
Profile
Fabian Abel
Data Scientist
Haves:
Interests:
web science
big data, hadoop skills & co.
Interactions
data
web
social media
clicks, shares,
ratings
big data
kununu
Interactions of
similar users
similar usershadoop
scala
![Page 7: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/7.jpg)
7
Relevance Estimation
Social Network
explicit and
implicit
connections
Profile
Fabian Abel
Data Scientist
Haves:
Interests:
web science
big data, hadoop skills & co.
Interactions
data
web
social media
clicks, shares,
ratings
big data
kununu
Interactions of
similar users
similar usershadoop
scala
Content-
based
features
Collaborative
features
Social
features
Usage
behavior
features
Relevance
Estimation(regression model)
Logistic Regression
P(relevant | x) = 1
1 + e -(b0 + bi xi)i
n
feature vector impact of feature xi
![Page 8: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/8.jpg)
8
Relevance Estimation + Additional Filters
Content-
based
features
Collaborative
features
Social
features
Usage
behavior
features
Relevance
Estimation(regression model)
Location-
based
filtering
Content-
based
diversification
Monetary-
based
diversification
Career Level
filtering
Filtering &
Diversification
0.92 0.8 0.76
…
![Page 9: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/9.jpg)
ChallengesIssues that we have to fight with…
9
![Page 10: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/10.jpg)
What John writes…
10
And what he means…
Recruiter-John
International Sales Manager Call Center Agent(10 EUR per hour)
Sales Manager Sales Manager for B2B
customers(80K EUR per year)
Data Scientist skilled in Hadoop,
Scala, Elasticsearch, … with PhD in …
Data Analyst(skilled in SAS or Excel)
![Page 11: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/11.jpg)
What Paul says he is…
11
And what he means…
Paul, the Candidate
CEO Network Engineer(currently unemployed)
BI Engineer(skilled in old-school ETL)
Shopman(in a kiosk)
Data Scientist with 100+ skills
Sales Manager
![Page 12: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/12.jpg)
12
Understanding the meaning of things that people write
in job postings and in their profiles is not trivial…
![Page 13: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/13.jpg)
13
Profiles vs. People’s wishes for
their future
past
past
Profile describes a
user‘s past/current
position(s), not future
wishes
![Page 14: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/14.jpg)
14
Career path patterns: locationsHow far away are the jobs that the users bookmark?
0-50 km
35%
51-200 km
22%
>200 km43%
![Page 15: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/15.jpg)
15
Career path patterns: locationsClimbing up the ladder?
junior
junior
senior
manger
senior manger
Today
Next
ste
p
53%
senior
72%
manger
54%
senior manger
52%
![Page 16: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/16.jpg)
CTR of job recommender over time
16
2014 2015 2016
CTR
New year,
new job
bad CTR
No Love for
job RecSys
Intensified
love for job
RecSys
B/A test: are the changes
in the job RecSys really
responsible for increased
CTRs?
Increase of job inventory
(from 100k to ca. 750k)
Using algorithms
from RecSys
Challenge
- Feedback App
- LSI for MoreLikeThis component
- Entity resolution
- Explanations
- …
Running out of ideas :-)
recsyschallenge.com
![Page 17: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/17.jpg)
RecSys Challengehttp://recsyschallenge.com
17
![Page 18: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/18.jpg)
Challenge
Given a user, the goal is to recommend job postings…
1. that the user may be interested in and
2. for which the user is an appropriate candidate.
18
Scala Dev
(m/w)
Scala
Engineer
Scala Dev,
Hamburg
user
job postings
Job
recommende
r
companies
![Page 19: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/19.jpg)
RecSys Challenge
Given a user, the goal is to predict those job postings that the
user will interact with.
19
Scala
(m/w)
?Scala Dev,
Hamburg
job postings
Scala
Engineer
2 months of impressions & interactions
click
bookmark
![Page 20: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/20.jpg)
Datasets
1. Training data:
• User demographics (jobtitle, discipline, industry, career level, # CV entries,
country, region) [1M]
• Job postings (title, discipline, industry, career level, country region) [1M]
• Interactions (user_id, item_id, interaction_type, timestamp) [10M, 2 months]
• Impressions (user_id, item_id, week) [30M, 2 months]
2. Task files:
• Users (= User IDs for whom recommendations should be computed) [150k]
• Candidate items (= item IDs that are allowed to be recommended) [300k]
3. Solution (secret)
• Interactions (user_id, item_id) [1M, 1 week]
Anonymization (Strings IDs; users and interactions are enriched with
artitificial noise) 20
http://recsyschallenge.com
![Page 21: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/21.jpg)
Evaluation Measure
Mixture of…
- Precision@k (k = 2, 4, 6, 20)= fraction of relevant items in the top k
- Recall@30 = fraction of relevant
items in the top k
- Success@30 = probability that at
least one relevant item was
recommended in the top 30
21
http://recsyschallenge.com
![Page 22: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/22.jpg)
Current Status
22
http://recsyschallenge.com
![Page 23: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/23.jpg)
Current Status (ordered by rank)
23
http://recsyschallenge.com
![Page 24: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/24.jpg)
Join the challenge!
• Deadline for submissions: June 26th 2016
Current leaders: >600k points (ca. 20% of max. possible points)
Prizes: 1st = 3,000 EUR; 2nd = 1,500 EUR, 3rd = 500 EUR
• Workshop at RecSys 2016 in Boston: Sep 15th
• RecSys Challenge 2017:
Dream = online evaluation
24
http://recsyschallenge.com
![Page 25: What's wrong with Recruiter-John? A non-trivial recommender challenge](https://reader031.vdocument.in/reader031/viewer/2022030318/5a6575ee7f8b9af3678b4f4f/html5/thumbnails/25.jpg)
Thank you @fabianabel
http://recsyschallenge.com
www.xing.com