1

ZenCrowd: Leveraging Probabilistic Reasoning andCrowdsourcing Techniques for Large-Scale Entity Linking

by Gianluca Demartini, Djellel Eddine Difallah, and Philippe Cudré-Mauroux

eXascale InfolabU. of Fribourg—Switzerland

{firstname.lastname}@unifr.ch

Pick-A-Crowd: Tell Me What You Like,and I’ll Tell You What to Do

byDjellel Eddine Difallah, Gianluca Demartini, and Philippe Cudré-Mauroux

eXascale InfolabU. of Fribourg—Switzerland

Presented by:Muhammad Nuruddin,student ID: 2961230,

email address: nuruddin@L3S.de,Internet Technologies and Information Systems(ITIS),

M.Sc. 4th SemesterLeibniz Universität Hannover

Course details:Advanced Methods of Information Retrieval

By: Dr. Elena Demidovaleibniz universität hannover

Presentation on the papers

2

Entity Linking

Entity linking Algorithm

( Probabilistic Reasoning based )

Entity Linking:A suggested way to automate the construction of a semantic web

Example: Wikipedia provide annotated pages

Military

Germany

Pacific Ocean

Historical Incidence

France

4

Crowdsourcing• Obtaining services, ideas, or content by asking contributions

from a large group of people, and especially from an online community.

• Example: - Wikipedia = Wiki + encyclopedia = quick + encyclopedia- IMDB movie top chart.- AMT ( AmazonMechanicalturk )

5

Paper1: ZenCrowd: Leveraging Probabilistic Reasoning andCrowdsourcing Techniques for Large-Scale Entity Linking

Entity linking Algorithm

( Probabilistic Reasoning based )

Crowdsourcing

Improvement between4% and 35%

6

Current techniques of Entity Linking

• Entity Linking is known to be extremely challenging, since parsing and disambiguating natural language text is still extremely difficult for machines.

• The current matching techniques:– Algorithmic Matching: Mostly based on probabilistic reasoning (e.g.

TF-IDF based). Not fully reliable as human manual matching.– Manual Matching: Fully reliable. Costly and time consuming. e.g. New

York Times (NYT) employs a whole team whose sole responsibility is to manually create links from news articles to NYT identifiers.

• This paper represents a step towards bridging the gap between those two classes.

7

System Architecture

The results of algorithmic matching are stored in a probabilistic network:Decision Engine decides:1. If results has very high probability value, it is directly linked to the entity.2. If results have very low confidence value, it is discarded and ignored.3. Promising but uncertain valued entities are passed to Micro-Task Manager

to crowdsource the problem and make a decision.

8

System Architecture

After getting vote from Crowdsourcing platform, all information gathered both from the algorithmic matchers and the crowd are fed into a scalable probabilistic store, and used by their decision engine to process all entities accordingly.

Lets have a look on decision engine’s mechanism to take a decision.

9

Example scenario

CountryJordan

River

Berkeley professor

CountryJordan

Entities

River

After the UNC workshop, Jordan gave a tutorial on nonparametric Bayesian methods.

Worker W1 Worker W2

l1

l2

l3

HTML page doc. 1

C11

Reliability factorPw1()Good, or Bad

Reliability factorPw1()Good, or Bad

C12 C13 C21 C22 C23

plj – Probability of lj computed from algorithmic matches.

pl1

pl2

pl3

LOD cloud

10

Decision Engine uses Factor-Graph

• Factor-Graph can deal with a complicated global problem by viewing it as a factorization of several local functions.

• l1,l2,l3 – 3 candidate entities for a linking.

• plj – Probability of lj computed from algorithmic matches.

• W1,W2 – two workers employed to check these l1,l2,l3 relevancy.

• Pw1, pw2 – worker w1 and w2s reliability factor.

• Lfi() – linking factor, connects li to related clicks (e.g. C11) and workers ( e.g W1).

• Sa1-2() – entities has SameAs link in LOD.

11

Equations used for Linking factor calculation in Factor-Graph

12

Reaching a Decision

• We will find a posterior probability for all the links running probabilistic inference in the network.

• Links with posterior probability > 0.5 are considered to be correct.

13

Updating Priors• As much as entity linkings come to a decision, workers working

profiles get updated.• From the result, workers accuracy of work can be calculated.

Reliability factor of W2Reliability factor of W1

14

EXPERIMENTS

Experimental setup• Collection consists of 25 English news articles• News from CNN.com, NYTimes.com, washingtonpost.com,

timesofindia.indiatimes.com, and swissinfo.com• 489 entities extracted using stanford parser.• Crowdsourcing was performed using Amazon Mturk• 80 distince workers• Precision, Recall and accuracy was measured.

15

Comparison of three matching techniques

16

Observations• A Hybrid model ( Based on both automated and manual

human experts)for entity linking.• 4% to 35% improvement than manually optimized agreement

voting approach.• Average 14% improvement over best automated system.• In both cases, the improvement is statistically signicant (t-test

p < 0.05)• Manual work makes the total annotation work significantly

slow. So there are some questions about time – quality tradeoff.

• They classified workers into {Good, Bad} manually and calculated workers reliability P(w) , but did not mention any relation between these two factors.

17

End of presentation of first paper

18

Paper 2: Pick-A-Crowd: Tell Me What You Like,and I’ll Tell You What to Do

• This paper is about a different Crowdsourcing approach based on push methodology.

• This new push methodology yields better results (29% more efficient) than usual pull strategies. Any worker

can pull any task

Figure: Traditional Crowdsourcing pull strategy

19

Example of traditional approach:

So what’s wrong with this?• Does not care about workers field of expertise.• Not all workers are a good fit for all tasks. tasks requiring background

knowledge is important.• “I had no idea what to answer to most questions...” was a comment of a

worker from AMT (Amazon Mechanical Turk).

Any worker can pull any

task[1]

[1] https://requester.mturk.com/images/graphic_process.png?1403199990

20

So how they are going to improve it?• Ranks/orders the workers according to the type of the work

and skill of the workers and pushes the work to the most suitable workers.

• At first they constructs user models for each workers in the crowd in order to assign HITs ( Human Intelligence Tasks) to the most suitable available worker.

• User model/user profile is built based on his social network usage, his fields of interest.

21

So how this system ranks/orders the workers? Recommender system

• Assigning HITs to workers is similar to the task performed by recommender systems.

• The recommender systems matches HITs (Human Intelligence tasks) to human workers (i.e. users) profiles that describe worker interests and skills.

• Then the system generates a ranking of candidate workers who can do the work better.

22

System Overview

23

Workflow of the system

• Calculate work difficulty:- Every work is different from other works. HIT Difficulty Assessor takes each HIT and determines a complexity score for it.

• Assess Worker skill:- System create workers profile considering his liked pages and previously work experiences.

• Calculate Reward for the work:- As every work is different and every workers ability differs from work to work, Rewards for different works and workers are different. System calculates rewards considering these factors.

• Assign works to top-k suitable candidates:- Recommender system finds k top most suitable candidates and assign (pushes ) the work only these n workers.

24

Calculate work difficulty3 different possible algorithms

1. Text Compare: Compare the textual description of the task with the skill description of each worker and assess the difficulty.

2. LOD(Linked Open Data) entity based: – Each Facebook page liked by the workers can be linked to its respective

LOD entities. – Then the set of entities related to HITs and the set of entities

representing the interests of the crowd can be directly compared. – The task is classified as difficult when the entities involved in the task

heavily differ from the entities liked by the crowd.

3. Machine Learning based: A classifier trained by means of previously completed tasks, their description and their result accuracy. The description of a new task is given as a test vector to the classifier.

25

4 possible way of Reward Estimation

Input: A monetary budget B, HIT hi.

1. Rewarding the same amount of money for each task of the same type.2. Taking into account the difficulty d() of the HIT h.

3. Computing a reward based on both the specific HIT as well as the worker skill who performs it.

4. Game theoretic based approaches to compute the optimal reward for paid crowdsourcing incentives in the presence of workers who collude in order to game the system.

26

Worker Profile Selector

• This module uses the similarity measure that used for matching workers to tasks .

• The entities included in the workers profiles can be considered. • The Facebook categories of their liked pages also plays

significant role.• A generic Similarity measurement equation is

A = set of candidate answers for task hi sim() = similarity between the worker profile and the task description.

• 3 Assignment models for hit ( Human Intelligence task) assignemnt.

27

HIT ASSIGNMENT MODELS• Category-based Assignment Model

– Tasks are assigned according to Facebook pages or page categories. (e.g. Entertainment -> Movie )

– Requestor mentions the category of the task.• Expert Profiling Assignment Model

– Scoring function is based on a voting model.– Voting model is based on no. of pages related to the tasks and no.

of pages user liked and how many are common.• Semantic-Based Assignment Model

– Answers and liked pages are linked to entities and Underlying graph structure is used to measure the distance ( similarity).

– Example SPARQL

28

Example: Expert Finding Voting Model

Figure: An example of the Expert Finding Voting Model. The final ranking identifies worker A as the top worker as he likes the most pages related to the query

29

Any worker can pull any task

Hit Assigner assigns tasks to suitable

workers

Summary of the system

30

Experimental Evaluation

• Experimental Setting:– 170 workers– Overall, more than 12K distinct liked Facebook pages– workers have been recruited via Amazon Mturk.– Task categories: actors, soccer players, anime characters,

movie actors, movie scenes, music bands, and questions related to cricket

– 50 images/category. – Precision, Recall over majority votes obtained over 3 or 5

workers.

31

Figure: Crowd performance on the cricket task. Square points indicate the 5 workers selected by their proposed system.

The best worker performing at 0.9 Precision and 0.9 Recall.

32

Figure: OpenTurk worker Accuracy vs no. of relevant Pages a worker likes.

Observations:More relevant pages in the worker profile (e.g., >30), more accuracy.

33

Table: Average Accuracy for different HIT assignment models assigning each HIT to 3 and 5 workers.

AMT – Amazon Mechanical Turk.Cagegory-based – Category of liked page and category of task based comparison.En. type 3/5 - Entity graph – Entity type in the DBPedia knowledge base assigning each HIT to 3 and 5 workers.Voting Model ti – Voting model based on page text relevant to the task.Voting Model Ai – Voting model based on all possible answer based similarity.1-step – Considers directly related entities within one step in the graph.

•Result is based on 320 questions.

• Voting Model ti achieves 29% relative improvement over the best accuracy obtained by the AMT model

34

Observations

• May lead to longer task completion times

• Real – time annotation is not possible.

• But obtaining high-quality answers is more important rather than getting real-time data in most of the cases.

35

Thank you!

Any Question?