a brief introduction to crowdsourcing for data collection

A BRIEF

INTRODUCTION TO

CROWDSOURCED

DATA COLLECTION

ELENA SIMPERL

UNIVERSITY OF SOUTHAMPTON

25-May-14

USEWOD@ESWC2014 1

EXECUTIVE SUMMARY

• Crowdsourcing can help

with research data forensics

• But

• There are things computers do better

than humans hybrid approaches are

the ultimate solution

• There is crowdsourcing and

crowdsourcing pick your faves and

mix them

• Human intelligence is a valuable

resource experiment design is key

2

CROWDSOURCING:

PROBLEM SOLVING VIA

OPEN CALLS

"Simply defined, crowdsourcing represents the act of a

company or institution taking a function once performed by

employees and outsourcing it to an undefined (and generally

large) network of people in the form of an open call. This can

take the form of peer-production (when the job is performed

collaboratively), but is also often undertaken by sole

individuals. The crucial prerequisite is the use of the open

call format and the large network of potential

.“

[Howe, 2006]

25-May-14

3

CROWDSOURCING

COMES IN DIFFERENT

FORMS AND FLAVORS

25-May-14

4

25-May-14

5

DIMENSIONS OF CROWDSOURCING

DIMENSIONS OF CROWDSOURCING

WHAT IS

OUTSOURCED

• Tasks based on

human skills not

easily replicable by

machines

• Visual recognition

• Language

understanding

• Knowledge acquisition

• Basic human

communication

• ...

WHO IS THE CROWD

• Open call (crowd accessible through a platform)

• Call may target specific skills and expertise (qualification tests)

• Requester typically knows less about the ‘workers’ than in other ‘work’ environments

25-May-14

6

See also [Quinn & Bederson, 2012]

USEWOD EXPERIMENT: TASK AND CROWD

WHAT IS

OUTSOURCED

• Annotating research papers with

data set information.

• Alternative representations of the

domain

• Bibliographic reference

• Abstract + title

• Paragraph

• Full paper

• What if the domain is not known in

advance or is infinite?

• Do we know the list of potential

answers?

• Is there only one correct solution to

each atomic task?

• How many people would solve the

same task?

WHO IS THE CROWD

• People who know the papers or the data sets

• Experts in the (broader ) field

• Casual gamers

• Librarians

• Anyone (knowledgeable of English, with a computer/cell phone…)

• Combinations thereof…

25-May-14

7

25-May-14

Tutorial@ISWC2013

CROWDSOURCING AS

‚HUMAN COMPUTATION‘

Outsourcing tasks that machines find difficult to solve

to humans

8

DIMENSIONS OF CROWDSOURCING (2)

HOW IS THE TASK OUTSOURCED

• Explicit vs. implicit participation

• Tasks broken down into smaller units

undertaken in parallel by different people

• Coordination required to handle cases with

more complex workflows

• Partial or independent answers consolidated

and aggregated into complete solution

25-May-14

9

See also [Quinn & Bederson, 2012]

EXAMPLE: CITIZEN SCIENCE

WHAT IS OUTSOURCED

• Object recognition, labeling,

categorization in media content

WHO IS THE CROWD

• Anyone

HOW IS THE TASK

OUTSOURCED

• Highly parallelizable tasks

• Every item is handled by multiple

annotators

• Every annotator provides an answer

• Consolidated answers solve scientific

problems

25-May-14

10

Users aware of how their

input contributes to the

achievement of

application’s goal (and

identify themselves with it)

vs.

Tasks are hidden behind

the application narratives.

Engagement ensured

through other incentives

25-May-14

11

EXPLICIT VS. IMPLICIT

CONTRIBUTION - AFFECTS

MOTIVATION AND ENGAGEMENT

USEWOD EXPERIMENT: TASK

DESIGN

HOW IS THE TASK OUTSOURCED:

ALTERNATIVE MODELS

• Use the data collected here to train a IE algorithm

• Use paid microtask workers to go a first screening, then expert crowd to sort out challenging cases

• What if you have very long documents potentially mentioning different/unknown data sets?

• Competition via Twitter

• ‘Which version of DBpedia does this paper use?’

• One question a day, prizes

• Needs golden standard to bootstrap and redundancy

• Involve the authors

• Use crowdsourcing to find out Twitter accounts, then launch campaign on Twitter

• Write an email to the authors…

• Change the task

• Which papers use Dbpedia 3.X? • Competition to find all papers

25-May-14

12

EXAMPLE: SOYLENT AND COMPLEX

WORKFLOWS

25-May-14

13

http://www.youtube.com/watch?v=n_miZqsPwsc

WHAT IS OUTSOURCED

• Text shortening, proof-

reading, open editing

WHO IS THE CROWD

• MTurk

HOW IS THE TASK

OUTSOURCED

• Text divided into paragraphs

• Select-fix-verify pattern

• Multiple workers in each step

See also [Bernstein et al., 2010]

DIMENSIONS OF CROWDSOURCING (3)

HOW ARE THE

RESULTS VALIDATED

• Solutions space closed vs. open

• Performance measurements/ground truth

• Statistical techniques employed to predict accurate solutions

• May take into account confidence values of algorithmically generated solutions

HOW CAN THE

PROCESS BE

OPTIMIZED

• Incentives and motivators

• Assigning tasks to people based on their skills and performance (as opposed to random assignments)

• Symbiotic combinations of human- and machine-driven computation, including combinations of different forms of crowdsourcing

25-May-14

14

See also [Quinn & Bederson, 2012]

USEWOD EXPERIMENT:

VALIDATION

• Domain is fairly restricted

• Spam and obvious wrong answers can be detected easily

• When are two answers the same? Can there be more

than one correct answer per question?

• Redundancy may not be the final answer

• Most people will be able to identify the data set, but

sometimes the actual version is not trivial to reproduce

• Make educated version guess based on time intervals

and other features

25-May-14

15

ALIGNING INCENTIVES

IS ESSENTIAL

Motivation: driving force that

makes humans achieve their

goals

Incentives: ‘rewards’ assigned

by an external ‘judge’ to a

performer for undertaking a

specific task

• Common belief (among

economists): incentives can be

translated into a sum of money

for all practical purposes.

Incentives can be related to

both extrinsic and intrinsic

motivations.

Extrinsic motivation if task is

considered boring, dangerous,

useless, socially undesirable,

dislikable by the performer.

Intrinsic motivation is driven by

an interest or enjoyment in the

task itself.

16

EXAMPLE: DIFFERENT

CROWDS FOR DIFFERENT

TASKS

Contest

Linked Data experts

Difficult task

Final prize

Find Verify

Microtasks

Workers

Easy task

Micropayments

TripleCheckMate [Kontoskostas2013]

MTurk

Adapted from [Bernstein2010] http://mturk.com

See also [Acosta et al., 2013]

17

IT‘S NOT ALWAYS

JUST ABOUT MONEY

25-May-14

18

http://www.crowdsourcing.org/editorial/how-to-motivate-the-crowd-infographic/

http://www.oneskyapp.com/blog/tips-to-motivate-participants-of-crowdsourced-

translation/

[Kaufmann, Schulze, Viet, 2011]

USEWOD EXPERIMENT:

OTHER INCENTIVES

MODELS

• Twitter-based contest

• ‘Which version of DBpedia does this paper use?’

• One question a day, prizes

• If question is not answered correctly, increase the prize

• If low participation, re-focus the audience or change the

incentive.

• Altruism: for each ten papers annotated we send a

student to ESWC…

25-May-14

19

PRICING ON MTURK: AFFORDABLE,

BUT SCALE OF EXPERIMENTS

DOES MATTER

25-May-14

20

[Ipeirotis, 2008]

USEWOD EXPERIMENT:

HYBRID APPROACH

• Use IE algorithm to select best candidates

• Use different types of crowds

• Publish results as Linked Data

25-May-14

21

See also [Demartini et al., 2012]

25-May-14

22

SUMMARY

SUMMARY AND FINAL

REMARKS

• There are things computers do

better than humans hybrid

approaches are the ultimate

solution

• There is crowdsourcing and

crowdsourcing pick your faves

and mix them

• Human intelligence is a valuable

resource experiment design is

key

25-May-14

23