Crowdsourcing: Achieving Data Quality with Impefect Humans

Panos Ipeirotis

New York University & oDesk

Joint work with: Jing Wang, Foster Provost, Josh Attenberg, and Victor Sheng; Special thanks to AdSafe Media

Twitter: @ipeirotis

“A Computer Scientist in a Business School”http://behind-the-enemy-lines.com

Brand advertising not fully embraced Internet advertising yet…

Afraid of improper brand placement

3 Gabrielle Giffords Shooting, Tucson, AZ, Jan 2011Gabrielle Giffords Shooting, Tucson, AZ, Jan 2011

4

5

New Classification Models Needed within days

Pharmaceutical firm does not want ads to appear:– In pages that discuss swine flu (FDA prohibited pharmaceutical

company to display drug ad in pages about swine flu)

Big fast-food chain does not want ads to appear:– In pages that discuss the brand (99% negative sentiment)– In pages discussing obesity, diabetes, cholesterol, etc

Airline company does not want ads to appear:– In pages with crashes, accidents, …– In pages with discussions of terrorist plots against airlines

6

7

Need to build models fast

Traditionally, modeling teams have invested substantial internal resources in data collection, extraction, cleaning, and other preprocessing

No time for such things… However, now, we can outsource preprocessing tasks,

such as labeling, feature extraction, verifying information extraction, etc.

– using Mechanical Turk, oDesk, etc.– quality may be lower than expert labeling (much?) – but low costs can allow massive scale

Amazon Mechanical Turk

Example: Build an “Adult Web Site” Classifier

Need a large number of hand-labeled sites Get people to look at sites and classify them as:G (general audience) PG (parental guidance) R (restricted) X (porn)

Cost/Speed Statistics Undergrad intern: 200 websites/hr, cost: $15/hr Mechanical Turk: 2500 websites/hr, cost: $12/hr

Bad news: Spammers!

Worker ATAMRO447HWJQ

labeled X (porn) sites as G (general audience)

Redundant votes, infer quality

Look at our lazy friend ATAMRO447HWJQtogether with other 9 workers

Using redundancy, we can compute error rates for each worker

1. Initialize“correct” label for each object (e.g., use majority vote)2. Estimate error rates for workers (using “correct” labels)3. Estimate “correct” labels (using error rates, weight worker

votes according to quality)4. Go to Step 2 and iterate until convergence

Algorithm of (Dawid & Skene, 1979) [and many recent variations on the same theme]

Iterative process to estimate worker error rates

Our friend ATAMRO447HWJQmarked almost all sites as G.

Clickety clickey click…

Error rates for ATAMRO447HWJQP[G → G]=99.947% P[G → X]=0.053%P[X → G]=99.153% P[X → X]=0.847%

Challenge: From Confusion Matrixes to Quality Scores

How to check if a worker is a spammer using the confusion matrix?(hint: error rate not enough)

Confusion Matrix for ATAMRO447HWJQ P[X → X]=0.847% P[X → G]=99.153% P[G → X]=0.053% P[G → G]=99.947%

Challenge 1: Spammers are lazy and smart!

Confusion matrix for spammer P[X → X]=0% P[X → G]=100% P[G → X]=0% P[G → G]=100%

Confusion matrix for good worker P[X → X]=80% P[X → G]=20% P[G → X]=20% P[G → G]=80%

Spammers figure out how to fly under the radar…

In reality, we have 85% G sites and 15% X sites

Error rate of spammer = 0% * 85% + 100% * 15% = 15% Error rate of good worker = 85% * 20% + 85% * 20% = 20%

False negatives: Spam workers pass as legitimate

Challenge 2: Humans are biased!

Error rates for CEO of AdSafeP[G → G]=20.0% P[G → P]=80.0% P[G → R]=0.0% P[G → X]=0.0%P[P → G]=0.0% P[P → P]=0.0% P[P → R]=100.0% P[P → X]=0.0%P[R → G]=0.0% P[R → P]=0.0% P[R → R]=100.0% P[R → X]=0.0%P[X → G]=0.0% P[X → P]=0.0% P[X → R]=0.0% P[X → X]=100.0%

We have 85% G sites, 5% P sites, 5% R sites, 5% X sites

Error rate of spammer (all G) = 0% * 85% + 100% * 15% = 15% Error rate of biased worker = 80% * 85% + 100% * 5% = 73%

False positives: Legitimate workers appear to be spammers(important note: bias is not just a matter of “ordered” classes)

Solution: Reverse errors first, compute error rate afterwards

When biased worker says G, it is 100% G When biased worker says P, it is 100% G When biased worker says R, it is 50% P, 50% R When biased worker says X, it is 100% X

Small ambiguity for “R-rated” votes but other than that, fine!

Error Rates for CEO of AdSafeP[G → G]=20.0% P[G → P]=80.0% P[G → R]=0.0% P[G → X]=0.0%P[P → G]=0.0% P[P → P]=0.0% P[P → R]=100.0% P[P → X]=0.0%P[R → G]=0.0% P[R → P]=0.0% P[R → R]=100.0% P[R → X]=0.0%P[X → G]=0.0% P[X → P]=0.0% P[X → R]=0.0% P[X → X]=100.0%

When spammer says G, it is 25% G, 25% P, 25% R, 25% X When spammer says P, it is 25% G, 25% P, 25% R, 25% X When spammer says R, it is 25% G, 25% P, 25% R, 25% X When spammer says X, it is 25% G, 25% P, 25% R, 25% X[note: assume equal priors]

The results are highly ambiguous. No information provided!

Error Rates for spammer: ATAMRO447HWJQP[G → G]=100.0% P[G → P]=0.0% P[G → R]=0.0% P[G → X]=0.0%P[P → G]=100.0% P[P → P]=0.0% P[P → R]=0.0% P[P → X]=0.0%P[R → G]=100.0% P[R → P]=0.0% P[R → R]=0.0% P[R → X]=0.0%P[X → G]=100.0% P[X → P]=0.0% P[X → R]=0.0% P[X → X]=0.0%

Solution: Reverse errors first, compute error rate afterwards

[***Assume misclassification cost equal to 1, solution generalizes]

• High cost: probability spread across classes• Low cost: “probability mass concentrated in one class

Assigned Label Corresponding “Soft” Label ExpectedLabel Cost

Spammer: G <G: 25%, P: 25%, R: 25%, X: 25%> 0.75

Good worker: P <G: 100%, P: 0%, R: 0%, X: 0%> 0.0

Expected Misclassification Cost

Quality Score

• A spammer is a worker who always assigns labels randomly, regardless of what the true class is.

• Scalar score, useful for the purpose of ranking workers• Threshold-ing rewards gives wrong incentives:

• Decent (but still useful) workers get fired• Uncertainty near the decision threshold

)Spammer()Worker(1)Worker(

CostCostQuality

Quality Score: A scalar measure of quality

HCOMP 2010

• Threshold-ing rewards gives wrong incentives: • Decent (but still useful) workers get fired• Uncertainty near the decision threshold

• Instead: Estimate payment level based on quality• Set acceptable quality (e.g., 99% accuracy)• For workers above quality specs: Pay full price• For others: Estimate level of redundancy to reach

acceptable quality (e.g., Need 5 workers with 90% accuracy or 13 workers with 80% accuracy to reach 99% accuracy;)

• Pay full price divided by level of redundancy

Instead of blocking: Quality-sensitive Payment

Example: Build an “Adult Web Site” Classifier

Get people to look at sites and classify them as:G (general audience) PG (parental guidance) R (restricted) X (porn)

But we are not going to label the whole Internet…ExpensiveSlow

22

40

50

60

70

80

90

100

1 20 40 60 80 100

120

140

160

180

200

220

240

260

280

300

AUC

Number of examples ("Mushroom" data set)

Quality and Classification Performance

Noisy labels lead to degraded task performanceLabeling quality increases classification quality increases

Quality = 50%

Quality = 60%

Quality = 80%

Quality = 100%

Single-labeler quality (probability of assigning correctly a binary label)

23

Tradeoffs: More data or better data?

Get more examples Improve classification Get more labels Improve label quality Improve classification

40

50

60

70

80

90

100

1 20 40 60 80 100

120

140

160

180

200

220

240

260

280

300

Number of examples (Mushroom)

Accu

racy

Quality = 50%

Quality = 60%

Quality = 80 %

Quality = 100%

KDD 2008, Best paper runner-up

24

Summary of Basic Results

We want to follow the direction that has the highest “learning gradient”– Estimate improvement with more data (cross-validation)

– Estimate sensitivity to data quality (introduce noise and measure degradation in quality)

Rule-of-thumb results:With high quality labelers (85% and above): Get more data (One worker per example)

With low quality labelers (~60-70%):Improve quality (Multiple workers per example)

25

Selective Repeated-Labeling

We do not need to label everything the same way

Key observation: we have additional information to guide selection of data for repeated labeling the current multiset of labels

the current model built from the data

Example: {+,-,+,-,-,+} vs. {+,+,+,+,+,+}– Will skip details in the talk, see “Repeated Labeling” paper,

for targeting using item difficulty, and other techniques

Improving worker participation

With just labeling, workers are passivelylabeling the data that we give them

But this can be wasteful when positive cases are sparse

Why not asking the workers to search themselves and find training data

26

27

Guided Learning

Ask workers to findexample web pages (great for “sparse” content)

After collecting enough examples, easy to build and test web page classifier http://url-collector.appspot.com/allTopics.jsp

KDD 2009

28

Limits of Guided Learning

No incentives for workers to find “new” content

After a while, submitted web pages similar to already submitted ones

No improvement for classifier

29

The result? Blissful ignorance…

Classifier seems great: Cross-validation tests show excellent performance

Alas, classifier fails: The “unknown unknowns” ™No similar training data in training set

“Unknown unknowns” = classifier fails with high confidence

30

Beat the Machine!

Ask humans to find URLs that the classifier will classify incorrectly another human will classify correctly

Example: Find hate speech pages that the machine will classify as benign

http://adsafe-beatthemachine.appspot.com/

31

Probes Successes

Error rate for probes significantly higherthan error rate on (stratified) random data (10x to 100x higher than base error rate)

32

Structure of Successful Probes

Now, we identify errors much faster (and proactively)

Errors not random outliers: We can “learn” the errors

Could not, however, incorporate errors into existing classifier without degrading performance

33

Unknown unknowns Known unknowns

Once humans find the holes, they keep probing (e.g., multilingual porn )

However, we can learn what we do not know (“unknown unknowns” “known unknowns”)

We now know the areas where we are likely to be wrong

34

Reward Structure for Humans

High reward higher when:– Classifier confident (but wrong) and – We do not know it will be an error

Medium reward when:– Classifier confident (but wrong) and – We do know it will be an error

Low reward when:– Classifier already uncertain about outcome

Workers reacting to bad rewards/scores

Score-based feedback leads to strange interactions:

The “angry, has-been-burnt-too-many-times” worker: “F*** YOU! I am doing everything correctly and you know

it! Stop trying to reject me with your stupid ‘scores’!”

The overachiever worker: “What am I doing wrong?? My score is 92% and I want to

have 100%”

35

National Academy of Sciences Dec 2010 “Frontiers of Science” conference

36

Your workers behave like my mice!

An unexpected connection…

37

Your workers behave like my mice!

Eh?

38

Your workers want to use only their motor skills,not their cognitive skills

The Biology Fundamentals

39

Brain functions are biologically expensive (20% of total energy consumption in humans)

Motor skills are more energy efficient than cognitive skills (e.g., walking)

Brain tends to delegate easy tasks to part of the neural system that handles motor skills

An unexpected connection at theNAS “Frontiers of Science” conf.

40

Your workers want to use only their motor skills,not their cognitive skills

Makes sense

An unexpected connection at theNAS “Frontiers of Science” conf.

41

And here is how I train my miceto behave…

The Mice Experiment

CognitiveSolve mazeFind pellet

42

MotorPush lever three times

Pellet drops

How to Train the Mice?

43

Confuse motor skills!Reward cognition!

I should try this the moment that I get back to my room

Punishing Worker’s Motor Skills

Punish bad answers with frustration of motor skills (e.g., add delays between tasks)– “Loading image, please wait…”– “Image did not load, press here to reload”– “404 error. Return the HIT and accept again”

→Make this probabilistic to keep feedback implicit

44

45

Rewarding (?) Cognitive Effort

Reward good answers by rewarding the cognitive part of the brain – Introduce variety– Introduce novelty– Give new tasks fast– Show score improvements faster (but not the opposite)

– Show optimistic score estimates

46

Experiments

Web page classification Image tagging Email & URL collection

47

Experimental Summary (I)

Spammer workers quickly abandon– No need to display scores, or ban– Low quality submissions from ~60% to ~3%– Half-life of low-quality from 100+ HITs to less than 5

Good workers unaffected– No significant effect on participation of workers with

good performance– Lifetime of participants unaffected– Reduction in response time (after removing the

“intervention delays”; that was puzzling)48

Experimental Summary (II)

Remember, scheme was for training the mice…

15%-20% of the spammers start submitting good work!

????

49

Two key questions

Why response time was slower for some good workers?

Why some low quality workers start working well?

????

50

System 1: “Automatic” actions

System 2:“Intelligent” actions

51

System 1 Tasks

52

System 2 Tasks

53

54

Status: Usage of System 1

(“Automatic”)

Status: Usage of System 2

(“Intelligent”)

Not Performing Well?Disrupt and Engage

System 2

Performing Well?Check if System 1

can Handle, removeSystem 2 stimuli

Performing Well?

Not Performing Well?Hell/Slow ban

Out

Thanks!

Q & A?