Slide 1

Crowdsourcing Quality Management and other stories Panos Ipeirotis New York University & Tagasauris

Slide 2

Amazon Mechanical Turk

Slide 3

patent pending Tagging

Slide 4

patent pending But what if MTurk workers submit incorrect tags?

Slide 5

Voting

Slide 6

But what if MTurk workers also vote incorrectly?

Slide 7

Voting But what if MTurk workers also vote incorrectly? Redundancy: 5+ votes per tag

Slide 8

But my task is not just right or wrong Creation HIT (e.g. describe the image) Improve HIT (e.g. improve description) Improve HIT (e.g. improve description) Compare HIT (voting) Which is better? Compare HIT (voting) Which is better? Describe this

Slide 9

version 1: A parial view of a pocket calculator together with some coins and a pen. version 2: A view of personal items a calculator, and some gold and copper coins, and a round tip pen, these are all pocket and wallet sized item used for business, writting, calculating prices or solving math problems and purchasing items. version 3: A close-up photograph of the following items: A CASIO multi- function calculator. A ball point pen, uncapped. Various coins, apparently European, both copper and gold. Seems to be a theme illustration for a brochure or document cover treating finance, probably personal finance. version 4: Various British coins; two of 1 value, three of 20p value and one of 1p value. version 8: A close-up photograph of the following items: A CASIO multi-function, solar powered scientific calculator. A blue ball point pen with a blue rubber grip and the tip extended. Six British coins; two of 1 value, three of 20p value and one of 1p value. Seems to be a theme illustration for a brochure or document cover treating finance - probably personal finance."

Slide 10

New York: Old and New http://newyorkoldandnew.blogspot.com/ An engagement experiment: Search the NYPL digital library for an old photo of New York, and then find a photo of the same location, today

Slide 11

1935

Slide 12

2007

Slide 13

1933

Slide 14

2007

Slide 15

1890

Slide 16

2007

Slide 17

1890

Slide 18

1900

Slide 19

1976

Slide 20

2008

Slide 21

The incomplete knowledge graph [Harsen Homestead, 72nd Street, ca. 1888.]

Slide 22

The incomplete knowledge graph Harsen Homestead ???

Slide 23

The incomplete knowledge graph Jacob Harsen, a farmer who started Harsenville, a village spanning 68th Street to 81st Street between Central Park West and the Hudson River. Source: http://www.oneshermansquare.com/HistoryofShermanSquare.aspx

Slide 24

The incomplete knowledge graph Crowd not only annotates historical archives but also contributes back to the knowledge graph [allowing easier annotation in the future]

Slide 25

New York Times, 1911

Slide 26

The incomplete knowledge graph Crowd participation not only annotates historical archives but also contributes back to the knowledge graph [allowing easier annotation in the future]