social participation: how collective activity can make change
Post on 30-Oct-2014
1.362 Views
Preview:
DESCRIPTION
TRANSCRIPT
Social Participation: How Collective Activity Can Make Change
Benjamin B. Bederson
Computer Science DepartmentHuman-Computer Interaction LabInstitute for Advanced Computer Studies
University of Maryland
www.cs.umd.edu/~bederson @bederson
Wednesday, October 3, 12
Approach
Let people collaborate with computersin a way that can be aggregated to provide value to a greater goal
Human Computation=> sometimes called
Blog commentingQ&A SitesTwitter & Facebook
Wednesday, October 3, 12
Approach
ThingsHUMANS
can do
ThingsCOMPUTERS
can do
TranslationPhoto tagging
Face recognitionHuman detection
Speech recognitionText analysis
Planning
Human Computation
Let people collaborate with computersin a way that can be aggregated to provide value to a greater goal
Wednesday, October 3, 12
Human Computation Taxonomy
SocialComputing
Data Mining
Collective Intelligence
Crowdsourcing
HumanComputation
Replace humans with humans
Replace computers with humans
Wednesday, October 3, 12
Trade-off Space
Quality
Spee
d, A
fford
abilit
y
Computers
Human Workers(traditional)
Human Computation
Wednesday, October 3, 12
Online Labor Markets
Wednesday, October 3, 12
Active Area
VizWiz - www.vizwiz.orgWednesday, October 3, 12
Active Area
Color Name Models - vis.stanford.edu/color-names/Wednesday, October 3, 12
A Real-World Problem: ICDL Now:
– 4,386 books– 54 languages– Translations in a few languages– 100K unique visitors/month
Goal:10,000 books100 languagesEvery book in every language!
www.childrenslibrary.orgWednesday, October 3, 12
Quality
Spee
d / A
fford
abilit
yMachineTransla,on
Professional0Bilingual0Human0Par,cipa,on
Amateur0Bilingual0Human0Par,cipa,on
Wednesday, October 3, 12
The key idea
Wednesday, October 3, 12
Translation with the Crowd
vs. 1,200,000 contributors Wikipedia: 900 translators
Translate with the Monolingual Crowd
Wednesday, October 3, 12
Quality
Spee
d / A
fford
abilit
yMachineTranslation
Professional Bilingual Human Participation
Amateur Bilingual Human Participation
MonolingualHumanPar,cipa,on
Philip ResnikLinguistics
Chang Hu, Ph.D.=> Microsoft
Wednesday, October 3, 12
The Protocol
Machine Translation
Machine Back-Translation
Stopping Condition
Source Sentence
Best Translation
Annotation Projection
Annotation Projection
Wednesday, October 3, 12
La0sonda0...
Wednesday, October 3, 12
Wednesday, October 3, 12
Google: 10% of sentences MonoTrans2: 68% of sentences
24
0"
20"
40"
60"
80"
100"
120"
140"
160"
1=None" 2" 3" 4" 5=All"
#"of"Sen
tences"
Sentences"per"Accuracy"Category"
Google"
MonoTrans2"
Accuracy
Wednesday, October 3, 12
Too many sentencesToo many tasks
MonoTrans2 UI ComplexityHard to get casual users
Wednesday, October 3, 12
MonoTrans Widgets
Too many sentences => One sentence at a time
Too many tasks => One task at a time
Wednesday, October 3, 12
27Wednesday, October 3, 12
28Wednesday, October 3, 12
Julio0say0that0the0mouse0needed0the0tooth0for0his0secret0formula.
Julio0said0that0the0mouse0needed0the0tooth0for0his0secret0formula.
29Wednesday, October 3, 12
DeploymentAug 2011 – May 2012
11 books (6 languages)
1,282 sentences
155,000 visits
31,000 tasks
Wednesday, October 3, 12
English
Widgets Evaluation
14 days
54 sentences
739 users2 books
Spanish
Wednesday, October 3, 12
Google: 31% of sentencesWidgets: 52% of sentences
0"
10"
20"
30"
40"
50"
1=None" 2" 3" 4" 5=All"
#"of"Sen
tences"
Sentences"per"Accuracy"Category"
Google"
Widgets"
Accuracy
Wednesday, October 3, 12
How0could0a0crowdhelp0you0make0an
importantdecision?
Alex QuinnCS Ph.D. student
Wednesday, October 3, 12
Wednesday, October 3, 12
AskSheet Example Problems
Pick a grad school
Where to buy groceries?
Where/when go on vacation?
Which papers should be accepted?
Wednesday, October 3, 12
Decision modeling process
Define the Problem
Develop a Model
Acquire Input Data
Develop a Solution
Test the Solution
Wednesday, October 3, 12
Decision modeling process
Acquire Input DataResearch focus
Frugal input acquisition
Wednesday, October 3, 12
Create a blank decision model with ASK formulas in cells C2:E6. AskSheet highlights priority cells in dark green.
Create Model
Wednesday, October 3, 12
Set up the task by specifying who will receive the HITs, instructions, and other details. The "Prioritization" slider controls how many inputs to include in each batch.
Configure Task
Wednesday, October 3, 12
At the bottom of the setup screen is a dynamically updated preview of what workers will see. The input form is automatically generated from the spreadsheet model.
Preview Task
Wednesday, October 3, 12
The system prioritizes the inputs. Obtaining the dark green cells first would provide the greatest opportunity to eliminate other cells. Braces expressions show the possible range of output values for each cell.
Data is Collected
Wednesday, October 3, 12
The model is now complete. Store C wins because its maximum possible price is $68, is lower than the minimum possible price of either Store A ($85) or Store B. ($71). The system avoided requesting 11 of the 27 inputs.
Task Completed
Wednesday, October 3, 12
“ASK” formula
A1: =ASK(1, 1, 5)
B1: =ASK(1, 1, 10)
C1: =IF(A1 < B1, "A", "B")
cost, min, max
Wednesday, October 3, 12
“ASK” formula
A1: =ASK(1, 1, 5)B1: =ASK(1, 1, 10)
C1: =IF(A1 < B1, "A", "B")
A1
B1
1 2 3 4 5 6 7 8 9 10
Task Prioritization
Wednesday, October 3, 12
AskSheet Trial
Answer rcvd in < 9 hrs
Eliminated 74% of work
Task: Find pediatrician that is rated well, accepted by insurance, and sufficiently close
Wednesday, October 3, 12
Open Problems
•Evaluation of student work in MOOCs
•Aggregation of comments on blogs / class forums
•Assuring quality of work (i.e., product reviews)
Wednesday, October 3, 12
Design Lessons
Assuring quality at a viable cost remains hard•Assume adversarial participants. •Best practices: redundancy, gold standards, inform you
are watching, treat workers as employees
The "motivation" problem remains hard•Don’t forget "what's in it for me".
Make it fundamental to primary task
Ethics
Wednesday, October 3, 12
Ethics of Human Computation
$$$ Anonymity
Wednesday, October 3, 12
• Bulk of problems are money/qualityrelated
• Workers complain about– Low wages–Not getting paid– Slow payment
• Requesters (who are less anonymous) complain about– Low quality work
• Also, significant other issues – Decontextualization– Tasks that are illegal or unacceptable– Privacy / anonymity
Money Problems
Wednesday, October 3, 12
Current Situation• Workers and Requesters alike build their own
reputation/quality mgmt systems
–Turker Nation
Wednesday, October 3, 12
Current Situation• Workers and Requesters alike build their own
reputation/quality mgmt systems
–Turker Nation–Turkopticon
Wednesday, October 3, 12
Current Situation• Workers and Requesters alike build their own
reputation/quality mgmt systems
–Turker Nation
–CrowdFlower–Turkopticon
Wednesday, October 3, 12
Questions?
www.cs.umd.edu/~bederson @bederson
MonoTrans0supported0by0Na,onal0Science0Founda,onMonoTrans0&0SearchParty0supported0by0Google
Wednesday, October 3, 12
top related