social participation: how collective activity can make change
DESCRIPTION
Keynote talk at South African Institute for Computer Scientists and Information Technologists (SAICSIT) Oct 2012TRANSCRIPT
![Page 1: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/1.jpg)
Social Participation: How Collective Activity Can Make Change
Benjamin B. Bederson
Computer Science DepartmentHuman-Computer Interaction LabInstitute for Advanced Computer Studies
University of Maryland
www.cs.umd.edu/~bederson @bederson
Wednesday, October 3, 12
![Page 2: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/2.jpg)
Approach
Let people collaborate with computersin a way that can be aggregated to provide value to a greater goal
Human Computation=> sometimes called
Blog commentingQ&A SitesTwitter & Facebook
Wednesday, October 3, 12
![Page 3: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/3.jpg)
Approach
ThingsHUMANS
can do
ThingsCOMPUTERS
can do
TranslationPhoto tagging
Face recognitionHuman detection
Speech recognitionText analysis
Planning
Human Computation
Let people collaborate with computersin a way that can be aggregated to provide value to a greater goal
Wednesday, October 3, 12
![Page 4: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/4.jpg)
Human Computation Taxonomy
SocialComputing
Data Mining
Collective Intelligence
Crowdsourcing
HumanComputation
Replace humans with humans
Replace computers with humans
Wednesday, October 3, 12
![Page 5: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/5.jpg)
Trade-off Space
Quality
Spee
d, A
fford
abilit
y
Computers
Human Workers(traditional)
Human Computation
Wednesday, October 3, 12
![Page 6: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/6.jpg)
Online Labor Markets
Wednesday, October 3, 12
![Page 9: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/9.jpg)
Active Area
VizWiz - www.vizwiz.orgWednesday, October 3, 12
![Page 10: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/10.jpg)
Active Area
Color Name Models - vis.stanford.edu/color-names/Wednesday, October 3, 12
![Page 11: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/11.jpg)
A Real-World Problem: ICDL Now:
– 4,386 books– 54 languages– Translations in a few languages– 100K unique visitors/month
Goal:10,000 books100 languagesEvery book in every language!
www.childrenslibrary.orgWednesday, October 3, 12
![Page 12: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/12.jpg)
Quality
Spee
d / A
fford
abilit
yMachineTransla,on
Professional0Bilingual0Human0Par,cipa,on
Amateur0Bilingual0Human0Par,cipa,on
Wednesday, October 3, 12
![Page 13: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/13.jpg)
The key idea
Wednesday, October 3, 12
![Page 14: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/14.jpg)
Translation with the Crowd
vs. 1,200,000 contributors Wikipedia: 900 translators
Translate with the Monolingual Crowd
Wednesday, October 3, 12
![Page 15: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/15.jpg)
Quality
Spee
d / A
fford
abilit
yMachineTranslation
Professional Bilingual Human Participation
Amateur Bilingual Human Participation
MonolingualHumanPar,cipa,on
Philip ResnikLinguistics
Chang Hu, Ph.D.=> Microsoft
Wednesday, October 3, 12
![Page 16: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/16.jpg)
The Protocol
Machine Translation
Machine Back-Translation
Stopping Condition
Source Sentence
Best Translation
Annotation Projection
Annotation Projection
Wednesday, October 3, 12
![Page 17: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/17.jpg)
La0sonda0...
Wednesday, October 3, 12
![Page 18: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/18.jpg)
Wednesday, October 3, 12
![Page 19: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/19.jpg)
Google: 10% of sentences MonoTrans2: 68% of sentences
24
0"
20"
40"
60"
80"
100"
120"
140"
160"
1=None" 2" 3" 4" 5=All"
#"of"Sen
tences"
Sentences"per"Accuracy"Category"
Google"
MonoTrans2"
Accuracy
Wednesday, October 3, 12
![Page 20: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/20.jpg)
Too many sentencesToo many tasks
MonoTrans2 UI ComplexityHard to get casual users
Wednesday, October 3, 12
![Page 21: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/21.jpg)
MonoTrans Widgets
Too many sentences => One sentence at a time
Too many tasks => One task at a time
Wednesday, October 3, 12
![Page 22: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/22.jpg)
27Wednesday, October 3, 12
![Page 23: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/23.jpg)
28Wednesday, October 3, 12
![Page 24: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/24.jpg)
Julio0say0that0the0mouse0needed0the0tooth0for0his0secret0formula.
Julio0said0that0the0mouse0needed0the0tooth0for0his0secret0formula.
29Wednesday, October 3, 12
![Page 25: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/25.jpg)
DeploymentAug 2011 – May 2012
11 books (6 languages)
1,282 sentences
155,000 visits
31,000 tasks
Wednesday, October 3, 12
![Page 26: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/26.jpg)
English
Widgets Evaluation
14 days
54 sentences
739 users2 books
Spanish
Wednesday, October 3, 12
![Page 27: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/27.jpg)
Google: 31% of sentencesWidgets: 52% of sentences
0"
10"
20"
30"
40"
50"
1=None" 2" 3" 4" 5=All"
#"of"Sen
tences"
Sentences"per"Accuracy"Category"
Google"
Widgets"
Accuracy
Wednesday, October 3, 12
![Page 28: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/28.jpg)
How0could0a0crowdhelp0you0make0an
importantdecision?
Alex QuinnCS Ph.D. student
Wednesday, October 3, 12
![Page 29: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/29.jpg)
Wednesday, October 3, 12
![Page 30: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/30.jpg)
AskSheet Example Problems
Pick a grad school
Where to buy groceries?
Where/when go on vacation?
Which papers should be accepted?
Wednesday, October 3, 12
![Page 31: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/31.jpg)
Decision modeling process
Define the Problem
Develop a Model
Acquire Input Data
Develop a Solution
Test the Solution
Wednesday, October 3, 12
![Page 32: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/32.jpg)
Decision modeling process
Acquire Input DataResearch focus
Frugal input acquisition
Wednesday, October 3, 12
![Page 33: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/33.jpg)
Create a blank decision model with ASK formulas in cells C2:E6. AskSheet highlights priority cells in dark green.
Create Model
Wednesday, October 3, 12
![Page 34: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/34.jpg)
Set up the task by specifying who will receive the HITs, instructions, and other details. The "Prioritization" slider controls how many inputs to include in each batch.
Configure Task
Wednesday, October 3, 12
![Page 35: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/35.jpg)
At the bottom of the setup screen is a dynamically updated preview of what workers will see. The input form is automatically generated from the spreadsheet model.
Preview Task
Wednesday, October 3, 12
![Page 36: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/36.jpg)
The system prioritizes the inputs. Obtaining the dark green cells first would provide the greatest opportunity to eliminate other cells. Braces expressions show the possible range of output values for each cell.
Data is Collected
Wednesday, October 3, 12
![Page 37: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/37.jpg)
The model is now complete. Store C wins because its maximum possible price is $68, is lower than the minimum possible price of either Store A ($85) or Store B. ($71). The system avoided requesting 11 of the 27 inputs.
Task Completed
Wednesday, October 3, 12
![Page 38: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/38.jpg)
“ASK” formula
A1: =ASK(1, 1, 5)
B1: =ASK(1, 1, 10)
C1: =IF(A1 < B1, "A", "B")
cost, min, max
Wednesday, October 3, 12
![Page 39: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/39.jpg)
“ASK” formula
A1: =ASK(1, 1, 5)B1: =ASK(1, 1, 10)
C1: =IF(A1 < B1, "A", "B")
A1
B1
1 2 3 4 5 6 7 8 9 10
Task Prioritization
Wednesday, October 3, 12
![Page 40: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/40.jpg)
AskSheet Trial
Answer rcvd in < 9 hrs
Eliminated 74% of work
Task: Find pediatrician that is rated well, accepted by insurance, and sufficiently close
Wednesday, October 3, 12
![Page 41: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/41.jpg)
Open Problems
•Evaluation of student work in MOOCs
•Aggregation of comments on blogs / class forums
•Assuring quality of work (i.e., product reviews)
Wednesday, October 3, 12
![Page 42: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/42.jpg)
Design Lessons
Assuring quality at a viable cost remains hard•Assume adversarial participants. •Best practices: redundancy, gold standards, inform you
are watching, treat workers as employees
The "motivation" problem remains hard•Don’t forget "what's in it for me".
Make it fundamental to primary task
Ethics
Wednesday, October 3, 12
![Page 43: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/43.jpg)
Ethics of Human Computation
$$$ Anonymity
Wednesday, October 3, 12
![Page 44: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/44.jpg)
• Bulk of problems are money/qualityrelated
• Workers complain about– Low wages–Not getting paid– Slow payment
• Requesters (who are less anonymous) complain about– Low quality work
• Also, significant other issues – Decontextualization– Tasks that are illegal or unacceptable– Privacy / anonymity
Money Problems
Wednesday, October 3, 12
![Page 45: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/45.jpg)
Current Situation• Workers and Requesters alike build their own
reputation/quality mgmt systems
–Turker Nation
Wednesday, October 3, 12
![Page 46: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/46.jpg)
Current Situation• Workers and Requesters alike build their own
reputation/quality mgmt systems
–Turker Nation–Turkopticon
Wednesday, October 3, 12
![Page 47: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/47.jpg)
Current Situation• Workers and Requesters alike build their own
reputation/quality mgmt systems
–Turker Nation
–CrowdFlower–Turkopticon
Wednesday, October 3, 12
![Page 48: Social Participation: How Collective Activity Can Make Change](https://reader034.vdocument.in/reader034/viewer/2022051608/5453a553af795904308b5442/html5/thumbnails/48.jpg)
Questions?
www.cs.umd.edu/~bederson @bederson
MonoTrans0supported0by0Na,onal0Science0Founda,onMonoTrans0&0SearchParty0supported0by0Google
Wednesday, October 3, 12