social machines: theory design and incentives
Post on 09-Jan-2017
164 Views
Preview:
TRANSCRIPT
-
SOCIAL MACHINESTECHNOLOGY DESIGN AND INCENTIVES
Elena Simperle.simperl@soton.ac.uk @esimperlJuly 4th, 2016
1
-
TUTORIAL OVERVIEW
Social machines as engine of creativity, growth, and sociality
This tutorial covers Introduction to social machines How-to crowdsourcing design How-to study citizen science projects Examples from current research
TUTORIAL@ISWC2013
-
THE ORDER OF SOCIAL MACHINES
Real life is and must be full of all kinds of social constraint the very processes from which society arises. Computers can help if we use them to create abstract social machines on the Web: processes in which the people do the creative work and the machine does the administration []
The stage is set for an evolutionary growth of new social engines.
Tim Berners-Lee, Weaving the Web, 1999
3
-
PEOPLE SUPPLY, CURATE AND ANALYZE DATA
-
OR CREATE CONTENT FROM SCRATCH
-
FOR FREE OR AGAINST A REWARD
-
PEOPLE ARE (ELEMENTARY) PROBLEM SOLVERS
-
SOMETIMES ON ANY GIVEN TOPIC
-
PEOPLE LIKE TO SHARE THEIR VIEWS
9
-
SOMETIMES THEY WORK TOGETHER
10
-
SOMETIMES THEY COMPETE
11
-
IT DOESNT GO WELL ALL THE TIME
12
-
BUT THE ONES THAT DO WELL SEEM TO HAVE THINGS IN COMMON
i. Problems solved by the scale of human participation
ii. Timely mobilisation of people, technology, and information resources
iii. Access to (or else the ability to generate) large amounts of relevant data
iv. Confidence in the quality of the data
v. Efficient, effective and equitable
vi. Trust in the agents and process
vii. Aligned incentives and network effects
viii. Intuitive, user-centred interfaces
ix. Cross platform support
x. Exploit the power of the open
-
sociam.org
@project_sociam
-
WHAT IS A SOCIAL MACHINE? Exercise 115
-
Based on the definition and examples mentioned so far, discuss examples of social machines you are familiar with.
When is a social machine successful?
How do social machines compare with: human computation, crowdsourcing, collective intelligence, social computing, wisdom of the crowds, open innovation, socio-technical systems etc.?
16
-
SOCIAL MACHINES AS CROWDSOURCING
17
-
CROWDSOURCING:PROBLEM SOLVING VIA OPEN CALLS"Simply defined, crowdsourcing represents the act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call. This can take the form of peer-production (when the job is performed collaboratively), but is also often undertaken by sole individuals. The crucial prerequisite is the use of the open call format and the large network of potential laborers.
[Howe, 2006]
7/4/2016 18
-
DIFFERENT FORMS AND PLATFORMS TO CHOOSE FROM
19
Macrotasks
Microtasks
Challenges
Self-organized crowds
Crowdfunding
Source:
[Prpi et al., 2015]
-
FOR EXAMPLE: MICROTASK CROWDSOURCING
7/4/2016 20
Work is broken down into smaller pieces that can be solved independently
-
Are social machines just crowdsourcing?
-
CROWDSOURCING & SOCIALITY
-
CROWDSOURCING CREATIVE TASKS
-
CROWDSOURCING & ETHICS
[Difallah et al, 2015]
-
MANY QUESTIONS TO ANSWER
TASK DESIGNWORKFLOW DESIGN AND EXECUTION
TASK INTERFACES QUALITY ASSURANCE
TASK ASSIGNMENT
CROWD TRAINING AND
FEEDBACKINCENTIVES
ENGINEERING
COLLABORATION, COMPETITION,
SELF-ORGANIZATION
REAL-TIME DELIVERY NICHESOURCING
EXTENSIONS TO TECHNOLOGIES
SOCIAL MACHINES
ENGINEERING
-
LUNCH BREAKe.simperl@soton.ac.uk@esimperl
sociam.org26
-
CROWDSOURCING DESIGN27
-
CROWDSOURCING DESIGN 101
What: Goal
Who: Contributors
How: Process
Why: Motivation and incentives
http://sloanreview.mit.edu/article/the-collective-intelligence-genome
28
Image at http://jasonbiv.files.wordpress.com/2012/08/collectiveintelligence_
genome.jpg
http://sloanreview.mit.edu/article/the-collective-intelligence-genome/http://jasonbiv.files.wordpress.com/2012/08/collectiveintelligence_genome.jpg
-
EXAMPLE: ZOONIVERSE
WHAT IS OUTSOURCED
Object recognition, labeling, categorization in media content
WHO IS THE CROWD
Anyone
HOW IS THE TASK OUTSOURCED
Highly parallelizable tasks
Every item is handled by multiple annotators
Every annotator provides an answer
Consolidated answers solve scientific problems
WHY DO PEOPLE CONTRIBUTE
Fun, interest in science, desire to contribute to science
7/4/2016 29
zooniverse.org
-
DESIGN YOUR OWN Exercise 230
-
IN THIS TALK: CROWDSOURCING DATA CITATIONThe USEWOD experiment Goal: collect information about the usage of Linked
Data sets in research papers Explore different crowdsourcing methods Online tool to link publications to data sets (and
their versions) 1st feasibility study with 10 researchers in May
2014
7/4/2016 31
http://prov.usewod.org/
9650 publications
-
Discuss how you would crowdsource the problem illustrated on the previous slide.
Consider using the schema presented earlier.
Think about how you would ensure that the data is accurate.
(15 minutes discussions in groups, then plenary discussion of results)
32
-
DIMENSIONS OF CROWDSOURCING33
-
WHAT
In general: something you cannot do (fully) automatically
In our example: annotating research papers with data set information Alternative representations of the domain What if the paper is not available? What if the domain is not known in advance or is infinite? Do we know the list of potential answers? Is there only one correct solution to each atomic task? How many people would solve the same task?
7/4/2016 34
What Who
How Why
-
BROAD RANGE OF TASKS
7/4/2016 35
What Who
How Why
-
WHOIn general An open (unknown) crowd Scale helps Qualifications might be a pre-requisite
In our example People who know the papers or the data sets Experts in the (broader) field Casual gamers Librarians Anyone (knowledgeable of English, with a computer/cell phone etc) Combinations thereof
7/4/2016 36
What Who
How Why
-
Country of residence
United States: 46.80% India: 34.00% Miscellaneous: 19.20%
7/4/2016 37
A LARGE, BUT NOT ALWAYS DIVERSE CROWD
What Who
How Why
-
SIGNIFICANT AMOUNT OF RESOURCES AND TIMELY DELIVERY
7/4/2016 38
What Who
How Why
-
SIGNIFICANT AMOUNT OF RESOURCES AND TIMELY DELIVERY (2)
39
http://www.mturk-tracker.com/
60% of workers spend more than 4 hours a week on MTurk
What Who
How Why
-
HOW: PROCESSIn general: Explicit vs. implicit participation
Affects motivation
Tasks broken down into smaller units undertaken in parallel by different people Does not apply to all forms of crowdsourcing
Coordination required to handle cases with more complex workflows Sometimes using different crowds for workflow parts
Example: text summarization
Task assignment to match skills, preferences, and contribution history Example: random assignment vs meritocracy vs full autonomy
Partial or independent answers consolidated and aggregated into complete solution Example: challenges (e.g., Netflix) vs aggregation (e.g., Wikipedia)
7/4/2016 40
See also [Quinn & Bederson, 2012]
What Who
How Why
-
HOW: PROCESS (2)In our example: Use the data collected here to train a IE algorithm Use paid microtask workers to go a first screening, then expert crowd to sort out
challenging cases What if you have very long documents potentially mentioning different/unknown
data sets? Competition via Twitter
Which version of DBpedia does this paper use? One question a day, prizes Needs golden standard to bootstrap and redundancy
Involve the authors Use crowdsourcing to find out Twitter accounts, then launch campaign on
Twitter Write an email to the authors
Change the task Which papers use Dbpedia 3.X?
Competition to find all papers
7/4/2016 41
What Who
How Why
-
HOW: VALIDATION In general: Solutions space closed vs. open
Redundancy vs. iterations/peer-review Performance measurements/ground truth
Available and accepted Automated techniques employed to predict accurate solutions
In our example: Domain is fairly restricted
Spam and obvious wrong answers can be detected easily When are two answers the same? Can there be more than one correct answer per question?
Redundancy may not be the final answer Most people will be able to identify the data set, but sometimes the actual
version is not trivial to reproduce Make educated version guess based on time intervals and other features
7/4/2016 42
What Who
How Why
-
WHYMoney (rewards)
Love
Glory
Love and glory reduce costs
Money and glory make the crowd move faster
Intrinsic vs extrinsic motivation
Rewards/incentives influence motivation. They are granted by an external judge
Successful volunteer crowdsourcing is difficult to predict or replicate Highly context-specific Not applicable to arbitrary tasks
Reward models often easier to study and control (if performance can be reliably measured) Not always easy to abstract from social
aspects (free-riding, social pressure) May undermine intrinsic motivation
43
What Who
How Why
-
WHY (2)
In our example:
Who benefits from the results
Who owns the results
Money Different models: pay-per-time, pay-per-unit, winner-takes-it-all Define the rewards, analyze trade-offs accuracy vs costs, avoid spam
Love Authors, researchers, Dbpedia community, open access advocates, publishers, casual gamers etc.
Glory Competitions, mentions, awards
7/4/2016 44
Image from Nature.com
What Who
How Why
-
PRICING ON MTURK
7/4/2016 45
[Ipeirotis, 2008]
-
ITS NOT ALWAYS JUST ABOUT MONEY
7/4/2016 46
http://www.crowdsourcing.org/editorial/how-to-motivate-the-crowd-infographic/
http://www.oneskyapp.com/blog/tips-to-motivate-participants-of-crowdsourced-translation/
[Kaufmann, Schulze, Viet, 2011]
-
CURRENT RESEARCH47
-
CROWDSOURCING RESEARCH
TASK DESIGN TASK ASSIGNMENT QUALITY ASSURANCEINCENTIVES
ENGINEERING
WORKFLOW DESIGN AND EXECUTION
CROWD TRAINING SELF-ORGANIZING CROWDSCOLLABORATIVE
CROWDSOURCING
REAL-TIME DELIVERY EXTENSIONS TO TECHNOLOGIESPROGRAMMING
LANGUAGESAPPLICATION SCENARIOS
-
IMPROVING PAID MICROTASKS @WWW15Compared effectivity of microtasks on CrowdFlower vs self-developed game Image labelling on ESP data set as gold standard Evaluated
accuracy #labels cost per label avg/max #labels/contributor
For three types of tasks Nano: 1 image Micro: 11 images Small: up to 2000 images
Probabilistic reasoning to personalize furtherance incentives
Findings Gamification and payments work well together Furtherance incentives particularly interesting for top contributors
-
CROWDSOURCING DBPEDIA CURATION @ISWC13
-
ENTITY CLASSIFICATION IN DBPEDIA@SEMANTICWEBJOURNAL15Study different ways to crowdsource entity typing using paid microtasks.
Three workflows Free associations Validating the machine Exploring the DBpedia ontology
Findings Shortlists are easy & fast
Popular classes are not enough Alternative ways to explore the taxonomy
Freedom comes with a price Unclassified entities might be unclassifiable Different human data interfaces
Working at the basic level of abstraction achieves greatest precision But when given the freedom to choose, users suggest more specific classes
4.58Mthings
-
HYBRID NER ON TWITTER @ESWC15
Identified content and crowd factors that impact effectivity #entities in post types of entities content sentiment skipped TP posts avg. time/task UI interaction
Findings Shorter tweets with fewer entities work better Crowd is more familiar with people and places from recent news MISC as a NER category sometimes confusing but useful to identify partial and
implicitly named entities
-
OBSERVE AND IMPROVE* Elena Simperl, University of Southampton, UK
*with slides by Ramine Tinati, Markus Luczak-Rsch, and others
-
OVERVIEW
Background to citizen science
Citizen science & human computation
Citizen science & online communities
Citizen science & science
Methods and design guidelines
-
WHAT IS CITIZEN SCIENCE
-
EXAMPLE: ZOONIVERSE
-
EXAMPLE: PYBOSSA
-
EXAMPLE: VOLUNTEER SCIENCE
-
EXAMPLE: EYEWIRE
-
STUDYING CITIZEN SCIENCE: HUMAN COMPUTATION
Task design
Task assignment
Answer validation and aggregation
Contributors performance
Motivation and incentives
-
STUDYING CITIZEN SCIENCE: ONLINE COMMUNITY
Roles and activities
Patterns of participation
Motivation and incentives
Evolution in time
-
STUDYING CITIZEN SCIENCE: ESCIENCE
New workflows
New best practices
New publishing models
-
A FOOTER
When is a citizen science project successful?
-
LEVELS OF ENGAGEMENT
0
2
4
6
8
10
12
14
16
18
20
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31
Act
ive
user
s in
%
Month since registration
~1% of participants contribute 72% of Talk & 29% of Task
-
BEYOND DATA COLLECTION AND ANALYSIS
Discussion and community engagement are integral part of the experience
-
WORK VS TALK
40.5%
Classifications
Talk
con
trib
utio
ns
Classifications
-
CHAT AND INSTANT MESSAGING
Microposts
PH SG SW NN GZ CC PF SF AP WS
91%2
0
6
4
10
8
-
DISCUSSION PROFILES
Deeply engaged volunteers, few
threads but multiple posts within them
9 0.1%
Content producers,
posting across many boards and threads
7 0.1%
Thread followers and PM (one-to-one) talkers
8 0.4%
First to respond and question
answerers 4 1%
Highly active thread starters and answerers across a wide
range of topics
1 2.8%
Infrequent volunteers, single thread posts, no
personal messages
5 5.5%
Watcher and starter of many threads, but not
first to reply3 6.5%
Highly active thread starters
and first to reply back
2 14.6%
Long active volunteers (the
core group), posting
sporadically
6 69.0%
-
TALKING DOES NOT MEAN LESS WORK
How long do games take to complete depending on when the chat messages are made?
Games took longer to complete when chat messages were being made during the
gaming session, compare to games that had messages before or after.
-
ECOSYSTEM EFFECTS
Project A
Project B
Project C
Participant X
Part. Y
-
DESIGNING PLATFORMS
Task specificity
Community development
Task design PR and engagement
Bootstrapping the community
Serendipitous scientific discovery
Engaging with people, supporting profession team
Supporting individuals, finding new scientific discoveries
Obtaining new citizen scientists
Retaining people
Supporting people, improving task completion
Obtaining new citizen scientists
Reinvigorating old users
-
WHATS NEXT?
Human computation Task assignment: what tasks are interesting/relevant for whom? Peer review, collaborative approaches The role of gamification: is science a game?
Online community Mapping design patterns to online community design frameworks: do CS form a community? Making discussions more effective
Science Citizen science platforms that everyone can use
-
PUBLICATIONS
R. Tinati, M. Van Kleek, E. Simperl, M. Luczak-Rsch, R. Simpson and N. Shadbolt. Designing for citizen data analysis: a cross-sectional case study of a multi-domain citizen science platform. Proceedings of ACM CHI Conference on Human Factors in Computing Systems 2014 (CHI2015), pages 1-10, 2015.
M. Luczak-Rsch, R. Tinati, E. Simperl, M. Van Kleek, N. Shadbolt and R. Simpson. Why Won't Aliens Talk to Us? Content and Community Dynamics in Online Citizen Science. Proceedings of the International Conference on Weblogs and Social Media (ICWSM2014), 2014.
A FOOTER
-
E.SIMPERL@SOTON.AC.UK@ESIMPERL
WWW.SOCIAM.ORGWWW.STARS4ALL.EU
7/4/2016 74
mailto:e.simperl@SOTON.AC.UKhttp://www.sociam.org/http://www.stars4all.eu/
-
FURTHER READING
7/4/2016 75
Social machinestechnology DESIGN and incentivesTutorial overviewTHE ORDER OF SOCIAL MACHINESPEOPLE SUPPLY, CURATE AND ANALYZE DATAOR CREATE CONTENT FROM SCRATCHFOR FREE OR AGAINST A REWARDPEOPLE ARE (ELEMENTARY) PROBLEM SOLVERSSOMETIMES ON ANY GIVEN TOPICPeople like to share their viewsSometimes they work togetherSometimes they competeIt doesnt go well all the timeBut the ones that do well seem to have things in commonSlide Number 14What is a social machine?Slide Number 16SOCIAL MACHINES As crowdsourcingCrowdsourcing:problem solving via open callsDifferent Forms and platforms to CHOOSE FROMFor example: Microtask crowdsourcingSlide Number 21Crowdsourcing & socialITYCrowdSourcing Creative TASKSCrowdSourcing & EthicsMANY QUESTIONS TO ANSWERLunch breakCrowdsourcing designCrowdsourcing design 101Example: ZooniverseDesign your ownIn this talk: CROWDSOURCING data citationSlide Number 32Dimensions of crowdsourcingwhatBroad range of tasksWhoA Large, but Not always diverse crowdSignificant amount of resources and timely deliverySignificant amount of resources and timely delivery (2)HOW: PROCESSHOW: PROCESS (2)How: Validation WHYWhy (2)Pricing on MTurkIts not always JUST about moneyCurrent researchCrowdsourcing Researchimproving paid microtasks @WWW15Crowdsourcing Dbpedia curation @ISWC13Entity classification in DBPedia @SemanticWebJournal15Hybrid NER on Twitter @ESWC15Observe and improve* OverviewWhat is citizen scienceExample: ZooniverseExample: PybossaExample: Volunteer ScienceExample: EyeWireStudying citizen science: human computationStudying citizen science: online communityStudying citizen science: eScienceSlide Number 63Levels of engagementBeyond data collection and analysisWork vs talkChat and instant messagingDiscussion profilesTalking does not mean less workEcosystem effectsDesigning platformsWhats next?Publicationse.simperl@SOTON.AC.UK@esimperlwww.sociam.orgwww.stars4all.eu Further reading
top related