boomerang - stanford hci group€¦ · boomerang rebounding the consequences of reputation feedback...
TRANSCRIPT
![Page 1: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/1.jpg)
Boomerang REBOUNDING THE CONSEQUENCES OF REPUTATION FEEDBACK ON CROWDSOURCING PLATFORMS
NEIL (SNEHALKUMAR) S. GAIKWAD, DURIM MORINA, ADAM GINZBERG, CATHERINE MULLINGS, SHIRISH GOYAL, DILRUKSHI GAMAGE, CHRISTOPHER DIEMERT, MATHIAS BURTON, SHARON ZHOU, MARK WHITING, KAROLINA ZIULKOSKI, ALIPTA BALLAV, AARON GILBEE, SENADHIPATHIGE S. NIRANGA, VIBHOR SEHGAL, JASMINE LIN, LEONARDY KRISTIANTO, ANGELA RICHMOND-FULLER, JEFF REGINO, NALIN CHHIBBER, DINESH MAJETI, SACHIN SHARMA,KAMILA MANANOVA, DINESH DHAKAL, WILLIAM DAI, VICTORIA PURYNOVA, SAMARTH SANDEEP, VARSHINE CHANDRAKANTHAN, TEJAS SARMA, SEKANDAR MATIN, AHMED NASSER, ROHIT NISTALA, ALEXANDER STOLZOFF, KRISTY MILLAND, VINAYAK MATHUR, RAJAN VAISH, MICHAEL S. BERNSTEIN
STANFORD CROWD RESEARCH COLLECTIVE
![Page 2: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/2.jpg)
Stanford Crowd Research Collective
![Page 3: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/3.jpg)
a self-governed crowdsourcing marketplace coming soon
![Page 4: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/4.jpg)
Reputation Systems
4
![Page 5: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/5.jpg)
Reputation establishes trust
5
Trust is a central currency of crowdsourcing marketplaces Crowdsourcing marketplaces such as Amazon Mechanical Turk and Upwork rely on reputation systems to convey who is trustable.
![Page 6: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/6.jpg)
Reputation inflation
[Horton and Golden 2015]
![Page 7: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/7.jpg)
Incentives are misaligned! Social costs of negative feedback lead to generous
rounding up
! Many requesters do not reject work — even terrible work — so they don’t have to deal with complaints.
! Many workers give requesters 5 stars because they fear getting blocked from future work.
7
![Page 8: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/8.jpg)
A trust breakdown produces a market for lemons
These reputation systems remain untrusted, prompting requesters to offer lower wages to hedge the uncertainty.
8
![Page 9: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/9.jpg)
Incentive Compatibility (IC)! Game theory: incentive compatibility holds when
people maximize their own outcomes by sharing their information truthfully.
! Removes the need for strategic behavior.! Example: online advertisement auctions
! Traditional auctions: item goes to the top bidder, at their bid! Incentive-compatible single item second-price auction: item goes
to the top bidder, at the second-highest bid.
9
![Page 10: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/10.jpg)
IC second-price auction
10
A’s Bid Second Highest A’s Bid
Second Highest A’s Bid Second
Highest
A’s True Value
A Pays
A’s Bid HighestBid
Second Highest
![Page 11: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/11.jpg)
However, wicked problems aren’t easily modeled! Traditional game theory requires proofs via
mathematical modeling of the game’s rules.
! These models cannot straightforwardly apply to the socio-technical design of a crowdsourcing system: agents do not share a clearly-definable strategy, and the set of possible behaviors are unknown.
11
![Page 12: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/12.jpg)
Our goal: incentive compatible interaction design (ICID)! Can we adapt incentive compatibility to the broader
class of interaction design problems where utility is difficult to model?
! We seek socio-technical designs where the user benefits more by sharing honest information with the platform than by sharing inaccurate information. 12
![Page 13: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/13.jpg)
Could we design a reputation system where people benefit the most by rating truthfully?
13
![Page 14: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/14.jpg)
Boomerang
14
ブーメラン
![Page 15: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/15.jpg)
Rebounding the consequences of rating feedback! Boomerang is a reputation system that rebounds
ratings to directly impact the rater’s later activities on the platform.
! We implemented Boomerang on the Daemo crowdsourcing platform [Gaikwad et al., 2015] daemo.stanford.edu
15
![Page 16: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/16.jpg)
Workers
16
√+
√
√-
Requester
√+
√√-
![Page 17: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/17.jpg)
Boomerang releases tasks first to high-rated workers, then sequentially to lower and lower scores.
Inflating the rating for a low-quality worker will increase the probability the worker will “boomerang” around and do more work for you.
17
√+ √ √-
Early Access Late Access
Requesters’ incentive: quality
![Page 18: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/18.jpg)
18
Cascading task release
utilization ≤ λ
√+ √ √-
Time
Util
izat
ion
utilization =
CompletedTasks(t� T, t)
max
ti2[tinit, t�T ]CompletedTasks(ti, ti + T )
λutilization ≤ λ
![Page 19: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/19.jpg)
Workers’ incentive: task search
19
! Workers spend significant time finding tasks [Chilton et al. 2010]
! Making more efficient task search a significant motivator.
![Page 20: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/20.jpg)
Boomerang feed ranking
20
! Boomerang uses workers’ ratings to rank their task feed: high-rated requesters on top, low-rated requesters at the bottom.
! Giving high ratings to a poor requester thus muddies the task feed.
![Page 21: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/21.jpg)
Boomerang task feed
21
Verify facts in a short video √Alice$0.90
Tracribe data
Describ Img
√+
√+
$0.05
$0.01
Richard
Ana
√-Collect Product Information$0.50 Leonardo
21
![Page 22: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/22.jpg)
Evaluation
22
![Page 23: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/23.jpg)
Field experiment
23
N=35 requesters rated workers’ submissions to three crowdsourcing tasks in sequence• Control condition: ratings had no effect• Boomerang condition: ratings impacted
the probability that each worker would return for the next task (high rating = 2x baseline probability).
√+
√-
√√+
√√√-
√+
√+
√+
√√-
√-
√-
![Page 24: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/24.jpg)
Measure: predictive accuracy
24
After rating, requesters performed ten forced-choice decisions between sets of three workers each, deciding which of the three workers had been the highest quality.
We used ratings to predict which worker the requester would choose.
score =
8><
>:
1, if correct
1/n, if correct and n-way rating tie
0, otherwise
![Page 25: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/25.jpg)
Boomerang produces more accurate ratings
25
t(33) = 2.66 p<.01 d=0.91
Similar results for workers
Scor
e
0
1
2
3
4
5
6
Boomerang Control
![Page 26: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/26.jpg)
Boomerang deflates ratings
26
Rat
ings
per
req
uest
er
0
4
8
12
16
√- √ √+
Control
![Page 27: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/27.jpg)
Boomerang deflates ratings
27
Boomerang led to 0.5x as many √+’s, 2.5x as many √’s, and 1.8x as many √-’s.
Rat
ings
per
req
uest
er
0
4
8
12
16
√- √ √+
Boomerang Control
![Page 28: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/28.jpg)
Applications
28
![Page 29: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/29.jpg)
Rejection transparency
29
! Problem: Requesters reject infrequently or indiscriminately.
! Boomerang: show requesters’ rejection rate.
! Never rejecting means attracting spam, always rejecting means work won’t get done.
![Page 30: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/30.jpg)
Rejection transparency
30
Verify facts in a short video checkAlice Rejection Rate 0%$0.20
Transcribe data
Describe Images
check+
check+
$1.50
$2.50
Richard Rejection Rate 10%
Sanjay Rejection Rate 80%
check-Collect Product Information$0.51 Ryo Rejection Rate 16%
![Page 31: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/31.jpg)
Discussion
31
![Page 32: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/32.jpg)
Limitations ! Boomerang’s incentives currently depend on workers
and requesters using the platform repeatedly over time.
! Boomerang is not a forcing function, but a nudge.
32
![Page 33: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/33.jpg)
ICID for sharing economy
! Can we motivate pro-social behavior on other social computing platforms?! Uber: give priority to riders who don’t cancel often.! Waze: give early access to traffic reports to those
who make reports themselves.
33
![Page 34: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/34.jpg)
Boomerang REBOUNDING THE CONSEQUENCES OF REPUTATION FEEDBACK ON CROWDSOURCING PLATFORMS
STANFORD CROWD RESEARCH COLLECTIVEhttps://daemo.stanford.edu/
![Page 35: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/35.jpg)
Crowd Guilds: Worker-led Reputation and Feedbackon Crowdsourcing Platforms
Mark E. Whiting, Dilrukshi Gamage, Aaron Gilbee, Snehal Gaikwad, Shirish Goyal, Alipta Ballav,Dinesh Majeti, Nalin Chhibber, Freddie Vargus, Teo Moura, Angela Richmond-Fuller, VarshineChandrakanthan, Gabriel Bayomi Tinoco Kalejaiye, Tejas Seshadri Sarma, Yoni Dayan, AdamGinzberg, Mohammed Hashim Kambal, Kristy Milland, Sayna Parsi, Catherine A. Mullings,
Henrique Orefice, Sekandar Matin, Vibhor Sehgal, Sharon Zhou, Akshansh Sinha, Jeff Regino,Rajan Vaish, Michael S. BernsteinStanford Crowd Research Collective
ABSTRACTCrowd workers are distributed and decentralized. While de-centralization is designed to utilize independent judgment topromote high-quality results, it paradoxically undercuts be-haviors and institutions that are critical to high-quality work.Reputation is one central example: crowdsourcing systemsdepend on reputation scores from decentralized workers andrequesters, but these scores are notoriously inflated and unin-formative. In this paper, we draw inspiration from historicalworker guilds (e.g., in the silk trade) to design and implementcrowd guilds: centralized groups of crowd workers who col-lectively certify each other’s quality through double-blind peerassessment. A two week field experiment compared crowdguilds to a traditional decentralized crowd work model. Crowdguilds produced reputation signals more strongly correlatedwith ground-truth worker quality than signals available oncurrent platforms, and more accurate than in the traditionalmodel.
Author Keywordscrowdsourcing platforms; human computation
ACM Classification KeywordsH.5.3. Group and Organization Interfaces
INTRODUCTIONCrowdsourcing platforms such as Amazon Mechanical Turkdecentralize their workforce, designing for distributed, inde-pendent work [16, 42]. Decentralization aims to encourageaccuracy through independent judgement [59]. However, bymaking communication and coordination more difficult, de-centralization disempowers workers and forces worker collec-tives off-platform [41, 64, 16]. The result is disenfranchise-ment [22, 55] and an unfavorable workplace environment [41,Paste the appropriate copyright statement here. ACM now supports three differentcopyright statements:• ACM copyright: ACM holds the copyright on the work. This is the historical ap-proach.• License: The author(s) retain copyright, but ACM receives an exclusive publicationlicense.• Open Access: The author(s) wish to pay for the work to be open access. The addi-tional fee must be paid to ACM.This text field is large enough to hold the appropriate release statement assuming it issingle spaced.Every submission will be assigned their own unique DOI string to be included here.
Figure 1. Crowd guilds aim to provide accurate reputation signalsthrough re-centralized feedback in a paid crowdsourcing platform.
42]. Worse, while decentralization is motivated by a desirefor high-quality work, it paradoxically undercuts behaviorsand institutions that are critical to high-quality work. In manytraditional organizations, for example, centralized worker coor-dination is a keystone to behaviors that improve work quality,including skill development [3], knowledge management [35],and performance ratings [58].
In this paper, we focus on reputation as an exemplar challengethat arises from worker decentralization: effective reputationsignals are traditionally reliant on centralized mechanismssuch as performance reviews [58, 23]. Crowdsourcing plat-forms rely heavily on their reputation systems, such as taskacceptance rates, to help requesters identify high-quality work-ers [22, 43]. On Mechanical Turk, as on other on-demandplatforms such as Upwork and Uber, these reputation scoresare derived from decentralized feedback via many independentrequesters. However, the resulting reputation scores are no-toriously inflated and noisy, making it difficult for requestersto find high-quality workers and difficult for workers to becompensated for their quality [43, 20].
To address this reputation challenge, and with an eye towardother challenges that arise from decentralization, we draw in-spiration from a historical labor strategy for coordinating adecentralized workforce: guilds. Worker guilds arose in theearly middle ages, when workers in a trade such as silk weredistributed across a large region, as bounded sets of laborerswho shared an affiliation. These guilds played many roles,including training apprentices [18, 44], setting prices [45],and providing mechanisms for collective action [52, 49]. Es-
Crowd Guilds: Worker-led Reputation and Feedbackon Crowdsourcing Platforms
Mark E. Whiting, Dilrukshi Gamage, Aaron Gilbee, Snehal Gaikwad, Shirish Goyal, Alipta Ballav,Dinesh Majeti, Nalin Chhibber, Freddie Vargus, Teo Moura, Angela Richmond-Fuller, VarshineChandrakanthan, Gabriel Bayomi Tinoco Kalejaiye, Tejas Seshadri Sarma, Yoni Dayan, AdamGinzberg, Mohammed Hashim Kambal, Kristy Milland, Sayna Parsi, Catherine A. Mullings,
Henrique Orefice, Sekandar Matin, Vibhor Sehgal, Sharon Zhou, Akshansh Sinha, Jeff Regino,Rajan Vaish, Michael S. BernsteinStanford Crowd Research Collective
ABSTRACTCrowd workers are distributed and decentralized. While de-centralization is designed to utilize independent judgment topromote high-quality results, it paradoxically undercuts be-haviors and institutions that are critical to high-quality work.Reputation is one central example: crowdsourcing systemsdepend on reputation scores from decentralized workers andrequesters, but these scores are notoriously inflated and unin-formative. In this paper, we draw inspiration from historicalworker guilds (e.g., in the silk trade) to design and implementcrowd guilds: centralized groups of crowd workers who col-lectively certify each other’s quality through double-blind peerassessment. A two week field experiment compared crowdguilds to a traditional decentralized crowd work model. Crowdguilds produced reputation signals more strongly correlatedwith ground-truth worker quality than signals available oncurrent platforms, and more accurate than in the traditionalmodel.
Author Keywordscrowdsourcing platforms; human computation
ACM Classification KeywordsH.5.3. Group and Organization Interfaces
INTRODUCTIONCrowdsourcing platforms such as Amazon Mechanical Turkdecentralize their workforce, designing for distributed, inde-pendent work [16, 42]. Decentralization aims to encourageaccuracy through independent judgement [59]. However, bymaking communication and coordination more difficult, de-centralization disempowers workers and forces worker collec-tives off-platform [41, 64, 16]. The result is disenfranchise-ment [22, 55] and an unfavorable workplace environment [41,Paste the appropriate copyright statement here. ACM now supports three differentcopyright statements:• ACM copyright: ACM holds the copyright on the work. This is the historical ap-proach.• License: The author(s) retain copyright, but ACM receives an exclusive publicationlicense.• Open Access: The author(s) wish to pay for the work to be open access. The addi-tional fee must be paid to ACM.This text field is large enough to hold the appropriate release statement assuming it issingle spaced.Every submission will be assigned their own unique DOI string to be included here.
Figure 1. Crowd guilds aim to provide accurate reputation signalsthrough re-centralized feedback in a paid crowdsourcing platform.
42]. Worse, while decentralization is motivated by a desirefor high-quality work, it paradoxically undercuts behaviorsand institutions that are critical to high-quality work. In manytraditional organizations, for example, centralized worker coor-dination is a keystone to behaviors that improve work quality,including skill development [3], knowledge management [35],and performance ratings [58].
In this paper, we focus on reputation as an exemplar challengethat arises from worker decentralization: effective reputationsignals are traditionally reliant on centralized mechanismssuch as performance reviews [58, 23]. Crowdsourcing plat-forms rely heavily on their reputation systems, such as taskacceptance rates, to help requesters identify high-quality work-ers [22, 43]. On Mechanical Turk, as on other on-demandplatforms such as Upwork and Uber, these reputation scoresare derived from decentralized feedback via many independentrequesters. However, the resulting reputation scores are no-toriously inflated and noisy, making it difficult for requestersto find high-quality workers and difficult for workers to becompensated for their quality [43, 20].
To address this reputation challenge, and with an eye towardother challenges that arise from decentralization, we draw in-spiration from a historical labor strategy for coordinating adecentralized workforce: guilds. Worker guilds arose in theearly middle ages, when workers in a trade such as silk weredistributed across a large region, as bounded sets of laborerswho shared an affiliation. These guilds played many roles,including training apprentices [18, 44], setting prices [45],and providing mechanisms for collective action [52, 49]. Es-
CSCW 2017
![Page 36: Boomerang - Stanford HCI Group€¦ · boomerang rebounding the consequences of reputation feedback on crowdsourcing platforms neil (snehalkumar) s.gaikwad, durim morina, adam ginzberg,](https://reader034.vdocument.in/reader034/viewer/2022051806/60029cc36e309f666a50aac4/html5/thumbnails/36.jpg)
Boomerang REBOUNDING THE CONSEQUENCES OF REPUTATION FEEDBACK ON CROWDSOURCING PLATFORMS
STANFORD CROWD RESEARCH COLLECTIVEhttps://daemo.stanford.edu/