crisp answers to fuzzy questions: design lessons for crowdsourcing decision inputs alex quinn, ben...
TRANSCRIPT
![Page 1: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/1.jpg)
Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs
Alex Quinn, Ben Bederson
![Page 2: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/2.jpg)
“Market research firm J.D. Power and Associates
says […] more than 80% of buyers have
already spent an average of 18 hours online researching models and prices, according to
Google data.” Wall Street Journal, 2/26/2013
R. L. Polk & Co. / Autotrader.com, Automotive Buyer Study, 2011
![Page 3: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/3.jpg)
Vacationitinerary
Location forheadquarters
Grad schoolapplications
Pediatrician Car Smartphone
DATA-DRIVEN DECISIONS
![Page 4: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/4.jpg)
![Page 5: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/5.jpg)
Building blocks for Mechanical Turk:
HITs(human intelligence task)
![Page 6: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/6.jpg)
![Page 7: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/7.jpg)
![Page 8: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/8.jpg)
![Page 9: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/9.jpg)
![Page 10: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/10.jpg)
![Page 11: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/11.jpg)
Keep instructions short.
Input labels should be unambiguous
HITs must be grouped by
common templates
See Mechanical Turk Requester Best Practices Guide
![Page 12: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/12.jpg)
Example #1: Find a pediatrician
Requirements
• Accepts my insurance
• ≥4 stars at RateMDs.com
• >80% positive at HealthGrades.com
• ≤15 minutes drive from home
![Page 13: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/13.jpg)
![Page 14: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/14.jpg)
![Page 15: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/15.jpg)
![Page 16: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/16.jpg)
![Page 17: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/17.jpg)
![Page 18: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/18.jpg)
![Page 19: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/19.jpg)
![Page 20: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/20.jpg)
![Page 21: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/21.jpg)
![Page 22: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/22.jpg)
![Page 23: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/23.jpg)
![Page 24: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/24.jpg)
![Page 25: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/25.jpg)
![Page 26: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/26.jpg)
![Page 27: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/27.jpg)
![Page 28: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/28.jpg)
![Page 29: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/29.jpg)
Effort should be proportional to
the reward.
HITs in a group share a common
base price.
Information sources should be
traceable.
See Mechanical Turk Requester Best Practices Guide
![Page 30: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/30.jpg)
Example #2: Buy a stroller
Requirements
• Fits a 30-pound baby
• Reclines for sleeping
• Medium/large-sized soft tires
• Can purchase online in US
![Page 31: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/31.jpg)
Bonus offers allow rewardto scale with effort
Find creative ways to track sources
![Page 32: Crisp Answers to Fuzzy Questions: Design lessons for crowdsourcing decision inputs Alex Quinn, Ben Bederson](https://reader036.vdocument.in/reader036/viewer/2022062409/56649f575503460f94c7c35d/html5/thumbnails/32.jpg)
Design lessons
1) Consider effort-reward balance from the start.
2) Look for implicit ways of capturing sources.
3) Use word economy to conserve vertical space.
4) Choose unambiguous input labels.
Alex [email protected]