cs 5306 info 5306: crowdsourcing and human computationhirsh/5306/lecture4.pdf · 2017-09-05 ·...
TRANSCRIPT
![Page 1: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/1.jpg)
CS 5306INFO 5306:
Crowdsourcing andHuman Computation
Lecture 48/31/17
Haym Hirsh
![Page 2: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/2.jpg)
Upcoming Speakers
• Thursday, Aug 31, 4:15 (after class)Louis Hyman. The Return of The Independent Workforce: The History and The Future of Work(extra credit)
• Thursday, Sep 7, 4:15 (after class)Henry Kautz, Mining Social Media to Improve Public Health(extra credit)
• Tuesday, Sep 12 (in class)Serge Belongie
• Thursday, Sep 28, 4:15 (after class)Michael Bernstein(extra credit)
![Page 3: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/3.jpg)
“Prehistory” of Human Computation
![Page 4: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/4.jpg)
![Page 5: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/5.jpg)
![Page 6: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/6.jpg)
Characterizing Crowdsourcing and Human Computation Systems
• “Overt” vs “Covert”: Are human participants explicitly participating to achieve the collective outcomes, or is some form of mining of human activity achieving the collective outcomes
– Overt: Amazon reviews, Wikipedia
– Covert (“Crowd Mining”): Google, Amazon recommendations
![Page 7: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/7.jpg)
Characterizing Crowdsourcing and Human Computation Systems
• What are they doing?– Collecting
– Collaborative Creation
– Smartest in the Crowd
– Collaborative Decisions
– Micro-Labor
– Mining User Behavior• Search logs
• Social media
![Page 8: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/8.jpg)
Characterizing Crowdsourcing and Human Computation Systems
• What are they doing?– Collecting
– Collaborative Creation
– Smartest in the Crowd
– Collaborative Decisions
– Micro-Labor
– Mining User Behavior• Search logs
• Social media
![Page 9: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/9.jpg)
![Page 10: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/10.jpg)
![Page 11: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/11.jpg)
![Page 12: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/12.jpg)
![Page 13: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/13.jpg)
![Page 14: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/14.jpg)
![Page 15: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/15.jpg)
![Page 16: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/16.jpg)
Characterizing Crowdsourcing and Human Computation Systems
• What are they doing?– Collecting
– Collaborative Creation
– Smartest in the Crowd
– Collaborative Decisions
– Micro-Labor
– Mining User Behavior• Search logs
• Social media
![Page 18: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/18.jpg)
![Page 19: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/19.jpg)
![Page 20: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/20.jpg)
Characterizing Crowdsourcing and Human Computation Systems
• What are they doing?– Collecting
– Collaborative Creation
– Smartest in the Crowd
– Collaborative Decisions
– Micro-Labor
– Mining User Behavior• Search logs
• Social media
![Page 21: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/21.jpg)
![Page 22: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/22.jpg)
![Page 23: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/23.jpg)
![Page 24: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/24.jpg)
![Page 25: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/25.jpg)
![Page 26: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/26.jpg)
Characterizing Crowdsourcing and Human Computation Systems
• What are they doing?– Collecting
– Collaborative Creation
– Smartest in the Crowd
– Collaborative Decisions
– Micro-Labor
– Mining User Behavior• Search logs
• Social media
![Page 27: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/27.jpg)
![Page 28: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/28.jpg)
![Page 29: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/29.jpg)
“Prediction Markets”
![Page 30: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/30.jpg)
![Page 31: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/31.jpg)
![Page 32: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/32.jpg)
Characterizing Crowdsourcing and Human Computation Systems
• What are they doing?– Collecting
– Collaborative Creation
– Smartest in the Crowd
– Collaborative Decisions
– Micro-Labor
– Mining User Behavior• Search logs
• Social media
![Page 33: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/33.jpg)
![Page 34: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/34.jpg)
Copyright 2011 Haym Hirsh
![Page 35: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/35.jpg)
![Page 36: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/36.jpg)
“Games with a Purpose”
![Page 37: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/37.jpg)
![Page 38: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/38.jpg)
![Page 39: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/39.jpg)
• Introduction: Broad overview of collective intelligence, a framework for understanding it, and its various connections to machine learning– Animals to humans images/videos
– CI mosaic (icons, then list of keywords)
– CI, crowdsourcing, human computation
– Overt vs covert• Collaborative creation
• Collaborative decision making
• Smartest in the crowd / contests
• HC and micro-labor markets
• Crowd mining
– Roles of Machine Learning
![Page 40: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/40.jpg)
Characterizing Crowdsourcing and Human Computation Systems
• What are they doing?– Collecting
– Collaborative Creation
– Smartest in the Crowd
– Collaborative Decisions
– Micro-Labor
– Mining User Behavior• Search logs
• Social media
![Page 41: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/41.jpg)
5 September 2017 Copyright 2011 Haym Hirsh 41
![Page 42: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/42.jpg)
5 September 2017 Copyright 2011 Haym Hirsh 42
![Page 43: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/43.jpg)
5 September 2017 Copyright 2011 Haym Hirsh 43
![Page 44: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/44.jpg)
5 September 2017 Copyright 2011 Haym Hirsh 44
![Page 45: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/45.jpg)
5 September 2017 Copyright 2011 Haym Hirsh 45
![Page 46: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/46.jpg)
“Games with a Purpose”Luis von Ahn
IEEE Computer, June 2006
![Page 47: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/47.jpg)
![Page 48: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/48.jpg)
“A game-theoretic analysis of games with a purpose”S. Jain and D. Parkes
Internet and Network Economics, pp.342-350, 2008
![Page 49: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/49.jpg)
Important Game Theory Concepts
• Nash Equilibrium: A strategy for each player to take actions wherein if either player changes his/her strategy, the outcome will be worse for that actor
• Bayes Game: Each player has information that is unknown to the other player, but the other player has a probability distribution over what that information might be
• Bayes Nash Equilibrium: If either player changes his/her strategy, the expected value of the outcome will be worse, given the probability distribution
![Page 50: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/50.jpg)
Result
• The ESPGame incentivizes players to take a strategy of playing the most common words first• It’s a Bayes Nash Equilibrium
• It yields the best response even for playing otherwise
• This might not match up with the goals of the game designer
![Page 51: CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 · Upcoming Speakers •Thursday, Aug 31, 4:15 (after class) Louis Hyman. The Return of](https://reader033.vdocument.in/reader033/viewer/2022042802/5f3b56803c445c1e5020aaa7/html5/thumbnails/51.jpg)
Readings for Next Time
• Yu, L., André, P., Kittur, A. and Kraut, R., 2014, February. A comparison of social, learning, and financial strategies on crowd engagement and output quality. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing (pp. 967-978). ACM.
• Mason, Winter, and Duncan J. Watts. "Financial incentives and the performance of crowds." Proceedings HComp 2009. http://crowdsourcing-class.org/readings/downloads/econ/financial-incentives-and-the-performance-of-crowds.pdf