crowd-based evaluation methods
TRANSCRIPT
![Page 1: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/1.jpg)
Social Computing @ Know-Center
1
. Christoph Trattner 15.12.2015 – Graz, Austria
Crowd-Based Evaluations
Christoph TrattnerKnow-Center
@Graz University of Technology, Austria
![Page 2: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/2.jpg)
Social Computing @ Know-Center
2
. Christoph Trattner 15.12.2015 – Graz, Austria
Before I will start in this class, I will talk a bit about
myself (background), what I am doing, etc.
![Page 3: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/3.jpg)
Social Computing @ Know-Center
3
. Christoph Trattner 15.12.2015 – Graz, Austria
Where do I come from (Austria)?
![Page 4: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/4.jpg)
Social Computing @ Know-Center
4
. Christoph Trattner 15.12.2015 – Graz, Austria
Graz
![Page 5: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/5.jpg)
Social Computing @ Know-Center
5
. Christoph Trattner 15.12.2015 – Graz, Austria
Trondheim
![Page 6: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/6.jpg)
Social Computing @ Know-Center
6
. Christoph Trattner 15.12.2015 – Graz, Austria
Academic Back-Ground?
Studies Computer Science at Graz University of
Technology & University of Pittsburgh
Worked since 2009 as scientific researcher at the KMI &
IICM (BSc 2008, MSc 2009)
My PhD thesis was on the Search & Navigation in Social
Tagging Systems (defended 2012)
Since Feb. 2013 @ Know-Center Leading the SC Area @ KC
@TUG
WebScience
Semantic Technologies
@NTNU
Datamining
![Page 7: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/7.jpg)
Social Computing @ Know-Center
7
. Christoph Trattner 15.12.2015 – Graz, Austria
What is my group doing?
… we research on novel methods and tools that exploit
social data to generate a greater value for the
individual, communities, companies and the society as
whole.
Our competences:• Network & Web Science
• Science 2.0
• Crowdsourcing
• Predictive Modeling
• Social Network Analysis
• Information Quality Assessment
• User Modeling
• Machine Learning and Data Mining
• Collaborative Systems
Our Services:• Social Analytics: Hub-, Expert -, Community -
, Influencer -, Information Flow-, Trend
(Event) Detection, etc.
• Information Quality Assessment
• Social & Location-based Recommander
Systems
• Customer Segmentation
• Social Systems Design
![Page 8: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/8.jpg)
Social Computing @ Know-Center
8
. Christoph Trattner 15.12.2015 – Graz, Austria
Ok, lets start….
![Page 9: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/9.jpg)
Social Computing @ Know-Center
9
. Christoph Trattner 15.12.2015 – Graz, Austria
Crowd-Based Evaluations
Definition
Crowd1. A large number of people gathering together.
2. A group of people.
3. A large number of things positioned or considered together.
Source: http://www.thefreedictionary.com/crowd
![Page 10: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/10.jpg)
Social Computing @ Know-Center
10
. Christoph Trattner 15.12.2015 – Graz, Austria
Crowd-Based Evaluations
Definition
Crowdsourcing
is the process of obtaining needed services, ideas, or
content by soliciting contributions from a large group
of people, and especially from an online community,
rather than from traditional employees or suppliers.
Source: http://en.wikipedia.org/wiki/Crowdsourcing
![Page 11: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/11.jpg)
Social Computing @ Know-Center
11
. Christoph Trattner 15.12.2015 – Graz, Austria
Crowd-Based Evaluation
And what is crowd-based evaluation?
• Evaluation with a bunch (a lot) of people
• In the literature also often referred as:• „crowdsourcing“
• Evaluation with crowdsourcing platforms
![Page 12: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/12.jpg)
Social Computing @ Know-Center
12
. Christoph Trattner 15.12.2015 – Graz, Austria
Crowd-Based Studies vs. Lab Studies
Crowdsourcing
• Study-type: Controlled
• Payment: 0.1 USD
• Design: Between
subject
• Duration: Short
(seconds)
• Users: > 100
• Duration: minutes
Lab Studies
• Study-type: Controlled
• Payment: > 30USD
• Design: Within subject
• Duration: Long (hours)
• Users: < 100
• Duration: days/weeks
![Page 13: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/13.jpg)
Social Computing @ Know-Center
13
. Christoph Trattner 15.12.2015 – Graz, Austria
Is crowdsourcing something new?
1714 – The Longitude Prize: When the British government was
trying to find a way to measure a ship’s longitude, they offered
the public a monetary prize to whomever came up with the best
solution. (2014 http://longitudeprize.org/)
1884 – Oxford English Dictionary (OED): 800 volunteers
catalogued words to create the first fascicle of the OED
2001 – Launch of Wikipedia
2004 – Toyota’s first “Dream car art” contest: Children were asked
globally to draw their ‘dream car of the future.’
2006 – Jeff Howe coined the term crowdsourcing in Wired
(magazine).
![Page 14: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/14.jpg)
Social Computing @ Know-Center
14
. Christoph Trattner 15.12.2015 – Graz, Austria
Crowdsourcing Plattforms
![Page 15: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/15.jpg)
Social Computing @ Know-Center
15
. Christoph Trattner 15.12.2015 – Graz, Austria
![Page 16: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/16.jpg)
Social Computing @ Know-Center
16
. Christoph Trattner 15.12.2015 – Graz, Austria
Crowdsourcing Platforms for Evaluation
Amazon Mechanical Turk
URL: https://www.mturk.com/mturk/welcome
• Founded March 2007- 100,000 workers in over 100
countries.
• 2011 - over 500,000 workers from over 190 countries
in January 2011.
• Tasks = Hits
• Workers = Turkers
![Page 17: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/17.jpg)
Social Computing @ Know-Center
17
. Christoph Trattner 15.12.2015 – Graz, Austria
Mturk - workflow
![Page 18: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/18.jpg)
Social Computing @ Know-Center
18
. Christoph Trattner 15.12.2015 – Graz, Austria
What do I have to do
...as a turker?
![Page 19: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/19.jpg)
Social Computing @ Know-Center
19
. Christoph Trattner 15.12.2015 – Graz, Austria
Mtruk.com
![Page 20: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/20.jpg)
Social Computing @ Know-Center
20
. Christoph Trattner 15.12.2015 – Graz, Austria
![Page 21: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/21.jpg)
Social Computing @ Know-Center
21
. Christoph Trattner 15.12.2015 – Graz, Austria
![Page 22: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/22.jpg)
Social Computing @ Know-Center
22
. Christoph Trattner 15.12.2015 – Graz, Austria
![Page 23: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/23.jpg)
Social Computing @ Know-Center
23
. Christoph Trattner 15.12.2015 – Graz, Austria
What do I have to do
...as a hit submitter?
![Page 24: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/24.jpg)
Social Computing @ Know-Center
24
. Christoph Trattner 15.12.2015 – Graz, Austria
![Page 25: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/25.jpg)
Social Computing @ Know-Center
25
. Christoph Trattner 15.12.2015 – Graz, Austria
![Page 26: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/26.jpg)
Social Computing @ Know-Center
26
. Christoph Trattner 15.12.2015 – Graz, Austria
![Page 27: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/27.jpg)
Social Computing @ Know-Center
27
. Christoph Trattner 15.12.2015 – Graz, Austria
Source: http://dev.globis.ethz.ch/crowdstudy/res/eics2013.pdf
![Page 28: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/28.jpg)
Social Computing @ Know-Center
28
. Christoph Trattner 15.12.2015 – Graz, Austria
![Page 29: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/29.jpg)
Social Computing @ Know-Center
29
. Christoph Trattner 15.12.2015 – Graz, Austria
![Page 30: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/30.jpg)
Social Computing @ Know-Center
30
. Christoph Trattner 15.12.2015 – Graz, Austria
![Page 31: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/31.jpg)
Social Computing @ Know-Center
31
. Christoph Trattner 15.12.2015 – Graz, Austria
![Page 32: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/32.jpg)
Social Computing @ Know-Center
32
. Christoph Trattner 15.12.2015 – Graz, Austria
Who are the Turkers?
![Page 33: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/33.jpg)
Social Computing @ Know-Center
33
. Christoph Trattner 15.12.2015 – Graz, Austria
Mturk Study
Ross, J., Irani, I., Silberman, M. Six, Zaldivar, A., and Tomlinson, B.
(2010). "Who are the Crowdworkers?: Shifting Demographics in Amazon
Mechanical Turk". In: CHI EA 2010. (2863-2872)
Survey of 573 people
![Page 34: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/34.jpg)
Social Computing @ Know-Center
34
. Christoph Trattner 15.12.2015 – Graz, Austria
Mturk Study
![Page 35: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/35.jpg)
Social Computing @ Know-Center
35
. Christoph Trattner 15.12.2015 – Graz, Austria
Mturk Study
![Page 36: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/36.jpg)
Social Computing @ Know-Center
36
. Christoph Trattner 15.12.2015 – Graz, Austria
Mturk Stats
Source: http://mturk-tracker.com/general/general-data/
![Page 37: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/37.jpg)
Social Computing @ Know-Center
37
. Christoph Trattner 15.12.2015 – Graz, Austria
Mturk Alternatives
Biggest Limitation:
„Officially“ only available (as Hit submitter) in the US!
Alternatives:
• CrowdFlower• Founded in 2007 in SF
• Over 153 other countries
• Jobs = Hits
• except from that more or less identical to MTurk
![Page 38: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/38.jpg)
Social Computing @ Know-Center
38
. Christoph Trattner 15.12.2015 – Graz, Austria
CrowdFlower.com
![Page 39: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/39.jpg)
Social Computing @ Know-Center
39
. Christoph Trattner 15.12.2015 – Graz, Austria
CrowdFlower - Stats
Source: http://venturebeat.com/2013/11/14/how-enterprises-use-crowdsourcing-infographic/
![Page 40: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/40.jpg)
Social Computing @ Know-Center
40
. Christoph Trattner 15.12.2015 – Graz, Austria
CrowdFlower - Stats
![Page 41: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/41.jpg)
Social Computing @ Know-Center
41
. Christoph Trattner 15.12.2015 – Graz, Austria
CrowdFlower - Stats
![Page 42: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/42.jpg)
Social Computing @ Know-Center
42
. Christoph Trattner 15.12.2015 – Graz, Austria
CrowdFlower - Stats
![Page 43: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/43.jpg)
Social Computing @ Know-Center
43
. Christoph Trattner 15.12.2015 – Graz, Austria
CrowdFlower - Stats
![Page 44: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/44.jpg)
Social Computing @ Know-Center
44
. Christoph Trattner 15.12.2015 – Graz, Austria
Mturk: Alternatives
Clickworker:
• Qualification: Little Knowledge – 2 pre-assessment
needed (language check)
• Registration Fee: free
• Min. Payment: 1USD
Example Tasks:
• Tagging
• Uploading videos
(on other sites)
Source: http://easyworkfromhomejobsonline.com/top-5-amazon-mechanical-turk-alternatives-sites-like-mturk/
http://clickworker.com
![Page 45: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/45.jpg)
Social Computing @ Know-Center
45
. Christoph Trattner 15.12.2015 – Graz, Austria
Mturk: Alternatives
Microworkers:
Qualification: Little Knowledge
Registration Fee: free
Min. Payment: 0.10 USD
Example Tasks:
• Twitter follower tasks
• Tagging
• Uploading videos
(on other sites)Source: http://easyworkfromhomejobsonline.com/top-5-amazon-mechanical-turk-alternatives-sites-like-mturk/
http://microworkers.com
![Page 46: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/46.jpg)
Social Computing @ Know-Center
46
. Christoph Trattner 15.12.2015 – Graz, Austria
...is crowdsourcing a reasonable alternative for „real“
user studies?
![Page 47: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/47.jpg)
Social Computing @ Know-Center
47
. Christoph Trattner 15.12.2015 – Graz, Austria
Crowdsourcing User Studies
• In 2008 Kittur et al. conducted 2 experiments to test
the utility of Mechanical Turk as a user study platform
• In both experiments workers had to rate quality of
Wikipedia articles
• Comparative study – expert raters vs. turkers
Kittur, A., Chi, E. H., & Suh, B. (2008, April). Crowdsourcing user studies with Mechanical Turk.
In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 453-456).
ACM.
![Page 48: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/48.jpg)
Social Computing @ Know-Center
48
. Christoph Trattner 15.12.2015 – Graz, Austria
Experiment 1
Study Design
• 14 radomly chosen articles
• 7-point likert scale
• writing, factually accuracy, neutrality, structure...
• Questions - Wikipedia “Featured article criteria“
Results
• 58 users provided 210 ratings
• 93 ratings within 24hrs (very fast)
• Correlation was only marginally significant
r = 0.50, p = .07
![Page 49: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/49.jpg)
Social Computing @ Know-Center
49
. Christoph Trattner 15.12.2015 – Graz, Austria
„The results from Experiment 1 provided only weak
support for the utility of Mechanical Turk as a user
measurement tool.“
![Page 50: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/50.jpg)
Social Computing @ Know-Center
50
. Christoph Trattner 15.12.2015 – Graz, Austria
Experiment 2
Study Design
• More or less indendical to Experiment 1
• Contrary to Exp. 1, users had to input how many
references, images, and sections the article had.
• After that they were asked the overall article quality
question
Results
• 124 users provided 277 ratings
• Correlation was higher AND statistically significant
r = 0.66, p = 0.01
![Page 51: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/51.jpg)
Social Computing @ Know-Center
51
. Christoph Trattner 15.12.2015 – Graz, Austria
Stats Mturk Experiment
![Page 52: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/52.jpg)
Social Computing @ Know-Center
52
. Christoph Trattner 15.12.2015 – Graz, Austria
Other Applications
CrowdSearch
Yan, T., Kumar, V., & Ganesan, D. (2010, June). Crowdsearch: exploiting crowds for accurate
real-time image search on mobile phones. In Proceedings of the 8th international conference on
Mobile systems, applications, and services (pp. 77-90). ACM.
![Page 53: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/53.jpg)
Social Computing @ Know-Center
53
. Christoph Trattner 15.12.2015 – Graz, Austria
Eval search index
![Page 54: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/54.jpg)
Social Computing @ Know-Center
54
. Christoph Trattner 15.12.2015 – Graz, Austria
Eval Search index
![Page 55: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/55.jpg)
Social Computing @ Know-Center
55
. Christoph Trattner 15.12.2015 – Graz, Austria
Eval Search Index
Lin, Y., Trattner, C., Brusilovsky, P. and He, D.: The Impact of Image Descriptions on User Tagging
Behavior: A Study of the Nature and Functionality of Crowdsourced Tags, Journal of the Association for
Information Science and Technology, 2015. (in press)
![Page 56: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/56.jpg)
Social Computing @ Know-Center
56
. Christoph Trattner 15.12.2015 – Graz, Austria
...ok that‘s basically it
![Page 57: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/57.jpg)
Social Computing @ Know-Center
57
. Christoph Trattner 15.12.2015 – Graz, Austria
Thank you!
Christoph Trattner
Email: [email protected]
Web: christophtrattner.info
Twitter: @ctrattner
Sponsors:
![Page 58: Crowd-Based Evaluation Methods](https://reader036.vdocument.in/reader036/viewer/2022062419/559ea1351a28abe2618b4591/html5/thumbnails/58.jpg)
Social Computing @ Know-Center
58
. Christoph Trattner 15.12.2015 – Graz, Austria
Any questions?