Download - Crowd-Based Evaluation Methods
Social Computing @ Know-Center
1
. Christoph Trattner 15.12.2015 – Graz, Austria
Crowd-Based Evaluations
Christoph TrattnerKnow-Center
@Graz University of Technology, Austria
Social Computing @ Know-Center
2
. Christoph Trattner 15.12.2015 – Graz, Austria
Before I will start in this class, I will talk a bit about
myself (background), what I am doing, etc.
Social Computing @ Know-Center
3
. Christoph Trattner 15.12.2015 – Graz, Austria
Where do I come from (Austria)?
Social Computing @ Know-Center
4
. Christoph Trattner 15.12.2015 – Graz, Austria
Graz
Social Computing @ Know-Center
5
. Christoph Trattner 15.12.2015 – Graz, Austria
Trondheim
Social Computing @ Know-Center
6
. Christoph Trattner 15.12.2015 – Graz, Austria
Academic Back-Ground?
Studies Computer Science at Graz University of
Technology & University of Pittsburgh
Worked since 2009 as scientific researcher at the KMI &
IICM (BSc 2008, MSc 2009)
My PhD thesis was on the Search & Navigation in Social
Tagging Systems (defended 2012)
Since Feb. 2013 @ Know-Center Leading the SC Area @ KC
@TUG
WebScience
Semantic Technologies
@NTNU
Datamining
Social Computing @ Know-Center
7
. Christoph Trattner 15.12.2015 – Graz, Austria
What is my group doing?
… we research on novel methods and tools that exploit
social data to generate a greater value for the
individual, communities, companies and the society as
whole.
Our competences:• Network & Web Science
• Science 2.0
• Crowdsourcing
• Predictive Modeling
• Social Network Analysis
• Information Quality Assessment
• User Modeling
• Machine Learning and Data Mining
• Collaborative Systems
Our Services:• Social Analytics: Hub-, Expert -, Community -
, Influencer -, Information Flow-, Trend
(Event) Detection, etc.
• Information Quality Assessment
• Social & Location-based Recommander
Systems
• Customer Segmentation
• Social Systems Design
Social Computing @ Know-Center
8
. Christoph Trattner 15.12.2015 – Graz, Austria
Ok, lets start….
Social Computing @ Know-Center
9
. Christoph Trattner 15.12.2015 – Graz, Austria
Crowd-Based Evaluations
Definition
Crowd1. A large number of people gathering together.
2. A group of people.
3. A large number of things positioned or considered together.
Source: http://www.thefreedictionary.com/crowd
Social Computing @ Know-Center
10
. Christoph Trattner 15.12.2015 – Graz, Austria
Crowd-Based Evaluations
Definition
Crowdsourcing
is the process of obtaining needed services, ideas, or
content by soliciting contributions from a large group
of people, and especially from an online community,
rather than from traditional employees or suppliers.
Source: http://en.wikipedia.org/wiki/Crowdsourcing
Social Computing @ Know-Center
11
. Christoph Trattner 15.12.2015 – Graz, Austria
Crowd-Based Evaluation
And what is crowd-based evaluation?
• Evaluation with a bunch (a lot) of people
• In the literature also often referred as:• „crowdsourcing“
• Evaluation with crowdsourcing platforms
Social Computing @ Know-Center
12
. Christoph Trattner 15.12.2015 – Graz, Austria
Crowd-Based Studies vs. Lab Studies
Crowdsourcing
• Study-type: Controlled
• Payment: 0.1 USD
• Design: Between
subject
• Duration: Short
(seconds)
• Users: > 100
• Duration: minutes
Lab Studies
• Study-type: Controlled
• Payment: > 30USD
• Design: Within subject
• Duration: Long (hours)
• Users: < 100
• Duration: days/weeks
Social Computing @ Know-Center
13
. Christoph Trattner 15.12.2015 – Graz, Austria
Is crowdsourcing something new?
1714 – The Longitude Prize: When the British government was
trying to find a way to measure a ship’s longitude, they offered
the public a monetary prize to whomever came up with the best
solution. (2014 http://longitudeprize.org/)
1884 – Oxford English Dictionary (OED): 800 volunteers
catalogued words to create the first fascicle of the OED
2001 – Launch of Wikipedia
2004 – Toyota’s first “Dream car art” contest: Children were asked
globally to draw their ‘dream car of the future.’
2006 – Jeff Howe coined the term crowdsourcing in Wired
(magazine).
Social Computing @ Know-Center
14
. Christoph Trattner 15.12.2015 – Graz, Austria
Crowdsourcing Plattforms
Social Computing @ Know-Center
15
. Christoph Trattner 15.12.2015 – Graz, Austria
Social Computing @ Know-Center
16
. Christoph Trattner 15.12.2015 – Graz, Austria
Crowdsourcing Platforms for Evaluation
Amazon Mechanical Turk
URL: https://www.mturk.com/mturk/welcome
• Founded March 2007- 100,000 workers in over 100
countries.
• 2011 - over 500,000 workers from over 190 countries
in January 2011.
• Tasks = Hits
• Workers = Turkers
Social Computing @ Know-Center
17
. Christoph Trattner 15.12.2015 – Graz, Austria
Mturk - workflow
Social Computing @ Know-Center
18
. Christoph Trattner 15.12.2015 – Graz, Austria
What do I have to do
...as a turker?
Social Computing @ Know-Center
19
. Christoph Trattner 15.12.2015 – Graz, Austria
Mtruk.com
Social Computing @ Know-Center
20
. Christoph Trattner 15.12.2015 – Graz, Austria
Social Computing @ Know-Center
21
. Christoph Trattner 15.12.2015 – Graz, Austria
Social Computing @ Know-Center
22
. Christoph Trattner 15.12.2015 – Graz, Austria
Social Computing @ Know-Center
23
. Christoph Trattner 15.12.2015 – Graz, Austria
What do I have to do
...as a hit submitter?
Social Computing @ Know-Center
24
. Christoph Trattner 15.12.2015 – Graz, Austria
Social Computing @ Know-Center
25
. Christoph Trattner 15.12.2015 – Graz, Austria
Social Computing @ Know-Center
26
. Christoph Trattner 15.12.2015 – Graz, Austria
Social Computing @ Know-Center
27
. Christoph Trattner 15.12.2015 – Graz, Austria
Source: http://dev.globis.ethz.ch/crowdstudy/res/eics2013.pdf
Social Computing @ Know-Center
28
. Christoph Trattner 15.12.2015 – Graz, Austria
Social Computing @ Know-Center
29
. Christoph Trattner 15.12.2015 – Graz, Austria
Social Computing @ Know-Center
30
. Christoph Trattner 15.12.2015 – Graz, Austria
Social Computing @ Know-Center
31
. Christoph Trattner 15.12.2015 – Graz, Austria
Social Computing @ Know-Center
32
. Christoph Trattner 15.12.2015 – Graz, Austria
Who are the Turkers?
Social Computing @ Know-Center
33
. Christoph Trattner 15.12.2015 – Graz, Austria
Mturk Study
Ross, J., Irani, I., Silberman, M. Six, Zaldivar, A., and Tomlinson, B.
(2010). "Who are the Crowdworkers?: Shifting Demographics in Amazon
Mechanical Turk". In: CHI EA 2010. (2863-2872)
Survey of 573 people
Social Computing @ Know-Center
34
. Christoph Trattner 15.12.2015 – Graz, Austria
Mturk Study
Social Computing @ Know-Center
35
. Christoph Trattner 15.12.2015 – Graz, Austria
Mturk Study
Social Computing @ Know-Center
36
. Christoph Trattner 15.12.2015 – Graz, Austria
Mturk Stats
Source: http://mturk-tracker.com/general/general-data/
Social Computing @ Know-Center
37
. Christoph Trattner 15.12.2015 – Graz, Austria
Mturk Alternatives
Biggest Limitation:
„Officially“ only available (as Hit submitter) in the US!
Alternatives:
• CrowdFlower• Founded in 2007 in SF
• Over 153 other countries
• Jobs = Hits
• except from that more or less identical to MTurk
Social Computing @ Know-Center
38
. Christoph Trattner 15.12.2015 – Graz, Austria
CrowdFlower.com
Social Computing @ Know-Center
39
. Christoph Trattner 15.12.2015 – Graz, Austria
CrowdFlower - Stats
Source: http://venturebeat.com/2013/11/14/how-enterprises-use-crowdsourcing-infographic/
Social Computing @ Know-Center
40
. Christoph Trattner 15.12.2015 – Graz, Austria
CrowdFlower - Stats
Social Computing @ Know-Center
41
. Christoph Trattner 15.12.2015 – Graz, Austria
CrowdFlower - Stats
Social Computing @ Know-Center
42
. Christoph Trattner 15.12.2015 – Graz, Austria
CrowdFlower - Stats
Social Computing @ Know-Center
43
. Christoph Trattner 15.12.2015 – Graz, Austria
CrowdFlower - Stats
Social Computing @ Know-Center
44
. Christoph Trattner 15.12.2015 – Graz, Austria
Mturk: Alternatives
Clickworker:
• Qualification: Little Knowledge – 2 pre-assessment
needed (language check)
• Registration Fee: free
• Min. Payment: 1USD
Example Tasks:
• Tagging
• Uploading videos
(on other sites)
Source: http://easyworkfromhomejobsonline.com/top-5-amazon-mechanical-turk-alternatives-sites-like-mturk/
http://clickworker.com
Social Computing @ Know-Center
45
. Christoph Trattner 15.12.2015 – Graz, Austria
Mturk: Alternatives
Microworkers:
Qualification: Little Knowledge
Registration Fee: free
Min. Payment: 0.10 USD
Example Tasks:
• Twitter follower tasks
• Tagging
• Uploading videos
(on other sites)Source: http://easyworkfromhomejobsonline.com/top-5-amazon-mechanical-turk-alternatives-sites-like-mturk/
http://microworkers.com
Social Computing @ Know-Center
46
. Christoph Trattner 15.12.2015 – Graz, Austria
...is crowdsourcing a reasonable alternative for „real“
user studies?
Social Computing @ Know-Center
47
. Christoph Trattner 15.12.2015 – Graz, Austria
Crowdsourcing User Studies
• In 2008 Kittur et al. conducted 2 experiments to test
the utility of Mechanical Turk as a user study platform
• In both experiments workers had to rate quality of
Wikipedia articles
• Comparative study – expert raters vs. turkers
Kittur, A., Chi, E. H., & Suh, B. (2008, April). Crowdsourcing user studies with Mechanical Turk.
In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 453-456).
ACM.
Social Computing @ Know-Center
48
. Christoph Trattner 15.12.2015 – Graz, Austria
Experiment 1
Study Design
• 14 radomly chosen articles
• 7-point likert scale
• writing, factually accuracy, neutrality, structure...
• Questions - Wikipedia “Featured article criteria“
Results
• 58 users provided 210 ratings
• 93 ratings within 24hrs (very fast)
• Correlation was only marginally significant
r = 0.50, p = .07
Social Computing @ Know-Center
49
. Christoph Trattner 15.12.2015 – Graz, Austria
„The results from Experiment 1 provided only weak
support for the utility of Mechanical Turk as a user
measurement tool.“
Social Computing @ Know-Center
50
. Christoph Trattner 15.12.2015 – Graz, Austria
Experiment 2
Study Design
• More or less indendical to Experiment 1
• Contrary to Exp. 1, users had to input how many
references, images, and sections the article had.
• After that they were asked the overall article quality
question
Results
• 124 users provided 277 ratings
• Correlation was higher AND statistically significant
r = 0.66, p = 0.01
Social Computing @ Know-Center
51
. Christoph Trattner 15.12.2015 – Graz, Austria
Stats Mturk Experiment
Social Computing @ Know-Center
52
. Christoph Trattner 15.12.2015 – Graz, Austria
Other Applications
CrowdSearch
Yan, T., Kumar, V., & Ganesan, D. (2010, June). Crowdsearch: exploiting crowds for accurate
real-time image search on mobile phones. In Proceedings of the 8th international conference on
Mobile systems, applications, and services (pp. 77-90). ACM.
Social Computing @ Know-Center
53
. Christoph Trattner 15.12.2015 – Graz, Austria
Eval search index
Social Computing @ Know-Center
54
. Christoph Trattner 15.12.2015 – Graz, Austria
Eval Search index
Social Computing @ Know-Center
55
. Christoph Trattner 15.12.2015 – Graz, Austria
Eval Search Index
Lin, Y., Trattner, C., Brusilovsky, P. and He, D.: The Impact of Image Descriptions on User Tagging
Behavior: A Study of the Nature and Functionality of Crowdsourced Tags, Journal of the Association for
Information Science and Technology, 2015. (in press)
Social Computing @ Know-Center
56
. Christoph Trattner 15.12.2015 – Graz, Austria
...ok that‘s basically it
Social Computing @ Know-Center
57
. Christoph Trattner 15.12.2015 – Graz, Austria
Thank you!
Christoph Trattner
Email: [email protected]
Web: christophtrattner.info
Twitter: @ctrattner
Sponsors:
Social Computing @ Know-Center
58
. Christoph Trattner 15.12.2015 – Graz, Austria
Any questions?