Crowd-Based Evaluation Methods

Social Computing @ Know-Center
1
Crowd-Based Evaluations
Christoph Trattner
Know-Center
ctrattner@know-center.at
@Graz University of Technology, Austria
. Christoph Trattner 15.12.2015 – Graz, Austria

2
Before I will start in this class, I will talk a bit about
myself (background), what I am doing, etc.

3
Where do I come from (Austria)?

4
Graz

5
Trondheim

6
Academic Back-Ground?
 Studies Computer Science at Graz University of
Technology & University of Pittsburgh
 Worked since 2009 as scientific researcher at the KMI &
IICM (BSc 2008, MSc 2009)
 My PhD thesis was on the Search & Navigation in Social
Tagging Systems (defended 2012)
 Since Feb. 2013 @ Know-Center
 Leading the SC Area @ KC
 @TUG
 WebScience
 Semantic Technologies
 @NTNU
 Datamining

7
What is my group doing?
… we research on novel methods and tools that exploit
social data to generate a greater value for the
individual, communities, companies and the society as
whole.
Our competences:
• Network & Web Science
• Science 2.0
• Crowdsourcing
• Predictive Modeling
• Social Network Analysis
• Information Quality Assessment
• User Modeling
• Machine Learning and Data Mining
• Collaborative Systems
Our Services:
• Social Analytics: Hub-, Expert -, Community -
, Influencer -, Information Flow-, Trend
(Event) Detection, etc.
• Information Quality Assessment
• Social & Location-based Recommander
Systems
• Customer Segmentation
• Social Systems Design

8
Ok, lets start….

9
Definition
Crowd
1. A large number of people gathering together.
2. A group of people.
3. A large number of things positioned or considered together.
Source: http://www.thefreedictionary.com/crowd

10
Definition
Crowdsourcing
is the process of obtaining needed services, ideas, or
content by soliciting contributions from a large group
of people, and especially from an online community,
rather than from traditional employees or suppliers.
Source: http://en.wikipedia.org/wiki/Crowdsourcing

11
Crowd-Based Evaluation
And what is crowd-based evaluation?
• Evaluation with a bunch (a lot) of people 
• In the literature also often referred as:
• „crowdsourcing“
• Evaluation with crowdsourcing platforms

12
Crowd-Based Studies vs. Lab Studies
Crowdsourcing
• Study-type: Controlled
• Payment: 0.1 USD
• Design: Between
subject
• Duration: Short
(seconds)
• Users: > 100
• Duration: minutes
Lab Studies
• Study-type: Controlled
• Payment: > 30USD
• Design: Within subject
• Duration: Long (hours)
• Users: < 100
• Duration: days/weeks

13
Is crowdsourcing something new?
1714 – The Longitude Prize: When the British government was
trying to find a way to measure a ship’s longitude, they offered
the public a monetary prize to whomever came up with the best
solution. (2014 http://longitudeprize.org/)
1884 – Oxford English Dictionary (OED): 800 volunteers
catalogued words to create the first fascicle of the OED
2001 – Launch of Wikipedia
2004 – Toyota’s first “Dream car art” contest: Children were asked
globally to draw their ‘dream car of the future.’
2006 – Jeff Howe coined the term crowdsourcing in Wired
(magazine).

14
Crowdsourcing Plattforms

15

16
Crowdsourcing Platforms for Evaluation
Amazon Mechanical Turk
URL: https://www.mturk.com/mturk/welcome
• Founded March 2007- 100,000 workers in over 100
countries.
• 2011 - over 500,000 workers from over 190 countries
in January 2011.
• Tasks = Hits
• Workers = Turkers

17
Mturk - workflow

18
What do I have to do
...as a turker?

19
Mtruk.com

20

21

22

23
What do I have to do
...as a hit submitter?

24

25

26

27
Source: http://dev.globis.ethz.ch/crowdstudy/res/eics2013.pdf

28

29

30

31

32
Who are the Turkers?

33
Mturk Study
Survey of 573 people
Ross, J., Irani, I., Silberman, M. Six, Zaldivar, A., and Tomlinson, B.
(2010). "Who are the Crowdworkers?: Shifting Demographics in Amazon
Mechanical Turk". In: CHI EA 2010. (2863-2872)

34
Mturk Study

35
Mturk Study

36
Mturk Stats
Source: http://mturk-tracker.com/general/general-data/

37
Mturk Alternatives
Biggest Limitation:
„Officially“ only available (as Hit submitter) in the US!
Alternatives:
• CrowdFlower
• Founded in 2007 in SF
• Over 153 other countries
• Jobs = Hits
• except from that more or less identical to MTurk

38
CrowdFlower.com

39
CrowdFlower - Stats
Source: http://venturebeat.com/2013/11/14/how-enterprises-use-crowdsourcing-infographic/

40
CrowdFlower - Stats

41
CrowdFlower - Stats

42
CrowdFlower - Stats

43
CrowdFlower - Stats

44
Mturk: Alternatives
Clickworker:
• Qualification: Little Knowledge – 2 pre-assessment
needed (language check)
• Registration Fee: free
• Min. Payment: 1USD
Example Tasks:
• Tagging
• Uploading videos
(on other sites)
Source: http://easyworkfromhomejobsonline.com/top-5-amazon-mechanical-turk-alternatives-sites-like-mturk/
http://clickworker.com

45
Mturk: Alternatives
Microworkers:
Qualification: Little Knowledge
Registration Fee: free
Min. Payment: 0.10 USD
Example Tasks:
• Twitter follower tasks
• Tagging
• Uploading videos
(on other sites)
Source: http://easyworkfromhomejobsonline.com/top-5-amazon-mechanical-turk-alternatives-sites-like-mturk/
http://microworkers.com

46
...is crowdsourcing a reasonable alternative for „real“
user studies?

47
Crowdsourcing User Studies
• In 2008 Kittur et al. conducted 2 experiments to test
the utility of Mechanical Turk as a user study platform
• In both experiments workers had to rate quality of
Wikipedia articles
• Comparative study – expert raters vs. turkers
Kittur, A., Chi, E. H., & Suh, B. (2008, April). Crowdsourcing user studies with Mechanical Turk.
In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 453-456).
ACM.

48
Experiment 1
Study Design
• 14 radomly chosen articles
• 7-point likert scale
• writing, factually accuracy, neutrality, structure...
• Questions - Wikipedia “Featured article criteria“
Results
• 58 users provided 210 ratings
• 93 ratings within 24hrs (very fast)
• Correlation was only marginally significant
r = 0.50, p = .07

49
„The results from Experiment 1 provided only weak
support for the utility of Mechanical Turk as a user
measurement tool.“

50
Experiment 2
Study Design
• More or less indendical to Experiment 1
• Contrary to Exp. 1, users had to input how many
references, images, and sections the article had.
• After that they were asked the overall article quality
question
Results
• 124 users provided 277 ratings
• Correlation was higher AND statistically significant
r = 0.66, p = 0.01

51
Stats Mturk Experiment

52
Other Applications
CrowdSearch
Yan, T., Kumar, V., & Ganesan, D. (2010, June). Crowdsearch: exploiting crowds for accurate
real-time image search on mobile phones. In Proceedings of the 8th international conference on
Mobile systems, applications, and services (pp. 77-90). ACM.

53
Eval search index

54
Eval Search index

55
Eval Search Index
Lin, Y., Trattner, C., Brusilovsky, P. and He, D.: The Impact of Image Descriptions on User Tagging
Behavior: A Study of the Nature and Functionality of Crowdsourced Tags, Journal of the Association for
Information Science and Technology, 2015. (in press)

56
...ok that‘s basically it 

57
Thank you!
Christoph Trattner
Email: ctrattner@know-center.at
Web: christophtrattner.info
Twitter: @ctrattner
Sponsors:

58
Any questions?

Crowd-Based Evaluation Methods

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (13)

Similar to Crowd-Based Evaluation Methods

Similar to Crowd-Based Evaluation Methods (20)

Recently uploaded

Recently uploaded (20)

Crowd-Based Evaluation Methods