1.
THE CROWD MACHINE Elena Simperl
e.simperl@soton.ac.uk

2.
CROWDSOURCING: PROBLEM SOLVING VIA OPEN
CALLS
2
"the act of a company or institution taking a function once performed by
employees and outsourcing it to an undefined (and generally large) network of
people in the form of an open call. This can take the form of peer-production
(when the job is performed collaboratively), but is also often undertaken by
sole individuals. The crucial prerequisite is the use of the open call format and
the large network of potential laborers.“
[Howe, 2006]

3.
TYPOLOGY OF CROWDSOURCING
Macrotasks Microtasks
Challenges
Self-
organized
crowds
Crowdfunding
3
Source:[Prpić, Shukla, Kietzmann and McCarthy 2015]

4.
SOCIAL MACHINES: PEOPLE DO THE CREATIVE
WORK AND MACHINES DO THE ADMINISTRATION
"Real life is and must be full of all kinds of social constraint – the very
processes from which society arises. Computers can help if we use them to
create abstract social machines on the Web: processes in which the people do
the creative work and the machine does the administration […] The stage is set
for an evolutionary growth of new social engines.
[Tim Berners-Lee, Weaving the Web, 1999
]

5.
Are social
machines just
crowdsourcing?

6.
CROWDSOURCING & SOCIALITY

7.
CROWDSOURCING CREATIVE TASKS

8.
CROWDSOURCING & ETHICS
[Difallah et al, 2015]

9.
CROWDSOURCING RESEARCH
TASK DESIGN TASK ASSIGNMENT QUALITY
ASSURANCE
INCENTIVES
ENGINEERING
WORKFLOW
DESIGN AND
EXECUTION
CROWD TRAINING SELF-ORGANIZING
CROWDS
COLLABORATIVE
CROWDSOURCING
REAL-TIME DELIVERY EXTENSIONS TO
TECHNOLOGIES
PROGRAMMING
LANGUAGES

10.
GAMIFYING PAID MICROTASKS
Improving paid microtasks through
gamification and adaptive
furtherance incentives
O Feyisetan, E Simperl, M Van
Kleek, N Shadbolt
WWW 2015, 333-343
11

11.
OVERVIEW
Make paid microtasks more
cost-effective though
gamification*
*use of game elements and mechanics in
a non-game context
12
[Source: http://www.hideandseek.net/wp-
content/uploads/2010/10/gamification_badges.jpg]

12.
PLATFORM
Image labelling tasks published on
microtask platform
Free-text labels, varying numbers of labels
per image, taboo words
1st setting: ‘standard’ tasks, including basic
spam control
2nd setting: same requirements and rewards,
but contributors were asked to carry out the
task in Wordsmith
13

13.
RESULTS
BETTER, CHEAPER, BUT FEWER WORKERS
14
Metric CrowdFlower Wordsmith
Total workers 600 423
Total keywords 1,200 41,206
Unique keywords 111 5,708
Avg. agreement 5.72% 37.7%
Mean images/person 1 32
Max images/person 1 200

14.
RESULTS (NO INCENTIVES)
COMPARABLE QUALITY, HIGHER UNIT COSTS, FEWER DROPOUTS
15
Metric CrowdFlower Wordsmith
Total workers 600 514
Total keywords 13,200 35,890
Unique keywords 1,323 4,091
Avg. agreement 6.32% 10.9%
Mean images/person 11 27
Max images/person 1 351

15.
RESULTS (WITH INCENTIVES)
Increased participation
 People come back (20 times) and play longer (43 hours vs. 3 hours without
incentives)
Targeted incentives work
 77% players stayed vs. 27% in the randomised condition
 19% more labels compared to no incentives condition
16

16.
CONCLUSIONS & FUTURE WORK
MAIN FINDINGS
Task design matters as much as
payment
Gamification achieves high
accuracy for lower costs and
improved engagement
Top contributors appreciate
social features, but their main
motivation is still task-driven
NEXT STEPS
The effect of individual
gamification elements
The effect of task autonomy
(allowing people to skip tasks)
Best ways to retain contributors
The effect of referrals
Application of SAPS to GWAPs
and financially motivated crowds
17