With the advent and growing use of crowdsourcing labor markets for a variety of applications, optimizing the quality of results produced is of prime importance. The quality of the results produced is typically a function of the performance of crowd workers. In this paper, we investigate the notion of treating crowd workers as ‘learners’ in a novel learning environment. This learning context is characterized by a short-lived learning phase and immediate application of learned concepts. We draw motivation from the desire of crowd workers to perform well in order to maintain a good reputation, while attaining monetary rewards successfully. Thus, we delve into training workers in specific microtasks of different types. We exploit (i) implicit training, where workers are provided training when they provide erraneous responses to questions with priorly known answers, and (ii) explicit training, where workers are required to go through a training phase before they attempt to work on the task itself. We evaluated our approach in 4 different types of microtasks with a total of 1200 workers, who were subjected to either one of the proposed training strategies or baseline case of no training. The results show that workers who undergo training depict an improvement in performance upto 5 %, and a reduction in the task completion time upto 41 %. Additionally, crowd training led to the elimination of malicious workers and a costs-benefit gain upto nearly 15%.

1. Training Workers for Improving Performance in
Crowdsourcing Microtasks
Ujwal Gadiraju,
Besnik Fetahu, and Ricardo Kawase
ECTEL’15
Toledo, Spain
16th
September, 2015

2. Source: altamartv

3. Source: http://www.mission4636.org/

4. Dalila: I need Thomassin Apo please
Apo: Kenscoff Route: Lat: 18.495746829274168,
Long:-72.31849193572998
Apo: This Area after Petion-Ville and Pelerin
5 is not on Google Map. We have no streets
name
Apo: I know this place like my pocket
Dalila: thank God u was here
“just got emergency SMS, child delivery,
USCG are acting, and the GPS
coordinates of the location we got from
the translators were 100% accurate!”

5. ● People from over 50 countries
participated in relief efforts
● Free phone number 4636
● Maps about aid stations and food
distribution centers
● Sustainability: Created 100 jobs
Ahead of the curve in all relief efforts!
Mission 4636
HOW ?!
A triumph of people working together and
doing their small bits.

6. Crowdsourcing-A Brief Introduction
“The whole is greater than
the sum of its parts.”
- Aristotle
● Accumulating small
contributions from
each crowd worker to
solve a bigger
problem.

7. Crowdsourcing-A Brief Introduction
“The whole is greater than
the sum of its parts.”
- Aristotle
● Accumulating small
contributions from
each crowd worker to
solve a bigger
problem.

11. Some Remarkable Outcomes

12. Crowdsourcing Platforms
PAID MICROTASK
CROWDSOURCING

13. Outline
● Introduction
● Motivation
● Methodology
● Analysis & Results
● Conclusions
13
Training Workers for
Improving Performance
in Crowdsourcing
Microtasks

14. Motivation
● Crowdsourcing labor markets are being widely used nowadays.
● Optimizing the quality of results produced is of prime importance.
● At the same time, crowd workers aim to maintain high reputation
scores.
○ A worker’s good reputation gives access to more tasks.
○ A worker consequently can earn more money.

15. Research Questions
❏ How can crowd workers be trained
within crowdsourcing microtasks?
❏ What is the impact of training on the
performance of crowd workers?
❏ How does the impact of training differ
across various types of microtasks?

16. Crowd Workers as Learners
We recognize a crowd worker as a learner
in an atypical learning environment :
● There is no information regarding the
background, knowledge, or skills of a crowd
worker.
● Given the short nature of crowdsourced
microtasks, workers face an ‘on-the-fly’ learning
situation.
○ Comparable to experiential learning and
microlearning.
● In many cases, workers have no time to apply
their gained experience.
● Often for single use, high % of new requesters.

17. Training Strategies
Implicit Training Explicit Training
● Workers are prompted with
training information on
providing erroneous
responses to questions with
priorly known answers.
● Workers are prompted with
training information before
they begin responding to
questions within the actual
task.

18. Taxonomy of Microtasks
Information
Finding
Verification &
Validation
Interpretation
& Analysis
Content
Creation
Surveys
Content
Access
A Taxonomy of Microtasks on the Web.
Ujwal Gadiraju, Ricardo Kawase and Stefan Dietze.
In Proceedings of the 25th ACM Conference on
Hypertext and Social Media. 2014.
(IF) Finding
Middle-names
(VV) Spam Detection
(IA) Sentiment Analysis
(CC) Image Transcription

19. Experimental Setup and Tasks Design
For each type of task:
● 3 Experiments
○ No-Training (NT)
○ Implicit Training (IT)
○ Explicit Training (ET)
● Tasks deployed on CrowdFlower
● Responses from 100 independent crowd
workers for each training setup
● Monetary compensation of 5 USD cents
on successful task completion
● Randomized order of units, clear
instructions
● Task lengths to beget similar task
completion time for different types

20. Information Finding Verification & Validation
● Find middle-names
● 20 units in each task
● No-Training
● Implicit Training on
erroneous responses
● 5 units of Explicit Training
● Identify spam SMS from a
set of 5 SMSes
● 30 units in each task
● No-Training
● Implicit Training on
erroneous responses
● 5 units of Explicit Training
Error Reason:
Seemingly fake advertisements or
meaningless offers are good candidates of
being spam messages.

21. HAM or SPAM
Example Unit

22. Interpretation &
Analysis
Content Creation
● Assess sentiment of tweets
● 30 units in each task
● No-Training
● Implicit Training on
erroneous responses
● 5 units of Explicit Training ● Transcribe the text from
images (CAPTCHAs)
● 30 units in each task
● No-Training
● Implicit Training on
erroneous responses
● 5 units of Explicit Training

23. ● No Training: 92%
● Implicit Training:
96%
● Explicit Training:
98%
Results 1/4
Worker Performance
(accuracy)

24. ● No Training: 82%
● Implicit Training:
65%
● Explicit Training:
77%
Results 2/4
Worker Performance
(accuracy)

25. ● No Training: 86%
● Implicit Training:
89%
● Explicit Training:
92%
Results 3/4
Worker Performance
(accuracy)

26. ● No Training: 91.6%
● Implicit Training:
96.37%
● Explicit Training:
98.14%
Results 4/4
Worker Performance
(accuracy)

27. Impact of Training on Elite Workers

28. Impact of Training on Unit Completion Time

29. Impact of Training on Malicious Workers
Percentage of Malicious Workers across different types of Tasks

30. Costs-Benefit Analysis

31. ● Information Finding : Explicit Training
improves the costs-benefit ratio by 8.5%.
● Verification & Validation : Explicit
Training improves the costs-benefit ratio
by 1.4%.
● Interpretation & Analysis : Explicit
Training improves the costs-benefit ratio
by 14.6%.
● Content Creation : Explicit Training
improves the costs-benefit ratio by 8.5%.
Costs-Benefit Analysis

32. Caveats, Limitations and Future Work
● Quantum of Training
We need further experiments to
quantify the ideal duration/length of
training excerpts (both implicit and
explicit).
● Quality of Training Excerpts
Experiments that assess the impact of
varying training ‘quality’ on crowd
worker performance.
● Cost overheads due to training
● Impact of Task Complexity

33. Conclusions
● Training crowd workers in microtasks can lead
to substantial improvements in their
performance.
● Explicit Training leads to the most
improvement in individual worker performance
and overall quality of the results.
● We studied the effect of training across
different types of microtasks.
RQ #1
RQ #2
RQ #3
Training workers in crowdsourcing microtasks makes for an
interesting learning environment with opportunities galore!

34. Contact Details :
gadiraju@l3s.de
http://www.L3S.de
SLIDES: http://www.
slideshare.net/ujwal07/
34

35. Geographical Distribution of Workers
● No restrictions on worker
participation.
○ Information Finding: 50% from
India, 25% from Turkey
○ Verification and Validation: 53%
from Spain, 24% from Italy
○ Interpretation and Analysis: 37%
from Bosnia and Herzegovina,
31% from Turkey
○ Content Creation: 40% from
Philipines, 30% from Great Britain