1. Pick-A-Crowd: Tell Me What You Like,
and I’ll Tell You What to Do
A Crowdsourcing Platform for Personalized
Human Intelligence Task Assignment Based on Social
Networks
Djellel E. Difallah, GianlucaDemartini, Philippe Cudré-Mauroux
eXascaleInfolab
University of Fribourg, Switzerland
15th May 2013, WWW 2013 - Rio De Janeiro, Brazil
1

2. Crowdsourcing
• Exploit human intelligence to solve tasks that
are simple for Humans and complex for
machines
• Examples:
– Wikipedia, reCaptcha, Duolingo
• Incentives
– Financial, fun, visibility
2

3. Motivation
• The Pull Methodology is suboptimal
Actual workers
Max Overlap
Effective workers
3

4. Motivation
• The Push Methodology is a Task-to-Worker
Recommender System.
4

5. Contribution and Claim
• Pick-A-Crowd: A system architecture that uses
Task-to-Worker matching:
– The worker’s social profile
– The task context
• Workers can provide higher quality answers
on tasks they relate to
5

6. Worker Social Profiling
“YouAreWhatYouLike”
7

7. Problem Definition (1)The Human Intelligence Task (HIT)
Categorization
Survey
Image Tagging
Data Collection
Batch of Tasks:
Title
Batch Instruction
Specific task instruction*
Task data:
- Text.
- Options.
- Additional data (image, Url)
List of categories*
8

8. ProblemDefinition (2)The Worker
Completed HITs: 256
Approval Rate: 96%
Qualification Types
Generic Qualifications
Page:
Page:
Page:
- -Title
Title
- Title
- -Category
Category
- Category
- -Description
Description
- Description
- -Feed, etc.
Feed, etc.
- Feed, etc.
9

9. Problem Definition (3) –
Task-to-Worker Matching
Batch of Tasks:
Title
Batch Instruction
Specific task instruction*
Task data:
- Text.
- Options.
- Additional data (image, Url)
List of categories*
Page:
Page:
Page:
- -Title
Title
- Title
- -Category
Category
- Category
- -Description
Description
- Description
- -Feed, etc.
Feed, etc.
- Feed, etc.
1- Task-to-Page Matching Function
- Category
- Expert finding
- Semantic
2- Worker Ranking
10

10. Matching Models (1/3)–
Category Based
• The requester provides a list of categories related to the batch
• We create a subset of pages whose category is in the category
list of the batch
• Rank the workers by the number of liked pages in the subset
11

11. Matching Models (2/3) –
Expert Finding
•
•
•
Build an inverted index on the pages’ titles and description
Use the title/description of the tasks as a key word query on the
inverted index and get a subset of pages
Rank the workers by the number of liked pages in the subset
12

12. Matching Models (3/3) –
Semantic Based
•
•
Link the context to an external knowledge base (e.g., DBPedia)
Exploit the underlying graph structure to determine the Hits and Pages similarity
– Assumption that a worker who likes a page is able to answer questions about related entities
– Worker who likes a page is able to answer questions about entities of the same type
•
Rank the workers by the number of liked pages in the subset
Similarity
Relatedness
HIT
FB Pages
Type-Similarity
13

13. Pick-A-Crowd Architecture
15

14. Experimental Evaluation
• The Facebook app OpenTurkimplements part
of the Pick-A-Crowd architecture:
– More than 170 registered workers participated
– Over 12k pages crawled
• Covered both multiple answer questions as
well as open-ended questions
– 50 images with multiple choice question and 5 candidate answers
(Soccer, Actors, Music, Authors,Movies, Animes)
– Answer 20 open-ended questions related to the topic (Cricket)
16

15. OpenTurk app
18

16. Evaluation -
WORKER PRECISION
Correlation between the crowd accuracy and
the number of relevant likes (Category Based)
NUMBER OF RELEVANT LIKES
19

17. Evaluation (Baseline) –
Amazon Mechanical Turk (AMT)
AMT 3 = Majority vote of 3 workers
AMT 5 = Majority vote of 5 workers
20

18. Evaluation –
HIT Assignment Models
CATEGORY APPROACH
21

19. Evaluation –
HIT Assignment Models
EXPERT FINDING BASED
TITLE/INSTRUCTION
CONTENT
22

20. Evaluation –
HIT Assignment Models
SEMANTIC BASED
TYPE
RELATEDNESS
23

21. PICK-A-CROWD
AMT
Evaluation Comparison With Mechanical Turk
24

22. Conclusions and Future Work
• Pull vs. Pushmethodologies in Crowdsourcing
• Pick-A-Crowd system architecture with Taskto-Worker recommendation
• Experimental comparison with AMT shows a
consistent quality improvement
“Workers Know what they Like”
• Exploit more of the social activity, and handle
content-less tasks
25

23. Next Step
• We are building a Crowdsourcing platform for
the research community
• Pre-register on:
www.openturk.com
Thank You!
26