1. Chris Conley
chris@chrisconley.me

2. Mechanical Turk
Marketplace for Work
• HITs - Human Intelligence Tasks
• Turkers - Mechanical Turk Workers
• Requesters ~ Employers
• Example HITs:
• Product Categorization ($0.03)
• Edit Zappos Product Reviews ($0.05)
• Find Company Contact Info ($0.02)

3. Mechanical Turk
Marketplace for Work
• HITs - Human Intelligence Tasks
• Turkers - Mechanical Turk Workers
• Requesters ~ Employers
• Example HITs:
• Product Categorization ($0.03)
• Edit Zappos Product Reviews ($0.05)
• Find Company Contact Info ($0.02)
Artificial Artificial Intelligence

4. Problem
• Time and effort still required for
• Qualifying Turkers
• Reviewing Work

5. Problem
• Time and effort still required for
• Qualifying Turkers
• Reviewing Work
Automatic
Artificial Artificial Intelligence

6. Trustification Tactics
• Qualification Tests
• Classifying Turkers
• Turker Reviewed Work
• Spam Filtering

7. Qualification Tests
• Qualification AND Training

8. Qualification Tests
• Qualification AND Training
Work Quality 5
Worker Pool 1
Ease of Implementation 4
Automatic-ness 3

9. Classifying Workers
• Track Trusted / Rejected

10. Classifying Workers
• Track Trusted / Rejected
Work Quality 5
Worker Pool 5
Ease of Implementation 3
Automatic-ness 3

11. Turker Reviewed Work
• Compare / Majority Vote
• Work Then Verify

12. Turker Reviewed Work
• Compare / Majority Vote
• Work Then Verify
Work Quality 3.5
Worker Pool 5
Ease of Implementation 1
Automatic-ness 5

13. Spam Filtering
• Examples: Avg work duration, # of approved
HITs, answer content

14. Spam Filtering
• Examples: Avg work duration, # of approved
HITs, answer content
Work Quality 3
Worker Pool 5
Ease of Implementation 2
Automatic-ness 4

15. Trustification Tactics
• Qualification Tests
• Classifying Turkers
• Turker Reviewed Work
• Spam Filtering
Questions?

16. Event data collection with Mechanical Turk
- Andy Guy
Basecamp Business

17.  Business Events
Calendar
− Wide coverage
− Input events to
sustain coverage
 Sultan
− Internal web app
− Large API use
We are the Sultans

18.  External form
 Worker Qualifications
− Test, Hand picked
 Event split into 2 HITs
− Event, Address
 Multiple (2) HITs per
event
Process Highlights

19. Sultan Event Acquisition Process
Source
URL

20. Sultan Event Acquisition Process
Source
URL
HIT HIT HIT HIT
Event Event Address Address

21. Sultan Event Acquisition Process
Source
URL
HIT HIT HIT HIT
Event Event Address Address
E E E E E E E E A A A A A A A A
E E E E E E E E A A A A A A A A

22. Sultan Event Acquisition Process
Source
URL
HIT HIT HIT HIT
Event Event Address Address
E E E E E E E E A A A A A A A A
E E E E E E E E A A A A A A A A
Group
Name
Start Date
Start Time

23. Sultan Event Acquisition Process
Source
URL
HIT HIT HIT HIT
Event Event Address Address
E E E E E E E E A A A A A A A A
E E E E E E E E A A A A A A A A
Group E E
Name
Start Date E E
Start Time
E E

24. Sultan Event Acquisition Process
Source
URL
HIT HIT HIT HIT
Event Event Address Address
E E E E E E E E A A A A A A A A
E E E E E E E E A A A A A A A A
Group E E Match
Name Name
Start Date E E Start Date
Start Time URL
E E

25. Sultan Event Acquisition Process
Source
URL
HIT HIT HIT HIT
Event Event Address Address
E E E E E E E E A A A A A A A A
E E E E E E E E A A A A A A A A
Group E E Match
Name Name
Start Date E E Start Date
Start Time URL
E E
Reviewable
Event

26. Sultan Event Acquisition Process
Source
URL
HIT HIT HIT HIT
Event Event Address Address
E E E E E E E E A A A A A A A A
E E E E E E E E A A A A A A A A
Group E E Match
Name Name
Start Date E E Start Date
Start Time URL
E E
Resolve discrepancies Reviewable
and Event
Approve

27.  Identify workers
 Communicate and motivate workers
 Fine tune price
Keys

28. Satyajeet Shahade

31. 6- 25 /hr $400 -
$1600
$1500 –
$500 – $4500
$3000

33. Outsourcing on oDesk
Creating a Worldwide Team
Phil Ives
April 20, 2010 @ Philly Startup
Leaders

34. Process for oDesk Projects
• Define value of in-house staff – stick to it
• Scope Small
• How to hire… or how I learned to
understand the NCAA tournament seeds
• Trust, but verify
• Build a real team, get related
• SCR-AGILEo – My day to day process

35. What would you say ya do here?

36. Define Value in the process
• Hard • Harder
– Define your and local Stick to what you said
staff’s value. When timelines slip…
– What can’t be stick to what you said
outsourced? When people turnover…
– If you don’t have know.. stick to what you said
better rethink your
career

37. Bite size is better

38. Scope Small
• Scope early and often
• Keep tasks small for accountability
• Check in with the freelancer early and
often

39. How to Hire
• What’s your time worth?
• What would an interview process on
email and annoying Skype give you?
• Why would you do that to yourself?
• Hire 2x - 3x actual need and weed out.

40. Trust, but Verify
• Like anything important to your business
– Verify that work is correct and complete
– Create tests and validation procedures
– Spot check
– Use the tools you have: Team Room, Bugzilla,
Skype
– Have a conversation….

41. Build a Team
• Really, they aren’t actually “mechanical”
• Manage them like your local team
• Use the technology to have conversations
• Get acquainted: Who are they / what goals?
• Where possible, let them create from
nothing
• Inspire them with the goals of your project

42. SCR-AGILEo
• Loose agile process
• Daily Check-in
• Timelines, coordinate time of days
• Don’t work with consultants who can’t
work on your schedule - make them
stick to it.
• Iterate both the product and your
process for managing it