1. Houdini
On Demand Labor API
Easy Mechanical Turk Integration
with Houdini
Philly AWS December Meetup
Date: December 13, 2010
By: Chris Conley and Jameson Detweiler
12/13/2010
Tuesday, December 14, 2010

2. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Mechanical Turk
• Marketplace for work
• ~ 500k workers
• Accessible via API
12/13/2010
Tuesday, December 14, 2010

3. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Task Examples
• Moderation - images, comments, reviews, video
• Categorization - products, best image, tagging
• Ads - moderation, keywords
• Audio Transcription / Video Captioning
• Cleanup - proofreading, contact reconciliation, review editing
• PDF/Image/Handwriting to Text
• Research - contact info, document search
12/13/2010
Tuesday, December 14, 2010

4. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Turk Overview
• Workers and Requesters
• HITs
• Questions
• Assignments
• Qualifications
• Notifications
• Blocking Workers
12/13/2010
Tuesday, December 14, 2010

5. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Worker View
12/13/2010
Tuesday, December 14, 2010

6. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Creating HITs
WYSIWYG API
• Easy • More Flexible
• Great for testing • But Complicated
the waters
• Programmatic
• No integration Integration
12/13/2010
Tuesday, December 14, 2010

7. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
WYSIWYG
12/13/2010
Tuesday, December 14, 2010

8. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
WYSIWYG
12/13/2010
Tuesday, December 14, 2010

9. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
API
http://docs.amazonwebservices.com/AWSMturkAPI/2008-08-02/index.html
12/13/2010
Tuesday, December 14, 2010

10. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
HIT Questions
• XML
• External Question
12/13/2010
Tuesday, December 14, 2010

11. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Turk API Libraries
• ruby-aws http://rubyforge.org/projects/ruby-aws/
• rTurk https://github.com/mdp/rturk
• Turkee https://github.com/aantix/turkee
12/13/2010
Tuesday, December 14, 2010

12. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Questions about Turk?
12/13/2010
Tuesday, December 14, 2010

13. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Houdini
• Simple
• Qualified results
• Automatic
• Trusted Workers
12/13/2010
Tuesday, December 14, 2010

14. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Houdini APIs
Generic Optimized
• Simple • Image Moderation
• Exact • Ad Moderation
• Multi-answer • Data Reconciliation
• Data Extraction
• ...
12/13/2010
Tuesday, December 14, 2010

15. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Simple API
• One answer per task
• Cheap
• Image Moderation, Low profile content
12/13/2010
Tuesday, December 14, 2010

16. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Simple API Example
12/13/2010
Tuesday, December 14, 2010

17. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Exact API
• Multiple workers
• Ensures all answers match
• Returns the “final answer”
• If not, escalated to trusted worker group
• More expensive than Simple
• Image Moderation, Data Harvesting, etc
12/13/2010
Tuesday, December 14, 2010

18. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Exact API Example
12/13/2010
Tuesday, December 14, 2010

19. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Multi-answer API
• Ask multiple turkers same question
• All answers are returned
• Surveys, feedback, answer qualification
on your end
12/13/2010
Tuesday, December 14, 2010

20. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Image Moderation API
• No need to build form_html
• Just send the image url
• Optionally send custom instructions
and possible responses
12/13/2010
Tuesday, December 14, 2010

21. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Image Moderation Example
12/13/2010
Tuesday, December 14, 2010

22. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Image Moderation Demo
Source: https://github.com/chrisconley/cat_or_not
12/13/2010
Tuesday, December 14, 2010

23. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Admin Console
• Review Work
• Perform “Trusted” tasks
• Worker Insights: Avg Completion Time,
# of workers
• Remove bad performing workers from
specific task groups
12/13/2010
Tuesday, December 14, 2010

24. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Getting Started
• Get an API Key - http://houdiniapi.com/sign-up/
• Tunnlr - http://tunnlr.com
• Sandbox - http://houdini-sandbox.heroku.com
12/13/2010
Tuesday, December 14, 2010

25. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Pricing Example
Monthly Fee Images Per Month
$20 200
$99 1500
12/13/2010
Tuesday, December 14, 2010

26. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Other Solutions
• Crowdflower
• Microtask
• CrowdCloud
12/13/2010
Tuesday, December 14, 2010

27. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Resources
Houdini Docs:
http://houdiniapi.com/documentation
Mturk Docs:
http://docs.amazonwebservices.com/AWSMturkAPI/2008-08-02/index.html
12/13/2010
Tuesday, December 14, 2010

28. Houdini Easy Mechanical Turk Integration with Houdini
On Demand Labor API Philly AWS December Meetup
Thanks!
Chris Conley Jameson Detweiler
chris@houdiniapi.com jameson@houdiniapi.com
@chrisconley @jaymstr
12/13/2010
Tuesday, December 14, 2010