These are slides from a presentation I gave on the TurKit paper and toolkit by Greg Little and others at MIT.

1. TurKit: Tools for Iterative Tasks on Mechanical Turk Paper by Greg Little, Lydia B. Chilton, Rob Miller, and Max Goldman (MIT CSAIL) Presented by Sanjay Kairam (Stanford)

2. Human Computation There are still some tasks that are too difficult for computers to do well. Examples: Labeling Images Tagging Documents Proofreading Text Writing Novel Content Simple solution: Get humans to do it!

3. Motivation and Participation Why would humans want to do these things? Reputation (Q&A Sites, Review Sites) Contribution (Wikipedia) Fun (Games with a Purpose) $$$

4. Amazon Mechanical Turk Marketplace for “Human Intelligence Tasks” (HITs). Small amounts of money per task.

5. Amazon Mechanical Turk

6. Traditional Workflow HIT HIT Data Collected in CSV File Requester Posts HIT Group to Mechanical Turk HIT Data Exported for Use HIT HIT HIT

7. Traditional Workflow: Pros & Cons Easy to run simple, parallelized tasks. Not so easy to run tasks in which turkers improve on or validate each others’ work. TurKit to the rescue!

8. Iterative Improvement and Voting improved text output original Vote Improve

9. “Improve” and “Vote” Tasks Improve-HIT: Improve a given body of text according to some instructions. Vote-HIT: Choose which of two bodies of text is better according to some criteria.

10. “Improve Task” Please improve the description for this image. People will vote whether to approve your changes. Use no more than 500 characters. “Vote Task” Please select the better description for this image. Your vote must agree with the majority to be approved. Differences are highlighted in yellow. Image Description

11. Image Description Version 1: A parial view of a pocket calculator together with some coins and a pen.

12. Image Description Version 2: A view of personal items a calculator, and some gold and copper coins, and a round tip pen, these are all pocket and wallet sized item used for business, writing, calculating prices or solving math problems and purchasing items. [Voted down in favor of Version 1]

13. Image Description Version 3: A close-up photograph of the following items: A CASIO multi-function calculator A ball point pen, uncapped Various coins, apparently European, both copper and gold Seems to be a theme illustration for a brochure or document cover treating finance, probably personal finance.

14. Image Description Version 4: [Builds on Version 3] …Various British coins; two of £1 value, three of 20p value and one of 1p value... Further versions continued to iterate on Version 3.

15. Experiment: Iterative vs. Single HIT 11 image description tasks In each task: Image selected randomly from set of 10 Budget chosen (either $0.25 or $0.50) Iterative HIT Condition: $0.02/Improve & $0.01/Vote Single HIT Condition: Single HIT with $0.25 or $0.50 Reward

16. Experiment: Iterative vs. Single HIT 20 Turkers voted on each pair. Votes favored the iteratively generated description in 9 out of 11 cases (82%).

17. Writing Tasks: Outline to Prose Improve letter to convey this outline: Somehow I need this to sound diplomatic: I am grateful for job offer I got another offer they pay $10,000 more I’ll work for you if you can match their offer Version 1: I want to thank you for your gracious job offer. However, I have had another job offer willing to pay me $10,000 more for the same job. But, I am willing to forgoe that job offer because working with your company would be a great opportunity, that is, if you would like to match their offer.

18. Writing Tasks: Outline to Prose Improve letter to convey this outline: Somehow I need this to sound diplomatic: I am grateful for job offer I got another offer they pay $10,000 more I’ll work for you if you can match their offer Version 3: I would like to thank you very much for your offer, however, I have received a better package from another firm. I still would like to work for company and would like to speak with you to see if there is anyway we could make that occur.

19. Writing Tasks: Outline to Prose Improve letter to convey this outline: Somehow I need this to sound diplomatic: I am grateful for job offer I got another offer they pay $10,000 more I’ll work for you if you can match their offer Version 9: Thank you very much for your offer. At this time, I have a competing offer on the table at a higher salary. If the remuneration packages were equal I would much prefer to work for yourselves, and I am open to discussing the possibility of your matching this offer. If this is possible, please contact me at your earliest convenience.

20. Handwriting Recognition Version 1: You (?) (?) (?) (work). (?) (?) (?) work (not) (time). I (?) (?) a few grammatical mistakes. Overall your writing style is a bit too (phoney). You do (?) have good (points), but they got lost amidst the (writing). (signature)

21. Handwriting Recognition Version 2: You (?) (?) (?) (work). (?) (?) (?) work (not) (time). I (?) (?) a few grammatical mistakes. Overall your writing style is a bit too (phoney). You do (?) have good (points), but they got lost amidst the (writing). (signature)

22. Handwriting Recognition Version 4: You (misspelled) (several) (words). (?) (?) (?) work next (time). I also notice a few grammatical mistakes. …

23. Handwriting Recognition Version 5: You (misspelled) (several) (words). (Plan?) (spellcheck) (your) work next time. I also notice a few grammatical mistakes. Overall your writing style is a bit too phoney. You do make some good (points), but they got lost amidst the (writing). (signature)

24. Handwriting Recognition Final Version: You (misspelled) (several) (words). Please spellcheck your work next time. I also notice a few grammatical mistakes. Overall your writing style is a bit too phoney. You do make some good (points), but they got lost amidst the (writing). (signature)

25. Task Time & Cost

26. The TurKit Toolkit Turkers Arrows indicate the flow of information. Programmer writes 2 sets of source code: HTML files for web servers JavaScript executed by TurKit Output is retrieved via a JavaScript database. Mechanical Turk Web Server TurKit JavaScript Database *.js *.html Programmer

27. TurKit APIs MTurk API: JavaScript wrapper for the MTurk API Trace API: Uses the database to store information about program execution Utility API: Covers some common higher level MTurk tasks. waitForHit: accepts a HIT ID and returns a JavaScript object containing answers. vote: manages a HIT where turkers vote between two or more options. sort: takes two parameters and a comprator

28. TurKit Demo Iterative Text Improvement

29. TurKit Demo: Properties Fields to define: Mode = {Sandbox, Offline, Real} maxMoney & maxHITs = budget, HIT limits repeatInterval = wait time before re-running script You will also need: AWS Developer Access Key AWS Secret Key

30. TurKit Demo: Code // improve text varhitID = createImproveHIT(text, 0.02) var hit = mturk.waitForHIT(hitId) varnewText = hit.assignments[0].answer.newText print(“-------------”) print(newText) print(“-------------”)

31. TurKit Demo: Code // verify improvement if (vote(text, newText, 0.01)) { text = newText mturk.approveAssignment(hit.assignments[0]) print(“vote = keep”) } else { Mturk.rejectAssignment(hit.assignments[0]) print(“vote = reject”)

32. TurKit Demo: Code HITs are created using XML schemas defined by Amazon. Referred to by URL: Example:http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2005-10-01/QuestionForm.xsd;