Extracts and addendums from an earlier talk, for those interested in ethics and related issues in regard to crowdsourcing, particularly research uses. Slides updated Sept. 2, 2013.

1. Extracts and Addendums:
Ethics & Crowd Computing
Matt Lease
School of Information @mattlease
University of Texas at Austin ml@ischool.utexas.edu

2. What about context?
By minimizing context, greater task efficiency
can often be achieved
– e.g. “Can you name who is in this photo?”
2

3. Context & Informed Consent
• But people need context for informed consent
• Who will benefit from it and how?
Jonathan Zittrain, Harvard 3

4. What about requestor fraud?
• Refusing to pay for work, installing malware
• Using crowds to defeat online metrics/security
4

5. Robert Sim, MSR Summit’12
5

6. What about regulation?
• Wolfson and Lease. Look before you leap: Legal pitfalls of
crowdsourcing. Proceedings of the American Society for
Information Science and Technology (ASIS&T), 2011.
• As usual, technology is ahead of the law
– employment law
– patent inventorship
– data security and the Federal Trade Commission
– copyright ownership
– securities regulation of crowdfunding
• Take-away: don’t panic, but be mindful
– Understand risks of “just in-time compliance”
6

7. Who are
the workers?
• A. Baio, November 2008. The Faces of Mechanical Turk.
• P. Ipeirotis. March 2010. The New Demographics of
Mechanical Turk
• J. Ross, et al. Who are the Crowdworkers? CHI 2010.
7

8. What about ethics?
• Six Silberman, Lilly Irani, and Joel Ross. Ethics and
tactics of professional crowdwork. XRDS: Crossroads,
The ACM Magazine for Students 17.2 (2010): 39-43.
– “How should we… conceptualize the role of these people
who we ask to power our computing?”
– Power dynamics
– “Abstraction hides detail”
• Fort, Karën, Gilles Adda, and K. Bretonnel Cohen.
Amazon mechanical turk: Gold mine or coal mine?.
Computational Linguistics 37.2 (2011): 413-420.
– “…opportunities for our community to deliberately
value ethics above cost savings.”
8

9. Davis et al. (2010) The HPU.
HPU
9

10. HPU: “Abstraction hides detail”
10

11. Requester Fraud on MTurk
“Do not do any HITs that involve: filling in
CAPTCHAs; secret shopping; test our web page;
test zip code; free trial; click my link; surveys or
quizzes (unless the requester is listed with a
smiley in the Hall of Fame/Shame); anything
that involves sending a text message; or
basically anything that asks for any personal
information at all—even your zip code. If you
feel in your gut it’s not on the level, IT’S NOT.
Why? Because they are scams...”
11

12. Fraud example
• Is this really the requester’s captcha?
12

13. Captcha Fraud
• Severity?
13

14. • “…not only do malicious crowd-sourcing
systems exist, but they are rapidly growing…”
14

15. Digital Dirty Jobs
• The Googler who Looked at the Worst of the Internet
• Policing the Web’s Lurid Precincts
• Facebook content moderation
• The dirty job of keeping Facebook clean
• Even linguistic annotators report stress &
nightmares from reading news articles!
15

16. Crowdsourcing and Vigilantism
• http://searchenginewatch.com/article/2221527/Western-Human-Flesh-
Search-Chases-Wrong-Guy
16

17. What about freedom?
• Crowdsourcing vision: empowering freedom
– work whenever you want for whomever you want
• Risk: people being compelled to perform work
– Chinese prisoners farming gold…
– Digital sweat shops? Digital slaves?
– We really don’t know (and need to learn more…)
– Traction? Human Trafficking at MSR Summit’12
17

18. Mechanical Turk is Not Anonymous
Matthew Lease, Jessica Hullman, Jeffrey P. Bigham, Michael S. Bernstein, Juho Kim,
Walter S. Lasecki, Saeideh Bakhshi, Tanushree Mitra, and Robert C. Miller.
Online: Social Science Research Network, March 6, 2013
ssrn.com/abstract=2190946

19. Worker Privacy
Each worker is assigned an alphanumeric ID
19

20. Requesters see only Worker IDs
20

21. Safeguarding Personal Data
•
“What are the characteristics of MTurk workers?... the MTurk
system is set up to strictly protect workers’ anonymity….”
21

22. Broad Perception of Anonymity
22

23. `
Amazon profile page
URLs use the same
IDs used on MTurk !
Who knew?
23

24. Workers’ Views: Survey & Forums
• “... my reviewer profile is linked to my Mturk number! I had
no idea...”
• “...Amazon needs to separate the Mturk numbers from
seller numbers to protect our privacy…”
• “I think this is outrageous though. Makes me concerned
about trusting privacy agreements.”
• “Mine pulled up my Amazon wish list which revealed my
identity. It seems to me that so called ”anonymous” tasks
on mTurk (like surveys) are not anonymous after all.”
24

25. Risks to
Workers
• Inadvertent disclosure of PII or private data
• Loss of blind hiring practices online
• Greater risk of exploitation, reputation damage,
loss of income, or even physical harm…
25

26. Risks to Researchers
• Exposing participants to undocumented risks
• Having disclosed WorkerIDs (e.g., online)
• Having not restricted access to the internally
– Potential harm to participants
– Lack of compliance with Federal/IRB governance
of human subjects research
– Being required to discard collected data
– Delays or inability to conduct future MTurk studies
26

27. Risks to Service Provider
• Workers/Requesters abandoning MTurk
• The US Federal Trade Commission (FTC) has
recently begun to aggressively protect consumers
from data breaches by commercial entities,
including release of supposedly “anonymous” data
– Inadequate protection of customer records: BJWC
– De-anonymized customer records: AOL, Netflix
– Did workers have a reasonable expectation of privacy
in their use of MTurk which has been violated? 27

28. The Future of Crowd Work, CSCW’13
Kittur, Nickerson, Bernstein, Gerber,
Shaw, Zimmerman, Lease, and Horton
28

29. Additional References
• Paul Hyman. Software Aims to Ensure Fairness in Crowdsourcing Projects.
Communications of the ACM, Vol. 56 No. 8, Pages 19-21, August 2013.
• Irani, Lilly C. The Ideological Work of Microwork. In preparation, draft
available online.
• Irani, Lilly C., and Six Silberman. Turkopticon: Interrupting worker
invisibility in amazon mechanical turk. ACM SIGCHI Conference, 2013.
• Adda, Gilles, et al. Crowdsourcing for language resource development:
Critical analysis of amazon mechanical turk overpowering use. Proceedings
of the 5th Language and Technology Conference (LTC). 2011.
• Adda, Gilles, and Joseph J. Mariani. Economic, Legal and Ethical analysis of
Crowdsourcing for Speech Processing. (2013).
• Harris, Christopher G., and Padmini Srinivasan. Crowdsourcing and Ethics.
Security and Privacy in Social Networks. 67-83. 2013.
• Harris, Christopher G. Dirty Deeds Done Dirt Cheap: A Darker Side to
Crowdsourcing. IEEE 3rd conference on social computing (socialcom). 2011.
• Horton, John J. The condition of the Turking class: Are online employers
fair and honest?. Economics Letters 111.1 (2011): 10-12.
29

30. A Few Legal References
• Felstiner, Alek. Working the Crowd: Employment
and Labor Law in the Crowdsourcing Industry.
Berkeley Journal of Employment and Labor Law
32.1 (2011).
• Felstiner, Alek. Sweatshop or Paper Route?: Child
Labor Laws and In-Game Work. Proceedings of
CrowdConf (2010).
• Zittrain, Jonathan. Ubiquitous human computing.
Philosophical Transactions of the Royal Society A:
Mathematical, Physical and Engineering Sciences
366.1881 (2008): 3813-3821.
30

31. Join the conversation online
• Crowdwork-ethics: “an informal and
occasional newsletter for researchers
interested in ethical issues in crowd work.”
– Email Six Silberman to subscribe (manual)
31