A wide-scale bottom-up approach to the creation and management of open data has been demonstrated by projects like Freebase, Wikipedia, and DBpedia. This talk explores how to involving a wide community of users in collaborative management of open data activities within a Smart City. The talk discusses how crowdsourcing techniques can be applied within a Smart City context using crowdsourcing and human computation platforms such as Amazon Mechanical Turk, Mobile Works, and Crowd Flower.

1. Crowdsourcing Approaches for
Smart City Open Data Management
Edward Curry & Adegboyega Ojo
Insight @ NUI Galway
ed.curry@insight-centre.org
www.edwardcurry.org

2. About Me
• Researcher in both Computer
Science and Information
Systems
• Green and Sustainable IT
Research Group Leader in
DERI/Insight NUI Galway

3. Some Background
Multi-year research on state
of research and practice of
smart cities to inform Next
Generation Smart City Design
and Policy
Part of an International Smart Cities
Research/Practice Consortium
composed of international research
teams from the US, Canada, Mexico,
Colombia, China and Ireland.

4. Designing Next Generation Smart City
Initiatives - SCID
Ojo, A., Curry, E., and Janowski, T. 2014. “Designing Next Generation Smart City Initiatives - Harnessing
Findings And Lessons From A Study Of Ten Smart City Programs,” in 22nd European Conference on
Information Systems (ECIS 2014)

5. Open Data as a Smart City Imitative
Ojo, A., Curry, E., and Sanaz-Ahmadi, F. 2015. “A Tale of Open Data Innovations in Five Smart
Cities,” in 48th Annual Hawaii International Conference on System Sciences (HICSS-48)

6. Open Data Powering Smart Cities
Economy Energy Environment Education
Health &
Wellbeing
Tourism Mobility Grovenance

7. An Open Innovation Economy
Initial findings of the study are consistent and
support the notion of an open data oriented
smart city as an:
“Open Innovation Economy”
We are now investigating Crowdsourcing as a
means of increasing Citizen engagement and
participation within a smart city’s open
innovation ecosystem

8. Introduction to Crowdsourcing
 Coordinating a crowd (a large group of workers)to do
micro-work (small tasks) that solves problems (that
computers or a single user can’t)
 A collection of mechanisms and associated
methodologies for scaling and directing
crowd activities to achieve goals
 Related Areas
 Collective Intelligence
 Social Computing
 Human Computation
 Data Mining
A. J. Quinn and B. B. Bederson, “Human computation: a survey and taxonomy of a growing field,” in
Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems, 2011, pp. 1403–
1412.

9. 9
Crowdsourcing Landscape

10. When Computers Were Human
 Maskelyne 1760
Used human computers to
created almanac of moon
positions
– Used for shipping/navigation
Quality assurance
– Do calculations twice
– Compare to third verifier
D. A. Grier, When Computers Were Human, vol.
13. Princeton University Press, 2005.

11. When Computers Were Human

12. Audio Tagging - Tag a Tune

13. Image Tagging - Peekaboom

14. Protein Folding - Fold.it/

15. ReCaptcha
 OCR
 ~ 1% error rate
 20%-30% for 18th and 19th
century books
 40 million ReCAPTCHAs
every day” (2008)
 Fixing 40,000 books a day

16. Enterprise Examples
 Categorize millions of products on eBay’s catalog
with accurate and complete attributes
 Combine the crowd with machine learning to
create an affordable and flexible catalog quality
system
 Understanding customer sentiment for launch
of new product around the world.
 Implemented 24/7 sentiment analysis system
with workers from around the world.
 90% accuracy in 95% on content

17. Spatial Crowdsourcing
 Spatial Crowdsoucring requires a person to travel to a
location to preform a spatial task
 Helps non-local requesters through workers in targeted spatial
locality
 Used for data collection, package routing, citizen actuation
 Usually based on mobile applications
 Closely related to social sensing, participatory sensing, etc.
 Early example Ardavark social search

18. Sensing
Credits: Albany Associates, stuartpilrow, Mike_n (Flickr)
Computation Actuation
Human Powered
Smart Cities
Leverages human capabilities in conjunction
with machine capabilities for optimizing
processes in the cyber-physical-social
environments

19. Citizen Sensors
“…humans as citizens on the ubiquitous Web, acting as
sensors and sharing their observations and views…”
 Sheth, A. (2009). Citizen sensing, social signals, and enriching human
experience. Internet Computing, IEEE, 13(4), 87-92.
Air Pollution

20. Crisis Response

22. Citizens as Sensors

23. Haklay, M., 2013, Citizen Science and Volunteered Geographic Information – overview and typology of participation in Sui,
D.Z., Elwood, S. and M.F. Goodchild (eds.), 2013. Crowdsourcing Geographic Knowledge: Volunteered Geographic
Information (VGI) in Theory and Practice . Berlin: Springer.
23

24. Human vs Machine Affordances
Human
 Visual perception
 Visuospatial thinking
 Audiolinguistic ability
 Sociocultural awareness
 Creativity
 Domain knowledge
Machine
 Large-scale data
manipulation
 Collecting and storing
large amounts of data
 Efficient data movement
 Bias-free analysis
R. J. Crouser and R. Chang, “An affordance-based framework for
human computation and human-computer collaboration,” IEEE
Trans. Vis. Comput. Graph., vol. 18, pp. 2859–2868, 2012.

25. Generic Architecture
Platform/Marketplace
(Publish Task, Task Management)
Workers
Requestors
1.
2.
4.
3.

26. Platforms and Marketplaces

27. Core Design Questions
Goal
What
Workers Who Why Incentives
How
Process
Malone, T. W., Laubacher, R., & Dellarocas, C. N.
Harnessing crowds: Mapping the genome of collective intelligence. MIT Sloan Research Paper 4732-09, (2009).

28. Setting up a Crowdsourcing Process
1 – Who is doing it?
 Hierarchy (Assignment), Crowd (Choice)
2 – Why are they doing it?
 Money ($$££), Glory (reputation/prestige), Love (altruism, socialize,
enjoyment), Unintended by-product (e.g. re-Captcha, captured in
workflow), Self-serving resources (e.g. Wikipedia, product/customer
data), Part of their job description
 Determine pay and time for each task
 Marketplace: Delicate balance (Money does not improve quality but can increase
participation)
 Internal Hierarchy: Engineering opportunities for recognition: Performance review, prizes for
top contributors, badges, leaderboards, etc.
3 – What is being done?
 Creation Tasks: Create/Generate/Find/Improve/ Edit / Fix
 Decision (Vote) Tasks: Accept/Reject, Thumbs up / Down, Vote
4 – How is it being done?
 Identify the workflow: Integrate in workflow (“rating” algorithm)
 Identify the platform (Internal/Community/Public)
 Identify the Algorithm (Data quality, Image recognition, etc.)

29. Summary
29
Analytics &
Algorithms
Entity Linking
Data Fusion
Relation Extraction
Human
Computation
Relevance Judgment
Data Verification
Disambiguation
Better Data
Internal Community
- Domain Knowledge
- High Quality Responses
- Trustable
Web Data
Databases
Sensor Data
Programmers Managers
External Crowd
- High Availability
- Large Scale
- Expertise Variety

30. References & Further Information
 Ojo, A., Curry, E., and Janowski, T. 2014. “Designing
Next Generation Smart City Initiatives - Harnessing
Findings And Lessons From A Study Of Ten Smart City
Programs,” in 22nd European Conference on Information
Systems (ECIS 2014)
 Ojo, A., Curry, E., and Sanaz-Ahmadi, F. 2015. “A Tale
of Open Data Innovations in Five Smart Cities,” in 48th
Annual Hawaii International Conference on System
Sciences (HICSS-48)
 Curry, E., Freitas, A., and O’Riáin, S. 2010. “The Role of
Community-Driven Data Curation for Enterprises,” in
Linking Enterprise Data, D. Wood (ed.), Boston, MA:
Springer US, pp. 25–47.