Presented at AAAI 2016 Spring Symposium: Intelligent systems for supporting distributed human teamwork. Position paper available at: http://juhokim.com/files/AAAISymposium2016-Organic-Crowdsourcing-Systems.pdf
6. User: personally
meaningful activity
Human computation artifact:
immediately available in UI
for future users
System:
algorithmic
coordination
Organic crowdsourcing system
7. • Learning via online videos
• Planning a large conference
• Understanding a government budget
Three case studies of
organic crowdsourcing
8. • Learning via online videos
• Planning a large conference
• Understanding a government budget
Three case studies of
organic crowdsourcing
14. How to extract & present
solution structure at scale?
Subgoal labeling is a costly operation:
Requires domain experts and knowledge extraction
experts to work together. [Catrambone, 2011]
Insight:
Subgoal labeling can be a good learning exercise!
Solution: LEARNERSOURCING
21. Crowdy evaluation
Does participating in learnersourcing
improve learning?
Does the learnersourcing workflow
produce good subgoal labels?
Retention test: baseline < Crowdy
Matching expert quality labels
26. Community-guided
conference planning
Attendees / Authors / Committee members
explore program & express own preferences.
Final schedule
with less conflicts
& more preferences
Scheduling tool
encodes community
input & recommends
schedule changes.
organizer
29. Authorsourcing
8,651 preferences and constraints from
645 authors, covering 87% of accepted papers
my paper
fits with my paper does not fit
paper E
paper
D
paper B
paper C
paper A
paper Z
30. Mixed-initiative scheduling tool
Cobi: A Community-Informed Conference Scheduling Tool.
Juho Kim, Haoqi Zhang, Paul André, Lydia B. Chilton, Wendy Mackay, Michel Beaudouin-Lafon, Robert C. Miller, Steven P. Dow.
UIST 2013.
32. • Learning via online videos
• Planning a large conference
• Understanding a government budget
Three case studies of
organic crowdsourcing
33.
34.
35.
36. BudgetMap:
Issue-driven budget navigation
BudgetMap: Engaging Taxpayers in the Issue-Driven Classification of a Government Budget.
Nam Wook Kim, Jonghyuk Jung, Eun-Young Ko, Songyi Han, Chang Won Lee, Juho Kim, Jihee Kim.
CSCW 2016.
38. Organic crowdsourcing for
civic engagement
increased awareness &
understanding of
government budget
dynamically generated
social issue – budget program links
39. Organic crowdsourcing
design principles
Paid crowdsourcing
simple and concrete task
quality control
microscopic, focused task
to workers
cost: money
Organic crowdsourcing
personally meaningful task
incentive design
overall goal & contribution
awareness
cost: crowd’s time & effort
40. Organic Crowdsourcing Systems
Juho Kim | juho@juhokim.com | @imjuhokim | juhokim.com
User: personally
meaningful activity
Human computation artifact:
immediately available in UI
for future users
System:
algorithmic
coordination