1. Jurix 2012 Workshop on
Argumentation Technology for Policy Deliberations
A Collective Intelligence Tool for Evidence
Based Policy & Practice
Anna De Liddo & Simon Buckingham Shum
Knowledge Media Institute
The Open University, Milton Keynes, MK7 6AA, UK
http://evidence-hub.net/
2. Human-‐Centred
Compu/ng
for
CCI
(Argumenta/on-‐based
Collec/ve
Intelligence)
Practice:
CoPHV
Policy:
Research:
Olnet, Ed
Rcyp Hub
Future
Human Dynamics of Engagements
UX design Computational
Services/
Cohere/free
semantics Analytics NLP, XIP,
discourse
analysis
EH/simplified model, Structure
widget/threaded searches/
interface Agents
3. Collective Intelligence:
How do we Crowdsource Policy
Deliberation?
Collective intelligence is a new umbrella term used to express the augmented
functions that can be enabled by the existence of a community.
It refers to the intelligence that emerges by the coexistence of multiple people
in the same environment. This environment can be both a real world
environment and a virtual one.
We are looking at the Web and at what intelligent hints, supports or behaviors
can be tracked and emerge by the coexistence of a collective of people online.
4. Collective Intelligence
Nowadays successful CI tools have been developed, especially in the IT
business and e-commerce sector, that use those users traces to depict
users profiles and suggest user actions based on those profiles.
These CI tools support quite simple user objectives, such as: deciding what
book to buy (Amazon), finding the picture or video that mach their needs (Flickr
and Youtube), or deciding what music to listen (LastFM).
Collecting fragmented users traces seems to work to collect and exploit CI in
the business and commerce sector.
5. Contested Collective Intelligence
On the other hand if we look at the social and political sector, or at higher level
organizational and business strategy issues we need to support more complex
users goals such as i.e.:
understanding policy actions; learning environmental responses to adapt
organizational actions; understanding the economical crisis and the possible
implications for the community etc.
CI tools that aims at supporting users in more complex knowledge works
need to be thought and designed so that users collective intelligence can be
captured and shared in a much richer and explicit way.
6. Contested Collective Intelligence
In the design space of CI systems
• where there is insufficient data to confidently compute an answer,
• when there is ambiguity about the trustworthiness of environmental
signals,
• and uncertainty about the impact of actions,
…then a more powerful scaffolding for thinking and discourse is required, in
order to support the emergence of CI around complex socio political dilemmas.
7. First Prototype Tool for Contested
Collective Intelligence (CCI)
With Cohere users can make their thinking visible and sharable with online
communities by:
ü collaboratively annotating the Web,
ü leveraging lists of annotations into meaningful knowledge maps and
ü Engaging in structured online discussions.
8. Cohere Conceptual Model
Cohere
builds
on
a
conceptual
model
which
consists
of
four
main
users
ac/vi/es
through
which
users
can
make
their
thinking
visible
and
contribute
to
the
development
of
Collec/ve
Intelligence
around
specific
issues:
11. Explore, Filter and Makesense
Watch the demo video at: http://www.youtube.com/watch?v=Fcn2ab9PYo4
Watch the Open Deliberation model video at: http://www.youtube.com/watch?v=vthygbKA2Mg
12. Despite the success of the web annotation
paradigm…People seems to struggle to
make semantic connections, moreover too
many semantics produce often
redundancy and duplication.
This brought to the second design iteration:
A new simplified data model and a new
interface for connection making….
13. Experimenting CCI in a real case of
Educational Policy: The OLnet project
The issue: lack of evidence of OER effectiveness
olnet.org
14. OLnet project:
The wider research question
RQ: How can we help researchers and practitioners in
the OER field to contribute to the evidences of OER
effectiveness and to investigate these evidences
collaboratively?
15. Approach:
Contested Collective Intelligence
Our approach to CI focuses on:
capturing the hidden knowledge of the OER movement
and leveraging it so that can be:
ü debated (building and confronting arguments),
ü evaluated (assessing evidence), and
ü put in value (distilling claims used to inform OER
policy and practice)
16. What&Why:
The Evidence Hub provides
ü the OER community with a space to harvest the
evidence of OER effectiveness
ü policy makers with a community-generated
knowledge base to make evidence based decision
on Educational Policy.
17. The Evidence Hub: Mapping the social and
discourse ecosystem
Social
Ecosystem
• People
(Contributors)
• Projects
• Organiza/ons
Discourse
Ecosystem
• Key
challenges
Themes
• Issues,
• Solu/ons,
• Claims
• Evidence
• Resources
olnet.org
41. The Evidence Hub
Some Facts and Figures
The Evidence Hub alpha version launched in April 2011.
With 50 users, from 35 different countries, including key OER people.
42. Some Facts and Figures
Opened to the public at OpenEd11 in Utah.
olnet.org
43. Some facts and figures:
Engagement
- 108 contributors,
- received 3,054 visits from 1,053 unique visitors from
57 different countries
olnet.org
44. Some facts and figures on Content
304 OER projects and organizations
129 OER research claims
79 OER issues
89 proposed solutions
323 Evidence and
553 Resources
olnet.org
45. Reflection on initial
User Testing & Interviews
Feedback
from
users
shows
that
the
EH
is
perceived
as:
“relevant”,
“organized”,
“desirable”
and
“engaging”
but
some/mes
“sophis1cated”
and
“complex”.
improving
the
user
experience
by
crea4ng
summary
views,
facilitate
and
simplify
content
seeding,
be:er
displays
and
filters
on
the
content.
olnet.org
46. Feedback from Lab-Based User Testing
Fragmented approach to argument construction: widget
interface
• Easy to contribute to but
• Increases miscathegorization : interpretation biases on
how content should be labeled under specific
argumentation categories;
• Increases duplication of content
• Decreases argumentation coherence
This lead to the third design Iteration….
48. Research
by
Children
and
Young
People
Evidence
Hub:
A
mixed
threaded/widget
interface
49.
50. Collective Intelligence
Development Trajectories:
Facilitating content seeding
1) Web Annotation to support seeding
Evidence Hub bookmarklet to allow people to capture evidence by
performing annotation of free web resources and OERs.
Allows users to
highlight and
annotate Web
resources
through an
Evidence Hub
bookmarklet
olnet.org
51. 2) Combining Human and Machine Annotation:
The Hewlett Grant Reports Project
template
report
RESULTS
XIP-annotated report
De Liddo, A., Sándor, Á. and Buckingham Shum, S. (2012) Contested Collective Intelligence: Rationale,
Technologies, and a Human-Machine Annotation Study, Computer Supported Cooperative Work (CSCW)
Journal : Volume 21, Issue 4 (2012), Page 417-448
52. Discourse analysis with the Xerox
Incremental Parser
Detection of salient sentences based on rhetorical markers:
BACKGROUND KNOWLEDGE: NOVELTY: OPEN QUESTION:
Recent studies indicate … ... new insights provide direct … little is known …
… the previously proposed … evidence ...... we suggest a new ... … role … has been elusive
approach ... Current data is insufficient …
… is universally accepted ...
... results define a novel role ...
CONRASTING IDEAS: SIGNIFICANCE: SUMMARIZING:
… unorthodox view resolves … studies ... have provided important The goal of this study ...
paradoxes … advances Here, we show ...
In contrast with previous hypotheses ... Knowledge ... is crucial for ... Altogether, our results ... indicate
... inconsistent with past findings ... understanding
valuable information ... from studies
GENERALIZING: SURPRISE:
... emerging as a promising approach We have recently observed ...
Our understanding ... has grown surprisingly
exponentially ... We have identified ... unusual
... growing recognition of the The recent discovery ... suggests
importance ... intriguing roles
De Liddo, A., Sándor, Á. and Buckingham Shum, S. (2012) Contested Collective Intelligence: Rationale, Technologies, and a
Human-Machine Annotation Study, Computer Supported Cooperative Work (CSCW) Journal : Volume 21, Issue 4 (2012),
Page 417-448
53. XIP annotations to Cohere
CONTRAST converts into semantic connection’s
label :””describes contrasting ideas in”
PROBLEM_CONTRAST_ First, we discovered that there is no empirically based
understanding of the challenges of using OER in K-12 settings.
PROBLEM converts into
Annotation node icon: Name entities extracted by XIP convert
“Issue”=“Light Bulb” into Tags
Annotation Node Report Node
De Liddo, A., Sándor, Á. and Buckingham Shum, S. (2012) Contested Collective Intelligence: Rationale, Technologies, and a
Human-Machine Annotation Study, Computer Supported Cooperative Work (CSCW) Journal : Volume 21, Issue 4 (2012),
Page 417-448
54. Human annotation and machine annotation
1.
~19 sentences annotated 22 sentences annotated
11 sentences = human annotation
2 consecutive sentences of human
annotation
2. 71 sentences annotated 59 sentences annotated
42 sentences = human annotation
De Liddo, A., Sándor, Á. and Buckingham Shum, S. (2012) Contested Collective Intelligence: Rationale, Technologies, and a
Human-Machine Annotation Study, Computer Supported Cooperative Work (CSCW) Journal : Volume 21, Issue 4 (2012),
Page 417-448
55. 3)
Collabora4ve
PDF
annota4on
A high % of policy report and documents are in PDF format: we have a
concept demo of direct PDF annotation shared back to Cohere
Future developments
could be devoted to
power the Evidence
Hub with PDF
annotation so that
users can share
evidence of Policy
arguments and
impact directly
working with PDF.
Steve Pettifer, Utopia:
http://getutopia.com/
olnet.org
56. Collective Intelligence
Development Trajectories:
4) Better visualization and filtering of content
5)Discourse Analytics
When social and discourse elements become too many in
number and complexity how can we make sense of
them?
Toward CI visualization and analysis…
adding more formal logics to evaluate arguments
&
developing discourse analytics to create summaries,
identify gaps, localize interests, focus contributions
57. Discourse Network Visualization
Watch the demo video at: http://www.youtube.com/watch?v=Fcn2ab9PYo4
Watch the Open Deliberation model video at: http://www.youtube.com/watch?v=vthygbKA2Mg
59. Theoretical questions for future work
• How to evaluate arguments? - authomatic (based on argument
computation) vs community lead mechanisms (such as voting and
reputation systems)
• How to make optimal use of both human and machine
annotation & argumentation skills?
– How to exploit machine consistency while reducing information
overload and noise?
– How to exploit the unique human capacities to abstract, filter for
relevance etc.?
• How to cope with visual complexity (new search interface,
focused and structured network searches, collective filtering,
identifying argument structures)?
• How do we crowdsource Policy Deliberation? What is the right
interface? What is the architecture of Participation?
60. References
• De
Liddo,
A.,
Sándor,
Á.
and
Buckingham
Shum,
S.
(2012)
Contested
Collec/ve
Intelligence:
Ra/onale,
Technologies,
and
a
Human-‐Machine
Annota/on
Study,
Computer
Supported
Coopera/ve
Work
(CSCW)
Journal
:
Volume
21,
Issue
4
(2012),
Page
417-‐448
• Buckingham
Shum,
Simon
(2008).
Cohere:
Towards
Web
2.0
Argumenta/on.
In:
Proc.
COMMA'08:
2nd
Interna4onal
Conference
on
Computa4onal
Models
of
Argument,
28-‐30
May
2008,
Toulouse,
France.
Available
at:hap://oro.open.ac.uk/
10421/
• De Liddo, Anna and Buckingham Shum, Simon (2010). Cohere: A prototype for contested collective intelligence.
In: ACM Computer Supported Cooperative Work (CSCW 2010) - Workshop: Collective Intelligence In
Organizations - Toward a Research Agenda, February 6-10, 2010, Savannah, Georgia, USA.
Available
at: http://
oro.open.ac.uk/19554/
• Buckingham
Shum,
Simon
and
De
Liddo,
Anna
(2010).
Collec/ve
intelligence
for
OER
sustainability.
In:
OpenED2010:
Seventh
Annual
Open
Educa4on
Conference,
2-‐4
Nov
2010,
Barcelona,
Spain.
Available
at:
hap://oro.open.ac.uk/23352/
• De
Liddo,
Anna
(2010).
From
open
content
to
open
thinking.
In:
World
Conference
on
Educa4onal
Mul4media,
Hypermedia
and
Telecommunica4ons
(Ed-‐Media
2010),
29
Jun,
Toronto,
Canada.
Available
at:
hap://oro.open.ac.uk/22283/
• De
Liddo,
Anna
and
Alevizou,
Panagiota
(2010).
A
method
and
tool
to
support
the
analysis
and
enhance
the
understanding
of
peer-‐-‐to-‐-‐peer
learning
experiences.
In:
OpenED2010:
Seventh
Annual
Open
Educa4on
Conference,
2-‐4
Nov
2010,
Barcelona,
Spain.
Available
at:
hap://oro.open.ac.uk/23392/
• Buckingham
Shum,
Simon
(2007).
Hypermedia
Discourse:
Contes/ng
networks
of
ideas
and
arguments.
In:
Priss,
U.;
Polovina,
S.
and
Hill,
R.
eds.
Conceptual
Structures:
Knowledge
Architectures
for
Smart
Applica4ons.
Berlin:
Springer,
pp.
29–
44.
Thanks for Your Attention!
Anna De Liddo
anna.deliddo@open.ac.uk
http://people.kmi.open.ac.uk/anna/
