Enabling reuse of arguments and opinions in open collaboration systems PhD viva 2013 10-01
1. Copyright 2011 Digital Enterprise Research Institute. All rights reserved.
Digital Enterprise Research Institute www.deri.ie
Enabling Networked Knowledge
Enabling reuse
of arguments and opinions
in open collaboration systems
Jodi Schneider
Tuesday 1 October 2013
1
Ph.D. viva
Galway, Ireland
4. Reuse arguments & rationales
o How do we make arguments more clear to BOTH
humans and machines?
o Explicit arguments are not available
• Important in bug reports, political commentary, product
reviews, etc.
o Machine-readable arguments could help
• Gather information – e.g. finding issues, claims, and
opinion clusters
• Connect opinions to explicit evidence
• Navigate claims networks
9. Open collaboration systems
―people form ties with others & create things together‖
(Forte and Lampe 2013)
Examples:
o Wikipedia
o HTML5 working group
o OpenStreetMap
o Project Gutenberg – Distributed Proofreaders
o Apache projects, Mozilla Firefox, …
10. How do we enable the reuse of
arguments and opinions
in open collaboration systems?
11. Use case: deletion in Wikipedia
o 1 in 4 new Wikipedia articles is deleted – within
minutes or hours
o Demotivating!
• 1 in 3 newcomers start by writing a new article
• 7X less likely to stay if their article is deleted!
o Can we support editor retention?
Source: http://enwp.org/Wikipedia:Wikipedia_Signpost/2011-04-04/Editor_retention
12. Thousands of new editors each month
Source: http://reportcard.wmflabs.org/
English
All languages
13. Policies grew 10-15X (2002-2008)
See e.g. Butler, Joyce, and Pike. CHI 2008
"Don't look now, but we've created a bureaucracy:
the nature and roles of policies and rules in Wikipedia."
14. Supporting open collaboration systems
o Can we support editor retention?
o Make criteria explicit to:
• Explain community expectations (how to be convincing)
• Support making & auditing decisions
16. How do we enable
the reuse of arguments and opinions
on the World Wide Web?
o RQ1: What are the opportunities and
requirements for providing argumentation support?
o RQ2: Which arguments are used in open
collaboration systems?
o RQ3: How can we structure and display opinions
and arguments to support their use and reuse?
17. How do we enable
the reuse of arguments and opinions
on the World Wide Web?
o RQ1: What are the opportunities and requirements
for providing argumentation support?
o RQ2: Which arguments are used in open
collaboration systems?
o RQ3: How can we structure and display opinions
and arguments to support their use and reuse?
18. How do we enable
the reuse of arguments and opinions
on the World Wide Web?
o RQ1: What are the opportunities and requirements
for providing argumentation support?
o RQ2: Which arguments are used in open
collaboration systems?
o RQ3: How can we structure and display opinions
and arguments to support their use and reuse?
19. How do we enable
the reuse of arguments and opinions
on the World Wide Web?
o RQ1: What are the opportunities and requirements
for providing argumentation support?
Netnography
o RQ2: Which arguments are used in open
collaboration systems?
Iterative Annotation
o RQ3: How can we structure and display opinions
and arguments to support their use and reuse?
Semantic Web Systems Development
20. RQ1: What are the opportunities and
requirements for providing
argumentation support?
22. Methodology: Netnography
1. Planning and community selection
2. Participant observation and data collection
3. Data analysis and iterative interpretation
4. Presenting results
23. Results: Sample corpus
72 discussions started on 1 day
o Each discussion has:
• 3—33 messages
• 2—15 participants
o 741 messages contributed by 244 users.
Each message has 3—350+ words.
o 98 printed A4 sheets
27. Results: Important tasks
for consensus discussions
1. Determine one’s personal position
2. Express one’s personal position in accordance with
community norms
3. Determine the consensus
29. Methods
o Use corpus of 72 discussions
o Two types of annotation: 2 argumentation theories
o Iterative annotation for each theory
• Multiple annotators
• Refine to get good inter-annotator agreement
• 4 rounds of annotation
30. Two argumentation theories
o Walton’s Argumentation Schemes
(Walton, Reed, and Macagno 2008)
• Informal argumentation
(philosophical & computational argumentation)
• Identify & prevent errors in reasoning (fallacies)
• 60 patterns
o Factors Analysis
(Ashley 1991)
• Case-based reasoning
• E.g. factors for deciding cases in trade secret law,
favoring either party (the plaintiff or the defendant).
31. Walton’s Argumentation Schemes
Example Argumentation Scheme:
Argument from Rules – “we apply rule X”
Critical Questions
1. Does the rule require carrying out this type of
action?
2. Are there other established rules that might conflict
with or override this one?
3. Are there extenuating circumstances or an excuse
for noncompliance?
Walton, Reed, and Macagno 2008
36. Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
• 91% of comments
• 70% of discussions
―Other‖ as 5th catchall
Factor Example (used to justify
`keep')
Notability Anyone covered by another
encyclopedic reference is
considered notable enough for
inclusion in Wikipedia.
Sources Basic information about this
album at a minimum is certainly
verifiable, it's a major label
release, and a highly notable
band.
Maintenance …this article is savable but at its
current state, needs a lot of
improvement.
Bias It is by no means spam (it does
not promote the products).
**Other I'm advocating a blanket
"hangon" for all articles on newly-
drafted players
37. RQ3: How can we structure and
display opinions and arguments to
support their use and reuse?
38. Methodology
o Linked Data Application Development
o User testing – 20 users
Original
Discussion
Ontology
Semantic
Enrichment
Semantically
Enriched
RDFa
Querying
Queryable
User Interface
With Barchart
45. Use semantic structure
Implementation based on Jodi Schneider and Krystian Samp
“Alternative Interfaces for Deletion Discussions in Wikipedia: Some
Proposals Using Decision Factors. [Demo]” In WikiSym2012.
52. PU* - Perceived usefulness
PE* - Perceived ease of use
DC -Decision completeness
PF - Perceived effort
IC* - Information
completeness
Statistical Significance
PU* p < .001
PE* p .001
IC* p .039
54. Results: 84% prefer our system
―Information is structured and I can quickly get an
overview of the key arguments.‖
―The ability to navigate the comments made it a bit
easier to filter my mind set and to come to a
conclusion.‖
―It offers the structure needed to consider each factor
separately, thus making the decision easier. Also, the
number of comments per factor offers a quick
indication of the relevance and the deepness of the
decision.‖
Based on a 20 participant user test.
1 participant did not take the final survey
55. Overall contributions
o A procedure for providing argumentation support
o A demonstration of this procedure, including
• A requirements analysis
• A categorization of the most common arguments used
according to two theories
o Walton’s argumentation schemes
o Factors-dimensions theory
• An ontology for argumentation in Wikipedia deletion
discussions.
• An argumentation visualization system that structures
arguments with decision factors.
56. Main papers used in the thesis
o Jodi Schneider, Krystian Samp, Alexandre Passant, and
Stefan Decker. ―Arguments about Deletion: How Experience
Improves the Acceptability of Arguments in Ad-hoc Online
Task Groups‖. In CSCW 2013.
o Jodi Schneider, Tudor Groza, Alexandre Passant, ―A Review
of Argumentation for the Social Semantic Web.‖ Semantic
Web – Interoperability, Usability, Applicability, 2013, 4(2),
159-218.
o Jodi Schneider and Krystian Samp. ―Alternative Interfaces for
Deletion Discussions in Wikipedia: Some Proposals Using
Decision Factors. [Demo]‖ In WikiSym2012.
o Jodi Schneider, Alexandre Passant, and Stefan Decker.
―Deletion Discussions in Wikipedia: Decision Factors and
Outcomes.‖ In WikiSym2012.
57.
58. Summary
o We need better ways of structuring arguments
on the Web.
o Arguments vary across Social Media.
o Different theories of argumentation stress different
aspects.
o Factors analysis is useful for providing a brief
summary of discussions. This can help find
consensus.
Notes de l'éditeur
Arguments are everywhere – we would like to reuse argumentsincluding on the Web.From http://www.sendareview.com/The Web is full of opinions & commentary.A lot of it disagrees.How do we learn from other people, when they disagree?
1“only 0.6 percent of those whose articles are met with deletion stayed editing, compared to 4.4 percent of the users whose articles remained”, http://enwp.org/Wikipedia:Wikipedia_ Signpost/2011-04-04/Editor_retentionOriginal stats from Mr.Z-man: http://en.wikipedia.org/wiki/User:Mr.Z-man/newusers
***Get just blue figures***
Image from http://breakingenergy.com/2013/04/08/energy-legal-work-takes-the-number-two-spot-growth-forecast/***Picture conveying social norms/policies – e.g. growth over time from Butler (if there’s some image of that)
Image from http://kozinets.net/archives/509
Technically started or relistedCorpus is https://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletion/Log/2011_January_29
Categories (Walton’s argumentation schemes) vs. process (factors analysis)
Major Premise: If carrying out types of actions including A is the established rule for x, then (unless the case is an exception), a must carry out A.Minor Premise: Carrying out types of actions including A is the established rule for a.Conclusion: Therefore, a must carry out A.
Earlier in CSCW: Jodi Schneider, KrystianSamp, Alexandre Passant, Stefan Decker. “Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups”. In Computer Supported Cooperative Work and Social Computing (CSCW). San Antonio, TX, February 23-27, 2013.Used as categoriesInitial annotation60 categories (each Walton argumentation scheme)all arguments in each messageRound 415 most common argumentation schemesmain argument in each messageGood inter-annotator agreement for hard task:54% agreement (compared to 12% chance) among 2 annotators
20 novice participants used both systems“The ability to navigate the comments made it a bit easier to filter my mind set and to come to a conclusion.”“summarise and, at the same time, evaluate which factor should be considered determinant for the final decision”