How do we support people in using/reusing arguments and opinions on the World Wide Web? WIMMICS, INRIA seminar based on my PhD viva slides.

1. Jodi Schneider
WIMMICS seminar
INRIA Sophia Antipolis – Méditerranée & I3S
vendredi 21 février 2014

3. How can we make sense of disagreement?

4. Arguments & opinions give a rationale

5. Reuse arguments & rationales
o How do we make arguments more clear to BOTH
humans and machines?
o Explicit arguments are not available
• Important in bug reports, political commentary, product
reviews, etc.
o Machine-readable arguments could help
• Gather information – e.g. finding issues, claims, and
opinion clusters
• Connect opinions to explicit evidence
• Navigate claims networks

6. Arguments in collective decision-making

7. Arguments about content deletion

8. Arguments in open collaboration systems

9. Open collaboration systems
―people form ties with others & create things together‖
(Forte and Lampe 2013)

10. Open collaboration systems
―people form ties with others & create things together‖
(Forte and Lampe 2013)
Examples:
o Wikipedia
o HTML5 working group
o OpenStreetMap
o Project Gutenberg – Distributed Proofreaders
o Apache projects, Mozilla Firefox, …

11. How do we enable the reuse of
arguments and opinions
in open collaboration systems?

12. Use case: deletion in Wikipedia
o 1 in 4 new Wikipedia articles is deleted – within
minutes or hours
o Demotivating!
• 1 in 3 newcomers start by writing a new article
• 7X less likely to stay if their article is deleted!
o Can we support editor retention?
Source: http://enwp.org/Wikipedia:Wikipedia_Signpost/2011-04-04/Editor_retention

13. Thousands of new editors each month
Source: http://reportcard.wmflabs.org/
English
All languages

14. Policies grew 10-15X (2002-2008)
See e.g. Butler, Joyce, and Pike. CHI 2008
"Don't look now, but we've created a bureaucracy:
the nature and roles of policies and rules in Wikipedia."

15. Supporting open collaboration systems
o Can we support editor retention?
o Make criteria explicit to:
• Explain community expectations (how to be convincing)
• Support making & auditing decisions

16. Research Questions

17. How do we enable
the reuse of arguments and opinions
on the World Wide Web?
o RQ1: What are the opportunities and
requirements for providing argumentation support?
o RQ2: Which arguments are used in open
collaboration systems?
o RQ3: How can we structure and display opinions
and arguments to support filtering?

18. How do we enable
the reuse of arguments and opinions
on the World Wide Web?
o RQ1: What are the opportunities and requirements
for providing argumentation support?
o RQ2: Which arguments are used in open
collaboration systems?
o RQ3: How can we structure and display opinions
and arguments to support filtering?

19. How do we enable
the reuse of arguments and opinions
on the World Wide Web?
o RQ1: What are the opportunities and requirements
for providing argumentation support?
o RQ2: Which arguments are used in open
collaboration systems?
o RQ3: How can we structure and display opinions
and arguments to support filtering?

20. How do we enable
the reuse of arguments and opinions
on the World Wide Web?
o RQ1: What are the opportunities and requirements
for providing argumentation support?
Netnography
o RQ2: Which arguments are used in open
collaboration systems?
Iterative Annotation
o RQ3: How can we structure and display opinions
and arguments to support filtering?
Semantic Web Systems Development

21. RQ1: What are the opportunities and
requirements for providing
argumentation support?

22. Methodology: Netnography
Kozinets, Robert V.
Netnography: Doing ethnographic research online.
Sage Publications, 2010.

23. Methodology: Netnography
1. Planning and community selection
2. Participant observation and data collection
3. Data analysis and iterative interpretation
4. Presenting results

24. Results: Sample corpus
72 discussions started on 1 day
o Each discussion has:
• 3—33 messages
• 2—15 participants
o 741 messages contributed by 244 users.
Each message has 3—350+ words.
o 98 printed A4 sheets

25. Example from Corpus

26. Example from Corpus

27. Results: Terminology and policy
knowledge becomes an obstacle

28. Results: Important tasks
for consensus discussions
1. Determine one’s personal position
2. Express one’s personal position in accordance with
community norms
3. Determine the consensus

29. RQ2: Which arguments are used in
open collaboration systems?

30. Methods
o Use corpus of 72 discussions
o Two types of annotation: 2 argumentation theories
o Iterative annotation for each theory
• Multiple annotators
• Refine to get good inter-annotator agreement
• 4 rounds of annotation

31. Two argumentation theories
o Walton’s Argumentation Schemes
(Walton, Reed, and Macagno 2008)
• Informal argumentation
(philosophical & computational argumentation)
• Identify & prevent errors in reasoning (fallacies)
• 60 patterns
o Factors Analysis
(Ashley 1991)
• Case-based reasoning
• E.g. factors for deciding cases in trade secret law,
favoring either party (the plaintiff or the defendant).

32. Walton’s Argumentation Schemes
Example Argumentation Scheme:
Argument from Rules – “we apply rule X”
Critical Questions
1. Does the rule require carrying out this type of
action?
2. Are there other established rules that might conflict
with or override this one?
3. Are there extenuating circumstances or an excuse
for noncompliance?
Walton, Reed, and Macagno 2008

33. Walton’s Argumentation Schemes
Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker.
―Arguments about Deletion: How Experience Improves the Acceptability of Arguments in
Ad-hoc Online Task Groups‖. In CSCW 2013.
! "#$%&' ()*+((&"' *"&, +-&' . &
! "#$%&' ()*"+%), -./&' 0&)(+)123+(4&5.5 67897:
! "#$%&' ()*"+%); $<&5 6=87>:
? +(& 6@8=7:
! "#$%&' (A(.+' )*"+%)BA<$&5 C89>:
! "#$%&' ()*"+%)? &&/)*+")1&<3 C869:
! "#$%&' ()*"+%)D.A5 @8EF:
? +)"&A5+' )#.- &' @899:
! "#$%&' ()*"+%)G+5.(.+' )(+)H' +I @8>J :
! "#$%&' ()*"+%)G"&0&/&' ( @8>J :
! "#$%&' ()*"+%)K#' +"A' 0& 987F:
! "#$%&' ()*"+%)L+%3+5.(.+' 98J =:
! "#$%&' ()*"+%)LA$5&)(+), **&0( 98@6:
! "#$%&' ()*"+%)! ' A<+#2 989@:
! "#$%&' ()*"+%)MA5(& 989@:
G"A0(.0A<); &A5+' .' # 989@:
! "#8)*"+%)B&"NA<)L<A55.*.0A(.+' 98>=:

34. Factors Analysis
Factors determined
by iterative annotation

35. Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
• 91% of comments
• 70% of discussions

36. Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
• 91% of comments
• 70% of discussions
―Other‖ as 5th catchall

37. Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
• 91% of comments
• 70% of discussions
―Other‖ as 5th catchall
Factor Example (used to justify
`keep')
Notability Anyone covered by another
encyclopedic reference is
considered notable enough for
inclusion in Wikipedia.
Sources Basic information about this
album at a minimum is certainly
verifiable, it's a major label
release, and a highly notable
band.
Maintenance …this article is savable but at its
current state, needs a lot of
improvement.
Bias It is by no means spam (it does
not promote the products).
**Other I'm advocating a blanket
"hangon" for all articles on newly-
drafted players

38. RQ3: How can we structure and
display opinions and arguments to
support filtering?

39. Methodology
o Linked Data Application Development
o User testing – 20 users
Original
Discussion
Ontology
Semantic
Enrichment
Semantically
Enriched
RDFa
Querying
Queryable
User Interface
With Barchart

40. Add a discussion summary

41. Add a discussion summary

45. Semantically enrich messages

46. Use semantic structure
Implementation based on Jodi Schneider and Krystian Samp
“Alternative Interfaces for Deletion Discussions in Wikipedia: Some
Proposals Using Decision Factors. [Demo]” In WikiSym2012.

47. Experimental design

48. System A (Control)

49. System B (Experimental)

50. Experimental design

51. Experimental design

53. PU* - Perceived usefulness
PE* - Perceived ease of use
DC -Decision completeness
PF - Perceived effort
IC* - Information
completeness
Statistical Significance
PU* p < .001
PE* p .001
IC* p .039

54. Final survey

55. Results: 84% prefer our system
―Information is structured and I can quickly get an
overview of the key arguments.‖
―The ability to navigate the comments made it a bit
easier to filter my mind set and to come to a
conclusion.‖
―It offers the structure needed to consider each factor
separately, thus making the decision easier. Also, the
number of comments per factor offers a quick
indication of the relevance and the deepness of the
decision.‖
Based on a 20 participant user test.
1 participant did not take the final survey

56. Overall contributions
o A procedure for providing argumentation support
o A demonstration of this procedure, including
• A requirements analysis
• A categorization of the most common arguments used
according to two theories
o Walton’s argumentation schemes
o Factors-dimensions theory
• An ontology for argumentation in Wikipedia deletion
discussions.
• An argumentation filtering system
that visually summarizes arguments
with bar charts of the decision factors.

57. Main papers used in the thesis
o Jodi Schneider, Krystian Samp, Alexandre Passant, and
Stefan Decker. ―Arguments about Deletion: How Experience
Improves the Acceptability of Arguments in Ad-hoc Online
Task Groups‖. In CSCW 2013.
o Jodi Schneider, Tudor Groza, Alexandre Passant, ―A Review
of Argumentation for the Social Semantic Web.‖ Semantic
Web – Interoperability, Usability, Applicability, 2013, 4(2),
159-218.
o Jodi Schneider and Krystian Samp. ―Alternative Interfaces for
Deletion Discussions in Wikipedia: Some Proposals Using
Decision Factors. [Demo]‖ In WikiSym2012.
o Jodi Schneider, Alexandre Passant, and Stefan Decker.
―Deletion Discussions in Wikipedia: Decision Factors and
Outcomes.‖ In WikiSym2012.

59. Summary
o We need better ways of structuring arguments
on the Web.
o Arguments vary across Social Media.
o Different theories of argumentation stress different
aspects.
o Factors analysis is useful for providing a brief
summary of discussions. This can help find
consensus.

60. Copyright 2011 Digital Enterprise Research Institute. All rights reserved.
Digital Enterprise Research Institute www.deri.ie
Enabling Networked Knowledge
Identifying, Annotating, and Filtering
Arguments and Opinions in Open
Collaboration Systems
Jodi Schneider
vendredi 21 fev 2014
60
WIMMICS seminar
INRIA Sophia Antipolis - Méditerranée and I3S