2013 EEE and HICC conferences--3 presentations See socialdeliberativeskills.com

1. 1. Supporting Social Deliberative Skills Online:
the Effects of Reflective Scaffolding Tools
2. A Prototype Facilitators Dashboard: Assessing and visualizing dialogue
quality in online deliberation for education and work
3. Text Analysis of Deliberative Skills in Undergraduate
Online Dialogue: Using L1 Regularized Logistic Regression
with Psycholinguistic Features
Tom Murray,
Xiaoxi Xu, Beverly Woolf, Leah Wing, Lynn Stephens,
Natasha Shrikant, Lori Clarke, Lee Osterweil
EEE 2013 & HICC 2013
Supporting Social Deliberative Skills in Online Contexts

2. Project Overview
2

3. Overview
1. Background–Social Deliberative Skills
2. Classroom Studies
3. Facilitator Dashboard
4. Automated Text Analysis/Classification
5. Conclusions

4. 1. Background on
Social Deliberative Skills

5. Group &
Collaborative
Learning/Work
Conflict
Resolution
Meaning
Negotiation
Problem
solving
Planning
Brainstorming
Inquiry
Decision
making
Knowledge
building
Group
dynamics
(form, storm,
norm)
Peer
help/tutoring

6. Meaning Negotiation/
Conflict Resolution
Scope: Support the skills needed to bridge
different perspectives to build mutual
understanding and mutual regard

7. Educational Priorities
• King & Baxter (2005) note that “In times of
increased global interdependence, producing
interculturally competent citizens who can
engage in informed, ethical decision-making
when confronted with problems that involve a
diversity of perspectives is becoming an urgent
educational priority [however, these skills] are
what corporations find in shortest supply
among entry-level candidates"

8. Areas of Application:
Dialog/Deliberation
Dispute/Conflict Resolution
• Civic engagement/public dialogue
• International & inter-group conflict
• Labor/management, consumer disputes
alternative dispute resolution
• Interpersonal disputes / mediation
• Deliberative decision making
(school, work, home)

9. Social Deliberative Skills:
Social/Emotional/Reflective
• 1. Social perspective taking
(cognitive empathy, reciprocal role
taking...)
• 2. Social perspective seeking (social
inquiry, question asking skills...)
• 3. Social perspective monitoring
(self-reflection, meta-dialogue...)
• 4. Social perspective weighing
(reflective reasoning; comparing and
contrasting views...)
9

10. Text Coding Scheme
10

11. Social Deliberative Skill:
application of HOSs to me/you/we
Higher Order Skills
• argumentation
• critical thinking
• explanation & clarification
• inquiry/curiosity
(question asking & investigation)
• reflective judgment
• meta-cognition
• epistemic reasoning
Apply these skills, not to
EXTERNAL REALITY (“IT”/problem
domain) but to the
INTERSUBJECTIVE domain
Higher Order Skills applied to:
SELF
goals; level of certainty;
feelings, values, assumptions…
YOU
goals, assumptions, feelings,
values; perspective taking;
"believing" & cognitive empathy…
WE
agreements, goals; quality of
the discourse/collaboration;
differences and similarities in
values, beliefs, goals, power, roles…

12. Corpora and Rater Agreement

13. Examples of Social Deliberative
Skills/Behavior
From authentic dialogues in our online
corpora
“ I am probably extremely biased because I am
under 21 years old and in college. I wonder if as a
45 year old I will feel differently. ” (self reflection)
“I can’t help but imagine what that is like, for her
and for her family.” (perspective taking)
13

14. Support/Scaffolding (vs. “Education”)
Online Dialogue &
DELIBERATION
Outcomes:
- Agreements/solutions
- Relationship, Trust (social capital)
- SKILL USE (and practice)
Existing
Skills
Adaptive
Support
(4th party)
Passive
Support
(interface)
Facilitator
Support
(Dashboard)

15. 2. Classroom Studies

16. [CURRENT] WEEK 1: Discuss the pros and cons of leg...
UPDATE PROFILE
LOG OUT
HOME
Logged in as tomm
[CURRENT] WEEK 1: Discuss the pros and cons of legalizing marijuana.[CURRENT] WEEK 1: Discuss the pros and cons of legalizing marijuana.
To focus the conversation, we invite you to assume you are on an advisory panel for the state
legislature, having some preliminary conversations online, and you will eventually be drafting
a group recommendation. Consider not only your own preferences but what is best for the
state (or society).
edit delete
CONTRIBUTE YOUR THOUGHTS
14:53 EDT Sunday, November 13 by tomm
tomm has joined the conversation
23:53 EDT Saturday, November 12 by ines- v
ines-v added a resource: 'Getting a Fix'
23:52 EDT Saturday, November 12 by ines- v
I have to disagree with your third point that marijuana is a gateway drug. Of
all the people I know that smoke marijuana, they do not do any hard drugs.
I do agree that gateway drugs exist, however I feel like that typically
happens from one hard drug to another when one doesn't seem to be
enough. But if you want to talk about gateway drugs we would also have to
mention alcohol and cigarettes which many people consume and smoke.
Alcohol and cigarettes are also drugs and often considered gateway drugs.
They are both legal so that option is void in regards to marijuana.
You also mentioned cancer and other lung related issues. Marijuana is a
natural plant. Cigarettes are made up of extremely harmful chemicals that
cause lung related issues and cancer much faster than marijuana ever could.
Yet, they are still legal. If anything, cigarettes should be illegal when
considering public health. Marijuana is a lot safer than cigarettes.
I do appreciate you playing Devil's advocate though!
I'd like to explain how I see it differently (ines-v)
18:26 EDT Friday, November 11 by arthur- x
It seems like the vast majority is supportive of the legalization of marijuana,
so I'm going to play devil's advocate in order to bring the opposition's side
to the table.
First off, research has demonstrated that marijuana use reduces learning
ability by limiting the capacity to absorb and retain information. A 1995
study of college students discovered that the inability of heavy marijuana
users to focus, sustain attention, and organize data persists for as long as 24
hours after their last use of the drug. Earlier research, comparing cognitive
abilities of adult marijuana users with non-using adults, found that users fall
short on memory as well as math and verbal skills. Although it has yet to be
proven conclusively that heavy marijuana use can cause irreversible loss of
intellectual capacity, animal studies have shown marijuana-induced
ines-v
arthur-x
joseph-t
laura-t
rtwells
matthew-s
tomm
DIALOGUE TABLE
Everyone (no demographics set)
16

17. 17
Mediem
Opinion Sliders

18. Study: Classroom Dialog
• 26 College Students in 2 week
online discussion
• 2 Topics: Trayvon Martin Shooting
& Gun Control
• 3 Experimental conditions/ 3
discussion groups
• 829 text segments from 369 posts
• 43% of the segments coded as
"deliberate skill”

19. Experimental Conditions
Exp Group N Gender Grade
Vanilla 8 (5 Female,
3 Male)
4 soph, 4 juniors,
0 seniors
Reflective Tools 8 (5 Female,
3 Male)
4 soph, 2 juniors,
2 seniors
(Sliders) 8 (Group omitted due to interaction issues)
19
• Sliders group omitted (did not use tools; poor group dynamics)
• V&R groups: 241 posts and 516 segments (average
of 15.06 (SD = 7.45) posts/student)
• Mean words/post = 54 (SD = 42); mean
characters/post = 299 (SD = 242)

20. Total Skill score adds:
• Intersubjectivity: perspective taking or question
asking
• Meta-dialogue, discussing the quality of the
dialogue
• Meta-Topic: Birds eye or systemic view of the topic
• Appreciation (Gratitude, affirmation of another's
idea or situation)
• Source Reference (Mentioning a source, with a
reference or description; without a fact)
• Apology
20

21. Total Skill vs. Condition
21

22. Intersub vs. Condition
22

23. Main Effect
Exp. Group Total_
SD_Skill
Intersubjective
speech acts
Vanilla (N = 8) 0.29 (0.07) 0.20 (0.09)
Reflective Tools (N = 8) 0.40 (0.08) 0.30 (0.08)
23
• A significant difference and main effect between
Total-SD-Score and grouping, F(1, 14) = 6.89, p =
0.02*, d = 1.46 (a large effect) in favor of the
Reflective Tools group
• A significant relationship between Intersub and
grouping, F(1, 14) = 4.81, p = 0.05*, d = 1.05 (a large
effect) in favor of the Reflective Tools group

24. Comparison of Discussion Topics
24
Reflective tool use vs. topic
Story words vs. topic

25. Other Results
• No effects of sub-skill vs. gender, except females
scored higher on Appreciation
• Positive correlation between Total Skill and post-survey
scores on self-scored Engagement (r = 0.44) and
Learning (r = 0.21) (no correlation vs. Enjoyment question)
• No correlation between Replies-from and Replied-to
vs. experimental group
• The main effect of Condition vs. Total-skill came from
the Trayvon discussion (Gun Control topic had less engagement)
• Most of main effect of Total-skill from the Intersub sub-
skill

26. 3. Facilitator Dashboard

27. 27

28. View by Gender
28

29. Linguistic Features – LIWC
80+
features
5
categories
Linguistic process
(e.g., total words
per sentence, %
of pronouns)
Psychological
process (e.g., affect,
cognition)
Paralinguistic
dimensions (e.g.,
assents, fillers)
Punctuation (e.g.,
quotation marks,
exclamation marks)
Contents (excluded
from this study)
29

30. Dashboard Text Tagging
30

31. 31

32. Advice Screen

33. Settings

34. Next: Linked
Representations
34

35. Future: Additional Metrics
Common problems encountered in online facilitation
• Low or no participation of individuals or groups, or
silences or lulls on the part of individuals, the entire
group, or sub-groups
• Conversation domination by an individual or group
• Inappropriate or disrespectful behavior
• Off-topic conversation
• Tension-filled disagreements, or high emotional
content
• Too much agreement or politeness
• Misunderstanding due to missing communication skills
normally available in face-to-face communication

36. 4. Automated Text Analysis/Classification
Text Analysis of Deliberative Skill:
Using L1 Regularized Logistic Regression
with Psycholinguistic Features

37. Research Approach
• Analyze online dialogues through a
variety of lexical, discourse, and
gender demographic features
• Create machine learning classifiers to
recognize social deliberative skills
– “Total Skill”
– Individual Skills (future work)
37

38. Text Coding Scheme
38

39. Total Skill score adds:
• Intersubjectivity: perspective taking or question
asking
• Meta-dialogue, discussing the quality of the
dialogue
• Meta-Topic: Birds eye or systemic view of the topic
• Appreciation (Gratitude, affirmation of another's
idea or situation)
• Source Reference (Mentioning a source, with a
reference or description; without a fact)
• Apology
39

40. Corpora and Rater Agreement

41. Demographic Features – Gender
• Data distribution
41
Motivation: Woolley et. al, have shown that women score higher on
social sensitivity than men do.

42. Code Frequencies (classroom)
Inter-
sub
Meta_
Dialogue
Meta_
Topic
Apology Apprecia
tion
Fact_
Source
Source_
Ref
#students 22 5 15 1 8 1 4
%segmts 25% 0.9% 5.5% 0.2% 1.3% 0.3% 1.2%
42

43. Code Frequencies in Several Domains
Exp. Group Total_
SD_Skill
Intersubjective
speech acts
Vanilla (N = 8) 0.29 (0.07) 0.20 (0.09)
Reflective Tools (N = 8) 0.40 (0.08) 0.30 (0.08)
43
• A significant difference and main effect between
Total-SD-Score and grouping, F(1, 14) = 6.89, p =
0.02*, d = 1.46 (a large effect) in favor of the
Reflective Tools group
• A significant relationship between Intersub and
grouping, F(1, 14) = 4.81, p = 0.05*, d = 1.05 (a large
effect) in favor of the Reflective Tools group

44. Linguistic Features – LIWC
80+
features
5
categories
Linguistic process
(e.g., total words
per sentence, %
of pronouns)
Psychological
process (e.g., affect,
cognition)
Paralinguistic
dimensions (e.g.,
assents, fillers)
Punctuation (e.g.,
quotation marks,
exclamation marks)
Contents (excluded
from this study)
44

45. Discourse Features – Coh-Metrix
100+
features
8
categories
Narrativity
Referential
cohesion
Syntactic
simplicity
Word
concreteness
Causal
cohesion
Verb
cohesion
Logical
cohesion
Temporal
cohesion
45

46. Machine Learning Method
• L1 Regularized Logistic Regression
–Auto-select features while learning
–High generalizability via minimizing
training loss and selecting a sparse model
–High transparency like a “glass-box”
model
46

47. Performance Metrics
• Accuracy
What percent of all predictions were correct?
Precision
What percent of the positive predictions were correct?
• Recall
What percent of the positive cases were caught?
• F2
Weighted average of precision and recall that weights
recall twice as high
47

48. Study 1. Single Classroom Dialog
• 26 College Students in 2 week online
discussion
• 3 small discussion groups (of 8 or 9)
• 2 Topics: Trayvon Martin Shooting & Gun
Control
• 829 text segments from 369 posts
• 43% of the segments coded as "deliberate
skill”

49. Predictive performance (in %) of
L1 regularized logistic regression
built using different type of features

50. Results
• Moderate Recall (68%) and F2 (65%)
• LIWC features outperformed Coh-Metrix
• Adding gender and grade level features did
not improve performance
• Possibly encoded within LIWC/Coh-Metrix

51. Study 2: Multi-Domain
Dialogue Analysis
• College dialogues – 4 college classes
– Posts from college students from a variety of disciplines
participating in e-discussions, about controversial topics.
• Civic deliberation
– E-Democracy.org, neighborhood discussion about ethnic
tensions in a multi-racial community.
• Professional community negotiation
– Email exchanges among faculty of two academic
communities deciding where to schedule a meeting
51

52. Preliminary ML methods comparison
52
Conclusion:
• L1-LRL slightly
outperformed
Naive Bayes and
SVM

53. Study 2 Experimental Design
(using L1-RLR)
• Goals
– Study feature effects on prediction performance
of machine learning models
• Design of 2 scenarios and 54 experiments
– In-domain analysis (6*3 evaluations)
• Evaluate six possible feature configurations in
each domain
– Cross-domain analysis (6*6 evaluations)
• Evaluate six possible feature configurations in
six domain pairs (training and testing)
53

54. 54
Precision 52.7
Recall 100.0
F2 84.8
Accuracy 56.2
Precision 56.9
Recall 69.7
F2 66.7
Accuracy 53.4
Precision 53.5
Recall 90.0
F2 79.2
Accuracy 56.2
Precision 56.8
Recall 70.6
F2 67.3
Accuracy 53.9
Precision 53.9
Recall 87.0
F2 77.5
Accuracy 54.1
Precision 54.0
Recall 87.0
F2 77.5
Accuracy 52.7
Precision 53.1
Recall 88.7
All
LIWC+Gender
Cohmetrix
+Gender
LIWC
+Cohmetrix
Cohmetrix
Gender
LIWC

55. 55
In-Domain
Training

56. In-domain training: Results
• Gender did not predict; nor did adding it to
other features improve prediction
• Classroom domain had poor performance
(probably due to data skew)
• LIWC performed better in Faculty dialogue
recall 90%)
• Coh-Metrix performed better in Civic dialogue
(recall 84%)

57. Data Skew per Domain

58. Cross-Domain Training
Training
corpus
Civic
deliberation
Civic
deliberation
Professional
community
negotiation
Professional
community
negotiation
College
dialogues
College
dialogues
Testing
corpus
Professional
community
negotiation
College
dialogues
Civic
deliberation
College
dialogues
Civic
deliberation
Professional
community
negotiation
Accuracy 52.7 31.7 56.8 31.7 43.2 47.3
Precision 52.7 31.7 56.8 31.7 0.0 0.0
Recall 100.0 100.0 100.0 100.0 0.0 0.0
F2 84.8 69.9 86.8 69.9 0.0 0.0
Accuracy 56.2 47.1 59.6 39.3 42.4 47.3
Precision 56.9 33.4 59.6 32.7 46.8 50.0
Recall 69.7 67.6 89.3 86.9 9.8 2.6
F2 66.7 56.1 81.2 65.2 11.6 3.2
Accuracy 53.4 36.4 53.3 39.0 41.9 47.0
Precision 53.5 30.7 58.3 30.2 36.8 42.9
Recall 90.0 80.0 62.2 70.3 3.1 1.3
F2 79.2 60.5 61.4 55.5 3.8 1.6
Cohmetrix
Cross-domain
Gender
LIWC

59. Cross-Domain Training Results
• College domain prediction much better when
training in other domains
• Faculty domain best for training overall
(Recall: 89% Civic; 87% College; 90% Faculty)
• In general LIWC features do slightly better
than CohMetrix, and combining them does
not improve performance

60. 60

61. Conclusions

62. Future Work-Text Analysis
Create multi-task machine learning
models with advanced regularizes
(e.g., sparse group Lasso) to
simultaneously identify each
component social deliberative skills
from online communication
62

64. extra slides

65. Top 10 LIWC features learned by L1
regularized logistic regression

66. Total Skill component correlations