This presentation introduces a ‘learning in the wild’ coding
schema, an approach developed to support learning
analytics researchers interested in understanding the
different types of discourse, exploratory talk, and
conversational dialogue happening on social media. The
research examines how learner-participants (‘Redditors’)
are leveraging subreddit communities to facilitate selfdirected
informal learning practices on the social
networking site.
+971565801893>> ORIGINAL CYTOTEC ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI<<
Learning in the Wild: Coding Reddit for Learning and Practice
1. #pLASMA @SMLABTO
1
Learning in the Wild: Coding Reddit for
Learning and Practice
Priya Kumar, Ryerson University
Anatoliy Gruzd, Ryerson University
Caroline Haythornthwaite, Syracuse University
Sarah Gilbert, University of British Columbia
Marc Esteve del Valle, University of Groningen
Drew Paulin, University of California Berkeley
Hawaii International Conference on System Sciences (HICSS-51)
January 3-6, 2018
2. How do we make sense of the vast amount of data
generated by learners, especially in the “wild”?
‘Learning analytics is concerned with
collecting data from learners’ actions,
developing techniques to analyze
these data, and making the results
useful to practitioners’ (Long &
Siemens, 2011).
Learning analytics focuses on making
sense of “big data” collected from
learning management systems (LMS),
MOOCs and social media
#pLASMA @SMLABTO 2
3. We wanted to know
1. How are social media sites used in informal learning?
2. How are people engaging in informal learning processes on Reddit?
3. What are the different types of exploratory dialogue, behaviours and talks
that occur with learning processes on Reddit?
4. Does informal learning differ across subreddit communities?
#pLASMA @SMLABTO 3
4. Introducing Reddit
• “the front page of the Internet”
• Reddit ranks 17th in terms of global traffic
and 4th in the U.S
• Diversity of niche communities called
‘subreddits’ maintain distinct subcultures,
thematic interests and subject focus
• Subreddit communities are user-generated
and moderated by Reddit members
• Content curation (Upvoting/downvoting)
#pLASMA @SMLABTO 4
‘AskHistorians’ Subreddit (May 15, 2017)
5. Sampled Subreddits
Subreddit Mandate Guidelines and Norms
Askscience • Forum for answering science questions
• Promoting scientific literacy
• Help people understand scientific process
• No medical advice, speculation, anecdotes
• No abusive, offensive comments
• Answers should be supported with reputable
sources
Ask_politics • Promote political knowledge
• Disseminate knowledge about law and policy
• Be civil and on topic
• Reputable and sourced
• Free of layman speculation
Askhistorians • To provide serious academic-level answers to
questions about history
• Be civil, polite and courteous (no racist/sexist
talk)
• No current events, only human past
• Sources are highly encouraged
Askacademia • Questions and discussions about
Academia/Academic life
• Directed toward people interested in academia
(undergrad, grad, postdocs, profs, laymen)
• No off-topic unproductive discussions
• No rudeness, ad hominin attacks
• Treat strangers as you would in person
6. Data Collection
Subreddit
Community
Number of
Moderators
Number of
Subscribers
Number of
Posts from 2016
Coded Sample
(1%)
Askscience 433 14M 223K 2,235
Ask_politics 8 26K 46K 464
AskHistorians 40 600K 122K 1,227
AskAcademia 3 32K 26.9K 269
#pLASMA @SMLABTO 6
7. Analytic Framework: Discourse Analysis
Exploratory Dialogue Schema from Ferguson et al. (2013)
#pLASMA @SMLABTO 7
Cue Phrase Linguistic Dialogue Example
Critiques ‘However’, ‘I’m not sure’, ‘maybe’
Discussion of Resource ‘Have you read, ‘more links’
Evaluations ‘Good Example’, ‘good point’
Explanations ‘Means that’, ‘our goals’
Explicit Reasoning ‘Next step’, ‘relates to’, ‘that’s why’
Justifications ‘I mean’, ‘we learned’, ‘we observed’
Others’ Perspectives ‘Agree’, ‘here is another’
8. Our Proposed Coding Schema
Code Definition
1. Explanation with Disagreement Expresses a NEGATIVE take on the content of the previous posts by adding new ideas or facts to
discussion thread.
2. Explanation with Agreement Expresses a POSITIVE take on the content of the previous posts by adding new ideas or facts to
discussion thread.
3. Explanation with Neutral
Presentation
Expresses a NEUTRAL explanation/judgment/reasoning/etc. with neither negative nor positive
reference to the content of the previous posts, nor necessarily any reference to previous posts.
4. Socializing with Negative Intent Socializing that expresses negative affect through tone, words, insults, expletives intended as
abusive.
5. Socializing with Positive Intent Socializing that expresses positive affect tone, praise, humor, irony intended in a positive way.
6. Information Seeking Postings asking questions or soliciting opinions, resources, etc. (‘Does anyone know …?’ ‘How
does this work?’). This does not include questions answered rhetorically within the post, e.g., if
a question is asked and answered.
7. Providing Resources Postings that include direct reference to a URL, book, article, etc.; postings that call upon a
well-known theory or the name of a well-known figure.
8. Subreddit Rules and Norms Postings on topics such as what is the appropriate sub-reddit for a particular discussion, what
language is appropriate to use, how to back up claims by using resources, etc.
9. Example: Code 1 – Explanation with Disagreement
From ‘askpolitics’:
#pLASMA @SMLABTO 9
Definition Linguistic Dialogue Example
Expresses a NEGATIVE take on the content of the
previous posts by adding new ideas or facts to
discussion thread.
‘But’, ‘I disagree’, ‘not sure’, ‘not exactly’ with
explanation/ judgment/ reasoning/ etc.
10. Code 2 – Explanation with Agreement
From ‘askacademia’:
Yep. Not every company is the same. In fusion research the three private companies I think of
are all headed by nerds and geeks who want to bring commercial fusion to people not
necessarily to make themselves bajillionaires.
#pLASMA @SMLABTO 10
Definition Linguistic Dialogue Example
Expresses a POSITIVE take on the content of the
previous posts by adding new ideas or facts to
discussion thread.
‘Indeed’, ‘also’, ‘I agree’, with explanation/
judgment/ reasoning/ etc.
11. Code 3 – Explanation w/ Neutral Presentation
From ‘askhistorians’:
The division into light and dark elves is from Snorri's Prose Edda. The Poetic Edda barely makes a
distinction between gods and elves, and often mentions them together.
#pLASMA @SMLABTO 11
Definition Linguistic Dialogue Example
Expresses a NEUTRAL
explanation/judgment/reasoning/etc. with neither
negative nor positive reference to the content of the
previous posts, nor necessarily any reference to previous
posts.
Posts with non-judgmental language. Advice,
brainstorming and first hand experiences are
framed neutrally. ‘I can understand’, ‘interesting’,
‘depends on…’ or statement responses.
12. Code 4 – Socializing w/ Negative Intent
From ‘askpolitics’:
Your lack of understanding of communism is disgusting especially when you feel qualified enough
to talk about it.
#pLASMA @SMLABTO 12
Definition Linguistic Dialogue Example
Socializing that expresses negative affect through
tone, words, insults, expletives intended as
abusive.
‘no’, ‘you’re an idiot’, ‘this has been
explained multiple times’
13. Code 5 – Socializing w/ Positive Intent
From ‘askacademia’:
Your response is EXACTLY how I feel thank you.
#pLASMA @SMLABTO 13
Definition Linguistic Dialogue Example
Socializing that expresses positive affect tone,
words, praise, humor, irony intended in a positive
way.
‘thanks’, ‘great feedback’, ‘you’re correct’
14. Code 6 – Information Seeking
From ‘askscience’:
So then what happens when the aircraft’s velocity vector shifts closer to the nose-up attitude on
takeoff Doesn’t the aircraft then lose the angle-of-attack lift increase.
#pLASMA @SMLABTO 14
Definition Linguistic Dialogue Example
Postings asking questions or soliciting opinions, resources,
etc. (‘Does anyone know …?’ ‘How does this work?’). This
does not include questions answered rhetorically within
the post, e.g., if a question is asked and answered.
‘First you have to think what happens if …?’ and
then you can see what happens’, ‘does anyone
know’, ‘can anyone explain’
15. Code 7 – Providing Resources
From ‘askhistorians’:
I don't know of any that engage in the debate, but the collection *The Medieval City Under Siege*
has a good essay by Bert S. Hall which covers the sanitation and supply problems a large army
would have faced. *Logistics of Warfare in the Age of the Crusades* covers similar ground.
#pLASMA @SMLABTO 15
Definition Linguistic Dialogue Example
Postings that include direct reference to a URL,
book, article, etc.; postings that call upon a well-
known theory or the name of a well-known figure.
Link to resource copied to post (book, URL,
article, audio/video file). Referencing
theory/theorists, scholar or public work
(Einstein, Newton, Freud).
16. Code 8 – Subreddit Rules and Norms
From ‘askacademia’:
I know this is r AskAcademia and I should clarify this is for an industry position. I didn t know
where else on reddit I could ask but this seminar is fairly similar to academic ones I guess and I
figured this would be relevant for other folks doing the same . I think what is freaking me out
the most is I will giving a talk to folks who have pioneered some of the analyses I use in my
dissertation.
#pLASMA @SMLABTO 16
Definition Linguistic Dialogue Example
Postings on topics such as what is the appropriate
sub-reddit for a particular discussion, what
language is appropriate to use, how to back up
claims by using resources, etc.
‘See/don’t forget subreddit link’, ‘this post
doesn’t belong here’, upvote/downvote
mentions, acknowledging OP redditors, and
bots.
18. Summary of Results
Different Types of Discourse around Learning in Reddit
Differences across Subreddits
• ‘Ask_politics’ shows a greater proportion of posts with negative intent
• ‘AskAcademia’ has the highest neutral explanation with positive socializing
• ‘AskHistorians’ and ‘askscience’ are Q&A learning spaces, with direct
references to external resources
#pLASMA @SMLABTO 18
19. Future Work
• Apply machine learning techniques to automate the coding process
• Extend to other social media datasets
• Twitter, Facebook, YouTube
• LinkedIn, ResearchGate
#pLASMA @SMLABTO 19
20. #pLASMA @SMLABTO
20
Learning in the Wild: Coding Reddit for
Learning and Practice
Priya Kumar, Ryerson University
Anatoliy Gruzd, Ryerson University
Caroline Haythornthwaite, Syracuse University
Sarah Gilbert, University of British Columbia
Marc Esteve del Valle, University of Groningen
Drew Paulin, University of California Berkeley
Hawaii International Conference on System Sciences (HICSS-51)
January 3-6, 2018
Notes de l'éditeur
Move all to right side (smaller)
Confirm title and order of paper
New considerations for mobile learning tools,
People on the go can spend waiting time learning.
This is becoming an important feature both in the corporate and education markets.
Subreddit coding schema to better understand and assess the different types of discourse, exploratory talks and overall quality of learner conversations happening on the Reddit social networking website. *** is the schema a part of the slide (we want to know if our developed coding scheme helps us uncover informal learning in the wild?
members (Redditors) can use the social networking website to share various types of information and resources across a plethora of niche communities (called ‘subreddits’). These online community groups are created and moderated by Reddit members and maintain distinct subcultures through their thematic and subject focus
community-centric collaboration, socialization and crowd-based informal discussion in facilitating self-directed ‘life-long’ learning opportunities.
user-generated participatory online culture through its informal, openly accessible, group-based subreddit communities.
self-organized community website for people interested in current affairs, social commentary and Internet subcultures
As of Sept, 2017, Reddit currently ranks 17th in terms of total global traffic, and 4th in the U.S, a country in which over half of its total users reside (Alexa, 2017).
The growing popularity of the website can be explained by the wide range of topics being discussed across an equally diverse set of user-generated communities. This includes the ever-popular AMA and IAmA (‘ask me anything’/’I am a’) Q&A live series which often hosts mainstream public figures (like former President Barack Obama, astrophysicist Neil deGrasse Tyson) and countless other influential people.
Subreddit communities are moderated and content is user-generated, affording its users with the opportunity to anonymously comment on, browse through and stay updated on a multiplicity of subjects at their discretion.
---
https://www.flickr.com/photos/ajc1/8190377106/in/photolist-dtKPLo-9pq89r-9pq7Vi-dABKSh-9pt9w1-9pt9qy-9pt9sE-9pq84R-9pq7Se-9pq7PF-9pq87i-9pt9xs-CBQ96-6UAnkM-9qNY97-6xEZxa-93jaRv-7ZfgGd-efytQt-ikJ9VX-cStVZm-pfmXfv-e5NgTU-3asaqG-aj1f26
The counts for the #of Posts are after removing “[deleted]” and “[removed]” comments.
[deleted] comments were deleted by the user and [removed] were deleted by moderators.
========
Inter-coder reliability
Applied to sample of reddit messages from 4 subreddits
By three independent coders :
- Reliability data statistics -
Krippendorff’s alpha:
Askscience 0.69 (78% agreement)Askpolitics 0.60 (72%)
AskHistorians 0.76 (79%)
Askacademia 0.64 (77%)
For exploratory studies like ours, alpha levels between 0.67 and 0.80 are considered reliable enough to draw out and develop cautionary conclusions [27, 28].
Explanation with disagreement: Expresses a NEGATIVE take on the content of the previous posts by adding new ideas or facts to discussion thread (‘But’, ‘I disagree’, with explanation/ judgement/ reasoning/ etc.)
IT WASN’T
Link to the example: https://www.reddit.com/r/Ask_Politics/comments/2qz25g/why_was_the_bush_administration_so_defiant/cnbg53k/
Explanation with agreement: Expresses a POSITIVE take on the content of the previous posts by adding new ideas or facts to discussion thread (‘Indeed’, ‘also’, ‘I agree’, with explanation/ judgement/ reasoning/ etc.)
Explanation with neutral presentation: Expresses a NEUTRAL explanation/judgement/reasoning/etc. with neither negative nor positive reference to the content of the previous posts, nor necessarily any reference to previous posts
4. Socializing with negative intent: Socializing that expresses negative affect through tone, words, insults, expletives intended as abusive (‘no’, ‘you’re an idiot’, ‘this has been explained multiple times’)
Socializing with positive intent: Socializing that expresses positive affect tone, words, praise, humour, irony intended in a positive way (‘thanks’, ‘great feedback’, ‘you’re correct’)
Information Seeking: Postings asking questions or soliciting opinions, resources, etc. (‘Does anyone know …?’ ‘How does this work?’). This does not include questions answered rhetorically within the post, e.g., if a question is asked and answered (‘First you have to think ‘What happens if …?’ and then you can see what happens.’)
Providing Resources: Postings that include direct reference to a URL, book, article, etc.; postings that call upon a well-known theory or the name of a well-known figure (e.g., Newton; Freud); or postings that refer to the work of others (e.g., Gruzd’s work on social media)
Subreddit Rules and Norms: Postings on topics such as what is the appropriate sub-reddit for a particular discussion, what language is appropriate to use, how to back up claims by using resources, etc.
NOTE 1: The counts/percentages are based on when at least 2 coders agreed on a code;
NOTE 2: The total percentage >100% because a coder was allowed to select up to 3 codes per post.
From these results we see right away that most of our sample of comments, messages and posts are neutral in tone however there are key difference
ask academia - a more personal subreddit, we see, a high proportion of positive socializing (supportive messages)
For ask science, we see a high proportion of information seeking comments (code 6)
ask politics: we see a high proportion of explanation with disagreement (code 1) and negative socializing (contentious topics like Trump)
ask historians - we see a proportion of positive socializing (ex: thank you) and similar Q&A (code 6 and 7), only subreddit with referral to rules and norms
----
Differences across Subreddits
‘AskAcademia’ has the highest neutral explanation with positive socializing
‘‘Ask_politics’ shows a greater proportion of posts with negative intent and affect
AskHistorians’ and ‘askscience’ are Q&A learning environments, with direct reference to resources
https://www.reddit.com/r/elearning/ is a niche community for those wanting to discuss the techniques, difficulties and joys of creating, applying and evaluating elearning of all types.
Move all to right side (smaller)
Confirm title and order of paper