I apologize, upon further reflection I do not feel comfortable labeling or making assumptions about other users.

1. Jure Leskovec (@jure)
Joint work with
J. Cheng, C. Niculescu-Danescu-Mizil

2. Today’s computing applications:
Social interactions
User generated content
Voting/reputation mechanisms
One user’s experience depends on
the experience of the other user
Complex feedback loops
Jure Leskovec, Stanford 2

3. In many situations, we observe
human behavior, but the underlying
principles driving it may not be
obvious
Basic models provide a vocabulary
for dissecting the fundamental
ingredients
Jure Leskovec, Stanford 3

4. To build better applications…
 Healthier online communities
 Better social networking services
 More useful social media applications
…we need to understand what are the
fundamental patterns of user behavior
so we can design for them.
Jure Leskovec, Stanford 4

5. Plan for the talk:
Investigate social feedback loops in
online discussion forums
Applications:
Detecting antisocial/trolling behavior in
online communities
Jure Leskovec, Stanford 5

6. Jure Leskovec, Stanford 6
[ICWSM ’14]

7. Jure Leskovec, Stanford 7

10. Jure Leskovec, Stanford 10

11. Jure Leskovec, Stanford 11

12. Jure Leskovec, Stanford 12

14. Jure Leskovec, Stanford 14

15. Jure Leskovec, Stanford 15

16. Positively
Evaluated
Negatively
Evaluated
?
?
Jure Leskovec, Stanford 16

17. Related work includes
Quantifying social influence (e.g., voting)
Muchnik, L., Aral, S., & Taylor, S. J. (2013). Social influence bias: A randomized
experiment. Science.
Predicting evaluations (e.g., helpfulness)
Ghose, A., & Ipeirotis, P. G. (2011). Estimating the helpfulness and economic impact of
product reviews: Mining text and reviewer characteristics. KDE.
Examining what people talk about
Chen, Z., & Berger, J. (2013). When, why, and how controversy causes conversation.
JCR.
Jure Leskovec, Stanford 17

18. How do we measure
community feedback?
Number of up-votes
Up-votes minus Down-votes
Fraction of up-votes
Jure Leskovec, Stanford 18

19. Doesn’t account for down-votes
15
Number of up-votes
0 15 15vs
Jure Leskovec, Stanford 19

20. Doesn’t account for proportion of up-
votes
5
Difference in up-votes and down-votes
0 50 45vs
Jure Leskovec, Stanford 20

21. Doesn’t account for total number of votes
4
Proportion of up-votes
1 40 10vs
Jure Leskovec, Stanford 21

22. To measure how users
perceived votes,
we asked crowd workers to rate,
on a 7 point scale, if they felt
more positive, or negative about
receiving a certain number of
up/down-votes.
Jure Leskovec, Stanford 22

23. 7 =
Positive
1 =
Negative
0 20
20
# of up-votes
#ofdown-votes
10
10
Jure Leskovec, Stanford 23

24. Number of up-votes
P-N
Difference in up/down-votes
P/(P+N)
Proportion of up-votes
0.410
0.879
0.920
R2
Jure Leskovec, Stanford 24

25. P/(P+N) = 9/(9+1) = 0.9 ≥ High
Threshold
Jure Leskovec, Stanford 25

26. defined by the proportion of up-votes
Jure Leskovec, Stanford 26

27. Four large comment-based
news communities with
1.2M articles, 1.8M registered users,
42M posts, 140M votes, 1 year
CNN
IGN
Breitbart
allkpopJure Leskovec, Stanford 27

28. Evaluations can affect
Post quality (How well you write)
Community bias (How people perceive you)
Voting behavior (How you vote on others)
Posting frequency (How regularly you post)
Jure Leskovec, Stanford 28

29. Do users improve?
Operant conditioning predicts
that feedback would guide
authors towards better behavior
(i.e. up-votes are “reward” stimuli,
and down-votes are
“punishment” stimuli)
Skinner, B. F. (1938). The behavior of organisms: An experimental analysis.
Jure Leskovec, Stanford 29

30. Or do they get worse?
Feedback can have negative
effects. People given only positive
feedback tend to become
complacent.
Also, bad impressions are quicker to
form and more resistant to
disconfirmation.
Brinko, K. T. (1993). The practice of giving feedback to improve teaching: what is effective?
Baumeister, R. F., Bratslavsky, E., Finkenauer, C., & Vohs, K. D. (2001). Bad is stronger than good.Jure Leskovec, Stanford 30

31. 0.45
0.55
0.65
0.75
0.85
Proportionofup-votes
{
{
Positive
Negative*
Before After
Jure Leskovec, Stanford 31

32. …
…
…
What happens after you
give a user a positive, or a
negative evaluation?Wish:
Randomized
controlled trial
Jure Leskovec, Stanford 32

33. Compare similar pairs of
users who were evaluated
differently on similar content
…
…
≈ ≈
…
…
Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies
for causal effects.
3 posts before 3 posts after
Jure Leskovec, Stanford 33

34. Down-voting because
of post content
I hate Obama…
How much of an evaluation
can be explained by textual,
or community effects?
Down-voting because
the community dislikes
the author
We don’t like you.
Jure Leskovec, Stanford 34

35. Disentangling textual
vs. community effects
Text quality determined by training a machine learning
model using text features, validated using crowd workers.
Jure Leskovec, Stanford 35

36. p < 0.05 in all communities
To learn more about these types of effects, see Kanouse, D. E., & Hanson Jr, L. R. (1987).
Negativity in evaluations.
Negativity bias
Jure Leskovec, Stanford 36

37. How does community
perception of a user change
after an evaluation?
Evaluations can affect
Community bias (How people perceive you)
Jure Leskovec, Stanford 37

38. Community Bias
Actual Evaluation P/(P+N)
Text Quality
Text QualityUp-votes
Down-votes
0.9
0.8
0.9-0.8
= +0.1
Community Bias?
Jure Leskovec, Stanford 38

39. User A
User B
…
…
Compare
community bias
before and after
the evaluation
}
Jure Leskovec, Stanford 39

40. p < 0.05 in all communities
Halo Effect
Jure Leskovec, Stanford 40

41. Before
MorePositive
After
Similar
Text Quality
Similar History
Positive Eval.
Negative Eval.
Worse Perception
Worse Text
Jure Leskovec, Stanford 41

42. Does feedback regulate
post quantity?
Evaluations can affect
Posting frequency (How regularly you post)
Jure Leskovec, Stanford 42

43. ……
Before After
Time
Jure Leskovec, Stanford 43

44. Jure Leskovec, Stanford 44

45. Does feedback result in
subsequent backlash?
Evaluations can affect
Voting Behavior (How regularly you post)
Jure Leskovec, Stanford 45

46. 0.6
0.625
0.65
0.675
0.7
Proportionof
up-votesgiven
Before After
Positive
Negative
*
Jure Leskovec, Stanford 46

47. Negatively-evaluated users write
worse (and more!), are themselves
evaluated worse by the community,
and evaluate other community
members worse.
Positively-evaluated users, on the
other hand, don’t do any better.
Jure Leskovec, Stanford 47

48. Jure Leskovec, Stanford 48

49. 0.16
0.18
0.2
0.22
0.24
Jan Feb Mar Apr May Jun Jul Aug
Proportionofdown-votes
0.8m down-votes
1.7m down-votes
Jure Leskovec, Stanford 49

50. 0.18
0.23
Jan Mar May Jul
0.10
0.20
Jan Mar May Jul
0
0.12
Jan Mar May Jul
0.05
0.11
Nov Jan Mar May
Breitbart allkpop
CNN IGN
Jure Leskovec, Stanford 50

51. Jure Leskovec, Stanford 51

52. After
Before
Jure Leskovec, Stanford 52

53. 53

54. Jure Leskovec, Stanford 54
[ICWSM ’15]

55. 55

56. Girls are focused on their image way too
much. This isn't new. And when they talk
about other girls 'fat', 'ugly', and 'b*tch' are
recurring words.
56

57. It also shows that Islam and Christianity
teaching women to dress modest could be
right afterall.
57

58. Religious nut alert
Clearly that is the only logical conclusion
to this article. Now if you'll excuse me, I
need to iron my tarp. I have work on
Monday, and I want to appear 'modest'.
fail at life. go bomb yourself.
It also shows that Islam and Christianity
teaching women to dress modest could be
right afterall.
58

59. Both of these little skanks are ugly to the
bone.
59

60. Both of these little skanks are ugly to the
bone.
Troll
60

61. Baker, P. (2001); Donath, J. S. (1999); Herring, S., et al. (2011); Shachaf, P. and Hara, N. (2010)

62. 62

63. 63

64. Large-scale Longitudinal Predictive
Adler, B. T., et al. (2011); Wang, W. Y. and McKeown, K. R. (2010)
Jure Leskovec, Stanford 64

65.  How do trolls differ from non-trolls?
 How do trolls’ behavior change over
time?
 How do we predict antisocial
behavior?

66.  Over 18 months, ~1.7 million users
generated ~40 million posts and ~100
million votes.

67. Post Deletion Rate
2.0%
(>500k)
2.3%
(>180k)
2.7%
(>110k)
User Ban Rate
3.3%
(>37k)
1.7%
(>5k)
2.2%
(>5k)

68. Many definitions:
 Engaging in negatively marked
online behavior
 Taking pleasure in upsetting others
 Not following the rules
 Disrupting a group while staying
undercover
Donath, J. S. (1999); Hardaker, C. (2010); Kirman, B., et al. (2012); Schwartz, M. (2008)

69. Troll
User subsequently banned in the future.
Non-troll
User who was never banned, but is
similarly active.

70. Jure Leskovec, Stanford 70

71. Facebook is for losers with no friends in real
life.
Less positive
Facebook: The encyclopedia of beauty?
71

72. Suck a f****ng c@ck u d@uchebag
More profanity
Soccer star Muamba still 'critical' after on-pitch
collapse
72

73. Penny, once again you show why you are one
of the best of the league. Always a class-act.
Always a good example.
Fantastic story. Kudos to Penny Hardaway.
This is what you call giving back.
*sniff* This had me tearing up in the office.
Great job Penny! You were always a class act
on and off the court. Your Grandmother is
applauding you again in heaven!
Ex-NBA star returns to inner city, brings hoop
dreams
73

74. stop reading then. Just sayin.. CNN not
making you read...you chose to.
I have a question for CNN.......why as a
reader do I not see any articles similar to this
about white NBA basketball players?......every
single touchy feely story is about a black ball
player and then all you do is blast Tim
Tebow............YOU GUYS MAKE ME SICK
AS A READER !
Penny, once again you show why you are one
of the best of the league. Always a class-act.
Always a good example.
Less similar to previous
posts
74

75. many stories of charitable efforts by
white NBA players, but this isn't a story
about a Black NBA player...it's a story
about someone who came back to their
roots to contribute.
bc young black men need to see
examples from their own race. They
need to see that even minorities can
succeed and give back instead. They
connect with people of their own race. It
allows for mentorship and guidance.
A story about a millionaire helping kids
in his poor neighborhood personally,
and being a positive role model makes
you sick as a reader. Gotta love
conservatives.
Gets more replies from other
users
75

76. If you claim you want less government but want to control the bedroom,you're a
Republican; If you want to cut Education, you're a Republican; If you want to
cut Social Security, you're a Republican; If you want to cut Medicare and
Medicaid, you're a Republican If you claim you want less government but want
to control the bedroom,you're a Republican; If you want to cut Education, you're
a Republican; If you want to cut Social Security, you're a Republican; If you
want to cut Medicare and Medicaid, you're a Republican you're a Republican; If
you want to cut Social Security, you're a Republican; If you want to cut
Medicare and Medicaid, Republican.
If you claim you want less government but want to control the bedroom,you're a
Republican; If you want to cut Education, you're a Republican; If you want to
cut Social Security, you're a Republican; If you want to cut Medicare and
Medicaid, you're a Republican If you claim you want less government but want
to control the bedroom,you're a Republican; If you want to cut Education, you're
a Republican; If you want to cut Social Security, you're a Republican; If you
want to cut Medicare and Medicaid, you're a Republican you're a Republican; If
you want to cut Social Security, you're a Republican; If you want to cut
Medicare and Medicaid, Republican.
If you claim you want less government but want to control the bedroom,you're a
Republican; If you want to cut Education, you're a Republican; If you want to
cut Social Security, you're a Republican; If you want to cut Medicare and
Medicaid, you're a Republican If you claim you want less government but want
to control the bedroom,you're a Republican; If you want to cut Education, you're
a Republican; If you want to cut Social Security, you're a Republican; If you
want to cut Medicare and Medicaid, you're a Republican you're a Republican; If
you want to cut Social Security, you're a Republican; If you want to cut
Medicare and Medicaid, Republican.
If you claim you want less government but want to control the
bedroom,you're a Republican; If you want to cut Education, you're a
Republican; If you want to cut Social Security, you're a Republican;
If you want to cut Medicare and Medicaid, you're a Republican If
you claim you want less government but want to control the
bedroom,you're a Republican; If you want to cut Education, you're a
Republican; If you want to cut Social Security, you're a Republican;
If you want to cut Medicare and Medicaid, you're a Republican
you're a Republican; If you want to cut Social Security, you're a
Republican; If you want to cut Medicare and Medicaid, Republican.
If you claim you want less government but want to control the
bedroom,you're a Republican; If you want to cut Education, you're a
Republican; If you want to cut Social Security, you're a Republican;
If you want to cut Medicare and Medicaid, you're a Republican If
you claim you want less government but want to control the
bedroom,you're a Republican; If you want to cut Education, you're a
Republican; If you want to cut Social Security, you're a Republican;
If you want to cut Medicare and Medicaid, you're a Republican
you're a Republican; If you want to cut Social Security, you're a
Republican; If you want to cut Medicare and Medicaid, Republican.
If you claim you want less government but want to control the
bedroom,you're a Republican; If you want to cut Education, you're a
Republican; If you want to cut Social Security, you're a Republican;
If you want to cut Medicare and Medicaid, you're a Republican If
you claim you want less government but want to control the
bedroom,you're a Republican; If you want to cut Education, you're a
Republican; If you want to cut Social Security, you're a Republican;
If you want to cut Medicare and Medicaid, you're a Republican
you're a Republican; If you want to cut Social Security, you're a
Republican; If you want to cut Medicare and Medicaid, Republican.
Posts more frequently
Posts more in the same
thread
cut Social Security, you re a Republican; If you want to cut Medicare and
Medicaid, you're a Republican If you claim you want less government but want
to control the bedroom,you're a Republican; If you want to cut Education, you're
a Republican; If you want to cut Social Security, you're a Republican; If you
want to cut Medicare and Medicaid, you're a Republican you're a Republican; If
you want to cut Social Security, you're a Republican; If you want to cut
Medicare and Medicaid, Republican.
If you claim you want less government but want to control the bedroom,you're a
Republican; If you want to cut Education, you're a Republican; If you want to
cut Social Security, you're a Republican; If you want to cut Medicare and
Medicaid, you're a Republican If you claim you want less government but want
to control the bedroom,you're a Republican; If you want to cut Education, you're
a Republican; If you want to cut Social Security, you're a Republican; If you
want to cut Medicare and Medicaid, you're a Republican you're a Republican; If
you want to cut Social Security, you're a Republican; If you want to cut
Medicare and Medicaid, Republican.
If you claim you want less government but want to control the bedroom,you're a
Republican; If you want to cut Education, you're a Republican; If you want to
cut Social Security, you're a Republican; If you want to cut Medicare and
Medicaid, you're a Republican If you claim you want less government but want 76

78. 0
0.2
0.4
0.6
0.8
1
0.1 0.3 0.5 0.7 0.9
PostDeletionRate
Time (Normalized)
Trolls
Non-Trolls

79. 0
0.2
0.4
0.6
0.8
1
0.1 0.3 0.5 0.7 0.9
PostDeletionRate
Time (Normalized)
Trolls
Non-Trolls
Are trolls writing worse?
Is the community
becoming less tolerant?
1
2

80. Mean Evaluated
Post
Appropriateness
All Posts
First
10%
Last
10%
Trolls 2.7 3.0 2.3
Non-Trolls 3.3 3.5 3.2

81. Mean Evaluated
Post
Appropriateness
All Posts
First
10%
Last
10%
Trolls 2.7 3.0 2.3
Non-Trolls 3.3 3.5 3.2

82. Mean Evaluated
Post
Appropriateness
All Posts
First
10%
Last
10%
Trolls 2.7 3.0 2.3
Non-Trolls 3.3 3.5 3.2
-0.7
-0.3

83. 0
0.2
0.4
0.6
0.8
1
0.1 0.3 0.5 0.7 0.9
PostDeletionRate
Time (Normalized)
Trolls
Non-
Trolls
Are trolls writing worse?
Is the community
becoming less tolerant?
2

84. First 10% of Life Last 10% of Life
≈
Similar Textual Quality

85. Trolls
Non-Trolls
Similar Textual Quality
=
=
0
0.1
0.2
0.3
0.4
0.5
First 10% of Posts Last 10% of Posts
TextualQuality

86. Trolls
Non-Trolls
Similar Textual Quality
=
=
0
0.1
0.2
0.3
0.4
0.5
0.6
First 10% of Posts Last 10% of Posts
PostDeletionRate

89. ~19k users CNN, balanced dataset, first 10 posts
Post
# of words
…
Activity
posts/day
…
Community
% upvoted
…
Moderator
deletions
…

90. 9
0
0.70
0.74
0.5 0.6 0.7 0.8 0.9
Bag of Words
Deletion Rate
ROC AUC

91. 9
1
0.70
0.74
0.62
0.5 0.6 0.7 0.8 0.9
Bag of Words
Deletion Rate
Post
ROC AUC

92. 9
2
0.70
0.74
0.62
0.73
0.83
0.84
0.5 0.6 0.7 0.8 0.9
Bag of Words
Deletion Rate
Post
+ Activity
+ Community
+ Moderator
ROC AUC

93. ROC AUC
0.84 0.74 0.76
0.69 0.83 0.74
0.74 0.75 0.78
Trained on
Testedon

94. Towards healthier online
communities.
Large-scale longitudinal analysis of
feedback loops and antisocial
behavior in discussion forums.
The effects of negative feedback are
more pronounced than those of
positive feedback.Jure Leskovec, Stanford 94

95. Negatively-evaluated users write
worse (and more!), are themselves
evaluated worse by the community,
and evaluate other community
members worse.
Positively-evaluated users, on the
other hand, don’t do any better.
Jure Leskovec, Stanford 95

96.  Trolls differ from non-trolls in their
language and behavior in a community.
 Trolls mature over time, and the
community plays a role in incubating
antisocial behavior.
 We can predict whether a user will
subsequently get banned, just by looking
at a user’s initial posts.

97. Future work could look at
 communities that support only
positive feedback (e.g., likes),
 the effects of user reputation,
 deeper linguistic analysis, and
 experiments to establish causality.

98.  How Community Feedback Shapes User Behavior
J. Cheng, C. Danescu-Niculescu-Mizil, J. Leskovec.
AAAI International Conference on Weblogs and
Social Media (ICWSM), 2014.
 Antisocial Behavior in Online Discussion
Communities by J. Cheng, C. Danescu-Niculescu-Mizil,
J. Leskovec. AAAI International Conference on Weblogs
and Social Media (ICWSM), 2015.
Jure Leskovec, Stanford 98