can anyone become an internet troll? - web science · 2016-06-07 · complementing observational...

Can Anyone Become an Internet Troll?

Jure Leskovec (@jure)Including joint work with Justin Cheng,M. Bernstein, C. Danescu-Niculescu-Mizil

Two Metaphors for the WebWeb as a library Web as a social place

+–

You?

Me! No, me!

Web as a Social Place

3

Crucial for the success of the Web is the ability of

users to contribute content and provide feedback

Two Metaphors for the WebWeb as a library§ One static

version§ Everyone’s

experience is fixed

§ No user interactions

Web as a social place§ Complex dynamic

feedback effects§ One user’s

experience is a function of other users’ experiences

§ User interactions

ChallengesWant users to engage and contribute (good) content (and make others’ experiences better)§ How do we keep users engaged?§ How do we promote good behavior?§ How do we ensure health of the

community and the content ecosystem?

Challenges§ We know a lot about overall average user

behavior in online communities§ But many times few users can have a

disproportionate (negative) effect on the entire community

§ How do we study and model behavior of such users?

§ Have do we use the insights to improve online communities?

Antisocial BehaviorInternet Troll is a person who sows discord on the Internet§ by starting arguments and upsetting

people§ by posting inflammatory, extraneous,

or off-topic messages in an online community

11

[wikipedia]

13

Girls are focused on their image way too much. This isn't new. And when they talk about other girls 'fat', 'ugly', and 'b*tch' are recurring words.

14

It also shows that Islam and Christianity teaching women to dress modest could be right afterall.

15

Religious nut alert

Clearly that is the only logical conclusion to this article. Now if you'll excuse me, I need to iron my tarp. I have work on Monday, and I want to appear 'modest'.

fail at life. go bomb yourself.

It also shows that Islam and Christianity teaching women to dress modest could be right afterall.

16

Both of these little skanks are ugly to the bone.

17

Both of these little skanks are ugly to the bone.

Troll

PROBLEM: Trolls disrupt online discussions

Baker, P. (2001); Donath, J. S. (1999); Herring, S., et al. (2011); Shachaf, P. and Hara, N. (2010)

18

Facebook is for losers with no friends in real life.

Less positive

19

Facebook:Theencyclopediaofbeauty?

Suck a f****ng c@ck u d@uchebag

More profanity

20

SoccerstarMuamba still'critical'afteron-pitchcollapse

I have a question for CNN.......why as a

Penny, once again you show why you are one of the best of the league. Always a class-act. Always a good example.

Fantastic story. Kudos to Penny Hardaway. This is what you call giving back.

*sniff* This had me tearing up in the office. Great job Penny! You were always a class act on and off the court. Your Grandmother is applauding you again in heaven!

21

Ex-NBAstarreturnstoinnercity,bringshoopdreams

stop reading then. Just sayin.. CNN not making you read...you chose to.

How many white NBA players grew up

I have a question for CNN.......why as a reader do I not see any articles similar to this about white NBA basketball players?......every single touchy feely story is about a black ball player and then all you do is blast Tim Tebow............YOU GUYS MAKE ME SICK AS A READER !

Penny, once again you show why you are one of the best of the league. Always a class-act. Always a good example.

This is what you call giving back.

Off-topic: Less similar to previous posts

22

many stories of charitable efforts by white NBA players, but this isn't a story about a Black NBA player...it's a story about someone who came back to their roots to contribute.

b\c young black men need to see examples from their own race. They need to see that even minorities can succeed and give back instead. They connect with people of their own race. It allows for mentorship and guidance.

A story about a millionaire helping kids in his poor neighborhood personally, and being a positive role model makes you sick as a reader. Gotta love conservatives.

Gets more replies from other users

23

If y ou clai m yo u w an t le ss go ve rn me nt bu t w an t t o c on tr ol t he b ed ro om ,yo u'r e a R ep ublic an ; I f y ou wa nt tocut Ed uca tio n, yo u're a R ep ublic an ; I f y ou w an t t o cut So cial Sec uri ty, yo u'r e a Re pu blica n; If yo u wa nt tocut M edic ar e an d Me dic aid, yo u'r e a R ep ublic an If yo u clai m y ou wa nt les s g ov er nm en t but wa nt to co nt ro lthe be dr oo m,y ou' re a Re pu blic an; I f yo u wa nt t o cu t Ed uca tio n, yo u'r e a R ep ubli ca n; If y ou w an t to c u tSocial Sec uri ty, yo u'r e a Re pu blic an; If yo u wa nt t o c ut M edi ca re an d Me dic aid, y ou'r e a R ep ubli ca n y ou' rea Re pu blica n; I f yo u wa nt t o c ut So cial Se cu rity, yo u're a R ep ublic an ; If y ou w an t to cut Me dic ar e a ndMedicaid, Republican.



If yo u cl aim yo u wa nt l ess go ve rn me nt but wa nt t o c ont rol th e b ed ro om ,yo u'r e aRep ubli ca n; If y ou w an t to c ut Ed uc atio n, y ou' re a Re pu blica n; If y ou wa nt to cut S ocialSecu rity , yo u're a Re pu blic an; I f yo u wa nt t o cut Me dic ar e a nd Me dicai d, y ou' re aRep ubli ca n I f y ou cl aim y ou wa nt les s gov er n me nt bu t w an t t o co ntr ol th e be dr oo m, yo u'rea R ep ubli ca n; If yo u wa nt to cu t E du cati on , y ou' re a R ep ublic an ; I f yo u w an t t o cu t S ocialSecu rity , yo u're a Re pu blic an; I f yo u wa nt t o cut Me dic ar e a nd Me dicai d, y ou' re aRep ubli ca n y ou' re a Re pu blic an; If y ou wa nt to c ut S oci al S ecu rity , y ou' re a Re pu blic an; Ifyou want to cut Medicare and Medicaid, Republican.If yo u cl aim yo u wa nt l ess go ve rn me nt but wa nt t o c ont rol th e b ed ro om ,yo u'r e aRep ubli ca n; If y ou w an t to c ut Ed uc atio n, y ou' re a Re pu blica n; If y ou wa nt to cut S ocialSecu rity , yo u're a Re pu blic an; I f yo u wa nt t o cut Me dic ar e a nd Me dicai d, y ou' re aRep ubli ca n I f y ou cl aim y ou wa nt les s gov er n me nt bu t w an t t o co ntr ol th e be dr oo m, yo u'rea R ep ubli ca n; If yo u wa nt to cu t E du cati on , y ou' re a R ep ublic an ; I f yo u w an t t o cu t S ocialSecu rity , yo u're a Re pu blic an; I f yo u wa nt t o cut Me dic ar e a nd Me dicai d, y ou' re aRep ubli ca n y ou' re a Re pu blic an; If y ou wa nt to c ut S oci al S ecu rity , y ou' re a Re pu blic an; Ifyou want to cut Medicare and Medicaid, Republican.If yo u cl aim yo u wa nt l ess go ve rn me nt but wa nt t o c ont rol th e b ed ro om ,yo u'r e aRep ubli ca n; If y ou w an t to c ut Ed uc atio n, y ou' re a Re pu blica n; If y ou wa nt to cut S ocialSecu rity , yo u're a Re pu blic an; I f yo u wa nt t o cut Me dic ar e a nd Me dicai d, y ou' re aRep ubli ca n I f y ou cl aim y ou wa nt les s gov er n me nt bu t w an t t o co ntr ol th e be dr oo m, yo u'rea R ep ubli ca n; If yo u wa nt to cu t E du cati on , y ou' re a R ep ublic an ; I f yo u w an t t o cu t S ocialSecu rity , yo u're a Re pu blic an; I f yo u wa nt t o cut Me dic ar e a nd Me dicai d, y ou' re aRep ubli ca n y ou' re a Re pu blic an; If y ou wa nt to c ut S oci al S ecu rity , y ou' re a Re pu blic an; Ifyou want to cut Medicare and Medicaid, Republican.

Posts more frequently

Posts more in the same thread



If y ou clai m yo u w an t le ss go ve rn me nt bu t w an t t o c on tr ol t he b ed ro om ,yo u'r e a R ep ublic an ; I f y ou wa nt tocut Ed uca tio n, yo u're a R ep ublic an ; I f y ou w an t t o cut So cial Sec uri ty, yo u'r e a Re pu blica n; If yo u wa nt tocut M edic ar e an d Me dic aid, yo u'r e a R ep ublic an If yo u clai m y ou wa nt les s g ov er nm en t but wa nt to co nt ro l 24

Discussion Communities

Complete data over 18 months, ~1.7 million users generated ~40 million posts and ~100 million votes

25

Post Deletion Rate 21.4% 2.3% 2.7%

User Ban Rate 3.3% 1.7% 2.2%

How Common is Bad Behavior?

26

This Talk: Trolling§ Large-scale investigation of

trolling in online discussion communities

§ Common theme:Complementing observational data with experiments

27

This Talk: Quesitons1) Who are trolls?

§ Mechanisms behind trolling behaviors: Mood and discussion context

2) Does feedback mitigate trolling?§ Votes lead to cascading of bad behaviors

3) Can we detect trolling early?§ Predicting whether a given user will be

blocked from the community

Who are these people?What makes them troll?Why is trolling so common?

29

30

124

Donate (1999), Baker (2001), Schwartz (2008)

124


124


Studies on TrollingPrior research suggests: § Trolls are born and not made

[Baker ‘01; Herring et al. ‘11; Shachaf-Hara ‘10]§ Trolls have particular personality

[Buckels et al. ’14; Raine ‘02]§ Trolls are sociopathic individuals

[Rensin ’14; Schwartz ‘08]

§ Q: Are trolls really special people or can anyone become a troll?

Hypotheses: Mood & Context

33

How can we study these?

130

Mood Discussion ContextH1: Negative mood increases a user’s likelihood of trolling

H2: The discussion context affects a user’s

susceptibility to troll.

Designing an Experiment

MTukers only care about being paid.How do we design an experiment?

34131

Designing an ExperimentTwo-by-two experimental design:§ Step 1: Mood – Take a test (5min)

§ Easy test & tell people they did well§ Hard test & tell people they did poorly

§ Step 2: Context – Be in a discussion§ Neutral context of neutral comments§ Negative context of negative comments

People do not know the real purpose. Are debriefed at the end.

35

Controlling Mood: POSMOOD

36

Controlling for Mood

132

Controlling Mood: NEGMOOD

Context: NEUTRALCONTEXT

Context: NEGCONTEXT

39

Controlling for Discussion Context

133

Mechanical Turk Experiment

667 participants, 40% female, age 34.2, 54% Democrat, 25% Moderate, 21% Republican

Contributed 791 posts (37.8 words/post), 1392 votes40131

Results: Test Performance§ POSMOOD (easy test):

11 out of 15 questions correct§ Outperforming stated avg. score of 8

§ NEGMOOD (hard test):1.9 out of 15 questions correct§ After below stated avg. score of 8

Does the test score affect user’s mood?

Results: Does it affect mood?§ After the test: 65 questions on how

people were feeling § Based on the Profile of Mood States

§ People in NEGMOOD felt worse:§ Higher anger, confusion, depression,

fatigue, and tension scores

§ How good were the comments?42

Comments: MOOD

§ Users in the negative mood condition are more likely to troll (~80% higher odds, p < 0.05)

43

Prop. Troll Posts Negative Affect (LIWC)POSMOOD NEGMOOD POSMOOD NEGMOOD

NEUTRALCONTEXT 0.35 0.49 0.011 0.014NEGCONTEXT 0.47 0.68 0.023 0.029

Table 1: Both the proportion of user-written posts that werelabeled as trolling and negative affect are lowest in the(POSMOOD, NEUTRALCONTEXT) condition, and highest inthe (NEGMOOD, NEGCONTEXT) condition.

NEGCONTEXT condition, the first three comments were trollposts:

Oh yes. By all means, vote for a Wall Street sellout – alying, abuse-enabling, soon-to-be felon as our next Pres-ident. And do it for your daughter. You’re quite the rolemodel.

In the NEUTRALCONTEXT, they were more innocuous:

I’m a woman, and I don’t think you should vote for awoman just because she is a woman. Vote for her be-cause you believe she deserves it.

These comments were slightly edited from real commentsposted by users in comments in the original article, as wellas other online discussion forums discussing the issue (e.g.,Reddit).

Does this explain universes well? To ensure that the effectswe observed were not path-dependent (i.e., if a discussionbreaks down by chance because of a single user), we createdeight separate “universes” for each condition [71], for a totalof 32 universes. To replicate the interactions that may takeplace in actual online discussions, multiple participants wereassigned to the same “universe” within each condition. Par-ticipants assigned to the same universe could see and respondto other participants who had commented prior, but not inter-act with participants from other universes.

Measuring discussion quality. We evaluated discussionquality in two ways: whether subsequent posts written weretroll posts, and the difference in negative affect of these posts.To evaluate whether a post was a troll post or not, two ex-perts independently labeled posts as being troll or non-trollposts, with disagreements resolved through discussion. Bothexperts reviewed CNN.com’s community guidelines [21] forcommenting – at a high level, posts that were offensive, irrel-evant, or designed to elicit an angry response, whether inten-tional or not, were flagged as trolling. To measure the neg-ative affect of a post, we used LIWC [65] (Vader [39] givesempirically similar results).

[MSB: I’m increasingly a fan of describing the statistical testsup in the method section. What tests did you run on the data?]Like every single statistical test?

Results667 participants (40% female, mean age of 34.2, 54% Demo-crat, 25% Moderate, 21% Republican) completed the experi-ment, with an average of 21 participants in each universe. In

Fixed Effects Coef. SE z(Intercept) �0.70⇤⇤⇤ 0.17 �4.23NEGMOOD 0.64⇤⇤ 0.24 2.66NEGCONTEXT 0.52⇤ 0.23 2.38NEGMOOD ⇥ NEGCONTEXT 0.41 0.33 1.23Random Effects Var. SEUser 0.41 0.64

Table 2: A mixed effects logistic regression reveals a signif-icant effect of both NEGMOOD and NEGCONTEXT on trollposts (⇤: p<0.05, ⇤⇤: p<0.01, ⇤⇤⇤: p<0.001). In other words,both negative mood and the presence of initial troll posts in-creases the probability of trolling.

aggregate, these workers contributed 791 posts (with an aver-age of 37.8 words written per post) and 1392 votes.

Manipulation checks. On average, participants in the POS-MOOD condition obtained an average of 11.2 out of 15 ques-tions correct, performing above the stated “average” score of8. In contrast, participants in the NEGMOOD condition an-swered only an average of 1.9 questions correctly, perform-ing significantly worse (t(594)=63.2, p<0.001), and belowthe stated “average”. Correspondingly, the post-test POMSquestionnaire confirmed that participants in the NEGMOODcondition experienced higher mood disturbance on all axes,with higher anger, confusion, depression, fatigue, and tensionscores, and a lower vigor score (t>7.0, p<0.001). Total mooddisturbance, where a higher scores correspond to more nega-tive mood, was 12.2 for participants in the POSMOOD condi-tion (comparable to a baseline level of disturbance measuredamong athletes [81]), and 40.8 in the NEGMOOD condition.

Similarly, the initial posts in the NEGCONTEXT conditionobtained a significantly lower proportion of up-votes (0.36)as compared to those in the NEUTRALCONTEXT condition(0.90) (t(507)=15.7, p<0.001), demonstrating that partici-pants did perceive these initial troll posts worse.

Negative mood and prior troll posts reduce discussionquality. Table 1 demonstrates how the proportion of trollposts and negative affect (or the proportion of negative words)differ in each condition. The proportion of troll posts washighest in the (POSMOOD, NEUTRALCONTEXT) condition,drops in both the (NEGMOOD, NEUTRALCONTEXT) and(POSMOOD, NEGCONTEXT) conditions, and is lowest in the(NEGMOOD, NEGCONTEXT) condition, which suggests amain effect of either condition. For negative affect, we ob-serve similar, but smaller differences.

Fitting a mixed effects logistic regression model, with thetwo conditions as fixed effects, user as a random effect, andwhether a post was trolling or not as the outcome variable,we do observe a significant effect of both NEGMOOD andNEGCONTEXT in this model (p < 0.05) (Table 2), confirm-ing both H1 and H2. In fact, the presence of prior troll postsincreases the odds of trolling by 64%, and negative mood in-creases the odds of trolling by 79%. An ANOVA comparingmodels without these effects also revealed significant differ-

5

Comments: CONTEXT

§ Users in the negative context condition are more likely to troll (~60% higher odds, p < 0.05)

44




















5

Comments: Negativity

§ Negative mood and context lead to more negative posts

45




















5




















5

Bad mood and context make people write bad posts.

How do we combat that?

46

Sites allow users to vote on the content and hopefully bad posts get voted down

47

51

When a user evaluatesa comment, she is indirectly

evaluating the author

Do such evaluations help authors contribute better

content?How Community Feedback Shapes User BehaviorJ. Cheng, C. Danescu-Niculescu-Mizil, J. LeskovecAAAI ICWSM, 2014

PositiveVote

NegativeVote

?

?

52

§ Number of up-votes?§ Up-votes minus Down-votes?§ Fraction of up-votes?

How to Summarize Feedback?

53

Doesn’t account for down-votes

15

P: Up-Votes

54

Number of up-votes:

0 15 15vs

Doesn’t account for proportion of up-votes

5

P–N : Up – Down Votes

55

Difference in up-votes and down-votes:

0 50 45vs

Doesn’t account for total number of votes

4

P/(P+N): Fraction Upvotes

56

Proportion of up-votes:

1 40 10vs

How is Feedback Perceived?Measure how users perceive votes

§ Crowdsourcing task:Workers rate, on a 7 point scale, if they felt more positive, or negative about receiving a certain number of up/down-votes.

57

7 = Positive

1 = Negative

0 20

20

# of up-votes

# of

dow

n-vo

tes

10

10

Results: Perceiving Votes

User ratings are independent of the total number of votes

58

Number of up-votes

P-NDifference in up/down-votes

P/(P+N)Proportion of up-votes

0.410

0.879

0.920

R2

59

P

Evaluation of a Post§ A post is positively/negatively

evaluated when the proportion of up-votes is higher/lower than a given threshold.

60

P/(P+N) = 9/(9+1) = 0.9 ≥ High Threshold

How does positive/negativefeedback influencesubsequent behavior?

defined by the proportion of up-votes

61

Effects of Evaluations§ Post quality

§ How well you write§ Community bias

§ How people evaluate you§ Posting frequency

§ How regularly you post§ Voting behavior

§ How you vote on others62

Skinner, B. F. (1938). The behavior of organisms: An experimental analysis.

Do Users Improve?Operant conditioning predicts that feedback would guide authors towards better behavior: § up-votes are “reward” stimuli§ down-votes are “punishment” stimuli

63

Brinko, K. T. (1993). The practice of giving feedback to improve teaching: what is effective?Baumeister, R. F., Bratslavsky, E., Finkenauer, C., & Vohs, K. D. (2001). Bad is stronger than good.

Or Do They Get Worse?Feedback can have negative effects:§ People given only positive feedback

tend to become complacent

§ Also, bad impressions are quicker to form and more resistant to disconfirmation

64

…

…≈ ≈

…

…

Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects.

3 posts before 3 posts after

Matching Similar UsersCompare pairs of users who were evaluated differently on similar content

65

Details of MatchingMatch pairs of users where one got positive and one got negative evaluation§ Match based on

§ history text quality§ number of posts§ post length§ overall proportion of up-votes

66

Text quality determined by training a machine learning model using text features, validated using crowd workers.

0.45

0.55

0.65

0.75

0.85

Prop

ortio

n of

up

-vot

es

{ {

Positiveevaluation

Negativeevaluation

*

Before After

Results: Neg-Votes Increase

67

Down-voting because of post content

I hate …

Down-voting because the community dislikes the author

We don’t like you.

Content vs. CommunityCan valuations be explained by textual or community effects?

68

Results: Negativity Bias§ Text quality drops significantly

after a negative evaluationbut does not change after a positive evaluation

p < 0.05 in all communities

To learn more about these types of effects, see Kanouse, D. E., & Hanson Jr, L. R. (1987). Negativity in evaluations.

Evaluations can Affect…

…Community Bias (How people perceive you)

§ How does community perception of a user change after an evaluation?

70

Community Bias

Actual Evaluation P/(P+N)

Text Quality

Text QualityUp-votesDown-votes

0.9

0.8

0.9-0.8= +0.1

Community Bias?

71

User A

User B

…

…

Comparecommunity bias before and afterthe evaluation

}72

Communities DiscriminatePosts made after a negative evaluation were perceived worse than those made after a positive evaluation

§ On CNN, on average, your posts are being evaluated 48% worse than would be predicted by text quality

p < 0.05 in all communities

Before

Mor

e Pos

itive

After

SimilarText Quality

Similar History

Positive Eval.

Negative Eval.

Worse Perception

Worse Text

Summary of Effects

74


…Posting Frequency (How often you post)

§ Does feedback regulate posting quantity?

75

Feedback and Posting Activity§ Do users post more after a

positive/negative evaluation?

76

……Before After

Time

Negativity Increases Activity

Users who receive negative feedback post more frequently

77


…Voting Behavior (How you vote on others)

§ Does feedback result in subsequent backlash of negative behavior?

78

0.60.625

0.650.675

0.7U

p-vo

tes

give

n

Before After

Positive

Negative*

More Down-Votes Given

Users who receive negative feedback are more likely to down-vote others

79

Summary so far…§ Negatively-evaluated users

§ Write worse (and more!), § Are evaluated worse by the community§ And evaluate others worse

§ Positively-evaluated usersdon’t do any better

80

Is there a downward spiral in communities?

81

82

124


83

124


124


124


124


124


124


124


124


84

125


Downward Spiral in CNN

Proportion of down-votes is increasing over time

85

0.16

0.18

0.2

0.22

0.24

Jan Mar May Jul

Prop

.dow

n-votes

0.8m down-votes

1.7m down-votes

0.18

0.23

Jan Mar May Jul0.10

0.20

Jan Mar May Jul

0

0.12

Jan Mar May Jul0.05

0.11

Nov Jan Mar May

Breitbart allkpop

CNN IGN

86

Can we identify antisocial behavior quickly and prevent downward spirals?

88

Antisocial Behavior in Online Discussion CommunitiesJ. Cheng, C. Danescu-Niculescu-Mizil, J. Leskovec.AAAI ICWSM, 2015.

Post# of words…

Activityposts/day…

Community% upvoted…

Moderatordeletions…

Prediction Task§ Task: Detect which users will be

banned in the future

§ ~19k users CNN, balanced dataset, first 10 posts

89

Prediction Results

90

0.70

0.74

0.5 0.6 0.7 0.8 0.9

BagofWords

DeletionRate

ROCAUC

Prediction Results

91

0.70

0.74

0.62

0.5 0.6 0.7 0.8 0.9

BagofWords

DeletionRate

Post

ROCAUC

Prediction Results

92

0.700.74

0.620.73

0.830.84

0.5 0.6 0.7 0.8 0.9

BagofWordsDeletionRate

Post+Activity

+Community+Moderator

ROCAUC

Time to Detection

93

0.70

0.75

0.80

0.85

0.90

0 10 20 30 40Using First N Posts

AUC

>= N5−10

10−2020−40

40−8080+

(a) Performance against # posts

●●

●

●●●

●

●

●

●●●●

●●

●

●●●

●

●

●

●

●

●

●●●●●●●

●

●●●●

●

●●●

●●●●

●●●●●

●●●●

●●●

●

0.825

0.850

0.875

0.900

0 10 20 30Starting from Nth Last Post (Window=5)

AUC

● ● ●10−20 20−40 40−80

(b) Performance against time

Figure 7: (a) Prediction performance increases as more postsare observed. Additionally, users who live longer are moredifficult to identify early on. (b) Using a sliding window offive posts, performance decreases with increasing temporaldistance from the time of deletion.

To understand the relative importance of these features,we computed the classification performance of each in-dividual feature using logistic regression. Unsurprisingly,moderator features are the strongest predictors of beingsubsequently banned (individual feature set AUC=0.75 forCNN), with the most performant feature being the propor-tion of posts deleted (0.73). Community features were nextstrongest (0.75), with a lower proportion of up-votes re-ceived (0.67) and a higher number of reported posts (0.66)both indicators of antisocial behavior. The text similarity ofa post with previous posts in a thread, while correlated withpost deletion, does not improve classifier performance. Ac-tivty features follow (0.66), with the number of posts perday the most indicative (0.64) of being subsequently banned.Post features were collectively the weakest predictors ofwhether a user will ultimately get banned (0.62). Futurework could involve identifying better textual features (e.g.,phrase structure), and take the context of the post (i.e., thesurrounding posts) into account.

How does prediction performance change with the num-ber of posts observed? If the classifier only has access tothe first five posts, it can still predict whether a user will getbanned with an AUC of 0.80 (across all communities). Moregenerally, performance seems to peak near ten posts (Figure7a). The same figure also shows how classifier performancechanges for users with different post counts: the more postsa user eventually makes, the more difficult it is to predictwhether they will get eventually banned later on.

How does prediction performance change with “dis-tance” from when a user gets banned? Instead of lookingat a user’s initial posts, we now consider sliding windowsfive posts in width, starting from the last five posts of a user,to understand if posts made further in the past are as effectiveas predicting whether a user will get banned. As Figure 7bshows, it becomes increasingly difficult to predict whethera user will subsequently get banned the further in time theexamined posts are from when the user gets banned. Thissuggests that changes in both user or community behaviordo occur leading up to a ban.

How does the classifier perform on different types ofusers? We previously identified two types of FBUs: those

Trained onCNN IGN Breitbart

Test

edon CNN 0.84 0.74 0.76

IGN 0.69 0.83 0.74Breitbart 0.74 0.75 0.78

Table 5: Cross-domain classifier performance (AUC) is rela-tively high, suggesting that these learned models generalizeto multiple communities.

with high deletion rates (Hi-FBUs), and those with lowdeletion rates (Lo-FBUs). Overall, the classifier identifiesHi-FBUs (mean recall=0.99) more reliably than Lo-FBUs(mean recall=0.41). As Hi-FBUs exhibit high deletion ratesfrom the beginning of their life while Lo-FBUs do not, fea-tures such as the proportion of deleted posts are highly in-formative in the former case but not the latter. Further, Lo-FBUs write more posts in total than Hi-FBUs, so overall weare examining a smaller fraction of a Hi-FBU’s total life ina community. Still, by assigning higher weights to instancesof Lo-FBUs during training, we can maintain a similar AUC,while increasing recall of Lo-FBUs to 0.57 (but decreasingthat of Hi-Del users to 0.97). If we are only interested dif-ferentiating Lo-FBUs from other users, a classifier trainedon the same features obtains a mean AUC of 0.79 across allcommunities (recall of Lo-FBUs=0.83). In this case, we mayuse other mechanisms to identify Hi-FBUs separately.

How generalizable are these classifiers? Using a modelthat uses all four feature sets, we find that cross-domainperformance is high (mean AUC=0.74) relative to within-domain performance (Table 5), suggesting not only the ap-plicability of these features, but also the generalizability ofmodels learned on single communities. Most striking is thata classifier trained on the Breitbart community only per-forms slightly worse if tested on CNN (0.76) or IGN (0.74)than on Breitbart itself (0.78). We further note that bag-of-words classifiers do not generalize as well to other commu-nities (mean AUC=0.58).

Discussion & ConclusionThis paper presents a data-driven study of antisocial behav-ior in online discussion communities by analyzing users thatare eventually banned from a community. This leads to acharacterization of antisocial users and to an investigation ofthe evolution of their behavior and of community response:users that will eventually be banned not only write worseposts over time, but the community becomes less tolerant ofthem. Next, it proposes a typology of antisocial users basedon post deletion rates. Finally, introduces a system for iden-tifying undesired users early on in their community life.

By using explicit signals of undesirability (i.e., perma-nent banning), we are able to study users engaged in awide variety of antisocial behavior. While scalable, our ap-porach has several limitations. A more fine-grained label-ing of users (perhaps through crowdsourcing), may reveal agreater range of behavior. Similarly, covert instances of an-tisocial behavior (e.g., through deception) might be signif-

User lifetime:

Results: Cross-Domain

94

ROC AUC

0.84 0.74 0.76

0.69 0.83 0.74

0.74 0.75 0.78

Trained on

Test

ed o

n

Conclusion§ Web is a social place!

But people are less inhibited online

§ Important to understand interaction mechanisms and behavioral patterns

§ Towards healthier and more welcoming online communities

95

SummaryLarge-scale longitudinal analysis of antisocial behavior in discussion forums

Combination of data analysis and human experiments

Trolls are not “special” people§ Mood and discussion context

increase the probability of trolling96

SummaryBad behavior can spread§ The effects of negative feedback

are more pronounced§ Write worse and more§ Are evaluated worse by others§ Evaluate others worse

§ Positively-evaluated users, on the other hand, don’t do any better

97

Summary§ Prediction task: Predicting whether a

user will be banned in the future

§ After observing just 10 posts, we can accurately predict whether the user will be banned in the future

§ Generalization performance across communities

98

Further Questions§ Social feedback loops: Are trolls

socially created by harsh feedback?§ Reputation: Effects of user reputation§ Role of moderators: Are moderators

effectively helping and moderating§ Linguistic norms: Linguistic analysis§ Feedback mechanism: Communities

with only positive feedback (likes)99

References§ How Community Feedback Shapes User Behavior

J. Cheng, C. Danescu-Niculescu-Mizil, J. Leskovec. AAAI International Conference on Weblogs and Social Media (ICWSM), 2014.

§ Antisocial Behavior in Online Discussion Communities by J. Cheng, C. Danescu-Niculescu-Mizil, J. Leskovec. AAAI International Conference on Weblogs and Social Media (ICWSM), 2015.

101

can anyone become an internet troll? - web science · 2016-06-07 · complementing observational...

Documents