Enquiring Minds: Early Detection of Rumors in Social Media from Enquiry Posts

Zhe Zhao Paul Resnick Qiaozhu Mei

PRESENTATION GROUP 2

Outline

Introduction Background Study Approach For Detection Experimental Setup Evaluation Conclusion

WHAT IS RUMOR?

Rumor is a controversial, fact-checkable statement

Rumor is a controversial, fact-checkable statement

Malaysia airline MH370 is missing

Malaysia airline MH370 crashed

Rumor is a controversial, fact-checkable statement

Malaysia airline MH370 is missing

Malaysia airline MH370 crashed

Recreational Marijuana should be made legal

Recreational Marijuana becomes legal in Michigan

Introduction

It is very difficult to claim that every post on social media is a factual claim

The broad success of online social media has created fertile soil for the emergence and fast spread of rumors.

This paper proposes an automated tool to identify potential Rumors

Spread of Rumor

Oh my god is this real? RT @AP: Breaking: Two Explosions in the White House and Barack Obama is injured

Is this true? Or hacked account? RT @AP Breaking: Two Explosions in the White House and Barack Obama is injured

Is this real or hacked? RT @AP: Breaking: Two Explosions in the White House and Barack Obama is injured

Is this legit? RT @AP Breaking: Two Explosions in the White House and Barack Obama is injured

Detecting Rumor

Rumors are basically judge on the key phrases it has – “Is this true?”

“Really?”

“What?

The paper proposes algorithm for identifying newly emerging, controversial topics that is scalable to massive stream of tweets i.e. signal tweets

Then it identifies a set of regular expressions that define the set of signal tweets

Related Work

Detection Problems in Social Media!

The work on detecting rumor has started in recent years.

Sharing/ Retweeting / Trending determines it’s a rumor or not.

Question Asking in Social Media

Another detection feature used in related work is question asking. Mendoza et al. found on a small set of cases that false tweets were questioned much more than confirmed truths.

Detection using question mark!

Previous work has shown that only one third of tweets with question marks are real questions, and not all questions are related to rumors.

Problem Statement

Rumor Cluster We define a rumor cluster R as a group of social media posts that are either declaring, questioning, or denying the same fact claim, s, which may be true or false. Let S be the set of posts declaring s, E be the set of posts questioning s, and C be the set of tweets denying s, then R = S ∪ E ∪ C. We say s is a candidate rumor if S ≠ ∅ and E ∪ C ≠ ∅.

The paper’s objective is to minimize the delay from the time when the first tweet about the rumor is posted to the detection time.

RUMOR

Fact Checka

ble

Controversial

Approach for Detection

Detection of rumors

1) Identify Signal Tweets

2) Identify Signal Clusters

3) Detect Statements

4) Capture Non-signal Tweets

5) Rank Candidate Rumor Cluster

Identify Signal Rumor

If we want to detect rumors, the first thing we should know is what rumors look like.

Author defines rumors as a verification of a piece of factual knowledge, i.e. “According to the Mayan Calendar, does the world end on Dec 16th, 2013?”.

Or as corrections (debunks) of a question. i.e. “This new is true!”

What we need is more than theory

Using Porter Stemmer and Chi-Squared algorithm on 10417 tweets, with 3423 tweets labeled as verification or correction, and we draw the pattern of good signals.

Identify Signal Clusters

What is Signal Cluster?

After a rumor tweet emerges, people might follow, i.e. retweet it or come up with a new one containing similar information, thus forming a group or cluster.

What? An eight year girl died at Boston marathon explosion.

Is it true? Two explosions in the White house and Barack Obama is injured!

The shocking new is tested be to wrong!

How do we do it?

Use connected component clustering algorithm, Jaccard Similarity algorithm and Minhash algorithm to achieve it.

What??!! Two Explosions in the White House and Barack Obama is Injured in head.Is it true?? Two Explosions in the White House and Barack Obama is Injured on arm.Really?? @AP: Two Explosions in the White House and Barack Obama is not Injured.

Detect Statement

Right now what we get is a few clusters of potential rumors, not sure about the content.

Our goal is the rumor content, not the pattern.

Which one to draw out?

A way out

Just pick out the statement that appears more often than 80% of other statements.

Why 80? Have higher probability to be a rumor!

What??!! Two Explosions in the White House and Barack Obama is InjuredIs it true?? Two Explosions in the White House and Barack Obama is InjuredReally?? @AP: Two Explosions in the White House and Barack Obama is Injured

Compare Non-Signal Tweets

Remember when we detect rumor clusters, using signals.

Tweets not belong to verification or correction, but also can bear rumor information.

Match those statements with non-signal tweets.

Also using Jaccard similarity. If the score > 0.6, we can say they matched.

Rank candidate rumor clusters

Till now, in network, we have got several rumor clusters.

Each cluster stands for one rumor statement.

But output should be one, the most potential rumor.

Popularity? NO! i.e. funny picture or touching

video.

Ranking rumor clusterPercentage of signal tweets

Entropy ratio

Tweet lengths

Retweets

URLs

Hashtags

@ Mentions

Experimental Setup

Data Sets

BOSTON MARATHON BOMBING (high-profile event) Two bombs exploded at the finish line of the annual Boston Marathon competition on April 15th, 2013 which contains 30,340,218 unique tweets.

GARDENHOSE (random sample) Collected a tweet stream in a random month of the year 2013 (November 1 to November 30, 2013) which contains 1,242,186,946 tweets.

Baselines and Variants of Methods

1. Trending Topics

2. Hash tag Tracking

3. Corrections Only

4. Enquiries and Corrections

Rank candidate rumors purely by popularity, the number of tweets in the cluster.(identify signal tweets)

5. SVM ranking

6. Decision tree ranking

Use both enquiry and correction tweets as signals.(rank the candidate rumor clusters)

Effectiveness of Enquiry Signals

Precision of Candidate Rumor Clusters

Precision of rumor detection using different signals. Candidate rumors ranked by popularity only. Maximum number of output rumor clusters: 10 per hour for BOSTON and 50 per day for GARDENHOSE.

Effectiveness of Enquiry Signals

Earliness of Detection

Earliness of detection comparing to Enquiries+ Corrections: enquiry signals help to detect rumors hours

Ranking Candidate Rumor Clusters

@N is the percentage of real rumors among the top N candidate rumor clusters output by the a method.

Precision@N of different ranking methods

Effectiveness of Enquiry Signals

In order to verify that the ranking algorithm is not overfitting only one data set, We also applied the decision tree trained using 7 days of labeled results in GARDENHOSE data set to rank rumor clusters detected hourly from BOSTON data set.

Precision@N if rumor clusters are ranked by the Decision Tree. One third of top 50clusters are real rumors.

Efficiency of Framework

Filtering of tweets

Clustering

Potential rumor statements

The cost is significantly reduced as compared to approach which first generates trending topics and then identify rumors.

Time Comparison

Trending Topics:

Clustering

Hashtag Tracking:

Filtering & Clustering

This Method:

Filtering, Clustering then

retrieving back Same clustering and ranking implementation was used except filtering tweets with enquiry and tweets were not retrieved back after clustering.

Tracking Rumor Using Enquiry Method

Tracking detected rumors about Boston Marathon Bombing

Conclusion

Method which capitalizes on verification questions which also appear sooner facilitating early detection.

Cluster only those tweets that contain enquiry patterns, extract the statements and use them to pull back in the rest of the non-signal tweets.

Robust even with tweets exceeding 100 million.

Future work-

• Signal labelled by humans to have iterative improvements

• Improving the filtering of enquiry and correction signal by training a classifier

Questions ?