twitter trend detection and analysis

Trend Detection and Analysison Twitter

Benjamin RäthleinHenning MuszynskiLukas Masuch

2

Agenda

Motivation

Architecture

Data Preparation

Trend Analysis

Analyzed Trends

Conclusion

3

Motivation

Predict the stock market in real timesource

source

Detecting influenza epidemics

Automatic crime predictionsource

“Successful results of mainly research-based projects

helped to open up new business opportunities”

http://www.fastcoexist.com/1681873/twitter-can-predict-the-stock-market-if-youre-reading-the-right-tweets

http://static.googleusercontent.com/media/research.google.com/en/archive/papers/detecting-influenza-epidemics.pdfDE83tYmC53DGQ&bvm=bv.86475890,d.bGQ

http://ptl.sys.virginia.edu/ptl/sites/default/files/sbp2012_wgb.pdf

4

Twitter

5

Early Trend Detector

Bag-of-words (Hashtags, Mentions)

Twitter Streaming API (Twython)

Architecture

Bag of Words

Bags Count

#newyear 7

#christmas 6

@bigdata 2

@sap 3

6

Statistical MeasurementEarly Trend Detector



Architecture

Statistical Measurement(growth, average usage, retweets, participating users…)

Report statistics (every 20 minutes):• Total hashtags & user mentions• Hashtag/mentions count• Usage growth per hashtag/mention• Participating users per hashtag/mention• Retweet count per hashtag/mention

7




Architecture


Anomaly Detection

Time Series Analysis

Calculated for every hashtag / user mentionEvery 2 / 4 hours based on reports

Anomaly detection using:• Relative & absolute fluctuation• Total occurrences (sum)• Minimum occurrences• Maximum occurrences• Average occurrences


8


Architecture

Trend Analyzer

Text Preprocessing (Python NLTK)

Lowercasing & tokenizing

URL & stopword removal

Stop Word Removal

This sample text shows which words willbe removed when applying stop word removal. Mostly words like the, a or and.

This sample text shows which words will be removed when applying stop word removal. Mostly words like the, a or and.

9


Architecture

Trend Analyzer




Word stemming

Stemming

Amazing

Amazement

Amazed

amaze

10


Architecture

Trend Analyzer




Word stemming

Sentiment Analysis

Sentiment Analysis

I love cookiesI hate cookies

11


Architecture

Trend Analyzer




Word stemming

Sentiment Analysis

Topic Modeling (LDA)

Topic Modeling

Topics• …• …• …

Trend Classification

14

Trend Analyzer




Word stemming

Sentiment Analysis

Topic Modeling (LDA)

Wordcloud Visualization

Wordfreq.js

Wordcloud2.js

GeoSpatial Visualization

CartoDB



Anomaly Detection



Trend Classification


Architecture

15

Analyzed Trends

16

Limitations

Tweets collected: 38 million (70GB)

Only English tweets from the USA

Twitter Streaming API

17

New Year

Time Series

18

New Year

Word Cloud

19

New Year

Geospatial Analysis

Midnight Los Angeles Midnight New York

20

New Year

Sentiment Analysis

Positive Neutral Negative

Home sick on #nye. Horrible timing

stupid cold. Ugh. My date is my

couch & pillow watching.

#HappyNewYear everyone.

#HappyNewYear from the Youth for

Astronomy and Engineering Program

at Space Telescope Science Institute!

Happy New Year! Last year was

amazing, and here’s to another great

year of love & happiness! #NYE2015

21

Air Asia Tragedy

22

Air Asia Tragedy

Time Series

23

Air Asia Tragedy

Word Cloud

24

Air Asia Tragedy

Topic Modeling

Newsairasia, missing, flight, air, Indonesia, singapore, asia

Search for the Planeairasia, missing, plane, find, plane, world, technology

SympathyPrayers, families, thoughts, airasia, crash, thought, airfrance

Causeairasia, weather, flight, pilots, fly, bad, path

International Helpraaf, butterworth, china, australia, Russia, trndnl, trending

25

Air Asia Tragedy

Sentiment Analysis

Neutral Negative Positive

Prayers are USELESS! Stop repeating

meaningless crap, pretending that

you care … #PrayForAirAsia #QZ8501

#GrowABrain #ReligousNonsense

#BREAKING #AirAsia Flight #8501

likely “at the bottom of the sea”

rescue officials says.

May God’s great love shine on the

families and loved ones of all

passengers and crew #AirAsia #8501

26

Air Asia Tragedy

Google Trends Comparison

Google Trends Twitter Sample

27

Air Asia Tragedy

Google Trends Comparison

Google Trends Twitter Sample

28

Sony Hack

29

Sony Hack

Time Series

30

Sony Hack

Word Cloud

31

Sony Hack

Topic Modeling

Christmas Releasetheinterview, christmas, day, theaters, freedom, theater, showing

Reviewstheinterview, jamesfrancotv, sethrogen, movie, interview, funny, hilarious

Suspicionsnorthkorea, sonyhack, korea, north, internet, sony, amp

Newstheinterview, sonypictures, sony, movie, korea, north, interview

Insider Joketheinterview, aint, hate, cuz, jealous, anus, peanutbutter

32

Sony Hack

Geospatial Analysis

33

Sony Hack

Sentiment Analysis


#TheInterview SUCKS!!! @sethrogen

Like I knew it would #Stupid

#NotFunny

#Sony says #TheInterview made

more than $1 million at the box office

on in 1 single day on Dec. 25.

Happy I joined my fellow Americans

in the great #TheInterview Christmas

Day Viewing. Plus it was pretty funny,

truth be told.

34

Network Outage

35

Network Outage

Time Series

36

Network Outage

Word Cloud

37

Network Outage

Topic Modeling

Network Errorxbox, psn, sign, connect, live, error, account, issues

Connection between Hacksxbox, playstation, watch, movie, fuckcrucifix, north, korea, interview

Xbox Downxbox, christmas, play, xboxlivedown, live, xboxlive, xboxsupport, day

Caused Damageplaystation, dollar, psn, company, lizardsquad, sony, billion, multi

Hacker Groupfuckcrucifix, lizardmafia, lizardsquad, fuck,lizard, squad, finestsquad, stop

Restoredpsn, back, playstation, online, askplaystation, network, psndown, working

38

Network Outage

Sentiment Analysis


@XboxSupport f*** your servers, a

big ass company like you should

handle these teenage kids, terrible

@AskPlayStation when will the

service be back online because it says

there’s maintenance?

@PlayStation thanks for the great

year. I am sure this new year will be

amazing. Don’t allow yourselves to

be hacked ever again.

39

Conclusion

High quality insights into world’s interest

Twitter is very good for detecting and predicting trends

Maintaining a high data quality is important

40

#Questions

Benjamin Räthlein@B3nRa

Henning Muszynski@henningmus

Lukas Masuch@LukasMasuch

twitter trend detection and analysis

Technology