{ trends in social network m. tech project presentation by : pranay agarwal 2008cs50220 guides :...
TRANSCRIPT
![Page 1: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/1.jpg)
{
Trends in Social Network
M. Tech Project Presentation
By :Pranay Agarwal2008CS50220
Guides :Amitabha BagchiMaya Ramanath
![Page 2: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/2.jpg)
Introduction Why twitter ? Filter model for tweets Social Graph construction Evolving graphs for topics Results and Conclusion Future work
Outline
![Page 3: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/3.jpg)
Introduction : Trends
• 2.4 B Online
• Facebook 1.2 B• Twitter 200 M [1]
Age old query :
“what are people talking about” ?
![Page 4: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/4.jpg)
Search Engines
News Media Online
Contents E-Commerce
![Page 5: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/5.jpg)
200 Millions users, 400 Millions tweets everyday. [2]
All major news sources, Govt. offices etc. Fast and real time Data exposed via Twitter API.
Why Twitter
![Page 6: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/6.jpg)
Needle in Haystack
![Page 7: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/7.jpg)
Tweet based Meta data NLP tags
Word count Retweet count NERs
Mentions count Followers count POS
Hash tags count URLs count
- Source (mobile, web)
S(t) = Σ ( Wi ∗ Fi(t) ) S (t) = score of tweet tWi = Weight of feature IFi(t) = feature i value for tweet t
Model Features
Filter Model for tweets
![Page 8: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/8.jpg)
Too short tweets (<= 2 words) are always “chat” tweets.
40 % of the “chat” tweets had one of the stop words while only 2% of the “informative” tweets had it.
Stop words = (I, me, mine, you, yours etc.)
Two Phase model
![Page 9: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/9.jpg)
Goal : “Enhance Timesense (Yahoo! proprietary) capabilities using twitter.”
Tasks : Marcelo Filter model Trend prediction using Social Graph.
Yahoo! Proposal
![Page 10: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/10.jpg)
Collecting data from Twitter API. Storing and processing Social Graph. Topic wise clusters of tweets.
Spatio Temporal Analysis of Topic Popularity in Twitter
Evolving graph for a topicThe hypothesis was that during evolution of graph, its structure and topology gives rise to patterns, which could act as distinct features to distinguish “trending” topic from “non-trending” topic.
![Page 11: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/11.jpg)
Wrote a python library to communicate with the API.
Collecting friends and followers relations Several instances of nodes making calls to Twitter
API under normal rate limit.[3] Frequent outages in the API service causing
further delay and blocking. Resolve first for “Good” users, who will be
involved in creating and sharing of “informative” tweets.
Twitter API
![Page 12: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/12.jpg)
Node : Each node is a user, which also contains several other details of user profiles. We also label users for which we have resolved all the relations.
Edge : An directed edge edge(u → v) represents user “u” follows user “v” stored in Adjacency List.
Index : We need to make several queries where given a user details we want to get it’s all followers and friends. To make this query fast and efficient we indexed the graph by a unique key “uid” as user id of all user nodes. This “uid” is same as the twitter user id, which is already present in the tweet object.
Graph Database Neo4j [4]
![Page 13: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/13.jpg)
30 Millions nodes and 60 Millions edges.
Our graph is only 10% of the whole twitter graph.
Validation Almost 95% of the top celebs present Around 60% of the users of the second
set present in our graph
This is a very strong indication that our graph mostly contains “active” and “good” users while there could be significant fraction of twitter users as “inactive”
![Page 14: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/14.jpg)
“Timesense”, a Yahoo! Proprietary service, which gives list of topic search queries along with “buzz” score which indicates the “trendiness”
The search queries returned by Timesense are not clustered together, which means different search queries related to same event is given as different queries
Tweets Clustering
mark appel houston mlb draft mark appel and pat appel mark appel 2013 mlb draft mark appel contract what high school did mark appel go to mark appel major stanfordmark appel stanford baseball baseball player mark appel of stanford
![Page 15: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/15.jpg)
Users in twitter use different variants of the same topic
We implemented a Bi-gram matching algorithm to cluster together search queries like these.
One pass of all the public tweets and fetch the tweet if it contains any of the bi-gram terms pair in it.
N gram Matching
![Page 16: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/16.jpg)
“Topics that are going to become very popular witness intense discussion within communities at first. When the level of intensity rises then the users who bridge communities enter the discussion in a big way causing a merging of what were earlier disjoint discussions.”
Evolving Graphs
![Page 17: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/17.jpg)
The vertex set of Gt0 comprises the users V0 who tweet about t on window 0 (the edge set is empty)
The vertex set Vit of Gti is the set of all users who have tweeted on a topic in windows 0 through i
An edge(u ← v) is added to Eit if u ∈ V (Current set) and v has tweeted about t on window i
Window = 30 Minutes.
Algorithm
![Page 18: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/18.jpg)
Topic no.
Trend Type
Topic
1 High IRS Scandal of Obama administration
2 High Angelina Jolie going through mastectomy
3 Low Cannes Film Festival 2013
4 Low The Great Gatsby (Movie 2013)
Experiments
![Page 19: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/19.jpg)
High Trend : First row, Low Trend : Second Row
![Page 20: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/20.jpg)
High Trends
![Page 21: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/21.jpg)
Low Trends
![Page 22: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/22.jpg)
Largest component size increases for all the topics. But the increase in the size is much more significant in case of topic 1 and 2
Topics 1 and 2 contain most of its nodes in the largest low ratio c1/c2 for topics 2 and 3 shows that there are
many small independent clusters of communities discussing among themselves without leading to a large component component.
Topic C1 C1/C2
1 1732 346
2 974 44
3 127 8
4 22 3
![Page 23: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/23.jpg)
Users who bridge communities enter the discussion
Bridge users serve as a barometer of the topics rising popularity
External edges = {(u → v) : u ∈ S, v ∈ V \ S}Total Edges = |{(u → v) : u ∈ S}|
φ(S) = External Edges / Total edges
Conductance
![Page 24: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/24.jpg)
Resolve Missing Edges
Topic No.
Nature Topic
1 Very Low Mattapoisett Car accident
2 Mild Jeep Patriot new model car
3 Trend Mayor Bloomberg
4 High trend Mark Appel
• Collecting tweets clusters for these topics using above Algorithm
• Resolving all relations for all the authors in these tweets.
![Page 25: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/25.jpg)
First row : low, Second row: high Trend
![Page 26: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/26.jpg)
Low Trends
![Page 27: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/27.jpg)
High Trends
![Page 28: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/28.jpg)
Table shows the higher conversion of external to internal edges, in case of trending topics, which means more behavior influence and spreading to followers in case of trends.
Largest connected component contains around 15 % and 35 % of all users in case of topic 1 and 2 respectively, while in case of topic 3 it is 80 % and > 90% in case of topic 4.This strongly supports the hypothesis that in case of trending topics, users form large connected community.
Topic E/N L/N %
Fall in conductance
1 0.09 8.7 0.3
2 0.38 36.7 0.1
3 0.83 82.1 0.5
4 0.98 97.98 1.2
![Page 29: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/29.jpg)
Resolve all relations ? May be NOT..
Limitations
0.6 Drop
0.1Drop
![Page 30: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/30.jpg)
Identify good “sensors” or users Resolve as many relations possible Better topic detection and clustering
from tweets. Efficient Graph processing data
Structure Inter relation of topics
Future Work
![Page 31: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/31.jpg)
Thanks
![Page 32: { Trends in Social Network M. Tech Project Presentation By : Pranay Agarwal 2008CS50220 Guides : Amitabha Bagchi Maya Ramanath](https://reader036.vdocument.in/reader036/viewer/2022062409/56649c895503460f94942cde/html5/thumbnails/32.jpg)
[1]http://www.internetworldstats.com/stats.htm
[2] Twitter official blog [3] Twitter api. 2013. [4] neo4j http://www.neo4j.org. 2013. S. Ardon, A. Bagchi, A. Mahanti, A.
Ruhela, A. Seth, R. M. Tripathy, and S. Triukose. Spatio-temporal analysis of topic popularity in twitter. CoRR, abs/1111.2904, 2011.
References