spectral analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf ·...
TRANSCRIPT
![Page 1: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/1.jpg)
SPECTRAL ANALYSIS OF REAL
WORLD NETWORKS By :-
Anshuman Tripathi (07CS3024)
Gautam Kumar (07CS1021)
Parin Chheda (07CS3023)
(under guidance of Animesh Shrivastav and Prof Niloy Ganguly)
![Page 2: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/2.jpg)
PROJECT GOAL
Collect Network data from real world networks
like World Wide Web, Facebook, Twitter … etc
Compute spectral properties of the graphs
Laplace spectrum
Adjacency spectrum
Degree distribution
Assortativity … etc
Study these spectral properties under certain
type of network attacks to conclude resilience of
these networks
![Page 3: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/3.jpg)
REAL – WORLD NETWORKS
Autonomous System Graph
Every AS router is viewed as a node in the graph
A trace route from a router to another router denotes
an edge
Every individual is a node
Friendship denotes an undirected edge
Followers (who follow „x‟) and Friends (who „x‟
follows) define directed edges adjacent to „x‟
![Page 4: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/4.jpg)
WORK FLOW
Collect Network
Data Prune Network Data
Perform Node
removal Attacks
Compute Spectral
Properties
Plot/Compare
Results
Conclude on
resilience of these
networks
![Page 5: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/5.jpg)
COLLECTING DATA (AS)
The network data for AS router network was
downloaded from
http://snap.stanford.edu/data/as-skitter.html
The Data organized in for of edge-list
Undirected Graph
Statistics :-
Number of nodes (|V|) 1696415 ~ 1.7M
Number of Edges (|E|) 11095298 ~ 11.1M
Highest Degree 1008
Assortativity 0.04
Clustering Coef. 0.2963
![Page 6: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/6.jpg)
COLLECTING DATA (FACEBOOK &
TWITTER)
Designed python based crawlers
Used cloudlight python module
The friend list dynamically fetched from Facebook server
Used mobile version of Facebook ( http://m.facebook.com ) to
browse friends (10 friends per page)
Crawled ~2000 nodes in 3 days
OAuth2 authentication
Used Twitter API for python (twython)
(https://github.com/ryanmcgrath/twython )
Crawling limited by number of api-calls per hour from a
client (350 calls/hour)
Crawled ~1900 nodes in 1 day
![Page 7: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/7.jpg)
COLLECTING DATA (FACEBOOK &
TWITTER)
Facebook data downloaded from
Twitter data downloaded from
Statistics
Facebook Twitter
Number of nodes
(|V|)
258912 ~ 2.5M 40103281 ~ 40M
Number of Edges
(|E|)
60022032 ~ 60M 1468365182 ~ 1.5B
diameter 6.5 5.9
![Page 8: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/8.jpg)
PRUNING OF NETWORKS
Data collected too huge for performing spectral
computations
Entire data is not necessary for studying
statistical properties
Prune the data obtained w.r.t degree of node
Selecting Threshold
Should conserve the degree distribution of the
original network
Should reduce number of nodes to computationally
feasible levels ~ 10K
![Page 9: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/9.jpg)
STATISTICS OF PRUNED NETWORKS Metric AS Facebook Twitter
Number of
nodes
9881 ~ 10K 10707 ~ 10K 1030869 ~ 1M
Number of
edges
403474 ~ 403K 328926 ~ 329K 55921630 ~ 55M
Threshold >175 >800 >100 and < 500
Assortativity 0.0398 0.3589 N/A2
Clustering
Coef.
0.3095 0.3143 N/A2
Diameter1 9 13 10
Size of Big
Component
99.78% 99.75% 99.99%
Number of
components
11 7 4
1 Diameter of the big component 2 unable to compute => graph too big
![Page 10: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/10.jpg)
PRUNING (AS)
Threshold = 175
![Page 11: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/11.jpg)
DEGREE DISTRIBUTION: AS (LOG-SCALE)
𝑙𝑜𝑔𝑒 𝑁𝑘
𝑙𝑜𝑔𝑒 𝑘≈ −0.645
![Page 12: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/12.jpg)
PRUNING (FACEBOOK)
Threshold = 800
![Page 13: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/13.jpg)
DEGREE DISTRIBUTION: FACEBOOK (LOG-
SCALE)
𝑙𝑜𝑔𝑒 𝑁𝑘
𝑙𝑜𝑔𝑒 𝑘≈ −1.2
![Page 14: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/14.jpg)
PRUNING (TWITTER)
Threshold = 100 to 500 (out-degree)
![Page 15: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/15.jpg)
DEGREE DISTRIBUTION: TWITTER (LOG-
SCALE)
Non linear curve
![Page 16: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/16.jpg)
SPECTRAL ANALYSIS (LAPLACE SPECTRUM)
AS
![Page 17: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/17.jpg)
SPECTRAL ANALYSIS (LAPLACE SPECTRUM)
![Page 18: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/18.jpg)
ADJACENCY SPECTRUM
AS
![Page 19: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/19.jpg)
ADJACENCY SPECTRUM
![Page 20: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/20.jpg)
NODE REMOVAL
Node removal: Top k node removed based on four
metrics
Random node removal ( „rand‟ attack)
Degree based ( nodes with high degree centrality)
(„deg‟ attack)
Based on betweeness centrality („bet‟ attack)
Based on closeness centrality („load‟ attack)
Sort the nodes based on a particular centrality
and remove Top „k‟ nodes : size of attack = k
![Page 21: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/21.jpg)
NODE REMOVAL ON AS (LAPLACE
SPECTRUM)
Bet Attack Deg Attack
Load Attack Rand Attack
![Page 22: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/22.jpg)
NODE REMOVAL AS (ASSORTATIVITY)
![Page 23: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/23.jpg)
NODE REMOVAL ON FACEBOOK
![Page 24: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/24.jpg)
NODE REMOVAL FACEBOOK (SIZE OF BIG
COMPONENTS)
![Page 25: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/25.jpg)
BIMODAL NETWORK SIMULATION
Bimodal networks are networks in which a node can have either low degree or high degree (super nodes)
Bimodal network simulation
Simulation done using a C code(courtesy Animesh Srivastav)
Variation of Assortativity with random node removal
Statistics of simulated Bimodal Network generated:
Low degree 5
High degree 20
Prob. Of low degree 0.8
Assortativity of network 0.5
![Page 26: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/26.jpg)
BIMODAL NETWORK SIMULATION
(ASSORTATIVITY VS. NODE REMOVAL)
Number of iterations = 10
![Page 27: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/27.jpg)
POST-MIDSEM
![Page 28: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/28.jpg)
ADJACENCY BINNED SPECTRUM
(FACEBOOK)
![Page 29: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/29.jpg)
ADJACENCY BINNED SPECTRUM (AS)
![Page 30: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/30.jpg)
LAPLACE BINNED SPECTRUM (FACEBOOK)
![Page 31: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/31.jpg)
LAPLACE BINNED SPECTRUM (AS)
![Page 32: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/32.jpg)
LAPLACE BINNED SPECTRUM OF
ATTACKED NETWORK (AS)
BET
LOAD
DEG
RAND
![Page 33: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/33.jpg)
DEDUCTIONS
From the attacks done on AS and Facebook
Network is clearly visible that attacks based on
load and betweenness centrality behave in same
way
The trend of change in assortativity is different
for AS and Facebook network.
Correlation between various centralities in AS
and Facebook ?
![Page 34: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/34.jpg)
CO-RELATION
AS (bet-deg) Facebook (bet-deg)
AS (bet-load) Facebook (bet-load)
𝜎𝑥𝑦 =
𝜎𝑥𝑦 = 𝜎𝑥𝑦 =
𝜎𝑥𝑦 =
![Page 35: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/35.jpg)
CO-RELATION
AS (bet-clust) Facebook (bet-clust)
AS (deg-load) Facebook(deg-load)
𝜎𝑥𝑦 = 𝜎𝑥𝑦 =
𝜎𝑥𝑦 = 𝜎𝑥𝑦 =
![Page 36: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/36.jpg)
CO-RELATION
AS (deg-clust)
AS (load-clust)
(deg-clust)
(load-clust)
𝜎𝑥𝑦 = 𝜎𝑥𝑦 =
𝜎𝑥𝑦 = 𝜎𝑥𝑦 =
![Page 37: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/37.jpg)
PEARSON CO-RELATION MATRIX
BET DEG LOAD CLUST
BET 1.0 0.3179 0.8307 -0.0225
DEG 1.0 0.3204 -0.0079
LOAD 1.0 -0.0229
CLUST 1.0
BET DEG LOAD CLUST
BET 1.0 0.1073 0.9641 -0.0052
DEG 1.0 0.1067 -0.0027
LOAD 1.0 -0.0052
CLUST 1.0
AS network
Facebook network
![Page 38: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/38.jpg)
CORRELATION (AS VS FACEBOOK)
AS network has considerably higher correlation
between betweenness and load centrality and
degree centrality
Means that nodes with high degree have higher
betweenness and closeness (typical of a router
network)
Facebook has correlation between the load,
betweeness centrality and degree centrality but it
is lower than AS networks
![Page 39: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/39.jpg)
CORRELATION (CONTD…)
In social network context degree does not dictate
the closeness of a node from other nodes.
In both cases Load centrality and Betweenness
centrality are highly correlated, more so in the
case of Facebook.
In both the cases, negligible negetive correlation
with clustering coefficients
![Page 40: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/40.jpg)
FUTURE WORKS
Perform the experiments on twitter dataset
Perform clustering coefficient based node
removal
Study the effect of attacks on network diameter
Compare the results obtained for the three data
sets
Simulate experiments with bimodal networks
![Page 41: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/41.jpg)
REFERENCES
[1]. S. N. Dorogovtsev, A. V. Goltsev, J. F. F. Mendes, and A. N. Samukhin, “Spectra of complex networks,” Phys. Rev. E, vol. 68, no. 4, p. 046109, Oct 2003.
[2]. E. Estrada, Spectral Theory of Networks: From Biomolecular to Ecological Systems., Jun 2009.
[3]. A. N. Samukhin, S. N. Dorogovtsev, and J. F. F. Mendes, “Laplacian spectra of complex networks and random walks on them: Are scale-free architectures really important?” Jun 2007.
[4]. http://snap.stanford.edu/data/as-skitter.html (Internet Data)
[5]. Kwak, Haewoon and Lee, Changhyun and Park, Hosung and Moon, Sue. “What is Twitter, a Social Network or a News Media?”. http://an.kaist.ac.kr/traces/WWW2010.html (Twitter data set)
[6]. On the Evolution of User Interaction in Facebook. (Facebook Data Set)
![Page 42: Spectral Analysis of real world networkscse.iitkgp.ac.in/~anshumant/reports/cnt.pdf · 2013-01-20 · REAL – WORLD NETWORKS Autonomous System Graph Every AS router is viewed as](https://reader034.vdocument.in/reader034/viewer/2022042304/5ecf946d0378236368511b56/html5/thumbnails/42.jpg)