the very small world of the well-connected xiaolin shi, matt bonner, lada adamic, anna gilbert
DESCRIPTION
Network or Hairball? Huge networks difficult to study, store, share.. Can we shrink or summarize a network? Starting point: important vertices Vertex-Importance Graph SynopsisTRANSCRIPT
The Very Small World
of theWell-Connected
Xiaolin Shi, Matt Bonner, Lada Adamic, Anna Gilbert
Outline VIGS: Vertex-Importance Graph Synopsis
Testing VIGS with different datasets and importance measures
Analytical expectations
Making guarantees about VIGS
Connectedness: KeepOne, KeepAll
Related Work
Graph Sampling, Rich Club, K-cores, Web Measure
Network or
Hairball?
Huge networks difficult to study, store, share..
Can we shrink or summarize a network? Starting point: important vertices
Vertex-Importance Graph Synopsis
Vertex-Importance Graph Synopsis
Create subgraph of important vertices
Study both key nodes and entire graph
Which vertices are important? High-traffic routers? The most quoted blog?
Standard, well-defined measures Degree, Betweenness, Closeness, PageRank
VIGS In Action• Starting point: random graph with 100 vertices• Select an importance measure - Degree• pick 9 highest degree vertices• keep only edges between these 9 vertices
average degree = 4 average degree = 0.9
Motivating example: citations among ACM
papers
500 random papers 500 most cited papers
Datasets Erdos-Renyi random graph and three real networks BuddyZoo - collection of buddy lists TREC - links between blogs Web - an older web crawl from PARC
Erdos-Renyi BuddyZoo TREC Web
Vertices 10,000 135,131 29,690 152,171
Edges 49,935 803,200 195,940 1,686,541
ASP 4.26 5.96 3.72 3.48
Directed false false true true
Importance measures degree (number
of connections) denoted by size
betweenness (number of shortest paths a vertex lies on) denoted by color
Importance measures degree (number
of connections) denoted by size
closeness (length of shortest path to all others) denoted by color
High correlation between different importance measurements
Undirected graphs - higher correlation Closeness has lowest correlation in all datasets
Correlation among measures
High correlation between different importance measurements Undirected graphs – higher orrelation Closeness has lowest correlation in all datasets
Correlation among measures
Assortativity In an assortative graph, high-value nodes
tend to connect to other high-value nodes Example: degree
assortative disassortative
Assortativity - Degree
• ER: Neutral
• BZ: Assortative
• TREC and Web: Disassortative
Assortativity
Degree distributions
Subgraphs
Apply VIGS! Select Degree, top 100 nodes Example: degree Substantial difference between datasets!
Subgraphs
The selection of an importance measure may have an impact, even in the same dataset
Connectivity: size of largest component
Proportion of nodes that are connected either directly or indirectly
Subgraph Connectivity - ER
• Highly connected, even with only a few vertices
• All importance measures almost completely connected by 2000 nodes
• Better performance than random
Subgraph Connectivity
subgraphs: density
average degree = 4 average degree = 0.9
What is the proportion of edges to nodes in the original graphs vs. subgraphs?
Subgraph Density - ER
• Black line slope = Edges/Vertices in entire network
• Lower dotted line = subgraph of random vertices
• VIGS subgraphs: lower than total density, higher than random subgraph density
Subgraph Density
Average Shortest Path‘ASP’
whole network ASP
ASP between IV’s in subgraph.
ASP between IV’s in whole graph
ER ASP shorter between IV’s, but higher in subgraph
Subgraph Average Shortest Path
‘ASP’ for Erdos Renyi
Subgraph ASP’s
Relative Rank of Vertices in Subgraph - ER
• Do IV’s maintain their relative rank in subgraphs?
• IV and edges only• ER - little correlation,
steadily increasing until all vertices are included
Relative Rank in Subgraph
TREC anomaly - closeness
Four Regions Four regions, highlighted in density plot:
OriginalCloseness only, Regions highlighted
Cause: Blog Aggregator One node has connections to 99% of the
nodes between 1 and 7961! (regions 1, 2, 3) This same node has only 1 connection to a
node beyond 7961 (region 4) Nodes between 5828 and 7961 (region 3)
have only 1 connection: to the aggregator Spam blogs? New blogs? Private blogs?
Examining Density
The first 3 regions feature nodes connected to the aggregator
R1: well connected blogs Average increase in total edges
per node added: 12.93 R2: far less connected, but
not quite barren Average increase per node: 3.2
R3: isolated spam/new blogs 1 edge per node increase
Examining Density
R4: well connected, but not linked to aggregator
Average increase even higher than region 1: 17.8
Aggregator inflated the closeness scores of connected nodes (R1, 2, 3) above those in region 4
Examining Avg Shortest Paths (ASP)
R1: ASP slightly below 2 Some nodes directly connected,
99%+ within 2 hops via aggregator R2 and 3: ASP levels at ~2
Fewer and fewer direct links, but all accessible via aggregator
R4: ASP’s begin to increase ASP doesn’t explode: ~70% of R4
links are to R1 or R2 nodes R3 only reachable from R4 via agr. Access to aggregator through
connected R1/R2 nodes: adds a hop to path
Examining Relative Ranking Correlation
R1-3: correlation steadily decreases
R4: rapid increase in correlation!
Spam blogs importance in subgraph initially inflated
Realigns when blogs in 4 connect with real blogs in 1-2
Localized to closeness Region 1, 2 and 3 nodes have high closeness
thanks to the aggregator Recall ASP graph - short distance to many, many
nodes via aggr. Connection to aggregator doesn’t confer high
degree, PageRank or Betweenness - nodes must ‘fend for themselves’ Degree: link to aggr. Is just 1 link. PR: aggr. ‘vote’ diluted by high degree Bet: Aggr. Is gateway to its children, could use any
child to reach aggr.
• VIGS results vary by graph and importance measure
• Still, subgraphs tended towards– High connectivity– Average or higher density– Shorter ASP’s– Maintain relative importance rank of vertices
– “spam” affects closeness primarily
Empirical Analysis Summary
Preserving Properties So far, just studying subgraphs Applying VIGS - may need guarantees Hard to make a guarantee?
Example property: subgraph is connected
Preserving Properties
Preserving Properties Is it difficult to guarantee the connectedness
of a VIGS subgraph? NP-complete: reducible to Steiner Minimum
Spanning Tree (MST) problem Resort to heuristics
KeepOne, KeepAll from Gilbert and Levchenko (2004)
KeepOne and KeepAll KeepOne - build an MST: drop as many vertices/edges as
possible while maintaining connectivity. Problem! ASP/diameter could increase
Solution: KeepAll - MST, but add all vertices/edges on a shortest path
Heuristic Performance - ER
• KO - did not have to add many vertices, but shortest path rather large (ER ASP was 4.26)
• KA - good improvement in path length, but huge increase in vertices
ASP
Heuristic Performance - BZ
• Similar performance to ER - KO results in significantly longer shortest paths, but KA adds many vertices
• Is 4000 too many vertices to add? Small compared to total graph, but huge compared to number of important vertices
ASP
Heuristic Performance - TREC
• Almost completely connected from the start
• KA adds only a few vertices, doesn’t change much
• Results for Web dataset similar
ASP
Related Work Graph sampling - Similar objective: synopsis
Concerned only with original graph Random sampling, snowball sampling… Lee, Kim, Jeong (2006), Leskovec, Faloutsos (2006), Li, Church, Hastie (2006)
Rich-club Concerned only with high degree nodes Zhou, Mondragon (2004), Colizza, Flammini, Serrano, Vespignani (2006)
Related Work K-cores
Subgraphs where each vertex has at least k-connections within the subgraph
Dorogovstev, Goltsev, Mendes (2006) Core connectivity
Smallest number of important vertices to remove before destroying largest component
Mislove, Marcon, Gummadi, Druschel, Bhattacharjee (2007)
VIGS wrap up vertex-importance graph synopsis
create a subgraph of important vertices to study both the full graph and these vertices in particular
properties of VIGS depend on entire network and importance measure
real world networks have dense, closely knit VIGS
in some cases easy to meet connectivity & ASP guarantees
Thanks to Xiaolin Shi
Matthew Bonner
Lada Adamic
NSF DMS 0547744