challenges and opportunities posed by power laws in network analysis

Challenges and Opportunities Posed by Power Laws in Network Analysis

Bruno RibeiroUMass Amherst

MURI REVIEW MEETINGBerkeley, 26th Oct 2011

2

Power Laws in Networks

→ Network topology:– power law distribution of node degrees

• AS topology, social networks (Facebook, etc)

→ Network traffic:

– Flow: subset of packets– Power law distribution of flow sizes

routerpacket stream

vertex degree - d

P[de

g >

d ] Flickr dataset

3

Characterizing Networks from Incomplete Data

This talk

→ Estimate distributions (of degrees, of flow sizes, …) from incomplete data (sampled edges, sampled packets, …)

→ Uncover central nodes in the network

4

Outline

→ Challenge: Estimating subset size distributions from incomplete data– Incomplete data:

• randomly sampled edges, randomly sampled packets, …– Impact of power laws on estimation accuracy– Impact of other distributions on estimation accuracy

→ Opportunity: Uncovering central nodes in power law networks

5

ESTIMATING SUBSET SIZE DISTRIBUTIONS FROM INCOMPLETE DATA

Part 1: Challenge

6

Subset size distributionsSet of fishes

Number of fishes(subset size)

types of fish (subsets)

distribution

x - subset size (number of fishes)

fractionof subsets (types of fish)with size x

7

Estimating subset size distributions

Set of fishes

randomlysample Nfishes(uniformly)

distribution

x - subset size (number of fishes)

fractionof subsets (types of fish)with size x

unbiasedestimate

sampledfishes

8

Questions

How many fishes need to catch toobtain accurate distribution estimates?

What is impact of distribution shapeon estimation accuracy?

10

Incomplete Data Estimation

randomsampling

estimation

IP flow size distribution

set of IP packets

Sampled packets

flow

11

→ Distribution of # incoming links to a webpage– Q: do we need to crawl most of web graph?

→ Incoming links observed as outgoing links from other webpages– set = set of links– subset = incoming links to a webpage– sampling: link sampling

Network-related subset sizedistributions (webgraph)

?in-degree:# of links to webpage

outgoing links

12

→ Distribution number of packets in a TCP flow

– Set = IP packets– Subset = a IP flow– Sampling: packet sampling

Network-related subset sizedistributions (IP traffic)

routerpacket stream

13

Incomplete Data, Edge Sampling Example

Original graph

Sampled in-degrees

3x Estimator

OriginalIn-Degree Distribution

14

Incomplete data model→ Set elements sampled with probability p– without replacement– independently

→ Model– : probability that j out

of i subset elements are sampled– i : fraction of subsets with i elements• e.g.: fraction of nodes with degree i, fraction of flows

with i packets

15

Model (cont)→ bij – binomial(i,j)→ i : fraction of subsets with i elements→ W : maximum subset size

→ : fraction of subsets with j sampled

elements– d0 is not observable

16

Mean Squared Error Question

→ i : unbiased estimate of of i

→ p : sampling probability→ N : sampled subsets (e.g. N sampled flows)

Exists an unbiased estimator that has small mean squared error: MSE(i)?

Try Maximum Likelihood Estimator (MLE)?

17

Maximum Likelihood Estimation→ Simulation: edge sampling→ Flickr network (photo-sharing), 1.5M nodes

in-degree

i

18

Cramer-Rao Lower Bound (CRLB)→ Let B = [bij] , d = [dj] , = [i]– Then

d = B→ D = diag(d) : diagonal matrix Djj = dj

→ i : unbiased estimate of of i

→ J : Fisher information matrix of N subsets– J = BT D B– lower bound Mean Squared Error of i :

MSE(i) (J-1)ii/NNeed to find J-1

19

Recap→ Interested in the inverse of Fisher information

matrix becauseMSE(i) (J-1)ii/N

→ N : # of subsets sampled (# of nodes, # of TCP flows)

→ : subset size distribution estimate (what we seek)

→ p : sampling probability (edges, packets)

→ W : maximum subset size

20

Results

21

Heavier than exponential subset size distribution tail

→ Theorem 1: Suppose that W decreases more slowly than exponential. More precisely assume –log(W) = o(W) error grows

with subset size W

22

Exponential subset size distribution tail

→ Theorem 2: Suppose that W decreases exponentially in W. More precisely assume -log(W) = W log a + o(W) as W ∞ for some 0 < a < 1

23

Lighter than exponential subset size distribution tail

→ Theorem 3: Suppose that W decreases faster than exponentially in W. More precisely assume -log(W) = 𝜔(W). Then it follows that

0 < p ≤ 1

24

Infinite support & power laws

→ If is power law with infinite support (W ∞)– if p < any unbiased estimator has ½“infinite” MSE• might as well output random estimates

– if p > estimates can be accurate if ½enough samples are collected

25

Estimating Subset Size Average

→ I : randomly chosen subset size→ Average subset size E[I]:

– E[I] ≤ ∞ & E[I2] = ∞ then estimation error is unbounded• Reason: inspection paradox• Sampling biased towards very large subsets

– Average size of sampled subsets E[I2]/2E[I]

– otherwise, error is bounded

26

IMPACT OF POWER LAWS ON SAMPLING CENTRAL NETWORK NODES

Part 2: Opportunity

27

→ Central nodes important in networks– Communication bottlenecks, trend setters, information aggregators

→ Notions of centrality. – betweenness, closeness, PageRank, degree

Challenge: identify top k central nodes exploring small fraction of network

Central Nodes

central nodes

28

Degree as a proxy for centrality→ Betweenness centrality: node is central if it belongs to many

shortest paths→ Closeness centrality: node is

central if has short paths to all other nodes

→ Rank correlation measures the degree of similarity between two rankings

→ Low rank correlation inplanar graphs (e.g. power grid)Set Type of Network # of nodes # of edges Description

AS-Snapshot Computer 22,963 48,436 Snapshot of Internet at level of ASca-CondMat Collaboration 23,133 186,936 ArXiv Condense Matter

ca-HepPh Collaboration 12,008 237,010 ArXiv High Energy Physicsemail-Enron Social 36,692 367,662 Email network from Enron

Rank correlation with Degree

29

Random walk in steady state visits node with probability proportional to node degree

In power law graphs such bias towards high degree nodes is strong

We observe that RWs more efficient than more evolved techniques (AXS, RXS)

Looking for high degree nodes

% of network sampled

30

Thank you