Statistical properties of network community structure

Jure Leskovec, CMUKevin Lang, Anirban Dasgupta and Michael MahoneyYahoo! Research

Network communities

Communities: Sets of nodes with lots

of connections inside and few to outside (the rest of the network)

Assumption: Networks are

(hierarchically) composed of communities (modules)

Communities, clusters, groups,

modules

Community score (quality)

How community like is a set of nodes?

Want a measure that corresponds to intuition

Conductance (normalized cut):Φ(S) = # edges cut / # edges inside

Small Φ(S) corresponds to more community-like sets of nodes

Community score (quality)

Score: Φ(S) = # edges cut / # edges inside

Community score (quality)

Score: Φ(S) = # edges cut / # edges inside

Bad communit

yΦ=5/7 = 0.7

Community score (quality)

Score: Φ(S) = # edges cut / # edges inside

Better communit

y

Φ=5/7 = 0.7

Bad communit

y

Φ=2/5 = 0.4

Community score (quality)

Score: Φ(S) = # edges cut / # edges inside

Better communit

y

Φ=5/7 = 0.7

Bad communit

y

Φ=2/5 = 0.4

Best communit

yΦ=2/8 = 0.25

Network Community Profile Plot We define:

Network community profile (NCP) plotPlot the score of best community of size k

Search over all subsets of size k and find best: Φ(k=5) = 0.25

NCP plot is intractable to compute Use approximation algorithm

NCP plot: Small Social Network

Dolphin social network Two communities of dolphins

NCP plotNetwork

NCP plot: Zachary’s karate club

Zachary’s university karate club social network During the study club split into 2 The split (squares vs. circles) corresponds to cut B

NCP plotNetwork

NCP plot: Network Science

Collaborations between scientists in Networks

NCP plotNetwork

Geometric and Hierarchical graphs

Hierarchical network

Geometric (grid-like) network – Small social

networks– Geometric and– Hierarchical networkhave downward NCP plot

Our work: Large networks

Previously researchers examined community structure of small networks (~100 nodes)

We examined more than 70 different large social and information networks

Large real-world networks look

completely different!

Example of our findings

Typical example:General relativity collaboration network (4,158 nodes, 13,422 edges)

NCP: LiveJournal (N=5M, E=42M)

Better and better

communities

Worse and worse

communities

Best community

has 100 nodes

Com

mu

nit

y s

core

Community size

Explanation: Downward part

Whiskers are responsible for downward slope of NCP plot

Whisker is a set of nodes connected to the network by a single

edge

NCP plot

Largest whisker

Explanation: Upward part

Each new edge inside the community costs more

NCP plot

Φ=2/1 = 2

Φ=8/3 = 2.6

Φ=64/11 = 5.8

Each node has twice as many

children

Suggested Network Structure

Network structure: Core-

periphery, jellyfish, octopus

Whiskers are responsible for

good communities

Denser and denser core

of the network

Core contains 60%

node and 80% edges

Caveat: Bag of whiskers

What if we allow cuts that give disconnected communities?

Cut all whiskers and compose

communities out of them

Caveat: Bag of whiskersC

om

mu

nit

y s

core

Community size

We get better community scores when

composing disconnected sets of

whiskers

Connected communitie

sBag of whiskers

Comparison to a rewired network

Rewired network: random

network with same degree distribution

What is a good model?

What is a good model that explains such network structure?

None of the existing models work

Pref. attachment Small World Geometric Pref. Attachment

FlatDown and Flat

Flat and Down

Forest Fire model works

Forest Fire: connections spread like a fire New node joins the network Selects a seed node Connects to some of its neighbors Continue recursively

As community grows it

blends into the core of

the network

Forest Fire NCP plot

rewired

network

Bag of whisk

ers

Conclusion and connections Whiskers:

Largest whisker has ~100 nodes Independent of network size Dunbar number: a person can maintain social

relationship to 150 people Bond vs. identity communites

Core: Newman et al. analyzed 400k node product network▪ Largest community has 50% nodes▪ Community was labeled “miscelaneous”

Conclusion and connections

NCP plot is a way to analyze network community structure

Our results agree with previous work on small networks

But large networks are fundamentally different

Large networks have core-periphery structure Small well isolated communities blend into the

core of the networks as they grow