An Extensible Graph Toolkit for Mathematica

Eytan Bakshy

SI 708

Introduction & Motivation

The Extensible Graph Toolkit for Mathematica, or EGraph, is a framework for studying networks. The toolkit is a lightweight, extensible object frame-work that poses significant advantages over the two standard methods of representing graphs in Mathematica: rule lists and Combinatorica graphs. Thistutorial demonstrates the usage of the kit in the analysis and visualization of static and evolving networks.

Graph Programming with Rule Lists

Rule-lists can be used with some of Mathematica's more modern graph utilities such as adaptive algorithms for computing all-pairs shortest paths, pseudo-diameters, PageRank, and finding strong and weakly connected components. The objects in rule lists can easily be plotted using GraphPlot[]. Since therules can contain arbitrary objects, text or graphics objects are automatically rendered inline as nodes.

Here are a few examples of how one constructs a graph as a rule list, finds its weakly connected components, and use GraphPlot to plot data

<<GraphUtilities`

ruleGraph = 81®"Î","Î"®3,4® 5,"a"®"Î","a"® 3,6® 7,7®1,7® "Ì","Ì"®1<;WeakComponents@ruleGraphDGraphPlot@ruleGraphDGraphPlot@ruleGraph,VertexLabeling® TrueD881, Î, 3, a, 6, 7, Ì<, 84, 5<<

1Î

3

45

a6

7

Ì

Pitfalls with Combinatorica Graph objects and the EGraph Structure

Combinatorica is a package that covers many areas of discrete mathematics, including graph theory. It has a generic Graph object, which allows users toadd labels, edge weights, and progressively consruct graphs. It also has a number of algorithms to compute basic graph measures. Unfortunately, thestructure is not well optimized for large graphs. In this section, we will compare the performance characteristics of the Combinatorica Graph structurewith the EGraph structure.

<<Combinatorica`

In the following series of experiments, it is shown that time to compute a vertex's nearest neighbor scales linearly with the number of nodes in the graph (inthis case, a random graph), using Mathematica's built-in graph structure. The EGraph structure on the other hand uses a double-hash structure so that theseoperations can be carried out in constant time.

GraphTiming = Table@8n,First�Timing@Neighborhood@ð,1,1DD<&@RandomGraph@n,2�nDD,8n,100,1000,100EGraphTiming = Table@8n,First�Timing@AllNeighbors@ð,1DD<&@RandomEGraph@n,2�nDD,8n,100,1000,100 ;

ListPlot@8GraphTiming,EGraphTiming< ,AxesOrigin® 80,0<,PlotLabel ® "Neighborhood access time vs number of nodes", AxesLabel® 8"num nodes","access time D

200 400 600 800 1000num nodes

0.005

0.010

0.015

0.020

access time

Neighborhood access time vs number of nodes

The difference in performance is even more pronounced when one compares Combinatorica's shortest path algorithm to the one used by EGraph.

H* gives the time to find the shortest path length for random nodes, averaged over 50 trials *LAveragePathLengthTiming@graph_,numNodes_,PathFunction_D:=

Mean@Table@First�Timing@PathFunction@graph,RandomInteger@81,numNodes<D,RandomInteger@81,numNodes ;

GraphTiming = Table@8n, AveragePathLengthTiming@RandomGraph@n,2�nD,n,ShortestPathD<,8n,100,1000EGraphTiming = Table@8n, AveragePathLengthTiming@RandomEGraph@n,2�nD,n,PathLengthD<,8n,100,1000 ;

ListPlot@8GraphTiming,EGraphTiming< ,AxesOrigin® 80,0<,PlotLabel ® "Shortest path calculation time vs number of nodes", AxesLabel® 8"num nodes","{ time D

200 400 600 800 1000num nodes

0.1

0.2

0.3

0.4

{ time

Shortest path calculation time vs number of nodes

2 EGraph Paper.nb

Getting Started with the EGraph Object

The EGraph structure consists of a double-hash sparse matrix representation of the graph, with forward and backwards index of vertex labels. These labelsmay be of any data format. In most circumstances, the labels will either be integers or strings, but the toolkit supports labels of any datatype. Finally, theEGraph object contains an attribute store, which can hold arbitrary data about the graph, including individual node properties.

An EGraph object can be instantiated from a list of vertex label pairs, or rules, or, as a sparse adjacency matrix

ToEGraph@880,1<,81,2<,82,3<,80,2<<DToEGraph@80®1,1®2,2®3,0®2<DToEGraph@SparseArray@8i_,j_<¦ If@RandomReal@1.0D<0.1,1,0D,84,4<DDEGraph of <<4 nodes>>

EGraph of <<4 nodes>>

EGraph of <<4 nodes>>

EGraph structures are capable of dealing with non-numerical data as well. Consider the network of trade partners in the UN

importLinks=Select@Flatten@Thread@Rule@CountryData@ð,"ImportPartners"D,ðDD&��CountryData@"UN"DexportLinks=Select@Flatten@Thread@Rule@ð,CountryData@ð,"ExportPartners"DDD&��CountryData@"UN"DtradeData = Join@importLinks,exportLinksD;Short@tradeDataDtradeGraph = ToEGraph@tradeDataD8Pakistan ® Afghanistan, UnitedStates ® Afghanistan,

�2048�, Zimbabwe ® Italy, Zimbabwe ® Germany<EGraph of <<198 nodes>>

The graph structure itself is smaller than the base data, because vertex labels are only stored in the backwards and forwards index hash. For largernetworks, like the Yahoo! QA network, this difference can be quite significant.

ByteCount@tradeDataDByteCount@tradeGraphD187584

48160

Labels can easily be extracted from the graph and used to label the output of network metrics

Short@VertexLabels@tradeGraphDDShort@PageRanks@tradeGraphDD8Afghanistan, Albania, Algeria, AmericanSamoa,Andorra, �188�, Venezuela, Vietnam, Yemen, Zambia, Zimbabwe<

8Afghanistan ® 0.00411284, Albania ® 0.00305773, �195�, Zimbabwe ® 0.00245865<

The EGraph structure can easily be converted into other commonly used representations, like matiricies and ordered pairs

EGraph Paper.nb 3

MatrixPlot@GraphMatrix@tradeGraphDDEGraphToPairs@tradeGraphD��Short

1 50 100 150 198

1

50

100

150

198

1 50 100 150 198

1

50

100

150

198

88Afghanistan, UnitedStates<, 8Afghanistan, Pakistan<,�1908�, 8Zimbabwe, Italy<, 8Zimbabwe, Germany<<

Here is an example of how the following data can be modified to visualize the rankings of countries based on prestige. (mouse over to see country)

ListPlot@Sort@PageRanks@tradeGraphD�.HRule@country_,pr_D¦ Tooltip@pr,countryDLD,AxesLabel® 8"Ordinal Rank","PageRank"<D

50 100 150 200Ordinal Rank

0.004

0.006

0.008

0.010

PageRank

Analysis of Dynamic Network Data

We will demonstrate various ways to analyze the dynamics of network models

The Diameter of an Evolving Graph

The first example shows how one can perform experiments with NetLogo and the EGraph structure to study properties of evolving networks. In thisexample, we use the modified preferential attachment model from class.

<<NetLogo`NLStart@"�Applications�NetLogo 4.0"D;NLLoadModel@"�Users�ebakshy�netlearn�RandAndPrefNetEB.nlogo"D;

To perform the experiment, we grow the graph to 5000 nodes and sample its structure every 100 model ticks. Because the behavior of the model mightdiffer from run to run, we perform 20 realizations of the model. The resulting network is stored in BAG[step, realization #] (warning, the simulation takesa considerable amount of time to run)

4 EGraph Paper.nb

To perform the experiment, we grow the graph to 5000 nodes and sample its structure every 100 model ticks. Because the behavior of the model mightdiffer from run to run, we perform 20 realizations of the model. The resulting network is stored in BAG[step, realization #] (warning, the simulation takesa considerable amount of time to run)

Do@NLCommand@"set m 2","setup"D;Do@

NLCommand@"repeat 100 @goD"D;BAG@i,jD=ToEGraph@NLGetGraph@D,Directed® FalseD;,8i,50<D,8j,20<D;

Here is how one would plot a simple statistic, like the degree distribution

ListLogLogPlot@Cumdist@OutDegrees@BAG@50,1DDD,AxesLabel ® 8"Degree","Count"<,PlotLabel® "Degree D

2 5 10 20 50 100Degree

5

10

50

100

500

1000

Count

Degree Distribution

To plot the effective diameter of the graph over time, we plot the average the results at each time step

AvgDiameter@i_D:=Mean@Table@First�First�PseudoDiameter@BAG@i,jDD,8j,20<DD;ListLinePlot@Table@8NumNodes@BAG@i,1DD, AvgDiameter@iD<,8i,50<D,AxesLabel® 8"Num Nodes","Pseudo D

1000 2000 3000 4000 5000Num Nodes

6.0

6.5

7.0

7.5

8.0

Pseudo Diameter

The Spread of Opinions in a Network

In this example, we will generate a network and observe the diffusion of opinions on the network. Then, the structure of the network will be used togenerate a two dimensional representation of agent influence over time.

NLLoadModel@"�Users�ebakshy�netlearn�PrefDiffusionEBOpinion.nlogo"D;NLCommand@"set num-nodes 300","set gamma 0.3","set p 0.14","set sorting? true","generate-topology

NLCommand@"setup"D;opGraph = ToEGraph@NLGetGraph@D,Directed® FalseD;H* this part is ugly but will be more straight forward in future versions *LopSeries = Apply@Rule,NLDoReport@"repeat 2 @spreadD","@Hlist who pro?LD of turtles", 200D,82<DagentStates = Map@N�VertexLabels@opGraphD �. ð &,opSeriesD¨;SetAttribute@opGraph,"Opinions",agentStatesD;

One way of tracking people's opinions over time is to represent each agent as a row in an ArrayPlot, with each column representing the agent's opinion attime t.

EGraph Paper.nb 5

One way of tracking people's opinions over time is to represent each agent as a row in an ArrayPlot, with each column representing the agent's opinion attime t.

OpinionPlot@opinions_D:=ArrayPlot@opinions �.8True® Red,False® Lighter@Pink,0.5D<,AspectRatio ;

Here are the opinions of all agents over time, sorted in random order. In this case, the spatial ordering of agents in no way reflects the structure ofinteraction among agents.

OpinionPlot@GetAttribute@opGraph,"Opinions"DD

Here is what the network looks like, using a ball and stick plot and a matrix plot

6 EGraph Paper.nb

GraphPlot@GraphMatrix@opGraphDDMatrixPlot@GraphMatrix@opGraphDD

1 100 200 304

1

100

200

304

1 100 200 304

1

100

200

304

We can compute the shortest path between all pairs of nodes and use hierarchical clustering to resolve community structure

<<HierarchicalClustering`

EGraph Paper.nb 7

gdm = GraphDistanceMatrix@GraphMatrix@opGraphDD;gdOrdering = ClusterFlatten@DirectAgglomerate@gdmDD;GraphicsRow@8

MatrixPlot@gdm, PlotLabel® "Vertex Distance Matrix HunorderedL"D,MatrixPlot@gdm@@gdOrdering,gdOrderingDD,PlotLabel® "Vertex Distance Matrix HclusteredL"D<D

1 100 200 304

1

100

200

304

1 100 200 3041

100

200

304

Vertex Distance Matrix HunorderedL

1 100 200 304

1

100

200

304

1 100 200 3041

100

200

304

Vertex Distance Matrix HclusteredL

This new ordering can be used to visualize agent states over time in a way in which agents' spatial proximity is related to their structural proximity in thenetwork

OpinionPlot@GetAttribute@opGraph,"Opinions"D@@gdOrderingDDD

Network Visualization

Fisher-Smith Egonet Plots

The kit has built-in support for "Fisher" plots, which have been used in our work on Yahoo! Answers. Here is one example using previous data set

8 EGraph Paper.nb

FisherSample@tradeGraph,20D

Using Attribute Stores for Enhancing Network Visualizations

This part is a bit experimental, but is a good demonstration of how the attribute object can be used to enhance network visualizations

importLinks=Select@Flatten@Thread@Rule@CountryData@ð,"ImportPartners"D,ðDD&��CountryData@"MiddleEastexportLinks=Select@Flatten@Thread@Rule@ð,CountryData@ð,"ExportPartners"DDD&��CountryData@"MiddleEastmidEastGraph = ToEGraph@Join@importLinks,exportLinksDD;

We assign each node a shape attribute

SetAttribute@midEastGraph,"shape",Tooltip@CountryData@ð,"Shape"D,ðD& �� VertexLabels@midEastGraph ;

EGraph Paper.nb 9

GraphPlot@GraphMatrix@midEastGraphD, VertexRenderingFunction®HInset@GetAttribute@tradeGraph, "shape"D@@ð2DD, ð1, Automatic, 80.5, 0.5<D &LD

10 EGraph Paper.nb

Supporting Code

<< GraphUtilities`;

Format@EGraph@sa_SparseArray,labels_, labelRules_, attributes_DD :=

"EGraph of <<" <> ToString@Length@labelsDD <> " nodes" <> ">>";GraphMatrix@g_EGraphD := g@@1DD;VertexLabels@g_EGraphD := g@@2DD;VertexLabelToIndex@g_EGraphD := g@@3DD;AttributeObject@g_EGraphD := g@@4DD;H*

PairIntersect@pairs,baseListDis a fast algorithm for reducing a list of ordered pairs to a base setreturns 8Ha,bL È Ha,bLÎ pairs ì a Î baseList ì b Î baseList<

*LPairIntersect@pairs_, baseList_D :=

Module@8membershipRules<,membershipRules = Dispatch@Thread@Rule@baseList, TrueDDD;DeleteCases@Pick@pairs, pairs �. membershipRules, 8True, True<D, 8<DD;

Options@ToEGraphD = 8VertexLabeling ® True, Directed ® True<;Clear@ToEGraphDToEGraph@g : 8Rule@_, _D ..<, opts : OptionsPattern@DD := ToEGraph@List ��� g, optsD;ToEGraph@g : 88_, _< ..<, opts : OptionsPattern@DD := Module@8allVertexLabels, labelToIndex,

gTally, storedVertexLabels, storedLabelToIndex, arrayRules, graphMatrix<,allVertexLabels = Union@Flatten@gDD;labelToIndex = Dispatch@MapIndexed@Rule@ð1, First@ð2DD &, allVertexLabelsDD;gTally = Tally@g �. labelToIndexD;storedVertexLabels = If@OptionValue@VertexLabelingD === True,allVertexLabels, Range@Length@allVertexLabelsDDD;

storedLabelToIndex = If@OptionValue@VertexLabelingD === True, labelToIndex, Rule@i_, iDD;arrayRules = gTally �. 88in_Integer, out_Integer<, count_Integer< ¦ Rule@8in, out<, countD;If@OptionValue@DirectedD === False,arrayRules = Join@arrayRules, arrayRules �. HRule@8i_, j_<, c_D ® Rule@8j, i<, cDLDD;graphMatrix = SparseArray@arrayRules,8Length@allVertexLabelsD, Length@allVertexLabelsD<, 0D;EGraph@graphMatrix, storedVertexLabels, storedLabelToIndex, Unique@attributeObjectDDD;

ToEGraph@m_SparseArray, opts : OptionsPattern@DD := ModuleA8allVertexLabels, labelToIndex,

gTally, storedVertexLabels, storedLabelToIndex, arrayRules, graphMatrix<,allVertexLabels = Range@Length@mDD;labelToIndex = Hi_ ® iL;If@OptionValue@DirectedD === False,arrayRules = Join@arrayRules, arrayRules �. HRule@8i_, j_<, c_D ® Rule@8j, i<, cDLDD;graphMatrix = MapAIf@ð > 0, 1, 0D &, m + m¨, 82<E; H* make symmetric *LEGraph@graphMatrix, allVertexLabels, labelToIndex, Unique@attributeObjectDD

E;EGraphToPairs@G_EGraphD := First �� Most@ArrayRules@GraphMatrix@GDDD �.8a_Integer, b_Integer< ¦ VertexLabels@GD@@8a, b<DD;Clear@RandomEGraphDRandomEGraph@n_, p_D := Module@8m<,

ToEGraph@SparseArray@88i_, j_< ¦ If@RandomReal@1.0D < p, 1, 0D �; i < j<, 8n, n<DDD;H* sets arbitrary attribute *L

EGraph Paper.nb 11

H* sets arbitrary attribute *LSetAttribute@G_EGraph, attr_String, data_D := HAttributeObject@GD@attrD = data;L;H* sets node attributes by encoding data by vertex label ® data rules *LSetAttribute@G_EGraph, attr_String, data : 8List@_, _D ..<, opts : OptionsPattern@DD :=

Module@8dataDispatch = Dispatch@Rule ��� dataD<,SetAttribute@G, attr, Table@vertex �. dataDispatch, 8vertex, VertexLabels@GD<DD;D;

GetAttribute@G_EGraph, attr_StringD := AttributeObject@GD@attrD;NumNodes@G_EGraphD := Length@GraphMatrix@GDD;InDegrees@G_EGraphD := Total@GraphMatrix@GDD;OutDegrees@G_EGraphD := Total �� GraphMatrix@GD;

OutNeighbors@G_EGraph, i_IntegerD :=

Cases@ArrayRules@GraphMatrix@GD@@iDDD, Rule@8node_Integer<, val_IntegerD ¦ node �; val > 0D;InNeighbors@G_EGraph, i_IntegerD := Cases@ArrayRules@GraphMatrix@GD@@All, iDDD,

Rule@8node_Integer<, val_IntegerD ¦ node �; val > 0D;OutNeighbors@G_EGraph, i_D := VertexLabels@GD@@OutNeighbors@G, i �. VertexLabelToIndex@GDDDD;InNeighbors@G_EGraph, i_D := VertexLabels@GD@@InNeighbors@G, i �. VertexLabelToIndex@GDDDD;AllNeighbors@G_EGraph, i_D := Join@InNeighbors@G, iD, OutNeighbors@G, iDD;AllNeighbors2@G_EGraph, i_D := Union@Flatten@AllNeighbors@G, ðD & �� AllNeighbors@G, iDDD;H* ordered pairs of node i and its in or out neighbors *LInNeighborPairs@G_EGraph, i_D := Thread@8InNeighbors@G, iD, i<D;OutNeighborPairs@G_EGraph, i_D := Thread@8i, OutNeighbors@G, iD<D;AllNeighborPairs@G_EGraph, i_D := Join@OutNeighborPairs@G, iD, InNeighborPairs@G, iDD;H* all second nearest neighbors *LAllNeighbors2Pairs@G_EGraph, i_D := Module@8myNeighbors<,

myNeighbors = AllNeighbors@G, iD;Union@Join@AllNeighborPairs@G, iD,Flatten@Map@AllNeighborPairs@G, ðD &, myNeighborsD, 1DDDD;

H* egonet is all nearest neighbors of i and the links between all nearest neighbors*LEgoNetPairs@G_EGraph, i_D :=

Join@AllNeighborPairs@G, iD, PairIntersect@AllNeighbors2Pairs@G, iD, AllNeighbors@G, iDDD;AllNeighbors2Graph@G_EGraph, i_D :=

ToEGraph@AllNeighbors2Pairs@G, iD �. 8a_Integer, b_Integer< ¦ VertexLabels@GD@@8a, b<DDD;EgoNetGraph@G_EGraph, i_D := ToEGraph@

EgoNetPairs@G, iD �. 8a_Integer, b_Integer< ¦ VertexLabels@GD@@8a, b<DDD;Unprotect@StrongComponents, WeakComponents, PseudoDiameter,PageRankVector, CommunityStructureAssignment, CommunityStructurePartition,CommunityModularity, PathLengthMatrix, PathLengthMatrixD;

StrongComponents@G_EGraphD := StrongComponents@GraphMatrix@GDD;PageRankVector@G_EGraph, opts___D := PageRankVector@GraphMatrix@GD, optsD;PageRanks@G_EGraph, opts___D := Thread@VertexLabels@GD ® PageRankVector@G, optsDD;WeakComponents@G_EGraphD := WeakComponents@GraphMatrix@GDD;PseudoDiameter@G_EGraph, opts___D := PseudoDiameter@GraphMatrix@GD, optsD;CommunityStructureAssignment@G_EGraph, opts___D :=

CommunityStructureAssignment@EGraphToPairs@GDD;CommunityStructurePartition@G_EGraph, opts___D :=

CommunityStructurePartition@EGraphToPairs@GDD;CommunityModularity@G_EGraph, part_List, opts___D :=

CommunityModularity@EGraphToPairs@GD, part, optsD;PathLengthMatrix@G_EGraphD := GraphDistanceMatrix@GraphMatrix@GDD;PathLength@G_EGraph, i_Integer, j_IntegerD := GraphDistance@GraphMatrix@GD, i, jD;

12 EGraph Paper.nb

PathLength@G_EGraph, i_Integer, j_IntegerD := GraphDistance@GraphMatrix@GD, i, jD;PathLength@G_EGraph, i_, j_D :=

GraphDistance@GraphMatrix@GD, i �. VertexLabelToIndex@GD, j �. VertexLabelToIndex@GDD;PathLength@G_EGraph, 8i_, j_<D := PathLength@G, i, jD;Protect@StrongComponents, WeakComponents, PseudoDiameter,PageRankVector, CommunityStructureAssignment, CommunityStructurePartition,CommunityModularity, GraphDistanceMatrix, GraphDistanceD;

H* cumulative binning for doing log plots *LCumdist@l_D := Module@8valueTally<,H*sorted in descending order by value*L

valueTally = Reverse@SortBy@Tally@lD, FirstDD;H*pair each value with the ð of values less than itself*LTranspose@8HFirst �� valueTallyL, Accumulate@Last �� valueTallyD<DD;

Clear@FisherSampleD;RandomizePairs@pairs_D := Module@8randList<,randList = Map@Rule@ð, RandomReal@DD &, Union@Flatten@pairsDDD;pairs �. randListD

FisherSample@G_EGraph, n_, opts___D := Module@8theGraph<,theGraph = Flatten@Table@Rule ���

RandomizePairs@EgoNetPairs@G, RandomInteger@81, Length@VertexLabels@GDD<DDD,8n<DD;GraphPlot@theGraph, DirectedEdges ® True, EdgeRenderingFunction®H8HBlackL, Arrowheads@0.01D, Arrow@ð1, 0.01D< &L, VertexLabeling ® False, optsDD;

EGraph Paper.nb 13