a pattern-aware graph mining system

A Pattern-Aware Graph Mining SystemKasra Jamshidi Rakesh Mahadasa Keval Vora

Simon Fraser University

https://github.com/pdclab/peregrine

Why should you pay attention?

executes 700x fasterPeregrine

consumes 100x less memoryPeregrine

scales to 100x larger datasetsPeregrine

On 8x fewer machines

With a more expressive API

a

db

a

bd

Data Graph

e

c

gf

Graph Mining

a

db

7

a

db

a

bd

Graph Mining

e

c

gf

Data Graph

a

db

8

a

db

a

bd

Graph Mining

e

c

gf

Data Graph

a

db

Subgraph

9

a

db

a

bd

Graph Mining

e

c

gf

Data Graph Pattern

10

Graph Mining

a

db

a

bd

Edge-Induced

11

Graph Mining

Edge-Induced

a

db

a

bd

12

Graph Mining

a

db

a

bd

Edge-Induced

13

Graph Mining

a

db

a

bd

Vertex-Induced

14

Graph Mining

a dba b d

Vertex-Induced

15

Graph Mining

a

db

a

bd

Vertex-Induced

16

Graph Mining

a

db

e

c

gf

Data Graph

17

Graph Mining

a

db

e

c

gf

Data Graph

a

db

a

bd

Vertex-Induced

18

Graph Mining

a

db

e

c

gf

Data Graph

a

db

a

b d

a

db

a

b d

a

db

a

b d

a

db

a

b d

Edge-Induced

a

db

a

bd

Vertex-Induced

19

Frequent Patterns(Edge-Induced)

Graph Mining

a

db

e

c

gf

Data Graph

20

Unlabeled Pattern Distribution(Vertex-Induced)

2 1 0

1 02

Graph Mining

a

db

e

c

gf

Data Graph

21

Scalability Challenge

22

• 4-motif counting on Orkut graph (|V| = 13M, |E| = 117M)


23


• 4-motif counting on Orkut graph (|V| = 13M, |E| = 117M)

123,503,340,341,270 subgraphs

24

System Requirements

25

System Requirements

Uniqueness

x

zy

x

zy

x

zy

26

System Requirements

Uniqueness

x

zy xz

y

x

z

y

27

System Requirements

Uniqueness

x

zy xz

y

x

z

y

28

System Requirements

Uniqueness

Structurex

zy

29

System Requirements

Uniqueness

Structurex

zy

30

System Requirements

Uniqueness

Structure

Interestingness

31

System Requirements

Uniqueness

Structure

Interestingness

32

System Requirements

Uniqueness

Structure

Interestingness

33

System Requirements

Uniqueness

Structure

Interestingness

34

Existing Work

Uniqueness

Structure

Interestingness

Arabesque (SOSP ‘15)RStream (OSDI ‘18)Fractal (SIGMOD ‘19)AutoMine (SOSP ‘19)

35

Existing Work

Uniqueness

Structure

Interestingness


Overlook user requirements

36

Existing Work

Uniqueness

Structure

Interestingness


Overlook user requirementsPer-subgraph computations

37

Pattern Awareness

Pattern Selection

Pattern Matching

39

Pattern Awareness

Pattern Selection

Pattern Matching

40

Pattern Awareness

Pattern Selection

41

Pattern Programming

#include “Peregrine.hh”using namespace Peregrine;

void motifCounting(int size){

DataGraph G(“path/to/graph/”);auto patterns = PatternGenerator::all(size, VERTEX_INDUCED);auto counts = count(G, patterns);

for (auto &[pattern, n] : counts)std::cout << pattern << “ ” << n << std::endl;

}

42

Pattern Programming





}

43

Pattern Programming





}

44

Pattern Programming





}

45

Pattern Programming

DataGraph G(“path/to/graph/”);auto patterns = PatternGenerator::all(size, VERTEX_INDUCED);

auto counts = count(G, patterns);

46

Pattern Programming

DataGraph G(“path/to/graph/”);auto patterns = PatternGenerator::all(size, VERTEX_INDUCED);patterns[0].set_labels({‘a’, ‘b’, ‘c’, ‘d’});


47

Pattern Programming

DataGraph G(“path/to/graph/”);auto patterns = PatternGenerator::all(size, VERTEX_INDUCED);patterns[0].set_labels({‘a’, ‘b’, ‘c’, ‘d’});patterns[0].add_edge(1, 5);


48

Pattern Programming

DataGraph G(“path/to/graph/”);auto patterns = PatternGenerator::all(size, VERTEX_INDUCED);patterns[0].set_labels({‘a’, ‘b’, ‘c’, ‘d’});patterns[0].add_edge(1, 5);patterns.emplace_back(“path/to/pattern.txt”);auto counts = count(G, patterns);

49

Pattern Programming

DataGraph G(“path/to/graph/”);auto patterns = PatternGenerator::all(size, VERTEX_INDUCED);auto pattern = Pattern().add_edge(1, 2)

.add_edge(1, 3)

.add_edge(2, 3);auto counts = count(G, {pattern});

50

Pattern Programming





}

51

Pattern Programming





}

52

Pattern Programming

void frequentSubgraphMining(){

DataGraph G(“path/to/graph/”);auto patterns = PatternGenerator::all(2, EDGE_INDUCED);auto mapDomain = [](auto &&match, auto &&aggregator)

{ aggregator.map(match.pattern, match.mapping); };

auto results = match<Pattern, Domain>(G, patterns, mapDomain);

for (auto &[pattern, frequency] : results)std::cout << pattern << “ ” << frequency << std::endl;

}

53

Pattern Programming






}

54

Pattern Programming






}

55

Pattern Programming






}

56

Pattern Awareness

Pattern Selection

Pattern Matching

57

Pattern Awareness

Pattern Selection

u

v

Anti-Edge

58

Pattern Awareness

Pattern Selection

u

v

Anti-Vertex

59

Pattern Awareness

Pattern Selection

Pattern Matching

60

Pattern Awareness

Pattern Selection

Pattern Matching

61

Pattern Awareness

Pattern Matching

• Symmetry breaking (RECOMB ‘07)

• Core pattern reduction (SIGMOD ‘16)

62

Symmetry Breaking

63

Symmetry Breaking

u x

y v

64

Symmetry Breaking

u x

y v

65

Symmetry Breaking

x

y

66

v

u

Symmetry Breaking

u x

y v

67

Symmetry Breaking

u y

x v

68

Symmetry Breaking

u x

y v

Worker 1

a

db

e

c

gf

Data Graph

69

b x

y v

Symmetry Breaking

a

db

e

c

gf

Data Graph Worker 1

70

b x

y d

Symmetry Breaking

Worker 1

a

db

e

c

gf

Data Graph

71

Symmetry Breaking

b x

y d

Worker 1

u x

y v

Worker 2

a

db

e

c

gf

Data Graph

72

Symmetry Breaking

b x

y d

Worker 1

d x

y v

Worker 2

a

db

e

c

gf

Data Graph

73

Symmetry Breaking

b x

y d

Worker 1

d x

y b

Worker 2

a

db

e

c

gf

Data Graph

74

Symmetry Breaking

b a

y d

Worker 1

d a

y b

Worker 2

a

db

e

c

gf

Data Graph

75

Symmetry Breaking

b a

g d

Worker 1

d a

g b

Worker 2

a

db

e

c

gf

Data Graph

76

Symmetry Breaking

b a

g d

Worker 1

d a

g b

Worker 2

a

db

e

c

gf

Data Graph

77

Symmetry Breaking

u < v

u

v

x

y

78

Symmetry Breaking

x < y

u

v

x

y

79

Symmetry Breaking

u < vx < y

u

v

x

y

80

Core Pattern

u

v

x

y

81

Core Pattern

u

v

x

y

82

Core Pattern

u

v

x

y

83

Core Pattern

b

d

Pattern-Unaware

a

db

e

c

gf

Data Graph

84

Core Pattern

b a

d

Pattern-Unaware

a

db

e

c

gf

Data Graph

85

Core Pattern

b a

d

Pattern-Unaware

a

db

e

c

gf

Data Graph

86

Core Pattern

b a

g d

Pattern-Unaware

a

db

e

c

gf

Data Graph

87

Core Pattern

b a

g d

Pattern-Unaware

a

db

e

c

gf

Data Graph

88

Core Pattern

b x

y d

Pattern-Aware

a

db

e

c

gf

Data Graph

89

Core Pattern

b x

y d

Pattern-Aware

a

db

e

c

gf

Data Graph

90

Core Pattern

a

db

e

c

gf

Data Graph

b x

y d

Pattern-Aware

91

Core Pattern

b a

g d

Pattern-Aware

a

db

e

c

gf

Data Graph

92

Core Pattern

b a

g d

Pattern-Aware

a

db

e

c

gf

Data Graph

h

93

Core Pattern

b a

h d

Pattern-Aware

b a

g d

a

db

e

c

gf

Data Graph

h

94

Core Pattern

b a

h d

Pattern-Aware

b a

g d

a

db

e

c

gf

Data Graph

h

95

Pattern Awareness

Pattern Matching

• Symmetry breaking (RECOMB ‘07)

• Core pattern reduction (SIGMOD ‘16)

96

Pattern Awareness

Pattern Matching

• Early termination

97

Early Termination

bool globalClusteringCoefficient(int bound){

DataGraph G(“path/to/graph/”);auto triplet = PatternGenerator::star(3);int numTriplets = count(G, {triplet});auto countAndCheck = [=](auto &&match, auto &&aggregator){int numTriangles = aggregator.readValue(match.pattern);if (3*numTriangles/numTriplets > bound) aggregator.stop();else aggregator.map(match.pattern, 1);

}auto triangle = PatternGenerator::clique(3);auto result = match<Pattern, int>(G, triangle, countAndCheck);return 3*result[triangle]/numTriplets > bound;

}

98

Early Termination




}

99

Early Termination




}

100

Triangle

Early Termination




}

101

TripletTriangle

Early Termination




}

102

TripletTriangle

Early Termination




}

103

TripletTriangle

Early Termination




}

104

TripletTriangle

Early Termination




}

105

TripletTriangle

Early Termination




}

106

TripletTriangle

Early Termination




}

107

TripletTriangle

Early Termination




}

108

Comparison with Existing Work

• Peregrine with 16 logical cores and 32GB RAM

• Arabesque & Fractal with 8x16 logical cores and 8x32GB RAM

• RStream with 96 logical cores and 192GB RAM

109


110


111

Effects of Pattern Awareness

112

Scalability

113

Scalability

114

Scalability

115

Scalability

116

Scalability

117

Scalability

118

Scalability

119

Pattern-aware programming and processing models

120


• Shift abstraction from subgraph to pattern

121



•User program is transparent to the system

122




•Up to 42x faster than pattern-unaware

•Up to 737x faster than state-of-the-art

123




•Up to 42x faster than pattern-unaware

•Up to 737x faster than state-of-the-art

https://github.com/pdclab/peregrine

124

a pattern-aware graph mining system

Documents