data modeling with neo4j training

(you)-[:know]->(Neo4j)

Training�� Neo4j��

© All Rights Reserved 2014 | Neo Technology, Inc.

with�� FIRST�� LAST�� @TWITTER��

EMAIL��

© All Rights Reserved 2014 | Neo Technology, Inc. © All Rights Reserved 2013 | Neo Technology, Inc. © All Rights Reserved 2013 | Neo Technology, Inc.

Welcome!��

Grab a USB key, and plug it in.!

Find a seat. Say hi!!1!

2!

Run the “Install Neo4j” application (Mac or Windows).!

When you see the README and the command prompt, you’re ready.!3!

4!Grab coffee and get ready to begin!!


Data�� Modeling�� With�� Neo4j��


Why�� did�� you�� come?��


Answer�� five�� questions��

• What�� is�� Neo4j?�� • How�� do�� I�� query�� data?�� • How�� do�� I�� represent�� my�� domain�� in�� a�� graph?�� • What�� are�� common�� graph�� structures?�� • How�� does�� my�� model�� evolve?��

and...�� 5


provide�� the�� confidence�� to�� start�� building��

a�� complex�� graph�� model��


Topics��

• Data�� complexity�� • Graph�� model�� building�� blocks�� • Quick�� intro�� to�� Cypher�� • Modeling�� guidelines�� • Common�� graph�� structures�� • Evolving�� a�� graph�� model��

7


Addressing�� Data�� Complexity�� With�� Graphs��


Data�� Complexity��

complexity = f(size, semi-structure, connectedness)


Social�� Network��

10


Network�� Impact�� Analysis��

11


Route�� Finding��

12


Recommendations��

13


Logistics��

14


Access�� Control��

15


Fraud�� Analysis��

16


Securities�� and�� Debt��

17

Image: orgnet.com


Graphs�� Are�� Everywhere��

18


Graph�� Databases��

• Store�� • Manage�� • Data��

19


Neo4j�� is�� a�� Graph�� Database��

20


Graph�� Model�� Building�� Blocks��


Labeled�� Property�� Graph�� Data�� Model��

22


Four�� Building�� Blocks��

• Nodes�� • Relationships�� • Properties�� • Labels��

23


Nodes��

24


Nodes��

• Used�� to�� represent�� entities�� and�� complex�� value�� types�� in�� your�� domain��

• Can�� contain�� properties�� • Used�� to�� represent�� entity�� attributes�� and/or�� metadata�� (e.g.�� timestamps,�� version)�� • Key-value�� pairs�� •  Java�� primitives�� including�� Strings�� •  Arrays�� •  null�� is�� not�� a�� valid�� value�� • Every�� node�� can�� have�� different�� properties��

25


Entities�� and�� Value�� Types��

• Entities�� • Have�� unique�� conceptual�� identity�� • Change�� attribute�� values,�� but�� identity�� remains�� the�� same�� • Value�� types�� • No�� conceptual�� identity�� • Can�� substitute�� for�� each�� other�� if�� they�� have�� the�� same�� value�� •  Simple:�� single�� value�� (e.g.�� colour,�� category)�� •  Complex:�� multiple�� attributes�� (e.g.�� address)��

26


Relationships��

27


Relationships��

• Every�� relationship�� has�� a�� name�� and�� a�� direction�� • Add�� structure�� to�� the�� graph�� • Provide�� semantic�� context�� for�� nodes�� • Can�� contain�� properties�� • Used�� to�� represent�� quality�� or�� weight�� of�� relationship,�� or�� metadata�� • Every�� relationship�� must�� have�� a�� start�� node�� and�� end�� node�� • No�� dangling�� relationships��

28


Relationships�� (continued)��

29

Nodes can have more than one

relationship Self relationships are allowed

Nodes can be connected by more than one

relationship


Variable�� Structure��

• Relationships�� are�� defined�� with�� regard�� to�� node�� instances,�� not�� classes�� of�� nodes��

• Two�� nodes�� representing�� the�� same�� kind�� of�� “thing”�� can�� be�� connected�� in�� very�� different�� ways��

•  Allows�� for�� structural�� variation�� in�� the�� domain�� • Contrast�� with�� relational�� schemas,�� where�� foreign�� key�� relationships�� apply�� to�� all�� rows�� in�� a�� table��

•  No�� need�� to�� use�� null�� to�� represent�� the�� absence�� of�� a�� connection��

30


Labels��

31


Labels��

• Every�� node�� can�� have�� zero�� or�� more�� labels�� • Used�� to�� represent�� roles�� (e.g.�� user,�� product,�� company)��

• Group�� nodes�� • Allow�� us�� to�� associate�� indexes�� and�� constraints�� with�� groups�� of�� nodes��

32


Four�� Building�� Blocks��

• Nodes�� • Entities�� • Relationships�� • Connect�� entities�� and�� structure�� domain�� • Properties�� • Entity�� attributes,�� relationship�� qualities,�� and�� metadata�� • Labels�� • Group�� nodes�� by�� role��

33


Cypher�� Query�� Language��


Nodes�� and�� Relationships��

35

()-->()


Labels�� and�� Relationship�� Types��

36

(:Person)-[:FRIEND]->(:Person)


Properties��

37

(:Person{name:'Peter'})-[:FRIEND]->(:Person{name:'Lucy'})


Identifiers��

38

(p1:Person{name:'Peter'})-[r:FRIEND]->(p2:Person{name:'Lucy'})


Cypher��

MATCH�� graph_pattern��

RETURN�� results��

39

http://docs.neo4j.org/chunked/milestone/query-match.html http://docs.neo4j.org/chunked/milestone/query-return.html


Example�� Query��

MATCH (p:Person{name:'Peter'})

-[:FRIEND]->(friends)

RETURN friends

40


Find�� This�� Pattern��



RETURN friends

41


Lookup�� Using�� Identifier�� +�� Label��



RETURN friends

42

Search nodes labeled ‘Person’, matching on

‘name’ property


Return�� Nodes��



RETURN friends

43


Exercise�� 1��

44

Example�� Graph��


Install�� Neo4j��

45

Windows��

• Run�� the�� installer�� and�� use�� the�� Neo4j�� Desktop�� to�� start�� and�� stop�� your�� server��

Linux/Mac��

• Unzip�� the�� distribution,�� then�� start�� the�� server�� from�� a�� terminal:��

• bin/neo4j�� start��


Open�� The�� Browser�� (Chrome)��

46


Clean�� The�� Database��

47


Exercise�� 1�� -�� Create�� Some�� Data��

48

• Clear�� the�� command�� bar�� • Drag�� setup.txt�� into�� the�� command�� bar�� • Click�� Execute��


View�� Some�� Data��

49

Try�� this�� query:��

MATCH�� (n)�� RETURN�� n��

��

Then�� this:��

MATCH�� (person:Person)-[r:HAS_SKILL]-(skill)��

RETURN�� person.username,�� skill.name,�� r.level��


Run�� Another�� Query��

50

Execute�� query-1.txt��

Find�� colleagues�� with�� similar�� skills�� to�� Ian��

MATCH (company)<-[:WORKS_FOR]-(me:Person{username:'ian'})

-[:HAS_SKILL]->(skill),

(company)<-[:WORKS_FOR]-(colleague)-[r:HAS_SKILL]->(skill)

WHERE r.level > 1

RETURN colleague.username AS username,

count(skill) AS score,

collect(skill.name) AS skills

ORDER BY score DESC


MATCH��

51



(company)<-[:WORKS_FOR]-(colleague)

-[r:HAS_SKILL]->(skill)

WHERE r.level > 1




ORDER BY score DESC


Anchor�� the�� Pattern��

52

Search nodes labeled ‘Person’, matching on ‘username’ property




WHERE r.level > 1




ORDER BY score DESC


Narrow�� Results�� Using�� WHERE��

53

Only match HAS_SKILL relationships whose level

property > 1




WHERE r.level > 1




ORDER BY score DESC


Column�� Alias��

54

http://docs.neo4j.org/chunked/milestone/query-return.html#return-column-alias




WHERE r.level > 1




ORDER BY score DESC


Count��

55

http://docs.neo4j.org/chunked/milestone/query-aggregation.html#_count




WHERE r.level > 1




ORDER BY score DESC


Collect��

56

http://docs.neo4j.org/chunked/milestone/query-aggregation.html#aggregation-collect




WHERE r.level > 1




ORDER BY score DESC


ORDER�� BY��

57

http://docs.neo4j.org/chunked/milestone/query-order.html




WHERE r.level > 1




ORDER BY score DESC


Modeling�� Guidelines��


Symmetric�� Relationships��

59

OR


Infer�� Symmetric�� Relationship��

60


Bi-Directional�� Relationships��

61


Use�� Single�� Relationship�� and�� Ignore�� Relationship�� Direction�� in�� Queries��

62

MATCH (p1{name:'Eric'})

-[:KNOWS]-(p2)

RETURN p2


Qualified�� Bi-Directional�� Relationships��

63

OR


Properties�� Versus�� Relationships��

64


Use�� Relationships�� When..��

• You�� need�� to�� specify�� the�� weight,�� strength,�� or�� some�� other�� quality�� of�� the�� relationship��

• AND/OR�� the�� attribute�� value�� comprises�� a�� complex�� value�� type�� (e.g.�� address)��

• Examples:�� • Find�� all�� my�� colleagues�� who�� are�� level�� 2�� or�� above�� (relationship�� quality)�� in�� a�� skill�� (attribute�� value)�� we�� have�� in�� common��

• Find�� all�� recent�� orders�� delivered�� to�� the�� same�� delivery�� address�� (complex�� value�� type)��

65


Example:�� Find�� Expert�� Colleagues��

66


Use�� Properties�� When..��

• There’s�� no�� need�� to�� qualify�� the�� relationship�� • AND�� the�� attribute�� value�� comprises�� a�� simple�� value�� type�� (e.g.�� colour)��

• Examples:�� • Find�� those�� projects�� written�� by�� contributors�� to�� my�� projects�� that�� use�� the�� same�� language�� (attribute�� value)�� as�� my�� projects��

67


Example:�� Similar�� By�� Language��

68


If�� Performance�� is�� Critical..��

• Small�� property�� lookup�� on�� a�� node�� will�� be�� quicker�� than�� traversing�� a�� relationship��

• But�� traversing�� a�� relationship�� is�� still�� faster�� than�� a�� SQL�� join…⋯�� • However,�� many�� small�� properties�� on�� a�� node,�� or�� a�� lookup�� on�� a�� large�� string�� or�� large�� array�� property�� will�� impact�� performance��

• Always�� performance�� test�� against�� a�� representative�� dataset��

69


Relationship�� Granularity��

70


Align�� With�� Use�� Cases��

• Relationships�� are�� the�� “royal�� road”�� into�� the�� graph��

• When�� querying,�� well-named�� relationships�� help�� discover�� only�� what�� is�� absolutely�� necessary��

• And�� eliminate�� unnecessary�� portions�� of�� the�� graph�� from�� consideration��

71


General�� Relationships��

• Qualified�� by�� property��

72


Specific�� Relationships��

73


Best�� of�� Both�� Worlds��

74



75

Creating�� Data��


Exercise�� 2�� -�� Create�� Some�� data��

76

• Clean�� the�� database�� • Execute�� create-1.txt�� • View�� the�� results�� • MATCH�� (n)�� RETURN�� n��


Create�� Nodes��

77

CREATE

(ben:Person{username:'ben'}),

(acme:Company{name:'Acme, Inc'}),

(neo4j:Skill{name:'Neo4j'}),

(rest:Skill{name:'REST'}),

(ben)-[:WORKS_FOR]->(acme),

(ben)-[:HAS_SKILL{level:1}]->(neo4j),

(ben)-[:HAS_SKILL{level:3}]->(rest)


Connect�� Nodes��

78

CREATE

(ben:Person{username:'ben'}),

(acme:Company{name:'Acme, Inc'}),

(neo4j:Skill{name:'Neo4j'}),

(rest:Skill{name:'REST'}),

(ben)-[:WORKS_FOR]->(acme),

(ben)-[:HAS_SKILL{level:1}]->(neo4j),

(ben)-[:HAS_SKILL{level:3}]->(rest)


Create�� Some�� More�� Data��

79

• Create�� more�� people�� •  Same�� skills�� (Neo4j�� and�� REST)�� •  Same�� company�� (Acme)��

• View�� the�� results�� • MATCH (n) RETURN n


ADVANCED�� LAB��

80

• Clean�� the�� database�� • Execute�� create-2.txt,�� create-3.txt�� and�� create-4.txt��

•  After�� each�� operation,�� view�� the�� results�� • What�� happens�� if�� you�� add�� or�� remove�� properties�� when�� specifying�� unique�� nodes�� and�� relationships?��


Creating�� Unique�� Nodes�� and�� Relationships��

81

MERGE (c:Company{name:'Acme'})

MERGE (p:Person{username:'ian'})

MERGE (s1:Skill{name:'Java'})

MERGE (s2:Skill{name:'C#'})

MERGE (s3:Skill{name:'Neo4j'})

MERGE (c)<-[:WORKS_FOR]-(p)

MERGE (p)-[r1:HAS_SKILL]->(s1)



SET r1.level = 2

SET r2.level = 2

SET r3.level = 3

RETURN c, p, s1, s2, s3


Create�� Unique�� Nodes��

82










SET r1.level = 2

SET r2.level = 2

SET r3.level = 3



Create�� Unique�� Relationships��

83










SET r1.level = 2

SET r2.level = 2

SET r3.level = 3



Set�� Relationship�� Properties��

84










SET r1.level = 2

SET r2.level = 2

SET r3.level = 3



MERGE��

85

http://docs.neo4j.org/chunked/milestone/query-merge.html

MERGE�� ensures�� that�� a�� pattern�� exists�� in�� the�� graph.�� Either�� the�� pattern�� already�� exists,�� or�� it�� needs�� to�� be�� created.��


Common�� Graph�� Structures��


Intermediate�� Nodes��

• Connect�� more�� than�� 2�� nodes�� in�� a�� single�� context��

• Hyperedges�� (n-ary�� relationships)�� • Relate�� something�� to�� a�� relationship��

87


Rich�� Context,�� Multiple�� Dimensions��

88


Intermediate�� Nodes�� Between�� Contexts��

89


Multiple�� Parties��

90


Considerations��

• An�� intermediate�� node�� provides�� flexibility��

• It�� allows�� more�� than�� two�� nodes�� to�� be�� connected�� in�� a�� single�� context��

• But�� it�� can�� be�� overkill,�� and�� will�� have�� an�� impact�� on�� performance��

91


Linked�� Lists��

• Entities�� are�� linked�� in�� a�� sequence�� • You�� need�� to�� traverse�� the�� sequence�� • You�� may�� need�� to�� identify�� the�� beginning�� or�� end�� (first/last,�� earliest/latest,�� etc.)��

• Examples�� • Event�� stream�� • Episodes�� of�� a�� TV�� series�� • Job�� history��

92


Job�� History��

93

NEXT from: 2005��to: 2007

from: 2007��to: 2008

from: 2009��to: 2011

from: 2011��to: 2014

NEXT NEXT

name: Patrick

name: Startup name: Acme title: Software

Developer title: QA

EMPLOYMENT EMPLOYMENT EMPLOYMENT

EMPLOYMENT

ROLE

name: MNC

ROLE ROLE ROLE COMPANY COMPANY COMPANY COMPANY


NEXT

Linked�� List��

94

from: 2005��to: 2007

from: 2007��to: 2008

from: 2009��to: 2011

from: 2011��to: 2014

NEXT NEXT


Doubly�� Linked�� List��

95

NEXT

from: 2005��to: 2007

from: 2007��to: 2008

from: 2009��to: 2011

from: 2011��to: 2014

PREV

NEXT

PREV

NEXT

PREV


Interleaved�� Linked�� Lists��

96


from: 2007��to: 2008

from: 2009��to: 2011

from: 2011��to: 2014

NEXT NEXT from: 2007��to: 2008

NEXT

NEXT_COMPANY

NEXT_COMPANY


Pointers�� to�� Head�� and�� Tail��

97


from: 2007��to: 2008

from: 2009��to: 2011

from: 2011��to: 2014

NEXT NEXT

name: Patrick

FIRST LAST



98

Linked�� List�� Example��


Patrick’s�� Job�� History��

99

• Add�� jobs�� as�� Patrick�� changes�� them�� •  Maintain�� pointer�� to�� FIRST�� and�� LAST�� jobs�� • Find�� all�� jobs�� he’s�� been�� in�� so�� far�� • Find�� his�� latest�� job��


Your�� Turn��

100

• Clean�� the�� database�� • Execute�� setup.txt�� •  Creates�� root�� Patrick�� node�� • Execute�� add-node.txt�� •  Adds�� Jobs�� • Modify�� the�� query�� to�� add�� more�� jobs�� in�� order��

• At�� each�� stage,�� view�� the�� results�� • MATCH�� (n)�� RETURN�� n��


Add�� Job�� to�� History��

101

MATCH (patrick:User {name: "Patrick"})

MERGE (newJob:Job {from: 2009, to:2010})

MERGE (patrick)-[:LAST]->(newJob)

WITH patrick,newJob,

CASE WHEN NOT ((patrick)-[:FIRST]->()) THEN [1] ELSE []

END AS firstExists

FOREACH (i IN firstExists | CREATE UNIQUE (patrick)-[:FIRST]->(newJob))

WITH patrick, newJob

MATCH (newJob)<-[:LAST]-(patrick)-[oldRel:LAST]->(oldLast)

DELETE oldRel

MERGE (oldLast)-[:NEXT]->(newJob)


Query-1�� -�� Find�� All�� Jobs�� So�� Far��

102

MATCH (patrick:User{name:"Patrick"})-[:FIRST]->(firstJob)-[:NEXT*0..]->(nextJob)

RETURN nextJob.from as nextJobFrom, nextJob.to as nextJobTo


Query-2�� -�� Find�� Last�� Job�� held��

103

MATCH (patrick:User{name:'Patrick'})-[:LAST]->(lastJob)

RETURN lastJob.from AS lastJobFrom, lastJob.to as lastJobTo


In-Graph�� Indexes��

• Indexes�� are�� graphs:�� • B-tree�� (binary�� search)�� • R-tree�� (spatial�� access,�� multi-dimensional�� information)��

104


Timeline�� Tree��

• Discrete�� events�� • No�� natural�� relationships�� to�� other�� events�� • You�� need�� to�� find�� events�� at�� differing�� levels�� of�� granularity��

• Between�� two�� days�� • Between�� two�� months�� • Between�� two�� minutes��

105


Example�� Timeline�� Tree��

106



107

Timeline�� Tree�� Example��



108

• Clean�� the�� database�� • Execute�� create-1.txt�� to�� create-6.txt�� in�� order��

• At�� each�� stage,�� view�� the�� results�� •  MATCH (n) RETURN n


Lazily�� Insert�� Date�� Elements��

109

MERGE (timeline:Timeline{name:'timeline-1'})

MERGE (timeline)-[:YEAR]->(year{value:2008})

MERGE (year)-[:MONTH]->(month{value:1})

MERGE (month)-[:DAY]->

(day{value:28, millis:1201478400000})

MERGE (day)<-[:OCCURRED]-

(n:Purchase{name:'purchase-6'})


Create�� or�� Insert�� Root��

110





(day{value:28, millis:1201478400000})




The�� Add�� ‘Locally�� Unique’�� Nodes��

111





(day{value:28, millis:1201478400000})




Query-1�� -�� Get�� All�� Events�� Between�� Two�� Dates��

112

MATCH (timeline:Timeline{name:'timeline-1'})

-[:YEAR]->(year)

-[:MONTH]->(month)

-[:DAY]->(day)

<-[:OCCURRED]-(n)

WHERE day.millis >= 1169596800000

AND day.millis < 1200268800000

RETURN n


Composing�� Structures��

113


Timeline�� Tree��

114


Intermediate�� Nodes��

115


Linked�� Lists��

116


Evolving�� a�� Graph�� Model��


Refactoring��

Definition��

• Restructure�� graph�� without�� changing�� informational�� semantics�� Reasons��

• Improve�� design�� • Enhance�� performance�� • Accommodate�� new�� functionality�� • Enable�� iterative�� and�� incremental�� development�� of�� data�� model��

118


Data�� Migrations��

• Execute�� in�� repeatable�� order�� • Backup�� database�� • Execute�� in�� batches�� • Unbounded�� results�� will�� generate�� large�� transactions�� and�� may�� trigger�� Out�� of�� Memory�� exceptions��

• Apply�� migrations�� to�� test�� data�� to�� ensure�� existing�� functionality�� doesn’t�� break��

• Ensure�� application�� can�� accommodate�� old�� and�� new�� structures�� if�� performing�� against�� live�� data��

119


Extract�� Node�� from�� Property��

Problem�� • You’ve�� modeled�� an�� attribute�� as�� a�� property�� with�� a�� simple�� value,�� but�� now�� need�� to:�� • Qualify�� the�� attribute�� semantics�� AND/OR�� • Introduce�� a�� complex�� value�� AND/OR�� • Reify�� the�� relationship�� represented�� by�� the�� value�� Solution�� • Create�� a�� new�� node�� per�� unique�� property�� value�� • Connect�� existing�� nodes�� to�� the�� new�� property�� nodes�� • Remove�� the�� old�� property��

120



121

Extract�� Node�� From�� Property��


Example:�� (n.currency)�� to�� (:Currency)��

122



123

• Clean�� the�� database�� • Execute�� setup.txt�� • View�� the�� results�� • MATCH�� (n)�� RETURN�� n�� • Execute�� update-1.txt�� repeatedly,�� until�� numberRemoved�� is�� zero��

• At�� each�� stage,�� view�� the�� results��


Extract�� Node�� From�� Property��

124

MATCH (t:Trade) WHERE has(t.currency)

WITH t LIMIT {batchSize}

MERGE (c:Currency{code:t.currency})

MERGE (t)-[:CURRENCY]->(c)

REMOVE t.currency

RETURN count(t) AS numberRemoved


Select�� Batch�� of�� Nodes�� With�� Property��

125





REMOVE t.currency



Create�� Property�� Node��

126

Copy property value from existing node





REMOVE t.currency



Relate�� Existing�� Node�� to�� Property�� Node��

127





REMOVE t.currency



Remove�� Old�� Property��

128

Repeat until numberRemoved

is zero





REMOVE t.currency



Extract�� Node�� from�� Array�� Property��

Problem�� • You’ve�� modeled�� an�� attribute�� as�� a�� property�� with�� an�� array�� value,�� but�� now�� need�� to:�� • Qualify�� the�� attribute�� semantics�� AND/OR�� • Introduce�� a�� complex�� value�� AND/OR�� • Reify�� the�� relationship�� represented�� by�� the�� value�� Solution�� • Create�� a�� new�� node�� per�� unique�� property�� value�� • Connect�� existing�� nodes�� to�� the�� new�� property�� nodes�� • Remove�� the�� old�� property��

129


Example:�� Extract�� Language�� Nodes��

130



131

Extract�� Node�� From�� Array�� Property��



132

• Clean�� the�� database�� • Execute�� setup.txt�� • View�� the�� results�� • MATCH�� (n)�� RETURN�� n�� • Execute�� update-1.txt�� repeatedly,�� until�� numberRemoved�� is�� zero��



Extract�� Node�� From�� Array�� Property��

133

MATCH (project:Project)

WHERE has(project.language)

WITH project LIMIT {batchSize}

FOREACH (l IN project.language |

MERGE (language:Language{value:l})

CREATE UNIQUE (project)-[:LANGUAGE]->(language))

REMOVE project.language

RETURN count(project) AS numberDeleted


Select�� Batch�� of�� Nodes�� With�� Property��

134










Loop�� Through�� Values�� in�� Array..��

135










Create�� New�� Unique�� Node�� Per�� Value��

136

Copy value from current iteration

MATCH�� (project:Project)��

WHERE�� has(project.language)��

WITH�� project�� LIMIT�� {batchSize}��

FOREACH�� (l�� IN�� project.language�� |��

�� MERGE�� (language:Language{value:l})��

�� CREATE�� UNIQUE�� (project)-[:LANGUAGE]->(language))��

REMOVE�� project.language��

RETURN�� count(project)�� AS�� numberDeleted��


Relate�� Existing�� Node�� to�� Value�� Node��

137










Remove�� Array�� Property��

138

Repeat until numberRemoved is

zero








RETURN count(project) AS numberRemoved


Extract�� Node�� From�� Relationship��

Problem��

• You’ve�� modeled�� something�� as�� a�� relationship�� (with�� properties),�� but�� now�� need�� to�� connect�� it�� to�� more�� than�� two�� things��

Solution��

• Extract�� relationship�� into�� a�� new�� node�� (and�� two�� new�� relationships)�� • Copy�� old�� relationship�� properties�� onto�� new�� node�� • Delete�� old�� relationship��

139



140



Example:�� [:EMAILED]�� to�� (:Email)��

141



142

• Clean�� the�� database�� • Execute�� setup.txt�� • View�� the�� results�� • MATCH�� (n)�� RETURN�� n�� • Execute�� update-1.txt�� repeatedly,�� until�� numberDeleted�� is�� zero��




143

MATCH (a:User)-[r:EMAILED]->(b:User)

WITH a, r, b LIMIT 2

MERGE (email:Email{content:r.content, _rid:id(r)})

MERGE (a)-[:SENT]->(email)-[:TO]->(b)

DELETE r

RETURN count(r) AS numberDeleted


Select�� Batch�� of�� Relationships��

144





DELETE r



Create�� New�� Node�� and�� Relationships��

145

“Refactoring ID” ensures uniqueness

Copy properties from old relationship MATCH (a:User)-[r:EMAILED]->(b:User)




DELETE r



Delete�� Old�� Relationship��

146

Repeat until numberDeleted is

zero





DELETE r



(Optional)�� Remove�� Refactoring�� IDs��

147

MATCH (n:Email)

WHERE has(n._rid)

WITH n LIMIT {batchSize}

SET n._rid = null

RETURN count(n) AS numberRemoved

Background

Business problem Solution & Benefits


Background



San Jose, CA

Cisco.com

Industry: Communications Use case: Recommendations

•  Call center volumes needed to be lowered by improving the efficacy of online self service

•  Leverage large amounts of knowledge stored in service cases, solutions, articles, forums, etc.

•  Problem resolution times, as well as support costs, needed to be lowered

•  Cisco.com serves customer and business customers with Support Services •  Needed real-time recommendations, to encourage use of online knowledge

base •  Cisco had been successfully using Neo4j for its internal master data

management solution. •  Identified a strong fit for online recommendations

•  Cases, solutions, articles, etc. continuously scraped for cross-reference links, and represented in Neo4j

•  Real-time reading recommendations via Neo4j •  Neo4j Enterprise with HA cluster

•  The result: customers obtain help faster, with decreased reliance on customer support

150

Support Case

Support Case

Knowledge Base

Article

Solution

Knowledge Base

Article

Knowledge Base

Article

Message

Background



Background



San Jose, CA

Cisco HMP

Industry: Communications Use case: Master Data Management

•  Sales compensation system had become unable to meet Cisco’s needs •  Existing Oracle RAC system had reached its limits:

•  Insufficient flexibility for handling complex organizational hierarchies and mappings

•  “Real-time” queries were taking > 1 minute! •  Business-critical “P1” system needs to be continually available, with zero

downtime

•  One of the world’s largest communications equipment manufacturers •  #91 Global 2000. $44B in annual sales. •  Needed a system that could accommodate its master data hierarchies in a

performant way

•  HMP is a Master Data Management system at whose heart is Neo4j. Data access services available 24x7 to applications companywide

•  Cisco created a new system: the Hierarchy Management Platform (HMP) •  Allows Cisco to manage master data centrally, and centralize data access and business

rules •  Neo4j provided “Minutes to Milliseconds” performance over Oracle RAC, serving

master data in real time •  The graph database model provided exactly the flexibility needed to support Cisco’s

business rules •  HMP so successful that it has expanded to��

include product hierarchy

151

Background



Background



Location

CompanyName

Industry: Logistics Use case: Parcel Routing

•  24x7 availability, year round •  Peak loads of 2500+ parcels per second •  Complex and diverse software stack •  Need predictable performance & linear scalability

•  Daily changes to logistics network: route from any point, to any point

•  One of the world’s largest logistics carriers •  Projected to outgrow capacity of old system •  New parcel routing system

•  Single source of truth for entire network

•  B2C & B2B parcel tracking •  Real-time routing: up to 5M parcels per day

•  Neo4j provides the ideal domain fit: •  a logistics network is a graph

•  Extreme availability & performance with Neo4j clustering •  Hugely simplified queries, vs. relational for complex routing

•  Flexible data model can reflect real-world data variance much better than relational •  “Whiteboard friendly” model easy to understand

152

Background



Background



Sausalito, CA

GlassDoor

Industry: Online Job Search Use case: Social / Recommendations

•  Wanted to leverage known fact that most jobs are found through personal & professional connections

•  Needed to rely on an existing source of social network data. Facebook was the ideal choice.

•  End users needed to get instant gratification •  Aiming to have the best job search service, in a very competitive market

•  Online jobs and career community, providing anonymized inside information to job seekers

•  First-to-market with a product that let users find jobs through their network of Facebook friends

•  Job recommendations served real-time from Neo4j •  Individual Facebook graphs imported real-time into Neo4j

•  Glassdoor now stores > 50% of the entire Facebook social graph •  Neo4j cluster has grown seamlessly, with new instances being brought online as graph

size and load have increased

153

Person

Company

Person

Person Company

KN

OW

S

WORKS_AT

WORKS_AT

Background



Background



San Jose, CA

Adobe

Industry: Web/ISV Use case: Content Management, Social, Access Control

•  Adobe needed a highly robust and available, 24x7 distributed global system, supporting collaboration for users of its highest revenue product line

•  Storing creative artifacts in the cloud meant managing access rights for (eventually) millions of users, groups, collections, and pieces of content

•  Complex access control rules controlling who was connected to whom, and who could see or edit what, proved a significant technical challenge

•  One of the ten largest software companies globally •  $4B+ in revenue. Over 11,000 employees. •  Launched Creative Cloud in 2012, allowing its Creative Suite users to

collaborate via the Cloud

•  Selected Neo4j to meet very aggressive project deadlines. The flexibility of the graph model, and performance, were the two major selection factors.

•  Easily evolve the system to meet tomorrow’s needs •  Extremely high availability and transactional performance requirements. 24x7 with no

downtime. •  Neo4j allows consistently fast response times with complex queries, even as the

system grows •  First (and possibly still only) database cluster to run across three Amazon EC2

regions: U.S., Europe, Asia

154

User-Content-Access Graph

Background



Background



Paris, France

SFR

Industry: Communications Use case: Network Management

•  Infrastructure maintenance took one full week to plan, because of the need to model network impacts

•  Needed rapid, automated “what if” analysis to ensure resilience during unplanned network outages

•  Identify weaknesses in the network to uncover the need for additional redundancy

•  Network information spread across > 30 systems, with daily changes to network infrastructure

•  Business needs sometimes changed very rapidly

•  Second largest communications company in France •  Part of Vivendi Group, partnering with Vodafone

•  Flexible network inventory management system, to support modeling, aggregation & troubleshooting

•  Single source of truth (Neo4j) representing the entire network •  Dynamic system loads data from 30+ systems, and allows new applications to access

network data •  Modeling efforts greatly reduced because of the near 1:1 mapping between the real

world and the graph •  Flexible schema highly adaptable to changing business requirements

155

Router

Service

Switch Switch

Router

Fiber Link Fiber Link

Fiber Link

Oceanfloor Cable

DEPEN

DS_O

N

DEPEN

DS_O

N

DEPENDS_ON

DEPENDS_ON

DEPENDS_ON

DEP

END

S_O

N

LINKED

DEPENDS_ON

Background



Background



Frankfurt, Germany

Deutsche Telecom

Industry: Communications Use case: Social gaming

•  The Fanorakel application allows fans to have an interactive experience while watching sports

•  Fans can vote for referee decisions and interact with other fans watching the game

•  Highly connected dataset with real-time updates •  Queries need to be served real-time on rapidly changing data •  One technical challenge is to handle the very high spikes of activity during

popular games

•  Europe’s largest communications company •  Provider of mobile & land telephone lines to consumers and businesses, as

well as internet services, television, and other services

•  Interactive, social offering gives fans a way to experience the game more closely •  Increased customer stickiness for Deutsche Telekom •  A completely new channel for reaching customers with information, promotions, and

ads

•  Clear competitive advantage

156

Interactive Television Programming

Background



Background



Global (U.S., France)

Hewlett Packard

Industry: Web/ISV, Communications Use case: Network Management

•  Use network topology information to identify root problems causes on the network

•  Simplify alarm handling by human operators •  Automate handling of certain types of alarms Help operators respond rapidly

to network issues •  Filter/group/eliminate redundant Network Management System alarms by

event correlation

•  World’s largest provider of IT infrastructure, software & services •  HP’s Unified Correlation Analyzer (UCA) application is a key application inside

HP’s OSS Assurance portfolio •  Carrier-class resource & service management, problem determination, root

cause & service impact analysis •  Helps communications operators manage large, complex and fast changing

networks

•  Accelerated product development time •  Extremely fast querying of network topology •  Graph representation a perfect domain fit •  24x7 carrier-grade reliability with Neo4j HA clustering

•  Met objective in under 6 months

157

Background



Background



Oslo, Norway

Telenor

Industry: Communications Use case: Resource Authorization & Access Control

•  Degrading relational performance. User login taking minutes while system retrieved access rights

•  Millions of plans, customers, admins, groups. ��Highly interconnected data set w/massive joins

•  Nightly batch workaround solved the performance problem, but meant data was no longer current

•  Primary system was Sybase. Batch pre-compute workaround projected to reach 9 hours by 2014: longer than the nightly batch window

•  10th largest Telco provider in the world, leading in the Nordics •  Online self-serve system where large business admins manage employee

subscriptions and plans •  Mission-critical system whose availability and responsiveness is critical to

customer satisfaction

•  Moved authorization functionality from Sybase to Neo4j •  Modeling the resource graph in Neo4j was straightforward, as the domain is

inherently a graph •  Able to retire the batch process, and move to real-time responses: measured in

milliseconds •  Users able to see fresh data, not yesterday’s snapshot •  Customer retention risks fully mitigated

158

Subscription

Account

Customer

Customer

SUBSCRIBED_BY

CONTROLLED_BY

PART_OF

User

USER_ACCESS

Background



Background



Silicon Valley & France

Viadeo

Industry: Professional Social Network Use case: Social, Recommendations

•  Business imperative for real-time recommendations: to attract new users and retain existing ones

•  Key differentiator: show members how they are connected to any other member

•  Real-time traversals of social graph not feasible with MySQL cluster. Batch precompute meant stale data.

•  Process taking longer & longer: > 1 week!

•  World’s second-largest professional network��(after LinkedIn)

•  50M members. 30K+ new members daily. •  Over 400 staff with offices in 12 countries

•  Neo4j solution implemented in 8 weeks with 3 part-time programmers •  Able to move from batch to real-time: improved responsiveness with up-to-date data. •  Viadeo (at the time) had 8M members and 35M relationships. •  Neo4j cluster now sits at the heart of Viadeo’s professional network, connecting

50M+ professionals

159

Background



Background



Hong Kong

Maaii

Industry: Communications Use case: Social, Mobile

•  Launched a new mobile communication app “Maaii” allowing consumers to communicate by voice & text��(Similar to Line, Viber, Rebtel, VoxOx...)

•  Needed to store & relate devices, users, and contacts

•  Import phone numbers from users’ address books. Rapidly serve up contacts from central database ��to the mobile app

•  Currently around 3M users w/200M nodes in the graph

•  Hong Kong based telephony infrastructure provider��(aka M800 aka Pop Media)

•  Exclusive China Mobile partner for international toll-free services. SMS Hub & other offerings

•  2012 Red Herring Top 100 Global Winner

•  Quick transactional performance for key operations: •  friend suggestions (“friend of friend”) •  updating contacts, blocking calls, etc. •  etc.

•  High availability telephony app uses Neo4j clustering •  Strong architecture fit: Scala w/Neo4j embedded

160

Background



Background



Zürich, Switzerland

Junisphere

Industry: Web/ISV, Communications Use case: Data Center Management

•  “Business Service Management” requires mapping of complex graph, covering: business processes--> business services--> IT infrastructure

•  Embed capability of storing and retrieving this information into OEM application

•  Re-architecting outdated C++ application based on relational database, with Java

•  Junisphere AG is a Zurich-based IT solutions provider •  Founded in 2001. Profitable. Self funded. •  Software & services. •  Novel approach to infrastructure monitoring: ��

Starts with the end user, mapped to business processes and services, and dependent infrastructure

•  Actively sought out a Java-based solution that could store data as a graph •  Domain model is reflected directly in the database:

•  “No time lost in translation” •  “Our business and enterprise consultants now speak the same language, and can

model the domain with the database on a 1:1 ratio.” •  Spring Data Neo4j strong fit for Java architecture

161

Background



Background



San Francisco, CA

Teachscape

Industry: Education Use case: Resource Authorization & Access Control

•  Neo4j was selected to be at the heart of a new architecture. •  The user management system, centered around Neo4j, will be used to

support single sign-on, user management, contract management, and end-user access to their subscription entitlements.

•  Teachscape, Inc. develops online learning tools for K-12 teachers, school principals, and other instructional leaders.

•  Teachscape evaluated relational as an option, considering MySQL and Oracle. •  Neo4j was selected because the graph data model provides a more natural fit

for managing organizational hierarchy and access to assets.

•  Domain and technology fit •  simple domain model where the relationships are relatively complex. Secondary factors included support

for transactions, strong Java support, and well-implemented Lucene indexing integration •  Speed and Flexibility

•  The business depends on being able to do complex walks quickly and efficiently. This was a major factor in the decision to use Neo4j.

•  Ease of Use •  accommodate efficient access for home-grown and commercial off-the-shelf applications, as well as ad-hoc

use. •  Extreme availability & performance with Neo4j clustering •  Hugely simplified queries, vs. relational for complex routing •  Flexible data model can reflect real-world data variance much better than relational •  “Whiteboard friendly” model easy to understand

162

Background



Background



Cambridge, Massachusetts

SevenBridges Genomics

Industry: Life Sciences Use case: Content Management

•  Neo4j is used to store metadata about each sequenced genome (including a pointer to the sequenced genome itself, which is a binary file stored on Amazon S3), and to support search and other forms of information processing against the genomic data.

•  graph database was chosen because “Our specific domain maps naturally onto graph paradigm”.

•  Bioinformatics company offering gene sequencing ��"as a service" (over the web)

•  Provider of genomic information services •  Needed a new platform to support storage & retrieval of sequenced genomes

in the cloud

•  Domain fit •  Domain naturally lends itself to a graph representation. •  Graph model determined to be a perfect fit.

•  Agility & Performance

•  Saved time with Neo4j as compared to the alternatives. •  Queries “practically write themselves.”

•  Solution Completeness •  “Neo4j is incomparably better than other graph databases.”

163


(More�� about�� Neo4j)��


Interacting�� with�� Neo4j��

165

1. Cypher�� for�� most�� work�� 2. REST�� for�� management�� 3. Plugin�� API�� for�� special�� cases��


server/embedded��

http://www.neo4j.org/develop/drivers��


Traverses�� 1,000,000+�� relationships�� /�� second�� on�� commodity�� hardware��


Scale:�� 34�� Bn�� nodes/relationships��


Licensing�� options��


(Next�� steps)��


The�� rise�� of�� the�� graph��


It's�� not�� just�� social��


Find�� the�� graphs�� in�� your�� domain,�� and�� model�� them��


Get�� involved�� in�� the�� community��


GraphGists-�� Share�� your�� model��

•  How do you share your graph model?

•  Whiteboard? •  Document? •  How do you demo queries? •  Need a live database? •  Modelling is incremental!


GraphGists-�� Share�� your�� model��

http://www.neo4j.org/learn/graphgist��



http://stackoverflow.com/questions/tagged/neo4j��



http://groups.google.com/group/neo4j��



https://github.com/neo4j/neo4j/issues��



http://neo4j.meetup.com/��



http://www.neo4j.org/participate/contributors#map��


(Thank�� You)��