real world applications with spring data neo4j

© 2013 SpringOne 2GX. All rights reserved. Do not distribute without permission.

Graph Databases Rock - Learn Why!By Michael Hunger

Neo Technology

Mittwoch, 11. September 13

The book on your seat! -“Graph Databases”

• The definitive guide on graph databases• discusses graph concepts, history and data model• compares with other NoSQL technologies• details use-cases from different domains• explains Neo4j‘s humane query language „Cypher“

Written by experts - Emil Eifrem, Ian Robinson, Jim WebberCompliments of Neo Technology


Bookmark: GraphConnect Oct 3-4 2013 San Francisco

4

$99 ticket coupon in your book


4

• Wed, Sep 11, 6-9PM• David's Restaurant,

5151 Stars and Stripes Drive• Just around the corner• Enjoy dessert, wine, beer• Chance to win a signed

Spring Data Book

Meetup - Tomorrow (Wed) EveningSpring Data Neo4j: Past, Current & Future

http://www.meetup.com/graphdb-sf/events/138428012/




(SDN)<-[:LEARN]-(YOU)-[:GET]->(Book)

YOU

Coffee? Awake?

Spring?

Book

GraphConnectticket

Spring DataNeo4j

NoSQL? Sleep?

Neo4j?


(Michael)-[:CODES]->(SDN)-[:USES]->(Neo4j)

ME

Spring Cloud

Community

Cypher

console

community graph

ServerCustomers

Kernel


Agenda

1. Connected Data, Graph Databases and Neo4j

2. Spring Data Neo4j

3. Twitter-Graph - Small Example App

4. Real World Customers and Use-Cases

5. Summary

6. Q & A


NOSQL


complexity = f(size, semi-structure, connectedness)

Data Complexity


NOSQL databases

NOSQL

RelationalGraph

DocumentKeyValueRiak

Column oriented

Redis

Cassandra

Mongo

Couch

Neo4j

MySQL Postgres


Volume vs. Complexity & NOSQL databases



Volume ~= Size



Den

sity

~=

Com

plex

ity

Volume ~= Size



Den

sity

~=

Com

plex

ity

Volume ~= Size

Key-ValueStore



Den

sity

~=

Com

plex

ity

ColumnFamily

Volume ~= Size

Key-ValueStore



Den

sity

~=

Com

plex

ity

ColumnFamily

Volume ~= Size

Key-ValueStore

DocumentDatabases



RDBMSD

ensi

ty ~

= C

ompl

exity

ColumnFamily

Volume ~= Size

Key-ValueStore

DocumentDatabases



RDBMSD

ensi

ty ~

= C

ompl

exity

ColumnFamily

Volume ~= Size

Key-ValueStore

DocumentDatabases

GraphDatabases



RDBMSD

ensi

ty ~

= C

ompl

exity

ColumnFamily

Volume ~= Size

Key-ValueStore

DocumentDatabases

GraphDatabases

90%of

usecases


Are Graphs Everywhere ?


The big graphs are visible

4Mittwoch, 11. September 13

The big graphs are visible

4

Facebook Open Graph


Social Network


(Network) Impact Analysis


Route Finding


Recommendations


Logistics


Access Control


Fraud Analysis


Graphs are everywhere !!

21

Science

Art

Friends

Neighbor

Work

Family

Biology

Chemistry

7

ComputersInternet

HistoryPoliticsSociety Reactions

Environments

Proteins

Genes

Users

Networks

Devices

ServicesSoftware

Stories

Creativity Visuals

Business

Geo

Retail

ERP

CloudTravel


Graph Databases


You know relational


You know relational

23

customer


You know relational

23

customer account


You know relational

23

customer accountcust_acct


You know relational

23

now consider relationships...


We're talking about a Property Graph



24

Nodes



24

Nodes

Relationships


Emil

Andrés

Lars

Johan

Allison

Peter

Michael

Tobias

Andreas

IanMica

Delia

knows

knows

knowsknows

knows

knows

knows

knows

knows

knowsMica

knowsknowsMica

Delia

knows


24

Nodes

Relationships

Properties (each a key+value)

+ Indexes (for easy look-ups)


How do I query this graph?

Andreas

// lookup starting point in an indexstart n=node:People(name = ‘Andreas’)

START user=node:User(name = ‘Andreas’)MATCH (user)<-[:OWNS]-(account)-[:CONTAINS]-> (booking)-[:TO]->(recipient) RETURN recipient, count(*)ORDER BY count DESC LIMIT 10


How do I query this graph?

26

SELECT skills.*, user_skill.* FROM users JOIN user_skill ON users.id = user_skill.user_id JOIN skills ON user_skill.skill_id = skill.id WHERE users.id = 1

START user = node:() MATCH (user)-[user_skill]->(skill) RETURN skill, user_skill


Whiteboard friendly


Whiteboard friendly

4

AndreasPeter

EmilAllison


Whiteboard friendly

4

AndreasPeter

EmilAllison

knows

knows knows

knows


(Neo4j) -[:IS_A]->(Graph Database)



4

- A Graph Database:



4

- A Graph Database:

- a schema-free Property Graph



4

- A Graph Database:


- perfect for complex, highly connected data



4

- A Graph Database:



- A Graph Database:



4

- A Graph Database:



- A Graph Database:

- reliable with real ACID Transactions



4

- A Graph Database:



- A Graph Database:


- scalable: billions of nodes and relationships



4

- A Graph Database:



- A Graph Database:



- fast with millons traversals / second



4

- A Graph Database:



- A Graph Database:




- Server with REST API, or Embeddable on the JVM



4

- A Graph Database:



- A Graph Database:




- Server with REST API, or Embeddable on the JVM

- higher-performance with High-Availability (read scaling)



4

JS

Clojure

Heroku

ACID TX

LuceneIndex

ServerRUNS_AS

HIGH_AVAIL.

SCALES_TO

embedded

RUNS_AS

RUNS_ON

Ruby

.net

PROVIDES

MySQL

Mongo

LICENSED_LIKE

INTEG

RATES

Master/Slave

billionsNodes >1

M trav/s


(Cypher) -[:IS_A]->(Graph Query Language)

4

• Declarative query language

- Describe what you want, not how

- Based on graph pattern matching

• Example:

START user=node:User(name=”Michael”), # index lookupMATCH (user)-[:SKILL]->()<-[rating:SKILL]-(colleague)WHERE rating.ability > 3RETURN colleague.name, count(*)ORDER BY count(*) DESCLIMIT 10


Spring Data Neo4j


Spring Data Neo4j

• Spring Data: SpringSource initiative to give Spring developers easy, consistent access to the world of NOSQL databases

• projects support different data-stores

• convenient, common access patterns

• but don‘t hide the power of the database

•Spring Data Neo4j is the integration library for Neo4j


O‘Reilly Spring Data Book„Modern Data Access for Enterprise Java“

•book by the Spring Data project leads

• introduction to Spring Data & SD-repositories

• covers all sub-projects

• consistent sample project across the sub-projects

• e-commerce domain

• I give one out at the Meetup tomorrow evening!


Spring Data Neo4j features• Annotation-based programming model• Spring Data Repository support• Neo4j Cypher Query and Traversal support

–on dynamic fields and via repository methods• Neo4j indexing support (includes fulltext and numeric

range queries)• Always possible to drop down to Neo4j APIs


Spring Data Neo4j features• Entity types stored in the graph database• Dynamic type projection (duck typing)• Cross-store support for partial JPA / graph entities• Support for JSR-303 (bean validation)• Support for the Neo4j Server (remote server and server

extension)• Neo4jTemplate with exception translation, optional

transaction management and more


Classic Neo4j domain classpublic class User { private final Node underlyingNode;

User( final Node node ) { underlyingNode = node; }

public Node getUnderlyingNode() { return underlyingNode; }

public final String getName() { return (String) underlyingNode.getProperty( “name” ); }

public void setName( final String name ) { underlyingNode.setProperty( “name”, name ); }}


@NodeEntitypublic class User { @GraphId private Long id; private String name;}

Spring Data Neo4j domain class


@NodeEntitypublic class User { @GraphId private Long id; private String name;}

@Indexed(unique = true)

Spring Data Neo4j domain class


Defining entity classes



• @NodeEntity•Represents a node in the graph•Fields saved as properties on node•References stored as relationships between nodes• Instantiated using Java ‘new’ keyword, like any POJO•Also returned by lookup mechanisms•Type information stored in the graph



• @RelationshipEntity•Represents a relationship in the graph•Fields saved as properties on relationship•Special fields for start- and end-nodes•Only returned by lookup methods


Tweet domain class

@NodeEntitypublic class Tweet { @GraphId private Long id;

@Indexed(unique=true) Long tweetId; String text;

@Fetch User sender; // eager loading

@RelatedTo(type="TAGGED") Collection<Tag> tags; @RelatedTo(type="MENTIONS") Set<User> mentions; @RelatedTo(type="SOURCE") Tweet source;}


Interface based Repositories• based on Repository infrastructure in Spring Data Commons• just define the interface and the namespace configuration• provide out-of-the-box support for

–CRUD-Operations–Index-Lookups–Traversal-Execution–Annotated Graph-Queries (Cypher)–Derived Queries–Spatial Queries

• extensible via custom methods with provided implementations


Repositories

43

interface TweetRepository extends GraphRepository<Tweet> { Tweet findByTweetId(String id); Collection<Tweet> findByTagsName(String name); @Query("start user=node({0}) match user<-[:SENDER]-tweet-[:TAGGED]->tag return distinct tag.name") Page<String> findUserTags(User user);}

<neo4j:repositories base-package="com.example.repositories"/>

@Controllerpublic class TwitterController { @Autowired TweetRepository tweetRepository;

@RequestMapping(value = "/tweet/{id}",...) public String show(Model model, @PathVariable String id) { Tweet tweet = tweetRepository.findByTweetId(id); model.addAttribute("tweet", tweet); return "/tweet/show"; }}


Neo4j-Template (I)

• well known Spring Template Pattern• Resource / Transaction Management• Convenience Methods• Nodes and Entities handling & conversion• Fluent Query Result Handling• Works also via REST with Neo4j-Server• Exception Translation


Small Example App

-uses Spring Social (minimally)

-Simple Domain: Tweets, Users, Tags

-connected Entities

-Repositories, Service

-standalone App with

-embedded Neo4j-Database

-Neo4j-Server

https://github.com/jexp/sdn-twitter




Whiteboard friendly - Abstract

:Tweet

:Tweet:User

:User

POSTED

POSTED

:TagTAGGED

TAGGED

:Tag

TAGG

ED


Whiteboard friendly - Abstract

MENTION

S

:Tweet

:Tweet:User

:User

POSTED

POSTED

:TagTAGGED

TAGGED

:Tag

TAGG

ED


Whiteboard friendly - Example

Attending the #S2GX

#SDN session by @mesirii

Looking forward to my #SDN

session



Attending the #S2GX



session@mesirii

@peterneubauer

POSTED

POSTED



MENTION

S

Attending the #S2GX



session@mesirii

@peterneubauer

POSTED

POSTED



MENTION

S

Attending the #S2GX



session@mesirii

@peterneubauer

POSTED

POSTED

#S2GXTAGGED

TAGGED

#SDN

TAGG

ED


Twitter - Graph Data Model


Spring Data Neo4j - Sample Domain Class & Service


Neo4j - Sample Graph Query

START tag1=node:Tag("tag:*obama*")> MATCH tag1<-[:TAG]-tweet-[:sender]->user, user<-[:sender]-tweet2-[:TAG]->tag2> WHERE tag2.tag =~ "(?i).*romney.*"> RETURN count(*) as cnt, tag1.tag,tag2.tag> ORDER BY count(*) DESC> LIMIT 10;

4

+--------------------------------------+| cnt | tag1.tag | tag2.tag |+--------------------------------------+| 440 | "obama" | "voteromney" || 148 | "obama" | "romneyryan2012" || 65 | "obama" | "romney" || 56 | "voteobama" | "romneyryan" || 42 | "voteobama" | "romney" |+--------------------------------------+5 rows194 ms


Real World Use-Cases with


(Neo4j) -[:LOVES]-> (Our Customers)


• hard to get permission to talk in public• complex domain models, challenging implementations• hard to get insights in implementation• challenges / feedback seldomly communicated

• this presentation is a „best effort“ mix of – actual,– inferred,– adapted information

Challenges with „Real World Use-Cases“


Airline: Media Asset Management


• Many movies, music, tv-series• Each of them with publishers, publication rights• Permissions, Broadcasting rules• Asset Management/Tracking

– also technical components, virtual containers• Streaming Box Management (What is where, Documentation)• Impact analytics of legal changes or program updates• Decision support system• Complex domain model



• Neo4j Embedded• SDN Advanced Mapping• REST Server with Spring WebMVC• Thymeleaf• Rich UI with

– CRUD for all 50 elements– dependency network visualization, with interactive navigation– advanced query generator / console

Airline: Media Asset Management (Technical)



57

Media-Box

Aircraft

Aircraft

TV-Series

Movie

Media-Box Recording

Metadata

Metadata

APPL

IES_

TOA

PP

LIE

S_T

O

INSTALLED

INST

ALLE

D

Route

Airport AirportCountry

REGULATES

Metadata

STO

RE

D_O

N


Junisphere: Complex structural document handling


• Business proccess to IT systems mapping (Exec View)• Conways Law

– Business Process Structure resembles Information Architecture it runs on

• Map and trace both–Initial and continous updates

• Optimize both sides–Servers and Services

• http://www.neotechnology.com/junispheres-usecase/



http://www.neotechnology.com/junispheres-usecase/

http://www.neotechnology.com/junispheres-usecase/


61

•Junisphere AG is a Zurich-based IT solutions provider

•Founded in 2001. Profitable. Self funded.

•Software & services.

•Novel approach to infrastructure monitoring: Starts with the end user, mapped to business processes and services, and dependent infrastructure



62

•“Business Service Management” requires mapping of complex graph, covering: business processes--> business services--> IT infrastructure

•Embed capability of storing and retrieving this information into OEM application

•Re-architecting outdated C++ application based on relational database, with Java



63

•Actively sought out a Java-based solution that could store data as a graph

•Domain model is reflected directly in the database:

•“No time lost in translation”

•“Our business and enterprise consultants now speak the same language, and can model the domain with the database on a 1:1 ratio.”

•Spring Data Neo4j strong fit for Java architecture


• dictionary contents is created by many parties• for each contribution you have to track the contracts

Publisher: Dictionary Translation Contract Handling


• Java Embedded Neo4j• Spring Data Neo4j• Integrated in Java Service Architecture• XML-Editor• Device/HW-Scanners report back model changes• Models are trees/networks,

– need to merge model-updates – with inferred updates/deltas from full models

Junisphere: eReality (technical)


66

DEFINES

Process

DEPENDS_ON

Activity

Service

Process

Activity

DEPENDS_ON

Flow Model

Device

KPI

KPI

RELATES_TO

DeviceFLOWS_TO

Function

DEPENDS_ON

RUNS_ON

KPI

DEPENDS_ON

Junisphere: eReality (Model)


67

Our requirements> Convenient modelling of our domain

> Pushing states from measurements in "real time" through the model (1000 times/s)

> Versioning of the model

> Multi tenancy support

> Different views of the same model− Availability, performance, etc. views

> Referencing properties of objects from other objects


67







Perfect fit


67







Perfect fit

Proof of concept app with 2000 traversals/s


67







Perfect fit


Proof of concept solution delivered by neo4j-team


67







Perfect fit


Proof of concept solution delivered by neo4j-teamDone with indices per tenant and filtering traversal results


67







Perfect fit


Proof of concept solution delivered by neo4j-teamDone with indices per tenant and filtering traversal resultsFeasible


67







Perfect fit


Proof of concept solution delivered by neo4j-teamDone with indices per tenant and filtering traversal resultsFeasible

Storing the properties in its own nodes


WhyOwnIt - Ownership Sharing Service


• Sharing Culture –You own things you only seldomly need–You need things you don‘t own

• Match needs and available devices/tools– add ratings, trust, recommendations– location based, close to you for pick-up

• Startup




70

Thing

Person

Category

Person

Person

HAS

OWNSCategory

NEEDS

TRUST

Person

RATED

RATED

ThingOWNS


• Classical Pay and Win Games (Poker & Co)• Spring-Web-App with Oracle backend• Add Social

– Referral System with complex rules for inferred bonuses– Multi-Level Buddy-System– Prevent Fraud– Add more social functions on top (Groups)– Recommendations

• http://www.neotechnology.com/gamesys/

Gamesys - Add Social to Gaming


http://www.neotechnology.com/gamesys/

http://www.neotechnology.com/gamesys/

Gamesys - Add Social to Gaming

74

Player

PLAYS

Game

Round

Player

PLAY

S

Player

BUDDY

Player

Player Player

BUDDYBUDDY

REFERS

REFERSBUDDY


Music Sharing/Discovery/Recommendation Service


• Recommend new artists and songs to your users– take their previous listen-history into account– look at what‘s hot amongst their friends– infer broader selection via bands and genres

• Detect duplicates using media-fingerprints– copyright infringements– de-duplicate data




77

Song

Artist

Band

Label

Song

Song

Artist Song

User

User

BYLISTENED

FOLLOWS

RAT

ED

BYPU

BLIS

HESW

ORKS_W

ITH


• Financial Services - Asset / Account Management• Bank - Permission Management and Authorization

– Entitlement• Data Software Provider - Impact analysis of changes

– Implication of changes on the company core asset: DATA– Change tracking– Root Cause Analysis

Other Use-Cases


• Networked Storage Boxes– many components, disks, controllers, power supply, sensors

• Tracking of measurement data• Impact analytics• Early detection of anomalies• Intelligent fail over• Huge number of domain entities (~450)

– generating domain classes and repositories

SAN/NAS Provider - Lifetime device tracking


• Buy and Download & Game on Demand• Fraud detection

–delayed payments, charge backs, fradulent keys are invalid• Convert rules to traversals

–co-occurrance of sudden large sales–fradulent e-mail patterns

• Combination with Oracle Infrastructure• Cross-Store

Game Retailer


Open Source Projects


Cineasts.net - a social movie database

Check Out: http://spring.neo4j.org/tutorial


https://github.com/jexp/cineasts

https://github.com/jexp/cineasts

Spring Data Neo4j Guidebook“Good Relationships”

- Spring Data Neo4j comes with a great Guide Book, featuring:

- Forewords by Rod Johnson and Emil Eifrem

- An easy to read, narrative tutorial walkthrough for cineasts.net

- A comprehensive reference for all the details

- Check it out: http://bit.ly/sd-book

- Free e-book and printed

“I’m excited about Spring Data Neo4j.... makes working with Neo4j amazingly easy, and therefore has the potential to make you more successful as a developer.”

Rod Johnson, founder of Spring


http://cineasts.net/

http://cineasts.net/

http://www.infoq.com/minibooks/good-relationships-spring-data

http://www.infoq.com/minibooks/good-relationships-spring-data

• Boston Bike Sharing Network• Import using Spring Data Neo4j• Visualization with Cypher & d3.js (Chord)

Hubway Data Challenge - Import

http://bit.ly/neo_hubway85


http://blog.neo4j.org/2012/10/using-spring-data-neo4j-for-hubway-data.html

http://blog.neo4j.org/2012/10/using-spring-data-neo4j-for-hubway-data.html

Agent Spring - Energy Market Modeling


• Research project (open source)• Model energy producers and consumers• Trace dependencies, prices, usage, demands, market

activities• Simulate impact of changes over the course of several years• Sustainable energy vs. nuclear,

private usage vs. industry demand• Used in Latvia for decision making in energy politics

Agent Smith - Energy Market Modeling

87

https://github.com/alfredas/AgentSpring




How do I get started?



4

๏ Documentation



4

๏ Documentation

• http://spring.neo4j.org


http://spring.neo4j.org



4

๏ Documentation


• Developer-Notes





4

๏ Documentation


• Developer-Notes

• Spring Data Book





4

๏ Documentation


• Developer-Notes


• Good Relationships





4

๏ Documentation


• Developer-Notes



๏ Get Neo4j





4

๏ Documentation


• Developer-Notes



๏ Get Neo4j

• http://neo4j.org/download




http://neo4j.org/download



4

๏ Documentation


• Developer-Notes



๏ Get Neo4j


๏ Participate







4

๏ Documentation


• Developer-Notes



๏ Get Neo4j


๏ Participate

• http://neo4j.org/forums






http://groups.google.com/group/neo4j



4

๏ Documentation


• Developer-Notes



๏ Get Neo4j


๏ Participate


• http://spring.neo4j.org/discussions








http://spring.neo4j.org/discussions



4

๏ Documentation


• Developer-Notes



๏ Get Neo4j


๏ Participate



• http://neo4j.meetup.com










http://neo4j.meetup.com



4

๏ Documentation


• Developer-Notes



๏ Get Neo4j


๏ Participate



• http://neo4j.meetup.com

• a session like this one ;)












The book on your seat! -“Graph Databases”

• The definitive guide on graph databases• discusses graph concepts, history and data model• compares with other NoSQL technologies• details use-cases from different domains• explains Neo4j‘s humane query language „Cypher“

Written by experts - Emil Eifrem, Ian Robinson, Jim WebberCompliments of Neo Technology


Bookmark: GraphConnect Oct 3-4 2013 San Francisco

4

$99 ticket coupon in your book


91

• Wed, Sep 11, 6-9PM• David's Restaurant,

5151 Stars and Stripes Drive• Just around the corner• Enjoy dessert, wine, beer• Chance to win a signed

Spring Data Book

Meetup - Tomorrow (Wed) EveningSpring Data Neo4j: Past, Current & Future





Thank you !Questions ?
