Download - Squire: A polyglot application combining Neo4j, MongoDB, Ruby and Scala @ FOSDEM GraphDevRoom 2013

Squire: A polyglot application combining Neo4j, MongoDB, Ruby and Scala

Alberto Perdomo@albertoperdomo

FOSDEM 2013Graph Processing DevRoom

About me

About me

Co-Founder of Aentos

About me


Living my dream* (CNY)

About me


Living my dream* (CNY)

Web developer

Features

Features

★ Extensive movies & TV-shows data

Features


★ Info & price on streaming & rent services:

iTunes, Amazon, Hulu, Netflix...

Features




★ Social features (contacts, comments)

Features





★ Ratings

Features





★ Ratings

★ Suggestions by users

Features





★ Ratings

★ Suggestions by users

★ System recommendations

UnconventionalSystem

Architecture*

Ingredients

✔ MongoDB

Ingredients

✔ MongoDB

✔ Neo4j

Ingredients

✔ MongoDB

✔ Neo4j

✔ Ruby

Ingredients

✔ MongoDB

✔ Neo4j

✔ Ruby

✔ Scala

Ingredients

⚛ System Overview ⚛iPad

Client

Main API(Ruby)

Recommend. Engine(Scala)

MongoDB Neo4jUsersRatingsMovies and TV shows

All data

API: Ruby + MongoDB

API: Ruby + MongoDB

⚛ REST+JSON

API: Ruby + MongoDB

⚛ REST+JSON

⚛ Accounts & sessions

API: Ruby + MongoDB

⚛ REST+JSON


⚛ Movies & TV-Shows

API: Ruby + MongoDB

⚛ REST+JSON



⚛ Contacts

API: Ruby + MongoDB

⚛ REST+JSON



⚛ Contacts

⚛ Ratings

API: Ruby + MongoDB

⚛ REST+JSON



⚛ Contacts

⚛ Ratings

⚛ User suggestions & recommendations

Recommendations: Scala + Neo4j


⚛ Movies & TV-shows (IDs & rel.)



⚛ Users



⚛ Users

⚛ Ratings

Communication (RabbitMQ)

iPad Client

Main API(Ruby)


MongoDB Neo4j

RabbitMQ + JSON

JSON array of IDs

GET /suggestions

Query params in JSON

UsersRatingsMovies and TV shows

JSON array of data

Retrieve datafor response Collect IDs of suggestions

All dataGraph reset*

Issues

Issues

☠ API server waiting for query in Neo4j

Issues


☠ Query MongoDB for add. data

Issues



☠ Transform data

Issues



☠ Transform data

☠ Ruby server is blocking

Communication (DB)

iPad Client

Main API(Ruby)


MongoDB Neo4j

GET /suggestions


Redirect to Recomm.Eng. w/ hashed params

Retrieve datafor query params Collect IDs of suggestions

All data

JSON array of suggested results

Retrieve data for response

Follow redirect link

Graph reset*

Improvements

Improvements

✌ No more long blocking requests in Ruby server

Improvements


✌ Move heavy lifting to Spray/Scala (async)

Improvements


✌ Move heavy lifting to Spray/Scala (async)

✌ Add cache layer

Additional Cache Layer


* Read-Through


* Read-Through

* Stores transformed data


* Read-Through

* Stores transformed data

* Implemented using spray-caching

Total Response Time

40s → <1s

Delta Updates

Delta Updates

* API: New ratings in queue collection (MongoDB)

Delta Updates

* API: New ratings in queue collection (MongoDB)

* Recommendations:Backend checks for updates in queue and applies them in graph

Delta Updates

iPad Client

Main API(Ruby)


MongoDB Neo4j

POST /rating


Store rating +Put rating in queue Collect IDs of suggestions

Check if updatesin queue

GET /suggestions

Apply updatesif pending

Retrieve data for response

(Re)loading data in graph


1. Create new Neo4j instance



2. Import data from MongoDB




3. Hot swap




3. Hot swap

★ Allows for massive updates in the primary

database




3. Hot swap

★ Allows for massive updates in the primary

database

★ Used only for content-related data and user

Graph DB as an index for relations*

Graph DB as an index for relations*powerful queries on relations

Graph DB as an index for relations*powerful queries on relations

⁂ like full text search for strings

Design decisions: Graph


✍ Copy subsets of data in Neo4j



✍ Rebuild graph with no downtime*




✍ Massive updates blazingly fast




✍ Massive updates blazingly fast

✍ Less critical pieces

The right tools, for us! ✌

API in Ruby & MongoDB


API in Ruby & MongoDBMaintanability, flexibility, libraries,


API in Ruby & MongoDBMaintanability, flexibility, libraries,

schema-free!, development speed


Recommendations


RecommendationsScala: Performance + good interface for Neo4j


RecommendationsScala: Performance + good interface for Neo4j

Neo4j: Graph problem

Thanks! ☃

Alberto Perdomo@albertoperdomo

❤ Credits ❤

Entering Hyperspace by Éole Wind

http://www.flickr.com/photos/eole/380316678/

http://www.flickr.com/photos/eole/380316678/

Download - Squire: A polyglot application combining Neo4j, MongoDB, Ruby and Scala @ FOSDEM GraphDevRoom 2013

Top Related