Download - Dex Technical Seminar (April 2011)
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Technical Seminar
Sergio Gómez
April, 2011
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Introduction
Basic Concepts
Database construction
Validate construction
Query database
Graph algorithms
Script loaders
Tips & tricks
Index
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Index
Introduction
Basic Concepts
Database construction
Validate construction
Query database
Graph algorithms
Script loaders
Tips & tricks
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Introduction
Graph database
Graph databases focus on the structure of the model. Nodes and edges instead of tables. Implicit relation in the model.
DEX is a programming librarywhich allows to manage agraph database.
Very large datasets. High performance
query processing.
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
IntroductionDEX Definition
Persistent and temporary graph management programming library.
Data model: Typed and attributed directed multigraph.
Typed: Node and edge instances belong to a type (label). Attributed: Node and edge instances may have attribute
values. Directed: Edge can be directed or undirected. Multigraph: Multiple edges between two nodes.
Type of edges: Materialized: directed and undirected. Virtual: constrained by the values of two attributes
(foreign keys) Just for navigation
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
IntroductionGraph Model
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Introduction
Basic Concepts
Database construction
Validate construction
Query database
Graph algorithms
Script loaders
Tips & tricks
Index
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Basic Concepts
Java library: jdex.jar public API Native library
Linux: libjdex.so Windows: jdex.dll
System requirements: Java Runtime Environment, v1.5 or higher. Operative system:
Windows – 32 bits (64 bits to be supported in new release)
Linux – 32 and 64 bits Soon to announce Mac OS
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Basic Concepts
DEX Session1
N
DbGraph11
RGraph
1
N
Graph
Graph factory Persistent DB
TemporaryObjects
1
N
GraphPool
1
N
Set of OIDs
Class Diagram
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Basic ConceptsMain methods
DEX
open(filename) GraphPoolcreate(filename) GraphPoolclose()
GraphPoolnewSession() Session
Session
getDbGraph() DbGraphnewGraph() Rgraphclose()
Graph
newNodeType(name) intnewEdgeType(name) intnewNode(type) longnewEdge(type) longnewAttribute(type, name) longsetAttribute(oid, attr, value)getAttribute(oid, attr) value
select(type) Objectsselect(attr, op, value) Objectsexplode(oid, type) Objects
Objects.Iterator
hasNext() booleannext() long
Objects
add(long)exists(long)
copy(objs)union(objs)Intersection(objs)difference(objs)
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Index
Introduction
Basic Concepts
Database construction
Validate construction
Query database
Graph algorithms
Script loaders
Tips & tricks
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Database constructionGraph
DEX: Loads library and manages graph db instances. Graph Pool: Manages a graph db instance. Session: Manages a set of queries and temporary data.
Nodes & Edges Type:
DEX identifier (integer) Public identifier (string)
Instance: DEX identifier (long) – OID belongs to a type
Attributes Attribute:
DEX identifier (long) public identifier (string) Scope: type or global
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Database construction
GraphPool DEX#create(String name)Creates a new graph database instance.Returns the GraphPool instance to manage the new graph database instance.
GraphPool DEX#open(String name)Opens an existing graph database instance.Returns the GraphPool instance to manage the existing graph database instance.
Session GraphPool#newSession()Initiates a new user Session.
DbGraph Session#getDbGraph()Gets the Graph instance representing the persistent graph
database.
Create a graph database
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Database construction
import edu.upc.dama.dex.core.*;…DEX dex = new DEX();GraphPool gpool = dex.create(“C:/image.dex”);Session s = gpool.newSession();……s.close();gpool.close();dex.close();
Create a graph database example
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Database construction
int Graph#newNodeType(String name)Creates a new node type with the given unique name.Returns the DEX node type identifier.
long Graph#newNode(int nodeType)Creates a new node belonging to the given node type. Returns the DEX object identifier.
Add nodes
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Database construction
int Graph#newEdgeType(String name, bool directed)Creates a new edge type with the given unique name. Directed
or undirected edge type. Returns the DEX edge type identifier.
int Graph#newRestrictedEdgeType(String name, int srcNodeType, int dstNodeType)
Creates a new directed edge type with the given unique name.(Integrity restriction) Source and destionation of the edge
instances are restricted to the given node types.Returns the DEX edge type identifier.
long Graph#newEdge(long tail, long head, int edgeType)
Creates a new edge belonging to the given edge type. Tail is the source and head is the target (iff directed).Returns the DEX edge identifier.
Add edges
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
…DbGraph dbg = s.getDbGraph();
int person = dbg.newNodeType(“PERSON”);long p1 = dbg.newNode(person);long p2 = dbg.newNode(person);long p3 = dbg.newNode (person);
int friend = dbg.newUndirectedEdgeType(“FRIEND”);long e1 = dbg.newEdge(p1, p2, friend);long e2 = dbg.newEdge(p2, p3, friend);int loves = dbg.newEdgeType(“LOVES”);long e3 = dbg.newEdge(p1, p3, loves);…
Add nodes and edges example
Database construction
p1
p2
p3
p1
p2
p3
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Database construction
class ValueEncapsulates a value and its domain (data type): String, Integer, Long, Double, Boolean, Timestamp. Use them to set and get attribute values for the objects.
long Graph#newAttribute(int type, String name, short dataType, short kind)Creates a new attribute with the given unique name for the given node or edge type. Returns the DEX attribute identifier.
“dataType” can be: Value#STRING, Value#INT, Value#LONG, Value#DOUBLE, Value#BOOL, Value#TIMESTAMP.“kind” can be:
Graph#ATTR_KIND_BASIC. Basic attribute (just set and get values).Grahp#ATTR_KIND_INDEXED. Indexed attribute (set and get values as well as select operations)Graph#ATTR_KINDUNIQUE. Indexed attribute. Unique (PK).
Manage attributes
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Database construction
Graph#setAttribute(long oid, long attr, Value v)Sets the given Value for the given attribute to the given object identifier. Given attribute must be defined for the object’s type. Value ‘s data type must match attribute’s data type or NULL.
Graph#getAttribute(long oid, long attr, Value v)Gets the Value for the given attribute and for the given object identifier. Given attribute must be defined for the object’s type.
Manage attributes
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Database construction
…long name = dbg.newAttribute(person, “NAME”, Value.STRING);long age= dbg.newAttribute(person, “AGE”, Value.INT);Value v = new Value();
dbg.setAttribute(p1, age, v.setInt(18));dbg.setAttribute(p2, name, v.setString(“KELLY"));dbg.setAttribute(p3, name, v.setString(“MARY"));…
long since = dbg.newAttribute(friend, “SINCE”, Value.INT);
dbg.setAttribute(e1, since, v.setInt(2000));dbg.setAttribute(e2, since, v.setInt(1995));…
JOHN18
KELLY
MARY
1995
2000JOHN18
KELLY
MARY
Manage attributes example
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Database construction
…int phones = dbg.newEdgeType("phones");long when = dbg.newAttribute(phones, "when", Value.STRING);
long e4 = dbg.newEdge(p1, p3, phones);dbg.setAttribute(e4, when, v.setString("4pm")));
long e5 = dbg.newEdge(p1, p3, phones);dbg.setAttribute(e5, when, v.setString("5pm"));
long e6 = dbg.newEdge(p3, p2, phones);dbg.setAttribute(e6, when, v.setString("6pm"));…
System.out.println(dbg.getAttribute(p1, name));System.out.println(dbg.getAttribute(e4, when));System.out.println(dbg.getAttribute(e5, when));
1995
2000JOHN18
KELLY
MARY
4pm5pm
6pm
Manage attributes example
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Index
Introduction
Basic Concepts
Database construction
Validate construction
Query database
Graph algorithms
Script loaders
Tips & tricks
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Validate construction
GraphPool#dumpData(File f)Dumps a summary of the logical content of the graph database.
GraphPool#dumpStorage(File f)Dumps internal information about storage content of the graph database.
Graph#export(PrintWriter pw, short kind, Export e)Exports the graph to an external format.“kind” can be: GRAPHVIZ or YGRAPHML.Export defines the visualization (if null, default export).
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Validate construction
import java.io.PrintWriter;import java.io.FileWriter;…
//Default Export is used in the example by stating null. //Personalized export can be created implementing class Export PrintWriter stdOut = new PrintWriter(
new FileWriter("out.graphml"));dbg.export(stdOut, Type.YGRAPHML, null);pw.close();
//Dump data filegpool.dumpData(“out.data”);
//Dump internal storage filegpool.dumpStorage(“out.storage”);
Validate construction example
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Validate constructionData file example
-- 1:2 = Friend-- 2:3 = Loves-- 3:4 = phonesedge types = 3-- 4:1 = Person------- #00000400 [Person] ATTR=1 //Person//age{18} TO [Friend] #00000401 [Person] ATTR=1 //Person//name{kelly} TO [Loves] #00000402 [Person] ATTR=1 //Person//name{mary} TO [phones] #00000402 [Person] ATTR=1 //Person//name{mary} #00000402 [Person] ATTR=1 //Person//name{mary}…------- 3 nodes.node types = 4
Edge types
# of edge types
Node of type person
Value for “age” attribute
# of node types
“Phones” out-going edges
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Validate constructionInternal storage file example
2011-02-09 12:12:45.330 ...SIZE: 262KB / 2688KB NODES: 3 EDGES: 6 1. ... - NODES Person : N=3... ATTR DATA: 4 ... - //Friend//since[INDEXED|INDEXED LINK] 2/2... [1995] - [2000] - //Person//age[INDEXED|INDEXED LINK] 1/1 [18] - [18]...
Number of edgesNumber of nodes
Number of “Person”s
Number of attributesName and kind of the attribute
Minimum and maximum value
# different values / # values
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Validate construction
Export file example, yEd Graph Editor<?xml version="1.0" encoding="ISO-8859-1"?><graphml xmlns="http://graphml.graphdrawing.org/xmlns/graphml"
xmlns:y="http://www.yworks.com/xml/graphml" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns/graphml http://www.yworks.com/xml/schema/graphml/1.0/ygraphml.xsd"><key id="d0" for="node" yfiles.type="nodegraphics"/><key id="d1" for="edge" yfiles.type="edgegraphics"/><graph id="DB" edgedefault="undirected">
<node id="0"><data key="d0" ><y:ShapeNode><y:Geometry height="19.0" width="20"/><y:Fill color="#a5c3f6"/><y:BorderStyle color="#a5c3f6"/><y:NodeLabel fontSize="10" textColor="#000000">1024</y:NodeLabel><y:Shape type="rectangle"/> </y:ShapeNode></data></node>…</graph>
</graphml>
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Index
Introduction
Basic Concepts
Database construction
Validate construction
Query database
Graph algorithms
Script loaders
Tips & tricks
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Query Database
Objects
class ObjectsUnordered set of OIDs for large collections.Implements Set<Long>, Iterable<Long>, Closeable.
boolean Objects#add(long oid)Adds the given OID to the collection.Returns true if added, false if the OID was already into the collection.
boolean Objects#exists(long oid)Returns true if the given OID exists into the collection, false otherwise.
boolean Objects#remove(long oid)Removes the given OID from the collection.Returns true if removed or false if the OID was not into the collection.
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Query Database
Objects
long Objects#union(Objects objs)this = this UNION objsReturns the new size of the collection.
long Objects#intersection(Objects objs)this = this INTERSECTION objsReturns the new size of the collection.
long Objects#difference(Objects objs)this = this DIFFERENCE objsReturns the new size of the collection.
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Query Database
Objects Graph#select(int t)Retrieves object identifiers belonging to the given node or edge type.
Objects Graph#select(long attr, short op, Value v)
Retrieves object identifiers which satisfay the query.“op” can be: Graph#OPERATION_{EQ|NE|GT|GE|LT|LE|LIKE|ERE}
long Graph#findObj(long attr, Value v)Randomly retrieves an object identifier which has the given value for the given attribute (or INVALID_OID if not found).Useful for unique atributes.
Retrieve data
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Query Database
Objects Graph#explode(long oid, int edgeType, short direction)
Retrieves out-going or in-going edges (or both) from or to the given node identifier and for the given edge type.“direction” can be: Graph#EDGES_IN, Graph#EDGES_OUT, Graph#EDGES_BOTH.
Objects Graph#neighbors(long oid, int edgeType, short direction)
Retrieves neighbor nodes to the given node identifier which can be reached through the given edge type and direction.“direction” can be: Graph#EDGES_IN, Graph#EDGES_OUT, Graph#EDGES_BOTH.
Navigation
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Query Database
…DbGraph dbg = s.getDbGraph();Objects persons = dbg.select(person);Objects.Iterator it = persons.iterator();while (it.hasNext()) {
long p = it.next();String name = dbg.getAttribute(p, name).toString();
}it.close();persons.close();…
JOHN18
KELLY
MARY
JOHN18
KELLY
MARY
1995
4pm
5pm
6pm
2000
Retrieve data example
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Query Database
…Objects objs1 = dbg.select(when, >=, 5pm);// objs1 = { e5, e6 }Objects objs2 = dbg.explode(p1, phones, OUT);// objs2 = { e4, e5 }Objects objs = objs1.intersection(objs2);// objs = { e5, e6 } ∩ { e4, e5 } = { e5 }…objs.close();objs1.close();objs2.close();…
JOHN18
KELLY
MARY
JOHN18
KELLY
MARY
1995
4pm
5pm
6pm
2000
Navigation & Objects operations example
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Index
Introduction
Basic Concepts
Database construction
Validate construction
Query database
Graph algorithms
Script loaders
Tips & tricks
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Graph Algorithms
Traversals
class TraversalImplements Iterator<Long>: hasNext(), next().Traverses the graph from a given node identifier.Node and edge types can be filtered out.
Class TraversalBFSUses a BFS (breadth-first search ) algorithm.
Class TraversalDFSUses a DFS (depth-first search ) algorithm.
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Graph Algorithms
Traversals example
Graph graph = … // graph instancelong idsource = … // source nodeTraversalBFS bfs = new TraversalBFS(graph, idsource);
// Adding the allowed edge types for traversing the graph fs.addEdge(graph.findType("street"), Algorithm.NAVIGATION_FORWARD);bfs.addEdge(graph.findType("road"), Algorithm.NAVIGATION_FORWARD);
// Adding the allowed node types for traversing the graphbfs.addAllNodes();
// Running the algorithm long idnode;while (bfs.hasNext()) {
idnode = bfs.next();}
// Closing the traversalbfs.close();
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Graph Algorithms
Shortest paths
class SinglePairShortestPathFinds the shortest path between two given node identifiers.Node and edge types can be filtered out.
Class SinglePairShortestPathBFSUses a BFS traversal to find the shortest path.For un-weighted edges.
Class SinglePairShortestPathDijkstraUses a Dijkstra algorithm to find the shortest path.For weighted edges.
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Graph Algorithms
ShortestPath example
Graph graph = … // graph instancelong idsource = … // source nodelong iddestination = … // destination nodeSinglePairShortestPathBFS sp =
new SinglePairShortestPathBFS(graph, idsource, iddestination);
// Adding the allowed edge types for traversing the graphsp.addEdge(graph.findType("street"), Algorithm.NAVIGATION_FORWARD);sp.addEdge(graph.findType("road"), Algorithm.NAVIGATION_UNDIRECTED);
// Setting the maximum hopssp.setMaximumHops(7);
// Running the algorithmsp.run();
// Retrieving the generated resultsif (sp.existsShortestPath()) {
long[] spAsNodes = sp.getPathAsNodes();long[] spAsEdges = sp.getPathAsEdges();int hopsDone = sp.getCost();
}// Closing the instancesp.close();
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Graph Algorithms
Connected components
class ConnectivityRetrieves connected components of the given graph.Node and edge types can be filtered out.Connected components can be materialized into an attribute.
class StrongConnectivityGabowRetrieves strong connected components in a directed graph.Uses the Gabow algorithm.
class WeakConnectivityDFSRetrieves weakly connected components in a undirected graph or in a directed graph (ignoring directions).Uses a DFS algorithm.
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Index
Introduction
Basic Concepts
Database construction
Validate construction
Query database
Graph algorithms
Script loaders
Tips & tricks
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Script loaders
CREATE DBGRAPH alias INTO filename
CREATE NODE node_type_name "(“[attribute_name
(INT|LONG|DOUBLE|STRING|BOOLEAN|TIMESTAMP|TEXT)[INDEXED|UNIQUE|BASIC]
, ...]")“
CREATE [UNDIRECTED|VIRTUAL] EDGE edge_type_name[FROM node_type_name[.attribute_name] TO node_type_name[.attribute_name]] "(“
[attribute_name(INT|LONG|DOUBLE|STRING|BOOLEAN|TIMESTAMP|TEXT) [INDEXED|UNIQUE|BASIC]
, ...]") [MATERIALIZE NEIGHBORS]"
Schema definition
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Script loaders
LOAD NODES file_nameCOLUMNS attribute_name [alias_name], …INTO node_type_name[IGNORE (attribute_name|alias_name), …][FIELDS
[TERMINATED char][ENCLOSED char][ALLOW_MULTILINE]]
[FROM num][MAX num][MODE (ROWS|COLUMNS [SPLIT [PARTITIONS num]])]
Load nodes
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Script loaders
LOAD EDGES file_nameCOLUMNS attribute_name [alias_name], …INTO node_type_name[IGNORE (attribute_name|alias_name), …]WHERETAIL (attribute_name|alias_name) = node_type_name.attribute_nameHEAD (attribute_name|alias_name) = node_type_name.attribute_name[FIELDS
[TERMINATED char][ENCLOSED char][ALLOW_MULTILINE]]
[FROM num][MAX num][MODE (ROWS|COLUMNS [SPLIT [PARTITIONS num]])]
Load edges
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Index
Introduction
Basic Concepts
Database construction
Validate construction
Query database
Graph algorithms
Script loaders
Tips & tricks
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Tips & tricks
Shell java –cp jdex.jar edu.upc.dama.dex.shell.Shell
Explore database: Schema Instances
Execute basic queries: Select Explode Neighbors
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Tips & tricks
Index or not? Attributes:
Attributes used at select operations must be indexed.
Optionally, index once the attribute has been created/loaded.
Neighbors: Edge types used at neighbors
operations it is recommended to be indexed.
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Tips & tricks
String attributes String
Maximum length = 2048. Indexed or not.
Select [==, !=, >, >=, <, <=, ERE] Text (Character large object)
Unlimited length. Not Indexed.
Just get and set. Streaming read and write operations.
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Tips & tricks
Others: DB cross-platform format.
32 – 64 bits, OS independent. Just considere platform endianess.
Read only mode. Configuration:
edu.upc.dama.dex.utils.DEXConfig Set the maximum memory usage.
0 means unlimited. License.
No license means evaluation version.
DEX
Gra
ph
Data
base
http://www.sparsity-technologies.com
Thanks for yourattention
Any questions?
SPARSITY-TECHNOLOGIESJordi Girona, 1-3, Edifici K2M
08034 [email protected]
http://www.sparsity-technologies.com