introducing neo4j 3.1: new security and clustering architecture
TRANSCRIPT
Introducing Neo4j 3.1New Security and Clustering Architecture
●●
●○○○
3.1
New Binary Protocol ProceduresNew Language Drivers
3.1
3.1
Massive Throughput
Data Redundancy
Data Redundancy
Data Redundancy
Data Redundancy
High Availability
High Availability
High Availability
Error!503: Service Unavailable
High Availability
Error!503: Service Unavailable
High Availability
Error!503: Service Unavailable
High Availability
Error!503: Service Unavailable
High Availability
✓
Error!503: Service Unavailable
Data RedundancyMassive Throughput High Availability
Data RedundancyMassive Throughput High Availability
3.0
Data RedundancyMassive Throughput High Availability
3.0
Bigger Clusters Consensus Commit Built-in load balancing
3.1Causal
Clustering
Replica
Core
• Small group of Neo4j databases• Fault-tolerant Consensus Commit • Responsible for data safety
Core
Writing to the Core Cluster
Neo4j Driver Neo4j Cluster
Writing to the Core Cluster
Neo4j Driver
CREATE (:User {...})
✓
Neo4j Cluster
Writing to the Core Cluster
Neo4j Driver
CREATE (:User {...})
✓
Neo4j Cluster
Writing to the Core Cluster
Neo4j Driver
CREATE (:User {...})
✓
✓
✓
Neo4j Cluster
Writing to the Core Cluster
Neo4j Driver
CREATE (:User {...})
✓
✓
✓
Neo4j Cluster
Writing to the Core Cluster
Neo4j Driver
CREATE (:User {...})
✓
✓
✓
Neo4j Cluster
Writing to the Core Cluster
Neo4j Driver
✓
✓
✓
Success
Neo4j Cluster
Writing to the Core Cluster
Neo4j Driver
✓
✓
✓
Success
Neo4j Cluster
✓
✓
• Small group of Neo4j databases• Fault-tolerant Consensus Commit • Responsible for data safety
Core
• For massive query throughput• Read-only replicas• Not involved in Consensus Commit • Disposable, suitable for auto-scaling
Replica
Propagating updates to the Replica
Neo4j Driver Neo4j Cluster
Propagating updates to the Replica
Neo4j Driver Neo4j Cluster
Write
Propagating updates to the Replica
Neo4j Driver Neo4j Cluster
Write
Reading from the Replica
Neo4j Driver Neo4j Cluster
Read
Replica
Core Updating the graph
Queries, analysis, reporting
:sysinfo
Writing an application forNeo4j Causal Clustering
App Server
Neo4j Driver
Bolt protocol
Java<dependency> <groupId>org.neo4j.driver</groupId> <artifactId>neo4j-java-driver</artifactId></dependency>
Python
pip install neo4j-driver
.NET
PM> Install-Package Neo4j.Driver
JavaScript
npm install neo4j-driver
https://neo4j.com/developer/language-guides
bolt://
GraphDatabase.driver( "bolt://aServer" )
bolt+routing://
GraphDatabase.driver( "bolt+routing://aCoreServer" )
GraphDatabase.driver( "bolt+routing://aCoreServer" )
Bootstrap: specify any core server to route load across the whole cluster
bolt+routing://
Application Server
Neo4j DriverMax
Jim
Jane
Mark
Routed write statements
driver = GraphDatabase.driver( "bolt+routing://aCoreServer" );
try ( Session session = driver.session( AccessMode.WRITE ) )
{
try ( Transaction tx = session.beginTransaction() )
{
tx.run( "MERGE (user:User {userId: {userId}})",
parameters( "userId", userId ) );
tx.success();
}
}
Routed read queries
driver = GraphDatabase.driver( "bolt+routing://aCoreServer" );
try ( Session session = driver.session( AccessMode.READ ) )
{
try ( Transaction tx = session.beginTransaction() )
{
tx.run( "MATCH (user:User {userId: {userId}})-[*]-(:Product) RETURN *",
parameters( "userId", userId ) );
tx.success();
}
}
Consistency
Register
Login
You need to login in
to continue your
purchase!
Register
Login
You need to login in
to continue your
purchase!
Username:
Password:
Create Account
Register
Login
You need to login in
to continue your
purchase!
Username:
jim_w
Password:
********
Create Account
Register
Login
You need to login in
to continue your
purchase!
Username:
Password:
Login
Username:
jim_w
Password:
********
Login
Purchase
Login Successful
Try again
No account found!Username:
jim_w
Password:
********
Login
Username:
jim_w
Password:
********
A few moments later...
✓
Login
Purchase
Login SuccessfulUsername:
jim_w
Password:
********
Login
A few moments later...
✓
Q Why didn’t this work?A Eventual Consistency
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
Create Account
App Server A Driver
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
CREATE (:User)Create Account
App Server A Driver
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
CREATE (:User)Create Account
App Server A Driver
0 1 2 3 4 5 6 7 8 9 10 11CREATE (:User)Create Account
App Server A Driver
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10 11CREATE (:User)Create Account
App Server A Driver
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
CREATE (:User)Create Account
App Server A Driver
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
CREATE (:User)Create Account
App Server A Driver
MATCH (:User)Login
App Server B Driver
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
CREATE (:User)Create Account
App Server A Driver
MATCH (:User)Login
App Server B Driver
Bookmark
• Session token• String (for portability)• Opaque to application• Represents ultimate user’s most recent
view of the graph• More capabilities to come
Let’s try again, with Causal Consistency
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
Create Account
App Server A Driver
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
CREATE (:User)Create Account
App Server A Driver
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
CREATE (:User)Create Account
App Server A Driver
0 1 2 3 4 5 6 7 8 9 10 11CREATE (:User)Create Account
App Server A Driver
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10 11CREATE (:User)Create Account
App Server A Driver
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
CREATE (:User)Create Account
App Server A Driver
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
CREATE (:User)Create Account
App Server A Driver
MATCH (:User)Login
App Server B Driver
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
CREATE (:User)Create Account
MATCH (:User)Login
App Server A
App Server B
Driver
Driver
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
CREATE (:User)Create Account
MATCH (:User)Login
App Server A
App Server B
Driver
Driver
11
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
CREATE (:User)Create Account
MATCH (:User)Login
App Server A
App Server B
Driver
Driver
11
Replica
Core Updating the graph
Queries, analysis, reporting
Neo4j 3.0 Neo4j 3.1High Availability Cluster Causal Cluster
Master-Slave architecture
Paxos consensus used for master election
Raft protocol used for leader election, membership changes and commitment of all transactions
Two part cluster: writeable Core and read-only read replicas.
Transaction committed once written durably on the master
Transaction committed once written durably on a majority of the core members
Practical deployments: 10s servers Practical deployments: 100s servers
3.1
•
•
•
# Choose LDAP connector as both authentication and authorization providerdbms.security.auth_provider=ldap
# Configure LDAP connector to point to the AD serverdbms.security.ldap.host=ldap://myactivedirectory.example.com
# In case where defined users are not allowed to search for themselves,# we can specify credentials for user with read access to all users and groupsdbms.security.ldap.authorization.use_system_account=truedbms.security.ldap.system_username=CN=admin,OU=people,DC=example,DC=comdbms.security.ldap.system_password=admin-password
# Provide details on user structure within LDAPdbms.security.ldap.user_dn_template=CN={0},OU=people,DC=example,DC=comdbms.security.ldap.authorization.user_search_base=OU=people,dc=example,dc=comdbms.security.ldap.authorization.user_search_filter=(&(objectClass=*)(CN={0}))dbms.security.ldap.authorization.group_membership_attributes=memberOf
./conf/neo4j.conf
dbms.security.ldap.authorization.group_to_role_mapping= \ "CN=Neo4j Read Only,OU=groups,DC=example,DC=com" = reader; \ "CN=Neo4j Read-Write,OU=groups,DC=example,DC=com" = publisher; \ "CN=Neo4j Schema Manager,OU=groups,DC=example,DC=com" = architect; \ "CN=Neo4j Administrator,OU=groups,DC=example,DC=com" = admin; \ "CN=Neo4j Procedures,OU=groups,DC=example,DC=com" = allowed_role
./conf/neo4j.conf
# Configure mapping between groups in the LDAP and roles in Neo4j dbms.security.ldap.authorization.group_to_role_mapping= \ “CN=Neo4j Accounting,OU=groups,DC=example,DC=com” = accounting; \ “CN=Neo4j Operator,OU=groups,DC=example,DC=com” = operator
CALL dbms.security.createRole(‘accounting’)CALL dbms.security.addRoleToUser(‘accounting’, ‘bobsmith’)
•••
••
3.1
•••
https://neo4j.com/blog/cypher-graphql-neo4j-3-1-preview/
https://neo4j.com/docs/operations-manual/beta/tools/cypher-shell/
neo4j-admin restore --from=<backup-directory> --database=<database-name> [--force]
Restore a backed up database.
neo4j-admin dump
neo4j-admin load
neo4j-admin backup [--from=<address>] --to=<backup-path> [--check-consistency] [--additional-config=<config-file-path>] [--timeout=<timeout>]
Perform a backup, over the network, from a running Neo4j server into a local copy of the database store (the backup).
neo4j-admin check-consistency --database=<database> [--additional-config=<file>] [--verbose]
Check the consistency of a database.
neo4j-admin import --mode={database|csv} --database=<database-name>
Import a collection of CSV files with --mode=csv, or a database from a pre-3.0 installation with --mode=database.
●○
●○
●○
●
