Introducing N1QL: New SQL-Based Query Language
for JSON
Keshav Murthy| @rkeshavmurthyDirector, Query | Couchbase
Team @N1QL
NoSQL
NoRDBMSNoRelationalNoSchema
NoTransactions
NoACID
NoConstraintsNoTriggers
NoScaleUp
#First get all the customerscustomerlist = self.customer.find({},{"c_id": 1, "c_name":1 })
#initialize the total listtotlist = set()
#For each customer in customer list#Create a pipeline to get total order PER CUSTOMER#Get the SUM of ol_amount for the customer#Add the customerid, customer name and total to the totlist (list of tuples)for cl in customerlist: pipe = [{'$unwind': "$order_line"}, {'$group':{'_id':cl, 'total':{'$sum':'$ol_amount'}}}] csum = db.orders.aggregate(pipeline=pipe) totlist.add(cl["c_id"], cl["c_name"], csum["total"])
#sort this full list by the total in the reverse order.data.sort(key=lambda tup: tup[2], reverse=True)
Application Developers Started Writing Complex
Code for Queries…With Serious
Performance Issues
©2015 Couchbase Inc. 5
SQL
©2015 Couchbase Inc. 6
SQLResultS
et
©2015 Couchbase Inc. 7
{ "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2842-2847-3909", "expiry" : "2019-03" } ], "Connections" : [ { "CustId" : "XYZ987", "Name" : "Joe Smith" }, { "CustId" : "PQR823", "Name" : "Dylan Smith" } { "CustId" : "PQR823", "Name" : "Dylan Smith" } ], "Purchases" : [ { "id":12, item: "mac", "amt": 2823.52 } { "id":19, item: "ipad2", "amt": 623.52 } ]}
LoyaltyInfo ResultDocuments
PricingInfo
CUSTOMER
©2015 Couchbase Inc. 8
N1QL examplesSELECT d.C_ZIP, SUM(ORDLINE.OL_QUANTITY) AS TOTALQTYFROM CUSTOMER d UNNEST ORDERS as CUSTORDERS UNNEST CUSTORDERS.ORDER_LINE AS ORDLINEWHERE d.C_STATE = ”NY”GROUP BY d.C_ZIPORDER BY TOTALQTY DESC;
INSERT INTO CUSTOMER("PQR847", {"C_ID":4723, "Name":"Joe"});
UPDATE CUSTOMER c SET c.STATE=“CA”, c.C_ZIP = 94501 WHERE c.ID = 4723;
Data in JSON
©2015 Couchbase Inc. 10
Data Comes from the Real World
©2015 Couchbase Inc. 11
Properties of Real-World Data
Rich structure– Attributes, Sub-structure
Relationships– To other data
Value evolution– Data is updated
Structure evolution– Data is reshaped
Customer
Name
DOB
Billing
Connections
Purchases
Jane Smith
Jan-30-1990
©2015 Couchbase Inc. 12
Modeling Customer Data in Relational World
Billing
Connections
Purchases
Contacts
Customer
Rich structure Normalize & JOIN
Queries Relationships
JOINS and Constraints Value evolution
Additional ROWS, JOINS Structure evolution
ALTER TABLE Application Downtime Application Migration Application Versioning
©2015 Couchbase Inc. 13
Using JSON For Real World Data
CustomerID
Name DOB
CBL2015 Jane Smith 1990-01-30
Table: Customer{ "Name" : "Jane Smith", "DOB" : "1990-01-30”}
Customer DocumentKey: CBL2015
The primary (CustomerID) becomes the DocumentKey
Column name-Column value become KEY-VALUE pair.
{ "Name" : { "fname": "Jane ", "lname": "Smith” } "DOB" : "1990-01-30”}
OR
©2015 Couchbase Inc. 14
Using JSON to store Data
CustomerID
Name DOB
CBL2015 Jane Smith 1990-01-30
Table: Customer
{ "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" } ]}
Customer DocumentKey: CBL2015
CustomerID
Type Cardnum
Expiry
CBL2015 visa 5827… 2019-03
Table: Billing
Rich Structure & Relationships– Billing Information is stored as a sub-document– There could be more than a single credit card. So, use an array.
©2015 Couchbase Inc. 15
Using JSON to store Data
CustomerID
Name DOB
CBL2015 Jane Smith 1990-01-30
Table: Customer
{ "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2542-5847-3949", "expiry" : "2018-12" } ]}
Customer DocumentKey: CBL2015
CustomerID
Type Cardnum
Expiry
CBL2015 visa 5827… 2019-03
CBL2015 master
6274… 2018-12
Table: Billing
Value evolution Simply add additional array
element Relationships are implied
©2015 Couchbase Inc. 16
Using JSON to store Data
CustomerID
ConnId Name
CBL2015 XYZ987 Joe Smith
CBL2015 SKR007 Sam Smith
Table: Connections{ "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2542-5847-3949", "expiry" : "2018-12" } ], "Connections" : [ { "ConnId" : "XYZ987", "Name" : "Joe Smith" }, { "ConnId" : ”SKR007", "Name" : ”Sam Smith" }}
Customer DocumentKey: CBL2015
Structure evolution Simply add new key-value pairs No downtime to add new KV
pairs Applications can validate data Structure evolution over time.
Relations via Reference
©2015 Couchbase Inc. 17
Using JSON to store Data{ "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2842-2847-3909", "expiry" : "2019-03" } ], "Connections" : [ { "CustId" : "XYZ987", "Name" : "Joe Smith" }, { "CustId" : "PQR823", "Name" : "Dylan Smith" } { "CustId" : "PQR823", "Name" : "Dylan Smith" } ], "Purchases" : [ { "id":12, item: "mac", "amt": 2823.52 } { "id":19, item: "ipad2", "amt": 623.52 } ]}
Customer DocumentKey: CBL2015
©2015 Couchbase Inc. 18
Models for Representing Data
Data Concern Relational Model JSON Document Model (NoSQL)
Rich Structure Multiple flat tables Constant assembly /
disassembly Documents No assembly required!
Relationships Represented Queried (SQL)
Represented Queried? Not until
now…Value Evolution Data can be updated Data can be updated
Structure Evolution
Uniform and rigid Manual change
(disruptive) Flexible Dynamic change
The N1QL Language
©2015 Couchbase Inc. 20
The N1QL Language
GoalGive developers and enterprises an expressive, powerful, and complete language for querying, transforming, and manipulating JSON data.
©2015 Couchbase Inc. 21
The N1QL LanguageQuery Features SQL N1QL
Statements SELECT, INSERT, UPDATE, DELETE, MERGE
SELECT, INSERT, UPDATE, DELETE, MERGE
Query Operations
Select, Join, Project, Subqueries
Strict Schema Strict Type checking
Select, Join, Project, Subqueries
Nest & Unnest Look Ma! No Type Mismatch
Errors! JSON keys act as columns
Schema Predetermined Columns Fully addressable JSON Flexible document structure
Data Types SQL Data types Strings, Integer, Float,
Boolean, arrays, objects Scalar conversion Functions
Query Processing
INPUT: Rows OUPUT: Rows
INPUT: JSON OUTPUT: JSON
©2015 Couchbase Inc. 22
SELECT Statement
SELECT [ DISTINCT ] …FROM … JOIN …WHERE …GROUP BY … HAVING …ORDER BY …LIMIT …OFFSET …( UNION | INTERSECT | EXCEPT ) [ ALL ] …
©2015 Couchbase Inc. 23
SELECT StatementSELECT customers.id, customers.NAME.lastname, customers.NAME.firstname Sum(orderline.amount) FROM orders UNNEST orders.lineitems AS orderline JOIN customers ON KEYS orders.custid WHERE customers.state = 'NY'GROUP BY customers.id, customers.NAME.lastname HAVING sum(orderline.amount) > 10000 ORDER BY sum(orderline.amount) DESC
• Dotted sub-document reference
• Names are CASE-SENSITIVE
• UNNEST to flatten the arrays
JOINS with Document KEY of customers
©2015 Couchbase Inc. 24
SELECT StatementSELECT * FROM ( SELECT a, b, c FROM cust WHERE x = 1 ORDER BY x LIMIT 100 OFFSET 0 UNION ALL SELECT a, b, c FROM cust WHERE y = 2 ORDER BY x LIMIT 100 OFFSET 0) AS newtab LEFT OUTER JOIN contacts ON KEYS newtab.c.contactidORDER BY a, b, cLIMIT 10 OFFSET 0
©2015 Couchbase Inc. 25
SELECT Statement Highlights Querying across relationships
– JOINs– Subqueries
Aggregation– MIN, MAX– SUM, COUNT, AVG, ARRAY_AGG [ DISTINCT ]
Combining result sets using set operators– UNION, UNION ALL, INTERSECT, EXCEPT
©2015 Couchbase Inc. 26
Data Modification Statements
UPDATE … SET … WHERE … DELETE FROM … WHERE … INSERT INTO … ( KEY, VALUE ) VALUES
… INSERT INTO … ( KEY …, VALUE … )
SELECT … MERGE INTO … USING … ON …
WHEN [ NOT ] MATCHED THEN …Note: Couchbase Server provides per-document
atomicity.
©2015 Couchbase Inc. 27
Data Modification Statements
INSERT INTO ORDERS (KEY, VALUE) VALUES ("1.ABC.X382", {"O_ID":482, "O_D_ID":3, "O_W_ID":4)};
UPDATE ORDERS SET O_CARRIER_ID = ”ABC987” WHERE O_ID = 482 AND O_D_ID = 3 AND O_W_ID = 4
DELETE FROM NEW_ORDER WHERE NO_D_ID = 291 AND NO_W_ID = 3482 AND NO_O_ID = 2483
©2015 Couchbase Inc. 28
Data Modification Statements
INSERT INTO ORDERS (KEY, VALUE) VALUES ("1.ABC.X382", {"O_ID":482, "O_D_ID":3, "O_W_ID":4)}),VALUES ("1.ABC.X383", {"O_ID":422, "O_D_ID":3, "O_W_ID":4)}),VALUES ("1.ABC.X384", {"O_ID”:782, "O_D_ID":3, "O_W_ID":4)}),VALUES ("1.ABC.X352", {"O_ID”:282, "O_D_ID":3, "O_W_ID":4)}),VALUES ("1.ABC.X362", {"O_ID”:582, "O_D_ID":3, "O_W_ID":4)}),VALUES ("1.ABC.X322", {"O_ID”:682, "O_D_ID":3, "O_W_ID":4)}),VALUES ("1.ABC.X389", {"O_ID”:982, "O_D_ID":3, "O_W_ID":4)}),VALUES ("1.ABC.X392", {"O_ID”:182, "O_D_ID":3, "O_W_ID":4)});
©2015 Couchbase Inc. 29
Index Statements
CREATE INDEX ON … DROP INDEX … EXPLAIN …Highlights Functional indexes
– on any data expression
Partial indexes
©2014 Couchbase, Inc. ©2015 Couchbase Inc. 30
N1QL Expressions from SQL
30
Literals• Primitives [ 0, ‘hello’, TRUE ]• NULL
Operators
• Arithmetic [ +, -, *, /, % ]• Logical [ AND, OR, NOT ]• Comparison [ <, <=, =, !=, >=, >, BETWEEN, IS NULL ]• Pattern matching [ LIKE ]• Conditional [ CASE ]
Scalar functions
• Numeric [ trigonometric, ROUND, TRUNC, … ]• String [ UPPER, LOWER, TRIM, SUBSTR, … ]• Date [ string and numeric dates, NOW, date arithmetic, …
]Aggregate functions • MIN, MAX, SUM, AVG, COUNT [ DISTINCT ]
Subqueries • Subqueries are full expressions
N1QL Extensions to SQL
©2015 Couchbase Inc. 32
N1QL Query Operators [ 1 of 2 ]
USE KEYS …– Direct primary key lookup bypassing index scans– Ideal for hash-distributed datastore– Available in SELECT, UPDATE, DELETE
JOIN … ON KEYS …– Nested loop JOIN using key relationships– Ideal for hash-distributed datastore
©2015 Couchbase Inc. 33
N1QL Query Operators [ 2 of 2 ]
NEST– Special JOIN that embeds external child documents
under their parent– Ideal for JSON encapsulation
UNNEST– Flattening JOIN that surfaces nested objects as top-
level documents– Ideal for decomposing JSON hierarchies
JOIN, NEST, and UNNEST can be chained in any
combination
©2014 Couchbase, Inc. ©2015 Couchbase Inc. 34
N1QL Expressions for JSON
34
Ranging over collections
• WHERE ANY c IN children SATISFIES c.age > 10 END• WHERE EVERY r IN ratings SATISFIES r > 3 END
Mapping with filtering • ARRAY c.name FOR c IN children WHEN c.age > 10 END
Deep traversal, SET, and UNSET
• WHERE ANY node WITHIN request SATISFIES node.type = “xyz” END
• UPDATE doc UNSET c.field1 FOR c WITHIN doc END
Dynamic Construction
• SELECT { “a”: expr1, “b”: expr2 } AS obj1, name FROM … // Dynamic object
• SELECT [ a, b ] FROM … // Dynamic array
Nested traversal • SELECT x.y.z, a[0] FROM a.b.c …
IS [ NOT ] MISSING • WHERE name IS MISSING
©2015 Couchbase Inc. 35
N1QL Data Types from JSON
N1QL supports all JSON data types Numbers Strings Booleans Null Arrays Objects
©2015 Couchbase Inc. 36
N1QL Data Type HandlingNon-JSON data types MISSING BinaryData type handling Date functions for string and numeric
encodings Total ordering across all data types
– Well defined semantics for ORDER BY and comparison operators
Defined expression semantics for all input data types– No type mismatch errors
The Query Servicein Couchbase Server 4.0
©2015 Couchbase Inc. 38
Couchbase Server Cluster Architecture
38
STORAGE
Couchbase Server 1
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Managed CacheStorage
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 2
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 3
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 4
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 5
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 6
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Service
Index Service
Query Service
Managed CacheStorage
Managed CacheStorage
Managed CacheStorage
Managed CacheStorage
Managed CacheStorage
©2015 Couchbase Inc. 39
Couchbase Server Cluster Service Deployment
39
STORAGE
Couchbase Server 1
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Managed CacheStorage
Data Servi
ce STORAGE
Couchbase Server 2
Managed Cache
Cluster ManagerCluster Manager
Data Servi
ce STORAGE
Couchbase Server 3
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Data Servi
ce STORAGE
Couchbase Server 4
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Query
Service
STORAGE
Couchbase Server 5
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Query
Service
STORAGE
Couchbase Server 6
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Index
Service
Managed CacheStorage
Managed CacheStorage Storage
STORAGE
Couchbase Server 6
SHARD7
SHARD9
SHARD5
SHARDSHARDSHARD
Managed Cache
Cluster ManagerCluster Manager
Index
Service
StorageManaged Cache
Managed Cache
©2015 Couchbase Inc. 40
N1QL: Query Execution Flow
Clients
1. Submit the query over REST API
8. Query result
2. Parse, Analyze, create Plan
7. Evaluate: Documents to results
3. Scan Request;
index filters6. Fetch the documents
Index
Service
Query
Service Data
Service
4. Get qualified doc keys
5. Fetch Request, doc keys
SELECT c_id, c_first, c_last, c_max FROM CUSTOMER WHERE c_id = 49165;
{ "c_first": "Joe", "c_id": 49165, "c_last": "Montana", "c_max" : 50000}
©2015 Couchbase Inc. 41
Query Execution
Client
FetchParse Plan Join FilterPre-Aggregate
Offset Limit ProjectSortAggregateScan
Query ServiceInde
x Servi
ce
Data Servi
ceData-parallel — Query latency scales with coresIn-memory, streamedPluggable architecture — datastore, indexer, client, …
Query
Resultset
Benefits of N1QLfor your Application and
Enterprise
©2015 Couchbase Inc. 43
Benefits for your Application and Enterprise Model your Data Cleanly
– Model once, query– Use both relationships and embedding
Query your Data with Flexibility– Query across structure, relationships, and datasets– Query across change and heterogeneity
Develop Rich Applications with Agility– Deliver features that transform, combine, and aggregate data– Use your favorite clients, frameworks, and interfaces
Integrate your Applications and Data– Leverage ecosystem: Connectivity, Analytics, Cloud and Packaged Apps
Community and Participation
©2015 Couchbase Inc. 45
Community and Participation
Enterprise and Community Editions of Couchbase 4.0 Community Ecosystem
– Build complementary tools, products, and drivers– Build and integrate via our open APIs
©2015 Couchbase Inc. 46
Community and Participation in Action
Couchbase N1QL Query Runner– by Warren Postma, Beta customer– 30-minute quick start with Python’s Tkinter GUI package
Getting Started
©2015 Couchbase Inc. 48
Getting Started
Test drive Couchbase Server 4.0 Now!Visit query.couchbase.comPlay with the online tutorialAsk us questions
– Couchbase forums, Stack Overflow, @N1QL
N1QL Sessionsat Couchbase Live
NewYork 2015
©2015 Couchbase Inc. 50
2:00pmLevel 2
Developer
3:50pmLevel 2
Architecture
3:50pmLevel 2
Architecture
4:35pmLevel 2
Developer
4:35pmLevel3
Architecture
Sampling of N1QL Sessions by Couchbase
Data Modeling with Couchbase Server
Evolution of Couchbase at CenterEdge: From Cache to Query
Nielsen’s Interactive Data Analytics with Couchbase N1QL
Deep Dive into N1QL with Global Secondary Indexes
N1QL and SDK Support for Java, .NET, and Node.js
Q & A
Thank you.Keshav Murthy, Couchbase
@rkeshavmurthyTeam @N1QL