This session introduces N1QL and sets the stage for the rich selection of N1QL-related sessions at Couchbase Connect 2015. N1QL is SQL for JSON, extending the querying power of SQL with the modeling flexibility of JSON. In this session, you will get an introduction to the N1QL language, architecture, and ecosystem, and you will hear the benefits of N1QL for developers and for enterprises.

1. Introducing N1QL:
New SQL-Based Query Language for JSON
Keshav Murthy| @rkeshavmurthy
Director, Query | Couchbase
Team @N1QL

2. NoSQL
NoRDBMS
NoRelational
NoSchema
NoTransactions
NoACID
NoConstraints
NoTriggers
NoScaleUp

3. #First get all the customers
customerlist = self.customer.find({},{"c_id": 1, "c_name":1 })
#initialize the total list
totlist = set()
#For each customer in customer list
#Create a pipeline to get total order PER CUSTOMER
#Get the SUM of ol_amount for the customer
#Add the customerid, customer name and total to the totlist (list of tuples)
for cl in customerlist:
pipe = [{'$unwind': "$order_line"},
{'$group':{'_id':cl, 'total':{'$sum':'$ol_amount'}}}]
csum = db.orders.aggregate(pipeline=pipe)
totlist.add(cl["c_id"], cl["c_name"], csum["total"])
#sort this full list by the total in the reverse order.
data.sort(key=lambda tup: tup[2], reverse=True)
Application Developers Started
Writing Complex Code for Queries
…With Serious Performance Issues

5. ©2015 Couchbase Inc. 5
SQL

6. ©2015 Couchbase Inc. 6
ResultSet

7. ©2015 Couchbase Inc. 7
{
"Name" : "Jane Smith",
"DOB" : "1990-01-30",
"Billing" : [
{
"type" : "visa",
"cardnum" : "5827-2842-2847-3909",
"expiry" : "2019-03"
},
{
"type" : "master",
"cardnum" : "6274-2842-2847-3909",
"expiry" : "2019-03"
}
],
"Connections" : [
{
"CustId" : "XYZ987",
"Name" : "Joe Smith"
},
{
"CustId" : "PQR823",
"Name" : "Dylan Smith"
}
{
"CustId" : "PQR823",
"Name" : "Dylan Smith"
}
],
"Purchases" : [
{ "id":12, item: "mac", "amt": 2823.52 }
{ "id":19, item: "ipad2", "amt": 623.52 }
]
}
LoyaltyInfo ResultDocuments
PricingInfo
CUSTOMER

8. ©2015 Couchbase Inc. 8
N1QL examples
SELECT d.C_ZIP, SUM(ORDLINE.OL_QUANTITY) AS TOTALQTY
FROM CUSTOMER d
UNNEST ORDERS as CUSTORDERS
UNNEST CUSTORDERS.ORDER_LINE AS ORDLINE
WHERE d.C_STATE = ”NY”
GROUP BY d.C_ZIP
ORDER BY TOTALQTY DESC;
INSERT INTO CUSTOMER("PQR847", {"C_ID":4723, "Name":"Joe"});
UPDATE CUSTOMER c SET c.STATE=“CA”, c.C_ZIP = 94501
WHERE c.ID = 4723;

9. Data in JSON

10. ©2015 Couchbase Inc. 10
Data Comes from the Real World

11. ©2015 Couchbase Inc. 11
Properties of Real-World Data
 Rich structure
– Attributes, Sub-structure
 Relationships
– To other data
 Value evolution
– Data is updated
 Structure evolution
– Data is reshaped
Customer
Name
DOB
Billing
Connections
Purchases

12. ©2015 Couchbase Inc. 12
Modeling Customer Data in RelationalWorld
Billing
ConnectionsPurchases
Contacts
Customer
 Rich structure
 Normalize & JOIN Queries
 Relationships
 JOINS and Constraints
 Value evolution
 Additional ROWS, JOINS
 Structure evolution
 ALTERTABLE
 Application Downtime
 Application Migration
 ApplicationVersioning

13. ©2015 Couchbase Inc. 13
Using JSON For RealWorld Data
CustomerID Name DOB
CBL2015 Jane Smith 1990-01-30
Table: Customer
{
"Name" : "Jane Smith",
"DOB" : "1990-01-30”
}
Customer DocumentKey: CBL2015
 The primary (CustomerID) becomes the
DocumentKey
 Column name-Column value become
KEY-VALUE pair.
{
"Name" : {
"fname": "Jane ",
"lname": "Smith”
}
"DOB" : "1990-01-30”
}
OR

14. ©2015 Couchbase Inc. 14
Using JSON to store Data
CustomerID Name DOB
CBL2015 Jane Smith 1990-01-30
Table: Customer
{
"Name" : "Jane Smith",
"DOB" : "1990-01-30",
"Billing" : [
{
"type" : "visa",
"cardnum" : "5827-2842-2847-3909",
"expiry" : "2019-03"
}
]
}
Customer DocumentKey: CBL2015
CustomerID Type Cardnum Expiry
CBL2015 visa 5827… 2019-03
Table: Billing
 Rich Structure & Relationships
– Billing Information is stored as a sub-document
– There could be more than a single credit card. So, use an array.

15. ©2015 Couchbase Inc. 15
Using JSON to store Data
CustomerID Name DOB
CBL2015 Jane Smith 1990-01-30
Table: Customer
{
"Name" : "Jane Smith",
"DOB" : "1990-01-30",
"Billing" : [
{
"type" : "visa",
"cardnum" : "5827-2842-2847-3909",
"expiry" : "2019-03"
},
{
"type" : "master",
"cardnum" : "6274-2542-5847-3949",
"expiry" : "2018-12"
}
]
}
Customer DocumentKey: CBL2015
CustomerID Type Cardnum Expiry
CBL2015 visa 5827… 2019-03
CBL2015 master 6274… 2018-12
Table: Billing
 Value evolution
 Simply add additional array element
 Relationships are implied

16. ©2015 Couchbase Inc. 16
Using JSON to store Data
CustomerID ConnId Name
CBL2015 XYZ987 Joe Smith
CBL2015 SKR007 Sam Smith
Table: Connections
{
"Name" : "Jane Smith",
"DOB" : "1990-01-30",
"Billing" : [
{
"type" : "visa",
"cardnum" : "5827-2842-2847-3909",
"expiry" : "2019-03"
},
{
"type" : "master",
"cardnum" : "6274-2542-5847-3949",
"expiry" : "2018-12"
}
],
"Connections" : [
{
"ConnId" : "XYZ987",
"Name" : "Joe Smith"
},
{
"ConnId" : ”SKR007",
"Name" : ”Sam Smith"
}
}
Customer DocumentKey: CBL2015
 Structure evolution
 Simply add new key-value pairs
 No downtime to add new KV pairs
 Applications can validate data
 Structure evolution over time.
 Relations via Reference

17. ©2015 Couchbase Inc. 17
Using JSON to store Data
{
"Name" : "Jane Smith",
"DOB" : "1990-01-30",
"Billing" : [
{
"type" : "visa",
"cardnum" : "5827-2842-2847-3909",
"expiry" : "2019-03"
},
{
"type" : "master",
"cardnum" : "6274-2842-2847-3909",
"expiry" : "2019-03"
}
],
"Connections" : [
{
"CustId" : "XYZ987",
"Name" : "Joe Smith"
},
{
"CustId" : "PQR823",
"Name" : "Dylan Smith"
}
{
"CustId" : "PQR823",
"Name" : "Dylan Smith"
}
],
"Purchases" : [
{ "id":12, item: "mac", "amt": 2823.52 }
{ "id":19, item: "ipad2", "amt": 623.52 }
]
}
Customer
DocumentKey:
CBL2015

18. ©2015 Couchbase Inc. 18
Models for Representing Data
Data Concern Relational Model
JSON Document Model
(NoSQL)
Rich Structure
 Multiple flat tables
 Constant assembly / disassembly
 Documents
 No assembly required!
Relationships
 Represented
 Queried (SQL)
 Represented
 Queried? Not until now…
Value Evolution  Data can be updated  Data can be updated
Structure Evolution
 Uniform and rigid
 Manual change (disruptive)
 Flexible
 Dynamic change

19. The N1QL Language

20. ©2015 Couchbase Inc. 20
The N1QL Language
Goal
Give developers and enterprises an expressive,
powerful, and complete language for querying,
transforming, and manipulating JSON data.

21. ©2015 Couchbase Inc. 21
The N1QL Language
Query Features SQL N1QL
Statements
 SELECT, INSERT, UPDATE, DELETE,
MERGE
 SELECT, INSERT, UPDATE, DELETE,
MERGE
Query Operations
 Select, Join, Project, Subqueries
 Strict Schema
 StrictType checking
 Select, Join, Project, Subqueries
 Nest & Unnest
 Look Ma! NoType Mismatch Errors!
 JSON keys act as columns
Schema  Predetermined Columns
 Fully addressable JSON
 Flexible document structure
DataTypes  SQL Data types
 Strings, Integer, Float, Boolean,
arrays, objects
 Scalar conversion Functions
Query Processing
 INPUT: Rows
 OUPUT: Rows
 INPUT: JSON
 OUTPUT: JSON

22. ©2015 Couchbase Inc. 22
SELECT Statement
SELECT [ DISTINCT ] …
FROM … JOIN …
WHERE …
GROUP BY … HAVING …
ORDER BY …
LIMIT …
OFFSET …
( UNION | INTERSECT | EXCEPT ) [ ALL ] …

23. ©2015 Couchbase Inc. 23
SELECT Statement
SELECT customers.id,
customers.NAME.lastname,
customers.NAME.firstname
Sum(orderline.amount)
FROM orders UNNEST orders.lineitems AS orderline
JOIN customers ON KEYS orders.custid
WHERE customers.state = 'NY'
GROUP BY customers.id,
customers.NAME.lastname
HAVING sum(orderline.amount) > 10000
ORDER BY sum(orderline.amount) DESC
• Dotted sub-document reference
• Names are CASE-SENSITIVE
• UNNEST to flatten the arrays
JOINS with Document KEY of
customers

24. ©2015 Couchbase Inc. 24
SELECT Statement
SELECT *
FROM
(
SELECT a,
b,
c
FROM cust
WHERE x = 1
ORDER BY x LIMIT 100 OFFSET 0
UNION ALL
SELECT a,
b,
c
FROM cust
WHERE y = 2
ORDER BY x LIMIT 100 OFFSET 0) AS newtab
LEFT OUTER JOIN contacts ON KEYS newtab.c.contactid
ORDER BY a, b, c
LIMIT 10 OFFSET 0

25. ©2015 Couchbase Inc. 25
SELECT Statement Highlights
 Querying across relationships
– JOINs
– Subqueries
 Aggregation
– MIN, MAX
– SUM, COUNT, AVG,ARRAY_AGG [ DISTINCT ]
 Combining result sets using set operators
– UNION, UNIONALL, INTERSECT, EXCEPT

26. ©2015 Couchbase Inc. 26
Data Modification Statements
 UPDATE … SET … WHERE …
 DELETE FROM … WHERE …
 INSERT INTO … ( KEY,VALUE ) VALUES …
 INSERT INTO … ( KEY …,VALUE … ) SELECT …
 MERGE INTO … USING … ON …
WHEN [ NOT ] MATCHEDTHEN …
Note: Couchbase Server provides per-document atomicity.

27. ©2015 Couchbase Inc. 27
Data Modification Statements
INSERT INTO ORDERS (KEY, VALUE)
VALUES ("1.ABC.X382", {"O_ID":482, "O_D_ID":3, "O_W_ID":4)};
UPDATE ORDERS
SET O_CARRIER_ID = ”ABC987”
WHERE O_ID = 482 AND O_D_ID = 3 AND O_W_ID = 4
DELETE FROM NEW_ORDER
WHERE NO_D_ID = 291 AND NO_W_ID = 3482
AND NO_O_ID = 2483

28. ©2015 Couchbase Inc. 28
Data Modification Statements
INSERT INTO ORDERS (KEY, VALUE)
VALUES ("1.ABC.X382", {"O_ID":482, "O_D_ID":3, "O_W_ID":4)}),
VALUES ("1.ABC.X383", {"O_ID":422, "O_D_ID":3, "O_W_ID":4)}),
VALUES ("1.ABC.X384", {"O_ID”:782, "O_D_ID":3, "O_W_ID":4)}),
VALUES ("1.ABC.X352", {"O_ID”:282, "O_D_ID":3, "O_W_ID":4)}),
VALUES ("1.ABC.X362", {"O_ID”:582, "O_D_ID":3, "O_W_ID":4)}),
VALUES ("1.ABC.X322", {"O_ID”:682, "O_D_ID":3, "O_W_ID":4)}),
VALUES ("1.ABC.X389", {"O_ID”:982, "O_D_ID":3, "O_W_ID":4)}),
VALUES ("1.ABC.X392", {"O_ID”:182, "O_D_ID":3, "O_W_ID":4)});

29. ©2015 Couchbase Inc. 29
Index Statements
 CREATE INDEX ON …
 DROP INDEX …
 EXPLAIN …
Highlights
 Functional indexes
– on any data expression
 Partial indexes

30. ©2015 Couchbase Inc. 30
N1QL Expressions from SQL
©2014 Couchbase, Inc.
30
Literals
• Primitives [ 0, ‘hello’,TRUE ]
• NULL
Operators
• Arithmetic [ +, -, *, /, % ]
• Logical [AND,OR, NOT ]
• Comparison [ <, <=, =, !=, >=, >, BETWEEN, IS NULL ]
• Pattern matching [ LIKE ]
• Conditional [ CASE ]
Scalar functions
• Numeric [ trigonometric, ROUND,TRUNC, … ]
• String [ UPPER, LOWER,TRIM, SUBSTR, … ]
• Date [ string and numeric dates, NOW, date arithmetic, … ]
Aggregate functions • MIN, MAX, SUM, AVG,COUNT [ DISTINCT ]
Subqueries • Subqueries are full expressions

31. N1QL Extensions to SQL

32. ©2015 Couchbase Inc. 32
N1QL Query Operators [ 1 of 2 ]
 USE KEYS …
– Direct primary key lookup bypassing index scans
– Ideal for hash-distributed datastore
– Available in SELECT, UPDATE, DELETE
 JOIN … ON KEYS …
– Nested loop JOIN using key relationships
– Ideal for hash-distributed datastore

33. ©2015 Couchbase Inc. 33
N1QL Query Operators [ 2 of 2 ]
 NEST
– Special JOIN that embeds external child documents under their parent
– Ideal for JSON encapsulation
 UNNEST
– Flattening JOIN that surfaces nested objects as top-level documents
– Ideal for decomposing JSON hierarchies
JOIN, NEST, and UNNEST can be chained in any combination

34. ©2015 Couchbase Inc. 34
N1QL Expressions for JSON
©2014 Couchbase, Inc.
34
Ranging over collections
• WHERE ANY c IN children SATISFIES c.age > 10 END
• WHERE EVERY r IN ratings SATISFIES r > 3 END
Mapping with filtering • ARRAY c.name FOR c IN children WHEN c.age > 10 END
Deep traversal, SET, and
UNSET
• WHERE ANY node WITHIN request SATISFIES node.type = “xyz” END
• UPDATE doc UNSET c.field1 FOR c WITHIN doc END
DynamicConstruction
• SELECT { “a”: expr1, “b”: expr2 } AS obj1, name FROM … // Dynamic object
• SELECT [ a, b ] FROM … // Dynamic array
Nested traversal • SELECT x.y.z, a[0] FROM a.b.c …
IS [ NOT ] MISSING • WHERE name IS MISSING

35. ©2015 Couchbase Inc. 35
N1QL DataTypes from JSON
N1QL supports all JSON data types
 Numbers
 Strings
 Booleans
 Null
 Arrays
 Objects

36. ©2015 Couchbase Inc. 36
N1QL DataType Handling
Non-JSON data types
 MISSING
 Binary
Data type handling
 Date functions for string and numeric encodings
 Total ordering across all data types
– Well defined semantics for ORDER BY and comparison operators
 Defined expression semantics for all input data types
– No type mismatch errors

37. The Query Service
in Couchbase Server 4.0

38. ©2015 Couchbase Inc. 38
Couchbase Server Cluster Architecture
38
STORAGE
Couchbase Server 1
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed
Cache
Cluster
ManagerCluster
Manager
Managed Cache
Storage
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 2
Managed
Cache
Cluster
ManagerCluster
Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 3
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed
Cache
Cluster
ManagerCluster
Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 4
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed
Cache
Cluster
ManagerCluster
Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 5
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed
Cache
Cluster
ManagerCluster
Manager
Data Service
Index Service
Query Service
STORAGE
Couchbase Server 6
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed
Cache
Cluster
ManagerCluster
Manager
Data Service
Index Service
Query Service
Managed Cache
Storage
Managed Cache
Storage
Managed Cache
Storage
Managed Cache
Storage
Managed Cache
Storage

39. ©2015 Couchbase Inc. 39
Couchbase Server Cluster Service Deployment
39
STORAGE
Couchbase Server 1
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed
Cache
Cluster
ManagerCluster
Manager
Managed Cache
Storage
Data
Servic
e
STORAGE
Couchbase Server 2
Managed
Cache
Cluster
ManagerCluster
Manager
Data
Servic
e
STORAGE
Couchbase Server 3
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed
Cache
Cluster
ManagerCluster
Manager
Data
Servic
e
STORAGE
Couchbase Server 4
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed
Cache
Cluster
ManagerCluster
Manager
Query
Servic
e
STORAGE
Couchbase Server 5
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed
Cache
Cluster
ManagerCluster
Manager
Query
Servic
e
STORAGE
Couchbase Server 6
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed
Cache
Cluster
ManagerCluster
Manager
Index
Servic
e
Managed Cache
Storage
Managed Cache
Storage Storage
STORAGE
Couchbase Server 6
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed
Cache
Cluster
ManagerCluster
Manager
Index
Servic
e
Storage
Managed Cache Managed Cache

40. ©2015 Couchbase Inc. 40
N1QL: Query Execution Flow
Clients
1. Submit the query over RESTAPI 8. Query result
2. Parse, Analyze, create Plan 7. Evaluate: Documents to results
3. Scan Request;
index filters
6. Fetch the documents
Index
Servic
e
Query
Service
Data
Servic
e
4. Get qualified doc keys
5. Fetch Request,
doc keys
SELECT c_id,
c_first,
c_last,
c_max
FROM CUSTOMER
WHERE c_id = 49165;
{
"c_first": "Joe",
"c_id": 49165,
"c_last": "Montana",
"c_max" : 50000
}

41. ©2015 Couchbase Inc. 41
Query Execution
Client
FetchParse Plan Join Filter
Pre-Aggregate
Offset Limit ProjectSortAggregateScan
Query Service
Index
Servic
e
Data
Servic
e
Data-parallel — Query latency scales with cores
In-memory, streamed
Pluggable architecture — datastore, indexer, client, …
Query Resultset

42. Benefits of N1QL
for your Application and
Enterprise

43. ©2015 Couchbase Inc. 43
Benefits for your Application and Enterprise
 Model your Data Cleanly
– Model once, query
– Use both relationships and embedding
 Query your Data with Flexibility
– Query across structure, relationships, and datasets
– Query across change and heterogeneity
 Develop Rich Applications with Agility
– Deliver features that transform, combine, and aggregate data
– Use your favorite clients, frameworks, and interfaces
 Integrate your Applications and Data
– Leverage ecosystem: Connectivity, Analytics, Cloud and Packaged Apps

44. Community and Participation

45. ©2015 Couchbase Inc. 45
Community and Participation
 Enterprise and Community Editions of Couchbase 4.0
 Community Ecosystem
– Build complementary tools, products, and drivers
– Build and integrate via our open APIs

46. ©2015 Couchbase Inc. 46
Community and Participation in Action
Couchbase N1QL Query Runner
– byWarren Postma, Beta customer
– 30-minute quick start with Python’sTkinter GUI package

47. Getting Started

48. ©2015 Couchbase Inc. 48
Getting Started
Test drive Couchbase Server 4.0 Now!
Visit query.couchbase.com
Play with the online tutorial
Ask us questions
– Couchbase forums, Stack Overflow, @N1QL

49. N1QL Sessions
at Couchbase Live NewYork 2015

50. ©2015 Couchbase Inc. 50
2:00pm
Level 2
Developer
3:50pm
Level 2
Architecture
3:50pm
Level 2
Architecture
4:35pm
Level 2
Developer
4:35pm
Level3
Architecture
Sampling of N1QL Sessions by Couchbase
Data Modeling with
Couchbase Server
Evolution of
Couchbase at
CenterEdge: From
Cache to Query
Nielsen’s Interactive
Data Analytics with
Couchbase N1QL
Deep Dive into
N1QL with Global
Secondary Indexes
N1QL and SDK
Support for Java,
.NET, and Node.js

51. Q & A

52. Thank you.
Keshav Murthy, Couchbase
@rkeshavmurthy
Team @N1QL