Silicon Valley NoSQL Meetup - Nov 2012. View with animations: video version here: https://vimeo.com/54691785
http://www.meetup.com/Silicon-Valley-NoSQL/events/88257222/
For more information visit: www.couchbase.com
3. Couchbase Server
Easy Consistent High
Scalability Performance
Grow cluster without Consistent sub-millisecond
application changes, without read and write response times
downtime with a single click with consistent high throughput
Always Flexible Data
On JSON
JSON JSO
JSON
JSON
N
Model
24x365
No downtime for software JSON document model with
upgrades, hardware no fixed schema.
maintenance, etc.
3
4. Flexible Data Model
{
“ID”: 1,
“FIRST”: “Dipti”,
“LAST”: “Borkar”,
“ZIP”: “94040”,
“CITY”: “MV”,
“STATE”: “CA”
} JSON JSON
JSON
JSON
• No need to worry about the database when changing your
application
• Records can have different structures, there is no fixed
schema
• Allows painless data model changes for rapid application
development
4
5. New in 2.0
JSON support Indexing and Querying
JSON
JSON JSO
JSON N
JSON
Incremental Map Reduce Cross data center replication
5
6. Couchbase Server 2.0 Architecture
8092 11211 11210
Query API Memcapable 1.0 Memcapable 2.0
Moxi
Query Engine
REST management API/Web UI
vBucket state and replication manager
Memcached
Global singleton supervisor
Rebalance orchestrator
Configuration manager
Node health monitor
Process monitor
Heartbeat
Couchbase EP Engine
Data Manager Cluster Manager
storage interface
New Persistence Layer http on each node one per cluster
Erlang/OTP
HTTP Erlang port mapper Distributed Erlang
8091 4369 21100 - 21199
6
7. Couchbase Server 2.0 Architecture
8092 11211 11210
Query API Memcapable 1.0 Memcapable 2.0
Moxi
Query Engine
REST management API/Web UI
vBucket state and replication manager
Object-level Cache
Global singleton supervisor
Rebalance orchestrator
Configuration manager
Node health monitor
Process monitor
Heartbeat
Couchbase EP Engine
storage interface
New Persistence Layer http on each node one per cluster
Erlang/OTP
HTTP Erlang port mapper Distributed Erlang
8091 4369 21100 - 21199
7
8. Couchbase deployment
Web
Application
Couchbase
Client Library
Data Flow
Cluster Management
8
10. Single node - Couchbase Write Operation
2
Doc 1
App Server
3 2 3
Managed Cache
To other node Replication
Doc 1
Queue
Disk Queue
Disk
Couchbase Server Node 10
11. Single node - Couchbase Update Operation
2
Doc 1’
App Server
3 2 3
Managed Cache
To other node Replication
1
Doc 1’
Queue
Disk Queue
Disk
Doc 1
Couchbase Server Node 11
12. Single node - Couchbase Read Operation
2
Doc 1
GET
App Server
3 2 3
Managed Cache
To other node Replication
Queue Doc 1
Disk Queue
Disk
Doc 1
Couchbase Server Node 12
13. Single node - Couchbase Cache Eviction
2
Doc 6
2
3
4
5
App Server
3 2 3
Managed Cache
To other node Replication
Doc 1
Queue
Disk Queue
Disk
Doc 1
Doc 6 Doc 5 Doc 4 Doc 3 Doc 2
Couchbase Server Node 13
14. Single node – Couchbase Cache Miss
2
Doc 1
GET
App Server
3 2 3
Managed Cache
To other node Replication
Queue Doc 1
Doc 5 4 4
Doc
Doc Doc 3 2
Doc
Disk Queue
Disk
Doc 1
Doc 6 Doc 5 Doc 4 Doc 3 Doc 2
Couchbase Server Node 14
15. Cluster wide - Basic Operation
APP SERVER 1 APP SERVER 2
COUCHBASE Client Library COUCHBASE Client Library
CLUSTER MAP CLUSTER MAP
READ/WRITE/UPDATE
SERVER 1 SERVER 2 SERVER 3 • Docs distributed evenly across
ACTIVE ACTIVE ACTIVE
servers
Doc 5 Doc Doc 4 Doc Doc 1 Doc • Each server stores both active and
replica docs
Doc 2 Doc Doc 7 Doc Doc 2 Doc Only one server active at a time
• Client library provides app with
Doc 9 Doc Doc 8 Doc Doc 6 Doc
simple interface to database
REPLICA REPLICA REPLICA • Cluster map provides map
to which server doc is on
Doc 4 Doc Doc 6 Doc Doc 7 Doc App never needs to know
Doc 1 Doc Doc 3 Doc Doc 9 Doc • App reads, writes, updates docs
Doc 8 Doc Doc 2 Doc Doc 5 Doc • Multiple app servers can access same
document at same time
COUCHBASE SERVER CLUSTER
User Configured Replica Count = 1
15
16. Cluster wide - Add Nodes to Cluster
APP SERVER 1 APP SERVER 2
COUCHBASE Client Library COUCHBASE Client Library
CLUSTER MAP CLUSTER MAP
READ/WRITE/UPDATE READ/WRITE/UPDATE
SERVER 1 SERVER 2 SERVER 3 SERVER 4 SERVER 5 • Two servers added
ACTIVE ACTIVE ACTIVE ACTIVE ACTIVE
One-click operation
Doc 5 Doc Doc 4 Doc Doc 1 Doc • Docs automatically
rebalanced across
Doc 2 Doc Doc 7 Doc Doc 2 Doc cluster
Even distribution of docs
Minimum doc movement
Doc 9 Doc Doc 8 Doc Doc 6 Doc
• Cluster map updated
REPLICA REPLICA REPLICA REPLICA REPLICA
• App database
Doc 4 Doc Doc 6 Doc Doc 7 Doc calls now distributed
over larger number of
Doc 1 Doc Doc 3 Doc Doc 9 Doc servers
Doc 8 Doc Doc 2 Doc Doc 5 Doc
COUCHBASE SERVER CLUSTER
User Configured Replica Count = 1
16
17. Cluster wide - Fail Over Node
APP SERVER 1 APP SERVER 2
COUCHBASE Client Library COUCHBASE Client Library
CLUSTER MAP CLUSTER MAP
SERVER 1 SERVER 2 SERVER 3 SERVER 4 SERVER 5 • App servers accessing docs
ACTIVE ACTIVE ACTIVE ACTIVE ACTIVE
• Requests to Server 3 fail
Doc 5 Doc Doc 4 Doc Doc 1 Doc Doc 9 Doc Doc 6 Doc
• Cluster detects server failed
Promotes replicas of docs to
Doc 2 Doc Doc 7 Doc Doc 2 Doc Doc 8 Doc Doc active
Updates cluster map
Doc 1 Doc 3
• Requests for docs now go to
REPLICA REPLICA REPLICA REPLICA REPLICA appropriate server
Doc 4 Doc Doc 6 Doc Doc 7 Doc Doc 5 Doc Doc 8 Doc • Typically rebalance
would follow
Doc 1 Doc Doc 3 Doc Doc 9 Doc Doc 2 Doc
COUCHBASE SERVER CLUSTER
User Configured Replica Count = 1
17
19. Indexing and Querying – The basics
• Define materialized views on JSON documents and
then query across the data set
• Using views you can define
• Primary indexes
• Simple secondary indexes (most common use case)
• Complex secondary, tertiary and composite indexes
• Aggregations (reduction)
• Indexes are eventually indexed
• Queries are eventually consistent with respect to
documents
• Built using Map/Reduce technology
• Map and Reduce functions are written in Javascript
19
20. Eventually indexed Views – Data flow
2
Doc 1
App Server
Couchbase Server Node
3 2 3
Managed Cache
To other node Replication
Queue Doc 1
Disk Queue
Disk
Doc 1
View engine
20
21. Cluster wide - Indexing and Querying
APP SERVER 1 APP SERVER 2
COUCHBASE Client Library COUCHBASE Client Library
CLUSTER MAP CLUSTER MAP
Query
SERVER 1 SERVER 2 SERVER 3 • Indexing work is distributed
ACTIVE ACTIVE ACTIVE
amongst nodes
Doc 5 Doc Doc 5 Doc Doc 5 Doc • Large data set possible
Doc 2 Doc Doc 2 Doc Doc 2 Doc
• Parallelize the effort
Doc 9 Doc
• Each node has index for data stored
Doc 9 Doc Doc 9 Doc
on it
REPLICA REPLICA REPLICA • Queries combine the results from
Doc 4 Doc
required nodes
Doc 4 Doc Doc 4 Doc
Doc 1 Doc Doc 1 Doc Doc 1 Doc
Doc 8 Doc Doc 8 Doc Doc 8 Doc
COUCHBASE SERVER CLUSTER
User Configured Replica Count = 1
21
22. Cross Data Center Replication – The basics
• Replicate your Couchbase data across clusters
• Clusters may be spread across geos
• Configured on a per-bucket basis
• Supports unidirectional and bidirectional operation
• Application can read and write from both clusters
(active – active replication)
• Replication throughput scales out linearly
• Different from intra-cluster replication
22
23. Cross data center replication – Data flow
2
Doc 1
App Server
Couchbase Server Node
3 2 3
Managed Cache
To other node Replication
Doc 1
Queue
Disk Queue
Disk
Doc 1
XDCR Queue
To other cluster
23
24. Cluster wide - XDCR
SERVER 1 SERVER 2 SERVER 3
ACTIVE ACTIVE ACTIVE COUCHBASE SERVER CLUSTER
Doc Doc Doc NY DATA CENTER
Doc 2 Doc Doc
Doc 9 Doc Doc
RAM RAM RAM
Doc Doc Doc Doc Doc Doc Doc Doc Doc
DISK DISK DISK
SERVER 1 SERVER 2 SERVER 3
ACTIVE ACTIVE ACTIVE
Doc Doc Doc
Doc 2 Doc Doc
Doc 9 Doc Doc
RAM RAM RAM
COUCHBASE SERVER CLUSTER Doc Doc Doc Doc Doc Doc Doc Doc Doc
SF DATA CENTER
DISK DISK DISK 24
26. Demo: The next big social game
3 Objects (documents) within game:
• Players
• Monsters
• Items
Gameplay:
• Players fight monsters
• Monsters drop items
• Players own items
26
27. Player Document
{
"jsonType": "player",
"uuid": "35767d02-a958-4b83-8179-616816692de1",
"name": "Keith4540",
"hitpoints": 75,
Player ID
"experience": 663,
"level": 4,
"loggedIn": false
}
27
28. Item Document
{
Item ID
"jsonType": "item",
"name": "Katana_e5890c94-11c6-65746ce6c560",
"uuid": "e5890c94-11c6-4856-a7a6-65746ce6c560",
"ownerId": "Dale9887"
}
Player ID
28
31. Full Text Search Integration
• Elastic Search is good for ad-hoc queries and faceted browsing
• Couchbase adapter uses XDCR to push mutations to ESDocs
are indexed by Elastic Search
• Couchbase ES Adapter is cluster-aware
ElasticSearch
Unidirectional Cross Data Center Replication
31
32. Full Text Search
Application Server
Couchbase SDK ES Queries over HTTP
Do
ta TS c
Da ery Qu Re
Qu er fs
M R y
Couchbase Server Cluster ElasticSearch Server Cluster
MR MR MR MR
Views Views Views Views
XDCR-based
Cross Data Center Replication CB-ES Transport
32
33. Couchbase SDKs
Java SDK User Code
.Net SDK Java client API
CouchbaseClient cb = new CouchbaseClient(listURIs,
"aBucket", "letmein");
cb.set("hello", 0, "world");
cb.get("hello");
Couchbase Java Library
PHP SDK (spymemcached)
Ruby SDK
Couchbase Server
…and many
more
http://www.couchbase.com/develop
33
Hello everyone, I am diptiborkar, director of product management at couchbase. I manage the product roadmap of couchbase server, our flagship product. And I’m really excited to talk to you about couchbase server 2.0
Rapid app devwithouth the need to perform an expensive alter table operation.
JSON support – natively stored as json, whne you build an app, there is not conversion required. New doc viewing , editing capability. Indexing and querying – look inside your json, build views and query for a key, for ranges or to aggregate data Incremental mapreduce – powers indexing. Build complex views over your data. Great for real-time analytics XDCR – replicate information from one cluster to another cluster
1. A set request comes in from the application .2. Couchbase Server responses back that they key is written3. Couchbase Server then Replicates the data out to memory in the other nodes4. At the same time it is put the data into a write que to be persisted to disk
1. A set request comes in from the application .2. Couchbase Server responses back that they key is written3. Couchbase Server then Replicates the data out to memory in the other nodes4. At the same time it is put the data into a write que to be persisted to disk
1. A set request comes in from the application .2. Couchbase Server responses back that they key is written3. Couchbase Server then Replicates the data out to memory in the other nodes4. At the same time it is put the data into a write que to be persisted to disk
1. A set request comes in from the application .2. Couchbase Server responses back that they key is written3. Couchbase Server then Replicates the data out to memory in the other nodes4. At the same time it is put the data into a write que to be persisted to disk
1. A set request comes in from the application .2. Couchbase Server responses back that they key is written3. Couchbase Server then Replicates the data out to memory in the other nodes4. At the same time it is put the data into a write que to be persisted to disk
Bulletize the text. Make sure the builds work.
Bulletize the text. Make sure build work properly.
Bulletize the text. Make sure build work properly.
1. A set request comes in from the application .2. Couchbase Server responses back that they key is written3. Couchbase Server then Replicates the data out to memory in the other nodes4. At the same time it is put the data into a write que to be persisted to disk
Bulletize the text. Make sure the builds work.
Overview of what this feature is
1. A set request comes in from the application .2. Couchbase Server responses back that they key is written3. Couchbase Server then Replicates the data out to memory in the other nodes4. At the same time it is put the data into a write que to be persisted to disk
Break out to demo here
ElasticSearch cluster is fed the documents from the Couchbase Server clusterElastic search indexes the fields(configurable which ones) and by default will only store references back to the document idThe application does document access via the Couchbase Server Cluster and uses The Views and incremental map reduce on the Couchbase cluster.For full text queries it queries the Ealstic search cluster directly (simple Http and JSON interface)The full text queries typically returns the ids of the matching documents.Documents are then retrieved from the Couchbase Server cluster.This way the high throughput document access always comes from high performance Couchbase Cluster.