Everything You Need to Know About Membase

San Diego

Thursday, January 6, 2011

What is Membase?


Membase is a distributed database

Application user

Web application server Web application server Web application server

Membase Servers

In the data center On the administrator console

3


Membase is Simple, Fast, Elastic

Five minutes or less to a working
cluster
• Downloads for Linux and Windows
• Start with a single node
• One button press joins nodes to a cluster
Easy to develop against
• Just SET and GET – no schema required
• Drop it in. 10,000+ existing applications
already “speak membase” (via memcached)
• Practically every language and application
framework is supported, out of the box
Easy to manage
• One-click failover and cluster rebalancing
• Graphical and programmatic interfaces
• Conﬁgurable alerting

4



Predictable
• “Never keep an application waiting”
• Quasi-deterministic latency and throughput
Low latency
• Built-in Memcached technology
High throughput
• Multi-threaded
• Low lock contention
• Asynchronous wherever possible
• Automatic write de-duplication

5



Zero-downtime elasticity
• Spread I/O and data across commodity
servers (or VMs)
• Consistent performance with linear cost
• Dynamic rebalancing of a live cluster
All nodes are created equal
• No special case nodes
• Any node can replace any other node, online
• Clone to grow
Extensible
• Filtered TAP interface provides hook points
for external systems (e.g. full-text search,
backup, warehouse)
• Data bucket – engine API for specialized
container types

6


Built-in Memcached Caching Layer

Memcached Membase Cache

Membase Database Membase Database

Memcached Mode Membase Mode

Fact: Membase development team has also contributed over
half of the code to the Memcached project. 7


Use Cases


Ad targeting

40 milliseconds to come
up with an answer.

profiles, real time campaign
3 statistics

2
1 profiles, campaigns
events

9


Search and Gaming Portal

Database

10


Membase Caching at The Knot
Jason Sirota


Membase Architecture


Clustering
• Underlying cluster
functionality based on
erlang OTP
• Have a custom, vector
clock based way of storing
and propagating...
– Cluster topology
– vBucket mapping
• Collect statistics from many
nodes of the cluster
– Identify hot keys, resource
utilization

13


TAP

• A generic, scalable method of streaming mutations
from a given server
– As data operations arrive, they can be sent to arbitrary TAP
receivers
• Leverages the existing memcached engine interface,
and the non-blocking IO interfaces to send data
• Three modes of operation
Data
Mutations Working set

Data
Mutations Working set Working set

15


Membase data ﬂow – under the hood

SET request arrives at SET acknowledgement
KEY’s master server 1 2 returned to application

3 Listener-Sender
3

RAM

membase storage engine
4

Disk Disk Disk

Disk Disk Disk

Replica Server 1 for KEY Master server for KEY Replica Server 2 for KEY
16


Clients, nodes and other nodes

Client moxi + Client

port 11210
port 11211 memcached operations REST/comet
memcached operations cluster topology
and vbucket map

memcached operations
moxi ns_server

membase ns_server
(memcached + membase engine)

TAP
memcached operations
with tap commands vbucketmigrator

17


Data buckets are secure membase “slices”

Application user

Web application server

Bucket 1
Bucket 2
Aggregate Cluster Memory and Disk Capacity

Membase data servers

In the data center On the administrator console
18


vBucket mapping

19


Disk > Memory

Dataset may have many memory quota

Bucket Conﬁguration
items infrequently accessed. mem_high_wat

However, memcached has mem_low_wat
different behavior (LRU) than
wanted with membase.

Still, traditional (most)
RDBMS implementations are
not 100% correct for us
either. The speed of a miss
is very, very important.

20


Membase Demo


Key-Value Patterns


Key-Value

23


Key-Value

Items have: Operations include:
Key Get/Set
Value Increment/Decrement
Expiration Append/Prepend
Flags
CAS (more on this later)
23


Key-Value

(with a replica )
Image courtesy http://www.flickr.com/photos/brenda-starr/3509344100/sizes/m/in/photostream/

23


Membase Datatypes

24


Membase Datatypes

• byte[]
– Does your data have
1s and 0s?

24


Membase Datatypes

• byte[]
1s and 0s?

“Any customer can have
a car painted any colour
that he wants so long as
it is black.”

24


Membase Datatypes

• byte[]
1s and 0s?
• Items do have flags
– Many clients use flags
– Data type options
• Google protobuf “Any customer can have
a car painted any colour
• Thrift that he wants so long as
• Avro it is black.”

24


Transactions

• Lock == slow me down
• CAS operations User 1 User 2
– Optimistic locking

Su
• Very useful with complex

cc

il!
es

Fa
datatypes

s
– Imagine two clients trying to
update a complex item
• You’re likely using CAS
already... if you use a CPU

25


Common Use: Sessions

• Web user sessions
– Highly read, less writes in many case
– Protocol advantage of memcached
• Options already for PHP, Ruby and Java
• Application state
– Not necessarily “entity” style things
– May be appropriate for a “cache” pool

26


Common Use (cache): Rate Limiting

• Want to provide API calls
into the system Your Users

– Twitter search h !
uc
– Google search services ¡O

• Use the atomic increment Your App

– Set an item with a unique ID
– Upon API request,
increment and check
• HTTP 420: go away and come
back later

27


Looking Ahead: NodeCode


NodeCode – Motivation

Beyond key-value
• Indexing/Range Queries
• Advanced Data Structures
• Sub-object direct manipulation

Validation and In-ﬂight transformation
• Block mutations failing validation
• Enrich or transform objects

Connectors (Integrate easily with other systems)
• Solr
• Hadoop
• MySQL

29


Q&A

31


Attributions

• http://commons.wikimedia.org/wiki/File:Flag_of_China.png
• http://commons.wikimedia.org/wiki/File:Flag_of_South_Korea.svg
• http://commons.wikimedia.org/wiki/File:Flag_of_Japan.svg

32


Everything You Need to Know About Membase

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (9)

Similaire à Everything You Need to Know About Membase

Similaire à Everything You Need to Know About Membase (20)

Dernier

Dernier (20)

Everything You Need to Know About Membase