Developing high-throughput services with no sql ap-is to innodb and mysql cluster

Developing High-Throughput
Services with NoSQL APIs to
InnoDB and MySQL Cluster
Andrew Morgan – MySQL Product Management
John Duncan – MySQL Cluster Engineering

1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012

Program Agenda

 Why NoSQL
 MySQL Cluster Attributes
 NoSQL APIs for MySQL Cluster
 Memcached API for InnoDB
 Node.js – Native javascript access
 Next Steps


Disclaimer

The f ll i is intended to tli
Th following i i t d d t outline our general product di ti
l d t direction. It i
is
intended for information purposes only, and may not be incorporated
into
i t any contract. It is not a commitment to deliver any material,
t t i t it t t d li t i l
code, or functionality, and should not be relied upon in making
purchasing d i i
h i decisions. Th d The development, release, and ti i of any
l t l d timing f
features or functionality described for Oracle’s products remains at
the l discretion of O l
th sole di ti f Oracle.


NoSQL – Why the demand?

2.1BN USERS
 Web
W b applications
li ti 8X DATA GROWTH IN 5 YRS 900M USERS
demanding: 40% DATA
70+ NEW DOMAINS EVERY 60 SECONDS 20M APPS PER DAY

 Development velocity: GROWTH PER
YEAR 60 HOURS
Simplicity & flexibility of UPLOADED
EVERY MINUTE
data model & APIs
 Scalability & $1TR BY 2014
350m TWEETS
performance: high write PER DAY $700BN IN 2011

throughput and
Key/Value access
 Support for “Big Data” 5.9BN MOBILE SUBS IN 2011
1 BILLION iOS & ANDROID 85% HANDSETS
SHIPPED WITH A
APPS DOWNLOADED PER WEEK
BROWSER


What NoSQL must deliver

 Massive scalability
Scalability
 No application-level sharding
o app cat o e e s a d g
Performance  Performance
 High Availability/Fault Tolerance
HA
 Ease of use
Ease of use  Simple operations/administration
 Simple APIs
 Quickly evolve application & schema


Still a role for the RDBMS?

 No best single solution
fits all NoSQL RDBMS
 Mix and match
Simple
Si l access patterns
tt Complex queries with j i
C l i ith joins
Scalability
Compromise on consistency
Performance ACID transactions
for performance

HA
Ad-hoc data format Well defined schemas
Ease o use
ase of

SQL/Joins Simple operation Rich set of tools

ACID Transactions


MySQL Cluster introduction
Carrier Grade • Shared-nothing in-memory parallel database server
Database • ACID compliant relational database

• Five nines (99.999%) availability
Highly Available • Self-healing, sub-second failover

Real-time • High load, real-time performance
Performance • Predictable low latency, bounded access times

• Incrementally grow out with application demands
Linearly Scalable • Linearly scale with distribution awareness
y

• Open Source, multiple data access
Open Development • High performance APIs (C++, Java, SQL, LDAP)
(C ,


Who’s Using MySQL Cluster?


Scalability

MySQL Cluster Architecture Performance

HA

Ease of use

SQL/Joins
SQL/J i 
Clients ACID Transactions 

Application Layer

Data Layer

Management

MySQL Cluster Data Nodes


Scalability 

HA

Ease of use

SQL/Joins
SQL/J i 

Application Layer

Data Layer

Management
Management



Scalability 

HA 
Ease of use

SQL/Joins
SQL/J i 

Application Layer

Data Layer

Management
Management



Scalability 
On-line Scheduled Maintenance Performance

HA 
Ease of use

SQL/Joins
SQL/J i 
ACID Transactions 

Scale Backup

Evolve Upgrade
pg


Scalability 
Extreme performance Performance 
HA 
Ease of use

SQL/Joins
SQL/J i 

writes


Creating & running your first Cluster
The traditional way (pre-MCM) – Up and running in 15 mins

Download & Configure Start processes
Extract •Cluster-wide
•Cluster wide •Management Nodes
•edelivery.oracle.com “config.ini” •Data Nodes
•www.mysql.com •Per-mysqld “my.cnf” •MySQL Servers
•dev.mysql.com

 Up & running in 10-15 minutes using Quick Start guides from http://dev mysql com/downloads/cluster/
http://dev.mysql.com/downloads/cluster/
– Versions for Linux, Windows & Solaris

Scalability 
Performance 
HA 
Ease of use 
SQL/Joins 

MySQL Cluster Manager
Bootstrap single h t Cl t
B t t i l host Cluster

1. Download MCM/Cluster package from edelivery.oracle.com:
2. Unzip
3. Run agent, define, create & start Cluster!
$ binmcmd bootst ap
$> b c d –bootstrap
MySQL Cluster Manager 1.1.2 started
Connect to MySQL Cluster Manager by running "D:AndrewDocumentsMySQLmcmbinmcm" -a NOVA:1862
Configuring default cluster 'mycluster'...
Starting default cluster 'mycluster'...
Cluster 'mycluster' started successfully
ndb_mgmd NOVA 1186
db d NOVA:1186
ndbd NOVA
ndbd NOVA
mysqld NOVA:3306
mysqld NOVA:3307
ndbapi *
Connect to the database by running "D:AndrewDocumentsMySQLmcmclusterbinmysql" -h NOVA -P 3306 -u root


MySQL Cluster 7.3 labs: Auto-Installer

• Fast configuration Specify
p y Auto-
Workload Discover
• Auto-discovery

• Workload ti i d
W kl d optimized

• Repeatable best
practices
ti

• For MySQL Cluster Confirm
7.2 7 3
7 2 + 7.3 Deploy Topology


MySQL Cluster 7.3 labs: Auto-Installer

• Fast configuration

• Auto-discovery

• Workload ti i d
W kl d optimized

• Repeatable best
practices
ti

• For MySQL Cluster
7.2 7 3
7 2 + 7.3


Scalability 
Performance 
HA 
Ease of use 
SQL/Joins 

NoSQL Access to MySQL Cluster data

Apps Apps Apps Apps Apps Apps Apps Apps Apps Apps Apps Apps

JPA
ClusterJPA

PHP PERL Python Ruby JDBC ClusterJ JSON Apache Memcached

Node.js mod-ndb ndb-eng
MySQL JNI
NDB API (C++)



MySQL Cluster 7.1: ClusterJ/JPA
• Domain Object Model Persistence API (ClusterJ):
• Java API
• High performance, low latency
• Feature rich
• JPA interface built upon this new Java layer:
• Java Persistence API compliant
• Implemented as an OpenJPA plugin
• Uses ClusterJ where possible, reverts to JDBC for
some operations
• Higher performance than JDBC
• More natural for most Java designers
• Easier Cluster adoption for web applications
Cl ster eb


MySQL Cluster 7.2: Memcached

• Memcached is a distributed memory based hash-
key/value t
k / l store with no persistence to disk
ith i t t di k
• NoSQL, simple API, popular with developers
• MySQL Cluster already provides scalable, in-
y yp
memory performance with NoSQL (hashed) access
as well as persistence
• Provide the Memcached API but map to NDB
API calls
• Writes-in-place, so no need to invalidate cache
• Simplifies
Si lifi architecture as caching & d t b
hit t hi database
integrated into 1 tier
• Access data from existing relational tables


Traditional Memcached Architecture
Two l
T levels of hashing
l f h hi

httpd memcached

hash key
PHP/Perl memcached
to find data
Memcache
friends:12389 memcached
memcache key


Memcache Protocol Elements

• Key Up to 250 characters
• Expire Time A number in seconds, up to 30 days
seconds
• Flags 32-bit number for application use
• CAS ID 64-bit incrementing version number on value
64 bit i ti i b l
• Value Value stored for key; traditionally up to 1 MB.


Memcache Protocol Commands (1)

• get k
t key Fetch value for this key
al e ke
• set key value Store this data
• add key value Store this data (strict insert)
• replace key value Store this data (strict update)
• cas key value cas-id Update with version number check


Memcache Protocol Commands (2)

• append key value Append text to current value
• prepend key value Insert text before current value
• incr key amount Add to current stored integer value
• decr key amount Subtract from current stored value
• delete key Delete a key and value
• flush_all Delete all stored values on the server
• stats stat-name Fetch server statistics


NDB & Memcache Architecture:
Memcache protocol + NDB storage
M h t l t

MySQL
y Q
Application Cluster
memcached Data Node

Memcache
NDB Engine MySQL
Client
Cluster
Cl
Data Node


Goals

• Access stored data directly from memcache client
• Memcached perspective:
• MySQL Cluster is a write-scalable, replicated data store
• with reliable in-memory storage,
• plus on-disk storage when data is too big for memory.
• MySQL Cluster perspective:
• memcache is a high performance API
• providing easy access to in-memory data,
• plus an extra layer of caching when data is on disk.


Memcached/MySQL Cluster latency
memcachetest -t 2 -M 7000 -c 25000


Cluster & Memcached – Schema-Free

key value
<town:maidenhead,SL6>
Application i
A li ti view
SQL view key value


Key Value
town:maidenhead SL6
generic table


Cluster & Memcached – Configured Schema

key value
Application i
A li ti view
SQL view prefix key value


Prefix Table Key-col Val-col policy town ... code ...
town: map.zip town code cluster maidenhead ... SL6 ...

Config tables
C fi t bl map.zip
i


Memcached NoSQL Access with InnoDB
• Memcached as a plugin of
Application
MySQL Server; same
process space, with very low
latency access to data SQL Memcached protocol
• Memcapable: supports both mysqld

memcached ascii protocol
MySQL Server
and binary protocol memcached plugin

• Support multiple columns:
local cache
users can map multiple innodb_memcache
(optional)
columns into “value” Handler API InnoDB API
• Optional local caching:
“innodb-only”, “cache-only”, InnoDB Storage Engine
and “caching”
• Batch operations for
performance

MySQL Cluster 7.3 EA: Node.js NoSQL API
 Native JavaScript access to MySQL Cluster
S SQ C
– End-to-End JavaScript: browser to the app and
Clients database
– Storing and retrieving JavaScript objects directly
in MySQL Cluster
y
– Eliminate SQL transformation
V8 JavaScript Engine  Implemented as a module for node.js
p j
MySQL Cluster Node.js Module – Integrates full Cluster API library within the web
app
 Couple high performance, distributed apps, with
high performance distributed database


MySQL Cluster NoSQL API for Node.js
Application Code
A li ti C d

// Constructor
function Tweet(user, message) {
this.id = UUID.generate();
this.timestamp = Date.now();
p
this.user = user;
this.message = message;
message
}



• Modular connector with
// Server Startup various back-end adapters:
• ndb: low-level native
access to MySQL Cluster
var nosql = require('mysql-js');
• mysql: access to any
MySQL server (translates
y (
var sessionFactory = operations to SQL
nosql.connectSync('ndb'); statements)

nosql.mapClass(Tweet, 'tweets');



// Create a tweet

function newTweet(err, dbSession, httpReq) {
var tweet = new Tweet(
httpReq.user,
httpReq.message);
dbSession.persist(tweet);
}



function onNewTweetRequest(err, httpReq) {
sessionFactory.openSession(null, newTweet, httpReq);
}


Sync s Async Execution
S nc vs. As nc E ec tion
 If you had all synchronous operations
y y p
– Single execution context blocked for I/O.
– Low performance
p
 If you had all async operations
– Complex code
– Each individual operation nested within its own callback function.
 The right balance
– Define a database operation as an immediate call (no I/O)
– Execute database operations in async calls with callbacks

Next Steps
Learn More
• www.mysql.com/cluster
• A th ti M SQL C i l
Authentic MySQL Curriculum:
http://oracle.com/education/mysql

Try O t
T it Out
• dev.mysql.com/cluster
• labs.mysql.com
y

Let us know what you think
• clusterdb.com
• @clusterdb
• forums mysql com/list php?25
forums.mysql.com/list.php?25


Graphic Section Divider


Developing high-throughput services with no sql ap-is to innodb and mysql cluster

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to Developing high-throughput services with no sql ap-is to innodb and mysql cluster

Similar to Developing high-throughput services with no sql ap-is to innodb and mysql cluster (20)

More from Andrew Morgan

More from Andrew Morgan (11)

Developing high-throughput services with no sql ap-is to innodb and mysql cluster