State of the art of MySQL replication and clustering

State of the Art in
MySQL HA and Replication

Robert Hodges, CEO
Giuseppe Maxia, Director of QA

©Continuent 2012.

Tuesday, November 13, 12 1

Introducing Continuent

• The leading provider of clustering and
replication for open source DBMS

• Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance
scaling and data management for MySQL
• Replication - Flexible, high-performance data
movement

©Continuent 2012 2


Quick Continuent Facts

• Largest Tungsten clustering installation
processes about 500M transactions each day

• Tungsten Replicator was application of the
year at the 2011 MySQL User Conference

• Continuent is well known in the MySQL
community for its clustering and replication

• Over 90 customers currently deployed

©Continuent 2012 3


A Few Continuent Customers

©Continuent 2012 4


Future-Proof HA and
Replication for MySQL

©Continuent 2012 5


What is Future-Proo!ng?

• Future-proo!ng = creating systems that last
while parts change and improve

• MySQL is not losing out to other solutions for
data management

• The real problem is using MySQL as a building
block with other technologies that are
changing rapidly

©Continuent 2012 6


But Wait... Isn’t MySQL Dead??

MySQL
“You were so open”
Born
May 25, 1995
Died
2008, 2009, 2010,
2011, 2012, ...

©Continuent 2012 7


Oracle/MySQL 5.6 Replication
(On the Way)

• Global transaction IDs

• Parallel slave threads

• Crash safe slaves

• Optimized row updates

• Time delay replication

• Informational log events

©Continuent 2012 8


Galera Synchronous Clusters
(New Arrival in 2012)

• Multi-master for InnoDB

• Synchronous replication avoids data loss

• Automatic cluster membership management

• Simple node provisioning model

• Based on innovative state-machine work from
Fernando Pedone

©Continuent 2012 9


Tungsten Async Replication
(2011 Belle of the Ball)

• All 5.6 features including parallel replication
for MySQL 5.0 onwards

• Optimized support for failover

• Programmable transaction !ltering

• Backup/restore integration

• Multi-master, fan-in, and star replication

• Replication to/from Oracle and batch loading
of data warehouses

• Replication to MongoDB
©Continuent 2012 10


Not Bad for a Dying Market

THE$TOTAL$MARKET$FOR$MYSQL,$NOSQL$$
My/NewSQL,' AND$NEWSQL$$
5%'
NewSQL,'2%' 1000"
900"
NoSQL,'5%' 800"
700"
600" Text
500"
MySQL,'88%' 400"
300"
200"
100"
0"
(451Group) 2011" 2012" 2013" 2014" 2015"



MySQL Is Part of Bigger Trends
• Cloud and Big Data investment dwarf the
MySQL marketplace



21st Century CRM/Call Center

Fat Client CRM Web CRM Web
Accounting Application Application

Call
Session
Data
Online
Legacy
Transaction Hadoop
Oracle
Processing Analytics

On-Premise Amazon Web Services Telephony Telephony
Data Streams Data Streams



Revolutionary Challenges for MySQL

• Failures in unstable cloud environments

• Zero-downtime maintenance

• Rapidly growing data volumes, esp. in cloud

• Distributing data to geographical regions

• Integration between MySQL, NoSQL,
commercial RDBMS

• Supplying real-time analytics

• Technology upgrade/replacement



Future-Proofed HA and Replication

Data Fabric
Encapsulated, fault-tolerant,
horizontally scalable, globally
accessible, integrated data



Data Fabric Design Patterns

• Fabric Connector

• Fault-Tolerant Data Service

• Sharded Data Service

• Multi-Site Data Service

• Real-Time Data Bridge



1. Fabric Connector

• Single point of entry for applications

• Service catalog locates data within fabric

• Transparent connectivity, multiple protocols

• Security, auditing, performance management

Fabric
Connector

Application
Application
Stack
Stack

Fabric
Connector



2. Fault-Tolerant Data Service

• Encapsulate redundant database copies

• Sync and async clustering models

• Protect against local DBMS failure

• Rolling maintenance of replicas

Master/ Multi-
Slave Master



3. Sharded Data Service

• Partition datasets based on RAM / storage
speed / resident set size

• Multiple buckets per server

• Look-up methods for locating data in buckets

• Re-sharding / migration for high growth



4. Multi-Site Data Service

• Robust protection against region/site failures

• Geographic distribution of data

• Primary/DR vs. multi-master

• Eventually consistent replication for SQL
Amazon East
EU West

US East APAC Tokyo Rackspace DFW



5. Real-Time Data Bridge

• Replicate from logs in real-time

• Enable apps to get data from one data source

• Heterogeneous transfer and transformation

• High performance, low application impact

Bridge
Bridge
Bridge



Future-Proofed CRM/Call Center Data
Fat Client CRM Web CRM Web Telephony Telephony
Accounting Application Application Data Streams Data Streams

Fabric Fabric Fabric Fabric Fabric
Connector Connector Connector Connector Connector

Bridge Bridge Call
Session
Bridge Bridge Data

Legacy Oracle OLTP
Primary
Bridge Hadoop
Analytics

OLTP DR Data Marts



Enabling Data Fabric With Tungsten



Tungsten Fault Tolerant Data Service
GonzoPortal.com
Tungsten clusters combine o"-
the-shelf open source DBMS
servers into data services with:
apache
/php
• 24x7 data access
• Scaling of load on replicas Connector Connector

• Simple management commands
...without app changes or data
migration
Amazon
US West



Doesn’t Replication Do This Already?
Application Application
Libmysqlclient.a Libmysqlclient.a

Db2 Db1 Db3

Replicator Replicator Replicator

Slave Master Slave


Replication Limitations
Replication alone cannot enable constant access
to data...

• What happens if the master fails?

• What happens if a slave is lagging too far
behind to be usable?

• How do I maintain the DBMS without stopping
apps?



From Replicating to a Data Service
Application Application
Tungsten Connector Tungsten Connector
Monitoring and Control

Monitoring and Control
Db2 Db1 Db3

Manager Manager Manager
Replicator Replicator Replicator

Slave Master Slave
Data Service: sjc


Basic Operations

• Switch -- Promote a slave node to master
• Shun -- Kick node out of cluster
• Welcome -- Return node to cluster
• Backup -- Run a backup
• Restore -- [Re-]provision a node
• Recover -- Return a broken node to slave
• Heartbeat -- Check cluster liveness



Automatic Failover

Application
Tungsten Connector



Automatic Failover

Application
Tungsten Connector

2. Halt master connections

1. Detect non-
responsive
master



Automatic Failover

Application
Tungsten Connector

3. Select most advanced
slave and wait for pending
transactions to apply



Automatic Failover

Application
Tungsten Connector

4. Promote to master ,
reconfigure slave(s), and
re-enable connections



Automatic Failover

Application
Tungsten Connector

5. Recover failed master



Rolling SQL Schema Upgrade

Slave Upgrade Slave Upgrade Switch Final Upgrade

• Shun slave 1, • Repeat for • Switch • Upgrade old
run upgrade, additional master to master.
and return to slave(s) promote an Cluster
cluster. upgraded schema is now
Discard and slave fully upgraded
recover on
failure



Backup, Recovery, and Restore

• Backups
/cluster > datasource DB1 backup

• Recover
/cluster > datasource DB2 recover

• Restore
/cluster > datasource DB2 restore
• Out-of-box support for mysqldump/
XtraBackup; simple integration for others



Data Fabric Connectivity



Tungsten Connector Basics

• Any MySQL/PostgreSQL client can connect

• Initiates connections using JDBC

App Native
protocol Connector
COM_QUERY
COM_INIT_DB
COM_DROP_DB



Tungsten Connector Basics

• Pass-through operation after connection

• Full transparency & low overhead for clients
Protocol packet
COM_QUERY
select * from t

App
Connector
Protocol packet
OK
ResultSet Rows:1



Simple Read/Write Splitting

• Very fast, easy-to-understand scaling model

• Transactions go to master

• Auto-commit SELECTs go to slaves, if
available, but...

• Reads potentially outdated data

• Ephemeral SQL objects won’t be available

• May miss: select update_function()



SmartScale Session Load Balancing
Master Transaction
DBMS History Log

Connect/Write
Session x
App Session y
Connector
Slave
replay
position

Slave
DBMS



Master Transaction
DBMS History Log

Read
Session x
App Session y
Connector
Slave
replay
position

Slave
DBMS



Master Transaction
DBMS History Log

Read
Session x
App
Connector
Slave
replay
position

Slave
DBMS



Multi-Site Data Service



Multi-Master Pattern

• Updates on 2+ sites (active-active mode)

• Enables geographic distribution of data

• No failover necessary if network fails or site
becomes unavailable

• Not all applications can handle multi-master
• Applications must avoid con!icts

• Careful testing required

• Restoration of broken systems may not be easy



Simple Multi-Master Con!guration

NYC Replicator Frankfurt
Replicator
fra (slave) fra (master)

nyc (master) nyc (slave)

Database-to-Database



Clustered Multi-Master Con!guration

NYC Frankfurt

master master
fra nyc

slave fra nyc slave




NYC Frankfurt

slave master
fra nyc

master fra nyc slave

NYC Master Ofﬂine




NYC Frankfurt

master master
fra nyc

slave fra nyc slave

NYC Replicator Ofﬂine



Complex Multi-Master Topologies

All
Masters

Star
Replication



Primary/DR Pattern

• Updates go to primary site master

• DR site receives updates in real time

• Simple command promotion

• Tungsten connectors can route to primary
and DR sites

• No application changes required

• Simplest way to protect against region/site
failures



Tungsten Composite Data Services
App Server App Server
+ Connector + Connector

San Jose New York

sj_nyc
Master Slave



Real-Time Data Bridge



Problem: Real-Time Order Ful!llment

Web-Facing Sales Purchase Ordering

PHP/ Java/
Perl JBoss

1-2M transactions daily 3M
Order
Items
Real-Time Replication
MySQL Oracle



How Does Tungsten Do That?

MySQL Tungsten Master Tungsten Slave Oracle
Replicator Replicator
Service oracle Service oracle

MySQLExtractor Special Filters
Special Filters * Ignore extra tables
* Transform enum to string * Map names to upper case
* Optimize updates to
MySQL remove unchanged columns
Binlog

binlog_format=row



Batch Loading to Data Warehouses

MySQL Vertica
Master Data
Replicator Warehouse
Service batch
London

Transaction Csv Csv Csv
History Log THL
File File ... File



Possibilities for Analytics...

Complex, near
real-time
MySQL Master reporting

Light-weight,
real-time
operational
status
OLTP Data
Web-facing mini-
data marts for
SaaS users



Getting Started



How Do I Obtain Tungsten?

• Register at Continuent to get downloads for Oracle
replication:
http://www.continuent.com/downloads/software

• Documentation is available on Continuent website

• MySQL replication is available from Tungsten
Replicator project on code.google.com
http://code.google.com/p/tungsten-replicator/



560 S. Winchester Blvd., Suite 500 Our Blogs:
San Jose, CA 95128 http://scale-out-blog.blogspot.com
Tel +1 (866) 998-3642 http://datacharmer.org/blog
Fax +1 (408) 668-1009 http://www.continuent.com/news/blogs
e-mail: sales@continuent.com

Continuent Web Page:
http://www.continuent.com

Tungsten Replicator 2.0:
http://code.google.com/p/tungsten-replicator

©Continuent 2012.


State of the art of MySQL replication and clustering

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à State of the art of MySQL replication and clustering

Similaire à State of the art of MySQL replication and clustering (20)

Plus de Giuseppe Maxia

Plus de Giuseppe Maxia (20)

Dernier

Dernier (20)

State of the art of MySQL replication and clustering