MySQL cluster workshop

MySQL Cluster
Workshop
王郁萍 2017-12-09

Agenda
» Basics concepts of MySQL
» About MySQL HA
» MySQL NDB Cluster overview
» Installation
» Configuration of NDB Cluster
» NDB Cluster programs
» Management of NDB Cluster
» NDB Cluster replication

連接層
SQL層
儲存層
MySQL Architecture

MySQL
伺
服
器
端
連接層
通訊協定處理、線程處理、身份驗證、安全性的驗證
SQL層
權限判斷、SQL解析、優化路徑、資料緩存、查詢執
行及返回等
儲存層
儲存引擎
- 硬碟：InnoDB、MyISAM、TokuDB等
- 記憶體：MEMORY
- 網路：NDB
日誌：Slow、Binary、Error、General、Redo等
Applications
(JDBC, ODBC, Python…etc)
MySQL Architecture

Connectors
C, JDBC, ODBC, .NET, PHP …etc
Per-Thread buffer
- sort_buffer_size
- read_buffer_size
- join_buffer_size
- tmp_table_size
Privileges Cache
MySQL Server
Thread Cache
Holds pre-allocated threads for quick re-use
Query Cache
Holds query results as preformatted network packets
Parser
MyISAM Key Cache
Holds MyISAM
index data in
memory for reuse
InnoDB Buffer Cache
Holds InnoDB data and
index pages within
memory for reuse
Table Cache File
descriptor (MyISAM)
Session Level，即每個
線程都會分配一次
Global Level，即開機時只
會分配一次，全域共用
Memory Usage

MySQL HA Overview
Master Slave
AP
R/W R
Active Stand by
AP
R/W
Failover
Master Master
AP
R/W
Master
R/W
R/W
MySQL
HA
讀寫
分流
主備
架構
多主
架構
MySQL
Cluster
AP
R/W
Data Node
SQL Node
Data Node
SQL Node
R/W

Master Slave
AP
R/W R
讀寫
分流
多階
架構
一主
多從
一主
一從
AP
R/W
R
Master Slave_1 Slave_2
R
Master
Master/Slave
AP
R/W
Slave_1 Slave_2 Slave_n
Read
MySQL Replication

MySQL Replication
非同步複製

MySQL Replication
半同步複製

主備
架構
Shared
Storage
DRBD
Storage
透過Clusterware軟
體來進行VIP管理、
故障監控以及故障切
換
有單點故障風險
經由底層以Block為
單位將進行資料同步
若有磁區損壞，另一
台也同時會損壞

多主
架構
群組
複製
多源
複製
Master Master
AP
R/W
Master
R/W
R/W
DB_1 /
Master
DB_2 /
Master
AP
R/W
DB_3 /
Master
R/
W
R/W
Slave
Replication
Read
類似方案：
• MySQL Group Replication
• Galera Cluster For MySQL
• Percona Xtradb Cluster
+Servers ≠+Speed
Have to waiting for
other servers commit
均為獨立資料庫
最後統一複製到
單一資料庫中

M
M M
S1 S2 S3 S4 S...
HA M
M M
S1 S2 S3 S4 S...
HA
MySQL connector
Applications
MySQL Router
MySQL connector
Applications
MySQL Router
MySQL
Shell
MySQL
InnoDB
Cluster

MySQL
NDB
Cluster
MySQL
InnoDB
ClusterMySQL
with
Hardware
Cluster

Agenda
» About MySQL HA
» MySQL NDBCluster overview
» Installation

MySQL NDB Cluster Architecture
SQL Node SQL Node SQL Node SQL Node
NDB
NDB
NDB
NDB
MySQL Client PHP Connector/J
NDB
Management
Server
ndb_mgmd
NDB
Management
Client
ndb_mgm
連接層
SQL層
儲存層

NDB
NDB
NDB
NDB
NDB
Management
Server
ndb_mgmd
NDB
Management
Client
ndb_mgm
Heartbeat

NDB
NDB
NDB
NDB
NDB
Management
Server
ndb_mgmd
NDB
Management
Client
ndb_mgm
Heartbeat
若ndb_mgmd異常時，會由其中一個
可運作的SQL Node當成新的仲裁者

NDB Terminology
» Node: A component of NDB Cluster
» Local Checkpoint(LCP)
⋄ Specific to a single node
⋄ Saving all of a node's data to disk
⋄ Usually occurs every fewminutes
» Global Checkpoint(GCP)
⋄ Occurs every fewseconds
⋄ When transactions for all nodes are synchronized
⋄ The REDOlog is flushed to disk
» Fragment
⋄ Contains a portion of a database table
⋄ Also called a partition
» Replica: Fragment's replicas in order to provide redundancy
User Manual: https://dev.mysql.com/doc/ndbapi/en/overview-terminology.html

NDB Terminology
SQL Node SQL Node SQL Node
NDB
NDB
NDB
NDB
1.Write to SQL Node
2.Insert into Data Node
3.SYNC5.GCP
6. Redo Log
4.ACK
Flush
7.GCP
8. Fragment
Write
Default Replica is 2
Sync 2 Data Nodes

»Management node
⋄To manage the other nodes
⋄Providing configuration data
⋄Performing starting and stopping nodes
⋄Running backups
⋄Arbitrator
NDB
Management
Server
ndb_mgmd
NDB
Management
Client
ndb_mgm
MySQL Cluster Components

»Management node Client
⋄Connect to the management server
⋄Provide commands:
⋄Start and stop nodes
⋄Showing node versions and status
⋄Starting and stopping backups
NDB
Management
Server
ndb_mgmd
NDB
Management
Client
ndb_mgm

»Data node
⋄Stores cluster data
⋄Located on a separate computer
⋄Stored completely in memory
⋄There are as many data nodes as there are
replicas, times the number of fragments

»SQL node
⋄Accesses the cluster data
⋄A traditional MySQL server
⋄Uses the NDBCLUSTER storage engine
⋄Just a specialized type of API node
⋄Standard MySQL Client
⋄ndb_restore
⋄NDB API:C++API, ClusterJ API...etc
NDB API
NoSQL
Interface

Types of installation
» MySQL Cluster Auto-Installer
» Traditionway
» MySQL Cluster Manager(MCM)

Supported Platforms
Operating System Architecture 7.5 7.4
Red Hat / Oracle Linux 7 x86_64 V V
Red Hat / Oracle Linux 6 x86_32, x86_64 V V
Red Hat / Oracle Linux 5 x86_32, x86_64 V X
Ubuntu 16.04 LTS x86_32, x86_64 V X
Ubuntu 14.04 LTS x86_32, x86_64 V X
SUSE Enterprise Linux 12 x86_64 V V
SUSE Enterprise Linux 11 (11.4+) x86_64 X V
Debian GNU/Linux 8 x86_32, x86_64 V X
Microsoft Windows 2012 Server (incl R2) x86_64 V X
Microsoft Windows 2008 Server (incl R2) x86_64 X V

Requirements
»Supported Web browsers
» Required software—setup host
⋄ Python 2.6 or higher
⋄ Paramiko 1.7.7.1 or higher
⋄ Pycrypto version 2.6 or higher
» Authentication and security
⋄ SSH
⋄ HTTPS

shell> ndb_setup.py
Create New NDB Cluster
Continue Previous
Cluster Configuration

定義Cluster名稱
Cluster所有host IP
寫入負載
Low, Medium, High
選擇應用程式的類別
Simple testing
Web, Real-Time
步驟一：定義Cluster 步驟二：定義Hosts
它會自動帶出前面所設
定的Host IP內容
若有不正確或找不到的
情況，可以手動修改

會自動依前面所
設定的Host數量
來決定幾個節點
也可以自行修改
節點的數量/類型
步驟三：定義Processes

步驟四：定義屬性
可以針對各節點
的內容做修改

若設定沒問題，直接執行
Deploy and start cluster
就會開行安裝並啟動
步驟四：部署設定

shell> tar -zxvf /opt/mysql-cluster-version-x86_64.tar.gz
shell> ln -s /opt/mysql-cluster-version-x86_64 /opt/data_node
shell> chown -R mysql:mysql /opt/data_node/
shell> su - mysql
shell> cd /opt/data_node
shell> ndbmtd -c mgm_ip --bind-address=ndb_ip --initial
shell> tar -zxvf /opt/mysql-cluster-version-x86_64.tar.gz
shell> ln -s /opt/mysql-cluster-version-x86_64 /opt/mgm_node
shell> chown -R mysql:mysql /opt/mgm_node/
shell> su - mysql
shell> cd /opt/mgm_node
shell> vi config.ini
shell> ./bin/ndb_mgmd -f <config_file> --configdir=</path/to/config>
--bind-address=<IP,IP2> --initial
shell> ./bin/ndb_mgm -c <IP>
管理節點
資料節點
所有資料節
點同時啟動

shell> cd /tmp
shell> tar -C /var/lib/ -xzvf mysql-cluster-version-x86_64.tar.gz
shell> ln -s /var/lib/mysql-cluster-version-x86_64.tar.gz
/var/lib/mysql
shell> chown -R mysql:mysql /var/lib/mysql/
shell> su - mysql
shell> cd /var/lib/mysql
shell> mysqld --datadir=/var/lib/mysql/data --initialize
shell> vi /etc/my.cnf
shell> ./bin/mysqld_safe --defaults-file=/etc/my.cnf --ndbcluster
--ndb-connectstring=<mgmIP> &
SQL節點

shell> /opt/mgm_node/bin/ndb_mgm -c mgmIP1,mgmIP2 -e"show"
管理節點-檢查狀態

MySQL Cluster Architecture
NDB
NDB
NDB
NDB
NDB
Manager
mcmd
NDB
Manager
Client
mcm
mcmd agent mcmd agent mcmd agent mcmd agent
mcmd agent
mcmd agent
mcmd agent
mcmd agent

Concept of MCMNDB
Manager
mcmd
Site
ndb-7.3
ndb-7.4
ndb-7.5
PackagesCluster
To define the set of MySQL
NDB Cluster nodes (processes)
The set of hosts to be managed
Provide the location of the MySQL
NDB Cluster software on all hosts

shell> cd /opt
shell> tar -xzvf mysql-cluster-mcm-version-x86_64.tar.gz
shell> mv mysql-cluster-mcm/mcm_1.4.4 /opt/mcm-1.4.4
Shell> mv mysql-cluster-mcm/cluster /opt/ndb-7.5.7
shell> chown -R mysql:mysql /opt/mcm-1.4.4/ /opt/ndb-7.5.7
shell> su - mysql
shell> cd /opt/mcm-1.4.4
shell> vi etc/mcmd.ini
shell> /opt/mcm-1.4.4/etc/init.d/mcmd start
MCM安裝
所有節都
需要安裝

shell> /opt/ndb-7.5.7/bin/mysql -u<user> -p<pwd> -h<IP> -P<port>
--prompt="mcm> "
mcm> create site -h
192.168.0.1,192.168.0.2,192.168.0.3,192.168.0.4,192.168.0.5 mysite;
+---------------------------+
| Command result |
+---------------------------+
| Site created successfully |
+---------------------------+
1 row in set (40.52 sec)
建立Site

--prompt="mcm> "
mcm> add package -b /opt/ndb-7.5.7 ndb-7.5.7;
+----------------------------+
| Command result |
+----------------------------+
| Package added successfully |
+----------------------------+
建立package

--prompt="mcm> "
mcm> create cluster -P ndb-7.5.7 -R
ndb_mgmd:1@192.168.0.5,ndbmtd:4@192.168.0.1,ndbmtd:5@192.168.0.2,n
dbmtd:6@192.168.0.3,ndbmtd:7@192.168.0.4,mysqld:21@192.168.0.1,mys
qld:22@192.168.0.2,mysqld:23@192.168.0.3,mysqld:24@192.168.0.4,mys
qld:25@*,mysqld:26@* mycluster;
+------------------------------+
| Command result |
+------------------------------+
| Cluster created successfully |
+------------------------------+
建立cluster

--prompt="mcm> "
mcm> start cluster mycluster;
+------------------------------+
| Command result |
+------------------------------+
| Cluster started successfully |
+------------------------------+
啟動cluster

--prompt="mcm> "
mcm> show status -r mycluster;
Cluster狀態

Agenda
» About MySQL HA
» Installation
» Configuration of NDBCluster

Configuration file
»Two types of configuration file
⋄ NDB management node: config.ini
⋄ SQL Node: my.cnf
»Beginning from[xxx]
⋄ [NDBDDEFAULT], [NDBD]
⋄ [MGM DEFAULT], [NDB_MGMD]
⋄ [MYSQLD]
User Manual: https://dev.mysql.com/doc/refman/5.7/en/mysql-cluster-quick.html

[MGM DEFAULT]
PortNumber=1186
DataDir=/opt/mysql/mgm_data
[NDB_MGMD]
NodeId=1
HostName=192.168.1.54
LogDestination=FILE:filename=ndb_1_cluster.log,maxsi
ze=100000000,maxfiles=6
Defining MGM Node
所有管理節點相同
的設定，可以放在
MGM DEFAULT裡

[NDBD DEFAULT]
NoOfReplicas=2
DataDir=/opt/mysql/ndb_data
FileSystemPath=/opt/mysql/ndb_file
BackupDataDir=/logs/mysqlcluster/backup/
DataMemory=4028M
IndexMemory=1024M
LockPagesInMainMemory=1
[NDBD]
NodeId=4
HostName=192.168.1.56
Defining Data Node
所有資料節點相同
的設定，可以放在
NDB DEFAULT裡

»Node Group
⋄Consists of one or more nodes, and stores
partitions, or sets of replicas
⋄Number of Redundancy:
[# of node groups] =[# of data nodes] / NoOfReplicas
»Replica
⋄copy of a cluster partition
Defining Data Node
Node Group 0
Node Group 1
4 Data Nodes:
NoOfReplicas=1
 4 node groups
NoOfReplicas=2
 2 node groups
NoOfReplicas=4
 1 node group

Defining Data Node
Primary
Backup

[MYSQLD]
NodeId=11
HostName=192.168.1.58
[MYSQLD]
====================================================
# my.cnf
[mysqld]
ndbcluster
[mysql_cluster]
connect-string=192.168.1.54
config-file=/etc/config.ini
Configure SQL Node
[MYSQLD]裡不見
得一定要設定內容
建議多放幾個空的
區域給MGM備用

Agenda
» About MySQL HA
» Installation
» NDBCluster programs

Commonly used
» ndbd - The NDB Cluster Data Node Daemon
» ndbmtd - The NDB Cluster Data Node Daemon (Multi-Threaded)
» ndb_mgmd - The NDB Cluster Management Server Daemon
» ndb_mgm- The NDB Cluster Management Client
» ndb_restore - Restore an NDB Cluster Backup

ndb_mgmd, ndb_mgm
ndb_mgmd -f <config_file> --configdir=</path/to/config>
--bind-address=<IP:port,IP2:port> --initial
ndb_mgmd -f <config_file> --configdir=</path/to/config>
--bind-address=<IP:port,IP2:port> --reload
ndb_mgm -c <IP:port>
ndb_mgm> show;
ndb_mgm> shutdown;
ndb_mgm> <ids> stop;
ndb_mgm> <ids> restart;
ndb_mgm> start backup;
ndb_mgm> help;

ndbd, ndbmtd
ndbmtd -c <mgm_IP> --bind-address=<IP> --initial
ndbmtd -c <mgm_IP> --bind-address=<IP>
ndbmtd -c <mgm_IP> --bind-address=<IP> --nowait-nodes=ids

ndb_restore
shell> ndb_restore --nodeid=1 --backupid=1 --restore_data
--backup_path=<BKPath>/BACKUP/BACKUP-1 --disable-indexes
--backup_path=<BKPath>/BACKUP/BACKUP-1 --disable-indexes
--backup_path=<BKPath>/BACKUP/BACKUP-1 --rebuild-indexes
User Manual: https://dev.mysql.com/doc/mysql-cluster-excerpt/5.7/en/mysql-cluster-programs-ndb-restore.html

For Table used
» ndb_delete_all - Delete All Rows from an NDB Table
» ndb_desc - Describe NDB Tables
» ndb_drop_index - Drop Index from an NDB Table
» ndb_drop_table - Drop an NDB Table
» ndb_index_stat - NDB Index Statistics Utility
» ndb_move_data - NDB Data Copy Utility
» ndb_select_all - Print Rows from an NDB Table
» ndb_select_count - Print RowCounts for NDB Tables
» ndb_show_tables - Display List of NDB Tables

ndb_error_reporter
/opt/ndb-7.5.7/bin/ndb_error_reporter ./config.ini
2017-12-05 23:49:51 [ndb_config] WARNING -- at line 337: Cluster
configuration warning:
arbitrator with id 1 and db node with id 7 on same host 192.168.101.75
Running arbitrator on the same host as a database node may
cause complete cluster shutdown in case of host failure.
root@192.168.101.75's password:
ndb_1_cluster.log 100% 419KB 5.5MB/s 00:00
ndb_1_out.err 100% 0 0.0KB/s 00:00
ndb_1_out.log 100% 421KB 14.2MB/s 00:00
2017-12-05 23:57:44 [ndb_config] WARNING -- at line 337: Cluster
configuration warning:
.....
Copying configuration file...
/opt/mcm-1.4.4/clusters/mycluster/1/cfg/config.ini
Please attach ndb_error_report_20171205235655.tar.bz2 to your error report

Agenda
» About MySQL HA
» Installation
» Management of NDBCluster

NDB
NDB
NDB
NDB
NDB
Management
Server
ndb_mgmd
NDB
Management
Client
Ndb_mgm
Start in order

Start types
» Initial start
⋄ The cluster starts with a clean file system on all data nodes
» System restart
⋄ This occurs when the cluster has been shut down
» Node restart
⋄ This is the online restart of a cluster node
» Initial node restart
⋄ The node is reinitialized and started with a clean file system

Start phases
» Phase -1
» Phase 0
» Phase 1
» Phase 2
» Phase 3
» Phase 4
» Phase 5
User Manual: https://dev.mysql.com/doc/refman/5.7/en/mysql-cluster-start-phases.html
» Phase6
» Phase 7
» Phase 8
» Phase 9
» Phase 100
» Phase 101

Start phases
» Setup and initialization (phase -1)
1. Obtain a node ID
2. Fetch configuration data
3. Allocate ports to be used for inter-node communications
4. Allocate memory according to settings obtained from the
configuration file

Start phases
» Phase 0
⋄ The NDBFS and NDBCNTR blocks start
⋄ Data node file systems are cleared(With --ininialize)
» Phase 1
⋄ All remaining NDB kernel blocks are started
⋄ Connections are set up
⋄ Inter-block communications are established
⋄ Heartbeats are started
⋄ API node connections are also checked(Node restart)

Hang in Phase 1
» Usually network problems
⋄ Having multiple network interfaces
⋄ Blocking of TCP/IP ports needed

Start phases
» Phase 2
⋄ Checks the states of all existing nodes
⋄ The master node is chosen
⋄ The cluster schema file is initialized
» Phase 3
⋄ For initial starts, the log files must be initialized
⋄ Obtains the states of the data nodes

Start phases
» Phase 4
⋄ For initialize or initialize restart
• The redo log files are created
• The number of these files is equal to NoOfFragmentLogFiles
⋄ For a system restart:
• Read schema or schemas
• Read data from the local checkpoint
• Apply all redo information until the latest restorable global
checkpoint has been reached
⋄ For a node restart, find the tail of the redo log

Common Errors
» Reconfigure NoOfFragmentLogFiles without --initial-start
» Local checkpoint(LCP) files are broken
» Global checkpoint(GCP) files are broken
» Redo log files are broken
More Details:
https://dev.mysql.com/doc/ndb-internals/en/ndb-internals-start-phases-
system-restart-phase-4.html

Start phases
» Phase 5
⋄ For an initial start or system restart
• Local checkpoint(LCP) is executed
• Global checkpoint(GCP) is executed
⋄ Checks of memory usage
⋄ Any required node takeovers are performed
» Phase 6
⋄ Node groups are defined and set up

Start phases
» Phase 7
⋄ The arbitrator node is selected and begins to function
⋄ The next backup IDis set
⋄ Nodes reaching this start phase are marked as Started
⋄ It is nowpossible SQL nodes to connect to the cluster
» Phase 8
⋄ If this is a system restart, all indexes are rebuilt

Start phases
» Phase 9
⋄ The node internal startup variables are reset
» Phase 100
⋄ During a node restart or initial node restart, SQL nodes could
connect to the node and begin to receive events
» Phase 101
⋄ The SUMA handover phase
⋄ GCP is negotiated and used as a point of reference for changing
the source of event

Data node log files
nodeid
ndb_node_id_out.log
ndb_node_id.pid
data-nodeid.dat undo-nodeid.dat
ndb_nodeid_fs
1 2
LCP
DBDICT D8
D1
D9
D10 D11DBDI
Define as datadir
in config.ini
Define as FileSystemPath in config.ini
Data dictionary,
GCI, Redo Log

Upgrade
User Manual: https://dev.mysql.com/doc/mysql-cluster-excerpt/5.7/en/mysql-cluster-upgrade-downgrade.html
mgm>1 stop
2.STOP mgm_node
shell>ndb_mgmd --initial
3.RESTART mgm_node
4.Rolling restart
Data Nodes
with --initial
5.Rolling restart
SQL Nodes
1.Replcae package
on each node
6.Running mysql_upgrade

Add data node
User Manual: https://dev.mysql.com/doc/refman/5.7/en/mysql-cluster-online-add-node-basics.html
1.Edit config.ini
mgm>1 stop
2.STOP mgm_node
shell>ndb_mgmd --reload
3.RESTART mgm_node
4.Rolling restart
Data Nodes
5.Rolling restart
SQL Nodes
6.Start the new data nodes
NDB
NDB
NDB
NDBNDB
NDB
NDBNDB
7.Execute CREATE NODEGROUP
commands mysql> ALTER TABLE ... ALGORITHM=INPLACE,
REORGANIZE PARTITION
8.Redistribute partitions

BackupNDB
Management
Client
ndb_mgm
shell> ndb_mgm -e "START BACKUP "
ndb_mgm> START BACKUP
Waiting for completed, this may take
several minutes
Node 2: Backup 1 started from node 1
Node 2: Backup 1 started from node 1
completed
StartGCP: 177 StopGCP: 180
#Records: 7362 #LogRecords: 0
Data: 453648 bytes Log: 0 bytes
ndb_mgm>
User Manual: https://dev.mysql.com/doc/refman/5.7/en/mysql-cluster-backup-using-management-client.html

Restore
User Manual: https://dev.mysql.com/doc/mysql-cluster-excerpt/5.6/en/ndb-restore-to-more-nodes.html
1.Restore schema 2.Restore the data 3. Rebuild the indexes
ndb_restore --nodeid=1 --backupid=1
--restore_data --backup_path=<PATH>/BACKUP-1
--disable-indexes
ndb_restore --nodeid=1 --backupid=1
--restore_data --backup_path=<PATH>/BACKUP-1
--rebuild-indexes

Agenda
» About MySQL HA
» Installation
» NDBCluster replication

# In Master Server
shell> mysql -u<user> -p<pwd>
mysql> create user 'repl'@'172.16.0.%';
Query OK, 0 rows affected (0.01 sec)
mysql> set password for 'repl'@'172.16.0.%'=password('repl');
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> grant replication slave on *.* to 'repl'@'172.16.0.%';
Query OK, 0 rows affected (0.00 sec)

# In Slave Server
mysql> use mysql;
mysql> CREATE TABLE `mysql`.`ndb_apply_status` (
`server_id` int(10) unsigned NOT NULL,
`epoch` bigint(20) unsigned NOT NULL,
`log_name` varchar(255) CHARACTER SET latin1 COLLATE latin1_bin
NOT NULL,
`start_pos` bigint(20) unsigned NOT NULL,
`end_pos` bigint(20) unsigned NOT NULL,
PRIMARY KEY (`server_id`) USING HASH
) ENGINE=innodb;

# In Slave Server
mysql> change master to master_host='192.168.0.4',
master_port=3306, master_log_file='mysql-bin.000001',
master_log_pos=120, master_user='repl', master_password='repl';
mysql> start slave;
mysql> show slave statusG
*************************** 1. row ***************************
...(以上資料略過)...
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
...(以下資料略過)...
＊檢查上面2個參數是否為Yes，若是則代表設定成功

THANKS!
Any questions?
You can find meat
» jeaimerwang@gmail.com

MySQL cluster workshop

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à MySQL cluster workshop

Similaire à MySQL cluster workshop (20)

Plus de 郁萍王

Plus de 郁萍王 (16)

Dernier

Dernier (20)