MySQL Cluster 8.0 tutorial text

============================================================
              MySQL Cluster 8.0 tutorial
              MySQL preFOSDEM days 2020
============================================================
This file contains instructions which can be followed as
part of the MySQL preFOSDEM MySQL Cluster 8.0 tutorial.
This is a text file to make copying and pasting as easy as
possible.
Please ask questions !

Contents
1.     Installation of MySQL Cluster 8.0 software 2
1.1      VirtualBox VM 3
1.1.1      Importing the NdbVM appliance to create a VM 3
1.1.2      Using the NdbVM VM 3
1.1.3      Checking the installed software 4
2.     Configuration of MySQL Cluster 5
2.1      Creating the directories 5
2.2      Creating the cluster configuration file 6
2.3      Creating the MySQLD configuration files 7
2.4      Initialising the MySQLD data directories 11
3      Starting MySQL Cluster     13
3.1      Starting the Management server (MGMD) 13
3.1.1      Looking at the MGMD process 14
3.1.2      OPT : Looking at the MGMD files 14
3.2      Starting the Data nodes (NDBD) 14
3.2.1      Looking at the data node processes 16
3.2.2      OPT : Looking at the Data node files 17
3.3      Starting the MySQL Server processes (MySQLD) 18
3.3.1      Connecting as a MySQL client 18
4      Exploring the cluster 20
4.1      Using ndb_mgm 21
4.2      OPT : Looking at logs 23
4.3      Work with data 25
4.3.1      Insert data 26
4.3.2      Read data using SQL 27
4.3.3      Read data using NdbApi 28
4.4      OPT : Look at binary logs 29
5      High Availability 34
5.1      Kill one data node process 35
5.2      OPT : Suspend another data node process 37
5.3      Restart one data node process and kill another 41
5.4      Kill one binlogging MySQLD process 43
5.5      OPT : Kill all MySQLD processes 45
5.6      OPT : Restart all MySQLD processes 46
5.7      Kill all data nodes 47
5.8      Restart all data nodes 49
MySQL Cluster 8.0 tutorial v1.5 1/92

6      Backup and Restore 50
6.1      Taking a backup 51
6.2      OPT : Looking at backup filesets 52
6.3      OPT : Restoring a backup 54
6.3.1      Restoring metadata 55
6.3.2      Restoring data 57
6.3.3      Checking restored data 59
7      Joins, Foreign keys, JSON 60
7.1      Joining 61
7.2      Foreign keys 64
7.3      OPT : JSON 67
8      OPT : NdbInfo views 71
9      OPT : Synchronized Privileges 77
10     OPT : Addnode and rebalance data 80
10.1     Add nodes to configuration 82
10.2     Perform rolling restart of data nodes 84
10.3     Perform rolling restart of the MySQLD nodes 86
10.4     Start the new data nodes and create a nodegroup 87
10.5     Rebalance data across the nodes 89
11     Other features of MySQL Cluster 92

=============================================
1  Installation of MySQL Cluster 8.0 software
=============================================
1.1      VirtualBox VM 3
1.1.1      Importing the NdbVM appliance to create a VM 3
1.1.2      Using the NdbVM VM 3
1.1.3      Checking the installed software 4
MySQL Cluster 8.0 can be deployed on a number of platforms
including Linux, Solaris (SPARC/x86), MacOS and Windows.
On these platforms there are also multiple ways to install
the MySQL Cluster software including :
Using packages from Oracle for your OS
Using packages from your OS distributor
Using Docker container images from Oracle
Using generic binaries from Oracle
Additionally there are a number of ways to setup and run
MySQL Cluster once the software is installed :
Manually
Using MySQL Cluster Manager (MCM)
Using MySQL Cluster Installer
Using dbDeployer or other tools
These options (and more) are documented online at :

   https://dev.mysql.com/doc/refman/8.0/en/mysqlclusterinstallation.html
The documentation acts as a tutorial and should be consulted
if attempting this elsewhere.

1.1  VirtualBox VM
==================
For consistency and efficiency in this tutorial, a VirtualBox
VM 'appliance' has been made available which comprises :
A 'minimal' Ubuntu Desktop 18.04 OS
MySQL Cluster 8.0.19 (Community GPL) installed from Debian
   packages
This VM requires that the free, opensource VirtualBox software
is available on your machine, and that you have sufficient memory
and disk space to run the VM.
VirtualBox software for your system can be downloaded from here
   https://www.virtualbox.org/wiki/Downloads
Documentation including how to install is available here :
   https://www.virtualbox.org/manual/UserManual.html
1.1.1  Importing the NdbVM appliance to create a VM

The steps are :
   1.  Copy the NdbVM.ova virtual appliance file onto your machine
   2.  From the VirtualBox manager GUI, select File > Import Appliance
   3.  Select the NdbVM.ova file
   4.  On the Appliance Settings dialog, ensure that the appliance is
       configured with 4GB of (virtual) RAM
   5.  Click Import and Wait for it to complete
   6.  Select the NdbVM created, and start it up
1.1.2  Using the NdbVM VM

The NdbVM will boot to a Desktop, already logged in as the
Ndb user (username : Ndb password : ndbcluster).
A terminal icon is available to open a terminal window running
the bash shell.
Extra tabs in the terminal can be created by selecting the
File > New Tab option.
The prompt of the bash shell looks like this :
ndb@ndbVirtualBox:~$
As you change directories, the prompt changes to show
the current path :
ndb@ndbVirtualBox:~$ cd cluster1
ndb@ndbVirtualBox:~/cluster1$

When showing commands, the command is the content after the $.
The VM includes a few editors :
Commandline:
  nano
  vi    (Exit vi with ESC :!q or ESC :!wq )
Visual:
  gedit
1.1.3  Checking the installed software

For this tutorial the MySQL Cluster 8.0 software is already
installed.
You can check this by running e.g. :
Shell >

mysql version

    Example output :
      ndb@ndbVirtualBox:~$ mysql version
      mysql  Ver 8.0.19cluster for Linux on x86_64 (MySQL Cluster Community Server
GPL)

Note that the version includes the cluster suffix this
is important as it indicates that we have MySQL Cluster 8.0.19 and
not just MySQL 8.0.19.
1.1.4   Binaries used in this tutorial

   Name                  Description
   ndb_mgmd              Management server process
   ndbmtd                Data node process
   mysqld                MySQL Server
   ndb_mgm               Management client shell
   mysql                 MySQL client shell
   mysqlbinlog           MySQL binlog file tool
   mysqladmin            MySQL Server admin tool

ndb_restore           Backup restoration tool
   ndb_desc              NdbApi table description tool
   ndb_select_all        NdbApi table SELECT * tool
   ndb_select_count      NdbApi table SELECT COUNT(1) tool
   ndb_error_reporter    Error report gathering tool
   ndbinfo_select_all    NdbInfo tool
   ndb_waiter            State change handling tool
=================================
2  Configuration of MySQL Cluster
=================================
2.1      Creating the directories 5
2.2      Creating the cluster configuration file 6
2.3      Creating the MySQLD configuration files 7
2.4      Initialising the MySQLD data directories 11
For this tutorial, MySQL cluster will be setup in a single VM.
This kind of setup is useful for understanding cluster, and for
software development purposes.
A real deployment would spread a cluster across a number of hosts
to take advantage of the High Availability and scalability features
of MySQL Cluster.
The MySQL cluster software binaries are already installed, but
we need a location for the software to store configuration and
data.
For this tutorial we will store the configuration and binaries
in the home directory this is typical of a development
installation.
2.1   Creating the directories
==============================
For this tutorial we want to create a cluster with :

  1 Management node (MGMD)
  4 Data nodes (NDBD)
  3 MySQLD nodes (MYSQLD)
MySQL Cluster is a shared nothing cluster, so we will give each
node its own directory.
Create this directory structure :
Shell >

cd /home/ndb
mkdir /home/ndb/cluster1
mkdir /home/ndb/cluster1/mgmd1
mkdir /home/ndb/cluster1/mgmd1/config

mkdir /home/ndb/cluster1/ndbd1
mkdir /home/ndb/cluster1/mysqld1

2.2   Creating the cluster configuration file
=============================================
Most of the distributed aspects of MySQL Cluster are configured
using a text configuration file.
This file is read by the cluster management node, which then
serves it up to data node or other cluster processes on demand.
This allows most processes in a cluster to be started using only
by the URL of the management server and an identity.
The configuration file below contains a simple configuration
for a 4 node, 2 replica cluster.  There are a few configuration
items related to the amount of memory to allocate, which are
kept low for this tutorial.
To input this configuration file you can either paste the
cat command and all the content up to ENDOFCONF into a shell,
or you can paste the content between #FILESTART and #FILEEND
into the file
  /home/ndb/cluster1/cluster1_conf.ini
Shell >

cat > /home/ndb/cluster1/cluster1_conf.ini <<ENDOFCONF
#FILESTART
# cluster1_conf.ini
[ndbd default]
NoOfReplicas = 2
# Small memory footprint
DataMemory = 50M
TransactionMemory = 20M
DiskPageBufferMemory = 20M
SharedGlobalMemory = 20M
[ndbd]
NodeId = 1
HostName = localhost

DataDir = /home/ndb/cluster1/ndbd1
[ndbd]
NodeId = 2
[ndbd]
NodeId = 3
[ndbd]
NodeId = 4
[ndb_mgmd]
NodeId = 10
DataDir = /home/ndb/cluster1/mgmd1/
# mysqlds
[api]
[api]
[api]
# extras
[api]
[api]
[api]
[api]
#FILEEND
ENDOFCONF

You can check that the config file is created as expected by looking
at it with the more command.
Shell >

more cluster1/cluster1_conf.ini

2.3 Creating the MySQLD configuration files
=============================================
The MySQL Cluster configuration file read by the MGMD defines the

size and topology of the cluster, as well as a number of details
of the operation of the management node and data nodes.
Each MySQL Server (MySQLD) that connects to a cluster is a standalone
MySQL Server, with all of the normal infrastructure and configuration
that implies.
Therefore each MySQL Server needs some configuration of the Server
components as normal, plus some configuration which tells it how
to connect to the cluster.
For this tutorial, we will run the MySQL Cluster management server
on its default port (1186), which means that all of the other cluster
processes (data nodes, MySQLDs etc) will be able to connect to the
management server without specifically setting the management server
URL.
To input this configuration file you can either paste the
cat command and all the content up to ENDOFCONF into a shell,
or you can paste the content between #FILESTART and #FILEEND
into the file
  /home/ndb/cluster1/mysqld1/mysqld.cnf
Shell >

cat > /home/ndb/cluster1/mysqld1/mysqld.cnf <<ENDOFCONF
#FILESTART
# mysqld1
# mysqld.cnf
[mysqld]
#         paths for this MySQLD
datadir = /home/ndb/cluster1/mysqld1/data
socket = /home/ndb/cluster1/mysqld1/mysqld.sock
log_error = /home/ndb/cluster1/mysqld1/mysqld.err
#         Port which this MySQLD will listen on for mysql
#         client protocol connections
port=3001
#         Binary log filename prefix
log_bin = mysqld1_binary
server_id = 1
#         Activate Ndb cluster storage engine
ndbcluster
#         Record Ndb cluster changes in the Binlog
ndb_log_bin = 1
#         Default connectstring, but added for clarity
ndb_connectstring=localhost:1186
#FILEEND
ENDOFCONF

You can check that the config file is created as expected by looking
at it with the more command.
Shell >

more cluster1/mysqld1/mysqld.cnf

Now do the same for the other two MySQLD configuration files.
Note that the third configuration file has ndb_log_bin=0, so that its
binary log will *not* contain changes made to the cluster.
Shell >

#FILESTART
# mysqld2
# mysqld.cnf
[mysqld]
port=3002
server_id = 2

ndbcluster
#         Record Ndb cluster changes in the Binlog
ndb_log_bin = 1
#FILEEND
ENDOFCONF

Shell >

#FILESTART
# mysqld3
# mysqld.cnf
[mysqld]
port=3003
server_id = 3
ndbcluster
#         Do not record Ndb cluster changes in the Binlog
ndb_log_bin = 0
#FILEEND
ENDOFCONF

You can use the more command as described above to check
that the files are as expected.

MySQLD Configuration summary

Three MySQLD config files have been created, one for each
MySQLD we plan to use.
Each MySQLD has its own data directory, socket file, error log
and listening port (3001, 3002, 3002)
All MySQLDs connect to the same cluster URL (localhost:1186).
MySQLDs 1 + 2 are recording Ndb changes in their Binlogs, MySQLD
3 is not.
2.4 Initialising the MySQLD data directories
==============================================
All MySQL Servers require that their data directories are
initialised before they can be started up for normal service.
This can be done before or after setting up MGMD or NDBD nodes.
As it is mostly a onetime activity, we will do it now.
For this tutorial we are using the initializeinsecure
option for simplicity.
A real deployment would normally use a randomly generated
root account password which is put into the mysqld error
log, and can be read from there to allow initial login.
Shell >

cd /home/ndb/cluster1/mysqld1
mysqld initializeinsecure datadir=/home/ndb/cluster1/mysqld1/data log
error=/home/ndb/cluster1/mysqld1/mysqld.err

    Example output
      ndb@ndbVirtualBox:~$ cd /home/ndb/cluster1/mysqld1
      ndb@ndbVirtualBox:~/cluster1/mysqld1$ mysqld initializeinsecure
datadir=/home/ndb/cluster1/mysqld1/data log
      ndb@ndbVirtualBox:~/cluster1/mysqld1$ cd /home/ndb/cluster1/mysqld2

Each initialisation takes a little time, and creates a number of files
in the MySQLD's data directory, for example :
      ndb@ndbVirtualBox:~/cluster1/mysqld3$ ls
      data  mysqld.cnf  mysqld.err
      ndb@ndbVirtualBox:~/cluster1/mysqld3$ ls data/
       auto.cnf          ibdata1         ndbinfo              sys
       cakey.pem        ib_logfile0     performance_schema   undo_001
       ca.pem            ib_logfile1     private_key.pem      undo_002
       clientcert.pem  '#innodb_temp'   public_key.pem
       clientkey.pem    mysql           servercert.pem
       ib_buffer_pool    mysql.ibd       serverkey.pem
We can see that there are a number of Server files, InnoDB files and
database directories created.
Note that the ndbinfo database (directory) is created, indicating that
the mysqld which we have installed is compiled for mysql cluster.
      ndb@ndbVirtualBox:~/cluster1/mysqld3$ mysqld version
      /usr/sbin/mysqld  Ver 8.0.19cluster for Linux on x86_64 (MySQL Cluster Community
Server GPL)

==========================
3   Starting MySQL Cluster
==========================
3.1      Starting the Management server (MGMD) 13
3.1.1      Looking at the MGMD process 14
3.1.2      OPT : Looking at the MGMD files 14
3.2      Starting the Data nodes (NDBD) 14
3.2.1      Looking at the data node processes 16
3.2.2      OPT : Looking at the Data node files 17
3.3      Starting the MySQL Server processes (MySQLD) 18
3.3.1      Connecting as a MySQL client 18
Now that we have the binaries installed, and directories and
configuration files created, we are ready to start the cluster.
The cluster will be started in these steps :
   Start management server (MGMD)
   Start data nodes (NDBD)
   Initialise and start MySQL Servers (MySQLD)
3.1   Starting the Management server (MGMD)
===========================================
The management server needs to be given :
   The path to the cluster configuration file
   A path to a location where it can store processed
    configuration files.
Shell >

cd /home/ndb/cluster1/mgmd1
ndb_mgmd f /home/ndb/cluster1/cluster1_conf.ini
configdir=/home/ndb/cluster1/mgmd1/config

    Example output
      ndb@ndbVirtualBox:~/cluster1/mgmd1$ ndb_mgmd f
/home/ndb/cluster1/cluster1_conf.ini configdir=/home/ndb/cluster1/mgmd1/config
      MySQL Cluster Management Server mysql8.0.19 ndb8.0.19


3.1.1   Looking at the MGMD process

The management server process can be seen using ps e.g.
Shell >

ps ef | grep ndb_mgmd

    Example output
      ndb       1658     1  0 17:17 ?        00:00:01 ndb_mgmd f
/home/ndb/cluster1/cluster1_conf.ini configdir=/home/ndb/cluster1/mgmd1/config
      ndb       1836  1810  0 17:22 pts/1    00:00:00 grep color=auto ndb_mgmd

3.1.2   Looking at the MGMD files

*** OPTIONAL : Read, or follow along in your VM ***
The management server writes a number of files e.g. :
      ndb@ndbVirtualBox:~/cluster1/mgmd1$ ls
      config  ndb_10_cluster.log  ndb_10_out.log  ndb_10.pid
These are :
  ndb_10_cluster.log    A centralised log of interesting events that have
                        occurred on this cluster
  ndb_10_out.log        The output of the MGMD process
  ndb_10.pid            Pid lock file
3.2   Starting the Data nodes (NDBD)
====================================
The data nodes need to be given :
   The URL (connectstring) of the management server
   Their own id (nodeid)
As the management server in this tutorial is using the default
connectstring (localhost:1186), it need not be provided, but
normally it would be.

Confusing point :
    The data nodes are sometimes referred to as NDBD processes
    (Ndb daemon), and there are two data node binaries :
      ndbd     Single Threaded data node binary
      ndbmtd   Multi Threaded data node binary
    In this tutorial we will use the ndbmtd binary, as it is
    preferred in most situations.  We will refer to this as
    'data node' or 'ndbd'.
Shell >

cd /home/ndb/cluster1/ndbd1
ndbmtd ndbnodeid=1
ndbmtd ndbnodeid=2
ndbmtd ndbnodeid=3
ndbmtd ndbnodeid=4

    Example output
      ndb@ndbVirtualBox:~$ cd /home/ndb/cluster1/ndbd1
      ndb@ndbVirtualBox:~/cluster1/ndbd1$ ndbmtd ndbnodeid=1
      20200122 18:41:45 [ndbd] INFO      Angel connected to 'localhost:1186'
      20200122 18:41:45 [ndbd] INFO      Angel allocated nodeid: 1
      ndb@ndbVirtualBox:~/cluster1/ndbd1$
      ndb@ndbVirtualBox:~/cluster1/ndbd1$ cd /home/ndb/cluster1/ndbd2

Once all 4 processes are started, they will e.g.

Form a cluster
   Elect a leader
   Perform internal allocation and initialisation
   Start distributed protocols
   Start providing NdbApi database service
This takes some time.
As this is the first time that data node processes are being started
in these directories, there are no existing files.
The data nodes will detect this, and create the files they need.
On subsequent starts, the data nodes will attempt to reload state
from the files rather than create files.
3.2.1   Looking at the data node processes

The data node processes can be seen using ps e.g.
Shell >

ps ef | grep ndbmtd

    Example output
      ndb@ndbVirtualBox:~/cluster1/ndbd4$ ps ef | grep ndbmtd
      ndb       1849     1  0 17:28 ?        00:00:00 ndbmtd ndbnodeid=1
      ndb       2131  1810  0 17:47 pts/1    00:00:00 grep color=auto ndbmtd

Note that there are 8 ndbmtd processes, and we only started 4!
This is because by default, the data node process that you start
forks itself into two processes.
The parent process is known as the Angel process, and the child
process is known as the Worker.
Having separate processes like this allows some error situations
to be handled in a cleaner way.  The Angel process is single
threaded and does very little other than monitoring the Worker.

The Worker process is multithreaded and implements all of the
logic of the data node.
In the example above, we can see e.g. that process #1849 is the
parent of process #1850, therefore 1849 is the Angel, and 1850 is
the worker for nodeid1.  The same applies to the other processes.
  NodeId    Angel pid    Worker pid
       1         1849          1850
       2         1928          1930
       3         1935          1937
       4         2000          2001
3.2.2   Looking at the Data node files

Each data node writes a number of files in its data directory, e.g. :
      ndb@ndbVirtualBox:~/cluster1/ndbd4$ ls
      ndb_4_fs  ndb_4_out.log  ndb_4.pid
These are :
  ndb_4_fs       A directory containing all of the persistent
                 state files of this data node, e.g. redo logs,
                 checkpoints, schema information, tablespaces,
                 undo log files...
  ndb_4_out.log  The output of the data node process, can be
                 useful in error situations
  ndb_4.pid      Pid lock file
Each data node has a similar directory structure, with the
nodeid in each filename changed (e.g. ndb_3_fs, ndb_2_fs....)
Looking inside the ndb_X_fs directory, we can see some of the
different file types stored there :
      ndb@ndbVirtualBox:~/cluster1/ndbd4$ # Schema files
      ndb@ndbVirtualBox:~/cluster1/ndbd4$ ls ndb_4_fs/D1/DBDICT
      P0.SchemaLog  T1  T2  T3
      ndb@ndbVirtualBox:~/cluster1/ndbd4$ # Redo log files
      ndb@ndbVirtualBox:~/cluster1/ndbd4$ ls ndb_4_fs/D8/DBLQH
      S0.FragLog   S12.FragLog  S15.FragLog  S3.FragLog  S6.FragLog  S9.FragLog
      S10.FragLog  S13.FragLog  S1.FragLog   S4.FragLog  S7.FragLog
      S11.FragLog  S14.FragLog  S2.FragLog   S5.FragLog  S8.FragLog
      ndb@ndbVirtualBox:~/cluster1/ndbd4$ # Checkpoint files
      ndb@ndbVirtualBox:~/cluster1/ndbd4$ ls ndb_4_fs/LCP/0
      T2F1.ctl   T2F3.ctl   T3F1.ctl   T3F3.ctl
      T2F1.Data  T2F3.Data  T3F1.Data  T3F3.Data
The filenames used are minimalistic, but in general Tx refers to
table x and Fy refers to Fragment y.

3.3   Starting the MySQL Server processes (MySQLD)
==================================================
We want to start three MySQLD processes to connect to our
running MGMD and data nodes.  The configuration files and
data directories for these processes have already been
created above.
The MySQLD processes must be given the path to their
configuration files, all other configuration can be
read from there.
Shell >

mysqld defaultsfile=/home/ndb/cluster1/mysqld1/mysqld.cnf &

    Example output
      ndb@ndbVirtualBox:~/cluster1/ndbd4$ cd /home/ndb/cluster1/mysqld1
      ndb@ndbVirtualBox:~/cluster1/mysqld1$ mysqld defaults
file=/home/ndb/cluster1/mysqld1/mysqld.cnf &
      [1] 2586
      ndb@ndbVirtualBox:~/cluster1/mysqld1$
      [2] 2587
      ndb@ndbVirtualBox:~/cluster1/mysqld2$
      [3] 2588

3.3.1   Connecting as a MySQL client

You now have a cluster running in your VM with 3 MySQLDs attached to it.
The MySQLD configurations offer a few ways to connect :
  Server     TCP port      Socket
  MySQLD1    3001          /home/ndb/cluster1/mysqld1/mysqld.sock
As a result of the initializeinsecure option, there is a root
account created on each MySQLD which can connect from localhost.
Using the mysql client, you could connect to MySQLD1 using either
e.g.
  # TCP
  mysql protocol=tcp host=localhost port=3001 uroot
or
  # Socket
  mysql socket=/home/ndb/cluster1/mysqld1/mysqld.sock uroot
For this tutorial, it is recommended to setup some bash aliases
for connecting to each MySQL Server.  This saves on typing and
gives the client on each MySQL Server a different prompt which
helps avoid mistakes.
These aliases are using the Unix socket to connect, and using
the root account for simplicity.
Shell >

alias mysql1="mysql socket=/home/ndb/cluster1/mysqld1/mysqld.sock uroot
prompt="mysql1> ""
prompt="mysql2> ""
prompt="mysql3> ""

To use an alias, just type it in and you will enter a mysql
client shell connected to the MySQLD.
To exit the mysql client shell, type

  exit
e.g.

ndb@ndbVirtualBox:~$ mysql1
      Welcome to the MySQL monitor.  Commands end with ; or g.
      Your MySQL connection id is 14
      Server version: 8.0.19cluster MySQL Cluster Community Server GPL
      Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
      Oracle is a registered trademark of Oracle Corporation and/or its
      affiliates. Other names may be trademarks of their respective
      owners.

      Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

      mysql1> show variables like 'port';
      +++
      | Variable_name | Value |
      +++
      | port          | 3001  |
      +++
      1 row in set (0.00 sec)

      mysql1> exit
      Bye
      ndb@ndbVirtualBox:~$ mysql3
      Welcome to the MySQL monitor.  Commands end with ; or g.
      Your MySQL connection id is 10
      Server version: 8.0.19cluster MySQL Cluster Community Server GPL

      Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

      Oracle is a registered trademark of Oracle Corporation and/or its
      affiliates. Other names may be trademarks of their respective
      owners.

      Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

      mysql3> show variables like 'port';
      +++
      +++
      | port          | 3003  |
      +++
      1 row in set (0.00 sec)

      mysql3> exit
      Bye
Note that if you accidentally kill your shell process, and start another,
you may need to recreate these aliases.
=========================
4   Exploring the cluster
=========================
4.1      Using ndb_mgm 21
4.2      OPT : Looking at logs 23
4.3      Work with data 25

4.3.1      Insert data 26
4.3.2      Read data using SQL 27
4.3.3      Read data using NdbApi 28
4.4      OPT : Look at binary logs 29
In this section we will have a look at the cluster which has been
started.
4.1 Using ndb_mgm
=================
The easiest way to get an overview of your cluster is to use the
management client, known as ndb_mgm.
This ndb_mgm tool provides a command line shell allowing queries
and commands to be sent over TCP to the management server (MGMD).
In this tutorial the management server is running on the default
host and port, so rather than issue :
  ndb_mgm ndbconnectstring=localhost:1186
we can just type :
  ndb_mgm
Shell >

ndb_mgm

    Example output
      ndb@ndbVirtualBox:~$ ndb_mgm
       NDB Cluster Management Client
      ndb_mgm>

The management client can be exited by typing :
  exit
The most commonly used command in ndb_mgm is SHOW.
ndb_mgm >


show

    Example output
      ndb_mgm> show
      Connected to Management Server at: localhost:1186
      Cluster Configuration

      [ndbd(NDB)] 4 node(s)
      id=1 @127.0.0.1  (mysql8.0.19 ndb8.0.19, Nodegroup: 0, *)
      id=2 @127.0.0.1  (mysql8.0.19 ndb8.0.19, Nodegroup: 0)

      [ndb_mgmd(MGM)] 1 node(s)
      id=10 @127.0.0.1  (mysql8.0.19 ndb8.0.19)

      [mysqld(API)] 7 node(s)
      id=11 @127.0.0.1  (mysql8.0.19 ndb8.0.19)
      id=12 @127.0.0.1  (mysql8.0.19 ndb8.0.19)
      id=13 @127.0.0.1  (mysql8.0.19 ndb8.0.19)
      id=14 (not connected, accepting connect from any host)

This output shows the 4 data nodes, 1 MGMD and 3 MySQLDs connected to
our cluster.  All are connected from localhost (127.0.0.1).
As the cluster configuration is set to NoOfReplicas=2, each Nodegroup
contains 2 data nodes, and with 4 nodes, we have 2 Nodegroups (0 and 1)
Note that the ndb_mgm command can take a command to execute as a
parameter, so that it is also possible to run the show command
directly from the shell
e.g. :
Shell >

ndb_mgm e show

ndb_mgm has a number of other commands that can be seen using
the help command, including:

START BACKUP
  STATUS
  STOP
  RESTART
  SHUTDOWN
These commands are useful for managing the cluster.
Some commands take a nodeid as a prefix so that the command
can be targeted to a single node, with ALL meaning to target
to all applicable nodes.
e.g.
      ndb_mgm> 3 status
      Node 3: started (mysql8.0.19 ndb8.0.19)
      ndb_mgm> all status
4.2   Looking at logs
=====================
There are a number of log files that are recorded by MySQL Cluster :
  Type           Description                    Path in tutorial
  Cluster log    Unified cluster log written
/home/ndb/cluster1/mgmd1/ndb_10_cluster.log
                 by MGMD
  Data node log  Data node output log           /home/ndb/cluster1/ndbdXX/ndb_XX_out.log

  MySQLD log     Standard MySQLD error log      /home/ndb/cluster1/mysqldYY/mysqld.err
When investigating issues, investigation normally starts at the highest
level (the clusterlog) and then moves to the more detailed local logs of
the data nodes or MySQLDs if necessary.
Have a look at each type of log to get a sense of what they
contain.
Shell >


more /home/ndb/cluster1/mgmd1/ndb_10_cluster.log

    Example output
      ndb@ndbVirtualBox:~$ more /home/ndb/cluster1/mgmd1/ndb_10_cluster.log
      20200122 18:12:53 [MgmtSrvr] INFO      Got initial configuration from
'/home/ndb/cluster1/cluster1_conf.ini', will try to set it when all ndb_mgmd(s) started
      20200122 18:12:53 [MgmtSrvr] INFO      Node 10: Node 10 Connected
      20200122 18:12:53 [MgmtSrvr] INFO      Id: 10, Command port: *:1186
      20200122 18:12:53 [MgmtSrvr] INFO      MySQL Cluster Management Server mysql
8.0.19 ndb8.0.19 started
      20200122 18:12:53 [MgmtSrvr] INFO      Node 10 connected
      20200122 18:12:53 [MgmtSrvr] INFO      Starting initial configuration change
      20200122 18:12:53 [MgmtSrvr] INFO      Configuration 1 commited
      ...

Shell >

more /home/ndb/cluster1/ndbd1/ndb_1_out.log

    Example output
      ndb@ndbVirtualBox:~$ more /home/ndb/cluster1/ndbd1/ndb_1_out.log
      20200122 18:13:21 [ndbd] INFO      Angel pid: 2338 started child: 2339
      20200122 18:13:21 [ndbd] INFO      Normal start of data node using checkpoint
and log info if existing
      20200122 18:13:21 [ndbd] INFO      Configuration fetched from
'localhost:1186', generation: 1
      20200122 18:13:21 [ndbd] INFO      Changing directory to
'/home/ndb/cluster1/ndbd1'
      ThreadConfig: input:  LockExecuteThreadToCPU:  => parsed: main,ldm,recv,rep
      20200122 18:13:21 [ndbd] INFO      MaxNoOfTriggers set to 1400
      NDBMT: MaxNoOfExecutionThreads=4
      NDBMT: workers=1 threads=1 tc=0 send=0 receive=1
      20200122 18:13:21 [ndbd] INFO      NDB Cluster DB node 1
      20200122 18:13:21 [ndbd] INFO      mysql8.0.19 ndb8.0.19
      20200122 18:13:21 [ndbd] INFO      Memory Allocation for global memory pools
Starting
      ...


Shell >

more /home/ndb/cluster1/mysqld1/mysqld.err

    Example output :
      ndb@ndbVirtualBox:~$ more /home/ndb/cluster1/mysqld1/mysqld.err
      20200122T18:08:05.731861Z 0 [System] [MY013169] [Server] /usr/sbin/mysqld
(mysqld 8.0.19cluster) initializing of server in progress as process 2166
      20200122T18:08:09.118723Z 5 [Warning] [MY010453] [Server] root@localhost is
created with an empty password ! Please consider switching off the initializei
nsecure option.
      20200122T18:16:00.772969Z 0 [System] [MY010116] [Server] /usr/sbin/mysqld
(mysqld 8.0.19cluster) starting as process 2586
      20200122T18:16:04.667277Z 0 [Warning] [MY010068] [Server] CA certificate ca.pem
is self signed.
      20200122T18:16:04.822200Z 0 [System] [MY010931] [Server] /usr/sbin/mysqld:
ready for connections. Version: '8.0.19cluster'  socket: '/home/ndb/cluster1/mysq
ld1/mysqld.sock'  port: 3001  MySQL Cluster Community Server GPL.
      20200122 18:16:05 [NdbApi] INFO      Flushing incomplete GCI:s < 75/9
      20200122 18:16:05 [NdbApi] INFO      Flushing incomplete GCI:s < 75/9
      ...

If a data node encounters a bug or crashing situation then it will
additionally write two extra types of files :
  Data node error log  (ndb_XX_error.log)
  Data node trace log  (ndb_XX_trace.log.n)
The data node error log contains a short summary of the error, and
the data node trace log files contain detailed trace information
allowing MySQL Support to investigate further.
If support is required then the ndb_error_reporter tool can be used
to gather cluster logs, data node logs, error and trace files for
upload to MySQL Support.
4.3   Work with data
====================
Let's create a database, then create a table and insert
some data, and then look at that data.
We will use SQL to do this work, and refer to clients on
different MySQLD servers (1..3) as mysql1, mysql2, mysql3.

The aliases setup in section 3.3.1 can be used to quickly
start a mysql client connected to a given server.
4.3.1   Insert data

mysql1 >

create database tutorial;
use tutorial;
# Note the 'engine=ndb' suffix
create table attendees(person_id int primary key, name varchar(30)) engine=ndb;
insert into attendees values (1, "P Calculus");
insert into attendees values (2, "B Castafiore");
insert into attendees values (3, "Snowy Dog");
insert into attendees values (4, "Nestor");
insert into attendees values (5, "Rastapopoulos");
insert into attendees values (6, "Dr Muller");
insert into attendees values (7, "Alcazar");
insert into attendees values (8, "Ottokar");
insert into attendees values (9, "R Rackham");

    Example output
      mysql1> create database tutorial;
      Query OK, 1 row affected (0.14 sec)
      mysql1>
      mysql1> use tutorial;
      Database changed
      mysql1>
      mysql1> # Note the 'engine=ndb' suffix
      mysql1> create table attendees(person_id int primary key, name varchar(30))
engine=ndb;
      Query OK, 0 rows affected (0.53 sec)
      mysql1>
      mysql1> insert into attendees values (1, "P Calculus");
      mysql1> insert into attendees values (2, "B Castafiore");
      mysql1> insert into attendees values (3, "Snowy Dog");

...
      mysql1> insert into attendees values (9, "R Rackham");

4.3.2   Read data using SQL

Now we can look at the data using SQL :
mysql1 >

select * from attendees;

    Example output :
      mysql1> select * from attendees;
      +++
      | person_id | name          |
      +++
      |         9 | R Rackham     |
      |         7 | Alcazar       |
      |         3 | Snowy Dog     |
      |         6 | Dr Muller     |
      |         2 | B Castafiore  |
      |         4 | Nestor        |
      |         5 | Rastapopoulos |
      |         1 | P Calculus    |
      |         8 | Ottokar       |
      +++
      9 rows in set (0.00 sec)

Note that Ndb does not necessarily return rows in
insert order, or in a stable order, unless an explicit
ORDER BY is given.
Now let's show that the data we have inserted is visible
from all three attached MySQLDs :
mysql2 >


show databases;
use tutorial;

mysql3 >

show databases;
use tutorial;

4.3.3   Read data using NdbApi

MySQL Cluster ships with some tools which allow tables to be read directly
via NdbApi without involving the MySQLD Servers at all.  NdbApi tools
are given a cluster connectstring (URL) among other parameters, and use
that to connect to a MGMD, and then to the data nodes to be able to read
and write data there.
One such tool is ndb_select_all, which acts somewhat like issuing a SQL
SELECT * query.
This tool can be used to show the data we have just inserted.
Shell >

ndb_select_all dtutorial attendees


Example output :
      ndb@ndbVirtualBox:~$ ndb_select_all dtutorial attendees
      person_id name
      9       "R Rackham"
      7       "Alcazar"
      3       "Snowy Dog"
      6       "Dr Muller"
      2       "B Castafiore"
      4       "Nestor"
      5       "Rastapopoulos"
      1       "P Calculus"
      8       "Ottokar"
      9 rows returned

      NDBT_ProgramExit: 0 OK

4.4   Look at binary logs
=========================
The MySQLD configuration is set up so that MySQLD1 and MySQLD2 write
binary logs (binlogs) containing changes to data stored in MySQL
Cluster, but MySQLD3 does not.
Note that MySQLD3 is configured to have a Binary log, but it will
only contain changes made locally on MySQLD3 to local schema objects.
mysql1 >

show variables like '%log_bin';
show binlog events;

    Example output :
      mysql1> show variables like '%log_bin';
      +++
      +++
      | log_bin       | ON    |
      | ndb_log_bin   | ON    |
      | sql_log_bin   | ON    |
      +++

      mysql1> show binlog events;

+++++
+
+
      | Log_name              | Pos  | Event_type     | Server_id | End_log_pos | Info
|
      +++++
+
+
      | mysqld1_binary.000001 |    4 | Format_desc    |         1 |         124 | Server
ver: 8.0.19cluster, Binlog ver: 4
|
      | mysqld1_binary.000001 |  124 | Previous_gtids |         1 |         155 |
|
      | mysqld1_binary.000001 |  155 | Anonymous_Gtid |         1 |         232 | SET
@@SESSION.GTID_NEXT= 'ANONYMOUS'
|
      | mysqld1_binary.000001 |  232 | Query          |         1 |         352 | create
database tutorial /* xid=77 */
|
|
      | mysqld1_binary.000001 |  431 | Query          |         1 |         605 | use
`tutorial`; create table attendees(person_id int primary key, name varchar(30))
engine=ndb /* xid=82 */ |
|
      | mysqld1_binary.000001 |  684 | Query          |         1 |         755 | BEGIN
|
      | mysqld1_binary.000001 |  755 | Table_map      |         1 |         822 |
table_id: 257 (tutorial.attendees)
|
table_id: 172 (mysql.ndb_apply_status)
|
      | mysqld1_binary.000001 |  894 | Write_rows     |         1 |         959 |
table_id: 172
|
table_id: 257 flags: STMT_END_F
|
      | mysqld1_binary.000001 | 1130 | Query          |         1 |        1202 | COMMIT
|
      +++++
+
+

In the example above we can see
   create database statement
   create table statement
   1 binlog transaction (BEGIN > COMMIT)
    containing
     Two table map events
     Two WRITE_ROWS events including
      on on the tutorial.attendees table
The Server_id data for every event indicates server 1

(aka MySQLD1).
To look more closely at the binlog transaction
we can use the mysqlbinlog tool as follows.
Shell >

mysqlbinlog verbose /home/ndb/cluster1/mysqld1/data/mysqld1_binary.000001

    Example output :
      ndb@ndbVirtualBox:~$ mysqlbinlog verbose
/home/ndb/cluster1/mysqld1/data/mysqld1_binary.000001
      /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
      /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
      DELIMITER /*!*/;
      # at 4
      #200122 21:37:29 server id 1  end_log_pos 124 CRC32 0xf32794c4 Start: binlog v
4, server v 8.0.19cluster created 200122 21:37:29 at startup
      # Warning: this binlog is either in use or was not closed properly.
      ROLLBACK/*!*/;
      BINLOG '
      mcAoXg8BAAAAeAAAAHwAAAABAAQAOC4wLjE5LWNsdXN0ZXIAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
      AAAAAAAAAAAAAAAAAACZwCheEwANAAgAAAAABAAEAAAAYAAEGggAAAAICAgCAAAACgoKKioAEjQA
      CgHElCfz
      '/*!*/;
      # at 124
      #200122 21:37:29 server id 1  end_log_pos 155 CRC32 0xb083b518 PreviousGTIDs
      # [empty]
      # at 155

      ...

      ### INSERT INTO `tutorial`.`attendees`
      ### SET
      ###   @1=3
      ###   @2='Snowy Dog'
      ### SET
      ###   @1=6
      ###   @2='Dr Muller'
      ### SET
      ###   @1=7
      ###   @2='Alcazar'
      ### SET
      ###   @1=9
      ###   @2='R Rackham'
      ...

      ### SET
      ###   @1=2
      ###   @2='B Castafiore'
      ### SET
      ###   @1=4
      ###   @2='Nestor'

      ...

The mysqlbinlog verbose mode shows the contents of the Write_rows
event as pseudoSQL and shows the Inserts that we made.
Now check what MySQLD2 has stored in its Binlogs.
mysql2 >

show binlog events;

    Example output :
      +++
      +++
      | log_bin       | ON    |
      | ndb_log_bin   | ON    |
      +++

      +++++
+
+
|
      +++++
+
+
ver: 8.0.19cluster, Binlog ver: 4
|
|

|
      | mysqld2_binary.000001 |  232 | Query          |         2 |         330 | create
database tutorial
|
|
      | mysqld2_binary.000001 |  407 | Query          |         2 |         559 | use
`tutorial`; create table attendees(person_id int primary key, name varchar(30))
engine=ndb |
|
      | mysqld2_binary.000001 |  638 | Query          |         2 |         709 | BEGIN
|
table_id: 198 (tutorial.attendees)
|
table_id: 153 (mysql.ndb_apply_status)
|
table_id: 153
|
table_id: 198 flags: STMT_END_F
|
      | mysqld2_binary.000001 | 1084 | Query          |         2 |        1156 | COMMIT
|
      +++++
+
+

The output shows that MySQLD2 has equivalent Binlog content to MySQLD1,
including the create database statement, create table statement and
the Binlog data transaction.
Comparing with the Binlog content on MySQLD1, we can see that :
All events logged on MySQLD2 have server_id 2
There are some minor differences between the Binlog
   contents, but they are logically equivalent.
    Therefore the same content may be at different
     offsets in both Binlogs.
Looking at MySQLD3, we don't expect to see any Ndb related content there :
mysql3 >

show binlog events;

    Example output :
      +++
      +++
      | log_bin       | ON    |
      | ndb_log_bin   | OFF   |
      +++

      +++++
++
|
      +++++
++
ver: 8.0.19cluster, Binlog ver: 4 |
|
      +++++
++

=====================
5   High Availability
=====================
5.1      Kill one data node process 35
5.2      OPT : Suspend another data node process 37
5.3      Restart one data node process and kill another 41
5.4      Kill one binlogging MySQLD process 43
5.5      OPT : Kill all MySQLD processes 45
5.6      OPT : Restart all MySQLD processes 46
5.7      Kill all data nodes 47
5.8      Restart all data nodes 49
MySQL Cluster is designed for high availability, and can be configured
with redundant components so that there is no single point of failure
(SPOF).
  Component    Redundancy notes


  MGMD         Two MGMDs recommended
  Data node    1 to 4 replicas in each NodeGroup

MySQLD       n+k redundancy model.  2 or 3 for binlogging
The cluster data nodes monitor each other and the MGMD and API nodes
connected to them using a heartbeating mechanism.
When a node failure is detected, the cluster excludes the failed
node automatically, and aborts any inprogress transactions
or operations involving the failed node.
When a data node recovers and restarts, the cluster automatically
resynchronises it with data from its nodegroup peers, including
it in transaction processing when it has recovered.
MySQL Cluster is designed to remain in service where possible,
but some scenarios can result in a loss of service :
Loss of all nodes in a single nodegroup
   No service with an incomplete dataset.
Network partitioning with no quorum or arbitrator
   Consistency preferred over availability in a network
   partition
5.1   Kill one data node process
================================
For this tutorial, all data node processes are running in the same
VM, but we can identify them by their nodeid= suffix.
We can use this suffix to kill both the Angel and Worker processes
around the same time using the pkill command.
Shell >

ps ef | grep ndbmtd
pkill f nodeid=1
ps ef | grep ndbmtd

    Example output :
      ndb@ndbVirtualBox:~$ ps ef | grep ndbmtd

      ndb@ndbVirtualBox:~$ pkill f nodeid=1

Data node 1 has gone, let's look at the view from ndb_mgm.
Shell >

ndb_mgm e show

    Example output :
      ndb@ndbVirtualBox:~$ ndb_mgm e show

      id=1 (not connected, accepting connect from localhost)

      id=10 @127.0.0.1  (mysql8.0.19 ndb8.0.19)

      id=11 @127.0.0.1  (mysql8.0.19 ndb8.0.19)
      id=12 @127.0.0.1  (mysql8.0.19 ndb8.0.19)
      id=13 @127.0.0.1  (mysql8.0.19 ndb8.0.19)

The data node has gone, but we can see that all the other
cluster nodes are still connected and healthy.

Now let's check that we can still access our data via
MySQLD.
Shell >

mysql1 e "select * from tutorial.attendees"

    Example output:
      ndb@ndbVirtualBox:~$ mysql1 e "select * from tutorial.attendees"
      +++
      +++
      |         4 | Nestor        |
      ...
      |         3 | Snowy Dog     |
      |         6 | Dr Muller     |
      +++
      +++
      +++
      |         4 | Nestor        |
      ...
      |         3 | Snowy Dog     |
      |         6 | Dr Muller     |
      +++

The data is accessible with data node 1 missing.
5.2  Suspend another data node process
======================================
This cluster can survive the failure of a further data node, in
Nodegroup 1 either node 3 or node 4.
If node 2 fails then there will be no data nodes in NodeGroup 0 and
the cluster will as it has an incomplete data set.

Let's kill node 4.
Rather than killing the process directly, this time we will suspend
the process so that we rely on the cluster data node heartbeats
to detect the problem.
This takes longer to detect a failure, but will handle the failure
in a similar time.
First we will suspend the process with a SIGSTOP signal.
Shell >

ps ef | grep ndbmtd
pkill signal=SIGSTOP f nodeid=4

Data node 4 is now suspended, and the other data nodes
will eventually detect this.
Looking in the cluster log we should see messages to this
effect.
Shell >

more /home/ndb/cluster1/mgmd1/ndb_10_cluster.log

    Example output :
      ...
      20200122 22:49:35 [MgmtSrvr] INFO      Node 4 disconnected in state: 0
      20200122 22:49:35 [MgmtSrvr] ALERT     Node 10: Node 4 Disconnected
      20200122 22:49:39 [MgmtSrvr] WARNING   Node 2: GCP Monitor: GCP_COMMIT lag 10
sec (no max lag), [SignalCounter: m_count=0 0000000000000000000000000000000000000000]
      20200122 22:49:41 [MgmtSrvr] WARNING   Node 2: Node 4 missed heartbeat 2
      20200122 22:49:50 [MgmtSrvr] WARNING   Node 2: GCP Monitor: GCP_COMMIT lag 20
sec (no max lag), [SignalCounter: m_count=0 0000000000000000000000000000000000000000]
      20200122 22:49:51 [MgmtSrvr] ALERT     Node 2: Node 4 declared dead due to
missed heartbeat
      20200122 22:49:51 [MgmtSrvr] INFO      Node 3: Communication to Node 4 closed
      20200122 22:49:51 [MgmtSrvr] INFO      Node 2: Communication to Node 4 closed

20200122 22:49:51 [MgmtSrvr] ALERT     Node 2: Arbitration check won all
node groups and more than 1/2 nodes left
      20200122 22:49:51 [MgmtSrvr] INFO      Node 2: President restarts arbitration
thread [state=6]
      20200122 22:49:51 [MgmtSrvr] INFO      Node 2: NR Status: node=4,OLD=Initial
state,NEW=Node failed, fail handling ongoing
      20200122 22:49:51 [MgmtSrvr] INFO      Node 3: NR Status: node=4,OLD=Initial
state,NEW=Node failed, fail handling ongoing
      20200122 22:49:51 [MgmtSrvr] INFO      Node 2: NR Status: node=4,OLD=Node
failed, fail handling ongoing,NEW=Node failure handling complete
      20200122 22:49:51 [MgmtSrvr] INFO      Node 3: NR Status: node=4,OLD=Node
failed, fail handling ongoing,NEW=Node failure handling complete

We can use ndb_mgm to see the cluster state :
Shell >

ndb_mgm e show

    Example output :

      ...

The cluster has handled the failure, and is providing service as
can be seen using the MySQL client
Shell >



Now we should unsuspend data node 4 it has been evicted
by the other nodes and when it resumes it will immediately
shutdown.
As it has been evicted in an error scenario it will record
error and trace logs.
Shell >

pkill signal=SIGCONT f nodeid=4
ps ef | grep ndbmtd
ls /home/ndb/cluster1/ndbd4
more /home/ndb/cluster1/ndbd4/ndb_4_error.log
more /home/ndb/cluster1/ndbd4/ndb_4_trace.log.1

    Example output :
      ndb@ndbVirtualBox:~$ pkill signal=SIGCONT f nodeid=4
      ndb@ndbVirtualBox:~$ ls /home/ndb/cluster1/ndbd4
      ndb_4_error.log  ndb_4.pid             ndb_4_trace.log.1_t2
      ndb_4_fs         ndb_4_trace.log.1     ndb_4_trace.log.1_t3
      ndb_4_out.log    ndb_4_trace.log.1_t1  ndb_4_trace.log.next
      ndb@ndbVirtualBox:~$ more /home/ndb/cluster1/ndbd4/ndb_4_error.log
      Current byteoffset of filepointer is: 1067
      Time: Wednesday 22 January 2020 22:59:45
      Status: Temporary error, restart node
      Message: Node declared dead. See error log for details (Arbitration error)
      Error: 2315
      Error data: We(4) have been declared dead by 2 (via 2) reason: Heartbeat
failure(4)
      Error object: QMGR (Line: 4744) 0x00000002
      Program: ndbmtd
      Pid: 3161 thr: 0
      Version: mysql8.0.19 ndb8.0.19
      Trace file name: ndb_4_trace.log.1
      Trace file path: /home/ndb/cluster1/ndbd4/ndb_4_trace.log.1 [t1..t4]
      ***EOM***

      ndb@ndbVirtualBox:~$ more /home/ndb/cluster1/ndbd4/ndb_4_trace.log.1
      JAM CONTENTS up>down left>right
      SOURCE FILE          LINE  LINE  LINE  LINE  LINE  LINE  LINE  LINE  LINE

DbtcMain.cpp         09866 09866 09866 09866 09866 09866 09866 09866 09866
                           09866 09866 09866 09866 09866 09866 09866 09866 09866
                           09866 09866 09866 09866 09866 09866 09866 09866 09866
                           09866 09866 09866 09866 09866 09866 09866 09866 09866
                           09866 09866 09866 09866 09866 09866 09866 09866 09866
      ...
      > signal
      DbdihMain.cpp        17801
      > signal
      QmgrMain.cpp         00392 04641
      SimulatedBlock.cpp   02089

Signal
r.bn: 252 "QMGR", r.proc: 4, r.sigId: 4985773 gsn: 254 "FAIL_REP" prio: 0
s.bn: 252 "QMGR", s.proc: 2, s.sigId: 5208106 length: 3 trace: 0 #sec: 0 fragInf
: 0
FailedNode: 4, FailCause: 4
Signal
r.bn: 246 "DBDIH", r.proc: 4, r.sigId: 4985772 gsn: 278 "GCP_PREPARE" prio: 0
s.bn: 246 "DBDIH", s.proc: 2, s.sigId: 5202317 length: 3 trace: 2 #sec: 0 fragIn
f: 0
H'00000002 H'00001f11 H'00000007
5.3   Restart one data node process and kill another
====================================================
Restart data node 1 and wait for it to fully start by using
the ndb_mgm e show command repeatedly.
Shell >

ndbmtd ndbnodeid=1
ndb_mgm e show
#...
ndb_mgm e show

    Example output :
      ndb@ndbVirtualBox:~$ cd /home/ndb/cluster1/ndbd1
      ndb@ndbVirtualBox:~/cluster1/ndbd1$ ndb_mgm e show


      ...

      ...
      id=1 @127.0.0.1  (mysql8.0.19 ndb8.0.19, starting, Nodegroup: 0)

      ...

      ...

Data node 1 has recovered and synchronised data from its
Nodegroup peer node 2.
Now we can kill node 2 and see that data is still served.
Shell >

pkill f nodeid=2
ps ef | grep ndbmtd

    Example output :
      ndb@ndbVirtualBox:~$ pkill f nodeid=2
      person_id name

9       "R Rackham"
      7       "Alcazar"
      3       "Snowy Dog"
      6       "Dr Muller"
      2       "B Castafiore"
      4       "Nestor"
      5       "Rastapopoulos"
      1       "P Calculus"
      8       "Ottokar"
      9 rows returned

5.4   Kill one binlogging MySQLD process
========================================
Next we kill one binlogging MySQLD, then start it again.
Shell >

ps ef | grep mysqld
pkill f mysqld1
ps ef | grep mysqld

    Example output :
      ndb@ndbVirtualBox:~$ ps ef | grep mysqld
      ndb       2586     1  0 18:15 ?        00:02:55 mysqld defaults
file=/home/ndb/cluster1/mysqld1/mysqld.cnf
      ndb       4201  3648  0 23:13 pts/2    00:00:00 grep color=auto mysqld
      ndb@ndbVirtualBox:~$ pkill f mysqld1

Show that we can still connect to e.g. MySQLD3 and perform
DDL, modify data.
mysql3 >

use tutorial;
create table incident(report varchar(50)) engine=ndb;
insert into incident values ("Numerous process failures");

Show that we can still connect to MySQLD2 and access data,
and the recent changes are being binlogged by MySQLD2 :
mysql2 >

use tutorial;
select * from incident;
show binlog events;

Now, let's restart the killed MySQLD process.
Shell >



5.5   Kill all MySQLD processes
===============================
Now we will get rid of all of the MySQLDs.
Shell >

killall mysqld
ps ef | grep mysqld
ndb_mgm e show

    Example output :
      ndb@ndbVirtualBox:~$ killall mysqld

      id=10 @127.0.0.1  (mysql8.0.19 ndb8.0.19)


With all the MySQLDs down, we cannot use SQL to access the data, but
we can still use an NdbApi client.
Shell >

ndb_select_all dtutorial incident

    Example output :
      person_id name
      9       "R Rackham"
      7       "Alcazar"
      3       "Snowy Dog"
      6       "Dr Muller"
      2       "B Castafiore"
      4       "Nestor"
      5       "Rastapopoulos"
      1       "P Calculus"
      8       "Ottokar"
      9 rows returned


      ndb@ndbVirtualBox:~$ ndb_select_all dtutorial incident
      report $PK
      "Numerous process failures" 1
      1 rows returned


Data is still available for access over NdbApi with no MySQLDs
available.
5.6   Restart all MySQLD processes
==================================
Restart the 3 MySQLD processes, then check that data is
accessible via SQL again.

Shell >

mysql1 e "select * from tutorial.attendees";

5.7   Kill all data node processes
==================================
Now we will kill all of the data nodes, showing that cluster
data becomes unavailable, though noncluster data is still
accessible.
Shell >

killall ndbmtd
ps ef | grep ndbmtd
ndb_mgm e show
ps ef | grep mysqld

    Example output :
      ndb@ndbVirtualBox:~$ killall ndbmtd


      id=10 @127.0.0.1  (mysql8.0.19 ndb8.0.19)
      ndb       4547  3648  1 23:45 pts/2    00:00:03 mysqld defaults

Notes
   The data nodes have all failed.
   The management node is available.
   The management node reports that no MySQLDs are
    available.
    However they are running, just unable to connect to
    the cluster as the data nodes are not up.
Now try to access cluster table data via mysql, this will fail.
Access to noncluster tables is ok.
Shell >

mysql2 e "select count(1) from mysql.user"

    Expected output :
      ERROR 1296 (HY000) at line 1: Got error 4009 'Cluster Failure' from NDBCLUSTER
      ndb@ndbVirtualBox:~$ mysql2 e "select count(1) from mysql.user"
      ++

MySQL Cluster 8.0 tutorial text

MySQL Cluster 8.0 tutorial text

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à MySQL Cluster 8.0 tutorial text

Similaire à MySQL Cluster 8.0 tutorial text (20)

Plus de Frazer Clement

Plus de Frazer Clement (6)

Dernier

Dernier (20)

MySQL Cluster 8.0 tutorial text