2. About Mydbops
● Founded in 2015, HQ in Bangalore India with 150+ customer base across the globe.
● Mydbops is on Database Consulting with core specialization on MySQL and MongoDB Administration and
Support.
● We have expert team with 20+ certified DBA’s providing full time support and currently managing 300+
servers.
● Mydbops was created with a motto of developing a DevOPS model for Database administration offering
24*7 expert remote DBA support.
● We help organisations to architect and scale systems in MySQL/Mongo by implementing the advanced
technologies in industry which are completely open source.
3. Mydbops is into MySQL/MongoDB Support and Consulting. It is founded by experts
who have scaled database at Yahoo! ,Percona and Datavail. We are providing an
expert level support and 24*7 monitoring for MySQL databases and its related
technologies like MariaDB , Percona ( also clustering ) . We support modern
database technologies in MySQL which includes Galera ( Clustering ), Group
Replication , SQL aware Load balancers like Maxscale / ProxySQL.
About Mydbops
6. 1. Introduction to InnoDB Cluster
2. Project Background
3. Challenges faced
4. Future Enhancements planned
Table of Contents
7. ● Group Replication Library
● Virtually Synchronous replication (MySQL 5.7 )
● GCS is based on Paxos ( Variant of Mencius )
● High Availability solution on InnoDB
● Consistent data across nodes
● Shared Nothing Cluster
● Supported on all Platforms
● Inbuilt in MySQL ( No third party plugin )
Introduction : InnoDB Cluster
10. 1. Introduction to InnoDB Cluster
2. Project Background
3. Challenges faced
4. Future Enhancements planned
Table of Contents
11. A largest retail store in USA makes a migration to Open source Platform.
● Migration to Cloud ( Google Cloud )
● Monolithic to Microservice Platform
● Application servers on Containers ( Dockers )
● MySQL and MongoDB as backend data stores.
● Redis as a caching layer
● Consul for service discovery and Ansible for Config management.
Project Background
12. MySQL Info
● Automated Deployment with MySQL Shell ( In GCC)
● MySQL Enterprise servers.
● Virtual Environment
● Single Primary Mode cluster
● MySQL Version 5.7.17 ( started with )
● No read / write Split up
Project Background
13. 1. Introduction to InnoDB Cluster
2. Project Background
3. Challenges faced
4. Future Enhancements planned
Table of Contents
14. Case 1) Heavy Data loading.
Initial Loading of CSV files of heavy size breaks
the cluster and rollback the transaction.
Becomes our first blocker as data validating can
be made once the data is loaded.
Challenges Faced
15. Solutions Proposed.
● Stop the group replication reset the Master GTID’s and make the sync back in async mode ( normal
replication )
● Use a hot backup solution like MySQL Enterprise backup and rebuilt it.
The Solution 1 was comparatively faster in comparing the data set we had.
Challenges Faced
16. Case 2 ) Setting up HA for MySQL Router.
MySQL router can become a single point of failure
Challenges Faced
17. Solutions Proposed.
Implement Router as a layer with docker containers as
a part of application deployment and maintain the config
repository based on microservices.
Challenges Faced
MR MR MR MR MR
18. Case 3: MySQL bugs affected the Stress tests
MySQL Version : 5.7.17
Replication: In previous versions issuing STOP GROUP_REPLICATION stopped the plugin but the server
still accepted transactions. This meant the transactions were not transmitted to the group. To make STOP
GROUP_REPLICATION safer, now super_read_only is set to ON immediately upon issuingSTOP
GROUP_REPLICATION, which ensures no transactions are accepted. (Bug #25495067, Bug #84795)
Replication: Interleaved transactions could sometimes deadlock the slave applier when the transaction
isolation level was set to REPEATABLE-READ. (Bug #25040331)
Challenges Faced
19. Solutions Proposed.
MySQL Version : Those bugs were fixed in MySQL 5.7.20 and MySQL 5.7.19 respectively. We made
MySQL 5.7.20 as a standard in our repository for production.
Note :
Group_replication_transaction_size_limit was added in MySQL 5.7.19
Challenges Faced
20. Case 4: Isolation Level and Open transaction
Repeatable read can be performance killer in a cluster ( optimistic locking ). And open transactions from
application or users make it much worse.
Solution Proposed :
Read Committed was made as a standard isolation level and open transaction alerts based on
performance_schema was set to find application code bugs or user level
Challenges Faced
21. Case 5: Schema changes
The heavy DDL can block the parallelism in the cluster and can make the secondary to RECOVERY
state.Even INNODB Online alter can't help much as it needs a table rebuild.
Solution Proposed :
Third party Tools can be saviour - Percona toolkit or Gh-OST ( can be )
Challenges Faced
22. Case 6 : InnoDB cluster Can’t rejoin .
When the node is rebuild using MEB , the Group Replication is running fine. But the InnoDB cluster can’t
find the right UUID and marks the cluster as failed node. Can be validated via MySQL Shell
select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | b0b1603f-83ef-11e6-85a6-080027de0e0e | box01 | 3306 | ONLINE |
| group_replication_applier | bb29750c-83ef-11e6-8b4f-080027de0e0e | box02 | 3306 | ONLINE |
| group_replication_applier | bbu3761b-83ef-11e6-894c-080027de0t0e | box03 | 3306 | ONLINE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
Challenges Faced
23. mysql> show tables;
+-----------------------------------------+
| Tables_in_mysql_innodb_cluster_metadata |
+-----------------------------------------+
| clusters |
| hosts |
| instances |
| replicasets |
| schema_version |
+-----------------------------------------+
5 rows in set (0,00 sec)
Solution and fix ( Temporary Patch ).
The cluster metadata table has to be updated manually post a rebuild if there is a mismatch in UUID.
Challenges Faced
24. 1. Data loading
2. Setting up HA for MySQL Router.
3. MySQL and Router Version.
4. Isolation level and open transaction
5. Handling the DDL’s
6. DB Recovery and InnoDB Cluster.
Challenges Faced
25. 1. Introduction to InnoDB Cluster
2. Project Background
3. Challenges faced
4. Future Enhancements planned
Table of Contents
26. ● Read Write Split up from App / ProxySQL
● Planning to Upgrade MySQL 5.7.22 or MySQL 8.0
● Migration to Bare Metal server.
● Single Click Upgrades on Ansible
Future Enhancement