This is presented at LSPE meet up on 17-06-2017.
ProxySQL and MaxScale are the next-generation database proxy that manages security, scalability, high availability and data streaming in scale-out deployments
ProxySQL and MaxScale have been most widely adapted in industry these days. They can be used in smaller platforms to bigger platforms. The Objective of this talk is to make awareness about the next-gen database proxies and their value addition.
2. Mydbops is into MySQL Support and Consulting. It is founded by experts who have
scaled MySQL database at Yahoo! ,Percona and Datavail. We are providing an expert level
support and monitoring for MySQL databases and its related technologies like MariaDB ,
Percona ( also clustering ) . We support modern database technologies in MySQL which
includes Galera ( Clustering ), Group Replication , SQL aware Load balancers like
Maxscale / ProxySQL, MySQL Engine like InnoDB, TokuDB and MyRocks.
We offer seamless end to end DB solutions like migration, installation, up-gradation, design,
performance tuning, troubleshooting, monitoring, recovery with 24x7 on call support and
Remote DBA support inevitably decreasing the downtime by 99.9%. Our services will improve
your DB Performance and helps in saving cost on IT.
www.mydbops.com mysqlsupport@mydbops.com
About Mydbops
4. Karthik P R
Founder of Mydbops. Has an experience of around 7 years in handling MySQL servers .
Handled large scale systems at Yahoo! ( 3 years ). Active MySQL speakers and blogger.
Currently providing solutions to many established startups and organisation to scale in
MySQL.
Email : prk@mydbops.com Twitter: @prkart
Kabilesh P R
Lead Consultant in Mydbops. Has an experience of about 5 years. Worked with large remote
database support company Datavail on multi customer platform. Expertise in monitoring of
MySQL and actively working with mysql on cloud
www.mydbops.com mysqlsupport@mydbops.com
About US
5. ● Need for Load Balancer
● SQL Awareness Load Balancers
● Introduction to Maxscale
● Maxscale use cases
● Introduction to ProxySQL
● ProxySQL use cases
● Maxscale Vs ProxySQL
www.mydbops.com mysqlsupport@mydbops.com
Agenda
6. ● Load balancer distributes the incoming traffic efficiently to underlying backend servers
● Provides efficient use of the underlying hardware
● Avoids single point of failures ie., High Availability
● Provides way for easy switchover and failover
● Additional Features
www.mydbops.com mysqlsupport@mydbops.com
Need for a Load balancer
7. Types Of Load Balancers
● Reverse Proxies
○ HAProxy
○ Ngnix
● SQL aware Proxies
○ Maxscale
○ MySQL Router
○ ProxySQL
www.mydbops.com mysqlsupport@mydbops.com
Need for a Load balancer
8. ● SQL aware load balancer, forward the queries/statements by understanding the
underlying state of servers.
● Read/Write splitting.
● Support for external tools and scripts.
● Monitoring of backend servers.
● Query Caching.
● Data Streaming
www.mydbops.com mysqlsupport@mydbops.com
SQL Aware Load Balancer
10. ● Maxscale is an intelligent SQL Aware proxy.
● Built by MariaDB corporation
● Pluggable router
● Easy to configure
● Highly Scalable and flexible
● Provides High Availability
www.mydbops.com mysqlsupport@mydbops.com
Introduction To Maxscale
11. ● Maxscale comes with two license model they are GPL (Open Source License) and BSL
(Business Source License).
● Till version 1.4 Maxscale comes with GPL
● From version 2.0 Maxscale comes with BSL
● Usage Limitation of BSL : The usage of software is free for when you are using total or
less than of 3 server for production purposes.
● Change Date of BSL : 2019-01-01
“Business source is simply proprietary software released under a Microsoft-esque shared source license that magically
becomes fully open source after a period of time.”
www.mydbops.com mysqlsupport@mydbops.com
License In Maxscale
12. ● Maxscale Comes with a simple pluggable modular architecture
● Modules are loaded dynamically on demand
● The maxscale core provides the following
Network layer interface
Buffer management
Config Management
Synchronization
www.mydbops.com mysqlsupport@mydbops.com
Maxscale Architecture
15. ● Maxscale support the below routing algorithms for various use cases
○ ReadConnRoute - Connection based load balancing
○ ReadWriteSplit - Read/Write aware statement based routing
○ SchemaRouter - Simple Sharding based on database level
○ Binlogrouter - As Binlog server
www.mydbops.com mysqlsupport@mydbops.com
Maxscale Routers
16. ● Maxscale monitor module monitors the underlying cluster/server and provides the status
information to the router based on which the query is routed to the server, the various
supported monitor modules include
○ MySQL Monitor(mysqlmon) - Monitors M-S replication and assign roles
○ Galera Monitor(galeramon) - Monitors the SYNC status of the each cluster node
○ NDBCluster Monitor(ndbclustermon) - Monitors the NDB cluster
○ Multi-Master Monitor(mmmon) - monitors the multi-master replication cluster.
www.mydbops.com mysqlsupport@mydbops.com
Maxscale Monitor
17. ● Configuration file to store the maxscale configurations (/etc/maxscale.cnf) default.
● Similar interface like mysql cmd line
● Has query parser similar to mysql server
● Default port for maxscaled 6603
● Default password is ‘mariadb’ for maxadmin (admin cmd line)
www.mydbops.com mysqlsupport@mydbops.com
Maxscale Configuration
24. ● Compression is not supported in mysql server handshake
● With R/W read queries routed to master when they are executed as transaction, in case
of prepared statement, procedure call and multiple statement in one query.
● Maxscale cannot manage authentication with % as hostname, the only accepted wildcard
will be with IP as ‘192.167.2.%’
● Has some limitation with client session variable
www.mydbops.com mysqlsupport@mydbops.com
Maxscale Limitations
25. ● From 2.1 maxscale accepts runtime changes to most of the config
● Support for Aurora Cluster (AWS)
● Selective data Masking
● Secure binlog server using SSL
● Great improvements with query caching.
● Inbuild failover management for promoting slave ( no more external scripts )
www.mydbops.com mysqlsupport@mydbops.com
Maxscale Advanced Features
27. ● Built for DBA’s by a DBA.
● Major Contribution by René Cannaò (Founder)
● Winner MySQL Community Contributor (2017)
● Light weight application.
● Advanced Multi Core Architecture.
● Complete Open Source GPL License (GPL 3.0)
● Current Version 1.3.7 and 1.4 is Beta. ( 17-06-2017 )
www.mydbops.com mysqlsupport@mydbops.com
ProxySQL Introduction
28. ● More Granular Configuration
● SQL based Administration.
● Dynamic Configuration
● Advanced Metric Collections.
● Supports More Complex Topologies.
● Seamless Integrations.
● Manage database at large scale ( 1000’s )
www.mydbops.com mysqlsupport@mydbops.com
ProxySQL Introduction
30. ● SQLite database to store config.
● MySQL Compatible interface.
● There is no real parser in ProxySQL
● Multi Layer Configuration
● Default port 6032
$mysql -u admin -padmin -h 127.0.0.1 -P6032
www.mydbops.com mysqlsupport@mydbops.com
ProxySQL Configuration
31. ● SQLite database to store config.
● MySQL Compatible interface.
● There is no real parser in ProxySQL
● Multi Layer Configuration
● Default port 6032
$mysql -u admin -padmin -h 127.0.0.1 -P6032
www.mydbops.com mysqlsupport@mydbops.com
ProxySQL Configuration
32. Memory: Represent SQLite3 in memory
database
Runtime: In memory store of ProxySQL with
RUNTIME Config.
Disk: On Disk location of SQLite3. Contains
Persistent storage of config.
Config File: Load the config (proxysql.cnf)
when there is no disk file ( proxysql.db)
www.mydbops.com mysqlsupport@mydbops.com
ProxySQL Configuration
33. Loading configuration three levels:
SQL (writes ) are made to table in Memory.
Then load to RUNTIME ( Dynamically ) and
saved to DISK for persistence.
Eg)
Load MYSQL USERS TO RUNTIME;
SAVE MYSQL USERS TO DISK;
www.mydbops.com mysqlsupport@mydbops.com
ProxySQL Configuration
34. Scenario Loading a User
Host groups 101 - Writer , 102 -Reader
insert into mysql_users(username,password,active,default_hostgroup,default_schema) values
('writer_app','test',1,101,'sbtest');
insert into mysql_users(username,password,active,default_hostgroup,default_schema) values
('reader_app','test',1,102,'sbtest');
LOAD MYSQL USERS TO RUNTIME;
SAVE MYSQL USERS TO DISK;
www.mydbops.com mysqlsupport@mydbops.com
ProxySQL Configuration
36. Loading Server Info:
INSERT INTO mysql_servers ( hostname,hostgroup_id,port,weight) VALUES ('192.168.1.1',101,3306,1000);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES
('192.168.1.2',102,3306,10000,2);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES
('192.168.1.3',102,3306,10000,2);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES
('192.168.1.4',102,3306,1,2);
INSERT INTO mysql_replication_hostgroups VALUES (101,102);
Loading Config
LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;
www.mydbops.com mysqlsupport@mydbops.com
ProxySQL Replication ( case 1 )
37. ● ProxySQL can be Integrated with application server to reduce latency.
● Read_only flag defines the availability of readers
● Group of ProxySQL can be designed for complex Infra.
● Schedulers can be made for custom solutions.
● HA Solution for ProxySQL can be made with Keepalived.
● Prepared Statements are not supported in Read/write split
www.mydbops.com mysqlsupport@mydbops.com
ProxySQL Replication ( case 1 )
38. ● ProxySQL has in built Query caching.
● Caching is based on the query Rules.
● Cache invalidation is through cache_ttl ( in mysql_query_rules)
● mysql_query_cache_size_MB variables controls it.
● mysql_threshold_resultset_size defines the Maximum size of can be buffered.
● stats_mysql_global has metric related to Query Cache too.
● More Granular control over caching ( Query , User , Schema , Hint )
Note : Query Cache inbuilt with MySQL is Retiring in MySQL 8.0
www.mydbops.com mysqlsupport@mydbops.com
ProxySQL Caching ( case 2)
40. ● Caching Rules has to be made in Query rules. ( digest,cache_ttl )
● Rules are made based on digest ( stats_mysql_query_digest )
● TTL defines the invalidation in ms . ( cache_ttl)
● Top Queries can loaded based on Count of execution, Sum of execution time.
● Cache size helps in tuning it based on global stats.
● Background threads take care of memory cleanup and expired entries.
● It can replace Redis , memcache and other 3rd party caching for DB.
www.mydbops.com mysqlsupport@mydbops.com
ProxySQL Caching ( case 2)
41. ProxySQL Caching Vs MySQL Query Cache from official proxysql website
www.mydbops.com mysqlsupport@mydbops.com
ProxySQL Caching ( case 2 )
42. ● Galera is cluster based on InnoDB engine.
● Developed by Codership and it is an open source.
● Synchronous replication ( Zero lag )
● Multi Threaded replication.
● Automated node Recovery in case of crash.
● Distributions - Percona XtraDB Cluster , MariaDB Cluster and MySQL ( with plugin ).
● Galera with ProxySQL makes a better High Availability.
www.mydbops.com mysqlsupport@mydbops.com
ProxySQL For Galera ( case 3 )
44. ● Inbuilt schedulers to supports
● Support third party schedulers
● Awares on Galera States during SST / IST
● Desync states in galera.
● Easy Maintenance with status ( ONLINE , SHUNNED,OFFLINE_SOFT,OFFLINE_HARD
)
www.mydbops.com mysqlsupport@mydbops.com
ProxySQL For Galera ( case 3)
45. Integrations:
● ProxySQL is bundled in Percona XtraDB Cluster 5.7
● Proxy-admin tool by Percona to ease configuration.
● Orchestrator has integration for proxysql.
● Ansible modules helps in easy deployment ( 7 modules are present )
● Prometheus exporters for metrics.
● Integrated with PMM ( Grafana based Dashboards )www.mydbops.com mysqlsupport@mydbops.com
ProxySQL Integration & Deployment
46. ● Query Rewrite
● Sharding ( Schema , User , Query , Hints )
● Persistent Connections
● NDB Cluster Support
● Mirroring
● Query Filter
● Data Masking and more
www.mydbops.com mysqlsupport@mydbops.com
ProxySQL Other use cases
47. www.mydbops.com mysqlsupport@mydbops.com
Maxscale Vs ProxySQL
Features Maxscale ProxySQL
License BSL/GPL GPL
Group Replication Not Supported Supported
Galera Support Supported Supported
Read/Write Split Query Module ( Classifier ) REGEX
Aurora Support Supported(2.1) Not Supported
Binlog router Supported Not Supported
Kafka Streaming Supported Not Supported
48. www.mydbops.com mysqlsupport@mydbops.com
Maxscale Vs ProxySQL
Features Maxscale ProxySQL
Multiplexing Not Supported Supported
Caching Control Not Supported Supported
Query Digest Not Supported Supported
Ansible No Modules Ansible Modules
NagiOS Plugins are present No Official
Prometheus No exporters Exporters are present
Clustering Not Supported Not Supported