At Pinterest, the ProxySQL infrastructure fronts numerous heterogeneous databases. Due to the high number of unique configurations and dynamic nature of cloud deployments, it’s a challenge to reliably provision, change, auto-scale, and monitor ProxySQL servers. Applying Infrastructure as Code principles using Terraform, we made it possible to manage such a large fleet of ProxySQL instances confidently. Come learn how we automated provisioning, testing, and monitoring of ProxySQL at scale.

1. ProxySQL at Scale on AWS
Percona Live 2022

2. Challenges
● Over 100 diverse backend types, 400+ EC2 instances
○ Different query routing
○ Different users and configurations
● Cloud environment
○ Short lived instances
○ Shorter lifetime
● Disposable infrastructure
○ Configuration drift
○ History of changes

3. Three level provisioning
● AWS resources
○ Load balancer
○ Auto Scaling Group
○ CNAME
● Zero downtime
migrations
● OS level
● Packages
● Configs
● Services
● Config Agent
○ ProxySQL final
configuration
○ Updating configs
○ Restarts
● Healthchecks
○ Flexible logic

4. Guiding principles
● Infrastructure as code
○ Git as UI
● Declarative is favored over imperative
● Terraform - one time change
● Puppet - rare changes
● Python - frequent changes

5. ProxySQL Unit
P
ASG
N
N
N
Load Balancer
AZ1
AZ2
AZ3
P
P
P
P
P
to provision

6. Terraform

7. Example: Terraform
module "sandboxdb" {
source = "git::ssh://git@p.com/terraform-aws-proxysql.git?ref=4.0.0"
security_group_ids = [
data.aws_security_group.proxysql["vpc-db-automation"].id,
data.aws_security_group.proxysql["proxysql_use1_healthcheck"].id,
data.aws_security_group.proxysql["proxysql_data"].id,
data.aws_security_group.proxysql["vpc-dev-data-layer"].id
]
iam_instance_profile = data.aws_iam_instance_profile.proxysql.arn
aws_ami = var.aws_ami_map["default"]
subnets = var.subnets
backend_type = "sandboxdb"
enable_deletion_protection = false
health_check_type = "EC2"
health_check_grace_period = 1800
healthcheck_port = "traffic-port"
elb_zone = var.elb_zone
min_size = length(var.subnets)
ports = [3306]
nlb_cname = "sandbox.proxysql"
aws_instance_type = "c5.xlarge"
pinfo_env = "sandbox"
}

8. WhyTF module
● Cookiecutter
○ Repeatable infrastructure
○ Good for juniors
● Splits complexity:
○ module maintainer vs module user
● Versioning
● Testable

9. What module creates
resource "aws_launch_configuration" "proxysql_zero" {...}
resource "aws_autoscaling_group" "proxysql_zero" {...}
resource "aws_placement_group" "proxysql_placement" {...}
resource "aws_autoscaling_lifecycle_hook" "terminating" {...}
resource "aws_autoscaling_policy" "cpu_load" {...}
auto_scaling.tf dns.tf
load_balance.tf outputs.tf

10. What module creates
resource "aws_lb" "proxysql" {...}
resource "aws_lb_listener" "proxysql" {...}
resource "aws_lb_target_group" "proxysql" {...}
auto_scaling.tf dns.tf
load_balance.tf outputs.tf

11. What module creates
resource "aws_route53_record" "elb" {
name = "${var.backend_type}.proxysql"
type = "CNAME"
ttl = local.dns_ttl
zone_id = data.aws_route53_zone.nlb_zone.id
records = [
aws_lb.proxysql.dns_name
]
}
auto_scaling.tf dns.tf
load_balance.tf outputs.tf

12. What module creates
output "nlb_dns_name" {
value = aws_lb.proxysql.dns_name
}
output "nlb_cname" {
value = aws_route53_record.elb.name
}
auto_scaling.tf dns.tf
load_balance.tf outputs.tf

13. Healthchecks
Two decisions to make:
1. Shall I forward traffic to X?
2. Shall I terminate X?
?
Load
Balancer
?
resource "aws_autoscaling_group" "proxysql_zero" {
…
health_check_type = “EC2|ELB”
…
}

14. ELB type Healthchecks
● Traffic port vs health check port
● TCP connection success?
○ Yes
■ Forward traffic
○ No
■ Do not forward traffic
■ Terminate instance
:8080
:3306

15. EC2 type Healthchecks
● Traffic port == health check port
● TCP connection success?
○ Yes
■ Forward traffic
○ No
■ Do not forward traffic
● ProxySQL healthcheck agent
:8080
:3306
try:
self.check_health(cursor)
self.check_at_least_one_live(cursor)
self.check_whg_count(cursor)
self.check_instance_state(local_instance)
self.mark_healthy(local_instance)
except:
self.mark_unhealthy(local_instance)

16. Zero downtime migration
● Needed when change:
○ AMI, subnet, security group, IAM role
○ ami-12345678 -> ami-87654321
● Blue/green deployment
● Must be tested for a proof
Load
Balancer

17. Zero downtime migration: Step 1
resource "aws_launch_configuration" "proxysql_zero" {
…
}
resource "aws_autoscaling_group" "proxysql_zero" {
name =
"${local.asg_prefix}-${aws_launch_configuration.proxysql_zero.name}"
target_group_arns = aws_lb_target_group.proxysql[*].arn
lifecycle {
create_before_destroy = true
}
}

18. Zero downtime migration: Step 2
resource "aws_launch_configuration" "proxysql_zero" {
…
}
resource "aws_autoscaling_group" "proxysql_zero" {
name =
"${local.asg_prefix}-${aws_launch_configuration.proxysql_zero.name}"
target_group_arns = aws_lb_target_group.proxysql[*].arn
lifecycle {
create_before_destroy = true
}
}

19. Zero downtime migration: Step 3
resource "aws_launch_configuration" "proxysql_zero" {
…
}
resource "aws_autoscaling_group" "proxysql_zero" {
name =
"${local.asg_prefix}-${aws_launch_configuration.proxysql_zero.name}"
target_group_arns = aws_lb_target_group.proxysql[*].arn
lifecycle {
create_before_destroy = true
}
}

20. Zero downtime migrations: How to test
Must be tested (tests/zero_downtime/test_zero_downtime.py)
Plan:
● ProxySQL acts as an HTTP server
● Create pool A
● Start a probe in a loop
● Create pool B
● Watch for probe errors

21. Testing a module: terraform-ci
def test_zero_downtime(ec2_client):
# Pool A
with open(osp.join(terraform_dir, "configuration.tfvars"), "w") as fp:
fp.write(configuration_template.format(payload=”old payload”))
with terraform_apply(terraform_dir) as tf_output:
proxysql_nlb = tf_output["proxysql_nlb"]["value"]
# Probe
que = Queue()
child = Process(target=ping, args=(url, que))
# Pool B
with open(osp.join(terraform_dir, "configuration.tfvars"), "w") as fp:
fp.write(configuration_template.format(payload=”new payload”))
with terraform_apply(terraform_dir):
child.terminate()
result = que.get()
assert result["error"] == 0
terraform-ci · PyPI

22. test_data/main.tf
module "proxysql" {
source = "../"
...
iam_instance_profile = aws_iam_instance_profile.ProxySQL_development.arn
ports = [22, local.traffic_port]
user_data_base64 = data.template_cloudinit_config.cloud_config["proxysql"].rendered
aws_instance_type = "t3.micro"
wait_for_capacity_timeout = "10m"
}

23. ProxySQL user-data
userdata_proxysql = format(
"#cloud-confign%s",
yamlencode(
{
write_files : [
{
content : var.index_content,
path : "/root/index.html"
},
]
package_update : true
packages : [...],
runcmd : [
"hostname proxysql",
"pip3 install webdev",
"cd /root; webdev -p ${local.traffic_port} &",
"/usr/local/bin/drain.sh &"
]
}
)
)

24. Test results
Destroy complete! Resources: 24 destroyed.
2022-05-16 13:53:39,416: INFO: __init__.execute():416: Executing: terraform destroy
-var-file=configuration.tfvars -input=false -auto-approve
Changes to Outputs:
- subnet_ids = [
- "subnet-5c19e837",
- "subnet-53d7411f",
- "subnet-43d5fa39",
] -> null
- traffic_port = 13 -> null
- vpc_id = "vpc-2ef53c45" -> null
- zone_id = "Z07524281VZN9F9H2TZCX" -> null
You can apply this plan to save these new output values to the Terraform
state, without changing any real infrastructure.
Destroy complete! Resources: 0 destroyed.
PASSED
======================== 1 passed in 1098.00s (0:18:18) ========================

25. Puppet
(rare changes)

26. Bootstrapping instance
AWS Starts EC2 instance
AMI starts cloud-init
cloud-init starts puppet

27. User-data
[root@proxysql-0a01b51c:~]# ec2metadata --user-data
#cloud-config
pinfo_team: proxysql
pinfo_env: sandbox
pinfo_role: sandboxdb
backend_type: sandboxdb
spiffe_id: spiffe://pin220.com/ec2/proxysql/sandboxdb

28. Puppet: what to run?!
[root@proxysql-0a01b51c:~]# puppet lookup
--facts /etc/facter/facts.d/pinfo-$HOSTNAME.yaml
--merge deep
--hiera_config /mnt/puppet/hiera.yaml
--render-as json classes
["pinterest::profile::proxysql"]

29. Example: Puppet
---
proxysql_manager::package_version: "latest"
pinterest::profile::proxysql::mtls::enabled: true
pinterest::profile::proxysql::package_version: "2.3.2"
class pinterest::profile::proxysql (...) {
service { 'proxysql':
ensure => running,
provider => 'debian',
require => Exec['generate_proxysql_cnf']
}
service { "proxysql-manager-agent":
ensure => $enabled ? { true => running, default => stopped },
subscribe => Package['proxysql-manager'],
require => [
Package['proxysql-manager'],
File["/etc/proxysql-manager-agent.conf"],
File["/etc/systemd/system/proxysql-manager-agent.service"],
]
}
}

30. Bootstrap ProxySQL configuration
[root@proxysql:~]# proxysql-manager generate-config --save-to=/dev/stdout
datadir="/var/lib/proxysql"
errorlog="/var/lib/proxysql/proxysql.log"
admin_variables={
admin_credentials="superadmin:qwerty"
mysql_ifaces="127.0.0.1:6032"
}2022-05-16 19:05:25,546: INFO: MainProcess(30966) generate_config.generate_config():35: Successfully saved
ProxySQL config in /dev/stdout

31. ProxySQL Manager logs
class pinterest::profile::proxysql_efs (
$efs_dir,
$efs_target,
$efs_mount_options = 'nfsvers=4.1',
) {
file { $efs_dir:
ensure => directory,
owner => 'root',
group => 'root',
mode => '0755',
}
mount { $efs_dir:
ensure => mounted,
atboot => true,
device => $efs_target,
fstype => 'nfs4',
options => $efs_mount_options,
require => [
Package['nfs-common'],
File[$efs_dir],
]
}
}

32. ProxySQL Manager logs
[root@proxysql-sharddb-prod-0a019037:~]# df -h /var/log/proxysql
Filesystem Size Used Avail Use% Mounted on
proxysql-efs-prod.pinadmin.com:/ 8.0E 102G 8.0E 1% /var/log/proxysql
# ll /var/log/proxysql-manager.log
lrwxrwxrwx 1 root root 58 May 13 02:01 /var/log/proxysql-manager.log ->
/var/log/proxysql/i-0ec0c1de5b2190f4a/proxysql-manager.log

33. Python:
frequent changes

34. Arch & Workflow (GitOps)
VCS
Jenkins
SoT
ProxySQL
Node
agent
ProxySQL
Node
agent
ProxySQL
Node
agent
git
test
deploy
test
watch

35. Data model
CREATE TABLE global_variables (
variable_name VARCHAR NOT NULL PRIMARY KEY,
variable_value VARCHAR NOT NULL
)

36. Table is a Class
[root@proxysql-sharddb-prod-0a019037:~]# proxysql-manager admin login -e "show tables"
+----------------------------------------------------+
| tables |
+----------------------------------------------------+
| global_variables |
| mysql_aws_aurora_hostgroups |
| mysql_collations |
| mysql_firewall_whitelist_rules |
| mysql_firewall_whitelist_sqli_fingerprints |
| mysql_firewall_whitelist_users |
| mysql_galera_hostgroups |
| mysql_group_replication_hostgroups |
| mysql_query_rules |
| mysql_query_rules_fast_routing |
| mysql_replication_hostgroups |
| mysql_servers |
| mysql_users |
| scheduler |
+----------------------------------------------------+

37. mysql_servers

38. PSList (table == list of records)

39. Source of Truth(s)
● SourceOfTruth(ABC):
● SOTLocalProxy(SourceOfTruth):
● SOTRemoteMySQL(SourceOfTruth):

40. ProxySQL Manager
# proxysql-manager
Usage: proxysql-manager [OPTIONS] COMMAND [ARGS]...
ProxySQL manager helps to manage local ProxySQL instance
Options:
--debug Print debug messages
-q, --quiet Print only errors
--logfile TEXT Log messages to this file [default: /var/log/proxysql-
manager.log]
--version Show the version and exit.
--help Show this message and exit.
Commands:
admin ProxySQL admin credentials
deregister-server Deregister MySQL server from a ProxySQL Source of...
deregister-unknown Deregister non-ZK MySQL servers.
expert ProxySQL manager advanced commands.
generate-config Generate ProxySQL config and save it on disk.
is-server-registered The tool will inspect the latest ProxySQL config...
register-server Register MySQL server in a ProxySQL Source of Truth.
show-backends Show all backend types and their NLB DNS name.
show-instance Show ProxySQL instance for given backend_type
show-instances Same as ``proxysql-manager show-instance`` command.
start-agent Update latest configuration to local ProxySQL.
start-healthcheck Health check for upstream NLB.
sync-sot Synchronize ProxySQL config to remote Source of...

41. ProxySQL Manager: Agent
def _watch_table(self, klass, interval):
"""
Start agent daemon.
:param klass: Table class to watch
:param int interval: Time in seconds between SoT
checks.
"""
while True:
with global_lock():
if klass == PSMysqlUsers:
self._sync_mysql_users_with_knox()
else:
self.update_one_table(klass)
sleep(interval)

42. ProxySQL Manager: Agent (cont)
def update_one_table(self, tblclass):
try:
pslist = self._source_of_truth.read(tblclass, self._backend_type)
if pslist.version and pslist.version > self.versions[tblclass]:
# pslist is empty or pslist has newer version
LOG.info(
"It is a new version %d for table %s",
pslist.version,
tblclass.TABLE,
)
self._local_proxysql.write(pslist)
self.versions[tblclass] = pslist.version
self.update_versions_file(tblclass=tblclass, version=pslist.version)
self._handle_fallback(pslist)

43. ProxySQL Manager: Healthcheck
def watcher(self, credentials):
while True:
try:
with connection.cursor() as cursor:
self.check_health(cursor)
self._check_at_least_one_live(cursor)
self._check_whg_count(cursor)
self._check_instance_state(local_instance)
self._mark_healthy(local_instance)
except (
ProxySQLNotHealthy,
MySQLError,
ConnectionRefusedError,
) as err:
self._unhealthy_count += 1
LOG.error(err)
if self._unhealthy_count > self._healthy_threshold:
self._mark_unhealthy(local_instance)
else:
LOG.warning("Health check failed %d times", self._unhealthy_count)
sleep(self._probe_interval)

44. Source of Truth configuration
mysql_query_rules.json
[
{
"proxy_port": 3306,
"active": 1,
"rule_id": 1,
"destination_hostgroup": 1
}
]
mysql_users.json
[
{
"username": "knox://<knox key>",
"default_hostgroup": 3,
"attributes": {
"spiffe_id": "spiffe://pin220.com/foo"
},
"comment": {
"fallback_group": 4
}
},
]
● global_variables.json
● mysql_replication_hostgroups.json

45. Thank you!
https://twitter.com/OleksandrKuzm