Using SSL/TLS to protect network traffic to and from a MySQL database

1. Protecting MySQL network traffic
Daniël van Eeden | 25 April 2017

2. Booking.com at a glance
● Started in 1996; still based in Amsterdam
● Member of the Priceline Group since 2005 (stock: PCLN)
● Amazing growth; continuous scaling challenges
● Online Hotel/Accommodation/Travel Agent (OTA):
● Over 1.2 million active properties in 227 countries
● Over 1.2 million room nights reserved daily
● 40+ languages (website and customer service)
● Over 13,000 people working in 187 offices in 70 countries
● We use a lot of MySQL and MariaDB:
● Thousands (1000s) of servers, ~90% replicating
● >150 masters: ~30 >50 slaves & ~10 >100 slaves
2

3. Why protect MySQL network
traffic?
● Protect leaking of authentication data (passwords, etc)
● Protect leaking of sensitive data (PII, credit card numbers,
medical records)
● Ensure data is not tampered with.
● Because of regulations
● Because why not? Are you still using telnet to manage
servers?

4. How?
● Use SSL!
● Done!

5. SSL Support in MySQL
● MySQL doesn't have SSL support
● MySQL never had any SSL support
● MySQL has TLS support.. this is what is called SSL but isn't
● Supported since 4.0.0 (~2003)
● For now just assume SSL and TLS are the same

6. What is NOT protected by TLS
● Data-at-rest
○ InnoDB and MyISAM data files
○ Binlogs, redo logs, slow query logs
○ Backups
● Does not protect against a DoS
○ e.g. corrupting traffic
● Might not protect the query text
○ performance_schema etc.
● Does not hide the traffic pattern

7. First steps with TLS
1. Get a certificate
2. Restart MySQL
3. Enable TLS on the client
4. Check if the connection actually uses TLS

8. Generating the certificate
● With 5.7 and up: Might already be done by your installation
● If not use mysql_ssl_rsa_setup
● For older versions: https://github.com/dveeden/mysslgen
● Or use the openssl commandline utilities as described in the reference
manual on
https://dev.mysql.com/doc/mysql/en/creating-ssl-files-using-openssl.html
● Did you know MySQL Workbench has a SSL Wizard?

9. Configuration
● On 5.7+: Place the ca.pem, server-cert.pem and
server-key.pem in your datadir. (already the case if you
use mysql_ssl_rsa_setup)
● Or set ssl-ca, ssl-key, ssl-cert in your my.cnf
● Restart MySQL
● Enable SSL in your application. You probably want to copy
your ca.pem file to your client

10. Checking your
connection
● 'status' or s
● Look for 'Cipher in use'
● Or check the 'Ssl_cipher' session
status.

11. What if it doesn't work?
● Check your mysqld.log
● Check the permissions on the pem files
○ Should be readable for the mysql user
● Try to connect with --ssl-mode=REQUIRED
● Use the OpenSSL commandline tools to see what's in the certificate.
● Use Wireshark.
ERROR 2026 (HY000): SSL connection error: error:00000001:lib(0):func(0):reason(1)
ERROR 2026 (HY000): SSL connection error: unknown error number
ERROR 2026 (HY000): SSL connection error: SSL certificate validation failure
ERROR 2026 (HY000): SSL connection error: SSL_CTX_set_default_verify_paths failed
ERROR 2026 (HY000): SSL connection error: protocol version mismatch
ERROR 2026 (HY000): --ssl-mode=REQUIRED option forbids non SSL connections
ERROR 2026 (HY000): SSL connection error: Failed to set ciphers to use
ERROR 2026 (HY000): SSL connection error: Unable to get certificate
ERROR 2026 (HY000): SSL connection error: Unable to get private key

12. Now let's make it more secure
● Require the use of TLS on the server
● Require the use of TLS on the client
● Enable more security checks
● Security updates

13. Make TLS a requirement
● On a per user basis:
○ ALTER USER foo REQUIRE SSL
○ Undo: ALTER USER foo REQUIRE NONE
● But what happens if you accidentally create a user?
○ e.g. GRANT on a nonexistent user?
○ Set: sql_mode=NO_AUTO_CREATE_USER,…
● On a server level:
○ SET GLOBAL require_secure_transport=ON
○ This still allows UNIX socket connections w/o TLS

14. Issues with full-on TLS
● Is your monitoring capable of using TLS connections?
● What about load balancer health checks?

15. On the client
● Use --ssl-mode=REQUIRED
○ The default in 5.7 is PREFERRED
○ Older releases default to DISABLED
● This only makes a TLS connection a requirement
○ Does not check if issued by a trusted CA
○ Does not check if the hostname matches the cert
○ To do this use VERIFY_CA or VERIFY_IDENTITY
● On older versions:
○ Use --ssl-ca to allow TLS and enable CA checks
○ Use --ssl-verify-server-cert to do hostname checks.
○ Often not possible to force the use of TLS: this is the
BACKRONYM vulnerability
● Use --ssl-ca=/path/to/ca.pem to specify which CA(s) are
trusted.

16. Client checks
● The client could do these checks:
○ Is the certificate signed by a trusted CA?
○ Does the CommonName (CN) in the certificate match
the hostname we are connecting to?
○ Is the certificate expired?

17. Certificate Authority validation
● Validates that the server certificate is signed by one of the CA's
present in the specified CA file.
● Note that a CA file can have multiple CA's
● There is also a CA path option.
● The auto generated certificates from mysql_ssl_rsa_setup all
have their own CA.

18. Hostname validation
● mysql_ssl_rsa_setup generates certificates with
○ CN=MySQL_Server_5.7.18_Auto_Generated_Server_Certificate
● So generate the certificates manually if want this to match your
hostname
● A certificate can have a list of hostnames in SubjectAltName
○ MySQL doesn't check those... Bug #68052
● So if you use a virtual-IP, cname, etc. it might be difficult to match
this.
● What if your clients connect on a CNAME and your replicas connect
on the hostname? You can't have both!

19. Security updates
● I reported a few issues to Oracle.
● CVE-2017-3590 for Connector/Python
● CVE-2017-3469 for MySQL Workbench
● CVE-2017-3467 for libmysqlclient
● Those are fixed. See the Oracle Critical Patch Update for
details.
● But if you care about security you should follow the release
notes and Critical Patch Update anyways...

20. What library does MySQL use?
● Community Edition: YaSSL
○ Because GPL and the OpenSSL license are not really
compatible
○ This library is maintained by WolfSSL
○ This not CyaSSL/WolfSSL
○ WolfSSL made a patch to include WolfSSL in MySQL
5.6.30 (https://github.com/wolfSSL/mysql-patch)
● Enterprise Edition: OpenSSL
● If you build MySQL yourself: you can compile against either
of them.

21. Why not TLS?
● Because it is SLOW!
● Because we trust our network!
● Because we encrypt with:
○ The application (store encrypted data)
○ SSH (Also works great with Workbench)
○ VPN
● Because we want to inspect our network traffic!
○ Wireshark can decrypt it if you hand over your private
key. Some ciphers require you to somehow extract
session keys.

22. How slow is slow?
● Overhead in milliseconds for
setting up a TLS connection on
localhost with TCP.
● Client: go
● 5.7 is faster than 5.6
● OpenSSL is faster than YaSSL
● Using TLS tickets (OpenSSL only)
helps
● Best case: 0.99ms (5.7 OpenSSL w/
tickets) vs. 0.60ms (no TLS)
● TLS does need more roundtrips,
but this will change with TLS 1.3
● OpenSSL performs better because
it uses AVX2 and AES-NI

23. Bulk transfer performance
● Easy to test: mysqldump with and without TLS
● Different ciphers do make a difference.
mysqldump performance with MySQL 5.6.35 (YaSSL)
No TLS 4.5s
TLS Default 10.4s
RC4-MD5 7.1s
DES-CBC3-SHA 23.2s

24. Monitoring
● Monitor the Expiry of certificates
● Not just the certificate on disk, also the one in memory.
● Use TLS for your monitoring on 5.6 and earlier, otherwise
you might not see the status vars
● Performance schema can show you the ciphers and TLS
versions in use by all connections
● Using SYS is even easier:
○ SELECT * FROM session_ssl_status

26. Client certificates
● This allows mutual authentication
● Often used together with a password
● You might want to use REQUIRE SUBJECT or REQUIRE
ISSUER on accounts.
● At least use REQUIRE X509 instead of REQUIRE SSL

27. Replication
● Use CHANGE MASTER TO MASTER_SSL=1, etc
● Think about what happens if your certificate(s) expire
● Does the hostname match the certificate?

28. Changing certificates
● Needs restart
● Moving slaves around might not work until you restart..
● Same for a switchover.

29. CRL and OCSP
● Only possible with OpenSSL
● Does not auto download the CRL from the distribution
point
● Does not use OCSP
● Basically restart MySQL every time your CRL changes.. which
is not practical

30. Where to get your certificate?
● Official CA?
● Internal CA?
● Self signed?

31. TLS handshake with MySQL
● server helo with ssl flag set
● 'empty' login packet with ssl flag set
● Start SSL handshake
● Basically STARTTLS-ish
○ SSL and non-SSL on the same port

32. Protection of authentication data
● native password with nonce
● sha256 password with RSA keys or TLS
● cleartext plugin

33. TLS ciphers
● Possible to set restriction on Server and Client
● How are you going manage and maintain that?
● 'REQUIRE cipher' also requires client certificates
● One practical use case would be to use a faster cipher for
mysqldump
● Might help with compliance
● 5.7.10 already places more strict requirements on the list of
ciphers

34. TLS versions
● Can be limited on the server and client
● Note that YaSSL only has TLS 1.0 and TLS 1.1 support
● Minimum is TLS 1.0

35. What about MariaDB?
● Doesn't use --ssl-mode
● Does have good TLS support
● MariaDB Connector/C has support for
○ fingerprint verification
○ password protected private keys
● 19 Open MDEV's tagged with SSL

36. Connector support
● Works for C, C++, Python (multiple), Perl, Java, ODBC, Go, etc
● The Go MySQL driver lets you specify a TLS Config, which is
really flexible.
● Do update your Connector.. Many connectors did have
security updates related to TLS.

37. Don't forget these
● MySQL Cluster (NDB) communication within the cluster
● Galera communication
● Sending backups to a central location (xbstream etc)
● Network traffic for iSCSI, FCP, NFS

38. Future
● TLSv1.3 with 0-RTT
● WolfSSL?

39. Oh, and Booking.com is hiring!
● Almost any role:
● MySQL Engineer / DBA
● System Administrator
● System Engineer
● Site Reliability Engineer
● Developer
● Designer
● Technical Team Lead
● Product Owner
● Data Scientist
● And many more…
● https://workingatbooking.com/39

40. Thank you!
All references to “Booking.com", including any mention of “us”, “we” and “our” refer to Booking.com BV, the company behind Booking.com™