Valerie Parham-Thompson
Lead Database Consultant with Pythian
Find more by Valerie Parham-Thompson: https://speakerdeck.com/dataindataout
All Things Open
October 26-27, 2016
Raleigh, North Carolina
3. OVERVIEW
PROTECTING DATA
It has been possible to protect data in transit
(moving over the network), and proper
authentication can be used to protect data in
use (being used by an application), but data at
rest (sitting on a file system) has been an area
of risk in MySQL/MariaDB.
4. OVERVIEW
WHAT ISTHE RISK?
Without encryption of data at rest, a bad actor
with access to the file system supporting the
database can view data even without proper
database-level permissions.
Having root access to the file system is very common!
5. OVERVIEW
WHAT IS SENSITIVE DATA?
PCI: credit card data
PII: names, drivers license, financial data, social
security number
HIPAA: medical, health insurance data
See: https://en.wikipedia.org/wiki/California_S.B._1386
7. EXPOSED!
DATA FILES
[root@encr_maria ~]# ls -al /var/lib/mysql/
allthingsopen/
total 108
drwx------. 2 mysql mysql 45 Oct 19 15:09 .
drwxr-xr-x. 6 mysql mysql 4096 Oct 19 15:07 ..
-rw-rw----. 1 mysql mysql 65 Oct 19 15:07 db.opt
-rw-rw----. 1 mysql mysql 932 Oct 19 15:09 t1.frm
-rw-rw----. 1 mysql mysql 98304 Oct 19 15:09 t1.ibd
[root@encr_maria ~]# strings /var/lib/mysql/
allthingsopen/t1.ibd
...
mysecret123-45-6789
9. EXPOSED!
REDO LOG
[root@encr_maria ~]# yum install vim-common
...
[root@encr_maria ~]# xxd /var/lib/mysql/ib_logfile0 | grep -v
"0000"
...
018a580: 0110 8000 0001 8000 0002 6d79 7365 6372 ..........mysecr
018a590: 6574 3132 332d 3435 2d36 3738 3982 0081 et123-45-6789...
[root@encr_comm ~]# xxd /var/lib/mysql/ib_logfile0 | grep -v
“0000"
...
0269ec0: 011b 0110 8000 0001 8000 0002 6d79 7365 ............myse
0269ed0: 6372 6574 3132 332d 3435 2d36 3738 3937 cret123-45-67897
(xxd is another command-line tool that allows you to see
text within binary files. Install vim-common to use it.)
10. BUT FIRST… SOME ALTERNATIVES
OVERVIEW
ALTERNATIVESYES, DEAR
KEY ROTATION
KEY STORAGE
THIRD-PARTYTOOLS
BACKUPS
PERFORMANCE
SUMMARY
11. ALTERNATIVES
OVERVIEW
• Don’t store sensitive data.
• Encrypt data from the application.
• Use column-level encryption.
• Encrypt the file system.
12. ALTERNATIVE
DON’T DO IT
• Outsource data storage.
• Don’t store data if you don’t have a legitimate
need for it.
• Regularly archive data from departed users or
data that has aged out.
13. ALTERNATIVE: ENCRYPT FROM APP
EXAMPLE OF ENCRYPTED DATA
MariaDB [allthingsopen]> select * from t1G
intcol1: 1
intcol2: 2
charcol1: my
charcol2: secret
charcol3: “?????-{??S@?/@%?>??????9?
14. ALTERNATIVE: ENCRYPT FROM APP
NOTVISIBLE IN DATA FILES
[root@encr_maria ~]# strings /var/lib/mysql/
allthingsopen/t1.ibd
...
mysecret"?????-{??S@?/
%?>???
???9?
26. YES, DEaR, MARIADB
ENCRYPTION CONFIGS ON
MariaDB [(none)]> show global variables like '%encr%';
+------------------------------------------+---------+
| Variable_name | Value |
+------------------------------------------+---------+
| aria_encrypt_tables | OFF |
| encrypt_binlog | ON |
| encrypt_tmp_disk_tables | ON |
| encrypt_tmp_files | ON |
| file_key_management_encryption_algorithm | aes_cbc |
| innodb_default_encryption_key_id | 1 |
| innodb_encrypt_log | ON |
| innodb_encrypt_tables | ON |
| innodb_encryption_rotate_key_age | 1 |
| innodb_encryption_rotation_iops | 100 |
| innodb_encryption_threads | 4 |
+------------------------------------------+---------+
27. YES, DEaR, MARIADB
ENCRYPTION METHODS
There are three ways to encrypt table data in MariaDB 10.1:
• innodb_encrypt_tables in the my.cnf will encrypt all tables (unless
specified as “encrypted=no”).
• innodb_encrypt_tables=FORCE in the my.cnf will encrypt all new
tables created, and will prevent table creation with “encrypted=no.”
• If the configuration innodb_encrypt_tables is not in the my.cnf, but
the plugin is present, you can explicitly encrypt a table by including
“encrypted=yes” in the table creation statement.
(If innodb_encryption_threads is set higher than 0, then existing
tables will be encrypted in the background.)
28. YES, DEaR, MARIADB
INNODB_ENCRYPTION_THREADS=4
[root@encr_maria ~]# tail -f /var/lib/mysql/
encr_maria.err
...
2016-10-19 19:44:31 140629860751488 [Note] InnoDB:
Creating #1 thread id 140629124273920 total threads 4.
2016-10-19 19:44:31 140629860751488 [Note] InnoDB:
Creating #2 thread id 140629115881216 total threads 4.
2016-10-19 19:44:31 140629860751488 [Note] InnoDB:
Creating #3 thread id 140629107488512 total threads 4.
2016-10-19 19:44:31 140629860751488 [Note] InnoDB:
Creating #4 thread id 140629099095808 total threads 4.
...
29. YES, DEaR, MARIADB
EFFECTS OF BACKGROUNDTHREADS
MariaDB [(none)]> select name from
information_schema.innodb_tablespaces_encryption where
encryption_scheme=1;
Empty set (0.00 sec)
MariaDB [(none)]> select name from
information_schema.innodb_tablespaces_encryption where
encryption_scheme=1;
+--------------------------+
| name |
+--------------------------+
...
| allthingsopen/t1 |
+--------------------------+
5 rows in set (0.00 sec)
35. YES, DEaR, MARIADB
REMAINING EXPOSURES
Not all data on disk is encrypted:
• slow log
• error log
• general log
• audit log
• relay logs on unencrypted slaves
36. YES, DEaR, MARIADB
VISIBLE IN SLOW LOG
MariaDB [allthingsopen]> set global slow_query_log=on;
MariaDB [allthingsopen]> set global long_query_time=0;
MariaDB [(none)]> update allthingsopen.t2 set
charcol3='456-78-9123' where intcol1>1;
# Time: 161019 20:55:33
# User@Host: root[root] @ localhost []
# Thread_id: 6 Schema: QC_hit: No
# Query_time: 0.003517 Lock_time: 0.000118 Rows_sent: 0
Rows_examined: 1
# Rows_affected: 0
SET timestamp=1476924933;
update allthingsopen.t2 set charcol3='456-78-9123' where
intcol1>1;
See: https://jira.mariadb.org/browse/MDEV-9639
37. YES, DEaR, MARIADB
VISIBLE IN ERROR LOG
MariaDB [(none)]> set global log_warnings=2;
MariaDB [(none)]> insert into allthingsopen.t2 (select * from
allthingsopen.t1 where charcol3 like '123-45-6789' limit 1);
Query OK, 0 rows affected, 1 warning (0.00 sec)
Records: 0 Duplicates: 0 Warnings: 1
[root@encr_maria ~]# less /var/lib/mysql/encr_maria.err
...
2016-10-19 20:58:53 140158235900672 [Warning] Unsafe statement
written to the binary log using statement format since
BINLOG_FORMAT = STATEMENT. The statement is unsafe because it
uses a LIMIT clause. This is unsafe because the set of rows
included cannot be predicted. Statement: insert into
allthingsopen.t2 (select * from allthingsopen.t1 where charcol3
like '123-45-6789' limit 1)
38. YES, DEaR, MARIADB
VISIBLE IN GENERAL LOG
MariaDB [(none)]> set global general_log=on;
MariaDB [(none)]> insert into allthingsopen.t2 values
(5, 6, 'my', 'secret', '987-65-4321');
[root@encr_maria ~]# less /var/lib/mysql/encr_maria.log
/usr/sbin/mysqld, Version: 10.1.18-MariaDB (MariaDB
Server). started with:
Tcp port: 3306 Unix socket: /var/lib/mysql/mysql.sock
Time Id Command Argument
161019 21:00:42 6 Query insert into
allthingsopen.t2 values (5, 6, 'my', 'secret',
'987-65-4321')
39. YES, DEaR, MARIADB
VISIBLE IN AUDIT LOG
MariaDB [(none)]> INSTALL PLUGIN server_audit SONAME
‘server_audit.so';
MariaDB [(none)]> set global server_audit_mode=1;
MariaDB [(none)]> SET GLOBAL
server_audit_events=‘CONNECT,QUERY,TABLE';
MariaDB [(none)]> SET GLOBAL server_audit_logging=ON;
MariaDB [(none)]> insert into allthingsopen.t1 values
(6, 6, 'my', 'secret', ‘777-77-7777');
[root@encr_maria ~]# tail -f /var/lib/mysql/
server_audit.log
20161019 21:12:30,encr_maria,root,localhost,
7,6,QUERY,allthingsopen,'insert into allthingsopen.t1
values (6, 6, 'my', 'secret', '777-77-7777')',0
40. YES, DEaR, MARIADB
VISIBLE IN RELAY LOGS
[root@encr_maria_slave ~]# xxd /var/lib/mysql/
encr_maria_slave-relay-bin.000002
...
0000300: 0008 0061 6c6c 7468 696e 6773 6f70 656e ...allthingsopen
0000310: 0069 6e73 6572 7420 696e 746f 2074 3320 .insert into t3
0000320: 7661 6c75 6573 2028 342c 2035 2c20 276d values (4, 5, 'm
0000330: 7927 2c20 2773 6563 7265 7427 2c20 2731 y', 'secret', '1
0000340: 3233 2d34 352d 3637 3839 2729 f513 0858 23-45-6789')...X
...
encrypt_binlog will encrypt both binlogs and relay logs on the
configured server, but relay logs on any attached slaves are not
encrypted without configuration on those slaves. Watch those
database permissions!
41. YES, DEaR, MARIADB
DISABLING ENCRYPTION
MariaDB [(none)]> set global innodb_encryption_threads=0;
MariaDB [(none)]> select name from
information_schema.innodb_tablespaces_encryption where
encryption_scheme=1;
| test/sbtest1 |
MariaDB [allthingsopen]> alter table t2 encrypted=‘no';
MariaDB [allthingsopen]> select name from
information_schema.innodb_tablespaces_encryption where
encryption_scheme=1;
Empty set (0.00 sec)
# remove encryption configs from configurationn files
[root@centosbase ~]# sudo systemctl restart mariadb
42. YES, DEaR, ORACLE
EDIT MY.CNFTO INSTALL PLUGIN
[mysqld]
early-plugin-load=keyring_file.so
keyring_file_data=/var/lib/mysql/mysql-keyring/keyring
[root@encr_percona ~]# service mysql start
mysql> SELECT PLUGIN_NAME, PLUGIN_STATUS FROM
INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_NAME LIKE 'keyring%';
+--------------+---------------+
| PLUGIN_NAME | PLUGIN_STATUS |
+--------------+---------------+
| keyring_file | ACTIVE |
+--------------+---------------+
mysql> show global variables like '%keyring%';
+-------------------+--------------------------------------+
| Variable_name | Value |
+-------------------+--------------------------------------+
| keyring_file_data | /var/lib/mysql/mysql-keyring/keyring |
+-------------------+--------------------------------------+
Store your key outside the data directory.
48. YES, DEaR, ORACLE
REMAINING EXPOSURES
Only the tablespace is encrypted, leaving the following
exposed:
• binlogs
• redo log
• relay logs on unencrypted slaves
• slow log
• error log
• general log
• audit log
49. YES, DEaR, ORACLE
DON’T LOSETHE MASTER KEY
[root@encr_percona ~]# ls -l /var/lib/mysql/mysql-
keyring/
total 0
-rw-r-----. 1 mysql mysql 0 Oct 19 16:18 keyring
mysql> select * from t2;
ERROR 3185 (HY000): Can't find master key from
keyring, please check keyring plugin is loaded.
Special note for Xtrabackup later.
51. KEY ROTATION: ORACLE
ROTATINGTHE MASTER KEY
[root@encr_percona ~]# ls -al /var/lib/mysql/mysql-
keyring/keyring
-rw-r-----. 1 mysql mysql 795 Oct 19 20:31 /var/
lib/mysql/mysql-keyring/keyring
[root@encr_percona ~]# mysql -e'alter instance
rotate innodb master key’;
[root@encr_percona ~]# ls -al /var/lib/mysql/mysql-
keyring/keyring
-rw-r-----. 1 mysql mysql 923 Oct 19 20:58 /var/
lib/mysql/mysql-keyring/keyring
52. KEY ROTATION: ORACLE
ROTATINGTHE HEADER KEY
[root@encr_percona ~]# ls -al /var/lib/mysql/
allthingsopen/t2.ibd
-rw-r-----. 1 mysql mysql 98304 Oct 19 21:08 /var/
lib/mysql/allthingsopen/t2.ibd
[root@encr_percona ~]# mysql -e'alter instance
rotate innodb master key’;
[root@encr_percona ~]# ls -al /var/lib/mysql/
allthingsopen/t2.ibd
-rw-r-----. 1 mysql mysql 98304 Oct 19 21:09 /var/
lib/mysql/allthingsopen/t2.ibd
Try at home: look at the header contents of the .ibd file.
53. KEY ROTATION: MARIADB
SOME ALTERNATIVES
Default encryption plugin: no rotation
Key rotation available with:
• Amazon Web Services (AWS) Key Management
Services (KMS) (https://mariadb.com/kb/en/mariadb/
aws-key-management-encryption-plugin/)
• eperi Gateway for Databases (http://eperi.de/mariadb)
54. KEY ROTATION: MARIADB
MAINTENANCE: DECRYPT
# decrypt
MariaDB [(none)]> set global innodb_encryption_threads=0;
MariaDB [(none)]> select name from
information_schema.innodb_tablespaces_encryption where
encryption_scheme=1;
| test/sbtest1 |
MariaDB [allthingsopen]> alter table t2 encrypted=‘no';
MariaDB [allthingsopen]> select name from
information_schema.innodb_tablespaces_encryption where
encryption_scheme=1;
Empty set (0.00 sec)
55. KEY ROTATION: MARIADB
MAINTENANCE: RE-ENCRYPT
[root@centosbase mysql]# mv keys.txt keys.bak
# comment out encryption configs from my.cnf
[root@centosbase ~]# sudo systemctl restart mariadb
# re-encrypt; create new keys.txt and uncomment encryption
configs in my.cnf
[root@centosbase ~]# sudo systemctl restart mariadb
MariaDB [(none)]> alter table allthingsopen.t2 encrypted='yes';
56. DON’T PUT YOUR KEY AT THE FRONT DOOR
OVERVIEW
ALTERNATIVES
YES, DEAR
KEY ROTATION
KEY STORAGETHIRD-PARTYTOOLS
BACKUPS
PERFORMANCE
SUMMARY
57. KEY STORAGE
CAN ENCRYPT KEYFILE
[root@maria101 mysql]# openssl enc -aes-256-cbc -md
sha1 -k mypassword -in /var/lib/mysql/keys.txt -out /
var/lib/mysql/keys.enc
[root@maria101 mysql]# cat /etc/my.cnf | grep key-
management
file-key-management-filename = /var/lib/mysql/keys.enc
file-key-management-filekey = mypassword
…but of course, the password is exposed in the my.cnf on disk
58. KEY STORAGE
CLEVER IDEAS
Store the keyfile on a USB stick.
Store the keyfile on a directory mounted only during
database startup.
Your clever idea!
60. WHAT TOOLS CAN STREAM DATA OR
LOGS FROM YOUR SERVER?
OVERVIEW
ALTERNATIVES
YES, DEAR
KEY ROTATION
KEY STORAGE
THIRD-PARTY TOOLS
BACKUPS
PERFORMANCE
SUMMARY