advanced databasing

1. (R)DBMS
Advanced techniques
Bogdan Kecman
MySQL Principal Technical Engineer @ORACLE
bogdan.kecman@oracle.com
bogdan.kecman@bad-team.net
GrowIT, Novi Sad, December 2018.

2. Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied
upon in making purchasing decisions. The development, release, and timing of any
features or functionality described for Oracle’s products remains at the sole discretion of
Oracle.
2

3. 3
Follow the talk: https://joind.in/talk/c03bc

4. 4
Speed!!! Availability?!? Reliability?!? Scalability?!

5. 5
Total Cost of Operation

6. Conventional setup
6
PRESENTATION LAYER
CLIENTS
MIDDLEWARE
LAYER LOGIC
INTEGRATION
PLUGINS and ADAPTERS
WRAPPERS
EVENT SOURCES BUSINESS MODULES RESOURCE MGM
MAGIC
MONITORING
DATA STORAGE
?

7. MAGIC?
• Education?
• Information volume?
• Information complexity?
• Not sexy enough?
• Money not good?
• Requirement?
Why is so little attention given to monitoring and data storage?
7

8. How to uncover the MAGIC?
●
Understand the problem
●
Check out all available tools
●
Think OUTSIDE THE BOX but KNOW THE BOX!!!
Any sufficiently advanced technology is indistinguishable from magic!
8

9. Problems?
●
Availability & Reliability – robustness
●
Scalability – “all directions”
●
Performance
●
TCO optimization
For every expert, there is an equal and opposite expert!
9

10. Know your tools
Any sufficiently advanced technology is indistinguishable from magic!
10

11. Examples – MySQL
MySQL environment can be very complex or very simple!
11
MySQL Cluster Data Nodes InnoDB SE s1 SE s2MyISAM

12. Examples – MySQL InnoDB
●
SQL (via MySQL connector/php,java,ruby,python..., odbc, C-API..)
●
MEMCACHED (via memcached plugin on the mysqld)
●
InnoDB API (C-API)
How can you access data stored in InnoDB table?
12

13. Examples – MySQL HTTP plugin
●
SQL over HTTP(s)
– http(s)://host:port/sql/SELECT+%2A+FROM+t1+ORDER+BY+f1
●
CRUD (aka REST) over HTTP(s)
– http(s)://host:port/crud/database/table/primaryKey
●
JSON over HTTP(s)
– http(s)://host:port/doc/database/table
– http(s)://host:port/doc/database/table/key
Other ways to access data?
13

14. Results – MySQL InnoDB
●
70000 TPS vs 10000 TPS
14

15. Examples – MySQL Cluster
●
SQL (via MySQL connector/php,java,ruby..., odbc, MySQL C-API..)
●
MEMCACHED (add ndbcluster driver to memcached server)
●
ClusterJ, JPA, ClusterJPA, LDAP
●
ndbAPI (C/C++ API)
MySQL Cluster is a system that runs 80% of mobile traffic in the World!
15
ClusterJ
MySQL
JDBC
Apps
JPA
JNI
Python Ruby
ClusterJPA
Apps Apps Apps Apps Apps
Node.js
JSON
Apps
mod-ndb
Apache
Apps
ndb-eng
Memcached
Apps Apps
NDB API (C++)
MySQL Cluster Data Nodes
Apps
PHP PERL
Apps

16. Examples – MySQL Cluster
●
SQL (via MySQL C-API or JDBC..) to create and backup/restore schema (DDL)
●
ClusterJ, LDAP to access and manipulate data (DML)
MySQL Cluster how are Telco’s accessing it
16
Visitor Location Register
Authentication Center
Home Location Register
NDB API (C++)
MySQL Cluster Data Nodes
Management
Apps
C API / JDBC ClusterJ
JNIMySQL LDAP

17. Examples – MySQL Cluster as persistent memcached
Memcached is fast as very usable, can we get similar performance but keep
data persistent ?
17
hash key
to find data
Memcache
httpd memcached
memcached
memcached
memcache key
PHP/Perl
friends:12389
MySQL
Cluster
Data Nodememcached
Memcache
Client
Application
MySQL
Cluster
Data Node
NDB Engine
Traditional
Persistent with MCCGE

18. Results - MySQL Cluster as persistent memcached
memcachetest -t 2 -M 7000 -c 25000
18

19. Results – MySQL Cluster SQL vs NDB API
Reads / seconds – note logarithmic Y axis
19

20. PostgreSQL Columnar Store
●
Compression
●
Column Projection
●
Different / no Indexes
Have Analytics Workload, running trough OLAP cubes
20

21. PostgreSQL GPU Acceleration
●
CUDA 7.0 or later
●
Accelerates sequential scan, hash based table join and aggregate
functions
PG Strom
21

22. Results – PostgreSQL
Data courtesy of Nikola Krgović from Twin Star Systems
22
PSQL Columnar GPU GPU+Columnar
test1 10.4 143.3 9.1 28.1
test2 45.6 41.5 36.1 41.2
test3 825 820 859 832

23. Tokutek - TokuDB Storage Engine, TokuMX
Fractals anyone?
23

24. Server monitoring & profiling
●
MySQL Enterprise Monitor
●
MySQL SYS schema
●
pgBadger
●
pgFouine
●
Nagios and Zabbix plugins to monitor (r)DBMS state
I believe in having each device secured and monitoring each device, rather
than just monitoring holistically on the network, and then responding in
short enough time for damage control.
24

25. DB Model optimization
●
First Normal Form
●
Second Normal Form
●
Third Normal Form
●
Boyce – Codd normal form
●
Fourth Normal Form
●
5th
, Domain Key and 6th
Normal Form.
In order to denormalize db model you first need to achieve it’s NF
25

26. Query profiling and tuning / optimization
●
ORM
●
ORM?
●
ORM REALLY?
●
Profile queries!
●
Choose a right (R)DBMS
●
Use CTE (common table expressions)
Do you really trust ORM to write a query for you?
26

27. Example - Query profiling and tuning / optimization
Original:
DELETE FROM t1 WHERE id NOT IN (SELECT id FROM t1_data);
Query time 3.5s - 5s
Optimized to:
DELETE t1 FROM t1 LEFT OUTER JOIN t1_data
ON msg.id=msg_data.id WHERE t1_data.id IS NULL;
Optimized query time below 1.5s! More then 3 times faster!
Large number of entries inside IN() clause is not smart
27

28. Example - Query profiling and tuning / optimization
Do you really trust ORM to write a query for you?
28
Original query (ORM generated) takes 10 minutes to execute:
SELECT t1.wid,
t2.wname,
Date(t1.t_stamp) AS 'dateOnly',
t1.rpd + ( t1.runminpd / 60 ) AS 'PDRuntime',
Min(t1.t_stamp),
l.name,
fg.fieldgroupname
FROM arch.lfedata plc
LEFT JOIN arch.lfe_wellsites w
ON t1.wid = t2.id
LEFT JOIN arch.lfe_locations l
ON t2.locationid = l.id
LEFT JOIN arch.lfe_fieldgroups fg
ON l.fieldgroupid = fg.fieldgroupid

29. Example - Query profiling and tuning / optimization
Do you really trust ORM to write a query for you?
29
WHERE t1.t_stamp BETWEEN '2016-06-01 00:00:00' AND '2016-08-31 23:00:00'
AND quality_code = 192
AND t2.compreport = 1
GROUP BY t1.wid,
Date(t1.t_stamp)
UNION
(SELECT 2,
'NONE',
'2000-00-00 00:00:00',
0,
'2000-00-00 00:00:00',
'ZZ_NoName',
'NoFieldGroup'
ORDER BY name, wname, dateonly DESC);

30. Example - Query profiling and tuning / optimization
Do you really trust ORM to write a query for you?
30
Final optimized query takes 10 seconds to execute:
SELECT t.wid, t.wname, t. 'dateOnly',
plc.rpd +(plc.RunMinPD/60) as 'PDRuntime', l.name, fg.FieldGroupName
FROM (SELECT plc.wid, DATE(plc.t_stamp) as 'dateOnly', MIN(lfedata_ndx) as 'ndx', t2.wname, t2.locationId
FROM arch.lfedata plc INNER JOIN arch.lfe_wellsites w ON plc.wid = t2.id
WHERE plc.t_stamp BETWEEN '2016-06-01 00:00:00' AND '2016-08-31 23:55:00'
AND plc.quality_code = 192
AND t2.compReport = 1
GROUP BY plc.wid, DATE(plc.t_stamp) ) t
INNER JOIN arch.lfedata plc
ON t.ndx = plc.lfedata_ndx
LEFT JOIN arch.lfe_locations l
ON t.locationId = l.id
LEFT JOIN arch.lfe_fieldgroups fg
ON l.fieldGroupId = fg.FieldGroupId;

31. Sources
●
Voxxed Days – Bogdan Kecman: https://youtu.be/u0A0i9OEohs
●
mDevDay – Bogdan Kecman: https://youtu.be/0vnSchjQHoE
●
Data Science – Nikola Krgović: https://youtu.be/MhE1skm0y7k
●
MySQL Documentation: https://dev.mysql.com/doc/
●
PostgreSQL Documentation: https://www.postgresql.org/docs/
●
The one book to rule them all: https://www.amazon.com/dp/0321197844/
●
TokuDB Fractal Index WP: http://www.miniuri.com/306
The only source of knowledge is experience!
31

32. HVALA!
Bogdan Kecman
MySQL Principal Technical Engineer
bogdan.kecman@oracle.com
bogdan.kecman@bad-team.net
GrowIT, Novi Sad, December 2018.