Slides from a presentation given at Percona Live MySQL Conference 2013 in Santa Clara, US.
Topics include:
- How to look for performance bottlenecks
- Foreign Key performance in MySQL Cluster 7.3
- Sharding and table partitioning
- efficient use of datatypes (e.g. BLOBS vs varbinary)
3. 7.3 Feature Update
Node.js Connector
JavaScript (V8 engine) to access data directly in the
Data nodes
No SQL – bypasses the MySQL Server lower latency,
high throughput for simple queries (like PK operations,
simple scans from one table)
3
Copyright 2011 Severalnines AB
4. 7.3 Feature Update
FOREIGN KEYs finally supported!
Implemented at the Data Node level
But..
ERROR 1506 (HY000): Foreign key clause is not yet
supported in conjunction with partitioning
Hopefully fixed for the GA release
What about the performance penalty?
4
Copyright 2011 Severalnines AB
5. 7.3 Foreign Key Perf
create table users_posts (
uid integer ,
fid integer ,
pid integer auto_increment,
message varchar(1024),
primary key(uid,fid, pid),
constraint fk_forum foreign key(fid) references forum(fid) on delete cascade,
constraint fk_user foreign key(uid) references users(uid) on delete cascade
) engine=ndb;
Compare INSERT performance with and w/o FKs
With FK, must check that forum(fid) and users(uid) exists.
Populate with 1M records
5
Copyright 2011 Severalnines AB
6. 7.3 Foreign Key Perf
Bencher drivers the load:
https://github.com/severalnines/bencher
4 threads, 2 data nodes, 4 cores,
App mysqld data nodes
FOREIGN KEYs enabled
Summary:
--------------------------
Average Throughput = 1274.58 tps (stdev=59.71)
6
Copyright 2011 Severalnines AB
7. 7.3 Foreign Key Perf
Bencher drivers the load:
https://github.com/severalnines/bencher
4 threads, 2 data nodes, 4 cores,
App mysqld data nodes
Not using FOREIGN KEYs
Summary:
--------------------------
Average Throughput = 1428.57 tps (stdev=57.10)
Foreign keys gave ~11% drop in performance.
7
Copyright 2011 Severalnines AB
8. 7.2 7.3 Caveats
Rolling upgrade from 7.2.10 to 7.3.1 works!
A little gotcha:
--engine-condition-pushdown no longer supported in
MySQL 5.6
Mysqld will fail to start
Take it out from my.cnf before upgrading!
8
Copyright 2011 Severalnines AB
9. Facts
A single query will never run as fast as on Innodb
(served from RAM)
Network latency is a issue
More data nodes does not speed up query execution
time.
9
Copyright 2011 Severalnines AB
10. OS Tuning
Disable NUMA in /etc/grub.conf
echo ‘0’ > /proc/sys/vm/swappiness
echo ‘vm.swappiness=0’ >> /etc/sysctl.conf
Bind data node threads to CPUs/cores
cat /proc/interrupts | grep eth
cpu0 cpu1 cpu2 cpu3
44: 31 49432584 0 0 xen-dyn-event eth0"
45: 1633715292 0 0 0 xen-dyn-event eth1"
10
Copyright 2011 Severalnines AB
Avoid!
OK!
In config.ini [ndbd default]:
ThreadConfig=ldm={count=1,cpubind=1,2},main={cpubind=3} ..
11. Stability Tuning
Tuning the REDO log is key
FragmentLogFileSize=256M
NoOfFragmentLogFiles=<4-6> X DataMemory in MB / 4 x
FragmentLogFileSize
RedoBuffer=64M for a write busy system
Disk based data:
SharedGlobalMemory=4096M
In the LOGFILE GROUP: undo_buffer_size=128M
Or higher (max is 600M)
11
Copyright 2011 Severalnines AB
12. Stability Tuning
Make sure you don’t have more “execution threads” than cores
You want to have
Major page faults low
Involuntary context switches low
mysql> SELECT node_id, thr_no,thr_nm , os_ru_majflt,
os_ru_nivcsw FROM threadstat;
+---------+--------+--------+--------------+--------------+
| node_id | thr_no | thr_nm | os_ru_majflt | os_ru_nivcsw |
+---------+--------+--------+--------------+--------------+
| 3 | 0 | main | 1 | 541719 |
| 4 | 0 | main | 0 | 561769 |
+---------+--------+--------+--------------+--------------+
2 rows in set (0.01 sec)
12
Copyright 2011 Severalnines AB
13. Application Design
Define the most typical Use Cases
List all my friends, session management etc etc.
Optimize everything for the typical use case
Engineer schema to cater for the Use Cases
Keep it simple
Complex access patterns does not scale
Simple access patterns do ( Primay key and Partitioned Index Scans )
Note! There is no parameter in config.ini that affects
performance – only availability.
Everything is about the Schema and the Queries.
Tune the mysql servers (sort buffers etc) as you would for innodb.
13
Copyright 2011 Severalnines AB
14. Simple Access
PRIMARY KEY lookups are HASH lookup O(1)
INDEX searches a T-tree and takes O(log n) time.
In 7.2 and later JOINs are ok, but in 7.1 you should try
to avoid them.
14
Copyright 2011 Severalnines AB
15. Identifying Bottlenecks
A lot of CPU is used on the data nodes
Probably a lot of large index scans and full table scans are used.
Check Slow query log or a query monitor
A lot of CPU is used on the mysql servers
Probably a lot of GROUP BY/DISTINCT or aggregate functions.
Hardly no CPU is used on either mysql or data nodes
Probably low load
Time is spent on network (a lot of “ping pong” to satisfy a request).
System is running slow in general
Disks (io util), queries, swap (must never happen), network
16. Need To Add Data Nodes?
(adding mysql servers is easy)
top –Hd1
Is any of data nodes threads at 100%?
Yes: add more data nodes (online)
No: do nothing
17. Detecting Query Problems
Here is a standard method for how to attack the problem.
Performance tuning is a never-ending loop:
BEGIN
– Capture information – e.g, slow query log
• Change long_query_time if needed
– EXPLAIN the queries
• What indexes are used?
• Are tables JOINed in the correct order (small to big)
– Re-run the optimized typical use cases using bencher/
mysqlslap
GOTO BEGIN;
END;
Never tune unless you can measure and test!
Don't optimize unless you have a problem!
18. Enable Logging
Slow query log
set global slow_query_log=1;
set global long_query_time=0.01;
set global log_queries_not_using_indexes=1;
General log (if you don’t get enough info in the Slow
Query Log)
Activate for a very short period of time (30-60seconds) –
intrusive
Can fill up disk very fast – make sure you turn it off.
set global general_log=1;
Use Severalnines ClusterControl
Includes a Cluster-wide Query Monitor.
Query frequency, EXPLAINs, lock time etc.
Performance Monitor and Manager.
20. Sharding
By default, all index scans hit all data nodes
good if result set is big – you want as many CPUs as possible to
help you.
For smaller result sets (~a couple of hundred records) Partition
Pruning is key for scalability.
User-defined partitioning can help to improve equality index
scans on part of a primary key.
CREATE TABLE t1 (uid,
fid,
somedata,
PRIMARY KEY(uid, fid))
PARTITION BY KEY(userid);
All data belonging to a particular uid will be on the same
partition.
Great locality!
select * from user where uid=1;
Only one data node will be scanned (no matter how many
nodes you have)
21. Sharding
mysql> show global status like 'ndb_pruned_scan_count’;
+-----------------------+-------+
| Variable_name | Value |
+-----------------------+-------+
| Ndb_pruned_scan_count | 0 |
+-----------------------+-------+
CREATE TABLE t1( … ) PARTITION BY KEY (userid);
An run query, and verify it works:
select * from user where userid=1;
mysql> show global status like 'ndb_pruned_scan_count’;
+-----------------------+-------+
| Variable_name | Value |
+-----------------------+-------+
| Ndb_pruned_scan_count | 1 |
+-----------------------+-------+
22. Sharding
mysql> show global status like 'ndb%pruned%';
| Ndb_api_table_scan_count | 264 |
| Ndb_api_range_scan_count | 18 |
| Ndb_api_pruned_scan_count | 3 |
24. Sharding – EXPLAIN PARTITIONS
mysql> explain partitions select * from users_posts u where u.uid=1G
id: 1
select_type: SIMPLE
table: u
partitions: p0,p1
type: ref
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: const
rows: 2699
Extra: NULL
With PARTITION BY KEY (UID)
mysql> explain partitions select * from users_posts2 u where
u.uid=1G
id: 1
select_type: SIMPLE
table: u
partitions: p0
type: ref
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: const
rows: 2699
Extra: NULL
25. Data Types
BLOBs/TEXTs vs VARBINARY/VARCHAR
BLOB/TEXT columns are stored in an external hidden table.
First 256B are stored inline in main table
Reading a BLOB/TEXT requires two reads
One for reading the Main table + reading from hidden
table
Change to VARBINARY/VARCHAR if:
Your BLOB/TEXTs can fit within an 14000 Bytes record
(record size is currently 14000 Bytes)
Reading/writing VARCHAR/VARBINARY is less expensive
Note 1: BLOB/TEXT are also more expensive in Innodb as BLOB/TEXT data is
not inlined with the table. Thus, two disk seeks are needed to read a
BLOB.
Note 2: Store images, movies etc outside the database on the filesystem.
26. Query Tuning
MySQL Cluster 7.2 and later has pushed down joins joins
are performed in the data nodes.
OPTIMIZER in MySQL Cluster 7.1 and earlier is weak
Statistics gathering is non-existing
Optimizer thinks there are only 10 rows to examine in each
table!
FORCE INDEX / STRAIGHt_JOIN to get queries run the way you
want
27. Query Tuning
if you have two similar indexes:
index(a)
index(a,ts)
on the following table
CREATE TABLE `t1` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`a` bigint(20) DEFAULT NULL,
`ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idx_t1_a` (`a`),
KEY `idx_t1_a_ts` (`a`,`ts`)) ENGINE=ndbcluster DEFAULT CHARSET=latin1
28. Query Tuning
mysql> select count(id) from t1 where a=5;
+-----------+
| count(id) |
+-----------+
| 3072000 |
+-----------+
1 row in set (0.02 sec)
mysql> select count(id) from t1 where a=5
and ts>'2013-04-18 14:34:08’;
+-----------+
| count(id) |
+-----------+
| 512 |
+-----------+
1 row in set (0.00 sec)
29. Query Tuning Pre 7.2
mysql> explain select * from t1 where a=2 and ts='2011-10-05 15:32:11';
+----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref |
rows | Extra |
+----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+
| 1 | SIMPLE | t1 | ref | idx_t1_a,idx_t1_a_ts | idx_t1_a | 9 | const | 10 |
Using where |
+----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+
Use FORCE INDEX(..) ...
mysql> explain select * from t1 FORCE INDEX (idx_t1_a_ts) where a=2 and ts='2011-10-05
15:32:11;
+| 1 | SIMPLE | t1 | ref | idx_t1_a_ts | idx_t1_a_ts | 13 | const,const | 10 |
Using where |
1 row in set (0.00 sec)
..to ensure the correct index is picked!
The difference can be 1 record read instead of any
number of records!
30. Index Statistics
explain select * from t1 where a=5 and ts>'2013-04-18 14:34:08' G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: t1
type: range
possible_keys: idx_t1_a,idx_t1_a_ts
key: idx_t1_a
key_len: 9
ref: const
Rows: 17
Extra: Using where with pushed condition
31. Index Statistics
mysql> analyze table t1;
+---------+---------+----------+----------+
| Table | Op | Msg_type | Msg_text |
+---------+---------+----------+----------+
| test.t1 | analyze | status | OK |
+---------+---------+----------+----------+
1 row in set (3.40 sec)
32. Index Statistics
Mysql> explain select * from t1 where a=5
and ts>'2013-04-18 14:34:08' G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: t1
type: range
possible_keys: idx_t1_a,idx_t1_a_ts
key: idx_t1_a_ts
key_len: 13
ref: NULL
rows: 253
Extra: Using where with pushed condition; Using MRR
1 row in set (0.00 sec)
33. Ndb_cluster_connection_pool
Problem:
A Sendbuffer on the connection between mysqld and the
data nodes is protected by a Mutex.
Connection threads in MySQL must acquire Mutex and the
put data in SendBuffer.
Many threads gives more contention on the mutex
Must scale out with many MySQL Servers.
Workaround:
Ndb_cluster_connection_pool (in my.cnf) creates more
connections from one mysqld to the data nodes
Threads load balance on the connections gives less
contention on mutex which in turn gives increased scalabilty
Less MySQL Servers needed to drive load!
www.severalnines.com/cluster-configurator allows you to
specify the connection pool.
34. Ndb_cluster_connection_pool
Gives atleast 70% better performance and a MySQL Server
that can scale beyond four database connections.
Set Ndb_cluster_connection_pool=2x<CPU cores>
It is a good starting point
One free [mysqld] slot is required in config.ini for each
Ndb_cluster_connection.
4 mysql servers,each with Ndb_cluster_connection_pool=8
requires 32 [mysqld] in config.ini
Note that also memcached and node.js, cluster/j etc also has
the concept of the ndb_cluster_connection_pool.
36. Resources
MySQL Cluster Configurator
www.severalnines.com/config
MySQL Cluster Management + Monitoring
www.severalnines.com/cmon
MySQL Cluster Training Slides
www.severalnines.com/mysql-cluster-training
My Blog
johanandersson.blogspot.com