5. • ON by default
• Only global, thread, statements and
transactions instrumentation enabled
• All other consumers are disabled
Performance Schema defaults
5
6. • Memory allocated on demand
• You don’t need to limit size of tables anymore
• Sys schema included into standard MySQL
distribution
• You can turn statistics on or off for
particular host and/or user
• Size of SQL DIGEST is tunable
Configuraiton improvements in 5.7
6
7. • We will turn required instrumentation ON
for each example separately
• We will use pattern
update performance_schema.setup_consumers set enabled=’yes’
where name like ’OUR_REQUIREMENT_%’;
update performance_schema.setup_instruments set enabled=’yes’, timed=’yes’
where name like ’OUR_REQUIREMENT_%’;
• Be careful!
• They are memory and CPU intensive
• Do not turn them all ON until needed
Prepare
7
8. • We will turn required instrumentation ON
for each example separately
• Or easier
call sys.ps_setup_enable_consumer(YOUR_CONSUMER);
call sys.ps_setup_enable_instrument(YOUR_INSTRUMENT);
• Be careful!
• They are memory and CPU intensive
• Do not turn them all ON until needed
Prepare
7
10. • Table METADATA LOCKS
• Which thread is waiting for a lock
• Which thread holds the lock
• Not only for talbes:
GLOBAL, SCHEMA, TABLE, FUNCTION, PROCEDURE, EVENT, COMMIT, USER LEVEL LOCK,
TABLESPACE
MDL
9
12. • Login into EC2 instance
• Login: see your card
• Password: see your card
• Run load
./test1.sh
CALL help_task()G
CALL help_solve()G
CALL task_prepare();
• We need to find out what prevents ALTER
from finishing
MDL: practice
11
13. • Table TABLE HANDLES
• Not only locks, but also information about
open tables
• FLUSH TABLES removes data from this table
Table locks
12
14. mysql1> select count(*) from employees where first_name like ’Svet%’;
• While running, check what is going on in
parallel connection:
mysql2> select * from table_handlesG
*************************** 1. row ***************************
OBJECT_TYPE: TABLE
OBJECT_SCHEMA: employees
OBJECT_NAME: employees
OBJECT_INSTANCE_BEGIN: 140544885988272
OWNER_THREAD_ID: 23
OWNER_EVENT_ID: 818320
INTERNAL_LOCK: NULL
EXTERNAL_LOCK: READ EXTERNAL -- Table lock!
1 row in set (0.00 sec)
Table locks: example
13
15. mysql1> select count(*), sleep(10) from employees where emp_no=10001;
• In parallel connection:
mysql2> select * from table_handlesG
*************************** 1. row ***************************
OBJECT_TYPE: TABLE
OBJECT_SCHEMA: employees
OBJECT_NAME: employees
OBJECT_INSTANCE_BEGIN: 140544885988272
OWNER_THREAD_ID: 23
OWNER_EVENT_ID: 1011419
INTERNAL_LOCK: NULL
EXTERNAL_LOCK: NULL -- Now everything is good: index access
1 row in set (0.00 sec)
Table locks: example
14
16. • Run load
./tables.sh
CALL help_task()G
CALL help_solve()G
• We need to find out why so many threads
are waiting for a lock and fix the issue
• We can also examine table cache content
Table Handles: practice
15
18. • You could not diagnose where was memory
gone before version 5.7
• Buffers?
• Temporary tables?
• Internal structures which are out of user
control?
• There is no leak, simply OS did not show
memory as freed yet?
Why these are our favorite improvements?
17
19. • free
• top
• vmstat
• Investigation
• There was no way to know how exactly
memory was allocated
Diagnostic tools before 5.7
18
20. • free
$free
total used free shared buffers cached
Mem: 16149184 6223916 9925268 317536 1048 3655160
-/+ buffers/cache: 2567708 13581476
Swap: 2110460 0 2110460
• top
• vmstat
• Investigation
Diagnostic tools before 5.7
18
21. • free
• top
$top
Tasks: 295 total, 3 running, 292 sleeping, 0 stopped, 0 zombie
%Cpu(s): 3.0 us, 0.8 sy, 0.1 ni, 95.4 id, 0.8 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 16149184 total, 6231688 used, 9917496 free, 1048 buffers
KiB Swap: 2110460 total, 0 used, 2110460 free. 3670752 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1914 mysql 20 0 670m 95m 1296 S 0.7 1.2 2:42.14 mysqld
• vmstat
• Investigation
Diagnostic tools before 5.7
18
22. • free
• top
• vmstat
$vmstat -t 5 3
procs ----------------------memory---------------------- ------swap---- --------
r b swpd free buff cache si so bi bo in cs us sy id wa...
2 0 0 9923160 1048 3662724 0 0 168 86 167 674 3 1 87...
0 0 0 9923252 1048 3662904 0 0 30 122 1168 5264 3 1 96...
0 0 0 9922864 1048 3663120 0 0 25 128 1191 5342 2 1 96...
• Investigation
Diagnostic tools before 5.7
18
23. • free
• top
• vmstat
• Investigation
• Total size of buffers
• Number of temporary tables
• Number of parallel connections
Diagnostic tools before 5.7
18
25. mysql> select * from sys.memory_by_thread_by_current_bytes
-> order by current_allocated descG
*************************** 1. row ***************************
thread_id: 152
user: lj@127.0.0.1
current_count_used: 325
current_allocated: 36.00 GiB
current_avg_alloc: 113.43 MiB
current_max_alloc: 36.00 GiB
total_allocated: 37.95 GiB
...
• Finding connections, using too much
memory, now is matter of seconds!
Threads Statistics
20
26. • memory summary by account by event name
• memory summary by host by event name
• memory summary by thread by event name
• memory summary by user by event name
• memory summary global by event name
• You must enable memory instrumentation!
• sys schema includes user name
RAW Performance Schema tables
21
27. • NAME@HOST - regular user
• System users
• sql/main
• innodb/*
• ...
• Data comes from table THREADS
Users in sys.memory * tables
22
28. • Run load
./test2.sh
CALL help_task()G
CALL help_solve()G
CALL task_prepare();
• We need to find out how much memory
uses SysBench load, running in parallel
• To identify how much RAM used by whole
server run
select * from sys.memory_global_total;
Memory usage: practice
23
31. • What happens inside the routine
• Queries, called from the routine
• statement/sp/stmt
Stored routines instrumentation
26
32. • We will use this procedure
CREATE DEFINER=‘root‘@‘localhost‘ PROCEDURE ‘sp_test‘(val int)
BEGIN
DECLARE CONTINUE HANDLER FOR 1364, 1048, 1366
BEGIN
INSERT IGNORE INTO t1 VALUES(’Some string’);
GET STACKED DIAGNOSTICS CONDITION 1 @stacked_state = RETURNED_SQLSTATE;
GET STACKED DIAGNOSTICS CONDITION 1 @stacked_msg = MESSAGE_TEXT;
END;
INSERT INTO t1 VALUES(val);
END
• When HANDLER called?
Stored routines: example
27
33. mysql> call sp_test(1);
Query OK, 1 row affected (0.07 sec)
mysql> select thread_id, event_name, sql_text from events_statements_history
-> where event_name like ’statement/sp%’;
+-----------+-------------------------+----------------------------+
| thread_id | event_name | sql_text |
+-----------+-------------------------+----------------------------+
| 24 | statement/sp/hpush_jump | NULL |
| 24 | statement/sp/stmt | INSERT INTO t1 VALUES(val) |
| 24 | statement/sp/hpop | NULL |
+-----------+-------------------------+----------------------------+
3 rows in set (0.00 sec)
Correct value
28
35. • Run load
./crazy_timing.sh
CALL help_task()G
CALL help_solve()G
CALL task_prepare();
• We need to find out why procedure takes
different time each run
• For better output set pager to less:
mysql> P less
Stored routines: practice
30
37. • Contains current prepared statements
• Statistics by
• Which thread owns the statement
• How many times executed
• Optimizer statistics, similar to
events statements *
Table prepared statements instances
32
38. mysql1> prepare stmt from ’select count(*) from employees where hire_date > ?’;
Query OK, 0 rows affected (0.00 sec)
Statement prepared
mysql1> set @hd=’1995-01-01’;
Query OK, 0 rows affected (0.00 sec)
mysql1> execute stmt using @hd;
+----------+
| count(*) |
+----------+
| 34004 |
+----------+
1 row in set (1.44 sec)
• Try EXECUTE with different variable values
Example: prepared statement
33
39. mysql2> select statement_name, sql_text, owner_thread_id, count_reprepare,
-> count_execute, sum_timer_execute from prepared_statements_instancesG
*************************** 1. row ***************************
statement_name: stmt
sql_text: select count(*) from employees where hire_date > ?
owner_thread_id: 22
count_reprepare: 0
count_execute: 3
sum_timer_execute: 4156561368000
1 row in set (0.00 sec)
mysql1> drop prepare stmt;
Query OK, 0 rows affected (0.00 sec)
mysql2> select * from prepared_statements_instancesG
Empty set (0.00 sec)
Example: diagnosis
34
40. • Run load
./prepared.sh
CALL help_task()G
CALL help_solve()G
• We need to find out how effective is
prepared statement
Prepared statements: practice
35
42. • Data from SHOW SLAVE STATUS available
in replication * tables
• Support of Replication Channels
(Multi-master slave)
• More instruments for GTID
Major improvements
37
43. • No need to parse SHOW output
• Configuration
• IO thread
• SQL thread
SLAVE STATUS
38
44. • No need to parse SHOW output
• Configuration
• replication connection configuration
• replication applier configuration
• IO thread
• SQL thread
SLAVE STATUS
38
45. • No need to parse SHOW output
• Configuration
• IO thread
• replication connection status
• SQL thread
SLAVE STATUS
38
46. • No need to parse SHOW output
• Configuration
• IO thread
• SQL thread
• replication applier status
• replication applier status by coordinator - MTS only
• replication applier status by worker
SLAVE STATUS
38
48. • State of IO Thread
mysql> select * from replication_connection_statusG
*************************** 1. row ***************************
CHANNEL_NAME:
GROUP_NAME:
SOURCE_UUID: d0753e78-14ec-11e5-b3fb-28b2bd7442fd
THREAD_ID: 21
SERVICE_STATE: ON
COUNT_RECEIVED_HEARTBEATS: 17
LAST_HEARTBEAT_TIMESTAMP: 2015-06-17 15:49:08
RECEIVED_TRANSACTION_SET:
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
1 row in set (0.00 sec)
SLAVE STATUS
40
49. • Coordinator thread for multiple workers
mysql> select * from replication_applier_status join
-> replication_applier_status_by_coordinator using(channel_name)G
*************************** 1. row ***************************
CHANNEL_NAME:
SERVICE_STATE: ON
REMAINING_DELAY: NULL
COUNT_TRANSACTIONS_RETRIES: 0
THREAD_ID: 22
SERVICE_STATE: ON
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
1 row in set (0.00 sec)
• Other cases
Performance Schema: State of SQL Thread
41
50. • Coordinator thread for multiple workers
• Other cases
mysql> select * from replication_applier_status join
-> replication_applier_status_by_worker using(channel_name)G
*************************** 1. row ***************************
CHANNEL_NAME: master-1
SERVICE_STATE: OFF
REMAINING_DELAY: NULL
COUNT_TRANSACTIONS_RETRIES: 0
WORKER_ID: 0
THREAD_ID: NULL
SERVICE_STATE: OFF
LAST_SEEN_TRANSACTION: ANONYMOUS
LAST_ERROR_NUMBER: 1032
LAST_ERROR_MESSAGE: Could not execute Update_rows...
Performance Schema: State of SQL Thread
41
51. • Coordinator thread for multiple workers
• Other cases
*************************** 2. row ***************************
CHANNEL_NAME: master-2
SERVICE_STATE: ON
REMAINING_DELAY: NULL
COUNT_TRANSACTIONS_RETRIES: 0
WORKER_ID: 0
THREAD_ID: 42
SERVICE_STATE: ON
LAST_SEEN_TRANSACTION: ANONYMOUS
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
2 rows in set (0,00 sec)
Performance Schema: State of SQL Thread
41
52. • RECEIVED TRANSACTION SET
in table replication connection status
• LAST SEEN TRANSACTION
in replication applier status by worker
GTID diagnostics
42
53. • Single-threaded slave
mysql> select cs.CHANNEL_NAME, cs.SOURCE_UUID, cs.RECEIVED_TRANSACTION_SET,
-> asw.LAST_SEEN_TRANSACTION, aps.SERVICE_STATE from
-> replication_connection_status cs join replication_applier_status_by_worke
-> asw using(channel_name) join replication_applier_status aps
-> using(channel_name) G
*************************** 1. row ***************************
CHANNEL_NAME:
SOURCE_UUID: 9038967d-7164-11e6-8c88-30b5c2208a0f
RECEIVED_TRANSACTION_SET: 9038967d-7164-11e6-8c88-30b5c2208a0f:1-2
LAST_SEEN_TRANSACTION: 9038967d-7164-11e6-8c88-30b5c2208a0f:2
SERVICE_STATE: ON
1 row in set (0,00 sec)
• Multi-threaded
GTID: all in one place
43
54. • Single-threaded slave
• Multi-threaded
*************************** 1. row ***************************
THREAD_ID: 30
SERVICE_STATE: ON
RECEIVED_TRANSACTION_SET: 9038967d-7164-11e6-8c88-30b5c2208a0f:1-3
LAST_SEEN_TRANSACTION:
...
*************************** 8. row ***************************
THREAD_ID: 37
SERVICE_STATE: ON
RECEIVED_TRANSACTION_SET: 9038967d-7164-11e6-8c88-30b5c2208a0f:1-3
LAST_SEEN_TRANSACTION: 9038967d-7164-11e6-8c88-30b5c2208a0f:3
8 rows in set (0,00 sec)
GTID: all in one place
43
55. • Tables in mysql schema
• slave master info
• slave relay log info
• slave worker info
• Join with Performance Schema tables
• New instruments
• memory
• wait
• stage
More diagnostic
44
56. • Run load
./repl.sh
CALL help_task()G
CALL help_solve()G
• We need to find out why replication is
broken and fix it
Replication: practice
45
58. • Variables
• Status variables
• show compatibility 56 = 0
Variables instrumentation
47
59. • Variables
• global variables
• session variables
• user variables by thread
• variables by thread
• Status variables
Variables instrumentation
47
60. • Variables
• Status variables
• global status
• session status
• status by [account|host|thread|user]
Variables instrumentation
47
61. • Same information which is in
• SHOW [GLOBAL] STATUS
• I S.GLOBAL VARIABLES (deprecated in 5.7)
• I S.SESSION VARIABLES (deprecated in
5.7)
• Helps to watch session variables changes
Global and session variables
48
62. • Same information which is in
• SHOW [GLOBAL] STATUS
• I S.GLOBAL STATUS (deprecated in 5.7)
• I S.SESSION STATUS (deprecated in 5.7)
Status variables
49
63. mysql> SELECT ss.variable_name, ss.variable_value FROM session_status ss
-> LEFT JOIN global_status gs USING(variable_name)
-> WHERE ss.variable_value != gs.variable_value OR gs.variable_value IS NULL
-> AND ss.variable_value>0;
+----------------------------+----------------+
| variable_name | variable_value |
+----------------------------+----------------+
| Bytes_sent | 197774 |
| Handler_commit | 0 |
| Handler_external_lock | 44 |
| Handler_read_first | 3 |
| Handler_read_key | 523 |
| Handler_read_next | 0 |
| Handler_read_rnd_next | 7241 |
| Opened_table_definitions | 0 |
...
Status variables
50
64. • variables by thread
• status by
• account
• host
• thread
• user
Possible to group
51
65. • variables by thread
mysql> select * from variables_by_thread where variable_name=’tx_isolation’;
+-----------+---------------+-----------------+
| THREAD_ID | VARIABLE_NAME | VARIABLE_VALUE |
+-----------+---------------+-----------------+
| 71 | tx_isolation | REPEATABLE-READ |
| 83 | tx_isolation | REPEATABLE-READ |
| 84 | tx_isolation | SERIALIZABLE |
+-----------+---------------+-----------------+
3 rows in set, 3 warnings (0.00 sec)
• status by
Possible to group
51
66. • variables by thread
• status by
mysql> select * from status_by_thread where variable_name=’Handler_write’;
+-----------+---------------+----------------+
| THREAD_ID | VARIABLE_NAME | VARIABLE_VALUE |
+-----------+---------------+----------------+
| 71 | Handler_write | 94 |
| 83 | Handler_write | 477 | -- Most writes
| 84 | Handler_write | 101 |
+-----------+---------------+----------------+
3 rows in set (0.00 sec)
Possible to group
51
67. • Grouped by connection
• Sometimes can help to find tricky bugs with
persistent connections
mysql> select * from user_variables_by_thread;
+-----------+---------------+----------------+
| THREAD_ID | VARIABLE_NAME | VARIABLE_VALUE |
+-----------+---------------+----------------+
| 71 | baz | boo |
| 84 | foo | bar |
+-----------+---------------+----------------+
2 rows in set (0.00 sec)
User variables
52
68. • VARIABLES INFO in 8.0
• Source of variable
COMPILED
EXPLICIT
COMMAND LINE
DYNAMIC
• Path of option file if specified
• Minimum and maximum values
Variables info
53
69. • VARIABLES INFO in 8.0
mysql> select * from variables_info G
*************************** 1. row ***************************
VARIABLE_NAME: auto_increment_increment
VARIABLE_SOURCE: COMPILED
VARIABLE_PATH:
MIN_VALUE: 1
MAX_VALUE: 65535
*************************** 2. row ***************************
VARIABLE_NAME: basedir
VARIABLE_SOURCE: EXPLICIT
VARIABLE_PATH: /home/sveta/build/mysql-8.0/mysql-test/var/my.cnf
MIN_VALUE: 0
MAX_VALUE: 0
...
Variables info
53
70. • VARIABLES INFO in 8.0
• Source of variable
COMPILED
EXPLICIT
COMMAND LINE
DYNAMIC
• Path of option file if specified
• Minimum and maximum values
• No variable values in this table!
Variables info
53
71. • Run load
./variables.sh
CALL help_task()G
CALL help_solve()G
CALL task_prepare();
• We need to watch progress of INSERT
command, running by stored routine.
• Note what there is parallel load, caused by
SysBench. We are not interested in its
statistics.
Variables: practice
54
73. • Traditionally aggregated
• events errors summary by account by error
• events errors summary by host by error
• events errors summary by thread by error
• events errors summary by user by error
• events errors summary global by error
• All tables have similar structure
Errors Summary Tables in 8.0
56