Using MySQL in Automated Testing

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Using
MySQL
in

Automated
Testing
Morgan
Tocker

MySQL
Community
Manager

February
2015

Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Safe
Harbor
Statement
The
following
is
intended
to
outline
our
general
product
direction.

It
is
intended
for
information
purposes
only,
and
may
not
be

incorporated
into
any
contract.
It
is
not
a
commitment
to
deliver

any
material,
code,
or
functionality,
and
should
not
be
relied
upon

in
making
purchasing
decisions.
The
development,
release,
and

timing
of
any
features
or
functionality
described
for
Oracle’s

products
remains
at
the
sole
discretion
of
Oracle.
3

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
4
Today’s
Agenda
Introduction

Tools

Bootstrapping

Pushing
Database
Changes

Metrics
to
Capture
1
2
3
4
5

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Introduction
• Database
are
not
an
outlier

• Can
be
automated
/
scripted
/
managed
as
part
of
a

deployment

• Many
good
tools
exist
5

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
History
of
MySQL
specific
versions
• MySQL
5.1
and
below:

• No
Online
DDL

• Adding
indexes
to
InnoDB
tables
causes
rebuild

• Supports
triggers
from
5.0
(!)
6

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
History
of
MySQL
specific
versions
(cont.)
• MySQL
5.5
(2010):

• InnoDB
as
default

• Fast
ALTER
TABLE
7

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
History
of
MySQL
specific
versions
(cont.)
• MySQL
5.6
(2013):

• Online
DDL

• Performance
Schema
by
default
8

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
History
of
MySQL
specific
versions
(cont.)
• MySQL
5.7
(In
Development):

• Multiple
triggers
per
table
9

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Online
DDL
(MySQL
5.6+)
• Can
add
indexes,
many
other
changes
without
blocking

READS
or
WRITES:

• http://dev.mysql.com/doc/refman/5.6/en/innodb-‐
create-‐index-‐overview.html

• Still
requires
application
to
handle
two
versions
of

schema.

• Does
not
facilitate
READ/WRITE
split
with
slaves.
10

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

| 11
In-Place? Copies Table? Allows Concurrent
DML?
Allows Concurrent
Query?
CREATE INDEX,ADD INDEX Yes* No* Yes Yes
ADD FULLTEXT INDEX Yes No* No Yes
DROP INDEX Yes No Yes Yes
OPTIMIZE TABLE Yes Yes Yes Yes
Set default value for a column Yes No Yes Yes
Change auto-increment value for a column Yes No Yes Yes
Add a foreign key constraint Yes* No* Yes Yes
Drop a foreign key constraint Yes No Yes Yes
Rename a column Yes* No* Yes* Yes
Add a column Yes Yes Yes* Yes
Drop a column Yes Yes Yes Yes
Reorder columns Yes Yes Yes Yes
Change ROW_FORMAT property Yes Yes Yes Yes
Change KEY_BLOCK_SIZE property Yes Yes Yes Yes
Make column NULL Yes Yes Yes Yes
Make column NOT NULL Yes* Yes Yes Yes
Change data type of column No Yes No Yes
Add primary key Yes* Yes Yes Yes
Drop primary key and add another Yes Yes Yes Yes
Drop primary key No Yes No Yes
Convert character set No Yes No Yes
Specify character set No Yes No Yes
Rebuild with FORCE option No Yes No Yes

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
12
Today’s
Agenda
Introduction

Tools

Bootstrapping

Pushing
Database
Changes

Metrics
to
Capture
2
3
4
2
1
5

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Worth
Considering
• pt-‐online-‐schema-‐change

• MySQL
Sandbox

• SYS

• Outbrain
Propagator

• Liquibase

• ORM
specific
migrations

• libeatmydata
13

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
pt-‐online-‐schema-‐change
• Part
of
“Percona
Toolkit”.

• Simulates
Online
DDL
by
creating
a
shadow
table
and

adding
triggers
to
the
original
table.

• Later
performs
a
hot
switchover.

• Useful
for
statements
that
do
not
support
Online
DDL.

• Required
in
substitution
of
Online
DDL
in
the
case
slaves

are
required
for
active
use.
14

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
MySQL
Sandbox
• Allows
you
to
run
multiple
copies
at
once,
each
self

contained.

• Can
bootstrap
a
fresh
install
from
a
.tar.gz
of
MySQL.

• Can
bootstrap
common
topologies.

• i.e.
Master
with
a
slave
15

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
SYS
• Common
set
of
views
on
MySQL’s
Performance
Schema.

• Entirely
SQL
view
of
all
of
MySQL’s
Performance.

• Could
potentially
problematic
queries
in
QA
during
test

suite
execution,
etc…
16

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Outbrain
Propagator
• Based
on
SQL
Scripts

• Concept
of
Environments
17

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

| 18

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Liquibase
• Specify
change
in
XML/Yaml/JSON/SQL
19

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
ORM
Specific
• Rails
Migrations

• Django
Migrations
(1.7+)
20

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
libeatmydata
• Disables
fsyncs
and
related
functions

• Packaged
in
Ubuntu

• apt-‐get
install
eatmydata
21
root@mysqldb:/usr/local/mysql# eatmydata ./bin/
mysqld —user=mysql

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
22
Today’s
Agenda
Introduction

Tools

Bootstrapping

Pushing
Database
Changes

Metrics
to
Capture
2
3
4
1
5
3

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Building
a
new
QA
Environment
• Recommend
starting
with
either
MySQL::Sandbox
or

official
repos.

• Can
restore
data
from
mysqldump,
or
copying
data
data

directory
in
place.
23

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Official
Repos
24
sudo yum localinstall http://dev.mysql.com/get/
mysql-community-release-el6-5.noarch.rpm
yum install mysql-community-server
• More
Information:

• http://dev.mysql.com/downloads/repo/yum/

• http://dev.mysql.com/downloads/repo/apt/

• http://dev.mysql.com/downloads/repo/suse/

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
MySQL::Sandbox
25
$ make_sandbox mysql-5.6.23-osx10.9-x86_64.tar.gz
unpacking /Users/morgo/Downloads/mysql-5.6.23-osx10.9-x86_64.tar.gz
Executing low_level_make_sandbox --basedir=/Users/morgo/Downloads/
5.6.23
--sandbox_directory=msb_5_6_23
--install_version=5.6
--sandbox_port=5623
--no_ver_after_name
--my_clause=log-error=msandbox.err
The MySQL Sandbox, version 3.0.47
(C) 2006-2013 Giuseppe Maxia
installing with the following parameters:
upper_directory = /Users/morgo/sandboxes
sandbox_directory = msb_5_6_23
sandbox_port = 5623
..
do you agree? ([Y],n) Y
loading grants
.. sandbox server started
Your sandbox server was installed in $HOME/sandboxes/msb_5_6_23

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
My
Local
Sandboxes
26
morgo@Rbook:~/sandboxes$ ls
clear_all msb_5_5_40 msb_5_6_15 msb_5_6_22
msb_5_7_3 msb_5_7_5.bak sandbox_action status_all
msb_5_5_32 msb_5_6_12 msb_5_6_16 msb_5_6_23
msb_5_7_4 plugin.conf send_kill_all stop_all
msb_5_5_36 msb_5_6_13 msb_5_6_17 msb_5_7_1
msb_5_7_5 restart_all start_all use_all
morgo@Rbook:~/sandboxes$ cd msb_5_6_23
morgo@Rbook:~/sandboxes/msb_5_6_23$ ./use
Welcome to the MySQL monitor. Commands end with ; or g.
Your MySQL connection id is 3
Server version: 5.6.23 MySQL Community Server (GPL)
..
Type 'help;' or 'h' for help. Type 'c' to clear the current input
statement.
mysql [localhost] {msandbox} ((none)) >
Version
5.6.23

=
Port
5623

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Restoring
Data
• Some
testing
environments
(i.e.
staging)
may
have
a

snapshot
of
actual
data.

• This
is
actually
a
very
good
practice
if
your
data

security
policy
allows
for
it.

• Ensures
highest
chance
of
same
performance

characteristics
as
production.
27

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Restoring
Data
(cont.)
• mysqldump
can
be
very
slow
to
load
data
into
your

newly
setup
environment.

• Consider
copying
data
directory
in
raw
format
instead.
28

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
How
to
Configure
MySQL
in
QA?
• It’s
up
to
you!

• For
CI
rule
of
thumb
says
that
the
test
suite
should

execute
in
less
than
5
minutes.
29

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Alternative
#1
-‐
mysql-‐reckless.cnf
• Disable
MySQL’s
durability:

• Warning:
Not
100%
a
best
practice.
30
sync_frm=0
sync_binlog=0
innodb-flush-log-at-trx-commit=0
innodb-doublewrite=0
innodb_support_xa=0
innodb_checksum_algorithm=none
innodb_adaptive_flushing=OFF
Source:
https://github.com/morgo/mysql-‐compatibility-‐config/blob/master/mysql-‐56/mysql-‐56-‐reckless.cnf

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Alternative
#2
-‐
Disable
fsync
and
friends
• At
a
lower
level,
kill
durability.

• Keeps
MySQL
functionality
consistent
between
QA
and

Production.

• Possibly
requires
Linux.
31
root@mysqldb:/usr/local/mysql# eatmydata ./bin/
mysqld —user=mysql

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
32
Today’s
Agenda
Introduction

Tools

Bootstrapping

Pushing
Database
Changes

Metrics
to
Capture
2
3
4
1
5
4

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Two
Categories
of
Change
• Data
Definition
Language

• aka
DDL

• Adding/Dropping
Indexes

• Adding/Dropping
Columns

• ..

• Data
manipulation
language

• aka
DML

• One
off
tasks
to
modify

data

• Repairing
incorrect
values

• ..
33

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
DDL
Considerations
• DDL
does
not
support
transactions

• Is
still
atomic
on
a
per-‐table
basis

• Grouping
multiple
tables
may
complicate
deployment:
34
ALTER TABLE t1 ADD INDEX (a);
ALTER TABLE t2 ADD INDEX (a); // t2 fails
ALTER TABLE t3 ADD INDEX (a);

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
DDL
Considerations
(cont.)
• Good
practice
to
group
all
changes
for
one
table
to

single
chained
ALTER
command.

• Saves
on
table
rebuilds
where
required:
35
ALTER TABLE my_table ADD my_column1 VARCHAR(255),
ADD my_column2 VARCHAR(255);

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
DDL
Considerations
(cont.)
• Don’t
play
a
guessing
game
with
Online
DDL.

• ALTER
TABLE
supports
LOCK=NONE
extension:
36
CREATE TABLE acdc (a int);
ALTER TABLE acdc ADD PRIMARY KEY (a), LOCK=NONE;
Query OK, 0 rows affected (0.29 sec)
Records: 0 Duplicates: 0 Warnings: 0
ALTER TABLE acdc DROP PRIMARY KEY, LOCK=NONE;
ERROR 1846 (0A000): LOCK=NONE is not supported. Reason:
Dropping a primary key is not allowed without also
adding a new primary key. Try LOCK=SHARED.

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
DDL
Considerations
(cont.)
• Any
time
you
change
indexes:

• Existing
query
plans
can
change

• Diligent
DBAs
watch
for
regressions
:)
37

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
DML
Example
• Added
a
new
column
called
is_paying_user
on
users

table.

• New
code
correctly
sets
is_paying_user
to
0
or
1.

• All
5M
existing
users
will
need
the
is_paying_user

property
set.
38

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
DML
Considerations
• SQL
is
not
guaranteed
idempotent

• Best
to
make
sure
that
any
script
fails
or
does
nothing
if

executed
twice.
39
UPDATE t SET a = a+1; // potentially bad.
UPDATE t SET a = 11; // better
UPDATE t SET a = 11 WHERE a = 10; // safest

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
DML
Considerations
(cont.)
• Modifications
via
temporary
tables
are
unsafe
for

Statement-‐based
replication:
40
CREATE TEMPORARY TABLE tmp_user_payments AS
SELECT DISTINCT(user_id) as user_id FROM payments
WHERE payment_total > 0;
/* Slave can disconnect here and lose tmp table */
UPDATE users, tmp_user_payments SET
is_paying_user=1 WHERE user
tmp_user_payments.user_id=users.user_id;

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Suggested
Fixes:
• Use
Row-‐Based
replication
(proposed
default
in
5.7)

• Use
base
table
instead
of
temporary
table.

• Perform
change
in
single
step,
such
as
via
sub
query

(really
requires
5.6+)
41

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
DML
Considerations
(cont.)
• Locks
are
held
for
the
duration
of
a
transaction

• Scripts
that
appear
safe
in
QA
may
still
create

production
problems

• Common
practice
to
chunk-‐split
batch
operations
to

yield
locks
(i.e.
10K
row
modifications
per
transaction.)

• Not
strictly
ACID
:)
42

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Distributed
Systems
• i.e.
Sharded
of
Functional
Partitioned
Deployments

• Presents
special
challenges

• Application
typically
must
talk
to
two
versions
of
a

schema
at
once.

• Requirement
still
true
for
non
distributed
systems…
43

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Why
two
versions
of
a
Schema?
• Even
when
DDL
is
online,
can’t
control
when
it
finishes

and
new
schema
is
immediately
visible
to
application.

• Guaranteed
safe
way:

• Stop
old
version
of
Application

• Apply
Database
Changes

• Start
new
version
of
Application
44

Painful

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Examples
of
Incompatibilities
• Renaming
a
column

• $row[4];
//
Using
nth
column.

Can’t
guarantee
new
column
preserves
order.

• INSERT
into
my_table
VALUES
(1,2,3);
//
Non-‐full
inserts
if
column

count
changes

• SELECT
*
FROM
a
UNION
SELECT
*
FROM
b;
//
Column
count

mismatch
if
a
adds
a
column

• INSERT
INTO
a
SELECT
*
FROM
b
//
Column
count
mismatch
if
a
adds

a
column
45

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
How
to
make
an
application
speak
two

schemas?
• Introduce
Social
Change

• Developers
will
not
always
consider
deployment

issues

• Easy
way
to
force
them
is
to
make
database
changes

separate
from
other
changes
46

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
When
to
add
that
column?
• Make
the
change
happen
in
the
release
before.

• If
it’s
easier
to
explain,
have
two
types
of
software

releases:

• Database
Changes

• Code
Changes
47

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
When
to
add
that
column?
(cont.)
• Can
also
be
handled
as
push
prerequisites
or
post-‐
requisites.

• Adds
to
complexity.

• Release
steps
should
not
boil
the
ocean
-‐
hard
to
keep

simple
and
automated.
48

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
“Bad”
Things
• Not
always
likely
to
catch
slow
queries
in
QA
while
tests

execute.

• Data
set
can
not
always
reflect
production.

• May
be
able
to
catch
symptoms.
51

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Symptoms
• SHOW
GLOBAL
STATUS
LIKE
'Select_full_join';

• Slow
queries

• Unhandled
errors
or
warnings

• In
non-‐strict
versions
of
MySQL
“warnings”
are

probably
“bugs”.

• Not
recommended
to
use
deprecated
features
from

Day
#1.
52

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Slow
Queries
54
mysql> select * from statement_analysisG 
*************************** 1. row *************************** 
query: SELECT `sys` . `format_stateme ... cy` , `sys` . `format_time` ( 
db: sys 
full_scan: * 
exec_count: 4 
err_count: 0 
warn_count: 0 
total_latency: 100.45 ms 
max_latency: 30.83 ms 
avg_latency: 25.11 ms 
lock_latency: 1.59 ms 
rows_sent: 683 
rows_sent_avg: 171 
rows_examined: 1367 
rows_examined_avg: 342 
rows_affected: 0 
rows_affected_avg: 0 
tmp_tables: 0 
tmp_disk_tables: 0 
rows_sorted: 683 
sort_merge_passes: 0 
digest: 0efbdc5d2727eea1b943011cde39e375 
first_seen: 2015-02-20 07:53:52 
last_seen: 2015-02-20 07:54:10

Copyright
©
2015
Oracle
and/or
its
affiliates.
All
rights
reserved.

|
Runtime
Stats
mysql> select * from wait_classes_global_by_latency; 
+-------------------+--------------+---------------+-------------+-------------+-------------+ 
| event_class | total_events | total_latency | min_latency | avg_latency | max_latency | 
+-------------------+--------------+---------------+-------------+-------------+-------------+ 
| wait/io/file | 550470 | 46.01 s | 19.44 ns | 83.58 µs | 4.21 s | 
| wait/io/socket | 228833 | 2.71 s | 0 ps | 11.86 µs | 29.93 ms | 
| wait/io/table | 64063 | 1.89 s | 99.79 ns | 29.43 µs | 68.07 ms | 
| wait/lock/table | 76029 | 47.19 ms | 65.45 ns | 620.74 ns | 969.88 µs | 
| wait/synch/mutex | 635925 | 34.93 ms | 19.44 ns | 54.93 ns | 107.70 µs | 
| wait/synch/rwlock | 61287 | 7.62 ms | 21.38 ns | 124.37 ns | 34.65 µs | 
+-------------------+--------------+---------------+-------------+-------------+-------------+
55

Using MySQL in Automated Testing

Using MySQL in Automated Testing

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (20)

Similaire à Using MySQL in Automated Testing

Similaire à Using MySQL in Automated Testing (20)

Plus de Morgan Tocker

Plus de Morgan Tocker (11)

Dernier

Dernier (20)

Using MySQL in Automated Testing