MySQL Optimizer Overview

MySQL Optimizer Overview
Olav Sandstå
MySQL Optimizer Team


Program Agenda

§  Introduction to MySQL optimizer
§  Query transformations
§  Selecting data access method
§  Join optimizer
§  Subquery optimizations


Query Optimizer

What does the optimizer do?

SELECT a, b JOIN
FROM t1, t2, t3
WHERE t1.a = t2.b
AND t2.b = t3.c Optimizer JOIN
Table
scan
AND t2.d > 20
AND t2.d < 30; Range Ref t1
scan access

t2 t3
Metadata: Statistics:
-  Index information -  Table sizes
-  Uniqueness -  Cardinality
-  Nullability


MySQL Architecture
SQL query Parser

Preprocessor:
Semantic check,name resolution

Optimizer
Logical transformations

Cost-based optimizer:
Join order and access methods

Plan refinement

Query execution
plan

Query result Query execution

Storage Engine
InnoDB MyISAM


MySQL Optimizer Characteristics

§  Produce a query plan that uses least resources
–  IO and CPU JOIN

§  Optimizes a single query
Table
JOIN
–  No inter-query optimizations scan

§  Produces left-deep linear query execution plan Table t4
JOIN scan

Range Ref t3
scan access

t1 t2


Optimizer Overview
Main phases:
Negation elimination
Logical Equality and constant propagation
transformations Conversion of outer to inner joins
Subquery transformations
Evaluation of constant expressions

Ref access analysis
Prepare for Estimation of condition fanout
cost-based Const table detection
optimization Range analysis

Cost-based Access method selection
optimization JOIN optimizer

Table condition pushdown
Plan Access method adjustments
refinement Sort avoidance
Index condition pushdown


Logical Transformations



§  Logical transformations of query conditions:
–  Negation elimination
–  Equality propagations
–  Evaluate constant expressions
–  Remove trivial conditions
§  Conversion of outer to inner join
§  Subquery transformations



Example:
SELECT * FROM t1, t2 WHERE
t1.a = t2.a AND t2.a = 9 AND (NOT (t1.a > 10 OR t2.b > 3) OR (t1.b = t2.b + 7 AND t2.b = 5));
Negation
elimination
t1.a = t2.a AND t2.a = 9 AND (t1.a <= 10 AND t2.b <= 3 OR (t1.b = t2.b + 7 AND t2.b = 5));
Equality/const
propagation
t1.a = 9 AND t2.a = 9 AND (9 <= 10 AND t2.b <= 3 OR (t1.b = 5 + 7 AND t2.b = 5));
Evaluate const
expressions
t1.a = 9 AND t2.a = 9 AND (9 <= 10 AND t2.b <= 3 OR (t1.b = 12 AND t2.b = 5));
Trivial condition
=TRUE removal
t1.a = 9 AND t2.a = 9 AND (t2.b <= 3 OR (t1.b = 12 AND t2.b = 5));


Cost-based Optimization


Cost-based Query Optimization

JOIN
General idea:
§  Assign cost to operations JOIN
Table
scan
§  Assign cost to partial or alternative plans
Range Ref t3
§  Search for plan with lowest cost scan access

t1 t2
§  Cost-based optimizations:

Access method Join order Subquery strategy


Optimizer Cost Model

The ”cost” for executing a query
§  Cost unit: read of a random data page
§  Main cost factors:
–  IO cost:
§  #pages read from table
§  #pages read from index
–  CPU cost:
§  Cost for evaluating query conditions
§  Cost for comparing keys/records


Input to Cost Model

§  IO-cost: §  Statistics:
–  Estimates from storage engine –  Number of records in table
based on number of pages to –  Key distribution/Cardinality:
read
§  Average number of records
–  Both index and data pages per key value
§  Schema: §  Only for indexed columns
–  Length of records and keys §  Maintained by storage engine
–  Uniqueness for indexes –  Number of records in an index
–  Nullability range


Cost Model Example

SELECT * FROM t1 WHERE a BETWEEN 20 and 30;

Table scan:
§  IO-cost: #pages in table
§  CPU cost: #records * ROW_EVALUATE_COST

Range scan (on secondary index):
§  IO-cost: #pages to read from index + #records_in_range
§  CPU cost: #records_in_range * ROW_EVALUATE_COST


Selecting Access Method


Selecting Access Method

Finding the optimal method to read data from storage engine
§  For each table, find the best access method: Main access methods:
§  Table scan
§  1. Check if the access method is useful §  Index scan
§  2. Estimate cost of using access method §  Ref access
§  3. Select the cheapest to be used §  Range scan
§  Index merge
§  Choice of access method is cost based §  Loose index scan


Ref Access

§  Read all records with a given key value using an index:
§  Examples:
SELECT * FROM t1 WHERE t1.key = 7;
SELECT * FROM t1, t2 WHERE t1.key = t2.key;

§  “eq_ref”:
–  Reading from a unique index, max one record returned
§  “ref”:
–  Reading from a non-unique index or a prefix of an index, multiple records
returned


Ref Access Analysis

§  Determine which indexes that can be used for ref access in a join
SELECT City.Name AS Capital, Language FROM CountryLanguage, Country, City
WHERE City.CountryCode = Country.Code
AND City.ID = Country.Capital
AND CountryLanguage.Country = Country.Code

CountryLanguage
Country
Country City
Code CountryCode
Capital ID


Range Optimizer

§  Goal: find the “minimal” ranges for each index that needs to be read
§  Example:
SELECT * FROM t1 WHERE (key1 > 10 AND key1 < 20) AND key2 > 30
§  Range scan using INDEX(key1):

10 20

§  Range scan using INDEX(key2):

30


Range Optimizer, cont.

§  Range optimizer selects the “useful” parts of the WHERE condition:
–  Conditions comparing a column value with a constant:
key > 3 key BETWEEN 4 AND 6 key IS NULL
key = 4 key IN (10,12,..) key LIKE ”abc%”
–  Nested AND/OR conditions are supported
§  Result: list of disjoint ranges that need to be read from index:

§  Cost estimate based on number of records in each range:
–  Record estimate is found by asking the Storage Engine (“index dives”)


Range Access for Multi-part Index

Example table with multi-part index
§  Table:
pk a b c

§  INDEX idx(a, b, c);

§  Logical storage layout of index:

a 10 11 12 13
b 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
c


Range Access for Multi-part Index, cont

§  Equality on 1st index part?
–  Can add condition on 2nd index part in range condition
§  Example:
SELECT * from t1 WHERE a IN (10,11,13) AND (b=2 OR b=4)
a 10 11 12 13
b 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
c

§  Resulting range scan:
2 4 2 4 2 4


Range Access for Multi-part Index, cont

§  Non-Equality on 1st index part:
–  Can NOT add condition on 2nd index part in range condition
§  Example:
SELECT * from t1 WHERE a > 10 AND a < 13 AND (b=2 OR b=4)
a 10 11 12 13
b 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
c

§  Resulting range scan:
a >10 AND a < 13


Index Merge

§  Uses multiple indexes on the same table
§  Implemented index merge strategies:
–  Index Merge Union
§  OR-ed conditions between different indexes
–  Index Merge Intersect
§  AND conditions between different indexes
–  Index Merge Sort-Union
§  OR-ed conditions where condition is a range


Index Merge Union

§  Single index cannot handle ORed conditions on different columns
§  Example:
SELECT * FROM t1 WHERE a=10 OR b=10
§  Index Merge Union:
INDEX(a) 10

INDEX(b) 10

Union

Result: a=10 OR b=10

Loose Index Scan

§  Optimization for GROUP BY and DISTINCT:
SELECT a, b FROM t1 GROUP BY a, b;
SELECT DISTINCT a, b FROM t1;
SELECT a, MIN(b) FROM t1 GROUP BY a;
§  GROUP BY/DISTINCT must be on the prefix of the index

a 10 11 12 13
b 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
c


Join Optimizer


Join Optimizer

”Greedy search strategy”
§  Goal: Given a JOIN of N tables, find the best JOIN ordering
§  Strategy:
–  Start with all 1-table plans
t1 t2 t3 t4
–  Expand each plan with remaining tables
§  Depth-first t2 t3 t4

–  If “cost of partial plan” > “cost of best plan”:
t3 t4 t2 t4 t2 t3
§  “prune” plan
t4 t3 t4 t2 t3 t2


JOIN Optimizer Illustrated
SELECT City.Name, Language FROM Language, Country, City
WHERE City.CountryCode = Country.Code
AND City.ID = Country.Capital
AND City.Population >= 1000000
AND Language.Country = Country.Code;
start

Language Country City
cost=862

Country City Language City Language Country
cost=1245

City Country City Language Country Language

cost=26568 cost=32568 cost=627


Complexity and Cost of Join Optimizer

Join of N tables: N! possible plans to evaluate
Heuristics to reduce the number of plans to
evaluate:
§  Use optimizer_search_depth to limit the
number of tables to consider
§  Pre-sort tables on size and key
dependency order (Improved in MySQL
5.6)
§  When adding the next table to a partial
plan, add all tables that it has an equality
reference to (New in MySQL 5.6)

Plan Refinements


Finalizing Query Plan

Main optimizations:
§  Assigning conditions to tables
–  Evaluate conditions as early as possible in join order
§  ORDER BY optimization: avoid sorting
–  Change to different index
–  Read in descending order
§  Change to a cheaper access method
–  Example: Use range scan instead of table scan or ref access
§  Index Condition Pushdown (New in MySQL 5.6)


Index Condition Pushdown New

§  Pushes conditions that can be evaluated on the MySQL server

index down to storage engine Query
conditions
–  Works only on indexed columns
§  Goal: evaluate conditions without having to
access the actual record
–  Reduces number of disk/block accesses Index

–  Reduces CPU usage Table data
Storage engine
§  Both conditions on a single index and
conditions on earlier tables in a JOIN can be
pushed down

Index Condition Pushdown

How it works

Without ICP: MySQL server With ICP:
Optimizer
Storage Engine: Storage Engine:
1. Reads index Execution 4. 1. Reads index and
2. Reads record evaluates pushed index
3. Returns record condition
3. 2. Reads record
Server: 1. 3. Returns record
Index
4. Evaluates condition 2.
Table data
Storage engine
Server:
4. Evaluates rest of condition


Index Condition Pushdown Example

CREATE TABLE person (
name VARCHAR(..),
height INTEGER,
postalcode INTEGER,
age INTEGER,
INDEX (postalcode, age)
); Evaluated by server

SELECT name FROM person
WHERE postalcode BETWEEN 5000 AND 5500 AND age BETWEEN 21 AND 22 AND height>180;

Pushed to storage engine
Evaluated on index entries


Subqueries


Overview of Subquery Optimizations
Subquery category: Strategy: New

§  IN (SELECT …) •  Semi-Join
•  Materialization
•  IN ➜ EXISTS
§  NOT IN (SELECT …)

§  FROM (SELECT …) •  Derived table

§  <CompOp> ALL/ANY (SELECT ..) •  MAX/MIN re-write

§  EXISTS/other •  Execute subquery

Optimization of IN subqueries
New
A. Semi-join Transformation
1.  Transform IN (and =ANY) subquery to semi-join:
SELECT * FROM t1
WHERE query_where AND outer_expr IN (SELECT inner_expr FROM t2 WHERE cond2)

SELECT * FROM t1 SEMIJOIN t2 ON outer_expr = inner_expr
WHERE query_where AND cond2

2.  Apply transformations/strategies for avoiding/removing duplicates:
Duplicate
Table pullout Weedout First Match
Semi-join
LooseScan materialization
3.  Optimize using cost-based JOIN optimizer

Optimization of IN Subqueries, cont.
New
B. Subquery Materialization
§  Only for non-correlated subqueries
§  Execute subquery once
–  store result in temporary table with unique index (removes duplicates)
§  Outer query does lookup in temporary table

Temporary
SELECT title FROM film

Index

table
Lookup
WHERE film_id IN
(SELECT film_id FROM actor WHERE name=“Bullock”)
Materialize


Optimization of IN Subqueries, cont.

C. IN è EXISTS transformation
§  Convert IN subquery to EXISTS subquery by “push-down” IN-equality
to subquery:
WHERE film_id IN (SELECT film_id FROM actor WHERE name=“Bullock”)

WHERE EXISTS (SELECT 1 FROM actor
WHERE name=“Bullock” AND film.film_id = actor.film_id)

§  Benefit: subquery will evaluate fewer records
§  Note: special handling if pushed down expressions can be NULL

Query Execution Plan


Understanding the Query Plan

EXPLAIN
§  Use EXPLAIN to print the final query plan:

EXPLAIN SELECT * FROM t1 JOIN t2 ON t1.a = t2.a WHERE b > 10 AND c > 10;
+----+-------------+-------+-------+---------------+-----+---------+------+------+-----------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+-----+---------+------+------+-----------------------+
| 1 | SIMPLE | t1 | range | PRIMARY,idx | idx | 4 | NULL | 12 | Using index condition |
| 2 | SIMPLE | t2 | ref | key1 | key1| 4 | t1.a | 1 | NULL |
+----+-------------+-------+-------+---------------+-----+---------+------+------+-----------------------+

§  Structured EXPLAIN (New in MySQL 5.6)
–  JSON format
–  Includes more information about query (eg. pushed index conditions)


Understanding the Query Plan

Structured Explain Example:
EXPLAIN FORMAT=JSON SELECT * FROM t1 WHERE b > 10 AND c > 10 AND d > 10;
EXPLAIN
{
"query_block": {
"select_id": 1,
"table": {
"table_name": "t1",
"access_type": "range",
"possible_keys": [
”idx"
],
"key": ”idx",
"key_length": "4",
"rows": 12,
"filtered": 100,
"index_condition": "((`test`.`t1`.`b` > 10) and (`test`.`t1`.`c` > 10))",
"attached_condition": "(`test`.`t1`.`d` > 10)"
}
}
}

Optimizer Traces "table": "`t1`",
"range_analysis": {
(New in MySQL 5.6) "table_scan": {
"rows": 54,
Understand HOW a query gets optimized },
"cost": 13.9

"best_covering_index_scan": {
§  Tracing of the main steps and decisions "index": ”idx",
"cost": 11.903,
done by the optimizer "chosen": true
},
"analyzing_range_alternatives": {
"range_scan_alternatives": [
§  Example: {
"index": ”idx",
"ranges": [
"10 < a"
SET optimizer_trace=”enabled=on”; ],
SELECT * FROM t1 WHERE a > 10; "rowid_ordered": false,
SELECT * FROM "using_mrr": false,
"index_only": true,
INFORMATION_SCHEMA.OPTIMIZER_TRACE; "rows": 12,
"cost": 3.4314,
"chosen": true
}


Influencing the Optimizer

When the optimizer does not do what you want:
§  Add indexes
§  Force use of specific indexes:
–  USE INDEX, FORCE INDEX, IGNORE INDEX
§  Force specific join order:
–  STRAIGHT_JOIN
§  Adjust optimizer_switch flags:
–  set optimizer_switch=“index_merge=off”
§  Ask question in the MySQL optimizer forum


Query Execution


Nested Loops Join

Execution strategy for joins
SELECT City.Name, Language FROM Language, Country, City WHERE …..

QEP: Execution:
Result
City Range scan
Ref Table
range access scan

Language
Country
Index

Index
City
Country
ref access

City
table scan


Presentations by the MySQL Optimizer Team

Sessions:
§  Overview of New Optimizer Features in MySQL 5.6
§  Query Performance Comparison of MySQL 5.5 and MySQL 5.6
§  Powerful EXPLAIN in MySQL 5.6

BOF:
§  Query Optimizations: What Is New and What Is Coming?


Summary

§  Query transformations
§  Selecting data access method
§  Join optimizer
§  Subquery optimizations
§  Plan refinements

Questions?

Graphic Section Divider


MySQL Optimizer Overview

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (11)

Similaire à MySQL Optimizer Overview

Similaire à MySQL Optimizer Overview (20)

Plus de MYXPLAIN

Plus de MYXPLAIN (17)

Dernier

Dernier (20)

MySQL Optimizer Overview