SlideShare une entreprise Scribd logo
1  sur  46
Télécharger pour lire hors ligne
Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
SQL window functions for MySQL
Dag H. Wanvik
Senior database engineer,
Oracle
2Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not
a commitment to deliver any material, code, or functionality, and should not be
relied upon in making purchasing decisions. The development, release, and
timing of any features or functionality described for Oracle’s products remains at
the sole discretion of Oracle.
3Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
Agenda
Quick intro to window functions
Types of window functions
The window specification
Evaluation and optimizations
More on non-aggregate wfs
Implicit and explicit windows
Q & A
1
2
3
4
5
6
7
4Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
Window functions: what are they?
●
A window function performs a calculation across a set of rows that are
related to the current row, similar to what can be done with an aggregate
function.
●
But unlike traditional aggregate functions, a window function does not
cause rows to become grouped into a single output row.
●
So, similar to normal function, but can access values of other rows “in
the vicinity” of the current row
5Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
SELECT name, department_id, salary,
SUM(salary) OVER (PARTITION BY department_id)
AS department_total
FROM employee
ORDER BY department_id, name;
Window function example, no frame
The OVER keyword
signals a window function
6Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
Partitions: 10, 20, 30
+---------+---------------+--------+------------------+
| name | department_id | salary | department_total |
+---------+---------------+--------+------------------+
| Newt | NULL | 75000 | 75000 |
| Dag | 10 | NULL | 370000 |
| Ed | 10 | 100000 | 370000 |
| Fred | 10 | 60000 | 370000 |
| Jon | 10 | 60000 | 370000 |
| Michael | 10 | 70000 | 370000 |
| Newt | 10 | 80000 | 370000 |
| Lebedev | 20 | 65000 | 130000 |
| Pete | 20 | 65000 | 130000 |
| Jeff | 30 | 300000 | 370000 |
| Will | 30 | 70000 | 370000 |
+---------+---------------+--------+------------------+
Partition == disjoint
set of rows in result set
Here: all rows in partition
are peers
7Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
SELECT name, department_id, salary,
SUM(salary) AS department_total
FROM employee GROUP BY department_id
ORDER BY department_id, name;
ERROR 1055 (42000): Expression #1 of SELECT list is not in GROUP BY clause and
contains nonaggregated column 'mysql.employee.name' which is not functionally dependent
on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by
With GROUP BY
8Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
SELECT /* name, */ department_id, /* salary,*/
SUM(salary) AS department_total
FROM employee GROUP BY department_id
ORDER BY department_id /*, name */;
+---------------+------------------+
| department_id | department_total |
+---------------+------------------+
| NULL | 75000 |
| 10 | 370000 |
| 20 | 130000 |
| 30 | 370000 |
+---------------+------------------+
With GROUP BY
9Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
SELECT name, department_id, salary,
SUM (salary) OVER (PARTITION BY department_id
ORDER BY name
ROWS 2 PRECEDING) total
FROM employee
ORDER BY department_id, name;
Window function example, frame
10Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
Partitions: 10, 20, 30
ORDER BY name
within each
partition
+---------+---------------+--------+--------+
| name | department_id | salary | total |
+---------+---------------+--------+--------+
| Newt | NULL | 75000 | 75000 |
| Dag | 10 | NULL | NULL |
| Ed | 10 | 100000 | 100000 |
| Fred | 10 | 60000 | 160000 |
| Jon | 10 | 60000 | 220000 |
| Michael | 10 | 70000 | 190000 |
| Newt | 10 | 80000 | 210000 |
| Lebedev | 20 | 65000 | 65000 |
| Pete | 20 | 65000 | 130000 |
| Jeff | 30 | 300000 | 300000 |
| Will | 30 | 70000 | 370000 |
+---------+---------------+--------+--------+
moving window frame:
SUM (salary)
...
ROWS 2 PRECEDING
a frame is a subset of a
partition
11Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
SELECT name, department_id, salary,
AVG(salary) OVER w AS `avg`,
salary - AVG(salary) OVER w AS diff FROM employee
WINDOW w as (PARTITION BY department_id)
ORDER BY diff DESC;
Window function example
i.e. find the employees with the largest difference between their wage
and that of the department average
Note: explicit window definition of “w”
12Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
Partitions: 10, 20, 30
+---------+---------------+--------+-----------+------------+
| name | department_id | salary | avg | diff |
+---------+---------------+--------+-----------+------------+
| Jeff | 30 | 300000 | 185000.00 | 115000.00 |
| Ed | 10 | 100000 | 74000.00 | 26000.00 |
| Newt | 10 | 80000 | 74000.00 | 6000.00 |
| Newt | NULL | 75000 | 75000.00 | 0.00 |
| Pete | 20 | 65000 | 65000.00 | 0.00 |
| Lebedev | 20 | 65000 | 65000.00 | 0.00 |
| Michael | 10 | 70000 | 74000.00 | -4000.00 |
| Jon | 10 | 60000 | 74000.00 | -14000.00 |
| Fred | 10 | 60000 | 74000.00 | -14000.00 |
| Will | 30 | 70000 | 185000.00 | -115000.00 |
| Dag | 10 | NULL | 74000.00 | NULL |
+---------+---------------+--------+-----------+------------+
13Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
Types of window functions
●
Aggregates
●
Ranking
●
Analytical
COUNT, SUM, AVG + more to come
RANK, DENSE_RANK, PERCENT_RANK,
CUME_DIST, ROW_NUMBER
NTILE, LEAD, LAG, NTH, FIRST_VALUE,
LAST_VALUE
Blue ones use frames, all obey partitions
14Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
Anatomy of a window specification
window specification ::=
[ existing window name ]
[PARTITION BY expr-1, ... ]
[ORDER BY expr-1, ... [DESC] ]
[ frame clause ]
–
15Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
frame clause bound
partition
CURRENT ROW
UNBOUNDED
PRECEDING
UNBOUNDED
FOLLOWING
n PRECEDING
m PRECEDING
16Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
frame clause
frame clause ::= { ROWS | RANGE } { start | between }
start ::= { CURRENT ROW | UNBOUNDED PRECEDING | n PRECEDING}
between ::= BETWEEN bound-1 AND bound-2
bound ::= start | UNBOUNDED FOLLOWING | n FOLLOWING
●
“start” form implies upper is CURRENT ROW (or its peer iff RANGE)
●
An empty OVER () specification, says that all rows in the partition are
peers and included in frame.
●
An ORDER BY without frame implies frame:
RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
●
Limitation: frame exclusion not supported
17Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
frame clause
●
ROWS or RANGE (GROUPS unit not supported)
●
ROWS: bounds are physical row offsets
●
RANGE: logical offset based on current row's value
Ex: RANGE BETWEEN INTERVAL 1 WEEK PRECEDING AND CURRENT ROW
●
RANGE requires ORDER BY on one numeric expression
●
Limitation: bounds must be static
18Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
SELECT dat, amount, SUM(amount) OVER w FROM payments
WINDOW w AS (ORDER BY dat RANGE
BETWEEN INTERVAL 1 WEEK PRECEDING AND
CURRENT ROW)
ORDER BY dat;
RANGE frame example
i.e. find the sum of payments within the last 7 days
19Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
RANGE frame example
+------------+--------+--------+
| dat | amount | sum |
+------------+--------+--------+
| 2017-01-01 | 100.50 | 300.50 |
| 2017-01-01 | 200.00 | 300.50 |
| 2017-01-02 | 200.00 | 500.50 |
| 2017-01-03 | 200.00 | 700.50 |
| 2017-01-05 | 200.00 | 900.50 |
| 2017-01-10 | 200.00 | 700.00 |
| 2017-01-10 | 100.00 | 700.00 |
| 2017-01-11 | 200.00 | 700.00 |
+------------+--------+--------+
SELECT dat, amount,
SUM(amount) OVER w AS `sum` FROM payments
WINDOW w AS
(ORDER BY dat
RANGE BETWEEN INTERVAL 1 WEEK PRECEDING AND
CURRENT ROW)
ORDER BY dat;
Current row's date is the 10th
, so first row in range is the 3rd
.
Frame cardinality is 4 due to peer in next row.
For Jan 5, the frame cardinality is 5, and sum is 900.50.
20Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
When are they evaluated?
●
after GROUP BY/ HAVING
●
before final ORDER BY, DISTINCT, LIMIT
●
you can have several window functions and several different
windows
●
To filter on wf's value, use a subquery, e.g.
SELECT * FROM (SELECT SUM(salary) OVER (PARTITION BY department_id) `sum`
FROM employee) AS s
WHERE `sum` < 100000;
+-------+
| sum |
+-------+
| 75000 |
+-------+
21Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
Logical flow
JOIN
GROUP
BY
WINDOW
1
WINDOW
n
ORDER BY/
DISTINCT/
LIMIT
Sort for
PARTITION BY
and
ORDER BY
● Tmp file between each windowing step
(in-mem if result set can fit †)
● Streamable wfs vs buffered
● Depends on wf and frame
● Buffered: re-read rows
● O(rows * frame size) optimize
● Move frame for SUM 1 row: invert by
subtraction, add new row.
† cf. variables tmp_table_size, max_heap_table_size
22Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
Streamable evaluation
SELECT name, department_id, salary,
SUM(salary) OVER (PARTITION BY department_id ORDER BY name
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS `sum`
FROM employee;
+---------+---------------+--------+--------+
| name | department_id | salary | sum |
+---------+---------------+--------+--------+
| Newt | NULL | 75000 | 75000 |
| Dag | 10 | NULL | NULL |
| Ed | 10 | 100000 | 100000 | Just accumulate as we see rows
| Fred | 10 | 60000 | 160000 |
| Jon | 10 | 60000 | 220000 |
| Michael | 10 | 70000 | 290000 |
| Newt | 10 | 80000 | 370000 |
| Lebedev | 20 | 65000 | 65000 |
| Pete | 20 | 65000 | 130000 |
| Jeff | 30 | 300000 | 300000 |
| Will | 30 | 70000 | 370000 |
+---------+---------------+--------+--------+
23Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
Logical flow
JOIN
GROUP
BY
WINDOW
1
WINDOW
n
ORDER BY/
DISTINCT/
LIMIT
Row
addressable
buffer
in-mem:
overflows to disk
Permits re-reading
rows when frame
moves
24Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
Non-streamable evaluation
SELECT name, department_id, salary,
SUM(salary) OVER (PARTITION BY department_id ORDER BY name
ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) AS `sum`
FROM employee;
+---------+---------------+--------+--------+
| name | department_id | salary | sum |
+---------+---------------+--------+--------+
| Newt | NULL | 75000 | 75000 |
| Dag | 10 | NULL | NULL |
| Ed | 10 | 100000 | 100000 |
| Fred | 10 | 60000 | 160000 |
| Jon | 10 | 60000 | 220000 |
| Michael | 10 | 70000 | 190000 |
| Newt | 10 | 80000 | 210000 |
| Lebedev | 20 | 65000 | 65000 |
| Pete | 20 | 65000 | 130000 |
| Jeff | 30 | 300000 | 300000 |
| Will | 30 | 70000 | 370000 |
+---------+---------------+--------+--------+
When eval'ing Michael, subtract Ed's
contribution, add Michael
or just evaluate entire frame over again
(non-optimized). In both cases we need
re-visit rows.
25Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
Optimizations
●
Accumulate frames by inversion (discussed; done) ¹
●
Avoid materializing each step in temporary file if possible (done)
●
Sort only once for windows with the same PARTITION/ORDER BY
requirements (done)
●
Sort only once for windows with subset relation on above (not yet)
●
Eliminate sorts by using indexes (not yet)
¹not done by default for floats due to possible under/-overflow errors, but can be enabled using a variable.
Enabling it is crucial for performance for large frames due to the combinatorial hardness.
26Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
Explain, last query
EXPLAIN FORMAT=JSON SELECT name, department_id, salary,
SUM(salary) OVER (PARTITION BY department_id ORDER BY name
ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) AS `sum`
FROM employee;
... "windows": [
{"name": "<unnamed window>",
"evalated as #": 1,
"using sorting": true,
"using temporary file": false,
"uses frame buffer": true,
"optimized frame evaluation": true,
"functions used": [
"sum" ]}
],
"buffer_result": {
:
27Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
More on non-aggregate wfs
●
RANK, DENSE_RANK, PERCENT_RANK, CUME_DIST, ROW_NUMBER
●
LEAD, LAG, FIRST_VALUE, LAST_VALUE, NTH_VALUE
28Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
RANK
SELECT name, department_id AS dept, salary,
RANK() OVER w AS `rank`
FROM employee
WINDOW w AS (PARTITION BY department_id
ORDER BY salary DESC);
+---------+------+--------+------+
| name | dept | salary | rank |
+---------+------+--------+------+
| Newt | NULL | 75000 | 1 |
| Ed | 10 | 100000 | 1 |
| Newt | 10 | 80000 | 2 |
| Fred | 10 | 70000 | 3 |
| Michael | 10 | 70000 | 3 |
| Jon | 10 | 60000 | 5 |
| Dag | 10 | NULL | 6 |
| Pete | 20 | 65000 | 1 |
| Lebedev | 20 | 65000 | 1 |
| Jeff | 30 | 300000 | 1 |
| Will | 30 | 70000 | 2 |
+---------+------+--------+------+
29Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
DENSE_RANK
SELECT name, department_id AS dept, salary,
RANK() OVER w AS `rank`, DENSE_RANK() OVER w AS dense
FROM employee
WINDOW w AS (PARTITION BY department_id
ORDER BY salary DESC);
+---------+------+--------+------+-------+
| name | dept | salary | rank | dense |
+---------+------+--------+------+-------+
| Newt | NULL | 75000 | 1 | 1 |
| Ed | 10 | 100000 | 1 | 1 |
| Newt | 10 | 80000 | 2 | 2 |
| Fred | 10 | 70000 | 3 | 3 |
| Michael | 10 | 70000 | 3 | 3 |
| Jon | 10 | 60000 | 5 | 4 |
| Dag | 10 | NULL | 6 | 5 |
| Pete | 20 | 65000 | 1 | 1 |
| Lebedev | 20 | 65000 | 1 | 1 |
| Jeff | 30 | 300000 | 1 | 1 |
| Will | 30 | 70000 | 2 | 2 |
+---------+------+--------+------+-------+
DENSE_RANK doesn't skip
30Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
ROW_NUMBER
SELECT name, department_id AS dept, salary,
RANK() OVER w AS `rank`, DENSE_RANK() OVER w AS dense,
ROW_NUMBER() OVER w AS `#`
FROM employee
WINDOW w AS (PARTITION BY department_id
ORDER BY salary DESC);
+---------+------+--------+------+-------+---+
| name | dept | salary | rank | dense | # |
+---------+------+--------+------+-------+---+
| Newt | NULL | 75000 | 1 | 1 | 1 |
| Ed | 10 | 100000 | 1 | 1 | 1 |
| Newt | 10 | 80000 | 2 | 2 | 2 |
| Fred | 10 | 70000 | 3 | 3 | 3 |
| Michael | 10 | 70000 | 3 | 3 | 4 |
| Jon | 10 | 60000 | 5 | 4 | 5 |
| Dag | 10 | NULL | 6 | 5 | 6 |
| Pete | 20 | 65000 | 1 | 1 | 1 |
| Lebedev | 20 | 65000 | 1 | 1 | 2 |
| Jeff | 30 | 300000 | 1 | 1 | 1 |
| Will | 30 | 70000 | 2 | 2 | 2 |
+---------+------+--------+------+-------+---+
31Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
CUME_DIST
SELECT name, department_id AS dept, salary,
RANK() OVER w AS `rank`, DENSE_RANK() OVER w AS dense,
ROW_NUMBER() OVER w AS `#`, CUME_DIST() OVER w AS cume
FROM employee
WINDOW w AS (PARTITION BY department_id
ORDER BY salary DESC);
+---------+------+--------+------+-------+---+---------------------+
| name | dept | salary | rank | dense | # | cume |
+---------+------+--------+------+-------+---+---------------------+
| Newt | NULL | 75000 | 1 | 2 | 1 | 1 |
| Ed | 10 | 100000 | 1 | 2 | 1 | 0.16666666666666666 |
| Newt | 10 | 80000 | 2 | 3 | 2 | 0.3333333333333333 |
| Fred | 10 | 70000 | 3 | 4 | 3 | 0.6666666666666666 |
| Michael | 10 | 70000 | 3 | 4 | 4 | 0.6666666666666666 |
| Jon | 10 | 60000 | 5 | 5 | 5 | 0.8333333333333334 |
| Dag | 10 | NULL | 6 | 6 | 6 | 1 |
| Pete | 20 | 65000 | 1 | 2 | 1 | 1 |
| Lebedev | 20 | 65000 | 1 | 2 | 2 | 1 |
| Jeff | 30 | 300000 | 1 | 2 | 1 | 0.5 |
| Will | 30 | 70000 | 2 | 3 | 2 | 1 |
+---------+------+--------+------+-------+---+---------------------+
Cumulative distribution
“For a row R, if we assume ascending
ordering, CUME_DIST of R is the
number of rows with values <= the value
of R, divided by the number of rows
evaluated in the partition. “
32Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
PERCENT_RANK
SELECT name, department_id AS dept, salary,
RANK() OVER w AS `rank`, DENSE_RANK() OVER w AS dense,
ROW_NUMBER() OVER w AS `#`, CUME_DIST() OVER w AS cume,
PERCENT_RANK() OVER w AS p_r
FROM employee WINDOW w AS (PARTITION BY department_id
ORDER BY salary DESC);
+---------+------+--------+------+-------+---+---------------------+-----+
| name | dept | salary | rank | dense | # | cume | p_r |
+---------+------+--------+------+-------+---+---------------------+-----+
| Newt | NULL | 75000 | 1 | 2 | 1 | 1 | 0 |
| Ed | 10 | 100000 | 1 | 2 | 1 | 0.16666666666666666 | 0 |
| Newt | 10 | 80000 | 2 | 3 | 2 | 0.3333333333333333 | 0.2 |
| Fred | 10 | 70000 | 3 | 4 | 3 | 0.6666666666666666 | 0.4 |
| Michael | 10 | 70000 | 3 | 4 | 4 | 0.6666666666666666 | 0.4 |
| Jon | 10 | 60000 | 5 | 5 | 5 | 0.8333333333333334 | 0.8 |
| Dag | 10 | NULL | 6 | 6 | 6 | 1 | 1 |
| Pete | 20 | 65000 | 1 | 2 | 1 | 1 | 0 |
| Lebedev | 20 | 65000 | 1 | 2 | 2 | 1 | 0 |
| Jeff | 30 | 300000 | 1 | 2 | 1 | 0.5 | 0 |
| Will | 30 | 70000 | 2 | 3 | 2 | 1 | 1 |
+---------+------+--------+------+-------+---+---------------------+-----+
(rank - 1) / (total rows - 1)
33Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
NTILE
SELECT name, department_id AS dept, salary,
...
PERCENT_RANK() OVER w AS p_r, NTILE(3) OVER w AS `ntile`
FROM employee WINDOW w AS (PARTITION BY department_id
ORDER BY salary DESC);
+---------+------+--------+------+-------+---+---------------------+-----+-------+
| name | dept | salary | rank | dense | # | cume | p_r | ntile |
+---------+------+--------+------+-------+---+---------------------+-----+-------+
| Newt | NULL | 75000 | 1 | 2 | 1 | 1 | 0 | 1 |
| Ed | 10 | 100000 | 1 | 2 | 1 | 0.16666666666666666 | 0 | 1 |
| Newt | 10 | 80000 | 2 | 3 | 2 | 0.3333333333333333 | 0.2 | 1 |
| Fred | 10 | 70000 | 3 | 4 | 3 | 0.6666666666666666 | 0.4 | 2 |
| Michael | 10 | 70000 | 3 | 4 | 4 | 0.6666666666666666 | 0.4 | 2 |
| Jon | 10 | 60000 | 5 | 5 | 5 | 0.8333333333333334 | 0.8 | 3 |
| Dag | 10 | NULL | 6 | 6 | 6 | 1 | 1 | 3 |
| Pete | 20 | 65000 | 1 | 2 | 1 | 1 | 0 | 1 |
| Lebedev | 20 | 65000 | 1 | 2 | 2 | 1 | 0 | 2 |
| Jeff | 30 | 300000 | 1 | 2 | 1 | 0.5 | 0 | 1 |
| Will | 30 | 70000 | 2 | 3 | 2 | 1 | 1 | 2 |
+---------+------+--------+------+-------+---+---------------------+-----+-------+
Divides an ordered
partition into a
specified number of
groups aka buckets
and assigns a bucket
number to each row in
the partition.
34Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
LEAD, LAG
Returns value evaluated at the row that is offset rows after/before the current row within the
partition; if there is no such row, instead return default (which must be of the same type as
value).
Both offset and default are evaluated with respect to the current row. If omitted, offset
defaults to 1 and default to null
lead or lag function ::= { LEAD | LAG } ( expr [ , offset [ , default expression>] ] )
[ RESPECT NULLS ]
Note: “IGNORE NULLS” not supported, RESPECT NULLS is default but can be specified.
35Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
+---------+------+--------+-------+
| name | dept | salary | lead |
+---------+------+--------+-------+
| Newt | NULL | 75000 | NULL |
| Ed | 10 | 100000 | 80000 |
| Newt | 10 | 80000 | 70000 |
| Fred | 10 | 70000 | 70000 |
| Michael | 10 | 70000 | 60000 |
| Jon | 10 | 60000 | NULL |
| Dag | 10 | NULL | NULL |
| Pete | 20 | 65000 | 65000 |
| Lebedev | 20 | 65000 | NULL |
| Jeff | 30 | 300000 | 70000 |
| Will | 30 | 70000 | NULL |
+---------+------+--------+-------+
LEAD
SELECT name, department_id AS dept, salary,
LEAD(salary, 1) OVER w AS `lead`
FROM employee
WINDOW w AS (PARTITION BY department_id
ORDER BY salary DESC);
36Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
+---------+------+--------+--------+-------+
| name | dept | salary | lag | lead |
+---------+------+--------+--------+-------+
| Newt | NULL | 75000 | NULL | NULL |
| Ed | 10 | 100000 | NULL | 80000 |
| Newt | 10 | 80000 | 100000 | 70000 |
| Fred | 10 | 70000 | 80000 | 70000 |
| Michael | 10 | 70000 | 70000 | 60000 |
| Jon | 10 | 60000 | 70000 | NULL |
| Dag | 10 | NULL | 60000 | NULL |
| Pete | 20 | 65000 | NULL | 65000 |
| Lebedev | 20 | 65000 | 65000 | NULL |
| Jeff | 30 | 300000 | NULL | 70000 |
| Will | 30 | 70000 | 300000 | NULL |
+---------+------+--------+--------+-------+
LAG
SELECT name, department_id AS dept, salary,
LAG(salary, 1) OVER w AS `lag`,
LEAD(salary, 1) OVER w AS `lead`
FROM employee
WINDOW w AS (PARTITION BY department_id
ORDER BY salary DESC);
37Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
FIRST_VALUE, LAST_VALUE, NTH_VALUE
Returns value evaluated at the first, last, nth in the frame of the current row
within the partition; if there is no nth row (frame is too small), the NTH_VALUE
returns NULL.
first or last value ::= { FIRST_VALUE | LAST_VALUE } ( expr )
[ RESPECT NULLS ]
nth_value ::= NTH_VALUE ( expr, nth-row ) [FROM FIRST]
[ RESPECT NULLS ]
Note: “IGNORE NULLS” is not supported, RESPECT NULLS is used but can
be specified.
Note: For NTH_VALUE, “FROM LAST” is not supported, FROM FIRST is used
but can be specified
38Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
SELECT name, department_id AS dept, salary,
SUM(salary) OVER w AS `sum`,
FIRST_VALUE(salary) OVER w AS `first`
FROM employee WINDOW w AS (PARTITION BY department_id
ORDER BY name
ROWS BETWEEN 2 PRECEDING AND CURRENT ROW)
+---------+------+--------+--------+--------+
| name | dept | salary | sum | first |
+---------+------+--------+--------+--------+
| Newt | NULL | 75000 | 75000 | 75000 |
| Dag | 10 | NULL | NULL | NULL |
| Ed | 10 | 100000 | 100000 | NULL |
| Fred | 10 | 60000 | 160000 | NULL |
| Jon | 10 | 60000 | 220000 | 100000 |
| Michael | 10 | 70000 | 190000 | 60000 |
| Newt | 10 | 80000 | 210000 | 60000 |
| Lebedev | 20 | 65000 | 65000 | 65000 |
| Pete | 20 | 65000 | 130000 | 65000 |
| Jeff | 30 | 300000 | 300000 | 300000 |
| Will | 30 | 70000 | 370000 | 300000 |
+---------+------+--------+--------+--------+
FIRST_VALUE “in frame”
Current row: Jon
FIRST_VALUE in frame is: Ed
39Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
SELECT name, department_id AS dept, salary,
SUM(salary) OVER w AS `sum`,
FIRST_VALUE(salary) OVER w AS `first`, LAST_VALUE(salary) OVER w AS `last`
FROM employee WINDOW w AS (PARTITION BY department_id
ORDER BY name
ROWS BETWEEN 2 PRECEDING AND CURRENT ROW)
+---------+------+--------+--------+--------+--------+
| name | dept | salary | sum | first | last |
+---------+------+--------+--------+--------+--------+
| Newt | NULL | 75000 | 75000 | 75000 | 75000 |
| Dag | 10 | NULL | NULL | NULL | NULL |
| Ed | 10 | 100000 | 100000 | NULL | 100000 |
| Fred | 10 | 60000 | 160000 | NULL | 60000 |
| Jon | 10 | 60000 | 220000 | 100000 | 60000 |
| Michael | 10 | 70000 | 190000 | 60000 | 70000 |
| Newt | 10 | 80000 | 210000 | 60000 | 80000 |
| Lebedev | 20 | 65000 | 65000 | 65000 | 65000 |
| Pete | 20 | 65000 | 130000 | 65000 | 65000 |
| Jeff | 30 | 300000 | 300000 | 300000 | 300000 |
| Will | 30 | 70000 | 370000 | 300000 | 70000 |
+---------+------+--------+--------+--------+--------+
LAST_VALUE “in frame”
Current row: Jon
LAST_VALUE in frame is: Jon
40Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
SELECT name, department_id AS dept, salary,
SUM(salary) OVER w AS `sum`,
NTH_VALUE(salary, 2) OVER w AS nth
FROM employee WINDOW w AS (PARTITION BY department_id
ORDER BY name
ROWS BETWEEN 2 PRECEDING AND CURRENT ROW)
+---------+------+--------+--------+--------+
| name | dept | salary | sum | nth |
+---------+------+--------+--------+--------+
| Newt | NULL | 75000 | 75000 | NULL |
| Dag | 10 | NULL | NULL | NULL |
| Ed | 10 | 100000 | 100000 | 100000 |
| Fred | 10 | 60000 | 160000 | 100000 |
| Jon | 10 | 60000 | 220000 | 60000 |
| Michael | 10 | 70000 | 190000 | 60000 |
| Newt | 10 | 80000 | 210000 | 70000 |
| Lebedev | 20 | 65000 | 65000 | NULL |
| Pete | 20 | 65000 | 130000 | 65000 |
| Jeff | 30 | 300000 | 300000 | NULL |
| Will | 30 | 70000 | 370000 | 70000 |
+---------+------+--------+--------+--------+
NTH_VALUE “in frame”
Current row: Jon
NTH_VALUE(.., 2) in frame is: Fred
41Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
Implicit and explicit windows
●
Windows can be implicit and unnamed:
COUNT(*) OVER (PARTITION BY DEPARTMENT_ID)
●
Windows can be defined and named via the windows clause clause:
SELECT COUNT(*) OVER w FROM t
WINDOW w as (PARTITION BY department_id)
●
This allows easy sharing of windows between several window functions
and also avoids redundant windowing steps since more functions can be
evaluated in the same step.
●
Limitation: equivalent windows are not yet merged
42Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
Implicit and explicit windows
●
A window definition can inherit from another window definition in its
specification, adding detail (no override)
SELECT name, department_id,
COUNT(*) OVER w1 AS cnt1,
COUNT(*) over w2 AS cnt2 FROM employee
WINDOW w1 AS (PARTITION BY department_id),
w2 AS (w1 ORDER BY name)
ORDER BY department_id, name;
+---------+---------------+------+------+
| name | department_id | cnt1 | cnt2 |
+---------+---------------+------+------+
| Newt | NULL | 1 | 1 |
| Dag | 10 | 6 | 1 |
| Ed | 10 | 6 | 2 |
| Fred | 10 | 6 | 3 |
| Jon | 10 | 6 | 4 |
| Michael | 10 | 6 | 5 |
| Newt | 10 | 6 | 6 |
| Lebedev | 20 | 2 | 1 |
| Pete | 20 | 2 | 2 |
| Jeff | 30 | 2 | 1 |
| Will | 30 | 2 | 2 |
+---------+---------------+------+------+
43Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
Time-line
●
Work in progress, no due date yet
44Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
Q & A
45Copyright © 2017 Oracle and/or its affiliates. All rights reserved.
SQL window functions for MySQL

Contenu connexe

Tendances

Oracle sql analytic functions
Oracle sql analytic functionsOracle sql analytic functions
Oracle sql analytic functionsmamamowebby
 
Oracle basic queries
Oracle basic queriesOracle basic queries
Oracle basic queriesPRAKHAR JHA
 
Introduction of sql server indexing
Introduction of sql server indexingIntroduction of sql server indexing
Introduction of sql server indexingMahabubur Rahaman
 
Survey of High Performance NoSQL Systems
Survey of High Performance NoSQL SystemsSurvey of High Performance NoSQL Systems
Survey of High Performance NoSQL SystemsScyllaDB
 
MySQL Tutorial For Beginners | Relational Database Management System | MySQL ...
MySQL Tutorial For Beginners | Relational Database Management System | MySQL ...MySQL Tutorial For Beginners | Relational Database Management System | MySQL ...
MySQL Tutorial For Beginners | Relational Database Management System | MySQL ...Edureka!
 
Procedures and triggers in SQL
Procedures and triggers in SQLProcedures and triggers in SQL
Procedures and triggers in SQLVikash Sharma
 
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...Aaron Shilo
 
Advanced MySQL Query Tuning
Advanced MySQL Query TuningAdvanced MySQL Query Tuning
Advanced MySQL Query TuningAlexander Rubin
 

Tendances (20)

Oracle SQL Advanced
Oracle SQL AdvancedOracle SQL Advanced
Oracle SQL Advanced
 
Sql joins
Sql joinsSql joins
Sql joins
 
Sql and Sql commands
Sql and Sql commandsSql and Sql commands
Sql and Sql commands
 
Oracle sql analytic functions
Oracle sql analytic functionsOracle sql analytic functions
Oracle sql analytic functions
 
Oracle basic queries
Oracle basic queriesOracle basic queries
Oracle basic queries
 
Introduction of sql server indexing
Introduction of sql server indexingIntroduction of sql server indexing
Introduction of sql server indexing
 
Sql query patterns, optimized
Sql query patterns, optimizedSql query patterns, optimized
Sql query patterns, optimized
 
Survey of High Performance NoSQL Systems
Survey of High Performance NoSQL SystemsSurvey of High Performance NoSQL Systems
Survey of High Performance NoSQL Systems
 
Sql Basics And Advanced
Sql Basics And AdvancedSql Basics And Advanced
Sql Basics And Advanced
 
Sql commands
Sql commandsSql commands
Sql commands
 
MySQL Tutorial For Beginners | Relational Database Management System | MySQL ...
MySQL Tutorial For Beginners | Relational Database Management System | MySQL ...MySQL Tutorial For Beginners | Relational Database Management System | MySQL ...
MySQL Tutorial For Beginners | Relational Database Management System | MySQL ...
 
Ms sql-server
Ms sql-serverMs sql-server
Ms sql-server
 
Sql
SqlSql
Sql
 
SQL Overview
SQL OverviewSQL Overview
SQL Overview
 
Procedures and triggers in SQL
Procedures and triggers in SQLProcedures and triggers in SQL
Procedures and triggers in SQL
 
MySql slides (ppt)
MySql slides (ppt)MySql slides (ppt)
MySql slides (ppt)
 
Vertica-Database
Vertica-DatabaseVertica-Database
Vertica-Database
 
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
 
Advanced MySQL Query Tuning
Advanced MySQL Query TuningAdvanced MySQL Query Tuning
Advanced MySQL Query Tuning
 
SQL Commands
SQL Commands SQL Commands
SQL Commands
 

En vedette

MySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table Expressions MySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table Expressions oysteing
 
MySQL Server Defaults
MySQL Server DefaultsMySQL Server Defaults
MySQL Server DefaultsMorgan Tocker
 
MySQL 8.0: GIS — Are you ready?
MySQL 8.0: GIS — Are you ready?MySQL 8.0: GIS — Are you ready?
MySQL 8.0: GIS — Are you ready?Norvald Ryeng
 
What you wanted to know about MySQL, but could not find using inernal instrum...
What you wanted to know about MySQL, but could not find using inernal instrum...What you wanted to know about MySQL, but could not find using inernal instrum...
What you wanted to know about MySQL, but could not find using inernal instrum...Sveta Smirnova
 
MySQL 8.0 & Unicode: Why, what & how
MySQL 8.0 & Unicode: Why, what & howMySQL 8.0 & Unicode: Why, what & how
MySQL 8.0 & Unicode: Why, what & howBernt Marius Johnsen
 
Proxysql use case scenarios fosdem17
Proxysql use case scenarios    fosdem17Proxysql use case scenarios    fosdem17
Proxysql use case scenarios fosdem17Alkin Tezuysal
 
MySQL Group Replication
MySQL Group ReplicationMySQL Group Replication
MySQL Group ReplicationKenny Gryp
 
How Booking.com avoids and deals with replication lag
How Booking.com avoids and deals with replication lagHow Booking.com avoids and deals with replication lag
How Booking.com avoids and deals with replication lagJean-François Gagné
 
Using Optimizer Hints to Improve MySQL Query Performance
Using Optimizer Hints to Improve MySQL Query PerformanceUsing Optimizer Hints to Improve MySQL Query Performance
Using Optimizer Hints to Improve MySQL Query Performanceoysteing
 
Jeudis du Libre - MySQL InnoDB Cluster
Jeudis du Libre - MySQL InnoDB ClusterJeudis du Libre - MySQL InnoDB Cluster
Jeudis du Libre - MySQL InnoDB ClusterFrederic Descamps
 
Jeudis du Libre - MySQL comme Document Store
Jeudis du Libre - MySQL comme Document StoreJeudis du Libre - MySQL comme Document Store
Jeudis du Libre - MySQL comme Document StoreFrederic Descamps
 
MySQLerの7つ道具 plus
MySQLerの7つ道具 plusMySQLerの7つ道具 plus
MySQLerの7つ道具 plusyoku0825
 
MySQL 5.7の次のMySQL 8.0はどんなものになるだろう
MySQL 5.7の次のMySQL 8.0はどんなものになるだろうMySQL 5.7の次のMySQL 8.0はどんなものになるだろう
MySQL 5.7の次のMySQL 8.0はどんなものになるだろうyoku0825
 
MySQL Cloud Service Deep Dive
MySQL Cloud Service Deep DiveMySQL Cloud Service Deep Dive
MySQL Cloud Service Deep DiveMorgan Tocker
 
MySQLアンチパターン
MySQLアンチパターンMySQLアンチパターン
MySQLアンチパターンyoku0825
 
Polyglot Database - Linuxcon North America 2016
Polyglot Database - Linuxcon North America 2016Polyglot Database - Linuxcon North America 2016
Polyglot Database - Linuxcon North America 2016Dave Stokes
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performanceoysteing
 
How to analyze and tune sql queries for better performance percona15
How to analyze and tune sql queries for better performance percona15How to analyze and tune sql queries for better performance percona15
How to analyze and tune sql queries for better performance percona15oysteing
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performanceoysteing
 
MySQL EXPLAIN Explained-Norvald H. Ryeng
MySQL EXPLAIN Explained-Norvald H. RyengMySQL EXPLAIN Explained-Norvald H. Ryeng
MySQL EXPLAIN Explained-Norvald H. Ryeng郁萍 王
 

En vedette (20)

MySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table Expressions MySQL 8.0: Common Table Expressions
MySQL 8.0: Common Table Expressions
 
MySQL Server Defaults
MySQL Server DefaultsMySQL Server Defaults
MySQL Server Defaults
 
MySQL 8.0: GIS — Are you ready?
MySQL 8.0: GIS — Are you ready?MySQL 8.0: GIS — Are you ready?
MySQL 8.0: GIS — Are you ready?
 
What you wanted to know about MySQL, but could not find using inernal instrum...
What you wanted to know about MySQL, but could not find using inernal instrum...What you wanted to know about MySQL, but could not find using inernal instrum...
What you wanted to know about MySQL, but could not find using inernal instrum...
 
MySQL 8.0 & Unicode: Why, what & how
MySQL 8.0 & Unicode: Why, what & howMySQL 8.0 & Unicode: Why, what & how
MySQL 8.0 & Unicode: Why, what & how
 
Proxysql use case scenarios fosdem17
Proxysql use case scenarios    fosdem17Proxysql use case scenarios    fosdem17
Proxysql use case scenarios fosdem17
 
MySQL Group Replication
MySQL Group ReplicationMySQL Group Replication
MySQL Group Replication
 
How Booking.com avoids and deals with replication lag
How Booking.com avoids and deals with replication lagHow Booking.com avoids and deals with replication lag
How Booking.com avoids and deals with replication lag
 
Using Optimizer Hints to Improve MySQL Query Performance
Using Optimizer Hints to Improve MySQL Query PerformanceUsing Optimizer Hints to Improve MySQL Query Performance
Using Optimizer Hints to Improve MySQL Query Performance
 
Jeudis du Libre - MySQL InnoDB Cluster
Jeudis du Libre - MySQL InnoDB ClusterJeudis du Libre - MySQL InnoDB Cluster
Jeudis du Libre - MySQL InnoDB Cluster
 
Jeudis du Libre - MySQL comme Document Store
Jeudis du Libre - MySQL comme Document StoreJeudis du Libre - MySQL comme Document Store
Jeudis du Libre - MySQL comme Document Store
 
MySQLerの7つ道具 plus
MySQLerの7つ道具 plusMySQLerの7つ道具 plus
MySQLerの7つ道具 plus
 
MySQL 5.7の次のMySQL 8.0はどんなものになるだろう
MySQL 5.7の次のMySQL 8.0はどんなものになるだろうMySQL 5.7の次のMySQL 8.0はどんなものになるだろう
MySQL 5.7の次のMySQL 8.0はどんなものになるだろう
 
MySQL Cloud Service Deep Dive
MySQL Cloud Service Deep DiveMySQL Cloud Service Deep Dive
MySQL Cloud Service Deep Dive
 
MySQLアンチパターン
MySQLアンチパターンMySQLアンチパターン
MySQLアンチパターン
 
Polyglot Database - Linuxcon North America 2016
Polyglot Database - Linuxcon North America 2016Polyglot Database - Linuxcon North America 2016
Polyglot Database - Linuxcon North America 2016
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performance
 
How to analyze and tune sql queries for better performance percona15
How to analyze and tune sql queries for better performance percona15How to analyze and tune sql queries for better performance percona15
How to analyze and tune sql queries for better performance percona15
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performance
 
MySQL EXPLAIN Explained-Norvald H. Ryeng
MySQL EXPLAIN Explained-Norvald H. RyengMySQL EXPLAIN Explained-Norvald H. Ryeng
MySQL EXPLAIN Explained-Norvald H. Ryeng
 

Similaire à SQL window functions for MySQL

20180420 hk-the powerofmysql8
20180420 hk-the powerofmysql820180420 hk-the powerofmysql8
20180420 hk-the powerofmysql8Ivan Ma
 
Dublin 4x3-final-slideshare
Dublin 4x3-final-slideshareDublin 4x3-final-slideshare
Dublin 4x3-final-slideshareDag H. Wanvik
 
Windowing Functions - Little Rock Tech Fest 2019
Windowing Functions - Little Rock Tech Fest 2019Windowing Functions - Little Rock Tech Fest 2019
Windowing Functions - Little Rock Tech Fest 2019Dave Stokes
 
Windowing Functions - Little Rock Tech fest 2019
Windowing Functions - Little Rock Tech fest 2019Windowing Functions - Little Rock Tech fest 2019
Windowing Functions - Little Rock Tech fest 2019Dave Stokes
 
MySQL 8.0 Preview: What Is Coming?
MySQL 8.0 Preview: What Is Coming?MySQL 8.0 Preview: What Is Coming?
MySQL 8.0 Preview: What Is Coming?Gabriela Ferrara
 
MySQL 8.0 Released Update
MySQL 8.0 Released UpdateMySQL 8.0 Released Update
MySQL 8.0 Released UpdateKeith Hollman
 
Developers' mDay 2017. - Bogdan Kecman Oracle
Developers' mDay 2017. - Bogdan Kecman OracleDevelopers' mDay 2017. - Bogdan Kecman Oracle
Developers' mDay 2017. - Bogdan Kecman OraclemCloud
 
Developers’ mDay u Banjoj Luci - Bogdan Kecman, Oracle – MySQL Server 8.0
Developers’ mDay u Banjoj Luci - Bogdan Kecman, Oracle – MySQL Server 8.0Developers’ mDay u Banjoj Luci - Bogdan Kecman, Oracle – MySQL Server 8.0
Developers’ mDay u Banjoj Luci - Bogdan Kecman, Oracle – MySQL Server 8.0mCloud
 
MySQL Optimizer: What’s New in 8.0
MySQL Optimizer: What’s New in 8.0MySQL Optimizer: What’s New in 8.0
MySQL Optimizer: What’s New in 8.0oysteing
 
What's New MySQL 8.0?
What's New MySQL 8.0?What's New MySQL 8.0?
What's New MySQL 8.0?OracleMySQL
 
The Current State of Table API in 2022
The Current State of Table API in 2022The Current State of Table API in 2022
The Current State of Table API in 2022Flink Forward
 
Automating Networks by Converting into API/Webs
Automating Networks by Converting into API/WebsAutomating Networks by Converting into API/Webs
Automating Networks by Converting into API/WebsAPNIC
 
MySQL 8.0 Optimizer Guide
MySQL 8.0 Optimizer GuideMySQL 8.0 Optimizer Guide
MySQL 8.0 Optimizer GuideMorgan Tocker
 
Data Love Conference - Window Functions for Database Analytics
Data Love Conference - Window Functions for Database AnalyticsData Love Conference - Window Functions for Database Analytics
Data Love Conference - Window Functions for Database AnalyticsDave Stokes
 
MySQL Goes to 8! FOSDEM 2020 Database Track, January 2nd, 2020
MySQL Goes to 8!  FOSDEM 2020 Database Track, January 2nd, 2020MySQL Goes to 8!  FOSDEM 2020 Database Track, January 2nd, 2020
MySQL Goes to 8! FOSDEM 2020 Database Track, January 2nd, 2020Geir Høydalsvik
 
Machine learning for developers
Machine learning for developersMachine learning for developers
Machine learning for developersSource Ministry
 
Automating Networks by using API
Automating Networks by using APIAutomating Networks by using API
Automating Networks by using API一清 井上
 
Oracle Database Advanced Querying
Oracle Database Advanced QueryingOracle Database Advanced Querying
Oracle Database Advanced QueryingZohar Elkayam
 
Modern query optimisation features in MySQL 8.
Modern query optimisation features in MySQL 8.Modern query optimisation features in MySQL 8.
Modern query optimisation features in MySQL 8.Mydbops
 

Similaire à SQL window functions for MySQL (20)

20180420 hk-the powerofmysql8
20180420 hk-the powerofmysql820180420 hk-the powerofmysql8
20180420 hk-the powerofmysql8
 
Dublin 4x3-final-slideshare
Dublin 4x3-final-slideshareDublin 4x3-final-slideshare
Dublin 4x3-final-slideshare
 
Windowing Functions - Little Rock Tech Fest 2019
Windowing Functions - Little Rock Tech Fest 2019Windowing Functions - Little Rock Tech Fest 2019
Windowing Functions - Little Rock Tech Fest 2019
 
Windowing Functions - Little Rock Tech fest 2019
Windowing Functions - Little Rock Tech fest 2019Windowing Functions - Little Rock Tech fest 2019
Windowing Functions - Little Rock Tech fest 2019
 
MySQL 8.0 Preview: What Is Coming?
MySQL 8.0 Preview: What Is Coming?MySQL 8.0 Preview: What Is Coming?
MySQL 8.0 Preview: What Is Coming?
 
MySQL 8.0 Released Update
MySQL 8.0 Released UpdateMySQL 8.0 Released Update
MySQL 8.0 Released Update
 
Developers' mDay 2017. - Bogdan Kecman Oracle
Developers' mDay 2017. - Bogdan Kecman OracleDevelopers' mDay 2017. - Bogdan Kecman Oracle
Developers' mDay 2017. - Bogdan Kecman Oracle
 
Developers’ mDay u Banjoj Luci - Bogdan Kecman, Oracle – MySQL Server 8.0
Developers’ mDay u Banjoj Luci - Bogdan Kecman, Oracle – MySQL Server 8.0Developers’ mDay u Banjoj Luci - Bogdan Kecman, Oracle – MySQL Server 8.0
Developers’ mDay u Banjoj Luci - Bogdan Kecman, Oracle – MySQL Server 8.0
 
MySQL Optimizer: What’s New in 8.0
MySQL Optimizer: What’s New in 8.0MySQL Optimizer: What’s New in 8.0
MySQL Optimizer: What’s New in 8.0
 
What's New MySQL 8.0?
What's New MySQL 8.0?What's New MySQL 8.0?
What's New MySQL 8.0?
 
The Current State of Table API in 2022
The Current State of Table API in 2022The Current State of Table API in 2022
The Current State of Table API in 2022
 
Php forum2015 tomas_final
Php forum2015 tomas_finalPhp forum2015 tomas_final
Php forum2015 tomas_final
 
Automating Networks by Converting into API/Webs
Automating Networks by Converting into API/WebsAutomating Networks by Converting into API/Webs
Automating Networks by Converting into API/Webs
 
MySQL 8.0 Optimizer Guide
MySQL 8.0 Optimizer GuideMySQL 8.0 Optimizer Guide
MySQL 8.0 Optimizer Guide
 
Data Love Conference - Window Functions for Database Analytics
Data Love Conference - Window Functions for Database AnalyticsData Love Conference - Window Functions for Database Analytics
Data Love Conference - Window Functions for Database Analytics
 
MySQL Goes to 8! FOSDEM 2020 Database Track, January 2nd, 2020
MySQL Goes to 8!  FOSDEM 2020 Database Track, January 2nd, 2020MySQL Goes to 8!  FOSDEM 2020 Database Track, January 2nd, 2020
MySQL Goes to 8! FOSDEM 2020 Database Track, January 2nd, 2020
 
Machine learning for developers
Machine learning for developersMachine learning for developers
Machine learning for developers
 
Automating Networks by using API
Automating Networks by using APIAutomating Networks by using API
Automating Networks by using API
 
Oracle Database Advanced Querying
Oracle Database Advanced QueryingOracle Database Advanced Querying
Oracle Database Advanced Querying
 
Modern query optimisation features in MySQL 8.
Modern query optimisation features in MySQL 8.Modern query optimisation features in MySQL 8.
Modern query optimisation features in MySQL 8.
 

Dernier

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
Kubernetes go-live checklist for your microservices.pptx
Kubernetes go-live checklist for your microservices.pptxKubernetes go-live checklist for your microservices.pptx
Kubernetes go-live checklist for your microservices.pptxPrakarsh -
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...OnePlan Solutions
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfBrain Inventory
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
Mastering Kubernetes - Basics and Advanced Concepts using Example Project
Mastering Kubernetes - Basics and Advanced Concepts using Example ProjectMastering Kubernetes - Basics and Advanced Concepts using Example Project
Mastering Kubernetes - Basics and Advanced Concepts using Example Projectwajrcs
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdfMeon Technology
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyRaymond Okyere-Forson
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9Jürgen Gutsch
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsJaydeep Chhasatia
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesSoftwareMill
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 

Dernier (20)

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
Kubernetes go-live checklist for your microservices.pptx
Kubernetes go-live checklist for your microservices.pptxKubernetes go-live checklist for your microservices.pptx
Kubernetes go-live checklist for your microservices.pptx
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdf
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
Mastering Kubernetes - Basics and Advanced Concepts using Example Project
Mastering Kubernetes - Basics and Advanced Concepts using Example ProjectMastering Kubernetes - Basics and Advanced Concepts using Example Project
Mastering Kubernetes - Basics and Advanced Concepts using Example Project
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdf
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human Beauty
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9
 
Sustainable Web Design - Claire Thornewill
Sustainable Web Design - Claire ThornewillSustainable Web Design - Claire Thornewill
Sustainable Web Design - Claire Thornewill
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retries
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 

SQL window functions for MySQL

  • 1. Copyright © 2017 Oracle and/or its affiliates. All rights reserved. SQL window functions for MySQL Dag H. Wanvik Senior database engineer, Oracle
  • 2. 2Copyright © 2017 Oracle and/or its affiliates. All rights reserved. Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
  • 3. 3Copyright © 2017 Oracle and/or its affiliates. All rights reserved. Agenda Quick intro to window functions Types of window functions The window specification Evaluation and optimizations More on non-aggregate wfs Implicit and explicit windows Q & A 1 2 3 4 5 6 7
  • 4. 4Copyright © 2017 Oracle and/or its affiliates. All rights reserved. Window functions: what are they? ● A window function performs a calculation across a set of rows that are related to the current row, similar to what can be done with an aggregate function. ● But unlike traditional aggregate functions, a window function does not cause rows to become grouped into a single output row. ● So, similar to normal function, but can access values of other rows “in the vicinity” of the current row
  • 5. 5Copyright © 2017 Oracle and/or its affiliates. All rights reserved. SELECT name, department_id, salary, SUM(salary) OVER (PARTITION BY department_id) AS department_total FROM employee ORDER BY department_id, name; Window function example, no frame The OVER keyword signals a window function
  • 6. 6Copyright © 2017 Oracle and/or its affiliates. All rights reserved. Partitions: 10, 20, 30 +---------+---------------+--------+------------------+ | name | department_id | salary | department_total | +---------+---------------+--------+------------------+ | Newt | NULL | 75000 | 75000 | | Dag | 10 | NULL | 370000 | | Ed | 10 | 100000 | 370000 | | Fred | 10 | 60000 | 370000 | | Jon | 10 | 60000 | 370000 | | Michael | 10 | 70000 | 370000 | | Newt | 10 | 80000 | 370000 | | Lebedev | 20 | 65000 | 130000 | | Pete | 20 | 65000 | 130000 | | Jeff | 30 | 300000 | 370000 | | Will | 30 | 70000 | 370000 | +---------+---------------+--------+------------------+ Partition == disjoint set of rows in result set Here: all rows in partition are peers
  • 7. 7Copyright © 2017 Oracle and/or its affiliates. All rights reserved. SELECT name, department_id, salary, SUM(salary) AS department_total FROM employee GROUP BY department_id ORDER BY department_id, name; ERROR 1055 (42000): Expression #1 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'mysql.employee.name' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by With GROUP BY
  • 8. 8Copyright © 2017 Oracle and/or its affiliates. All rights reserved. SELECT /* name, */ department_id, /* salary,*/ SUM(salary) AS department_total FROM employee GROUP BY department_id ORDER BY department_id /*, name */; +---------------+------------------+ | department_id | department_total | +---------------+------------------+ | NULL | 75000 | | 10 | 370000 | | 20 | 130000 | | 30 | 370000 | +---------------+------------------+ With GROUP BY
  • 9. 9Copyright © 2017 Oracle and/or its affiliates. All rights reserved. SELECT name, department_id, salary, SUM (salary) OVER (PARTITION BY department_id ORDER BY name ROWS 2 PRECEDING) total FROM employee ORDER BY department_id, name; Window function example, frame
  • 10. 10Copyright © 2017 Oracle and/or its affiliates. All rights reserved. Partitions: 10, 20, 30 ORDER BY name within each partition +---------+---------------+--------+--------+ | name | department_id | salary | total | +---------+---------------+--------+--------+ | Newt | NULL | 75000 | 75000 | | Dag | 10 | NULL | NULL | | Ed | 10 | 100000 | 100000 | | Fred | 10 | 60000 | 160000 | | Jon | 10 | 60000 | 220000 | | Michael | 10 | 70000 | 190000 | | Newt | 10 | 80000 | 210000 | | Lebedev | 20 | 65000 | 65000 | | Pete | 20 | 65000 | 130000 | | Jeff | 30 | 300000 | 300000 | | Will | 30 | 70000 | 370000 | +---------+---------------+--------+--------+ moving window frame: SUM (salary) ... ROWS 2 PRECEDING a frame is a subset of a partition
  • 11. 11Copyright © 2017 Oracle and/or its affiliates. All rights reserved. SELECT name, department_id, salary, AVG(salary) OVER w AS `avg`, salary - AVG(salary) OVER w AS diff FROM employee WINDOW w as (PARTITION BY department_id) ORDER BY diff DESC; Window function example i.e. find the employees with the largest difference between their wage and that of the department average Note: explicit window definition of “w”
  • 12. 12Copyright © 2017 Oracle and/or its affiliates. All rights reserved. Partitions: 10, 20, 30 +---------+---------------+--------+-----------+------------+ | name | department_id | salary | avg | diff | +---------+---------------+--------+-----------+------------+ | Jeff | 30 | 300000 | 185000.00 | 115000.00 | | Ed | 10 | 100000 | 74000.00 | 26000.00 | | Newt | 10 | 80000 | 74000.00 | 6000.00 | | Newt | NULL | 75000 | 75000.00 | 0.00 | | Pete | 20 | 65000 | 65000.00 | 0.00 | | Lebedev | 20 | 65000 | 65000.00 | 0.00 | | Michael | 10 | 70000 | 74000.00 | -4000.00 | | Jon | 10 | 60000 | 74000.00 | -14000.00 | | Fred | 10 | 60000 | 74000.00 | -14000.00 | | Will | 30 | 70000 | 185000.00 | -115000.00 | | Dag | 10 | NULL | 74000.00 | NULL | +---------+---------------+--------+-----------+------------+
  • 13. 13Copyright © 2017 Oracle and/or its affiliates. All rights reserved. Types of window functions ● Aggregates ● Ranking ● Analytical COUNT, SUM, AVG + more to come RANK, DENSE_RANK, PERCENT_RANK, CUME_DIST, ROW_NUMBER NTILE, LEAD, LAG, NTH, FIRST_VALUE, LAST_VALUE Blue ones use frames, all obey partitions
  • 14. 14Copyright © 2017 Oracle and/or its affiliates. All rights reserved. Anatomy of a window specification window specification ::= [ existing window name ] [PARTITION BY expr-1, ... ] [ORDER BY expr-1, ... [DESC] ] [ frame clause ] –
  • 15. 15Copyright © 2017 Oracle and/or its affiliates. All rights reserved. frame clause bound partition CURRENT ROW UNBOUNDED PRECEDING UNBOUNDED FOLLOWING n PRECEDING m PRECEDING
  • 16. 16Copyright © 2017 Oracle and/or its affiliates. All rights reserved. frame clause frame clause ::= { ROWS | RANGE } { start | between } start ::= { CURRENT ROW | UNBOUNDED PRECEDING | n PRECEDING} between ::= BETWEEN bound-1 AND bound-2 bound ::= start | UNBOUNDED FOLLOWING | n FOLLOWING ● “start” form implies upper is CURRENT ROW (or its peer iff RANGE) ● An empty OVER () specification, says that all rows in the partition are peers and included in frame. ● An ORDER BY without frame implies frame: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ● Limitation: frame exclusion not supported
  • 17. 17Copyright © 2017 Oracle and/or its affiliates. All rights reserved. frame clause ● ROWS or RANGE (GROUPS unit not supported) ● ROWS: bounds are physical row offsets ● RANGE: logical offset based on current row's value Ex: RANGE BETWEEN INTERVAL 1 WEEK PRECEDING AND CURRENT ROW ● RANGE requires ORDER BY on one numeric expression ● Limitation: bounds must be static
  • 18. 18Copyright © 2017 Oracle and/or its affiliates. All rights reserved. SELECT dat, amount, SUM(amount) OVER w FROM payments WINDOW w AS (ORDER BY dat RANGE BETWEEN INTERVAL 1 WEEK PRECEDING AND CURRENT ROW) ORDER BY dat; RANGE frame example i.e. find the sum of payments within the last 7 days
  • 19. 19Copyright © 2017 Oracle and/or its affiliates. All rights reserved. RANGE frame example +------------+--------+--------+ | dat | amount | sum | +------------+--------+--------+ | 2017-01-01 | 100.50 | 300.50 | | 2017-01-01 | 200.00 | 300.50 | | 2017-01-02 | 200.00 | 500.50 | | 2017-01-03 | 200.00 | 700.50 | | 2017-01-05 | 200.00 | 900.50 | | 2017-01-10 | 200.00 | 700.00 | | 2017-01-10 | 100.00 | 700.00 | | 2017-01-11 | 200.00 | 700.00 | +------------+--------+--------+ SELECT dat, amount, SUM(amount) OVER w AS `sum` FROM payments WINDOW w AS (ORDER BY dat RANGE BETWEEN INTERVAL 1 WEEK PRECEDING AND CURRENT ROW) ORDER BY dat; Current row's date is the 10th , so first row in range is the 3rd . Frame cardinality is 4 due to peer in next row. For Jan 5, the frame cardinality is 5, and sum is 900.50.
  • 20. 20Copyright © 2017 Oracle and/or its affiliates. All rights reserved. When are they evaluated? ● after GROUP BY/ HAVING ● before final ORDER BY, DISTINCT, LIMIT ● you can have several window functions and several different windows ● To filter on wf's value, use a subquery, e.g. SELECT * FROM (SELECT SUM(salary) OVER (PARTITION BY department_id) `sum` FROM employee) AS s WHERE `sum` < 100000; +-------+ | sum | +-------+ | 75000 | +-------+
  • 21. 21Copyright © 2017 Oracle and/or its affiliates. All rights reserved. Logical flow JOIN GROUP BY WINDOW 1 WINDOW n ORDER BY/ DISTINCT/ LIMIT Sort for PARTITION BY and ORDER BY ● Tmp file between each windowing step (in-mem if result set can fit †) ● Streamable wfs vs buffered ● Depends on wf and frame ● Buffered: re-read rows ● O(rows * frame size) optimize ● Move frame for SUM 1 row: invert by subtraction, add new row. † cf. variables tmp_table_size, max_heap_table_size
  • 22. 22Copyright © 2017 Oracle and/or its affiliates. All rights reserved. Streamable evaluation SELECT name, department_id, salary, SUM(salary) OVER (PARTITION BY department_id ORDER BY name ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS `sum` FROM employee; +---------+---------------+--------+--------+ | name | department_id | salary | sum | +---------+---------------+--------+--------+ | Newt | NULL | 75000 | 75000 | | Dag | 10 | NULL | NULL | | Ed | 10 | 100000 | 100000 | Just accumulate as we see rows | Fred | 10 | 60000 | 160000 | | Jon | 10 | 60000 | 220000 | | Michael | 10 | 70000 | 290000 | | Newt | 10 | 80000 | 370000 | | Lebedev | 20 | 65000 | 65000 | | Pete | 20 | 65000 | 130000 | | Jeff | 30 | 300000 | 300000 | | Will | 30 | 70000 | 370000 | +---------+---------------+--------+--------+
  • 23. 23Copyright © 2017 Oracle and/or its affiliates. All rights reserved. Logical flow JOIN GROUP BY WINDOW 1 WINDOW n ORDER BY/ DISTINCT/ LIMIT Row addressable buffer in-mem: overflows to disk Permits re-reading rows when frame moves
  • 24. 24Copyright © 2017 Oracle and/or its affiliates. All rights reserved. Non-streamable evaluation SELECT name, department_id, salary, SUM(salary) OVER (PARTITION BY department_id ORDER BY name ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) AS `sum` FROM employee; +---------+---------------+--------+--------+ | name | department_id | salary | sum | +---------+---------------+--------+--------+ | Newt | NULL | 75000 | 75000 | | Dag | 10 | NULL | NULL | | Ed | 10 | 100000 | 100000 | | Fred | 10 | 60000 | 160000 | | Jon | 10 | 60000 | 220000 | | Michael | 10 | 70000 | 190000 | | Newt | 10 | 80000 | 210000 | | Lebedev | 20 | 65000 | 65000 | | Pete | 20 | 65000 | 130000 | | Jeff | 30 | 300000 | 300000 | | Will | 30 | 70000 | 370000 | +---------+---------------+--------+--------+ When eval'ing Michael, subtract Ed's contribution, add Michael or just evaluate entire frame over again (non-optimized). In both cases we need re-visit rows.
  • 25. 25Copyright © 2017 Oracle and/or its affiliates. All rights reserved. Optimizations ● Accumulate frames by inversion (discussed; done) ¹ ● Avoid materializing each step in temporary file if possible (done) ● Sort only once for windows with the same PARTITION/ORDER BY requirements (done) ● Sort only once for windows with subset relation on above (not yet) ● Eliminate sorts by using indexes (not yet) ¹not done by default for floats due to possible under/-overflow errors, but can be enabled using a variable. Enabling it is crucial for performance for large frames due to the combinatorial hardness.
  • 26. 26Copyright © 2017 Oracle and/or its affiliates. All rights reserved. Explain, last query EXPLAIN FORMAT=JSON SELECT name, department_id, salary, SUM(salary) OVER (PARTITION BY department_id ORDER BY name ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) AS `sum` FROM employee; ... "windows": [ {"name": "<unnamed window>", "evalated as #": 1, "using sorting": true, "using temporary file": false, "uses frame buffer": true, "optimized frame evaluation": true, "functions used": [ "sum" ]} ], "buffer_result": { :
  • 27. 27Copyright © 2017 Oracle and/or its affiliates. All rights reserved. More on non-aggregate wfs ● RANK, DENSE_RANK, PERCENT_RANK, CUME_DIST, ROW_NUMBER ● LEAD, LAG, FIRST_VALUE, LAST_VALUE, NTH_VALUE
  • 28. 28Copyright © 2017 Oracle and/or its affiliates. All rights reserved. RANK SELECT name, department_id AS dept, salary, RANK() OVER w AS `rank` FROM employee WINDOW w AS (PARTITION BY department_id ORDER BY salary DESC); +---------+------+--------+------+ | name | dept | salary | rank | +---------+------+--------+------+ | Newt | NULL | 75000 | 1 | | Ed | 10 | 100000 | 1 | | Newt | 10 | 80000 | 2 | | Fred | 10 | 70000 | 3 | | Michael | 10 | 70000 | 3 | | Jon | 10 | 60000 | 5 | | Dag | 10 | NULL | 6 | | Pete | 20 | 65000 | 1 | | Lebedev | 20 | 65000 | 1 | | Jeff | 30 | 300000 | 1 | | Will | 30 | 70000 | 2 | +---------+------+--------+------+
  • 29. 29Copyright © 2017 Oracle and/or its affiliates. All rights reserved. DENSE_RANK SELECT name, department_id AS dept, salary, RANK() OVER w AS `rank`, DENSE_RANK() OVER w AS dense FROM employee WINDOW w AS (PARTITION BY department_id ORDER BY salary DESC); +---------+------+--------+------+-------+ | name | dept | salary | rank | dense | +---------+------+--------+------+-------+ | Newt | NULL | 75000 | 1 | 1 | | Ed | 10 | 100000 | 1 | 1 | | Newt | 10 | 80000 | 2 | 2 | | Fred | 10 | 70000 | 3 | 3 | | Michael | 10 | 70000 | 3 | 3 | | Jon | 10 | 60000 | 5 | 4 | | Dag | 10 | NULL | 6 | 5 | | Pete | 20 | 65000 | 1 | 1 | | Lebedev | 20 | 65000 | 1 | 1 | | Jeff | 30 | 300000 | 1 | 1 | | Will | 30 | 70000 | 2 | 2 | +---------+------+--------+------+-------+ DENSE_RANK doesn't skip
  • 30. 30Copyright © 2017 Oracle and/or its affiliates. All rights reserved. ROW_NUMBER SELECT name, department_id AS dept, salary, RANK() OVER w AS `rank`, DENSE_RANK() OVER w AS dense, ROW_NUMBER() OVER w AS `#` FROM employee WINDOW w AS (PARTITION BY department_id ORDER BY salary DESC); +---------+------+--------+------+-------+---+ | name | dept | salary | rank | dense | # | +---------+------+--------+------+-------+---+ | Newt | NULL | 75000 | 1 | 1 | 1 | | Ed | 10 | 100000 | 1 | 1 | 1 | | Newt | 10 | 80000 | 2 | 2 | 2 | | Fred | 10 | 70000 | 3 | 3 | 3 | | Michael | 10 | 70000 | 3 | 3 | 4 | | Jon | 10 | 60000 | 5 | 4 | 5 | | Dag | 10 | NULL | 6 | 5 | 6 | | Pete | 20 | 65000 | 1 | 1 | 1 | | Lebedev | 20 | 65000 | 1 | 1 | 2 | | Jeff | 30 | 300000 | 1 | 1 | 1 | | Will | 30 | 70000 | 2 | 2 | 2 | +---------+------+--------+------+-------+---+
  • 31. 31Copyright © 2017 Oracle and/or its affiliates. All rights reserved. CUME_DIST SELECT name, department_id AS dept, salary, RANK() OVER w AS `rank`, DENSE_RANK() OVER w AS dense, ROW_NUMBER() OVER w AS `#`, CUME_DIST() OVER w AS cume FROM employee WINDOW w AS (PARTITION BY department_id ORDER BY salary DESC); +---------+------+--------+------+-------+---+---------------------+ | name | dept | salary | rank | dense | # | cume | +---------+------+--------+------+-------+---+---------------------+ | Newt | NULL | 75000 | 1 | 2 | 1 | 1 | | Ed | 10 | 100000 | 1 | 2 | 1 | 0.16666666666666666 | | Newt | 10 | 80000 | 2 | 3 | 2 | 0.3333333333333333 | | Fred | 10 | 70000 | 3 | 4 | 3 | 0.6666666666666666 | | Michael | 10 | 70000 | 3 | 4 | 4 | 0.6666666666666666 | | Jon | 10 | 60000 | 5 | 5 | 5 | 0.8333333333333334 | | Dag | 10 | NULL | 6 | 6 | 6 | 1 | | Pete | 20 | 65000 | 1 | 2 | 1 | 1 | | Lebedev | 20 | 65000 | 1 | 2 | 2 | 1 | | Jeff | 30 | 300000 | 1 | 2 | 1 | 0.5 | | Will | 30 | 70000 | 2 | 3 | 2 | 1 | +---------+------+--------+------+-------+---+---------------------+ Cumulative distribution “For a row R, if we assume ascending ordering, CUME_DIST of R is the number of rows with values <= the value of R, divided by the number of rows evaluated in the partition. “
  • 32. 32Copyright © 2017 Oracle and/or its affiliates. All rights reserved. PERCENT_RANK SELECT name, department_id AS dept, salary, RANK() OVER w AS `rank`, DENSE_RANK() OVER w AS dense, ROW_NUMBER() OVER w AS `#`, CUME_DIST() OVER w AS cume, PERCENT_RANK() OVER w AS p_r FROM employee WINDOW w AS (PARTITION BY department_id ORDER BY salary DESC); +---------+------+--------+------+-------+---+---------------------+-----+ | name | dept | salary | rank | dense | # | cume | p_r | +---------+------+--------+------+-------+---+---------------------+-----+ | Newt | NULL | 75000 | 1 | 2 | 1 | 1 | 0 | | Ed | 10 | 100000 | 1 | 2 | 1 | 0.16666666666666666 | 0 | | Newt | 10 | 80000 | 2 | 3 | 2 | 0.3333333333333333 | 0.2 | | Fred | 10 | 70000 | 3 | 4 | 3 | 0.6666666666666666 | 0.4 | | Michael | 10 | 70000 | 3 | 4 | 4 | 0.6666666666666666 | 0.4 | | Jon | 10 | 60000 | 5 | 5 | 5 | 0.8333333333333334 | 0.8 | | Dag | 10 | NULL | 6 | 6 | 6 | 1 | 1 | | Pete | 20 | 65000 | 1 | 2 | 1 | 1 | 0 | | Lebedev | 20 | 65000 | 1 | 2 | 2 | 1 | 0 | | Jeff | 30 | 300000 | 1 | 2 | 1 | 0.5 | 0 | | Will | 30 | 70000 | 2 | 3 | 2 | 1 | 1 | +---------+------+--------+------+-------+---+---------------------+-----+ (rank - 1) / (total rows - 1)
  • 33. 33Copyright © 2017 Oracle and/or its affiliates. All rights reserved. NTILE SELECT name, department_id AS dept, salary, ... PERCENT_RANK() OVER w AS p_r, NTILE(3) OVER w AS `ntile` FROM employee WINDOW w AS (PARTITION BY department_id ORDER BY salary DESC); +---------+------+--------+------+-------+---+---------------------+-----+-------+ | name | dept | salary | rank | dense | # | cume | p_r | ntile | +---------+------+--------+------+-------+---+---------------------+-----+-------+ | Newt | NULL | 75000 | 1 | 2 | 1 | 1 | 0 | 1 | | Ed | 10 | 100000 | 1 | 2 | 1 | 0.16666666666666666 | 0 | 1 | | Newt | 10 | 80000 | 2 | 3 | 2 | 0.3333333333333333 | 0.2 | 1 | | Fred | 10 | 70000 | 3 | 4 | 3 | 0.6666666666666666 | 0.4 | 2 | | Michael | 10 | 70000 | 3 | 4 | 4 | 0.6666666666666666 | 0.4 | 2 | | Jon | 10 | 60000 | 5 | 5 | 5 | 0.8333333333333334 | 0.8 | 3 | | Dag | 10 | NULL | 6 | 6 | 6 | 1 | 1 | 3 | | Pete | 20 | 65000 | 1 | 2 | 1 | 1 | 0 | 1 | | Lebedev | 20 | 65000 | 1 | 2 | 2 | 1 | 0 | 2 | | Jeff | 30 | 300000 | 1 | 2 | 1 | 0.5 | 0 | 1 | | Will | 30 | 70000 | 2 | 3 | 2 | 1 | 1 | 2 | +---------+------+--------+------+-------+---+---------------------+-----+-------+ Divides an ordered partition into a specified number of groups aka buckets and assigns a bucket number to each row in the partition.
  • 34. 34Copyright © 2017 Oracle and/or its affiliates. All rights reserved. LEAD, LAG Returns value evaluated at the row that is offset rows after/before the current row within the partition; if there is no such row, instead return default (which must be of the same type as value). Both offset and default are evaluated with respect to the current row. If omitted, offset defaults to 1 and default to null lead or lag function ::= { LEAD | LAG } ( expr [ , offset [ , default expression>] ] ) [ RESPECT NULLS ] Note: “IGNORE NULLS” not supported, RESPECT NULLS is default but can be specified.
  • 35. 35Copyright © 2017 Oracle and/or its affiliates. All rights reserved. +---------+------+--------+-------+ | name | dept | salary | lead | +---------+------+--------+-------+ | Newt | NULL | 75000 | NULL | | Ed | 10 | 100000 | 80000 | | Newt | 10 | 80000 | 70000 | | Fred | 10 | 70000 | 70000 | | Michael | 10 | 70000 | 60000 | | Jon | 10 | 60000 | NULL | | Dag | 10 | NULL | NULL | | Pete | 20 | 65000 | 65000 | | Lebedev | 20 | 65000 | NULL | | Jeff | 30 | 300000 | 70000 | | Will | 30 | 70000 | NULL | +---------+------+--------+-------+ LEAD SELECT name, department_id AS dept, salary, LEAD(salary, 1) OVER w AS `lead` FROM employee WINDOW w AS (PARTITION BY department_id ORDER BY salary DESC);
  • 36. 36Copyright © 2017 Oracle and/or its affiliates. All rights reserved. +---------+------+--------+--------+-------+ | name | dept | salary | lag | lead | +---------+------+--------+--------+-------+ | Newt | NULL | 75000 | NULL | NULL | | Ed | 10 | 100000 | NULL | 80000 | | Newt | 10 | 80000 | 100000 | 70000 | | Fred | 10 | 70000 | 80000 | 70000 | | Michael | 10 | 70000 | 70000 | 60000 | | Jon | 10 | 60000 | 70000 | NULL | | Dag | 10 | NULL | 60000 | NULL | | Pete | 20 | 65000 | NULL | 65000 | | Lebedev | 20 | 65000 | 65000 | NULL | | Jeff | 30 | 300000 | NULL | 70000 | | Will | 30 | 70000 | 300000 | NULL | +---------+------+--------+--------+-------+ LAG SELECT name, department_id AS dept, salary, LAG(salary, 1) OVER w AS `lag`, LEAD(salary, 1) OVER w AS `lead` FROM employee WINDOW w AS (PARTITION BY department_id ORDER BY salary DESC);
  • 37. 37Copyright © 2017 Oracle and/or its affiliates. All rights reserved. FIRST_VALUE, LAST_VALUE, NTH_VALUE Returns value evaluated at the first, last, nth in the frame of the current row within the partition; if there is no nth row (frame is too small), the NTH_VALUE returns NULL. first or last value ::= { FIRST_VALUE | LAST_VALUE } ( expr ) [ RESPECT NULLS ] nth_value ::= NTH_VALUE ( expr, nth-row ) [FROM FIRST] [ RESPECT NULLS ] Note: “IGNORE NULLS” is not supported, RESPECT NULLS is used but can be specified. Note: For NTH_VALUE, “FROM LAST” is not supported, FROM FIRST is used but can be specified
  • 38. 38Copyright © 2017 Oracle and/or its affiliates. All rights reserved. SELECT name, department_id AS dept, salary, SUM(salary) OVER w AS `sum`, FIRST_VALUE(salary) OVER w AS `first` FROM employee WINDOW w AS (PARTITION BY department_id ORDER BY name ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) +---------+------+--------+--------+--------+ | name | dept | salary | sum | first | +---------+------+--------+--------+--------+ | Newt | NULL | 75000 | 75000 | 75000 | | Dag | 10 | NULL | NULL | NULL | | Ed | 10 | 100000 | 100000 | NULL | | Fred | 10 | 60000 | 160000 | NULL | | Jon | 10 | 60000 | 220000 | 100000 | | Michael | 10 | 70000 | 190000 | 60000 | | Newt | 10 | 80000 | 210000 | 60000 | | Lebedev | 20 | 65000 | 65000 | 65000 | | Pete | 20 | 65000 | 130000 | 65000 | | Jeff | 30 | 300000 | 300000 | 300000 | | Will | 30 | 70000 | 370000 | 300000 | +---------+------+--------+--------+--------+ FIRST_VALUE “in frame” Current row: Jon FIRST_VALUE in frame is: Ed
  • 39. 39Copyright © 2017 Oracle and/or its affiliates. All rights reserved. SELECT name, department_id AS dept, salary, SUM(salary) OVER w AS `sum`, FIRST_VALUE(salary) OVER w AS `first`, LAST_VALUE(salary) OVER w AS `last` FROM employee WINDOW w AS (PARTITION BY department_id ORDER BY name ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) +---------+------+--------+--------+--------+--------+ | name | dept | salary | sum | first | last | +---------+------+--------+--------+--------+--------+ | Newt | NULL | 75000 | 75000 | 75000 | 75000 | | Dag | 10 | NULL | NULL | NULL | NULL | | Ed | 10 | 100000 | 100000 | NULL | 100000 | | Fred | 10 | 60000 | 160000 | NULL | 60000 | | Jon | 10 | 60000 | 220000 | 100000 | 60000 | | Michael | 10 | 70000 | 190000 | 60000 | 70000 | | Newt | 10 | 80000 | 210000 | 60000 | 80000 | | Lebedev | 20 | 65000 | 65000 | 65000 | 65000 | | Pete | 20 | 65000 | 130000 | 65000 | 65000 | | Jeff | 30 | 300000 | 300000 | 300000 | 300000 | | Will | 30 | 70000 | 370000 | 300000 | 70000 | +---------+------+--------+--------+--------+--------+ LAST_VALUE “in frame” Current row: Jon LAST_VALUE in frame is: Jon
  • 40. 40Copyright © 2017 Oracle and/or its affiliates. All rights reserved. SELECT name, department_id AS dept, salary, SUM(salary) OVER w AS `sum`, NTH_VALUE(salary, 2) OVER w AS nth FROM employee WINDOW w AS (PARTITION BY department_id ORDER BY name ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) +---------+------+--------+--------+--------+ | name | dept | salary | sum | nth | +---------+------+--------+--------+--------+ | Newt | NULL | 75000 | 75000 | NULL | | Dag | 10 | NULL | NULL | NULL | | Ed | 10 | 100000 | 100000 | 100000 | | Fred | 10 | 60000 | 160000 | 100000 | | Jon | 10 | 60000 | 220000 | 60000 | | Michael | 10 | 70000 | 190000 | 60000 | | Newt | 10 | 80000 | 210000 | 70000 | | Lebedev | 20 | 65000 | 65000 | NULL | | Pete | 20 | 65000 | 130000 | 65000 | | Jeff | 30 | 300000 | 300000 | NULL | | Will | 30 | 70000 | 370000 | 70000 | +---------+------+--------+--------+--------+ NTH_VALUE “in frame” Current row: Jon NTH_VALUE(.., 2) in frame is: Fred
  • 41. 41Copyright © 2017 Oracle and/or its affiliates. All rights reserved. Implicit and explicit windows ● Windows can be implicit and unnamed: COUNT(*) OVER (PARTITION BY DEPARTMENT_ID) ● Windows can be defined and named via the windows clause clause: SELECT COUNT(*) OVER w FROM t WINDOW w as (PARTITION BY department_id) ● This allows easy sharing of windows between several window functions and also avoids redundant windowing steps since more functions can be evaluated in the same step. ● Limitation: equivalent windows are not yet merged
  • 42. 42Copyright © 2017 Oracle and/or its affiliates. All rights reserved. Implicit and explicit windows ● A window definition can inherit from another window definition in its specification, adding detail (no override) SELECT name, department_id, COUNT(*) OVER w1 AS cnt1, COUNT(*) over w2 AS cnt2 FROM employee WINDOW w1 AS (PARTITION BY department_id), w2 AS (w1 ORDER BY name) ORDER BY department_id, name; +---------+---------------+------+------+ | name | department_id | cnt1 | cnt2 | +---------+---------------+------+------+ | Newt | NULL | 1 | 1 | | Dag | 10 | 6 | 1 | | Ed | 10 | 6 | 2 | | Fred | 10 | 6 | 3 | | Jon | 10 | 6 | 4 | | Michael | 10 | 6 | 5 | | Newt | 10 | 6 | 6 | | Lebedev | 20 | 2 | 1 | | Pete | 20 | 2 | 2 | | Jeff | 30 | 2 | 1 | | Will | 30 | 2 | 2 | +---------+---------------+------+------+
  • 43. 43Copyright © 2017 Oracle and/or its affiliates. All rights reserved. Time-line ● Work in progress, no due date yet
  • 44. 44Copyright © 2017 Oracle and/or its affiliates. All rights reserved. Q & A
  • 45. 45Copyright © 2017 Oracle and/or its affiliates. All rights reserved.