mysql 8.0: common table expressions - percona · title: title author: eivin hatvik subject:...
TRANSCRIPT
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
MySQL 8.0: Common Table Expressions
Øystein Grøvlen – Senior Principal Software Engineer MySQL Optimizer Team, Oracle
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Non-recursive common table expressions
Recursive common table expressions
1
2
2
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Non-recursive common table expressions
Recursive common table expressions
1
2
3
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Common Table Expression (MySQL 8.0.0 labs release)
• A derived table is a subquery in the FROM clause
SELECT … FROM (subquery) AS derived, t1 ...
• Common Table Expression (CTE) is just like a derived table, but its declaration is put before the query block instead of in FROM clause
WITH derived AS (subquery) SELECT … FROM derived, t1 ...
• A CTE may precede SELECT/UPDATE/DELETE including sub-queries
WITH derived AS (subquery) DELETE FROM t1 WHERE t1.a IN (SELECT b FROM derived);
4
Alternative to derived table
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
CTE
WITH cte_name [( <list of column names> )] AS
(
SELECT ... # Definition
)
[, <any number of other CTE definitions> ]
<SELECT/UPDATE/DELETE statement>
5
Syntax
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Common Table Expression (CTE)
WITH qn AS (SELECT a FROM t1) SELECT * from qn;
WITH qn AS (SELECT a+2 AS a, b FROM t1) UPDATE t1, qn SET t1.a=qn.a + 10 WHERE t1.a - qn.a = 0;
WITH qn(a, b) AS (SELECT a+2, b FROM t2) DELETE t1 FROM t1, qn WHERE t1.a - qn.a = 0;
INSERT INTO t2 WITH qn AS (SELECT 10*a AS a FROM t1) SELECT * from qn;
SELECT * FROM t1 WHERE t1.a IN (WITH cte as (SELECT * FROM t1 AS t2 LIMIT 1) SELECT a + 0 FROM cte);
6
Examples
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Common Table Expression versus Derived Table
Better readability
Can be referenced multiple times
Can refer to other CTEs
Improved performance
7
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Better readability
• Derived table:
SELECT … FROM t1 LEFT JOIN ((SELECT … FROM …) AS dt JOIN t2 ON …) ON …
• CTE:
WITH dt AS (SELECT ... FROM ...) SELECT ... FROM t1 LEFT JOIN (dt JOIN t2 ON ...) ON ...
8
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Can be referenced multiple times
• Derived table can not be referenced twice:
SELECT ... FROM (SELECT a, b, SUM(c) s FROM t1 GROUP BY a, b) AS d1 JOIN (SELECT a, b, SUM(c) s FROM t1 GROUP BY a, b) AS d2 ON d1.b = d2.a;
• CTE can:
WITH d AS (SELECT a, b, SUM(c) s FROM t1 GROUP BY a, b) SELECT ... FROM d AS d1 JOIN d AS d2 ON d1.b = d2.a;
9
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Can refer to other CTEs
• Derived tables can not refer to other derived tables: SELECT … FROM (SELECT … FROM …) AS d1, (SELECT … FROM d1 …) AS d2 …
ERROR: 1146 (42S02): Table ‘db.d1’ doesn’t exist
• CTEs can refer other CTEs: WITH d1 AS (SELECT … FROM …), d2 AS (SELECT … FROM d1 …) SELECT FROM d1, d2 …
10
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Chained CTEs
WITH cte1(txt) AS (SELECT "This "),
cte2(txt) AS (SELECT CONCAT(cte1.txt,"is a ") FROM cte1),
cte3(txt) AS (SELECT "nice query" UNION
SELECT "query that rocks" UNION
SELECT "query"),
cte4(txt) AS (SELECT concat(cte2.txt, cte3.txt) FROM cte2, cte3)
SELECT MAX(txt), MIN(txt) FROM cte4;
+----------------------------+----------------------+
| MAX(txt) | MIN(txt) |
+----------------------------+----------------------+
| This is a query that rocks | This is a nice query |
+----------------------------+----------------------+
1 row in set (0,00 sec)
11
Neat, but not very useful example
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Better performance
• Derived table:
– For derived tables that are materialized, two identical derived tables will be materialized. Performance problem (more space, more time, longer locks)
– Similar with view references
• CTE: –Will be materialized once, regardless of how many references
12
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
DBT3 Query 15
CREATE VIEW revenue0 (supplier_no, total_revenue) AS SELECT l_suppkey, SUM(l_extendedprice * (1-l_discount)) FROM lineitem WHERE l_shipdate >= '1996-07-01' AND l_shipdate < DATE_ADD('1996-07-01‘, INTERVAL '90' day) GROUP BY l_suppkey;
SELECT s_suppkey, s_name, s_address, s_phone, total_revenue FROM supplier, revenue0 WHERE s_suppkey = supplier_no AND total_revenue = (SELECT MAX(total_revenue) FROM revenue0) ORDER BY s_suppkey;
Top Supplier Query
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Materialization of view
DBT-3 Query 15
ORDER JOIN
Supplier (eq_ref)
GROUP Lineitem (range)
Materialize
GROUP Lineitem (range)
Materialize
SELECT
Sub
qu
ery
Revenue0
Revenue0
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
DBT-3 Query 15 EXPLAIN
CTE
id select type
table type possible keys key rows filtered Extra
1 PRIMARY <derived3> ALL NULL NULL 4801074 10.00 Using where; Using temporary; Using filesort
1 PRIMARY supplier eq_ref PRIMARY PRIMARY 1 100.00 NULL
3 DERIVED lineitem range i_l_shipdate, ... i_l_shipdate 4801074 100.00 Using temporary
2 SUBQUERY <derived4> ALL NULL NULL 4801074 100.00 NULL
4 DERIVED lineitem range i_l_shipdate, ... i_l_shipdate 4801074 100.00 Using temporary
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
CTE Materialization
WITH revenue0 (supplier_no, total_revenue) AS (SELECT l_suppkey, SUM(l_extendedprice * (1-l_discount)) FROM lineitem WHERE l_shipdate >= '1996-07-01' AND l_shipdate < DATE_ADD('1996-07-01‘, INTERVAL '90' day) GROUP BY l_suppkey)
SELECT s_suppkey, s_name, s_address, s_phone, total_revenue FROM supplier, revenue0 WHERE s_suppkey = supplier_no AND total_revenue = (SELECT MAX(total_revenue) FROM revenue0) ORDER BY s_suppkey;
DBT3 Query 15: Top Supplier Query
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
DBT-3 Query 15, CTE EXPLAIN
CTE
id select type
table type possible keys key rows filtered Extra
1 PRIMARY <derived2> ALL NULL NULL 4801074 10.00 Using where; Using temporary; Using filesort
1 PRIMARY supplier eq_ref PRIMARY PRIMARY 1 100.00 NULL
3 SUBQUERY <derived2> ALL NULL NULL 4801074 100.00 NULL
2 DERIVED lineitem range i_l_shipdate, ... i_l_shipdate 4801074 100.00 Using temporary
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
DBT-3 Query 15 Materialization of CTE
ORDER JOIN
Supplier (eq_ref)
GROUP Lineitem (range)
Materialize
SELECT
Sub
qu
ery
Revenue0
FROM
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
DBT-3 Query 15
Confidential – Oracle Internal/Restricted/Highly Restricted 19
Query Performance
0
2
4
6
8
10
12
14
16
18
View CTE
Qu
ery
Exe
cuti
on
Tim
e (
seco
nd
s)
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Non-recursive CTE
• Find best and worst month:
WITH sales_by_month(month,total) AS
# first CTE: one row per month, with amount sold on all days of month
(SELECT MONTH(day_of_sale), SUM(amount) FROM sales_days
WHERE YEAR(day_of_sale)=2015
GROUP BY MONTH(day_of_sale)),
best_month(month, total, award) AS # second CTE: best month
(SELECT month, total, "best" FROM sales_by_month
WHERE total=(SELECT MAX(total) FROM sales_by_month)),
worst_month(month, total, award) AS # 3rd CTE: worst month
(SELECT month, total, "worst" FROM sales_by_month
WHERE total=(SELECT MIN(total) FROM sales_by_month))
# Now show best and worst:
SELECT * FROM best_month UNION All SELECT * FROM worst_month;
20
A more useful example
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Non-recursive CTE
• Result:
+-------+-------+-------+
| month | total | award |
+-------+-------+-------+
| 1 | 300 | best |
| 3 | 11 | worst |
+-------+-------+-------+
21
A more useful example
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Non-recursive common table expressions
Recursive common table expressions
1
2
22
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Recursive CTE
• A recursive CTE refers to itself in a subquery
• The “seed” SELECT is executed once to create the initial data subset, the recursive SELECT is repeatedly executed to return subsets of data until the complete result set is obtained.
• Recursion stops when an iteration does not generate any new rows
• Useful to dig in hierarchies (parent/child, part/subpart)
23
WITH RECURSIVE cte AS ( SELECT ... FROM table_name /* "seed" SELECT */ UNION ALL SELECT ... FROM cte, table_name) /* "recursive" SELECT */ SELECT ... FROM cte;
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Recursive CTE
24
A simple example
Print 1 to 10 : WITH RECURSIVE qn AS ( SELECT 1 AS a UNION ALL SELECT 1+a FROM qn WHERE a<10 ) SELECT * FROM qn;
a 1 2 3 4 5 6 7 8 9 10
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Recursive CTE
25
INSERT
Insert 1 to 10 : INSERT INTO numbers WITH RECURSIVE qn AS ( SELECT 1 AS a UNION ALL SELECT 1+a FROM qn WHERE a<10 ) SELECT * FROM qn;
SELECT * FROM numbers; a 1 2 3 4 5 6 7 8 9 10
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Date sequence
26
Missing dates
SELECT orderdate, SUM(totalprice) sales FROM orders GROUP BY orderdate ORDER BY orderdate;
+------------+-----------+
| orderdate | sales |
+------------+-----------+
| 2016-09-01 | 43129.83 |
| 2016-09-03 | 218347.61 |
| 2016-09-04 | 142568.40 |
| 2016-09-05 | 299244.83 |
| 2016-09-07 | 185991.79 |
+------------+-----------+
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Date sequence
27
All dates
WITH RECURSIVE dates(date) AS ( SELECT '2016-09-01' UNION ALL SELECT DATE_ADD(date, INTERVAL 1 DAY) FROM dates WHERE date < '2016-09-07‘ ) SELECT dates.date, COALESCE(SUM(totalprice), 0) sales FROM dates LEFT JOIN orders ON dates.date = orders.orderdate GROUP BY dates.date ORDER BY dates.date;
+------------+-----------+
| date | sales |
+------------+-----------+
| 2016-09-01 | 43129.83 |
| 2016-09-02 | 0.00 |
| 2016-09-03 | 218347.61 |
| 2016-09-04 | 142568.40 |
| 2016-09-05 | 299244.83 |
| 2016-09-06 | 0.00 |
| 2016-09-07 | 185991.79 |
+------------+-----------+
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
n un unp 1 1 1 1 2 1 2 3 2 3 4 3 5 5 5 8 6 8 13 7 13 21 8 21 34 9 34 55 10 55 89
Fibonacci Numbers
28
WITH RECURSIVE qn(n, un, unp1) AS ( SELECT 1, 1 , 1 UNION ALL SELECT 1+n, unp1, un+unp1 FROM qn WHERE n<10) SELECT * FROM qn;
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Hierarchy Traversal
29
Employee database
CREATE TABLE employees ( id INT PRIMARY KEY, name VARCHAR(100), manager_id INT, FOREIGN KEY (manager_id) REFERENCES employees(id) );
INSERT INTO employees VALUES (333, "Yasmina", NULL), # CEO (198, "John", 333), # John reports to 333 (692, "Tarek", 333), (29, "Pedro", 198), (4610, "Sarah", 29), (72, "Pierre", 29), (123, "Adil", 692);
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Hierarchy Traversal
30
List reporting chain
WITH RECURSIVE emp_ext (id, name, path) AS ( SELECT id, name, CAST(id AS CHAR(200)) FROM employees WHERE manager_id IS NULL UNION ALL SELECT s.id, s.name, CONCAT(m.path, ",", s.id) FROM emp_ext m JOIN employees s ON m.id=s.manager_id ) SELECT * FROM emp_ext ORDER BY path;
id name path 333 Yasmina 333 198 John 333,198 692 Tarek 333,692 29 Pedro 333,198,29 123 Adil 333,692,123 4610 Sarah 333,198,29,4610 72 Pierre 333,198,29,72
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Hierarchy Traversal
31
List descendants
WITH RECURSIVE emp_ext (id, name, path) AS ( SELECT id, name, CAST(id AS CHAR(200)) FROM employees WHERE manager_id IS NULL UNION ALL SELECT s.id, s.name, CONCAT(m.path, ",", s.id) FROM emp_ext m JOIN employees s ON m.id=s.manager_id ) SELECT * FROM emp_ext ORDER BY path;
id name path 333 Yasmina 333 198 John 333,198 29 Pedro 333,198,29 4610 Sarah 333,198,29,4610 72 Pierre 333,198,29,72 692 Tarek 333,692 123 Adil 333,692,123
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Recursive CTE
• WITH RECURSIVE cte_name [list of column names ] AS (
SELECT ... <-- specifies initial set
UNION ALL
SELECT ... <-- specifies initial set
UNION ALL
...
SELECT ... <-- specifies how to derive new rows
UNION ALL
SELECT ... <-- specifies how to derive new rows
...
)
[, any number of other CTE definitions ]
32
Complete syntax
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Recursive CTE
• A recursive SELECT must not contain
– GROUP BY
– Aggregate functions (like SUM)
–ORDER BY
– LIMIT
– DISTINCT
• A recursive SELECT must reference the CTE only once and only in its FROM clause, not in any subquery
• A recursive SELECT can not be the inner table of an outer join
Confidential – Oracle Internal/Restricted/Highly Restricted 33
Restrictions
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Want to try it out?
• Goto labs.mysql.com
• Select MySQL Server 8.0.0 Optimizer
• Available as
– Linux binaries
– Source code
• Warning!
– For testing purposes only!
– NOT FIT FOR PRODUCTION.
34
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Other New Features in 8.0.0 Optimizer Labs Release
• MERGE() and NO_MERGE() hints WITH cte AS ( ... ) SELECT /*+ NO_MERGE(cte) */ ... FROM cte ...
• Column name aliases for derived tables
SELECT ... FROM (SELECT ...) dt(a, b, c) ...
• Descending index
• Join order hints SELECT /*+ JOIN_ORDER(t1, t2) */ * FROM t1 JOIN t2 ...
• JSON aggregation function
– JSON_ARRAYAGG
– JSON_OBJECTAGG
Confidential – Oracle Internal/Restricted/Highly Restricted 35
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Want to learn more?
• MySQL Server Team blog
– http://mysqlserverteam.com/
– http://mysqlserverteam.com/mysql-8-0-labs-recursive-common-table-expressions-in-mysql-ctes/
–More to come
• My blog:
– http://oysteing.blogspot.com/
• MySQL forum
–Optimizer & Parser: http://forums.mysql.com/list.php?115