Download - Sql query patterns, optimized
1
© 2014 PERCONA
MySQL Query Patterns, Optimized
Bill Karwin, Percona
1
© 2014 PERCONA
Welcome Everybody! • Bill Karwin
– Senior Knowledge Manager in Percona Support – Joined Percona in 2010
• Percona offers MySQL services – Consulting – Support for MySQL and variants – Remote DBA – Development – Training
2
1
© 2014 PERCONA
How Do We Optimize? • Identify queries. • Measure optimization plan and performance.
– EXPLAIN – SHOW SESSION STATUS – SHOW PROFILES
• Add indexes and/or redesign the query.
1
© 2014 PERCONA
Testing Environment • MySQL 5.7.5-m15 • CentOS 6.5 running under VirtualBox on my
Macbook Pro (non-SSD) • MySQL Workbench 6.2
1
© 2014 PERCONA
Example Database
cast_info
name
char_name
role_type >tle kind_type
1
© 2014 PERCONA
Common Query Patterns 1. Exclusion Joins 2. Random Selection 3. Greatest per Group 4. Dynamic Pivot 5. Relational Division
1
© 2014 PERCONA
EXCLUSION JOINS Query Patterns
1
© 2014 PERCONA
Assignment:
“I want to find recent movies that had no director.”
1
© 2014 PERCONA
Not Exists Solution SELECT t.title!FROM title t!WHERE kind_id = 1 !AND production_year >= 2005 !AND NOT EXISTS (! SELECT * FROM cast_info c! WHERE c.movie_id = t.id ! AND c.role_id = 8 /* director */!);!
Movies
In the range of recent years
Correlated subquery to find a director for each movie
1
© 2014 PERCONA
Not Exists Solution SELECT t.title!FROM title t!WHERE kind_id = 1 !AND production_year >= 2005 !AND NOT EXISTS (! SELECT * FROM cast_info c! WHERE c.movie_id = t.id ! AND c.role_id = 8!);!
???s
I gave up after waiting > 1 hour
1
© 2014 PERCONA
EXPLAIN: the Not-Exists Solution id select_type table type key ref rows Extra
1 PRIMARY t ALL NULL NULL 1598319 Using where
2 DEPENDENT SUBQUERY
c ALL NULL NULL 24149504 Using where
The correlated subquery is executed 1.6 M times!
Both tables are table scans
And scans 24 M rows each time, totalling 3.8 × 1013 row comparisons
Dependent subquery executes once for each set of values in outer
1
© 2014 PERCONA
Indexes: the Not-Exists Solution CREATE INDEX k_py !ON title (kind_id, production_year);!!CREATE INDEX m_r!ON cast_info (movie_id, role_id);!
1
© 2014 PERCONA
EXPLAIN: the Not-Exists Solution id select_type table type key ref rows Extra
1 PRIMARY t range k_py NULL 189846 Using index condi>on; Using where
2 DEPENDENT SUBQUERY
c ref m_r t.id, const
3 Using index
The correlated subquery is executed 189k times!
At least both table references use indexes
A covering index is best—if the index fits in memory
Dependent subquery executes once for each set of values in outer
1
© 2014 PERCONA
Visual Explain in Workbench
1
© 2014 PERCONA
Not Exists Solution SELECT t.title!FROM title t!WHERE kind_id = 1 !AND production_year >= 2005 !AND NOT EXISTS (! SELECT * FROM cast_info c! WHERE c.movie_id = t.id ! AND c.role_id = 8!);!
5.34s
Better, but when the indexes aren’t in memory, it’s still too slow
1
© 2014 PERCONA
Buffer Pool • It’s crucial that queries read an index from memory;
I/O during an index scan kills performance. [mysqld]!innodb_buffer_pool_size = 64M # wrong!innodb_buffer_pool_size = 2G # better!
1
© 2014 PERCONA
Not Exists Solution SELECT t.title!FROM title t!WHERE kind_id = 1 !AND production_year >= 2005 !AND NOT EXISTS (! SELECT * FROM cast_info c! WHERE c.movie_id = t.id ! AND c.role_id = 8!);!
3.25s
much faster after increasing size of
buffer pool
1
© 2014 PERCONA
SHOW SESSION STATUS • Shows the real count of row accesses for your
current session. mysql> FLUSH STATUS;!mysql> ... run a query ...!mysql> SHOW SESSION STATUS LIKE 'Handler%';!
1
© 2014 PERCONA
Status: the Not-Exists Solution +----------------------------+--------+!| Variable_name | Value |!+----------------------------+--------+!| Handler_commit | 1 |!| Handler_delete | 0 |!| Handler_discover | 0 |!| Handler_external_lock | 6 |!| Handler_mrr_init | 0 |!| Handler_prepare | 0 |!| Handler_read_first | 0 |!| Handler_read_key | 186489 |!| Handler_read_last | 0 |!| Handler_read_next | 93244 |!| Handler_read_prev | 0 |!| Handler_read_rnd | 93244 |!| Handler_read_rnd_next | 0 |!| Handler_rollback | 0 |!| Handler_savepoint | 0 |!| Handler_savepoint_rollback | 0 |!| Handler_update | 0 |!| Handler_write | 0 |!+----------------------------+--------+!
read_key: lookup by index, e.g. each lookup in cast_info, plus the first row in title
read_next: advancing in index order, e.g. the range query for rows in title after the first row
read_rnd: fetch a row from a table based on a row reference (probably by MRR)
1
© 2014 PERCONA
SHOW PROFILE • Enable query profiler for the current session.
mysql> SET PROFILING = 1;!
• Run a query. mysql> SELECT t.title FROM title t ...!
• Query the real execution time. mysql> SHOW PROFILES;!
• Query detail for a specific query. mysql> SHOW PROFILE FOR QUERY 1;!
1
© 2014 PERCONA
Profile: the Not-Exists Solution +----------------+----------+!| Status | Duration |!+----------------+----------+!| Sending data | 0.000029 |!| executing | 0.000004 |!| Sending data | 0.000075 |!. . .!| executing | 0.000004 |!| Sending data | 0.000023 |!| executing | 0.000004 |!| Sending data | 0.000023 |!| executing | 0.000004 |!| Sending data | 0.000081 |!| executing | 0.000009 |!| Sending data | 0.000029 |!| end | 0.000007 |!| query end | 0.000022 |!| closing tables | 0.000028 |!| freeing items | 0.000344 |!| cleaning up | 0.000025 |!+----------------+----------+!
Thousands of iterations of correlated subqueries caused the profile information to overflow!
1
© 2014 PERCONA
Not-In Solution SELECT title!FROM title!WHERE kind_id = 1 !AND production_year >= 2005 !AND id NOT IN (! SELECT movie_id FROM cast_info! WHERE role_id = 8!);!
1.62s
Not a correlated subquery
1
© 2014 PERCONA
Indexes: the Not-In Solution CREATE INDEX k_py !ON title (kind_id, production_year);!!CREATE INDEX m_r!ON cast_info (movie_id, role_id);!
1
© 2014 PERCONA
EXPLAIN: the Not-In Solution id select_type table type key ref rows Extra
1 PRIMARY >tle range k_py NULL 189846 Using index condi>on; Using where; Using MRR
1 DEPENDENT SUBQUERY
cast_info
index_subquery m_r func, const
1 Using index; Using where
But somehow MySQL doesn’t report a different select type
picks only 1 row per subquery execution
1
© 2014 PERCONA
Visual Explain
1
© 2014 PERCONA
Status: the Not-In Solution +----------------------------+--------+!| Variable_name | Value |!+----------------------------+--------+!| Handler_commit | 1 |!| Handler_delete | 0 |!| Handler_discover | 0 |!| Handler_external_lock | 6 |!| Handler_mrr_init | 0 |!| Handler_prepare | 0 |!| Handler_read_first | 0 |!| Handler_read_key | 186489 |!| Handler_read_last | 0 |!| Handler_read_next | 93244 |!| Handler_read_prev | 0 |!| Handler_read_rnd | 93244 |!| Handler_read_rnd_next | 0 |!| Handler_rollback | 0 |!| Handler_savepoint | 0 |!| Handler_savepoint_rollback | 0 |!| Handler_update | 0 |!| Handler_write | 0 |!+----------------------------+--------+!
1
© 2014 PERCONA
Profile: the Not-In Solution +----------------------+----------+!| Status | Duration |!+----------------------+----------+!| starting | 0.000128 |!| checking permissions | 0.000014 |!| checking permissions | 0.000012 |!| Opening tables | 0.000043 |!| init | 0.000059 |!| System lock | 0.000030 |!| optimizing | 0.000022 |!| statistics | 0.000127 |!| preparing | 0.000036 |!| optimizing | 0.000013 |!| statistics | 0.000028 |!| preparing | 0.000017 |!| executing | 0.000008 |!| Sending data | 1.623482 |!| end | 0.000024 |!| query end | 0.000025 |!| closing tables | 0.000020 |!| freeing items | 0.000329 |!| cleaning up | 0.000029 |!+----------------------+----------+!
Most of the time spent in “Sending data” (moving rows around)
1
© 2014 PERCONA
Outer-Join Solution SELECT t.title!FROM title t!LEFT OUTER JOIN cast_info c ! ON t.id = c.movie_id ! AND c.role_id = 8!WHERE t.kind_id = 1 !AND t.production_year >= 2005 !AND c.movie_id IS NULL;!
1.58s
Try to find a director for each movie using a join
If no director is found, that’s the one we want
1
© 2014 PERCONA
Indexes: the Outer-Join Solution CREATE INDEX k_py !ON title (kind_id, production_year);!!CREATE INDEX m_r!ON cast_info (movie_id, role_id);!
1
© 2014 PERCONA
EXPLAIN: the Outer-Join Solution id select_type table type key ref rows Extra
1 SIMPLE t range k_py NULL 189846 Using index condi>on
1 SIMPLE c ref m_r t.id, const
1 Using where; Using index; Not exists
Special “not exists” optimization
1
© 2014 PERCONA
Visual Explain
1
© 2014 PERCONA
Status: the Outer-Join Solution +----------------------------+-------+!| Variable_name | Value |!+----------------------------+-------+!| Handler_commit | 1 |!| Handler_delete | 0 |!| Handler_discover | 0 |!| Handler_external_lock | 4 |!| Handler_mrr_init | 0 |!| Handler_prepare | 0 |!| Handler_read_first | 0 |!| Handler_read_key | 93245 |!| Handler_read_last | 0 |!| Handler_read_next | 93244 |!| Handler_read_prev | 0 |!| Handler_read_rnd | 0 |!| Handler_read_rnd_next | 0 |!| Handler_rollback | 0 |!| Handler_savepoint | 0 |!| Handler_savepoint_rollback | 0 |!| Handler_update | 0 |!| Handler_write | 0 |!+----------------------------+-------+!
Fewest row reads
1
© 2014 PERCONA
Profile: the Outer-Join Solution +----------------------+----------+!| Status | Duration |!+----------------------+----------+!| starting | 0.000161 |!| checking permissions | 0.000014 |!| checking permissions | 0.000013 |!| Opening tables | 0.000051 |!| init | 0.000056 |!| System lock | 0.000040 |!| optimizing | 0.000000 |!| statistics | 0.000263 |!| preparing | 0.000064 |!| executing | 0.000012 |!| Sending data | 1.581615 |!| end | 0.000021 |!| query end | 0.000022 |!| closing tables | 0.000016 |!| freeing items | 0.000324 |!| cleaning up | 0.000026 |!+----------------------+----------+!
1
© 2014 PERCONA
Summary: Exclusion Joins Solu7on Time Notes Not-‐Exists 3.25s dependent subquery Not-‐In 1.62s dependent subquery Outer-‐Join 1.58s “not exists” op>miza>on
1
© 2014 PERCONA
RANDOM SELECTION Query Patterns
1
© 2014 PERCONA
Assignment:
“I want a query that picks a random movie.”
1
© 2014 PERCONA
Naïve Order-By Solution SELECT *!FROM title!WHERE kind_id = 1 /* movie */!ORDER BY RAND()!LIMIT 1;!
0.98s
1
© 2014 PERCONA
EXPLAIN: the Order-By Solution id select_type table type key ref rows Extra
1 SIMPLE >tle ref k_py const 790882 Using temporary; Using filesort
1
© 2014 PERCONA
Visual Explain
1
© 2014 PERCONA
Status: the Order-By Solution +----------------------------+--------+!| Variable_name | Value |!+----------------------------+--------+!| Handler_commit | 2 |!| Handler_delete | 0 |!| Handler_discover | 0 |!| Handler_external_lock | 4 |!| Handler_mrr_init | 0 |!| Handler_prepare | 0 |!| Handler_read_first | 0 |!| Handler_read_key | 2 |!| Handler_read_last | 0 |!| Handler_read_next | 947164 |!| Handler_read_prev | 0 |!| Handler_read_rnd | 2 |!| Handler_read_rnd_next | 947166 |!| Handler_rollback | 0 |!| Handler_savepoint | 0 |!| Handler_savepoint_rollback | 0 |!| Handler_update | 0 |!| Handler_write | 947164 |!+----------------------------+--------+!
What is this?
1
© 2014 PERCONA
Profile: the Order-By Solution +----------------------+----------+!| Status | Duration |!+----------------------+----------+!| starting | 0.000074 |!| checking permissions | 0.000032 |!| Opening tables | 0.000035 |!| System lock | 0.000012 |!| init | 0.000025 |!| optimizing | 0.000004 |!| statistics | 0.000014 |!| preparing | 0.000010 |!| Creating tmp table | 0.000245 |!| executing | 0.000003 |!| Copying to tmp table | 4.875666 |!| Sorting result | 3.871513 |!| Sending data | 0.000059 |!| end | 0.000005 |!| removing tmp table | 0.058239 |!| end | 0.000018 |!
| query end | 0.000064 |!| closing tables | 0.000034 |!| freeing items | 0.000210 |!| logging slow query | 0.000003 |!| cleaning up | 0.000005 |!+----------------------+----------+!
MySQL 5.5 saves to a temp table, then sorts by filesort.
1
© 2014 PERCONA
Profile: the Order-By Solution +----------------------+----------+!| Status | Duration |!+----------------------+----------+!| starting | 0.000111 |!| checking permissions | 0.000015 |!| Opening tables | 0.000030 |!| init | 0.000045 |!| System lock | 0.000019 |!| optimizing | 0.000013 |!| statistics | 0.000106 |!| preparing | 0.000019 |!| Creating tmp table | 0.000050 |!| Sorting result | 0.000007 |!| executing | 0.000005 |!| Sending data | 0.836566 |!| Creating sort index | 0.145061 |!| end | 0.000018 |!| query end | 0.000021 |!| removing tmp table | 0.002427 |!
| query end | 0.000516 |!| closing tables | 0.000037 |!| freeing items | 0.000234 |!| cleaning up | 0.000023 |!+----------------------+----------+!
MySQL 5.7 creates a sort index for the temp table on the fly
1
© 2014 PERCONA
Offset Solution SELECT ROUND(RAND() * COUNT(*)) !FROM title!WHERE kind_id = 1;!!SELECT *!FROM title!WHERE kind_id = 1!LIMIT 1 OFFSET $random;!
0.45s
1
© 2014 PERCONA
Indexes: the Offset Solution CREATE INDEX k!ON title (kind_id);!
1
© 2014 PERCONA
EXPLAIN: the Offset Solution id select_type table type key ref rows Extra
1 SIMPLE >tle ref k const 787992
1
© 2014 PERCONA
Visual Explain
1
© 2014 PERCONA
Status: the Offset Solution +----------------------------+--------+!| Variable_name | Value |!+----------------------------+--------+!| Handler_commit | 1 |!| Handler_delete | 0 |!| Handler_discover | 0 |!| Handler_external_lock | 2 |!| Handler_mrr_init | 0 |!| Handler_prepare | 0 |!| Handler_read_first | 0 |!| Handler_read_key | 1 |!| Handler_read_last | 0 |!| Handler_read_next | 473582 |!| Handler_read_prev | 0 |!| Handler_read_rnd | 0 |!| Handler_read_rnd_next | 0 |!| Handler_rollback | 0 |!| Handler_savepoint | 0 |!| Handler_savepoint_rollback | 0 |!| Handler_update | 0 |!| Handler_write | 0 |!+----------------------------+--------+!
Query must read OFFSET + COUNT rows. A high random value makes the query take longer.
1
© 2014 PERCONA
Profile: the Offset Solution +----------------------+----------+!| Status | Duration |!+----------------------+----------+!| starting | 0.000099 |!| checking permissions | 0.000012 |!| Opening tables | 0.000034 |!| init | 0.000045 |!| System lock | 0.000019 |!| optimizing | 0.000013 |!| statistics | 0.000114 |!| preparing | 0.000021 |!| executing | 0.000005 |!| Sending data | 0.459230 |!| end | 0.000018 |!| query end | 0.000018 |!| closing tables | 0.000017 |!| freeing items | 0.000194 |!| cleaning up | 0.000025 |!+----------------------+----------+!
Many rows moving from storage layer to SQL layer, only to be discarded.
1
© 2014 PERCONA
Primary Key Solution SELECT ROUND(RAND() * MAX(id)) !FROM title!WHERE kind_id = 1;!!SELECT *!FROM title!WHERE kind_id = 1 AND id > $random!LIMIT 1;!
0.0008s
1
© 2014 PERCONA
EXPLAIN: the Primary Key Solution id select_type table type key ref rows Extra
1 SIMPLE >tle ref k NULL 268254 Using index condi>on
Strange that the optimizer estimates this many rows
1
© 2014 PERCONA
Visual Explain
1
© 2014 PERCONA
Status: the Primary Key Solution +----------------------------+-------+!| Variable_name | Value |!+----------------------------+-------+!| Handler_commit | 1 |!| Handler_delete | 0 |!| Handler_discover | 0 |!| Handler_external_lock | 2 |!| Handler_mrr_init | 0 |!| Handler_prepare | 0 |!| Handler_read_first | 0 |!| Handler_read_key | 1 |!| Handler_read_last | 0 |!| Handler_read_next | 0 |!| Handler_read_prev | 0 |!| Handler_read_rnd | 0 |!| Handler_read_rnd_next | 0 |!| Handler_rollback | 0 |!| Handler_savepoint | 0 |!| Handler_savepoint_rollback | 0 |!| Handler_update | 0 |!| Handler_write | 0 |!+----------------------------+-------+!
Just one row read after the index lookup.
1
© 2014 PERCONA
Profile: the Primary Key Solution +----------------------+----------+!| Status | Duration |!+----------------------+----------+!| starting | 0.000119 |!| checking permissions | 0.000016 |!| Opening tables | 0.000036 |!| init | 0.000053 |!| System lock | 0.000027 |!| optimizing | 0.000021 |!| statistics | 0.000180 |!| preparing | 0.000112 |!| executing | 0.000016 |!| Sending data | 0.000175 |!| end | 0.000010 |!| query end | 0.000017 |!| closing tables | 0.000015 |!| freeing items | 0.000028 |!| cleaning up | 0.000028 |!+----------------------+----------+!
Everything is fast when we search a index by value rather than by position.
1
© 2014 PERCONA
Summary: Random Selection Solu7on Time Notes Order-‐By Solu>on 0.98s Offset Solu>on 0.45s Requires the COUNT() Primary Key Solu>on 0.0008s Requires the MAX()
1
© 2014 PERCONA
GREATEST PER GROUP Query Patterns
1
© 2014 PERCONA
Assignment:
“I want the last episode of every TV series.”
1
© 2014 PERCONA
Getting the Last Episode SELECT tv.title, ep.title, MAX(ep.episode_nr) AS last_ep !FROM title ep!JOIN title tv ON tv.id = ep.episode_of_id!WHERE ep.kind_id = 7 /* TV show */!GROUP BY ep.episode_of_id ORDER BY NULL;!
This is not the title of the last episode!
1
© 2014 PERCONA
Why Isn’t It? • The query doesn’t necessarily return the title from
the row where MAX(ep.episode_nr) occurs. • Should the following return the title of the first
episode or the last episode? SELECT tv.title, ep.title, MIN(ep.episode_nr) AS first_ep ! MAX(ep.episode_nr) AS last_ep !FROM . . .!
1
© 2014 PERCONA
Exclusion Join Solution SELECT tv.title, ep1.title, ep1.episode_nr!FROM title ep1!LEFT OUTER JOIN title ep2 ! ON ep1.kind_id = ep2.kind_id! AND ep1.episode_of_id = ep2.episode_of_id! AND ep1.episode_nr < ep2.episode_nr!JOIN title tv ON tv.id = ep1.episode_of_id!WHERE ep1.kind_id = 7 ! AND ep1.episode_of_id IS NOT NULL! AND ep1.episode_nr >= 1! AND ep2.episode_of_id IS NULL;!
71.20
Try to find a row ep2 for the same show with a greater episode_nr
If no such row is found, then ep1 must be the last episode for the show
1
© 2014 PERCONA
Indexes: the Exclusion-Join Solution CREATE INDEX k_ep_nr !ON title (kind_id, episode_of_id, episode_nr);!
1
© 2014 PERCONA
EXPLAIN: the Exclusion-Join Solution id select_type table type key ref rows Extra
1 SIMPLE ep1 ref k_py const 787992 Using where
1 SIMPLE tv eq_ref PRIMARY ep1.episode_of_id 1
1 SIMPLE ep2 ref k_ep_nr const, ep1.episode_of_id
22 Using where; Using index
1
© 2014 PERCONA
Visual Explain
1
© 2014 PERCONA
Status: the Exclusion-Join Solution +----------------------------+-----------+!| Variable_name | Value |!+----------------------------+-----------+!| Handler_commit | 1 |!| Handler_delete | 0 |!| Handler_discover | 0 |!| Handler_external_lock | 6 |!| Handler_mrr_init | 0 |!| Handler_prepare | 0 |!| Handler_read_first | 0 |!| Handler_read_key | 693046 |!| Handler_read_last | 0 |!| Handler_read_next | 254373071 |!| Handler_read_prev | 0 |!| Handler_read_rnd | 0 |!| Handler_read_rnd_next | 0 |!| Handler_rollback | 0 |!| Handler_savepoint | 0 |!| Handler_savepoint_rollback | 0 |!| Handler_update | 0 |!| Handler_write | 0 |!+----------------------------+-----------+!
Unfortunately, this seems to be O(n2)
1
© 2014 PERCONA
Profile: the Exclusion-Join Solution +----------------------+-----------+!| Status | Duration |!+----------------------+-----------+!| starting | 0.000147 |!| checking permissions | 0.000009 |!| checking permissions | 0.000004 |!| checking permissions | 0.000009 |!| Opening tables | 0.000038 |!| init | 0.000049 |!| System lock | 0.000032 |!| optimizing | 0.000025 |!| statistics | 0.000195 |!| preparing | 0.000045 |!| executing | 0.000006 |!| Sending data | 71.195693 |!| end | 0.000021 |!| query end | 0.000021 |!| closing tables | 0.000017 |!| freeing items | 0.000864 |!| logging slow query | 0.000135 |!| cleaning up | 0.000027 |!+----------------------+-----------+!
A lot of time is spent moving rows around
1
© 2014 PERCONA
Derived-Table Solution SELECT tv.title, ep.title, ep.episode_nr!FROM (! SELECT kind_id, episode_of_id, ! MAX(episode_nr) AS episode_nr! FROM title! WHERE kind_id = 7! GROUP BY kind_id, episode_of_id!) maxep!JOIN title ep USING (kind_id, episode_of_id, episode_nr)!JOIN title tv ON tv.id = ep.episode_of_id;!
0.60s
Generate a list of the greatest episode number per show
1
© 2014 PERCONA
Indexes: the Derived-Table Solution CREATE INDEX k_ep_nr !ON title (kind_id, episode_of_id, episode_nr);!
1
© 2014 PERCONA
EXPLAIN: the Derived-Table Solution id select_type table type key ref rows Extra
1 PRIMARY <derived2> ALL NULL NULL 751752 Using where
1 PRIMARY tv eq_ref PRIMARY maxep.episode_of_id 1 NULL
1 PRIMARY ep ref k_ep_nr maxep.kind_id, maxep.episode_id, maxep.episode_nr
2 NULL
2 DERIVED >tle range k_ep_nr NULL 24544 Using where; Using index for group-‐by
1
© 2014 PERCONA
Visual Explain
1
© 2014 PERCONA
Status: the Derived-Table Solution +----------------------------+--------+!| Variable_name | Value |!+----------------------------+--------+!| Handler_commit | 1 |!| Handler_delete | 0 |!| Handler_discover | 0 |!| Handler_external_lock | 6 |!| Handler_mrr_init | 0 |!| Handler_prepare | 0 |!| Handler_read_first | 0 |!| Handler_read_key | 110312 |!| Handler_read_last | 1 |!| Handler_read_next | 28989 |!| Handler_read_prev | 0 |!| Handler_read_rnd | 0 |!| Handler_read_rnd_next | 30324 |!| Handler_rollback | 0 |!| Handler_savepoint | 0 |!| Handler_savepoint_rollback | 0 |!| Handler_update | 0 |!| Handler_write | 30323 |!+----------------------------+--------+!
Evidence of a temporary table, even though EXPLAIN didn’t report it
1
© 2014 PERCONA
Profile: the Derived-Table Solution +----------------------+----------+!| Status | Duration |!+----------------------+----------+!| starting | 0.000600 |!| checking permissions | 0.000013 |!| checking permissions | 0.000007 |!| checking permissions | 0.000009 |!| Opening tables | 0.000111 |!| init | 0.000037 |!| System lock | 0.000027 |!| optimizing | 0.000006 |!| optimizing | 0.000011 |!| statistics | 0.000196 |!| preparing | 0.000030 |!| Sorting result | 0.000012 |!| statistics | 0.000057 |!| preparing | 0.000023 |!| executing | 0.000016 |!| Sending data | 0.000010 |!
| executing | 0.000004 |!| Sending data | 0.607434 |!| end | 0.000020 |!| query end | 0.000028 |!| closing tables | 0.000005 |!| removing tmp table | 0.000014 |!| closing tables | 0.000070 |!| freeing items | 0.000235 |!| cleaning up | 0.000029 |!+----------------------+----------+!
Evidence of a temporary table, even though EXPLAIN didn’t report it
1
© 2014 PERCONA
Summary: Greatest per Group Solu7on Time Notes Exclusion-‐join solu>on 71.20 Bad when each group has
many entries. Derived-‐table solu>on 0.60s
1
© 2014 PERCONA
DYNAMIC PIVOT Query Patterns
1
© 2014 PERCONA
Assignment:
“I want the count of movies, TV, and video games per year—in columns.”
1
© 2014 PERCONA
Not Like This SELECT k.kind, t.production_year, COUNT(*) AS Count!FROM kind_type k!JOIN title t ON k.id = t.kind_id !WHERE production_year BETWEEN 2005 AND 2009!GROUP BY k.id, t.production_year;!!+-------------+-----------------+--------+!| kind | production_year | Count |!+-------------+-----------------+--------+!| movie | 2005 | 13807 |!| movie | 2006 | 13916 |!| movie | 2007 | 14494 |!| movie | 2008 | 18354 |!| movie | 2009 | 23714 |!| tv series | 2005 | 3248 |!| tv series | 2006 | 3588 |!| tv series | 2007 | 3361 |!| tv series | 2008 | 3026 |!| tv series | 2009 | 2572 |!
1
© 2014 PERCONA
Like This +----------------+-----------+-----------+-----------+-----------+-----------+!| kind | Count2005 | Count2006 | Count2007 | Count2008 | Count2009 |!+----------------+-----------+-----------+-----------+-----------+-----------+!| episode | 36138 | 24745 | 22335 | 16448 | 12917 |!| movie | 13807 | 13916 | 14494 | 18354 | 23714 |!| tv movie | 3541 | 3561 | 3586 | 3025 | 2778 |!| tv series | 3248 | 3588 | 3361 | 3026 | 2572 |!| video game | 383 | 367 | 310 | 300 | 215 |!| video movie | 7693 | 7671 | 6955 | 5808 | 4090 |!+----------------+-----------+-----------+-----------+-----------+-----------+!
1
© 2014 PERCONA
Do It in One Pass SELECT k.kind,! SUM(production_year=2005) AS Count2005, ! SUM(production_year=2006) AS Count2006, ! SUM(production_year=2007) AS Count2007, ! SUM(production_year=2008) AS Count2008, ! SUM(production_year=2009) AS Count2009 !FROM title t!JOIN kind_type k ON k.id = t.kind_id!GROUP BY t.kind_id ORDER BY NULL;!!
0.77s
SUM of 1’s = COUNT where condition is true
1
© 2014 PERCONA
Indexes: the One-Pass Solution CREATE INDEX k_py !ON title (kind_id, production_year);!
1
© 2014 PERCONA
EXPLAIN: the One-Pass Solution id select_type table type key ref rows Extra
1 SIMPLE k index kind NULL 7 Using index; Using temporary
1 SIMPLE t ref k_py k.id 167056 Using index
reading title table second unfortunately causes the group by to create a temp table
1
© 2014 PERCONA
Visual Explain
1
© 2014 PERCONA
Status: the One-Pass Solution +----------------------------+---------+!| Variable_name | Value |!+----------------------------+---------+!| Handler_commit | 1 |!| Handler_delete | 0 |!| Handler_discover | 0 |!| Handler_external_lock | 4 |!| Handler_mrr_init | 0 |!| Handler_prepare | 0 |!| Handler_read_first | 1 |!| Handler_read_key | 1543727 |!| Handler_read_last | 0 |!| Handler_read_next | 1543726 |!| Handler_read_prev | 0 |!| Handler_read_rnd | 0 |!| Handler_read_rnd_next | 7 |!| Handler_rollback | 0 |!| Handler_savepoint | 0 |!| Handler_savepoint_rollback | 0 |!| Handler_update | 1543713 |!| Handler_write | 6 |!+----------------------------+---------+!
title table has 1.5M rows; that’s how many times it increments counts in the temp table
1
© 2014 PERCONA
Profile: the One-Pass Solution +----------------------+----------+!| Status | Duration |!+----------------------+----------+!| starting | 0.000162 |!| checking permissions | 0.000034 |!| checking permissions | 0.000011 |!| Opening tables | 0.000043 |!| init | 0.000052 |!| System lock | 0.000030 |!| optimizing | 0.000017 |!| statistics | 0.000112 |!| preparing | 0.000025 |!| Creating tmp table | 0.000069 |!| executing | 0.000009 |!| Sending data | 0.772317 |!| end | 0.000025 |!| query end | 0.000022 |!| removing tmp table | 0.000036 |!| query end | 0.000007 |!
| closing tables | 0.000018 |!| freeing items | 0.000485 |!| cleaning up | 0.000030 |!+----------------------+----------+!
majority of time spent building temp table
1
© 2014 PERCONA
One-Pass with Straight-Join Optimizer Override
SELECT STRAIGHT_JOIN k.kind,! SUM(production_year=2005) AS Count2005, ! SUM(production_year=2006) AS Count2006, ! SUM(production_year=2007) AS Count2007, ! SUM(production_year=2008) AS Count2008, ! SUM(production_year=2009) AS Count2009 !FROM title t!JOIN kind_type k ON k.id = t.kind_id!GROUP BY t.kind_id ORDER BY NULL;!!
7.18s
1
© 2014 PERCONA
Indexes: the Straight-Join Solution CREATE INDEX k_py !ON title (kind_id, production_year);!
1
© 2014 PERCONA
EXPLAIN: the Straight-Join Solution id select_type table type key ref rows Extra
1 SIMPLE t index k_py NULL 1537429 Using index
1 SIMPLE k eq_ref PRIMARY t.kind_id 1
no "Using temporary" because forcing title table to be read first means it scans the index in index order, avoiding the temp table—in this case
1
© 2014 PERCONA
Visual Explain
1
© 2014 PERCONA
Status: the Straight-Join Solution +----------------------------+---------+!| Variable_name | Value |!+----------------------------+---------+!| Handler_commit | 1 |!| Handler_delete | 0 |!| Handler_discover | 0 |!| Handler_external_lock | 4 |!| Handler_mrr_init | 0 |!| Handler_prepare | 0 |!| Handler_read_first | 1 |!| Handler_read_key | 7 |!| Handler_read_last | 0 |!| Handler_read_next | 1543719 |!| Handler_read_prev | 0 |!| Handler_read_rnd | 0 |!| Handler_read_rnd_next | 0 |!| Handler_rollback | 0 |!| Handler_savepoint | 0 |!| Handler_savepoint_rollback | 0 |!| Handler_update | 0 |!| Handler_write | 0 |!+----------------------------+---------+!!
really one-pass
1
© 2014 PERCONA
Profile: the Straight-Join Solution +----------------------+----------+!| Status | Duration |!+----------------------+----------+!| starting | 0.000172 |!| checking permissions | 0.000176 |!| checking permissions | 0.000012 |!| Opening tables | 0.000169 |!| init | 0.000067 |!| System lock | 0.000034 |!| optimizing | 0.000018 |!| statistics | 0.000055 |!| preparing | 0.000036 |!| Sorting result | 0.000016 |!| executing | 0.000007 |!| Sending data | 7.277106 |!| end | 0.000022 |!| query end | 0.000024 |!| closing tables | 0.000019 |!| freeing items | 0.000236 |!
| cleaning up | 0.000026 |!+----------------------+----------+!
majority of time spent just moving rows
no temporary table!
1
© 2014 PERCONA
Scalar Subquery Solution SELECT k.kind,!(SELECT COUNT(*) FROM title WHERE kind_id = k.id AND production_year = 2005) AS Count2005,!(SELECT COUNT(*) FROM title WHERE kind_id = k.id AND production_year = 2006) AS Count2006,!(SELECT COUNT(*) FROM title WHERE kind_id = k.id AND production_year = 2007) AS Count2007,!
(SELECT COUNT(*) FROM title WHERE kind_id = k.id AND production_year = 2008) AS Count2008,!(SELECT COUNT(*) FROM title WHERE kind_id = k.id AND production_year = 2009) AS Count2009!FROM kind_type k;!
0.001
1
© 2014 PERCONA
Indexes: the Scalar Subquery Solution CREATE INDEX k_py !ON title (kind_id, production_year)!!CREATE UNIQUE INDEX kind !ON kind_type (kind);!
1
© 2014 PERCONA
EXPLAIN: the Scalar Subquery Solution id select_type table type key ref rows Extra
1 PRIMARY k index kind NULL 7 Using index
6 DEPENDENT SUBQUERY
>tle ref k_py k.id, const
1781 Using index
5 DEPENDENT SUBQUERY
>tle ref k_py k.id, const
1781 Using index
4 DEPENDENT SUBQUERY
>tle ref k_py k.id, const
1781 Using index
3 DEPENDENT SUBQUERY
>tle ref k_py k.id, const
1781 Using index
2 DEPENDENT SUBQUERY
>tle ref k_py k.id, const
1781 Using index
1
© 2014 PERCONA
Visual Explain
1
© 2014 PERCONA
Status: the Scalar Subquery Solution +----------------------------+--------+!| Variable_name | Value |!+----------------------------+--------+!| Handler_commit | 1 |!| Handler_delete | 0 |!| Handler_discover | 0 |!| Handler_external_lock | 12 |!| Handler_mrr_init | 0 |!| Handler_prepare | 0 |!| Handler_read_first | 1 |!| Handler_read_key | 36 |!| Handler_read_last | 0 |!| Handler_read_next | 262953 |!| Handler_read_prev | 0 |!| Handler_read_rnd | 0 |!| Handler_read_rnd_next | 0 |!| Handler_rollback | 0 |!| Handler_savepoint | 0 |!| Handler_savepoint_rollback | 0 |!| Handler_update | 0 |!| Handler_write | 0 |!+----------------------------+--------+!
good use of indexes
1
© 2014 PERCONA
Profile: the Scalar Subquery Solution +----------------------+----------+!| Status | Duration |!+----------------------+----------+!| checking permissions | 0.000005 |!| checking permissions | 0.000010 |!| Opening tables | 0.000143 |!| init | 0.000071 |!| System lock | 0.000047 |!| optimizing | 0.000007 |!| statistics | 0.000022 |!| preparing | 0.000020 |!| executing | 0.000005 |!| Sending data | 0.000620 |!. . .!| executing | 0.000013 |!| Sending data | 0.001620 |!| executing | 0.000008 |!| Sending data | 0.000978 |!| executing | 0.000010 |!| Sending data | 0.000384 |!| end | 0.000011 |!| query end | 0.000027 |!| closing tables | 0.000019 |!| freeing items | 0.000460 |!| cleaning up | 0.000030 |!+----------------------+----------+!
scary long profile, but still fast
1
© 2014 PERCONA
Summary: Dynamic Pivot Solu7on Time Notes One-‐pass solu>on 0.77s Straight-‐join solu>on 7.18s was much bejer in 5.5 Scalar Subquery solu>on 0.001s
1
© 2014 PERCONA
RELATIONAL DIVISION Query Patterns
1
© 2014 PERCONA
Assignment:
“I want to see movies with all three of keywords espionage, nuclear-bomb,
and ejector-seat.”
1
© 2014 PERCONA
Not Movies with One Keyword SELECT t.title, k.keyword FROM keyword k !JOIN movie_keyword mk ON k.id = mk.keyword_id !JOIN title t ON mk.movie_id = t.id !
WHERE k.keyword IN ('espionage', 'nuclear-bomb', 'ejector-seat');!!+--------------------------+--------------+!| title | keyword |!
+--------------------------+--------------+!| 2 Fast 2 Furious | ejector-seat |!| Across the Pacific | espionage |!| Action in Arabia | espionage |!
. . .!| You Only Live Twice | espionage |!| Zombie Genocide | nuclear-bomb |!
| Zombies of the Strat | espionage |!+--------------------------+--------------+!705 rows in set (12.97 sec)!
1
© 2014 PERCONA
This Won’t Work SELECT t.title, k.keyword !FROM keyword k !JOIN movie_keyword mk ON k.id = mk.keyword_id !
JOIN title t ON mk.movie_id = t.id !WHERE k.keyword = 'espionage'! AND k.keyword = 'nuclear-bomb'! AND k.keyword = 'ejector-seat';!!0 rows in set (12.97 sec)!
!
It’s impossible for one column to have three values on a given row
1
© 2014 PERCONA
Only Movies with All Three +------------+-------------------------------------+!| title | keywords |!+------------+-------------------------------------+!| Goldfinger | ejector-seat,espionage,nuclear-bomb |!+------------+-------------------------------------+!
1
© 2014 PERCONA
Group-by Solution SELECT t.title, GROUP_CONCAT(k.keyword) AS keywords!FROM title t !JOIN movie_keyword mk ON t.id = mk.movie_id !JOIN keyword k ON k.id = mk.keyword_id!WHERE k.keyword IN ! ('espionage', 'nuclear-bomb', 'ejector-seat')!GROUP BY mk.movie_id!HAVING COUNT(DISTINCT mk.keyword_id) = 3 !ORDER BY NULL;!!
0.02s
1
© 2014 PERCONA
Indexes CREATE INDEX k_i!ON keyword (keyword, id);!!CREATE INDEX k_m!ON movie_keyword (keyword_id, movie_id);!
1
© 2014 PERCONA
EXPLAIN: the Group-by Solution id select_type table type key ref rows Extra
1 SIMPLE k range k_i NULL 3 Using where; Using index; Using temporary; Using filesort
1 SIMPLE mk ref k_m k.id 23 Using index
1 SIMPLE t eq_ref PRIMARY mk.movie_id 1 NULL
1
© 2014 PERCONA
Visual Explain
1
© 2014 PERCONA
Status: the Group-by Solution +----------------------------+-------+!| Variable_name | Value |!+----------------------------+-------+!| Handler_commit | 1 |!| Handler_delete | 0 |!| Handler_discover | 0 |!| Handler_external_lock | 6 |!| Handler_mrr_init | 0 |!| Handler_prepare | 0 |!| Handler_read_first | 0 |!| Handler_read_key | 710 |!| Handler_read_last | 0 |!| Handler_read_next | 708 |!| Handler_read_prev | 0 |!| Handler_read_rnd | 705 |!| Handler_read_rnd_next | 706 |!| Handler_rollback | 0 |!| Handler_savepoint | 0 |!| Handler_savepoint_rollback | 0 |!| Handler_update | 0 |!| Handler_write | 705 |!+----------------------------+-------+!
building and reading a temporary table
1
© 2014 PERCONA
Profile: the Group-by Solution +----------------------+----------+!| Status | Duration |!+----------------------+----------+!| starting | 0.000267 |!
| checking permissions | 0.000010 |!| checking permissions | 0.000004 |!| checking permissions | 0.000008 |!
| Opening tables | 0.000041 |!| init | 0.000078 |!| System lock | 0.000037 |!| optimizing | 0.000024 |!
| statistics | 0.000166 |!| preparing | 0.000028 |!| Creating tmp table | 0.000199 |!
| Sorting result | 0.000013 |!| executing | 0.000005 |!| Sending data | 0.009532 |!
| Creating sort index | 0.010310 |!| end | 0.000066 |!| query end | 0.000027 |!| removing tmp table | 0.000176 |!
| query end | 0.000021 |!| removing tmp table | 0.000010 |!| query end | 0.000006 |!
| closing tables | 0.000022 |!| freeing items | 0.000016 |!| removing tmp table | 0.000007 |!| freeing items | 0.000009 |!
| removing tmp table | 0.000005 |!| freeing items | 0.000449 |!| cleaning up | 0.000028 |!
+----------------------+----------+!
building & tearing down temp table
1
© 2014 PERCONA
Self-Join Solution SELECT t.title, CONCAT_WS(',', k1.keyword, k2.keyword, k3.keyword) AS keywords !FROM title t !JOIN movie_keyword mk1 ON t.id = mk1.movie_id !JOIN keyword k1 ON k1.id = mk1.keyword_id !JOIN movie_keyword mk2 ON mk1.movie_id= mk2.movie_id !JOIN keyword k2 ON k2.id = mk2.keyword_id !JOIN movie_keyword mk3 ON mk1.movie_id = mk3.movie_id !JOIN keyword k3 ON k3.id = mk3.keyword_id !WHERE (k1.keyword, k2.keyword, k3.keyword) ! = ('espionage', 'nuclear-bomb', 'ejector-seat');!
0.015s
1
© 2014 PERCONA
EXPLAIN: the Self-Join Solution id select_type table type key ref rows Extra
1 SIMPLE k1 ref k_i const 1 Using index
1 SIMPLE k2 ref k_i const 1 Using index
1 SIMPLE k3 ref k_i
const 1 Using index
1 SIMPLE mk1 ref k_m k1.id 17 Using index
1 SIMPLE t eq_ref PRIMARY mk1.movie_id 1 NULL
1 SIMPLE mk2 ref k_m k2.id, mk1.movie_id
1 Using index
1 SIMPLE mk3 ref k_m k3.id, mk1.movie_id
1 Using index
1
© 2014 PERCONA
Visual Explain
1
© 2014 PERCONA
Status: the Self-Join Solution +----------------------------+-------+!| Variable_name | Value |!+----------------------------+-------+!| Handler_commit | 1 |!| Handler_delete | 0 |!| Handler_discover | 0 |!| Handler_external_lock | 14 |!| Handler_mrr_init | 0 |!| Handler_prepare | 0 |!| Handler_read_first | 0 |!| Handler_read_key | 1218 |!| Handler_read_last | 0 |!| Handler_read_next | 613 |!| Handler_read_prev | 0 |!| Handler_read_rnd | 0 |!| Handler_read_rnd_next | 0 |!| Handler_rollback | 0 |!| Handler_savepoint | 0 |!| Handler_savepoint_rollback | 0 |!| Handler_update | 0 |!| Handler_write | 0 |!+----------------------------+-------+!
minimal rows, good index usage
1
© 2014 PERCONA
Profile: the Self-Join Solution +----------------------+----------+!| Status | Duration |!+----------------------+----------+!| starting | 0.000205 |!| checking permissions | 0.000011 |!| checking permissions | 0.000006 |!| checking permissions | 0.000005 |!| checking permissions | 0.000005 |!| checking permissions | 0.000005 |!| checking permissions | 0.000005 |!| checking permissions | 0.000010 |!| Opening tables | 0.000061 |!| init | 0.000069 |!| System lock | 0.000056 |!| optimizing | 0.000029 |!| statistics | 0.000106 |!| preparing | 0.000049 |!| executing | 0.000008 |!| Sending data | 0.013485 |!
| end | 0.000018 |!| query end | 0.000022 |!| closing tables | 0.000020 |!| freeing items | 0.000683 |!| cleaning up | 0.000034 |!+----------------------+----------+!
who says joins are slow?
1
© 2014 PERCONA
Summary: Relational Division Solu7on Time Notes Group-‐by solu>on 0.02s much improved in 5.7 Self-‐join solu>on 0.015s
1
© 2014 PERCONA
PERFORMANCE SCHEMA Query Patterns
1
© 2014 PERCONA
Performance Schema • New tools to instrument and profile queries.
– Use PERFORMANCE_SCHEMA with MySQL 5.6+
• SHOW PROFILES is deprecated in 5.6.7+ and will be removed in a future release
1
© 2014 PERCONA
Sys Schema • Get the MySQL-sys schema for handy views,
functions, and procedures. – https://github.com/MarkLeith/mysql-sys
1
© 2014 PERCONA
Sys Schema Caveats • However, it’s still incomplete and tricky…
– Installation fails; the views reference some P_S tables that don’t exist yet as of MySQL 5.7.5 (e.g. memory_%).
– Documentation includes at least one sys procedure that isn’t in the download (ps_trace_statement_digest()).
• We all need to use it and give feedback to Mark!
1
© 2014 PERCONA
Sys Schema Setup mysql> SOURCE sys_57.sql;!!mysql> CALL sys.ps_setup_enable_instrument('stage/%');!
+-------------------------+!| summary |!+-------------------------+!| Enabled 108 instruments |!+-------------------------+!!
mysql> CALL sys.ps_truncate_all_tables(false);!+---------------------+!| summary |!+---------------------+!| Truncated 31 tables |!+---------------------+!
!
1
© 2014 PERCONA
Statement Analysis mysql> select * from sys.statement_analysis limit 1\G!*************************** 1. row ***************************! query: SELECT `t` . `title` FROM `ti ... id` AND `c` . `role_id` = ? ) ! db: imdb! full_scan: ! exec_count: 1! err_count: 0! warn_count: 0! total_latency: 2.27 s! max_latency: 2.27 s! avg_latency: 2.27 s! lock_latency: 296.00 us! rows_sent: 6056! rows_sent_avg: 6056! rows_examined: 180432!rows_examined_avg: 180432! tmp_tables: 0! tmp_disk_tables: 0! rows_sorted: 0!sort_merge_passes: 0! digest: 4f8e3695ecf7c8518a1e1defe2ff323c! first_seen: 2014-09-28 02:21:21! last_seen: 2014-09-28 02:21:21!
similar to pt-query-digest output with Percona’s verbose slow query log
1
© 2014 PERCONA
SHOW PROFILE Workalike mysql> select * from sys.x$user_summary_by_stages;!+------+--------------------------------+-------+---------------+-----------+!| user | event_name | total | wait_sum | wait_avg |!+------+--------------------------------+-------+---------------+-----------+!| root | stage/sql/Sending data | 93246 | 2053496638000 | 22022000 |!| root | stage/sql/executing | 93247 | 214310855000 | 2298000 |!| root | stage/sql/System lock | 27 | 6200477000 | 229647000 |!| root | stage/sql/Opening tables | 126 | 3940994000 | 31277000 |!| root | stage/sql/checking permissions | 31 | 2309620000 | 74503000 |!| root | stage/sql/closing tables | 126 | 697965000 | 5539000 |!| root | stage/sql/statistics | 3 | 452872000 | 150957000 |!| root | stage/sql/freeing items | 2 | 421142000 | 210571000 |!| root | stage/sql/query end | 126 | 384577000 | 3052000 |!| root | stage/sql/starting | 2 | 271533000 | 135766000 |!| root | stage/sql/preparing | 3 | 119699000 | 39899000 |!| root | stage/sql/init | 3 | 82758000 | 27586000 |!| root | stage/sql/optimizing | 4 | 49758000 | 12439000 |!| root | stage/sql/removing tmp table | 1 | 8568000 | 8568000 |!| root | stage/sql/cleaning up | 2 | 7443000 | 3721000 |!| root | stage/sql/Sorting result | 1 | 7039000 | 7039000 |!| root | stage/sql/end | 2 | 6986000 | 3493000 |!+------+--------------------------------+-------+---------------+-----------+!
totals for all queries by user, not just in current session
1
© 2014 PERCONA
CONCLUSIONS Query Patterns
1
© 2014 PERCONA
Conclusions • Use all tools to measure query performance
– EXPLAIN – Session Status – Query Profiler – Performance Schema and Sys Schema
• Test with real-world data, because the best solution depends on the volume of data you’re querying.
• Allocate enough memory to buffers so the indexes you need stay resident in RAM.
1
© 2014 PERCONA
License and Copyright Copyright 2014 Percona
Released under a Creative Commons 3.0 License: http://creativecommons.org/licenses/by-nc-nd/3.0/
You are free to share—to copy, distribute and transmit this work, under the following conditions:
Attribution. ���You must attribute this work to
Percona
Noncommercial. ���You may not use this work for
commercial purposes.
No Derivative Works. ���You may not alter, transform, or build
upon this work.