top five mysql query tuning tips - percona · top five mysql query tuning tips janis griffin,...

Top Five MySQL Query Tuning TipsJanis Griffin, Database Performance Evangelist

• Senior DBA / Performance Evangelist for SolarWinds

• Janis.Griffin@solarwindscom

• Twitter® - @DoBoutAnything

• Current – 25+ Years in Oracle®, SQL Server®, ASE, and now MySQL®

• DBA and Developer

• Specialize in Performance Tuning

• Review Database Performance for Customers and Prospects

• Common Question – How do I tune it?

© 2017 SolarWinds Worldwide, LLC. All rights reserved.

Who Am I

mailto:Janis.Griffin@solarwindscom

• Challenges Of Tuning

• Top Five Tuning Tips

• Monitor Wait Time

• Find the right SQL statements to work on

• Get Baseline Metrics

• Review the Execution Plan

• Know which Optimizer Features are being used

• Gather Object Information

• Review Table, Column, Index, and Constraint Information

• Understand Column Selectivity, and Statistics

• Find the Driving Table

• Consider SQL Diagramming

• Engineer out the Stupid


Agenda

• SQL Tuning is Hard

• Who should tune – DBA or Developer?

• Which SQL to tune

• Requires Expertise in Many Areas

• Technical – Plan, Data Access, SQL Design

• Business – What is the Purpose of SQL?

• Tuning Takes Time

• Large Number of SQL Statements

• Each Statement is Different

• Low Priority in Some Companies

• Vendor Applications

• Focus on Hardware or System Issues

• Never Ending


Challenges Of Tuning


Tuning Query Performance

Consider these tips as

building blocks for your

‘Tuning’ toolbox.

• Understand the total time a Query spends in Database

• Measure time while Query executes

• MySQL helps by providing Wait Events and Thread States


1. Monitor Wait Time

Focus on Total Wait Time

• Greatly improved.

• More information / more instrumentation

• Current / historical events at all levels

• Statements

• Stages

• Waits

• Performance_Schema consumers now ON by default

• Storage Engine now defaults to INNODB

• 5.7 Performance_Schema

• 35 new tables – 5.7 has 87, 5.6 had 52, 5.5 had 17

• Reduced overhead and footprint

• 1005 Instruments (5.7.10)

• New ones for transactions, metadata locks, memory usage

• SYS Schema – now provided by default

• 100 views – User, Host, IO, Innodb, Memory Summaries

• Wait analysis and statement level


5.6+ Performance_Schema


MySQL Wait Time Data

Information_Schema

INSERT INTO wta_data SELECT current_timestamp, t.processlist_id AS conn_id, t.processlist_user AS user, t.processlist_host AS host, t.processlist_db AS db,

LOWER(t.processlist_state) AS state, t.processlist_info AS current_sql, MD5(t.processlist_info) AS current_sql_md5,

t.processlist_command AS command, w.event_id AS wait_event_id, w.end_event_id AS wait_event_end_id,

w.event_name AS wait_event_name, w.operation AS wait_operation, w.object_schema AS wait_schema, w.object_name AS wait_object_name,

w.object_type AS wait_type, w.index_name AS wait_index, s.event_name AS statement_event, s.sql_text AS statement_sql,

MD5(s.sql_text) AS statement_sql_md5, CASE WHEN RIGHT(s.digest_text, 3) = '...' THEN 1 ELSE 0 END AS statement_digest_truncated,

s.digest AS statement_digest, s.event_id AS statement_event_id, s.end_event_id AS statement_event_end_id,

conn.attr_value AS program_name, trx.trx_state, trx.trx_operation_state, trx.blocking_id

FROM performance_schema.threads AS t LEFT JOIN performance_schema.events_waits_current AS w ON w.thread_id = t.thread_id AND w.end_event_id IS NULL AND w.event_name <> 'idle'

LEFT JOIN performance_schema.events_statements_current AS s ON s.thread_id = t.thread_id AND s.digest IS NOT NULL

LEFT JOIN (SELECT processlist_id, GROUP_CONCAT(attr_value ORDER BY attr_name DESC SEPARATOR '~^~') attr_value

FROM performance_schema.session_connect_attrs WHERE attr_name IN ('program_name','_client_name') GROUP BY processlist_id ) conn

ON t.processlist_id = conn.processlist_id

LEFT JOIN (SELECT trx.trx_mysql_thread_id processlist_id, blocking_trx.trx_mysql_thread_id blocking_id, trx.trx_state, trx.trx_operation_state

FROM information_schema.innodb_trx trx

LEFT JOIN (SELECT requesting_trx_id, MIN(blocking_trx_id) blocking_trx_id FROM information_schema.innodb_lock_waits

GROUP BY requesting_trx_id) lock_waits ON trx.trx_id=lock_waits.requesting_trx_id

LEFT OUTER JOIN information_schema.innodb_trx blocking_trx ON lock_waits.blocking_trx_id=blocking_trx.trx_id) tr ON trx.processlist_id = t.processlist_id

WHERE t.processlist_id IS NOT NULL AND t.instrumented = 'YES‘ AND t.processlist_id <> connection_id()

AND (t.processlist_info IS NOT NULL OR s.sql_text IS NOT NULL)

AND (t.PROCESSLIST_COMMAND <> 'Sleep')© 2017 SolarWinds Worldwide, LLC. All rights reserved.

Base Query – Not Rocket Science

10

select w.sql_text, w.wait_operation, w.time_in_seconds, tot.tot_time from

(select substr(current_sql, 1, 60) sql_text, wait_operation, count(*) time_in_seconds from wta_data w group by substr(current_sql, 1, 60), wait_operation) w

(select substr(current_sql, 1,60) sql_text, count(*) tot_time from wta_data group by substr(current_sql, 1,60)) tot

where w.wait_operation is not null

and w.sql_text = tot.sql_text

order by tot.tot_time, time_in_seconds;


Quickly Find Queries Spending the Most Time in DB

Wait Time & Events = Wait Time Analysis (WTA)

Almost 16 hours

spent on ‘waiting for

table metadata lock’

100% of time on

‘sending data’

Over 20 hours spent!


WTA Benefits


WTA Benefits – Blocking Issues


• Get baseline metrics.

• How long does it take now?

• What is acceptable? (Ten seconds? Two minutes? One hour?)

• Record ‘rows examined’ and ‘rows sent’ for tuning comparison

• Collect wait or thread states.

• Locking / blocking (system lock)

• I/O problem (sending data)

• Calculating statistics (statistics)

• Network slowdown (writing to net)

• May be multiple issues

• All have different resolutions


Benefits of WTA – Cont.


WTA Example

SELECT dayofweek ( payment_date ) day_of_week , staff . first_name , staff. last_name ,

city . city , country . country , category . name AS category , SUM ( payment . amount ) AS total_sales

FROM payment

INNER JOIN rental ON payment . rental_id = rental . rental_id

INNER JOIN inventory ON rental . inventory_id = inventory . inventory_id

INNER JOIN film ON inventory . film_id = film . film_id

INNER JOIN film_category ON film . film_id = film_category . film_id

INNER JOIN category ON film_category . category_id = category . category_id

INNER JOIN store ON store . store_id = inventory . store_id

INNER JOIN address ON address . address_id = store . address_id

INNER JOIN city ON city . city_id = address . city_id

INNER JOIN country ON country . country_id = city . country_id

INNER JOIN staff ON staff . staff_id = rental . staff_id

WHERE quarter ( rental_date ) = ?

AND return_date IS NOT ?

GROUP BY dayofweek ( payment_date ) , staff . first_name ,

staff . last_name , city . city , country . country , category . name

ORDER BY total_sales DESC


WTA – Baseline Metrics

• Explain

• Explain Extended

• Explain FORMAT=JSON

• Optimizer Trace 5.6.3+

• MySQL Workbench


2. Review the Execution Plan

Tip: Try to avoid using

table aliases—they don’t

translate in plan.

• http://d1kgm347lql2s5.cloudfront.net/blog/wp-content/uploads/explain-diagram1.pdf


EXPLAIN Cheat Sheet

http://d1kgm347lql2s5.cloudfront.net/blog/wp-content/uploads/explain-diagram1.pdf

• Shows optimizer transformation of query

• Can help you write better SQL © 2017 SolarWinds Worldwide, LLC. All rights reserved.

EXPLAIN EXTENDED Example

• JSON format shows extended and partition information


EXPLAIN FORMAT=JSON


OPTIMIZER_TRACE

• Stored in information_schema.optimizer_trace

• Default is 16k of Memory

• set optimizer_trace_max_mem_size=1000000;

• set optimizer_trace_features="greedy_search=OFF";

• Not as verbose


MySQL Workbench

• Find Expensive Operators

• Examine IO costs, rows examined vs. rows sent

• Look for full table or clustered index scans

• Review the Predicate Information

• Know how parameters are being interpreted

• Review the data types

• Implicit conversions?

• Know which step filtering predicate is applied

• Review Temporary and Filesort activities in Extra column

• Try to fix them


Examine the Execution Plan

• Table Definition

• Is it really a table or is it a view?

• Get size of tables

• Quick way: mysqlshow –status database {table} {column}

• Examine Columns in Where Clause

• Cardinality of columns / data skew

• Review the ‘Selected’ Columns

• Use of ‘*’ or scalar functions


3. Gather Object Information

• Know Existing Indexes

• Columns those indexes contain

• If multi-column, know the left leading column

• Cardinality

• Make sure the optimizer can use the index

• Functions on indexed columns can turn off index

• Consider rewriting

• Look for implicit conversions

• Get sample parameter values

• Review existing keys and constraints

• Know Multi-Table Relationships (ERD)

• Primary key and foreign definitions

• Check and not null constraints

• Tip: Keys and constraints help create better execution plans


Review Indexes and Constraints

• No Relationships?


Review Indexes and Constraints – Cont.

• Need to know the size of the actual data sets of each step

• In Joins (Right, Left, Outer)

• What are the filtering predicates

• When is each filtering predicate applied

• Try to filter earlier rather than later

• Compare size of final result set with data examined

• Find the driving table

• To reduce rows examined


4. Find the Driving Table

Who registered yesterday for SQL Tuning?

SELECT s.fname, s.lname, r.signup_date

FROM student s

INNER JOIN registration r

ON s.student_id = r.student_id

INNER JOIN class c

ON r.class_id = c.class_id

WHERE c.name = 'SQL TUNING'

AND r.signup_date BETWEEN

DATE_SUB('@BillDate', INTERVAL 1 DAY) and '@BillDate'

AND r.cancelled = 'N';


Case Study

• “SQL Tuning” by Dan Tow

• Great book that teaches SQL Diagramming

• http://www.singingsql.com


SQL Diagramming

select count(*) from registration r

where r.signup_date BETWEEN DATE_SUB('@BillDate', INTERVAL 1 DAY) and

'@BillDate' AND r.cancelled = 'N';

4,228 / 79,981 * 100 = 5.2%

select count(0) from class where name = 'SQL TUNING'

2 / 1,000 * 100 = .2%

registration

student class

5

1

30

1

5%

.2%

ALTER TABLE registration ADD FOREIGN KEY (student_id) REFERENCES student(student_id);

ALTER TABLE registration ADD FOREIGN KEY (class_id) REFERENCES class(class_id);

SELECT table_name,

column_name,constraint_name,

referenced_table_name,referenced_column_name

FROM information_schema.key_column_usage

WHERE table_schema = 'csu' AND table_name = 'registration'

AND referenced_column_name IS NOT NULL;


Add and View Foreign Keys


Any Change?

F Keys

added

CREATE INDEX class_nm ON class(name);


Drive the Query By Class


Did We Improve It?

• Try to reduce Logical IO (rows examined)

• Adding indexes is not always the right thing to do!

• Consider Insert, Update, and Delete activity

• Covering index

CREATE INDEX cidx_registration ON registration(class_id, signup_date, cancelled);

• Include all columns in

• Reads only the index – doesn’t go back to table

• Partial index

ALTER TABLE class ADD INDEX (description(3));

• description varchar(200)

SELECT * FROM class WHERE description LIKE 'SQL%‘;


Are We Using the Best Index?

Create an index on a Virtual Column in MySQL 5.7.7

Kind of like a function index

Persists the data in an index NOT the table

SELECT signup_dayofweek, count(*)

FROM vregistration r WHERE r.cancelled = 'N‘

GROUP BY signup_dayofweek HAVING count(*) < 10000;

Consider other specialized index types

Fulltext

Spatial

Hash for memory engine

Beyond scope of this presentation


Other Types of Indexes

CREATE INDEX cidx_registration ON registration(class_id, signup_date, cancelled);


Add a Covering Index

• Can’t cover it because of range

AND r.signup_date BETWEEN

DATE_SUB('@BillDate', INTERVAL 1 DAY) and '@BillDate'


Is the Covering Index Working?


Did We Improve It?

• Look for performance inhibitors

• Cursor or row by row processing

• Parallel query processing.

• Not always bad, but…

• Percona_Alex_Rubin_increasing-slow-query-performance-with-parallel-query-execution

• Hard-coded hints

• logicalread.solarwinds.mysql-query-hints-improve-performance

• Nested views

• Also, sub-queries

• Abuse of Wild Cards (*) or No Where Clause

• Code-based SQL Generators (e.g. PHP generator, LINQ; nHibernate)

• Implicit data conversions

• Non-SARG-able / scalar functions

• Select… where upper(first_name) = ‘JANIS’


5. Engineer Out the Stupid

https://www.percona.com/blog/2014/01/07/increasing-slow-query-performance-with-parallel-query-execution/

http://logicalread.solarwinds.com/mysql-query-hints-improve-performance-mc12/#.Vwfog3qtcws


Data Types


Fix the Query


Did Performance Improve?


Another Case Study – Sales Quarterly Contest

SELECT dayofweek ( payment_date ) day_of_week , staff . first_name , staff. last_name ,


FROM payment













GROUP BY dayofweek ( payment_date ) , staff . first_name ,




WTA – Baseline Metrics

• Explain

• Explain Extended

• Explain FORMAT=JSON

• Optimizer Trace 5.6.3+

• MySQL Workbench


Review the Execution Plan

Tip: Try to avoid using

table aliases—they don’t

translate in plan.


Gather Object Information


Find the Driving Table

FILTERING PREDICATES:

select avg(cnt) from (select quarter(rental_date), count(*) cnt

from rental

where return_date is not null group by quarter(rental_date)) a;

select 125557.5000 / 8918218 * 100 = 1.4%

JOIN RESULTS EXAMPLE FOR CITY TO ADDRESS:

select avg(cnt) from (select city_id, count(*) cnt

from address group by city_id) a;

64 rows

category

inventory

rental

3

1

94

1%

film store

address

1

1

91

1

1

city

country

film_category

payment

1

510

1

2

1

5

1

64

1

1

staff1

317

• Create virtual column on rental_date

CREATE INDEX rental_qtr_idx ON rental(quarter_date, return_date);


Drive the Query by Rental

select count(*) from rental

where quarter(rental_date)

and return_date is not null

ALTER TABLE payment ADD COLUMN payment_dayofweek TINYINT

GENERATED ALWAYS AS (dayofweek(payment_date)) VIRTUAL;

CREATE INDEX pay_idx ON payment(rental_id, payment_dayofweek, amount);


Create Virtual Column and Add Covering Index for Payment

SELECT payment_dayofweek day_of_week , staff . first_name , staff. last_name ,


FROM payment













GROUP BY payment_dayofweek, staff . first_name ,



• Can’t use virtual column on Rental


Still Driving by Payment Using Index

select day_of_week, sales.first_name, sales.last_name, sales.city, sales.country, sales.category, sum(pay.amount) total_sales

from

(select rental.rental_id, rental.staff_id, staff.first_name, staff.last_name, city.city, country.country, category.name category

from rental

INNER JOIN inventory ON rental.inventory_id = inventory.inventory_id

INNER JOIN film ON inventory.film_id = film.film_id

INNER JOIN film_category ON film.film_id = film_category.film_id

INNER JOIN category ON film_category.category_id = category.category_id

INNER JOIN store ON store.store_id = inventory.store_id

INNER JOIN address ON address.address_id = store.address_id

INNER JOIN city ON city.city_id = address.city_id

INNER JOIN country ON country.country_id = city.country_id

INNER JOIN staff ON staff.staff_id = rental.staff_id

WHERE quarter_date =1

AND return_date is not null) sales

inner join

(select payment_dayofweek day_of_week, rental_id, staff_id, payment.amount from payment) pay

on pay.rental_id = sales.rental_id and pay.staff_id = sales.staff_id

group by day_of_week, sales.first_name, sales.last_name, sales.city, sales.country, sales.category

order by day_of_week, total_sales desc;


Rewrite Query?

• Using where; Using MRR; Using temporary; Using filesort

• Multi-Range Read Optimization

• Using a range scan on a secondary index can result in many random disk accesses

• when the table is large and not stored in the storage engine's cache.

• MySQL tries to reduce random disk access for range scans by first scanning the index collecting the keys for relevant rows.


Query Uses Both Virtual Columns and Indexes


Another Stupid Example


Sloppy Coding – Need to Fix Query


Changed the Code: last_name like ‘griffin%’

• There are a lot of challenges in Query Tuning

• If you remember the Top 5 Tips, they should take you a long way

• 1. Monitor Wait Time

• Look at wait events, record baseline metrics

• 2. Review the Execution Plan

• Look for expensive steps, know what’s optimizer features are supporting the plan

• 3. Gather Object Information

• For expensive objects, know what the optimizer knows

• 4. Find the Driving Table

• Consider SQL Diagramming techniques

• 5. Engineer out the Stupid


Summary

• Try Database Performance Analyzer FREE for 14 days

• Improve root cause of slow performance• Quickly identify root cause of issues that impact end-user response time

• See historical trends over days, months, and years

• Understand impact of VMware® performance

• Agentless architecture with no dependence on Oracle® Packs; installs in minutes


Resolve performance issues quickly

www.solarwinds.com/dpa-download/

http://www.solarwinds.com/dpa-download/

http://www.solarwinds.com/dpa-download/

The SolarWinds, SolarWinds & Design, Orion, and THWACK trademarks are the

exclusive property of SolarWinds Worldwide, LLC or its affiliates, are registered

with the U.S. Patent and Trademark Office, and may be registered or pending

registration in other countries. All other SolarWinds trademarks, service marks,

and logos may be common law marks or are registered or pending

registration. All other trademarks mentioned herein are used for identification

purposes only and are trademarks of (and may be registered trademarks) of their

respective companies.

top five mysql query tuning tips - percona · top five mysql query tuning tips janis griffin,...

Documents