explain explain

Explaining EXPLAIN: Queries and what happens when you execute them.

Kristian Köhntopp

kk@php.net

What is an index?

Library Example

How to find data… in a library

• Assume the following: • We have one million books in no particular order. • Task #1: Find all books written by Larry Wall. • Task #2: Do we have any books by Larry Wall at all?

• How many books do you need to touch to answer?

4

How to find data… fast.

Logarithmic complexity

• 2x = n,for n = 1 000, 1 000 000, 1 000 000 000 books

• 210 = 1024 ~ 1 000 (Kilo) • 220 ~ 1 000 000 (Mega) • 230 ~ 1 000 000 000 (Giga)

• x = log2(n) = ln(n)/ln(2) lookups to a sorted author file.

6

The Index…7

.MYD file

RecordRecordRecordRecordRecordRecordRecorddeletedRecordRecorddeletedRecord

.MYI File

Index 1 Index 2

Logarithmic complexity

• One Index Block = Fanout > 2

• size(block)/size(index record) • e.g. 16K/32 = 512

• Fanout 50-500

8

.MYD file

RecordRecordRecordRecordRecordRecordRecorddeletedRecordRecorddeletedRecord

.MYI File

Index 1 Index 2

How many lookups to find anything in a database?

• Bad case:x = ln(1 000 000 000)/l(50)

• = 5.2 • Good case:

x = ln(1 000 000)/l(500) • = 2.2

9

How many lookups to find anything in a database?

• Bad case:x = ln(1 000 000 000)/l(50)

• = 5.2 • Good case:

x = ln(1 000 000)/l(500) • = 2.2

9

~4 is not a bad estimation in many cases

The number of disk accesses to find anything through an index (no caching)

3-410

Going to the disk

Access times12

Access times13

Access times13

Access times

• Disk • Linear read/write: 100-200 MB/s • 200k - 400k records/s, 500 Byte records

• Random read/write: 100-200 IOPS (5-10ms) • Yes, that’s about million times slower than RAM.

14

Access times

• SSD • Linear read/write: 200-400 MB/s • 400k - 800k records/s, 500 Byte records

• Random read/write: 10k-20k IOPS

15

Fast queries are aboutnot going to the disk.

16

Missing indexes?

Am I using enough indexes?18

Am I using enough indexes?18

Bad

Am I using enough indexes?

• show global status like ‘handler_%’;

19

Am I using enough indexes?

• show global status like ‘handler_%’;

19

Index

Scan

Am I using enough indexes?

• Handler_read_first + Handler_read_key + Handler_read_next + Handler_read_prev

• Handler_read_rnd + Handler_read_rnd_next

20

Index

Scan

An index is missing…

• … and data cannot be kept in memory: • Plenty of disk read activity • “Slow queries”

• … and data can be kept in memory: • High CPU, maybe not even slow query log entries

21

sys Schema

PERFORMANCE_SCHEMA (“P_S”)

• P_S appeared in 5.5, useful since 5.6 • Wait-Free, Allocation-Free Interface

• “rather be inaccurate or incomplete than slow down the server”

• Hard to read units, hard to decode references • Query Interface (“select * from …”)

23

sys

• http://github.com/MarkLeith/mysql-sys • started out as ps_helper, renamed “sys” • Part of a standard install since 5.7.7

• Series of views on p_s, installed in sys.* • additional stored procedures to handle p_s config

24

sys

• All views as name and x$name • name for human consumption • useful units and formatting

• x$name for programs • same data, no units and formatting

25

sys

• Am I using enough indexes? • schema_tables_with_full_table_scans • statements_with_full_table_scans

26

Running Queries…

SELECT * FROM a JOIN b ON a.aid = b.aid WHERE <acond> AND <bcond>

• MySQL always uses “nested loop join” • for each row in a matching <acond>

• look up aid in b, using b.aid index, • filter more, using <bcond>, …

• If you can use indices: • Fast and memory efficient, streamable

• Otherwise: Horrible.

28

EXPLAIN output

• Fixed plan: “Nested Loop Join” • Variables:

• Table order • Lookup methods

29

EXPLAIN this…

• EXPLAIN SELECT …FROM B_Rate_Room_Directory brrdJOIN B_CP_PolicyGroupType pgt ON (pg.pg_type_id = pgt.pg_type_id )

JOIN B_CP_PolicyGroup pg ON ( pg.object_id = brrd.default_policy_group_id and pg.eff_status='C' )

JOIN Translation t ON (t.item_id = pg.pg_type_id and t.language = 'en' and t.groupname='policy_group_name' )

WHERE brrd.active <> 0 and brrd.room_id = '25725602' and brrd.default_policy_group_id IS NOT NULL and brrd.default_policy_group_id <> 0

30

EXPLAIN this…31

EXPLAIN this…31

all simple joins, no unions or subselects.

EXPLAIN this…31

Join order, may be different

from order of tables as written in the SELECT

EXPLAIN this…31

Connection between tables

EXPLAIN this…31

Keys that could be used to resolve

the query

EXPLAIN this…31

Actual key chosen and the length of the prefix in bytes

EXPLAIN this…31

Badness (2x1x1314x1) number of lookup

operations

EXPLAIN this…31

Additional messages about query resolution

EXPLAIN this…31

Join Types

• The type can be:

• system, const - table has at most one matching row.The optimizer replaces the table with the value.

• eq_ref - compare using '=' operator, using a 1:1 relationship.

• ref, ref_or_null - using an index, but not on a 1:1 basis.

• fulltext - using a MyISAM FULLTEXT index.

• index_merge - using two (or more) indexes, intersecting the two partial results.

32

Join Types

• The type can be:

• range - using a part of the index to select values. SELECT … FROM t WHERE a < …SELECT … FROM t WHERE id IN (…, …, …)SELECT … FROM t WHERE v BETWEEN … AND …

• index - A full scan on an index. possible_keys is NULL, keys is not.

• ALL - A full table scan.

33

An unusual example

• select translation_id, translation, statusfrom B_CP_Translationwhere language_code = 'fr' and project_id = '0'

34

No compound index,

two individual indices

An unusual example• Explain Result:

id: 1select_type: SIMPLEtable: B_CP_Translationtype: index_mergepossible_keys: language_code,project_id key: language_code,project_id key_len: 6,4ref: NULLrows: 734Extra: Using intersect(language_code,project_id); Using where

35

Extra: Temporary tables

• Things that are not so good: • using filesort - extra sorting is necessary, rows not retrieved in

requested order.This can force an extra lookup, or a table materialization (using temporary also shown).

• using temporary - the result is pushed into a temp table at some point, and then processed further. The temp table can be in memory or go to disk.

36

Extra: Temporary Tables37

10:1 or better

Extra: Temporary Tables

• sys query: • select *

from sys.statements_with_temp_tables where db not in ('sys', 'mysql', 'information_schema', ‘performance_schema');

38

Other extra notices

• Not so bad news:

• Impossible WHERE noticed after reading const tables - there is no work left to do.

• Select tables optimzied away - the query contained aggregates that could be resolved at the optimizer stage already. There is no work left to do.

• Using index - Covering index, data need not be read.

• Distinct - implementing 'distinct' operator using a lose index scan.

• Using where - a where clause is being used to filter down a result from an index even more.

39

When and how to index

When to index

• Things to remember:

• The point of an index: Less disk operations. • Rows are on disk in pages, data is always read in pages. • If a wanted row is in a page, all of the page is read.

41

When to index

• Selectivity: • The percentage of rows selected by a condition.

• SELECT id, name FROM Person WHERE sex = ‘f' • Cardinality: 2 (or 3, when nullable) • Assumption: Equidistribution. Is it valid for our data?

42

INDEX(a,b,c)

• Compound index on three columns. • Used left to right,

for many conjunction (AND) of equalities/eq_ref/ref and one trailing relatation/range

• One point or subtree in the index.

43

INDEX(a,b,c)

• WHERE a = 10 AND b = 20 AND c = 30 • uses (a,b,c)

• WHERE a = 10 AND b = 20 ORDER BY c • uses (a,b,c)

• WHERE a = 10 AND b < 20 ORDER BY c • uses (a,b)

44

INDEX(a,b,c)

• WHERE a<10 AND b = 10 • uses (a)

• WHERE a<10 AND b<10 • uses (a)

• WHERE (a-10)<0 • no index used

45

Are we done, yet?

Is my query optimized enough?47

Remember this badness calculation?It is supposed to be the number of lookups.

How many lookups are minimal?

Is my query optimized enough?48

2 x 1 x 1314 x 1 ~ 2600 Lookups

Raw (before grouping, limit) result set size?

2600 lookups for 2600 result rows are ok. 2600 lookups for 26 result rows probably have potential for optimization.