performance schema in mysql (danil zburivsky)

Performance Schema in MySQL Danil Zburivsky

MySQL DBA and Team Lead at Pythian

Tuesday, October 23, 12

About myself• MySQL DBA and Team

Lead at Pythian

• Managing dozens of customers and thousands of MySQL servers

• http://www.pythian.com/news/author/zburivsky/

• @zburivsky

Debugging MySQL Performance

MySQL is a great database, but instrumentation sucks

SHOW ENGINE INNODB STATUS

Slow query log

OS STATSSHOW

PROCESSLIST

I think there is a problem with database...

• Identify timeframe (use trending tools)

• Collect data during incident

• Collect more data

• Collect as much data as you can!

• Try to make sense of it

• Tune settings, SQL, hardware. Did it help?

• I think there is a problem with database...

Tools that might help you

• pt-stalk (http://www.percona.com/doc/percona-toolkit/2.1/pt-stalk.html)

• innotop (http://code.google.com/p/innotop/)

• Or you write your own stuff

What do “Big Boys” have?

• Oracle: SQL-traces, kernel events timings, tons of books on performance tuning

• SQL Server: SQL profiler, Data Management Views, Extended Events,tons of books on performance tuning

Performance schema in MySQL 5.5

The Idea

• Executing a query breaks down to hundreds of smaller tasks

• There are background tasks as well

• We want to instrument it all to know where server is spending time

Implementation

• Instrumentation: measuring when event begins and ends

• Implemented in MySQL code on server end storage engine level

• Can be enabled/disabled or customized

Schema and a storage engine+----------------------------------------------+| Tables_in_performance_schema |+----------------------------------------------+| cond_instances || events_waits_current || events_waits_history || events_waits_history_long || events_waits_summary_by_instance || events_waits_summary_by_thread_by_event_name || events_waits_summary_global_by_event_name || file_instances || file_summary_by_event_name || file_summary_by_instance || mutex_instances || performance_timers || rwlock_instances || setup_consumers || setup_instruments || setup_timers || threads |+----------------------------------------------+17 rows in set (0.00 sec)

Instruments+------------------------------------------------------------+| NAME |+------------------------------------------------------------+| wait/synch/mutex/sql/PAGE::lock || wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_sync || wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_active || wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_pool |..... || wait/synch/rwlock/sql/LOCK_grant || wait/synch/rwlock/sql/LOGGER::LOCK_logger |

......| wait/synch/cond/sql/COND_thread_count | | wait/synch/cond/sql/COND_thread_cache |..... || wait/io/file/sql/binlog || wait/io/file/sql/binlog_index |+------------------------------------------------------------+

Consumers

SELECT * FROM setup_consumers;+----------------------------------------------+---------+| NAME | ENABLED |+----------------------------------------------+---------+| events_waits_current | YES || events_waits_history | YES || events_waits_history_long | YES || events_waits_summary_by_thread_by_event_name | YES || events_waits_summary_by_event_name | YES || events_waits_summary_by_instance | YES || file_summary_by_event_name | YES || file_summary_by_instance | YES |+----------------------------------------------+---------+8 rows in set (0.00 sec)

What are top wait events in my server?

SELECT COUNT_STAR, SUM_TIMER_WAIT, AVG_TIMER_WAIT FROM events_waits_summary_by_instance ORDER BY SUM_TIMER_WAIT DESC LIMIT 10;+---------------------------------------------+------------+-------------------+----------------+| EVENT_NAME | COUNT_STAR | SUM_TIMER_WAIT | AVG_TIMER_WAIT |+---------------------------------------------+-----------------------+---------------+---------|| wait/io/file/innodb/innodb_data_file | 3347631 | 13547848609831118 | 4046995803 || wait/synch/rwlock/innodb/index_tree_rw_lock | 69535441 | 2000129499580014 | 28764173 || wait/synch/mutex/innodb/buf_pool_mutex | 800909600 | 433331246983734 | 541048 || wait/synch/mutex/innodb/log_sys_mutex | 181356697 | 420610626124392 | 2319245 || wait/synch/rwlock/innodb/index_tree_rw_lock | 3060704 | 232839801243104 | 76073936 || wait/io/file/innodb/innodb_data_file | 84259 | 163336572508164 | 1938505946 || wait/synch/rwlock/innodb/btr_search_latch | 192529196 | 161285917556962 | 837721 || wait/synch/mutex/sql/LOCK_open | 285022231 | 154822982324406 | 543196 || wait/synch/mutex/innodb/kernel_mutex | 403820715 | 130291049763820 | 322645 || wait/synch/mutex/mysys/THR_LOCK::mutex | 285017708 | 121061567338910 | 424751 |+---------------------------------------------+------------+-------------------+----------------+10 rows in set (0.03 sec)

innodb_buffer_pool_instances=10

SELECT * FROM events_waits_summary_by_instance ORDER BY SUM_TIMER_WAIT DESC LIMIT 10;+-----------------------------------------------+------------+-------------------+----------------+| EVENT_NAME | COUNT_STAR | SUM_TIMER_WAIT | AVG_TIMER_WAIT |+-----------------------------------------------+------------+-------------------+----------------+| wait/io/file/innodb/innodb_data_file | 3600080 | 14130028463169872 | 3924920685 || wait/synch/rwlock/innodb/index_tree_rw_lock | 69313641 | 2045091134703870 | 29504886 || wait/synch/mutex/innodb/trx_doublewrite_mutex | 6432834 | 592011487365760 | 92029654 || wait/synch/mutex/innodb/log_sys_mutex | 183016201 | 459928639055248 | 2513048 || wait/synch/rwlock/innodb/index_tree_rw_lock | 3239574 | 277212671866944 | 85570717 || wait/io/file/innodb/innodb_data_file | 400484 | 219787308922610 | 548804219 || wait/synch/rwlock/innodb/btr_search_latch | 190363676 | 183251774563990 | 962640 || wait/synch/mutex/sql/LOCK_open | 285022522 | 144911560909258 | 508421 || wait/synch/mutex/mysys/THR_LOCK::mutex | 285018012 | 130317830152708 | 457226 || wait/synch/mutex/innodb/kernel_mutex | 401734769 | 126815854421630 | 315670 |+-----------------------------------------------+------------+-------------------+----------------+10 rows in set (0.02 sec)

buf_pool_mutex is no longer in top 10 wait events

Which files are being accessed the most?

mysql> SELECT SUM_NUMBER_OF_BYTES_READ, SUM_NUMBER_OF_BYTES_WRITE FROM file_summary_by_instance ORDER BY SUM_NUMBER_OF_BYTES_READ+SUM_NUMBER_OF_BYTES_WRITE DESC LIMIT 10;+------------------------------+--------------------------+---------------------------+| FILE_NAME |SUM_NUMBER_OF_BYTES_READ | SUM_NUMBER_OF_BYTES_WRITE |+------------------------------+--------------------------+---------------------------+| /var/lib/mysql/ibdata1 | 166920192 | 363649155072 || /var/lib/mysql/WIR.ibd | 180310720512 | 114136367104 || /var/lib/mysql/EMIWPR.ibd | 51803815936 | 71776468992 || /var/lib/mysql/EWAPR.ibd | 38596755456 | 54104932352 || /var/lib/mysql/OFPR.ibd | 24171593728 | 33964572672 || /var/lib/mysql/OFIR.ibd | 38002688000 | 14265303040 || /var/lib/mysql/EQPO.ibd | 12718882816 | 21407596544 || /var/lib/mysql/FIR.ibd | 19646103552 | 10337140736 || /var/lib/mysql/EIR.ibd | 14708899840 | 12814794752 || /var/lib/mysql/FPR.ibd | 6360350720 | 9764585472 |+------------------------------+--------------------------+---------------------------+10 rows in set (0.01 sec)

Threads table(performance_schema) > SELECT * FROM threads;+-----------+----------------+----------------------------------------+| THREAD_ID | PROCESSLIST_ID | NAME |+-----------+----------------+----------------------------------------+| 0 | 0 | thread/sql/main || 12 | 0 | thread/innodb/srv_lock_timeout_thread || 14 | 0 | thread/innodb/srv_monitor_thread || 16 | 0 | thread/innodb/srv_purge_thread || 13 | 0 | thread/innodb/srv_error_monitor_thread || 2 | 0 | thread/innodb/io_handler_thread || 1 | 0 | thread/innodb/io_handler_thread || 9 | 0 | thread/innodb/io_handler_thread || 25 | 8 | thread/sql/one_connection || 3 | 0 | thread/innodb/io_handler_thread || 15 | 0 | thread/innodb/srv_master_thread || 10 | 0 | thread/innodb/io_handler_thread || 5 | 0 | thread/innodb/io_handler_thread || 8 | 0 | thread/innodb/io_handler_thread || 7 | 0 | thread/innodb/io_handler_thread || 17 | 0 | thread/sql/signal_handler || 4 | 0 | thread/innodb/io_handler_thread || 6 | 0 | thread/innodb/io_handler_thread |+-----------+----------------+----------------------------------------+18 rows in set (0.00 sec)

Average wait per thread (I)

SELECT e.THREAD_ID, e.EVENT_NAME, MAX(e.AVG_TIMER_WAIT)FROM events_waits_summary_by_thread_by_event_name e LEFT JOIN threads t ON t.THREAD_ID = e.THREAD_IDWHERE event_name LIKE 'wait/synch/mutex/innodb/%' AND t.NAME='thread/sql/one_connection'AND AVG_TIMER_WAIT > 0GROUP BY t.thread_id, e.EVENT_NAMEORDER BY t.thread_id, MAX(e.AVG_TIMER_WAIT)

Average wait per thread (II)

+-----------+-------------------------------------------------------+-------------------------+| THREAD_ID | EVENT_NAME | MAX(AVG_TIMER_WAIT) |+-----------+-------------------------------------------------------+-------------------------+| 27 | wait/synch/mutex/innodb/mutex_list_mutex | 49463 || 27 | wait/synch/mutex/innodb/purge_sys_bh_mutex | 83247 |

.....| 27 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 115681 || 27 | wait/synch/mutex/innodb/ibuf_pessimistic_insert_mutex | 168537 || 27 | wait/synch/mutex/innodb/ibuf_mutex | 282400 || 27 | wait/synch/mutex/innodb/log_sys_mutex | 1690694 |----------------------------------------------------------------------------------------------| 28 | wait/synch/mutex/innodb/mutex_list_mutex | 49504 || 28 | wait/synch/mutex/innodb/flush_list_mutex | 52375 || 28 | wait/synch/mutex/innodb/log_flush_order_mutex | 62040 |

.....| 28 | wait/synch/mutex/innodb/autoinc_mutex | 63012 || 28 | wait/synch/mutex/innodb/dict_sys_mutex | 181203 || 28 | wait/synch/mutex/innodb/ibuf_mutex | 222376 || 28 | wait/synch/mutex/innodb/innobase_share_mutex | 239514 || 28 | wait/synch/mutex/innodb/ibuf_pessimistic_insert_mutex | 266684 || 28 | wait/synch/mutex/innodb/log_sys_mutex | 1718503 |

Average wait per thread (III)

+-----------+--------------------------------------------------+-------------------------+| THREAD_ID | EVENT_NAME | MAX(ebt.AVG_TIMER_WAIT) |+-----------+--------------------------------------------------+-------------------------+| 27 | wait/synch/mutex/innodb/ibuf_mutex | 55872 || 27 | wait/synch/mutex/innodb/flush_list_mutex | 59495 || 27 | wait/synch/mutex/innodb/mutex_list_mutex | 59499 || 27 | wait/synch/mutex/innodb/autoinc_mutex | 64136 |

.....| 27 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 88242 || 27 | wait/synch/mutex/innodb/buf_pool_mutex | 101598 || 27 | wait/synch/mutex/innodb/kernel_mutex | 148332 || 27 | wait/synch/mutex/innodb/log_sys_mutex | 4671112 |-----------------------------------------------------------------------------------------| 28 | wait/synch/mutex/innodb/ibuf_mutex | 42886 || 28 | wait/synch/mutex/innodb/mutex_list_mutex | 56646 || 28 | wait/synch/mutex/innodb/flush_list_mutex | 58105 || 28 | wait/synch/mutex/innodb/autoinc_mutex | 66873 |

....| 28 | wait/synch/mutex/innodb/purge_sys_bh_mutex | 83545 || 28 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 87697 || 28 | wait/synch/mutex/innodb/buf_pool_mutex | 101365 || 28 | wait/synch/mutex/innodb/kernel_mutex | 149135 || 28 | wait/synch/mutex/innodb/log_sys_mutex | 4829134 |

innodb_change_buffering = none [Don’t try this at home! ]

Picoseconds, shmicosecond! ps_helper to rescue!

• http://www.markleith.co.uk/ps_helper/

• Great examples of using performance_schema

• Useful tools for converting time, bytes, paths into human readable format

Example: top_global_io_consumers_by_bytes_usage

(ps_helper) > SELECT event_name, total_read, avg_read, total_written, avg_written,avg_latency FROM top_global_io_consumers_by_bytes_usage;+-------------------------+------------+-----------+---------------+-------------+-------------+| event_name | total_read | avg_read | total_written | avg_written | avg_latency |+-------------------------+------------+-----------+---------------+-------------+-------------+| innodb/innodb_data_file | 10.76 GiB | 16.00 KiB | 16.05 GiB | 30.90 KiB | 10.96 ms || innodb/innodb_log_file | 68.00 KiB | 11.33 KiB | 1.03 GiB | 369.66 KiB | 2.33 ms || myisam/dfile | 197.79 KiB | 8.60 KiB | 187.04 KiB | 5.84 KiB | 9.89 µs || sql/FRM | 142.28 KiB | 491 bytes | 4.50 KiB | 242 bytes | 3.66 µs || sql/ERRMSG | 43.68 KiB | 14.56 KiB | 0 bytes | 0 bytes | 15.42 µs || myisam/kfile | 13.22 KiB | 226 bytes | 5.96 KiB | 24 bytes | 8.26 µs || mysys/charset | 17.83 KiB | 17.83 KiB | 0 bytes | 0 bytes | 10.68 µs || sql/partition | 1.06 KiB | 32 bytes | 0 bytes | 0 bytes | 4.38 µs || sql/pid | 0 bytes | 0 bytes | 6 bytes | 6 bytes | 34.78 µs || sql/global_ddl_log | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 3.69 µs || archive/data | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 4.35 µs || sql/dbopt | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 4.55 µs || sql/casetest | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 28.08 µs |+-------------------------+------------+-----------+---------------+-------------+-------------+13 rows in set (0.00 sec)

No free lunch: PFS overhead

• Instrumentation doesn’t come for free

• In CPU-bound workloads overhead can be ~15-20%

• In IO-bound — 5%-8%

• Significant improvements in 5.6

Performance schema in MySQL 5.6

What’s new in 5.6?

• performance_schema enabled by default!

• Less overhead: 5%-10% for CPU-bound workload

• Statements, Stages, Actors and Objects

Actors: filter events by user

(performance_schema) > SELECT * FROM setup_actors;+------+------+------+| HOST | USER | ROLE |+------+------+------+| % | % | % |+------+------+------+1 row in set (0.00 sec)

Objects: filter out events by database/table

(performance_schema) > SELECT * FROM setup_objects;+-------------+--------------------+-------------+---------+-------+| OBJECT_TYPE | OBJECT_SCHEMA | OBJECT_NAME | ENABLED | TIMED |+-------------+--------------------+-------------+---------+-------+| TABLE | mysql | % | NO | NO || TABLE | performance_schema | % | NO | NO || TABLE | information_schema | % | NO | NO || TABLE | % | % | YES | YES |+-------------+--------------------+-------------+---------+-------+4 rows in set (0.00 sec)

StatementsSELECT * FROM events_statements_summary_by_digest\G

*************************** 4. row *************************** DIGEST: f60b0866d7cbd78d39caf5fb52f43bfb DIGEST_TEXT: SELECT * FROM `table_io_waits_summary_by_index_usage` ORDER BY `SUM_TIMER_WAIT` DESC LIMIT ? COUNT_STAR: 1 SUM_TIMER_WAIT: 511539000 MIN_TIMER_WAIT: 511539000 AVG_TIMER_WAIT: 511539000 MAX_TIMER_WAIT: 511539000 SUM_LOCK_TIME: 72000000 SUM_ERRORS: 0 SUM_WARNINGS: 0 SUM_ROWS_AFFECTED: 0 SUM_ROWS_SENT: 5 SUM_ROWS_EXAMINED: 91SUM_CREATED_TMP_DISK_TABLES: 0 SUM_CREATED_TMP_TABLES: 0 SUM_SELECT_FULL_JOIN: 0 SUM_SELECT_FULL_RANGE_JOIN: 0 SUM_SELECT_RANGE: 0 SUM_SELECT_RANGE_CHECK: 0 SUM_SELECT_SCAN: 1 SUM_SORT_MERGE_PASSES: 0 SUM_SORT_RANGE: 0 SUM_SORT_ROWS: 5 SUM_SORT_SCAN: 1 SUM_NO_INDEX_USED: 1 SUM_NO_GOOD_INDEX_USED: 0 FIRST_SEEN: 2012-10-16 21:50:13 LAST_SEEN: 2012-10-16 21:50:13

Summary. Pros.

• Provides insight into internal server metric

• Flexible

• Relational model allows to build your own views on top of this data

Summary. Cons.

• Steep learning curve: not very well documented

• Overhead for CPU-bound loads can be significant

Thank you!

performance schema in mysql (danil zburivsky)

instances events

history events

instance events

current events

long events

instances file

extended events

instances setup

Documents

postgresql, sqlalchemy, and schema-less data. - wmmi.net ·...

mysql performance schema missing_manual_flossuk

mysql information schema · pdf filethis is the mysql...

openworld 2014 - schema management: versioning and...

mysql performance schema : fossasia

mysql large table schema changes

using mysql 5.6 performance schema to …...2 about...

phew. break time. :)real world mysql tuning schema and index...

mysql performance schema · this is the mysql performance...

mysql 5.7 + json - fosdem · • no painful schema changes*...

mysql performance schema - percona.com · metadata always...

online schema change in mysql casual #1(2010/12/11)

mysql performance schema, open source india, 2015

mysql migration toolkit · chapter 2. introduction to the...

mysql performance schema in action

mysql information schema

mysql workbench - alexandru ioan cuza university · visual...

the mysql performance schema & new sys schema

mysql workbench tutorialthe schema into mysql using mysql...

profiling with performance schema - mysql · an...