ougls 2016: how profiling works in mysql

How Profiling Works in MySQL

How Profiling Works in MySQLGeorgi KodinovMySQL team leadOracle Confidential Internal/Restricted/Highly Restricted

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Oracle Confidential Internal/Restricted/Highly Restricted2

Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracles products remains at the sole discretion of Oracle.

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |AgendaOracle Confidential Internal/Restricted/Highly Restricted3Status variablesPERFORMANCE_SCHEMASYS schemaAnalyzing MySQL performance

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Status variablesThe traditional toolOracle Confidential Internal/Restricted/Highly Restricted4

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Status variables: how do they work ?Global and/or sessionOne mutex protected global copyEach session has a copy of ittaken at session creation timeSession variable updates are done on the THD copy: no synchronizationSession results are aggregated to the global copy at thread terminationOr FLUSH STATUSYou can observe P_S.SESISON_STATUS (global) and P_S.GLOBAL_STATUS (current session) or P_S.STATUS_BY_THREAD (all sessions)Oracle Confidential Internal/Restricted/Highly Restricted5

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Interesting status variablesVariableScopeWhat ?Threads_connectedGlobalNumber of open connectionsCreated_tmp_disk_tablesGlobal/SessionNumber of temp tables created by statements (sort etc)Handler_read_firstGlobal/SessionIndication of index scansInnodb_buffer_pool_wait_freeGlobalWaits for pages to be flushedMax_used_connectionsGlobalHigh watermark of the active connectionsSlow_queriesGlobal/SessionNumber of queries taking more than long_query_timeSelect_full_joinGlobal/SessionNumber of joins performing full table scans (indexes ?)

Oracle Confidential Internal/Restricted/Highly Restricted6

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Innodb_buffer_pool_wait_free high means innodb_buffer_pool_size 6

PERFORMANCE_SCHEMAOracle Confidential Internal/Restricted/Highly Restricted7

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |ServerPERFORMANCE_SCHEMAOverview of performance_schema Instrumentation

Queries

_history

_summary_ObjectsObjectsObjects

EventsEventsEvents


8

How Does Instrumentation Work in MySQL ?Tracks the duration of various eventsDuration can be exposed with up to a picosecond precisionThe instrumentation of each event is configurableData are stored into fixed size ring buffers in memoryThe collected data are directed into consumersVarious aggregations are automatically calculatedData collection is lockless !

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |What is PERFORMANCE_SCHEMA ?A generic system database for tables with volatile dataPowered by an actual built-in MySQL storage engineMajority of tables expose instrumented event latencyThere are configuration tables that define the extent of instrumentation

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Instrument NamingHierarchical : component/subcomponent/sub-subcomponent/Currently (5.7) 6 top level components:WaitStageStatementIdleMemoryTransaction

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Wait InstrumentsEventName MaskTable lockswait/lock/table/%Network I/Owait/io/socket/%Table I/Owait/io/table/%File I/Owait/io/file/%Mutexeswait/synch/mutex/%Read/Write lockswait/synch/rwlock/%Conditionswait/synch/cond/%

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Stage InstrumentsStage//Code area is e.g. sql, myisam, etcStage name is a set of well known query execution stagesPreparingExecutingOpening tablesetc

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Statement InstrumentsEventName MaskRPC commandstatement/com/%SQL statementstatement/sql/%Abstract statementstatement/abstract/%

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Abstract statements are needed since instrumentation of SQL statements starts before the actual parsing and its not known what type of statement that is14

Idle InstrumentTracks idle time on inactive socketsSocket is inactive when waiting for a request from the clientIts different from waiting on an active socket !

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Memory Instrumentsmemory//Code area is e.g. sql, myisam, etcInstrument name is a named memory allocation counter

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Transaction InstrumentNo further components

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Types of Instrumentation Data TablesInstances: socket_instances, file_instances, mutex_instances, cond_instances, etcExpose a current list and state of instrumented objectsEvents: events_waits_current, events_statements_history, etcAccess to the a timeline of event instrumentationsTypically 3 kinds per instrument : *_current, *_history, *_history_longSummary: events_waits_summary_by_instance, events_statements_summary_by_thread_by_event_name, etcAggregate events so you dont have to

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |How to Configure InstrumentationCompile TimeCompletely or selectively removeServer StartupDisable completelySelectively enable instruments and consumersMid FlightUpdate the setup tablesChanges effective immediately for all but setup_actors !

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Setup Tables ExamplesTableExplanationsetup_actorsWhat worker threads to monitorsetup_consumersWhat PERFORMANCE_SCHEMA data tables to fillsetup_instumentsWhat instruments to enablesetup_objectsWhat tables to monitorsetup_timersPrecision to use when collecting instrument timing

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Introduction to Analyzing MySQL Performanceusing PERFORMANCE_SCHEMAOracle Confidential Internal/Restricted/Highly Restricted21

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Profiling ConsiderationsIt is easy to just enable everything, but there are overhead concernsOn busy systems mutexes can be locked millions of times per secondIt is best to just pick higher latency event typesStatements, Stages, Table IO, File IO, maybe Network IOIt is good to enable all %_current consumers, and statement historyFor concurrency/profiling debugging toggle other instruments on an as needed basis

22

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Profiling TypesOnce youve narrowed down what youre interested in, there are two ways to start monitoringView raw data in the summary viewsGives you an overall picture of usage on the instanceSnapshot data, and compute deltas over timeGives you an idea of the rates of change for the events


Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Analyzing Global WaitsSome waits can include other waitsTable IO latency may also include some mutex and file IO latencyWait times include concurrency across all threadsDo not assume you can sum all events in global tables and compare to wall clock timesOracle Confidential Internal/Restricted/Highly Restricted24

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Top Waits By Latencymysql> select event_name, -> count_star as count, -> sys.format_time(sum_timer_wait) as total_latency, -> sys.format_time(avg_timer_wait) as avg_latency, -> sys.format_time(max_timer_wait) as max_latency -> from events_waits_summary_global_by_event_name -> where event_name != 'idle' -> order by sum_timer_wait desc limit 5;+--------------------------------------+----------+---------------+-------------+-------------+| event_name | count | total_latency | avg_latency | max_latency |+--------------------------------------+----------+---------------+-------------+-------------+| wait/io/table/sql/handler | 21888502 | 1.27h | 208.06 us | 2.21 s || wait/io/file/innodb/innodb_data_file | 4276800 | 00:48:12.49 | 676.32 us | 1.49 s || wait/io/file/innodb/innodb_log_file | 1948199 | 00:25:24.36 | 782.45 us | 1.30 s || wait/io/file/myisam/kfile | 4566406 | 00:13:45.92 | 180.87 us | 1.17 s || wait/io/file/myisam/dfile | 1277589 | 00:05:46.23 | 271.01 us | 1.18 s |+--------------------------------------+----------+---------------+-------------+-------------+Oracle Confidential Internal/Restricted/Highly Restricted25

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Some Mutex events That Can Affect Global Concurrency(if high in list)wait/synch/mutex/innodb/buf_pool_mutexIncrease innodb_buffer_pool_instanceswait/synch/mutex/sql/Query_cache::structure_guard_mutexLook in to disabling the Query Cachewait/synch/mutex/myisam/MYISAM_SHARE::intern_lockUse Innodb Oracle Confidential Internal/Restricted/Highly Restricted26Analyzing Global Waits

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Some File IO Events To Watch For (if high in list):wait/io/file/sql/FRMTune table_open_cache / table_definition_cachewait/io/file/sql/file_parser (view definition parsing)If high on 5.5, upgrade to 5.6, (which can cache these like tables)wait/io/file/sql/query_log and wait/io/file/sql/slow_logDisable the general logDisable or tune what is logged to the slow log (decent long_query_time)

Oracle Confidential Internal/Restricted/Highly Restricted27Analyzing Global Waits

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Top Files By Total IO mysql> select sys.format_path(file_name) as file, -> count_read, -> sys.format_bytes(sum_number_of_bytes_read) as total_read, -> sys.format_bytes(IFNULL(sum_number_of_bytes_read / count_read, 0)) as avg_read, -> count_write, -> sys.format_bytes(sum_number_of_bytes_write) as total_written, -> sys.format_bytes(IFNULL(sum_number_of_bytes_write / count_write, 0)) as avg_write, -> sys.format_bytes(sum_number_of_bytes_read + sum_number_of_bytes_write) as total, -> IFNULL(ROUND(100-((sum_number_of_bytes_read/(sum_number_of_bytes_read + sum_number_of_bytes_write))*100), 2), 0.00) as write_pct -> from file_summary_by_instance -> order by (sum_number_of_bytes_read + sum_number_of_bytes_write) desc limit 5;+----------------------------------+------------+------------+-----------+-------------+---------------+-----------+-------------+-----------+| file | count_read | total_read | avg_read | count_write | total_written | avg_write | total | write_pct |+----------------------------------+------------+------------+-----------+-------------+---------------+-----------+-------------+-----------+| @@datadir/ibdata1 | 888 | 15.84 MiB | 18.27 KiB | 1089824 | 61.99 GiB | 59.64 KiB | 62.00 GiB | 99.98 || @@datadir/mem__events/events.ibd | 114 | 1.80 MiB | 16.14 KiB | 117370 | 2.14 GiB | 19.14 KiB | 2.14 GiB | 99.92 || @@datadir/cerberus-bin.000010 | 296103 | 1.01 GiB | 3.59 KiB | 362852 | 1.00 GiB | 2.89 KiB | 2.01 GiB | 49.63 || @@datadir/ib_logfile0 | 6 | 68.00 KiB | 11.33 KiB | 506837 | 1.46 GiB | 3.03 KiB | 1.46 GiB | 100.00 || @@datadir/ib_logfile1 | 0 | 0 bytes | 0 bytes | 476961 | 1.45 GiB | 3.19 KiB | 1.45 GiB | 100.00 |+----------------------------------+------------+------------+-----------+-------------+---------------+-----------+-------------+-----------+

Oracle Confidential Internal/Restricted/Highly Restricted28High IO on InnoDB per-tablespace tables can show candidates for a separate mountpoint/disk using DATA DIRECTORY in CREATE TABLE

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Analyzing User ActivityAll event_% summaries are exposed with a number of dimensionsTo analyze connection activity you can do this in 4 waysBy UserBy HostBy Account (User@Host)By ThreadThe follow examples are by user, but could be replaced with the host, account or thread summary views


Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Top Users By Statement Latencymysql> select user, -> sum(count_star) as statements, -> sys.format_time(sum(sum_timer_wait)) as total_latency, -> sys.format_time(sum(sum_timer_wait) / sum(count_star)) as avg_latency -> from events_statements_summary_by_user_by_event_name -> where user is not null -> group by user -> order by sum(sum_timer_wait) desc;+------+------------+---------------+-------------+| user | statements | total_latency | avg_latency |+------+------------+---------------+-------------+| root | 7229032 | 15.17h | 7.55 ms || mark | 3072 | 1.77 s | 575.29 us |+------+------------+---------------+-------------+


Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Top Users By IO Latencymysql> select user, sum(count_star) as count, -> sys.format_time(sum(sum_timer_wait)) as total_latency -> from events_waits_summary_by_user_by_event_name -> where event_name like 'wait/io/file/%' -> and user is not null -> group by user -> order by sum(sum_timer_wait) desc;+------+----------+---------------+| user | count | total_latency |+------+----------+---------------+| root | 10892980 | 00:46:19.67 || mark | 20043 | 346.79 ms |+------+----------+---------------+

Oracle Confidential Internal/Restricted/Highly Restricted31Replace with some other pattern here for Top User by wait class

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Top Users By Connectionsmysql> select * from users where user is not null order by current_connections desc;+------+---------------------+-------------------+| USER | CURRENT_CONNECTIONS | TOTAL_CONNECTIONS |+------+---------------------+-------------------+| root | 8 | 151655 || mark | 1 | 1 |+------+---------------------+-------------------+2 rows in set (0.00 sec)

mysql> select * from accounts where user is not null order by current_connections desc;+------+-----------+---------------------+-------------------+| USER | HOST | CURRENT_CONNECTIONS | TOTAL_CONNECTIONS |+------+-----------+---------------------+-------------------+| root | localhost | 39 | 151698 || mark | localhost | 1 | 1 |+------+-----------+---------------------+-------------------+2 rows in set (0.00 sec)Oracle Confidential Internal/Restricted/Highly Restricted32

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |SYS SCHEMAOracle Confidential Internal/Restricted/Highly Restricted33

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |What is SYS Schema ?A set of objects that help interpreting data collected by PERFORMANCE_SCHEMAIncludes:Views that summarize Performance Schema data into more easily understandable form.Stored procedures that perform operations such as Performance Schema configuration and generating diagnostic reports. Stored functions that query Performance Schema configuration and provide formatting services. Installed together with the server since 5.7CALL sys.ps_setup_reset_to_default(TRUE);Oracle Confidential Internal/Restricted/Highly Restricted34

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |MySQL SYS FunctionsMake the raw data more readable to a humanformat_time() / format_bytes()Compress data for CLI outputformat_statement() / format_path()Extract some data to assist with JOIN in certain casesextract_[schema|table]_from_file_name()Other functionsps_is_account_enabled() / ps_thread_stack()


Copyright 2015, Oracle and/or its affiliates. All rights reserved. |MySQL SYS ViewsReference set of views solving various admin use casesBuild upon both Performance Schema and INFORMATION_SCHEMABoth formatted and raw views are availableAll raw views are prefixed with x$Allows tooling to poll raw views, but humans to use normal onesOracle Confidential Internal/Restricted/Highly Restricted36

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |High level user overviewBreakdowns of IO usageDrill in to stages per userDrill in to statement details per usermysql> show tables like 'user%';+-----------------------------------+| Tables_in_sys (user%) |+-----------------------------------+| user_summary || user_summary_by_file_io || user_summary_by_file_io_type || user_summary_by_stages || user_summary_by_statement_latency || user_summary_by_statement_type |+-----------------------------------+Oracle Confidential Internal/Restricted/Highly Restricted37User Summary Views

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |IO breakdown by threadGlobal summaries by file and wait class, by both bytes and latencyStream of last raw file IO statsmysql> show tables like 'io_%';+------------------------------+| Tables_in_sys (io_%) |+------------------------------+| io_by_thread_by_latency || io_global_by_file_by_bytes || io_global_by_file_by_latency || io_global_by_wait_by_bytes || io_global_by_wait_by_latency || latest_file_io |+------------------------------+Oracle Confidential Internal/Restricted/Highly Restricted38IO Summary Views

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Object overviewTable usage statsIndex usage statsmysql> show tables like 'schema%';+-------------------------------------+| Tables_in_sys (schema%) |+-------------------------------------+| schema_index_statistics || schema_object_overview || schema_table_statistics || schema_table_statistics_with_buffer || schema_tables_with_full_table_scans || schema_unused_indexes |+-------------------------------------+Oracle Confidential Internal/Restricted/Highly Restricted39Schema Analysis Views

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Buffer usage by schemaBuffer usage by tablemysql> show tables like 'innodb%';+-------------------------------+| Tables_in_sys (innodb%) |+-------------------------------+| innodb_buffer_stats_by_schema || innodb_buffer_stats_by_table |+-------------------------------+Oracle Confidential Internal/Restricted/Highly Restricted40InnoDB Buffer Usage Views

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Statement overviewFind statements by errors, full scans, sorting, temporary tablesFind the statements with the longest runtimesmysql> show tables like 'statement%';+---------------------------------------------+| Tables_in_sys (statement%) |+---------------------------------------------+| statement_analysis || statements_with_errors_or_warnings || statements_with_full_table_scans || statements_with_runtimes_in_95th_percentile || statements_with_sorting || statements_with_temp_tables |+---------------------------------------------+Oracle Confidential Internal/Restricted/Highly Restricted41Statement Analysis Views

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Wait summaries by class and per instrument globallyWait details per usermysql> show tables like 'wait%';+------------------------------------+| Tables_in_sys (wait%) |+------------------------------------+| wait_classes_global_by_avg_latency || wait_classes_global_by_latency || waits_by_user_by_latency || waits_global_by_latency |+------------------------------------+Oracle Confidential Internal/Restricted/Highly Restricted42Wait Analysis Views

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |MySQL SYS Procedures - P_S Analysis HelpersDump the data to create a graph dot file for a thread traceps_trace_thread()Look in the statement history table trying to capture more info ps_trace_statement_digest()Reset all summary data ps_truncate_all_tables()Oracle Confidential Internal/Restricted/Highly Restricted43

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Tracing Statement Digestsps_trace_statement_digest() analyzes live traffic looking for certain statement digestCaptures statistics on each matching statement it findsReturns a report of the statisticsAn overall summaryA break down for the longest running exampleAn EXPLAIN (if the statement is not truncated)Oracle Confidential Internal/Restricted/Highly Restricted44

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |Tracing Threadsps_trace_thread() monitors a specific thread for a periodCaptures as much information on the thread activity as possibleReturns a dot formatted file, that can graph the event hierarchy as previously seenhttp://en.wikipedia.org/wiki/DOT_(graph_description_language)


Copyright 2015, Oracle and/or its affiliates. All rights reserved. |MySQL SYS Procedures - P_S Setup Helpersps_setup_show_disabled() / ps_setup_show_enabled()ps_setup_disable_thread() / ps_setup_enable_thread()ps_setup_disable_background_threads() / ps_setup_enable_background_threads()ps_setup_disable_instrument() / ps_setup_enable_instrument()ps_setup_disable_consumer() / ps_setup_enable_consumer()ps_setup_save() / ps_setup_reload_saved()ps_setup_reset_to_default()ps_truncate_all_tables()




ougls 2016: how profiling works in mysql

Software