mysqlnd query cache plugin statistics and tuning

MySQL native driver for PHP:Dig deeper with QC statistics

mysqlnd_qc:
Dig deeper with statistics

Ulf Wendel, Andrey HristovMySQL Connectors TeamSun Microsystems

Table of Contents

OverviewSources of statistics

Core statisticsConfiguration and Access

Listing

AdvancedQuery statistics

Storage handler statistic

Tuning screws (statistics) all over

mysqlnd, mysqlnd_qc core, mysqlnd_qc storage handler

ext/*mysql*ext/*mysql*mysqlndmysqlnd

Query Cache Plugin (mysqlnd_qc core)Storage handler (mysqlnd_qc)

The 150 statistics of mysqlnd

Collected by: mysqlnd

Scope: process, connectionProcess: mysqli_get_client_stats(), phpinfo()

Connection: mysqli_get_connection_stats()

Aggregated values from all MySQL PHP APIs

Contents: wide rangeNetwork related

Result set related

Connection related

Client-Server Protocol related

20+ statistics of the query cache core

Collected by: query cache plugin

Scope: processmysqlnd_qc_get_core_stats()


Contents: wide rangeCache usage related

Network related

Timings

Query back traces and statistics

Collected by: query cache plugin

Scope: processmysqlnd_qc_get_query_trace_log()

mysqlnd_qc_get_normalized_query_trace_log()

Contents: Source code back trace showing query origin
(mysqlnd_qc_get_query_trace_log())

Cache related e.g. hit, miss, added, occurences...

Timings

Query cache storage handler stats

Collected by: query cache storage handler

Scope: cache entry (= request or process)Depends on storage handler

mysqlnd_qc_get_cache_info()


Contents: none or assortedDepends on storage handler

APC: timings, hit ratio, result set size

Default: like APC plus result set meta data

Runtime configuration

Table of Contents



Listing



Core statistics: basic monitoring

Measure cache efficiencyDetermine if query caching gives a speed-up

Check cache hit/miss ratio

Check how many queries get cached

Check collisions/slam defense efficiency

Available with all handlers

Aggregated dataYou cannot measure individual queries

Accessing core statistics

Change runtime configuration to collect statistics!

array mysqlnd_qc_get_core_statistics()

Returns a list of statistics collected by the coreof the query cache plugin, if the PHP configurationsetting mysqlnd_qc_collect_statistics is set to 1.The statistics are provided by the core and thereforeavailable with all build-in storage handlers and whenusing user-defined storage handlers.


mysqlnd_qc.collect_statisticsPHP_INI_ALL, default 0, boolean

enable/disable all core statistics

mysqlnd_qc.time_statisticsPHP_INI_ALL, default 1, boolean

use of gettimeofday() system call

enable/disable: *run_time*, *store_time*

mysqlnd_qc.slam_defensePHP_INI_SYSTEM, default 0, boolean

enable/disable: slam_stale_*

Scope of core statistics

PHP processData from one or multiple requests

Aggregated for all PHP MySQL extensions:
ext/mysql, ext/mysqli, PDO_MySQL

Aggregated for all cache entries

Watch out when interpreting the figures different processes may show different figures!

Lifting scope/life time limitations

Manually collect statistics at script endphp configuration directive: auto_append_file

register_shutdown_function()

Choose persistent storage mediumMySQL fire and forget with ASYNC

Files

Network

Memcache, APC, ...

Manual aggregation using MySQL

Fire and forget - works well for all kinds of QC stats!

array mysqlnd_qc_get_core_statistics()

Returns a list of statistics collected by the coreof the query cache plugin, if the PHP configurationsetting mysqlnd_qc_collect_statistics is set to 1.The statistics are provided by the core and thereforeavailable with all build-in storage handlers and whenusing user-defined storage handlers.

Cache access statistics (I)

cache_hitStatement is considered cacheable...

and cached data has been reused

and we had a miss but the statement got cached by someone else while we process it

Cache access statistics (II)

cache_missStatement is considered cacheable...

.. and has been added to cache right now

...but cache_no_table = 1 prevented caching

...but unbuffered result set used

...but buffered result set was empty

cache_putStatement is considered cacheable and has been added to cache

Cache access statistics (III)

cache_putStatement is considered cacheable and has been added to cache

Take care when calculating derived statistics. Handler with a storage life time beyond process scope may report cache_put = 0 together with cache_hit > 0, if another process has filled the cache. You may want to use num_entries from mysqlnd_cache_info(), given the handler supports it (Default, APC).

Query related statistics (I)

query_should_cacheStatement is considered cacheable based on query string analysis

It is yet unknown if it will end up in the cache!

query_should_not_cacheStatement is considered not cacheable based on
query string analysis

Derived: # of queries inspected by QCquery_should_cache + query_should_not_cache

Query related statistics (II)

query_not_cachedStatement is considered not cacheable...

or storage handler has returned no hash key

Derived: # of empty hash keysquery_not_cached query_should_cache

A value >0 indicates storage issues (errors, cache full, - details are unknown)

Query related statistics (III)

query_could_cacheStatement is considered cacheable...

and statement executed without errors

and meta data shows at least one column

It may or may not be in the cache already

It may or may not be added to cache later on

Query related statistics (IV)

query_found_in_cacheStatement is considered cacheable....

and we have found it in the cache ...

but we have not yet replayed cached data

No cache hit yet: client might not fetch data

No cache hit yet: cached data may be faulty

Query related statistics (V)

query_uncached_otherStatement is considered cacheable ...

it may or may not be in the cache already

but either replaying cached data failed, no result set is available or some other error happened

Query related statistics (VI)

query_uncached_no_tableQuery would have been cached if cache_no_table = 1

Query related statistics (VII)

query_uncached_use_resultQuery would have been cached if a buffered result set had been used.

Incremented together with cache_miss

Derived: # of cacheable queries if no issues:query_uncached_other + query_uncached_no_table + query_uncached_use_result

Run times (I)

query_aggr_run_time_cache_hitAggregated run times (ms) of queries which are considered as a cache hit

Derived: average run time of a cached queryquery_aggr_run_time_cache_hit / cache_hit

Run times (II)

query_aggr_run_time_cache_putAggregated run times (ms) of queries which are considered as a cache put

Derived: average run time improvement of cached queries (Default handler only!)(query_aggr_run_time_cache_put / cache_put) / (query_aggr_run_time_cache_hit / cache_hit)

Run times (III)

query_aggr_run_time_totalAggregated run time (ms) of all queries run by the query cache

Derived: aggr. run time of uncached queriesquery_aggr_run_time_total query_aggr_run_time_cache_hit

Derived: avg. run time of uncached queries(query_aggr_run_time_total query_aggr_run_time_cache_hit) / cache_miss

Store times (I)

query_aggr_store_time_cache_hitAggregated store times (ms) of queries which are considered as a cache hit

Derived: average store time for a cached queryquery_aggr_store_time_cache_hit / cache_hit

Store times (II)

query_aggr_store_time_cache_putAggregated store times (ms) of queries which are considered as a cache put

Derived: average store time improvement of a cached query (Default handler only!)(query_aggr_store_time_cache_put / cache_put) / (query_aggr_store_time_cache_hit / cache_hit)

Store times (III)

query_aggr_store_time_totalAggregated store time (ms) of all queries run by the query cache

Derived: aggr. store time of uncached queriesquery_aggr_store_time_total query_aggr_store_time_cache_hit

Derived: avg. store time of uncached queries(query_aggr_store_time_total query_aggr_store_time_cache_hit) / cache_miss

Network traffic (I)

receive_bytes_recordedRecorded traffic from MySQL to PHP

The data may or may not be added to the cache at some point (depends on hit or miss)

receive_bytes_replayedReplayed recorded traffic from MySQL to PHP

This is the total amount of incoming traffic saved by using the query cache plugin

Network traffic (II)

send_bytes_recordedRecorded traffic from PHP to MySQL

The data may or may not be added to the cache at some point (depends on hit or miss)

send_bytes_replayedReplayed recorded traffic from PHP to MySQL

This is the total amount of outgoing traffic saved by using the query cache plugin

Network traffic (III)

Derived: total network traffic savings in MB(receive_bytes_replayed + send_bytes_replayed) / 1024 / 1024

Slam defense

slam_stale_refreshNumber of cache misses which triggered serving stale data until the client causing the cache miss has refreshed the cache entry

slam_stale_hitNumber of cache hits while a stale cache entry gets refreshed

Table of Contents



Listing



Query back trace

php.ini setting: mysqlnd_qc.collect_query_trace = 1

array mysqlnd_qc_get_query_trace_log()

Returns a list query back traces for every query thathas been inspected by the query cache regardless if the query ended up being cached or not. The trace tells you where a query has been issues (see alsodebug_backtrace()). Together with the back traceyou get run and store times and information on ifthe query has been cached.


mysqlnd_qc.collect_query_tracePHP_INI_SYSTEM, default 0, boolean

enable/disable query back trace

mysqlnd_qc.query_trace_bt_depthPHP_INI_SYSTEM, default 3, integer

limits code back trace depth

Scope of the query back trace

PHP ProcessData from one or multiple requests

Data from all PHP MySQL APIs:

Query back trace array elements (I)

querystring: SQL statement

Trace contains all inspected queries

Trace contains cached and uncached queries

Same SQL statement can appear multiple times!

originstring: code back trace

Back trace to the origin of the query

Back trace begins with, e.g. mysqli_query()

ini setting: mysqlnd_qc.query_trace_bt_depth

Query back trace array elements (II)

run_timeint: query run time

ini setting: mysqlnd_qc.time_statistics (enabled by default)

store_timeint: time required to fetch and store result set

ini setting: mysqlnd_qc.time_statistics(enabled by default)

Query back trace array elements (III)

eligible_for_cachingboolean

Set to true if the query qualifies for cachingaccording to the query analysis

The query might or might not be cached

no_tableboolean

Set to true if any column from the result sethas no table name, e.g. SELECT NOW()

Query back trace array elements (IV)

was_addedboolean

Set to true if the query has been added to thecache.

was_already_in_cahceboolean

Set to true in case of a cache hit.

Normalized query back trace

mysqlnd_qc.collect_normalized_query_trace = 1

array mysqlnd_qc_get_normalized_query_trace_log()

Similar to mysqlnd_qc_get_query_trace_log() but withSQL statements normalized and aggregated by thenormalized query string. Normalization refers toreplacing actual parameters, for examplein WHERE a > 1, with questionmarks likeWHERE a > ?. WHERE a > ? will match any valuefor ?, for example 1, 2, 'abc' but not otheridentifiers.


mysqlnd_qc.collect_normalized_query_tracePHP_INI_SYSTEM, default 0, boolean

enable/disable normalized query trace

Scope of the normalized query log

PHP ProcessData from one or multiple requests


Aggregated by normalized query

Normalized query trace elements (I)

querystring: normalized SQL statement

Trace contains all inspected queries

Aggregated by normalized SQL statement

Normalized: parameter replaced by placeholderSELECT 1, SELECT 'a' SELECT ?

occurencesint

Number of SQL statements summarized in this record set

Normalized query trace elements (II)

eligible_for_cachingboolean

Set to true if the query qualifies for cachingaccording to the query analysis

The query might or might not be cached

Normalized query trace elements (III)

min_run_time

avg_run_time

max_run_timeint

Minimum, average and maximum run time of all queries aggregated in this record set


Normalized query trace elements (IV)

min_store_time

avg_store_time

max_store_timeint

Minimum, average and maximum store time of all queries aggregated in this record set


Storage handler statistics

Optional handler duty!

Default handlerResult set size

Access statistics

Meta data

APCAccess statistics

Memcacheno handler statistics available

Accessing storage handler statistics

User storage handler need to provide their own API!

array mysqlnd_qc_get_cache_info()

Returns information on the active handler and, if available, storage handler statistics. Storagehandler statistics are provided under the data keyof the hash. Different storage handler return differentstatistics, if any. User defined storage handler cannot return their statistics through this function. They need to provide their own API foraccessing storage handler level statistics.

Hot tables and guessing speed-up

Set myslqnd_qc.cache_by_default = 1Accept and ignore false results

Set mysqlnd_qc.collect_statistics = 1

Set mysqlnd_qc.time_statistics = 1

Analyze meta data of cache entriesmysqlnd_qc_get_cache_info() data hash

Default and APC handler provide meta data

Search and account table names from meta data

Analyze recorded run and store times

Hot tables and guessing speed-up

Set myslqnd_qc.cache_by_default = 0

Set mysqlnd_qc.collect_statistics = 1

Set mysqlnd_qc.time_statistics = 1

Read user handler specific presentation! :-)


mysqlnd_qc.collect_statisticsno impact on storage handler statistics

background: storage format with/wo statistics

mysqlnd_qc.time_statisticsuse of gettimeofday() system call

enable/disable: all timings

background: handler make use of core function

Scope of handler statistics

Scope of handler storageDefault: Process (one or multiple requests)

APC: machine (multiple processes)


Default handler: result set size

rowsNumber of rows

stored_sizeSize in bytes of the raw network traffic result setadded to the cache

See http://blog.ulf-wendel.de/?p=198 for high-level discussion of the raw network format

Default handler: access and timings

cache_hitsNumber of cache hits

run_timeRun time of the uncached query

Set once when adding the query to the cache

store_timeStore time of the uncached query


Default handler: timings

min_run_time

avg_run_time

max_run_timeMinimum, average and maximum query runtime measured during a cache hit

Default handler: timings

min_store_time

avg_store_time

max_store_timeMinimum, average and maximum storetime measured during a cache hit

APC handler: access, size, timings

cache_hitsNumber of cache hits

rowsNumber of rows

run_timeRun time of the uncached query


store_timeStore time of the uncached query


APC handler: timings

min_run_time

avg_run_time

max_run_time

min_store_time

avg_store_time

max_store_timeSee Default handler statistics

NULL if no cache hit

Tooling: Cache Monitor web script

web/mysqlnd_qc_monitor.php

Demonstrates usage of statistics

Not a production level tool

Basics onlyConfiguration

Per-process statistics

Cache contents (storage handler statistics)

Per-machine statistics using slow auto_append example script

Tooling: per machine statistics

web/auto_append_persist_qc_stats.php

auto_append to persist per-process data

Demonstrates aggregation of core statistics

Not a production level toolSlow

MySQL storage only

The End
Feedback: [email protected]

The End

Feedback:
[email protected],
[email protected]

Sun Microsystems, Inc.

Page

Click to edit the title text format

Click to edit the outline text formatSecond Outline Level

Click to edit the notes format

Page

Click to edit the title text format

Presenters NamePresenters TitlePresenters Company

Click to edit the notes format

Page

PHP ini settingScopeDefaultmysqlnd_qc.collect_statisticsINI_ALL0 mysqlnd_qc.time_statisticsINI_ALL1 mysqlnd_qc.collect_query_traceINI_SYSTEM0 mysqlnd_qc.query_trace_bt_depthINI_SYSTEM3 mysqlnd_qc.collect_normalized_query_traceINI_SYSTEM0 mysqlnd_qc.slam_defenseINI_SYSTEM0

???Page ??? (???)08.02.2010, 12:51:08Page /

mysqlnd query cache plugin statistics and tuning

Technology