Extending MEM, Custom Advisors, Graphs and Data Collections

Mark LeithSun Microsystems

(leith@sun.com)

Agenda

Quick MEM Agent Architecture Creating Data Collections Viewing Data Collection Information Creating Custom Graphs Creating and Modifying Advisors Questions / Input

Quick MEM Agent Architecture

Architecture – MEM Agent

Creating Data Collections

Data Collection Overview

All Data Collections defined on each Agent installation locally

Defaults are within the “./agent/share/mysql-proxy” directory

SQL and Lua based collections supported

SQL statements defined in XML format QUAN based collections are all Lua via

the Proxy

Internal Data Collection Format

Defined in XML files in the Advisor bundles Here's our definition of the

innodb_buffer_pool_size variable:

<itemList><attribName>innodb_buffer_pool_size</attribName>

<nameSpace>mysql</nameSpace> <attribType>INTEGER</attribType> <className>variables</className> <categoryName>Memory</categoryName> <subCategoryName>Buffers</subCategoryName> <isCounter>false</isCounter> </itemList>

Internal Data Collection Format

Types: nameSpace::className::attribName i.e: mysql::variables::innodb_buffer_pool_size or os::cpu::cpu_user

attribType – INTEGER | VARCHAR | DECIMAL isCounter – Whether the variable is a counter

(true) or point in time value (false) Counter based variables are always evaluated

using the delta between “now” and “now – 1” collections

Multiple instances of attributes supported internally (for things such as CPUs, disks, etc.)

Single instance collections use a “local” instance

SQL Based Data Collections Defaults in: ./agent/share/mysql-proxy/items/items-mysql-monitor.xml

<class><namespace>mysql</namespace><classname>classTitle</classname><query><![CDATA[SELECT ...]]></query>

</class>

SELECT must return one row (others ignored with warnings in the log file)

Can return multiple columns in 2.x●Assign <classname> in column order

Note: <isCounter> is not available until 2.1 here

SQL Based Data Collections Collections can be added to the default, or a new file New files should be added to the agent-item-files

variable in the mysql-monitor-agent.ini file Let's try an example with the new

PERFORMANCE_SCHEMA!

<class><namespace>mysql</namespace><classname>wait_sum_open_mutex</classname>

<query><![CDATA[

SELECT sum_timer_wait FROM PERFORMANCE_SCHEMA.EVENTS_WAITS_SUMMARY_BY_EVENT_NAME WHERE EVENT_NAME = 'wait/synch/mutex/sql/LOCK_open'

]]></query>

</class>

Lua Based Data Collections

Lua is a powerful embedded language – many possibilities exist to collect data (running external commands, gathering external data by reading files, etc.)

Lua collections also go to: ./agent/share/mysql-proxy/items/ They should also be added to the agent-items-

files variable in the mysql-monitor-agent.ini file We build up the items, attributes and instances

as associative arrays within Lua

Basic Lua Collectionsif not items then

items = { }

end

if not items.oldag then

items["oldag"] = { }

end

items["oldag"]["children"] = {

instances = { { name = "ethan" } },

attributes = {

["couch-dives"] = {

returns = "int"

}

},

collector = function() return { ["couch-dives"] = math.random(100) + 120 } end

}

Matching Lua Arrays to Types

The namespace is the top level of the items array● “oldag”

The classname / type name comes next ● “children”

We then define the instances of the namespace::classname by name

● “ethan” And the attributes and their return types

● “couchdives” returning an int Finally we define a collector function that pushes the

values we want to collect by attribute

Data Collections in the Dashboard

Data collection metadata is stored in the “inventory_*” tables

inventory_namespaces (namespace info) inventory_types (maps to “classname”

information) inventory_attributes (attributes for

namespace::type) inventory_instances (the instance names of

each attribute) inventory_instance_attributes (pivot like table for

attributes to instances mappings)

Data Collections in the Dashboard A useful SQL statement to get DC information:

SELECT DISTINCT namespace, type_name, attribute_name, instance_name FROM inventory_namespaces iin, inventory_types iit, inventory_instance_attributes iia, inventory_instances ii, inventory_attributes ia WHERE iia.instance_id = ii.instance_id AND ii.type_id = iit.type_id AND iin.namespace_id = iit.namespace_id AND iia.attribute_id = ia.attribute_id;

*************************** 1. row *************************** namespace: os type_name: fsattribute_name: fs_avail instance_name: mac:{0025003d17370000}./

Data Collections in the Dashboard Actual data is stored within the “dc_ng_*” tables The collected data is stored in it's raw form, whether it

is a counter based variable or not Currently data is only purged, the infrastructure is

ready for aggregation and aging for historical data in 2.x (we want to test more!)

dc_ng_long_now (integer based data) dc_ng_double_now (doubles) dc_ng_string_now (strings) dc_ng_(long|double)_age(0|1|2) (the rollup tables)

String data obviously can not be aggregated

Data Collections in the Dashboard A useful SQL statement to get the collected data:

SELECT namespace, type_name, attribute_name, instance_name, value, from_unixtime(end_time/1000) FROM inventory_namespaces iin, inventory_types iit, dc_ng_long_now dcn, inventory_instance_attributes iia, inventory_instances ii, inventory_attributes ia WHERE dcn.instance_attribute_id = iia.instance_attribute_id AND iia.instance_id = ii.instance_id AND ii.type_id = iit.type_id AND iin.namespace_id = iit.namespace_id AND iia.attribute_id = ia.attribute_id AND namespace = '...' AND type_name = '...' AND attribute_name = '...';

Creating Custom Graphs

Creating Custom Graphs Graphs also use XML for their structure Graphs can deal with both counter and non-counter

variables 2.0 now allows you to import your own graphs:

Graph XML Template<com_mysql_merlin_server_graph_Design>

<version>...</version> /* Internal graph version numbering */

<uuid>...</uuid> /* Internal graph unique identifier */

<tag>...</tag> /* Tags for the graph (will be exposed soon) */

<name>...</name> /* Visible graph name */

<rangeLabel>...</rangeLabel> /* Visible Y axis range label */

<series>

<label>...</label> /* Visible Series name / label */

<expression>...</expression> /* The expression used for the series */

</series>

<variables>

<name>...</name> /* Arbitrary name for variable in expression */

<dcItem>

<nameSpace>...</nameSpace>

<className>...</className>

<attribName>...</attribName>

</dcItem>

<instance>...</instance> /* The instance of the dc item to be used */

</variables>

</com_mysql_merlin_server_graph_Design>

Building a Graph - Metadata So let's build one up! Who wants to monitor disks? Start with the basic info

<com_mysql_merlin_server_graph_Design><version>1.0</version><uuid>a57c2bba-ea9b-102b-b396-94aca32bee28</uuid>

<name>filesystem usage - /</name> <rangeLabel>MB</rangeLabel> <frequency>00:05:00</frequency>

Note <frequency>, this is not required, and defaults to 1 minute

We use 5 minutes here to share with current collection

Building a Graph - Variables Next, skip to the end, and define the variables

<variables>

<name>used_fs</name>

<dcItem>

<nameSpace>os</nameSpace>

<className>fs</className>

<attribName>fs_used</attribName>

</dcItem>

<instance>/</instance>

</variables>

<variables>

<name>total_fs</name>

<dcItem>

<nameSpace>os</nameSpace>

<className>fs</className>

<attribName>fs_total</attribName>

</dcItem>

<instance>/</instance>

</variables>

</com_mysql_merlin_server_graph_Design>

Building a Graph - Series Now finally define each series

<series><label>used</label><expression>used_fs/1024/1024</expression>

</series><series>

<label>total size</label><expression>total_fs/1024/1024</expression>

</series>

View the Graph!

Graph PERFORMANCE_SCHEMA Viewing Global Mutex Information

Graph PERFORMANCE_SCHEMA Tracking Temporary File IO!

Graphs – Current Caveats

Currently only line graphs are supported Graphs are shared across all servers – so

they will show up even if the instances/collections are not available on a specific server

Graphs sort alphabetically – if you want custom graphs to sort together, keep that in mind

There is no “Delete” for graphs

Creating and Modifying Advisors

Modifying Current Rules

All of the current rules are conservative – they should certainly be modified to your environment

You can edit current rues on the “Manage Rules” page within the Advisors tab

Editing only allows you to modify the THRESHOLD and frequency values for the rule

Copying allows you to alter the expression, and advice or problem descriptions etc., as well as add new variables

Modifying Current Rules

Creating New Rules

Also done on the Manage Rules page Any data collection can be used in a rule, with no limit

on the number of data collections that can be used We still only support the default 3 THRESHOLD values

– Info / Warning / Critical Wrapping the variable names in %percents% allows us

to do variable value substitution within the Advice and Recommended Action sections

We can also format numerical output nicely with

'{'formatNumber:%variable_name%'}' You can make parts bold with two underscores on

either side of a section of the advice etc. too

Creating New Rules

Creating New Rules

Creating New Rules Once saved, you will be able to schedule the rule

within the Advisors->Add to Schedule page

Questions?

Ask me now! Join us at the “Stump the DBA” BoF

tonight Ask a MySQL Support Engineer http://support.mysql.com