alerting with mysql and nagios sheeri cabral senior db admin/architect mozilla @sheeri
DESCRIPTION
Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri. What is Monitoring?. Threshold alerting Graphing/trending. Why Monitor?. Problem alerting Find patterns Early warning for potential issues. What to Alert?. Problems you can fix max_connections - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/1.jpg)
Alerting With MySQL and Nagios
Sheeri CabralSenior DB Admin/Architect
Mozilla@sheeri
![Page 2: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/2.jpg)
What is Monitoring?
Threshold alerting
Graphing/trending
![Page 3: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/3.jpg)
Why Monitor?
Problem alerting
Find patterns
Early warning for potential issues
![Page 4: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/4.jpg)
What to Alert?
Problems you can fix
– max_connections
– long running queries
– locked queries
– backup disk space
![Page 5: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/5.jpg)
What to Alert?
Problems you can fix
– max_connections
– long running queries
– locked queries
– backup disk space
![Page 6: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/6.jpg)
Open Source Alerting Solutions
Nagios Cacti Munin Zabbix Zenoss OpenNMS
![Page 7: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/7.jpg)
Commercial AlertingSolutions
MySQL Enterprise Monitor
MONyog
Oracle Grid Control
Hyperic HQ
![Page 8: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/8.jpg)
Best to use what you already have
![Page 9: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/9.jpg)
Nagios is great because...
...anyone can write a plugin
![Page 10: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/10.jpg)
The problem with Nagios...
...anyone can write a plugin
![Page 11: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/11.jpg)
Nagios Flexibility
Templates Page after x failures
– Network, backup disk Time between checks Escalations Dependencies Time periods
![Page 12: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/12.jpg)
Official Nagios Pluginsfor MySQL
check_mysql
check_mysql_query
![Page 13: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/13.jpg)
check_mysql
db connectivity slave running slave lag using seconds_behind_master
![Page 14: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/14.jpg)
check_mysql_query
Checks the output of a query is within a certain range (numerical)
![Page 15: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/15.jpg)
System vars Status vars Caching Calculations
check_mysql yes yes no no
check_mysql (standard)
no yes no no
check_mysqld_status no yes no no
check_mysql_stats yes no yes no
check_mysqld no many no no
check_mysql_health yes yes no Hard-coded
check_mysql yes yes no Hard-coded
Third party plugins
![Page 16: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/16.jpg)
“Let us know what you'd like to see!”
![Page 17: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/17.jpg)
What I wanted
System vars Status vars Caching Calculations
mysql_health_check.pl yes yes yes flexible
![Page 18: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/18.jpg)
Caching
Save information to a file --cache-dir /path/to/dir/
Use the file instead of connecting again
--max-cache-age <seconds>
--no-cache to force connection
![Page 19: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/19.jpg)
--mode=varcomp
%metadata{varstatus} SHOW GLOBAL VARIABLES SHOW GLOBAL STATUS
--expression allows word replacement
--warning --critical are flexible
![Page 20: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/20.jpg)
Sample Command Definitiondefine command {
command_name check_mysql_tmp_tables
command_line $USER1$/mysql_health_check.pl
--hostname $HOSTADDRESS$ --user myuser --password mypass
--cache-dir=/var/lib/nagios/mysql_cache
--max-cache-age=300
--mode=varcomp
--expression=
"Created_tmp_tables/Uptime_since_flush_status"
--warning=">(8/60)" --critical=">(10/60)"
}
![Page 21: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/21.jpg)
Sample Service Definition
define service { use generic-service host_name __HOSTNAME__ service_description MySQL Temp Tables check_command check_mysql_tmp_tables}
![Page 22: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/22.jpg)
--mode=long-query--mode=locked-query
%metadata{proc_list} SHOW FULL PROCESSLIST
![Page 23: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/23.jpg)
Rates
Compare to last run mode=lastrun-varcomp
![Page 24: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/24.jpg)
Sample Command Definition
define command {
command_name check_mysql_locked_queries
command_line $USER1$/mysql_health_check.pl
--hostname $HOSTADDRESS$ --user myuser --password mypass
--cache-dir=/var/lib/nagios/mysql_cache
--max-cache-age=300
--mode=locked-query
--warning=$ARG1$ --critical=$ARG2$
}
![Page 25: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/25.jpg)
Extending Information
sub fetch_server_meta_data {}
add a new hash key to %metadata
$metadata{innodb_status} =
$dbh->selectall_arrayref(
q|SHOW ENGINE INNODB STATUS|);
![Page 26: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/26.jpg)
Currently Unused
• %metadata{innodb_status}
– SHOW ENGINE INNODB STATUS
• %metadata{master_status}
– SHOW MASTER STATUS
• %metadata{slave_status}
– SHOW SLAVE STATUS
![Page 27: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/27.jpg)
“Standard” checks using this plugin
High Priority:
• % max connections
--expression 'Threads_connected/max_connections*100'
• locked queries
• long-running queries
• InnoDB enabled
--expression
“have_innodb” --comparison=”ne 'YES'”
![Page 28: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/28.jpg)
“Standard” checksusing this plugin
% of sleeping connections
# connected, # running, # max connections
--expression="(Threads_connected-Threads_running)/max_connections * 100" --comparison=">$ARG1$"
slaves set as read-only, masters not
--expression “read_only” --comparison=”ne 'YES'”
![Page 29: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/29.jpg)
Limitations
One check/calculation per Nagios service
– Many variables
– Cache output Does not output for performance data
![Page 30: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/30.jpg)
Nagios Plugin for Partitions
table_partitions.pl --host
--user --pass
--database --table
--range [days|weeks|months]
--verify #
![Page 31: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/31.jpg)
Nagios Plugin for Partitions
table_partitions.pl --host db1.mozilla.com
--user nagiosuser --pass nagiospass
--database addons --table user_addons
--range months
--verify 3
![Page 32: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/32.jpg)
Other monitoring tools
Sar, ksarIostat
VmstatNetstat
pt-config-diffpt-stalk
![Page 33: Alerting With MySQL and Nagios Sheeri Cabral Senior DB Admin/Architect Mozilla @sheeri](https://reader035.vdocument.in/reader035/viewer/2022062222/56814cac550346895db9b265/html5/thumbnails/33.jpg)
Download Today!
www.palominodb.com->Community->Projects
OurSQL podcast (oursql.com)
MySQL Administrator's Bible
youtube.com/tcation
http://planet.mysql.com