nagios conference 2014 - mike weber - expanding nrds capabilities on linux systems

PowerPoint Presentation

NRDS:
Enhancements

Mike Weber

[email protected]

NRDS: Introduction

Basic Functionality
Passive checks with Active Config Updates

Multiple Configurations
Group Check Categories
Create Multiple Check Times

Store-and-Forward
Create timestamps to maintain data in disconnects.

Basic Functions

Passive Checks

Active Configuration Updates


Multiple Group Checks
Grouping checks based on type provide more efficient management aspects. (Linux_OS, Oracle, Apache, etc.)

Multiple Check Times
Multiple check times lessens the load on the server and provides best practice use of resources. (5 min., 30 min.,etc.)

Multiple Download Options
NRDS can be used to download, packages, plugins, scripts,etc.


Why Multiple ConfigurationsReduce Load on Server

Logs Analyzed for Summaries
daily, weekly, monthly

Low Tech Skills

Limited Host Access


Spreading the Load on a Busy Servercommand[Calls: Daily Guest]=/usr/lib/nagios/plugins/daily_guest.sh 2amcommand[Calls: Daily Summary]=/usr/lib/nagios/plugins/daily_summary.shcommand[Calls: Daily Voicemail]=/usr/lib/nagios/plugins/daily_voice.sh2.10amcommand[Daily Web Consol Use]=/usr/lib/nagios/plugins/check_web_consolcommand[Calls: Daily Room]=/usr/lib/nagios/plugins/daily_room.sh2.20amcommand[Calls: Daily Room-Room]=/usr/lib/nagios/plugins/daily_room_room.sh2.30amcommand[Calls: Daily Admin]=/usr/lib/nagios/plugins/daily_admin.sh2.35amcommand[Calls: Daily Admin-Guest]=/usr/lib/nagios/plugins/daily_admin_guest.sh2.45amcommand[Calls: Daily Admin-Room]=/usr/lib/nagios/plugins/daily_admin_room.sh2.55amcommand[Calls: Daily Admin-VoiceM]=/usr/lib/nagios/plugins/daily_admin_voicemail.sh3.05amcommand[Calls: Weekly Guest]=/usr/lib/nagios/plugins/weekly_guest.sh 2.40am on Mondaycommand[Calls: Weekly Voicemail]=/usr/lib/nagios/plugins/weekly_voice.sh2.50am on Mondaycommand[Calls: Weekly Room]=/usr/lib/nagios/plugins/weekly_room.sh3am on Monday


calls/min/ave/maxCalls: Daily Guest2013-09-05 12:31:46from_guest:1054/0/18/446Calls: Daily Room2013-09-01 20:07:23from_room:127/0/15/194Calls: Daily Room-Room2013-09-01 20:07:23from_room_to_room:2/0/0/1Calls: Daily Voicemail2013-09-01 20:07:23from_guest_to_voicemail:24/3/15/56Calls: Monthly Guest2013-09-05 03:20:26from_guest:1054/0/18/446Calls: Monthly Room2013-09-05 03:41:04from_room:127/0/15/194Calls: Monthly Room-Room2013-09-05 03:51:08from_room_to_room:2/0/0/1Calls: Monthly Voicemail2013-09-05 03:30:38from_guest_to_voicemail:24/3/15/56Calls: Weekly Guest2013-09-01 20:07:23from_guest:2108/0/18/446Calls: Weekly Room2013-09-05 04:00:00from_guest_to_voicemail:24/3/15/56Calls: Weekly Room-Room2013-09-05 03:10:34from_room:127/0/15/194Calls: Weekly Voicemail2013-09-01 20:07:23from_guest:1054/0/18/446



Install Procedure
SSH into the hotel pbx and become root.sudo su
cd /tmp
wget --no-check-certificate -Oinstall.tar.gz "https://192.168.1.1/nrdp/?cmd=nrdsgetclient&token=Gh78TFcD&configname=
Unpack the packagetar xzvf install.tar.gzInstall the packagecd clients
./installnrds install 30Note, this install script will be used for every hotel. It is going to install NRDS and download the package for the rest of the installation.Any host called install in the Unconfigured Objects should be removed, the hostname will be changed in the next step.


Install the Packagecd /tmp
tar xvf package64.tar
cd packageNow you will execute a script that will ask for the name of the hotel server and the IP Address of the hotel server, be sure these are absolutely correct as the script will change all of the files to reflect the options entered.Enter the host when asked.Now execute the install script../install.sh


Purpose of package64.tarDownload All Plugins

Download All NRDS Groups
When the script executes it will detect and setup the proper groups.

Download install.sh




Group Configuration Files All Point to One Hostnameadmin_server.cfg admin_server.pl admin_server_updater.plapp_server.cfg app_server.pl app_server_updater.plcustom.cfg custom.pl custom_server_updater.ploracle_db_server60.cfg oracle_db_server60.pl oracle_db_server60_updater.ploracle_db_server.cfg oracle_db_server.pl oracle_db_server_updater.pl


Admin_Server.cfgCONFIG_VERSION="0.2"CONFIG_OS="Linux"CONFIG_NAME="oracle_admin_server"URL="http://192.168.5.5/nrdp/"TOKEN="j5pj5adf89"PLUGIN_DIR="/usr/local/nagios/libexec"SEND_NRDP="/usr/local/nrdp/clients/send_nrdp.sh"TMPDIR="/usr/local/nrdp/clients/tmp"COMMAND_PREFIX=""LOG_FILE=""UPDATE_CONFIG="1"UPDATE_PLUGINS="1"
command[ADMIN]=/usr/local/nagios/libexec/check_url.pl http://example.com:8000
command[Apps Listener]=/usr/local/nagios/libexec/check_procs -c 1:1 -a APPS_$NAGSID -C tnslsnr -u $USER
command[Apache Server]=/usr/local/nagios/libexec/check_http -H example.com -p 8000 -w 5 -c 10
command[OACore Server]=/usr/local/nagios/libexec/check_procs -w 2:5 -c 1:5 -a name=oacore -C java -u $USER
command[OAFM Server]=/usr/local/nagios/libexec/check_procs -w 1:5 -c 1:5 -a name=oafm -C java -u $USER


Admin_Server.plRequired Changes to Custom Groups

my $nrds_updater = "/usr/local/nrdp/clients/nrds/admin_server_updater.pl";

my $configfile = "/usr/local/nrdp/clients/nrds/admin_server.cfg";


Admin_Server_updater.plRequired Changes to Custom Groups

my $configfile = "/usr/local/nrdp/clients/nrds/admin_server.cfg";


Install.sh#!/bin/bashCHECK_ORACLE_HEALTH_FILE=/usr/local/nagios/libexec/check_oracle_healthDEFAULT_CONFIG_FILE="/tmp/package64/nrds/custom.cfg"##### 64 Bit Nagios Monitoring Install #####arch=$(uname -m)if [ ${arch} == 'x86_64' ]; thenecho "64_bit Operating System Detected"yum install -y libaiorpm -Uvh /tmp/package64/rpms/oracle-instantclient11.2-basic-11.2.0.2.0.x86_64.rpmrpm -Uvh /tmp/package64/rpms/oracle-instantclient11.2-devel-11.2.0.2.0.x86_64.rpmrpm -Uvh /tmp/package64/rpms/oracle-instantclient11.2-sqlplus-11.2.0.2.0.x86_64.rpmecho "#####"elseecho "This is not a 64_bit Operating System, this script can only be used for 64_bit Systems"exitfi


Install.sh##### Install Nagios Directories and Plugins #####useradd nagios
mkdir -p /usr/local/nagios/libexec
cd plugins
cp check* /usr/local/nagios/libexec/
cp urlize /usr/local/nagios/libexec/
cp negate /usr/local/nagios/libexec/
chown -R nagios:nagios /usr/local/nagios/libexec/
chmod +x /usr/local/nagios/libexec/*


Install.sh##### Install check_oracle_health #####cd ..cd supportexport ORACLE_HOME="/usr/lib/oracle/11.2/client64"export LD_LIBRARY_PATH="$ORACLE_HOME/lib"tar zxvf check_oracle_health-1.7.8.1.tar.gzcd check_oracle_health-1.7.8.1./configuremakemake installmkdir /var/tmp/check_oracle_healthchown -R nagios /var/tmp/check_oracle_healthecho "ORACLE_HOME=/usr/lib/oracle/11.2/client64"echo "LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib"cp /tmp/package64/support/check_oracle_health /usr/local/nagios/libexec


Install.sh##### VNC Ports #####config_file=${DEFAULT_CONFIG_FILE}VNC_START_PORT=5900

vnc_ports=$(sudo netstat -anptu | grep 'vnc[[:space:]]*$'| tr -s ' '|cut -d ' ' -f4|cut -d ':' -f2)

i=0for port in ${vnc_ports} ; do : $((i++)) printf "command[Check VNC %02d]=/usr/local/nagios/libexec/check_tcp -H localhost -p %d -w 5 -c 8 -e\n" ${i} ${port} >> ${config_file}done


Install.sh##### CPU Core Set Up #####arg0=2arg1=1.5arg2=1arg3=3arg4=2.5arg5=0.5cpu_nb=$(grep -c ^processor /proc/cpuinfo)arg0=$((cpu_nb * arg0))arg1=$(echo "scale=1;${cpu_nb} * ${arg1}" | bc)arg1=$(echo ${arg1} | sed -e 's/\.0$//')arg2=$((cpu_nb * arg2))arg3=$((cpu_nb * arg3))arg4=$(echo "scale=1;${cpu_nb} * ${arg4}" | bc)arg4=$(echo ${arg4} | sed -e 's/\.0$//')arg5=$(echo "scale=1;${arg5} + 2 * ${cpu_nb}" | bc)arg_list=$(printf "%s %s,%s,%s %s %s,%s,%s" \ "-w" ${arg0} ${arg1} ${arg2} \ "-c" ${arg3} ${arg4} ${arg5})echo "command[Check Load]=/usr/local/nagios/libexec/check_load ${arg_list}" >>\ ${config_file}


Install.sh##### Set Hostname #####host=$(hostname)echo $hostecho "#####"echo "Use the hostname above as a base as you type the server name"read nameecho "#####"oldString="txb"newString="$name"dpath="/tmp/package64/nrds/*.cfg"sed -i "s/${oldString}/${newString}/g" /tmp/package64/crontabfor i in $dpath; dosed -i "s/${oldString}/${newString}/g" $idone


Install.sh##### Determine Linux Version #####version=$(lsb_release -rs|cut -f1 -d.)echo $version

if [ $version == '6' ]; thenecho "version 6"yum install -y glibc\* perl-YAML perl-Time-HiReselif [ $version == '5' ]; thenecho "version 5"yum install -y glibc\* perl-YAMLrpm -ivh /tmp/package64/rhel5/perl-Time-HiRes-1.9724-1.el4.rfx.x86_64.rpmelseecho "This install script will not work for RHEL 4"exitfi


Install.sh##### Manage Nagios Environement #####echo 'NAGPATH=/usr/local/nagios/libexec; export NAGPATH' >> /home/nagios/.bash_profileecho 'LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib; export LD_LIBRARY_PATH' >> /home/nagios/.bash_profileecho 'ORACLE_HOME=/usr/lib/oracle/11.2/client64; export ORACLE_HOME' >> /home/nagios/.bash_profileecho 'TNS_ADMIN=/home/nagios; export TNS_ADMIN' >> /home/nagios/.bash_profileecho 'PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/nagios/bin:$ORACLE_HOME/bin; export PATH' >> /home/nagios/.bash_profileecho 'HOME=/home/nagios; export HOME' >> /home/nagios/.bash_profile


Install.sh##### Oracle DB Plugins Install #####cp /tmp/package64/nrds/oracle_db* /usr/local/nrdp/clients/nrds/


Install.sh##### App Server Install #####APP_SERVER=$(ps -ef|grep tnslsnr|grep -v grep|wc -l)if [ ${APP_SERVER} == '1' ]; thenecho "Installing Application Server"cp /tmp/package64/nrds/oracle_app* /usr/local/nrdp/clients/nrds/sed -i 's/\#\*\/5/\*5/' /tmp/package64/crontabelseecho "No Application Server Installed...no plugins will be installed for App Server"fi


Install.sh##### Admin Server Install #####ADMIN_SERVER=$(ps -ef|grep FNDLIBR|grep -v grep|wc -l)if [ ${ADMIN_SERVER} == '1' ]; thenecho "Installing Admin Server"cp /tmp/package64/nrds/oracle_admin* /usr/local/nrdp/clients/nrds/sed -i 's/\#\*\/6/\*6/' /tmp/package64/crontabelseecho "No Admin Server Installed...no plugins will be installed for Admin Server"fi


Install.sh##### Move New Configs #####cd /tmp/package64/nrdscp nrds* /usr/local/nrdp/clients/nrds/cp custom* /usr/local/nrdp/clients/nrds/chown nagios:nagios /usr/local/nrdp/clients/nrds/*

##### Install Cron Jobs #####/tmp/package64/crontab

exit


Crontab Script#!/bin/bashcrontab -u nagios -rcrontab -u nagios -l | { cat; echo "### Cron Jobs for nagios ######## OS Metric Checks and Oracle Server Checks #####*/5 * * * * /usr/local/nrdp/clients/nrds/nrds.pl -H txb 2>&1*/4 * * * * /usr/local/nrdp/clients/nrds/custom.pl -H txb 2>&1*/6 * * * * /usr/local/nrdp/clients/nrds/oracle_db_server.pl -H txb 2>&1#*/5 * * * * /usr/local/nrdp/clients/nrds/oracle_app_server.pl -H txb 2>&1#*/6 * * * * /usr/local/nrdp/clients/nrds/oracle_admin_server.pl -H txb 2>&101 * * * * /usr/local/nrdp/clients/nrds/oracle_db_server60.pl -H txb 2>&1"; } | crontab -u nagios -exit

Store-and-Forward

Basic Problem
Network connections broken for multiple days.

Solution: NRDS Timestamps
Using timestamps NRDS is able to rebuild data and graphing on the Nagios server.

Store-and-Forward

Example: Broken Connection
Network connections breaks 5:17 AM.

Store-and-Forward

Network connections breaks 5:17 AM.

Store-and-Forward

Network connections restored 6:30 AM.

Store-and-Forward

Network connections restored 6:30 AM.

Store-and-Forward

Network connections breaks 2:20 PM.

Store-and-Forward

Network connections breaks 2:20 PM.

Store-and-Forward

Network connections restored 6:26 AM (16 hours).

Store-and-Forward

Network connections restored 6:26 AM (16 hours).

Store-and-Forward

NRDS ClientAdd a cache line in /usr/local/nrdp/clients/nrds/nrds.cfgCACHEDIR="/usr/local/nrdp/clients/cache"

Create folder /usr/local/nrdp/clients/cacheThe cache needs to collect timestamped data in order to rebuild the information later.mkdir /usr/local/nrdp/clients/cache
chown nagios.nagios /usr/local/nrdp/clients/cache
chmod 0775 /usr/local/nrdp/clients/cacheTimestamps Being Createddrwxr-xr-x 2 nagios nagios 4096 Sep 20 07:39 mylock -rw------- 1 nagios nagios 1742 Sep 20 07:24 nrdp.6pvqRR1IJ -rw------- 1 nagios nagios 1742 Sep 20 07:30 nrdp.B6qVekdsa -rw------- 1 nagios nagios 1742 Sep 20 07:36 nrdp.biayklzbM -rw------- 1 nagios nagios 1742 Sep 20 07:33 nrdp.CuPu74NX6

Store-and-Forward

NRDS ClientReplace /usr/local/nrdp/clients/send_nrdp.sh original if [ "x$url" == "x" -o "x$token" == "x" ] echo "Usage: send_nrdp -u url -t token"

Reference to cache directory if [ "x$url" == "x" -o "x$token" == "x" -o "x$directory" == "x" ] echo "Usage: send_nrdp -u url -t token -D directory"

Store-and-Forward

NRDS ClientReplace /usr/local/nagios/libexec/process_perfdata.pl chown nagios.nagios /usr/local/nagios/libexec/process_perfdata.plchmod 0755 /usr/local/nagios/libexec/process_perfdata.pl

if ( $NAGIOS{PERFDATA} =~ /\s*__check_timestamp__=(\d+)/ ) { $NAGIOS{TIMET} = $1; print_log( "Found encoded timestamp ($NAGIOS{TIMET}) in $NAGIOS{HOSTNAME} / $NAGIOS{SERVICEDESC} ($NAGIOS{PERFDATA})", 1 ); # remove encoded timestamp $NAGIOS{PERFDATA} =~ s/(\s*__check_timestamp__=\d+)//; print_log( "Perfdata now is: ($NAGIOS{PERFDATA})", 2 ); 218a231,232 } print_log( "Processing Line [$_]", 2 );

Script Downloads

NRDS Modifications:

Beginlinuxservers.com/nagiosconf

User: nagiosconference
Password: 54TBwh9

Only available during conference.

Conclusion

Mike Weber

[email protected]

Click to edit Master title style

Click to edit Master text styles

Second level

Third level

Fourth level

Fifth level

9/22/14


Click to edit Master subtitle style

9/22/14


Second level
Third level
Fourth level
Fifth level

9/22/14



9/22/14


Second level
Third level
Fourth level
Fifth level

Second level
Third level
Fourth level
Fifth level

9/22/14



Second level
Third level
Fourth level
Fifth level


Second level
Third level
Fourth level
Fifth level

9/22/14


9/22/14

9/22/14


Second level
Third level
Fourth level
Fifth level


9/22/14



9/22/14



Second level

Third level

Fourth level

Fifth level

9/22/14



Second level

Third level

Fourth level

Fifth level

9/22/14

PRESENTATION TITLE

Presenter Name

9/22/14

[email protected]



Second level

Third level

Fourth level

Fifth level

9/22/14

PRESENTATION TITLE

Presenter Name

9/22/14

[email protected]


Second level
Third level
Fourth level
Fifth level

9/22/14



9/22/14


Second level
Third level
Fourth level
Fifth level

Second level
Third level
Fourth level
Fifth level

9/22/14



Second level
Third level
Fourth level
Fifth level


Second level
Third level
Fourth level
Fifth level

9/22/14


9/22/14

9/22/14


Second level
Third level
Fourth level
Fifth level


9/22/14



9/22/14



Second level

Third level

Fourth level

Fifth level

9/22/14



Second level

Third level

Fourth level

Fifth level

9/22/14

nagios conference 2014 - mike weber - expanding nrds capabilities on linux systems

Technology