bsd-basedsystemsmonitoringwithicinga2andopensshbcr/tutorials/icinga.pdf ·...

BSD-based Systems Monitoring with Icinga2 and OpenSSH


Benedict [email protected]

BSDCan 2019 Tutorial

1 / 66


Monitoring

When we talk about monitoring, we talk about collecting valuable information from targetsystems to help system administrators and others stay informed about their ever-increasinghardware and software landscape. Problems can be detected early and proactively, without littleor no visible impact to the users of these systems. It is also a tool for decision-makers abouthow well service-levels are provided, outdated systems and services that could be replaced, aswell as new systems to be introduced and monitored the same way as done on previous systems.Typically, monitoring deals with the following three aspects:

Availability - Is the host or service available on the network?Metrics - Data collected from hosts and services

Logs - Messages (errors, warnings) written to log files from hosts and servicesWe’ll cover the first two aspects in these slides. The third one warrants its own due to size andcomplexity of the topic. Icinga is used to demonstrate one of many monitoring solutions due toits programmability, flexibility, and scalability with the number of hosts and services to monitor.

2 / 66

BSD-based Systems Monitoring with Icinga2 and OpenSSHIntroduction to Icinga

Overview

1 Introduction to Icinga

2 Installing the Monitoring SystemDatabase SetupNginx ConfigurationIcingaweb2 Setup

3 Configuring Icinga2Introduction to Monitoring ObjectsPlugin IntegrationMonitoring via OpenSSH

SSH SetupCreating SSH Checks

4 Restarting Services Automatically

3 / 66

BSD-based Systems Monitoring with Icinga2 and OpenSSHIntroduction to Icinga

Introduction to Icinga2

In this tutorial, we’ll take a closer look at using and configuring the unofficial Nagiosreplacement called Icinga2 to cover our monitoring needs. Icinga is a modern system thatbuilds on many of the concepts that made Nagios great, but adds their own, including amodern web interface called Icingaweb2. Plugins from Nagios can be used in Icinga withoutmodifications and there is a whole ecosystem of new plugins available for Icinga.Icinga was developed due to lack of response from the Nagios authors to react to bugs andfeature requests. The Icinga project was created to integrate those and has since rewrittenmany of the backend components, while still maintaining compatibility to Nagios plugins asmuch as possible. Icinga offers its own ecosystem of plugins and there is an active open sourcecommunity that develops the core Icinga system and creates new plugins(https://www.monitoring-plugins.org).There is also training, consulting, and support available for large and complex Icinga2installations.

4 / 66

https://www.monitoring-plugins.org

BSD-based Systems Monitoring with Icinga2 and OpenSSHInstalling the Monitoring System

Overview






5 / 66


Installing Required Software Packages

Icinga2 and its Icingaweb2 graphical component is a classic LAMP stack, but can also use ofsome other components. In this tutorial, we’ll use Nginx as the webserver and PostgreSQL asdatabase backend. The port is still using PostgreSQL 9.5 at the time of this writing. Newerversions of PHP should either be pulled in as dependencies automatically or can be selectedbased on your own experiences. There is no need to use the latest versions and teh monitoringsystem will run just fine with older versions.We’ll install the following packages (the rest is pulled in as dependencies):• pkg install icinga2 icingaweb2 postgresql95-server nginxImageMagick-nox11 php72-pecl-imagick

The ImageMagick ports are required for by Icingaweb2 during setup and are used for reportgeneration. If that part is not used, omit this port.

6 / 66


Adding Services to /etc/rc.conf

We need to start three components of our monitoring system:1 The backing PostgreSQL database2 The Icinga2 monitoring core component3 The Nginx webserver with fastCGI support for Icingaweb2

Using sysrc, this is not difficult:

sysrc icinga 2_ enable=yessysrc postgresql_enable=yessysrc nginx_enable=yes

Don’t start these services just yet, we have a bit of configuration ahead of us! But before weforget, we should rotate our log files:

# cp /usr/local/share/examples/icinga2/newsyslog/icinga2 /etc/newsyslog.conf.d/

We’ll start our configuration with the database setup.

7 / 66


Database Setup

Overview






8 / 66


Database Setup

Database Setup

First of all, when running on OpenZFS, we’ll create a dataset for PostgreSQL and set amountpoint. Lastly, we’ll change ownership to the pgsql user account that will run the servicelater.

# zfs create -o mountpoint =/usr/local/pgsql/data sys/pgdata# zfs set recordsize =8k sys/pgdata# zfs set logbias=throughput sys/pgdata# zfs set redundant_metadata=most sys/pgdata# zfs set primarycache=metadata sys/pgdata# chown pgsql:pgsql /usr/local/pgsql/data

The ZFS dataset settings are typical for any kind of PostgreSQL database. There are no heavyqueries to the database, but Icinga2 still needs to store a lot of information about monitoringobjects, it’s internal state, user accounts, etc.

9 / 66


Database Setup

Initializing the Database ClusterWe’ll switch to the pgsql user account and initialize the database cluster.# su pgsql$ cd$ initdb -D ./data --no-locale --encoding=utf -8 --lc -collate=C$ pg_ctl start -D ./data

After we started the database cluster, we’ll create a user account in the database for Icinga andset a password. We need this later when setting up the icinga tables and Icingaweb2. Then, weset up a database with the same name to hold the tables.$ createuser -dPrs icingaPassword:Re -type password:$ createdb -O icinga -E UTF8 icinga

Then, we create local access entries for the Icinga user in pg_hba.conf. Make sure not tochange the trust parts into ident as this will not work.local icinga icinga md5host icinga icinga 127.0.0.1/32 md5host icinga icinga ::1/128 md5

10 / 66


Database Setup

Importing the Icinga2-Schema

The database tables and indexes that Icinga2 needs to store various information are stored in/usr/local/share/icinga2-ido-pgsql/schema/pgsql.sql as a schema.

$ psql -U icinga -d icinga < /usr/local/share/icinga2-ido-pgsql/schema/pgsql.sql

Future versions of Icinga2 might require schema upgrades when updating to a newer version.The upgrade instructions will tell you when and how to perform the schema updates.Intermediate upgrades need to be applied as well, but they are all timestamped and can beimported in the same way as described here for the initial schema. The upgrades are located in/usr/local/share/icinga2-ido-pgsql/schema/upgrade.

11 / 66


Database Setup

Configuring the IDO databaseThe IDO (Icinga Data Output) will store information about our monitoring instance in thePostgreSQL database. We need to store our database connection credentials in the file/usr/local/etc/icinga2/features-enabled/ido-pgsql.conf with the following content:object IdoPgsqlConnection "ido -pgsql" {

user = "icinga"password = "<the_password_for_the_database_user_icinga_we_defined_earlier >"host = "localhost"database = "icinga"

}

Next, we log out of the psql user account and activate the ido-pgsql, api, and commandfeatures using the icinga2 CLI.# icinga2 feature enable ido -pgsql api command# icinga2 feature list

The feature list subcommand should list the ido-pgsql under the Enabled features. Theonly thing left for the database part is to restart the postgresql service and icinga2.# service postgresql restart# service icinga2 start

12 / 66


Nginx Configuration

Overview






13 / 66


Nginx Configuration

Nginx ConfigurationIcinga2 will work fine without the Icingaweb2 component and is in fact running along on manymachines. However, a central management interface is good to get an overview and centralpoint of data collection where administrators can see what kind of events are generated. Also,users can be given access to a subset of the monitored hosts and/or services without being ableto see and manage other objects. For these users, it is often easier to use a web interfacerather than a CLI. The first part of the webserver setup deals with getting PHP setup correctlyas a fastCGI process that Nginx runs. First, we enable both nginx and php_fpm in ourrc.conf, but do not start the services yet.# sysrc php_fpm_enable=yes# sysrc nginx_enable=yes

Then, we’re using sed to uncomment some options in www.conf in the/usr/local/etc/php-fpm.d/ directory.WWWCONF =/usr/local/etc/php -fpm.d/www.conf# sed -i '' "s/listen\ =\ 127.0.0.1:9000/ listen\ =\ \/var\/run\/php5 -fpm.sock/" $WWWCONF# sed -i '' "s/; listen.owner/listen.owner/" $WWWCONF# sed -i '' "s/; listen.group/listen.group/" $WWWCONF# sed -i '' "s/; listen.mode/listen.mode/" $WWWCONF

14 / 66


Nginx Configuration

Changes to nginx.conf

We need to add the following lines before location / { ... } in/usr/local/etc/nginx/nginx.conf:

location ~ ^/ icingaweb2/index\.php (.*)$ {fastcgi_pass unix:/var/run/php5 -fpm.sock;fastcgi_index index.php;include fastcgi_params;fastcgi_param SCRIPT_FILENAME /usr/local/www/icingaweb2/public/index.php;fastcgi_param ICINGAWEB_CONFIGDIR /usr/local/etc/icingaweb2;fastcgi_param REMOTE_USER $remote_user;

}

location ~ ^/ icingaweb2 (.+)? {alias /usr/local/www/icingaweb2/public;index index.php;try_files $1 $uri $uri/ /icingaweb2/index.php$is_args$args;

}

15 / 66


Nginx Configuration

Setting up the Icinga CA

Icinga2 creates it’s own certificate authority for trusted API access. We activated the apifeature already (check the output of icinga2 feature list to confirm). We need toinitialize the CA to create keys and certificates using:

# icinga2 api setup

Next, we go into Icinga2’s main configuration directory (/usr/local/etc/icinga2/conf.d/)and edit api-users.conf. It defines all the users that can access the API and theirpermissions. We need to create a separate ApiUser for the Icingaweb2 interface. Add a sectionlike this to the file:

object ApiUser "icingaweb2" {password = "<mypassword >"permissions = [ "status/query", "actions /*", "objects/modify /*", "objects/query/*" ]

}

This user will be needed in the Icingaweb2 setup that is coming soon.

16 / 66


Nginx Configuration

Finishing Up the Nginx Configuration

A few more things remain: we copy the php.ini intended for production use to the properlocation and change the timezone the server is in.Then, we can start both the php fastCGI process and the nginx webserver:

# service nginx start# service php -fpm start

Once these services are running, we can start configuring the Icingaweb2 interface from a webbrowser of choice. Browse to http://local.IP.or.domain/icingaweb2/setup

17 / 66


Icingaweb2 Setup

Overview






18 / 66


Icingaweb2 Setup

Setup Steps in the Browser: Setup Token Creation

First we need to enter the setup token. It protects the installation from other people whoencounter this page while the setup is not completed yet. Without the proper token, they cannot proceed.

19 / 66


Icingaweb2 Setup

Setup Steps in the Browser: Setup Token CreationTo create the token, run the following:sudo /usr/local/www/icingaweb2/bin/icingacli setup token create \--config=/usr/local/etc/icingaweb2Example output: The newly generated setup token is: ba63907b6ce1ecacMake sure to set owner and group properly:sudo chown -R www:www /usr/local/etc/icingaweb2Then, back in the browser window, enter the generated token into the form and press the"Next" button:

20 / 66


Icingaweb2 Setup

Setup Steps in the Browser: Select Modules

In this screen, select at least the monitoring module. Documentation is recommended fornewcomers. You can find all the information on icinga.com, too.

21 / 66

icinga.com


Icingaweb2 Setup

Setup Steps in the Browser: Check Dependencies

The setup checks if all components for Icingaweb2 are installed. Make sure that no item is red.You can ignore the yellow message in the "Linux Platform" section.

22 / 66


Icingaweb2 Setup

Setup Steps in the Browser: Authentication Setup

Here we define how Icingaweb2 should authenticate within the backend services. We selectdatabase here.

23 / 66


Icingaweb2 Setup

Setup Steps in the Browser: Configure Database Backend

We enter the database information of your previously setup postgres database. Check it with"Validate Configuration".

24 / 66


Icingaweb2 Setup

Setup Steps in the Browser: Setup Administrator Account for Icingaweb2

Next, we set up the administrator account. Chose a strong password. This account can createother accounts later.

25 / 66


Icingaweb2 Setup

Setup Steps in the Browser: Configure the Icingaweb2 Application/Logging

Here we chose to store all logging and application information in our database. Select"Webserver Log" as the logging type.

26 / 66


Icingaweb2 Setup

Setup Steps in the Browser: Summary

This is just a summary screen about what we configured so far.

27 / 66


Icingaweb2 Setup

Setup Steps in the Browser: Configure Monitoring Module

Next, we set up the monitoring module we selected earlier.

28 / 66


Icingaweb2 Setup

Setup Steps in the Browser: Configure IDO as Monitoring Backend

We chose to retrieve and store all monitoring data in the IDO (Icinga Data Output).

29 / 66


Icingaweb2 Setup

Setup Steps in the Browser: Configure IDO Ressource

The IDO itself is stored in our postgresql database, so we enter the connection information hereagain. Click "Validate Configuration" to see if it can establish a connection before proceeding.

30 / 66


Icingaweb2 Setup

Setup Steps in the Browser: Command Transport

Icinga2 can be controlled by a comprehensive API, for which we defined a separate ApiUsercalled icingaweb2 before. We need to enter it here and the password from/usr/local/etc/icinga2/conf.d/api-users.conf.

31 / 66


Icingaweb2 Setup

Setup Steps in the Browser: Protected Custom Variables

Leave these default settings as they are.

32 / 66


Icingaweb2 Setup

Setup Steps in the Browser: Summary

We’re almost done at this point. We just get another summary of the settings we’ve made.

33 / 66


Icingaweb2 Setup

Setup Steps in the Browser: Success!

34 / 66

Time for a break

BSD-based Systems Monitoring with Icinga2 and OpenSSHConfiguring Icinga2

Introduction to Monitoring Objects

Overview






36 / 66



Creating and including our own configuration filesIcinga2 provides a number of default configuration files in/usr/local/etc/icinga2/conf.d/ with basic settings. For a beginner, they can be a bitoverwhelming, so we move them out of the way (for reference) and create our own config filesone after the other. That way, we understand how these files work together and we’re notsurprised by things we have not defined yet.

cd /usr/local/etc/icinga 2//usr/local/etc/icinga 2# mv conf.d conf.orig/usr/local/etc/icinga 2# mkdir tutorial

We need to tell Icinga to include that directory and all it’s subdirectories we’re going to create.This is done in /usr/local/etc/icinga2/icinga2.conf. We’ll comment the lineinclude_recursive "conf.d" and add one for our newly created tutorial directory belowit:

// include_recursive "conf.d"include_recursive "tutorial"

Make sure to copy the api-users.conf and other files we created to the tutorial directory.37 / 66



Monitoring Objects in IcingaIcinga used various objects in it’s configuration files that have certain attributes. They areencoded in a human-readable way. The general syntax is as follows:

object <Object > "Name" {attribute(s) = value

}

Object types could be Host, Service, User, Template, and many others.The following value types can be used:

Type ExampleNumber 42Duration 5mString "This is a comment"Boolean trueArray [ "value1", "value2" ]Dictionary "key1" = "value1", "key2" = 123

38 / 66



Defining a HostCreating a host object is one of the basic tasks that should be performed at the beginningwhen setting up a new monitoring system. It helps us structure our environment and list hoststhat we want to monitor, defines common checks, names, etc. A basic host object is defined in/usr/local/etc/icinga2/tutorial/host.conf like this:

object Host "Hal9000" {address = "1.2.3.4"check_command = "hostalive"

}

We’ve defined a basic host object and assigned a basic service check that will alert us if thehost is unreachable.To validate and make the new object known to Icinga, we run the following command, whichwill fail if there are any config errors:

# icinga2 daemon -C && service icinga2 reload

If all went well, we should see the new host in Icingaweb2.

39 / 66



More Host Attributes

The Host object can take more optional parameters, but check_command must always beprovided to be a valid host object. Other interesting attributes that we are going to use laterare listed below:display_name (string): An alternative name to be displayed instead of the object name.address6 (string): The IPv6 address of the host.groups (string): A list of groups this host is a member of.vars (Dictionary): Custom attributes specific to this host than can be freely defined.check_interval (Duration): The interval in which the host should be checked in seconds.event_command (Object name): Name of an event command to be executed if the host

changes state.

40 / 66



Displaying Host AttributesThe Icinga CLI can display the current and default attributes for an object using the followingsyntax:

icinga2 object list --type Host --name Hal9000

The information displayed is valuable in debugging (where are objects defined) and what kindof values are currently set, without having to look into individual config files.

Object 'Hal9000' of type 'Host ':% declared in '/usr/local/etc/icinga 2/ tutorial/hal 9000. conf ', lines 2:1 -2:17* __name = "Hal 9000"* action_url = ""* address = "1.2.3.4"* check_command = "hostalive"

% = modified in '/usr/local/etc/icinga 2/ tutorial/hal 9000. conf ', lines 11:3 -11:29* check_interval = 300* check_period = ""* check_timeout = null* command_endpoint = ""* display_name = "Hal 9000"...

Remove the filters --type and --name to display all objects that Icinga2 knows about.41 / 66



Service Objects

We’re not only interested in monitoring server uptime, but also whether individual servicesrunning on these hosts are working fine (whatever that means). For that, we can define Serviceobjects in a file called services.conf. Again, we use the general syntax, but use Service asthe object type. Each service needs to be associated with a certain host object (attributehost_name), since not all hosts are running all the services.

object Service "ping4" {host_name = "Hal9000"check_command = "ping4"

}

Let Icinga check it’s configuration and reload the icinga service to see the newly created object.

42 / 66



Defining GroupsGroup objects help to logically group hosts or services together based on similar attributes. Forexample, hosts from a certain subnet mask could form a group or those with a certain numberof CPUs. The most common way for defining groups is based on variables. We’ll add a variablecalled vars.os to our host object to filter on. The file looks like this now:

object Host "Hal9000" {address = "1.2.3.4"check_command = "hostalive"vars.os == "Linux"

}

We can then define the group object in the file groups.conf in the following way:

object HostGroup "Linux Machines" {assign where host.vars.os == "Linux"

}

object HostGroup "FreeBSD Machines" {assign where host.vars.os == "FreeBSD"

}

43 / 66



Assigning Services to Hosts using Apply Rules

As our infrastructure and the number of objects (hosts and services) grows, it becomes tediousto add a service check for each new host and services running on that host. Instead, it wouldbe great if Icinga would apply service checks to new hosts automatically, based on certainattributes. This can be achieved by defining variables like vars.os for example and then use socalled apply rules to assign services based the variable values. That way, we only need todefine host objects and add new services when they are introduced and let Icinga take care ofthe mapping for us. We also don’t clutter our config files with a lot of definitions for each andand every host or service, but keep them orderly and small enough to manage.

44 / 66


Plugin Integration

Overview






45 / 66


Plugin Integration

Integrating External PluginsIcinga2 comes with a range of plugins for checking various standard system parameters (CPUand memory load, disk space, etc). In case this is not enough and other things should bemonitored, we need to integrate these external plugins into our monitoring infrastructure. Wefind these plugins either on sites like https://www.monitoring-plugins.org or in the portscollection. Remember that Icinga can also run Nagios plugins without modification. Forexample, these are just a few plugins that are available on FreeBSD1:

# pkg install nagios-check_zpools nagios-check_cpu_usage nagios-check_ports

Some plugins for the BSD community:• https://exchange.icinga.com/araujobsd/icinga2-network-plugin• https://exchange.icinga.com/bmccorkle/check_bsd_mem• https://exchange.icinga.com/anaumov/OpenBSD%20SNMPv3%20general%20check

The plugins are installed in the /usr/local/libexec/nagios/ directory.

1Thank you porters!46 / 66


https://exchange.icinga.com/araujobsd/icinga2-network-plugin

https://exchange.icinga.com/bmccorkle/check_bsd_mem

https://exchange.icinga.com/anaumov/OpenBSD%20SNMPv3%20general%20check


Plugin Integration

Testing the External Plugins

First, we need to see whether the plugins work as expected for our monitored infrastructure. Tothat end, we execute them manually from /usr/local/libexec/nagios/ to check theiroutput or if they require any parameters. For example, many plugins require the warning (-w)and critical (-c) thresholds to be set to send alerts when they are reached.

/usr/local/libexec/nagios/check_zpools -p ALL -w 80 -c 90

In this case, we define that all ZFS pools should be checked and when they reach 80% full, aWARNING status is sent to Icinga. Similarly, when 90% are reached, the plugin emitsCRITICAL.

47 / 66


Plugin Integration

Integrating the Plugin into our InfrastructureEach machine that should be checked needs to have the plugin installed/copied to/usr/local/libexec/nagios/. First, we add a new CheckCommand entry for the plugin in/usr/local/etc/icinga2/tutorial/nagios_plugins.conf:object CheckCommand "zpools" {

import "plugin -check -command"command = [ PluginDir + "/check_zpools" ]arguments = {

"-p" = "ALL""-w" = "80""-c" = "90"

}}

In services.conf, we define which hosts should execute the plugin:apply Service "zpools" {

import "generic -service"check_command = "zpools"assign where host.vars.os == "FreeBSD"

}

Next, restart Icinga2: sudo icinga2 daemon -C && sudo service icinga2 restart48 / 66


Plugin Integration

Example Output in Icingaweb2 for the check_ports Plugin

49 / 66


Monitoring via OpenSSH

Overview






50 / 66



Rationale for Monitoring Hosts via OpenSSH

There are a number of benefits for using OpenSSH to execute checks on remote hosts andreport the results back to the Icinga server.

Availability Most, if not all Unix hosts have SSH enabled for remote management, de factostandard. Little to no extra setup required.

Secure Check results represent the state of remote system and can contain data thatcan be valuable to an attacker. SSH encrypts all the data to and from theremote machine so that listening on the wire will not provide more informationto an attacker.

One downside is that logs are increasing as every connection (depending on the check interval)will be logged auth.log). This may not be ideal on systems with not a lot of disk space(embedded, network equipment) available and without a proper log analysis solution, mighthide other important authentication log entries in the flood. With proper log rotation andretention, this should be manageable. Logging to a central log host will also solve this problem.

51 / 66



Overview






52 / 66



Generating the SSH Key PairTo generate a new key for the icinga user, use ssh-keygen:

icinga2@host$ ssh -keygen -t ed25519 -f icingaGenerating public/private ed25519 key pair.Enter passphrase (empty for no passphrase ):Your identification has been saved in /var/spool/icinga /.ssh/ansible.Your public key has been saved in /var/spool/icinga /.ssh/icinga.pub.The key fingerprint is:SHA 256: Hcl0VflRPR 12ebzG2F/EoM5z3NvFSf5mgI8N6Sl/5Tg icinga@hostThe key 's randomart image is:+--[ED25519 256]--+| . o..+%|| o o o=B|| +.+ +. . =B|| . .= Bo=|| S .+ o=O+|| o-. o .ooO+|| +..+o=|| o.C o|| .o |+----[SHA256]-----+

53 / 66



Remote Machine User Setup

Create a user called icinga on each host that should be monitored via SSH. Define a shell,/bin/sh is usually enough. Set the home directory to /var/spool/icinga (optional). Usessh-copy-id to copy the public key to the users home directory and register it to log inpasswordless. (You can also create a Match rule in sshd for the icinga user in/etc/ssh/sshd_config).Some plugins must be run with root privileges (like check_hdd_health). We also want tomake sure that no one can run arbitrary commands as the icinga user on the remote system.Therefore, we set up sudo to run only certain commands. Here is an example from/usr/local/etc/sudoers:

icinga ALL=(ALL) NOPASSWD: /usr/local/libexec/nagios/*

The icinga plugins are located in /usr/lib/nagios/plugins on Linux hosts.Test the passwordless login and the privilege execution of the plugins before writing checks forit on the Icinga master.

54 / 66



Service Checks Using the by_ssh PluginHere is a simple service check for the load plugin. It requires thecheck_command = "by_ssh", which will execute the plugin on the remote system. It needs toknow where the plugin is located on the remote machine. The location and any other requiredparameters can be passed to the plugin using the vars.by_ssh_command line. Don’t forget todefine to which hosts this plugin should be assigned to. The execution intervall of these checksis defined in the "generic-service" template in templates.conf.

1 apply Service "load_ssh" {2 import "generic -service"3 check_command = "by_ssh"4 vars.by_ssh_command = PluginDir + "/check_load -w 5,5,5 -c 8,8,8"

6 assign where host.vars.os == "FreeBSD"7 }

The PluginDir variable is defined in /usr/local/etc/icinga2/constants.conf as:

const PluginDir = "/usr/local/libexec/nagios"

55 / 66



Running checks depending on host OS VariablesSince we defined the vars.os variable in our hosts.conf file for each host to monitor, we canmake Icinga run checks based on that variable value.

1 apply Service "load_ssh" {2 import "generic -service"3 check_command = "by_ssh"

5 if (host.vars.os == "Linux ") {6 vars.by_ssh_command = LinuxPluginContribDir + "/check_load -w 5,5,5 -c 8,8,8"7 }

9 if (host.vars.os == "FreeBSD ") {10 vars.by_ssh_command = PluginDir + "/check_load -w 5,5,5 -c 8,8,8"11 }

13 assign where host.vars.os == "FreeBSD" || host.vars.os == "Linux"14 }

We defined the variable LinuxPluginContribDir in our constants.conf like this:

const LinuxPluginContribDir = "/usr/lib/nagios/plugins"

56 / 66

BSD-based Systems Monitoring with Icinga2 and OpenSSHRestarting Services Automatically

Overview






57 / 66


Rationale

Since we’re already monitoring whether a service is in a certain state (DOWN), we can alsotrigger actions on certain events. For example, when the service state changes from UP toDOWN and there is no planned downtime, it can be assumed that this is an unscheduledoutage. Icinga can issue service restart commands when the service stays down after a coupleof retries. Once the number of retries has passed and the service is still in the DOWN state,Icinga will automatically call the configured restart routines without any administratorintervention. This way, a service downtime is limited.A word of caution: Automatically restarting services does not solve all problems. Some servicestake longer or have other dependencies (i.e. dependent databases) that can’t be handled byIcinga. A service downtime might need further investigation by a sysadmin to figure out thecause for it to prevent it from happening in the future. For simple services, Icinga’s functionalityworks and is quicker than any admin could act. Don’t use is for more complicated setups.

58 / 66


Setting up local_eventcommands.confWe create a file called local_eventcommands.conf for define commands to run as templatethat we can import in the actual events:object EventCommand "event_by_ssh" {

command = [ "/usr/local/libexec/nagios/check_by_ssh" ]

arguments = {"-H" = "$event_by_ssh_address $""-p" = "$event_by_ssh_port$""-C" = "$event_by_ssh_command $""-l" = "$event_by_ssh_logname $""-i" = "$event_by_ssh_identity $""-q" = {

set_if = "$ event_by_ssh_quiet $"}"-w" = "$event_by_ssh_warn$""-c" = "$event_by_ssh_crit$""-t" = "$event_by_ssh_timeout $"

}

vars.event_by_ssh_address = "$ address $"vars.event_by_ssh_quiet = false

}

59 / 66


Defining the commands to runFirst, we import the event_by_ssh EventCommand. Then, we check the return code in$service.state_id$ using the test command. If we detect a non-zero return value, we letsudo execute the service command to restart the service defined by the$event_by_ssh_service$ variable.

object EventCommand "event_by_ssh_restart_service" {import "event_by_ssh"

//only restart the daemon if state > 0 (not -ok)// requires sudo permissions for the icinga uservars.event_by_ssh_command = "test $service.state_id$ -gt 0 && \

sudo service $event_by_ssh_service$ restart"}

This requires a line like this in /etc/sudoers to allow the icinga user to restart the service:

icinga ALL=(ALL) NOPASSWD: /usr/sbin/service nginx restart

60 / 66


Define the service to restart

For testing purposes, we define a new service in services.conf for the nginx webserver. Weapply it to a specific host afterwards:

apply Service "http" {import "generic -service"check_command = "http"event_command = "event_by_ssh_restart_service"vars.event_by_ssh_service = "$host.vars.httpd_name$"

assign where host.name == "BSDHost"}

61 / 66


Setting the service name to restart

In hosts.conf we set the variable vars.httpd_name to the service name (nginx in thisexample):

object Host "BSDHost" {import "generic -host"address = "IP.address.of.host"vars.httpd_name = "nginx"

}

62 / 66


Testing the Changes by Forcing an Automatic Service Restart

After testing the changes and restarting the icinga service, we want to test whether the servicerestart will trigger upon a service downtime. We manually stop the nginx service and look atthe icinga.log file. We find entries like the following and the service should be up again:

[2018 -11 -02 15:09:29 +0100] information/ExternalCommandListener: Executing externalcommand: [1541167769] SCHEDULE_FORCED_SVC_CHECK;BSDHost;http ;1541167769[2018 -11 -02 15:09:30 +0100] warning/PluginEventTask: Event command for object'BSDHost!http ' (PID: 6328, arguments: '/usr/local/libexec/nagios/check_by_ssh ' '-C''test 2 -gt 0 && sudo service nginx restart ' '-H' 'IP.address.of.host ') terminated withexit code 3, output: Remote command execution failed: nginx: the configuration file/usr/local/etc/nginx/nginx.conf syntax is ok

[2018 -11 -02 15:09:38 +0100] information/WorkQueue: #6 (ApiListener , RelayQueue) items: 1,rate: 12.3833/s (743/ min 743/5 min 743/15 min);[2018 -11 -02 15:10:13 +0100] warning/PluginEventTask: Event command for object'BSDHost!http ' (PID: 7134, arguments: '/usr/local/libexec/nagios/check_by_ssh ' '-C''test 0 -gt 0 && sudo service nginx restart ' '-H' 'IP.address.of.host ') terminated withexit code 1, output: WARNING - check_by_ssh: Remote command 'test 0 -gt 0 && sudoservice nginx restart ' returned status 1

63 / 66

Questions?


Thanks

• Lars Engels (lme@) for maintaining the Icinga2 port in FreeBSD, help and advice• Netways GmbH for their Icinga2 training that helped me fill in my Icinga2 knowledge gaps• Icinga2 community for their comprehensive documentation and willingness to help• Port maintainers and committers who maintain monitoring plugins in the ports collection• You for taking this tutorial. I hope it is useful to you. Happy monitoring!

65 / 66


Further Information

Icinga2 DocumentationIcinga 2 Documentationhttps://icinga.com/docs/icinga2/latest/

Installing Icinga Web 2 with Apache 2.4, Icinga 2 and MySQL on FreeBSDIcingaweb2 Setup Tutorial by Lars Engels (Icinga port maintainer)https://lme.postach.io/post/installing-icinga-web-2-with-apache-2-4-icinga-2-and-mysql-on-freebsd

Monitoring Plugins Websitehttps://www.monitoring-plugins.org

66 / 66

https://icinga.com/docs/icinga2/latest/

https://lme.postach.io/post/installing-icinga-web-2-with-apache-2-4-icinga-2-and-mysql-on-freebsd

https://lme.postach.io/post/installing-icinga-web-2-with-apache-2-4-icinga-2-and-mysql-on-freebsd


bsd-basedsystemsmonitoringwithicinga2andopensshbcr/tutorials/icinga.pdf ·...

Documents