monitoring using sensu
TRANSCRIPT
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Who am I?• Malta since December 2015
• Consultant for 20+ years
• Government, Finance, Health, Social Media, Fortune 50, Startups
• DevOps, Automation, Architect, Development
• Open Source @ github.com/ripienaar
• Linux since Kernel 99 alpha p11
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Checks
API
MetricsDynamic
External InputsSecure
Events
Plugins
JSON
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Architecture - Overview
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Architecture - Schedule
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Architecture - Event
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Architecture - Integrate
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
{ "checks": { "procs_rhel7": { "command": "/usr/lib64/nagios/plugins/check_procs -w 50 -c 70 -k", "interval": 300, "subscribers": [ "RedHat-7" ] } } }
Config - Checks
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
{ "checks": { "procs_rhel7": { "command": "/usr/lib64/nagios/plugins/check_procs -w 50 -c 70 -k", "interval": 300, "subscribers": [ "RedHat-7" ] } } }
Config - Checks
Name
Command to run
How often
Where
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
{ "client": { "name": "dev3.devco.net", "address": "139.162.163.118", "subscriptions": [ "all", "linode_fra", "de", "RedHat-7" ], "redact": [
], "socket": { "bind": "127.0.0.1", "port": 3030 }, "procs": { "total": { "warn": 80, "crit": 90 } } } }
Config - Clients
Local event sync
What am I?
Node Parameters
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
{ "client": { …, "procs": { "total": { "warn": 80, "crit": 90 } } } }
Config - Clients
Node Parameters
{ "checks": { "procs_rhel7": { "command": "/usr/lib64/nagios/plugins/check_procs -w 50 -c 70 -k", "interval": 300, "subscribers": [ "RedHat-7" ] } } }
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
{ "client": { …, "procs": { "total": { "warn": 80, "crit": 90 } } } }
Config - Clients
Node Parameters
{ "checks": { "procs_rhel7": { "command": “…/check_procs -w :::procs.total.warn|50::: -c :::procs.total.crit|70::: -k", "interval": 300, "subscribers": [ "RedHat-7" ] } } }
Check Tokens
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
require ‘sensu-plugin/check/cli'
class CheckFile < Sensu::Plugin::Check::CLI option :file, long: “—file FILE”, description: “The file to check”, default: nil
def run unknown(“no file specified, please use —file”) unless config[:file]
if File.exist?(config[:file]) ok(“file %s exist” % [config[:file]]) else critical(“file %s does not exist” % [config[:file]]) end end end
Checks - Writing in Ruby
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
require ‘sensu-plugin/check/cli'
class CheckFile < Sensu::Plugin::Check::CLI option :file, long: “—file FILE”, description: “The file to check”, default: nil
def run unknown(“no file specified, please use —file”) unless config[:file]
if File.exist?(config[:file]) ok(“file %s exist” % [config[:file]]) else critical(“file %s does not exist” % [config[:file]]) end end end
Checks - Writing in Ruby
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
require “sensu-plugin/check/cli”
class CheckFile < Sensu::Plugin::Check::CLI option :file, long: “—file FILE”, description: “The file to check”, default: nil
def run unknown(“no file specified, please use —file”) unless config[:file]
if File.exist?(config[:file]) ok(“file %s exist” % [config[:file]]) else critical(“file %s does not exist” % [config[:file]]) end end end
Checks - Writing in Ruby
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
# yum install nagios-plugins-all # /usr/lib64/nagios/plugin/check_file_age —help … Usage: check_file_age [-w <secs>] [-c <secs>] [-W <size>] [-C <size>] [-i] -f <file> check_file_age [-h | --help] check_file_age [-V | --version]
# /usr/lib64/nagios/plugins/check_file_age -f /keepalive; echo $? FILE_AGE UNKNOWN: No file specified 3
# /usr/lib64/nagios/plugins/check_file_age -f /keepalive; echo $? FILE_AGE CRITICAL: File not found - /keepalive 2
# /usr/lib64/nagios/plugins/check_file_age -f /keepalive; echo $? FILE_AGE WARNING: /keepalive is 353 seconds old and 0 bytes 1
# /usr/lib64/nagios/plugins/check_file_age -f /keepalive; echo $? FILE_AGE OK: keepalive is 4 seconds old and 0 bytes 0
Checks - Reuse
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
# yum install nagios-plugins-all # /usr/lib64/nagios/plugin/check_file_age —help … Usage: check_file_age [-w <secs>] [-c <secs>] [-W <size>] [-C <size>] [-i] -f <file> check_file_age [-h | --help] check_file_age [-V | --version]
# /usr/lib64/nagios/plugins/check_file_age -f /keepalive; echo $? FILE_AGE UNKNOWN: No file specified 3
# /usr/lib64/nagios/plugins/check_file_age -f /keepalive; echo $? FILE_AGE CRITICAL: File not found - /keepalive 2
# /usr/lib64/nagios/plugins/check_file_age -f /keepalive; echo $? FILE_AGE WARNING: /keepalive is 353 seconds old and 0 bytes 1
# /usr/lib64/nagios/plugins/check_file_age -f /keepalive; echo $? FILE_AGE OK: keepalive is 4 seconds old and 0 bytes 0
Checks - Reuse
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Checks - Reuse
http://sensu-plugins.io/
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
$ cat <<EOF > /dev/tcp/localhost/3030 { "name":"my_awesome_check", "output":"everything is fine", "ttl":600, “status”:0 } EOF
Checks - Your own App
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
{ "mailer": { "mail_from": "[email protected]", "mail_to": "[email protected]", "smtp_address": "smtp.example.org", "smtp_port": "25", "smtp_domain": "example.org", "template": “/etc/sensu/templates/mail.erb”, "subscriptions": { "subscription_name": { "mail_to": "[email protected]" } } } }
Handlers - Email
https://github.com/sensu-plugins/sensu-plugins-mailer
/etc/sensu/handlers/mailer.json
Custom mail body
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
<%= output %> Admin GUI: <%= admin_gui %> Host: <%= @event['client']['name'] %> Timestamp: <%= Time.at(@event['check']['issued']) %> Address: <%= @event['client']['address'] %> Check Name: <%= @event['check']['name'] %> Command: <%= command %> Status: <%= status_to_string %> Occurrences: <%= @event['occurrences'] %>
Handlers - Email
https://github.com/sensu-plugins/sensu-plugins-mailer
/etc/sensu/templates/mail.erb
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Handlers - Thirdparties
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
events = JSON.parse( RestClient.get(“http://example:4567/events").body )
statusmap = {0 => "OK", 1 => "WARNING", 2 => "CRITICAL", 3 => "UNKNOWN"}
puts "Found %d events" % [events.size]
events.each do |event| puts "%30s: %-10s %s" % [ event["client"]["name"], statusmap[event["check"]["status"]], event[“check"]["output"] ] end
API - JSON
Found 2 events puppet1.example.net: CRITICAL DISK CRITICAL - /run/docker/netns/default is not accessible: Permission denied dev1.example.net: WARNING PROCS WARNING: 183 processes
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Linkshttps://sensuapp.org/
https://uchiwa.io/https://graphiteapp.org/
http://grafana.org/http://sensu-plugins.io/
R.I.Pienaar | [email protected] | http://devco.net | @ripienaar
Questions?
twitter: @ripienaaremail: [email protected]: www.devco.net
github: ripienaarfreenode: Volcane
slack.puppet.com: ripienaar
https://www.devco.net/