using logstash and elasticsearch analytics capabilities as ... › eunis2015 › wp-content ›...

29
Pashalis Korosoglou, Pavlos Daoglou, Stefanos Laskaridis, Dimitris Daskopoulos | Aristotle University of Thessaloniki, IT Center Using Logstash and Elasticsearch analytics capabilities as a BI tool

Upload: others

Post on 29-May-2020

7 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

Pashalis Korosoglou, Pavlos Daoglou, Stefanos Laskaridis, Dimitris Daskopoulos | Aristotle University of Thessaloniki, IT Center

Using Logstash and Elasticsearch analytics capabilities as a BI tool

Page 2: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Outline

• Technical stuff (Logstash, Elastic, Kibana, Ansible)

• Motivation for monitoring Software licenses

• Other use cases

• Summary and next steps

Page 3: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Logstash

• Written in jRuby

• Applicable well beyond log files

• Plethora of core and community contributed plugins • I/O plugins

• Filtering plugins

• Codecs

• Take this “msg” and parse/compute/save stuff on the wire

Page 4: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

A very simple pipe example serviceURI: node03.domain.gr\nhostName: node03.domain.gr\nserviceFlavour: service_i\nsiteName: SITENAME\nmetricStatus: OK\nmetricName: org.ldap.Freshness\nsummaryData: OK: freshness=70s, entries=1\ngatheredAt: nagios.domain.gr\ntimestamp: 2015-06-08T19:42:31Z\nnagiosName: org.ldap.Freshness\nserviceType: service_i\nEOT\n

{

"@timestamp" => "2015-06-08T19:42:31.000Z”,

"hostName" => "node03.domain.gr",

"serviceFlavour" => "service_i",

"siteName" => "SITENAME",

"metricStatus" => "OK",

"metricName" => "org.ldap.Freshness",

"freshness" => 70

"entries" => 1

"gatheredAt" => "nagios.domain.gr",

"probe" => "org.ldap.Freshness",

"serviceType" => "service_i"

}

Page 5: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Logstash forwarders & Lumberjack

• Logstash-forwarder is a lightweight forwarding service • Keeps track of offset within log file

• Failure resistant

• Supports multiple file inputs

• Lumberjack is a collection service • Basically one of many input plugins available

• Uses zlib for compression

• Secure transmission of logs via OpenSSL

Page 6: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Architecture Overview

Page 7: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Architecture Overview

Logstash forwarder(s) configuration

Page 8: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

ElasticSearch (Elastic)

• Distributed data-store with near-real time search capabilities • Built on top of Apache Lucene

• Exposes HTTP RESTful API (i.e. for querying)

• Multitenant architecture

• Highly available • Shards replication

• Supports 3rd party plugins (i.e. HQ, head etc)

• Apache 2.0 license

Page 9: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Elastic, RDBMS & Hadoop concepts

• ES document -> Table row in a RDB

• ES Index -> RDB database

• A collection of documents

• ES Mapping -> RDB schema definition

• ES Shards -> Hadoop splits

• Each shard is actually a Lucene index

• ES index splits into shards

Page 10: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Elastic, RDBMS & Hadoop concepts Replication: 1 5 primary shards by default 1 replica for each shard Replicas can’t be assigned on the same node with the primary shard

Page 11: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Kibana

• Node.js frontend for Elastic

• Allows (realtime) visualisation of data

• Flexible interface

• One can add, remove, move, modify rows and graphs

• Allows different search queries

• Allows save, import, export and share operations for dashboards

Page 12: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Software @ Auth

• More than 20 annual contracts signed (+a few perpetual) • The majority relies on FlexLM service

• Expenditures/year: ~100K€

• Use cases • Which departments use software X?

• Which departments use software X for educational or research purposes?

• How often is software X’s Y component used?

Page 13: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Software @ Auth

• The problem(s) with flex logs:

23:29:06 (deamon) TIMESTAMP 6/3/2015 0:36:51 (deamon) OUT: "feature" someone@somewhere 0:39:04 (deamon) IN: "feature" someone@somewhere 0:54:47 (deamon) DENIED: "feature" someone@somewhere (Licensed number of users already reached. (-4,342)) 0:54:47 (deamon) UNSUPPORTED: "feature" (PORT_AT_HOST_PLUS ) someone@somewhere (License server system does not support this feature. (-18,327)) 0:54:47 (deamon) OUT: "feature" someone@somewhere (2 licenses) 1:08:08 (deamon) IN: "feature" someone@somewhere (2 licenses) 1:08:31 (deamon) OUT: "feature" someone@somewhere 1:10:09 (deamon) IN: "feature" someone@somewhere 1:13:43 (deamon) UNSUPPORTED: "feature" (PORT_AT_HOST_PLUS ) someone@somewhere (License server system does not support this feature. (-18,327)) 3:16:44 (lmgrd) TIMESTAMP 6/4/2015

Page 14: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Software @ Auth

• Our solution (via logstash filtering):

{ "_type": "deamon", "_source": { "message": "19:07:17 (deamon) IN: \"feature\" someone@somewhere", "@version": "1", "@timestamp": "2015-06-08T16:07:17.000Z", "host": "tracker01", "tags": [ "taskTerminated", "elapsed", "elapsed.match" ], "feature": "\"feature\"", "username": "someone", "hostname": "somewhere", "elapsed.time": 67, "elapsed.timestamp_start": "2015-06-08T16:06:10.000Z" } }

Page 15: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Software @ Auth (screenshots)

Page 16: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Software @ Auth (screenshots)

Page 17: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Software @ Auth

• The decision making on what contracts will continue and with how many seating licenses depend on our accounting monitoring

• Actual scenarios/decisions

• Renew contract for software X but reduce the number of floating licenses

• Renew annual license for software X but don’t renew component Y

Page 18: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Other Use Cases?

• Web services

• Accounting

• Resources usage

• Environmental monitoring

• Logins and brute force attempts

• Performance metrics

• Any log file (?)

Page 19: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Web services filter { if [type] == "httpd" { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } geoip { source => "clientip" } mutate { convert => { bytes => "integer" } } date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] } } }

Page 20: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Web services

Page 21: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Accounting (local HPC resource)

Page 22: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Accounting (local HPC resource)

Page 23: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Logins (successful and attacks)

Page 24: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Resources Usage

Page 25: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Re-playing

• Log files are still kept in central syslog

• Scratch elastic completely and everything is reproducible ad-hoc

• Filters (via Ansible)

• Log files (via central syslog)

Page 26: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Reporting

• Elastic API not reachable from outside

• What if we want to send reports to our users?

• Using phantomjs framework and rasterize.js we can generate:

• custom weekly or monthly or annual reports in pdf format and share with our users

Page 27: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Summary

• The wealth sometimes hidden away in our log files is enormous

• ELK should not be considered a replacement for central logging • Rather it’s best to treat it as an addition to an

existing stack

• ELK has helped us in… • Indexing data from log files • Searching through log files • Visualizing data

• …and gain useful business insight

Page 28: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Next steps

• Performance monitoring via Nagios/Icinga probes & metrics

• Combination with Hadoop stack • Safekeeping ‘cold’ data

• Performing combined aggregated queries • λ architectural prototype

• Upgrade elastic and kibana to 1.5.x

• Apply data data retention policies and use Elastic's repository features for long term storage

Page 29: Using Logstash and Elasticsearch analytics capabilities as ... › eunis2015 › wp-content › uploads › ... · • ELK should not be considered a replacement for central logging

it.auth

Questions

[email protected]