Splunk IntegrationGuide

Release 5.3.1

June 15, 2013

Contents

...............................................................................................1Legal

...............................................................................................2Overview

...............................................................................................3System Requirements

...............................................................................................4Installation

................................................................................................................ 4App Installation

................................................................................................................ 4Splunk Configuration

................................................................................................................ 5INETCO Insight syslog Forwarding

...............................................................................................6Testing

...............................................................................................7Operation

...............................................................................................8Appendix - Alert Data Fields

Legal Page 1

Legal

©Copyright 2013, INETCO Systems Ltd.

All Rights Reserved.

Unauthorized copying prohibited.

This document is the property of and is proprietary to INETCO Systems Limited. The information inthis document cannot be duplicated, used or disclosed in whole or in part for any purpose other thanfor that intended, unless otherwise expressly agreed in writing by INETCO Systems Limited.

This document is an unpublished expression which contains confidential and secret informationbelonging to INETCO Systems Limited.

The existence of the copyright notice is not to be construed as an admission or presumption thatpublication has occurred.

INETCO Systems Limited#258 - 4664 Lougheed HwyBurnaby, British ColumbiaCanadaV5C 5T5(604) 451-1567 (phone)(604) 451-1565 (fax)

www.inetco.com

INETCO, INETCO Insight, the INETCO logo, the INETCO Insight logo, the POSway logo, theBankLink logo, and the "Every transaction tells a story" tagline are trademarks or registeredtrademarks of INETCO Systems Ltd. All other trademarks are the property of their respectiveowners.

Splunk Integration Guide - Release 5.3.1

OverviewPage 2

Overview

INETCO Insight®is a real time Application Performance Monitoring (APM) productthat monitors the transaction network passively, reconstructs transactions from thenetwork flow, decodes transaction message content, calculates performance metricson individual and aggregated transactions, and outputs any of the resulting data ina wide variety of ways.

One of INETCO Insight’s features is the ability to alert on emergent transactionperformance problems and unexpected transaction patterns and to output theresulting alerts in different formats such as sy slo g. This document describes theintegration of INETCO Insight’s sy slo g output with Splunk, a product focused onmaking machine data accessible to everyone.

The following capabilities are provided:

Alerts emitted by INETCO Insight are logged by Splunk.

The fields within the INETCO Insight syslog header and message are extractedby Splunk.

The insertion of INETCO Insight alerts into Splunk, Splunk users to access the detailsof the INETCO Insight alerts for analysis, report generation, Splunk alert generation,and any other functionality which Splunk provides for processing events.

NOTE:This document assumes that the user is already familiar with both Splunk andINETCO Insight.

System Requirements Page 3

System Requirements

The following products must be installed and operational in order to integrateINETCO Insight syslog alerts with Splunk:

Splunk (www.splunk.com)

INETCO Insight version 5.3.1 and above.

The following documentation should also be available for reference:

INETCO Insight Configuration Guide. This is accessible through the INETCOInsight console interface.

Splunk Integration Guide - Release 5.3.1

InstallationPage 4

Installation

The following installation and configuration steps are required:

The INETCO Insight Alerts App for Splunk must be installed.

Splunk must be configured to receive INETCO Insight syslog alerts.

INETCO Insight must be configured to direct syslog alerts to the Splunk.

NOTE:To ensure maximum performance of both Splunk and INETCO Insight, it isrecommended that they be installed on different servers.

App Installation

The installation of the INETCO Insight Alerts App for Splunk is done as follows:

Go to the directory “INETCOInsight\Tools\Splunk” where INETCO Insight isinstalled and unzip the "INETCO_Insight_Splunk_Alerts.zip" file.

Open the Splunk manager.

Go to the "Apps" page.

Click the button at the top marked "Install app from file".

Select the "INETCO_Insight_Splunk_Alerts.tar.gz" file unzipped earlier as theapp to upload.

Click the button marked "Upload".

This will result in the INETCO Insight Alerts App for Splunk being installed in Splunk.

Splunk Configuration

Once the INETCO Insight Alerts App for Splunk has been installed, the next step is toconfigure Splunk to receive the INETCO syslog alerts. This is done as follows:

Open the Splunk manager.

Go to the "Data inputs" page.

Select "UDP".

Click the button marked "New" at the top of the page.

Enter the UDP port that will be receiving the INETCO Insight data.Alternatively, for "source name override", enter the "host:port" of the INETCOInsight server.

Set "sourcetype" to "From list".

On the drop down list that appears, select "INETCO_Insight_Syslog_Alerts".

Installation Page 5

Click the "Save" button.

Splunk is now configured to accept and process any alerts received from INETCOInsight.

INETCO Insight syslog Forwarding

]Once Splunk has been configured, INETCO Insight must be configured to directsyslog formatted alerts to it. This is done as follows.

Open up the "INETCO Insight Manager" and log in with the admin id andpassword.

Select the "Forwarding" tab.

Select the "syslog" tab.

Check the "Forward alerts to syslog" check box.

For "IP Address:Port" enter the IP address and port of the Splunk indexer orforwarder. The port will be the port you entered when setting up Splunk instep 2.

Click the "SAVE CHANGES" button in the top right corner.

Click the "APPLY" button in the top right corner. Any alerts that Insightgenerates will now be forwarded to Splunk.

NOTE:For more information regarding this and how to create Insight alerts, refer tothe “INETCO Insight Configuration Guide”

Splunk Integration Guide - Release 5.3.1

TestingPage 6

Testing

To test that the installation has been successful, configure an INETCO Insight Alert asper the “INETCO Insight Configuration Guide”. The trigging of an alert can beverified via the INETCO Insight console. The alert should also appear in Splunk.

If alerts are being generated, but are not being received by Splunk, a careful checkshould be done to ensure:

That the alert is, in fact, being generated.

INETCO Insight sy slo g forwarding has been appropriately configured.

One of servers executing the INETCO Insight or Splunk software is not blockingdata traffic (e.g. UDP traffic must not be blocked).

Splunk is running and has the INETCO Insight Alerts App for Splunk installed.

The Splunk query has been properly formatted.

NOTE:Contact INETCO Professional Services should problems persist.

Operation Page 7

Operation

Once the INETCO Insight to Splunk connection has been established and is operatingproperly, users are encouraged to experiment with the system so as to get a betteridea as to its capabilities within their IT environment. INETCO Professional Servicesmay also be contracted should users wish to make use of their experience in alertconfiguration, monitoring, and analysis..

NOTE:INETCO welcomes any feedback as to preferred mappings between INETCOInsight and Splunk, as well as any requests as to additional information whichmight be forwarded by INETCO Insight alerts..

Splunk Integration Guide - Release 5.3.1

Appendix - Alert Data FieldsPage 8

Appendix - Alert Data Fields

The following alert fields are output by INETCO Insight syslog:

syslog_monthThe month in which the alert occurred. This is from the syslog header and is thelocal time of the computer running the instance of INETCO Insight whichgenerated the alert.

syslog_dayThe local day in which the alert occurred. This is from the syslog header and isthe local time of the computer running the instance of INETCO Insight whichgenerated the alert.

syslog_hourThe local hour in which the alert occurred. This is from the syslog header and isthe local time of the computer running the instance of INETCO Insight whichgenerated the alert.

syslog_minuteThe local minute in which the alert occurred. This is from the syslog header andis the local time of the computer running the instance of INETCO Insight whichgenerated the alert.

syslog_secondThe local second in which the alert occurred. This is from the syslog header andis the local time of the computer running the instance of INETCO Insight whichgenerated the alert.

syslog_hostnameThe name of the computer running the instance of INETCO Insight whichgenerated the alert.

syslog_tagThe syslog "TAG" field. This is set to "Insight" to indicate the alert wasoriginated by INETCO Insight.

syslog_process_idThe process id of the instance of the INETCO Insight Processor subsystem whichgenerated the alert.

uuidA "Universally Unique Identifier" assigned to the alert by INETCO Insight touniquely identify it.

seq_numberThe "Sequence Number" assigned to the alert by INETCO Insight. INETCOInsight assigns sequential sequence numbers to alerts.

time_localThe local time on the computer running INETCO Insight at which the alertoccurred in msec. This is also known as the "Insight Local Time".

Appendix - Alert Data Fields Page 9

time_utcThe UTC time at which the alert occurred in msec.

entityThe Insight "Entity" associated with the alert in the INETCO Insight Alertconfiguration.

entity_groupThe Insight "Entity Group" associated with the alert in the INETCO Insightconfiguration.

idA unique "ID" assigned to the alert by the INETCO Insight system at the timethe alert was first created.

customer_idThe "Customer Alert ID" specified in the INETCO Insight Alert configuration.

active_timeThe name of the active time interval (i.e. the "Active Time") as specified in theINETCO Insight Alert configuration.

categoryThe "Category" specified in the INETCO Insight Alert configuration.

alert_descriptionThe "Description" entered at the time of INETCO Insight Alert configuration.

nameThe "Name" specified in the INETCO Insight Alert configuration.

severityThe "Priority" specified in the INETCO Insight Alert configuration. Valid valuesare "Critical", "Warning", and "Notice".

timezoneThe "Time Zone" specified in the INETCO Insight Alert configuration. Validvalues are "UTC", "Insight", and "Entity". "UTC" indicates the UTC timezone,"Insight" indicates the local timezone of the INETCO Insight Processor, and"Entity" indicates the Insight Entity timezone.

event_countSpecifies whether the alert occurred when there were "less then", "equal to",or "greater then" the number of events configured in the INETCO Insight alertconfiguration. Valid values are one of "lt" (less than), "eq" (equal to), and"gte" (greater than or equal to).

event_intervalSpecifies the interval in which the events conditions occurred to generate thealert. The value is in seconds.

alert_triggerSpecifies whether the alert occurred based on the "number of events", the

Splunk Integration Guide - Release 5.3.1

Appendix - Alert Data FieldsPage 10

"number of consecutive events", or the "percent matching vs total events" asconfigured in INETCO Insight. Valid values are "count", "consec", and"percent".

event_valueSpecifies the numerical count or percentage used for comparison in the rulewhich generated the alert, as configured in the INETCO Insight Alertconfiguration.

rule_numberSpecifies the "Rule #" of the Insight Alert configuration corresponding to thealert.

alert_typeSpecifies whether the alert is an "event" alert, a threshold "set" alert, or athreshold "clear" alert. Valid values are "event", "set", and "clear".

rule_descriptionA human readable description of the rule which generated the alert. Thisdescription is automatically generated for the rule by INETCO Insight.

notify_messageThe "Notification Message" configured for the alert in the INETCO Insight Alertconfiguration. This is also the message sent when an alert is emailed to anadministrator.