Modular Alert Actions with Splunk Add-on

Builder

George Starcher&

Duane Waddle

> | whoami• George Starcher

• Splunking since 2010

• Still in love with the Splunk HEC (in “memoriam" of Glenn Block)

• Favorite game is automating Splunk to earn its keep

• Duane Waddle

• Splunking since 2010

• Still dreams of one day being a helicopter pilot

• Our Other .conf Talks:

• .conf 2015: talks SSL and Advanced Lookups

• .conf 2014: talks on SSL and Alert Script automation

• .conf 2016: Duane on: Anti Patterns it seemed like a good idea at the time!

> | history | search alert actions

• Script Actions

• Modular Alerts

• Adaptive Response (Modular Alerts with metadata)

> | history | search script actions• If you see references to putting scripts in $SPLUNK_HOME/bin

• Yeah, don’t do that. Totally OLD, not permissions flexible and few features.

• https://docs.splunk.com/Documentation/Splunk/7.0.1/Alert/Runscriptaction“The run a script alert action is officially deprecated. It has been replaced with custom alert actions as a more scalable and robust framework for integrating custom actions. See About custom alert actions for implementation and migration information.”

| index=splunkdocs | where isnotnull(modular alert)

• Might or might not be George’s fault. Modular Alerts got bumped up Splunk focus after this:http://www.georgestarcher.com/splunk-alert-scripts-automating-control/

• Better way to make flexible custom alert actions tied to search results. Also scoped to app context for knowledge object control. http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModAlertsIntro

• Without Add-On Builder modular alerts still complex.https://www.splunk.com/blog/2016/08/22/how-to-create-a-modular-alert.html

| index=splunkdocs | where isnotnull(adaptive response)

• Sometimes you hear “Adaptive Response”. Those are just modular alerts with extra Common Information Model (CIM) metadata.

• Used with Splunk Enterprise Security.

• http://dev.splunk.com/view/enterprise-security/SP-CAAAFBE

• Still painful to hand make or convert Adaptive Responses.

> index=splunkbase app=“Add-On Builder”• https://splunkbase.splunk.com/app/2962/

• https://docs.splunk.com/Documentation/AddonBuilder/2.2.0/UserGuide/Overview

• This is a Splunk App meant for your development Splunk install to make Add-ons. Do not install on production systems.

• AOB provides a lot of wrapper capabilities for modular inputs and alerts.

• Great helpers:

• Encrypted Credentials scoped to your add on app.

• Easy GUI building for your action

• One checkbox for Adaptive Response support

Building Process• Get a stand alone working script (python)

Test your stand alone using: splunk cmd python myscript.py

• Make your App in AOB

• Paste in your working script

• Fix spacing and wire it to the action UI and search results

• Add modules to the APP your code imports that aren’t native

• Test and Done

TA-Send_to_HEC• Start with the HEC Python class: https://github.com/

georgestarcher/Splunk-Class-httpevent

• Want it as alert action can use to send search results across Splunk deployments

AOB Adaptive Response

AOB GUI Builder

Default Value Behavior• A big annoyance of setting a “default value” is that means AOB sends None if that

is the value chosen instead of the value you said should be default.

• u_senddatatype = helper.get_param("u_senddatatype")

• to get the value from the GUI

• BUT if its the default we get None. SO a trick is handle for your options and leave out the default and use an else for the "default" behavior

if u_senddatatype=="raw":

destCollector = http_event_collector(u_hectoken, u_splunkserver, 'raw', '', u_splunkserverport)

else:

destCollector = http_event_collector(u_hectoken, u_splunkserver, 'json', '', u_splunkserverport)

AOB Code Editor

AOB Mod Alert Tour• The py file you need to edit:

$YOURAPP$/bin/$yourmodalert$/modalert_$yourmodalert$_helper.py

• Drop modules to import in: $YOURAPP$/bin/$yourmodalert$/

• Provides lots of wrapper work for you: log_info, log_errorhelper.log_error("FATAL Empty Search Results, nothing to send.")

• Access to stored creds

• GUI for options

• Using the search results

Handle your imports try:

from splunk_http_event_collector import http_event_collector

import json

except ImportError as err_message:

helper.log_error("{}".format(err_message))

return 1

Tour of the modular alert helper• Getting User Creds: (not used for HEC app)

u_username = helper.get_param("u_username")user_account = helper.get_user_credential(u_username)

• To get events: searchResults = helper.get_events()

• helper.get_events is a generator you can ITER over

• You are expected to iterate over all search results and execute your code. This way the action can work with one or more results rows.

• Each row is a JSON dict. Easy to work with in Python

• Python generators are once and one way. If you need to use result set more than once copy to a list or make your code ITER friendly.searchResults = list(helper.get_events())

modalert_sendtohec_helper.py https://github.com/georgestarcher/TA-Send_to_HEC/blob/master/bin/ta_send_to_hec/modalert_sendtohec_helper.py

def process_event(helper, *args, **kwargs):

IMPORTS

LOG START

GET PARAMS

VALIDATE REQUIRED PARAMS

HANDLE DEFAULT PARAMS

for entry in searchResults:

DO STUFF

LOG STOP

EXIT

Everything under process_event is your stand alone

script wired to the params and action

taken for each search results event

row

Other Tips• If using fields from a search results row (event) use pythonic get

• event.get(‘host’)

• That returns None if value is missing so no crash

• The Splunk Python SDK is available to you

• import splunklib.client as splunkClient

• Undocumented: The helper provides the sessionKey to auth back insession_key = helper.session_key

• In HEC:

• Pop off empty fields

• http://www.georgestarcher.com/splunk-null-thinking/

Other Tips• You can index data from your action

# The following example adds two sample events ("hello", "world")

# and writes them to Splunk

# NOTE: Call helper.writeevents() only once after all events

# have been added

helper.addevent("hello", sourcetype="sample_sourcetype")

helper.addevent("world", sourcetype="sample_sourcetype")

helper.writeevents(index="summary", host="localhost", source="localhost")

Other Tips

• https://docs.splunk.com/Documentation/Splunk/7.0.2/Admin/Alertactionsconf

• Maxtime default 5m

Make Processing Functions import re

def tokenreplacement(event, fieldname):

if fieldname not in event.keys(): return tokenized_field = event.get(fieldname) eventDict = dict(event) retokenized_field = re.sub(r"\$(\w+?)\$", r"{\1}", tokenized_field) try: event.update({fieldname:retokenized_field.format(**eventDict)}) except Exception as err_message: pass return event

> [ | search …] import time import splunk.search

def runsearch(session_key,VALUEOFINTEREST): query = 'mysearch value_of_interest={} ' job = splunk.search.dispatch(query.format(VALUEOFINTEREST), sessionKey=session_key) endTime = time.time()+300 helper.log_info("Running Search: Job endTime is {}".format(endTime)) while not job.isDone and time.time()<endTime: time.sleep(1) jobResults = [] if job.isDone: helper.log_info(Search Job completed.") if time.time()>endTime: helper.log_error("Job endTime exceeded.")

try: jobResults = job.results except Exception as err_message: helper.log_error("Failed to get search results.”)

return jobResults

> | makeresults | eval status=“the end”

• Thanks!!!

• You have enough frame work now to take search results and do actions on them if you have a working python script.