Copyright © 2015 Splunk Inc.

The Hands-On Version

BurchSystems Engineer, IT Operations Analytics SME

Setup Before You Can PlayDownload this presentation slide deck: https://splunk.box.com/splunkliveitsi16 Follow the instructions on your paper hand-out to log in to your VM.

Please log in as either• user1@buttercupgms.com OR• user2@buttercupgms.com• Password is “Changeme1” or

“Changeme2”

After logging in, selectIT Service Intelligence from thelist of apps at the left

2

ITSI Core Concepts

3

4

Experience with ITSI? Splunk?

5

ITSI – The Burch Version

7

ITSI – The Burch Version

8

ITSI – The Texbook Version

What is a Service?

Service RequestsResponses

In ITSI, a Service is a logical group of technology components that a user deems need to be monitored together.

It can often be generalized as a “black box” which we send requests, and expect responses

9

What is a Service?

DNS RequestsResponses

Technical Services

Auth RequestsResponses

Web RequestsResponses

Services can be lower level (technical) …

10

What is a Service?

DNS RequestsResponses

Technical Services

Customer Transactions

RequestsResponses

Business Services

Auth RequestsResponses

Web RequestsResponses

Support Desk RequestsResponses

Services can also be higher level (business) …

11

What is a Service?

Packet Network

Hypervisor and Hosts

RBMDBs

Storage Tier

API Services

Web Services

Customer Transactions

Mobile

API/Middlew

are

Partner Portal

DNS

Services can encompass multiple tiers of the IT domain. Services may also depend upon other services

12

What is a KPI?

DNS RequestsResponses

KPI: Number of requestsKPI: Error rateKPI: Average response timeKPI: Server CPU loadKPI: Server network I/F errors

Customer Transactions

RequestsResponses

KPI: Number of transactionsKPI: Error rateKPI: Average response timeKPI: Count of Incident TicketsKPI: Synthetic Transx Health

KPIs and Health scores constitute the means by which Services are monitored.

13

14

Key Performance Indicators (KPIs)

A Key Performance Indicator (KPI) is a Splunk saved search created within the ITSI UI that helps monitor a specific field like CPU, Memory, Number of Errors

and so on. KPIs are contained within Services.

Service Health Scores

15

A Health score is a score form 0-100 (0 being critical and 100 being normal) that helps determine the health of a Service. It is calculated based on all KPIs

importance and its status (e.g. green, orange, red), once every minute.

Now in ITSI

16

28

New Requirements!● Create a new KPI for the DB Service:

● Network Utilization

● Modify the Executive Glass Tablein order to show off the servicesyou slave over

“WE only have about 15min TO DO WHAT ???!!???”

Think about how long this would take you today?

29

Configuration of DB Service

Click Configure > Click Services

30

Let’s Talk Entities

● Select DB Service

● Entities are the relevant things which support this service (usually hosts)

● Select the right entries with filters, ANDs, ORs● Original Entity list can come from CMDB,

spreadsheet, Splunk search, others

31

A KPI in 5 minutes? Absolutely!

Click New – Generic KPI

Select Data Model● Host Operating System● Network● # bytes● Next

32

KPIs Continued….

Splunk Builds Searches for you – Oh Yeah, that’s happening

● Select Yes for Split by & Filter options● Select host for Entity Lookup & Alias options● Click Next

33

Almost There…Select● KPI Search Schedule: Every Minute ● Entity Calculation: Average● Service/Agg Calculation: Average● Calculation Window: Last Minute● Click Next

● Unit: Bps● Click Next

34

Final Steps …Set your thresholds:

● Aggregate (All) ● Per Entity

● Click “Add Threshold” TWICE● Make the Neapolitan ice cream colors

Yellow, Green, Yellow● Drag the sliders around in order to get

the current data graph entirely inside the Green (normal) band

● Click Finish● Other options are also available,

including adaptive thresholds and anomaly detection

35

Name that KPI!

From the list of KPIs, select your new one (at the bottom)● Click on the little pencil next to the name● Call it “Network Utilization”,

with your username up front

● Click on Save at bottom right when finished!

Adaptive Thresholds

36

What if your KPI data looks like this?

37

Adaptive ThresholdsStatic thresholds will not work…

38

Adaptive ThresholdsAdaptive Thresholding works beautifully with cyclical (and other dynamic) data

39

Anomaly Detection

● Machine Learning

● Works well for data with patterns

● Requires some “training” (trial & error) to zero in on best sensitivity

● More sophisticated capabilities coming! (multivariate, more algorithms, etc)

40

Let’s Fix that Glass Table

41

Clone the Glass TableReturn to Saved Glass Tables page (click on Glass Tables in the upper menu bar)

CLICK Edit for “Buttercup Games Business Process (IN PROGRESS)”• Select Clone• Title: Add your username

to the front• Permissions: Shared in App• Click Clone Page

• Click on your new Glass Tablefrom the list, to view it

42

Edit & Have Fun!Click on Edit in the upper right corner of your Glass Table

Use the “Services” panel on the left to select Individual KPIs, or Aggregate Service Health Scores• Choose 2 KPIs from Online Store that would be useful in

the “Order Process” section• Drag the selected widgets onto the canvas, positioning in

the gray oval

• What’s the difference between the

and tools at the top left?

43

More Fun with the Glass Table Editor…Use the Configurations panel on the right to edit a selected widget• Can change the visualization type, drilldown

behavior, and other settings

• You should hit Save frequently• I wonder what Auto Layout does?• (YIKES!) Revert All Changes might be helpful

44

Finishing up …• Add a ServiceHealthScore widget for Online

Store under Buttercup• Choose a Viz Type with a sparkline graph, then

resize to make it look pretty• Modify the Custom Drilldown action to go to

the saved glass table, Buttercup Games Online Store

• Bonus Points: Make the label bigger, more readable

• Click Save• View when done

45

A Troubleshooting ExerciseLet’s use ITSI to troubleshoot an outage ● Start at your Glass Table, “<UserName> Buttercup Business Process”● Customer Care reports that unhappy customers are complaining of failures

and long delays when trying to purchase● The calls began coming in at around ten minutes after the hour.● In the upper right corner of the Glass Table, change the time picker from Now

to XX:10:00.0, where XX is the appropriate hour. For example, if it is currently 14:05, set the time picker to 13:10:00.0, then Apply

● This is how we can “time travel” back to see conditions at a particular outage– oh yeah!

46

A Troubleshooting Exercise, cont’d● The Online Store seems to be degraded, just as Customer Care reported.

Click on the widget under Buttercup to drill down further

47

A Troubleshooting Exercise, cont’d.● The Online Store Glass Table shows a much more detailed view, including the impacted customer-facing KPIs

at the far left (Revenue, etc)

● Based on this view of all the relevant services, where do you think the root cause lies?

● Which service should we troubleshoot first?● Click on Health widget for that service, to

drill down to a Deep Dive

48

Deep Dive

● Deep Dive shows multiple KPIs and Health Scores in parallel “swim lanes”.

● The Health Score for this Service is the top swim lane. Can you see when it begins to degrade from 100%?

● Mousing over this point in time, can you spot the KPI with the leading fault indication, i.e., what failed first?

51

Review● High-value services can be decomposed and modeled in ITSI, using machine data

from the relevant systems● Services and KPIs can be created in minutes, with sophisticated thresholding

techniques to distinguish “normal” from “not normal”● Glass Tables allow service health and KPI metrics to be displayed in a way that

makes sense to specific groups, such as Executive Leadership, Business Service Owners, the NOC, DevOps & Others

● Deep Dives allow KPIs to be compared side-by-side across any time range, accelerating root cause analysis and significantly reducing MTTR

● Multi-KPI Alerts and Notable Events reduce alert noise, producing actionable events and a means to manage them

● … and it’s fun to build!

52

PLAY TIME IS OVER!Everyone out of the sandbox!

NOT! You can have your very own 7-day free eval sandbox, to continue playing:

● http://splunk.com/ITSI Then select:

And a Guidebook to help you explore ITSI’s capabilities:● https://splunk.box.com/ITSI-Sandbox-Guidebook

53

SEPT 26-29, 2016WALT DISNEY WORLD, ORLANDOSWAN AND DOLPHIN RESORTS

• 5000+ IT & Business Professionals• 3 days of technical content• 165+ sessions • 80+ Customer Speakers• 35+ Apps in Splunk Apps Showcase• 75+ Technology Partners• 1:1 networking: Ask The Experts and Security

Experts, Birds of a Feather and Chalk Talks• NEW hands-on labs! • Expanded show floor, Dashboards Control

Room & Clinic, and MORE!

The 7th Annual Splunk Worldwide Users’ Conference

PLUS Splunk University• Three days: Sept 24-26, 2016• Get Splunk Certified for FREE!• Get CPE credits for CISSP, CAP, SSCP• Save thousands on Splunk education!

A flying start to Service Intelligence

Start With A problem worth solving

Collaborate with Subject Matter Experts

Design Before Configuring

Sign Up Here - We’re Here To Help!Harness the creativity and domain knowledge of your organization to unlock

the value of data and solve an important service problem through a joint service intelligence workshop with key stakeholders

Define methods for:• Proactive service monitoring• Reduced risk and failures• Faster issue resolution• Increased business

performance

What is it? • 1 Day Onsite Workshop• Tightly linked with value• Collaborative approach• Build your own Splunk

ITSI Glass Table……

Copyright © 2015 Splunk Inc.

Thank You

BurchSystems Engineer, IT Operations Analytics SME