Andi Mann @AndiMann @Splunk

Copyright © 2016 Splunk Inc.

Data-Driven DevOps

Mining Machine Data for ‘Metrics that Matter’

Andi Mann @AndiMann @Splunk

Andi Mann @AndiMann @Splunk

CAMS – as close to prescriptive as DevOps gets

Andi Mann @AndiMann @Splunk3

BUT WHAT SHOULD

YOU MEASURE?

Andi Mann @AndiMann @Splunk

I’m working super hard!!

That’s my stapler.

Andi Mann @AndiMann @Splunk5

Yeah, but … … what are you

achieving?

I’m gonna need you to come in Sunday.

Andi Mann @AndiMann @Splunk

Gartner’s DevOps ‘Metrics that Matter’

Gartner Inc., Data-Driven DevOps: Use Metrics to Help Guide Your Journey, 29 May 2014 G00264319, Analyst(s): Cameron Haight | Tapati Bandopadhyay

Andi Mann @AndiMann @Splunk

IDC’s DevOps ‘Metrics that Matter’

Andi Mann @AndiMann @Splunk

Some DevOps Metrics that Might Matter

Culturee.g.

• Retention

• Satisfaction

• Callouts

Processe.g.

• Idea-to-cash

• MTTR

• Deliver time

Qualitye.g.

• Tests passed

• Tests failed

• Best/worst

Systemse.g.

• Throughput

• Uptime

• Build times

Activitye.g.

• Commits

• Tests run

• Releases

Impacte.g.

• Signups

• Checkouts

• Revenue

Andi Mann @AndiMann @Splunk

But DevOps Is Always a Unique Journey

What Are Your‘Metrics That

Matter’?

Andi Mann @AndiMann @Splunk

Every tool, every process, every component, on-prem or off

One Constant: Machine Data

Andi Mann @AndiMann @Splunk

Common Data Fabric

11

API

SDKs UI

Other ToolsEscalation/

Collaboration

Visibility Across the Whole Dev Lifecycle

Plan Code Build Test/QA Stage Release Config Monitor

Andi Mann @AndiMann @Splunk

Common Data Fabric

12

API

SDKs UI

Server, Storage. N/W

Server Virtualisation

Operating Systems

Mobile Applications

Cloud Services

Other ToolsTicketing/Help

Desk

Custom Applications

Visibility Across the Whole Ops Environment

API ServicesInfrastructure Applications

Andi Mann @AndiMann @Splunk

Use Machine Data To Identify ‘Waste’

Plan

Develop (UI)

Develop (Db)

Develop (M’ware)

Develop (Backend)

SecurityTest

Monitor

Build(Prod)

Architect

Secure/Comply

DeployAccept

UnitTest

Document

Cap Plan

Train

Feedback

IntegrationTest

Configure

System Test

Launch

CAB

Develop(APIs)

Budget

Build(Dev)

Mgmt/Tooling

W

W

W

W

W

W

W

W

W

16 40 52 35 96 40 48 24 --8 2 5 6 8 2 12

Andi Mann @AndiMann @Splunk

Use Machine Data To Manage Testing and QA

• Release when ready, not a date!

• Best / worst developers

• Best / worst providers

• Impact of new code on ops

• Impact of new code on biz

Andi Mann @AndiMann @Splunk

Use Machine Data To Enable Continuous Improvement

Defect Information

CapacityPlanning

Quality Standards

Enhancement Requests

Integration Requirements

Acceptance Metrics

Service Levels and KPIs

Application Development Test and Acceptance Production

BuildCodePlan Test/QA Stage Release Config Monitor

InfrastructureDependencies

Andi Mann @AndiMann @Splunk

Use Machine Data To Accelerate Velocity

Pivot & improve with Continuous Insights

Product Managers identify new opportunities

Continuously delivered to market

… and Auditors are “happy”

Andi Mann @AndiMann @Splunk

Use Machine Data To Improve Quality

Code quality scans Static security scans

White BoxDeveloper checks in code

Automated Acceptance Tests

Dynamic Security Scans

Black Box

“Chaos Monkey” tests

Test Fail: Return

Test Fail: Return

X

X

Production

QA Prod Pattern

QA Pattern Library

Test Pass: Promote

Test Pass: Promote to Production

Pattern library used for test and

QA

Andi Mann @AndiMann @Splunk

Use Machine Data To Align Business Impact

Andi Mann @AndiMann @Splunk

Metrics that Matter Drive Better Feedback Loops

Improve Application

Velocity

Improve Application

Quality

Improve Application

Impact

Andi Mann @AndiMann @Splunk

Copyright © 2016 Splunk Inc.

Andi Mann

@AndiMann

We make machine data accessible, usable, and valuable to everyone