ops happens: devops beyond deployment - damon edwards

Ops Happens: DevOps Beyond Deployment

Damon Edwards @damonedwards

Damon Edwards

Ops Improvement

DevOps Consulting

Tools

I’ve gotten to see a lot…

Community

The systemic force behind most DevOps problems? Silos.

Dev Release OperatePlanning



Handoff

!Handoff

!Handoff

!



Application Knowledge

Handoff

!Handoff

!Handoff

!




Operational Knowledge

Handoff

!Handoff

!Handoff

!





Business Intent

Handoff

!Handoff

!Handoff

!





Business Intent

Handoff

!Handoff

!Handoff

!

Ownership but limited Accountability





Business Intent

Handoff

!Handoff

!Handoff

!

Ownership but limited Accountability

Accountability but no Ownership

Dev/Test Release Operate Planning

Common perception of Enterprise DevOps Transformations


Cross-Functional Product TeamsCross-Functional Product TeamsCross-Functional Product Teams

Common perception of Enterprise DevOps Transformations



?

Reality of most Enterprise DevOps Transformations

Reinforces siloed divide at most critical point in lifecycle



Handoff

!

Delays critical feedback loops



Prevents meaningful organizational improvement



Time to Market

Cost

Quality

Handoff

!

Enterprise Operations is under tremendous pressure

“Go faster!” “Open it up!”

“Be more secure!” “Be more reliable!”

Organizational complexity compounds the system complexity

Ticket System SysAdmin

NetEng

CDN

SysEng Ticket System

SecOps

Ticket System

DNS

PM

DBA

?I just need an environment! Something isn’t working!

Silos + Time = Islands of Automation

Puppet Chef

Shell ScriptsData ETL

PowershellScriptsNetwork

Management

Monitoring

Ansible

LegacyDatacenterAutomation

Container Management

SQL Tools

New

Tools

New

Tools

What has to happen to achieve end-to-end transformation?



2. Integrate3.Enable

Standard Platforms

Ops APIs

1. Simplify

Internal Service Providers



2. Integrate3.Enable

Standard Platforms

Ops APIs

1. Simplify

Internal Service Providers

What has to happen to achieve end-to-end transformation?

“Shift Left” as much Ops activity as possible

Writing / Running Automated Tests Writing / Exercising Deploy Automation Running Security Scanning Tools

“Deploy.”



Writing / Exercising Automated Runbooks Writing / Exercising Monitoring/Metrics Operational Control (safely!)

“Deploy.”

“Operate.”

HOW?



Writing / Exercising Automated Runbooks Writing / Exercising Monitoring/Metrics Operational Control (safely!)

“Deploy.”

“Operate.”

How do you “shift left” ops procedures and incident response?

Those who build something define the procedures to fix it Those who build something fix it when it breaks

1

2

But... How do you safely and securely give out access? How do you enable the experts to contribute remediations? How do you give the experts visibility into operations? How do you do postmortems days/weeks/months later?

Those who build something define the procedures to fix it Those who build something fix it when it breaks

1

2

How do you “shift left” ops procedures and incident response?

Step 1: Establish a Secure Ops Portal

Step 2: Establish a SDLC for Ops Procedures

Step 3: Connect with Enterprise Management Systems

Who created the procedure? Who reviewed it? Who? When? Where? Approval trail?

Step 4: Make Compliance Really Happy

Example of these principles at work

Mark Maun

Jody Mulkey

Justin Dean

Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html

Ticketmaster’s “Support at the Edge” model


Mark Maun

Jody Mulkey

Justin Dean


Ticketmaster’s “Support at the Edge” model• Automated Ops procedures written/vetted by the delivery teams


Mark Maun

Jody Mulkey

Justin Dean


Ticketmaster’s “Support at the Edge” model• Automated Ops procedures written/vetted by the delivery teams• Ops remained in full control of what can run and security policy


Mark Maun

Jody Mulkey

Justin Dean


Ticketmaster’s “Support at the Edge” model• Automated Ops procedures written/vetted by the delivery teams• Ops remained in full control of what can run and security policy• Empowered support teams with self-service ops tasks


Mark Maun

Jody Mulkey

Justin Dean


Ticketmaster’s “Support at the Edge” model• Automated Ops procedures written/vetted by the delivery teams• Ops remained in full control of what can run and security policy• Empowered support teams with self-service ops tasks• Empowered developers with limited self-service operations


Mark Maun

Jody Mulkey

Justin Dean


Ticketmaster’s “Support at the Edge” model• Automated Ops procedures written/vetted by the delivery teams• Ops remained in full control of what can run and security policy• Empowered support teams with self-service ops tasks• Empowered developers with limited self-service operations• Combined with new incident response model

Better for the business and better way to work

90% Reduction in MTTR 50% Reduction in escalations 55% Reduction of overall support costs

Recap

Automated Tests Deployment Automation Running Security Scanning Tools Automated Runbooks Monitoring/Metrics Operational Control

“Shift-Left” operational concernsTake the transformation

all the way to Operations

Explicit investment in process and tooling

Let’s talk…

@damonedwards

[email protected]

Our sponsors speak your language… DevOps.

Get my slides immediately

[email protected]

ops happens: devops beyond deployment - damon edwards

Technology