puppetconf 2016: there is no “i” in devops – bart driscoll, dell emc

Dell Technologies - Internal Use - Confidential1

There is no “I” in DevOps @ ScaleChange Culture in the Enterprise


Your Speaker

IT leader and evangelist within EMC Global ServicesPassionate about driving IT Performance through lean and agileFocused on application development and deployment operations (aka. DevOps)

R. Bart DriscollDirector, DevOps Services @bartdriscoll

https://infocus.emc.com/author/bart_driscoll/[email protected]

Austin, TX


There is no “I” in DevOps @ ScaleChange Culture in the Enterprise


Change Culture, Embracing DevOps

C H A N G E

I SPOSSIBLE


Change Culture, Embracing DevOps

1. MEASURE2. RADIATE3. EVOLVE

C H A N G E

I SPOSSIBLE

THREE KEYS:


• Previous experience

• Conference or training

• Personal network

• Blogs, articles, and books

• Other


DEVOPS


PROTOTYPEDEVOPS


PROTOTYPEDEVOPS

It will never work for my team!

Why should I change? Or care?

We already tried something like this before and it failed.

How is this better than what I am doing

today?

But, what about…

You can do that in PROD!

Cool! But, there is no money to scale?

Great! Too busy to help.

Yikes. Not ready for this.

Show me the ROI.


DEVOPS

Why?

Tools?

People?

Process?


Valley of Despair

TIME

ADOP

TION

Valley of DespairEARLY

PROTOTYPE


Missing the 3 Keys

MEASURE RADIATE

EVOLVE

Cannot overcome enterprise inertia.

Cannot unlock culture.


Culture is the way of life in an enterprise.

• Behaviors• Beliefs• Values• Symbols

Comprised of …… passed along by communication and imitation from one generation to the next

Generally accepted without thinking

about it.


It is the underlying, undocumented operating fabric.


HARDW H Y I S I T S O

T O C H A N G E ?


Culture is like a knit sweater. It is both pliable and resilient

to change.

Otherwise, it would shred and destroy

the enterprise at the first sign of stress.


TIME

PACE

OF

CHAN

GE

“Valley of Despair”Unravel

Re-stitc

h

It Takes Time


Overcoming InertiaUnraveling Enterprise Culture

MEASURE RADIATE EVOVLEDevelop a resilient,

scalable & extensible

system that adapts to change

Create a proven

destination & pathway for success with strong feed-back loops

Build experts and champions

to guide, mentor, and model new

culture


Why is Measuring Important

• Proof

• Visibility

• Feedback


My Favorite - Title Insurance Company

12,936 Hours of rework per year $970,200

Averaged 98 defects related to deployment

per release

4 Major releases a year for their

primary application

33 person hours to fix, test,

redeploy each deployment

defect

Average Chargeback

$75/person hour

264 Hours of rework per year $19,800

Averaged 2 defects related to deployment

per release

4 Major releases a year for their

primary application

33 person hours to fix, test,

redeploy each deployment

defect

Average Chargeback

$75/person hour

BASE

LINE

POST

-PILO

T

4 98 33x x

4 2 33x x


Why is Measuring Important

• Proof

• Visibility

• Feedback

Dell Technologies - Internal Use - Confidential25But what do we measure?

Andrew Clay Shafer @ Spring Conference 2016


Alignment

ENTERPRISE

IT DEPARTMENT

PRODUCT/APPLICATION

TEAM/INDIVIDUAL

FEED

BACK


Processes& Models

3-5 Year Objectives/Outcomes

AnnualObjectives/Outcomes

Improvement Targets

Improve Earnings Per Share

Driv

e Effi

cienc

yAc

cele

rate

Tim

e-to

-Mar

ket

Impr

ove

Qual

ity

By E

OY 2

016,

ena

ble

busin

ess t

o ca

nary

test

UI

feat

ures

with

Pro

duct

ion

user

com

mun

ity

By E

OY 2

016,

feat

ure-

driv

en

deve

lopm

ent (

FDD)

pr

actic

es a

re S

OP fo

r pilo

t te

am(s

)

By E

OY 2

016,

team

can

de

ploy

into

pro

duct

ion

ever

y 30

day

s or l

ess f

or o

f pi

lot a

pplic

atio

n(s)

By E

OY 2

016,

ope

ratio

ns

team

s can

supp

ort a

uto-

scal

e up

/dow

n fo

r web

and

ap

plica

tion

tiers

of p

ilot

appl

icatio

n(s)

By E

OY 2

016,

dev

elop

men

t an

d te

st te

ams h

ave

desig

ned,

dev

elop

ed a

nd

impl

emen

ted

CD fo

r DBs

in

supp

ort o

f pilo

t ap

plica

tion(

s)

Application Architecture PracticesPlatform/Server Management ProcessDatabase Development LifecycleRelease Management ProcessDeployment & Provisioning ProcessSoftware Development Lifecycle

AffectedProcesses &

Practices

S.M.A.R.T.Improvement Targets


Deploy Every 28 DaysTeam Level Metrics to Change Culture

Application Architecture Practices

Platform/Server Management Process

Database Development Lifecycle

Release Management Process

Deployment & Provisioning Process

Software Development Lifecycle

# of collaboration with Ops/Infra/Dev/QA on standards# automated tests created to validate standards

# code commits per week (configurations)# automated tests created per change (unit, integrate, capacity)# of conversations, calls, or webex with Infra, Ops, QA, and Dev

# code commits per week (database/schema)# automated tests created per change (unit, func., regression)# of conversations, calls, or webex with DBA, Ops, QA, and Dev

LATER (still manual promote to PROD)

# code commits per week (pipeline)# of collaboration with Ops/Infra on configuration changes

# code commits per week (application/tests)# automated tests created per story (unit, functional, regression)# of conversations, calls, or webex with ProdMgr, QA, and Dev


Metrics Tips

• Make sure you have (or easily can get) the data• Make sure it is measuring the desired outcome• Once new behavior is habit (aka. part of culture) remove

metric and add a new one; you can always bring it back if needed

• Make a big deal of achieving and beating metrics• Always stay aligned to goal and higher level goals/metrics


Overcoming InertiaMinimize the Valley of Despair

MEASURE RADIATE EVOVLEDevelop a resilient,



Create a proven




culture

Dell Technologies - Internal Use - Confidential31John Shook, 2010 MIT Sloan Management Review

Different Approach

CULTURE

VALUES AND ATTITUDES

WHAT WE DO

TRADITIONAL MODEL:Change thinking to change behavior

Distributethen

Inspect

Provethen

Radiate

DEVOPS MODEL:Change behavior to change thinking


APP GOAL+ +CD PIPELINE

The Pilot Team


Develop a Support system for changeCreate Champions

EXTERNAL COACH(es)

INTERNAL CHAMPION(s)

TECHNICAL SME(s)

Industry ExpertiseBroad Experience

‘Outsider’ Perspective

• Recognized leader• Politically savvy• ‘Insider’ Perspective• Pragmatic thought-leadership

• Known technical expert• Deep subject matter expert• Innovator/Early Adopter

• • •


• Cross-functional• Collaborative• Transparent• Iterative• Courageous• Retrospective

Tips for RadiationStart with the right team

“Why should we change?”

“Can’t do that because …?” “Impossible!”

“Can we do that?”

“Cool!”

CHAMPIONS


Tips for Radiation

• Be clear about why?• Do more and teach more• Expect failure, drive out fear• Collaborate and listen more;

compete less (internally)• Measure outcomes


Overcoming InertiaMinimize the Valley of Despair

MEASURE RADIATE EVOLVEDevelop a resilient,



Create a proven




culture


Organizations which design systems ... are constrained to produce designs which are copies of the communication structures of

these organizations

System Mirror and Enforce CultureConway’s Law of System Design

Conway, Melvin E. (April 1968), "How do Committees Invent?“, Datamation


AUTOMATION ALONE

DOESN’T MAKE YOU GO FASTER!

44 42


Organizations which design systems ... are constrained to produce designs which are copies of the communication structures of

these organizations

System Mirror and Enforce CultureConway’s Law of System Design

Conway, Melvin E. (April 1968), "How do Committees Invent?“, Datamation


IterateIt is an evolution, not a revolution

* Illustration is from Jeff Patton

LEARNADAPT

USE

LEARNADAPT

USE

LEARNADAPT

USE

LEARNADAPT

USE


Model System HolisticallyOrganize around value creation

TOOLS &PLATFORMS

STANDARDS &PRACTICES

PROCESS &STRUCTURE

SKILLS &TEAMS

Development & Delivery Pipeline


Map tools to workflowCreate Pipelines

Deployment Environments Management & Monitoring

Version Control Build Tools Continuous Integration Tools Test Frameworks

TestManager

ACCEPTANCE STAGINGPRE-COMMIT DEV/COMMIT PRODUCTION

Configuration Managment Artifact Repositories

API FRAMEWORK & LIBRARIES

DevOps Workflow

TEST

API FRAMEWORK & LIBRARIES


Think LayersDefine Policy, Standards, Stage Gates, Workflows, etc. bottoms-up

APPLICATION SPECIFIC

ARCHITYPE/PORTFOLIO

LINE-OF-BUSINESS

ENTERPRISE-WIDE


Dependent systems; shared standardsAnd Scale

CRM Functionality Removed from Navigator (Client/Contact/Activity), just Deal Management Remains.

CEDTX Data

Navigator Services

DEALSTX Data

GCIB MSCRMMS Dynamics CRM 2015

DACTX Data

GWB MSCRMMS Dynamics CRM 2015

The next Phase of the Sales Platform:

Navigator: All CRM functionality will be removed from Navigator which will retain only Deal Management functionality.

Informatica will no longer push Clients/Entities and will start pushing read-only Deal data.

CRM Integration Engine will add support for Client/Entity data and will not longer sync Activities or Contacts with Navigator. Its data services will be the provisioning point for this data.

GPDS will be refactored to not update CDB or the legacy CED Orgs. Downstreams must stop using legacy CED Orgs.

Client Plans must move to GCIB MSCRM.

Client/Entity updates are gathered from a CED MQ topic (must research what MRDS plans).

Sales ADS: Theoretically, if GWB MSCRM contains all contacts and activities, it could serve as the aggregation point for that data. However, the Sales ADS will eventually become the single provisioning point and we will incrementally build it to that point. Until GDH is replaced and its downstreams refactored, its data will still flow to GDH.

CRM Integration Engine

MULE

FAST Mart

SalesADS

Real-time inserts, updates, deletes: Activities, Contacts, Clients/Entities.

Deal & Admin Codes

DealsAssociates

GDH ProfitClient Financials

1x/month

4x/day

SOSRef DataClient 360

4x/day

GCIB Reporting Lines, Ref

Data

Manual Process

Web Service Direct Call

Informatica ETL

Legend

Out: Plug-inIn: CRM API

MS Dynamics MarketingCloud

Dynamics Marketing Connector

Azure Service Bus

GPBS

DACClient/Server

NavigatorWeblogic Portal

GCBRTX Data

GDHSales Referrals

Sales Data4x/day

GCB Referrals

GPBS MQ Topic

Sync Data and Aggregated

Contacts, Activities, Clients, Client Plans

Refe

renc

e Da

ta, R

efer

rals

, Clie

nt

Plan

s, D

eals

& C

lient

360


Tips for Evolution

• Iterate, iterate, iterate• Be holistic by design• API-driven architectures• Define in layers starting with the application• Use dependencies to guide scaling


1. MEASURE2. RADIATE3. EVOLVE

C H A N G E

I SPOSSIBLE

THREE KEYS:


What can you start doing next week?

• MEASURE– Create a team metric or product metric to start changing behavior– Challenge everyone to create a metric that aligns with enterprise goals

• RADIATE– Write a blog, white paper, or submit code to open source project– Create a meet-up, brown bag session– Train some one on your team

• EVOLVE– Revisit your current automation platform

• What are the inputs? Outputs?• How can you make upstream/downstream tasks easier?


Thank you!

puppetconf 2016: there is no “i” in devops – bart driscoll, dell emc

Technology