Session ID:

How We Think About ITPreordains Our Outcomes:

DevOps And More…

Gene Kim

Act I: IT Ops Fixing Fragile Artifacts

Act 2: The Product Managers

Act 3: The Developers

Act 4: Dev And IT Ops At War

8

The Downward SpiralOperations Sees… Fragile applications are prone to

failure Long time required to figure out “which

bit got flipped” Detective control is a salesperson Too much time required to restore

service Too much firefighting and unplanned

work Urgent security rework and

remediation Planned project work cannot complete Frustrated customers leave Market share goes down Business misses Wall Street

commitments Business makes even larger promises

to Wall Street

Dev Sees… More urgent, date-driven projects

put into the queue Even more fragile code (less

secure) put into production More releases have increasingly

“turbulent installs” Release cycles lengthen to

amortize “cost of deployments” Failing bigger deployments more

difficult to diagnose Most senior and constrained IT

ops resources have less time to fix underlying process problems

Ever increasing backlog of work that cold help the business win

Ever increasing amount of tension between IT Ops, Development, Design…

11

The IT Core Chronic Conflict

Every IT organization is pressured to simultaneously: Respond more quickly to urgent business needs Provide stable, secure and predictable IT service

Source: The authors acknowledge Dr. Eliyahu Goldratt, creator of the Theory of Constraints and author of The Goal, has written extensively on the theory and practice of identifying and resolving core, chronic conflicts.

Every Company Is An IT Company…

95% of all capital projects have an IT component…

50% of all capital spending is technology-related

We are here…

Where we need to be…

IT is always in the way

(again…)

There Must Be A Better Way…

13

Source: John Allspaw

Source: John Allspaw

Source: John Allspaw

Source: John Allspaw

Source: John Jenkins, Amazon.com

The First Way:Systems Thinking

The First Way:Systems Thinking (Left To Right)

Understand the flow of work Always seek to increase flow Never unconsciously pass defects downstream Never allow local optimization to cause global

degradation Achieve profound understanding of the system

“Annual business planning sessions can be madding. They think IT Operations is an ‘all you can eat buffet.’”

-Ben Rockwood, Director Systems Engineering, Joyent

Practice #1: Define The Work and Make It Visible

Business projects (e.g., new order system) Internal IT projects (e.g., Puppet automation) Changes (e.g., deploys, improve database

performance) Unplanned work (e.g., site down, site impaired)

22

Day 2: PMO Meeting

Practice #2: Create One Step Environment Creation Process

Make environments available early in the Development process

Make sure Dev builds the code and environment at the same time

Create a common Dev, QA and Production environment creation process

Change the Agile sprint policy:

“At the end of each sprint, we must have working code and the environment it runs in!!

The First Way:Outcomes

Creating single repository for code and environments

Determinism in the release process

Consistent Dev, QA, Int, and Staging environments, all properly built before deployment begins

Decreased cycle time

Reduce deployment times from 6 hours to 45 minutes Refactor deployment process that had 1300+ steps

spanning 4 weeks Faster release cadence

The Second Way:Amplify Feedback Loops

The Second Way:Amplify Feedback Loops (Right to Left)

Understand and respond to the needs of all customers, internal and external

Shorten and amplify all feedback loops: stop the line when necessary

Create quality at the source Create and embed knowledge where we need it

“We found that when we woke up developers at 2am, defects got fixed faster than ever”

-Patrick Lightbody, CEO, BrowserMob

Pattern #3: Embed Dev Into IT Ops

Embed Dev into IT Ops incident escalation process

Invite Dev to post-mortems/root cause analysis meeting

Have Dev and Infosec cross-train IT Operations Ensure application monitoring/metrics to aid in

Ops and Infosec work (e.g., incident/problem management)

The Second Way:Outcomes

Defects and security issues getting fixed faster than ever

Reusable Ops and Infosec user stories now part of the Agile process

All groups communicating and coordinating better

Everybody is getting more work done

The Third Way:Culture Of Continual Experimentation And Learning

The Third Way:Culture Of Continual Experimentation And Learning

Foster a culture that rewards: Experimentation (taking risks) and learning from

failure Repetition is the prerequisite to mastery

Why? You need a culture that keeps pushing into the danger

zone And have the habits that enable you to survive in the

danger zone

Break Things Early And Often

“Do painful things more frequently, so you can make it less painful… We don’t get pushback from Dev, because they know it makes rollouts smoother.”

-- Adrian Cockcroft, Architect, Netflix

Pattern #5: Inject Failures Often

You Don’t Choose Chaos Monkey…Chaos Monkey Chooses You

Pattern #6: Break Things Before Production

Enforce consistency in code, environments and configurations across the environments

Add your ASSERTs to find misconfigurations, enforce https, etc.

Add static code analysis to automated continuous integration and testing process

Pattern #6: Allocate 20% Of Cycles To Technical Debt Reduction

Recognize Compounding Technical Debt…

That Gets Worse…

And Fixing It…

Source: Pingdom

An Innovation Culture

“By installing a rampant innovation culture, they now do 165 experiments in the three months of tax season.

Our business result? Conversion rate of the website is up 50 percent. Employee result? Everyone loves it, because now their ideas can make it to market.”

--Scott Cook, Intuit Founder

42

Why Do I Think This IsImportant?

44

The Downward SpiralOperations Sees… Fragile applications are prone to failure Long time required to figure out “which

bit got flipped” Detective control is a salesperson Too much time required to restore

service Too much firefighting and unplanned

work Urgent security rework and

remediation Planned project work cannot complete Frustrated customers leave Market share goes down Business misses Wall Street

commitments Business makes even larger promises

to Wall Street

Dev Sees… More urgent, date-driven projects

put into the queue Even more fragile code (less

secure) put into production More releases have increasingly

“turbulent installs” Release cycles lengthen to

amortize “cost of deployments” Failing bigger deployments more

difficult to diagnose Most senior and constrained IT ops

resources have less time to fix underlying process problems

Ever increasing backlog of work that cold help the business win

Ever increasing amount of tension between IT Ops, Development, Design…

46

The Three Ways: Some Patterns

First Way Second Way Third WayDefine The Work And Make It Visible

Make Environments Available Early

Wake Up Developers

Embed Dev Into IT Operations

Break Things Early And Often

Reserve 20% Of Cycles For Technical Debt Reduction

Help The Business Win…

With Support From Your Peers…

And Do More With Less Effort…

50

When IT Fails: A Business Novel and The DevOps Cookbook

Coming January 15, 2013 and Q1 2013

“The lessons in When IT Fails might just save your business if IT fails for you. Every IT executive should share this book with their business peers.” -James Turnbull, VP Operations, Puppet Labs and author of “Pro Puppet”

“The greatest IT management book of our generation.” –Branden Williams, CTO Marketing, RSA

“This book will have a profound effect on IT, just as The Goal did for manufacturing.’ - Jez Humble, co-author of the Jolt award-winning book Continuous Delivery, and Principal at ThoughtWorks Studios.

Our Mission: Positively Impact The Lives Of One Million IT Workers By 2017

For these slides, the “Top 10 Things You Need To Know About DevOps,” Rugged DevOps resources, and updates on the book:

Sign up at http://itrevolution.com Email genek@realgenekim.me

Or text “[email] 74730” to +1 (858) 598-3980

Visit: http://www.instantcustomer.com/go/74730