DEVOPS IS A VERBDevOps UMN 2017

THIS IS AWESOME…..THANK YOU!!

@ajdomie

INTRODUCTIONANDY DOMEIER

• 12+ Years @ SPS Commerce• 24x7 System Operations• Reliability Engineering• Performance Engineering

• DevOps Days & DevOps Meetup Organizer• Optimistic by nature

• “The disk is half empty”• Twitter: @ajdomie• My wife thinks she’s winning

@ajdomie

YOUR JOB IS NOT YOUR JOB:IN ONE SENTENCE, WHAT’S YOUR JOB?

Fred Kofman - https://www.youtube.com/watch?v=6OI7REyatq4

@ajdomie

DO YOU………• Developer:

• Write Code• Build applications that improve the educational experience for UofM students

• Systems Engineer:• Build and Operate Servers• Provide a stable environment for applications to operate effectively

• Database Engineer:• Build and Operate Databases• Provide a highly performing data store for applications

@ajdomie

@ajdomie

@ajdomie

THAT’S DEVOPS!

“IT@UMN IS A COMMUNITY OF IT PROFESSIONALS ACROSS THE UNIVERSITY WORKING TOGETHER TO SUPPORT AND ADVANCE THE ACADEMIC MISSION OF THE UNIVERSITY OF MINNESOTA.”

Guiding Principals - https://it.umn.edu/about/guiding-principles

SUBD. 1. MISSION. THE UNIVERSITY OF MINNESOTA (UNIVERSITY), FOUNDED IN THE BELIEF THAT ALL PEOPLE ARE ENRICHED BY UNDERSTANDING, IS DEDICATED TO THE ADVANCEMENT OF LEARNING AND THE SEARCH FOR TRUTH; TO THE SHARING OF THIS KNOWLEDGE THROUGH EDUCATION FOR A DIVERSE COMMUNITY; AND TO THE APPLICATION OF THIS KNOWLEDGE TO BENEFIT THE PEOPLE OF THE STATE, THE NATION, AND THE WORLD.

UMN Mission Statement - http://regents.umn.edu/sites/regents.umn.edu/files/policies/Mission_Statement.pdf

@ajdomie

DOES THIS CHANGE• The way we communicate• The way we solve problems• The way we use technology• What we build• How we build• When we build

• …….Everything

@ajdomie

WHY NOW?SOFTWARE IS EATING THE WORLD!

@ajdomie

“DIGITAL TRANSFORMATION”

@ajdomie

COMMON TRAITS OF WINNERS• Ship Fast, Ship Often

• Fail Fast & Iterate• Prove theories & learn faster

• They Win Together• Tesla:

• Tesla software is no good without the physical car• No one will buy the car if it doesn’t look good• Battery life is critical

• Airbnb & Lyft:• No available & secure payment solution, no drivers or hosts• No drivers or hosts, no platforms needed

@ajdomie

THE CORE OF DEVOPS•Culture

•Automation

•Measurement

@ajdomie

CULTURE:

WINNING AS A TEAM

@ajdomie

@ajdomie

@ajdomie

YAY DEVOPS!

@ajdomie

CULTURE• Ok, win as a team… got it.

THE END!

QUESTIONS?

@ajdomie

THIS IS REALLY HARD• Aligning to a greater goal

• Team goals & incentives• Individual goals & incentives

• Triple Down on Communication• “You mean I have to physically talk to them?”

• (yes, that’s what I mean)• Effective communication takes time• Listening is usually more important

• Safety in Failure• Fail fast & iterate• The perception of failure

@ajdomie

PRIORITY CONFLICT• Feature v. Bug• New Server Now v. Automate• Patch v. Build New• Prod is Broke v. Life

@ajdomie

TRANSPARENCY

@ajdomie

WHERE TO START….• Build empathy

• Between teams• Between individuals• Sometimes offense needs to play defense

• Build trust• Between teams• Between individuals

• Service teams• Work as if everything your team builds is a service• Align your service with the mission

Jeff Sussna Empathy the essence of DevOps:http://blog.ingineering.it/post/72964480807/empathy-the-essence-of-devops

RESPOND IN ISOLATION

• Hey Danielle, It looks like the site is acting up and when looking around the only outlier I have found so far is a cpu spike on the DB. Can you help me investigate this a bit more?

RESPOND AS A TEAM

WHAT CAN YOU DO TOMORROW THAT CAN MAKE A DIFFERENCE?

@ajdomie

AUTOMATE:BE BETTER, FASTER & ALSO BETTER

TECHNOLOGY MAKES SOMETHING BETTER EVERY DAY!

@ajdomie

HOW DO WE BUILD IT?

@ajdomie

HOW DO WE SECURE IT?

@ajdomie

HOW DO WE SHIP IT?

@ajdomie

HOW DO WE OPERATE IT?

@ajdomie

SMARTER NOT HARDER*• If something is hard (but important), do it often

• Provision servers• Provision DB’s• Test your code• Ship your code

• Decrease Friction• Make building & shipping more consistent (not easy)

• Limit Risk• Automated testing• Reduce the impact of failure

• Phased deployments• Feature flags• Monitoring

*Note: please continue to work hard (And smart )

@ajdomie

DIGITAL TRANSFORMATION• Teams that ship fastest win

• Theories are proven (or disproven), learn quickly• Fail fast (and safely) & iterate• Competitive advantage

• Win together• Automate to deliver a service• No single service wins the game• Share and you learn faster

WHAT CAN YOU DO TOMORROW THAT CAN MAKE A DIFFERENCE?

MEASURE:

VALIDATE& ITERATE

@ajdomie

@ajdomie

ALIGN YOUR GOALS• Favor action and move forward

• Fail fast and iterate • Time to value for your customer

• Encourage technical pride• Seeing your efforts providing value in production is good for the soul

• Limit risk to enable failing fast• Foster innovation

@ajdomie

DELIVER DELIGHTFUL SERVICE• If it moves, graph it! - @etsy

• If it doesn’t move, graph it anyways! – also @etsy• Performance to promises

• Incidents• Cause• Monitoring• Recover efforts

@ajdomie

SHARE YOUR DATA

WHAT CAN YOU DO TOMORROW THAT CAN MAKE A DIFFERENCE?

@ajdomie

RELIABILITY @ SPS

@ajdomie

RELIABILITY’S JOURNEY• 2 years ago:

• Single monitoring stack• Manual setup• Lacked proper variety of solutions

• The Goal: Expand service offerings• Logs• User experience & availability• Application performance• Notification platform

@ajdomie

RELIABILITY’S JOURNEY• 1 year ago:

• Basic monitoring stacks• Manual setup• Custom monitors were time consuming• Consistency was a fun idea

• The Goal: Monitoring as a Service• Establish a baseline that’s automated• Custom was easier• Consistency in solution types• Established experts of the service “monitoring”

@ajdomie

RELIABILITY’S JOURNEY• Today:

• Monitoring Service• Event based serverless monitoring setup• Prescribed reusable patterns for various situations• Single pane of glass & improved collaboration

• The Goal: Operations Automation Framework• Query-able documentation (Dependencies)• Beta: Advanced Visualization - Vizceral• Event based automation improving communication & transparency• Event based automation improving time to remediation• Encourage shared learnings and efforts, reusable functions

@ajdomie

RELIABILITY’S JOURNEY• What hasn’t gone well?

• Under communicated motivation and goals (yr1)• Generated a “Who are you to tell me” response

@ajdomie

RELIABILITY’S JOURNEY• What has gone well?

• Reduced friction by simplifying serverless deployments• Limited risk with architecture (getting harder)• Failed fast & iterated• Double downed on transparency

• Internal presentations• Blog posts• Advocates• All PRs welcomed

• Invested in empathy & collaboration

@ajdomie

RELIABILITY’S RESULTS• Reduced incident rates & time to resolution

• Positively impacted:• Customer experience• Team moral• Engineering productivity• Cross team collaboration• Technical pride

• SPS Mission: To be the most chosen and relied upon retail network.

@ajdomie

WHAT NOW?

@ajdomie

TO DO LIST• Deliver Services (regardless of org)

• Encourage reuse, care about customers• Solve problems at scale with tools• Leverage tools consistently

• Be Empathetic• Create transparency to priorities & goals• Build trust

• Fail Fast & Iterate• Change is a good thing, get good at it• Improve agility & velocity

• Share• Knowledge• Experiences

BIGGEST BLOCKERS

@ajdomie

• Cultural

• Technical

• Measurements

@ajdomie

DEVOPS IS A VERB• It’s collaborating• It’s automating• It’s communicating• It’s iterating• It’s trusting each other• It’s winning

WHAT CAN YOU DO TOMORROW THAT CAN MAKE A DIFFERENCE? (X3)

“”

DEVOPS ISN’T A GAME YOU PLAY UNTIL YOU WIN, IT’S A THING YOU DO THAT RESULTS IN WINNING.

@ajdomie

THANK YOU!!

QUESTIONS, DEMOS, COMPLAINTS, ETC…

@ajdomie

ARTICLES• Your job is not your job - Fred Kofman

• https://www.youtube.com/watch?v=6OI7REyatq4• Empathy the essence of DevOps - Jeff Sussna

• http://blog.ingineering.it/post/72964480807/empathy-the-essence-of-devops• Promise Theory – Mark Burgess

• “Scaling cooperation with Promises, not obligations”• http://markburgess.org/blog_cooperate.html• DevOps as an example of promise-oriented cooperation, without money