dev ops with smell v1.2
TRANSCRIPT
DEVOPS ANTIPATTERNS
~WHOAMI
ANTONS KRANGA▸ Full stack developer ~ 15years
▸ Cloud Architect
▸ DevOps Advocate
▸ Full-Stack Developer
▸ Speaker
▸ Marathon runner@acankr
TALK STRUCTURE
PRESO PLAN
▸ Smell Symptoms: organizational anti patterns for devops
▸ Vitamins: proposed solutions from devops library
▸ Painkillers: deployment anti patterns
▸ Antibiotics: environment anti patterns brought by devops
pic from wonderful “matrix” movie
WHY DEVOPSDEVELOPERS VERSION
DEFINITION OF DEVOPS
What is DevOps?
pic from “hitchhiker's guide to the galaxy” movie
DEFINITION OF DEVOPS
What is DevOps?
BFG900 from classical game DOOM (c) id software
DEFINITION OF DEVOPS
What is DevOps?
▸ Development teams ▸ Operations
▸ Security and compliance ▸ Management
▸ Strategy, IT Strategy
BFG900 from classical game DOOM (c) id software
▸ QA▸ HR
DRIVERS FOR SOFTWARE DEVELOPMENT
▸ development price
BEFORE AFTER
▸ innovations speed
DRIVERS FOR SOFTWARE DEVELOPMENT
▸ development price
BEFORE AFTER
▸ innovations speed
▸ specialization silos
▸ ops comes first
▸ cross functional
▸ software defined data centers
▸ complex communication ▸ complexity theory
SMELL SYMPTOMDEVOPS ANTIPATTERNS
ANTIPATTERN # 0
BI MODAL IT
Sparky from Frankenweenie
▸ Slow IT
▸ Safe and Predictable
▸ IT Centric
▸ Secure and Regulated
▸ Fast IT
▸ Fast and Agile
▸ Business Centric
▸ Innovative
ANTIPATTERN # 1
DEVOPS IN A BOX
You cannot buy culture!
▸ Buy a Golden DevOps LVL5
▸ Become certified DevOps master
▸ Give me DevOps compliance checklist
▸ Give me 5 key DevOps control metrics
ANTIPATTERN # 1.1
REBRANDED IT
DevOps != Configuration ManagementDevOps != Release Management|DevOps != Product Management…DevOps != (*) Management
DEVOPS IS THE CULTURE YOU CANNOT FIND IN IN ITIL CATALOG
ANTIPATTERN # 2
DEVOPS AS SEPARATE TEAM
You don’t need to change anything in your org
You extend organization with new capabilities
ANTIPATTERN # 2
DEVOPS AS SEPARATE TEAM
Creates Unicorn vs Horses dilemma
DevOps builds their own Silo
Often loses focus on value and speed
ANTIPATTERN # 2.1
DEVOPS AS A SILO
DevOps teams build their own Silo
▸ You cannot talk to DevOps
▸ Use Jira instead !!!!
▸ RTFM Architecture
ANTIPATTERN # 2: PROPOSED SOLUTION
SORRY, NOT MY DEPARTMENT
Bread ownership and specialization with autonomous teams
vs
http://martinfowler.com/bliki/DevOpsCulture.html
ANTIPATTERN # 3
DEVOPS IN AGILE
▸ DoD fails with “ilities”
▸ Services can be easily “undone”
ANTIPATTERN # 4
FEAR OF RELEASE
RELEASE === RISK
NO RELEAE NO REVENUE
vs
ANTIPATTERN # 5
SNOWFLAKE SERVER
Applying changes to Server Instance manually leads to unique and distinct server configuration footprint (TECHNICAL DEBT)
http://martinfowler.com/bliki/SnowflakeServer.html
VITAMINSDEVOPS ANTIPATTERNS
VITAMINS
INFRASTRUCTURE AS CODE
ENV
ENV
ENV
DEV …CODE
VITAMINS
INFRASTRUCTURE AS CODE
DEV
Code is aContract OPS
Essence to collectively create and share knowledge
VITAMINS
INFRASTRUCTURE AS CODE
DEV
Code is aContract OPS
Enables: built in quality
VITAMINS
INFRASTRUCTURE AS CODE
DEV
Code is aContract OPS
Put infrastructure into SCM
VITAMINS
INFRASTRUCTURE AS CODE
DEV
Code is aContract OPS`
Make infrastructure part of app
VITAMINS
USE CI ENVIRONMENT
CI
test infra depl AT
...release
VITAMINS
USE CI ENVIRONMENT
▸ We need feedback not to be afraid
▸ Feedback != SPAM
▸ It’s about size of release not frequency
▸ Don’t judge for broken builds
▸ Go home when build is green
PAINKILLERSDEVOPS ANTIPATTERNS
CONTINUOUS DELIVERY
CI/CD PROCESS
CI
test infra depl AT
...release
ANTIPATTERN # 6
SLOW PIPELINES
CI
...release
test infra depl DT
test DT depl infra
infra depl DT AT
ANTIPATTERN # 6: PROPOSED SOLUTION
SLOW PIPELINES
`
`
`
Design for parallelization
ANTIPATTERN # 6
“JENKINS” DRIVEN DEVELOPMENT
With single master CI you easily get a single point of failure
CI master
ANTIPATTERN # 6: PROPOSED SOLUTION
MULTI MASTER “JENKINS”
Multi-master CI can dedicate CI Master per group of related components
CI master CI master CI master
ANTIPATTERN # 6: PROPOSED SOLUTION # 2
SHARE RESPONSIBILITY
CI users (DEVs or OPs) are best for managing their jobs
▸ Implement Pipeline as Code
▸ Store CI/CD pipelines in git
▸ Let pipeline evolve together with your app
ANTIPATTERN # 7
MANUAL PROMOTION
Tester PRODUATIntegr TestsDEV System Tests
ENVENVENV ENV
ANTIPATTERN # 7: PROPOSED SOLUTION
KILL SWITCH FOR MANUAL TESTING
PRODUATIntegr TestsDEV System Tests
ENVENVENV ENV
TEST
ANTIPATTERN # 7: PROPOSED SOLUTION
KILL SWITCH FOR MANUAL TESTING
PRODUATIntegr TestsDEV System Tests
ENVENVENV ENV
TEST
Time adjustableif TREND is good
ANTIPATTERN # 7: PROPOSED SOLUTION
AB TESTING
PROD A
Integr TestsDEV System Tests
ENV
ENVENV
TEST A
PROD B
ENV
TEST Bfeedback
feedback
measure
ANTIPATTERN # 7.1
CODEREVIEW
PRODUATIntegr Tests System Tests
ENVENVENV ENV
DEV B
feature branch DEV Ccode
review
ANTIPATTERN # 7.1: PROPOSED SOLUTION
REACTIVE CODEREVIEW
ReleaseIntegr Tests System Tests
ENVENV ENV
Full Regression
Calc technicaldebt
ENV
Nightly
DEV A
Standup
code review
…
ANTIPATTERN # 7.1
CODEREVIEW
PRODUATIntegr TestsDEV A System Tests
ENVENVENV ENVmaster
DEV B
feature branch DEV Ccode
review
ANTIPATTERN # 7.2
GIT FLOWS
▸ Git flow helps to work feature dev isolated
▸ Indirectly breaks Continuous Delivery Principle
▸ Complex merge routine distracts teams
http://classicprogrammerpaintings.com/post/142586036029/junior-programmer-learns-git-rebase
ANTIPATTERN # 7.2: PROPOSED SOLUTION
GIT FLOWS
develop
master
test
merge
CI
DEV
pull
push
ANTIPATTERN # 7.2: PROPOSED SOLUTION
GIT FLOWS
develop
master
test
merge
CI
DEV
DEVpull
push
ANTIPATTERN # 8
DEV DOESN’T CONTROL DEV ENV
DEVENV
DEVENV
DEVENV
…DEV CI
ANTIPATTERN # 8: PROPOSED SOLUTION
DEV DOESN’T CONTROL DEV ENV
DEV …NO CI
DEVENV
DEVENV
DEVENV
ANTIPATTERN # 8: PROPOSED SOLUTION
DEV DOESN’T CONTROL DEV ENV
ENV
ENV
DEVENV
…DEV NO CI
WANT TO HACK? SURE!
ANTIPATTERN # 8: PROPOSED SOLUTION
DEV DOESN’T CONTROL DEV ENV
ENV
ENV
DEVENV
…
YOU BROKE IT? YOU FIX IT!
DEV NO CI
ANTIBIOTICSDEVOPS ANTIPATTERNS
ANTIPATTERN # 9
GOLDEN IMAGE
VM
OS
Problems
▸ Maintained manually
▸ No collaboration
▸ Hard to distribute
▸ Non versioning
Chnorr Service
ANTIPATTERN # 9: PROPOSED SOLUTION
PROVISIONING
OS Chnorr Service
ANTIPATTERN # 9: PROPOSED SOLUTION
PROVISIONING
OS
Configure
Infrastructurecode
Harden Download Install
Chnorr Service
ANTIPATTERN # 9: PROPOSED SOLUTION
PROVISIONING
OS
Configure
Infrastructurecode
Harden Download Install
Chnorr Service
ANTIPATTERN # 9: PROPOSED SOLUTION
PROVISIONING
OS
Configure
Infrastructurecode
Harden Download Install
cmdb
Chnorr Service
ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructurecode
Harden Download Install
▸ Provisioning success < 100%
▸ Time to provision
Chnorr Service
ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructurecode
Harden Download Install
▸ Provisioning success < 100%
▸ Time to provision
Chnorr Service
ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructurecode
Harden Download Install
▸ Provisioning success < 100%
▸ Time to provision
Chnorr Service
ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructurecode
Harden Download Install
▸ Provisioning success < 100%
▸ Time to provision
Chnorr Service
ANTIPATTERN # A.1
PRIVATE DATA CENTER PROVISIONING
PRIVATE DATA CENTER: NO INTERNET!!!
OS
ConfigureHarden Download Install
Chnorr Service
ANTIPATTERN # A.1
PRIVATE DATA CENTER PROVISIONING
PRIVATE DATA CENTER: NO INTERNET!!!
OS
ConfigureHarden Download Install
…
apt yum gem
Chnorr Service
ANTIPATTERN # A.1
PRIVATE DATA CENTER PROVISIONING
PRIVATE DATA CENTER: NO INTERNET!!!
OS
ConfigureHarden Download Install
…
apt yum gem
complicated?
Chnorr Service
ANTIPATTERN # A.1: PROPOSED SOLUTION
STEM CELLS
static dynamic
OS
ConfigureHarden Download Install
Chnorr Service
ANTIPATTERN # A.1: PROPOSED SOLUTION
STEM CELLS
OS
ConfigureHarden Download Install
LAB PRIVATE DATA CENTER
Chnorr Service
ANTIPATTERN # A.1: PROPOSED SOLUTION
STEM CELLS
OS
ConfigureHarden Download Install
LAB PRIVATE DATA CENTER
code
packerPROVISIONMAKE
SNAPSHOT
docker
Chnorr Service
RECONFIGURE
ANTIPATTERN # B
DEPENDENCY HELL
OS
Configure
Infrastructurecode
Harden Download Install
▸ Version of libraries
▸ Version of packages
▸ Maintain dependencies
Chnorr Service
ANTIPATTERN # B
DEPENDENCY HELL
OS
Configure
Infrastructurecode
Harden Download Install
▸ Version of libraries
▸ Version of packages
▸ Maintain dependencies
▸ Version of your infra code
▸ Maintain dependencies
Chnorr Service
ANTIPATTERN # B: PROPOSED SOLUTION
CANARY BUILDS
Accept TestsCI
Unit Test Provision
…
PRECISE version libs
Accept TestsCI Unit Test Provision
…
LATEST version libs
Chnorr Service
Chnorr Service
ANTIPATTERN # C
INFRASTRUCTURE PETAttributes of Pet
▸ Have meaningful names
▸ Long living instance
▸ Often needs manual nursing
▸ Requires scary patching
▸ Leads to snowflakes
▸ PaaS is modern pet
ANTIPATTERN # C: PROPOSED SOLUTION
INFRASTRUCTURE CATTLEAttributes of Pet▸ Have numbers in its name
▸ Short living instance
▸ Immutable configuration
▸ Recreate instead of patching
▸ Requires careful planning
ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
Cattle
Pet
ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
Cattle
Pet
User Data
ANTIPATTERN # D
SECRETS LEAK
OS
Configure
Infrastructurecode
Install
cmdbwrong place
for your secretswrong place
for your secrets
Chnorr Service
ANTIPATTERN # D: PROPOSED SOLUTION
SECRETS LEAK
▸ Don’t store secrets with code
▸ Don’t store secrets with configuration
▸ Don’t leave secrets in service
Secrets DON’Ts
ANTIPATTERN # D: PROPOSED SOLUTION
SECRETS LEAK
▸ Secret can be leased and rotated
▸ tmpfs is your fiend
Secrets DOs
ANTIPATTERN # D: PROPOSED SOLUTION
SECURITY LEASING EXAMPLE
Chnorr Service vault
consul
IAM
Database
api
x hours leasing
AWS
TAKEAWAYSGOOD INFRA CODE
TAKEAWAYS
PATTERNS GOES BEFORE TOOLS
Patterns Tools
vs
TAKEAWAYS
EVERYTHING MUST HAVE AN API
REST
DSL
CLI
TAKEAWAYS
SPLIT YOUR SECRETS FROM CODE OR CONFIGURATION
(c) @eduardsi
TAKEAWAYS
SELF TESTABLE CODE
▸ Use assertions for infrastructure code
▸ Use acceptance test frameworks
Tests improves your confidence
CHEF MINITEST
package"mysql"
assert_file"/etc/my.cnf","mysql","mysql","644"
SHELL
apt-getinstall-yqqmysql
test!-f/etc/my.cnf||error_exit“boo”
TAKEAWAYS
HOW TO START DEVOPS
▸ Fastest way to start with Vitamins and Coaching
▸ Apply measures early
▸ Apply right measures
▸ Tools comes later
TAKEAWAYS
READING
▸ Book: The Phoenix Project
▸ Author: Gene Kim Kevin Behr George Stafford
▸ ISBN: 978-0988262508
TAKEAWAYS
READING
▸ Book: The Lean Startup
▸ Author: Eric Ries
▸ ISBN: 978-0307887894
TAKEAWAYS
READING
▸ Book: A Human Error Approach to Aviation Accident Analysis
▸ Author: Douglas A. Wiegmann Scott A. Shappell
▸ ISBN: 978-0754618737
THANK YOU
Download me here: http://www.slideshare.net/akranga/dev-ops-with-smell-v12
Twitter: @acankr