teaching an old dog new tricks ci/cd for the enterprise · ci/cd for the enterprise: teaching an...
TRANSCRIPT
![Page 1: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/1.jpg)
CI/CD FOR THE ENTERPRISE:TEACHING AN OLD DOG NEW TRICKS
Open Infrastructure Summit 2019
Patrick EastersSr. Software Applications EngineerMay 1, 2019
![Page 2: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/2.jpg)
2
THE PIPE DREAM
CODE QUALITY
SECURITY SCANNING
CHANGE MGMT
PORTFOLIO MGMT
BUILD TESTAUTOMATICALLY
DEPLOY TOPRODUCTION
CONTINUOUSINTEGRATION
CONTINUOUSDELIVERY
![Page 3: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/3.jpg)
3
THE SQUADWHO WE ARE● 22 people (managers, developers,
SREs, quality engineers, BAs) ● Distributed across 3 states in the
US and 3 continents
WHAT WE DO● Build and maintain applications to
help Red Hat customers manage their subscriptions and access content
![Page 4: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/4.jpg)
4
WHERE WE BEGIN● Early stages of adopting DevOps practices, but still very siloed● IT Operations group performs majority of releases● Good use of Configuration Management, though it was complex● Averaged 2 releases/mo for our flagship application
![Page 5: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/5.jpg)
5
THE TERRIBLE, HORRIBLE, NO GOOD,VERY BAD RELEASE PROCESS
QA
Ops
Dev
BA
Manager
Change Manager
Dev
QA
Stage
Prod
Product Owner
PHASE
![Page 6: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/6.jpg)
6
THE TERRIBLE, HORRIBLE, NO GOOD,VERY BAD RELEASE PROCESS
![Page 7: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/7.jpg)
7
OPPORTUNITIES FOR IMPROVEMENT● Releases were a significant source of toil● Infrequent releases led to long cycle times● Confusing process
![Page 8: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/8.jpg)
8
PAAS TO THE RESCUE?● Red Hat IT launched new PaaS offering based on OpenShift 3● Application teams given access to their apps in production● Automated build and release jobs “for free”
![Page 9: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/9.jpg)
9
![Page 10: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/10.jpg)
10
CI/CD
![Page 11: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/11.jpg)
11
SCALING OUT● So far we’ve only migrated a few low-touch apps● Migrated our flagship app: access.redhat.com/management● Started a greenfield app: api.access.redhat.com/management
![Page 12: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/12.jpg)
SOLVING CONSTRAINTS
![Page 13: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/13.jpg)
13
DEV-ON-DEMAND● A single dev environment was no longer sufficient● Moved to short-lived instances following Merge Requests● Shifted left: code quality scans, unit tests● Quick feedback for QEs and BAs
![Page 14: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/14.jpg)
14
OPERATION “GO GREEN”● Many automated tests flickered and became unstable over time● Manual triage required by Quality Engineers● Conscious choice to allocate time for fixing tests instead of
developing new ones
![Page 15: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/15.jpg)
15
MODERNIZING CHANGE MANAGEMENT● Better risk categorization● Removed arbitrary lead time requirement and change freezes● Used “Standard Change” type that is pre-approved● Used ServiceNow API to open/close changes
![Page 16: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/16.jpg)
16
USER STORIES, NOT FAKE NEWS● Up-to-date Kanban board is important
○ WIP Limits○ Current Status
● Integrated pipeline with Rally APIs to move stories and create milestones
![Page 17: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/17.jpg)
17
IT’S BEGINNING TO LOOK A LOT LIKE CI/CD
![Page 18: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/18.jpg)
MEASURING UP
![Page 19: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/19.jpg)
19
DELIVERY PERFORMANCE
ELITE HIGH MEDIUM LOW
Deployment Frequency on-demand hourly-daily weekly-monthly weekly-monthly
Change Lead Time <1h 1d - 1w 1w - 1m 1m - 6m
MTTR <1h <1d <1d 1w - 1m
Change Failure Rate 0-15% 0-15% 0-15% 46-60%
From DORA’s 2018 State of DevOps report
![Page 20: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/20.jpg)
20
RHSM WEB CHANGE RATECh
ange
Req
uest
s pe
r Mon
th
![Page 21: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/21.jpg)
21
LEAD TIME
![Page 22: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/22.jpg)
22
THE HUMAN FACTOR
“Our release process used to feel more like filing taxes, but now it’s more like sending money with Venmo”
![Page 23: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/23.jpg)
23
THE HUMAN FACTOR
“My favorite part of our CI/CD adoption has been the freedom to experiment: having disposable merge instances for giving early feedback, feature flags that let us try things with a targeted audience and fine-tune before turning on new functionality for the entire customer base, and even the experimentation that comes with every member of the team feeling empowered to try automating some piece of our process.”
![Page 24: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/24.jpg)
24
THE HE-MAN FACTOR
“I am basically He-Man. By the power of CI/CD, I have the power*!”
*To push changes to production myself when an appropriate threshold of quality and organizational preparedness, much of which is built into our pipelines, is met.
![Page 25: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/25.jpg)
Q+A
![Page 26: TEACHING AN OLD DOG NEW TRICKS CI/CD FOR THE ENTERPRISE · CI/CD FOR THE ENTERPRISE: TEACHING AN OLD DOG NEW TRICKS Open Infrastructure Summit 2019 Patrick Easters Sr. Software Applications](https://reader034.vdocument.in/reader034/viewer/2022042223/5ec97a4dff3e4d2cef4fe7db/html5/thumbnails/26.jpg)
THANK YOUEmail: [email protected]: @patrickeasters