a continuous delivery journey -...

Copy r i g ht © S A S I nst i t ut e I nc. A l l r i g ht s r e se r v e d.

A Continuous Delivery JourneySHOBHA SUBRAMONIAN

APRIL 05, 2018

Picture source: QuizzClub.com

2

INTRODUCTION

• A Case Study in moving to Continuous Delivery

• One organization’s journey: How we made the transition

• Practices that may work for others

C o p y r ig h t © S AS In st i tu t e In c. Al l r ig h t s re se r ve d .

3

AGENDA

• WHERE WE WERE

• OUR NEW DESTINATION

• CHALLENGES

• TRANSITION PLAN

• HYBRID APPROACH

• START OF THE JOURNEY

• ROCKY ROAD

• MINDSET SHIFT

• CODE DELIVERY MODEL

• WHERE WE ARE NOW

• WHERE WE WANT TO GET TO

• DEVOPS


4

WHERE WE WERE

• On premise deployments

• Releases ~ once a year

• Long development cycle

• Agile sprints: Theoretically release-ready at end of each sprint

• However we were not actually release-ready!

• Big difference between being actually ready to release versus theoretically done


5

WHERE WE WERE (Continued)

………Release Planning, Scoping

Development & Test Iterations Last BuildSignoffRelease/Ship


6

OUR NEW DESTINATION

• Hosted environment on the cloud (using Amazon Web Services)

• Multi-tenant

• Single code base for all customers

• Short development cycle

• Continuous delivery

• Release every 4 weeks

• 4 week Agile sprints – goal to be release-ready at end of each sprint


7

Application(Source code, OS, Hardware)

AWSCustomer A

Customer B Customer C

OUR NEW DESTINATION (Continued)

Customer A’s data

Customer B’s data

Customer C’s data


8

OUR NEW DESTINATION (Continued)

Release Planning, Scoping

Development & Test

Release


9

CHALLENGES

• Typically the key to continuous delivery or release is:

• But Automation is not achieved in a day


10

CHALLENGES (Continued)

What is the automation we are looking for ?

• Automation of Unit testing

• Automation of Functional / End-to-End testing

• Automation of Regression testing

• Automation of Builds

• Automation of Deployment

• Automation of Deployment validation / Health checks

• Automated promotion and deployment to Stage

• Automated promotion and deployment to Production


11

CHALLENGES (Continued)

Other challenges:

• New leading-edge Cloud technology

• Design of Architecture with new Amazon Infrastructure Services and Platform Services

• Design and layout of various Instances / Environments

• Design of Code promotion process

• Development practices for Cloud, Multi-tenancy

• Security, Stability, Performance

• Extensive training; Some Hiring of new talent


12 Slide from

13

TRANSITION PLAN

• An initial (Early Adopter / Beta) version would be released in ~10 months

• Allowed time for setup of our new delivery framework

• And training of resources

• Not going into detail about our Beta release…

• …but at the end of this phase, we had:• An operational software product in the cloud with 2 Early Adopter customers

• Next, we had to make the leap to continuous delivery• As we started bringing on additional customers• As we moved from Early Adopter / Beta to full Production mode

• For this we arrived at a Hybrid approach.


14

HYBRID APPROACH

• We would have a release every 4 weeks

• However, every release would follow a 3-sprint (12-week) cycle

• 3 Sprints / Phases:• Story Development and Testing

• Integration Testing

• Release Candidate and Production Deployment

• This essentially meant a small waterfall for each release

• Over time, we would reduce the length of this cycle


15C o p y r ig h t © S AS In st i tu t e In c. Al l r ig h t s re se r ve d .

16

HYBRID APPROACH (Continued)

• Dev & Test phase• Majority of work for release done in this phase

• Common sprint in Jira for all teams

• Same sprint schedule: Same start and end dates

• Cut over to Integration phase• Small team focused on Integration phase tasks

• Cut over to Release Candidate phase• Even Smaller team focused on RC phase tasks


17

START OF THE JOURNEY

• We were to start continuous delivery in September 2015.

• And put out a release every 4 weeks after that.

• We had a complex product suite that had:• 11 teams (most in Cary, NC; some in India, UK, and other locations)• Every team had 6-12 members (Dev + Test)• 4 project managers

• We had set standards across the program for: • Jira practices • Story template and Workflow• Acceptance criteria


18

ROCKY ROAD

• Most teams had been together only a few months

• Still going through Storming, Norming - Transitioning to Performing

Source: Okpalad, based on Tuckman and Jensen


19

ROCKY ROAD (Continued)

• First continuous delivery release did not come in September

• … or in October …

• Our scope was too large for one sprint

• We did not achieve Minimal Viable Product (MVP) in sprint

• Could not complete testing which was still largely manual

• We continued work to complete most of the scope

• And addressed build, deploy and other challenges

• We finally delivered in December – 3 months beyond plan


20


• Retrospectives to see how to reign in the scope

• Scope for next release slashed significantly

• Minimized dependencies between teams

• Scrum of scrums held for close inter-team collaboration

• Sharepoint and Jira process standardization for consistent use

• Incorporation of Early Adopter feedback


21


• In 2016, we missed 4 out of 13 planned releases.

• In 2017, we only missed 1 out of 12 planned releases.

• (We did not do December releases anymore.)


22


• Automation was building up

• But Master Builds were still breaking often

• We developed stringent practices for teams to merge code

• Must pass automated tests (Lev 0) before requesting merge

• Merge to Master was controlled


23

MINDSET SHIFT

• Meet the need to get new features to customers quickly• One of the biggest benefits of continuous delivery

• Plan significantly smaller scope to finish in a sprint

• Deliver basic functionality first – additions/embellishments after

• Not okay anymore to slip several half-finished stories

• Chunk up stories so they are “deliverable” in sprint

• Code must be production ready at end of sprint!

• No more moving out of planned dates

• Design of scope with the above in mind


24

CODE DELIVERY MODEL

• Several build processes experimented

• Automated tools but still manual handoffs

• Monolith code base… but moving now to Microservices

• Goals:• Code coverage target – 80%

• Sprint completion target – 80%

• Defect rate below .1 defects per Story Point

• Weekly and monthly reports to track all this


25

CODE DELIVERY MODEL (Continued)

INTEGRATION

MASTER / DEV

Features A & B complete -merged into MASTER

Feature C incomplete -

moves to next sprint

Features C, D, E complete -merged into MASTER

MASTER (Sprint N content) branched off as new release

Production

Rel moves from INT to RC phase –Rel 0.91

Rel deployedto Prod – Rel 0.91

Production release dateEnd of cycle for R0.91

Tool / process controlledpush

Start of cycle

Rel 0.91 Rel 0.92

RC (Release Candidate)

Sprint N Sprint N+1 Sprint N+2

Rel moves from INT to RC phase –Rel 0.92

MASTER (Sprint N+1 content)branched off as new release


26

CODE DELIVERY MODEL (Continued)

• Developers only push code to Feature branches

• When ready to merge to MASTER, they send a request to Merge Contacts.

• Merging code into INTEGRATION requires a higher level of care • Needs approval from manager

• Merging code into RC requires extreme care • Needs approval from manager and director


27

WHERE WE ARE NOW

• We are now delivering a release reliably every 4 weeks

• We have recently achieved 80% of planned scope delivery• This is intermittent though – and still needs work


28

WHERE WE WANT TO GET TO

• Ability to deploy once a day in production automatically• Especially for defect fixes

• Ability to deploy without any downtime / outage

• Ability to complete releases in 1 sprint versus 3 sprints

• Requires Automation of all testing and deployment pipeline

• Elimination of manual handoffs


29

WHERE WE WANT TO GET TO (Continued)

• Reduce build time. Currently 3+ hours

• Ability to deploy only a component versus monolith


30

DEVOPS

DevOps is:

• The collaboration between Development and Operations

• To identify efficiencies and techniques

• To Speed up the delivery of the release to the customer

31

DEVOPS (Continued)

32

Questions?


33

Contact info: [email protected]

linkedin.com/in/shobha-subramonian-1967016


https://www.linkedin.com/in/shobha-subramonian-1967016

a continuous delivery journey -...

Documents