‘agile on steroids’: devops led enterprise agiledocuments.grenadine.co/pmi bangalore india...

‘Agile on Steroids’: DevOps led Enterprise Agile

Transformation

A Case Study

Sub-Theme: Strategies to transform organisation

Abstract ID: PMIBC-17-1-002

Project Management Practitioners’ Conference 2017

www.pmibanga lorechapter .o rg

CONTENTS

Abstract ............................................................................................................................................................................................................... 3

Introduction.......................................................................................................................................................................................................... 3

Details of the paper ............................................................................................................................................................................................. 4

Conclusion ........................................................................................................................................................................................................ 19

References ........................................................................................................................................................................................................ 20



ABSTRACT

Meeting the demands of an evolving marketplace is a constant challenge for most organizations. Product

development teams are transitioning to Agile development after decades of Waterfall. These teams are pushing the

operations groups to move even faster; are consuming more and more resources; and stepping across traditional

boundaries once exclusively the domain of IT Operations.

The struggle between achieving business goals (Dev!) and maintaining stable and secure infrastructure (Ops!) can

bring these two ends of any organization to instability. How do we tightly align these two loosely coupled groups?

DevOps is quite a buzzword these days. Everyone is starting DevOps teams, hiring DevOps engineers, and buying

DevOps tools. However, hiring engineers and buying tools constitutes only a portion of the effort.

This paper is based on our experience of using DevOps as fail fast approach to bring in a DevOps cadence, quick

error detection, faster automated testing and overall delivery efficiency through a Deployment pipeline. Further, the

paper will discuss a case study in detail with the 4 pillars of DevOps strategy to achieve the desired agile

transformation - Culture and Mindset, Micro Services based architecture, Built in Quality and Tools &

Automation.

INTRODUCTION

With technology disrupting every industry, products and processes that we used in the past, no longer fits in the

solution framework for the future. Organizations are looking for a new, faster and innovative service delivery structure

to serve their customers. As organisation started evolving, silos started to build up.

Agile software development has broken down some of the silos between requirements analysis, testing and

development. Deployment, operations and maintenance are other activities which have suffered a similar separation

from the rest of the software development process. The DevOps movement is aimed at removing these silos and

encouraging collaboration between development and operations.

DevOps has become possible largely due to a combination of new operations tools and established agile engineering

practices, but these are not enough to realize the benefits of DevOps. Even with the best tools, DevOps is just

another buzzword if you don't have the right culture.

The primary characteristic of DevOps culture is increased collaboration between the roles of development and

operations. There are some important cultural shifts, within teams and at an organizational level, that support this



collaboration, aimed at increasing collaboration and cooperation among different functions of an organization to

deliver faster, better, reliable and high-quality software solutions.

As per Gartner and Riley, 2017 is the year of DevOps transformation in majority of organizations, where we stop

talking about DevOps and start doing it:

According to 2017 DevOps State Report, in 2017, high-performing DevOps teams deploy 46x faster, enjoy 440x

faster lead time for changes, recover on average 96x faster, and suffer 5x fewer change failures. These organizations

were more than twice as likely to benefit from higher quality and quantity of products and services, better operating

efficiency and higher customer satisfaction, among other business impact goals.

DETAILS OF THE PAPER

HOW IT ALL STARTED – Silos started building across the organisation

The DevOps movement started to coalesce sometime between 2007 and 2008, when IT operations and software

development communities got vocal about what they felt was a fatal level of dysfunction in the industry.

They railed against the traditional software development model, which called for those who write the code to be

organizationally and functionally apart from those who deploy and support that code.

Developers and IT/Ops professionals had separate (and often competing) objectives, separate department

leadership, separate key performance indicators by which they were judged, and often worked on separate floors or

even separate buildings. The result was siloed teams concerned only with their own fiefdoms, long hours, botched

releases, and unhappy customers.

Surely there’s a better way, they said.

THEN…… AGILE HAPPENNED

Delivering products faster with better quality has led to agile practices that are dramatically impacting how

organizations deliver software. As agile delivers shorter and smoother development cycles - known as sprints -

SILOS STARTED BUILDING ACROSS…

https://www.atlassian.com/it-unplugged/devops



organizations are enjoying less risk by deploying smaller development packages with shorter delivery cycles. There

is also a greater involvement of the part of development with understanding business needs and requirements.

Operations teams however, are finding themselves struggling to deal with releases on a much more accelerated

basis. This has created many software release challenges.

While agile has helped improve development productivity and quality, the next natural step of the software delivery

evolution was to take agile to production - linking development with operations - which gave birth to DevOps. DevOps

is a set of practices and principles intended to help development and operations work more effectively together. The

challenge is how to effectively implement DevOps in a demanding business setting.

• Focusing on faster development cycles packed with features…

• Documentation to working software each iteration

• Waterfall releases to Incremental high value feature releases

• Dev + Test – one agile team with cross functional skills

LEADING TO… ‘WALL OF CONFUSION’

Developers always want to deliver changes as soon as possible. Operations want reliability and stability.

Lee Thomson describes this as a wall of confusion between development and operations. This wall of confusion

not only exists between the mind sets of the two teams but also with the tools they use. Development uses some

tools and operation uses some other tools to perform the same stuff. Some of the paradoxical dilemma that exists

between Dev and Ops are as below:

Development Team Operations Team

Fast Innovation System Stability

Code Streaming Waterfall release

Easy Access Locked Security

Agile Delivery Project Planning

More features More simplicity

BUT SILOS CONTINUED DEV + TEST …….. OPS



Smoothen the development cycle Smoothen the delivery cycle

This wall of confusion has the following impact to the delivery and development cycles and ultimately to the customer:

• Too costly defects found in the production

• Delay in Speed time to market (production)

• Dev and ops with different mindsets

• Traceability and feedback becomes difficult

• Blame game starts due to Human errors leading to havoc in the system

• Fail to adapt to changing requirements

• Ultimately. Dissatisfied customers

DEVOPS ADOPTION – BREAKING DOWN THE WALLS OF CONFUSION

DevOps is a cultural shift that breaks the silos between operations, development, tools and business to deliver a

competitive edge to the business.

Patrick Debois, who’s often called “the father of DevOps”, coined the word “DevOps” in 2009. As the word depicts,

it was formed by combining two words: “development” and “operations”. DevOps is a collaborative way of developing

and deploying software. DevOps (a portmanteau of development and operations) is a software development method

that stresses communication, collaboration and integration between software developers and information technology

(IT) operation professionals.

• DevOps is an approach based on agile and lean principles in which business owners, development,

operations, and quality assurance team collaborate to deliver software in a continuous stable manner

• DevOps is an environment that promotes cross practicality, shared business tasks and belief

• DevOps is a movement that improves IT service delivery agility

• DevOps is a culture that promotes better working relationship within the company

• DevOps is a set of practices that provides rapid, reliable software delivery

DevOps break down the walls of confusion between development and operations team, unifying development to

operations for better, faster outcomes.



AGILE + DevOps – The new enterprise agility

Agile and DevOps is the new enterprise agility. While Agile is about building quality products with shorter and

smoother development cycles, Dev Ops is about shorter and smoother delivery cycles to take agile to production in

order to achieve the following results:



• Develop and test against production-like systems

• Speed time to market ---> speed time to production

• Shift left development and operation concerns

• Deploy with repeatable, reliable process

• continuously dev, test -- Continuously dev, test, build, and release

• Independent testable and deployable unit

• Reduces defects cycle time – ability to reproduce and fix defects faster

• Fast Feedback (including visual)

• Consumer-centric and Value driven



• Optimizing on speed and quality

• Minimizes Rollbacks

According to McKinsey. companies can reduce the average number of days required to complete code development

and move it into live production from 89 days to 15 days, a mere 17 percent of the original time (Exhibit 1).

Exhibit 1

Case Study – Leading Industrial Automation Company

HARMAN partnered with a leading Industrial Automation company to support their end-to-end PDLC services for

mission critical projects. The engagement spanned across Business Consulting, Agile Transformation, DevOps,

Development, QA/Automation, Release Management, Sustenance to accelerate product development, test and

deploy with discipline and rigor. Product was Envisioned to Re-platform and Integrate over 20 different products

developed across 6 different geographies from a Monolith Architecture to a Cloud First service driven platform

enabling Cross sell / Up sell features across products. They intended to increase their global foot print thereby

expanding their customer base into Commercial, Residential and Enterprise Space with an aim to reduce release

cycle times from 6 – 8 months to 2 – 3 weeks and stay current in the market



ENGAGEMENT SUMMARY

- Team size – 200+

- Scrum teams – 14 scrum teams + 5 supporting teams (HUI, DB, API, DEVOPS and Architectural Runway)

- Release trains - 2 (Software and Firmware)

- Program duration – 3 yrs.+ roadmap

- Complexity includes Software and Firmware development teams.

CHALLENGES

• Monolithic code base posing challenges to build, test and deploy for every changes

• 25 + Services with manual Deployment, Manual Testing, 500 odd config changes ---errors leading to

unstable build

• Turnaround time to handover a refreshed build for QA = >1 week

• Environment creation was manual and tedious ---lack of automation

• No Built in quality like Unit tests, Static code checkers etc

• Dev doing the ops with very limited knowledge of tools, release management, dependencies, config.

Changes

• Tools – Just Bamboo and Manual Deployment

• Limited Safety Nets no API and UI functional tests

SOLUTION TENETS

HARMAN leveraged DevOps and Automation to achieve Fail fast approach, seamless integration and release for

large teams bringing in efficiency and productivity gain of up to 30%. HARMAN created a DevOps roadmap that

covered people, process and technology. We designed a unique operating model that leveraged SAFE Agile to drive

frequent releases and Synergies. HARMAN support DevOps culture and defines 4 essential pillars that will help you

to map out a DevOps journey for your company.



Agile Transformation:

SAFe was implemented across globally distributed teams involving software, firmware and hardware. The teams

were distributed across the following locations:

• Bangalore (Harman-India)

• Bangalore (Customer-India)

• Madurai (India)

• Shanghai (China)

• Bruno

Culture and Mindset:

DevOps isn’t just another software development methodology. It is a culture; an intersection of people, products and

processes to develop a performance-driven team structure. It isn’t for specific people or team, but for everyone who

is involved in delivering value to end users.

For DevOps to create an effective and seamless collaboration, the entire organization needs to embrace the mindset

and culture of DevOps. For E.g. The Entire infrastructure of Amazon is set up to do things faster than anywhere else

with the basic principle ‘you build it, you run it’. Following were some of the implementations:

- Train the team and make them aware about the impact of their mistakes

- Build the dev-ops team with App experts and tool experts

- Integrate code daily (Continuous Integration)

- Run Continuous Delivery (CD- complimenting CI)



- Promote builds on various environments

- Set up DevOps Cadence (Sample)

- Avoid name and shame but educate the team on implications

- In certain organizations, there is a penalty or made them wear a cap for breaking the builds

etc. should be avoided)

- Honor and respect the Master branch and always maintain the sanctity of it

For successful adoption and transition to DevOps, one needs massive business transformation initiatives along with

transparent communication across teams to ensure proper alignment. Such DevOps practices will help organizations

to provide faster, stable and continuous deliveries to survive dynamic market conditions.

Adapting Micro Services Architecture:

Organizations have been using monolithic architecture for long now. Monolithic architecture is like your big container

that has all the business services tightly assembled and coupled together in a single unit.

Due to its tight packaging of software applications in a single unit, the deployment of services become difficult

because even if a small component in the application needs a change, the whole application needs to be re-packaged

and re-assembled. It leads to high cost, reduced agility and scalability.

But, with time, companies realized the importance of micro services over monolithic architecture. Organizations

started looking micro services as an enabler of faster and agile application delivery. Micro services architecture

breaks down massive software projects into smaller, independent modules which are then defined for specific tasks

which can communicate with each other through simple APIs.

Micro services gave the concept of ‘fail fast and fix faster’. With micro services, parts of applications with errors can

be rectified independently and can be deployed easily and separately without touching the entire application. Smaller

codebases can be easily maintained by smaller teams which results in increased agility, scalability and reliability.

Thus, micro services make the organization nimble and agile in their development efforts by improving the uptime

and speed of execution. Following were the implementations:

- Monolithic to Micro-services

- Windows Services

- Easily deployable for changes



Building Quality-driven Mindset:

Organization needs to maintain synergy between quality and speed for the successful business delivery.

Quality assurance in DevOps organization is not limited to finding bugs but to prevent them from arising in the first

place. Software Development Life Cycle (SDLC) in DevOps culture requires rapid software development followed by

continuous testing, continuous deployment and continuous monitoring. In such a process, manual testing has to be

simply ruled out and test automation has to be deployed.

Automating the deployment process in DevOps environment leads to faster time to market, higher quality software

solution and increased organizational effectiveness.

DevOps practices accelerate the process of software delivery using Continuous Integration (CI) and Continuous

Delivery (CD) which allow business units to collaborate and function faster.

Few more important methods for Quality improvement are:

• Developing a Quality Improvement Plan based on customer feedback and internal defects

• UI, API, UT should be made part of Sprint development process so that automation becomes integral part

of delivery which would otherwise lead to rise of technical debt

• Address the technical debt incrementally (first by removing blockers then by performing code refactoring

wherever possible and lastly by resolving critical issues)

• Design & develop quality gates (UT >=90%, Code Complexity <=15, Code duplication <=5%) and strictly

adhere to it

• Infuse code checks (SonarQube, PCLint, KlockWorks, Coverity, Fortify etc.)

Tools and Automation:

As the companies are rushing to take their software early-to-the-market, the entire software development life cycle

(SDLC) is going through a paradigm shift. In the DevOps environment, you need to maintain a continuous feedback

system and with a lot of information moving back and forth, logging everything becomes critical.

- Therefore, integrating a tool system that lets you collaborate across development, testing and deployment

would save considerable amount of time which otherwise would take days during the DevOps

implementation.

- Tool selection is also important as it should reflect and adhere to a common set of business objectives

while providing seamless collaboration and integration between tools.



- Tools can be used to capture request and feedback. They can be integrated with survey platforms and can

provide customer feedback while social media platforms can be integrated to provide the team with real-

time feedback. It can later help in optimizing the products and spotting the gaps, issues and inefficiencies

among the team.

- In DevOps environment, deploying the right set of tools to the right users isn’t enough. A suitable

connectivity between the DevOps tools is imperative to create a collaborative work environment among

the team.

- Bring redundancy in tool chain (especially on Web - Azure, AWS, Verizon). Even a minor interruption in

network can cause build/deployment failure. So, manual intervention is required. For ex. If build failed to get

a NuGet package, re-trigger. Unable to push package to Octopus (Deployment tool) due to network error,

re-push..

- Choose always minimum tool set or otherwise, providing stable cadence is challenge (DevOps cadence is

end-to-end delivery, so, breakage between any tool set would break the entire chain and hence, failure)

BENEFITS TO THE ORGANISATION AND CUSTOMER

Before After

• Multiple, inorganically grown product bases

• Large amount of feature overlap among products

• Micro Service based architecture

• Standardized repository layout



• Monolithic code base posing challenges to build,

test and deploy for every changes

• 25 + Services with manual Deployment, Manual

Testing, 500 odd config changes ---errors

leading to unstable build

• Build times in excess of an hour

• Tools used in silos

• Ad-hoc library distribution/non-reproducible builds

• Complex and manual release process

• Migration from SVN to modular Git repositories

initiation

• Any new environment creation was a hectic task

and usually took almost a week time with lot of

stabilization issues

• Replicating the same environments was a

daunting task

• Turnaround time to handover a refreshed build

for QA = >1 week

• Environment creation was manual and tedious ---

lack of automation

• Limited Built in quality like Unit tests, Static code

checkers etc

• Dev doing the ops with very limited knowledge of

tools, release management, dependencies, config.

Changes

• Tools – Just Bamboo and Manual Deployment

• Limited Safety Nets no API and UI functional

tests

• Faster CI build times (takes ~ 5 mins currently)

• Sane branching and streamlined code reviews

with PRs (Feature enabled by using Git – Code

reviews become an online discussion that are

always in context of the change)

• Nuget for package distribution along with proper

versioning

• Nightly build and automated deployment prototype

• Code refactored with dependency injection

• Guided CI adoption across ~9 teams

• Modular MSI installers for each service

• ~Zero manual installation steps

• Feature toggling – Any feature can enable/disable

just by toggling

• New infra creation time has almost come down to

1 day

• Replicating environments is a mouse click

Practice Benefits

Sanity of master build - Dev CI

after every commit

- Highly stable build consistently across all environments

- Proper branching strategy, code review & merging process

- Faster CI build times (takes ~ 5 mins currently)



Scalable deployment model using

Micro services

- Capability of deploying any micro-service with a single mouse

click

- Supported multi environment, geographically distributed

DevOps Cadence - Disciplined DevOps Delivery (DDD)

Automated and more frequent

deployments

- Overall cycle time to hand over build to QA has come down from

5 days to 3.30hrs.

Daily CI and CD Build - Better stability

- Fail fast mechanism and prompt feedback via mails on CD

- CI builds for rapid feedback on every push

Faster recovery from failures - Daily visibility

Built in quality and safety nets - Shift left quality

- Integrated Built-In-Quality (UT, Static Code Check), if

parameters doesn’t meet goal, build will fail

- Involvement of Code Committers as part of review for every Pull

Request

- Growing set of unit tests that provide fail-fast behavior

- SonarQube for transparency on Code Quality

- Code Coverage/Unit Test reports [SonarQube]

- Code analysis report from SonarQube

- Nightly Builds with automated E2E test execution - Always in a

‘KNOWN STATE’

- API test suite integrated and runs as part of CD

- Selenium integrated and runs as part of CD

More (environments) with less

with latest code base

- Automation

No weekend or late night shifts

during release times

- Better Work life balance

Single view of current state –

JIRA, Stash, Bamboo and

Octopus tools integrated

- Integrated tools for better E-E- traceability

Release velocity over the past 10

releases

While, few builds appear to be failed, in actual scenario, it involves

whole build creation of all 16 micro services, UT, Static code check,

MSI creation and push the package to Octopus for deployment and all



are successful. However, below builds are connected to one more

build plan called ISP. Deploy. Master which triggers the Octopus

Deployment, where it failed because of low API test percentage. But,

end-to-end deployment is successful.

Average velocity of past 10 release is around 208 minutes which

includes Compiling+ UT+ Static Code Check+ MSI creation+ pushing

the package to Octopus for further Deployment

CRITICAL SUCCESS FACTORS:

• Average time after a check-in to build, test and deploy to an environment

Average Build time: 208 minutes

API Test execution time: 45 minutes (Report doesn’t have time stamp)

Redeploying the database: 15 minutes (We are compelled to redeploy DB because some of API test cases

do direct transaction with DB and populates lot of data. Currently tearing down is not happening.

Run Selenium Test suite: 6hrs (There are around 1000+ test cases that are distributed in 5 production grade

servers which fires through batch scripts in parallel)

TOTAL TIME TAKEN: 10hrs 28 minutes is overall cycle time

• Build promotion process & automation

Inferences: Build from Dev to Functional QA environment once API test percentage is greater than 70%. From

FQA to BETA once Selenium is greater than 80%.

• Build traceability back to issue (JIRA) and commit (Version Control)

• Build reproducibility i.e. ability to re-generate any past build with 100% exactness

Octopus has a provision of restoring any package back to date and choose the package and deploy. However,

as per retention policy, we are keeping only 10 packages. Even said that, restoring single package alone may

DEV FQA BETA

API > 70% Selenium >

80%



lead to service disruption given the context of dependent services also should be deployed. The dependency

matrix is already available and based on that the Services will be chosen for rollback plan.

• Code review process: For eg. Is it possible to revisit an old ticket, see the code that was written and the code

review feedback for the said code

Before merging the code from Future branch to Master branch, developer has to raise the Pull Request (PR).

This Pull Request has to be approved by a role called Code Committer (CC). The review comments given by

CC has to be fixed and once CC approves PR, the commit is authorized to merge to Master

• Current system to manage code quality (static analysis, code duplications)

• DevOps Cadence



LESSONS LEARNT

Business today is compelled to deliver business technology in cycles of hours, or days. Faster cycles render

not only tradition “waterfall” processes and silo based IT obsolete, it also renders traditional metrics ineffective!

These arcane metrics no longer deliver the visibility and granularity tech pros need to fine-tune their delivery

capability. The mission has transitioned to RAPIDLY deliver high quality, high value solutions. For all, this is a

significant shift from the past, when the main points of focus were schedule, cost, and efficiency. Modern

software metrics — speed, quality, and value — are based on continuous feedback from business partners

and customers. Some of the key takeaways, listed and briefly described below.

CONCLUSION

Waterfall gave way to V-Model which in turn was replaced by Agile as the preferred choice for software

development. DevOps is a natural evolution of the software industry, enabling organizations to do more with fewer

resources while differentiating leading and successful companies from their competitors. DevOps aim is aligning

development and operations roles and processes in the context of shared business objectives, and organizing them

into principles and practices.

Big visible dashboards

Teams across the organization get real-time information on the state of the software system, including build status, customer metrics, and availability.

Colocation Teams are in physical proximity to one another to enhance communication.

Continuous integration

You build your software (environments, applications, and so on) with every change.

Cross-functional teams

Software-delivery teams are composed of experts across various disciplines, including programmers, testers, analysts, and operations.

Polyskilled experts

Reduce specialist silos by expanding skill sets on cross-functional teams.

Scripted deployments

Deploying software to environments is entirely scripted so that it can be run from a single command.

Scripted environments

Creation of environments is entirely scripted so that it can be run from a single command.

Self-service releases ("You build it, you run it.")

Any authorized person on the team can and does perform deployments to production.

Stop the line Anyone can and should stop the continuous-integration system when necessary.

Test-driven everything

Write automated tests for everything: application, infrastructure, everything. This might include writing unit, acceptance, load, and performance tests.

Version everything

Version all artifacts: infrastructure, configuration, application code, and data.



Successful DevOps requires a symbiotic blend of culture, tools and methods. These pillars will allow organizations

to achieve a 360-degree view on how to build a DevOps strategy for their business to function faster and reduce risk

during SDLC through automation or shorter development iterations. It’s a continuous improvement cycle that

software development models undergo from time-to-time. You need to embrace, understand and inculcate it. You

need to master the various automation and continuous integration tools, so that your automation efforts add value

to the chain and are lean enough to quickly adapt to changes.

DevOps is the future….

REFERENCES

[1] Tech Insights Report: What Smart Businesses Know About DevOps, September 2013

[2] https://www.linkedin.com/pulse/devops-creating-agile-steroids-culture-stefano-la-porta

[3] DevOps for Dummies – Sanjeev Sharma

[4] https://devops.com/devops-2017-reflections-predictions/

[5] http://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/beyond-agile-reorganizing-it-for-

faster-software-delivery

[6] https://puppet.com/resources/whitepaper/state-of-devops-report

[7] http://www.gartner.com/newsroom/id/2999017

[8] http://dev2ops.org/wp-content/uploads/2010/02/WallOfConfusion.png

[9] http://www.collab.net/solutions/devops http://www.collab.net/solutions/devops

https://www.linkedin.com/pulse/devops-creating-agile-steroids-culture-stefano-la-porta

http://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/beyond-agile-reorganizing-it-for-faster-software-delivery

http://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/beyond-agile-reorganizing-it-for-faster-software-delivery

https://puppet.com/resources/whitepaper/state-of-devops-report

http://www.gartner.com/newsroom/id/2999017

http://dev2ops.org/wp-content/uploads/2010/02/WallOfConfusion.png

http://www.collab.net/solutions/devops

‘agile on steroids’: devops led enterprise agiledocuments.grenadine.co/pmi bangalore india...

Documents