DevOPs/Continuous Delivery for Cloud Product

Experience Report

Summary

Aricent has implemented “DevOps” method for automating delivery pipeline of Aricloud Product/Framework

This experience report covers the following Objectives behind the DevOps implementation Details the steps involved in implementing the DevOps Highlights various technical and non-technical challenges

faced during DevOps implementation Key Learnings to overcome the challenges

Why DevOps for Aricloud Product?Aricent IT is potential customer for Aricloud Product – (Cloud

Monitoring System). Aricent IT is planned to use Aricloud Product for managing and monitoring their private cloud based data center.

After every release, Aricloud Development Team has to create and upload the Aricloud Package in a network and then needs to co-ordinate with IT Operations Team to deploy at IT Staging/Production environment for testing

Manual process of Aricloud deployment was more time consuming and “overall delivery to deployment” had taken more time and delayed the delivery milestones

“DevOps” methodology is selected to completely automate the entire delivery pipeline

Transitioning to DevOPs Old Process – Manual Integration & Delivery

BuildBuild TestingTesting DeployDeployCode Check-in

Code Check-in

manualmanual manualmanual manualmanual

• DevOps – Continuous Integration & Delivery

DevOps ImplementationDevOps implemented using our Jenkins CI tool capability by adding Plugins to automate the entire delivery pipeline

Steps Involved: Develop the DevOps pipeline Ability to disable specific stage(s) in pipeline Design & Implement build promotion logic taking promotion

decision based on output of current stage Setup Email alerts for build promotion success or failure at each

stage in pipeline Log all the DevOps events including failure details and all the Test

results Triggering of DevOps workflow - check-in driven & schedule

driven Automated logging of new defects into Bug Tracker

(Bugzilla/Mantis) on build/test failures

DevOps ImplementationFollowing Plug-in used with Jenkins: https://wiki.jenkins-ci.org/display/JENKINS/

Conditional+BuildStep+Plugin https://wiki.jenkins-ci.org/display/JENKINS/

Run+Condition+Plugin https://wiki.jenkins-ci.org/display/JENKINS/

Flexible+Publish+Plugin https://wiki.jenkins-ci.org/display/JENKINS/

Any+Build+Step+Plugin https://wiki.jenkins-ci.org/display/JENKINS/Build+Pipeline+Plugin https://wiki.jenkins-ci.org/display/JENKINS/

Delivery+Pipeline+Plugin https://wiki.jenkins-ci.org/display/JENKINS/Join+Plugin https://wiki.jenkins-ci.org/display/JENKINS/Multijob+Plugin

Challenges FacedTechnical Challenges

• Using each of the Plug-in involved to configure and make it work in Jenkins involved its learning curve.

• Integrating the links among various stages of pipeline took considerable amount of time

• Setting up the Email Alerts for build promotion success/failure involved its own set of challenges.

Team had overcome the challenges by exploring and experimenting the Jenkins Plug-in and its configurations and were able to make it work each of the stages correctly in DevOps pipeline

Challenges FacedCultural Issue/ Resistance from IT Operations:

IT Operations has low trust on “DevOps” as they feared low quality code may be deployed into Staging/Production environment due to automated continuous delivery. So there were initial resistance to implement DevOps for their deployment.IT Operations team was walk through the complete process by showing the adequate test coverage in Continuous Integration and given confident by showing “Fewer changes per release means more control for operations, even if those releases happen more frequently”

This challenge is overcome by setting the “ground rules” and ensuring smooth co-ordination and effective communication between development and operations team

Challenges FacedChange in Team’s Way of Working

Team’s Way of Working needs to be changed for adopting “Continuous Integration” and “Continuous Delivery” as part of DevOps. Aricloud product is developed by distributed Agile team across geographic, which added further complexity.This challenge was overcome by educating the teams on processes involved in Continuous Integration & Continuous Delivery and ensuring the effective communication between teams in DSMs.

Organisation Delivery processesIn traditional organisation delivery process, the Release Audit needs to be done prior to delivery /deployment of Aricloud product.This challenged was overcome by automating the Release Audit/checklist and ensured the delivery is done only if the release criteria met.

Test Automation for reliabilityAs the Testing stage in the DevOps pipeline involves running series of automated test cases on the newly deployed build, thus there was a need for sufficient test coverage. We automated the entire QA testing process using Selenium based implementation.

Breaking down language barriers between Development and operations teams:Development Team’s view and operations team’s view the from different perspectives and follows different approaches to same goal so the challenge included to bring both to common ground establishing a standard lexicon for both.

Learning new skills:The CI tools Jenkins/Sonar were new to the team. Also implementing DevOps on top of Jenkins tools involved understanding/configuring of so many new plugins on Jenkins each involving its own learning curve. Setting up the Email Alerts for build promotion success/failure involved its own set of challenges. Team worked on and and troubleshooting when issues do occur. Thus we built the proper consensus.

Challenges FacedTest Automation

As the Testing stage in the DevOps pipeline involves running series of automated test cases on the newly deployed build, thus there was a need for sufficient test coverage. This had also some challenges in allocating the budget for Test Automation.

This challenge was overcome initially by automating test cases based on “Risk based Testing” using Selenium. Later other test cases were automated in subsequent sprints for more test coverage.

Key Benefits Continuous Delivery reduces the time between Code checkins to

Production Deployment from weeks to few hours Improved communication, collaboration and integration between

Software Developers and Operations team Increased Customer satisfaction due to

Rapid turnaround on requested feature enhancements and bug fixes Improved deployment frequency Shortened lead time between fixes Faster mean time to recovery in the event of a new release crashing or

otherwise disabling the current system