Praqmatic Software Development

Continuous Delivery Maturity Model

Continuous Delivery is a software development approach that enables teams to always have releasable code in stock. Through the means of automation and extensive tool support, qual-ity assurance is literally built into every single step of the process and is being continuously enforced.

Learn more about the Continuous Delivery Maturity Model and be inspired to take the natural next-step for your current process.

Continuous Delivery Maturity Model

Continuous Delivery Defies Phases

Continuous delivery is a different approach for organizing the development process. It’s a strategy that defies phases. Instead it introduces extensive automation and execution of tests and verifications upon every single commit from the developers.

The doctrine is that each commit is a potential release candidate – you just have to test it to see if it is – or not. Continuous delivery advocates that you always have releasable code in stock, so when the release date comes, you simple push the release button.

Continuous Delivery and the Agile Manifesto

The Agile Manifesto is a manifest formulated by the founding fathers of the agile movement (Fowler, Beck, Sutherland, Cockburn and a handful more ...all the agile celebrities).

The manifest is stating four values that are the basis of the agile approach and twelve accompanying principles that should be seen as guides. If you are not already familiar with it, we recommend you to be inspired at agilemanifesto.org.

In this article we will just point out one of the values, the second, which states that:

We value working software over comprehensive documentation

Software development is iterative by nature. Each release is followed by another and each release is regarded as a separate cycle founded on the previous ones.

Traditionally the software development process is perceived as having phases such as development, integration, deployment, testing, and release, but unfortunately these phases are all notoriously unpredictable.

The release date can not be planned.

The solution is to automate all consolidation and verification steps and execute them upon every single commit.

Development Integration Testing Deployment Release

Development + Integration + Testing + Deployment + Release

Development + Integration + Testing + Deployment + Release

Development + Integration + Testing + Deployment + Release

Development + Integration + Testing + Deployment + Release

Development + Integration + Testing + Deployment + Release

Traditional iterative and incremental development

Continuous delivery

2

and then the first of the twelve principles that says:

Our highest priority is to satisfy the customer through early and continuous delivery

of valuable software

It is generally recognized, that it’s the focus on working software and this first principle of the agile manifesto that first cornered the phrase “continuous delivery” and thus gave name to a whole movement of a modern approach to quality assurance in software development.

A staged approach to Continuous Delivery

In order to implement the required amount of automation it’s generally accepted that continuous delivery needs extensive tool-support.

Sometimes these tools are referred to simply as the tool-stack.

It may include anything from test frameworks, version control systems, continuous integration servers, code coverage tools, build frameworks, style checkers, static code analysis tools, deployment consoles, change management systems, lots of virtual machines ...and potentially a lot more.

At first glance the tool-stack can seem overwhelming. However, it’s widely acknowledged that the human and organizational factors such as the required changes in the software development processes, in the project roles, potentially in the system architecture, and sometimes even in the line-organization itself impose a far larger challenge than the tools themselves.

The complexity involved calls for some kind of divide-and-conquer-approach.

Recall the continuous delivery doctrine that every commit is a potential release candidate. Now imagine drawing a state-transition diagram of the process that should validate a release candidate from the point where it’s committed to the VCS and through the verifications that would finally promote it to release candidate status.

The initial state is the creation of a new commit that the CI server finds. The desired end-state is to become a release candidate – but if some step in the state transition fails – then the commit will be rejected.

It could probably look something like this:

Analysis and metrics

Build &Unit test

Commit Rejected

Release Candidate

Failed

Failed

Failed

Failed

Failed

Failed

Passed

Passed

Passed

Passed

Passed

Passed

Deploy

Functional test

Review and doc.

Acceptance test

3

To the left is a screen dump of what an implementation of a continuous delivery pipeline could look like using Jenkins CI.

Each row is initiated by a separate commit to the VCS. If an action succeeds it will kick off the next action in the pipeline. If it reaches the fourth level it’s a release candidate. For this particular product shown here, releases are manual (political) and the last job which releases - in this case to an Open Source community - is run manually.

The staged approach to implementing continuous delivery is very popular because it supports first building a simple foundation with relatively few means and then later extend and improve it either by baby steps or giant leaps depending on the level of ambition in the organization. And it supports the mixed focus on both tools and organizational changes.

An implementation of this staged approach is often referred to as the pipeline.

The Corner Stone of the Continuous Delivery Pipeline

It is quite obvious, when looking at the state transition diagram, that each stage in the promotion of a commit consists of (at least) one action followed by a decision; a decision to either promote and start the succeeding action or to reject the commit.

The tool that is required as the foundation of the tool-stack is one that is capable of executing all kinds of various actions (potentially even some you can’t imagine today) and making decisions.

These tools are referred to by different names but are most often called either builds servers or

continuous integration servers – despite the fact they do much more than just build or integrate.

A handful of the popular and well-known build- and integration servers today are Jenkins CI, Travis, Hudson, CruiseControl, TeamCity, Buildbot, FinalBuilder, Bamboo, AnthillPro, Team Foundation Server ...and there are many more.

Signs of Continuous Delivery Maturity

Some of the first steps towards a continuous delivery pipeline can be very cheap low hanging fruits; ripe and juicy and simply there for the taking for any modern development team. Other steps might require such a large effort, that the return of investment is in the scope of the long term strategic planning in the line organization and potentially not interesting for the typical project oriented focus in software development. In other words; implementing continuous delivery is very much a discipline of recognizing when enough is enough.

During our work in Praqma with helping many different companies in their implementations of continuous delivery and through the seminars, conferences, and training classes we conduct, some of the most common questions we get in the field of continuous delivery are questions like “where do we start”, “what to do next”, “what’s the benefit of that particular step”.

When it comes to continuous delivery we have learned to recognize the various signs that indicate different levels of maturity, and we have used them to create a model. We use the model to systemize and categorize our findings, observations and recommendations.

4

In Praqma we are all very engaged in continuous delivery and we find the model useful when discussing this topic with clients, colleagues user group companions, audiences etc.

The Continuous Delivery Maturity Model

The model is organized so that the signs to the left are those we typically see with novice users and teams, while the ones to the right are typical signs of the experts.

Horizontally in the model we have organized the signs in different categories. Currently Build, Test & Quality Assurance, Software Configuration Management, and Visibility. But one thing we have learned since releasing this model is, that even though we have strived to used broad categories that could embrace most of our findings, we have realized that the disciplines that are influenced by continuous delivery are even more numerous than the categories we have in the model today.A natural development of the models would be

a horizontal expansion into more areas where especially “system design and architecture” and “organizational culture” are obvious candidates.The remainder of the article will go through each of the areas and discuss the rationale we have used to organize it the way we have.

Build

Build is the first and most obvious discipline that influences continuous delivery; building, linking, managing dependencies, optimizing builds performance, supporting build avoidance, keeping track of artifacts, and utilizing distributed resources.

An interesting perspective on this obvious discipline is that in some quite popular programming languages it’s not very obvious. In fact it’s simply not existing. This is true for interpreted languages such as PHP, Perl, Phyton, Groovy, Gradle etc.

Build

Test +QA

Visibility

SCM

5

The infant stage of building, where teams practically don’t show any sign of maturity at all, is clearly when there is no maintained build procedure at all. Individuals simply build by pressing the “build” button in the GUI of the IDE – and that’s it.

Novice signs of maturity in the build category is when some script or procedure is being maintained and developers use the formalized script

or procedure to verify their work before sharing it with their colleagues – ideally before a check into the integration stream.

At this level it’s also common to see that the team has a designated build server scheduled to run a nightly build.

Beginner signs of a more advanced build procedure is simply when this common build is upgraded from being a nightly build to a build that

is triggered automatically by every commit to the integration branch.

At this level we also see that the derived objects from the build process are managed somehow.

Intermediary signs in the area of building is when teams use build systems or frameworks that rely less on procedural scripting and more on

configurations. It’s at this level the home-grown perl, make and ANT scripts are being replaced by extendible frameworks, such as Gradle, Maven, Jenkins etc. A configuration approach to build management may be more difficult to set up, but once it’s in place it supports consistent and generic use much better than a procedural script. At this level we also see that build procedures become capable of actually using the maintained artifacts to manage dependencies, which improves the system architecture and enables build avoidance.

Advanced signs of build maturity involves a more sophisticated build environment, which supports distributed builds and load-balancing.

Either a build farm or a continuous integration server that supports a master/slave concept.

At this level we also typically see that builds, tests, and verifications are organized using a staged approach. The pipeline is now a reality.

Expert signs in this field is when your whole build environment is based on virtual machines and accessible to the software developers and testers

as a natural part of their tool box. Disk images and snapshots are being maintained to enable easy and controlled availability of more resources on demand.

Sometimes we even see that the load balancing and call for more resources are done automatically by the continuous integration server orchestrating the hypervisor or cloud.

Test & Quality Assurance

While the build area was mentioned first due to its natural initial focus on continuous delivery, Test, and Quality assurance must be mentioned next, simply because of its importance. Seen from the outside, nothing is more important than assuring the quality. This is done through testing and verification.

There are two important discussions that we often return to when dealing with this topic. They are not directly part of this model, but nevertheless important to understand our rationale and point of view so we will allow ourselves a short recap:

1st discussion: Verify vs. Validate:We often see that people use these terms interchangeably to express the same thing. They don’t! We use the terms as they are distinguished by The U.S. Food and Drug Association (FDA); to verify something means to prove that it’s correct according to its specification. To validate something means to prove that it has value – to the purpose it should serve. In short: if you get the requirements wrong and misinterpret the users needs you will be creating the perfect solution to the wrong problem. It may be compliant with the specs, but it has no value. This solution can be verified, but it can never be validated.

2nd discussion: Unit test vs. Functional test:We are frequently met with the understanding, that the tools used for testing are also used to define the different types of test; a test executed by a unit test framework (such as NUnit, Junit PHPUnit, or CppUnit) is a unit test - the rest are functional tests.

That’s not how we see it. Our understanding is, that unit tests are used to test only the semantics of the code; the purpose of unit tests is to verify

6

that the code behaves as specified. It’s important in unit testing that you don’t get tangled up in hard-to-handle dependencies such as loading databases instantiating long sequences of objects before you get to the point. In unit testing you should use mock-up frameworks, fakes, stubs, and proxies. Our rule of thumb is, that a full unit test should be able to run in the same amount of time it takes to compile the code – or less.

Functional testing is all about getting a good gut-feeling before you ship. That will typically include firing up all you dependencies, databases, connections, live-feeds, and what have you. A functional test is typically broken down into different test suites and to run all of them may take quite some time, depending on your gut-feeling. As opposed to unit testing, the functional test runs on a production-like environment, which indicates that you should probably deploy first and run your functional tests in the environment you deployed to.

Functional tests may also be executed from a unit test framework – we often do that - but it will not include mock-ups.

You should measure the coverage of your unit tests, but you shouldn’t measure the coverage of your functional tests.

The infant level of test and quality assurance, before you even get a score on the chart, is when you rely solely on the developer’s individual judgement and the statement “it worked on my machine when I shipped it”.

Novice signs in the area of Test & Quality assurance is to introduce a unit test framework and to start measuring the code coverage from

the unit tests. At this level developers start to see unit testing as an integrated part of the software development process and unit testing becomes as natural as compiling and linking. It’s being done in conjunction with every build.

Beginner signs is when teams start producing more sophisticated metrics. Examples could be style checkers, static code analysis,

calculating cyclomatic complexity, scanning code for fixmes and todos, compiling documentation from the embedded comments.

It’s also at this level that teams start introducing mock-up frameworks and start using stubs and proxies to get more efficient unit tests and raise the level of coverage well above the fifties.

Intermediary signs of test and quality assurance is when the functional tests are being automated as well. Teams start supporting their

functional tests by using test robots, GUI test frameworks, sophisticated test scripts, and test management systems.

It is also at this level that peer-reviews or formal reviews are being introduced.

Advanced signs in this field is when an automated deployment becomes part of your QA strategy. This enables you to test in target and production-

like environments with a full and managed set of dependencies.

At this level we also see that the management and maintenance of the test data is integrated in the strategy.

Within embedded software development, this is the level where teams start automating the tests on the actual target platform as opposed to using simulators.

Expert signs of testing and QA is to dare automate even the acceptance test. The acceptance test is typically when the test

starts changing its focus from verification to validation (as per the discussion earlier). Some people may claim that trying to do an automated validation is not advisable at all – this shouldn’t be automated; it should always involve humans. We do acknowledge that this is sometimes true, dependent on what type of software you are creating. But it’s far from always true and if you are an expert you would challenge such a statement.

7

Software Configuration Management

The version control system and the way the software configuration is managed are essentially the set of techniques and supporting tools that have the largest impact on the way the team is collaborating. It is within this field that we find the tools that utilize the team collaboration and keep track of the overwhelming complexity that can surround the software development processes.

If your software configuration is not managed – and if you score low in this field – then we suggest that this should be your main focus, even preceding build, test, and quality assurance.

The infant stage of software configuration management is of course when it’s unmanaged; if you are not using a version control system (VCS) at all or if the VCS you are using does not support branching – and merging. Or if the VCS you use only supports versioning of files and does not support versioning of directories (I can think of at least Visual Source Safe or CVS that fails this feature demand and therefore disqualifies as bearer of the title “version control system”).

Novice signs of maturity within software configuration management is when branches are utilized to some extend. Even if it means that

the branching model is “early branching”. The term refers to a branching strategy where branches are created proactively – early – and used mostly as a variant of labels for storing releases and planning ahead.

In “early branching” merges are quite rare, the branches have a tendency to stay apart and not integrate again. Subversion (SVN) is very good at branching but notoriously not very good at merging. Often SVN users get stuck at the “early branching” strategy unless they are very ambitious.

Beginner signs of a more mature branching strategy is referred to as “late branching” - which is an approach where branches or clones

are more often used to actually isolate work for shorter periods and where merges are quite frequent and used fearlessly among developers and release managers – The aim is that every branch created should eventually be merged

back to the mainline – or closed. This branching approach is very popular in distributed version control systems such as Git, Mercurial, and Bazaar. Although “late branching” is a term that has been used within SCM for many years, this strategy is more and more frequently being referred to as “the Git branching strategy”.

Intermediary signs is when commits in your VCS are tied together with tasks or work-items in your change management system – and you start

to maintain data, that will allow you to gain some kind of traceability overview.

At this stage we typically also see, that if you are in a Distributed VCS the developers will start using history to rewrite features in their local clones before pushing to the centralized repository – simply to tidy up things and keep information crisp.

Advanced signs are recognized when companies implement pre-verified commits to resemble the well known work process of most Open

Source communities, where benevolent dictators and their loyal lieutenants keep guards of OSS repos and don’t let the small fish alter the pristine repos themselves but must rely on pull requests and review queues.

This approach can be implemented on the CI server, which will then take on the role as the benevolent dictator – or in this case the willing serf. The desired benefit is that the project’s integration branch is kept forever pristine. It is simply impossible for a developer to break the build for his colleagues.

Commits that break the build or don’t pass the unit test are automatically rejected.

Expert signs: If you are an expert in this field you utilize all the meta data persisted in your SCM system and its integration to the change

management system – and potentially other systems as well – to automatically generate release notes, bill of materials, and traceability analysis reports upon every build.

8

Praqmatic Software Development

Visibility

We consider visibility to be a very important part of the continuous delivery strategy. The visibility helps build a healthy culture around continuous delivery. Everybody is constantly aware of the current status and progress of the software. Monitors are being set up in the canteen, at the entrance, and in work areas. Producing software and releasing it is a team effort and pipelines being visible or at least available to all stakeholders – potentially even end-users and customers – supports a healthy culture and builds confidence.

Novice signs of visibility maturity is when the result and status of the triggered action are instantly sent or notified to the person who initiated

the build – the same person who committed the change.

Beginner signs is when this information is not only sent to the one who is to blame, but also stored and made available to all interested

stakeholders.

Intermediary signs are shown when the build reports are analyzed as they roll in on the build or CI server and are being used to create trend

reports and graphs.

Another maturity sign at this level is that the information can be subscribed to – typically by feeds – so that people can compile their own individual information level.

Advanced signs: At this point reporting is typically so mature that it becomes really useful and work areas are being equipped with monitors

that show the real-time state of the software – across all products and their dependencies.

Expert signs: Real experts in this field trust their process to an extend where sharing this information with customers and potentially even the

public becomes a natural thing to do.

Measure your Continuous Delivery Maturity Index (CDMI)

We recognize that this is not a complete maturity chart. We knowingly use the statement that this is a chart of signs that indicates different levels of maturity. It could easily have included more categories such as design, system architecture, organizational culture, and probably even more.The fact that continuous delivery embraces such a diversity of disciplines is a point by itself. Caring for and developing your continuous delivery pipeline can be an important catalyst for driving the change that is required to enable a release strategy and software quality that is expected by todays end-user.

Poorly managed malfunctioning products are being punished by the end-users with the hardest sentence possible; they are simply being abandoned in favor of more mature and complete products.

Those companies who master continuous delivery in the near future may see this as a competitive advantage but we predict, that within just very few years it will be considered a competitive necessity.

Please got to our web and take the CDMI test: http://www.praqma.com/cdmi-test

We have prepared a handful of questions for you and based on you replies, we will calculate the Continuous Delivery Maturity Index and mail you back with a small report and analysis.

9

Praqma A/SAllerød Stationsvej 43450 AllerødDenmarkTel: +45 36 PRAQMAEmail: info@praqma.netWeb: www.praqma.net

Feed-back and discussions

We encourage you to actively join the ongoing debate about continuous delivery.

Sign up for the user group on LinkedIn “Continuous Delivery Users” and share your thoughts or follow the same user group on twitter @contdeliu.

Feel free to contact us in Praqma – we are eager to help.

Mail: info@praqma.com Tel: +45 36 PRAQMA

10