Whitepaper

Metrics-led Agile governance to reduce Agile software delivery risk

This is part of a series of abridged whitepapers intended as quick reference sources for busy managers interested in the subject matter and faced with

limited time to absorb lengthy research documentation.

It is based on research undertaken by Plandek drawn from anonymised data observed across a range of clients – from small start ups to large corporates

with large scale, distributed Agile teams.

IntroductionPurpose of the paper. Nature of analysis

The ResearchMetrics-led Agile governance to reduce Agile software delivery risk

SummaryConclusion

The ScopeThe Agile governance deficit. Metrics-led Agile governance

About PlandekPlandek is the leading Agile and

delivery metrics BI platform, providing an end-to-end view of

your software delivery cycle.

Our SaaS solution allows mining the data history from the toolsets that engineers use for actionable

insights.

We provide new insight derived from its unique end-to-end view of

the delivery process.

Introduction

The Agile manifesto is now nearly 20 years old. In that time, adoption of Agile has proliferated globally across technology teams and organisations of all shapes and sizes. Over 75% of Fortune 500 companies now adopt Agile in some part of their operation.

As with all new methodologies, Agile delivers many great benefits, but is not without its problems – particularly when applied at scale in larger organisations.

Purpose of this Paper Nature of the Analysis As a result, Agile governance is an increasingly hot topic. So how do you put in place effective governance and related metrics to give Agile teams the freedom to do what they do best – whilst ensuring that overall software delivery risk is effectively understood and mitigated?

This paper presents our view on effective Agile governance principles and approaches, based on our experience working with a great variety of Agile teams in organisations of all shapes and sizes.

The Agile governance deficitAgile was conceived with smaller organisations in mind. Whilst many of its principles translate very well into larger organisations (collaboration, improved communication etc) - some are paradoxical in large

and complex organisations.

Agile Paradox 1The first key paradox of Agile

for large organisations is that it is predicated on decentralisingdelivery responsibility to small

self-determining squads or teams.

Agile Paradox 2The second key paradox relates

to Agile’s focus on a more iterative and organic

development process, often with limited up-front estimation

of a task.

The Agile governance deficit

Agile Paradox 1:

→ This is an excellent principle, but whilst responsibility may be decentralised, accountability always remains more centralised in large organisations. So, the teams may be responsible, but the Delivery Manager remains accountable.

→ As such, it is vital that the Delivery Manager has a complete understanding of what teams are doing, if delivery risk is to be effectively understood and managed. Only then should responsibility be fully delegated to teams (in keeping with the key Agile principle).

→ If the Delivery Manager delegates delivery responsibility to teams and has little understanding of their daily delivery capability, then they are not doing their job.

Agile Paradox 2:

→ This has many benefits in many contexts (such as when responding to rapidly changing business needs in an organisation’s digital assets) - but can cause frustration with larger organisations’ commercial stakeholders who have expectations that software will be delivered at a certain time and budget.

→ All our experience and research show that stakeholder frustration (in delivery timing accuracy) is actually increasing as Agile is adopted.

The Agile paradox for larger organisations

Agile Small, decentralized, self-determining development teams (i.e decentralised responsibility but accountability remains centralised

Delivery risk if there is a lack of clear visibility across/within teams

Projects with fixed scope, time expectation and budget are not suited to a pure Agile methodology

Iterative, organic development process, test and learn – limited up-front estimation

Agile

01

02

These two paradoxes of Agile in larger organisations mean that effective Agile governance is vital if Agile software delivery is to thrive. On the contrary however, in our experience, Agile governance is often limited – with:- no agreed metrics to measure success; - limited manual reporting; and- a very limited understanding of delivery RISK across Agile teams (that may themselves be distributed in-house and offshore and involve contractor, vendor and in-house resource).

Metrics-led Agile governance

→ In our view, robust Agile governance is essential if the benefits of Agile are to be realised and delivery risk minimised. The objective of Agile governance is to understand and mitigate Agile delivery risk.

→ We see Agile delivery risk as principally concerned with:1. Risk of velocity problems and late delivery (deployment)2. Risk of quality and security issues3. Risk of productivity problems and process inefficiency 4. Risk of declining morale and staff turnover.

→ Meaningful metrics are critical to an objective assessment of risk. As such, effective Agile Governance tracks and measures risk in those 4 areas – with a set of end-to-end metrics that enable the risk to be quantified and mitigated where necessary.

→ Unfortunately, most organisations lack an analytics tool to provide a single source of metrics to track and understand the delivery risk. [This is why we developed Plandek, which has become the world’s leading delivery analytics platform, providing a complete set of end-to-end metrics (from pre-dev to deployment), to effectively understand and mitigate delivery risk.]

→ Examples of the key metrics used to understand and manage delivery risk are shown in the graphic on the next page. They are drawn from a number of data sources (see section 4 below), and provide a balanced set of metrics that together are deterministic of delivery risk – and can be used therefore to build an Agile Delivery Risk Profile.

Key metrics categories for effective Governance of Agile software delivery

DELIVERY

Flow efficiency Return rate

END-TO-END PROCESS ENGINEERING EXCELLENCE TALENT

Cycle & Lead Time Throughput Metrics

Delivery Risk scores

Engineer morale score

Engineer Sprint effectiveness score

“Reduce delivery risk and improve delivery effectiveness through robust governance’

%time Bugs/Features

Engineer team work rating

Vendor integration rating

Code metrics

Quality metrics

Deployment Metrics Sprint Completion

Key data sources and metrics to underpin effective Agile Governance

→ Effective Agile Governance requires surfacing metrics (in near real time) from a variety of different sources.

→ The key sources are shown in the graphic below. As is clear this sort of data collection is hard and time-consuming to do manually and really need a BI/analytics platform to do efficiently.

→ There are many potential data sources, but we believe that the most important are:

1. Workflow management tools (like Jira) to gain insight on the efficiency of the process of delivering software

2. Code repositories (and related code quality tools) to understand the quality of what’s written and the efficiency of the process of writing code (who wrote what, when and was it changed?)

3. Data drawn from CI/CD tools to understand the effectiveness of the build and deployment process; and

4. Collaboration hubs like Slack and Microsoft Teams which can be used to gain in-workflow feedback from engineers, which is a hugely valuable source of data when understand delivery risk (see section 5).

Key metrics categories for effective Governance of Agile software

delivery

Agile software delivery

cycle

Workflow Management software

Code repositories

Code quality tools

Time tracking software

Bitbucket, GitHub, GitLab, Azure DevOps

SonarQube

Harvest, TEMPO

CI/CD tools

Collaboration hubs

Jenkins, GoCD, TeamCity, Azure DevOps

Slack, Microsoft Teams

JIRA

Key metrics categories for effective Governance of Agile software delivery

→ These underlying systems provide a myriad of potential data, much of which is noise. In our view, the metrics that are the most valuable for effective Agile Governance are shown in the table below.

→ They are drawn from analysis of the end-to-end Agile delivery process and together they create a balanced view of delivery risk in the broadest sense.

→ We have placed them in the four categories of:

1. End-to-end software delivery process metrics – which consider the efficiency and stability of the process from pre-development through to deployment

2. Delivery (velocity, throughput and estimation) metrics – which consider how time to value and ability to estimate (and therefore forecast time to value) is changing

3. Engineering excellence metrics – which consider code quality and related infosec metrics representing another facet of Agile delivery risk

4. Talent metrics – which are collected from live polling of engineers to understand not only their morale, but also their view of process effectiveness. Software development is clearly all about the people involved, so we believe that this source of insight is critical to really understanding delivery risk.

Table showing a typical set of metrics used to determine an Agile delivery risk profile

Metric Area Metrics Relevance

End-to-end process Productive Time• % Time spent on New Features• % Time spent on Upkeep

Key to understand how this has trended over time. If we are spending more and more time fixing stuff, clearly this is going to impact our progress going forward.

Process Efficiency

• Flow efficiency (%)• Rework (days)

These metrics analyse the “friction” in our development process and how this has trended over time. Declining flow efficiency is a problem that can often be addressed. So, it is a key metric in accurate forecasting and mitigation.Rework is interesting as it shows trends in accumulated time spent reworking tickets that fail QA. This is another form of friction that can often be mitigated (e.g. by assisting engineers new to the code base).

Delivery Velocity and Time to Value

• Feature Tickets Completed• Cycle Time (days)• Lead Time (days)• Deployment frequency

Velocity metrics are always problematic, but a detailed understanding of trends in tickets completed (and story points per ticket) is a key thing to bear in mind when challenging forecasts.Critical too is an understanding of changes in Cycle and Lead Times. If they are lengthening accurate forecasting is tricky

Team Estimation Accuracy

• Overall Completion Rate (%)• Sprint Overall Completion (%)

These metrics quantify how effectively teams are able to deliver on their sprint goals. Inability to meet two weekly sprint goals, makes forecasting over longer periods very difficult. These metrics are therefore critical to forecasting accuracy.

Table showing a typical set of metrics used to determine an Agile delivery risk profile (continued)

Metric Area Metrics Relevance

Engineering excellence

Development process• Pull request age• Unreviewed pull requests• Commits per day

Important metrics to understand the efficiency of the review process – and track the velocity of commits.

Code quality• Productive lines• Cognitive Complexity• Code smells• Unresolved bugs • First time pass rate

These are a summary list of the metrics that we believe give a rounded view of code quality.

Security and Risk• Code ownership• Code review distribution• Commits without a pull request• Vulnerabilities

Infosec is always a primary concern, so these metrics form an important part of Agile governance. We focus particularly on process-based metrics that can be tracked to improve process and reduce risk (e.g. the elimination of commits without a pull request)

Talent Available resource• Active engineers versus plan

Quant. Engineer feedback

• Team morale• Sprint effectiveness • Quality of business sponsor input

Clearly key – shows whether we have the planned resource in place to deliver the work.

A metrics platform like Plandek enables the real-time polling of engineers through collaboration hubs like Slack. This provides quant data of engineers’ views on morale and process efficiency. Such insight is also key in effective forecasting and mitigation.

Adding team feedback metrics for excellence in Agile governance

→ We believe that the talent metrics (shown in the table) are so important, that they deserve a bit more consideration.

→ The software delivery process is clearly fundamentally dependent on the people driving it. As a result, any real understanding of delivery risk must include an understanding of:

1. How engineers are feeling – are the motivated and engaged?

2. Engineers’ views of the effectiveness of the delivery process and potential risk

3. Engineers’ feedback on potential quality issues.

→ The Plandek software delivery analytics platform includes an integration with Slack to enable the real-time collection of engineer feedback. Plandek is able to gather feedback at any point in the workflow/delivery process, triggered by events (such as merge, ticket complete and sprint end).

→ So, engineers can feedback on how sprints are progressing; how a particular part of the workflow is going; how they are feeling etc. Importantly respondees are encouraged to suggest improvements and call out bottlenecks.

→ These data are collated and presented for review by the teams themselves throughout the delivery process (e.g. in Sprint retrospectives) - but also can be collated more broadly and analysed over time, to form a view of delivery risk using Plandek.

Example of engineer feedback collected via real-time Slack polling

Understanding your Agile Delivery Risk Profile→ If you are collecting and analysing the broad data sets

discussed, you can start to build a better understanding of underlying delivery risk.

→ This can be represented using delivery risk RAG reporting –where trends over time our shown for a sample (or all of the metrics) and a “red”, “amber” or “green” status assigned to the value. The statuses chosen would take account of the metric trend in the context of the project/workstream in question to form a measured view of underlying risk.

→ The metrics also point teams in the direction of suitable mitigations to manage the identified risk. A good example of this is shown below in an example Root Cause RAG Progress Report designed to give internal clients and stakeholders a clear understanding of delivery progress, risk and the actions being taken to mitigate.

→ Root Cause RAG reports are far more effective than traditional RAG progress reporting which often sheds very little light on actually why a project is behind schedule and what needs to be done to bring it back on track.

→ In contrast to a traditional RAG approach, the Root Cause RAG Report (see the example below) clearly shows:

1. Our latest delivery forecast;2. The delivery metrics that support our forecast;3. Our mitigations (based around the Logical Six levers

that drive project lateness) – e.g. the need to increase productive time by reducing time diverted to upkeep; the need to improve Flow Efficiency by addressing the blockages in the dev process (e.g. QA wait time); or the need for improved stakeholder input (as shown in the quant engineer feedback);

4. Allocated responsibilities (across the development teams and stakeholders) to deliver the identified mitigations.

→ Done well, Root Cause RAG Reports can be a really effective means of presenting our (more accurate) forecasts in a way that stakeholders can understand and therefore can be an important step in reducing lateness and bringing the technology team and the internal client much closer together.

→ As discussed however, it relies on an understanding of the metrics that actually determine project lateness and a means of collecting those metrics.

Summary

→ By its very nature, Agile needs good governance.

→ Good agile governance is rare.

→ Good Agile governance requires metrics taken from a variety of underlying tools used through the end-to-end delivery cycle, as well as strong feedback from the engineers themselves.

→ In our view the critical metric sets needed for effective delivery risk management are:

1. End-to-end software delivery process metrics – which consider the efficiency and stability of the process from pre-development through to deployment;

2. Delivery (velocity, throughput and estimation) metrics – which consider how time to value and ability to estimate (and therefore forecast time to value) is changing;

3. Engineering excellence metrics – which consider code quality and related infosec metrics representing another facet of Agile delivery risk;

4. Talent metrics – which are collected from live polling of engineers to understand not only their morale, but also their view of process effectiveness. Software development is clearly all about the people involved, so we believe that this source of insight is critical to really understanding delivery risk.

Summary

→ These data and related metrics are very hard to collect (and review over time) unless you have access to an analytics and BI platform such as Plandek.

→ Once you have a view of these metrics you can start to build a Delivery Risk Profile and present the profile in RAG reports.

→ You can build Root Cause RAG Progress Reports to communicate with internal clients and stakeholders – to give them a much clearer picture of delivery risk and project progress (including mitigations to manage the identified risks).

Example Root Cause RAG Progress Report

Summary Commentary Overall velocity is acceptable

Delivery forecast changed

Scoping and estimation of effort Scope changed Est 1500 workdays added (described by functional area)

Tech miss-estimation Est 1000 workdays underestimate

Resource availability Planned workdays (IT) 5500 workdays

Actual workdays (IT) 4500 workdays (staff turnover in XX, late appointment of XX

Stakeholder resource input Rating 2/5 - ongoing issues with availability

Build efficiency % completion 80% of planned story points delivered PTD

% new features 80% of time on new features PTD

Quality impact Rework effort 500 workdays to rework returns from QA (poorly defined tickets)

% time on bug fixing 20%

Forecast MVP delivery date 31/1/19 (no delay v last reforecast)

Under/overspend £ On budget

Required Mitigations Freeze further scope change

10 new FTEs in place for XX/XX/XX

About the Author

Charlie Ponsonby

Charlie started his career as an economist working on trade policy in the developing world, before moving to Andersen Consulting in 1992. In 1996 he joined the Operating Board of Selfridges, before moving to Open interactive TV in 1999 and then Sky in 2001, until leaving to found and run Simplifydigital in 2007. Simplifydigital was heavily technology-led. It was three times in the Sunday Times Tech Track 100 and grew to become the UK’s largest broadband comparison service. It was acquired by Dixons Carphone plc in April 2016.

Charlie co-founded Plandek with Dan Lee in October 2017. Plandek as an end-to-end delivery metrics analytics and BI platform. It mines data from toolsets used by delivery teams (such as Jira, Git and CI/CD tools), to provide end-to-end delivery metrics to optimise software delivery forecasting, risk management and delivery effectiveness. Plandek is used by clients globally including Reed Elsevier, News Corporation, Autotrader.ca and Secret Escapes.

Plandek

Plandek is a UK-based Dev Tech business founded in 2017. It is a world leader in end-to-end software delivery metrics and governance, working with organisations across Europe and North America.

The Plandek analytics platform provides actionable insight into how your teams perform to help you deliver quality software earlier and more predictably. Access to the largest library of delivery metrics will help you to: derisk, forecast and report more effectively; improve delivery capability; accelerate Agile transformations; better integrate outsourced partners. Understand the full story of your end-to-end software delivery, from design and planning to integration.

For more content, drop us a message athello@plandek.com or visit our website plandek.com

What we doThe Plandek SaaS solution mines and

analyses data from key systems used by development teams and synthesises key

metrics from these disparate data sources to give unique insight across

your end-to-end delivery cycle.

Follow Plandek

www.linkedin.com/company/plandek

facebook.com/plandekuk

@_Plandek_

Plandek.com