awb - 08 - agile metrics

273 Agile White Book – AXA Emerging Markets EMEA-LATAM

Chapter 8 AGILE METRICS

V1.0


Contents

DIAGRAM ........................................................................................................................................................................................................ 276 THE METRICS DILEMMA ................................................................................................................................................................................... 277 PRINCIPLES OF AGILE METRICS ........................................................................................................................................................................... 279

THE METRICS ONION ............................................................................................................................................................ 281 SCOPE DEFINITION ........................................................................................................................................................................................... 282

CREATE METRICS AT TEAM LEVEL ........................................................................................................................................... 282

CREATE METRICS AT PRODUCT LEVEL ...................................................................................................................................... 286

GET BALANCED METRICS ....................................................................................................................................................... 289 HOW TO COLLECT INDICATORS ......................................................................................................................................................................... 290

MAKE THE BEST POSSIBLE DECISIONS ...................................................................................................................................... 291

GET THE RIGHT TOOLS TO MEASURE ........................................................................................................................................ 292 THE EXPECTED OUTCOME ................................................................................................................................................................................ 294 TAKE AWAY .................................................................................................................................................................................................. 295 CHECKLIST 8.1 .................................................................................................................................................................................................. 296


Agile metrics

Agile metrics focus on getting an understandable

set of indicators to support a clear business goal

while reinforcing a positive feedback loop towards

people´s behaviours and company processes.

The primary metric in Agile is piece of working

software that fulfils the client expectations.


Diagram

AGILE METRICS

I know when to use metrics.

I learnt some typical metrics in Agile.

BENEFITS

OF USING

METRICS

LEARN THE

METRICS

PRINCIPLES

I know the techniques to gather indicators.

KNOW HOW TO

CREATE METRICS

- At Team level

- At product level

- Balanced indicators

LEARN HOW TO

COLLECT

INDICATORS


People generally measure something with

the purpose of:

Making sure they are on track.

Keeping impediments and risks

under control.

Checking that the current system is

not deteriorating.

THE METRICS dilemma

Metrics are generally used to make better decisions, and in that way, people tend to add new

indicators when they can´t figure out an answer. Following that way of thinking, more metrics

mean wiser decisions. The feeling that not measuring something leads to deterioration and lack

of insights is implicit in there.

While these common-sense assumptions are also valid in Agile, we place a different focus in

order to make sure that we avoid some problems.

These are some of the issues we try to avoid:

Measuring things out of context.

Evaluating indicators that are the result of an effect instead of analysing its root-

cause.

Inciting inadequate behaviours or putting pressure on people.

Making too many assumptions with little information or no facts.

Analysing out-of-date data or not fully understanding the scope of the values presented.


Always use the rule of the “simplest set of

measures that can possibly work”, where no

other metric can satisfy the need.

THE METRICS dilemma

These five issues can add instability into the system and create a vicious circle that re-enforces the

need for new indicators. In order to break that, you need to make sure that all indicators

provide a clear value while keeping people away from adding processes which don’t

support value creation or continuous improvement.


PRINCIPLES of Agile metrics

The first objective of Agile metrics is to enable people to make better decisions. As Agile is

based on the fact that change is welcomed, such values should be adopted as a reflexion of the

reality instead of being an out-dated picture.

There are 3 areas that Agile focuses on:

1. Understanding why we need an indicator, which goal it supports and its benefits.

It should be clear for people what a specific indicator means and which assumptions

were made in order to elaborate it.

2. Supporting the continuous improvement.

They should support a continuous improvement process to help adapt practices and be a

topic of discussion during Retrospectives.

3. Reinforcing positive behaviours.

They should always reinforce positive behaviours in order not to create attitudes that

feed the wrong values into the system (i.e. “make up” some numbers to create a better

but unreal perception).



As you can see, adding an incorrect metric or in the wrong way can seriously impact Teams

and processes. For example, creating a metric that measures how many lines of code each

person writes per week could end up with disastrous effects.



THE METRICS ONION

We use the Metrics Onion to conceptualise all metrics as a system composed by interrelated

parts that cooperate with processes instead of being just an isolated group of indicators. The onion

has five-stages that help emphasize the right attributes at the time of defining the new metric.

An indicator should satisfy the metrics principles, have a defined and clear Scope, easily

identify a need, get an answer about how and what it can improve in the system and finally

establish the best way to get the data.

As a characteristic, good metrics always affirm and reinforce Agile Principles and are an

outcome from a Retrospective.


SCOPE definition The first thing to be defined is the Scope for a particular indicator. There can be many areas to

measure but, in this chapter, we will group all of them into two main groups:

1. Related to the Team

2. The ones that make more business sense (product level).

CREATE METRICS AT TEAM LEVEL

When establishing metrics at Team level, it is important to have in mind the processes impacted

and to be improved. They are a main tool to help the Team learn and adapt their practices.

We, Agile Teams, typically need metrics to answer questions like these:

How much value have we delivered to business?

How much value can the Team deliver within the remaining time?

How much work can we take?

Are we on track to deliver what we forecasted for the Sprint?


SCOPE definition

The team Agile Star assessment can help analyse different aspects of the Team’s “health”.

This analysis can be also more specific and target different roles as follows:

Development Team Product Owner


SCOPE definition

There is also a second type of metrics that Team creates to make sure Business understands

the current situation and the way they work. These are some of the indicators we generally

recommend (but not limited to):

Benefit

Expected result

Metric used

Client Expectation

Target tools and interactions to

improve people competences

(Skills, way they work, etc.).

Defects reported, Defects measure

Efficiency Smaller user stories and better

Acceptance User Criteria

User Stories started but not finished

Predictability Reduced variability, better planning for Potential Product

Increments

Team velocity vs. Capacity measure

Product experience

Better client experience, Increased Quality

Support calls, defects measure

Stable product Better cadence, less pressure on

team, better cross-functional team

% of tests automated, % test in Continuous Integration,

skills learnt and shared measure, Unit test coverage

Time to market

Increase Business Value, Reduce cost per feature

No. architectural refactors, Technical Debt Scoring measure

Value Delivery Focus on techniques to increase

requests (i.e. eXtreme

Programming techniques)

Requests per Sprint, Requests Resolved per Sprint measure

Effort Better time to market, improved planning, less variability on

estimates

Extra work discover during the Sprint measure (i.e. technical debt, documentation, database rollback scripts,

system testing, etc.)

Business Alignment

Increase team understanding of goals and vision, better visibility

and improved communication

with P.O.

% goal achieved, % User Stories accepted by product Owner, Product Mapping Changes metric.


SCOPE definition

You should end up seeing some charts like the following ones:

Never forget that Business Value can be impacted by a large number of factors and any

approach to be taken has to be analysed in the context of the whole ecosystem.

Throughput

Average Lead Time


Typically people from business in an Agile environment

need metrics to answer questions like these:

What is this really measuring?

Is this supporting our goal?

Who is the metric for?

Who should see it?

What behaviours risks will this drive?

Are we measuring at the right level?

SCOPE definition

CREATE METRICS AT PRODUCT LEVEL

At business level, you still keep the focus on the continuous improvement but will need a more

comprehensive set of Key Progress indicators; they give everyone an idea of internal and

external progress and help align expectations and determine business outcomes.

You should see something like this:

Business Value Points

Contracts


SCOPE definition

This is a list of different indicators we recommend and see in the Agile world:

Goal/Business

Reason

Benefit

Expected result

Metric Used

I want to sell more cars… I want better client retention I want to introduce new products and be faster in the market

Agility

Continuous improvement in team and program measures

Team/company assessment, release predictability

measure

Value Delivery Target right features Accumulated ROI measure,

Features Scoring measure (value points)

Client Expectations More satisfaction and understanding of client needs

Features and level of satisfaction measure, % goal

achieved measure

Customer Satisfaction Improve customer score and better relationship

Client score Survey

Efficiency Reduce average requirement cycle

time

Reduce average requirement

cycle time

Predictability Inspect and adapt & learn more about the client behaviour

Project Deviation measure

Productivity Better retention, more innovation Motivation measure

Time to Market More frequent releases Number of releases per period of time

Innovation Lead innovation (really low scores per

release means low innovation)

Features Scoring measure

(value points)


SCOPE definition

And never forget that indicators should always support a strategic plan and goal. A variation in

an indicator is never isolated but a part of a whole. Business metrics should also follow the

metrics principles.

Remember that the most important concept in Agile is creating as much Business Value as

possible; therefore most of the indicators pivot around this idea. Agile Metrics are used to deliver

better software and help people know where they are and what else they need.


SCOPE definition

GET BALANCED METRICS

Having a balanced set of metrics is important so that actions to achieve/improve do not

impact other metrics. Note that when you choose to measure a person or team in a certain

way, their actions can be conditioned in excess to meet that goal and also neglect other

important aspects such as quality, costs, risks, sustainability, etc.

The following are some possible metrics grouped in order to help to create a balanced scorecard:

Customer Expectations

User satisfaction Value or cumulative ROI

Cost Incurred cost and deviation Budget Etc.

Productivity

Lead Time (from Idea to Market)

Release cadence Completed requirements Flow efficiency (value time or points or

requirements/ elapsed time)

External Quality

Hours corrective vs. Total

Total of defects in production Defect density (by US, points, KLOC,

etc.)

Internal Quality

Product Complexity Test coverage

Team

Teamwork Motivation

Agile Team Maturity Agile Management Maturity Attrition rate


HOW TO COLLECT indicators

We are always looking for metrics that are easy to collect and don't become an impediment to

the people gathering them or need to spend more time than the process itself.

These are some of my suggestions:

Measure business outcomes instead of activity (e.g. value delivered vs. number of

tasks completed)

Look at trends, not just a single point in time.

Encourage whole team results over individual results.

Think of how the measurement will improve the way to do things.

Evaluate the impact in the company when adding new indicators.

Remember that an Agile metric is a real opportunity to give visibility, establish a

conversation and start a new improvement initiative.

Find below some advice you can follow when trying to measure things in Agile:

Objectively evaluate the outcome of the project/service to make wiser decisions.

Identify trends, to act on them and see if the actions produce an improvement and a

sustained impact.

You can introduce a new metric at a given time if it is needed just for specific

situations, but remove it once the issue is fixed.

Have a balanced set of metrics, so that actions to achieve/improve a metric do not

worsen the results in others.

Establish a small set of metrics, so that everyone knows their current value, context

and purpose, and are non-intrusive.

Perform cause-effect analysis of the metrics´ results and their relationships, so that

the actions taken truly help solve the problem.



MAKE THE BEST POSSIBLE DECISIONS

In Agile, we try to diminish assumptions as they produce a false sense of safety and

increase uncertainty/risk. For that reason, some indicators are not created to get a closer look

at reality but to validate an emerging hypothesis. Many Product Backlog Items are treated

as assumptions (or hypotheses) that must be validated by rapid experimentation in the

marketplace. The aim is to eliminate wasteful practices, increase value and diminish risk.

This process is generally performed during the Need stage in the Agile Onion. In these cases,

temporary indicators are created in order to make faster and wiser business decisions while

keeping assumptions and impact low.



GET THE RIGHT TOOLS TO MEASURE

Agile adds a broad number of tools that can be quantified and many relative metrics that can

help track and understand how much Business Value we deliver to our clients or company.

With this data we shape the whole vision.

These are some techniques that work for me in order to understand the ecosystem:

- Know how important a feature is for my client by using Business Story Points.

- Know how much a feature is worth for my client by using Kano Survey Model

(excitement, performance, basic needs).

- Establish how important a requirement is by applying MoSCow (Must, Should, Could,

Won’t).

- Know how motivated a Team is using Niko Niko Mood Tracking (Body metrics, mood

metrics, etc.), etc.

For example, you would like to know the amount of discovered work during a Sprint using Story

Points as it can affect your estimates, people’s morale, etc.



An additional example can be Business Value Points; BVP is a unit that allows companies to

measure what the Business Value of an expected set of features is. This scoring is typically

done by Stakeholders/Product Owner when they evaluate requirements. As a general rule, all

Stakeholders should be aware of the practice and participate in defining what is more valuable to

the client, without delegating the whole responsibility to Product Owner.

We suggest using values of 25, 50, 100 and 200 (higher number = higher importance) to score

it. You can think of this value as one or a mix of the followings:

Client desire (the more clients ask for it, the higher the number)

Role petition based scoring (CEO 200 points, Regional Manager 50, etc.)

Client opinions (Kano Model or any other kind of survey)

Revenue of the feature (the more ROI the higher)

Etc.

Once BVP is in place, the company knows how much Business Value is achieved in each Sprint,

they can detect a lack of innovation, etc.

Open the checklist “Creating Metrics” to see how to create them step-by-step.


THE EXPECTED outcome

After using some of the techniques learnt, you should expect to get a metric that:

Is understood by people and the goal it supports.

Have clear stated benefits.

Is easy to get.

Does not impact negatively in other indicators.

Does not have many assumptions (or a plan to diminish them).

Clearly identifies which positive behaviours it reinforces.


TAKE AWAY

REMEMBER

Everyone should fully understand the units to be used and indicators in order to be balanced.

Don’t forget to extend the Team involvement in the agile process and always strengthen the Agile Principles.

It is worth teaching Agile Product Management techniques to the Product Owner.

Encourage automation to reduce variability and measure better.

Refine metrics in order to identify if the business benefits of the continuous initiative are being achieved.

ADDITIONAL RESOURCES

Business Value Game

Measuring Success, Measuring Value

Moscow Method

Kano Model

Business Value Game

Optimising Value

BENEFITS

Helps get insight on the changes needed.

Keeps stakeholders informed and involved in a meaningful manner.

Helps redirecting priorities in a continuous way.

Builds a shared understanding and acceptance of what is measured.

Establishes a discussion to make better decisions.

http://www.agile42.com/en/agile-coaching-company/agile-scrum-tools/business-value-game/

https://www.scrum.org/Blog/ArtMID/1765/ArticleID/11/Measuring-Success-Measuring-Value

http://en.wikipedia.org/wiki/MoSCoW_method

http://en.wikipedia.org/wiki/Kano_model

http://www.agile42.com/en/agile-coaching-company/agile-scrum-tools/business-value-game/

http://www.alnhouston.org/documents/Summits/LeaderSummit-ValuePresentation-Reed-20110610.pdf


Creating Metrics

Checklist 8.1

Version 1.0

DATE: __________

Audience and Context

The checklist will be useful for anybody who thinks of establishing adequate metrics to measure

the efficiency and quality of the development team work.


__________________________

Task

Comments

Metrics Creation

We made sure that the metric enables people to make better decisions/better software (we performed a cause-effect analysis if needed).

There was no other metric to serve the purpose.

We checked and know a clear answer of why we need it.

We clearly know its benefits.

We made sure that the metric supports continuous improvement.

We made sure that the metric supports positive behaviour.

It does not negatively impact other indicators.

Everyone understood the scope of this metric.

We diminished assumptions when creating the metric.

We clearly knew how to get it.

awb - 08 - agile metrics

Leadership & Management