T H E G Q M A P P R O A C H T O E N T E R P R I S E A G I L E M E T R I C S

Version 1.42

2

JOHN TANNERSVP &Enterprise Transformation Consultant

John.Tanner@leadingagile.com757.580.2078

www.linkedin.com/in/tannerjswww.facebook.com/leadingagilewww.twitter.com/leadingagile

3

JOHN TANNER20+ years in Software Development

Roles in industry as Consultant, Engineer, Architect, Product & Project Manager

Implemented Agile in All the environments:

Non-profit

Corporate

Government

D E F I N I N G T H E P R O B L E M

5

We need maximum metrics, because we just don’t know what we don’t know

We need minimal metrics, because they just don’t add value

We need some metrics, because something is better than nothing

THREE VIEWS ON METRICS

6

“Metrics are how we measure success. So let’s capture metrics for everything.”

Metrics dashboards become prolific, and someone has to maintain them

Each Team winds up with a unique set of metrics to capture, but with no common expectation for using them

The organization spends more time creating metrics roll-ups than they do creating product

There is no cohesive strategy for tying the data together to make targeted improvements

WE NEED ALL OF THE METRICS!

7

“Metrics do not add value, so they are waste. Let’s eliminate them altogether.”

Assumes all teams are currently producing working, tested product at high levels of efficiency

Only works when the organization doesn’t care if they ever improve the system…

… doesn’t care if they aren’t producing value

… doesn’t get hung up on details like making payroll

WE NEED NONE OF THE METRICS!

8

“Metrics might help us improve, so we need measures of some sort.”

Without clarity, we often capture metrics for metrics’ sake

When we gather metrics without clear expectations around their use they can be wasteful, or even harmful

We focus on metrics in a way that ultimately drives “local optimization” at the expense of the overall system

We forget that the “why” and the “how” matters

WE NEED SOME METRICS…

T H E “ W H Y ” A N D T H E “ H O W ”

10

LET’S TALK ABOUT WHY, FIRST…

BACKLOGS TEAMS WORKING TESTED PRODUCT

11

In Agile, we feel pretty confident we know how to measure success because…

OUR APPROACH TO THE PROBLEM

... is Making and Meeting Commitments

... is Producing Quality Software

The Team ... has a Stable Velocity

12

OUR APPROACH TO THE PROBLEM

MAKES AND MEETS

COMMITMENTS

HAS STABLE VELOCITY

PRODUCES QUALITY SOFTWARE

METRIC:Story Point

Completion %

METRIC:Velocity Variance

METRIC:Escaped Defect Count

13

Yay! We did it team!

No, but seriously, many people think like that…

PROBLEM SOLVED?

14

HOW DOES THAT WORK AT SCALE?

PROGRAM TEAMS

PORTFOLIO TEAMS

DELIVERY TEAMS

G O A L - Q U E S T I O N -M E T R I C

16

The Goal-Question-Metric approach is a simple model where we approach measures from three distinct levels:

Conceptual level (Goal)

Operational level (Question)

Quantitative level (Metric)

WHAT IS GQM?

17

Conceptual level (Goal)

A goal is defined for a team or system, to meet a specific purpose, with a focus on demonstrating a meaningful outcome for the business.

WHAT IS GQM?

18

Operational level (Question)

A set of questions is crafted to define an objective model for assessing the achievement of our goals.

WHAT IS GQM?

19

Quantitative level (Metric)

A set of metrics, based upon the objective model, is associated with every question in order to answer it in a measurable way.

WHAT IS GQM?

20

Identify the right goals for a team or system

Clarify our goals by asking the right questions

Identify measures and metrics that can answer the questions

Find ways to collect the data necessary to realize our measures and metrics

Analyze the data to assess whether we are answering our questions and achieving our goals

Use what we discover to improve our goals, questions, and metrics

HOW DO WE USE GQM?

21

EXAMPLE: MY GOAL FOR TODAY

Goal

Myaudiencedoesn'tfallasleepinthenexthalf

hour

Question

Didtheyevenshowupinthefirstplace?

Havetheybeenthoroughlycaffeinated?

Aretheypayingattentionsofar?

Metric

EmptySeatvsSoldSeat(ActualvsExpected)

SnoretoSilenceRatio(SnoreComplete/SilenceCommitted)

EyeContactVariance

HecklingDensity

R E D E F I N I N G T H E P R O B L E M

23

In Agile, we feel pretty confident we know how to measure success because…

REDEFINING THE PROBLEM

... is meeting Commitments

... is Producing Quality Software

The Team ... has a Stable Velocity

24

InEnterpriseAgile,wefeelprettyconfidentweknowhowtomeasuresuccessbecause…

Let’s figure it out, GQM style

REDEFINING THE PROBLEM

25

Delivery Team Health (Scrum)

Technical Quality

Program Health

Product Quality

Portfolio Health

Portfolio Financials

GQM AREAS TO DISCUSS

G Q M A T T H E D E L I V E R Y T I E R

27

Teamscanplan,coordinate,anddeliverpredictably

enoughtomeetareleaselevelcommitment

Hastheteamestablishedastablevelocity?

Doestheteamhavetheresourcesitneedstobesuccessful?

Doestheteamdeliverthecommittedfunctionalityeachsprint?

StoryPointCompletion%(SPsComplete/SPsCommitted)

VelocityVariance

TeamStabilityIndex

Isanythingpreventingtheteamfromdeliveringoncommitments? Blockers

Doestheteamdeliverthroughoutthesprintoratthe“lastminute”?

SprintCompletionRatio(%SPsAcceptedbySprintEnd- 2)

Goal MetricQuestion

UserStoryCompletion%(USsComplete/USsCommitted)

Towhatdegreeistheteamencapsulated(i.e.make/meetcommitmentsonitsown)?

%UserStoriesw/outsidedependencies

DELIVERY TEAM HEALTH (SCRUM)

28

Teamscanfrequentlydeliverworking,tested

softwareofahighlevelofquality

Istheteamabletomaintaintheoveralltechnicalhealthoftheproduct?

Canthequalityoftheproductbeingshippedbeverified? OpenDefects

TechnicalDebt

Aretheteamsabletoaddresstechnicalproblemsinatimelymanner? DefectAging

Aretechnicalproblemsgettingpastourqualitychecks? EscapedDefects

Aretechnicalproblemsbeingdiscoveredinproduction? LatentDefects

Goal Question Metric

TECHNICAL QUALITY

G Q M A T T H E P R O G R A M T I E R

30

ProgramManagementcanmakedeliverycommitmentsandsuccessfullymanage

incomingrequests

IstheProgrameffectivelymanagingorchestration/mitigatingdependencies?

Istheprogramdeliveringcommittedfunctionalityeachrelease?

Istheroadmapoffeaturessufficienttoeffectivelymaintainareadybacklog? FeatureRoadmapVisibility

%ReadyBacklogUserStoriesw/opendependencies

FeatureCompletionRatio

HowlongdoesittakeforanOpportunitytomovefromIdeatoDelivery? FeatureLeadTime

Goal MetricQuestion

IstheReadyBacklogsufficientlygroomedtobepredictablyconsumedbytheDT? %ReadyBacklog

IstheProgram(POTeam)focusedondeliveringvalue? %ValueinSprint

PROGRAM HEALTH

31

Goal Question Metric

Theorganizationdeliversanintuitiveexperienceand

productfeaturesthatdelightourcustomersandproducethehighestbusinessvalue

possible

Didthedeliveredsolutionmeetthebusinessneed?

Product/MarketFit(ActualvsExpected)

DoestheOverallProductDelighttheCustomer? NetPromoterScore

DidtheFeatureDelighttheCustomer? PirateMetrics(Acquisition,Activation,

Retention,Revenue,Referral)

PRODUCT QUALITY

G Q M A T T H E P O R T F O L I O T I E R

33

PORTFOLIO HEALTH

33

Theorganizationdeliversproductfeaturesand

capabilitieswiththebestabilitytodeliverearlyROI

Howlongdoesittakeforanopportunitytomovefrom“goodidea”to”done”?

Isthesystemdeliveringconsistently?

EpicLeadTime

DueDatePerformance

Areweinvestingintherighttypeofwork? PortfolioInvestmentMix

Arewefocusingonlyonthemostimportantthings? ValueStreamWIP

AbandonedWorkArewestartingontoomanythings?

Goal Question Metric

34

PORTFOLIO FINANCIALS

34

Goal Metric

ThePortfolioismeetingitsstrategic,financialobligations

andobjectives

Arewegettingtherightreturnonourinvestments?

Arewespendingtoomuchjusttokeepthelightson?

ValuetoCostRatio

CAPEXtoOPEXRatio

ArewegettingtheexpectedresultsfromthisPortfolio?

VariancetoResults($or%MarketShare)

Question

Arewemeetingourtargetedspendrates? VariancetoSpendTarget

Arewehittingourtargetedvalueandmaintenanceblend?

VariancetoValue/MaintenanceTarget

($or%Spend)

R E D E F I N I N G T H E P R O B L E M ( S O L V E D )

36

The Portfolio

... consistently delivers product features & capabilities with a focus on time to market and early return on investment (Portfolio Health)

... consistently meets its strategic financial objects (Portfolio Financials)

The Program

... can make delivery commitments and successfully manage incoming requests (Program Health)

... can ensure the organization delivers the most effective experience and product features to delight our customers and produce the highest business value possible (Product Quality)

The Delivery Teams

... can plan, coordinate, and deliver predictably, in order to meet a release commitments (Team Health)

... can frequently deliver working tested code of a high level of quality (Technical Quality)

MEASURE SUCCESS-ENTERPRISE LEVEL

S O N O W W H A T ?

38

Identify the right goals for a team or system

Clarify our goals by asking the right questions

Identify measures and metrics that can answer the questions

Find ways to collect the data necessary to realize our metrics and measures

Analyze the data to assess whether we are answering our questions and achieving our goals

Use what we discover to improve our goals, questions, and metrics

NEXT STEPS

F I N A L T H O U G H T S