agile release planning, metrics, and retrospectives

53
MB Fullday Tutorial 6/3/2013 8:30 AM "Agile Release Planning, Metrics, and Retrospectives" Presented by: Michael Mah QSM Associates, Inc. Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 8882688770 9042780524 [email protected] www.sqe.com

Upload: techwellpresentations

Post on 10-May-2015

202 views

Category:

Technology


2 download

DESCRIPTION

How do you compare the productivity and quality you achieve with agile practices with that of traditional waterfall projects? Join Michael Mah to learn about both agile and waterfall metrics and how these metrics behave in real projects. Learn how to use your own data to move from sketches on a whiteboard to create agile project trends on productivity, time-to-market, and defect rates. Using recent, real-world case studies, Michael offers a practical, expert view of agile measurement, showing you these metrics in action on retrospectives and release estimation and planning. In hands-on exercises, learn how to replicate these techniques to make your own comparisons for time, cost, and quality. Working in pairs, calculate productivity metrics using the templates Michael employs in his consulting practice. You can leverage these new metrics to make the case for changing to more agile practices and creating realistic project commitments in your organization. Take back new ways for communicating to key decision makers the value of implementing agile development practices.

TRANSCRIPT

Page 1: Agile Release Planning, Metrics, and Retrospectives

 

 

MB Full‐day Tutorial 6/3/2013 8:30 AM 

       

"Agile Release Planning, Metrics, and Retrospectives"

   

Presented by:

Michael Mah QSM Associates, Inc.

         

Brought to you by:  

  

340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com

Page 2: Agile Release Planning, Metrics, and Retrospectives

Michael Mah QSM Associates, Inc.

With twenty-five years of industry experience Michael Mah teaches, writes, and consults for QSM Associates to tech companies on measuring and estimating software projects for offshore, waterfall, and agile. Michael and his QSM partners have researched thousands of projects worldwide. His work examines time-pressure dynamics of teams and their contribution to project success and failure. Michael’s clients include Boeing, Progressive, Verizon Wireless, Nationwide, JPMorganChase, Roche, and other Fortune 100 companies. He is the director of the Benchmarking Practice at the Cutter Consortium in the US. A private pilot, Michael lives in the mountains of western Massachusetts. qsma.com.

 

Page 3: Agile Release Planning, Metrics, and Retrospectives

1

Agile Release Planning, Metrics, and Retrospectives - Workshop Slides

Better Software Conference West Las Vegas, Nevada

June 2013Michael Mah

Managing PartnerQSM Associates, Inc.

3/26/2013

QSM Associates, Inc.75 South Church Street

Pittsfield, MA 01201413-499-0988

Fax 413-447-7322e-mail: [email protected]

Website: www.qsma.comBlog: www.optimalfriction.com

Michael Mah is the director of the Benchmarking Practice at the Cutter Consortium, a Boston-based IT think-tank, and served as past editor of the IT Metrics Strategies

Michael Mah

Boston based IT think tank, and served as past editor of the IT Metrics Strategies publication. He is also managing partner at QSM Associates Inc. based in Massachusetts USA.Michael teaches, writes, and consults to technology companies on measuring, estimating and managing software projects, whether in-house, offshore, waterfall, or agile. With over 25 years of experience, Michael and his partners have derived productivity patterns for thousands of software projects collected worldwide across engineering and business applications. His current work examines time-pressure dynamics of teams, and its role in project success and failure. In addition to his background in physics and electrical engineering, he is a mediator

Copyright QSM Associates, Inc. 3/26/2013 2

specializing in dispute resolution for technology projects. He has a degree in electrics engineering from Tufts University. His training on dispute resolution, mediation, and participatory processes is from the Program on Negotiation at Harvard Law School and the Radcliffe Institute for Advanced Study.He can be reached at [email protected].

Page 4: Agile Release Planning, Metrics, and Retrospectives

2

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we h t l

Manifesto for Agile Software Development

have come to value:Individuals and interactions over processes and tools working software over comprehensive documentation Customer collaboration over contract negotiation responding to change over following a plan That is, while there is value in the items on the right, we

Copyright QSM Associates, Inc. 3/26/2013 3

value the items on the left more.

© 2001 Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, Dave Thomas, Martin Fowler

“Without metrics,you’re just another personwith a different opinion.”

Copyright QSM Associates, Inc. 3/26/2013 4

Page 5: Agile Release Planning, Metrics, and Retrospectives

3

“Frothy eloquence neither convinces nor satisfies me. I am from Missouri. You have got to show me.”

- Missouri Congressman Willard Duncan Vandiver, 1899

Copyright QSM Associates, Inc. 3/26/2013 5

Missouri Congressman Willard Duncan Vandiver, 1899

“Metrics” Is Not a Dirty Waterfall Word…Metrics (or measures) can be as light or as heavy as you want them to be. Think of them as – “Information”Familiar examples of measures that inform us:

Stock Market IndexesBlood Tests (i.e. cholesterol)Newborn “Apgar Score”Astronomy – Distance (i.e. Light-Years, Astronomical Units (AUs)…) Others…

In technology we often want to know information about

Copyright QSM Associates, Inc. 3/26/2013 6

In technology, we often want to know information about what works and what doesn’t work, whether we’re “Better, Faster, Cheaper” or if we’re “more productive.”

Page 6: Agile Release Planning, Metrics, and Retrospectives

4

Measures that Inform On… Time (Faster) - Project Duration, perhaps by phase.Cost (Cheaper) - Effort (including labor rates)( p ) ( g )Quality (Better) – Bugs/Defects, User SatisfactionProject Scope – Features, Requirements, Use Cases, Stories etc.

One way of capturing this information is bydrawing a picture

Copyright QSM Associates, Inc. 3/26/2013 7

drawing a picture…

An Example of a Waterfall Picture

Copyright QSM Associates, Inc. 3/26/2013 8

Page 7: Agile Release Planning, Metrics, and Retrospectives

5

An Example of an Agile Picture

Copyright QSM Associates, Inc. 3/26/2013 9

Entering Metrics from Whiteboard into QSM SLIM Model

Copyright QSM Associates, Inc. 3/26/2013 10

Page 8: Agile Release Planning, Metrics, and Retrospectives

6

Project Interviews

Copyright QSM Associates, Inc. 3/26/2013 11

Short Exercise – Observations of the Two

What similarities do you observe? With regard to the What similarities do you observe? With regard to the following

Phases?Time?Staffing?Project Scope/Size?Defects?

Copyright QSM Associates, Inc. 3/26/2013 12

What differences do you observe?

Page 9: Agile Release Planning, Metrics, and Retrospectives

7

An Example of an Agile Picture

Copyright QSM Associates, Inc. 3/26/2013 13

An Example of a Waterfall Picture

Copyright QSM Associates, Inc. 3/26/2013 14

Page 10: Agile Release Planning, Metrics, and Retrospectives

8

An Example of a Waterfall Picture

Copyright QSM Associates, Inc. 3/26/2013 15

Exercise #1

Metrics Capture: Agile XP Release(Part 1)

Copyright QSM Associates, Inc. 3/26/2013 16

( )

Page 11: Agile Release Planning, Metrics, and Retrospectives

9

You will learn how to: (Part 1)“Translate” a description of a project into a

Objectives of this Exercise

p p jwhiteboard staffing sketch,illustrating the major phases of the work over time, and the staffing used for both the story phase and the build (iterations) phase. (Project interview)Extract the key information measures from the sketch (Metrics capture)

Copyright QSM Associates, Inc. 3/26/2013 17

sketch. (Metrics capture)

Exercise #1

Metrics Capture: Agile XP Release(Part 2)

Copyright QSM Associates, Inc. 3/26/2013 18

( )

Page 12: Agile Release Planning, Metrics, and Retrospectives

10

You will learn how to: (Part 2)Open a data collection template (SLIM-Datamanager)

Objectives of this Exercise

p p ( g )and use it to record this information into a file.Save the file.

Copyright QSM Associates, Inc. 3/26/2013 19

Learn to Measure (and Deal Explicitly with) Project Size

3/26/2013

Page 13: Agile Release Planning, Metrics, and Retrospectives

11

Learn to Measure Size

M l t d i j t i t f P Many people today size projects in terms of Person HoursThat’s not sizeThat’s effortSome also size a project as being “25 People”That’s not size

Copyright QSM Associates, Inc. 3/26/2013 21

That’s staffing (headcount)

How Far to the Airport?

If t ll 40 i t h ’t d If you tell me 40 minutes, you haven’t answered my questionDistance is not measured in minutesDistance is measured in miles, or feet, or kilometers, or …

Copyright QSM Associates, Inc. 3/26/2013 22

Page 14: Agile Release Planning, Metrics, and Retrospectives

12

How Far to the Airport?

40 i t i i t i40 minutes is assuming a certainVehicleRouteTime of dayTraffic patternSpeed

Copyright QSM Associates, Inc. 3/26/2013 23

Speed…This is an estimate of time

How Far to the Airport?

If t ll 30 il h d If you tell me 30 miles, you have answered my questionI can estimate the time for a future airport trip by considering my history for any given

VehicleRoute

Copyright QSM Associates, Inc. 3/26/2013 24

Time of dayTraffic pattern…

Page 15: Agile Release Planning, Metrics, and Retrospectives

13

Size CategoriesCreated fromScratch(adds work;

NewModifiedAdapted

(adds work;impacts Size)

(adds work;impacts Size)

Copyright QSM Associates, Inc. 3/26/2013 25

Unmodified Purchased / Reused(adds complexity;impacts Productivity)

What Do We Mean By Effective Size?Created fromScratch(adds work;

New20 Units

Modified50 UnitsAdapted

(adds work;impacts Size)

(adds work;impacts Size)

Copyright QSM Associates, Inc. 3/26/2013 26

70 UnitsNew ModifieEffectiv deS S S= + =

Page 16: Agile Release Planning, Metrics, and Retrospectives

14

Usefulness of Various Size UnitsThere are many different units people use to size

ftsoftwareThey are all related to what must be created, but at different levels of abstractionEach can be useful depending on where you

Copyright QSM Associates, Inc. 3/26/2013 27

depending on where you are in the software development lifecycle

What Do We Mean By Size?

Units of Need Units of WorkBusiness Concern:Business Concern: Value, PriceValue, PriceFocus:Focus: Bang for the BuckBang for the BuckSize Measures:Size Measures:

RequirementsRequirementsFunction PointsFunction PointsUse CasesUse CasesStories/Story PointsStories/Story PointsFeaturesFeatures

Business Concern:Business Concern: CostCostFocus:Focus: ProductivityProductivitySize Measures:Size Measures:

Lines of CodeLines of CodeStatementsStatementsProgram ActionsProgram ActionsModules, ObjectsModules, Objects

Development

U ts o o

Copyright QSM Associates, Inc. 3/26/2013 28

Development

ProcessProductNeed

Page 17: Agile Release Planning, Metrics, and Retrospectives

15

Dividing Units of NeedIt may be helpful to divide the Units of Need

into:into:SimpleMediumComplex

Copyright QSM Associates, Inc. 3/26/2013 29

Getting “Gearing Factors”Once you know what the Units of Need and Units of Work

are, you ask:How large is a simple one?How large is a medium one?How large is a large one?How many small, medium, and large ones will there be?

This gives you Gearing Factors

Copyright QSM Associates, Inc. 3/26/2013 30

Page 18: Agile Release Planning, Metrics, and Retrospectives

16

Gearing FactorIt is valuable to know how many Units of Work are typically associated with a given Unit of Need.This is the “Gearing Factor“ - It can be calculated from completed projects.It can be thought of as a “currency conversion,” informing us about how much software (Units of Work) it took to implement a feature, a story, or a requirement. Examples:

200 lines of code (source instructions) in a C++ Object.3 Objects to implement a simple feature. 6 Objects to implement a complex feature

Copyright QSM Associates, Inc. 3/26/2013 31

implement a complex feature.10 Stories in an agile Iteration.Others…

If desired, we can tally a total Units of Need and Units of Work in a spreadsheet.

# Component NameMost Likely g

Most Likely Expected

Code, Instructions, or Implementation Units (IUs) per Component Number of Components

in the “Cart”

Sizing Example – A Component “Shopping Cart”

# Component Name Likely g Likely Expected1 Simple Foundation Table 5 11 552 Average Foundation Table 15 25 3753 Complex Foundation Table 20 9 1804 Gaps Simple (PeopleSoft Customizations) 34 11 3743 Gaps Average (PeopleSoft Customizations) 66 25 16504 Gaps Complex (PeopleSoft Customizations) 345 9 31055 Business Rules 5 0 06 Data Conversion 6 2496 149769 Interface Simple 320 276 88320

Copyright QSM Associates, Inc. 3/26/2013 32

p10 Interface Average 620 127 7874011 Interface Complex 1520 21 3192013 PeopleSoft Upgrade Rework 0 0 014 Custom Report Simple 25 0 015 Custom Report Average 50 0 016 Custom Report Complex 100 47 4700

Estimated Total # of Implementation Units = 224,395

Page 19: Agile Release Planning, Metrics, and Retrospectives

17

Agile Teams Explicitly Deal with, and Measure Size…

Copyright QSM Associates, Inc. 3/26/2013 33

Example: On Sizing, Agile Teams Use… StoriesA story, or a feature, is described by the product owner. You might also describe it as a “requirement.”Not all stories are created eq al Some are smaller some are Not all stories are created equal. Some are smaller, some are larger than others.Story points are a unit of measure for expressing the overall size of a story. There is no set rule for this. It is an amalgamation of the effort, the complexity, the risk, etc. associated with a story.The range of the scale can be 1-10, 1-7 (or whatever), depending on which book you reference. Agile authors haven’t seemed to set any standard; they say that what’s important is that the numbers

Copyright QSM Associates, Inc. 3/26/2013 34

are relative. i.e. a story with 10 story points is 2x one that is scored at a 5, and if you’re using a 10 scale, then a 5 is “average.”

Page 20: Agile Release Planning, Metrics, and Retrospectives

18

It Takes a Certain Amount of Code to Produce a Story

Within a given iteration (say… 2 weeks), an agile team accomplishes the work to produce a certain number of stories, and the associated story points. The number of story points accomplished in an iteration is called “velocity.”Since we’re talking about creating “software” in a given iteration, these features/stories are manifested by programmers who create new code, and/or adapt (modify) existing code to produce the stories/points.In an iteration (and across an entire release), we can express the total amount of code that delivers these stories/points, and understand the proportional relationship between them. i.e. “It took about 14,000 lines

Copyright QSM Associates, Inc. 3/26/2013 35

p p p ,of code to produce 10 story points in this iteration, which translates to (on average) 1,400 lines of code per story point.If you suppose that a release is aiming for a total of 200 story points, it might involve 280,000 lines of new/modified code (1,400 x 200).

Exercise #2

Creating Schedule and Effort Trendlines

Copyright QSM Associates, Inc. 3/26/2013 36

Page 21: Agile Release Planning, Metrics, and Retrospectives

19

Objectives of this Exercise

You will learn how to:C t X Y G h f f j t Create an X-Y Graph for a group of projects –smaller releases on the left, larger releases on the right – for two metrics of interest: Schedule and Effort.Understand how to visually construct a quick Regression Fit through the data to determine the average trend, and the high-low.

Copyright QSM Associates, Inc. 3/26/2013 37

g , gUse this baseline as a framework to evaluate potential scenarios for a future project.

Learn to Measure (and Deal Explicitly with) Productivity

3/26/2013

Page 22: Agile Release Planning, Metrics, and Retrospectives

20

Software Development Core Metrics

Duration

EffortProducedSoftware

(Size)

How long?

How much?

Copyright QSM Associates, Inc. 3/26/2013 39

DiscoveredDefectsHow good?

How Would You Describe “Productivity Improvement?”

Producing a certain amount of functionality |or features, faster, with lower cost at the sameor higher level of quality… or

Within a given timeline, producing more functionalityor features, at lower cost, at the same or higher level of quality… or

Copyright QSM Associates, Inc. 3/26/2013 40

level of quality… or

… other variations on the theme

Page 23: Agile Release Planning, Metrics, and Retrospectives

21

Production Equation Conceptual Form

Copyright QSM Associates, Inc. 3/26/2013 41

DeliverableSize

is over Time at some ProductivityEffort

Production Equation (Rearranged) Conceptual Form

DeliverableSize

is

over

Copyright QSM Associates, Inc. 3/26/2013 42

Time

Productivity

Effort

and

Page 24: Agile Release Planning, Metrics, and Retrospectives

22

Size = 272,768 SLOC

Effort = 249 Person-MonthsWFSO 5.1 SIZE = 19.4

Production Equation (In Actual Practice)Calibration Form

Months

Time = 13 Months*PI = TIME EFFORT 19.4

Copyright QSM Associates, Inc. 3/26/2013 43

Productivity contributing elementsNobody knows how many elements effect a given environment’s ability to produce a systemTh t l t d h th dThere are at least dozens, perhaps thousandsNobody knows what the effect of their interaction is

Copyright QSM Associates, Inc. 3/26/2013 44

Page 25: Agile Release Planning, Metrics, and Retrospectives

23

Productivity typical factors

Tooling / Methods Personnel ProfilegInfrastructureToolsStandards

Technical DifficultyHardware ConstraintsAlgorithm ComplexityLogic Complexity

ManagementEnvironmentTeam Capabilities

Integration IssuesAmount of reused softwareIntegration complexity

Copyright QSM Associates, Inc. 3/26/2013 45

Logic ComplexityManagement ComplexityPlatform Stability

Integration complexityNumber of interfacesExisting documentation

Productivity Index (PI)(industry values by application type)

Business Information

Command and Control

Process Control

Scientific

System

Telecommunications

Engineering

Copyright QSM Associates, Inc. 3/26/2013 46

0 2 4 6 8 10 12 14 16 18 20 22 24

Productivity Index (PI) w/ ±1 Standard Deviation

Avionics

Microcode

Real Time

Real Time

Page 26: Agile Release Planning, Metrics, and Retrospectives

24

What’s a PI Worth?

Size = 350 Java Classes/ObjectsjBurdened Labor Rate = $120,000/PY

ProductivityIndex

Effort(PM)

Schedule(Mos)

Cost($)

18 42 9.4 420,000

MTTD(Days)

4.1

Copyright QSM Associates, Inc. 3/26/2013 47

18

17

16

42

56

78

9.4

10.5

11.6

420,000

560,000

780,000

4.1

3.5

2.8

Exercise #3

Assessing 6 Agile XP Releases

Copyright QSM Associates, Inc. 3/26/2013 48

Page 27: Agile Release Planning, Metrics, and Retrospectives

25

Objectives of this ExerciseYou will learn how to:

Look at productivity patterns for a group of Look at productivity patterns for a group of completed projects.Examine if productivity is rising or falling over time.Understand how demonstrated/accomplished schedules and effort (high – low) relate to derived productivity values (low-high).Create your own trendlines for schedule, staffing, and defects

Copyright QSM Associates, Inc. 3/26/2013 49

and defectsAssess productivity targets implied by proposed deadline-scope pairings and evaluate they are reasonable (or not) when “sanity checked” against past accomplishments.

Two Case Studies:

Co-located Agile XP and Distributed SCRUM

Copyright QSM Associates, Inc. 3/26/2013 50

Page 28: Agile Release Planning, Metrics, and Retrospectives

26

Co-Located XP Case Study — Follett Software

Team size24 Developers24 Developers7 Testers3 Customers3 Project Leaders

Code Base1,000,000 lines of code7,000 automated unit test

Copyright QSM Associates, Inc. 51

10,000 automated acceptance test

Why XP for Follett?

“XP allowed us to start building based on gcurrent assumptions”“XP approach allowed us to change directions when needed”“XP iterations gave us a “pilot project” test bed”

Copyright QSM Associates, Inc. 52

test bed“Focus on building customer value gave high visibility”

Page 29: Agile Release Planning, Metrics, and Retrospectives

27

Robert Lucas, Nobel Prize (Economics):

On Co-Location of Smart People

The force of concentration, or “clustering” of human creativity and talent … the powerful economic gains when smart and talented people locate in close proximity to one another.

“Human capital externalities”: the productivity and

Copyright QSM Associates, Inc. 3/26/2013 53

“Human capital externalities”: the productivity and innovation gains that occur when human beings cluster together.

Source: Richard FloridaFlight of the Creative Class

People ManagementXP says “XP works in small- to medium-sized t ”teams”How we evolved or extended this rule

Subteams1 large room is mandatory

Trade-offs Communication between

Copyright QSM Associates, Inc. 54

Communication between subteams1 room noise level (distractions)Lack of personal space

Page 30: Agile Release Planning, Metrics, and Retrospectives

28

Copyright QSM Associates, Inc. 3/26/2013 55

Copyright QSM Associates, Inc. 3/26/2013 56

Page 31: Agile Release Planning, Metrics, and Retrospectives

29

Copyright QSM Associates, Inc. 3/26/2013 57

Destiny Release 6.5 – Whiteboard Sketch

Copyright QSM Associates, Inc. 3/26/2013 58

Page 32: Agile Release Planning, Metrics, and Retrospectives

30

Input to SLIM-DataManager

Size Defects

Copyright QSM Associates, Inc. 3/26/2013 59

Time Effort

SLIM Replica – Destiny 6.5Staffing & Probability Analysis

Avg Staff (people)<Destiny Release 6.5>

40

50876543 21

R&DC&T

1 2 3 4 5 6 7 8 9 10 11 12Apr'05

May Jun Jul Aug Sep Oct Nov Dec Jan'06

Feb Mar Apr May Jun

0

10

20

30

40

Avg Staff (people)

Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9 - 99R 10 - 99.9R

Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9 - 99R 10 - 99.9R

Copyright QSM Associates, Inc. 3/26/2013 60

SOLUTION PANEL - <Destiny Release 6.5>

DurationEffortCost

Peak StaffMTTD

Start Date

C&T11.0400

3400.036.50.638

7/2/2005

Life Cycle12.0446

3791.036.50.638

6/1/2005

MonthsPM$ (K)peopleDays

PI=24.7 MBI=4.8 Eff SLOC=893,298

CONTROL PANEL - <Destiny Release 6.5>

PI 19.8 29.6

24.7

Peak Staff 29.2 43.8

36.5

Eff SLOC (K) 715 1072

893

Page 33: Agile Release Planning, Metrics, and Retrospectives

31

Trendline Assessment – Build Phase StaffingMain Build Peak Staff vs. Size

1,000

1

10

100

Peak S

taff (FTEs)

Rel 6.0

Rel 6.5

Rel 5.0

Rel 7.5 Rel 7.0 Rel 8.0Rel 6.0

Rel 6.5

Rel 5.0

Rel 7.5 Rel 7.0 Rel 8.0

Copyright QSM Associates, Inc. 3/26/2013 61

100 1,000Effective SLOC (thousands)

0.1

Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 BusinessAv g. Line Sty le 1 Sigma Line Sty le

Normal Staffing

Trendline Assessment – Build Phase ScheduleMain Build Phase Duration vs Size

100

10

Time (M

onths)Rel 5.0

Rel 6.0

Rel 6.5

Rel 7.5Rel 7.0

Rel 8.0Rel 5.0

Rel 6.0

Rel 6.5

Rel 7.5Rel 7.0

Rel 8.0

Copyright QSM Associates, Inc. 3/26/2013 62

100 1,000Effective SLOC (thousands)

1

Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 BusinessAv g. Line Sty le 1 Sigma Line Sty le

Schedules are Half Industry

Page 34: Agile Release Planning, Metrics, and Retrospectives

32

Trendline Assessment – Defects/QualityDefects During Test

10,000

100

1,000

Errors (S

ysInt-Del)

Rel 5.0

Rel 6.0 Rel 6.5Rel 7.5 Rel 7.0

Rel 8.0

Rel 5.0

Rel 6.0 Rel 6.5Rel 7.5 Rel 7.0

Rel 8.0

Copyright QSM Associates, Inc. 3/26/2013 63

100 1,000Effective SLOC (thousands)

10

Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 BusinessAv g. Line Sty le 1 Sigma Line Sty le

Far Fewer Defects: 50% - 66% Below Industry

Industry Average

Current Performance

Delta

Follett vs. Industry Average

Project Cost $3.5 Million $2.2 Million -$1.3M

Schedule 12.6 months 7.8 months -4.8 mos

Cumulative 2 890 1450 50%

Copyright QSM Associates, Inc. 3/26/2013 64

CumulativeDefects

2,890 1450 -50%

Staffing 35 35 n/a

* Using average project size of 500,000 lines of new and modified code

Page 35: Agile Release Planning, Metrics, and Retrospectives

33

Follett and XP: It has worked incredibly well…Destiny Library Manager:

Award of Excellence 2004, presented by Technology and Learning magazine (December 2004). Awards Portfolio 2004, presented by Media and Methodsmagazine (May/June 2004).Technology & Learning Award of Excellence 2006, 2007

Destiny Textbook ManagerAwards Portfolio 2005, presented by Media and Methodsmagazine (May/June 2005).Technology & Learning Award of Excellence 2007

Copyright QSM Associates, Inc. 65

gy gDestiny Enriched Services

Technology & Learning Award of Excellence 2007Follett Software provides Library Automation Solutions to 52% of the K12 market. Destiny Library Manager: Single largest product market share in K12 with 19% of the total market and continues to outpace the competition in market growth.

Copyright QSM Associates, Inc. 3/26/2013 66

Page 36: Agile Release Planning, Metrics, and Retrospectives

34

Distributed SCRUM Case Study — BMC SoftwareTeam size

90-95 Total90-95 Total33 Developers37 QA20-25 Architects, PMs, Mgrs

4 Locations

Copyright QSM Associates, Inc. 67

US and IndiaVery Large Releases7 SCRUM Teams

Benchmark Interview — Highlights

Method:Conducted on site interviews on both releasesConducted on-site interviews on both releases.Gathered industry standard core metrics for elapsed time, effort, size*, and defects.Benchmarked the results, calculated performance values, and compared them to the QSM database.Assessed schedule performance, FTE staffing levels, effort defects and productivity values for the Rqmts

Copyright QSM Associates, Inc. 3/26/2013 68

effort, defects, and productivity values for the Rqmts (Story) and Main Build development phases.

* Iterations, stories, and the resultant added/changed code

Page 37: Agile Release Planning, Metrics, and Retrospectives

35

Project Interviews

Copyright QSM Associates, Inc. 3/26/2013 69

Project Interviews

Copyright QSM Associates, Inc. 3/26/2013 70

Page 38: Agile Release Planning, Metrics, and Retrospectives

36

Whiteboard Sketch – Performance Mgr R2.3

Copyright QSM Associates, Inc. 3/26/2013 71

120

140

160

Defect Type (All)

Count of Severity* Release 2.3 Defect Rate

40

60

80

100

120

Product+*

Copyright QSM Associates, Inc. 3/26/2013 72

0

20

Create Date Status Mode Status Severity* TR-Version

Page 39: Agile Release Planning, Metrics, and Retrospectives

37

Input to SLIM-Data Manager

Size DefectsSize

Copyright QSM Associates, Inc. 3/26/2013 73

Time EffortTime

Staffing & Probability Analysis

Avg Staff (people)<Performance Manager Rel 2.3>

100

120109876543 21

R&DC&TP_Mnt

SLIM Replica — PerfMgr Rel 2.3

1 2 3 4 5 6 7 8 9Apr'06

May Jun Jul Aug Sep Oct Nov Dec Jan'07

Feb Mar

0

20

40

60

80

100

Avg Staff (people)

Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9 - 99R 10 - 99.9R

Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9 - 99R 10 - 99.9R

Copyright QSM Associates, Inc. 3/26/2013 74

06 0

SOLUTION PANEL - <Performance Manager Rel 2.3>

DurationEffortCost

Peak StaffMTTD

Start Date

C&T5.3488

4880.092.8

0.1047/2/2006

Life Cycle7.0556

5561.292.8

0.2326/1/2006

MonthsPM$ (K)peopleDays

PI=28.3 MBI=8.3 Eff SLOC=844,710

CONTROL PANEL - <Performance Manager Rel 2.3>

PI 22.6 33.9

28.3

Peak Staff 74.2 111.3

92.8

Eff SLOC (K) 676 1014

845

Page 40: Agile Release Planning, Metrics, and Retrospectives

38

Trendline AssessmentThe following graphs illustrate the staffing, schedule, and effort, and defects for the BUILD phase (vertical

i )axis).On each graph, projects of smaller, medium, and progressively larger sizes (e.g., number of stories) are shown along the horizontal axis. Release 2.4 is shown on the left at 526 stories (569k LOC), Release 2.3 on the right at 918 stories (845k LOC).The center line on the comparison graphs represents

Copyright QSM Associates, Inc. 3/26/2013 75

The center line on the comparison graphs represents the QSM Industry Average, while the upper and lower dashed lines are the +/- 1 standard deviation ranges of the reference database (16th and 84th percentiles).

Agile Assessment — ScheduleBUILD Phase Schedule

100

Agile projects are faster as a whole.

10

C&T D

uration (Months)

BMC Rel 2.4BMC Rel 2.3

BMC Rel 2.4BMC Rel 2.3

(BMC (and also Follett) are highlighted)

Copyright QSM Associates, Inc. 3/26/2013 76

10 100 1,000STORIES (thousands)

1

BMC Rel 2.4BMC Rel 2.4

Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style1 Sigma Line Style

Page 41: Agile Release Planning, Metrics, and Retrospectives

39

Agile Assessment — StaffingBUILD Phase Staffing

1,000

Agile Projects’ team sizes are fairly typicalBMC elects to run with large teams

10

100

C&T Peak Staff (People)

BMC Rel 2.4BMC Rel 2.3

BMC Rel 2.4BMC Rel 2.3

BMC elects to run with large teams.

Copyright QSM Associates, Inc. 3/26/2013 77

10 100 1,000STORIES (thousands)

1

Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style1 Sigma Line Style

Agile Assessment – QualityBugs

10,000

Follett and BMC bug rates are

100

1,000

Errors (SysInt-Del)

BMC Rel 2.4BMC Rel 2.3

BMC Rel 2.4BMC Rel 2.3

significantly lower

Copyright QSM Associates, Inc. 3/26/2013 78

10 100 1,000STORIES (thousands)

1

10

Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style1 Sigma Line Style

Page 42: Agile Release Planning, Metrics, and Retrospectives

40

Summary View — Agile DataMain Build Trends

BUILD Phase Schedule100

C

BUILD Phase Ef f ort

1,000

10,000

10 100 1,000STORIES (thousands)

1

10

C&T D

uration (Months)

10 100 1,000STORIES (thousands)

1

10

100

C&T Effort (P

M)

BUILD Phase Staf f ing1,000

Bugs10,000

Agile projects as a wholeachieve faster speed

Copyright QSM Associates, Inc. 3/26/2013 79

10 100 1,000

STORIES (thousands)

1

10

100

C&T

Peak Staff (P

eople)10 100 1,000

STORIES (thousands)

1

10

100

1,000

Errors (SysInt-Del)

Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Av g. Line Sty le 1 Sigma Line Sty le

Low Defects for BMC & Follett

Productivity Index AssessmentProductivity Index/Velocity

30

35

Agile projects as a wholetend to exhibit higher PIs

10

15

20

25

PI

tend to exhibit higher PIs(Follett/BMC are circled)

Copyright QSM Associates, Inc. 3/26/2013 80

10 100 1,000STORIES (thousands)

0

5

Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style1 Sigma Line Style

Page 43: Agile Release Planning, Metrics, and Retrospectives

41

Productivity Index: Five Companies Using AgileAvg, Min, Max PI vs Organization

Agile #1 - Follett BMC and Follett

Agile #2 - BMC

Company B

Company C

Organization

lead the pack

Copyright QSM Associates, Inc. 3/26/2013 81

Company D

0 5 10 15 20 25 30 35 40Avg, Min, Max PI

All Systems Avg. Line Style

Industry Average

Current Performance

Delta

BMC vs. Industry Average

Project Cost $5.5 Million $5.2 Million -$.3M

Schedule 15 months 6.3 months -8.7 mos

Defects 713 635 11%

Copyright QSM Associates, Inc. 3/26/2013 82

DefectsDuring QA

713 635 -11%

Staffing 40 92 +52

Page 44: Agile Release Planning, Metrics, and Retrospectives

42

BMC “Secret Sauce”

Copyright QSM Associates, Inc. 3/26/2013 83

BMC “Secret Sauce” (con’t)Buy-In

VP-Level (or higher) Senior Executive SponsorshipScrum Master TrainingCore Group Energized and Passionate

Staying “Releasable”Nightly Builds/Test2-week Iteration DemosFrequent, Rigorous Peer Code Review

Dusk to Dawn Teamwork

Copyright QSM Associates, Inc. 3/26/2013 84

Dusk-to-Dawn TeamworkCommunication Techniques for Information FlowWikis, Video-conferencing, Periodic On-Site MeetingsCo-Located Release PlanningScrum of Scrum Meetings (US Time)

Page 45: Agile Release Planning, Metrics, and Retrospectives

43

BMC “Secret Sauce” (con’t)Backlogs

One Master Backlog AND Multiple Backlog ManagementOne Setup for User Stories Across TeamsAdded “Requirements Architect” to Interface Product Mgt with R&D

“Holding Back the Waterfall”Test Driven DevelopmentRetrospective Meetings to Not Regress into old Waterfall HabitsOutside Source to Audit the Process

Copyright QSM Associates, Inc. 3/26/2013 85

Tying It All Together:Release and Iteration Planning

3/26/2013

Page 46: Agile Release Planning, Metrics, and Retrospectives

44

Collect and Use HistoryLearn from the Past

Observe and discover patternsObserve and discover patternsDetermine cause and effectBehave accordingly

Copyright QSM Associates, Inc. 3/26/2013 87

Your Software Development Core Metrics History

Duration

EffortProducedSoftware

(Size)

How long?

How much?

Copyright QSM Associates, Inc. 3/26/2013 88

DiscoveredDefectsHow good?

Page 47: Agile Release Planning, Metrics, and Retrospectives

45

Given a Certain Development Efficiency/Productivity from Observed Patterns

“Real World Deadline Driven Estimation”

And Given the DeadlineWith a Team of “X” People ...

How Much Functionality Can We Build?

Copyright QSM Associates, Inc. 3/26/2013 89

How Much Functionality Can We Build?How Much Functionality Should We Promise?

Rifkin’s Dicta

On Software Estimation:Commitments have to be based on work to be performed (scope/size); therefore, there must be agreement on this.Estimates have to be based on the work to be performed

Stan Rifkin Master Systems Inc.Carnegie Mellon SEI

Copyright QSM Associates, Inc. 3/26/2013 90

(scope/size) and historical records of performance.Commitments must not exceed the capability to perform,or else there is no reason to estimate.

Page 48: Agile Release Planning, Metrics, and Retrospectives

46

Productivity RelationshipConceptual Form

Copyright QSM Associates, Inc. 3/26/2013 91

DeliverableSize

is over Time at some ProductivityEffort

Productivity Relationship (Rearranged) Historical Productivity Measurement

DeliverableSize

is

over

Copyright QSM Associates, Inc. 3/26/2013 92

Time

Productivity

Effort

and

Page 49: Agile Release Planning, Metrics, and Retrospectives

47

Size = 272,768 SLOC

Effort = 249 Person-Months

Ti 13 M thWFS 5.1 SIZE = 19.4

Step 1 - Derive Productivity Index from History (preferably more than 1 project)

Time = 13 Months*PI = TIME EFFORT 19.4

Copyright QSM Associates, Inc. 3/26/2013 93

Direct Reading from SLIM-DataManager

Copyright QSM Associates, Inc. 3/26/2013 94

PI

Page 50: Agile Release Planning, Metrics, and Retrospectives

48

Step 2 - Identify Proposed Time and Effort

Time to June Deadline = 6 Months

Budgeted Effort = 24 Person-Months

Project Staffing Profile

3

5 5 54

456

Copyright QSM Associates, Inc. 3/26/2013 95

23

01234

Jan Feb Mar Apr May Jun

Step 3 - Derive Size Implied by PI

Copyright QSM Associates, Inc. 3/26/2013 96

DeliverableSize

is over Time at some ProductivityEffort

Page 51: Agile Release Planning, Metrics, and Retrospectives

49

Step 3 – Determine (triaged) Size

What Size?

Copyright QSM Associates, Inc. 3/26/2013 97

What PI?

Step 4 - Map to Functional Size Units

Based on resultant size translate that down to Based on resultant size, translate that down to additional size units, such as number of features, technical requirements, stories, story points, etc.For example, if typically, it takes 2 objects per technical requirement (from observed history), then try to promise no more than 2X objects – or X technical requirements - in a 6 month time frame.

Copyright QSM Associates, Inc. 3/26/2013 98

Page 52: Agile Release Planning, Metrics, and Retrospectives

50

Exercise #4

Release and Iteration Planning

Copyright QSM Associates, Inc. 3/26/2013 99

Objectives of this ExerciseYou will learn how to:

Look at productivity patterns for a group of completed projectsprojects.Determine what historical productivity values are relevant to help estimate a new project.Given a target deadline, “reverse calculating” the size/scope that would be possible within the schedule and allocated effort. You will do this in terms of Units of Work (C++ Objects) and Units of Need (Technical Requirements).If the project can not deliver the desired amount of (full) functionality, you’ll understand how to start negotiating for

Copyright QSM Associates, Inc. 3/26/2013 100

y y g gadditional time, effort, or how to make the case for reduced scope (or incremental releases), by making a data-driven argument.

Page 53: Agile Release Planning, Metrics, and Retrospectives

51

For Additional Information

Michael Mahemail: [email protected]: www.qsma.comblog: www.optimalfriction.comtwitter: @michaelcmahTel: 1 413-499-0988

Copyright QSM Associates, Inc. 3/26/2013 101

Andrea GelliEmail: [email protected]: www.qsm-europe.comTel: +41 79 379 9807