agile release planning, metrics, and retrospectives
DESCRIPTION
How do you compare the productivity and quality you achieve with agile practices with that of traditional waterfall projects? Join Michael Mah to learn about both agile and waterfall metrics and how these metrics behave in real projects. Learn how to use your own data to move from sketches on a whiteboard to create agile project trends on productivity, time-to-market, and defect rates. Using recent, real-world case studies, Michael offers a practical, expert view of agile measurement, showing you these metrics in action on retrospectives and release estimation and planning. In hands-on exercises, learn how to replicate these techniques to make your own comparisons for time, cost, and quality. Working in pairs, calculate productivity metrics using the templates Michael employs in his consulting practice. You can leverage these new metrics to make the case for changing to more agile practices and creating realistic project commitments in your organization. Take back new ways for communicating to key decision makers the value of implementing agile development practices.TRANSCRIPT
MB Full‐day Tutorial 6/3/2013 8:30 AM
"Agile Release Planning, Metrics, and Retrospectives"
Presented by:
Michael Mah QSM Associates, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com
Michael Mah QSM Associates, Inc.
With twenty-five years of industry experience Michael Mah teaches, writes, and consults for QSM Associates to tech companies on measuring and estimating software projects for offshore, waterfall, and agile. Michael and his QSM partners have researched thousands of projects worldwide. His work examines time-pressure dynamics of teams and their contribution to project success and failure. Michael’s clients include Boeing, Progressive, Verizon Wireless, Nationwide, JPMorganChase, Roche, and other Fortune 100 companies. He is the director of the Benchmarking Practice at the Cutter Consortium in the US. A private pilot, Michael lives in the mountains of western Massachusetts. qsma.com.
1
Agile Release Planning, Metrics, and Retrospectives - Workshop Slides
Better Software Conference West Las Vegas, Nevada
June 2013Michael Mah
Managing PartnerQSM Associates, Inc.
3/26/2013
QSM Associates, Inc.75 South Church Street
Pittsfield, MA 01201413-499-0988
Fax 413-447-7322e-mail: [email protected]
Website: www.qsma.comBlog: www.optimalfriction.com
Michael Mah is the director of the Benchmarking Practice at the Cutter Consortium, a Boston-based IT think-tank, and served as past editor of the IT Metrics Strategies
Michael Mah
Boston based IT think tank, and served as past editor of the IT Metrics Strategies publication. He is also managing partner at QSM Associates Inc. based in Massachusetts USA.Michael teaches, writes, and consults to technology companies on measuring, estimating and managing software projects, whether in-house, offshore, waterfall, or agile. With over 25 years of experience, Michael and his partners have derived productivity patterns for thousands of software projects collected worldwide across engineering and business applications. His current work examines time-pressure dynamics of teams, and its role in project success and failure. In addition to his background in physics and electrical engineering, he is a mediator
Copyright QSM Associates, Inc. 3/26/2013 2
specializing in dispute resolution for technology projects. He has a degree in electrics engineering from Tufts University. His training on dispute resolution, mediation, and participatory processes is from the Program on Negotiation at Harvard Law School and the Radcliffe Institute for Advanced Study.He can be reached at [email protected].
2
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we h t l
Manifesto for Agile Software Development
have come to value:Individuals and interactions over processes and tools working software over comprehensive documentation Customer collaboration over contract negotiation responding to change over following a plan That is, while there is value in the items on the right, we
Copyright QSM Associates, Inc. 3/26/2013 3
value the items on the left more.
© 2001 Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, Dave Thomas, Martin Fowler
“Without metrics,you’re just another personwith a different opinion.”
Copyright QSM Associates, Inc. 3/26/2013 4
3
“Frothy eloquence neither convinces nor satisfies me. I am from Missouri. You have got to show me.”
- Missouri Congressman Willard Duncan Vandiver, 1899
Copyright QSM Associates, Inc. 3/26/2013 5
Missouri Congressman Willard Duncan Vandiver, 1899
“Metrics” Is Not a Dirty Waterfall Word…Metrics (or measures) can be as light or as heavy as you want them to be. Think of them as – “Information”Familiar examples of measures that inform us:
Stock Market IndexesBlood Tests (i.e. cholesterol)Newborn “Apgar Score”Astronomy – Distance (i.e. Light-Years, Astronomical Units (AUs)…) Others…
In technology we often want to know information about
Copyright QSM Associates, Inc. 3/26/2013 6
In technology, we often want to know information about what works and what doesn’t work, whether we’re “Better, Faster, Cheaper” or if we’re “more productive.”
4
Measures that Inform On… Time (Faster) - Project Duration, perhaps by phase.Cost (Cheaper) - Effort (including labor rates)( p ) ( g )Quality (Better) – Bugs/Defects, User SatisfactionProject Scope – Features, Requirements, Use Cases, Stories etc.
One way of capturing this information is bydrawing a picture
Copyright QSM Associates, Inc. 3/26/2013 7
drawing a picture…
An Example of a Waterfall Picture
Copyright QSM Associates, Inc. 3/26/2013 8
5
An Example of an Agile Picture
Copyright QSM Associates, Inc. 3/26/2013 9
Entering Metrics from Whiteboard into QSM SLIM Model
Copyright QSM Associates, Inc. 3/26/2013 10
6
Project Interviews
Copyright QSM Associates, Inc. 3/26/2013 11
Short Exercise – Observations of the Two
What similarities do you observe? With regard to the What similarities do you observe? With regard to the following
Phases?Time?Staffing?Project Scope/Size?Defects?
Copyright QSM Associates, Inc. 3/26/2013 12
What differences do you observe?
7
An Example of an Agile Picture
Copyright QSM Associates, Inc. 3/26/2013 13
An Example of a Waterfall Picture
Copyright QSM Associates, Inc. 3/26/2013 14
8
An Example of a Waterfall Picture
Copyright QSM Associates, Inc. 3/26/2013 15
Exercise #1
Metrics Capture: Agile XP Release(Part 1)
Copyright QSM Associates, Inc. 3/26/2013 16
( )
9
You will learn how to: (Part 1)“Translate” a description of a project into a
Objectives of this Exercise
p p jwhiteboard staffing sketch,illustrating the major phases of the work over time, and the staffing used for both the story phase and the build (iterations) phase. (Project interview)Extract the key information measures from the sketch (Metrics capture)
Copyright QSM Associates, Inc. 3/26/2013 17
sketch. (Metrics capture)
Exercise #1
Metrics Capture: Agile XP Release(Part 2)
Copyright QSM Associates, Inc. 3/26/2013 18
( )
10
You will learn how to: (Part 2)Open a data collection template (SLIM-Datamanager)
Objectives of this Exercise
p p ( g )and use it to record this information into a file.Save the file.
Copyright QSM Associates, Inc. 3/26/2013 19
Learn to Measure (and Deal Explicitly with) Project Size
3/26/2013
11
Learn to Measure Size
M l t d i j t i t f P Many people today size projects in terms of Person HoursThat’s not sizeThat’s effortSome also size a project as being “25 People”That’s not size
Copyright QSM Associates, Inc. 3/26/2013 21
That’s staffing (headcount)
How Far to the Airport?
If t ll 40 i t h ’t d If you tell me 40 minutes, you haven’t answered my questionDistance is not measured in minutesDistance is measured in miles, or feet, or kilometers, or …
Copyright QSM Associates, Inc. 3/26/2013 22
12
How Far to the Airport?
40 i t i i t i40 minutes is assuming a certainVehicleRouteTime of dayTraffic patternSpeed
Copyright QSM Associates, Inc. 3/26/2013 23
Speed…This is an estimate of time
How Far to the Airport?
If t ll 30 il h d If you tell me 30 miles, you have answered my questionI can estimate the time for a future airport trip by considering my history for any given
VehicleRoute
Copyright QSM Associates, Inc. 3/26/2013 24
Time of dayTraffic pattern…
13
Size CategoriesCreated fromScratch(adds work;
NewModifiedAdapted
(adds work;impacts Size)
(adds work;impacts Size)
Copyright QSM Associates, Inc. 3/26/2013 25
Unmodified Purchased / Reused(adds complexity;impacts Productivity)
What Do We Mean By Effective Size?Created fromScratch(adds work;
New20 Units
Modified50 UnitsAdapted
(adds work;impacts Size)
(adds work;impacts Size)
Copyright QSM Associates, Inc. 3/26/2013 26
70 UnitsNew ModifieEffectiv deS S S= + =
14
Usefulness of Various Size UnitsThere are many different units people use to size
ftsoftwareThey are all related to what must be created, but at different levels of abstractionEach can be useful depending on where you
Copyright QSM Associates, Inc. 3/26/2013 27
depending on where you are in the software development lifecycle
What Do We Mean By Size?
Units of Need Units of WorkBusiness Concern:Business Concern: Value, PriceValue, PriceFocus:Focus: Bang for the BuckBang for the BuckSize Measures:Size Measures:
RequirementsRequirementsFunction PointsFunction PointsUse CasesUse CasesStories/Story PointsStories/Story PointsFeaturesFeatures
Business Concern:Business Concern: CostCostFocus:Focus: ProductivityProductivitySize Measures:Size Measures:
Lines of CodeLines of CodeStatementsStatementsProgram ActionsProgram ActionsModules, ObjectsModules, Objects
Development
U ts o o
Copyright QSM Associates, Inc. 3/26/2013 28
Development
ProcessProductNeed
15
Dividing Units of NeedIt may be helpful to divide the Units of Need
into:into:SimpleMediumComplex
Copyright QSM Associates, Inc. 3/26/2013 29
Getting “Gearing Factors”Once you know what the Units of Need and Units of Work
are, you ask:How large is a simple one?How large is a medium one?How large is a large one?How many small, medium, and large ones will there be?
This gives you Gearing Factors
Copyright QSM Associates, Inc. 3/26/2013 30
16
Gearing FactorIt is valuable to know how many Units of Work are typically associated with a given Unit of Need.This is the “Gearing Factor“ - It can be calculated from completed projects.It can be thought of as a “currency conversion,” informing us about how much software (Units of Work) it took to implement a feature, a story, or a requirement. Examples:
200 lines of code (source instructions) in a C++ Object.3 Objects to implement a simple feature. 6 Objects to implement a complex feature
Copyright QSM Associates, Inc. 3/26/2013 31
implement a complex feature.10 Stories in an agile Iteration.Others…
If desired, we can tally a total Units of Need and Units of Work in a spreadsheet.
# Component NameMost Likely g
Most Likely Expected
Code, Instructions, or Implementation Units (IUs) per Component Number of Components
in the “Cart”
Sizing Example – A Component “Shopping Cart”
# Component Name Likely g Likely Expected1 Simple Foundation Table 5 11 552 Average Foundation Table 15 25 3753 Complex Foundation Table 20 9 1804 Gaps Simple (PeopleSoft Customizations) 34 11 3743 Gaps Average (PeopleSoft Customizations) 66 25 16504 Gaps Complex (PeopleSoft Customizations) 345 9 31055 Business Rules 5 0 06 Data Conversion 6 2496 149769 Interface Simple 320 276 88320
Copyright QSM Associates, Inc. 3/26/2013 32
p10 Interface Average 620 127 7874011 Interface Complex 1520 21 3192013 PeopleSoft Upgrade Rework 0 0 014 Custom Report Simple 25 0 015 Custom Report Average 50 0 016 Custom Report Complex 100 47 4700
Estimated Total # of Implementation Units = 224,395
17
Agile Teams Explicitly Deal with, and Measure Size…
Copyright QSM Associates, Inc. 3/26/2013 33
Example: On Sizing, Agile Teams Use… StoriesA story, or a feature, is described by the product owner. You might also describe it as a “requirement.”Not all stories are created eq al Some are smaller some are Not all stories are created equal. Some are smaller, some are larger than others.Story points are a unit of measure for expressing the overall size of a story. There is no set rule for this. It is an amalgamation of the effort, the complexity, the risk, etc. associated with a story.The range of the scale can be 1-10, 1-7 (or whatever), depending on which book you reference. Agile authors haven’t seemed to set any standard; they say that what’s important is that the numbers
Copyright QSM Associates, Inc. 3/26/2013 34
are relative. i.e. a story with 10 story points is 2x one that is scored at a 5, and if you’re using a 10 scale, then a 5 is “average.”
18
It Takes a Certain Amount of Code to Produce a Story
Within a given iteration (say… 2 weeks), an agile team accomplishes the work to produce a certain number of stories, and the associated story points. The number of story points accomplished in an iteration is called “velocity.”Since we’re talking about creating “software” in a given iteration, these features/stories are manifested by programmers who create new code, and/or adapt (modify) existing code to produce the stories/points.In an iteration (and across an entire release), we can express the total amount of code that delivers these stories/points, and understand the proportional relationship between them. i.e. “It took about 14,000 lines
Copyright QSM Associates, Inc. 3/26/2013 35
p p p ,of code to produce 10 story points in this iteration, which translates to (on average) 1,400 lines of code per story point.If you suppose that a release is aiming for a total of 200 story points, it might involve 280,000 lines of new/modified code (1,400 x 200).
Exercise #2
Creating Schedule and Effort Trendlines
Copyright QSM Associates, Inc. 3/26/2013 36
19
Objectives of this Exercise
You will learn how to:C t X Y G h f f j t Create an X-Y Graph for a group of projects –smaller releases on the left, larger releases on the right – for two metrics of interest: Schedule and Effort.Understand how to visually construct a quick Regression Fit through the data to determine the average trend, and the high-low.
Copyright QSM Associates, Inc. 3/26/2013 37
g , gUse this baseline as a framework to evaluate potential scenarios for a future project.
Learn to Measure (and Deal Explicitly with) Productivity
3/26/2013
20
Software Development Core Metrics
Duration
EffortProducedSoftware
(Size)
How long?
How much?
Copyright QSM Associates, Inc. 3/26/2013 39
DiscoveredDefectsHow good?
How Would You Describe “Productivity Improvement?”
Producing a certain amount of functionality |or features, faster, with lower cost at the sameor higher level of quality… or
Within a given timeline, producing more functionalityor features, at lower cost, at the same or higher level of quality… or
Copyright QSM Associates, Inc. 3/26/2013 40
level of quality… or
… other variations on the theme
21
Production Equation Conceptual Form
Copyright QSM Associates, Inc. 3/26/2013 41
DeliverableSize
is over Time at some ProductivityEffort
Production Equation (Rearranged) Conceptual Form
DeliverableSize
is
over
Copyright QSM Associates, Inc. 3/26/2013 42
Time
Productivity
Effort
and
22
Size = 272,768 SLOC
Effort = 249 Person-MonthsWFSO 5.1 SIZE = 19.4
Production Equation (In Actual Practice)Calibration Form
Months
Time = 13 Months*PI = TIME EFFORT 19.4
Copyright QSM Associates, Inc. 3/26/2013 43
Productivity contributing elementsNobody knows how many elements effect a given environment’s ability to produce a systemTh t l t d h th dThere are at least dozens, perhaps thousandsNobody knows what the effect of their interaction is
Copyright QSM Associates, Inc. 3/26/2013 44
23
Productivity typical factors
Tooling / Methods Personnel ProfilegInfrastructureToolsStandards
Technical DifficultyHardware ConstraintsAlgorithm ComplexityLogic Complexity
ManagementEnvironmentTeam Capabilities
Integration IssuesAmount of reused softwareIntegration complexity
Copyright QSM Associates, Inc. 3/26/2013 45
Logic ComplexityManagement ComplexityPlatform Stability
Integration complexityNumber of interfacesExisting documentation
Productivity Index (PI)(industry values by application type)
Business Information
Command and Control
Process Control
Scientific
System
Telecommunications
Engineering
Copyright QSM Associates, Inc. 3/26/2013 46
0 2 4 6 8 10 12 14 16 18 20 22 24
Productivity Index (PI) w/ ±1 Standard Deviation
Avionics
Microcode
Real Time
Real Time
24
What’s a PI Worth?
Size = 350 Java Classes/ObjectsjBurdened Labor Rate = $120,000/PY
ProductivityIndex
Effort(PM)
Schedule(Mos)
Cost($)
18 42 9.4 420,000
MTTD(Days)
4.1
Copyright QSM Associates, Inc. 3/26/2013 47
18
17
16
42
56
78
9.4
10.5
11.6
420,000
560,000
780,000
4.1
3.5
2.8
Exercise #3
Assessing 6 Agile XP Releases
Copyright QSM Associates, Inc. 3/26/2013 48
25
Objectives of this ExerciseYou will learn how to:
Look at productivity patterns for a group of Look at productivity patterns for a group of completed projects.Examine if productivity is rising or falling over time.Understand how demonstrated/accomplished schedules and effort (high – low) relate to derived productivity values (low-high).Create your own trendlines for schedule, staffing, and defects
Copyright QSM Associates, Inc. 3/26/2013 49
and defectsAssess productivity targets implied by proposed deadline-scope pairings and evaluate they are reasonable (or not) when “sanity checked” against past accomplishments.
Two Case Studies:
Co-located Agile XP and Distributed SCRUM
Copyright QSM Associates, Inc. 3/26/2013 50
26
Co-Located XP Case Study — Follett Software
Team size24 Developers24 Developers7 Testers3 Customers3 Project Leaders
Code Base1,000,000 lines of code7,000 automated unit test
Copyright QSM Associates, Inc. 51
10,000 automated acceptance test
Why XP for Follett?
“XP allowed us to start building based on gcurrent assumptions”“XP approach allowed us to change directions when needed”“XP iterations gave us a “pilot project” test bed”
Copyright QSM Associates, Inc. 52
test bed“Focus on building customer value gave high visibility”
27
Robert Lucas, Nobel Prize (Economics):
On Co-Location of Smart People
The force of concentration, or “clustering” of human creativity and talent … the powerful economic gains when smart and talented people locate in close proximity to one another.
“Human capital externalities”: the productivity and
Copyright QSM Associates, Inc. 3/26/2013 53
“Human capital externalities”: the productivity and innovation gains that occur when human beings cluster together.
Source: Richard FloridaFlight of the Creative Class
People ManagementXP says “XP works in small- to medium-sized t ”teams”How we evolved or extended this rule
Subteams1 large room is mandatory
Trade-offs Communication between
Copyright QSM Associates, Inc. 54
Communication between subteams1 room noise level (distractions)Lack of personal space
28
Copyright QSM Associates, Inc. 3/26/2013 55
Copyright QSM Associates, Inc. 3/26/2013 56
29
Copyright QSM Associates, Inc. 3/26/2013 57
Destiny Release 6.5 – Whiteboard Sketch
Copyright QSM Associates, Inc. 3/26/2013 58
30
Input to SLIM-DataManager
Size Defects
Copyright QSM Associates, Inc. 3/26/2013 59
Time Effort
SLIM Replica – Destiny 6.5Staffing & Probability Analysis
Avg Staff (people)<Destiny Release 6.5>
40
50876543 21
R&DC&T
1 2 3 4 5 6 7 8 9 10 11 12Apr'05
May Jun Jul Aug Sep Oct Nov Dec Jan'06
Feb Mar Apr May Jun
0
10
20
30
40
Avg Staff (people)
Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9 - 99R 10 - 99.9R
Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9 - 99R 10 - 99.9R
Copyright QSM Associates, Inc. 3/26/2013 60
SOLUTION PANEL - <Destiny Release 6.5>
DurationEffortCost
Peak StaffMTTD
Start Date
C&T11.0400
3400.036.50.638
7/2/2005
Life Cycle12.0446
3791.036.50.638
6/1/2005
MonthsPM$ (K)peopleDays
PI=24.7 MBI=4.8 Eff SLOC=893,298
CONTROL PANEL - <Destiny Release 6.5>
PI 19.8 29.6
24.7
Peak Staff 29.2 43.8
36.5
Eff SLOC (K) 715 1072
893
31
Trendline Assessment – Build Phase StaffingMain Build Peak Staff vs. Size
1,000
1
10
100
Peak S
taff (FTEs)
Rel 6.0
Rel 6.5
Rel 5.0
Rel 7.5 Rel 7.0 Rel 8.0Rel 6.0
Rel 6.5
Rel 5.0
Rel 7.5 Rel 7.0 Rel 8.0
Copyright QSM Associates, Inc. 3/26/2013 61
100 1,000Effective SLOC (thousands)
0.1
Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 BusinessAv g. Line Sty le 1 Sigma Line Sty le
Normal Staffing
Trendline Assessment – Build Phase ScheduleMain Build Phase Duration vs Size
100
10
Time (M
onths)Rel 5.0
Rel 6.0
Rel 6.5
Rel 7.5Rel 7.0
Rel 8.0Rel 5.0
Rel 6.0
Rel 6.5
Rel 7.5Rel 7.0
Rel 8.0
Copyright QSM Associates, Inc. 3/26/2013 62
100 1,000Effective SLOC (thousands)
1
Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 BusinessAv g. Line Sty le 1 Sigma Line Sty le
Schedules are Half Industry
32
Trendline Assessment – Defects/QualityDefects During Test
10,000
100
1,000
Errors (S
ysInt-Del)
Rel 5.0
Rel 6.0 Rel 6.5Rel 7.5 Rel 7.0
Rel 8.0
Rel 5.0
Rel 6.0 Rel 6.5Rel 7.5 Rel 7.0
Rel 8.0
Copyright QSM Associates, Inc. 3/26/2013 63
100 1,000Effective SLOC (thousands)
10
Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 BusinessAv g. Line Sty le 1 Sigma Line Sty le
Far Fewer Defects: 50% - 66% Below Industry
Industry Average
Current Performance
Delta
Follett vs. Industry Average
Project Cost $3.5 Million $2.2 Million -$1.3M
Schedule 12.6 months 7.8 months -4.8 mos
Cumulative 2 890 1450 50%
Copyright QSM Associates, Inc. 3/26/2013 64
CumulativeDefects
2,890 1450 -50%
Staffing 35 35 n/a
* Using average project size of 500,000 lines of new and modified code
33
Follett and XP: It has worked incredibly well…Destiny Library Manager:
Award of Excellence 2004, presented by Technology and Learning magazine (December 2004). Awards Portfolio 2004, presented by Media and Methodsmagazine (May/June 2004).Technology & Learning Award of Excellence 2006, 2007
Destiny Textbook ManagerAwards Portfolio 2005, presented by Media and Methodsmagazine (May/June 2005).Technology & Learning Award of Excellence 2007
Copyright QSM Associates, Inc. 65
gy gDestiny Enriched Services
Technology & Learning Award of Excellence 2007Follett Software provides Library Automation Solutions to 52% of the K12 market. Destiny Library Manager: Single largest product market share in K12 with 19% of the total market and continues to outpace the competition in market growth.
Copyright QSM Associates, Inc. 3/26/2013 66
34
Distributed SCRUM Case Study — BMC SoftwareTeam size
90-95 Total90-95 Total33 Developers37 QA20-25 Architects, PMs, Mgrs
4 Locations
Copyright QSM Associates, Inc. 67
US and IndiaVery Large Releases7 SCRUM Teams
Benchmark Interview — Highlights
Method:Conducted on site interviews on both releasesConducted on-site interviews on both releases.Gathered industry standard core metrics for elapsed time, effort, size*, and defects.Benchmarked the results, calculated performance values, and compared them to the QSM database.Assessed schedule performance, FTE staffing levels, effort defects and productivity values for the Rqmts
Copyright QSM Associates, Inc. 3/26/2013 68
effort, defects, and productivity values for the Rqmts (Story) and Main Build development phases.
* Iterations, stories, and the resultant added/changed code
35
Project Interviews
Copyright QSM Associates, Inc. 3/26/2013 69
Project Interviews
Copyright QSM Associates, Inc. 3/26/2013 70
36
Whiteboard Sketch – Performance Mgr R2.3
Copyright QSM Associates, Inc. 3/26/2013 71
120
140
160
Defect Type (All)
Count of Severity* Release 2.3 Defect Rate
40
60
80
100
120
Product+*
Copyright QSM Associates, Inc. 3/26/2013 72
0
20
Create Date Status Mode Status Severity* TR-Version
37
Input to SLIM-Data Manager
Size DefectsSize
Copyright QSM Associates, Inc. 3/26/2013 73
Time EffortTime
Staffing & Probability Analysis
Avg Staff (people)<Performance Manager Rel 2.3>
100
120109876543 21
R&DC&TP_Mnt
SLIM Replica — PerfMgr Rel 2.3
1 2 3 4 5 6 7 8 9Apr'06
May Jun Jul Aug Sep Oct Nov Dec Jan'07
Feb Mar
0
20
40
60
80
100
Avg Staff (people)
Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9 - 99R 10 - 99.9R
Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9 - 99R 10 - 99.9R
Copyright QSM Associates, Inc. 3/26/2013 74
06 0
SOLUTION PANEL - <Performance Manager Rel 2.3>
DurationEffortCost
Peak StaffMTTD
Start Date
C&T5.3488
4880.092.8
0.1047/2/2006
Life Cycle7.0556
5561.292.8
0.2326/1/2006
MonthsPM$ (K)peopleDays
PI=28.3 MBI=8.3 Eff SLOC=844,710
CONTROL PANEL - <Performance Manager Rel 2.3>
PI 22.6 33.9
28.3
Peak Staff 74.2 111.3
92.8
Eff SLOC (K) 676 1014
845
38
Trendline AssessmentThe following graphs illustrate the staffing, schedule, and effort, and defects for the BUILD phase (vertical
i )axis).On each graph, projects of smaller, medium, and progressively larger sizes (e.g., number of stories) are shown along the horizontal axis. Release 2.4 is shown on the left at 526 stories (569k LOC), Release 2.3 on the right at 918 stories (845k LOC).The center line on the comparison graphs represents
Copyright QSM Associates, Inc. 3/26/2013 75
The center line on the comparison graphs represents the QSM Industry Average, while the upper and lower dashed lines are the +/- 1 standard deviation ranges of the reference database (16th and 84th percentiles).
Agile Assessment — ScheduleBUILD Phase Schedule
100
Agile projects are faster as a whole.
10
C&T D
uration (Months)
BMC Rel 2.4BMC Rel 2.3
BMC Rel 2.4BMC Rel 2.3
(BMC (and also Follett) are highlighted)
Copyright QSM Associates, Inc. 3/26/2013 76
10 100 1,000STORIES (thousands)
1
BMC Rel 2.4BMC Rel 2.4
Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style1 Sigma Line Style
39
Agile Assessment — StaffingBUILD Phase Staffing
1,000
Agile Projects’ team sizes are fairly typicalBMC elects to run with large teams
10
100
C&T Peak Staff (People)
BMC Rel 2.4BMC Rel 2.3
BMC Rel 2.4BMC Rel 2.3
BMC elects to run with large teams.
Copyright QSM Associates, Inc. 3/26/2013 77
10 100 1,000STORIES (thousands)
1
Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style1 Sigma Line Style
Agile Assessment – QualityBugs
10,000
Follett and BMC bug rates are
100
1,000
Errors (SysInt-Del)
BMC Rel 2.4BMC Rel 2.3
BMC Rel 2.4BMC Rel 2.3
significantly lower
Copyright QSM Associates, Inc. 3/26/2013 78
10 100 1,000STORIES (thousands)
1
10
Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style1 Sigma Line Style
40
Summary View — Agile DataMain Build Trends
BUILD Phase Schedule100
C
BUILD Phase Ef f ort
1,000
10,000
10 100 1,000STORIES (thousands)
1
10
C&T D
uration (Months)
10 100 1,000STORIES (thousands)
1
10
100
C&T Effort (P
M)
BUILD Phase Staf f ing1,000
Bugs10,000
Agile projects as a wholeachieve faster speed
Copyright QSM Associates, Inc. 3/26/2013 79
10 100 1,000
STORIES (thousands)
1
10
100
C&T
Peak Staff (P
eople)10 100 1,000
STORIES (thousands)
1
10
100
1,000
Errors (SysInt-Del)
Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Av g. Line Sty le 1 Sigma Line Sty le
Low Defects for BMC & Follett
Productivity Index AssessmentProductivity Index/Velocity
30
35
Agile projects as a wholetend to exhibit higher PIs
10
15
20
25
PI
tend to exhibit higher PIs(Follett/BMC are circled)
Copyright QSM Associates, Inc. 3/26/2013 80
10 100 1,000STORIES (thousands)
0
5
Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style1 Sigma Line Style
41
Productivity Index: Five Companies Using AgileAvg, Min, Max PI vs Organization
Agile #1 - Follett BMC and Follett
Agile #2 - BMC
Company B
Company C
Organization
lead the pack
Copyright QSM Associates, Inc. 3/26/2013 81
Company D
0 5 10 15 20 25 30 35 40Avg, Min, Max PI
All Systems Avg. Line Style
Industry Average
Current Performance
Delta
BMC vs. Industry Average
Project Cost $5.5 Million $5.2 Million -$.3M
Schedule 15 months 6.3 months -8.7 mos
Defects 713 635 11%
Copyright QSM Associates, Inc. 3/26/2013 82
DefectsDuring QA
713 635 -11%
Staffing 40 92 +52
42
BMC “Secret Sauce”
Copyright QSM Associates, Inc. 3/26/2013 83
BMC “Secret Sauce” (con’t)Buy-In
VP-Level (or higher) Senior Executive SponsorshipScrum Master TrainingCore Group Energized and Passionate
Staying “Releasable”Nightly Builds/Test2-week Iteration DemosFrequent, Rigorous Peer Code Review
Dusk to Dawn Teamwork
Copyright QSM Associates, Inc. 3/26/2013 84
Dusk-to-Dawn TeamworkCommunication Techniques for Information FlowWikis, Video-conferencing, Periodic On-Site MeetingsCo-Located Release PlanningScrum of Scrum Meetings (US Time)
43
BMC “Secret Sauce” (con’t)Backlogs
One Master Backlog AND Multiple Backlog ManagementOne Setup for User Stories Across TeamsAdded “Requirements Architect” to Interface Product Mgt with R&D
“Holding Back the Waterfall”Test Driven DevelopmentRetrospective Meetings to Not Regress into old Waterfall HabitsOutside Source to Audit the Process
Copyright QSM Associates, Inc. 3/26/2013 85
Tying It All Together:Release and Iteration Planning
3/26/2013
44
Collect and Use HistoryLearn from the Past
Observe and discover patternsObserve and discover patternsDetermine cause and effectBehave accordingly
Copyright QSM Associates, Inc. 3/26/2013 87
Your Software Development Core Metrics History
Duration
EffortProducedSoftware
(Size)
How long?
How much?
Copyright QSM Associates, Inc. 3/26/2013 88
DiscoveredDefectsHow good?
45
Given a Certain Development Efficiency/Productivity from Observed Patterns
“Real World Deadline Driven Estimation”
And Given the DeadlineWith a Team of “X” People ...
How Much Functionality Can We Build?
Copyright QSM Associates, Inc. 3/26/2013 89
How Much Functionality Can We Build?How Much Functionality Should We Promise?
Rifkin’s Dicta
On Software Estimation:Commitments have to be based on work to be performed (scope/size); therefore, there must be agreement on this.Estimates have to be based on the work to be performed
Stan Rifkin Master Systems Inc.Carnegie Mellon SEI
Copyright QSM Associates, Inc. 3/26/2013 90
(scope/size) and historical records of performance.Commitments must not exceed the capability to perform,or else there is no reason to estimate.
46
Productivity RelationshipConceptual Form
Copyright QSM Associates, Inc. 3/26/2013 91
DeliverableSize
is over Time at some ProductivityEffort
Productivity Relationship (Rearranged) Historical Productivity Measurement
DeliverableSize
is
over
Copyright QSM Associates, Inc. 3/26/2013 92
Time
Productivity
Effort
and
47
Size = 272,768 SLOC
Effort = 249 Person-Months
Ti 13 M thWFS 5.1 SIZE = 19.4
Step 1 - Derive Productivity Index from History (preferably more than 1 project)
Time = 13 Months*PI = TIME EFFORT 19.4
Copyright QSM Associates, Inc. 3/26/2013 93
Direct Reading from SLIM-DataManager
Copyright QSM Associates, Inc. 3/26/2013 94
PI
48
Step 2 - Identify Proposed Time and Effort
Time to June Deadline = 6 Months
Budgeted Effort = 24 Person-Months
Project Staffing Profile
3
5 5 54
456
Copyright QSM Associates, Inc. 3/26/2013 95
23
01234
Jan Feb Mar Apr May Jun
Step 3 - Derive Size Implied by PI
Copyright QSM Associates, Inc. 3/26/2013 96
DeliverableSize
is over Time at some ProductivityEffort
49
Step 3 – Determine (triaged) Size
What Size?
Copyright QSM Associates, Inc. 3/26/2013 97
What PI?
Step 4 - Map to Functional Size Units
Based on resultant size translate that down to Based on resultant size, translate that down to additional size units, such as number of features, technical requirements, stories, story points, etc.For example, if typically, it takes 2 objects per technical requirement (from observed history), then try to promise no more than 2X objects – or X technical requirements - in a 6 month time frame.
Copyright QSM Associates, Inc. 3/26/2013 98
50
Exercise #4
Release and Iteration Planning
Copyright QSM Associates, Inc. 3/26/2013 99
Objectives of this ExerciseYou will learn how to:
Look at productivity patterns for a group of completed projectsprojects.Determine what historical productivity values are relevant to help estimate a new project.Given a target deadline, “reverse calculating” the size/scope that would be possible within the schedule and allocated effort. You will do this in terms of Units of Work (C++ Objects) and Units of Need (Technical Requirements).If the project can not deliver the desired amount of (full) functionality, you’ll understand how to start negotiating for
Copyright QSM Associates, Inc. 3/26/2013 100
y y g gadditional time, effort, or how to make the case for reduced scope (or incremental releases), by making a data-driven argument.
51
For Additional Information
Michael Mahemail: [email protected]: www.qsma.comblog: www.optimalfriction.comtwitter: @michaelcmahTel: 1 413-499-0988
Copyright QSM Associates, Inc. 3/26/2013 101
Andrea GelliEmail: [email protected]: www.qsm-europe.comTel: +41 79 379 9807