quantitative software management · 2019. 5. 29. · © quantitative software management, inc. all...

© Quantitative Software Management, Inc. All rights reserved. Slide #1

Quantitative Software Management

Using Metrics to Develop aSoftware Project Strategy

Donald M. BeckettQSM, Inc.

2000 Corporate Ridge, Suite 900McLean, VA 22102

(360) 697-2640, fax: (703) 749-3795e-mail: [email protected]

http://www.qsm.com

© Quantitative Software Management, Inc. #2

OutlineOutline

•• OverviewOverview

•• Measurement, Expense or InvestmentMeasurement, Expense or Investment

•• State of the Industry: Project EstimationState of the Industry: Project Estimation

•• Staffing and ScheduleStaffing and Schedule

•• Understanding TradeUnderstanding Trade--offsoffs

•• ConclusionConclusion

•• Questions?Questions?


OverviewOverview

Does this sound familiar?


Measurement: Expense or InvestmentMeasurement: Expense or Investment

•• Software measurement (and process Software measurement (and process improvement) are viewed as improvement) are viewed as expensesexpenses: : OverheadOverhead–– Lean, agile organizations want to reduce overheadLean, agile organizations want to reduce overhead

–– But, how do organizations become But, how do organizations become ““lean & agilelean & agile””??

–– Measurement activities not directly related to revenue Measurement activities not directly related to revenue generationgeneration

•• Part of cost of doing business Part of cost of doing business –– 3 3 –– 5% on average5% on average

–– Project management averages 14%Project management averages 14%


Measurement: Expense or InvestmentMeasurement: Expense or Investment

•• What does software measurement provide?What does software measurement provide?1.1. Knowledge of an organizationKnowledge of an organization’’s capabilitiess capabilities2.2. Identifies patterns and trends (Strengths to leverage and Identifies patterns and trends (Strengths to leverage and

weaknesses to correct)weaknesses to correct)3.3. Insight into projects in time to make effective midInsight into projects in time to make effective mid--stream stream

correctionscorrections4.4. Ability to benchmark against competition or Ability to benchmark against competition or ““the industrythe industry”” in in

quality, productivity, and time to marketquality, productivity, and time to market5.5. Quantitative basis for evaluating project and organizational Quantitative basis for evaluating project and organizational

performanceperformance

•• Improves ability to meet commitments, avoid Improves ability to meet commitments, avoid pitfalls, and evaluate tradepitfalls, and evaluate trade--offsoffs


State of the Industry: Project State of the Industry: Project EstimationEstimation

•• Software estimates are Software estimates are notnot project plansproject plans

•• Estimates contain uncertainty about two key Estimates contain uncertainty about two key components:components:–– Scope of the requirements (project size)Scope of the requirements (project size)

–– Team productivityTeam productivity

•• Estimates define a range of possible outcomesEstimates define a range of possible outcomes–– TradeTrade--offs can be examinedoffs can be examined

–– Impossible solutions identifiedImpossible solutions identified


The Cone of UncertaintyThe Cone of Uncertainty

• Not enough information is available early in the development lifecycle to make accurate estimates

• Precision is not accuracy

Low level requirements

Project Definition


Actual vs. Estimated EffortActual vs. Estimated Effort

Effort Growth

0

10

20

30

40

50

60

Smaller At Size Larger

Actual vs. Estimated Effort

Perc

ent


Actual vs. Estimated ScheduleActual vs. Estimated Schedule

Schedule Growth

0

10

20

30

40

50

60


Actual vs. Estimated Schedule

Perc

ent

Percent


Actual vs. Estimated SizeActual vs. Estimated Size

Size Growth

0

10

20

30

40

50

60

70

80

90

100


Actual vs. Estimated Size

Perc

ent


Change HappensChange Happens

•• Software development is more of a discovery Software development is more of a discovery process than a manufacturing oneprocess than a manufacturing one

•• Software projects tend to grow as the implications Software projects tend to grow as the implications of requirements are better definedof requirements are better defined

•• Change must beChange must be–– Managed (Change control process)Managed (Change control process)

–– AnticipatedAnticipated

–– Its implications quantifiedIts implications quantified


In SummaryIn Summary

•• Average schedule growth is 8%Average schedule growth is 8%

•• Average cost/effort growth is 16%Average cost/effort growth is 16%

•• Average size growth is 15%Average size growth is 15%

•• So how can we use this information to create more So how can we use this information to create more accurate estimates?accurate estimates?


Modeling Increased SizeModeling Increased Size

•• Create best project estimate based on proposed Create best project estimate based on proposed sizesize–– Use historically based productivityUse historically based productivity

–– Account for project constraints (staff, effort, schedule)Account for project constraints (staff, effort, schedule)

•• Create estimate based on 15% size growthCreate estimate based on 15% size growth–– Does this account for projected schedule & effort growth?Does this account for projected schedule & effort growth?


S ta ffin g & P ro b a b ility An a lys is

Avg S ta ff L ife Cycle (p e o p le )< 500 F P projec t>

1 2 3 4 5 6 7 8 9 10 11O ct'07

N ov D ec Jan'08

F eb M ar A pr M ay Jun Jul A ug S ep

0

1

2

3

4

5

6

7

Avg S

taff Life Cycle (people)

87654321

M iles tones 0 - C S R 1 - SR R 2 - HLDR 3 - LLDR 4 - C U T 5 - IC 6 - ST C 7 - U A T 8 - F C R 9 - 99R 10 - 99 . 9R

M iles tones 0 - C S R 1 - SR R 2 - HLDR 3 - LLDR 4 - C U T 5 - IC 6 - ST C 7 - U A T 8 - F C R 9 - 99R 10 - 99 . 9R

S O L U TI ON P AN EL - <500 F P p ro je c t>

D ura tionE ffo r tC ost

P eak S ta ffM T T D

S tar t D a te

C & T6 .729

493 .45 .7

1 .82312 /23 /2007

L ife C yc le9 .437

643 .65 .7

1 .82310 /1 /2007

M on thsPM$ (K )peo p leD ay s

PI =16 .5 M B I =3 .6 E f f F P =50 0

9.4 months duration

37 person months effort

50% probability

500 FP Project


Ev a lu a te Pro b a b ility o f C u r re n t Es t im a te

L ife Du r at io n (M o n th s ) Ris k Pr o f ile<500 FP pro jec t>

0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 0

A s s uranc e Lev e l (% )

7

8

9

1 0

1 1

1 2

1 3

Life Duration (M

onths)

L ife Ef fo r t (PM ) Ris k Pr o f ile<500 FP pro jec t>

0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 0

A s s uranc e Lev e l (% )

0

2 0

4 0

6 0

8 0

1 0 0

Life Effort (PM)

Life D uration (Months ) R is k P rofi le - P robabi l i ty demo<5 0 0 FP projec t>

A s s uranceLeve l(% )

15

10152025303540455055606570758085

Life D uration(Months )

7.267.908.258.488.668.828.969.099.219.339.459.579.699.819.94

10.0810.2410.42

Life Effor t (P M) R is k P rofi le - P robabil i ty demo<5 0 0 FP projec t>

A s s uranceLeve l(% )

15

10152025303540455055606570758085

Life Effor t(P M)

0.005.11

12.2016.9920.7924.0526.9829.6932.2734.7537.2039.6542.1344.7147.4250.3553.6157.41

P ro jec t: P robabi l i ty dem o

Likely outcomes 10.2 months schedule, 43 effort months

500 FP Project


S ta ffin g & P ro b ab ility An a lys is

Avg S ta ff L ife Cycle (p e o p le )< 15% s iz e growth>

1 2 3 4 5 6 7 8 9 10 11O ct'07

N ov D ec Jan'08

F eb M ar A pr M ay Jun Jul A ug S ep

0

2

4

6

8

Avg S

taff Life Cycle (people)

87654321

M iles tones 0 - C S R 1 - S R R 2 - HLDR 3 - LLDR 4 - C U T 5 - IC 6 - S T C 7 - U A T 8 - F C R 9 - 99R 10 - 99 .9R

M iles tones 0 - C S R 1 - S R R 2 - HLDR 3 - LLDR 4 - C U T 5 - IC 6 - S T C 7 - U A T 8 - F C R 9 - 99R 10 - 99 .9R

S O L U TI ON P AN E L - <15% s ize g ro w th >

D ura tionE ffo r tC ost

P ea k S ta ffM T T D

S ta r t D a te

C & T7 .33 5

6 03 .76 .5

1 .6811 2 /28 /2007

L ife C yc le10 .246

787 .56 .5

1 .68110 /1 /2 007

M on thsP M$ (K )peop leD ay s

P I =16 .5 M B I =3 .4 E f f F P =575

15% Growth (575 FP)

10.2 months duration

46 person months effort


E v a lu a t e P r o b a b il i t y o f C u r r e n t E s t im a t e

L if e Du r a t io n ( M o n t h s ) R is k P r o f ile< 1 5 % s iz e g r o w th >

0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 0

A s s u r a n c e L e v e l ( % )

7

8

9

1 0

1 1

1 2

1 3

1 4

Life Duration (M

onths)

L if e Ef f o r t ( P M ) R is k P r o f ile< 1 5 % s iz e g r o w th >

0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 0

A s s u r a n c e L e v e l ( % )

0

2 0

4 0

6 0

8 0

1 0 0

1 2 0

Life Effort (PM)

L i fe D u r ati on (M on th s ) R i s k P r ofi l e - P r obabi l i ty de m o< 1 5 % s i z e g r owth >

A s s u r an c eL e ve l(% )

15

10152025303540455055606570758085

L i fe D u r ati on(M on th s )

7 .838 .518 .889 .129 .329 .489 .639 .779 .91

10 .0310 .1610 .2910 .4110 .5510 .6910 .8411 .0011 .20

L i fe Effor t (P M ) R i s k P r ofi l e - P r oba bi l i ty de m o< 1 5 % s i z e g r owth >

A s s u r an c eL e ve l(% )

15

10152025303540455055606570758085

L i fe Effo r t(P M )

0 .008 .61

16 .7622 .2726 .6430 .3933 .7636 .8939 .8542 .7145 .5248 .3351 .1954 .1557 .2860 .6564 .4068 .77

P ro je c t: P ro b a b i l i ty d e m o

15% Growth (575 FP)

Averages close to numbers predicted for effort and schedule growth (10.2 duration and 43 staff months of effort)


Staffing & ScheduleStaffing & Schedule

What is “normal” variability?

Validate Estimate with History

C&T Duration (Months) vs Effective FP

100 1,000Effective FP

1

10

100

C&

T Duration (M

onths)

C&T Effort (PHR) vs Effective FP

100 1,000Effective FP

0.1

1

10

100

1,000

C&

T Effort (P

HR

) (thousands)

Current Solution Historical Projects QSM 2005 Business FP Av g. Line Sty le 1 Sigma Line Sty le Project: Untitled

Schedule varies by a factor of 3.5 from +1σ to -1σ

Effort varies by a factor of 8 from -1σ to +1 σ


How Should Project Effort Be ExpendedHow Should Project Effort Be ExpendedA Case StudyA Case Study

•• 838 projects that had data reported for Analysis/Design as well 838 projects that had data reported for Analysis/Design as well as Construction and Test phasesas Construction and Test phases

•• Average Effort applied to Analysis/Design = 20%Average Effort applied to Analysis/Design = 20%

•• 474 projects in the sample used <= 20% analysis/design effort474 projects in the sample used <= 20% analysis/design effort–– Average Analysis/Design Effort = 11%Average Analysis/Design Effort = 11%

•• 364 projects in the sample used > 20% analysis/design effort364 projects in the sample used > 20% analysis/design effort–– Average Analysis/Design Effort = 33%Average Analysis/Design Effort = 33%

•• Size profiles of samples very similarSize profiles of samples very similar


ObservationsObservations

•• Projects with <20% effort in Requirements and Projects with <20% effort in Requirements and Design Design –– Took 12% longer to completeTook 12% longer to complete

–– Averaged 5.6% more effort (median 24.4% greater)Averaged 5.6% more effort (median 24.4% greater)

–– Had an average staff 14.6% higherHad an average staff 14.6% higher

•• But these projects did excel at one thing:But these projects did excel at one thing:–– Found 63.7% more defects in systems testFound 63.7% more defects in systems test

–– Had 127% more defects in the first 12 months after deliveryHad 127% more defects in the first 12 months after delivery


Understanding TradeUnderstanding Trade--offsoffs

1 43 3

a b

a b= =

× ×where and

Size = Effort Time Productivity

Additional schedule has a much larger impact on a software project than increased effort


The Estimating ConundrumThe Estimating ConundrumSchedule / Effort TradeoffSchedule / Effort Tradeoff

Duration

Effo

rt

Impossible Zone

Impractical ZoneFeasible Solutions

Uncertainty about Size and Productivity creates uncertainty about the Duration-Effort curve


ConclusionsConclusions

•• Measurement is an integral part of managementMeasurement is an integral part of management

•• Information required to make precise estimates is Information required to make precise estimates is unavailableunavailable at project startat project start--upup–– Estimate uncertainty decreases rapidly with more informationEstimate uncertainty decreases rapidly with more information

•• Project estimates historically understate effort, Project estimates historically understate effort, schedule, & sizeschedule, & size–– Estimating based on a larger size or at a higher assurance levelEstimating based on a larger size or at a higher assurance level

can account for thiscan account for this

•• The tradeThe trade--off between schedule & cost/effort is off between schedule & cost/effort is nonnon--linearlinear


ConclusionsConclusions

•• Effort spent in Analysis & Design pays Effort spent in Analysis & Design pays bigbigdividendsdividends–– Reduces overall project effort (cost$$$$)Reduces overall project effort (cost$$$$)

–– Reduces overall project scheduleReduces overall project schedule

–– Improves project qualityImproves project quality


QuestionsQuestions??

quantitative software management · 2019. 5. 29. · © quantitative software management, inc. all...

Documents