promise 2011: "an analysis of trends in productivity and cost drivers over years"

University of Southern CaliforniaCenter for Systems and Software Engineering

1© USC-CSSE

An Analysis of Trends in Productivity and Cost

Drivers over Years

Vu Nguyen1, LiGuo Huang2, Barry Boehm3

[email protected], [email protected], [email protected]

1 University of Science, National University, Vietnam2 Southern Methodist University

3 University of Southern California


2© USC-CSSE

Outline

• Introduction

• Research Method

• Results

• Conclusions

• Future Work


3© USC-CSSE

Motivation• Software practices and technologies have evolved

significantly over the last decades, e.g.,

– Process: sequential to concurrent, traditional plan-driven to agile and rapid development

– Tool and language: from basic code editors and debuggers to sophisticated development environments, from first to fourth generation languages

– Storage and processing capabilities no longer a major concern for software project

• It is important to understand how such advancements affect the productivity of software projects over the years


4© USC-CSSE

Objectives

• Analysis of productivity trends

– Productivity trends of completed projects over the last 40 years

– COCOMO cost drivers contributing to the productivity trends

• Analysis of productivity influencing factors

– Trends of COCOMO cost drivers over the period

• Proposing a measure namely Difficulty to explain productivity trends and variability


5© USC-CSSE

COCOMO• Cost Constructive Model (COCOMO) is a model for

estimating software cost

• COCOMO originally published in 1981

– Calibrated using 63 completed projects

– Uses software size and 15 cost drivers as input

• COCOMO II published in 2000

– Reflects changes in technologies and practices

– Uses 22 cost drivers plus software size

– Introduces 5 scale factors

– Calibrated using 161 completed projects


6© USC-CSSE

COCOMO (cont’d)COCOMO II estimates project effort as

Where,

PM: is estimated effort in person-month

Size: size of software project

A: a constant

SFi: each of 5 scale factors

EMi: each of 17 effort multipliers

β0, β1: constants

* *p

Bi

i

PM A Size EM= ∏

∑=

+=5

110i

iSFB ββ


7© USC-CSSE

Calibrating COCOMO• Process of fitting data to the model to adjust its

parameters and constants

Initial rating scales for

cost drivers

Sample data

Delphi survey of 8 experts(Expert-judgment estimates) Model Calibration

New rating scales for cost

drivers andconstants

Calibration Techniques:- Ordinary Least Squares Regression (OLS)- Bayesian Analysis

E.g., 341 data points for this study


8© USC-CSSE

Outline

• Introduction

• Research Method

• Results

• Conclusions

• Future Work


9© USC-CSSE

ProcedureOur analysis procedure consists of

• Review data set of 341 projects

• Compute and evaluate productivity trends

• Calibrate COCOMO using 341 projects to obtain rating scales for cost drivers

• Define a project Difficulty measure using the rating scales from step #3

• Compute and evaluate project Difficulty trends

• Evaluate trends of cost drivers


10© USC-CSSE

Data Set• Consisting of 341 projects completed between 1970 and 2009

• Collected from 25 organizations in 4 countries, US (251 projects), Brazil (16), Vietnam (70) and Thailand (4)

• 207 new development and 134 maintenance/enhancement projects

• Attributes of each data point:

Attribute Description

Effort Actual project effort in person-month (PM)

Size Size in KSLOC

Cost Drivers Twenty two COCOMO cost drivers and scale factors

Completion Year The year in which the project is completed

Project Type Either New Development or Maintenance

Country Country from which the project was collected


11© USC-CSSE

Data Set (cont’d)

4

1 2 2 31

9 8

14

4

0 0 0 0 0 0

31

3

10

2 3 2

7 8

18

13

48

19

7

43

18 18

9

14

21

13

6 6

1

0

5

10

15

20

25

30

35

40

45

50

'70 '71 '72 '73 '74 '75 '76 '77 '78 '79 '80 '81 '82 '83 '84 '85 '86 '87 '88 '89 '90 '91 '92 '93 '94 '95 '96 '97 '98 '99 '00 '01 '02 '03 '04 '05 '06 '07 '08 '09

Completion Year

Nu

mb

er o

f P

roje

cts

• Number of projects completed in each year


12© USC-CSSE

Calculating Productivity• We defined productivity as

PM - project actual effort in person-month

KSLOC - software size measured in thousand source lines of code

• This productivity is not adjusted by the productivity impact factors

• We referred this as raw productivity

PM

KSLOCtyProductivi =

PM

KSLOCtyProductivi =


13© USC-CSSE

Defining a Difficulty Measure• Difficulty is proposed to measure the overall impact of the COCOMO

cost drivers on productivity

• Difficulty is derived from the COCOMO effort equation

• Difficulty is defined as

Where,

EMi and SFj are COCOMO’s cost drivers (effort multipliers and scale factors)

β0, β1 are constants

• Rating scales of EM and SF are obtained from calibrating the COCOMO model using 341 projects

• Difficulty is computed for each project

100

*10017

1∏

== ii

B EMDifficulty

∑=

+=5

110i

iSFB ββ


14© USC-CSSE

Difficulty Measure• Basic characteristics

– Indicates how difficult it is for a project to build the software in terms of effort required

– Reflects the overall impact of the COCOMO cost drivers on the effort of a 100KLOC project

– Is always > 0

– If Difficulty = 1.0, the cost drivers overall do not affect the effort required for the project

– If Difficulty > 1.0, the overall impact of the cost drivers is to increase the effort

– If Difficulty < 1.0, the overall impact of the cost drivers is to decrease the effort


15© USC-CSSE

Evaluation Method• Grouping projects based on their completion years

• Trends evaluated by testing the significance of correlations between productivity, cost drivers and completion year

• Kendall’s rank correlation used to test the trends

• Mann-Whitney U test used to validate the difference between two groups

• Level of significance used: 0.05

• Statistical tests performed using 341 data points


16© USC-CSSE

Outline

• Introduction

• Research Method

• Results

• Conclusions

• Future Work


17© USC-CSSE

Productivity Trends

• Overall trend: productivity increases over the years (Kendall’s correlation between productivity and completion year: p-value = 0)

Pro

du

ctiv

ity

(KS

LO

C/P

M)

Project Completion Year

No. of projects


18© USC-CSSE

Productivity Trends (cont’d)• Productivity trends in new development and

maintenance projects

New Development ProjectsMaintenance Projects

Overall trend: productivity increases over the years (p-value = 1.8*10-5)

Overall trend: productivity increases over the years (p-value = 0.01 < 0.05)

Pro

du

cti

vity

(K

SL

OC

/PM

)

Pro

du

ctiv

ity

(KS

LO

C/P

M)


19© USC-CSSE

Difficulty Trends• How does Difficulty explain the productivity trends?


Pro

ject

Dif

fic

ult

y

• Overall trend: Difficulty of the projects decreases over the years in the opposite direction with productivity (Kendall’s correlation between Difficulty and completion year: p-value = 0)


20© USC-CSSE

Productivity and Difficulty of New Development and Maintenance Projects

• New development projects were less productive because they were more difficult

Pro

du

ctiv

ity

(K

SL

OC

/PM

)

Dif

fic

ult

yProductivity of New Dev. and Maintenance Projects Difficulty of New Dev. and Maintenance Projects


21© USC-CSSE

Trends in Cost Drivers• Trends tested using Kendall’s

rank correlation between each of the cost drivers and completion year

• 13 of 22 cost drivers found to have statistically significant correlations with the completion year (Kendall’s p-value < 0.05)

Significantly Correlated Cost driver Kendall’s p-value

TOOL Use of Software Tools 2.20E-16

PMAT Process Maturity 1.22E-13

STOR Main Storage Constraint 1.31E-11

TIME Execution Time Constraint 6.62E-10

PLEX Platform Experience 1.98E-05

PVOL Platform Volatility 2.04E-05

APEX Applications Experience 4.88E-05

LTEX Language and Tool Experience 2.84E-04

DATA Database Size 1.81E-03

RELY Required Software Reliability 1.42E-02

CPLX Product Complexity 1.58E-02

PREC Precedentedness of Application 2.13E-02

ACAP Analyst Capability 4.87E-02


22© USC-CSSE

Trends in Cost Drivers (cont’d)• Platform and process factors Use of Tools, Process

Capability, Main Storage Constraint, and Execution Time Constraint have significantly improved over the years

• Applications Experience and Analyst Capability were seen to decrease

• Platforms Experience tends to increase over the years (developers on recent projects are more experienced with platforms)

• Product Complexity tends to decrease


23© USC-CSSE

Trends in Cost Drivers (cont’d)

• Use of Software Tools (TOOL) factorT

OO

L R

atin

g S

cale

s


Very Low

Low

Nominal

High

Very High

TO

OL

Rat

ing

Le

vels


25© USC-CSSE

Conclusions• Trends of productivity and cost drivers analyzed using

341 projects completed between 1970 and 2009

• Productivity found to increase over the years

• Characterizing productivity trends using one or a few factors is insufficient. Thus, we proposed Difficulty to combine factors

• Productivity trends explained by the Difficulty measure which is based on the COCOMO’s form and cost drivers

• Ratings of 13 of 22 cost drivers found to have clear trends over the years. Hence, they affect the productivity trends


26© USC-CSSE

Limitations and Future Work• Some limitations

– Small data set, considering many software products developed over the years

– Analyzed only 22 COCOMO cost drivers. It’s possible that other factors are more influential

• Future work

– Further analysis and evaluation of the Difficulty measure

– Investigating how relative impact of cost drivers differ across projects

promise 2011: "an analysis of trends in productivity and cost drivers over years"

Technology

software size

size of software projecta

years cocomo cost drivers

national university

university of science

input cocomo

projects compute

size em