research profile systems engineering economics

RESEARCH PROFILE

Systems Engineering Economics

October 16, 2007

Dr. Ricardo Valerdi

Massachusetts Institute of Technology

[email protected]

© 2007 Massachusetts Institute of Technology© 2007 Massachusetts Institute of Technology 2

Research Cluster-Portfolio

Mapping

XSE Strategic

Guidance

XSE in the

Enterprise

XSE Economics

XDesigning for

Value

Robustness

XXXXSocio-Tech

Decision

Making

SE-SynthesisSE-FieldR-STARSV-STARS


Research Portfolio (3)

SYSTEMS ENGINEERING ECONOMICS

This research area aims at developing a new paradigm that encompasses an economics view of systems engineering to achieve measurable and predictable outcomes while delivering

value to stakeholders. Examples include:

• Measurement of productivity and quantifying the ROI of systems engineering

• Advanced methods for reuse, cost modeling, and risk modeling

• Application of real options in systems and enterprises

• Leading indicators for systems engineering effectiveness


Outline

• Motivation for economics in systems engineering

• Economic Principles – Cost

– Diseconomies of scale

– Productivity

– Consumer behavior

– Judgment and decision making

– Risk

– Reuse

– Value and Return on Investment

– Leading Indicators


Motivation

Daniel Kahneman

• Psychologist (Princeton University)

• Nobel laureate in Economics (2002)

Barry Boehm

• Computer Scientist (USC)

• Father of Software Engineering Economics


Definitions

Systems Engineering• An interdisciplinary approach and means to enable the realization of successful systems. It focuses on defining customer needs and required functionality early in the development cycle, documenting requirements, then proceeding with design synthesis and system validation while considering the complete problem.

Economics• The social science that deals with the production, distribution, and consumption of goods and services and with the theory and management of economies or economic systems.


All models are wrong…

…but some of them are useful.


Systems Engineering Knowledge

Hierarchy

�Prediction

�Symbolic

Manipulation

�Symbolic

Representatio

n

��Quantification

&

Measurement

��Abstraction

��Classification

��Observation

VBSSE[2

] (Jain &

Boehm

2006)

Ontologies

(Honour &

Valerdi

2006)

GUTSE[1]

(Friedman

2004)

DoD

Architecture

Framework

(DODAF

2004)

COSYSMO

(Valerdi et

al 2003)

Maturity

Models

(CMMI

2002)

SE

Standards

(ANSI/EIA

1999,

ISO/IEC

2002)

Vee Model

(Forsberg &

Mooz 1995)

Systems

Architecting

heuristics

(Rechtin

1991)

[1] Grand Unified Theory of Systems Engineering[2] Value-Based Systems & Software Engineering

Dixit, I., Valerdi, R., “Challenges in the Development of Systems Engineering as a Profession,” INCOSE Symposium, San Diego, CA, June 2007.


Outline




– Productivity



– Risk

– Reuse




COSYSMO

SizeDrivers

EffortMultipliers

Effort, $

Calibration

# Requirements# Interfaces# Scenarios# Algorithms

+3 Volatility Factors

- Application factors-8 factors

- Team factors-6 factors

Valerdi, R., Systems Engineering Cost Estimation with COSYSMO, Wiley, 2008.


Where:

PMNS = effort in Person Months (Nominal Schedule)

A = calibration constant derived from historical project data

k = {REQ, IF, ALG, SCN}

wx = weight for “easy”, “nominal”, or “difficult” size driver

= quantity of “k” size driver

E = represents diseconomy of scale

EM = effort multiplier for the jth cost driver. The geometric product results in an

overall effort adjustment factor to the nominal effort.

xΦ

COSYSMO Algorithm

∏∑=

⋅

Φ+Φ+Φ⋅=

14

1

,,,,,, )(j

j

E

k

kdkdknknkekeNS EMwwwAPM


Diseconomies of scale

(i.e., Non-linear effects of size)


Productivity

Y-intercept = 300 PM (25 people for 1 year)

Productivity = [0.004 min, 0.22 max] (Size/SE_Hr)

Range = Size[374 min, 8743 max]; SE_Hours[10,992 min, 1,749,372 max]

“x” = SW

“+” = HW

0

0.05

0.1

0.15

0.2

0.25

1 2 3 4 5 6 7 8


Does size matter?

ADJHRSSE

SizeP

__=

• Evaluated by productivity, P, as a function of Size & Effort

• Projects with extremely low and extremely high productivity introduced new questions

∑ ⋅+⋅+⋅+⋅= OPSCALGINTFREQSize ωωωω


Feasibility Plans/Rqts. Design Develop

and TestPhases and Milestones

Relative

Size

Range

Operational

Concept

Life Cycle

Objectives

Life Cycle

Architecture

Initial

Operating

Capability

x

0.5x

0.25x

4x

2x

Estimation Accuracy


Bayesian Approximation

1.06

Literature,

behavioral analysis

A-priori

Experts’ Delphi

Noisy data analysis

A-posteriori Bayesian update

Productivity Range =Highest Rating /Lowest Rating

1.451.51

1.41


Outline




– Productivity



– Risk

– Reuse




0

2

4

6

8

10

12

14

16

18

20

300 700 1100 1500 1900 2300 2700 More

Person Months

Frequency

0

5

10

15

20

25

200

385.71

4285

757

1.42

8571

475

7.14

2857

194

2.85

7142

911

28.571

429

1314

.285

714

More

Person Months

Frequency

0

5

10

15

20

25

30

35

40

150 300 450 600 750 900 1050 More

Person Months

Frequency

Intuitive Judgments

SE Effort given

Conceptualize

took 300 PM

SE Effort given

Conceptualize &

Develop

took 300 PM

SE Effort given

Conceptualize,

Develop, OT&E

took 300 PM

1386=µ 594=µ 390=µ

Valerdi, R., “Cognitive Limits of Software Cost Estimation,” 1st Conference on Empirical Software Engineering & Measurement, September 2007, Madrid, Spain.


594=µ 390=µ

Cone of Uncertainty

SE Effort given

Conceptualize

took 300 PM

SE Effort given

Conceptualize &

Develop

took 300 PM

SE Effort given

Conceptualize,

Develop, OT&E

took 300 PM

1386=µ

758=σ 241=σ 145=σ


What do decision makers trust?

0

5

10

15

20

25

90 94 98 102

106

110

114

118

122

126

130

Person Months

Frequency

0

5

10

15

20

25

30

900

940

98010

2010

6011

0011

4011

8012

2012

6013

00

Person Months

Frequency

110=µ

COSYSMO = 1,000 PM

Historical data = 1,100 PM

1122=µ

COSYSMO = 100 PM

Historical data = 110 PM

9=σ 111=σ


Person Months Risk

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

55%

60%

65%

70%

75%

80%

85%

90%

95%

100%

0 25000 50000 75000 100000

Person Months

Risk (= Prob. That Actual Person Months

Will Exceed Indicated, X-Axis, Figure)

Person Months Confidence (Cumulative Probability)

0%5%10%15%20%25%30%35%40%45%50%55%60%65%70%75%80%85%90%95%100%

0 25000 50000 75000 100000

Person Months

Cumulative Probability of Person

Months

Risk/Confidence Plots

Valerdi, R., Gaffney, J., “Reducing Risk and Uncertainty in COSYSMO Size and Cost Drivers: Some Techniques for Enhancing Accuracy,”

5th Conference on Systems Engineering Research, March 2007, Hoboken, NJ.


Systems Engineering Risk

Assessment


Reuse in Systems Engineering

• Let:

– NN=count of new requirements;

– NM=count of modified requirements;

– NR=count of reused requirements;

– ND=count of deleted requirements

• Then, NT=total count of

requirements=NN+NM+NR+ND

• Define: ET=NN+(cM*NM)+(cR*NR)+(cD*ND);

– Where: cM, cR, cD, are the unit effort values for a

modified, a reused, or a deleted requirement

respectively, relative to that for a new requirement.


Incremental Software Systems

Engineering ROI

0

2

4

6

8

10

Incremental RESL Improvement Investment

10,000KSLOC

RO

I

L N N H H VH VH EHVL L-2

1,000KSLOC

100KSLOC

10KSLOC

Boehm, B., Valerdi, R., Honour, E., “The ROI of Systems Engineering,” Systems Engineering, 2008. (in press)


How Much Architecting is

Enough?

0

20

40

60

80

100

0 10 20 30 40 50

% time added for architecture and risk resolution

% t

ime

ad

ded

to

overa

ll s

ch

ed

ule

10,000KSLOC

X

X

X

X

X

X

X

X

X

X

X

XX

XX

X

% of project schedule devoted to initial architecture and risk resolution

Added schedule devoted to rework (COCOMO II RESL factor)

Total % added schedule

+

+

+

+

++

Sweet Spot

++

+ ++

++

++

+

100KSLOC

10KSLOC


Systems Engineering Leading

Indicators

• Requirements Trends

• System Definition Change Backlog Trend

• Interface Trends

• Requirements Validation Trends

• Requirements Verification Trends

• Work Product Approval Trends

• Review Action Closure Trends

• Risk Exposure Trends

• Risk Handling Trends

• Technology Maturity Trends

• Technical Measurement Trends

• Systems Engineering Staffing & Skills Trends

• Process Compliance Trends

Requirements Trends

TIME

Requirements Growth Trends

TIME

NUMBER OF REQUIREMENTS

JulyMar Apr May JuneFebJan

LEGEND

Planned Number

Requirements

Actual Number

Requirements

Aug Sep Oct Nov Dec

Projected Number

Requirements

SRR PDR CDR ….

Corrective

Action Taken

Rhodes, D., Roedler, G., Systems Engineering Leading Indicators Guide, V. 1.0, June 15, 2007


Gelperin, D., New Approaches to Specifying Clear Testable System Requirements, Requirements Quarterly, RQ36, pp. 17, June 2005.


Summary

• Systems Engineering Economics– developing a new paradigm that encompasses an

economics view of systems engineering to achieve

measurable and predictable outcomes while

delivering value to stakeholders.

• Next segment– Research Report: Tsoline Mikaelian, Managing

Uncertainty in Complex Systems and Enterprises

using Real Options

– Brief Overview: Chris Roberts, Harmonizing Systems

Engineering of Enterprises and their Context

QUESTIONS

http://seari.mit.edu

[email protected]

research profile systems engineering economics

Documents