cocomo forum 2015 bob ferguson © 2015 carnegie mellon university distribution statement a: approved...

COCOMO Forum 2015Bob Ferguson© 2015 Carnegie Mellon University

Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Making Connections to Existing CERs with QUELCE*

COCOMO ForumNovember 2015Bob Ferguson

*Quantifying Uncertainty in Early LifeCycle Estimation

3COCOMO Forum 2015Bob Ferguson© 2015 Carnegie Mellon University


Session Agenda

Overview of QUELCE

CERs1 in Estimation

Connecting the QUELCE BBN2 to CERs – Glue Nodes

Summary

1 Cost Estimating Relationships2 Bayesian Belief Network



QUELCE

QUELCE is a method for analyzing the effects of changing assumptions and other risks inherent in product development.

Rather than focusing on the consequences of each risk, the method tries to analyze the cascading effects of multiple changes.

The current presentation describes ways to connect the QUELCE probability network (BBN) to many different cost estimation relationships.



QUELCE Intended to Inform CER Inputs (Example)Acquisition EnvironmentA.1 Acquisition Category (ACAT) StatusA.2 Governance, Policies, and StandardsA.3 External Interdependencies / CoordinationA.4 External StakeholdersA.5 External Events, FundingA.6 Other: Acquisition EnvironmentA.7 Capability DefinitionAcquisition ManagementB.1 Acquisition StrategyB.2 ContractingB.3 Management StructureB.4 Program Scope, RequirementsB.5 BudgetB.6 ScheduleB.7 StaffingB.8 Facilities, Support Technology, and EquipmentB.9 Program Information ManagementB.10 Program-Contractor Performance B.11 Other: Acquisition ManagementTechnical SolutionC.1 Conceptual Design / SpecificationC.2 System Architecture and DesignC.3 Production and ConstructionC.4 Certification and AccreditationC.5 Deployment, Operations, and SupportC.6 Technology Maturity / ReadinessC.7 Estimated Complexity / DifficultyC.8 Supply Chain ProductsC.9 Other: Engineering Solution / Work Products

Need to map

QUELCE change drivers

(from the left)

to the CER input

parameters (on the right)



Modeling Uncertainty

Complexity Reduction

1. Use QUELCE

Repository to Populate Driver State

Matrix

3. Develop BBN Model and

Assign Conditional

Probabilities to BBN Model

4. Calculate Cost Factor

Distributions for Program Execution

Scenarios

5. Monte Carlo Simulation to Compute Cost

Distribution

2. Evaluate Cause and Effect

Relationships and Reduce Explosion via Dependency Structure Matrix

Overview of QUELCE Method

Legend:

QUELCE Change Driver Repository

Change Driver Nominal State Alternative States

Scope Definition

Stable Users added Additional (foreign) customer

Additional deliverable (e.g. training & manuals)

Production downsized

Scope Reduction (funding reduction)

Mission / CONOPS defined New condition New mission New echelon Program

becomes Joint

Capability Definition

Stable Addition Subtraction Variance Trade-offs [performance vs affordaility, etc.]

Funding Schedule

Established Funding delays tie up resources {e.g. operational test}

FFRDC ceiling issue

Funding change for end of year

Funding spread out

Obligated vs. allocated funds shifted

Advocacy Change

Stable Joint service program loses particpant

Senator did not get re-elected

Change in senior pentagon staff

Advocate requires change in mission scope

Service owner different than CONOPS users

Closing Technical Gaps (CBA)

Selected Trade studies are sufficient

Technology does not achieve satisfactory performance

Technology is too expensive

Selected solution cannot achieve desired outcome

Technology not performing as expected

New technology not testing well

● ~~~~ ~~~~ ~~~~ ● ~~~~ ~~~~ ~~~~ ~~~~ ● ~~~~ ~~~~ ~~~~ ~~~~ ~~~~ ~~~~

1. Driver State Matrix

Change Drivers - Cause & Effects Matrix

Mis

sion

/ C

ON

OPS

Cha

nge

in S

trat

egic

Vis

ion

Cap

abilit

y D

efin

ition

Adv

ocac

y C

hang

e

Clo

sing

Tec

hnic

al G

aps

(CBA

)

Build

ing

Tech

nica

l Cap

abilit

y &

Cap

acity

(CB

A)

Inte

rope

rabi

lity

Sys

tem

s D

esig

n

Inte

rdep

ende

ncy

Func

tiona

l Mea

sure

s

Sco

pe D

efin

ition

Func

tiona

l Sol

utio

n C

riter

ia (m

easu

re)

Fund

ing

Sche

dule

Acq

uisi

tion

Man

agem

ent

Prog

ram

Mgt

- C

ontr

acto

r Rel

atio

ns

Proj

ect S

ocia

l / D

ev E

nv

Pro

g M

gt S

truc

ture

Man

ning

at p

rogr

am o

ffice

Mission / CONOPS 3 3 0 6 0Change in Strategic Vision 3 3 3 2 2 2 2 2 3 2 3 2 29 0Capability Definition 3 0 2 1 1 0 0 2 2 2 0 1 0 2 0 0 16 0Advocacy Change 2 1 1 1 1 6 0Closing Technical Gaps (CBA) 2 1 3 1 2 2 1 2 2 1 1 2 1 0 2 2 1 1 2 2 1 2 34 0Building Technical Capability & Capacity (CBA) 1 1 2 1 2 2 1 2 3 2 2 1 2 2 1 1 1 27 0Interoperability 1 2 1 1 1 1 1 1 1 1 2 1 1 2 1 1 3 1 2 2 2 29 1Systems Design 1 2 2 2 2 1 1 1 1 1 2 2 3 21 3Interdependency 1 2 2 1 1 1 1 1 1 1 1 1 2 1 2 2 1 1 1 1 1 2 2 3 33 5Functional Measures 2 2 2 1 1 1 1 1 1 1 2 1 16 0Scope Definition 1 1 3 5 0Functional Solution Criteria (measure) 1 2 2 1 1 2 1 10 1Funding Schedule 1 1 2 1 5 0Acquisition Management 1 1 2 3 1 1 2 2 1 1 1 2 1 19 2Program Mgt - Contractor Relations 1 1 2 1 1 1 1 2 2 12 2Project Social / Dev Env 1 1 1 2 2 1 1 2 1 1 1 14 2Prog Mgt Structure 1 2 1 2 6 1Manning at program office 2 1 2 5 2Scope Responsibility 1 1 1 1 1 1 6 5Standards/Certifications 1 1 1 1 1 1 3 1 10 2Supply Chain Vulnerabilities 1 1 1 1 2 1 7 4Information sharing 1 1 1 1 1 1 1 7 3PO Process Performance 2 2 4 0Sustainment Issues 0 0Contract Award 0 0Production Quantity 2 2 0Data Ownership 2 2 0Industry Company Assessment 0 0Cost Estimate 0 0Test & Evaluation 0 0Contractor Performance 2 2 0Size 0 0Project Challenge 0 0Product Challenge 0 0Totals 0 0 6 4 1 9 5 12 8 7 7 13 4 10 15 18 7 7 8 8 14 17 17 15 12 9 10 13 11 20 19 5 5 17 0Below diagonal 0 0 0 1 1 4 4 4 1 2 0 3 1 3 2 2 3 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Effects

Causes

2. Dependency Structure Matrix

3. BBN Model

4. Cost Factor Distributions by Scenario of Change

5. Monte Carlo with Cost Estimation Tools

Drivers XL VL L N H VH XH Product ProjectScale Factors

PREC 6.20 4.96 3.72 2.48 1.24 0.00 <X>FLEX 5.07 4.05 3.04 2.03 1.01 0.00 <X>RESL 7.07 5.65 4.24 2.83 1.41 0.00 <X>TEAM 5.48 4.38 3.29 2.19 1.10 0.00 <X>PMAT 7.80 6.24 4.68 3.12 1.56 0.00 <X>

Effort MultipliersRCPX 0.49 0.60 0.83 1.00 1.33 1.91 2.72 XRUSE 0.95 1.00 1.07 1.15 1.24 XPDIF 0.87 1.00 1.29 1.81 2.61 XPERS 2.12 1.62 1.26 1.00 0.83 0.63 0.50 <X>PREX 1.59 1.33 1.12 1.00 0.87 0.74 0.62 <X>FCIL 1.43 1.30 1.10 1.00 0.87 0.73 0.62 <X>SCED 1.43 1.14 1.00 1.00 1.00 <X>

SRDRDAESSARS



CERs in EstimationCAPE recommends four cost estimation methods:• Analogy: (early in program lifecycle -> MS A)• Parametric (Statistical) - CERs : (early in program lifecycle -> MS B)• Engineering (“bottoms up”): (MS C and later)• Actual Costs: (LRIP and later)

Different Types of Cost Estimation Relationships (CERs)• Elements of scope – things we build• Required process elements associated to scope• Required purchases

Scope• Estimate of volume, weight, size, capacity based on required deliverable

characteristics. This may be LOC, Function Points, requirements, etc…• Product characteristics (quality attributes) such as speed of performance• Quantity of deliverable

Required Process Elements Associated to Scope• Certifications



Methods Employed in CERs

Scope Estimates• CER is a parametric model - an equation used to estimate a given cost

element using an established relationship with one or more independent variables.• Analogy to past projects can be used to bound regression parameters.• Expert judgment recommendations based on experience and data.

Required Processes Associated to Scope• Built from the scope estimates by using internal history and a proportional

formula.• Some of the required processes will be covered by a parametric tool.

Other Required Processes• Typically based on assigning a resource cost for duration of schedule or

duration of development for a specific element of scope.Purchases• Fixed cost based on count of need.



What Are Glue Nodes?

Definition• A glue node is a way to connect the output from the BBN in step 3 of the

QUELCE method to the CERs for your estimation model.Rationale• Glue nodes are used to incorporate the front end uncertainty and scenario

analyses afforded by QUELCE in a practical manner by mapping the output of the BBN into existing parameters in the CER.

Example• The process elements of scope are uncertain based on need for innovation,

product complexity, etc. Hence, we attempt to identify the related nodes in the model. Those nodes are connected to a glue node.• The probability distribution is calculated as the joint probability distribution of

the parent nodes.• The glue nodes are mapped to selected parameters of the CER.



BBN Model with three Glue Nodes

Product Challenge

Program Challenge

Scope (Size)



How to Use Glue Nodes

Each element of scope is likely to require different glue nodes

Schedule• The overall schedule will drive significant costs not associated to a specific

scope, so different CERs are employed.• Schedule will be specific to a lifecycle.

Use• The BBN must be connected to the selected CERs via glue nodes.• The glue node embodies the joint probability of several other BBN nodes.• The glue nodes will be specific to the CER.• The BBN itself usually remains the same across different CERs.

BBN Analysis• Bayesia and AgenaRisk are example tools that support this function.• It is easy to add the needed joint probability distributions.



COCOMO II Example

We used 3 glue nodes

• Size – Mike Ross code growth provides a useful size range – Median and a high percentile value are calculated

• Product Challenge– We would like to think of product domain as providing a critical

combination such as real-time-embedded very high reliability– Also novelty or invention adds to complexity

• Program Challenge– Factors associated to contractor skill, program office flexibility, and

number of contractors



COCOMO II Example - Glue Nodes

COCOMO parameter mapping --• Select parameters for “Product Challenge” and “Program Challenge.”• Range is limited for each of the selected parameters.• Map BBN glue node values to COCOMO parameter values. • Some parameter scales had to be reverse ordered so that the effect had the

appropriate direction

Product Challenge factors (1=low…5=high)COCOMO Parameter XL VL L N H VH EHScale Factors PREC 1 3 5

FLEX 1 2 3 5RESL 1 2 3 4 5

Effort Multipliers RCPX 1 2 3 4 5PDIF 1 5RUSE 1 3 5

Project Challenge factors (1=low…5=high)COCOMO Parameter XL VL L N H VH EHScale Factors TEAM 1 3 5

PMAT 1 2 3 4 5Effort Multipliers PERS 1 3 5

PREX 1 2 3 4 5FCIL 1 3 5SCED 1 3 5

MappedCOCOMO

value

BBN Outputs



Product Challenge FactorsCOCOMO Parameter XL VL L N H VH EHScale Factors PREC 1 2 3 4 5

6.2 4.96 3.72 2.48 1.24 0FLEX 5 4 3 5.07 4.05 3.04 RESL 2.53 TEAM PMAT

Effort Multipliers PERS 5 4 3 2 1 1.62 1.26 1.00 0.83 0.63 0.49RCPX 1 2 3 4 5 0.49 0.6 0.83 1 1.33 1.91 2.72PDIF 1 1 3 4 5 0.87 1 1.29 1.81 2.61PREX FCIL RUSE 1 2 3 4 5 0.95 1 1.07 1.15 1.24SCED

Italicized values are derived from the BBN; corresponding COCOMO values are directly below the BBN values.



Program Challenge FactorsCOCOMO Parameter XL VL L N H VH EHScale Factors PREC

FLEX RESL Nominal 2.53 TEAM Joint Norm 3.95 0.99 PMAT 1 2 3 4 5 reversed values 0 0.91 1.82 2.73 3.64

Effort Multipliers PERS fixed range not modeled 0.83 1.62 RCPX PDIF PREX fixed ranges 1.12 0.74 FCIL 1.1 0.87 RUSE SCED reversed values 1-2 3 5 1 1.14 1.43

Italicized values are derived from the BBN; corresponding COCOMO values are directly below the BBN values.



Monte Carlo Modeling

MappedCOCOMO

value

BBN Outputs

Each CER parameter is assigned a distribution

based on the Glue node distribution

Monte Carlo simulation then enables a distribution of cost

using the CER with input distributions



SummaryAcquisition EnvironmentA.1 Acquisition Category (ACAT) StatusA.2 Governance, Policies, and StandardsA.3 External Interdependencies / CoordinationA.4 External StakeholdersA.5 External Events, FundingA.6 Other: Acquisition EnvironmentA.7 Capability DefinitionAcquisition ManagementB.1 Acquisition StrategyB.2 ContractingB.3 Management StructureB.4 Program Scope, RequirementsB.5 BudgetB.6 ScheduleB.7 StaffingB.8 Facilities, Support Technology, and EquipmentB.9 Program Information ManagementB.10 Program-Contractor Performance B.11 Other: Acquisition ManagementTechnical SolutionC.1 Conceptual Design / SpecificationC.2 System Architecture and DesignC.3 Production and ConstructionC.4 Certification and AccreditationC.5 Deployment, Operations, and SupportC.6 Technology Maturity / ReadinessC.7 Estimated Complexity / DifficultyC.8 Supply Chain ProductsC.9 Other: Engineering Solution / Work Products

Product Challenge

Program Challenge

Scope (Size)

Change Driver Scenarios modeled to inform CER inputs leading to

cost distributions



Contact Information

Robert FergusonSenior Member of Technical StaffSoftware Solutions Division, SEAPTelephone: +1 412-268-1121Email: [email protected]

U.S. MailSoftware Engineering InstituteCustomer Relations4500 Fifth AvenuePittsburgh, PA 15213-2612USA

Webwww.sei.cmu.eduwww.sei.cmu.edu/contact.cfm

Customer RelationsEmail: [email protected]: +1 412-268-5800SEI Phone: +1 412-268-5800SEI Fax: +1 412-268-6257



BACKUP SLIDES



Change Drivers

Definition: A change driver is any assumption or decision that has the potential to change.Decisions: • Selection of suppliers and contractors including test facilities• Selection of technology or component

Assumptions:• A technology, currently at TRL 6, will mature sufficiently by Milestone C.• A KPP goal (key performance parameter) can be achieved

(KPP goals for software defined radios were not achieved).• Scope and feature definitions are fixed by Milestone B.



Change Drivers and Dependency Structure Matrix

“Change Drivers” • The specific assumptions and risks identified by program subject matter

experts (SME)• SMEs are asked to make estimates of frequency of change

The Dependency Structure Matrix (DSM)• Relates changes by asking whether a second change event would be a

consequence of an earlier one.• The matrix tends to be very large and must be reduced and reorganized into

a triangular matrix in order to create the probability network.• SEI has an automated tool that makes this process much easier. It now takes

about 30 minutes or less.



Modeling Schedule Impacts

QUELCE Addresses Schedule Slip• Use lifecycle synchronization points such as PDR and CDR.• Delays are common and usually have identifiable causes that have been

modeled in the BBN.• The critical concern for schedule is whether specific change events occur that

may cause schedule delays.

Synchronization Concerns• A critical dependency during the lifecycle is the need to synchronize

development teams and tasks in order to do any form of product validation. Without this validation exercise, no forecast of delivery is trustworthy.• Schedule forecasts of deliverables are essential to the accuracy of the cost

estimate.



Schedule Example: Causes of Schedule Delay

Supply Chain • Obsolete parts• Change of supplier

Technology Readiness and Manufacturing Readiness• e.g. GN (gallium nitride) is a potential replacement for Gas (gallium arsenide).

Both electrical characteristics and manufacturing problems still need study. Software will interact with electrical characteristics.

Contractor – Program disputes• Approval of engineering change request• CDRL review delays

Limited availability of specific resources– Test equipment and facility

Funding Delays… several others that are familiar to experienced program managers



Bayesian Belief Network

Directed Graph• A directed graph is constructed from the triangular matrix.• Each node gets a probability of occurrence and a probability of change effect

from each parent node.• Probabilities are generated by SMEs and by data from past programs.

Stability of the BBN• Our expectation is the BBN should change rarely during most programs.• New risks or assumptions not tested would add a node.• As time passes, more and more nodes would be set to zero expectation of

change.• BBN probabilities would consistently narrow over the course of time.

cocomo forum 2015 bob ferguson © 2015 carnegie mellon university distribution statement a: approved...

Documents