Learning Objectives
3Cost Estimating
OVERALL: Given a DoD IT/SW system scenario, develop the software program office lifecycle cost and schedule effort projections for budget purposes.• Describe the DoD processes for generating program
office cost and schedule estimates.• Describe the basic cost and schedule estimating
methodologies used to budget for a software-reliant system.
• Given a DoD IT/SW scenario, develop cost and schedule budget estimates.
• Determine the impact of COTS and other cost drivers on achieving an accurate estimation of Program costs.
Today we will learn to:
Lesson Overview
4Cost Estimating
• Program Office Budgeting• Software Cost Estimating Intro• Costing Exercise/Demo• COTS/ERP Cost Estimation• JTAMS CE Exercise
Lesson Plan
Lesson Overview
5Cost Estimating
• Program Office Budgeting- Life cycle costs- Building the program budget- Required cost estimates
• Software Cost Estimating Intro• Costing Exercise/Demo• COTS/ERP Cost Estimation• JTAMS CE Exercise
Lesson Plan Status
6Cost Estimating
• Life cycle cost (LCC) “The total cost to the government of a program over its full life, including costs for research and development, testing, production, facilities, operations, maintenance, personnel, environmental compliance, and disposal.”
• Costs are viewed in differing ways depending on the stakeholder involved (Program Office, Congress, Contractor):
Life cycle cost categoriesFunding appropriation Work breakdown structure (WBS)
The Life Cycle Cost
7Cost Estimating
Ref: DoD 5000.04-M-1, Cost Analysis Guidance and Procedures
Congressional appropriation categories
RDT&E—Research, Development, Test and
EvaluationPROC—Procurement
O&M—Operations and Maintenance
MILCON—Military ConstructionMILPERS—Military Personnel
DoD life-cycle cost categories
R&D—Research & DevelopmentInvestment
O&S Operating and SupportDisposal
Life-cycle cost vs. Funding Appropriation
8Cost EstimatingRef: MIL-HDBK-881D, 9 April, 2018
• The WBS serves as a basis for:- Program and technical planning, cost estimating, resource allocations,
performance measurements, and status reporting. • It displays the total system as a product-oriented hierarchy composed
of hardware, software, services, data, and facilities; and relate the elements of work to one another, as well as to the end product.
• Where used: Cost breakouts by WBS elements are useful to the program office and contractors in managing the program.
By Work Breakdown Structure (WBS)
9Cost Estimating
All ACAT I programs require an ICE
• ICE (Independent Cost Estimate)- Done by: Director, Cost Assessment and Program
Evaluation (CAPE)• CCE (Component Cost Estimate)
- Component Cost Agency• PLCCE (Program Life Cycle Cost Estimate)
- Program Office• EA (Economic Analysis)
- Requirement for MAIS programs
Required Cost Estimates
10Cost EstimatingNote: FY 2014 constant dollars per DoDI 5000.02
ACAT Reason Decision AuthorityACAT 1 (MDAP)
>$480M RDT&E, >$2.79B PROCMDA designation as special interest
ACAT ID: DAE or asDelegated
ACAT IC: Head of the DoDComponent or, if delegated, the CAE
ACAT IA (MAIS)
>$40M Single year cost>$165M from Material Solution Analysis through full deployment>$520M from Material Solution Analysis through full useful lifeMDA designation as special interest
ACAT IAM: DAE or asDelegated
ACAT IAC: Head of the DoD Component or, if delegated, the CAE (not further delegable)
ACAT II >$185M RDT&E or >$835M PROC CAE or as designee
ACAT III Does not meet criteria for ACAT II or above AIS that is not a MAIS
Designated by CAE
ACAT Level Drives Cost Estimate Requirements
11Cost Estimating
Ref: Defense Acquisition Guidebook Chapter 3.6 - Economic Analysis (MAIS Only)
Economic Analysis is IT system focused
• Independent Cost Estimate- An independent estimate of program life-cycle costs
• Economic Analysis- Determining the best acquisition alternative by assessing
the net costs and benefits of the proposed program relative to the status quo.
Cost Estimate vs. Economic Analysis
12Cost Estimating
Reference: DoDI 7041.3, Economic Analysis for Decision Making
Alternatives1. Status quo or
current functional baseline
2. New acquisition3. Leasing 4. Modification of
existing assets
ECONOMIC ANALYSIS• Objective• Assumptions• Alternatives• Costs and Benefits of each
alternative• Comparison of the costs
and benefits of each alternative
• Sensitivity and Uncertainty Analyses
• Results and Recommendations
Elements of an Economic Analysis
Lesson Overview
14Cost Estimating
• Program Office Budgeting
• Software Cost Estimating Intro- Overview- Estimating methods- Parametric Modeling Tool
• Costing Exercise/Demo• COTS/ERP Cost Estimation• JTAMS CE Exercise
Lesson Plan Status
Software is Here to Stay!
0.00
5.00
10.00
15.00
20.00
25.00
30.00
F-22 Raptor(2005)
Boeing 787(2009)
F-35 Fighter(2015?)
Line
s of
Cod
e in
Milli
ons
Aircraft
0.00
10.00
20.00
30.00
40.00
50.00
60.00
Windows 2000(2000)
Windows XP(2001)
Windows Vista(2006)
Line
s of
Cod
e in
Milli
ons
Operating Systems
0.00
100.00
200.00
300.00
400.00
500.00
600.00
Facebook (withoutbackend code)
Healthcare.gov (2013)Line
s of
Cod
e in
Milli
ons
Web Pages
0.00
20.00
40.00
60.00
80.00
100.00
120.00
Chevy Volt (2010) Software in TypicalNew Car (2013)
Line
s of
Cod
e in
Milli
ons
Automobiles
15Cost Estimating
18Cost Estimating
Poor at Estimating SW Costs
• Cost overruns decreased from 54% in 2008 to 46% in 2010.
• Time overruns also have gone down, from 79% in 2008 to 71% in 2010. The high point in time overruns was 2004 (84%).
Source: Standish Group Chaos Database (2004)
So, what happened after 2004? ……
19Cost Estimating
• Intangible nature• Complex• Ill defined requirements• Code Growth during development• Optimism of Reuse• Optimism for COTS solutions• Software Tool unavailability/inexperience• Poor interface design
Causes of Poor Cost Estimating
21Cost Estimating
• Compares a new or proposed system with one other system in which the form, fit, and function are alike
• Done early in the life cycle when costing data points are lacking
• Used as a “sanity check” for other type estimates• Requires identifying “adjustment factors” and “cost
to cost relationships”• Pros: Fast and Inexpensive, easy to change• Cons: High degree of cost risk
Analogy
22Cost Estimating
Adjustment Factors:Inflation: $250,000 x 1.2184 = $304,601Complexity: $305,000 x 1.05 = $319,831
Estimating Cost of a 2017 Ferrari 612 GTO Skaglietti
Analogy Example
Available Data:• $250,000 in 2009• Inflation 2.5% over 8 yrs (1+.025)8 =
1.2184• Complexity/Innovation 5% (expert opinion)
23Cost Estimating
Usually done using software-based tools
1. Model derived from historical data of many developmental efforts.
2. Determine historical CERs (Cost estimating relationships) and SERs (Schedule estimating relationships).
3. Apply the CERs and SERs to a planned project.
Parametric Model
24Cost Estimating
The parametric model gives us a great understanding of the drivers of software cost.
Let’s dive into this model more deeply…
• Pros- Provides excellent visibility of software cost drivers - Computer-based tools allow for excellent conducting
“what-if” analyses• Cons
- Constrained by the amount and quality of the data- Sensitivity of the tool requires an expert assessment of
expected cost drivers- Must estimate project size prior to using the model (expert
judgment?)
Parametric Pros and Cons
26Cost Estimating
ESLOC = New Code + Modified Code + Reused Code
Cost Factors in Software Development
27Cost Estimating
Each software estimating model calculates ESLOC a little differently
Measures the size of a SW program by counting the number of lines of code that are designed, documented, and tested, but … which ones count?• Executable (Yes, but don’t count End or debug Statements)• Data Declarations (yes)• Compiler Directives (yes)• Format Statements (yes)• Comments (nope)
Equivalent or Effective Lines of Code = Resource estimates based on physical LOC for modified code and reusable code cannot account for the additional resource demands to reverse-engineer, test and integrate, so …we have to account for• New code• Modified code• Reused code and • Off-the-shelf components
Estimating Size
28Cost Estimating
• Required Software Reliability
• Data Base Size • Product Complexity• Developed for Reuse • Documentation Match to
Life-Cycle Needs
Cost Factors in Software Development
29Cost Estimating
• Execution Time Constraint • Main Storage Constraint • Platform Volatility
Cost Factors in Software Development
30Cost Estimating
• Analyst Capability • Programmer Capability • Personnel Continuity • Applications Experience • Platform Experience • Language and Tool
Experience
Cost Factors in Software Development
31Cost Estimating
• Use of Software Tools • Multisite Development• Required Development
Schedule
Cost Factors in Software Development
32Cost Estimating
• Precedentedness• Development
Flexibility• Architecture / Risk
Resolution• Team Cohesion • Process Maturity
Cost Factors in Software Development
33Cost Estimating
How do these effect the cost of software development?
What management decisions will reduce costs?
Cost Factors in Software Development
34Cost Estimating
The result of analyzing 100’s of software development efforts
(In Person Months)
Effort MultipliersNominal Values = 1Scale Factors
COCOMO IIA = 2.94B = 1.1 (Nominal)C = 3.67D = 0.28
COCOMO II—How it Works
Lesson Overview
35Cost Estimating
• Program Office Budgeting• Software Cost Estimating Intro
• Costing Exercise/Demo• COTS/ERP Cost Estimation• JTAMS CE Exercise
Lesson Plan Status
36Cost Estimating
• What if we doubled the size of the code?
• Example: How many person-months will my project take if the software size is 100 KSLOC and all cost drivers (scale factors and effort multipliers are nominal. Using COCOMO historical parameters we know that A =2.94 and B=1.1 and nominal values for EAF are all normalized to 1.
How does size effect the effort?
37Cost Estimating
• Scenario- As a result of uncertainty with funding from the
government for software development projects, your contractor is reporting personnel turnover moved from 12% per year to 48% per year. You originally estimated turnover to be at 12%. Given that all other effort multipliers (parameters) and scale drivers are nominal how may this effect the cost of your SW project?
Parametric Model Exercise 1 - Effort Multipliers(Ratio Technique)
38Cost Estimating
Refer to COCOMO Cost Estimation
Workbook, p34–35, 38
29% more
• What Effort Multiplier parameter does this apply to? • What value applies for 12%? 48%? • How much more will it cost?
Parametric Model Exercise 1 - Effort Multipliers Answer (Ratio Technique)
39Cost Estimating
The COCOMO II Cost Estimating Tool – so we don’t worry about the math!!
1. Open file “DAUCostApplication”2. Select “New” for Project Type3. Highlight Project 1 folder4. Click on new software item5. Highlight Software Item 1
COCOMO II Software Tool
41Cost Estimating
Scale Factors cause EXPONENTIAL increases to development effort
From a common sense viewpoint, what can we do to manage these?
• Precedentedness• Development Flexibility• Architecture / Risk Resolution• Team Cohesion • Process Maturity
Parametric Model-Scale Factors
42Cost Estimating
Your program office has been planning a new C4I capability for the Navy. The current estimate for the SW is 200KSLOC and 26 months at a cost close to $8M. This assumes nominal values for all cost drivers and scale factors and average SW Labor Rates of $8K per person-month. You, however, have a plan to significantly improve team cohesion from moderately cooperative to highly cooperative. Using the COCOMO Cost Estimation Workbook answer this question:
- What does it mean to move the team from nominally to highly cooperative?
• Using the online COCOMO modeling tool, answer these questions:- How will this change the cost?- Which scale drivers are manageable by the PMO to lower costs?
Run a sensitivity analysis (Instructor assignment)- What scale driver has the biggest effect on the cost of software?
Parametric Model Demo 2—Scale Factor
43Cost Estimating
1. Which three multiplicative cost drivers account for the largest variation in costs? Hint: Use Cost Factor Cheat Sheet.
2. Using a 350 KSLOC SW with no reuse or modification of code and $8K/person-month cost, what are the costs and schedule with factors rated (Hint: Use CE Tool to calculate):- all nominal - all 3 identified factors optimal (less effort/lower cost) - all 3 identified factors pessimistic (greater effort/higher
cost)
Costing Exercise 2: Biggest Cost Drivers
44Cost Estimating
Source: “Modern Tools to Support DoD Software Intensive System of Systems Cost Estimation”, By Jo Ann Lane and Barry Boehm, University of Southern California, 2007
• Software Engineering- COCOMO II — Boehm eta, 2005- CostXper—CostXpert Group, 2003- Price-S (PRICE, 2006)- SEER-SEM (Galorath, 2001)- SLIM (QSM, 2006)
• COTS Integration- COCOTS (Abts, 2004)- SEER-SEM (Galorath, 2001)
Other Cost Estimation Apps
Lesson Overview
45Cost Estimating
• Program Office Budgeting• Software Cost Estimating Intro• Costing Exercise/Demo
• COTS/ERP Cost Estimation-Commercial off the Shelf (COTS)
Software-Enterprise Resource Planning (ERP)
Software• JTAMS CE Exercise
Lesson Plan Status
46Cost Estimating
• Can provide “out-of-the-box functionality but …
• Consider- May be difficult to customize and integrate (glue code)- Might need additional testing- Upgrades might require re-integration (rewrite glue code)- Must still perform requirements definition, design and test- May be incompatible with other software packages- Use proprietary languages (hard to support in-house)- Need training and access to documentation- New releases (may cause unpredictable problems—time/$
to find and fix)
COTS
47Cost Estimating
Source: Pat Phelan, Estimating the Time and Cost of ERP Implementation Projects Is a 10-Step Process (Stamford, Conn: Gartner Inc., Apr. 27, 2006, p. 3.
• Special kind of COTS—business support system• Consider
- Contains thousands of packages—built from database tables—that need to be configured to match end business processes = requires a deep understanding of existing business operating processes
- Integrating and testing software links- Interfacing with legacy systems- Customizing—can be costly and should be carefully managed - Converting and analyzing data—did you account for converting data
into new format? Data warehouses—renaming? Programming customized?
- Following up installation—is your team changing?- Training
• Through 2006, actual time spent on 40% of ERP implementations will exceed original estimates by at least 50% (0.7 probability)*
ERP
48Cost Estimating
• Estimates are necessary to - Establish systems engineering and software development
budgets - Facilitate trade-offs/negotiations
• Estimation is not easy • Different approaches have different inherent problems • Best way to estimate is to use a combination of
techniques • Local calibration of parametric models provide best
accuracy
Cost Estimating Recap
Lesson Overview
49Cost Estimating
• Software Cost Estimating Intro• Costing Exercise• Program Office Budgeting
• JTAMS CE Exercise
Lesson Plan Status
50Cost Estimating
1. Task 1. (20 min) Using the chart (Individual Estimate of Cost Factors), individually analyze the scenario and estimate the value of each factor (Very Low-Extremely High). (Use the COCOMO II Cost Estimating Workbook/Factor cheat sheet used in the lesson.)
2. Task 2. (10 min) Compare your results with that of your team. Discuss and decide on an agreed set of parameters. Record your team results and justification. (whiteboard)
3. Task 3. (10 min) Using the COCOMO II modeling tool calculate effort, schedule and cost for your team estimate – place these values on your Whiteboard.
4. Task 4. (15 min) Compare your team results with the other teams. Where there are differences of opinion, share your insights as you discuss/defend your results. The object is to have a clearer understanding of the ratings so we all eventually agree. (Instructor facilitated discussion)
If time permits:1. Answer the program manager’s challenge. Record results in worksheet.2. Identify total cost savings using CE Tool if all suggested improvements were
implemented? Record results in worksheet.
JTAMS CE Exercise
Summary
51Cost Estimating
Today we learned to:
OVERALL: Given a DoD IT/SW system scenario, develop the software program office lifecycle cost and schedule effort projections for budget purposes.• Describe the DoD processes for generating program
office cost and schedule estimates.• Describe the basic cost and schedule estimating
methodologies used to budget for a software-reliant system.
• Given a DoD IT/SW scenario, develop cost and schedule budget estimates.
• Determine the impact of COTS and other cost drivers on achieving an accurate estimation of Program costs.
53Cost Estimating
Ref: DoDD 5000.04-M-1, Cost Analysis Guidance and Procedures
• Program Acquisition Cost- Development Costs
- RDT&E- Procurement Costs
- Weapons System Costs- Flyaway Cost
- Procurement (PME, SE/PM, STE)- Procurement (Support Items)
- Procurement (Initial Spares & Repair Parts)- MILCON (Facilities)
• O&S (MILPERS, O&M, Disposal)
More Costing Terminology
55Cost Estimating
Ref: DoD 5000.4 M, "DoD Cost Analysis Guidance and Procedures," Sec1
More than just documenting the life cycle costs
• Cost Analysis Requirements Description (CARD) Requirement:- Initial submission required at MS A and MS B- Updates required at Development RFP, MS C and Full Rate- Production/Fielding Decision- Due any time an Independent Cost Estimate or Economic
Analysis is required• Serves as the basis of all cost estimates• Prepared by the PMO and approved by the PEO
Building a program’s budget
56Cost Estimating
Ref: DoDD 5000.4 M, "DoD Cost Analysis Guidance and Procedures," Sec1
CARD• Program description• Program operational concept• Program data management
requirements• Program quantity requirements• Program manpower
requirements• Program fielding strategy• Program milestone schedule• Program acquisition plan or
strategy
SOURCE DOCUMENTSTechnical Readiness Assessment
(TRA)Capability Docs (ICD, CDD, CPD)
Acquisition Strategy (AS)Life Cycle Sustainment Plan
(LCSP)Technical Evaluation Strategy/
Test and Evaluation Master Plan (TES/TEMP)
Manpower EstimateSystems Engineering Plan (SEP)
CARD Content
57Cost Estimating
80% of F-22 functionality and other modern day A/C depend on SW
• The role of software as the most critical part of weapons systems is growing as is the % of DoD Budget
• $28B annually to support operations, maintenance and modernization of various DoD IT systems: - $9B for C3I systems - $19B for business systems..
Why Care About SW?
58Cost Estimating
Ref: Teaching Note “Budget Exhibits”
Used to report appropriations for FYDP by program element
“Code Word”“P-Forms” or “P-Docs”“R-Forms” or “R-Docs”“O-Forms”“M-Forms”“C-Forms”
Appropriation CategoryProcurement
RDT&EO&M
MILPERS MILCON
Documenting a Program Budget
60Cost EstimatingFigure Source: Boehm, Barry. Software Estimation with COCOMO II. Prentice Hall: Upper Saddle River, NJ (2000)
Some studies have shown that historically cost
estimates are best accurate to within 25% of actuals
only about 70% of the time!
We need to improve our abilityto predict costs