1 early systems costing prof. ricardo valerdi systems & industrial engineering...
TRANSCRIPT
1
Early Systems CostingProf. Ricardo ValerdiSystems & Industrial [email protected]
Dec 15, 2011INCOSE Brazil Systems Engineering Week
INPE/ITASão José dos Campos, Brazil
Take-Aways1. Cost ≈ f(Effort) ≈ f(Size) ≈ f(Complexity)
2. Requirements understanding and “ilities” are the most influential on cost
3. Early systems engineering yields high ROI when done early and well
Two case studies:• SE estimate with limited information• Selection of process improvement initiative
2
3
Cost Commitment on Projects
Blanchard, B., Fabrycky, W., Systems Engineering & Analysis, Prentice Hall, 2010.
4
Feasibility Plans/Rqts. Design Develop and Test
Phases and Milestones
Relative Size
Range
OperationalConcept
Life Cycle Objectives
Life Cycle Architecture
Initial Operating Capability
x
0.5x
0.25x
4x
2x
Cone of Uncertainty
Boehm, B. W., Software Engineering Economics, Prentice Hall, 1981.
The Delphic Sybil
Michelangelo Buonarroti
Capella Sistina, Il Vaticano (1508-1512)5
NASA data (Honour 2002)
Systems Engineering Effort vs. Program Cost
6
7
COSYSMO Origins
COSYSMO
Systems Engineering
1950
Software Cost Modeling
1980
CMMI*
1990
*Capability Maturity Model Integrated (Software Engineering Institute, Carnegie Mellon University)Warfield, J. N., Systems Engineering, United States Department of Commerce PB111801, 1956. Boehm, B. W., Software Engineering Economics, Prentice Hall, 1981.Humphrey, W. Managing the Software Process. Addison-Wesley, 1989.
(Humphrey 1989)
(Boehm 1981)
(Warfield 1956)
8
How is Systems Engineering Defined?
• Acquisition and Supply – Supply Process– Acquisition Process
• Technical Management– Planning Process– Assessment Process– Control Process
• System Design– Requirements Definition Process– Solution Definition Process
• Product Realization– Implementation Process– Transition to Use Process
• Technical Evaluation
– Systems Analysis Process
– Requirements Validation Process
– System Verification Process
– End Products Validation Process
EIA/ANSI 632, Processes for Engineering a System, 1999.
COSYSMO Data SourcesBoeing Integrated Defense Systems (Seal Beach, CA)
Raytheon Intelligence & Information Systems (Garland, TX)
Northrop Grumman Mission Systems (Redondo Beach, CA)
Lockheed Martin Transportation & Security Solutions (Rockville, MD)
Integrated Systems & Solutions (Valley Forge, PA)
Systems Integration (Owego, NY)
Aeronautics (Marietta, GA)
Maritime Systems & Sensors (Manassas, VA; Baltimore, MD; Syracuse, NY)
General Dynamics Maritime Digital Systems/AIS (Pittsfield, MA)Surveillance & Reconnaissance Systems/AIS (Bloomington, MN)
BAE Systems National Security Solutions/ISS (San Diego, CA)
Information & Electronic Warfare Systems (Nashua, NH)
SAIC Army Transformation (Orlando, FL)
Integrated Data Solutions & Analysis (McLean, VA)
L-3 Communications Greenville, TX
10
Modeling Methodology
11
Results of Bayesian Update: Using Prior and Sampling Information
1.06
Literature,behavioral analysis
A-prioriExperts’ Delphi
Noisy data analysis
A-posteriori Bayesian update
Productivity Range =Highest Rating /Lowest Rating
1.451.51
1.41
12
COSYSMO Scope• Addresses first four phases of the system
engineering lifecycle (per ISO/IEC 15288)
• Considers standard Systems Engineering Work Breakdown Structure tasks (per EIA/ANSI 632)
Conceptualize DevelopOper Test & Eval
Transition to Operation
Operate, Maintain, or Enhance
Replace orDismantle
COSYSMO
SizeDrivers
EffortMultipliers
Effort(schedule, risk)
Calibration
# Requirements# Interfaces# Scenarios# Algorithms
- Application factors-8 factors
- Team factors-6 factors
WBS guided by EIA/ANSI 632
COSYSMO Operational Concept
13
14
Software Cost Estimating Relationship
cSaMM e
cKDSIMM 05.1)(4.2
Boehm, B. W., Software Engineering Economics, Prentice Hall, 1981.
MM = Man monthsa = calibration constantS = size driverE = scale factorc = cost driver(s)KDSI = thousands of delivered source instructions
15
x
COSYSMO Model Form
14
1,,,,,, )(
jj
E
kkdkdknknkekeNS EMwwwAPM
Where: PMNS = effort in Person Months (Nominal Schedule)
A = calibration constant derived from historical project data k = {REQ, IF, ALG, SCN}wx = weight for “easy”, “nominal”, or “difficult” size driver
= quantity of “k” size driverE = represents diseconomies of scaleEM = effort multiplier for the jth cost driver. The geometric product results in an
overall effort adjustment factor to the nominal effort.
x
16
Size Drivers vs. Effort Multipliers
• Size Drivers: Additive, Incremental– Impact of adding a new item inversely proportional to
current size• 10 -> 11 rqts = 10% increase• 100 -> 101 rqts = 1% increase
• Effort Multipliers: Multiplicative, system-wide– Impact of adding a new item independent of current
size• 10 rqts + high security = 40% increase• 100 rqts + high security = 40% increase
4 Size Drivers
1. Number of System Requirements
2. Number of System Interfaces
3. Number of System Specific Algorithms
4. Number of Operational Scenarios
Weighted by complexity and degree of reuse
Number of System RequirementsThis driver represents the number of requirements for the system-of-interest at a specific level of design. The quantity of requirements includes those related to the effort involved in system engineering the system interfaces, system specific algorithms, and operational scenarios. Requirements may be functional, performance, feature, or service-oriented in nature depending on the methodology used for specification. They may also be defined by the customer or contractor. Each requirement may have effort associated with is such as V&V, functional decomposition, functional allocation, etc. System requirements can typically be quantified by counting the number of applicable shalls/wills/shoulds/mays in the system or marketing specification. Note: some work is involved in decomposing requirements so that they may be counted at the appropriate system-of-interest.
Easy Nominal Difficult
- Simple to implement - Familiar - Complex to implement or engineer
- Traceable to source - Can be traced to source with some effort
- Hard to trace to source
- Little requirements overlap
- Some overlap - High degree of requirements overlap
Counting Rules ExampleCOSYSMO example for sky, kite, sea,
and underwater levels where:
Sky level: Build an SE cost model
Kite level: Adopt EIA 632 as the WBS and ISO 15288 as the life cycle standard
Sea level: Utilize size and cost drivers, definitions, and counting rules
Underwater level: Perform statistical analysis of data with software tools and implement model in Excel
Source: Cockburn 2001
Easy Nominal Difficult
# of System Requirements 0.5 1.00 5.0
# of Interfaces 1.7 4.3 9.8
# of Critical Algorithms 3.4 6.5 18.2
# of Operational Scenarios 9.8 22.8 47.4
Size Driver Weights
21
UNDERSTANDING FACTORS– Requirements understanding – Architecture understanding– Stakeholder team cohesion – Personnel experience/continuity
COMPLEXITY FACTORS– Level of service requirements– Technology Risk– # of Recursive Levels in the Design– Documentation Match to Life Cycle Needs
OPERATIONS FACTORS– # and Diversity of Installations/Platforms– Migration complexity
PEOPLE FACTORS
– Personnel/team capability
– Process capability
ENVIRONMENT FACTORS
– Multisite coordination
– Tool support
Cost Driver Clusters
22
Stakeholder team cohesion Represents a multi-attribute parameter which includes leadership, shared vision, diversity of stakeholders, approval cycles, group dynamics, IPT framework, team dynamics, trust, and amount of change in responsibilities. It further represents the heterogeneity in stakeholder community of the end users, customers, implementers, and development team.
1.5 1.22 1.00 0.81 0.65
Viewpoint Very Low Low Nominal High Very High
Culture Stakeholders with diverse expertise, task nature, language, culture, infrastructure Highly heterogeneous stakeholder communities
Heterogeneous stakeholder communitySome similarities in language and culture
Shared project culture
Strong team cohesion and project cultureMultiple similarities in language and expertise
Virtually homogeneous stakeholder communitiesInstitutionalized project culture
Compatibility Highly conflicting organizational objectives
Converging organizational objectives
Compatible organizational objectives
Clear roles & responsibilities
Strong mutual advantage to collaboration
Familiarity and trust
Lack of trust Willing to collaborate, little experience
Some familiarity and trust
Extensive successful collaboration
Very high level of familiarity and trust
Technology RiskThe maturity, readiness, and obsolescence of the technology being implemented. Immature or obsolescent technology will require more Systems Engineering effort.
Viewpoint Very Low Low Nominal High Very High
Lack of Maturity
Technology proven and widely used throughout industry
Proven through actual use and ready for widespread adoption
Proven on pilot projects and ready to roll-out for production jobs
Ready for pilot use Still in the laboratory
Lack of Readiness
Mission proven (TRL 9)
Concept qualified (TRL 8)
Concept has been demonstrated (TRL 7)
Proof of concept validated (TRL 5 & 6)
Concept defined (TRL 3 & 4)
Obsolescence
- Technology is the state-of-the-practice- Emerging technology could compete in future
- Technology is stale- New and better technology is on the horizon in the near-term
- Technology is outdated and use should be avoided in new systems- Spare parts supply is scarce
Migration complexity This cost driver rates the extent to which the legacy system affects the migration complexity, if any. Legacy system components, databases, workflows, environments, etc., may affect the new system implementation due to new technology introductions, planned upgrades, increased performance, business process reengineering, etc.
Viewpoint Nominal High Very High Extra High
Legacy contractor
Self; legacy system is well documented. Original team largely available
Self; original development team not available; most documentation available
Different contractor; limited documentation
Original contractor out of business; no documentation available
Effect of legacy system on new system
Everything is new; legacy system is completely replaced or non-existent
Migration is restricted to integration only
Migration is related to integration and development
Migration is related to integration, development, architecture and design
25
Cost Driver Rating ScalesVery Low Low Nominal High Very High
Extra High EMR
Requirements Understanding 1.87 1.37 1.00 0.77 0.60 3.12
Architecture Understanding 1.64 1.28 1.00 0.81 0.65 2.52
Level of Service Requirements 0.62 0.79 1.00 1.36 1.85 2.98
Migration Complexity 1.00 1.25 1.55 1.93 1.93
Technology Risk 0.67 0.82 1.00 1.32 1.75 2.61
Documentation 0.78 0.88 1.00 1.13 1.28 1.64
# and diversity of installations/platforms 1.00 1.23 1.52 1.87 1.87
# of recursive levels in the design 0.76 0.87 1.00 1.21 1.47 1.93
Stakeholder team cohesion 1.50 1.22 1.00 0.81 0.65 2.31
Personnel/team capability 1.50 1.22 1.00 0.81 0.65 2.31
Personnel experience/continuity 1.48 1.22 1.00 0.82 0.67 2.21
Process capability 1.47 1.21 1.00 0.88 0.77 0.68 2.16
Multisite coordination 1.39 1.18 1.00 0.90 0.80 0.72 1.93
Tool support 1.39 1.18 1.00 0.85 0.72 1.93
26
Cost Drivers Ordered by Effort Multiplier Ratio (EMR)
ISO/IEC 15288
Conceptualize DevelopTransition to
Operation
Acquisition & Supply
Technical Management
System Design
Product Realization
Technical Evaluation
Operational Test &
Evaluation
AN
SI/E
IA 6
32
Effort Profiling
Benefits of Local Calibration
Before local calibration
After local calibration
Sy
ste
ms
En
gin
ee
rin
g E
ffo
rt (
SE
Ho
urs
)
System Size (eReq)
Sy
ste
ms
En
gin
ee
rin
g E
ffo
rt (
SE
Ho
urs
)
System Size (eReq)
29
Prediction Accuracy
PRED(30)
PRED(25)
PRED(20)
PRED(30) = 100% PRED(25) = 57%
Human Systems Integration in the Air Force
31
Joint HSI Personnel Integration Working Group. “Human System Integration (HSI) Activity in DoD Weapon Acquisition Programs: Part II Program Coverage and Return on Investment.” April 16, 2007.
U.S. Air Force Scientific Advisory Board. “Report on Human Systems Integration in Air Force Weapon Systems Development and Acquisition.” July, 2004.
HSI Early in F119 Development
AF Acquisition Community-led requirements definition studies
40% fewer parts than previous engines
32
Parametric Cost Estimation
33
Parametric Cost Estimation
34
n
ii
1
E EMSizeAPM
Requirements
Interfaces
Algorithms
Operational Scenarios
Tool Support
Architecture Understanding
Technology Risk
# and diversity of installations/platforms
Personnel/team capability
Illustrative Example
35
100 Difficult Requirements
200 Medium Requirements
200 Easy Requirements
Nominal System
SE Effort = 300 Person-months
Illustrative Example
36
110 Difficult Requirements
210 Medium Requirements
200 Easy Requirements
SE Effort = 327 Person-months
5th/95th Percentile Manpower
Tools reduction
Interactive Technical Manuals
20 minute component replacement
Size impactNominal System
Illustrative Example
37
110 Difficult Requirements
210 Medium Requirements
200 Easy Requirements SE Effort = 368 Person-months
effect of effort multipliers:
High Level of service requirements High HSI Tools support
Cost impactNominal System
ROI Calculation
• Cost: 68 Person Months @ $10,000/person month– $680,000 of Human Systems Integration
• Benefit: assuming 30 year life cycle– Manpower (one less maintainer): $200,000 * 30 = $6,000,000– Human factors (40% fewer parts): $300,000 * 30 = $9,000,000– Safety (one less repair accident): = $50,000– Survivability (one less engine failure): = $500,000
= $15,550,000
• Return on Investment
38
%2186
000,680$
000,680$000,550,15$
Note: time value of money excluded
Relative ROI
Activity % ROI
CMMI® 173%
ISO 9001 229%
Systems Engineering 1,700%
Human Systems Integration 2,186%
Software inspections 3,272%
Personal Software Process 4,133%
39
Brodman, J. G., Johnson, D. L., Return on Investment from Software Process Improvement as Measured by U.S. Industry, Crosstalk – The Journal of Defense Software, 1996.
Rico, D. F., ROI of Software Process Improvement: Metrics for Project Managers and Software Engineers , J. Ross Publishing, Boca Raton, FL, 2004.
Boehm, B. W., Valerdi, R. and Honour, E., “The ROI of Systems Engineering: Some Quantitative Results for Software-Intensive Systems,” Systems Engineering, 11(3), 221-234, 2008.
40
41
Case Study I: Albatross Budgetary Estimate• You, as the Albatross SE lead, have just been asked to provide your program
manager a systems engineering budgetary estimate for a new, believed-to-be critical function to the current baseline– This new functionality would add some new, nearly-standalone, capability
to your Albatross program, your best educated guess by looking at the emailed requirements provided by your customer is that it adds about 10% to the requirements baseline of 1,000 and two new interfaces, you guess we need at least one new Operational Scenario.
– The customer also stated that they really need this capability to be integrated into the next delivery (5 months from now)
– The PM absolutely has to have your SE cost estimate within the next two hours, as the customer representative needs a not-to-exceed (NTE) total cost soon after that!
• Information that may prove useful in your response– Albatross is well into system I&T, with somewhat higher than expected
defects– Most of the baseline test procedures are nearing completion– The SE group has lost two experienced people in the past month to
attrition– So far, the Albatross customer award fees have been excellent, with
meeting of schedule commitments noted as a key strength
42
Case Study I: In-Class Discussion Questions
• What are some of the risks?
• What additional questions could you ask of your PM?
• What additional questions could the PM (or the SE Lead) ask of the Customer Representative?
• What role could the Albatross PM play in this situation?
• Is providing only “a number” appropriate in this situation?
• What additional assumptions did you make that can be captured by COSYSMO?
43
Case Study II: Selection of Process Improvement Initiative
• You are asked by your supervisor to recommend which process improvement initiative should be pursued to help your project, a 5-year $100,000,000 systems engineering study. The options and their implementation costs are:
Lean $1,000,000
Six Sigma $600,000
CMMI $500,000
• Which one would you chose?
44
Case Study II: Selection of Process Improvement Initiative
Assumptions– Implementing Lean on your project will greatly improve your team’s
documentation process. Using COSYSMO, you determine that the Documentation cost driver will move from “Nominal” to “Low” yielding a 12% cost savings (1.00 – 0.88 = 0.12).
– Implementing Six Sigma will greatly improve your team’s requirements process. Using COSYSMO, you determine that the Requirements Understanding cost driver will move from “Nominal” to “High” yielding a 23% cost savings (1.00 – 0.77 = 0.23)
– Implementing CMMI will greatly improve team communication. Using COSYSMO, you determine that the Stakeholder Team Cohesion cost driver will move from “Nominal” to “High” yielding a 19% cost savings (1.00 – 0.81 = 0.19)
45
Cost Driver Rating ScalesVery Low Low Nominal High Very High
Extra High EMR
Requirements Understanding 1.87 1.37 1.00 0.77 0.60 3.12
Architecture Understanding 1.64 1.28 1.00 0.81 0.65 2.52
Level of Service Requirements 0.62 0.79 1.00 1.36 1.85 2.98
Migration Complexity 1.00 1.25 1.55 1.93 1.93
Technology Risk 0.67 0.82 1.00 1.32 1.75 2.61
Documentation 0.78 0.88 1.00 1.13 1.28 1.64
# and diversity of installations/platforms 1.00 1.23 1.52 1.87 1.87
# of recursive levels in the design 0.76 0.87 1.00 1.21 1.47 1.93
Stakeholder team cohesion 1.50 1.22 1.00 0.81 0.65 2.31
Personnel/team capability 1.50 1.22 1.00 0.81 0.65 2.31
Personnel experience/continuity 1.48 1.22 1.00 0.82 0.67 2.21
Process capability 1.47 1.21 1.00 0.88 0.77 0.68 2.16
Multisite coordination 1.39 1.18 1.00 0.90 0.80 0.72 1.93
Tool support 1.39 1.18 1.00 0.85 0.72 1.93
46
Case Study II: Selection of Process Improvement Initiative
Assumptions•The systems engineers on your project spend:
5% of their time doing documentation
30% of their time doing requirements-related work
20% of their time communicating with stakeholders
•What is the financial benefit of each process improvement initiative?•How does the financial benefit vary by project size?
Return-on-Investment Calculation
(benefit – cost) / cost
Best option is Six Sigma!
48
Six Sigma
CMMI
Lean
Return-on-Investment vs. Project Size
Reuse is a Universal Concept
49
50
Reusable Artifacts
• System Requirements • System Architectures • System Description/Design Documents • Interface Specifications for Legacy Systems • Configuration management plan • Systems engineering management plan • Well-Established Test Procedures • User Guides and Operation Manuals• Etc.
51
Reuse Principles
1. The economic benefits of reuse can be described either in terms of improvement (quality, risk identification) or reduction (defects, cost/effort, time to market).
2. Reuse is not free, upfront investment is required
(i.e., there is a cost associated with design for reusability).
3. Reuse needs to be planned from the conceptualization phase of program
52
Reuse Principles (con’t.)
4. Products, processes, and knowledge are all reusable artifacts.
5. Reuse is as much of a technical issue as it is an organizational one
6. Reuse is knowledge that must be deliberately captured in order to be beneficial.
53
Reuse Principles (con’t.)
7. The relationship between project size/complexity and amount of reuse is non-linear
8. The benefits of reuse are limited to closely related domains.
9. Reuse is more successful when level of service requirements are equivalent across applications.
54
Reuse Principles (con’t.)
10.Higher reuse opportunities exist when there is a match between the diversity and volatility of a product line and its associated supply chain.
11. Bottom-up (individual elements where make or buy decisions are made) and top-down (where product line reuse is made) reuse require fundamentally different strategies.
12.Reuse applicability is often time dependent; rapidly evolving domains offer fewer reuse opportunities than static domains.
55
Reuse Success Factors
• Platform– Appropriate product or technology, primed for reuse
• People– Adequate knowledge and understanding of both the heritage
and new products
• Processes– Sufficient documentation to acquire and capture knowledge
applicable to reuse as well as the capability to actually deliver a system incorporating or enabling reuse
56
Background on Software Reuse
Main size driver = KSLOC• Adapted Source Lines of Code (ASLOC)• Percent of Design Modification (DM)• Percent of Code Modification (CM)• Percent of Integration Required for Modified Software (IM)• Percentage of reuse effort due to Software Understanding (SU)• Percentage of reuse effort due to Assessment and Assimilation (AA)• Programmer Unfamiliarity with Software (UNFM)
From COCOMO II Model Definition Manual (p. 7-11)
AAF
Example COSYSMO 2.0 Estimate
Estimated as 129.1 Person-Months by COSYSMO (without reuse)
…a 30.4% difference
COSYSMO 2.0 Implementation Results
• Across 44 projects at 1 diversified organization
• Using COSYSMO:– PRED(.30) = 14%
– PRED(.40) = 20%
– PRED(.50) = 20%
– R2 = 0.50
• Using COSYSMO 2.0:– PRED(.30) = 34%
– PRED(.40) = 50%
– PRED(.50) = 57%
– R2 = 0.72
• Result: 36 of 44 (82%) estimates improved
September 10, 2009
59
Reuse Terminology• New:
– Items that are completely new
• Managed:– Items that are incorporated and require no added SE effort other than technical
management
• Adopted: – Items that are incorporated unmodified but require verification and validation
• Modified: – Items that are incorporated but require tailoring or interface changes, and
verification and validation
• Deleted: – Items that are removed from a legacy system, which require design analysis,
tailoring or interface changes, and verification and validation
Notes:• New items are generally unprecedented• Those items that are inherited but require architecture or implementation changes
should be counted as New
Reuse Continuum
Modified
Adopted
New 1.0
0
DeletedDeleted
Managed
Reu
se w
eig
ht
0.65
0.510.43
0.15
Modified vs. New Threshold
Evaluate
Analyze Available
Assets
Estimate Costs & Anticipate
Benefits
Develop Reuse Plan
Implement
Invest for Reusability
Prepare Available
Assets
Execute Reuse Plan
Capture Future Reuse Opportunities
Manage
Measure Cost/Benefit
Results
On-Going Project Activities
Completion
Systems Engineering Assets
Update Asset Index
Reuse Considerations
10 Academic Theses
Academic Curricula
Proprietary Implementations
• SEEMaP• COSYSMO-R• SECOST• Systems Eng.
Cost Tool
Commercial Implementations
Policy & Contracts
COSYSMO Model
14
1,,,,,, )(
jj
E
kkdkdknknkekeNS EMwwwAPM
Impact
Take-Aways1. Cost ≈ f(Effort) ≈ f(Size) ≈ f(Complexity)
2. Requirements understanding and “ilities” are the most influential on cost
3. Early systems engineering yields high ROI when done early and well
Two case studies:• SE estimate with limited information• Selection of process improvement initiative
63