2013 cocomo forum stoddard, 24 october 2013 © 2013 carnegie mellon university harvesting reference...
TRANSCRIPT
2013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
Harvesting Reference Points for Cost Estimation: A Step in the SEI’s Cost Estimation Method
Software Engineering InstituteCarnegie Mellon UniversityPittsburgh, PA 15213
Robert W. StoddardDennis GoldensonRhonda Brown
24 October 2013
32013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
Introduction
QUELCE (Quantifying Uncertainty for Early Lifecycle Cost Estimation) is a multi-year research project led by the Software Engineering Measurement and Analysis (SEMA) team within the SEI Software Solutions Division.
Research team membership comprises SEI technical staff with cost estimation background in collaboration with several external faculty (Dr. Ricardo Valerdi, Univ of Arizona, & Dr. Eduardo Miranda, CMU).
This research is motivated by (1) the WSARA Act requiring cost estimates pre-Milestone A and (2) DoD’s need for more accurate cost estimation methods that provide continuous monitoring of changing assumptions and constraints.
42013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
• Mission / CONOPS• Capability-based analysis
...
• KPP selection• Systems design• Sustainment issues
...
• Production quantity• Acquisition mgt• Scope definition/responsibility• Contract award
Technology Development Strategy
Operational CapabilityTrade-offs
System CharacteristicsTrade-offs
Proposed Material Solution & Analysis of Alternatives
Information from Analogous Programs/Systems
Program Execution Change Drivers
Probabilistic Modeling (BBN) &
Monte Carlo Simulation
Exp
ert
Jud
gm
ents
Information Flow for Early Lifecycle Estimation
Plans, Specifications, Assessments
• Analogy• Parametri
c Models
Cost EstimatesProgram Execution
Scenarios with Conditional Probabilities of Drivers/States
Driver States & Probabilities
• Engineering
• CERs
52013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
Repository: Analyze Existing Data to Model Program Execution Uncertainties – 1
Materiel Solution Analysis Phase – Pre-Milestone Estimate A
Program Change Repository
Prog State Driver
DDG51 cond 1 CONOPS
cond 2 System
cond 3 CapDef
JTRS cond 1 InterOp
cond 2 Produc
F22 cond 1 Contract
cond 2 Function
cond 3 CONOPS
For C2 systems, how often does Strategic Vision
change?
Records show that Strategic Vision changed in 45% of the
MDAPS
Change Driver Nominal State Alternative States
Scope Definition
Stable Users added Additional (foreign) customer
Additional deliverable (e.g. training & manuals)
Production downsized
Scope Reduction (funding reduction)
Mission / CONOPS
defined New condition New mission New echelon Program becomes Joint
Capability Definition
Stable Addition Subtraction Variance Trade-offs [performance vs affordaility, etc.]
Funding Schedule
Established Funding delays tie up resources {e.g. operational test}
FFRDC ceiling issue
Funding change for end of year
Funding spread out
Obligated vs. allocated funds shifted
Advocacy Change
Stable Joint service program loses particpant
Senator did not get re-elected
Change in senior pentagon staff
Advocate requires change in mission scope
Service owner different than CONOPS users
Closing Technical Gaps (CBA)
Selected Trade studies are sufficient
Technology does not achieve satisfactory performance
Technology is too expensive
Selected solution cannot achieve desired outcome
Technology not performing as expected
New technology not testing well
● ~~~~ ~~~~ ~~~~
● ~~~~ ~~~~ ~~~~ ~~~~
● ~~~~ ~~~~ ~~~~ ~~~~ ~~~~ ~~~~
Driver State Matrix
The Materiel Solution of a global network command and control system anticipates a possible change in Strategic Vision that will include allied participation.
Sharing information with allies creates new encryption requirements (a change in Mission/CONOPs).
These changes lead to changes in Capability Definition.
Repository identifies probability of change in MDAP cost drivers.
1. Identify Change Drivers &
States
3. Assign Conditional
Probabilities to BBN Model
4. Calculate Cost Factor
Distributions for Program Execution
Scenarios
5. Monte Carlo Simulation to Compute Cost
Distribution
2. Reduce Cause and Effect Relationships
via Dependency Structure Matrix
Techniques
62013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
Repository: Analyze Existing Data to Model Program Execution Uncertainties – 2
Materiel Solution Analysis Phase – Pre-Milestone Estimate A
If Strategic Vision changes, what else changes?
70% of the time the Mission/CONOPS changes
Change Driver Nominal State Alternative States
Scope Definition
Stable Users added Additional (foreign) customer
Additional deliverable (e.g. training & manuals)
Production downsized
Scope Reduction (funding reduction)
Mission / CONOPS
defined New condition New mission New echelon Program becomes Joint
Capability Definition
Stable Addition Subtraction Variance Trade-offs [performance vs affordaility, etc.]
Funding Schedule
Established Funding delays tie up resources {e.g. operational test}
FFRDC ceiling issue
Funding change for end of year
Funding spread out
Obligated vs. allocated funds shifted
Advocacy Change
Stable Joint service program loses particpant
Senator did not get re-elected
Change in senior pentagon staff
Advocate requires change in mission scope
Service owner different than CONOPS users
Closing Technical Gaps (CBA)
Selected Trade studies are sufficient
Technology does not achieve satisfactory performance
Technology is too expensive
Selected solution cannot achieve desired outcome
Technology not performing as expected
New technology not testing well
● ~~~~ ~~~~ ~~~~
● ~~~~ ~~~~ ~~~~ ~~~~
● ~~~~ ~~~~ ~~~~ ~~~~ ~~~~ ~~~~
Driver State MatrixChange Drivers - Cause & Effects Matrix
Mis
sio
n /
CO
NO
PS
Ch
an
ge
in
Str
ate
gic
Vis
ion
Ca
pa
bili
ty D
efin
itio
n
Ad
vo
ca
cy C
ha
ng
e
Clo
sin
g T
ech
nic
al G
ap
s (
CB
A)
Bu
ildin
g T
ech
nic
al C
ap
ab
ility
& C
ap
acity (
CB
A)
Inte
rop
era
bili
ty
Mission / CONOPS 3 3 0 6 0Change in Strategic Vision 3 3 3 2 2 2 2 2 3 2 3 2 29 0Capability Definition 3 0 2 1 1 0 0 2 2 2 0 1 0 2 0 0 16 0Advocacy Change 2 1 1 1 1 6 0Closing Technical Gaps (CBA) 2 1 3 1 2 2 1 2 2 1 1 2 1 0 2 2 1 1 2 2 1 2 34 0Building Technical Capability & Capacity (CBA) 1 1 2 1 2 2 1 2 3 2 2 1 2 2 1 1 1 27 0Interoperability 1 2 1 1 1 1 1 1 1 1 2 1 1 2 1 1 3 1 2 2 2 29 1Systems Design 1 2 2 2 2 1 1 1 1 1 2 2 3 21 3Interdependency 1 2 2 1 1 1 1 1 1 1 1 1 2 1 2 2 1 1 1 1 1 2 2 3 33 5Functional Measures 2 2 2 1 1 1 1 1 1 1 2 1 16 0Scope Definition 1 1 3 5 0Functional Solution Criteria (measure) 1 2 2 1 1 2 1 10 1Funding Schedule 1 1 2 1 5 0Acquisition Management 1 1 2 3 1 1 2 2 1 1 1 2 1 19 2Program Mgt - Contractor Relations 1 1 2 1 1 1 1 2 2 12 2Project Social / Dev Env 1 1 1 2 2 1 1 2 1 1 1 14 2Prog Mgt Structure 1 2 1 2 6 1Manning at program office 2 1 2 5 2Scope Responsibility 1 1 1 1 1 1 6 5Standards/Certifications 1 1 1 1 1 1 3 1 10 2Supply Chain Vulnerabilities 1 1 1 1 2 1 7 4Information sharing 1 1 1 1 1 1 1 7 3PO Process Performance 2 2 4 0Sustainment Issues 0 0Contract Award 0 0Production Quantity 2 2 0Data Ownership 2 2 0Industry Company Assessment 0 0Cost Estimate 0 0Test & Evaluation 0 0Contractor Performance 2 2 0Size 0 0Project Challenge 0 0Product Challenge 0 0Totals 0 0 6 4 1 9 5 12 8 7 7 13 4 10 15 18 7 7 8 8 14 17 17 15 12 9 10 13 11 20 19 5 5 17 0Below diagonal 0 0 0 1 1 4 4 4 1 2 0 3 1 3 2 2 3 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Effects
Causes
DSM Cause-Effect Matrix
Program Change Repository
Prog State Driver
DDG51 cond 1 CONOPS
cond 2 System De
cond 3 CapDef
JTRS cond 1 InterOpera
cond 2 Production
F22 cond 1 Contract
cond 2 Functional
cond 3 CONOPS
The Materiel Solution of a global network command and control system anticipates a possible change in Strategic Vision that will include allied participation.
Sharing information with allies creates new encryption requirements (a change in Mission/CONOPs).
These changes lead to changes in Capability Definition.
1. Identify Change Drivers &
States
3. Assign Conditional
Probabilities to BBN Model
4. Calculate Cost Factor
Distributions for Program Execution
Scenarios
5. Monte Carlo Simulation to Compute Cost
Distribution
2. Reduce Cause and Effect Relationships via Dependency Structure Matrix Techniques
72013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
Repository: Analyze Existing Data to Model Program Execution Uncertainties – 3
Materiel Solution Analysis Phase – Pre-Milestone Estimate A
Change Driver Nominal State Alternative States
Scope Definition
Stable Users added Additional (foreign) customer
Additional deliverable (e.g. training & manuals)
Production downsized
Scope Reduction (funding reduction)
Mission / CONOPS
defined New condition New mission New echelon Program becomes Joint
Capability Definition
Stable Addition Subtraction Variance Trade-offs [performance vs affordaility, etc.]
Funding Schedule
Established Funding delays tie up resources {e.g. operational test}
FFRDC ceiling issue
Funding change for end of year
Funding spread out
Obligated vs. allocated funds shifted
Advocacy Change
Stable Joint service program loses particpant
Senator did not get re-elected
Change in senior pentagon staff
Advocate requires change in mission scope
Service owner different than CONOPS users
Closing Technical Gaps (CBA)
Selected Trade studies are sufficient
Technology does not achieve satisfactory performance
Technology is too expensive
Selected solution cannot achieve desired outcome
Technology not performing as expected
New technology not testing well
● ~~~~ ~~~~ ~~~~
● ~~~~ ~~~~ ~~~~ ~~~~
● ~~~~ ~~~~ ~~~~ ~~~~ ~~~~ ~~~~
Driver State Matrix Change Drivers - Cause & Effects Matrix
Mis
sio
n /
CO
NO
PS
Ch
an
ge
in
Str
ate
gic
Vis
ion
Ca
pa
bili
ty D
efin
itio
n
Ad
vo
ca
cy C
ha
ng
e
Clo
sin
g T
ech
nic
al G
ap
s (
CB
A)
Bu
ildin
g T
ech
nic
al C
ap
ab
ility
& C
ap
acity (
CB
A)
Inte
rop
era
bili
ty
Mission / CONOPS 3 3 0 6 0Change in Strategic Vision 3 3 3 2 2 2 2 2 3 2 3 2 29 0Capability Definition 3 0 2 1 1 0 0 2 2 2 0 1 0 2 0 0 16 0Advocacy Change 2 1 1 1 1 6 0Closing Technical Gaps (CBA) 2 1 3 1 2 2 1 2 2 1 1 2 1 0 2 2 1 1 2 2 1 2 34 0Building Technical Capability & Capacity (CBA) 1 1 2 1 2 2 1 2 3 2 2 1 2 2 1 1 1 27 0Interoperability 1 2 1 1 1 1 1 1 1 1 2 1 1 2 1 1 3 1 2 2 2 29 1Systems Design 1 2 2 2 2 1 1 1 1 1 2 2 3 21 3Interdependency 1 2 2 1 1 1 1 1 1 1 1 1 2 1 2 2 1 1 1 1 1 2 2 3 33 5Functional Measures 2 2 2 1 1 1 1 1 1 1 2 1 16 0Scope Definition 1 1 3 5 0Functional Solution Criteria (measure) 1 2 2 1 1 2 1 10 1Funding Schedule 1 1 2 1 5 0Acquisition Management 1 1 2 3 1 1 2 2 1 1 1 2 1 19 2Program Mgt - Contractor Relations 1 1 2 1 1 1 1 2 2 12 2Project Social / Dev Env 1 1 1 2 2 1 1 2 1 1 1 14 2Prog Mgt Structure 1 2 1 2 6 1Manning at program office 2 1 2 5 2Scope Responsibility 1 1 1 1 1 1 6 5Standards/Certifications 1 1 1 1 1 1 3 1 10 2Supply Chain Vulnerabilities 1 1 1 1 2 1 7 4Information sharing 1 1 1 1 1 1 1 7 3PO Process Performance 2 2 4 0Sustainment Issues 0 0Contract Award 0 0Production Quantity 2 2 0Data Ownership 2 2 0Industry Company Assessment 0 0Cost Estimate 0 0Test & Evaluation 0 0Contractor Performance 2 2 0Size 0 0Project Challenge 0 0Product Challenge 0 0Totals 0 0 6 4 1 9 5 12 8 7 7 13 4 10 15 18 7 7 8 8 14 17 17 15 12 9 10 13 11 20 19 5 5 17 0Below diagonal 0 0 0 1 1 4 4 4 1 2 0 3 1 3 2 2 3 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Effects
Causes
DSM Cause-Effect MatrixBBN Model
When
both Strategic Vision & Mission/CONOPs
experience change, the BBN calculates that
Capability Definition will also change
95% of the time.
The Materiel Solution of a global network command and control system anticipates a possible change in Strategic Vision that will include allied participation.
Sharing information with allies creates new encryption requirements (a change in Mission/CONOPs).
These changes lead to changes in Capability Definition.
1. Identify Change Drivers &
States
3. Assign Conditional
Probabilities to BBN Model
4. Calculate Cost Factor
Distributions for Program Execution
Scenarios
5. Monte Carlo Simulation to Compute Cost
Distribution
2. Reduce Cause and Effect Relationships via Dependency Structure Matrix Techniques
82013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
Develop Efficient Techniques to Calibrate Expert Judgment of Program UncertaintiesSolution
Calibrated
Un-Calibrated
Estimate of SW Size
Program Change Repository
1) Size of ground combat vehicle targeting feature xyz in 2002 consisted of 25 KSLOC Ada
2) Size of Army artillery firing capability feature abc in 2007 consisted of 18 KSLOC C++
3) …
Step 1: Virtual training using reference points
Step 2: Iterate through a series of domain-specific tests
Step 3: Feedback on test performance
Outcome: Expert renders calibrated estimate of size
Calibrated = more realistic size and
wider range to reflect true expert
uncertainty
Used with permission from Douglas Hubbard Copyright HDR 2008 [email protected]
92013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
Research Challenge
QUELCE team found the “mining” of domain reference points challenging in terms of requisite knowledge, effort, and schedule.
The use of an NVivo-like platform was hypothesized to address the intermediate step of identifying potential change driver experiences documented in historical MDAP artifacts•without losing the context of a given artifact, •enabling queries by attributes across MDAPs, and•achieving review by others prior to committing to a refined domain reference
point statement
102013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
How CAQDAS May Help
Computer-Assisted/Aided Qualitative Data AnalysiS (CAQDAS) is the “use of computer software to aid qualitative research such as transcription analysis, coding and text interpretation, recursive abstraction, content analysis, discourse analysis, grounded theory methodology, etc.”1
CAQDAS software initially developed for social science researchers needed a way to assemble, annotate, and make sense out of a myriad of textual artifacts.
The QUELCE team conceived of the use of CAQDAS in the sense that “research” has always been inherent in the use of analogy for cost estimation. CAQDAS enables a more formal treatment of that activity.
1 Wikipedia, http://en.wikipedia.org/wiki/CAQDAS
112013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
NVivo Experiment
Phase 1: Domain experts coded an assigned set of DoD program artifacts likely to exhibit potential change drivers
• Checkpoint synchronization in the middle of this phase
Phase 2: Domain experts and the research team conducted queries against the artifacts using NVivo text analytic and search capabilities
Phase 3: Feedback gathered from all regarding NVivo usage• On the NVivo experience• On high-value documents• On the co-occurrence and/or cascading of change drivers
122013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
Experimental Procedure
Approximately 200 artifacts of an inventory of 1,500 artifacts were identified for coding.
We operated in standalone mode, with each domain expert assigned a unique NVivo project on the server.
Each NVivo project possessed a unique subset of the 200 artifacts (we balanced the workload among 7 experts).
Experts were asked to open assigned artifacts, one at a time, within NVivo and identify the apparent change drivers, if any.
We asked that experts to track their time in hours during the coding so that we could gauge the productivity rate in forecasting future coding projects.
132013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
Detailed Experimental Work Flow
STEP 1: Open NVivo project
STEP 2: Open a file to code
STEP 4: Sequentially identify change events
STEP 5: Within a change event, identify one or more
change instances
STEP 6: For each change instance, highlight the text
and tag with a Change Driver code
STEP 7: For each change instance, optionally highlight and tag
separate elements of Stimulus, Response, and Outcome
STEP 3: Peruse the entire artifact for
context
142013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
An NVivo Demonstration Platform
152013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
NVivo Coding Artifacts with Change Drivers
162013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
NVivo Summary of Change Driver Coding
172013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
NVivo Simple Query on a Change Driver
182013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
NVivo Search on “Aerospace” and Schedule Change Driver
192013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
NVivo Word Frequency
202013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
NVivo Word Search Cluster Analysis
212013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
NVivo Analysis of Reproducibility
222013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
NVivo Search on Co-occurrence of Change Drivers Within Artifacts
232013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
Early Experimental Feedback
Although individual NVivo projects may be employed and then merged together, efficiencies may be possible via the team server setup.
The NVivo performance across sensitive networks can be troublesome.
Installing NVivo with today’s security infrastructure remains quite challenging.
Experts found the coding within NVivo to be painless, e.g., as easy as highlight and drag or highlight and click on a pull-down list of change drivers.
Queries provide a means to scan a large portfolio of artifacts for key words, change driver codes, co-occurrence of change drivers, etc…
242013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
Future Work
Need to enlist additional experts to code the same artifact set and evaluate reproducibility.
Need to design and implement expert judgment experiments for improvement in judgment “calibration” using a coded NVivo repository as a reference.
Need to refine the query results from an extensive NVivo code repository with the aim of producing refined domain reference points.
Need to consider the at-scale implications and logistics using a tool such as NVivo, e.g., as a living DoD asset.
252013 COCOMO ForumStoddard, 24 October 2013© 2013 Carnegie Mellon University
Contact Information
Robert W. StoddardPrincipal ResearcherSoftware Solutions Division, SEAPTelephone: +1 412-268-1121Email: [email protected]
U.S. MailSoftware Engineering InstituteCustomer Relations4500 Fifth AvenuePittsburgh, PA 15213-2612USA
Webwww.sei.cmu.eduwww.sei.cmu.edu/contact.cfm
Customer RelationsEmail: [email protected]: +1 412-268-5800SEI Phone: +1 412-268-5800SEI Fax: +1 412-268-6257