© 2013 price systems, llc all rights reserved | decades of cost management excellence 1 data...
TRANSCRIPT
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 1© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence
Data Collection and Analysis Supporting Defendable Cost Estimates
Arlene Minkiewicz, Chief ScientistPRICE Systems, LLC
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 2
Agenda
Introduction Historical Prospective on Cost Estimation Data Driven Estimation and its Importance in the current
economic client Pilot Case Studies
– Software Data Collection– Automotive Data Collection and Analysis
Lessons Learned
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 3
Introduction
Cost estimation in the Aerospace and Defense world emerged as a necessary around the time of World War II
All sorts of mathematical and experiential models have been applied to the problem with varying degrees of success
More and more organizations are pushing for models tuned to their specific data and experience
Many of these organizations lack the infrastructure, processes and tools to collect and analyze project data efficiently
This paper presents two pilot projects (both on-going) to inject data driven estimation into two very different organizations.
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 4
Historical Perspective
General H.H. “Hap” Arnold at the end of WWII
– “During the war, the Army, Army Air Forces, and the Navy have made unprecedented use of scientific and industrial resources. The conclusion is inescapable that we have not yet established the balance necessary to insure the continuation of teamwork among the military, other government agencies, industry, and the universities. Scientific planning must be years in advance of the actual research and development work.”
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 5
Historical Perspective
Leaders took this to heart – state of the art weapons systems are necessary to ensure military superiority
Beginning of period of time during which the Department of Defense (DOD) is plagued with issues surrounding cost!
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 6
Historical Perspective
1930 1940 1950 1960 1970 1980 1990 2000 2010 2020
PRICE H as an RCA Only Tool
Operations Research applied to Military Affairs
Proliferation of Commercially Available Cost Estimation Models
Great Depression
World War II
Severe DoD Project Cost Overruns
OSDCAIG
Formed
War on TerrorismBerlin Wall
Breached
RAND Created
PMI Created
PCProjectMgmtEVMTools
Parametric Estimating Initiative
ISPAFounded
KoreanISPA
Chapter
Defense Transformation
Operations Research and Statistics applied to Industrial Affairs
Homeland Defense
Gov’t & Industry Business ModernizationCost Overruns
J. Gordon DavisFrank Frieman
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 7
Data Driven Estimation
Not a new idea – just a new focus on what’s behind an estimate
More credible and lends a a sense of defendability to any estimate
“Data-driven” estimation IS…
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 8
Data Driven Estimation
Based on data extrapolation from analogous systems
Based on a model using analogous systems to determine model inputs
Based on a model calibrated to historical data
A “data-driven” estimate MAY BE…
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 9
Data Driven Estimation
Transparent as to what data was used
Obtained from a credible, identifiable source
Clear about how that data was used
A “data-driven” estimate MUST BE…
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 10
Case Study Pilot – Software Data Collection
Software data collection and analysis to support more defendable estimate
– Army Program Executive Office for Simulation, Training, and Instrumentation (PEO STRI)
– No data collection processes in place
– Pilot involves one large, complex program – two contractors – one does primarily development, one primarily integration
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 11
Software Pilot - Motivation
PEO STRI personnel take their mission of delivering training to the warfighter seriously and intend to deliver best of breed solutions.
Estimates presented to the Office of the Deputy Assistant Secretary of Defense for Cost and Economics (ODASA-CE) are met with the question “Where’s the data that supports this estimate?”
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 12
Software Pilot - Motivation
Unsatisfactory responses inspire large risk margins resulting in less budget available to the program
Furthermore, without insight into projected costs for specific capabilities, it is difficult to make trade-offs to accomplish reduced costs
PEO STRI wants to establish a consistent, repeatable process to collect cost and technical data for each capability they develop in their programs
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 13
Data Collection – How will it be used
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 14
Reporting Strategy for Maximum Benefit
Report
Report
Report
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 15
Data Collection and Normalization
Once the need for data collection has been identified, an important first step is identifying what to collect. This requires
– Understanding the goal of the collection program– Understanding what data is necessary to support that goal– Understanding what data is available or could reasonably be made available
Normalization guidelines need to be developed The next step needs to be documentation of these
understandings and guidelines in a clear, concise, understandable and repeatable fashion.
Automation eases the collection process, ensures consistency in the process and facilitates the process of normalization
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 16
Data Collection – Keys to Success
Use of an Integrated Product Team (IPT)
Data Item Dictionary created to define….
– What data to collect– A clear understanding of the data
and potential uses– How often to collect it– How to align project specific data to
more general categories (across multiple contractors)
PEO STRI
PRICE
Contractors
Lots of collaboration between PEO STRI personnel, contractors and the PRICE team resulting in…..
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 17
Data Collection – Keys to Success
Requirement added to Contract Deliverable Requirements List (CDRL) making data collection a contractor requirement
Creation of automated code counting tools by wrapping USC’s Code Counting tool with Excel to help ensure consistency of data
Software Resource Report (SRR) developed to align with output of code counting tool, automate mapping processes and align with potential uses
Lots of collaboration between PEO STRI personnel, contractors and the PRICE team resulting in…..
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 18
Software Pilot – Software Resource Report
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 19
PEO STRI Data Collection and Preparation Process
Collect OneSAF Program data using SRR forms. Data collected includes:
– Code counts (New, Modified, Deleted, Unmodified)
– Effort by Activity– Effort by Capability– Factors relevant to project, team & iteration
Extract data and import into TrueFindings ™ format
Create TruePlanning® Product Breakdown Structure (PBS) by:
– Contractor– Build– Capability
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 20
PEO STRI Data Collection and Preparation Process
Calibrate the inputs of interest in TruePlanning® using data from TrueFindings ™
– Organizational Productivity– Functional Complexity by capability by language
Import the calibrated inputs into TrueFindings™ file
Drive inputs values using the “findings” from TrueFindings™ inside TruePlanning® to create defendable estimates.
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 21
Software Pilot – Progress to Date
Data collection occurs every 10 weeks.
The first iteration was a thrown away
– Automation failures in the contractor configuration
– Uncertainty of some data definitions
– Additional discussions required by IPT to iron out questions and uncertainties
Four successful iterations have occurred since then – each resulting in slight refinement to the process, DID and/or SRR
A calibration process has been created and continues to be refined
Collected data and calibration results have been integrated into a TrueFindings database to support future estimation efforts
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 22
Data Collection - Lessons Learned
Alleviate contractor concern about being ‘measured’ or misuse of data
Facilitate determination of data collection targets
Need buy in from the data collectors
Entire team needs to understand motivation for data collection
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 23
Data Collection - Lessons Learned
Creates consistency Creates efficiency Reduces impact on contractors Promotes harmony Only means to handle such large
code counts / bases Facilitates normalization and
enforces normalization guidelines
Automation is key to success
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 24
Data Collection - Lessons Learned
Communication, teamwork and building trust are paramount
Data needs to be inspected for quality and ported into an environment for easy analysis.
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 25
Conclusions
The need for data driven estimation is driving many organizations in Aerospace and Defense and industry in general to look for new ways to collect and use data
Data driven estimation is really nothing new – we have been using data (real or experiential) to justify our estimates forever. The new requirement is transparency and openness about the data and how it is being used.
To support this effort organizations may need to establish formal processes for collecting their project data.
When doing this it is important to understand the importance of – Communication– Automation– Flexibility– Buy in across all the stakeholders
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 26
Questions
© 2013 PRICE Systems, LLC All Rights Reserved | Decades of Cost Management Excellence 27© 2013 Financial Executives International
Backup SlidesData Collection and Analysis Supporting Defendable Cost Estimates
Arlene Minkiewicz, Chief ScientistPRICE Systems, [email protected]