pcec v2.2 overview - nasa · pcec overview • pcec is a parametric cost model developed and...

PCEC v2.2 Overview

2017 NASA Cost & Schedule Symposium

29 Aug 2017

NA

SA M

SFC

Vict

ory

Sol

utio

ns

SB Diversity

MIPSS Team

Outline

• PCEC v2.2 New Features

• Robotic Spacecraft Model Updates for v2.2

• Crewed & Space Transportation Systems (CASTS) Updates for v2.2

• REDSTAR Overview

• Future Work & Analyses

Victory Solutions MIPSS Team 2

Victory Solutions MIPSS Team

PCEC Overview

• PCEC is a parametric cost model developed and maintained by NASA

beginning in late 2013. It comprises the PCEC Interface and PCEC Library

– The PCEC Library is a collection of cost estimating artifacts used to create an

estimate of a space flight hardware system

– The PCEC Interface is an Excel Add-in with buttons to access artifacts from the

PCEC Library for building an estimate in Excel

• PCEC is currently best suited for estimating the cost to design, develop, and

produce the following types of space systems

– Spacecraft: Earth Orbiting Satellites, Planetary Probes, Rovers

– Launch Vehicles: Multi-stage rockets, liquid and solid engines

– Human Space Flight Systems: Crew Capsules, Orbiters, Habitats

• PCEC is available to the General Public via the NASA Software Catalog

and to NASA Civil Servants via the ONCE Database

3

PCEC v2.2 Overview

PCEC V2.2 NEW FEATURES

4


PCEC Status Update

• PCEC v2.1 is the

current active

release (Aug 2016)

• Current user

counts as of mid-

Aug 2017

– 350+ Users

– 45 Countries

– Big increase in users earlier this year due to media campaign

associated with the 2017/2018 NASA Software Catalog release

– Continue to add a couple of users each month

• PCEC v2.2 will be available this week

– Announcement and download link will be sent out to existing users

5


Summary of Major Changes in

PCEC v2.2

• Usability Improvements

– Launch Resume Capability

– Post-Launch Worksheet Linking

• Additional Phase E/Operations Estimating Options

– Mission Operations & Data Analysis (MO&DA) CERs

– Linking to Operations Cost Model (OCM) and Space Operations Cost

Model (SOCM) estimates

• CER Importer

• Other Improvements

– Compatibility with Mac Office

– CER Updates

– Template Updates

– Many under-the-hood enhancements, bug fixes, and corrections

6


PCEC v2.2

Launch Updates (1 of 2)

7

Launch now has two paths:

• Start New Estimate

• Resume Existing EstimateThe Launch Resume

capability allows the

user to pick up the

editing of a Launch-

created estimate

where they left off


PCEC v2.2

Launch Updates (2 of 2)

8

The Resume routine

reads WBS metadata

(CERs selected, options

for linked models, etc.)

and enables the same

WBS building as the

initial run of the LaunchThe routine updates

the estimate file

based on the

additions / deletions /

edits made during the

Resume session

• The Launch Resume feature currently only works with estimates

created with v2.2

• Other Launch updates in v2.2

– MOCET, OCM, and SOCM-linked elements can be added to the WBS

– New indicators for WBS line items

– Options for formatting templates


PCEC v2.2

Post-Launch Sheet Linking

9

New routines automate the

linking of a template added

after the Launch run to other

key worksheets in the estimate

• Link to Globals: Links all Global variables on the active sheet to

the Globals worksheet

• Link to Summary: Inserts a new line into the WBS (location

indicated by the user), and the active sheet is linked to it

• Link to Subsystem Inputs: Inserts a section on the Subsystem

Inputs worksheet for entering input data for the subsystem


PCEC v2.2

MO&DA CERs

The Mission Operations

& Data Analysis

(MO&DA) CERs provide

an alternative option for

estimating Phase E costs

for robotic spacecraft

missions, both Near Earth

and Planetary

10

MO&DA templates

have the same look

as other templates

and allow for

conducting

sensitivity analysis

and modeling

uncertainty

A separate SOCM Score

Calculator worksheet is included

with the MO&DA template to

centralize input of data for use

throughout the estimate


PCEC v2.2

Linking to OCM and SOCM

OCM and SOCM are not included

with the PCEC distribution but can

be requested for NASA use

– Distribution rules/process TBD

Both the Space Operations

Cost Model (SOCM) and

Operations Cost Model (OCM)

are now supported for

inclusion into a PCEC estimate

Estimate results are

pasted or linked into a

template for including in

the WBS like the models

PCEC already links to

(NICM and MOCET)


PCEC v2.2

CER Importer

12

PCEC now makes it

easier to bring in a user’s

custom CERs for use in

an estimate

Data must first be input on

the pre-formatted CER

worksheet (included) so it

can be read by the Interface

The CER(s) can then

be imported into the

user’s local copy of

the Interface for use

throughout the tool

The user

selects the

workbook

containing the

documentation

and the CERs

to be imported

Current Limitations:

– Only supports linear or power CERs

– Any independent variable used, if not in PCEC

already, must be added manually to the Interface


PCEC v2.2

Other Improvements

• Mac Compatibility of PCEC Interface

– Now runs on Office for Mac 2016

• Worksheet Template Updates

– First Pound Cost Templates

– MOS/GDS Development

– SOCM Score Calculator

– General template for user-added CERs

• Bug Fixes / Corrections

– LV and Crewed Templates

– Userform Search bug

– Running Launch multiple times

– Correlation matrices for v1 CERs

– Unit space CV calculation for all CERs

• Under-the-hood enhancements for maintainability and adding new

estimating capabilities

PCEC v2.2 Overview

ROBOTIC SPACECRAFT

MODEL UPDATES

14


Robotic SC Model Updates

Summary

Mission Operations & Data Analysis (MO&DA) and Other CER Updates

• MO&DA Objective/Approach

• MO&DA Input Collection

• MO&DA Data Analysis

• MO&DA (Phase E) CER development

• Other CER Updates


PCEC MO&DA Analysis

Development Approach

16

GOAL: Derive Phase E (MO&DA) CERs for PCEC Robotic SC

leveraging the MOS/GDS Development effort

PCA

Other Data

Analysis Methods

Constructive Processes

Acceptable CER

Performance?

Yes

No

Mocet Inputs

SOCM Inputs

PCEC Inputs

Key Driver

Input SetsSupports

Regression Based

CERs

CER

Performance

AnalysisAttempt to Identify

Common Attributes

to Explain

Error/Residuals

Multiple Regression

PCEC CERs

Candidate CERsNear Earth Prime Mission

Planetary – Cruise

Planetary - Encounter

New Input CandidatesDifferent Input Combinations;

Data Collection of Additional Info;

Combination Inputs (SOCM

Scores)


PCEC MO&DA Analysis

NASA WBS Elements Included

• The PCEC MO&DA analysis

includes elements of Mission

Operation Systems (MOS) and

Ground Data Systems (GDS)

from WBS 7.0 & 9.0 during

Phase E.

NASA WBS items

included for this

effort in Phase E

17


PCEC CADRe Data Normalization

Current Project Data Set

• Phase E data for the 42 missions

shown here were normalized

• The set covers recent missions

and includes representatives from

each NASA science discipline

• The normalization process shows

the traceability to the official

CADRe data

– All assumptions and changes have

been documented

• The normalized data for each of

these missions has been provided

to the lead organizations for their

review

18


PCEC MO&DA Analysis

Initial Input Candidates

• Multiple information sources have been reviewed to generate the

initial input candidate list:

– CADRe: Fields in Part B (technical)

– Cost Models: Space Operations Cost Model (SOCM), PCEC

Normalized Data Library, MOCET

– Over 150 input candidates were identified for each mission

19

INPUT CANDIDATES


PCEC MO&DA AnalysisSearch for New Input Candidates

• Assessments have been made to identify whether a single variable exists to provide a

‘reasonable’ estimate

– Many candidates have been explored without showing any clear trend

– Similar result to pre-SOCM efforts in the 1990’s

• Explored potential to build on SOCM analyses (by using intermediate SOCM scores

or other lower-level SOCM inputs as predictor variable)

20


PCEC MO&DA AnalysisRationale for Planetary/Near-Earth Split

• Initially, a CER was developed using the entire data set which

consists of both Planetary and Near Earth missions. The CERs

created from this initial effort produced less than desirable results.

• Ultimately, the best results were obtained by splitting the data set

into Near Earth and Planetary missions.

– The planetary data was then split into cruise and encounter phases to

capture the unique attributes of these operating phases.

• This split makes sense when you consider the unique aspects of

operating in the Near Earth environment when compared to deep

space operations.

– Near Earth missions have many options in terms communications

networks (NEN, TDRSS). Planetary missions rely on the Deep Space

Network (DSN).

– Near-Earth missions often employ high-heritage s/c and MOS/GDS

elements. Planetary s/c are typically customized to minimize mass and

tailored to their application.21


PCEC MO&DA Analysis

Leveraging of SOCM

• Even with the split in the data set, reasonable CERs still could not

be identified

• Several Space Operations Cost Model (SOCM) inputs were added

to the candidate input set

– SOCM was originally developed by a team of operations experts and

cost analysts

– SOCM Level 1 Overall Score and Payload Score were added; The

SOCM Level 1 Scores have remained unchanged over the last 2

updates (2000 & 2004); The Overall Score supported the Near Earth

CER and the Payload Score supported the Planetary CER

– Parametric performance statistics significantly improved with the added

SOCM Score inputs

• SOCM has been tested with all the PCEC Missions and shows

reasonable results

– SOCM estimates did better for Near Earth Primary Mission and

Planetary Encounter than for Planetary Cruise

22


PCEC MO&DA Analysis

SOCM Level 1 Score

• The SOCM Level 1 Scores for Near Earth and Planetary

applications are a composite of several input parameters

– Level 1 Score consists of two equations that determine level of effort of

the science and engineering teams. E.g., the relevant input parameters

needed to calculate the Near Earth score include:

23


PCEC MO&DA Analysis

SOCM Performance Testing

• SOCM estimates were developed for each of the PCEC missions

– Approach leverages expert judgement from operations specialists which results in ‘intuitive’

changes from inputs (most inputs are ‘causal’ vs ‘associative’)

– Used Level 1 inputs and a ‘select set’ of Level 2 inputs (5-6 inputs)

– Results appear reasonable

• Some correlation of the SOCM Level 1 score to MO&DA cost requirements appears

to be demonstrated by this analysis

– Results are used to identify ‘Combination Inputs’ to carry into the PCA/Regression

24


PCEC MO&DA Analysis

CER Results

• SOCM Level 1 Scoreinput requires a separatecalculator in PCEC

25

• Equations take the form of:– $k/month = (Parameter 1)α · (Parameter 2)β ·…….. · EXP(Constant)


PCEC MO&DA AnalysisModeling Performance Comparisons

Mean Error = 4%

Mean Abs. Error =

23%

Standard Dev. = 26%

NEAR EARTH PRIMARY MISSION

Mean Error = 4%

Mean Abs. Error = 27%

Standard Dev. = 33%

PLANETARY ENCOUNTER

26


MO&DA Phase E

Planetary Cruise Calculator

• Regression of the planetary cruise data did not result in an

acceptable CER

• This is likely due to the fact that the activities during cruise vary

drastically from mission to mission and the time to accomplish these

activities varies greatly depending on the time required to reach

each planetary destination.

• During the data analysis effort, we discovered that encounter cost is

a fairly good predictor of cruise cost

27


PCEC MO&DA AnalysisOverall MO&DA Error by Mission

• Derived MO&DA prediction CERs have reasonable performance.

• Overall prediction error is roughly +/- 50% which is a slight improvement

over the SOCM approach.

• Near Earth CER performs slightly better than the Planetary Encounter CER

% D

iffe

ren

ce =

(E

sti

mate

–A

ctu

al

)/A

ctu

al

28


PCEC CER Updates

• As PCEC continues to develop into a full lifecycle cost model and is

used more and more to support various NASA costing activities,

opportunities arise for the PCEC developers to improve the CERs in

the model.

• PCEC is a constantly evolving model that relies on user feedback

and continued expansion of the normalized data set.

• Thank you for providing your inputs – please continue to do so. They

will ultimately make PCEC a better model.

• The following charts describe the CER changes that have resulted

from this iterative community process and will be included in the

upcoming PCEC v2.2 release.

29


PCEC MOS/GDS Development

Updated CER Results

• Updated SOCM Level 1 Score in MO&DA development resulted in an update to the MOS/GDS Development CERs

• Equations take the form of:

– $k/month = (Parameter 1)α · (Parameter 2)β ·…….. · EXP(Constant)

30


PCEC MOS/GDS Development Performance Comparisons – Updated CER

Mean Error = 6%


Standard Dev. = 35%

NEAR EARTH MISSIONMean Error = 4%


Standard Dev. = 28%

PLANETARY MISSION

31


PCEC Robotic SC SubsystemsComm. (SSPA Planetary) - Updated CER

• Given the small number of planetary S/C using SSPAs in the dataset, there was concern that the previous CER was somewhat oversubscribed.

• In re-examining the data, we found that SSPA mass was the best predictor of cost with a very clear distinction between Mars operating SSPAs and the rest of the planetary SSPA data set.

• Equations take the form of:– $k = (Parameter 1)α · (Parameter 2)β ·…….. · EXP(Constant)

32


PCEC Robotic SC Subsystems

C&DH NRC/RC - Updated CER

33

• Concern was expressed that PCEC C&DH CERs for Recurring Cost (RC) and Non-Recurring Cost (NRC) did not fully capture the complexity of various C&DH implementations.

• In re-examining the data, Non-Recurring and Recurring costs were recombined in order to enable development of a reasonable CER. An updated parameter (# of Boards) was vetted and introduced in to CER process.



PCEC Robotic SC SubsystemsStructures & Mech. (RC) – Updated CER

34

• Mission Risk Class wasbehaving in acounterintuitive manner.Variable was removedwith negligible impact onCER performance.



PCEC Robotic SC Subsystem

ACS (NRC) – Updated CER

s

• Year of Launch input variable was behaving in a counterintuitive manner. Variable was removed with negligible impact on CER performance.

35



PCEC Robotic SC Subsystems

ACS (RC) – Updated CER

• Year of Launch input variable was behaving in a counterintuitive manner. Variable was removed with negligible impact on CER performance.

36



Summary & Findings

• The current PCEC approach has eliminated need for “wrap factors”

across all NASA WBS items except the Science Team

• Prior operations modeling efforts (SOCM) included participation from

operations experts; Current approach is attempting to build on the

insights from past efforts by statistically correlating identified

operations cost drivers to documented costs

• Use of the SOCM Level 1 scores strongly improved parametric

performance for MO&DA and MOS/GDS Development CERs

– The SOCM Level 1 score combines inputs characterizing all elements

and was developed by an integrated team of operations experts and

cost analysts

37

PCEC v2.2 Overview

CASTS MODEL UPDATES

38


CASTS UPDATES

in PCEC v2.2

• Expansion of Full Life Cycle Cost Capability: 2 updates

1. Facilities, Launch and Flight Operations add-ins

• Operations Cost Model (OCM)

2. Prototype multi-system integrated LCC capability

• Multiple vehicles and/or vehicle configurations over multiple years

• Flight-rate-sensitive CER’s

– Fixed and Variable cost as function of flight/production rate

– Cash flow (budget) per year: Spread & Non-spread cost

• Updated CERs

– Minor updates to 8 CERs

– Additional reference data points added to historical data for multiple CERs

• Documentation

– Updated CASTS User’s Guide for v2.2 to be available October

– Virtual Black Books

• PCEC v2.1 Cost Data Sheets completed and uploaded to REDSTAR for all systems included

in CASTS CERs; Version 2.2 updates in work

• REDSTAR Resource Data and system Technical Data sheets completed for Saturn and

Apollo elements (S1C, SII, SIVB, CSM, LM), Skylab Airlock and OWS, Shuttle Orbiter

39

Expanded CASTS LCC

Capability & Context

40

System Definition• Sketch Drawing

• Subsystem Weights & Descriptions

• Mission Profile

• Ops Flow

CASTSAcquisition• $ = f(Indep Var, CPLX, AF, …)

• Indep Var = wt, thrust, SLOC, etc

Outputs:

• DDT&E by WBS element

• TFU by WBS element

OCMFacilities• $ = f(Indep Var, CPLX1, CPLX2, …)

• Indep Var = Footprint, GLOW, thrust

Outputs:

• $ per Facility

Operations• $ = f(Indep Var, ratios, CPLX1, CPLX2, …)

• Indep Var = Mission characteristics, system

element definitions, propellant load, etc.

Outputs:

• CER’s: $/Year as f(flight rate)• $ = (Var CPF x Flt Rt) + Fixed CPY

• $ = [A x Cum Flts^b x Flts/Yr^c] x

Flts/Year

Mission Model

• Flights/Year by System Element & Market Segment: 1) NASA Manned – ISS 2) Unmanned USG 3)Unmanned Commercial

Integrated Model - LCC/Yearly Cash Flow

DDTE• DDT&E Cost Elements

spread by year

Facilities• Facility Cost by facility

as needed – f(capacity,

flight rate)

Production• Reusable Hardware by

element as needed –

f(capacity, flight rate)

Recurring Operations• Recurring Cost by WBS

element– f(flight rate)

DDTEFacilities

Production

Recur Ops


Updated CASTS WBS

• Updated CASTS Work Breakdown Structure

– Addition of Facilities, Launch, and Flight Operations cost elements

– Appears in the Launch Vehicle and Crewed WBSs

Program Segment

Program Mgt & Support

Systems Engr & Integ

Vehicle Segment

Integration, Ass'y, Checkout

Crew Structures

Thrust Structure

Adapters

Secondary/Support Structs

Tanks

Intertank

Mechanisms

Thrust Vector/Flight Control

Separation

Recovery

Other

Main Propulsion Systems

Vehicle Segment (cont'd)

Thermal Protection

Passive

Propulsion

Liquid Engines

Solid Motors

Reaction Ctl/Orb Maneuv Sys

Avionics & Power

Guidance, Nav, & Control

Telemetry & Tracking

Command, Ctl, Data Handling

Range Safety/Flt Termination

Electric Power

Shroud/Fairing

Crew Systems

Vehicle Segment (cont'd)

Software SegmentFlight Software

Ground Software

Test SegmentSystem Test Operations

System Test Hardware

Ground SegmentGround/Test Support Equip

Tooling

Facilities

Launch Operations

Flight Operations


Expanded CASTS WBS

• Expanded WBS elements for Facilities, Launch, and Flight Operations

Ground SegmentFacilities

1 Launch Pad

2 Vertical Processing Facility

3 Horizontal Processing Facility

4 Launch/Mission Control Center

5 Payload Processing Facility

6 Mobile Launch Platform

7 Landing Facility

8 Base Infrastructure

Ground Segment (cont'd)Launch Operations

L1 Vehicle Processing

L2 Processing Engineering

L3 Recovery Operations

L4 Program Mgt & Support

L5 Facility O&M

L6 Base Support

L7 Propellants

L8 GSE Spares

L9 Ground Software O&M

L10 Payload Processing

Ground Segment (cont'd)Flight Operations

F1 Flight Planning

F2 Mission Software

F3 Simulation & Training

F4 Mission Control O&M

F5 reserved

F6 Payload Analytical Integ

F7 Crew Operations

F8 Program Mgt & Support

F9 reserved

F10 Network Support

Taken together CASTS now provides capability to estimate all elements

and phases of crew and space transportation systems Life Cycle Cost.


REDSTAR Overview

43

• REDSTAR is the REsource Data STorage And

Retrieval System managed by the NASA/Marshall

Space Flight Center's Engineering Cost Office.

– Established in 1971

– NASA-wide library of cost, programmatic, and

technical data pertaining to the space program

• REDSTAR is both a physical library and an online

searchable database maintained in SharePoint

– Accessible by NASA Civil Servants and Support

Contractors

– Request access via NAMS (nams.nasa.gov)

• REDSTAR contains over 40,000 documents of

which over 12,500 are available electronically. The

data spans from the total program level down to the

subcomponent level for over a 150 space flight

systems dating back to the 1910’s.

For questions about REDSTAR or for research support,

contact Mary Ellen Harris, REDSTAR Librarian

[email protected]

REDSTAR Document History

mailto:[email protected]


Future Work & Summary

• Ongoing / Future Work

– Milestone Estimating Capability (Robotic SC)

• Currently exploring at correlations between cost inputs at the start of Phase

B, project changes, and Actual Costs and/or Reserve requirements

– CER Library Selection

– Advanced Life Cycle & Architecture Modeling

– PCEC Website Update

• PCEC v2.2 Summary

– Available this week to existing users / requesters

– Usability improvements should make it easier to build and update PCEC

estimates in the future

– More life cycle estimating capabilities will provide the ability to estimate

through the end of the mission

44

PCEC Email Contact: [email protected]

Application Website: https://software.nasa.gov/ , search for PCEC

https://software.nasa.gov/


We always welcome feedback on the

tool / analyses and suggestions on

future capabilities. Find us at the

Symposium or send us an e-mail.

Contact Info

45

Brian Alford

Booz Allen Hamilton

[email protected]

Mark Jacobs

TGS Consultants

[email protected]

Shawn Hayes

TGS Consultants

[email protected]

Richard Webb

KAR Enterprises

[email protected]

Gideon Bass

Booz Allen Hamilton

[email protected]

mailto:[email protected]

BACKUP

46


Principal Component Analysis Approach

1) A correlation matrix was generated

to get a sense of the of the

dependency between variables.

• Several of the variables appeared to be

correlated, making PCA an attractive

method to apply to the data set.

47

2) The principal components were

determined using an algorithm

developed in Python.

• The first 6 principal components which

account for 85% of variance in the data set

were selected and used to determine which

of the 20 variables were most likely related

to cost.

3) For each of the 21 data sets examined, 4 subsets of the 20

variables were run through a multiple regression routine to

determine the new cost estimating relationships.


Process

PCA Results

48

Run RegressionMinimal Model


NASA Space Missions

Modeling Lessons Learned

• PCA can help identify a manageable subset of potential costing inputs that are

the main contributors to cost differences from a much larger candidate set

• A consistent approach for data normalization is essential; Programmatic

differences between the projects can strongly influence official costs

– PCEC normalization adjusts the data to a defined set of rules/procedures

• Do not trust regression results without a thorough sanity check

– Often, “associative” instead of “causal” inputs can yield counter-intuitive results (that may be

misdirected); Best approach maximizes utilization of available “causal” inputs

– It is important to understand reasons for outliers, which can lead to model enhancements

• A combination of PCA, regression, and constructive modelling approaches

appears to offer many benefits over reliance on a single technique

– Enhances flexibility to capture unique aspects associated with NASA robotic science

missions

– Adjustments to regression results need to be supported by data analysis

• Accuracy of technical and cost data should always be reviewed and questioned

– differences often exist in assumptions behind different values for the same

item from different sources

49

pcec v2.2 overview - nasa · pcec overview • pcec is a parametric cost model developed and...

Documents