one nasa cost engineering database (once)

Office of the Chief Financial Officer

One NASA Cost Engineering Database (ONCE)

Overview & Update

2017 NASA Cost Symposium

NASA: James K Johnson and Eric Plumer

SAIC: Mike Blandford and Julie McAfee

1


Main Presentation Agenda

• ONCE and CADRE Background (5 mins)

– Brief overview

• By the Numbers (5 mins)

– CADRe Dashboard, Download Dashboard, User Dashboard, and Model Downloads

– Numbers Summary

• ONCE Enhancements Part 1: User-Side Expansions (15 mins)

– New CADRe User Dashboard

– Improved User Charts – Focus on ‘Historical Actuals’• (5) Total Cost, Total Mass, Program Development Duration, Spacecraft Mass, and Instruments by Type

– Data Enhancements• Workforce Data Entry Task, Phasing Allocation Task, Instrument Type & Builder, and Crewed/Manned Data

– New Dashboards• (3) Workforce Data, Cost by Lifecycle Phase, Phasing Allocation

– New Family Box Plots• (6) NAFCOM mass and cost, NICM mass and cost, CADRe mass and cost

• ONCE Enhancements Part 2: Other Expansions (5 mins)

– Online Web Model• ASCoT = Analogy Software Costing Tool, first ever NASA cost model run entirely online at ONCEData.com

– OCFO SID Data Merger• QUIK = Quarterly Information and KDP’s

• Conclusion

• Main Presentation Backup Materials

– Crewed/Manned Data Improvement, Improved Search, Model Portal Updates, Inactive Accounts Summary, and New User Process Details2

ONCE Breakout Session:

Thursday @ 11am

Location: 2nd Floor (2E39)

Note: Numbers shown current as 8 August


Last Time(s)…

– ONCE 2.0 is significant improvement and puts

the database at the center of CAD’s OCFO SID

efforts to build and improve the NASA

community.

– ONCE is now the center of OCFO SID’s

efforts to empower analysts and improve cost

estimating at NASA by providing access to:

• CADRe Data– Active filtering for custom user reports

– CADRe Library

• Auxiliary Data – Normalized datasets

• Reports and Insight– Dynamic graphical & tabular reports

– Structured database reporting

• Models and Tools – Model Portal sharing access across community

3

CADReData

Auxiliary Data

Reports and

Insight

Models and

Tools


What is ONCEData.com?

• www.ONCEDATA.com (aka https://oncedata.msfc.nasa.gov,

aka “ONCE”) is a government website managed by HQ

OCFO SID that provides access to technical, cost, and

other programmatic information about NASA Projects.

• The data primarily comes from CADRe documents which

have a Part A, B, and C– Part A = Long narrative (MS Word doc)

– Part B = Technical Data (MS Excel file)

– Part C = Cost/Programmatic Data (MS Excel file)

– CADRe is the Agency’s formal cost data collection initiative as outlined NPD

71205.E. CADRe is paid for by SID and performed during KDP’s on Projects.

• The website provides a user interface to search and

retrieve data from the CADRe’s– Enables analysts and estimators to quickly build analogy datasets, perform

historical analysis, develop cost estimating relationships, etc.

• Users can output the data retrieved from ONCE to MS Excel

for their own specific analysis needs– Project estimation, Project assessment, independent estimation, research,

proposal development, architecture studies, etc.4

http://www.oncedata.com/

https://oncedata.msfc.nasa.gov/


ONCE BY THE NUMBERS

5


CADRe Dashboard

174 GB & 54K Files

448 Total CADRe,

254 Completed

Post-Launch

6

CADRe’sAdded by

Year


User Dashboard

470 Total UsersAverage Yearly Growth 14%

7


Downloads Dashboard

50% of Downloads from CADRe Library

New Chart integrated to display most

downloaded CADRes

Top 5: Juno, MAVEN, MSL, NUSTAR, MMS,

GPM 8


Model Downloads

1002 Total Unique Model

Downloads Top Downloads: NICM, Polaris, ACEIT9*as of 10 August


Total Unique Model Downloads

• 1002 Total Unique Model Downloads

• Model Download trend is positive

• Updates to the Model Portal drive download

spikes

• Top Unique Downloads:

– NICM* (189) (*Includes CS + Contractor)

– Polaris (136)

– ACEIT* (122) (*Includes JACS)

– MOCET (84)

– TCASE (80)

– SMART (67)

10


Top Unique Downloads

• NICM – 189

• Update spike (Q1 2016)

• Polaris – 136

• Update spike (Q3&4 2016)

• ACEIT – 122

• Trending up? (Q2 2016+)

• MOCET – 84


• TCASE – 80

• Reducing?

• SMART – 67


• Argo – 63

• Now free via BAH11


ONCE Numbers Summary

• Total CADRe = 448

– 2017 = +15, 2016 = +41

• Total Projects = 142

– 2017 = +9

• Total User = 470

– 2017 = +54, 2016 = +138

• Total Size = 55k documents and 173GB

– CADRe Library and Symposium Library most popular

• Total Model Downloads = 1002

– Top Models are: NICM, Polaris, ACEIT, MOCET, TCASE, SMART

12


ONCE ENHANCEMENTS PART 1

13


NEW CADRe Dashboard

Drag these lines to

filter time period

14

Double-click any year to view by Month

Double-click any milestone to view by

Year


NEW User Charts – Historical ‘Actuals’

• 5 new User

Charts

– Data from

LRD, EOM,

& CADRe+

only

– Historical

‘Actuals’

– LONG chart

descriptions

1. Mass

– Total Mission

Mass (Kg)

2. Cost

– Total (BY14$M)

3. Program

Development

Duration

– Total (Mo.)

4. Spacecraft Mass

5. CADRe

Instruments by

Type

15

1 2 3 4 5


NEW User Charts – Cost

• Cost

– LRD, EOM, and

CADRe+ ONLY

– Total Cost

includes

development,

launch vehicle,

and operations.

– Inflation handling

to Base Year

2014 $M

– Scale in

Thousands (K)

from $0 to $5.4B

(e.g. 5.4K $M)

– $1.0B - $1.2B

appear to form a

‘step’

16


NEW User Charts

Program Development Duration• Program

Development

Duration

– LRD, EOM and

CADRe+ ONLY

– Development

Duration in

Months

– Mission level

SRR to LRD

– Source is CADRe

Part C, “Schedule

Data”

– Most results 80

months or less

17


NEW User Charts – Spacecraft Mass

• Spacecraft Mass

– LRD, EOM,

and CADRe+

ONLY

– Total dry mass

(Kg)

– Source is

CADRe Part B,

“Summary

Table”

– May include

mass of Rover,

EDL, or

Carrier/Orbiter,

or S/C(s)

– 9 results

greater than

1500 Kg

18

*MER, MMS, TDRS & STEREO, VABP


NEW User Charts

CADRe Instruments by Instrument Type• CADRe Instruments by

Instrument Type

– LRD, EOM, and

CADRe+ ONLY

– Instrument Name,

Instrument Type,

Builder Type,

Builder, and Center

– Ongoing cleanup of

Instrument Type

and Builder

– (55) Spectrometer &

(41) Radiometer

– Drag the scroll bar

to the right to see all

types

– Click to filter the

table19

*60 types


Dashboard - Data Enhancement

Workforce Data Entry Actions• Workforce data collected in

CADRe

– FTE & WYE by WBS & Total

– CADRe templates have

improved over time and efforts

have increased

– CAD/SID Merger provided

YEDO benefits

• Data Enhancement Task

complete to improve Workforce

data available in ONCE

– ONCE vs CADRe

– Source Documents

• Dashboard shows all actions

taken to enhance Workforce data

20



Cost by Lifecycle Phase A/B/C/D/E• Cost data collected in

CADRe

– Includes $ by Lifecycle

Phase A/B/C/D/E

• LRD, EOM, KDP-F, and

CADRe+ shown

• Enhancement complete to

improve and share this

data*

• Dashboard shows $ by

Phase per Project as

stacked bar chart

21*see later slides for info on ONCE cost phasing allocation task improvements



CADRe Phasing Allocation Task Dashboard• Cost Phasing in ONCE by

Lifecycle Phase A/B/C/D/E has

been implemented

– Major data enhancement with

total transparency on actions

taken

• Categories Created, in order

from no action to most action

– Project Phase All (No Action)

– CADRe Phasing (No Action)

– ONCE Phase A/B (Least Action)

– ONCE Phase C/D

– ONCE Phase A-D

– ONCE Phase All

– Fix Schedule Data (Most Action)

– Create Fiscal Years (Most Action)

22

236 Total CADRe’s

required no action (135+101)

10 Total CADRe’srequired the

most action (7+3)


Dashboard – Family Box Plots

Box Plots Using CADRe & NAFCOM & NICM Data• In addition to CADRe data, ONCE

contains datasets from NAFCOM

and NICM

– NICM v6 (~2014)

– NAFCOM 2012

• Box Plot shows

– Minimum/Maximum

– 1st & 3rd Quartile

– Mean Value (dot)

• Family series on X-Axis

– Directly from the data

• GIF shows removing 4 filters to

view robotic spacecraft cost

– First remove Crewed, then

Engine, then Launch Vehicle

– Results are Cost by Family in

2014$M• Families are Earth Orbiting & Planetary

– Then remove Planetary filter• Families are Earth Orbiting only

• Families Include:

– Communications Earth Orbiting

– Mapping Earth Orbiting

– Observatory Earth Orbiting

• (6) Family Box Plots Available:

– CADRe, NAFCOM, NICM

– Mass & Cost 23

Active Filters for NAFCOM: HW Class,

SC Class, Orbit

Use mouse rollover to view box plot results

Double Click any family for zoom and

table update


ONCE ENHANCEMENTS PART 2

24


NEW - ASCoT – Flight SW Model on the web

• ASCoT Overview

– Analogy Software Costing Tool

– Robotic Flight Software

– Suite of Methods

– Includes Database

• ASCoT/ONCE Details

– Hello World achieved on 18 May• Successful deployment of Django and

Python on ONCE to support ASCoT

• Windows & Open Source living together!

– Single sign-on• Log into ONCE, access ASCoT

– “ONCE Model Portal v2.0”

• Plug for Dr. Jairus Hihn of JPL & Model

Based Systems Engineering team

– ASCoT presentation at 2017 Symposium

25

•Spectral Clustering

•Development Effort Estimate

Cluster Analysis

•Linear Regression

•Development Cost Estimate

Regression Analysis

•Verified Reproduction

•SLOC/Cost/EfforCOCOMOII

•K Nearest Neighbor

•Development Effort Estimate

•K_NN


NEW – CFO SID Only

QUIK – Quarterly Information and KDPs

• SID previously operated the C&S Db (Cost & Schedule Database)

– CAD joined SID in 2016 and began leveraging ONCE to help SID

• Goal: Utilize capabilities of ONCE to quickly enhance the

availability of Project Quarterly data

• Source Data: Consolidated Quarterly data sheets by Project– Approximately 51 Projects

– Quarterly data back to approx. FY2007Q1

– 964 individual quarterly datasheets

• What is QUIK? Automated import, analysis, and review of Quarterly

Datasheets– Online SQL database that leverages the CADRe/ONCE architecture

– Supports automated review, import, analysis, and visualization of Quarterly data

– Provides a controlled and authoritative file library of Quarterlies and other valuable information

– Multiple dynamic outputs, charts, and dashboards with full export to MS Excel/PDF/Image

– Integration with NASA IDMax/NAMS and NDC for user credentials and permissions

26


QUIK – Home Screen

27

Complete User List Total Portfolio LCC $M Over Time Total Portfolio MD UFE $M Over Time


QUIK – Automated Review & Import QA

28

Stage 1 Import Metrics Report Stage 2 Verification Receipt• QUIK is currently

performing 2 primary

functions:

– Automated Review

of new incoming

– Archive & analysis of

historical

• 2-Stage Import Process

– Automated Review

– Import QA

• AR yields Import

Metrics with exceptions

flagged

– AR Import Metrics

Report

• QA yields quantitative

metrics on

copy/verification

– QA Verification

Receipt


QUIK – Import Health Dashboard

29

Health Metrics Over Time Goal = Zero Cost Exceptions

Quantitative Verification %

Exception Count and

Type

Exceptions by Project


QUIK – Main Dashboard

30

Tabbed Interface FY Quarter Selector

Template Legend Links to Tabs

Delta from Previous Quarter

Project Pie Chart LCC $

Program Pie Chart LCC $


Conclusion

• ONCE continues to grow and expand

– User side and other expansions can reach wide or specific audiences

• Enhancements are not just “bells and whistles”

– QA type activities, such as Workforce, Phasing, and Crewed/Manned

tasks lead to new Dashboards

– Data improvements are here to stay, charts and visualizations are

fleeting!

• Improvements for the end-user are the goal

– Access to more data, reliable information, models/tools, and context• Curated charts with specific (long) chart descriptions add context, legends, PDFs, etc.

• Exciting opportunities via OCFO merger and new technology

– OCFO data is amazing and can be complementary to CADRe• Quarterly cost and schedule, Key Decision Point datasheets, etc.

– Deployment of the first online web model & ETL advancements31


Main Presentation Backup Materials

Improved Search, Model Portal Updates, Inactive

Accounts Summary, and New User Process Details

32


Crewed/Manned Data Improvements

• Current # of Crewed/Manned CADRe’s in

ONCE: 11

– (5) SRR, (6) PDR

• Current # of Crewed/Manned Projects: 7

– Ares I, CxP GO, CxP MO, CxP EVA,

GSDO, MPCV Orion, SLS

• Major Challenges:

– Inability to query between different projects

– Minimal data available via ONCE database

– Added tabs in CADRe Part B for various

major HW elements

– Different technical parameters captured

– Significant usage of cell comments and

“notes” with tech info

• Data Expansions:

– ISS Modules & Elements33

Different tech parameters on tabs for major HW elements


Moving to a Crewed/Manned Database

• Data improvement task will move toward a Crewed/Manned database

• Historical CADRe’s will undergo detailed process

• New Crewed/Manned CADRe’s will utilize improved CADRe template

• New Crewed/Manned CADRe’s will utilize improved ONCE import

• SQL Server Integration Service (SSIS) will be leveraged for Extract,

Transform, and Load operations (ETL)

• Final result will provide end-users with ability to create charts, reports,

and queries on all data available in the CADRe

34

General Workflow Process for New Crewed/Manned Capability

Historical CADRe Consolidated File SSIS Code New SQL Db


Improved Search Functions – 4 Examples

• Search is

available &

improved

– CADRe &

Source

Document

– Technical

Parameters

– Symposium

Files

– File Types

• Mission name,

CADRe

Milestone, and

File Location

• Result Name

– Short

description

• Export to

Excel, Filters35

Search CADRe& Source

Documents for “Laser

Altimeter”

Search System Technical

Parameters for Data Rate

Search CADRe& Source Docs

for “Spacecraft” schedules .MPP

file type

Search Symposium files

for “Joint Confidence Level” with

filter by Year


Model Portal & Library Updates

• 2017 Updates to the Model Portal:• MOCET

– Mission Operations– Aerospace Corp. (M. Hayhurst)

• ACEIT– Suite including JACS– Tecolote Research Inc.

• PRICE TP– PRICE TruePlanning– PRICE LLC

• SMART– Schedule Analysis– NASA HQ (SCoPE)

• PCEC CS– Project Cost Estimating– NASA MSFC (A. Prince)

• Available Elsewhere:• Argo

– Free via BAH.com • PCEC

– software.nasa.gov• MOCET

– software.nasa.gov

• Detailed Documentation of Process:– Adding a new Model/Tool– Updating an Existing Model/Tool

ONCE Model Portal – 14 Available*

*some models or versions not available to non-CS36


Inactive Accounts Update

• In 2016 the ONCE team, per

NPR 2810 “Security of

Information Technology”,

performed the biennial

analysis of inactive user

accounts

– Inactive = No login in last

one (1) year

• Implemented tiered

approach for user

communication:

– Tier 1 = Informational

notice via email to login

– Tier 2 = Warning letter via

email to login, if no login,

then automatic

suspension

– Tier 3 = Automatic

suspension with attempt

to contact

• Of 59 total inactive

accounts identified, 25 were

retained through contact

– 21 total suspensions37

31

1513

TIER 1 TIER 2 TIER 3

TOTAL INACTIVE ACCOUNTS

Number of Inactive

59

21 2513

TOTAL INACTIVE

RESULT SUSPENSION

RESULT RETAINED

NO ACTION

RESULTS OF REVIEW


New User Process Summary

• Following slides provide details on New User Process for ONCE

accounts:

– Access Instructions & FAQ are available at www.ONCEDATA.com

(https://oncedata.msfc.nasa.gov)

– New users should request access to ONCE via NAMS

• A new NAMS form has been implemented:

– OGC approved language, terms, and conditions

– Clear instructions and sections for NASA CS vs Contractor

– Language that covers:• NASA Use Only, Eligible Users, Monitoring, Technical Data, Model Portal, and

OCFO

• Standard Training with pointers to help and instructional resources

will be provided

– New users are contacted via email after approval for training38

http://www.oncedata.com/

https://oncedata.msfc.nasa.gov/


New User Request Process Overview

1. A new ONCE user is often directed to the ONCE website to view the FAQ or Access Instructions documents (PDFs)

2. This ultimately sends the person to NASA IdMax/NAMS– NASA CS and Contractors with badges handled easily– Automatically filters those without NASA badge and access to

internal NASA network (e.g. able to access IdMax/NAMS and initiate request)

3. The new user searches for ONCE in NAMS and finds catalog entry for the agency asset

4. User fills out the required form information and agrees to terms of use

– NAMS offers ability to collect name, org, and other information

5. Request is sent through pre-determined approval chain– User specifies some information with their request– NASA Supervisor approval– Asset owner approval (e.g. Eric Plumer, James Johnson, etc.)

6. New account is created by ONCE Systems Administrators

7. New user receives email contact from ONCE & invitation for training

FAQ (PDF, 1 page)

Access Instructions (PDF, 5 pages)


• NAMS Form is what the user sees inside IdMax/NAMS after locating the ONCE asset via search

• Must complete this form to initiate request for access to ONCE

• NAMS Form includes fields for input (shown here) and terms and conditions language

• Fields:– Urgency– Request Type– Justification– Reason for Access– OCFO SID only– Contractor only

Updated NAMS Form - Fields

Request Type links to 3 permission

types

User supplies business justification

for access

User selects primary reason for

access

OCFO SID access requires SID

supervisor name

Contractors must provide additional

information


• Terms and Conditions fondly referred to as, the “Red Language”

– Appears at end of form and user must select the tickbox indicating that they agree to all terms

• Original “Red Language” inherited from legacy

– New language OGC APPROVED includes sections for: NASA Use Only, Eligible Users,

Monitoring, Technical Data, Model Portal, and OCFO

Updated NAMS Form - Terms & Conditions