perspectives on nasa mission cost and schedule performance ...€¦ · perspectives on nasa mission...

1© 2008 The Aerospace Corporation

Perspectives on NASA Mission Cost and Schedule Performance Trends

Presentation for the Future In-Space Operations (FISO) Colloquium

2 July 2008

Bob Bitten

Acknowledgments: Claude Freaner, Dave Bearden, Debra Emmons, Tom Coonce


Typical Questions

• What is the magnitude of cost and schedule growth?

• How reliable are projects’ estimates in the conceptual design stage?

• Why does cost growth occur?

• What is the relationship between cost, schedule and “complexity”?

• Are there any improvements that can be made in estimating the costs of future design concepts?


0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.0 0.5 1.0 1.5 2.0 2.5 3.0

Cost Ratio

Sche

dule

Rat

io

within cost and schedule

within 100% overrun

Forty NASA Robotic Science Missions Experienced 27% Cost and 22% Schedule Growth*

* “Using Historical NASA Cost and Schedule Growth to Set Future Program and Project Reserve Guidelines”, Bitten R., Emmons D., Freaner C.


While Significant Variability is Evident, for Every 10% of Schedule Growth, there is a Corresponding 12% Increase in Cost*

%Cost Growth = 1.2348 * %Schedule GrowthR2 = 0.6124

-50%

0%

50%

100%

150%

200%

0% 20% 40% 60% 80% 100%

Schedule Growth for Non-Restricted Launch Window Projects

Cost

Gro

wth



0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

0.3 m 0.3 m 0.5 m 0.85 m 0.95 m 2.4 m

Gro

wth

per

Dev

elop

men

t Yea

r

Comparison of Schedule Growth Data with Agency Guidelines: NASA Telescope Missions

7 7 13

21 16

36

0

10

20

30

40

50

60

70

80

90

100

110

120

0.3 m 0.3 m 0.5 m 0.85 m 0.95 m 2.4 m

Syst

em D

evel

opm

ent T

ime

(Mon

ths)

Schedule GrowthInitial Schedule

General Rule of Thumb1 Month per Year

NASA/JPL Guidance1.8 Month per Year

Four of Six Telescope Missions Exceeded Common Schedule

Reserve Guidelines

Four of Six Telescope Missions Exceeded Common Schedule

Reserve Guidelines

Growth shown is above planned schedule reserve


Comparison of Schedule Growth and Success for Planetary Missions vs. Earth-orbiting Missions*

0

12

24

36

48

60

72

84

96

0 12 24 36 48 60 72 84 96

Planned development time (mos)

Act

ual d

evel

opm

ent t

ime

(mos

)

Earth Orbiting MissionsPlanetary Missions

Planetary Earth-Orbiting

Sample Size 10 56

Schedule Growth 3.9% 38.3%

• Development times for Planetary missions less than Earth-orbiting missions due to constrained launch windows

• Planetary missions experienced less schedule slip on average than earth-orbiting missions

• However, planetary missions failed or impaired twice as often

0 10 20 30 40 50

Failures

Successes

Average Development Time

Outcome Planetary Earth-orbiting

% Successful 30% 84% % Partial 40% 7% % Catastrophic 30% 9%

* “The Effect of Schedule Constraints on the Success of Planetary Missions”, Bitten R.E., Bearden D.A., Lao N.Y. and Park, T.H.


Typical Questions







How Reliable are the Projects’ Estimates at the Conceptual Design Stage and How Does Confidence Progress?

Ten Missions Demonstrate How Accuracy of Project Estimates Increases Over Time however Cost Growth, Over and Above Reserves, Still Occurs

Deep into the Project Life Cycle

Ten Missions Demonstrate How Accuracy of Project Estimates Increases Over Time however Cost Growth, Over and Above Reserves, Still Occurs

Deep into the Project Life Cycle

39.3%

29.6%

33.7%

0%

10%

20%

30%

40%

50%

60%

70%Pe

rcen

t Cos

t Diff

eren

ce fr

om M

ilest

one

to F

inal

Cos

t (%

)

Avg.

Avg.

Avg.

Start of Phase B @ PDR @ CDR


In What Phase Does Cost Growth Occur?

Greatest Growth Occurs During Integration and Test (Phase D) When Trying to Get Hardware & Software to Function as Designed

Greatest Growth Occurs During Integration and Test (Phase D) When Trying to Get Hardware & Software to Function as Designed

13.3%

47.9%

6.4%

-20%

0%

20%

40%

60%

80%

100%

120%

140%Pe

rcen

t Cos

t Gro

wth

from

Pre

viou

s M

ilest

one

(%)

Avg.

Avg.

Avg.

ATP-PDR PDR-CDR CDR-LaunchPhase B Phase C Phase D


Typical Questions







Some of the Reasons

• Inadequate definition of technical and management aspects of a program prior to seeking approval

(NASA’s Project Management Study, 1980)

• Program and funding instability; difficulties in managing programs in an environment where funding must be approved annually and priorities change

(Advisory Committee on the Future of the U.S. Space Program, 1990)

• Lack of emphasis on technological readiness and requirements on the front end of a program

(NASA’s Roles and Missions Report, 1991)

• Program redesign, Technical Complexity, Budget Constraints, Incomplete Estimates

(GAO Report on NASA Program Costs, 1992)


The Reasons for Growth - Study of 40 NASA Missions:Internal versus External Factors Driven-Growth*

• Internal Growth (within Project’s control)

– Technical• Spacecraft development difficulties • Instrument development difficulties• Test failures• Optimistic heritage assumptions

– Programmatic• Contractor management issues• Inability to properly staff an activity

• External Growth (outside Project’s control)

– Launch vehicle delay– Project redesign– Requirements growth– Budget constraint– Labor strike– Natural disaster

External Only

20.0%

Both10.0%

No Growth12.5%

Internal Only

57.5%

Distribution of Growth

S/C Only22.2%

Other14.8%

Inst. Only33.3%

Both Inst & S/C29.6%

Launch83.3%

Other16.7%

Distribution of Internal Growth

Distribution of External Growth



Mass Growth Exceeds Typical Guidance*

• Average mass growth for ten missions studied is 43% which exceeds the typical industry guidelines of 30% mass reserves (over CBE) at the start of Phase B

43%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Mission#1

Mission#2

Mission#3

Mission#4

Mission#5

Mission#6

Mission#7

Mission#8

Mission#9

Mission#10

Average

Sate

llite

Mas

s G

row

th R

elat

ive

to C

BE

(%)

Typical Reserves

* “An Assessment of the Inherent Optimism in Early Conceptual Designs and its Effect on Cost and Schedule Growth”, Freaner C., Bitten R., Bearden D., and Emmons D.


Assessing Relationships for Causality:Inherent Optimism in Initial Design & Estimates*

Progression of Average Cost Growth for Discovery Selections May be Indicative of Competitive Pressures Leading to More Aggressive DesignsProgression of Average Cost Growth for Discovery Selections May be

Indicative of Competitive Pressures Leading to More Aggressive Designs

-8.4%

21.4%

40.3%42.7%

0.0%

-20%

-10%

0%

10%

20%

30%

40%

50%

1993 1995 1997 1999 2001

Discovery Selection Year

Aver

age

Perc

ent C

ost G

row

th fo

r Di

scov

ery

Mis

sion

s Se

lect

ed



Typical Questions







Hypothesis*

• Complexity Index could be derived using a broad set of parameters to arrive at a top-level representation of the system

• Correlation to spacecraft cost and/or development time based on actual program experience might be apparent

• Data assembled for most spacecraft launched during past two decades (1989 to present) including technical specifications, costs, development time, mass properties and operational status

• Complexity Index calculated based on performance, mass, power and technology choices for purposes of comparison

• Relationship between complexity and “failures” investigated compared with adequacy of cost and schedule resources

• Method to assess complexity at the system-level should allow more informed overall decisions to be made for new systems being conceived

Illustrations reprinted courtesy of NASA* “A Complexity-based Risk Assessment of Low-Cost Planetary Missions: When is a Mission Too Fast and Too Cheap?”, Bearden, D.A.


Payload Mass (kg) 0 265 6065 90 55%Payload Orbit Average Power (W) 0 166 1600 62 38%Payload Peak Power (W) 0 174 750 85 31%Payload Data Rate (average) (Kbps) 0 11678 304538 175 55%Number of Instruments 1 4 18 3 43%Aperture diameter (cm) 3 67 240 60 58%Foreign Partnership None GS, LV, SC Bus PL, mult PL 0 0%Mission Design Life (mos) 0 39 240 7 5%Launch Mass Margin (%) -4% 23% 60% 24% 43%Spacecraft Launch Mass (Wet) (kg) 12 1076 18189 973 73%Spacecraft Mass (Dry) (kg) 6 806 16329 916 77%Spacecraft Bus Dry Mass (kg) 26 598 10264 589 76%Spacecraft Heritage (%) 0% 42% 100% 57% 37%Radiation Total Dose (krads) 0 74 600 50 61%Level of Redundancy (%) 0% 36% 100% 25% 49%Orbit Regime STS/ISS, GEO LEO/MEO, H-LEO/Dip, NE Interplan (au) 3 90%

Payload Mass (kg) 0 265 6065 90 55%Payload Orbit Average Power (W) 0 166 1600 62 38%Payload Peak Power (W) 0 174 750 85 31%Payload Data Rate (average) (Kbps) 0 11678 304538 175 55%Number of Instruments 1 4 18 3 43%Aperture diameter (cm) 3 67 240 60 58%Foreign Partnership None GS, LV, SC Bus PL, mult PL 0 0%Mission Design Life (mos) 0 39 240 7 5%Launch Mass Margin (%) -4% 23% 60% 24% 43%Spacecraft Launch Mass (Wet) (kg) 12 1076 18189 973 73%Spacecraft Mass (Dry) (kg) 6 806 16329 916 77%Spacecraft Bus Dry Mass (kg) 26 598 10264 589 76%Spacecraft Heritage (%) 0% 42% 100% 57% 37%Radiation Total Dose (krads) 0 74 600 50 61%Level of Redundancy (%) 0% 36% 100% 25% 49%Orbit Regime STS/ISS, GEO LEO/MEO, H-LEO/Dip, NE Interplan (au) 3 90%

Factor Unit Min Mean Max ExampleFactor Unit Min Mean Max Example

Complexity Index Example

BOL Power (W) 12 761 8000 1750 89%EOL Power (W) 3 653 6600 1651 92%Solar Array Area (m^2) 0 5 58 7.5 82%Solar Cell Type/Power Source Si GaAs, GaAs-mult GaAs-conc, RTG/R GaAs-mult 75%Battery Type Lead-acid NiCd, SNiCd NiH2, Li-Ion Li-Ion 100%Battery Capacity (A-hr) 1 36 516 266 98%# Articulated Structures 0 1 6 2 87%# Deployed Structures 0 2 9 3 81%Mech. Degrees of Freedom (max) 0 1 6 2 63%Solar Array Configuration body-fixed deployed, single-axis articulated B 0%Structures Material Aluminum Al w/Comp-face, Exotic Composite Al 0%

Pointing Accuracy (deg) 0 2 35 0.0039 88%Pointing Knowledge (deg) 0 1 20 0.0036 80%Platform Agility (slew rate) (deg/sec) 0 1 5 0.62 80%Pointing Stability (Jitter) (urad/sec) 0 87 524 5.000 64%Number of Thrusters+Tanks (#) 0 6 26 18 86%Propulsion Type None, Cold-Gas Mono, Biprop-(blow,pres) OB+US, Ion mono 40%Total Impulse (delta-V) (m/sec) 0 314 5845 190 61%Downlink Comm Band UHF/VHF/SHF S, L, X K/Ka/Ku X 75%

Max Uplink Data Rate (kbps) 0 38 2000 2.0 27%Transmitter Power (peak) (W) 1 10 60 20 85%Central Processor Power (Mips) 0 58 1600 119 81%Onboard Software Code (KSLOC) 2 78 650 110 79%Flight Software Reuse (%) 0% 36% 90% 47% 44%Data Storage Capacity (Mbytes) 0 4186 136000 512.0 60%Thermal Type passive heaters, semi-active active, cryo heaters 25%Multi-Element System? single-sc CL, mult (aerobr, rend) entry/landed/dock mult 66%

Complexity IndexNormalized Complexity Index

60%79%


Schedule as Function of Complexity y = 20.084e1.7203x

R2 = 0.7165

01224364860728496

108120132144

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Complexity Index

Dev

elop

men

t Tim

e (m

onth

s)

Successful

Failed

Impaired

To-be-determined

When is a Mission Too Fast?*

* “A Complexity-based Risk Assessment of Low-Cost Planetary Missions: When is a Mission Too Fast and Too Cheap?”, Bearden, D.A.


Development Cost as Function of Complexity y = 5.6931e5.9893x

R2 = 0.8973

10

100

1000

10000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Complexity Index

Cos

t (FY

05$M

)

Successful

Failed

Impaired

To-be-determined

When is a Mission Too Cheap?*

* “A Complexity-based Risk Assessment of Low-Cost Planetary Missions: When is a Mission Too Fast and Too Cheap?”, Bearden, D. A.


3-D Trade Space – Intuitive Result: Missions that have the greatest complexity, are highest cost and longest development*

Higher Cost,Longer ScheduleMore Complex

Missions

* “A Quantitative Assessment of Complexity, Cost, And Schedule: Achieving A Balanced Approach For Program Success”, Bitten R.E., Bearden D.A., Emmons D.L.


2.5 Year$50MCap

Complexity Bands vs. Cost and Schedule Help Proposers Define Scope of Mission to Fit Fixed Cost & Schedule*

$50 $150 $250 $350 $450 $550 $65024

36

48

60

72

Complexity

Development Cost (FY04$M)

Sche

dule

(Mon

ths)

0.75-0.80.7-0.750.65-0.70.6-0.650.55-0.60.5-0.550.45-0.5

75% - 80%

70% - 75%

65% - 70%

60% - 65%

55% - 60%

45% - 50%

50% - 55%

4 Year$350M

Cap

* “A Quantitative Assessment of Complexity, Cost, And Schedule: Achieving A Balanced Approach For Program Success”, Bitten R.E., Bearden D.A., Emmons D.L.


NASA’s Report Card Following Mars ’98 Failures

• Complexity of Failed Missions High in Both Catagories!

• Planetary Missions are “Fastest”– But fail more often than earth-orbiters

• NASA Earth-Orbiting Missions are “Cheapest”

– But longer to develop than planetary• Overall Success Record is About 3

out of 4 !

NASA Planetary

NASA Earth Orbiting All NASA

Average Complexity of Failed/Impaired Missions 94% 91% 93%Average Complexity of Successful Missions 70% 55% 58%Overall Average Complexity 82% 60% 67%Success Ratio: "Better" 50% 86% 74%Average Development Time: "Faster" (mos) 41 46 44Total Spacecraft Cost: "Cheaper" ($M) 132 75 98

Reprinted with permission of Aviation Weekly and Space Technology

AW&ST 12 June 2000


0

18

36

54

72

90

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Complexity Index

Dev

elop

men

t Tim

e (m

onth

s)

For a project that has fixed requirements and schedule, the inevitable outcome is that cost will grow if developmental problems occur

• Case Study: Mars Exploration Rover (MER)– 90-day surface lifetime; ~9-mos cruise– Launch Mass: 1050 kg (Delta II)– Mobile platform: 1000-m range

• Assessment found that:– 33-month development appeared inadequate– “Open Checkbook” and heritage offset shortfall

• Mitigations:– Focused on rapidly deploying staff to front load

schedule (dual/triple shifts)– Developed extra hardware test-beds

• Cost grew from $299M to $420M

10

100

1000

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Complexity Index

Dev

elop

men

t Cos

t (FY

03$M

)

AW&ST, 26 May 2003

Reprinted with permission of Aviation Weekly and Space Technology


Typical Questions







Example: Substantial Differences Exist between STEREO Science Definition Team (SDT) and Final Implemented Configuration*

STEREO STEREOProgrammatics SDT Final

Schedule (months) 40 70Launch Vehicle Taurus Delta II

TechnicalMass (kg)

Satellite (wet) 211 612Spacecraft (dry) 134 414Payload 69 133

Power (W)Satellite (Orbit Average) 152 515Payload (Orbit Average) 58 108

OtherTransponder Power (W) 20 60Downlink Data Rate (kbps) 150 720Data Storage (Gb) 1 8

SDT Configuration

Final Configuration

Illustrations reprinted

courtesy of NASA



Effect of Increased Complexity on Flight System Cost:STEREO Complexity Increased from 40% to 60%*

System Cost as Function of Complexity y = 11.523e5.7802x

R2 = 0.8832

10

100

1000

10000

0% 20% 40% 60% 80% 100%

Complexity Index

Cos

t (FY

08$M

)

Successful

Failed

Impaired

To-be-determined

STEREO @ SDT

STEREO @Launch



Schedule as Function of Complexity y = 24.22e1.6479x

R2 = 0.6889

01224364860728496

108120132

0% 20% 40% 60% 80% 100%

Complexity Index

Dev

elop

men

t Tim

e (m

onth

s)

Successful

Failed

Impaired

To-be-determinedSTEREO @ SDT

STEREO @Launch

Effect of Increased Complexity on Development Time:STEREO Complexity Increased from 40% to 60%*



Typical Cost-risk Analyses Won’t Capture Large Changes During Concept Evolution*

0%10%20%30%40%50%60%70%80%90%

100%

$200 $250 $300 $350 $400 $450 $500 $550 $600 $650 $700 $750 $800Total Mission Cost (FY07$M)

Perc

ent L

ikel

ihoo

d of

Com

plet

ing

at C

ost

SDT DesignFinal Design

$299M Estimate with 20% Reserve

Final Cost$551M



Inadequate Budget Planning for One Project Results in a Domino Effect for Other Projects in the Program Portfolio

$0

$50

$100

$150

$200

1999 2000 2001 2002 2003 2004 2005 2006

MMSSTEREOSolar-BTIMED

$0

$50

$100

$150

$200

1999 2000 2001 2002 2003 2004 2005 2006

MMSSTEREOSolar-BTIMED

STP 2000 Planned Funding STP Actual Funding History

Total Program Funding 1999-2006• Planned = $689M• Actual = $715M

Although the total program funding remained essentially the same over this time period, implementation of successive missions (e.g. MMS) was

substantially affected

Although the total program funding remained essentially the same over this time period, implementation of successive missions (e.g. MMS) was

substantially affected

MMS

STEREO

MMS

STEREO


Summary

• Methods exist to estimate cost and schedule at the conceptual phase albeit with some level of uncertainty

• The greatest growth manifests itself late in project during Integration & Test

• Data highlighted that the primary reason for cost and schedule growth is internal project technical and development issues often associated with instruments

• Initial project estimates may be unreliable due to design and technology immaturity and inherent optimism

• Success dependence on system complexity and adequacy of resources observed with identification of a “no-fly zone”

• Better technical and programmatic appraisal early in lifecycle is needed along with independent assessment of design and programmatic assumptions


References and Further Reading

1) Bearden, David A., “A Complexity-based Risk Assessment of Low-Cost Planetary Missions: When is a Mission Too Fast and Too Cheap?”, Fourth IAA International Conference on Low-Cost Planetary Missions, JHU/APL, Laurel, MD, 2-5 May, 2000.

2) Dornheim, Michael, "Aerospace Corp. Study Shows Limits of Faster-Better-Cheaper", Aviation Week and Space Technology, 12 June 2000.

3) Bearden, David A., "Small Satellite Costs", Crosslink Magazine, The Aerospace Corporation, Winter 2000-2001.4) Dornheim, Michael, “Can $$$ Buy Time?", Aviation Week and Space Technology, 26 May 2003.

5) Bitten R.E., Bearden D.A., Lao N.Y. and Park, T.H., “The Effect of Schedule Constraints on the Success of Planetary Missions”, Fifth IAA International Conference on Low-Cost Planetary Missions, 24 September 2003.

6) Bearden, D.A., “Perspectives on NASA Robotic Mission Success with a Cost and Schedule-constrained Environment”, Aerospace Risk Symposium, Manhattan Beach, CA, August 2005

7) Bitten R.E., Bearden D.A., Emmons D.L., “A Quantitative Assessment of Complexity, Cost, And Schedule: Achieving A Balanced Approach For Program Success”, 6th IAA International Low Cost Planetary Conference, Japan, 11-13 October 2005.

8) Bitten R.E., “Determining When A Mission Is "Outside The Box": Guidelines For A Cost- Constrained Environment”, 6th IAA International Low Cost Planetary Conference, October 11-13, 2005.

9) Bitten R., Emmons D., Freaner C., “Using Historical NASA Cost and Schedule Growth to Set Future Program and Project Reserve Guidelines”, IEEE Aerospace Conference, Big Sky, Montana, March 3-10, 2007.

10) Emmons D., “A Quantitative Approach to Independent Schedule Estimates of Planetary & Earth-orbiting Missions”, 2008 ISPA-SCEA Joint International Conference, Netherlands, 12-14 May 2008.

11) Freaner C., Bitten R., Bearden D., and Emmons D., “An Assessment of the Inherent Optimism in Early Conceptual Designs and its Effect on Cost and Schedule Growth”, 2008 SSCAG/SCAF/EACE Joint International Conference, Noordwijk, The Netherlands, 15-16 May 2008.

perspectives on nasa mission cost and schedule performance ...€¦ · perspectives on nasa mission...

Documents