seari short course series
TRANSCRIPT
seari.mit.edu
SEAri Short Course Series Course: PI.27s Value-driven Tradespace Exploration for System Design Lecture: Lecture 5: Modeling and Exploring the Tradespace Author: Adam Ross and Donna Rhodes Lecture Number: SC-2009-1-5-1 Revision Date: August 11, 2009 A slightly earlier version of this course was taught at PI.27s as a part of the MIT Professional Education Short Programs in June 2009 in Cambridge, MA. The lectures are provided to satisfy demand for learning more about Multi-Attribute Tradespace Exploration, Epoch-Era Analysis, and related SEAri-generated methods. The course is intended for self-study only. The materials are provided without instructor support, exercises or “course notebook” contents. Do not separate this cover sheet from the accompanying lecture pages. The copyright of the short course is retained by the Massachusetts Institute of Technology. Reproduction, reuse, and distribution of the course materials are not permitted without permission.
PI.27s VALUE-DRIVEN TRADESPACE EXPLORATIONFOR SYSTEM DESIGN
Lecture 5Modeling and Exploring the Tradespace
Dr. Donna Rhodes and Dr. Adam M. RossMIT
seari.mit.edu © 2009 Massachusetts Institute of Technology 2
Simulating the Tradespace
• Determine Key Decision Makers• Scope and Bound the Mission• Elicit Attributes
–Determine Utilities• Define Design Vector Elements
–Includes Fixing Constants Vector• Develop Model(s) to link Design
and Attributes–Includes Cost Modeling
• Generate the Tradespace• Tradespace Exploration
MissionConcept
Attributes
Calculate Utility
Develop System Model
Estimate Cost
SystemTradespace
Define Design Vector
Decision Makers
Cost
Util
ity
Cost
Util
ity
seari.mit.edu © 2009 Massachusetts Institute of Technology 3
Outline
• Modeling Techniques– Defining analysis modules– Designing analysis system– Cost modeling– Verification of models
• Verification & Preliminary Exploration Techniques– Understanding sensitivities– Dealing with shifting User needs– Comparing to real systems
Additional Example System:
X-TOS Atmosphere Science Satellite
seari.mit.edu © 2009 Massachusetts Institute of Technology 4
Evaluating Value: Models & Simulations
• Models and Simulations…– Take advantage of advances in computation (hardware)– Can utilize advances in numerical techniques with consideration
for the appropriate level of fidelity (software)– Enable designers to “experiment” before implementing– Can incorporate humans into a real-time collaborative design
environment• Transform design variables into attributes
seari.mit.edu © 2009 Massachusetts Institute of Technology 5
Example Project: X-TOS
Number of Designs Explored: 50488
DESIGN VARIABLES• Mission Scenarios
– Single satellite, single launch– Two satellites, sequential launch– Two satellites, parallel
• Orbital Parameters– Apogee altitude (km) 150-1100– Perigee altitude (km) 150-1100– Orbit inclination 0, 30, 60, 90
• Physical Spacecraft Parameters– Antenna gain– communication architecture– propulsion type– power type– delta_v
kmKm
• Problem: Inadequate drag models cause low-orbit objects to become “lost”
• Need: Better information on atmospheric drag
• Concept: In-situ vehicle carrying known instrument suite
seari.mit.edu © 2009 Massachusetts Institute of Technology 6
X-TOS ScopeAtmosphereIonosphereSpacecraft
Atmospheric Physics Model
Global Atmospheric ModelCurrent State
Global Atmospheric ModelPredict Future State
User-SpecificSystem Integration
AFRL Model
output data “Scientist”User Set
“Space Weather”User Set
“Knowledgeable”User Set
Density, Ionospheric characteristics Database
Other Data Sources
(Various assets)
Ground Processing
Output Raw Data
seari.mit.edu © 2009 Massachusetts Institute of Technology 7
X-TOS ScopeAtmosphereIonosphereSpacecraft
Atmospheric Physics Model
Global Atmospheric ModelCurrent State
Global Atmospheric ModelPredict Future State
User-SpecificSystem Integration
AFRL Model
output data “Scientist”User Set
“Space Weather”User Set
“Knowledgeable”User Set
Density, Ionospheric characteristics Database
Other Data Sources
(Various assets)
Ground Processing
Output Raw Data
Spacecraft part of complex science, space weather, and end user system.
Scoping decision (only design spacecraft) made based on available resources and time, knowledge of team, and maturity of sub-systems
Science user needs chosen; well defined, users available
seari.mit.edu © 2009 Massachusetts Institute of Technology 8
X-TOS AttributesDefined by Science User
(5)(4)(3)
(2)(1)1) Data Life Span2) Data Altitude3) Latitude Diversity4) Time Spent at Equator5) Data Latency
kmkm
2004 2005
DATA
seari.mit.edu © 2009 Massachusetts Institute of Technology 9
Generating DV-Att Mappings
QFD X-TOS
CO
NST
RA
INTS
DESIGN-ATTRIBUTE
Uni
ts
% $ (2
002)
ATT
RIB
UTE
S
Dis
tribu
tion
of A
ltitu
des
Num
ber o
f Alti
tude
Ban
ds
Dis
tribu
tion
of L
atitu
des
Num
ber o
f Lat
itude
s
Dis
tribu
tion
of T
ime/
Cyc
les
Tota
l Dat
a P
oint
s
Sam
ple
Res
olut
ion
Sam
ple
Accu
racy
Dat
a C
ompl
eten
ess/
Mis
sion
Suc
ces
TOTA
L (U
SER
)
Life
cycl
e C
ost
Tota
l (C
UST
OM
ER)
Ease
of M
odel
ing
Tota
l (D
ESIG
NER
)
1 2 3 4 5 6 7 8 9 10 11
VARIABLES Units CONSTRAINTS Weighting1 Orbital parameters 6 param 9 9 9 9 9 9 9 1 3 67 0 02 Number of different orbits integer 9 9 9 9 9 9 1 0 9 64 0 03 Number of s/c in each orbit integer 0 0 0 0 3 9 9 0 9 30 0 04 Satellite lifetime months 0 0 0 0 9 9 0 0 9 27 0 05 Mission Scenario - 0 0 06 Shape - 0 0 0 0 0 3 0 1 9 13 0 07 Communication scenario - 0 0 0 0 0 0 0 0 0 0 9 9 08 Launch (vehicle, date) - 0 0 0 0 0 0 0 0 0 0 9 9 09 Redundancy/Functionality - 0 0 0 0 0 9 0 0 9 18 9 9 0
10 Time/Position determination - 0 0 0 0 0 0 9 9 0 18 0 011 Heritage - 0 0 0 0 0 0 9 9 9 27 9 9 012 Sequencing - 0 0 013 Number of mission cycles integer 0 0 014 Number of s/c in each cycle integer 0 0 0
TOTAL 18 18 18 18 30 48 37 20 57 36 0
Design Variables are proposed to
“perform” in the attributes.
Concepts are aggregation of
the DV
Attributes are elicited from Decision Makers
Qualitative filtering of design drivers and attributes (low,mid,high effect 1,3,9)
Qual. Effect
Poor
Good
Fair
DVMDesign-Value Mapping
seari.mit.edu © 2009 Massachusetts Institute of Technology 10
Generating DV-Att Mappings
QFD X-TOS
CO
NST
RA
INTS
DESIGN-ATTRIBUTE
Uni
ts
% $ (2
002)
ATT
RIB
UTE
S
Dis
tribu
tion
of A
ltitu
des
Num
ber o
f Alti
tude
Ban
ds
Dis
tribu
tion
of L
atitu
des
Num
ber o
f Lat
itude
s
Dis
tribu
tion
of T
ime/
Cyc
les
Tota
l Dat
a P
oint
s
Sam
ple
Res
olut
ion
Sam
ple
Accu
racy
Dat
a C
ompl
eten
ess/
Mis
sion
Suc
ces
TOTA
L (U
SER
)
Life
cycl
e C
ost
Tota
l (C
UST
OM
ER)
Ease
of M
odel
ing
Tota
l (D
ESIG
NER
)
1 2 3 4 5 6 7 8 9 10 11
VARIABLES Units CONSTRAINTS Weighting1 Orbital parameters 6 param 9 9 9 9 9 9 9 1 3 67 0 02 Number of different orbits integer 9 9 9 9 9 9 1 0 9 64 0 03 Number of s/c in each orbit integer 0 0 0 0 3 9 9 0 9 30 0 04 Satellite lifetime months 0 0 0 0 9 9 0 0 9 27 0 05 Mission Scenario - 0 0 06 Shape - 0 0 0 0 0 3 0 1 9 13 0 07 Communication scenario - 0 0 0 0 0 0 0 0 0 0 9 9 08 Launch (vehicle, date) - 0 0 0 0 0 0 0 0 0 0 9 9 09 Redundancy/Functionality - 0 0 0 0 0 9 0 0 9 18 9 9 0
10 Time/Position determination - 0 0 0 0 0 0 9 9 0 18 0 011 Heritage - 0 0 0 0 0 0 9 9 9 27 9 9 012 Sequencing - 0 0 013 Number of mission cycles integer 0 0 014 Number of s/c in each cycle integer 0 0 0
TOTAL 18 18 18 18 30 48 37 20 57 36 0
Design Variables are proposed to
“perform” in the attributes.
Concepts are aggregation of
the DV
Attributes are elicited from Decision Makers
Qualitative filtering of design drivers and attributes (low,mid,high effect 1,3,9)
Qual. Effect
Poor
Good
FairThis process is equivalent to a qualitative
modeling of the tradespaceGoals: 1. Ensure attributes are “driven”2. Remove non-value adding design variables
DVMDesign-Value Mapping
seari.mit.edu © 2009 Massachusetts Institute of Technology 11
Determining the Design VectorQFD X-TOS
CO
NST
RA
INTS
DESIGN-ATTRIBUTE
Uni
ts
degr
ees
year
s
% km min
utes
inte
ger
% $ (2
002)
$/yr
? time
? ?
ATT
RIB
UTE
S
Atm
osph
eric
Cov
erag
e
Life
time
% o
rbit
in d
ata
regi
on
Kno
wle
dge
accu
racy
of a
ltitu
de
Late
ncy
Num
ber o
f sim
ulta
neou
s da
ta p
oint
s
Qua
lity
of d
ata
/ dat
a co
mpl
eten
ess
TOTA
L (U
SER
)
Life
cycl
e C
ost
Res
ourc
e bu
rn
Mul
tiple
mis
sion
on
s/c
Sch
edul
e
Leav
e be
hind
cap
abili
ty
Ris
k
Tota
l (C
UST
OM
ER)
Eas
e of
Mod
elin
g
Tota
l (D
ESIG
NER
)
1 2 3 4 5 6 7 8 9 10 11 12 13 14
VARIABLES Units CONSTRAINTS Weighting1 Orbital parameters 6 param 9 9 9 9 9 9 9 63 1 3 4 02 Number of different orbits integer 9 9 9 9 9 9 1 55 0 9 9 03 Number of s/c in each orbit integer 0 0 0 0 3 9 9 21 0 9 9 04 Satellite lifetime months 0 0 0 0 9 9 0 18 0 9 9 05 Mission Scenario - 0 0 06 Shape - 0 0 0 0 0 3 0 3 1 9 10 07 Communication scenario - 0 0 0 0 9 0 0 9 0 0 9 9 08 Launch (vehicle, date) - 0 0 0 0 0 0 0 0 1 9 10 09 Redundancy/Functionality - 0 0 0 0 0 9 0 9 0 9 9 18 010 Time/Position determination - 0 0 0 0 0 0 9 9 9 0 9 011 Heritage - 0 0 0 0 0 0 9 9 9 9 9 27 012 Sequencing - 0 0 013 Number of mission cycles integer 0 0 014 Number of s/c in each cycle integer 0 0 0
TOTAL 18 18 18 18 39 48 37 0 0 0 21 57 36 0
QFD X-TOS
CO
NST
RA
INTS
0.5
- 11
150
- 100
0
0 - 1
80
0 - 2
4
1 - 1
20
0 -2
00
DESIGN-ATTRIBUTE
Uni
ts
Year
s
Km degr
ees
Hou
rs/d
a
Hou
rs
$M (2
002
ATT
RIB
UTE
S
Dat
a Li
fe S
pan
(Per
Sat
ellit
e)
Sam
ple
Altit
ude
Div
ersi
ty o
f Lat
itude
s co
ntai
ned
in
the
Dat
a Se
t
Tim
e Sp
ent i
n Eq
uato
rial R
egio
n
Late
ncy
TOTA
L (U
SER
)
Life
cycl
e C
ost
Func
tiona
lity
trade
abilit
y
Tota
l (C
UST
OM
ER)
Ease
of M
odel
ing
Tota
l (D
ESIG
NER
)
1 2 3 4 5 6 7 8
VARIABLES Units CONSTRAINTS Weighting1 Perigee Altitude m 150<hp<350 9 9 0 0 1 19 9 9 9 92 Apogee Altitude m 150<ha<1500 9 9 0 3 1 22 9 9 9 93 Inclination deg. 0<I<90 0 0 9 9 1 19 3 3 9 94 delta-V m/s 0<mass<500 9 0 0 0 0 9 9 9 3 35 Comm System Type - AFSCN or TDRSS 0 0 0 0 9 9 1 1 3 36 Antenna Gain - Low or High 0 0 0 0 9 9 3 3 3 37 Propulsion Type - Chemical or Hall 3 0 0 0 0 3 3 3 3 38 Power System Type - Solar or Fuel Cells 3 0 0 0 3 6 3 3 3 39 Mission Scenario - - 9 9 9 9 1 37 3 3 1 1
TOTAL 42 27 18 21 25 43 0 43
Example DVM (midstream) Example DVM (almost final)
Design-Value Matrices…
• Map potential Design Variables to Attributes
• Reflect qualitative assessment of strength of relationship
Goal: minimize number of DV elements and ensure adequate modeling
DVM DVM
seari.mit.edu © 2009 Massachusetts Institute of Technology 12
Growth of the Design Space
Tradespace Size vs. Num DV and Num Steps
1.E+001.E+011.E+021.E+031.E+041.E+051.E+061.E+071.E+081.E+091.E+101.E+111.E+121.E+13
1 2 3 4 5 6 7 8 9 10 11 12
Steps per DV
Trad
espa
ce S
ize
1 DV2 DV3 DV4 DV5 DV6 DV7 DV8 DV9 DV10 DV11 DV12 DV
Beware growth of tradespace size!
X-TOS
Space Tug
• Space Tug runs in seconds on a spreadsheet• X-TOS took a day on a workstation• System “A” (8 steps, 10 design vars) would take 27 years…
A
seari.mit.edu © 2009 Massachusetts Institute of Technology 13
Tradespace Exploration
Compares many designs on a common, quantitative basis– Maps structure of design space onto stakeholder value (attributes)– Uses computer-based models to assess thousands of designs, avoiding limits of local
point solutions– Simulation can be used to account for design uncertainties (e.g., cost, schedule,
performance uncertainty)
Design tradespaces provide high-level insights into system-level trade-offs
Each point represents a feasible solution
Constants
Design Variables Attributes
Model(s)
Typical goal: maximize aggregate benefit (utility) andminimize aggregate cost (lifecycle cost)
seari.mit.edu © 2009 Massachusetts Institute of Technology 14
Defining the Design Space
The potential design space is specified through enumeration, which lists the range and steps possible for each design variable
Sampling is the strategy for selecting a subset of designs to evaluate from the enumerated potential design space
The models to be developed must be capable of assessing the potential design space. The actual tradespace will reflect the sampled subset of
considered designs.
Each point represents a feasible solution
Constants
Design Variables Attributes
Model(s)
• Orbital Parameters– Apogee altitude (km) 150-1100 150, 650, 1100 – Perigee altitude (km) 150-1100 150, 650, 1100– Orbit inclination 0, 30, 60, 90 0, 60
• Physical Spacecraft Parameters– Antenna gain hi, low hi, low– communication architecture TDRSS, direct TDRSS– propulsion type elec, chem elec, chem– power type solar, battery solar– delta_v (km/s) 6-12 6, 9, 12
Enumerated SampledX-TOS Example DV
seari.mit.edu © 2009 Massachusetts Institute of Technology 15
Issues in Selecting DV Levels
• Tradespace usually a discretization of span of design vector set
• Size of tradespace grows quickly, analyst must tradeoff breadth versus depth– More detailed modeling requires more time (depth)– More designs explored requires more time (breadth)
• Various techniques exist to “sample” the tradespace without requiring full factorial– e.g. Random Sampling, Design of Experiments
• Question: Why not use Optimization?Must not loose sight of goal:
Understand Tradespace (not just pick optimum)
seari.mit.edu © 2009 Massachusetts Institute of Technology 16
Sampling Strategies
• For smaller design spaces, use full factorial (size depends on computation considerations)
• For larger ones, need a sampling strategy– Ordered sampling: Latin Hypercube,
Taguchi Design of Experiments– Random: Pick designs until patterns in tradespace
emerge• May need to use optimization techniques to find
patterns, Pareto Front of very large tradespaces
Be wary of how selected sampling strategy may introduce “false” patterns in the tradespace
seari.mit.edu © 2009 Massachusetts Institute of Technology 17
Considerations for Sampling
• Design spaces tend to be “bumpy”*– Non-linear and discontinuous dependencies– Numerous local minima
• Ordered searches presume linear or low-order dependencies, so may be valid only in local regions
• Recommended: – Flat-probability (every design vector element has an
equal chance to be chosen) random sampling– Frequent expert interpretation of resulting tradespace– Adding detail
• Globally• To “interesting” regions of tradespace
*Response Surface Models (RSM) are difficult to develop for such spaces…
seari.mit.edu © 2009 Massachusetts Institute of Technology 18
Example from X-TOS
• Several thousand points necessary to reveal tradespace features• Full enumeration necessary to find sharp knee in Pareto front
- find by local sampling?
Util
ity
Cost
Sampled Tradespace
seari.mit.edu © 2009 Massachusetts Institute of Technology 19
X-TOS Design Vector
Life span•Total V capability (200 to 1000 m/s)
Life span•Power type (fuel/solar)
Life span•Propulsion type (Hall/Chemical)
Latency•Comm Architecture (TDRSS/AFSCN)
Latency•Antenna gain (low/high)
Physical Spacecraft Parameters:
Life span, Altitude, Latitude Diversity, Time at Equator
•Orbit inclination (0 to 90 degrees)
Life span, Altitude•Perigee altitude (150 to 350 km)
Life span, Altitude•Apogee altitude (200 to 2000 km)
Orbital Parameters:
Latitude Diversity, Time at Equator, Cost
•Two satellites, parallel launch
Life span, Cost•Two satellites, sequential launch
Cost•Single satellite, single launch
Mission Scenarios:
First Order Effect:Variable:
NOTE
The contents of the Design
Vector determine the
concept.
Each concept is a set of
design variables.
seari.mit.edu © 2009 Massachusetts Institute of Technology 20
The Constants Vector
• To keep modeling general and adaptable to later changes, do not “hardwire” assumptions into code
• Instead, keep a list (data structure) where “constants” are kept
• Five types (at least):– True constants (g, ) These might still change if your unit system
does…– Constraints (policies, standards…)– Modeling assumptions ($/kg, W/GHz, Margins…)– Quantities associated with design vector choices – Potential design vector elements (things under designers control)
that have been fixed - record why
seari.mit.edu © 2009 Massachusetts Institute of Technology 21
Defining the “Constants” Space
Recall four types of constants:1. True constants (e.g., physical constants)2. Assumed quantities (e.g., technology levels)3. Associated variables (e.g., mass/power for a given design variable level)4. Potential design variables (e.g., “weak” value drivers… for now!)
The models to be developed must be capable of assessing the potential design space. The constants reflect important assumptions in the modeling and should
be explicitly “collected” and communicated when exploring the tradespace
Each point represents a feasible solution
Constants
Design Variables Attributes
Model(s)
• Universal– Radius of Earth 6378.137 km
• Spacecraft module– Payload data rate 5000 bps– Coefficient of drag (CD) 1.7– S/C aspect ratio and shape 2:1:1 cylinder– Van Allen belt altitude 1000 km– Radiation hardening scale factor 1.25– Discount rate 1.9%– Inflation rate (2002) 3.4%– Learning curve slope 95%
X-TOS Example Constants
seari.mit.edu © 2009 Massachusetts Institute of Technology 22
Modeling
Des
ign V
ars
Perigee
Apogee
Del
ta-V
Propuls
ion
Incl
inat
ion
Com
m S
yste
m
Ant.
Gai
n
Pow
er s
yste
m
Mis
sion S
cenar
io
Tota
l Im
pac
t
AttributesData Lifespan 9 9 9 6 0 0 0 6 9 48Sample Altitude 9 9 0 0 0 0 0 0 9 27Diversity of Latitudes 0 0 0 0 9 0 0 0 9 18Time at Equator 0 6 0 0 9 0 0 0 9 24Latency 3 3 0 0 3 9 9 6 3 36Total 21 27 9 6 21 9 9 12 39Cost 9 9 3 6 6 3 6 6 9Total w/Cost 30 36 12 12 27 12 15 18 48
• Start with DVM
Hard orbital mechanics (drag)
Easy orbital mechanics
Comm Sys Design
Number of Sats, etc.
seari.mit.edu © 2009 Massachusetts Institute of Technology 23
Modeling a Tradespace
• Map design variables to attributes (DVM)• Brainstorm “modules” that simulate design variables and
determine attribute values• Create an N-squared to determine information flow• Partition effort to develop software code
X-TOS Module list1. Orbits2. Spacecraft3. Launch4. Mission Scenario5. Cost6. Attributes7. Utility
Output
Satellitedatabase
Cost (lifecycle)
Utility
Mission Scenario
Orbits
Spacecraft
Launch
All variations on design
vector
Mission scenarios with acceptable
satellitesOutput
Satellitedatabase
Cost (lifecycle)
Utility
Mission Scenario
Orbits
Spacecraft
Launch
All variations on design
vector
Mission scenarios with acceptable
satellites
seari.mit.edu © 2009 Massachusetts Institute of Technology 24
Design Structure Matrix (DSM) Principles
• Mapping model modules against each other to clarify interactions - sometimes called N-squared matrix
• ‘X’ indicates that the module in the column provides input to the module in the row
Information Flow
See DSMweb.org for more information on DSMs
seari.mit.edu © 2009 Massachusetts Institute of Technology 25
Using the DSM
• Arrange rows and columns to group interactions, keep interactions “below the diagonal” or as close to the diagonal as possible
• If all the interactions are one way (below the diagonal) iterations can be eliminated (or at least kept within the modules)
seari.mit.edu © 2009 Massachusetts Institute of Technology 26
Using the DSM
• Arrange rows and columns to group interactions, keep interactions “below the diagonal” or as close to the diagonal as possible
• If all the interactions are one way (below the diagonal) iterations can be eliminated (or at least kept within the modules)
x
“above diagonal”interactions would require iteration of entire model (not good)
x“close to diagonal”interactions require
iteration only between two modules (consolidate?)
seari.mit.edu © 2009 Massachusetts Institute of Technology 27
(Parametric) Modeling with Simulation
• Parametric modeling linked with simulation provide both static and dynamic insights into tradespace
• Modular software design enables integration with various software tools and fidelity adjustments
An approach enabling broad tradespace numerical assessment
seari.mit.edu © 2009 Massachusetts Institute of Technology 28
Breadth versus Depth: Model Fidelity
Key considerations when choosing fidelity• “Smoothness” of objective space (enumeration)• “Consistency” of fidelities across models (consistency)• Parametric v. human-in-loop “models” (appropriateness)• “Hooks” for later feedback (verification)
w
w
sat
Low fidelity
Square cylinderAll dimensions in meters
Mid fidelity
Simple CAD model
Feedback
Fidelity feedback
e.g. inability to fit on launch vehicle
Appropriate fidelity choice critical in early design
seari.mit.edu © 2009 Massachusetts Institute of Technology 29
Cost Modeling:Criteria for selecting an Estimating Approach
• Have a balance of– Statistical significance (i.e., good fit to the data)– Practical significance (i.e., intuitively appealing cost drivers)
• Explainable to management– Defensible to skeptics– Reliable enough to justify $M or $B commitments
• General approaches are categorized as– Analogy– Parametric– Bottoms up/Engineering buildup– Extrapolation– Expert opinion (including usage of Delphi method)– Cost factors
seari.mit.edu © 2009 Massachusetts Institute of Technology 30
Selecting a Cost Methodology
NASA Cost Estimating Handbook, 2002.
seari.mit.edu © 2009 Massachusetts Institute of Technology 31
Variation/Uncertainty in Cost Models
• Cost estimation uncertainty may stem from
– inaccuracies in cost-schedule estimation models
– misuse (or misinterpretation) of cost-schedule data
– misapplied cost-schedule estimation methods
– economic uncertainties that influence the cost of technology
– labor force– geo-political policies
Garvey, P. R., “Probability Methods for Cost Uncertainty Analysis-A Systems Engineering Perspective”, CRC Press, 2000.
seari.mit.edu © 2009 Massachusetts Institute of Technology 32
The Cone of Uncertainty
Feasibility Plans/Rqts. Design Develop and TestPhases and Milestones
Relative Size
Range
OperationalConcept
Life Cycle Objectives
Life Cycle Architecture
Initial Operating Capability
x
0.5x
0.25x
4x
2x
Boehm, B., Software Engineering Economics, Prentice Hall, 1981.
seari.mit.edu © 2009 Massachusetts Institute of Technology 33
Unrealistic Cost Estimates of U.S. Space Systems
• U.S. Government Accountability Office found chronic cost growth– Not caused by poor cost estimating– Rather, pressures to secure funding that lead to unrealistic expectations
• Analysis of six space programs found unrealistic expectations
• There are larger pressures to produce low estimates that are more likely to win support for funding
GAO, “Space Acquisitions: DOD Needs to Take More Action to Address Unrealistic Initial Cost Estimates of SpaceSystems,” GAO-07-96, Washington, D.C.: November 17, 2006.
seari.mit.edu © 2009 Massachusetts Institute of Technology 34
Domain-specific Cost Models
Software Engineering• COCOMO II [Boehm et al, 2005]• CostXpert [Cost Xpert Group, 2003]• CoStar [SOFTSTAR, 2006]• Price-S [PRICE, 2006]• SEER-SEM [Galorath, 2001]• SLIM [QSM, 2006]
Systems Engineering• COSYSMO [Valerdi, 2005]
COTS Integration• COCOTS [Abts, 2004]• SEER-SEM [Galorath, 2001]
Hardware Engineering• Price-H [PRICE, 2006]• SEER-H [Galorath, 2001]
seari.mit.edu © 2009 Massachusetts Institute of Technology 35
Model Verification
• Good software practices - make sure code works
• Check sensitivities to both design vector and constants vector elements– Verify both model and assumptions (e.g. if results are
very sensitive to constants vector elements, reconsider assumptions)
– Discussed in tradespace exploration• Verify against more detailed analysis if available
– Conventional studies– Real products– ICE studies
seari.mit.edu © 2009 Massachusetts Institute of Technology 36
Coupled with Tradespace Exploration
More techniques, coupled to issues of modeling:• Examining the effects of uncertainty in “constants”• Uncertainty in user needs and interaction with users• Basic confidence check - Does MATE yield realistic
results?
Remember ultimate goal: turn data generated by model into knowledge for decisions makers.
In addition, generate confidence in model and understandings of its limits
The goal here is to get a feeling for models to verify they produce “reasonable” results before proceeding to deeper tradespace exploration
seari.mit.edu © 2009 Massachusetts Institute of Technology 37
Examining X-TOS Sensitivities
Total Lifecycle Cost
($M2002)
• Multi-satellite mission designs were found to have large extra costs and very small benefits, so were excluded
• Single satellite designs shown
• Pareto designs collect low altitude data (most valuable) at the expense of lifetime
Warning: “good” is not always in the upper left (read the axes!)
“Good” Direction
seari.mit.edu © 2009 Massachusetts Institute of Technology 38
Examining X-TOS Sensitivities
Total Lifecycle Cost
($M2002)
• Multi-satellite mission designs were found to have large extra costs and very small benefits, so were excluded
• Single satellite designs shown
• Pareto designs collect low altitude data (most valuable) at the expense of lifetime
Each point is a specific design
Pareto Front of “best”designs “Dominated”
designs are inferior in cost or utility
Warning: “good” is not always in the upper left (read the axes!)
“Good” Direction
seari.mit.edu © 2009 Massachusetts Institute of Technology 39
Parametric Uncertainty Sourcesin X-TOS
Use a tree diagram to identify key sources of
uncertainty
Utility
Altitude Lifetime
V Drag
atmos CDVelocity Area
AR Volume
s/c MassChosen for sensitivity analysis
seari.mit.edu © 2009 Massachusetts Institute of Technology 40
Sensitivity AnalysisSelect relevant set of designs
Vary key uncertain parameters and observe effect on selected designs’ performances
All Pareto Front designs
Green – Low altitude, short lifetime
Purple – Mid altitude, mid lifetime
Red – Higher altitude, higher lifetime
seari.mit.edu © 2009 Massachusetts Institute of Technology 41
Sensitivity to Satellite Density
• Three designs chosen (colored lines)• Density varied about a nominal value, effect on Utility and Cost plotted• Not very sensitive except at lowest densities
Effects small, but implies there is +/- 5% (?) uncertainty in resultsUncertainty does NOT affect all designs equally
seari.mit.edu © 2009 Massachusetts Institute of Technology 42
Sensitivity to Area Ratio (AR) and Coeff. of Drag (CD)
• Same trend as s/c
• Lifetime decreases as CD increase– Note the non-linear
relationship– Arises from non-linear
utility function for lifetime
AR
CD
seari.mit.edu © 2009 Massachusetts Institute of Technology 43
Sensitivity to Atmospheric Density
• Density varied about nominal value; mean value varied based on solar cycle; solar cycle state assumed constant throughout life
• Effect is most notable at solar max, and is quite large then• Effect does not affect all designs equally
Variation caused by the solar cycle
0.7 0.8 0.9 1 1.1 1.2 1.30.55
0.6
0.65
0.7
0.75
Util
itydensity_multiplier
0.7 0.8 0.9 1 1.1 1.2 1.30.55
0.6
0.65
0.7
0.75
Util
ity
density_multiplier0.8 0.9 1 1.1 1.2 1.3
0.55
0.6
0.65
0.7
0.75
Util
ity
density_multiplier
MIN MEAN MAX
seari.mit.edu © 2009 Massachusetts Institute of Technology 44
Sensitivity to Atmospheric Density
• Density is a key driver of utility• Its value is uncertain
– Uncertainty of launch date leads to uncertainty of location in solar cycle
– Current atmospheric model have large errors– The purpose of this mission is to study atmospheric density!
Design spacecraft to have enough fuel and thrust to dynamically change its orbit in response to current
atmospheric conditions as mission progresses
We will discuss such adaptable approaches later
Design Approach: Dynamically change orbit
seari.mit.edu © 2009 Massachusetts Institute of Technology 45
Sensitivity to User Preferences
User changed preference weighting for lifespan
Design tradespace reevaluated in less than one hour
X-TOS Case
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
Latency Latitude Equator Time Lifespan Altitude
Weight Factors of each Attribute (k values)
Original Revised
40 42 44 46 48 50 52 54 56 58 600.2
0.3
0.4
0.5
0.6
0.7
0.8
40 42 44 46 48 50 52 54 56 58 600.2
0.3
0.4
0.5
0.6
0.7
0.8
Original Revised
Util
ity
Util
ityCost Cost
seari.mit.edu © 2009 Massachusetts Institute of Technology 46
X-TOS Comparison with Real System
Blue points represent X-TOS initial tradespace exploration
Red points represent possible STEP 1 equivalent architectures
Important: Convert points back to attribute values for communicationSTEP 1 mission is X-TOS precursor flown in early 1990s (into ocean)Flown on multi-payload launch, resulting in sub-optimal system
Clearly dominated solutions
Provides info for
negotiation
“Best” set of designs
seari.mit.edu © 2009 Massachusetts Institute of Technology 47
Total Lifecycle Cost($M2002)
Streak - Successful System
X-TOS vs. Streak
“Ion gauge and atomic oxygen sensor”
Three (?)Instruments
75***75 - 72Cost $M
0.590.56 - 0.50 Modified Utility**
0.57 - 0.54*0.61 - 0.55 Utility
MinotaurMinotaurLaunch Vehicle
321a-296p -> 200 @ 96°300 -185 km @ 20°Orbit
12.3 - 0.5Lifetime (yrs)
420325 - 450Wet Mass kg
Streak (Oct 2005 launch)
XTOS (2002 study)
Streak
Inclination different A user preference for data at the terminator unexpressed at
time of X-TOS
• Streak launched 2005• Very similar to Pareto
X-TOS design
Streak values from information in published sources (Aviation Week)*Calculated with X-TOS preferences
**Modified to account for changed preferences***Estimate using X-TOS model
seari.mit.edu © 2009 Massachusetts Institute of Technology 48
Cost,Attributes
• Delta-V• Capability• Response time
Design Space>Manipulator Mass
– Low (300kg)– Medium (1000kg)– High (3000 kg)– Extreme (5000 kg)
>Propulsion Type– Storable bi-prop– Cryogenic bi-prop– Electric (NSTAR)– Nuclear Thermal
>Fuel Load - 8 levels
Simple performance model– Delta-V calculated from rocket equation– Capability solely dependent on manipulator mass– Response time binary (electric propulsion=slow)– Cost calculated from vehicle wet and dry mass
Examining Space Tug Sensitivities
Design Vector
Constants VectorModel(s)
Costs
Attributes/Utilities
Intermediate Variables
Utility functions were assumed, not assessed
How sensitive are the results?
seari.mit.edu © 2009 Massachusetts Institute of Technology 49
Space Tug Parametric Study:Impact of Shifts in User Needs
Unlimited DV demand favors high ISP propulsion
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
0 2000 4000 6000 8000 10000 12000
DV (m/sec)
Leo-Geo
Leo-Geo RT
0
500
1000
1500
2000
2500
3000
3500
4000
0.0 0.2 0.4 0.6 0.8 1.0
Utility (dimensionless)
Biprop CryoElectricNuclear
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
0 5000 10000 15000 20000 25000 30000 35000 40000
DV (m/sec)
Leo-GeoLeo-Geo RT
0
500
1000
1500
2000
2500
3000
3500
4000
0.0 0.2 0.4 0.6 0.8 1.0
Utility (dimensionless)
Biprop CryoElectricNuclear
Cos
t ($M
)
Cos
t ($M
)
Util
ity (d
imen
sion
less
)
Util
ity (d
imen
sion
less
)
seari.mit.edu © 2009 Massachusetts Institute of Technology 50
0
500
1000
1500
2000
2500
3000
3500
4000
0.0 0.2 0.4 0.6 0.8 1.0
Low Capability Medium CapabilityHigh CapablityExtreme Capability
Changing Weightings -Capability Stressed
Spreads front at high-performance end
SPACETUG• General
purpose orbit transfer vehicles
Cos
t ($M
)
Utility (dimensionless)
Utility-Cost Discriminated by Tugging Capability, varied weightings
seari.mit.edu © 2009 Massachusetts Institute of Technology 51
Changing Weightings -Response Time Stressed
Eliminates electric propulsion
0
500
1000
1500
2000
2500
3000
3500
4000
0.0 0.2 0.4 0.6 0.8 1.0
Biprop CryoElectricNuclear
SPACETUG• General
purpose orbit transfer vehicles
Cos
t ($M
)
Utility (dimensionless)
Utility-Cost Discriminated by Prop Subsys, varied weightings
seari.mit.edu © 2009 Massachusetts Institute of Technology 52
General InsightsBy varying the utility functions, can find designs
that may interest various customers
seari.mit.edu © 2009 Massachusetts Institute of Technology 53
Full Scale Development of an Electric Cruiser:Orbital Recovery Corp.
Orbital Recovery Corp.
“Orbital tugboat” to supply propulsion, navigation and
guidance to maintain a satellite in its orbital slot for 10+ years
Electric Cruiser (2002 study) CX-SLES (2009 launch) Wet Mass kg 1405 1400 Dry Mass kg 805 670* Propellant kg 600 730* Equipment kg 300 213* DV m/s 12000 – 16500*** 15900** Utility 0.69 0.69 Cost 148 130*
CX-OLEV values from information in published sources (ORC website)
*Back-calculated using models**,***Accounts for changed assumption on ISP Project cancelled??
ConeXpress – Orbital Life Extension Vehicle
seari.mit.edu © 2009 Massachusetts Institute of Technology 54
Orbital Express:An Experimental Tender
Orbital Express (OE)
On-orbit validation of autonomous docking, refueling, component
swapping; focus on non-proprietary interfaces
Flown 2007
Utility
Cos
t ($M
)
seari.mit.edu © 2009 Massachusetts Institute of Technology 55
Summary
• Modeling Techniques– Defining analysis modules– Designing analysis system– Cost modeling– Verification of models
• Related Tradespace Exploration Techniques– Understanding sensitivities in model and constants– Understanding sensitivities to DM needs– Comparing to real systems
seari.mit.edu © 2009 Massachusetts Institute of Technology 56
References for Cost Models• Boehm, B., Abts, C., Brown, A. W., Chulani, S., Clark, B. K., Horowitz, E., Madachy, R., Reifer,
D., Steece, B., Software Cost Estimation with COCOMO II, Prentice Hall, New Jersey, 2000.• Cost Xpert Group, Inc. Cost Xpert 3.3 Users Manual. San Diego, CA: Cost Xpert Group, Inc.,
2003.• PRICE, Program Affordability Management, http://www.pricesystems.com• QSM, SLIM-Estimate, http://www.qsm.com/slim_estimate.html • Galorath, Inc. SEER-SEM Users Manual. www.galorath.com• SOFTSTAR, http://softstarsystems.com• Valerdi, R., The Constructive Systems Engineering Cost Model (COSYSMO), PhD Dissertation,
University of Southern California, May 2005.• Abts, C., Extending the COCOMO II Software Cost Model to Estimate Effort and Schedule for
Software Systems Using Commercial-Off-The-Shelf (COTS) Software Components: the COCOTS model, PhD Dissertation, University of Southern California, 2004.
• Space Systems Cost Analysis Group (http://sscag.saic.com/)• Wertz, J. R. and Larson, W. J. (Eds.), Space Mission Analysis and Design, 3rd Edition, Microcosm
Press and Kluwer Academic Publishing, 1999. (Chapter 20 - Cost Modeling)• International Society of Parametric Analysts (http://www.ispa-cost.org/)• Valerdi, R., Wheaton, M. J., Fortune, J., “Systems Engineering Cost Estimation for Space
Systems,” AIAA Space 2007.