strategic risks: complex systems · environmental lens 1 $ 104.2 b $ 75.4 b $ (131.9) to $96.1 b...
Post on 20-Jul-2020
0 Views
Preview:
TRANSCRIPT
AFRL S5 Symposium 2014
Yuri Gawdiak Manager Strategic Analysis
Strategy, Architecture and Analysis Office (SAA)
Aeronautics Research Mission Directorate (ARMD)
National Aeronautics & Space Administration (NASA)
yuri.o.gawdiak@nasa.gov June 12, 2014
Strategic Portfolio Analysis:
Complex Systems
Development Risks
NextGen Enterprise Architecture 2009
$(150.00)
$(100.00)
$(50.00)
$-
$50.00
$100.00
$150.00
$200.00
$250.00
FY 2
010
FY 2
011
FY 2
012
FY 2
013
FY 2
014
FY 2
015
FY 2
016
FY 2
017
FY 2
018
FY 2
019
FY 2
020
FY 2
021
FY 2
022
FY 2
023
FY 2
024
FY 2
025
FY 2
026
FY 2
027
FY 2
028
FY 2
029
FY 2
030
FY 2
031
FY 2
032
FY 2
033
FY 2
034
FY 2
035
FY 2
036
FY 2
037
FY 2
038
FY 2
039
FY 2
040
FY 2
041
FY 2
042
FY 2
043
FY 2
044
FY 2
045
FY 2
046
FY 2
047
FY 2
048
FY 2
049
FY 2
050
$(150.00)
$(100.00)
$(50.00)
$-
$50.00
$100.00
$150.00
$200.00
$250.00
Cost
Benefit
NPV
Note: Costs represented in the table and charts are based on point estimate results; these figures do not reflect all
estimating uncertainty, the influence of any programmatic risk –adjustment, or the impact due to environmental
factors
Incremental Cumulative NextGen Net Cash Flows (PV $M)
$104.2 B
In B
illio
ns o
f D
iscounte
d D
olla
rs
NPV: $ 104.2 B Discounted Payback: 2027 Benefit/Cost Ratio: 2.00 IRR: 10%
The Society/Passengers NPV for the Initial Alternative results in a positive $104.2B, without accounting for environmental impact or risk
2027 2050
SOCIETAL NET PRESENT VALUE JPDO NGOps-5 Architecture by 2025
IPSA Results: Monetized impacts of policy relative to baseline (2009 billions of US dollars) for initial alternative
ENVIRONMENTAL ANALYSIS
• The table shows monetized environmental impacts of the NextGen N+1 policy relative to the baseline in billions of 2009 U.S. dollars for the lower, mid, and upper bound assumptions (including discount rates within the recommended OMB and EPA ranges).
• Lower discount rates place greater value on future years, which contributes to the higher cost of impacts, especially for CO2 effects on climate.
• Negative values represent projected impact reductions (improvements) for policy minus baseline; positive values are projected costs, which stem primarily from the increased number of flights.
• Climate results reflect global impacts and are integrated out to 800 years, which is consistent with EPA guidance. Noise and Air Quality results reflect regional impacts and are assumed to be realized during the period of operation (2006-2050).
• The monetized environmental impacts are sensitive to many assumptions, as reflected in the low, mid, and upper bound ranges. For instance, climate estimates are sensitive to the following assumptions: discount rates, climate sensitivity, damage coefficients, short-lived radiative forcing, background CO2 scenario, NOx effects.
• Important to Note: Results are highly sensitive to emissions, noise, and fuel burn inventories provided by IPSA
Environmental Impacts 2006-2050 Mid-range Upper Bound Lower Bound
Discount Rate 3.5% 2.0% 5.0%
Climate $63bn $808bn $2bn
Air Quality -$4bn -$36bn -$0.4bn
Noise (for ≥ 55 dB DNL) -$29bn -$229bn -$3bn
Total (NextGen Policy minus Baseline) $30bn $543bn -$1.4bn
When comparing programs from various studies and sources, the
average cost overrun percentage ranged from 28% to 79%
Program Type Number of
Programs
Average
Program Cost
Growth
Dollar
Weighted
Average
Growth*
Cost
Growth
Standard
Deviation
FAA 11 79% 112% 86%
NASA 83 45% 59% 67%
DoD Weapons
Systems 76 45% ~40% 80%
Public Works
Transportation 258 28% Not Available 39%
* Percent overrun weighted by the program dollar value relative to total dollar value of all programs
Cost Overruns of Federal Programs
When considering environment and programmatic risks, the
NPV for Society/Passengers drops substantially
Point
Estimate
Environment
Impact
Unmitigated Historical
Risk-Adjusted
Discounted Dollars NPV NPV NPV
Environmental Lens 1
$ 104.2
B $ 75.4 B $ (131.9) to $96.1 B
Environmental Lens 2
$ 104.2
B $ (439.9) B $ (647.2) to $ (419.1) B
Environmental Lens 3
$ 104.2
B $ 105.7 B $ (101.6) to $126.4 B
Based on an assessment of the Initial Alternative, environment and
risks severely impact the NPV for NextGen
• Environment can negatively impact the NPV by a factor of 5
• Risks can negatively impact the NPV by a factor of 3
Including Environment Including Historic Risk
INTEGRATED RESULTS, SUMMARY, and RECOMMENDATIONS
Question: How do we stand now? Five Years Later?
Complex Hardware & Software Systems key examples
The new program baseline projects total acquisition costs of $395.7 billion, an increase of $117.2 billion (42 percent) from the prior 2007 baseline. Full rate production is now planned for 2019, a delay of 6 years from the 2007 baseline. Unit costs per aircraft have doubled since start of development in 2001. Since 2002, the total quantity through 2017 has been reduced by three-fourths, from 1,591 to 365. The original plan was that about 70 percent of all the parts on the airplanes would be common; the actual figure today is about 25 percent.
FAA now projects that ERAM will be almost 4 years behind schedule, with an uncertain final completion date. If problems persist, cost increases could reach in excess of $500 million and interfere with program execution. FAA’s problems in advancing ERAM are attributable to a number of fundamental program management weaknesses that have impeded the Agency’s ability to effectively implement ERAM and effectively manage other major acquisitions.
JWST was projected to cost just $1 billion to build and launch. By 2011, however, the program had seen almost a decade of cost overruns and schedule delays. Under pressure from lawmakers, NASA rebaselined the program with a revised cost estimate of $8.8 billion, and a new launch date of October 2018 - 10 years later than the original date of 2008.
NASA JWST FAA ERAM
DoD JSF
Simplified FAA ATM System Diagram Existing Environments are Complex
And we want to add new Functions & More Complexity
Airport Robotic Systems
Control Towers
Safety & Security Systems
AutoMax Single Pilot Ops
Airline & Airport Ops Centers Passengers/Public Ticketing & Scheduling Systems
Manufacturers Operations Center (Boeing)
Weather & External Services
FAA Command Center
Pilots/Intelligent Flight Decks
Intelligent Autonomous Systems
Gates & Ramps
Gate to Gate TBO/4DT/Autonomous
Intelligent Swarming Systems
Complete Success Requires Addressing Complex Organizational
Environments
UAS Stakeholders – Joint Planning & Development Office Coordination
― Policy/Regulatory
― Research and Development
― Operators
― Operations
― Strategic
― NAS Community & Public Advocacy
― Manufacturers
This is a high-level view
of the UAS Stakeholders
and their associated
roles/functions, which
are categorized by the 7
bins represented in the
Key below
This is a one time slice. Hand generated, expensive and already out of date. V&V of complex systems/ops require this to be continuously updated.
Key Research & Development Related
Strategic Decisions*
* Results based on a 20% sampling of the JPDO Enterprise Architecture
RISK ANALYSIS & RESULTS 2009 JPDO IWP
Note: Intensity is a function of the frequency and complexity of decision events
Question: Are we missing something obvious in our complex
problem spaces?
Humans/Organizations V&V?: Configuration Management, Baselines, Test, Demonstrations & Formal Methods
V&V?:
Software Hardware
Total System of Systems
V&V?: Configuration Management, Baselines, Test, Demonstrations & Formal Methods
V&V?:
Software Hardware
Total System of Systems
Humans/Organizations
1. Most Dynamic 2. Greatest Lifecycle
Impact 3. Least Predictable 4. Least Calibrated 5. Least Tracked 6. Least Supported
Columbia Accident Investigation
Board Organizational Issues After the accident, Program managers stated privately that if engineers had a safety concern, they were obligated to communicate their concerns to
management. Managers did not seem to understand that as leaders they had a corresponding and perhaps greater obligation to create viable routes for the
engineering community to express their views and receive information. This barrier to communications not only blocked the flow of
information to managers, but it also prevented the downstream flow of information from managers to engineers, leaving
Debris Assessment Team members no basis for understanding the reasoning behind Mission Management Team decisions.[1]
Encouraging Minority Opinions: The Naval Reactor Program encourages minority opinions and “bad news.” Leaders continually emphasize that when
no minority opinions are present, the responsibility for a thorough and critical examination falls to management. Alternate perspectives and critical
questions are always encouraged. In practice, NASA does not appear to embrace these attitudes. Board interviews revealed that it is difficult for
minority and dissenting opinions to percolate up through the agency’s hierarchy, despite processes like the anonymous NASA
Safety Reporting System that supposedly encourages the airing of opinions.[2]
NASA had conflicting goals of cost, schedule and safety.[3]
Allegiance to hierarchy and procedure had replaced deference to NASA engineer’s technical expertise.[4]
The organizational structure and hierarchy blocked effective communication of technical problems. Signals were overlooked, people were
silenced, and useful information and dissenting views on technical issues did not surface at higher levels.[5]
Engineers at Thiokol who still objected to the decision later testified that they were intimidated by management authority, were accustomed to
turning their analysis over to managers and letting them decide, and did not have the quantitative data that would empower them to object further.[6]
In the more decentralized decision process prior to Columbia’s re-entry, structure and hierarchy again were responsible for an
absence of signals.[7]
As what the Board calls an “informal chain of command” began to shape STS-107’s outcome, location in the structure empowered some to
speak and silenced others.[8]
Strategies must increase the clarity, strength, and presence of signals that challenge assumptions about risk. NASA’s
challenge is to design systems that maximize the clarity of signals, amplify weak signals so they can be tracked, and account for missing signals.[9]
1
2
3
4
5
6
7
8
9
[Notes] Columbia Accident Investigation Board, Report Volume 1, pages 169, 200-201, 203
General Motors Switch Failure
GM Internal Assessment: 1. Some were dismissed due to misconduct or incompetence, while others simply did not do enough to fix the problem. 2. It represents a fundamental failure to meet the basic needs of these customers. 3. We simply didn't do our jobs. We failed these customers. 4. GM's personnel's inability to address the ignition switch problem, which persisted for more than 11 years, represents a history of failures. 5. While everybody who was engaged on the ignition switch issue had the responsibility to fix it, nobody took responsibility. 6. Throughout the entire 11-year history, there was no demonstrated sense of urgency, right to the very end. 7. A pattern of management deficiencies and misjudgments -- often based on incomplete data -- that were passed off at the time as business as usual. 8. But the lack of information at the highest executive levels, the report concluded, was at the heart of G.M.’s failings on this issue.
http://money.cnn.com/2014/06/05/news/companies/gm-recall-probe/index.html?iid=TL_Popular
http://www.nytimes.com/2014/06/06/business/gm-response-to-a-fatal-flaw-was-to-shrug.html?hp&_r=0
Wetware (highest risks)
Software (high risk)
Hardware (lowest risk)
Mature; rich set of standards;
explicit; significant laws of
physics constraints; more
limited interfaces and
behaviors; & extensive full-
lifecycle tools.
Human &
Organizational
Components
Intelligent,
Autonomous
Systems
Avionics, Operating Systems,
Embedded Systems, Apps, Tools,
etc. Physical,
Mechanical,
Components
& Systems
Evolving; growing set of
standards; highly dynamic;
limited laws of physics
constraints; highly susceptible
to unintended interfaces and
behaviors; & growing &
evolving full-lifecycle tools.
Significant gaps; very limited set
of standards & certifications;
highly dynamic; moderate to
limited laws of physics
constraints; very susceptible to
unintended interfaces and
behaviors; & a dearth of full-
lifecycle tools.
Complex Systems Matrix
Key First Principles for Human/Organizational Complex Systems
Risks
KEY PREMISE ON
HUMANITY:
MOST OF US HAVE THE
SAME STRATEGIC
GOALS
HUMAN/ORGANIZATION
GOALS
1. PEACE
2. DIGNITY
3. PROSPERITY
4. GROWTH
5. SUSTAINABILITY
HUMAN/ORGANIZATION
GOALS - Pedigree
1. PEACE
2. DIGNITY
3. PROSPERITY
4. GROWTH
5. SUSTAINABILITY
1. Life
2. Liberty
3. Pursuit of
Happiness
SYSTEMS OWNERS/DESIGNERS
HAVE SIMILAR GOALS
1. PEACE
2. DIGNITY
3. PROSPERITY
4. GROWTH
5. SUSTAINABILITY
HUMANS/ORGANIZATION
S
1. FUNCTIONALITY
2. INTEGRITY
3. STABILITY
4. MARKET SHARE
5. SUSTAINABILITY
SYSTEMS and/or SYSTEMS
OF SYSTEMS
1. PEACE
2. DIGNITY
3. PROSPERITY
4. GROWTH
5. SUSTAINABILITY
HUMANS/ORGANIZATION
S
1. FUNCTIONALITY
2. INTEGRITY
3. STABILITY
4. MARKET SHARE
5. SUSTAINABILITY
=
SYSTEMS OWNERS/DESIGNERS
HAVE SIMILAR GOALS
SYSTEMS and/or SYSTEMS
OF SYSTEMS
1. PEACE
2. DIGNITY
3. PROSPERITY
4. GROWTH
5. SUSTAINABILITY
HUMANS/ORGANIZATION
S
1. FUNCTIONALITY
2. INTEGRITY
3. STABILITY
4. MARKET SHARE
5. SUSTAINABILITY
SYSTEMS OWNERS/DESIGNERS
HAVE SIMILAR GOALS
SYSTEMS and/or SYSTEMS
OF SYSTEMS
HOWEVER INDIVIDUAL HUMANS,
ORGANIZATIONS, & SYSTEMS
DON’T HAVE COMMON ANCHORS:
1. STARTING POINTS
2. KNOWLEDGE/EXPERIENCE
3. BELIEFS
4. FILTERS
5. COMFORT LEVELS
6. RESOURCES
7. TIMEFRAMES
8. LOCAL ENVIRONMENTS
9. CONSTRAINTS
10. DRIVERS
THESE DIFFERENCES LEAD TO:
1. STARTING POINTS
2. KNOWLEDGE/EXPERIENCE
3. BELIEFS
4. FILTERS
5. COMFORT LEVELS
6. RESOURCES
7. TIMEFRAMES
8. LOCAL ENVIRONMENTS
9. CONSTRAINTS
10. DRIVERS
1. PERSPECTIVES
2. PRIORITIES
3. APPROACHES
4. RISK MITIGATION
STRATEGIES
5. IMPLICIT VS. EXPLICIT
POSITIONS/BEHAVIORS
6. EXTRAPOLATIONS INTO
THE FUTURE
7. EXPECTATIONS
ANCHORS POINTS INCONGRUENT
INCENTIVES/BEHAVIORS
THESE BEHAVIORS DRIVE:
1. STARTING POINTS
2. KNOWLEDGE/EXPERIENCE
3. BELIEFS
4. FILTERS
5. COMFORT LEVELS
6. RESOURCES
7. TIMEFRAMES
8. LOCAL ENVIRONMENTS
9. CONSTRAINTS
10. DRIVERS
1. PERSPECTIVES
2. PRIORITIES
3. APPROACHES
4. RISK MITIGATION
STRATEGIES
5. IMPLICIT VS. EXPLICIT
POSITIONS/BEHAVIORS
6. EXTRAPOLATIONS INTO THE
FUTURE
7. EXPECTATIONS
ROOT CAUSE OF
MOST GAPS,
SOURCES OF
FRICTION &
CONFLICTS
ANCHORS POINTS INCONGRUENT
INCENTIVES/BEHAVIORS
I believe a critical priority is to illuminate this area. To find sufficient situational
awareness for the entire systems of systems lifecycle
1. STARTING POINTS
2. KNOWLEDGE/EXPERIEN
CE
3. BELIEFS
4. FILTERS
5. COMFORT LEVELS
6. RESOURCES
7. TIMEFRAMES
8. LOCAL ENVIRONMENTS
9. CONSTRAINTS
10. DRIVERS
1. PERSPECTIVES
2. PRIORITIES
3. APPROACHES
4. RISK MITIGATION
STRATEGIES
5. IMPLICIT VS. EXPLICIT
POSITIONS/BEHAVIOR
S
6. EXTRAPOLATIONS
INTO THE FUTURE
7. EXPECTATIONS
ROOT CAUSE OF
MOST GAPS AND
SOURCES OF
FRICTION &
CONFLICTS
DIFFERENT ANCHOR
POINTS
1. PEACE
2. DIGNITY
3. PROSPERITY
4. GROWTH
5. SUSTAINABILITY
1. FUNCTIONALITY
2. INTEGRITY
3. STABILITY
4. MARKET SHARE
5. SUSTAINABILITY
COMMON
STRATEGIC GOALS INCONGRUENT
INCENTIVES/BEHAVIO
RS
Thank You
Questions?
top related