Upload File

usage of decision analysis methods outside of a classroom environment by aerospace researchers...

  • Home
  • Documents
  • Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C
37
Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C. Ariel Pinto and Resit Unal Old Dominion University James T. Luxhøj Rutgers University Hampton Roads Area INCOSE Conference on Decision Analysis and Its Applications to Systems Engineering, Newport News, VA (November 17-18, 2009) National Aeronautics and Space Administration www.nasa.gov

Upload: madison-cook

Post on 27-Mar-2015

214 views

Category:

Documents


0 download

Report

Tags:

TRANSCRIPT

Page 1: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace ResearchersSharon Monica Jones NASA Langley

Rafael E. Landaeta, C. Ariel Pinto and Resit Unal Old Dominion University

James T. Luxhøj Rutgers University

Hampton Roads Area INCOSE Conference on Decision Analysis and Its Applications to Systems Engineering, Newport News, VA (November 17-18, 2009)

National Aeronautics and Space Administration

www.nasa.gov

Page 2: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration Jones,, et al. (2009) 2

Outline• Background/Definitions

• Data Collection Process

• Results

• Concluding Remarks

Page 3: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Problem Definition• Aerospace technology managers need:

To predict future technologies

To assess progress toward R&D goals

• Aerospace technology portfolio decisions are difficult because:

Very little time to acquire background data

Uncertainty factors (e.g., politics, global economy, environment, funding)

• Prescriptive decision analysis methods

Have been used for financial portfolio assessment

Value for policy related decisions has been questioned

Jones, et al (2009) 3

Page 4: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Ralph Keeney’s Suggestions for Making Better Decision Makers*• Develop concepts, tools and procedures to help decision makers

“My experience is that many people, including well-educated people, have a very difficult time in structuring their decisions. They can get mixed up about the difference between fundamental concepts such as alternatives and objectives.”

• Use real decisions, not just laboratory problems in decision research

“We have learned a great deal from all the laboratory settings where decision experiments have been conducted. There have also been some research studies of real decision problems. I feel there is much more to be gained by having more of this type of research.”

• Teach people what they can and will learn and use

“…hundreds and thousands of people have had at least a course that included a substantial part on decision analysis and very few have probably ever conducted a formal decision analysis. Once we find out what people can and will learn and use, that should constitute the basis for much of our teaching of decision analysis.”

*Source: Ralph L. Keeney, “Making Better Decision Makers”, Decision Analysis, 1:4 (2004)

Jones,, et al. (2009) 4

Page 5: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Decision Analysis Usage in Aerospace Portfolio Development• Aerospace managers have investigated the use of decision analysis methods for portfolio

investment decisions:

Commercial Aviation Safety Team (CAST)

NASA Aviation Safety Program

Future Aviation Safety Team (FAST)

• These technology assessments involved:

Resource commitments (e.g., employee time, travel money, software acquisition, training)

Assumption that decision analysis methods would improve aerospace technology

assessment process

Jones,, et al. (2009) 5

Page 6: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Technology Assessment

“ A process for measuring the impact of established or new technologies”

*Hans Mohr, “Technology Assessment in Theory and Practice”, Society for Philosophy and Technology, 4:4 (Summer 1999)

Jones, et al (2009) 6

Page 7: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Aerospace Technology AssessmentThree different processes for examining impact of a set of technologies

Technology assessment

Technology forecasting

Technology foresight

Aerospace Technology Assessment

Jones, et al (2009) 7

Page 8: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Conceptual Model *

1. Identify Context & Understand Objectives

2. Identify Alternatives

Aerospace Technology Assessment

Decision Analysis Methods

3. Decompose & Model the Problema. Model of Problem Structureb. Model of Uncertaintyc. Model of Preferences

4. Choose the Best Alternative

5. Sensitivity Analysis

6. Implement the Chosen Alternative

*Adapted from Robert T. Clemen, Making Hard Decisions, 2nd Edition (1995)Jones, et al (2009) 8

Page 9: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Purpose of StudyWhat is known:

Decision analysis methods in financial portfolio assessment

Decision experiments in laboratory settings

Technology assessment in medical R&D

What is unknown:Decision analysis methods for policy related decisions

Decision analysis methods in real decision problems

Technology assessment in aerospace

Jones, et al (2009) 9

Page 10: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Study Overview• Population was aerospace researchers with experience in one or more of the following:

Aerospace program/project management

Aerospace technology assessment

Aerospace technology selection

Aerospace R&D portfolio development

• Methods that were investigated in the study

Decision trees

Influence diagrams

Criteria aggregation methods

Explicit tradeoff approaches

• Participants were questioned about their usage of these methods for aerospace technology

assessment

Jones, et al (2009) 10

Page 11: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Decision Trees

11

Buy shares of XYZ stock

Don’t buy XYZ stock

Value of XYZ stock increases

Value of XYZ stock unchanged

Value of XYZ stock decreases

Earn money

Total amount of money unchanged

Lose money

Total amount of money unchanged

= decision node

= chance node

= consequence node

Jones, et al (2009)

Page 12: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Influence Diagrams

12

Buy XYZ stock? Change in amount of money

Price of XYZ stock at

future point

decision node

chance node

consequence node

Jones, et al (2009)

Page 13: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Criteria Aggregation Methods• Methods in which two sets of aggregated indices are developed and used to evaluate the

alternatives in the decision problem

• Methods in the category include:

Analytical Hierarchy Process (AHP)

Weighted Sum Model (WSM)

13Jones, et al (2009)

*E. Triantaphyllou, Multi-Criteria Decision Making Methods: A Comparative Study(1995)

Example of Simple Weighted Sum Model *

n

A*WSM-score =

max

aij,wj, for 1=1,2,3,….n i

j = 1 where, A*WSM-score = the WSM of the best alternative n = the total number of criteria

aij = the score of the i-th alternative in terms of the j-th criterion wj = the weight of importance of the j-th criterion

Page 14: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Explicit Tradeoff Approaches• Decision analysis methods that are based on value functions

• Methods in this category include:

Multi-Attribute Utility Theory (MAUT)

Simplified Multi-Attribute Rating Approach (SMART)

14Jones, et al (2009)

Page 15: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Excluded Decision Analysis Methods • Avoided decision analysis methods that are not popular in U.S.

• Real world applications are complex with large amounts of uncertainty

• Specific decision analysis methods that were excluded from study:

Outranking methods (e.g., ELECTRE, TOPSIS)

Optimization methods

Analytic network process (ANP)

Jones, et al (2009) 15

Page 16: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Data Collection and Analysis ProcessRefine List of Candidate Survey Participants

Develop Web-Based Survey Instrument

Develop List of Pilot Participants

Conduct Pilot Survey

Review & Analyze Pilot Results

Modify Survey Instrument

AnalyzeResults

Conduct Survey

Legend Survey Development Data Collection Data Analysis

Jones, et al (2009) 16

Page 17: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Web-Based Instrument Development• Several web-based services examined

• Questions developed based on several sources:

Short surveys at professional meetings

Validated research in decision analysis literature

• Identities of survey participants remained anonymous

Jones, et al. (2009) 17

Page 18: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Pilot Survey• Conducted with subset of population (10 persons)

• Think aloud cognitive interviewing techniques used

Solicitation of all thoughts and comments

Manual recording of information during completion of online survey

De-identification of subjects in final documentation

18Jones, et al. (2009)

Page 19: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Survey Instrument Modification• Questions were modified, added or eliminated from the survey based on input from:

Pilot survey comments

Data analysis of pilot survey data

Additional comments from other reviews (e.g., ODU IRB)

• Number of survey questions reduced from 70 to 65

19Jones, et al. (2009)

Page 20: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Data Collection Overview• Approval to conduct survey was obtained from NASA Langley and ODU Institutional Review

Boards (IRB’s)

• E-mail invitation was sent to 260 persons

• 154 total visits to survey website

16 partial responses

99 completes surveys

• Out of the 99 completed surveys

76% male, 24% female

Highest degree level was 60% Masters, 21% Bachelors, 18% Doctorate, 1% Associates

72% employed as government civil servants

Jones, et al. (2009) 20

Page 21: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Job Functions

Jones, et al. (2009) 21

Page 22: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Aerospace Experience

Jones, et al. (2009) 22

Page 23: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Decision Trees

23Jones, et al. (2009)

Page 24: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Influence Diagrams

24Jones, et al. (2009)

Page 25: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Criteria Aggregation Methods

25Jones, et al. (2009)

Page 26: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Explicit Tradeoff Approaches

26Jones, et al. (2009)

Page 27: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration Jones, et al. (2009) 27

Usage of Decision Analysis Methods Outside of a Classroom Environment

Page 28: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration Jones, et al (2009) 28

Categories of Non-ATA Usage of DA Outside of a Classroom Environment

Page 29: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Additional Questions

29Jones, et al. (2009)

Page 30: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Additional Questions (cont’d)

30Jones, et al. (2009)

Page 31: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration 31Jones, et al. (2009)

Additional Questions (cont’d)

Page 32: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Additional Questions (cont’d)

32Jones, et al. (2009)

Page 33: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Additional Questions (cont’d)

33Jones, et al. (2009)

Page 34: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Likelihood of Future Usage of DA Methods

34Jones, et al. (2009)

Page 35: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Concluding Remarks• This is a subset of the total data

There are many additional questions in the study

More formal analysis of the data was conducted using structural equation modeling techniques to test a set of hypotheses

• Survey participants believed that the successful use of decision analysis methods depends on:

Selection criteria in the decision model

Experience of the person that implements the method

Reliability of the input data

• Training/education does not guarantee future use of a decision analysis method

35Jones, et al. (2009)

Page 36: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration

Questions?

36

Page 37: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C

National Aeronautics and Space Administration Your Title Here 37


Book of Kings Langley · Kings Langley Village of the Year Entry 2012 – Brochure Page 5 of 32 Kings Langley - Our Heritage Kings Langley, our village in Hertfordshire, lays claim

James M. Langley, President Amy Gregg, Senior Associate Langley Innovations

Langley Literacy Network - School District 35 Langley...Langley Literacy Network District Literacy Plan School District #35 (Langley) Board of Education Meeting June 21, 2016. Our

Cvpat Langley

Langley Real Estate

LANGLEY BOLLARD LBD100 - Langley Design

GREE BUILDING PRESENTED BY: Hector Landaeta Abdullah Namankani Marcos Soto

LANGLEY CITY SQUARE 19860-19898 LANGLEY BYPASS …...LANGLEY CITY SQUARE LANGLEY, BC FORM RETAIL ADVISORS INC. RENOVATION PLANS • Langley City Square will have appearance of a new

Langley Newsletter

Langley Saves

COMMUNITY PROFILE TOWNSHIP OF LANGLEY - …newtobc.ca/wp-content/uploads/2014/09/Township-of-Langley... · Township of Langley Immigrant Demographics I Page 2 TOWNSHIP OF LANGLEY

Sonic boom probe flies - NASA · Anthony Washburn (Langley Research Center in Virginia), Richard Wahls (Langley), Fayette Collier (Langley), Gaudy Bezos-O’Connor (Langley), Cathy

Exposición de la Dra María Eugenia Landaeta en el Foro Epidemiología de la Fiebre del Chikungunya en Venezuela

Pat Langley Center for the Study of Language and Information Stanford University, Stanford, California langley [email protected]

Landaeta Mejías Brothers et al v. Vene- zuela · 2017 Landaeta Mejías Brothers et al. v. Venezuela 1599 ligence gathering operation while driving in a private car in plain-clothes.19

Next Week March 2019.pdf · Langley 1-0 Kent College Langley 5-0 Borden Grammar Langley 3-0 Duke of York Langley 0-0 Simon Langton RGS, Guildford 1st Place Quarter Final Langley 1-0

LANGLEY - NASA

Rafael Riqueni - Analisis de Estilo - Rafael Riqueni

Used Car Dealer in Langley - Langley Hyundai (604) 539-8549

Rafael C Jimenez [email protected]

LANGLEY HOUSE TRUST CARE SERVICES · 4 Langley House Trust - Care Services Langley House Trust - Care Services 5 Langley House Trust is a national offender rehabilitation charity

LANGLEY WAKEMAN

Rafael Riqueni - La Guitarra Flamenca de Rafael Riqueni

Langley United SA

EMILWebinar Langley

INTEGRANTES DEL SERVICIO COMUNITARIO: MARITZA LINAREZ MERCEDES RIOS MILANYI LANDAETA

Federico Landaeta - Masoneria Dinamica (1971)

SAMUEL PIERPONT LANGLEY - National Academy of Sciencesnasonline.org/publications/biographical-memoirs/memoir... · 2017-09-22 · SAMUEL PIERPONT LANGLEY." Samuel Pierpont Langley,

Langley Visitor Guide

Challenge Langley

City of Langley - Langley, Washington · City of Langley PLANNING ADVISORY BOARD AGENDA April 2, 2014 – 3-5pm LANGLEY CITY HALL - 112 2nd Street, Langley WA 1) Call to Order and

LANGLEY CREW

Langley Connect, Langley, Slough SL3 6EG · Langley Connect, Langley, Slough SL3 6EG RATES Rateable Value Langley Connect (2017) - £28,492 Uniform Business Rate Multiplier (2019-2020)

Langley Search Overview

Langley primary