formulation of an experiment selection methodology … · 2014. 12. 11. · boeing 757 vt, where a...

FORMULATION OF AN EXPERIMENT SELECTIONMETHODOLOGY FOR UNCERTAINTY REDUCTION AND

APPLICATION TO ACTIVE FLOW CONTROL

Ryan B. Jacobs and Dimitri N. MavrisAerospace Systems Design Laboratory

School of Aerospace EngineeringGeorgia Institute of Technology

Atlanta, GA 30332

Keywords: physical experiment, technology development, uncertainty quantification, active flowcontrol

Abstract

Many technologies have the potential to alleviateconcerns related to the environmental impact ofcommercial aviation. However, there is consid-erable uncertainty surrounding the impacts thatintegration of these technologies will have on fu-ture aircraft. In this paper, the foundation isestablished for a physical experiment selectionmethodology that aims to maximize the reduc-tion of uncertainty and the maturation of tech-nologies. Throughout the methodology formu-lation, a case study is described for an active flowcontrol technology currently under development.

1 Introduction

Commercial aviation forecasts continue to pre-dict increased passenger traffic and fleet ex-pansion, and concerns about the environmentalimpact of the commercial aviation sector havegrown accordingly. Many public and private or-ganizations have set aggressive goals to addressthese concerns. For instance, the National Aero-nautics and Space Administration (NASA) Envi-ronmentally Responsible Aviation (ERA) projecthas been established to simultaneously meetgoals, shown in Table 1, for noise, emissions, andfuel burn. Reconfiguration and optimization oftraditional vehicle architectures is likely not suf-ficient for achieving these goals; thus, the matu-

Table 1 NASA ERA system-level metrics andgoals (adapted from Ref. [1])

Technology benefits1 N+2 (2020)Noise (cum. below stage 4) -42 dBLTO NOx (below CAEP6) -75%Cruise NOx2 -70%Aircraft fuel burn2 -50%1Referenced to a Boeing 777-200 with GE90 engines2Relative to 2005 best in class

ration of advanced vehicle concepts and enablingtechnologies is being pursued.

One of the technologies that has been identi-fied with the potential to alleviate environmentalconcerns, in addition to other integration issues,is active flow control (AFC). Although scientificAFC research has been ongoing since Prandtl’ssuction flow control experiments over 100 yearsago [2], few production vehicles currently oper-ate with these technologies. Most of the vehiclesare military aircraft that employ boundary layercontrol for wing lift augmentation. For example,the Mikoyan-Gurevich MiG-21 has an internally-driven boundary layer control system, and theBoeing C-17 Globemaster III uses externally-blown flaps. Traditional boundary layer controldevices utilize steady suction or blowing to con-trol the time-averaged flow state, whereas mod-ern AFC devices control flow instabilities. One

1

RYAN B. JACOBS AND DIMITRI N. MAVRIS

of the primary benefits of the modern AFC de-vices, from an integration viewpoint, is that theyare typically more energy efficient than boundarylayer control devices.

Unlike boundary layer control devices, mod-ern AFC devices are relatively immature; theshift from laboratory settings to real-world aero-nautical applications began around the year 2000[3]. Due to the immaturity of modern AFCdevices, system-level integration effects are notwell-understood. If uncertainty surroundingthese effects is not reduced during modern AFCtechnology development programs, then vehicleintegrators will face additional risk when thetechnologies are adopted.

Uncertainty surrounding technology integra-tion effects can be reduced by gaining knowl-edge from the "right" physical experiments. Forany advanced technology development program,justification of investments is usually based onthe premise that uncertainty in performance, cost,and schedule will be reduced [4]. These invest-ments should not be squandered by selecting ex-periments that result in minimal uncertainty re-duction. Technology readiness levels (TRLs) areused as a figure of merit for assessing and com-municating the maturity of novel technologies,and they provide vague guidelines for the exper-imental demonstrations that must occur to "grad-uate" each maturity level. However, TRLs donot explicitly capture the uncertainty reductionsthat complement maturation. A physical experi-ment that satisfies the qualitative requirements ofa given TRL does not necessarily guarantee thatsignificant uncertainty reduction will be attained.

Choosing the "right" physical experiments ischallenging. Many combinations of options suchas experimental facility, variable settings, and ge-ometry can be proposed. Decision makers (DMs)need a way to evaluate each of the proposedexperiments based on uncertainty reduction, re-source requirements, maturation, and other crite-ria. Adding to the difficulty, some of the criteriamay be conflicting. As an example, the most in-expensive experiment to perform will likely notprovide the most uncertainty reduction.

In an ideal technology development program,physical experiments will be optimally selected

to reduce uncertainty and increase maturity sothat the full potential of the technology can beexploited. The research presented in this paperprovides an important contribution that is neces-sary to enable this vision: formulation of an ex-periment selection methodology for reducing un-certainty surrounding the integration impacts ofimmature technologies.

The next section of this paper provides back-ground information about the case study and therelationship between uncertainty and knowledge.Then, key elements of the proposed methodologyare presented along with application to the casestudy. Finally, the last section ends with a sum-mary and the future direction for research. Notethat, throughout the rest of this paper, modernAFC devices will be referred to with the initial-ism AFC.

2 Background

Continuing with the AFC theme set by the moti-vation for this research, section 2.1 introduces anAFC technology that is currently under develop-ment by NASA and its partners. This technologyis used as a case study for the proposed method-ology. Then, section 2.2 discusses uncertainty,knowledge, and understanding in the technologydevelopment context.

2.1 Technology for the case study: the AFC-enhanced vertical tail

A commercial transport drag reduction approachthat is currently being investigated in the NASAERA project is to use AFC actuators installed justupstream of the rudder hinge line to enable verti-cal tail (VT) area reduction. This can be accom-plished by controlling rudder flow separation toenhance the side force generated by the VT dur-ing critical flight situations, such as asymmetricpower at takeoff. The AFC device that is be-ing used for actuation is called a sweeping jet.This is a type of fluidic oscillator that producesa pulsed jet when supplied with pressurized fluid[5]. A typical sweeping jet actuator configura-tion is depicted in Fig. 1. The jet that enters themain cavity of the sweeping jet attaches to one

2

FORMULATION OF AN EXPERIMENT SELECTION METHODOLOGY FOR UNCERTAINTYREDUCTION AND APPLICATION TO ACTIVE FLOW CONTROL

a Feedback

Supply

Interactionregion

Outlet

Feedback path

Fig. 1 Sweeping jet actuator that is installed justupstream of the rudder hinge line (from Ref. [5]).

side of the cavity wall, due to the Coanda effect[6]. Then, the pressure in the adjacent feedbackloop increases. The higher pressure forces the jetto the opposite cavity wall, and the same processrepeats cyclically. For separation control appli-cations, the sweeping motion of the jet is impor-tant for effectiveness. This is because, in addi-tion to the jets injecting momentum to the flow-field, the sweeping motion of adjacent jets pro-motes the formation of streamwise vortices thatremove low-momentum fluid from the boundarylayer and supply it with high-momentum fluidfrom the outer region [7]. An additional benefit isthat the sweeping motion enables larger spacingbetween actuators than non-oscillatory jets withthe same effectiveness.

The sweeping jet actuator was developedmore than 50 years ago at the Harry DiamondLaboratories and was originally used in analogcomputers and fluidic amplifiers [8]. More re-cent investigations have been conducted to ex-plore the use of sweeping jets for aeronauticalapplications (e.g., [8, 9, 10, 11]). The actua-tors have been tested at full scale on an isolatedBoeing 757 VT, where a 20–30% side-force en-hancement was demonstrated at conditions sim-ilar to takeoff and landing [12]. Results fromthe full-scale experiments are being used to de-termine optimal actuator placement and suppliedmass flow rate for flight experiments on Boeing’secoDemonstrator 757 aircraft.

Although VT area reduction would reducethe weight and drag of an aircraft, installationof an AFC system in the VT would degrade per-formance as well. A power distribution architec-

Figure 13: Sweep Jet Power Distribution

Fig. 2 AFC power distribution architecture de-signed by Boeing (from Ref. [13]).

ture is required to deliver pressurized air from asource, such as an auxiliary power unit (APU),to the sweeping jet actuators. An example of anAFC architecture is shown in Fig. 2. As seenin the figure, flow supply lines would be routedfrom the APU bleed point to the sweeping jet ac-tuators. Since the hot bleed flow from an APUmay transfer too much heat to the aircraft struc-ture, a pre-cooler would be required. Additionalcomponents such as these would add weight tothe vehicle. There would also be a fuel burnpenalty due to operating the APU for takeoff andlanding segments of the mission. Other concernsinclude increased costs, increased complexity, re-liability, and noise. However, the current appli-cation scenario being considered for this tech-nology is to install it only on some membersof a commercial transport family. The VT ofa commercial transport family is typically thesame area for all family members and sized forthe shortest member; thus, members with longerfuselages carry VTs with non-optimal areas. Theuse of AFC actuation will enable the VT to be

3


sized for the longest family member.

2.2 Uncertainty, knowledge, and under-standing

As a technology matures, its success is deter-mined by DMs who must balance and mitigatethe uncertain costs and benefits of introducing thetechnology to a vehicle development program.With this context in mind, the authors followNikolaidis [14] in defining uncertainty indirectlyfrom the definition of certainty. Nikolaidis de-fines certainty as the condition of possessing allknowledge that is needed to choose the actionwith the most desirable consequences. Uncer-tainty can then be defined as the gap betweencertainty and a DM’s present state of knowledge.This concept is illustrated in Fig. 3a.

Uncertainties are often categorized using ataxonomy that has been developed by the risk as-sessment community [15]:

• Aleatory uncertainty: uncertainty due toinherent randomness• Epistemic uncertainty: uncertainty due to

lack of knowledge

An example of a source of aleatory uncertaintyis Young’s modulus of a material. AlthoughYoung’s modulus is reported as a constant, thereis variability between material samples due to themanufacturing process. The inherent random-ness in Young’s modulus can be reduced by im-proving the manufacturing process, but sourcesof aleatory uncertainty are often treated as irre-ducible. The term "epistemic" comes from theGreek "episteme", meaning knowledge; hence,epistemic uncertainty can be reduced by acquir-ing additional knowledge. An example of anepistemic source of uncertainty is a calibrationparameter in a deterministic computer model ofa physical system. The value of the calibra-tion parameter, such that the model predictionswill match reality, is uncertain. After obtain-ing data from physical experiments, discrepan-cies between the model predictions and the sys-tem behavior can be minimized using a calibra-tion process. Aleatory (irreducible) and epis-temic (reducible) uncertainties are the compo-

Maximum uncertainty

Uncertainty

Completeignorance Present state of

knowledge

Certainty

Reducibleuncertainty

Irreducibleuncertainty

Present state ofknowledge

Perfect stateof knowledge

Certainty

(a)

(b)

Fig. 3 Depiction of the definition of uncertainty(adapted from Ref. [14]).

nents of a DM’s total uncertainty, as shown inFig. 3b.

Mathematical representation, or characteri-zation, of uncertainties depends on the catego-rization. The most common mathematical formused for aleatory uncertainty is a probability dis-tribution function (PDF). There is disagreementamong researchers about what mathematicalform epistemic uncertainties should take. Some,such as Oberkampf and Roy [15], argue thatepistemic uncertainties should not be endowedwith any probabilistic structure and should in-stead be represented as intervals. O’Hagan andOakley [16], among others, argue that probabil-ity is adequate for describing any type of un-certainty. Rational arguments have been of-fered in support of both approaches, and bothrepresentations are frequently used in the litera-ture. This suggests that there is not necessarilya correct choice for all problems. There is alsoa mixed aleatory/epistemic type of mathemati-cal representation. As an example, assume thatthe aleatory uncertainty source mentioned pre-viously, Young’s modulus, is characterized as anormal distribution with mean µ and standard de-viation σ. One may not know the precise valuesof µ and σ; thus, they are epistemic uncertainties.

The definition of epistemic uncertainty maylead one to ask, "What types of knowledge should

4


be acquired during a technology developmentprogram?" Epistemology, the study of knowl-edge, provides philosophical views that can helpanswer this question. Although there are vari-ous kinds of knowledge, such as knowing how todo something or knowing someone by acquain-tance, epistemologists typically focus on proposi-tional knowledge, also referred to as knowledge-that [17]. Propositional knowledge requires thata subject knows a proposition. For example,one may express propositional knowledge of anAFC impact on an aircraft as, "I know that thecruise drag reduction achieved by implementingthe active flow control technology is two per-cent." The particular knowledge that is of inter-est in this research is scientific knowledge, whichis propositional knowledge generated by the sci-entific method. Scientific experimentation alsogenerates information that promotes understand-ing. Many epistemologists consider understand-ing to be a type of knowledge, namely, knowl-edge of causes. Whether understanding is propo-sitional in nature is debatable. Nevertheless,propositional knowledge (knowledge-that) aloneis not sufficient for technology development; un-derstanding is required not only for uncertaintyreduction but also to improve performance.

3 Proposed methodology

Selection of physical experiments for a technol-ogy development program requires informed de-cision making. A formal decision process for in-telligently selecting from a set of alternatives isrequired. The top-down design decision supportprocess, developed by Mavris et al. [18], providesa foundation for this purpose:

1. Establish the need2. Define the problem3. Establish value objectives4. Generate feasible alternatives5. Evaluate alternatives6. Make decision

In order to evaluate alternatives based on multiplecriteria, modeling and simulation (M&S) is em-ployed to quantify metric values. In this research,

metrics are figures of merit that characterize theimpacts of technology integration and attributesof the physical experiments. A way to quantifyuncertainty in the metrics is also needed. A pro-cess extracted from steps commonly found in theM&S-based uncertainty quantification (UQ) lit-erature is used as a basis:

1. Identify sources of uncertainty2. Characterize uncertainty3. Propagate uncertainty4. Analyze impacts5. Reduce uncertainty

The proposed methodology for selection of phys-ical experiments, shown in Fig. 4, is a fusion ofthe top-down design decision support process andthe generic UQ process. Each step is described inthe following sections.

3.1 Definition of the need and the technology(step 1)

It is assumed that the need for the technology hasbeen established prior to the start of the devel-opment program. But, the need must be well-defined so that the objectives of the developmenteffort can be aligned with the system-level goals.System-level analysis should be used to estab-lish the need by demonstrating that the vehicleconcept is not technically feasible using conven-tional technology. Metrics that describe the per-formance gap, in addition to the performancegoal values, need to be tracked during technol-ogy development.

Defining the technology involves specifyingthe vehicle architecture and the integration ap-proach. This information is essential for latersteps. A diagram of the vehicle with the inte-grated technology and a description of the physi-cal principles that characterize the technology fa-cilitate communication of this information. Theengineering design literature also led the authorsto a system decomposition approach. There aretwo types of decomposition commonly utilizedin the design of complex systems: physical andfunctional [19]. A physical decomposition is aschematic diagram that illustrates the system sub-assemblies and components and how the parts

5


Fig. 4 Proposed methodology for selection of physical experiments.

connect. A functional decomposition results in adiagram called a function structure. This diagramuses function blocks to represent transformationsdone by the system components, with flows of in-formation, energy, material, etc., indicated by ar-rows. Since a physical decomposition is more ap-propriate for understanding the nature of technol-ogy integration, it is incorporated in this method-ology. As seen in the following example for thisstep, the physical decomposition can also be usedto clearly show metrics at all levels of the systemhierarchy.

3.1.1 Step 1 case study

The need for the AFC-enhanced VT technologyhas been defined by NASA. In the first phaseof the ERA project, a large technology portfoliowas explored, with system-level analysis, for en-abling advanced vehicles to meet the goals in Ta-ble 1. As part of the second phase of the project,a subset of technologies was selected for devel-opment. To contribute to the 50% fuel burn goal,one of the technologies selected was the AFC-enhanced VT. In addition to fuel burn, NASA

has identified total vehicle drag during cruise as akey performance metric for this technology. Onedrawback of this technology is that the actuatorswill produce noise during operation, so it is im-portant to track the degradation to the ERA noisegoal as well.

To guide subsequent steps, basic physicalprinciples that govern the operation of the AFC-enhanced VT are described in section 2.1 andBoeing’s diagram of a representative aircraft withthe integrated AFC system is shown in Fig. 2.A physical decomposition, shown in Fig. 5, hasbeen created based on this information. The lightgreen boxes contain aircraft components and thelight blue boxes contain metrics tracked at thelevel of the green box that they flow into. Di-rectional arrows have been used in this diagramto indicate the flow of information from the tech-nology level to the system level. At the top,relevant ERA goals are listed: noise and fuelburn. Four levels were defined for the systemhierarchy: ERA goals, component groups, com-ponents, and component breakdown. Below theERA goal level, the groupings are similar to an

6


Fig. 5 Physical decomposition for the AFC-enhanced VT technology.

aircraft weight breakdown. This type of break-down was selected for the physical decomposi-tion because it is exhaustive and can be used torepresent the system components at any desiredlevel of detail.

3.2 Identification and selection of value ob-jectives, criteria, and metrics (step 2)

The purpose of this step is to select which cri-teria will be used to evaluate the set of possi-ble physical experiments based on the objectivesand constraints established by DMs. Althoughobjectives will vary from one development pro-gram to another, there are a few that will likely becommon for all: uncertainty reduction, matura-tion, and performance improvement. Constraintsshould include budget, schedule, and the avail-ability of experimental facilities.

System-level performance improvement isquantified using metrics defined in step 1. As-suming that the M&S capability is available,system-level analysis can be employed to identify

important disciplinary impacts associated withinfusing the technology. A vetted approach forrapidly modeling technologies at the system levelis the use of "k-factors". These are dimension-less, multiplicative values that operate on disci-plinary input parameters in computer codes. Asan example for demonstrating the concept, con-sider the k-factor-modified Breguet range equa-tion, rearranged for calculating fuel burn, forflight at constant velocity V∞, thrust-specific fuelconsumption ct , and lift-to-drag ratio CL/CD:

Wf = kW0W0

1− exp

− kct ctRkCLCLkCDCD

V∞

(1)

where W0 is the gross weight of the aircraft withfull fuel tanks, Wf is the weight of fuel usedduring the mission of range R, and ki is a k-factor for variable i. To simulate the impact ofan aerodynamic technology such as AFC on fuelburn, the k-factors on the lift and drag coeffi-cients in Eq. (1) would be set to particular val-ues. A sensitivity analysis provides an indicationof how much each k-factor contributes to vari-ability in the system-level metrics. k-factors withthe largest impacts are the ones that should beidentified as important metrics since reducing un-certainty in these will maximize uncertainty re-duction in the system-level metrics. The chosenmetrics are also used to quantify performance im-provement at the technology level. A notional ex-ample of a graphical approach to sensitivity anal-ysis and k-factor selection is illustrated at the topof Fig. 6. The lines within each box are "slices"of the system-level metrics at a point in the k-space. k2 and k3 have been circled because theycontribute more to variability in the Ms than k1.

To quantify the knowledge gained from ex-perimental observations, an uncertainty reductionmeasure is required. A frequently used measurefor probabilistic assessments is variance reduc-tion. Entropy measures, borrowed from informa-tion theory, have also been applied. Based onan analysis of the uncertainty literature, Bjork-man [20] concluded that an entropy measure isa better option than variance. Bjorkman pro-vides multiple valid reasons for preferring en-tropy. But, since employing both does not in-

7


Sys

tem

-lev

el m

etri

cs

-factors

-fac

tors

Technology-level variables

Fig. 6 Notional graphical sensitivity analyses atthe system level (top) and the technology level(bottom). Important k-factors are selected for fur-ther investigation during the technology develop-ment program.

cur any substantial computational penalties, theauthors of this paper suggest doing so. If non-probabilistic methods are used to characterizeepistemic uncertainties, then alternative uncer-tainty reduction measures must be derived.

The most prevalent approach for representingthe maturity achieved by an experimental activ-ity is a TRL scale. TRL definitions have beenproposed by multiple organizations around theworld. Selection of a particular scale and thedetermination of whether a better alternative ex-ists is outside the scope of the work presented inthis paper. But, at least one measure of matu-rity should be chosen for use in this methodology.Selection of the best metrics for cost, schedule,and availability are also outside the scope of thiswork.

After DMs choose metrics that are crucial forevaluating the physical experiments, individualmetrics or functions of metrics become the cri-teria for the decision process. If so desired, DMscan place subjective weights on each of the crite-rion to represent relative importance. The criteriaand associated weights are used in later steps.


NASA’s objectives for development of the AFC-enhanced VT naturally include uncertainty re-duction, maturation, and the attainment of per-formance goals. Schedule and budget constraintshave also been established.

Mooney et al. [13] conducted a system-levelanalysis to quantify the manufacturing, opera-tional, and combined net present value for a mid-size aircraft with the AFC-enhanced VT technol-ogy. A method for modeling the performanceimpacts was used that is similar to the k-factorapproach. Results from sensitivity analyses indi-cate that the drag reduction achieved by imple-menting the technology is the primary contribu-tor to uncertainty in combined net present value.Other important performance impacts identifiedare detriments in thrust-specific fuel consumptionand weight. All three qualify as important met-rics and are carried forward, similar to k2 and k3in Fig. 6.

NASA selected all of the physical experi-ments for the second phase of ERA using a setof diverse criteria. For brevity, the process is notdiscussed in this paper.

3.3 Quantification of uncertainty for thepresent state of knowledge (step 3)

Before uncertainty reduction that is achievedby alternatives can be predicted, an uncertaintybenchmark must be determined. As stated previ-ously, M&S is required for UQ in this methodol-ogy. Construction or updating of the M&S envi-ronment is the starting point for this step beforeUQ can be carried out.

3.3.1 Construction or updating of modelingand simulation environment

Products from defining the technology in step 1and metrics identified in step 2 guide constructionof the M&S environment. The physical decom-position, explanation of the governing physics,and diagram of the integrated technology shouldbe used to ensure that all germane phenomena arecaptured. Assuming that the k-factor approachis utilized for system-level modeling, additional

8


M&S capability should provide quantification ofthe important k-factors. An example of this isshown in the notional sensitivity plot at the bot-tom of Fig. 6.

It is essential for the computational cost of theM&S environment to be affordable. This is be-cause uncertainty propagation techniques neces-sitate a large number of evaluations. Consider-ing the computational appetite of many physics-based computer codes, researchers often leveragethe power of design of experiments and surrogatemodels. For example, response surface method-ology [21] is frequently used to generate statisti-cal models, which can be evaluated virtually in-stantaneously, of computer codes in lieu of exe-cuting the code itself. If high-fidelity codes suchas computational fluid dynamics are not used tomodel the physics, then an alternative approachis to build statistical models from any existing ex-perimental data.

3.3.2 Identification and characterization of un-certainties

Identification of sources of uncertainty involvesdetermining where uncertainty in the metrics ofinterest stems from. Many of the importantsources are identified by enumerating variables atall levels of the system hierarchy that are uncer-tain. This exercise is facilitated by an understand-ing of how a given technology integrates with avehicle concept; thus, the products of step 1 areuseful for this task as well. The use of M&S forquantifying metrics introduces model form un-certainty, which is due to assumptions made inthe modeling of physics [15]. Another prevalentsource is measurement uncertainty.

Once the sources of uncertainty have beenidentified, they must be characterized. As dis-cussed in section 2.2, aleatory uncertainties aretypically represented with PDFs, and there aretwo primary options for epistemic uncertainties.DMs and analysts must decide how to representepistemic uncertainties. One advantage of us-ing the probabilistic approach is that it enablesBayesian inference, which is discussed in the laststep of the methodology. Assigning probabilitydistributions or intervals to each source is not a

trivial task, as one must translate one’s own be-liefs and/or the beliefs of others to mathematicallanguage. For aleatory sources, the distributionshape can be estimated based on existing exper-imental data. If this information does not exist,a distribution shape can be assumed and updatedif and when data are generated. Expert elicita-tion methods, such as O’Hagan’s SHELF [22],should be employed for specifying PDFs in thisstep. But, once a sensitivity analysis is carriedout in the propagation step, the analyst may de-termine that some sources of uncertainty are notimportant no matter what distribution is assumed.

3.3.3 Uncertainty propagation and sensitivityanalysis

It is known from probability theory that any func-tion of a random variable is also a random vari-able; thus, when uncertain inputs to an M&S en-vironment are characterized as random variables,the outputs from the environment are also ran-dom variables. As an example (adapted fromRef. [23]), consider a deterministic computa-tional model that maps an input from the realnumber line into an output from the real num-ber line: y = g(z). Suppose that an analyst hascharacterized z as an epistemic or aleatory sourceof uncertainty with a given PDF f (z). Now, z istreated as a random variable, which is denoted byZ. The uncertain output from the computationalmodel is denoted by Y , which is also a randomvariable. Obtaining the cumulative distributionfunction (CDF) and PDF of the model output Yis the objective of uncertainty propagation. TheCDF of Y is calculated as follows:

P(Y ≤ y) = P(g(Z)≤ y)= P(g(Z) ∈ (−∞,y])

= P(Z ∈ g−1(−∞,y])

=∫

g−1(−∞,y]f (z)dz (2)

Once the CDF has been computed, the PDFis found by differentiating the CDF. The struc-ture of most M&S environments requires thatthe integral in Eq. (2) be computed numeri-cally. If it is not possible to modify the com-putational codes within the environment, then a

9


non-intrusive propagation method must be se-lected. Non-intrusive propagation methods in-clude simulation-based, such as Monte Carlosimulation; local expansion-based, such as aTaylor series; most probable point-based, suchas the first-order reliability method; functionalexpansion-based, such as polynomial chaos ex-pansion; and numerical integration-based, suchas full factorial numerical integration [24]. Ifepistemic uncertainty is characterized with inter-vals, then propagation will result in a probabilitybox instead of one CDF for each output. Otherpropagation methods exist for this type of char-acterization, such as second order Monte Carlo.Many M&S environments are treated as black-box-type functions, and this limits the choice ofpropagation methods to non-intrusive. As previ-ously mentioned, surrogate modeling techniquescan be employed to enable computationally ex-pensive uncertainty propagation. The highestaccuracy and most expensive methods involveMonte Carlo simulation, which should be usedfor propagation, if computational resources al-low.

In addition to the propagation task, a sensi-tivity analysis is conducted during this step. Sen-sitivity analysis can be local or global. An ex-ample of a local sensitivity analysis is computingthe partial derivatives of a computational model’soutputs with respect to the input variables at agiven point in the input space. When inputs areuncertain, it is useful to understand which inputsources of uncertainty are the largest contributorsto uncertainty in the outputs, and this is what aglobal sensitivity analysis can reveal. There aremany available techniques for global sensitivityanalyses such as scatterplots and variance-basedmeasures [25]. A notional example of the re-sults from a variance-based technique is shownin Fig. 7. An important result of sensitivity anal-ysis is a ranking of epistemic and aleatory uncer-tainty sources based on their contribution to thesystem-level metric uncertainties.


The premier suite of aircraft system-level anal-ysis tools currently employed for the ERA

Contribution to metric variabilitybefore the experiment

Source 1

Source 2

Source 3

Source 4

(a)

Contribution to metric variabilityafter the experiment

Source 1

Source 2

Source 3

Source 4

(b)

Fig. 7 Notional variance-based sensitivity anal-yses for ranking the importance of sources ofuncertainty. A physical experiment reduces theepistemic uncertainty surrounding source 1.

project is called the Environmental Design Space(EDS) [26]. EDS is a physics-based, integrated,multidisciplinary M&S environment that consistsof core modules originally developed by NASA.The current EDS modeling capability for theAFC-enhanced VT technology is at the compo-nents level and up in Fig. 5. Additional M&Stools are being brought in to capture behavior atlower levels.

Both probabilistic and non-probabilistic ap-proaches to characterize epistemic uncertaintyare being considered for the case study. The au-thors are working closely with NASA subject-matter experts to elicit distributions and rangesfor sources of uncertainty. In addition to iden-tifying sources of uncertainty from the physicaldecomposition, shown in Fig. 5, measurementuncertainties and uncertainties associated withmodeling have also been identified. Upon com-pletion of the M&S envrionment, uncertainties atthe technology level will be propagated to metricsof interest, such as aircraft cruise drag, weight,thrust-specific fuel consumption, and fuel burn.

10


3.4 Design and evaluation of physical exper-iments (step 4)

In this methodology, it is assumed that physicalexperiments are designed and proposed by tech-nologists, but they must be aware of informationfrom previous steps in order to produce relevantexperiments. Of particular importance is the def-inition of the technology. This is because physi-cal experiments should be designed with system-level constraints in mind. For example, pneu-matic or electrical power availability for AFC ac-tuation will be limited on an aircraft; thus, it isnot always useful to collect experimental data un-der the assumption that any resources availablein the laboratory should be used to demonstratesignificant performance benefits. This is espe-cially true when the cost of experimentation isexorbitant. Another key result that can guide theexperimental design is the ranking of uncertain-ties, by importance, for critical metrics, as seen inFig. 7a. Technologists should also be informed ofthe criteria that DMs will use to evaluate the ex-periments.

Once experiments are designed, all criteriacan be determined for each. Criteria representingobjectives such as maturity, cost, and scheduleare relatively straightforward to estimate beforeany experiments are selected and conducted. Cri-teria associated with performance improvementand uncertainty reduction after the experimentare more complicated to predict a priori. Re-searchers such as Sankararaman et al. [27] haveproposed the use of Bayesian inference for thispurpose, but determination of the most appropri-ate approach for this methodology is part of cur-rent research activities.


Two of the experiments for the AFC-enhancedVT have been performed: a sub-scale wind tun-nel experiment and a full-scale wind tunnel ex-periment. These experiments are being stud-ied retrospectively to understand the logic behindtheir design. Lessons learned are being appliedto predict uncertainty reduction and performanceimprovement for the full-scale flight experiment.Data for quantities such as cost, schedule, and

QuantifiedgwithgM&Sandgexpertgelicitation

A B C D E

Eva

luat

iong

crit

eria

Physicalgexperimentsgdesignedgbygtechnologists

MCDM

Rankedgphysicalgexperiments

ExperimentgA

ExperimentgB

ExperimentgC

ExperimentgD

ExperimentgE

Fig. 8 Notional depiction of the decision processfor selecting physical experiments.

TRL are also available for use in applying thismethodology to a real program.

3.5 Selection via informed decision makingand execution of experiment(s) (step 5)

The objective of this step is to find the best com-promise physical experiment(s). The word "com-promise" is used here because it is usually thebest type of solution that can be found in multi-criteria problems. Conflicting criteria, such asuncertainty reduction and cost, result in trade-offs of performance in each. A plethora of multi-criteria decision-making (MCDM) tools exist foraiding DMs in finding the best alternative. Mostof the MCDM methods incorporate DM-suppliedcriteria preferences in the form of weights. Noone MCDM technique is available that is guar-anteed to work well for all problems, but multi-ple approaches to selecting an MCDM method,such as expert/intelligent systems, are designedfor facilitating selection. The result of employ-ing an MCDM tool is a ranking of alternatives, as

11


shown in Fig. 8. As part of the planned researchfor evolving this experiment selection methodol-ogy, MCDM methods will be investigated and themost suitable options chosen.

After an experiment or set of experimentsis selected and executed, DMs must determinewhether another experimentation iteration will bedone. If the feedback loop in Fig. 4 is followed,step 3 is carried out again to update uncertaintywith the recently acquired knowledge from ex-perimentation. Updates to the M&S environ-ment may be warranted, and new sources of un-certainty could be uncovered. The experimentaldata should be used to update the form of epis-temic uncertainty sources. If PDFs are used forcharacterization, a logical approach is to employBayesian inference. The goal of Bayesian infer-ence is to update prior knowledge of a distribu-tion parameter of interest using the observed data.The updated distribution is attained by applyingBayes’ theorem:

π(θ|D)︸︷︷︸posterior

=

likelihood︷︸︸︷f (D|θ)

prior︷︸︸︷π(θ)

m(D)︸︷︷︸marginal

(3)

The likelihood function summarizes informationfrom the experimental data D. The parameter θ

is not directly observable, but it is inferred. Forexample, if the data are generated by a normaldistribution, then θ could be the mean of that dis-tribution. The prior PDF represents the DM’sknowledge about the parameter θ before the ex-perimental data are observed, and the posteriorPDF represents the updated knowledge about theparameter after observing the data. The marginaldistribution can be thought of as a normalizingconstant, once the prior is specified and the dataare observed, which ensures that the posterior isa proper PDF. Results from updating with Eq. (3)can be used to produce a new sensitivity analysis,which will likely demonstrate a different prioriti-zation of uncertainties, as seen in Fig 7b. Thisinformation can then be leveraged for anotherround of experiment selection and execution byfollowing steps 4 and 5.


In future work, MCDM techniques will be usedto simulate the selection of AFC-enhanced VTphysical experiments. Results will be comparedwith the process used to determine which experi-ments to conduct in the ERA project. Input fromNASA DMs will be valuable for vetting the re-sults of applying this methodology.

4 Summary and future direction

In order to accelerate the reduction of uncer-tainty surrounding the impacts that technologieswill have on future aircraft, physical experimentsmust be carefully selected. As a contribution to-ward enabling an optimal technology develop-ment program, a methodology for strategic selec-tion of physical experiments has been proposed.An AFC technology applied to the VT of a com-mercial transport been presented as a case studythroughout the methodology formulation.

Additional research is necessary to maturethe experiment selection methodology. In par-ticular, future work will focus on four compo-nents: the prediction of uncertainty reduction andperformance improvement before an experimentis conducted, selection of the most appropriatesensitivity analysis methods, selection of suitableMCDM techniques, and the updating of uncer-tainties after an experiment is conducted.

Acknowledgments

The authors would like to thank Dr. Fay Collierat the NASA Langley Research Center for sup-porting this research under the EnvironmentallyResponsible Aviation project.

References

[1] Guynn, M D, Berton, J J, Tong, M T, and Haller,W J. Advanced single-aisle transport propulsiondesign options revisited. 13th AIAA AviationTechnology, Integration, and Operations Con-ference, Los Angeles, CA, AIAA Paper 2013-4330, 2013.

[2] Prandtl, L. Über Flüssigkeitsbewegung bei sehrkleiner reibung. Proceedings of the Third Inter-

12


national Mathematical Congress, Heidelberg,Germany, pp. 484–491, 1904.

[3] Williams, D R and MacMynowski, D G. Briefhistory of flow control, Fundamentals and ap-plications of modern flow control, edited by RD Joslin and D N Miller. AIAA, Reston, VA,pp. 1–20, 2009.

[4] Mankins, J C. Technology readiness and risk as-sessments: a new approach. Acta Astronautica,Vol. 65, No. 9-10, pp. 1208–1215, 2009.

[5] Cattafesta, L N and Sheplak, M. Actuators foractive flow control. Annual Review of Fluid Me-chanics, Vol. 43, No. 1, pp. 247–272, 2011.

[6] Coanda, H.. Device for deflecting a stream ofelastic fluid projected into an elastic fluid. U.S.Patent No. 2,052,869, September 1, 1936.

[7] Woszidlo, R.. Parameters governing separationcontrol with sweeping jet actuators. Ph.D. Dis-sertation, Department of Aerospace and Me-chanical Engineering, University of Arizona,Tucson, AZ, 2011.

[8] Seele, R, Graff, E, Lin, J, and Wygnanski,I. Performance enhancement of a vertical tailmodel with sweeping jet actuators. 51st AIAAAerospace Sciences Meeting including the NewHorizons Forum and Aerospace Exposition,Grapevine, TX, AIAA Paper 2013-0411, 2013.

[9] Phillips, E and Wygnanski, I. Use of sweep-ing jets during transient deployment of a con-trol surface. AIAA Journal, Vol. 51, No. 4, pp.819–828, 2013.

[10] Tewes, P, Taubert, L, and Wygnanski, I. On theuse of sweeping jets to augment the lift of alambda-wing. 28th AIAA Applied Aerodynam-ics Conference, Chicago, IL, AIAA Paper 2010-4689, 2010.

[11] Seele, R, Tewes, P, Woszidlo, R, McVeigh, M A,Lucas, N J, and Wygnanski, I J. Discrete sweep-ing jets as tools for improving the performanceof the V-22. Journal of Aircraft, Vol. 46, No. 6,pp. 2098–2106, 2009.

[12] Warwick, G. Boeing, NASA test activeflow-control tail. Aviation Week, Novem-ber 14, 2013, http://aviationweek.com/awin-featured-story/boeing-nasa-test-active-flow-control-tail [cited 26 June 2014].

[13] Mooney, H P, Brandt, J B, Lacy, D S, andWhalen, E A. AFC-enabled vertical tail sys-

tem integration study. NASA/CR-2014-218168,2014.

[14] Nikolaidis, E. Types of uncertainty in designdecision making, Engineering design reliabil-ity handbook, edited by E Nikolaidis, D Ghio-cel, and S Singhal. CRC Press, Boca Raton, FL,2005.

[15] Oberkampf, W L and Roy, C J. Verification andvalidation in scientific computing. CambridgeUniversity Press, New York, 2010.

[16] O’Hagan, A and Oakley, J E. Probability is per-fect, but we can’t elicit it perfectly. ReliabilityEngineering & System Safety, Vol. 85, No. 1-3,pp. 239–248, 2004.

[17] Hetherington, S. Knowledge. Internet Encyclo-pedia of Philosophy, http://www.iep.utm.edu/knowledg/ [cited 30 June 2014]".

[18] Mavris, D N, Baker, A P, and Schrage, D P.IPPD through robust design simulation for anaffordable short haul civil tiltrotor. AmericanHelicopter Society 53rd Annual Forum, VirginiaBeach, VA, 1997.

[19] Dieter, G E and Schmidt, L C. Engineering de-sign. 4th ed., McGraw-Hill, New York, 2009.

[20] Bjorkman, E A. Test and evaluation resourceallocation using uncertainty reduction as ameasure of test value. Ph.D. Dissertation,School of Engineering and Applied Science,George Washington University, Washington,D.C., 2012.

[21] Myers, R H, Montgomery, D C, and Anderson-Cook, C M. Response surface methodology:process and product optimization using de-signed experiments. 3rd ed., John Wiley & Sons,Hoboken, NJ, 2009.

[22] O’Hagan, A. SHELF: The Sheffield ElicitationFramework, http://www.tonyohagan.co.uk/shelf/index.html [cited 24 June2014].

[23] Grimmett, G R and Stirzaker, D R. Probabilityand random processes. 3rd ed., Oxford Univer-sity Press, New York, 2001.

[24] Lee, S H and Chen, W. A comparative studyof uncertainty propagation methods for black-box-type problems. Structural and Multidisci-plinary Optimization, Vol. 37, No. 3, pp. 239–253, 2008.

[25] Saltelli, A, Ratto, M, Andres, T, Campolongo, F,Cariboni, J, Gatelli, D, Saisana, M, and Taran-

13

http://aviationweek.com/awin-featured-story/boeing-nasa-test-active-flow-control-tail




http://www.iep.utm.edu/knowledg/

http://www.iep.utm.edu/knowledg/

http://www.tonyohagan.co.uk/shelf/index.html

http://www.tonyohagan.co.uk/shelf/index.html


tola, S. Global sensitivity analysis: the primer.John Wiley & Sons, Chichester, England, 2008.

[26] Kirby, M R and Mavris, D N. The environmen-tal design space. 26th International Congressof the Aeronautical Sciences, Anchorage, AK,ICAS 2008-4.7.3, 2008.

[27] Sankararaman, S, McLemore, K, Mahadevan,S, Bradford, S C, and Peterson, L D. Test re-source allocation in hierarchical systems usingBayesian networks. AIAA Journal, Vol. 51, No.3, pp. 537–550, 2013.

Contact author email address

For further information, please send an email to:[email protected]

Copyright statement

The authors confirm that they, and/or their company or or-ganization, hold copyright on all of the original materialincluded in this paper. The authors also confirm that theyhave obtained permission, from the copyright holder of anythird party material included in this paper, to publish it aspart of their paper. The authors confirm that they give per-mission, or have obtained permission from the copyrightholder of this paper, for the publication and distribution ofthis paper as part of the ICAS 2014 proceedings or as indi-vidual off-prints from the proceedings.

14

formulation of an experiment selection methodology … · 2014. 12. 11. · boeing 757 vt, where a...

Documents