cfd validation: what is it? & how does it affect us? aka ... · • validation: the process of...

CFD Validation:

What is it? &

How does it affect us?

aka:

“To validate, or not to

validate, that is the question”

N Taylor (MBDA UK Ltd)

RAeS Aerodynamics Conference

October 14-15 2019

This document and the information contained herein is proprietary information of MBDA and shall not be disclosed or reproduced without the prior authorisation of MBDA. © MBDA 2019.

Slide :

Outline

• Caveats

• This is a wide-ranging & complex topic …

• … which impinges on several other wide-ranging & complex

topics …

• So, inevitably, this presentation will not address everything

• Nor may it do justice to those aspects that are addressed

• CFD Validation: What is it?

• Underlying concepts

• How they are evolving

• CFD Validation: How does it affect us?

• Stakeholder engagement

• Towards a unifying perspective

• Potential opportunities

• Closing remarks

2


Slide : 3

CFD Validation: What is it?


Slide :

To validate …

• Some popular definitions of validation:[1]

• The action of checking or proving the validity or accuracy of something

• “The technique requires validation in controlled trials“

• The action or process of making or declaring something legally or officially

acceptable

• “Parking is free with validation of your ticket”

• Recognition or affirmation that a person or their feelings or opinions are

valid or worthwhile

• “As human beings we crave validation from the outside world”

• So, what is actually meant by CFD Validation?

4


Slide :

CFD Validation defined?

• When asked this question, the first point of reference is

usually:

• AIAA-G-077-1998: Guide for the Verification and Validation of

Computational Fluid Dynamics Simulations[2]

• This proposes the following definitions:

• Verification: The process of determining that a model implementation

accurately represents the developer’s conceptual description of the

model and the solution to the model

• Validation: The process of determining the degree to which a model is

an accurate representation of the real world from the perspective of

the intended uses of the model

• However:

• This definition of CFD Validation has

proved somewhat controversial

• In order to explain why, it is helpful

to review the historical context …

5


Slide :

A Brief Historical Perspective (1)

“Pre-formalisation” (~pre-1990s)

• 1960s - 70s

• Several fundamental CFD-related breakthroughs (s/w and h/w)

• Immediacy of potential utility oversold

• CFD will replace WT in ~10 years[3]

• 1970s - 80s

• Industrial use becomes more widely established

• Inviscid or viscous-coupled (e.g. TSP … Euler)

• NB: Calibration used to improve match with experimental data

• Calibration: The process of adjusting numerical or physical

modelling parameters in the computational model for the purpose

of improving agreement with experimental data[2]

6

see [3]


Slide :


Early Formalisation (~1990s):

• Emergence of RANS

• Turbulence modelling “arrives” – an additional basis for calibration

• Evaluation and Validation

• Publication of experience-based perspectives

• Interesting variances observed across the stakeholder community

(Researchers, Developers … Industrial end-users)

• Contemporary apocryphal quote: "Nobody believes analytical results except the man who programmed it.

Everybody believes wind tunnel results except the man who tested it.”[4]

• Establishment of various initiatives (e.g. ERCOFTAC)

as incubators of “Best Practice”

• AIAA Guide G-077-1998 first issued

• Re-affirmed in 2002

7


Slide :


Initial Community Response (1) : ~2000s

• Pace of improvement in CFD capability continues

• Complexity, scale, automation : DES(etc.)

• -> The end of calibration is nigh … (or is it?)

• Initiatives beyond the AIAA Guide

• Various community resources established • NPARC Alliance, NASA Turbulence Modelling Resource ...

• AIAA CFD Workshops

• ASME standards published

• Disagreements emerge wrt the meaning of CFD Validation

• Within the immediate community …

• Principals in the debate “switched sides”[5]

• … and beyond it (see next slide)

8


Slide :


Initial Community Response (2) : ~2000s

• Here we need to introduce “the” definition of CFD Prediction:

• Prediction: The use of a CFD model to foretell the state of a physical

system under conditions for which the CFD model has not been

validated[2]

‘CFD validation cannot consist of the comparison of the results of one code to

those of one experiment. Rather, it is the agglomeration of comparisons at

multiple conditions, code-to-code comparisons, an understanding of the wind

tunnel corrections, etc., that leads to the understanding of the CFD uncertainty

and validation of its use as an engineering tool. Examples include comparisons

of predictive CFD to subsequently acquired test data. The question is not can

CFD give a great answer for one or two test cases, but can the CFD

‘‘processes’’ give good answers for a range of cases when run by a competent

engineer? This is what validation for an intended purpose is all about.’’[7]

9


Slide :


Subsequent Refinements (1) : ~2000s - 2010s

• Limitations in AIAA Guide openly acknowledged by its

originators …

• “Restricted View” c.f. “Encompassing View”[8]

• … and others in the VnV community

10

see [10]

see [9]

see [8]


Slide :


Subsequent Refinements (2) : 2010s (cntd)

• Formalisation of alternative views[9]

• An “operational” definition:

“Model Validation is the compilation of useful

indicators regarding the accuracy and adequacy

of a model’s predictive capability for output

quantities of interest, where meaningful comparisons

of experiment and simulation results are

conducted at points in the modelling space

that present significant prediction tests for

anticipated uses of the model.”

• NASA Credibility Assessment

This considers eight factors that are assumed

“nearly orthogonal, i.e., largely independent”

and are evaluated in three categories:

- Development (data pedigree, verification, validation);

- M&S Use (input pedigree, uncertainty characterization, robustness); and

- Supporting Evidence (M&S history, M&S process/product management)

Each factor is evaluated with a five-level assessment scale.

But NB: Since M&S Use comes after verification and preferably after

validation, assumption of factor independence is questionable11


Slide :

So, where are we now? (1)

• Unfortunately:

• “Prediction Science is still a relatively young discipline and requires far

more development of effective and robust methodologies for

integrating experimental results and their uncertainties with modelling

and simulation, …” [9]

• Different approaches have been developed to

support validation in other fields of application.

However, apparently:

• “There is no agreement in the scientific community

today regarding best practices for validation of

extrapolative predictions made using computational models.” [11]

• Difficulties are not confined to predictions

• Even when physical test data is available, CFD Verification is not

always straightforward (even in

interpolative circumstances) …

• … nor is isolating its effects

from CFD Validation

12

see [10]

see [12]


Slide :

So, where are we now? (2)

• Thankfully, there appears to be growing acknowledgement

that:

• “Validation is not a procedure for testing scientific theory or for

certifying the ‘truth’ of current scientific understanding. … Validation

means that a model is acceptable for its intended use because it

meets specified performance requirements.” [13]

• “Validation, from a practical or engineering perspective, is not a

philosophical statement of truth” [14]

• Various means for distinguishing between the requirements of science

and engineering are also beginning to emerge (e.g.[15])

• However, IMO, recent publications on the subject are still

deficient in a number of regards, e.g.

• Reference to “V&V and UQ” [14]

(as if UQ is not an inherent part of Validation – or Verification)

• Little attention is drawn to matters beyond those pertaining to a

scientific perspective of the “Restricted View” of CFD Validation

• Work is underway to update the AIAA Guide …13


Slide : 14

CFD Validation: How does it affect us?


Slide :

Scope and Approach

• Responses to this question are diverse

• Total lack of interest (perceived relevance) … sleepless nights (&c)

• Different perceptions of CFD Validation can result from

fundamental clashes in culture:

• Philosophy-of-science v Practitioner’s views[6]

• CFD is not physics – rather embodiment(s) of models of the physics

• The extensive published material, while well-written and

logical, is lacking in practical guidance re its application

• Applicability may be clear in the minds of its originators, but the “value

added” is not always clearly transmitted in the open literature

• NB It is recognised that certain “fitness-for-purpose” aspects of

validation will be context specific – and therefore may not be shareable

• IMO, it is in the mutual interest of all stakeholders to build

better bridges wrt CFD Validation

• To illustrate my reasoning, I will start by considering the “Validation

Hierarchy”[6]

15


Slide :

Validation Hierarchy (1)

• Key Features:

• Conceived as a means for identifying

“Validation Experiments”

• Constructed top-down, from Complete

System to Unit Problem levels (4 tiers)

• Phenomena Identification and Ranking

Tables (PIRT) used to prioritise them

• NB: All published examples appear idealised

or, at best, highly simplified

• Some Open Questions:

• It is not clear how entries have been

selected or excluded

• What are the implications for a physical

phenomena not having an entry?

• How daunting is the population task?

• How many dependency levels are

required (and what is the significance

of each dependency)?

• What does such a hierarchy tell us

that we don’t know already?16

see [6]

see [6]


Slide :

Validation Hierarchy (2)

• Some Observations:

• In the absence of a clearer understanding of the

dependencies that may (or may not) be implied in a

Validation Hierarchy, it is interesting to compare it with ---->

• Extensive use of “Systems Models”

reflects industrial ways of working

• e.g. for requirements definition & identification

of pragmatic ways for addressing gaps

in demonstrated CFD capability

• A systems-oriented view highlights the importance

of coverage as well as focus

• NB Aerodynamics is not a system (or a sub-system)

Rather aerodynamic forces and moments are Emergent Properties

• Currently, Validation Hierarchy entries appear unduly “one-dimensional”, e.g.

• Phenomena are important, but so are transitional behaviours

• Why are the relationships with analogous modelling/algorithmic hierarchies absent?

• Would not other perspectives be beneficial too?

(Physical and functional architectures; levels of abstraction/fidelity) …

• The more granular one goes, the more generalised entries may become

• Potential for sharing lower tier entries with others …

• … including those operating within very different contexts (systems)?17

see [16]


Slide :

Towards a Unifying Perspective …

I offer two observations:

• (1) The business environments we work in (Academia, Research labs

… SMEs, Technology supply … Tier 1 OEM) exhibit strong mutual

dependencies, but our principal intended use(s) of CFD may differ

• Our primary requirements for validation will follow (“intended use”)

• (2) Product validation is, ultimately, always undertaken at the system

level, in accordance with local (Time, Cost, Quality) constraints

• There are strong context dependencies here, e.g. stage in lifecycle,

type of (product, market) …

• Wherever we find ourselves in the stakeholder community:

• Our competitiveness will suffer if we do not incorporate developments

made available by other stakeholders

• Our CFD processes will fail to gain/maintain credibility if these

developments are not used correctly

• Examples of “Vertical” (stakeholder engagement) and “Horizontal”

(diversity in approach) implications, drawn from personal

experience, follow …

18


Slide :

“Vertical” Implications (1) : Validation Experiments

• Perhaps understandably, the open literature exhibits a strong

focus on the additional requirements for physical testing

• VE conceived as means for supporting CFD Validation by generating

more detailed physical measurements than might otherwise be required

• IMO a more balanced approach is essential, e.g.

“No physical testing w/o first having conducted systematic

numerical studies”

• Help establish clear and precise objectives

(incl. priorities for physical measurements)

• Provide other insights (e.g. as pertaining to Verification)

• Identify other opportunities (e.g. alternative, potentially simpler,

stepping-stones towards meeting the overall objectives)

• Begs the question: which Validation Experiments to conduct?

• (How) Does a “Validation Hierarchy” help to identify good candidate

“points of entry”?

• How to develop and maintain the taxonomy and current status of

general (shared) entries?19


Slide :

“Vertical” Implications (2) : Turbulence Modelling

• Turbulence modelling – a waiting game

• Is there a “glass ceiling”?[17]

• When/how will HPC-induced obsolescence take effect?

• In seeking improved capability at manageable cost, where should investment

priorities lie? (e.g. [18])

• What are the latest developments – and how well prepared are we for their

adoption? e.g.

• Uniformity of understanding re underlying assumptions

• Consequent ability to undertake critical assessment

(incl. situational awareness: “Solution Integrity Checks”[20])20

see [19]


Slide :

“Vertical” Implications (3) : Community Initiatives

• There are many potential opportunities here, e.g.

• AIAA CFD workshops

• Establishment of CRM-HL eco-system[21] creates several

opportunities, potentially for all stakeholders e.g.

• Geometry & Mesh Generation: Special Session at AIAA

AVIATION 2020 (2D-section analysis)[22]

• Potential basis for fundamental CFD Validation activities

(detailed numerical assessments & physical measurements

for localised phenomena)

• MBDA Test Case (CFD_OTC1) )[20]

• Another industrially relevant test case with strong metrics

• Identifying issues other than those associated with physical

measurements

• Interactive community approach being adopted … with various

stakeholders … starting with CFD Verification

• Allowing the models and algorithms to do more of the talking

• Special Sessions being proposed for AIAA meetings in 202121


Slide :

“Horizontal” Implications (1) : Fitness-for-Purpose

• In an industrial context, validated CFD capabilities are only a

means to an end

• What we require are predictive capabilities that can be applied with confidence

(NB: Currently accumulated during the course of multiple learning processes)

• Predictive UQ – another waiting game

• When will the challenges of developing fully mathematically-based approaches

capable of producing reliable predictive UQ be met?

• How do we help accelerate the process of their development?

• Appealing to the “fitness for purpose” aspects of CFD

Validation may help

• Precise estimates of predictive UQ are not always required

• Look beyond the immediate mathematics

• A wide range of techniques contribute to assessments of fitness for

purpose: CFD does not exist in a vacuum

• Users play vital roles – but how to build confidence in practice (in the

presence of residual concerns about modelling assumptions, numerical

error …)?

• There are benefits to be gained from physical testing beyond obtaining

detailed measurements 22


Slide :

“Horizontal” Implications (2) : Cumulative Inheritance

• Validation of MBDA’s EST Process for missile concept

development:[23]

• Principal objective: to measure the EST-predicted

aerodynamic data against the benchmark of those

generated in the traditional (manual and interactive)

way, by experienced CFD users, using the tools (&c)

with which their practice has been established ...

• Primary focus placed on assessing overall

aerodynamic force and moment coefficients,

since these are the data most commonly used during

concept development

• So, is this CFD Validation?

• Despite the fact that no detailed physical

measurements were involved (and physical data

were, initially, only “inherited”), Yes

• Note: the context is important here

• CFD-specific benefits include:

• Permits substantially greater exercise

- reduced susceptibility to (various) localised bias(es)

- improved awareness of challenge landscape

• Identified industrially relevant point of entry to a Validation Hierarchy (with strong metric

characteristics and performance context): CFD_OTC1

23


Slide :

“Horizontal” Implications (3) : “Self-streamlining”

• Adaptive flexible-walled wind tunnel testing

• Early (first?) interactive use of CFD and

wind tunnel data (physical measurements)

• Two forms of Physical/Numerical boundary interaction[24]

• 2D : “self-streamlining” (interface matching)

• Transonic WT technique improvement ...

• 3D : target line techniques

• Manipulation / control of

model-wall interactions …

• Controlled interactions at well-defined

physical/numerical boundaries

• Not just a mechanism for modifying

longitudinal pressure gradient

• Powerful means for:

• Identifying “unknown unknowns”

• Keeping actors honest

• Permits low TRL level of entry (& discovery, insight …) 24

see [24]


Slide :

Closing Remarks

• Assessment of “fitness-for-purpose” is a fundamental

component of CFD Validation

• CFD Validation is more than a comparative analysis of physical

measurements and computed results – although the importance of

this is not in question

• CFD Validation is dependent on a number of “other” activities

• Broad and interactive stakeholder engagement will be required to

refine all aspects of CFD capability (including Validation)

• Hardware & software developments mean (Time, Cost,

Quality) balances are adjusting in favour of addressing long-

held modelling & simulation concerns more directly

• Industry-relevant points of entry are being defined to help accelerate

development at all TRL

• In spite of the needs for increased detail and the adoption of

increasingly systematic approaches, there is still plenty of

room for “discovery”

25


Slide :

End

26

Thank you for your attention

Any Questions?


Slide :

“To validate, or not to validate, that is the question”

27

“To be, or not to be: that is the question; Whether 'tis nobler in the mind to suffer

The slings and arrows of outrageous fortune,Or to take arms against a sea of troubles

And by opposing end them.”

Hamlet, W Shakespeare


Slide :

References

[1] Derived from result of searching for “validation meaning” @ google.co.uk

[2] “Guide for the Verification and Validation of Computational Fluid Dynamics Simulations”, AIAA-G-077-1998

[3] Chapman, D.R., “Aerodynamic Analyses Requiring Advanced Computers”, pp 4-7, NASA SP-347, 1975

[4] Bobbitt, P.J., “The Pros and Cons of Code Validation”, NASA TM 100657, 1988

[5] Roache, P.J., “Perspective: Validation—What Does It Mean?”, J Fluids Eng, v131, Mar 2009

[6] Oberkampf, W.L. & Trucano, T.G.,“Verification and Validation Benchmarks”, SAND2007-0853, 2007

[7] Kraft, E.M, “After 40 Years Why Hasn’t the Computer Replaced the Wind Tunnel?”, ITEAJ 31: 329–346, 2010

[8] Oberkampf, W.L. & Roy, C.J., “Verification and Validation in Scientific Computing”, CUP, 2010

[9] Mehta, U.B., (ed) “Simulation Credibility: Advances in Verification, Validation, and Uncertainty Quantification”, NASA TP-2016-219422, 2016

[10] Oberkampf, W.L et al, “Verification, validation, and predictive capability in computational engineering and physics”, Appl Mech Rev v57

pp345-384, 2004

[11] Oliver, T.A., et al, “Validating Predictions of Unobserved Quantities”, CompMethAppMechEng 283:1310-1335, 2018

[12] Ollivier-Gooch, C., “Is the problem with the Mesh, the Turbulence Model, or the Solver?”, AIAA 2019-1334

[13] Oberkampf, W.L. & Trucano, T.G.,“Verification and Validation in Computational Fluid Dynamics”, SAND2002-0529, 2002

[14] Lee, H.B., et al, “Development and Use of Engineering Standards for Computational Fluid Dynamics for Complex Aerospace Systems”,

AIAA 2016-3811

[15] Morrison, J.H., et al, “Observations on CFD Verification and Validation from the AIAA Drag Prediction Workshops”, AIAA 2014-0202

[16] https://www.dsiac.org/resources/journals/dsiac/fall-2014-volume-1-number-2/high-power-microwave-directed-energy-weapons

[17] Duraisamy, K. et al, “Status, Emerging Ideas and Future Directions of Turbulence Modeling Research in Aeronautics”, NASA/TM–2017–

219682

[18] Bush, R.H., et al, “Recommendations for Future Efforts in RANS Modeling and Simulation”, AIAA 2019-0317

[19] Slotnick, J., et al, “CFD Vision 2030 Study: A Path to Revolutionary Computational Aerosciences”, NASA/CR-2014-218178, 2014

[20] Taylor, N.J., “Separated Flow: Some Challenges and Research Priorities for Missile Aerdynamics”, STO-MP-AVT-307-23, 2019

[21] Slotnick, J., “Integrated CFD Validation Experiments for Prediction and Assessment of Separated Flows for Subsonic Transport Aircraft”,

STO-MP-AVT-307-6, 2019

[22] http://www.gmgworkshop.com/gmgw25.shtml

[23] Dodds, M., et al, “Validation of the EnGAM-SOLAR-TAU Process for Missile Concept Design” RAeS Applied Aerodynamics Conference,

Bristol 2014

[24] Ewald, B. (ed), “Wind Tunnel Wall Correction”, AGARD-AG-336, Chapter 10, 1998

28

https://www.dsiac.org/resources/journals/dsiac/fall-2014-volume-1-number-2/high-power-microwave-directed-energy-weapons

http://www.gmgworkshop.com/gmgw25.shtml

cfd validation: what is it? & how does it affect us? aka ... · • validation: the process of...

Documents