Progress and Challenges in Numerical Simulation of Multi-physics Turbulent Flows in

Aerospace Applications

Parviz MoinCenter for Turbulence Research

Royal Academy of Engineering, Spain

(2003 Estimate)

A Story from the aircraft industry

• In 2003, Boeing estimated that the number of wing tests for 787 would be 5, representing a significant reduction from 11 a decade earlier.

• Estimates were based in large part on the increased use of simulation and enormous increase in compute resources during the decade 1995 to 2005 (~1000x)

(2005 Data)

A Story from the aircraft industry

• By 2005, the actual number of wing tests required was 11, the same as a decade earlier

• Why? computer power was not the largest source of uncertainty in their predictions: it was model fidelity.

• High fidelity methods that incorporate more “first principles” are a path to predictive simulations because they can leverage the dramatic increase in compute power available

Turbulence

Turbulence is the chaotic state of fluid motion that arises when the flow speed is higher than just the creeping motion

It is the rule, not the exception, in fluid dynamics

Transition prediction: DNS, 1.2B, 32k Cores

Time Evolution at Fixed Position

Methods for numerical simulation of turbulent flows

9/17/2014 6

Reynolds Averaged Navier Stokes (RANS). Average

over all turbulence scales.

Direct numerical simulation (DNS). Not practical for

applications.

Large Eddy Simulation (capture large eddies and

model small scales)

Large Eddy Simulation of Turbulence

Resolve the large scale motions directly and “model” the effect of small scales

Turbulent fluctuations are obtained as part of the solution in LES; only small scale

phenomena (largely universal) are modeled

Useful for prediction of large scale mixing, stirring and engulfing, pressure

fluctuations, noise, distortion of electromagnetic waves, …

Multi-physics Turbulent Flows Modeling Challenges

Many modeling and simulation challenges can benefit from a high-fidelity approach (LES):

• Compressible flow with shocks and complex mixing

• Laminar/turbulent flow transition

• Two Phase flow

• Combustion dynamics and coupled thermo-acoustics

• Integrated system issues, e.g. combustor/Turbine

Goal for this talk is to illustrate where we are in many of these areas, and where we are going in the near future

Filtering

Computational Grid

u u

10

Differential filtering on unstructured gridsGermano 1986a,b

Previous filtering approaches on unstructured grids built weighted sums from surrounding point cloud (Marsden et al 2002, Haselbacher & Vasilyev 2002)

Not robust on arbitrary meshes & complex to implementDifficult to decouple filtration from grid topology

Alternatively, introduce a linear operator whose Green’s function can be manipulated through a specification of a filter width

,

For constant p on an unbounded domain:

Elements of Large Eddy Simulation (LES)

9/17/2014 11

Traditional components

• Filtering-- constitutive equations

• Subgrid scale modeling ---- Dynamic Modeling (1991)

• Wall modeling-------Slip wall (2012)

• Numerical Methods----Continued advances since 2010

New considerations

• Interlink among above components

• Computer science

• Multiphysics (Combustion, Multiphase, cavitation…)

Stand-alone research in anyone of these areas is not

going to have large engineering impact

• It is important for LES calculations to predict accurately the quantities that led to choosing LES in the first place (e.g., turbulent fluctuations, acoustic sources, mixing,…).

• Numerical dissipation present in most codes, originally designed for RANS, is inadequate for LES

• Dispersion errors important for compressible flow and prediction of aerodynamic noise

• LES imposes additional requirements on mesh quality and size

Not all LES’s are equal: Numerical Methods

Decision to go unstructured

Penalty on a per-node/cv basis (5x+), however:

Complex geometry (e.g. combustor + turbine stage)

Mesh Flexibility: adaptation and refinement

Massive parallelism

9/17/2014 13

14

General Unstructured Meshes----Adaptation

Solvers designed to handle:

General polyhedral grids with hanging nodes

Complex geometries with body-fitted meshes

Local refinement in regions of interest

Grid-sensitive operators to reduce numerical dissipation

Grid-sensitive operatorsLocalized mesh refinement

Directional-refinement capability in Charles

9/17/2014 15

Normal shock

due to

supersonic jet

impingement

In addition to handling complex

geometries, unstructured

directional adaptation also

supports complex physics by

focusing refinement exactly where

it is required:

Turbulent shear

layer

Flow physics of high speed jet impingement (ideally-expanded)

169/17/2014

Note interaction

of shear layer

and normal

shock

FWH approach for noise prediction from compressible flow solver

Acoustic computations: a challenging quantitative metric

17

end capsend caps

9/17/2014

Computation (ideally-

expanded)

Measurement

Predicted OASPL : 154 dB

Measured OASPL : 156 dB

9/17/2014

18

Local refinement to address SGS model shortcomings

SGS model inaccurate at sufficiently coarse resolutionTypically, global refinement is performed to obtain more accurate resultsGoal: locally increase resolution in regions to reduce stress from SGS model

Can scale SGS model contributions by k* (measure of SGS fluctuation energy) Use statistics of k* to guide adaptation

Local refinement done automatically using adapt (Cascade Tech) Parallel, anisotropic mesh refinement

19

k* adaptation for LES

Refine in regions where estimates of SGS fluctuations are largeNorm of SGS stress contribution scales with k*Refine grid (filter) in regions where k* > k*crit

k* is bounded from above:

Criteria will never refine in regions where Rkk ≤ 2k*crit

k* is O(∆f2) accurate to SGS kinetic energy

... versus gradient based refinement criteriaGradients large everywhere for turbulent flowsNo self-limiting mechanism; gradients continue to grow after refinement

20

k* adaptation applied to a 3D diffuserdiffuser midplane (z/H = 1.67)

logscale k* contours, instantaneous

x/H

y/H

mean k* contours

Refinement concentrated in separated shear layers (bottom/side walls)Homothetic refinement of flagged cells Adapted mesh results in increase of 15% in total cell count

21

Separated shear layer visualizationsdiffuser midplane (z/H = 1.67)

spanwise velocity contours Ux = 0 (blue)

spanwise velocity contours, coarse

LES

21

22

Streamwise mean velocity profilesdiffuser midplane (x/H = 1.67), post-adaptation

k* adapted mesh addresses shortcomings of coarser LES Accurate prediction of mean velocity in separated shear layer

Continued discrepancy near upper wall due to weak sep shear layer

LES, post-adaptation Kolade expt (2010)coarse LES

23

SGS-based mesh adaptationk* generalizations

Re = 8900 heated cylinder in crossflow

Mean Nusselt number

(Nakamura & Igarashi 2004)

Nested near wall mesh refinement using estimate of SGS temperature fluctuations

Wall Modeling- A pacing item for LES

A turbulent boundary layer and the required LES grid resolution:

Wall Model Implementation

Adopted strategy for general, unstructured-grid, massively

parallel solvers.

Additional cost of wall-model is only 6-7% thanks to dual-

constraint partitioning

26

Wall modeled LESA filtering perspective

Cost of LES for wall bounded flows driven by resolution of small scales in the near wall region (Choi & Moin, 2012)

(grid point estimates for attached flow over an airfoil)

Wall resolved LES: filter width tends to 0 at the wall → resolves near-wall structuresresolved scales obey no-slip boundary conditions

Wall modeled LES: filter width finite at the wall → averages over the inner layerwhat BCs do the resolved scales satisfy?

27

Differential filter based wall modeling: slip velocity BC

What happens if the near-wall filter width is not small?Expanding the differential filter expression at the wall:

Wall-resolved (no-slip)

Wall-modeled (slip BC)

28

Slip BC formulation (Bose and Moin, 2014)vs traditional wall models

No explicit computation of wall stress; no embedded law of the wall

No empirical parameters from inner layer RANS modelNo zonal decomposition for hybrid RANS/LES techniques

No sensitivity to matching locations or ad hoc parametersNo a priori specification of transition locations or separation points

No sensors needed to deactivate wall model to revert to no-slip

).

29

NACA 4412 at near-stall conditionsRec = 1.6 X 106, AoA = 13.86o

Dynamic slip BC model for non-trivial geometry with separation + varying resolutions

NASA LaRC turbulence modeling benchmark case (Rumsey et al, 2012) Compare with the expt of Wadcock (1978) (also Wadcock & Coles 1978)

Expt measures trailing edge separation x/c ≈ 0.85-0.86

Courtesy: NASA LaRC, Wilcox k-ω

Streamwise velocity

30

NACA 4412 at near-stall conditionsRec = 1.6 X 106, AoA = 13.86o

Dynamic slip BC model for non-trivial geometry with separation + varying resolutions

NASA LaRC turbulence modeling benchmark case (Rumsey et al, 2012) Compare with the expt of Wadcock (1978) (also Wadcock & Coles 1978)

Expt measures trailing edge separation x/c ≈ 0.85-0.86

Courtesy: NASA LaRC, Wilcox k-ω

Streamwise velocity

x/c = 0.953

31

NACA 4412: trailing edge separationWall-model streamwise slip velocity

No-slip achieved due to shear layer refinement

Unsteady trailing edge separation

Contours blanked for Us ≤ 0 indicating separation or no-slip

“Transition”

32

NACA 4412: pressure coefficientRec = 1.6 X 106, AoA = 13.86o

WM-LES

Wadcock expt 1978

Overall agreement satisfactory with experimentCaptures Cp flattening due to separation at trailing edgeOverprediction of Cp during “transition” on suction side

Subgrid scale modeling in two phase flow

• Common practice in CFD to inject distributions of Lagrangian drops to represent fuel spray

• Based heavily on empirical correlations and experimental data – not predictive

• Need to be able to simulate primary atomization of fuel with high-fidelity approaches

• Physics-based subgrid scale models of fuel breakup are required

Experiment (Marmottant et al.)

Numerical Simulation

349/17/2014

Physical Breakup Process: pinching-off

Experiment (Tjahjadi et al. JFM 1992)

35

Refined Level set Grid Method (Herrmann 2008)

• Capillary instability leads to formation of satellite drops

• Number and size of drops can be predicted using stability theory

Physical Breakup Process: pinching-off

36

Experiment (Marmottant et al.)

• Ligaments undergo similar instability, pinching off to form small drops

Subgrid scale modeling concept

Method proposed by Kim & Moin (2011):

1. Detect ligament using resolution criteria

2. Locally solve stability problem with interface geometry as initial condition

3. Replace ligament with drops in Lagrangian DPM

Experiment Coarse grid Fine Grid

Subgrid scale model in action

• Subgrid droplet model in action for the coaxial liquid jet simulation

Lagrangian drops

Ligament just above detection threshold Ligament replaced by satellite drops

Sub-grid scale model validation

• Quantitative comparison to measured droplet pdf

Experiment

(Marmottant et al. 2004)

Pd

f (1

/mm

)

Diameter (mm)

subgrid drops

D / D = 0.01

Reacting Flow Challenges

Several competing approaches differing in cost, turbulence closure, complexity

of chemical mechanism, combustion regime,

Flamelet/Progress-Variable approach

• Assumes thin flame structure

• Tabulation of complex chemistry -> Reasonable cost

• Must be extended to include complex effects

• Autoignition, heat transfer, slow species, different regimes

PDF/FDF Transport approaches

• Accurate chemistry and turbulence closure, but costly

• Issues with mixing closure

Reduced Mechanisms

• Turbulence closure problem

Advocate for a balanced approach that doesn’t preference chemical fidelity over

flow fidelity, geometric fidelity

41

Sandia Flame D

Mixture

Fraction TemperatureHeat

Release NOx

Growth in Computing Power

7 years

100 times

Closing the Efficiency Gap

Hejaz. et al. cavitation

simulations:

70% of peak

Made possible by a critical mass of technology and people co-located at Stanford

through nearly two decades of DOE/NNSA support. Acquired the multi-disciplinary

culture needed to build, run and validate LES with realistic hydrocarbon chemistry, 2-

phase flow in complex geometries, noise, aero-optics, shape optimization, flow

control, hypersonics, …

45

Hot Supersonic Over-Expanded Jet (Chevron Nozzle)

46

Hot Supersonic Over-Expanded Jet (Chevron Nozzle)

47

Hot Supersonic Over-Expanded Jet (Chevron Nozzle)

Noise Predictions (1 Million cores)

150 deg

Near-field noise prediction

Blind comparison with UTRC experiment

Far-field noise prediction

Mj =1.35

Ma = 1.84

Tj = 1.85

Rej = 130,000

mesh: 55M cells

Temperature

Conclusions and Outlook

• Numerical methods and numerical analysis (e.g. stability

of multi-physics coupling) remain critical

• Grid generation for turbulent flows

• Conservative interface tracking on unstructured mesh

• Computer power increasing at 100x/7yrs but architectures

changing rapidly due to power constraints:

• challenges in programming these heterogeneous

systems efficiently (e.g. Liszt DSL)

• challenges associated with truly massive parallelism:

e.g. 1,000,000 cores

• Physics-based subgrid models will remain an important

element of LES of multiphysics engineering systems

High-fidelity reacting turbulent flow simulation for simple geometries only

Multicode/Multiphysics integration non-existent

Parallel computations with modest (~64) number of processors

e.g. gas-phase combustion only

Computational Capabilities in 1997

64 processors!

Computational Capabilities in 2009

Integrated Multiphysics/Multicode Simulations to support analysis and design

e.g. realistic aircraft engine simulations using multi-code coupling on 1000’s of processors

2000 processors

Code 1

Code 2

Code 3

LES Cost Estimates

NASA Vision 2030 LES cost estimates are pessimistic

• 100x: Based on reported calculations and/or optimal resolution of unstable modes and/or unstructured mesh adaptation

• 10x: Most CFD codes operate at 3-5% peak performance; we can reach 30-50% with better memory movement and code analysis in partnership with our CS (e.g. DSLs)

At least a factor 1000 too conservative

A 5-Exaflop/s machine can design & optimize, not just produce a one-off calculation!

Multi-physics WMLES of combustors is used in the propulsion industry now

Engineering use of full-wing WMLES is feasible

CharLES on Sequoia + Vulcan system

Leverage of the “Early Science

program” at LLNL (PSAAP)

Unique combination of

Vulcan+Sequoia

1.9 million cores

24.5 petaflops

539/17/2014

54

Large-scale scalability of explicitly filtered LES

Strong scalability of coupled duct + diffuser LES, 55M cvs

Intrepid, BG/P (ANL)

~75% eff. with <2K cvs/core

Scalability achieved with differential filters, locally refined meshes, new SGS models, and coupled simulationsPoisson solver scalability still largest bottleneck

Not subject to the EAR per 15 C.F.R. Chapter 1, Part 734.3(b)(3).

55

Software infrastructureUnstructured solvers for engineering applications

Differential filters and SGS models implemented in CharLES

Allow locally refined meshes with pockets of hanging nodes 3D diffuser, NACA 4412 airfoil meshes locally refined

• Control with synthetic jet actuator

• CDP’s unstructured grid capability

• Spanwise vorticity ( )

velocity BC

synthetic jet actuator

uncontrolled

controlled

Flow Separation Control

56

uncontrolled

controlled

Uncontrolled Controlled

LES 0.83 1.43

EXP 0.82 1.41

Lift coefficientLines: LES

Symbols: Experiments (Gilarranz et al., JFE, ’05)

uncontrolled

controlled

Velocity in the wakeSurface pressure

Flow Separation Control

57

Unstructured grid LES Solver: Charles

• Unstructured LES solver Charles

• Second-order in space (low numerical dissipation & dispersion)

• Anisotropic grid refinement

• Arbitrary polyhedral elements

• Dynamic subgrid-scale LES turbulence models

• Massively Parallel MPI-basedinfrastructure (run up to 1.9Mcores)

Scaling performance

BG/P Argonne National Laboratory

- Intrepid

59

Streamwise velocity fluctuationsdiffuser midplane (x/H = 1.67), post-adaptation

Fluctuation prediction accurate for x/H < 10Centerline over-prediction for x/H > 12

possible causes: statistical under-sampling, fine → coarse resolution transition

LESKolade expt (2010)

60

CHARLES : Scaling on LLNL Sequoia

Problem: LES of jet

crackle

Mesh : 650M CVs

Cores : 131K to 1.05M

4000 to 620 CVs/core

Performance

88% parallel efficiency

at 1.05M cores (relative

to 131K cores)

217 = 131K

219 = 524K

220 = 1.05K

61

I/O Scaling

Competition between MPI ranks during I/O can hamper

scalability for I/O intensive problems

New, more flexible I/O model can lead to 6-8X speedup

Visual evidence of Numerical Dissipation in LES

From Liu et al.

AIAA J. 2009,

MILES

Supersonic Jet LES using MILES-base method

Supersonic Jet LES using low-dissipation method (Charles)

Power – and the Exaflop machine in 2020

DOE planning to build an exaflop machine by 2020 that uses 20MW (dramatically reduced power/flop)

However, scaling of our problems is hard: e.g. for a factor of 2 in grid length scale, we need a factor of ~2^4=16 in computation power, or about 4 years

For a factor of 10 in length scale, need ~13 years

In the next decade:

physics-based sub-grid modeling will remain a critical part of high-fidelity simulations

Methods should carefully focus increased fidelity to beat these estimates (e.g. unstructured grids, fidelity of chemistry)

639/17/2014