the dlr tau-code - recent and future...

The DLR TAU-Code - Recent and Future Developments

Norbert Kroll, Dieter SchwambornInstitute of Aerodynamics and Flow TechnologyCenter of Computer Applications in Aerospace Science and Engineering

IMPULSE progress meeting

Folie 2

slide 2

Slide 2

Tau User Meeting, DLR Braunschweig, 18-19. 10. 2011

Hyperflex

CFD

Implicit methods

Extended spatial discretization

Multigrid

improvement

Parallel efficiency

Grid adaptation

Physical Modeling

Chimera capability

Linear / adjoint

solver

Perspective

Content


Folie 3

slide 3

Slide 3


Improved CFD Capabilities HyperFlex

CFD

Goal Overall CFD process fully automatic Locally best fitted numerical approaches Combine structured & unstructured mesh regions and apply appropriate solver algorithms

Hyperflex

MesherStructured-dominant in near-fieldUnstructured / Cartesian in far-fieldFlexible interfaces (matching, overlapping, non-conforming)

Requirements –

Hyperflex

SolverApplication of locally best fitting numerical solver

courtesy

of ARA


Folie 4

slide 4

Slide 4


Hyperflex

CFD DLR Approach: Extension of unstructured edge-based flow solver TAU

Fully unstructured meshes: Improved base TAU solverUpwind-biased discretization

schemes Implicit Runge

Kutta

scheme with matrix-based point and line implicit preconditionersImproved algorithms for gradient computations

Block structured meshes:Advanced algorithms based on line information with equal or improved accuracy/performance compared to today’s structured solversNo explicit (I,J,K) structure required

Structured/unstructured meshes:Canonical switch from line-based algorithms (structured mesh regions) to point-based algorithms (unstructured mesh regions)Non matching interfaces of structured/unstructured meshes

Only unstructured data representation

offers full flexibility

TAU Canonical


Folie 5

slide 5

Slide 5


Standard Matrix Dissipation

(current Release)Unstrucutred

JST-scheme

Use grid scaling factor in dissipation (0/1): 1

2nd order dissipation coefficient: 0.5

Inverse 4th order dissipation coefficient: 64

Modified Matrix Dissipation

(current Release)Modified scaling of first and second order

dissipationUse grid scaling factor in dissipation (0/1): 0

Pressure switch weighting factor: 4

4th difference matrix dissipation coefficient: 0.15

Improved modified Matrix Dissipation (upcoming Release)

Improved pressure switch by Swanson & Turkel

for improved accuracy

Accuracy Improvement Central Scheme -

Matrix Dissipation

A. Schwöppe, S. Langer

NACA0012, M=0.8, AoA=1.250


Folie 6

slide 6

Slide 6


Standard Matrix Dissipation

(current Release)Unstrucutred

JST-scheme

Use grid scaling factor in dissipation (0/1): 1

2nd order dissipation coefficient: 0.5

Inverse 4th order dissipation coefficient: 64

Modified Matrix Dissipation

(current Release)Modified scaling of first and second order

dissipationUse grid scaling factor in dissipation (0/1): 0

Pressure switch weighting factor: 4

4th difference matrix dissipation coefficient: 0.15

Improved modified Matrix Dissipation (upcoming Release)

Improved pressure switch by Swanson & Turke

for improved accuracy

Accuracy Improvement Central Scheme -

Matrix Dissipation

NACA0012, M=0.8, AoA=1.250

A. Schwöppe, S. Langer


Folie 7

slide 7

Slide 7


Transition PredictionPerformance Improvement Implicit Methods

ApproachPreconditioned implicit multistage Runge-Kutta

(RK) method as multigrid

smoother Preconditioners:

Use only diagonal blocks of Jacobian

(point implicit)Use tri-diagonal blocks along predetermined lines (line implicit)Use full 1st order approximate Jacobian

(1st

order Jacobian)Matrix dissipation

GoalReduce stiffness (grid, turbulence)

robust & efficient solution algorithmEstablish canonical solver on hybrid meshes

S. Langer

(3)1)(

(2)3

(0)(3)

(1)2

(0)(2)

(0)1

(0)(1)

)((0)

WW

)W(RαWW

)W(RαWW

)W(RαWW

WW

n

n

tCFL

tCFL

tCFL

Explicit Runge

Kutta

method

Pi

: some approximation to the flux Jacobian

)3()1(

)2(13

)0()3(

)1(12

)1()2(

)0(11

)0()1(

)()0(

WW

)(WRPWW

)(WRPWW

)R(WPWW

WW

n

n

tCFL

tCFL

tCFL

Preconditioned Runge

Kutta

method


Folie 8

slide 8

Slide 8


Example: Laminar flat plate, M = 0.3, Re = 10000

Fine mesh

Lines identified

Stiffness

induced by grid is significantly

reduced

with line implicit approach

Performance Improvement Line Implicit Preconditioner

S. Langer

line implicit

expl. RK.


Folie 9

slide 9

Slide 9


Example: Laminar flat plate, M = 0.3, Re = 10000

Fine mesh

Lines identified

Comparison with LU-SGS.

Stiffness

induced by grid is significantly

reduced

with line implicit approach


S. Langer

line implicit

LU-SGS


Folie 10

slide 10

Slide 10


Example: Laminar flat plate, Ma = 0.3, Re = 10000

Convergence rate is comparable, even improved when compared with structured solver

FLOWerTAU, line-

implicit


Comparison with structured solver

S. Langer


Folie 11

slide 11

Slide 11


Example: RAE2822, Re=6.5e6, AoA

= 2.79°, M = 0.73, SA turbulence

model

Performance Improvement Comparison of Different Preconditioners

line implicit

full implicit 1st order Jacobian

explicit RK

Convergence of 1st

order implicit method 2 orders of magnitude faster compared to explicit RK

S. Langer


Folie 12

slide 12

Slide 12


Example: RAE2822, Re=6.5e6, AoA

= 2.79°, M = 0.73, SA turbulence

model


1st

order implicit:

Almost mesh independent convergence rate

Explicit RK: significant deterioration of convergence rate

S. Langer


Folie 13

slide 13

Slide 13


Example: High-lift, M = 0.22, Re=4e6, AoA

= 22.1°, SA turbulence

model


Convergence of 1st

order implicit method 2 orders of magnitude faster compared to explicit RK & LU-SGS

line implicit explicit RK

explicit RK


LU-SGS

S. Langer


Folie 14

slide 14

Slide 14


Example: High-lift, M = 0.22, Re=4e6, AoA

= 22.1°, SA turbulence

model

Performance Improvement Preconditioner: 1st

Order Jacobian

Lift converged after 150 MG cycle


S. Langer


Folie 15

slide 15

Slide 15



M=0.85, AoA

= 2.33°, Re = 5.x106

DPW4 configuration

S. Langer


Folie 16

slide 16

Slide 16


Performance Improvement Canonical Solver

Iteration

C-li

ft,C

-dra

g

C-m

y

0 2000 4000 6000 8000 10000

0.05

0.1

0.15

0.2

0.25

0.3

0.1

0.2

0.3

0.4

0.5

2010.2.0 - explicit, new matrix dissipation2010.2.0 - line implicit, new matrix dissipation

ONERA M6M=0.8395Re=11.7x106

=3.06mixed anisotropic mesh

C-drag

C-moment-y

C-lift

point implicit method

line implicit method

canonical solver

Example: ONERA M6 wing, M = 0.84, AoA

= 3.06o, Re = 11.7 x 106

S. Langer


Folie 17

slide 17

Slide 17


Performance Improvement Time Accurate Flows

Use line-implicit approach within dual time stepping method –

first results

pitch angle [° ]lif

t

-0.2 -0.15 -0.1 -0.05 0 0.05 0.1 0.15 0.2

0.29

0.3

0.31

0.32

0.33

0.34Runge-Kuttaline implicit

Lift vs. Pitch

angle for

3rd period

oscillating airfoil, turbulent flow oscillating LANN wing, turbulent flow

For given convergence level:Explicit Runge

Kutta: 1 physical time stepLine implicit Runge

Kutta: ~ 10-20 physical time steps

For given convergence levelRunge

Kutta: ~ 550 inner iterations Line implicit: ~ 125 inner iterations

S. Langer, Li Dian


Folie 18

slide 18

Slide 18


Point & line implicit preconditiondersStatus

Available in recent TAU version (developer) Designed for SA turbulence model

Ongoing work / open questions More efficient implementation Trade-off: convergence rate & robustness versus parallel efficiency Extension to 2-eq. turbulence models and RSM

Linearization of the models Best coupling strategy of flow and turbulent equations

1st

order Jacobian

based preconditionerTransfer into TAU repositoryInvestigation for more complex 3D applicationsTrade-off: convergence rate & robustness versus memory & parallel efficiency

Performance Improvement Implicit Methods


Folie 19

slide 19

Slide 19


Accuracy Canonical Switch from Structured to Unstructured Stencil

unstructured

lines 1-sided lines1-sided lines 2-sided lines

x

z0.99 1 1.01 1.02

0

0.01

0.02

TAU edge-based data structure extended to structured line data (angle criteria)

Line-edge-based data structure is equal to the block-structured data structure with respect to discretization

of convective & viscous fluxes, source terms and boundary conditionsLine-edge-based data structure is filled up during preprocessing

using structured informationExtension is independent from grid metric (Cell Vertex/Cell Centered)Parallel infrastructure exists in TAUCanonical switch is provided by existing TAU edge-colouring

A. Schwöppe


Folie 20

slide 20

Slide 20


x/c

cp

0 0.2 0.4 0.6 0.8 1

-0.500

0.000

0.500

1.000

FLOWer - cc - matrix - AoFTAU- cc - matrix - AoF - can.TAU- cc - matrix - AoF

NACA0012, ADIGMA MTC Level 3, Ma = 0.5, alpha = 2.0

x/c

tota

lpre

ssur

elo

ss

0 0.2 0.4 0.6 0.8 1-0.004

-0.002

0.000

0.002

0.004

0.006

0.008

0.010

FLOWer - cc - matrix - AoFTAU- cc - matrix - AoF - can.TAU- cc - matrix - AoF

NACA0012, ADIGMA MTC Level 3, Ma = 0.5, alpha = 2.0

Comparison of structured, unstructured and canonical discretizationCentral Scheme with matrix dissipation

Structured discretization can be reproduced with canonical discretization

Accuracy Canonical Switch from Structured to Unstructured Stencil

A. Schwöppe


Folie 21

slide 21

Slide 21


Performance Improvement Improved Multigrid

Agglomeration

Modified agglomeration procedure for semi-structured grid parts

1st

step: agglomeration of surface cells2nd

step: agglomeration along wall normal element piles in structured layer

Stand agglomeration outside structured layer Input: # surface cells to be fused

# cells in wall normal direction to be fused (e.g.: 4:2, 2:2 or 2:4)

standard versionnew version

S. Reuss, P. Lettich


Folie 22

slide 22

Slide 22

Tau User Meeting, DLR Braunschweig, 18-19. 10. 2011 Vortrag > Autor > Dokumentname > Datum

High lift configuration

7.6 M points

Runge-Kutta

method

central

scheme, SAO CFL = 1,8

3:2 3V

3:2 4W

New version is more robust: 3V and 4W possible from scratch

Performance Improvement Improved Multigrid

Agglomeration



Folie 23

slide 23

Slide 23


Transition Prediction

Current investigation Investigation of most suitable coupling of main & turbulent equations on given level Investigation of alternative multigrid

concepts

Re-discretizationGalerkin

projection

Galerkin

projectionCoarse grid operator: Consistent treatment of boundary conditionsAllows larger multigrid

cycles

(5W, 10V, 10W, …)

hHh

HhH ILIL :

8W, point implicit, Galerkin

NACA 0012

M=0.8, AoA=1.25

Grid: 320x64

5W, point implicit, re-discretization

5W, point implicit, Galerkin

NACA 0012

M=0.8, AoA=1.25

Grid: 640x128

Performance Improvement Alternative multigrid

concepts

S. Langer


Folie 24

slide 24

Slide 24


Parallelization of the TAU Code Domain Decomposition

Parallel PerformanceQuality of domain decomposition (“partitioning”) significantly affects performance w.r.t. simulations timesParticularly important for large #domains

some domains in a graph-partitioned grid

Partitioning in TAUWell known: TAU’s

“private”

partitioning

(based on recursive coordinate bisection, RCB)NEW: Graph-based partitioning using “Zoltan”

(Sandia National Labs library)NEW: Integrated “MPI rank optimization”

to reduce traffic

over the cluster interconnect

Wal

lclo

ckSe

cond

s/25

0 Ie

ratio

ns

P. Lettich, B. Brandfass, J. Jägersküpper


Folie 25

slide 25

Slide 25


Paralleliztion

of the TAU Code High-Performance Computing CapabilitiesDevelopment of computing hardware has changed dramatically: the “many-core shift”

a single compute core no longer gets fasterenergy considerations very importantgain in compute power increased core countmulti-core chips common today……many-core chips will be standard soon

Implications for the TAU codeDomain decomposition with one

domain per core

no longer adequateScalability flattens off as domains become

too small (communication overhead etc.)Plan: 2-level (“hybrid”) parallelization with

multi-threaded processing of each domainone domain per multi-core chip

enhances

TAU’s

scalability significantlymore cores can be used more effectively

80-core chip picture: Intel

2-socket 4-core x86-system

J. Jägersküpper, Th. Gerhold


Folie 26

slide 26

Slide 26


Grid adaptation: Preservation of Grid Structures (Hyperflex)

Strategies to improve adapted grids with structured regions (trade-off between local adaptation and preserving structures)

Methods to preserve grid structures after local adaptation (e.g. avoid single/few refined elements in unref. areas and vice versa)

results in smoothing the refinement depth.Implementation of strict preservation of defined structured grid

blocks

in several steps:

1st: fading out (defined) structured blocks/area from refinement, 2nd: refinement of entire structured blocks depending on indicators,

3rd: directional refinement of entire blocks depending on indicators.

M. Orlt


Folie 28

slide 28

Slide 28


Set-up of reliable reference cases by repeated global grid refinement including wall-normal subdivision, surface interpolation and

smoothing grid lines and spacing (planned)Evaluation of adjoint-based error indication in comparison with other methods regarding resulting accuracy and computational effort.First results showed, that wall-normal cell subdivision

in semi-structured layers on viscous walls can take large effects

Planned feature:

refine semi-structured elements in wall-normal direction in dependence of layer height and wall-tangential refinement level

Grid adaptation: Investigation on Adjoint-based Error Indicators



Folie 29

slide 29

Slide 29


Eddy Viscosity Models (EVM): No new model planned

Work on possible model improvements may result in model updates

Differential Reynolds-Stress-Models (DRSM):Further validation/industrialization of implemented DRSM

The Jakirlic-Hanjalic

εh

-RSM (see talk of A. Probst) will become available

A ωh

-flavour of the same model may become available in addition (depending on experience with this formulations it might later replace the εh

-Version)

Turbulence Modelling New Developments in 2012

RANS models


Folie 30

slide 30

Slide 30


Mainly work on assessment of available modelsAlthough SA and

SST

versions of DES, DDES

and

IDDES

are available

assessment has mainly been done for the SA variants. SST to follow

An RSM-based DES variant is also under assessment (see talk of A. Probst)

SAS

has only been tested for the kω-variant in THETA and for a very small number of test cases, further assessment of SAS/PANS

scheduled for 2012

Recommendations for the use of hybrid models will be issued in late 2012 based also on final results of the ATAAC project

For flows with massive separation the model is less critical than a good gridFor flows with “shallow”

separation artificial turbulence excitation is likely to

be necessary to improve the results. Best approach has yet to be found, not likely before end of 2012

Turbulence Modelling New Developments in 2012

Hybrid RANS-LES models


Folie 31

slide 31

Slide 31


ProblemRestart of transition locations via transition update information in parameter file

Concept for improvementRead-in of transition locations and laminar zones from solution fileExtension of restart file by wall distance arrayAt restart, read-in of wall distance and its algebraic signPoints with negative sign are laminar points.

Status-version of TAU including the feature exists and is going to be tested

OutlookPlanned for Release 2012.1.0

Transition Prediction Enhancement of Restart Functionality

N. Krimmelbein


Folie 32

slide 32

Slide 32


Transition Prediction Extension to Flexible ConfigurationsProblem

Streamline/cut coordinates on deformed body

ConceptStore ID of nearest

surface point IDTransfer deformation to

streamline/cut coord.

N. Krimmelbein


Folie 33

slide 33

Slide 33


a) Undeformed

grid

(initial

position)

Test: Prolate

spheroid

Transition Prediction Extension to Flexible Configurations

N. Krimmelbein


Folie 34

slide 34

Slide 34


c) Deformed

and rotated

Test: Prolate

spheroid


N. Krimmelbein


Folie 35

slide 35

Slide 35


b) Deformed

Test: Prolate

spheroid


N. Krimmelbein


Folie 36

slide 36

Slide 36


d) Deformed, rotated

and translated

StatusFirst tests successful

OutlookDemonstration for flexible wing-body configuration in 2011Planned for Release 2012.1.0

Test: Prolate

spheroid


N. Krimmelbein


Folie 37

slide 37

Slide 37


Transition Prediction Correlation-based -Reθt

Transition

Transport Model (Menter/Langtry)

Transition prediction based on two transport equationsIntermittency γ

: Probability that flow is turbulent at a certain location

γ

= 0: flow is laminar, γ

= 1: flow is turbulent

The intermittency is used to trigger transition onset

Reynolds number based on momentum thickness at transition onset Reθt

Transport of information about transition onset from outside to inside the boundary layer

Coupling to the Menter-SST k-ω

turbulence model

C. Seyfert


Folie 38

slide 38

Slide 38


Transition prediction based on transport equationsnon-restrictive parallelizationno search-algorithmsno integration along lines

Well compatible with modern CFD-codesunknown applicationsunstructured meshescomplex geometrie

Transition prediction for Natural transition (Tollmien-Schlichting)Separation-induced transitionBy-pass transition

Available TAU release 2012.1.0Extension to cross flow transition under way

Transition locations for NLF(1)-0416 airfoil

Transition Prediction Correlation-based -Reθt

Transition

Transport Model

C. Seyfert


Folie 39

slide 39

Slide 39


TAU Developments 2012 Improved Robustness & Handling of Chimera Technique

Deformation of hole-definition geometries in case of large deflections Improved handling of orphaned points (nearest neighbor interpolation)Additional output to analyze failure casesCapability to test hole cutting before simulation starts for prescribed trajectories

Undeformed

wing, undeformed

hole definition geometry


Folie 40

slide 40

Slide 40



Deformed wing, undeformed

hole definition geometry (hole cutting procedure will fail)



Folie 41

slide 41

Slide 41



Deformed wing, deformed hole definition geometry (hole cutting procedure works)



Folie 42

slide 42

Slide 42


TAU, Cartesian Solver, Domain couplingObjective: accurate simulation of interaction of aerospace configurations with flow structures convected

over large(r) distances, e.g. gusts, vortex wakes.

Division of the fluid domain: Nearby (TAU) < domaincoupling

(Chimera, code2code) > farfield

(Cartesian Solver)Cartesian Solver: higher order spatial resolution through Compact Finite Difference (PADE) schemes. (implicit)

1st version available, further development in 2012P. Kelleners


Folie 45

slide 45

Slide 45


TAU / FlowSimulator Trim of Aircraft

yg

yb

zgzb

FT,g

g

R

RV

mF ggT

2

,

Extension of existing trim-module Improved robustness & efficiency (surrogate-based strategy)Allows more complex trim-scenarios like

Engine failureCurve flightDamaged rudder, …

Integration in FlowSimulatorDemonstration for Airbus like configurations

Newton iterations

Ang

les

[°]

Thru

st[N

]

0 1 2 3 4 5 6

-4

-2

0

2

4

6

14820

14880

14940

15000

phithetapsialphabetaHTPAIL_LAIL_RThrust

horizontal flight with damaged rudder (fixed at 80)

6 control parameters

A. Michler, R. Heinrich


Folie 46

slide 46

Slide 46


TAU Developments 2012 Fast response CFD

To allow a more efficient simulation of attached flows, TAU will be coupled to the boundary layer code CALISTO (Airbus)Coupling strategy based on transpiration velocity BC approach 1st prototype for Euler-boundary layer coupling will be delivered in 2012

x

cp0

0 0.25 0.5 0.75 1

-1

-0.75

-0.5

-0.25

0

0.25

0.5

0.75

1

Slc: Z=0.65eta-0.650000650upcp0exp650lowcp0exp

EE

uulleerr-BL

NS

6.0103.7Re

82.06

Ma

LANN-wing, coupling TAU with BL-Code of DLR-AE


Folie 47

slide 47

Slide 47


Status Discrete formulation for Euler & RANS equations RANS with frozen and variable viscosity Differentiation of selected models, algorithms and boundary conditions by handParallel implementation

Available solution methodsPreconditioned GMRES (PETSc)Linear multigrid

with LU-SGS (FACEMAT)

Current

& future

workAlgebraic multigrid

with defect correction (current work, best practice)

Linear geometrical multigrid

with matix-based improved preconditionersLinearization of dedicated boundary conditions, discretzation

schemes, alternative turbulence models, ….

Linear and Adjoint

TAU

target: comparable convergence• achievable for simple test cases• challenge for complex problems

ideal case


Folie 48

slide 48

Slide 48


Linear Problems in TAU Algebraic Multigrid

Solver

Cooperation with Fraunhofer

SCAI InstituteLinear systems are solved with algebraic multilevel methodExtension of SCAI SAMG library, integrated in TAU Investigation of various solution strategies Best practice guideline underway

ExampleLANN wing, adjoint

problemGrid: 5.1 million points AoA=0.59, M=0.822, Re=5.43*106

SA turbulence modelSolver: AMG + GMRes(100)Runtime: 32min on 32 processors for

6 order residual reduction No convergence with linear geometric multigrid

(FACEMAT)

1 Iteration

= 1 iteration

of GMRes

+ 1 iteration

of AMGAMG smoother: def. corr. on finest

level, 2xILU(0) on coarse

levels

A. Naumovich

the dlr tau-code - recent and future...

Documents