turbulence simulations: multiscale modeling and data-intensive … · abstract in this two part...

Turbulence simulations: multiscale modeling and

data-intensive computing methodologies

by

Jason Graham

A dissertation submitted to The Johns Hopkins University in conformity with the

requirements for the degree of Doctor of Philosophy.

Baltimore, Maryland

January, 2014

c© Jason Graham 2014

All rights reserved

Abstract

In this two part work, methodologies for the multiscale modeling of complex tur-

bulent flows and data-intensive computing strategies for large-scale turbulent simu-

lations are developed and presented. The first part of this thesis is devoted to the

simulation of turbulent flows over objects characterized by hierarchies of length-scale.

Flows of this type present special challenges associated with the cost of resolving

small-scale geometric elements. During large eddy simulation (LES), their effects

on the resolved scales must be captured realistically through subgrid-scale models.

Prior work performed by Chester et al. [21] proposed a technique called renormal-

ized numerical simulation (RNS), which is applicable to objects that display scale-

invariant geometric (fractal) properties. The idea of RNS is similar to that of the

dynamic model used in LES to determine model parameters for the subgrid-stress

tensor model in the bulk of the flow. In RNS, drag forces from the resolved elements

that are obtained during the simulation are re-scaled appropriately by determining

drag coefficients that are then applied to specify the drag forces associated with the

subgrid-scale elements. In the current work we introduce a generalized framework for

ii

ABSTRACT

describing and implementing the RNS methodology thereby extending the method-

ology first presented by Chester et al. [21]. Furthermore, we present various other

possible practical implementations of RNS that differ on important, technical aspects

related to 1) time averaging, 2) spatial localization, and 3) numerical representation

of the drag forces. The new RNS framework is then applied to fractal tree canopies

consisting of fractal-like trees with both planar cross-section and three dimensional

orientations. The results indicate that the propsed time averaged, local, and explicit

formulation of RNS is superior to the predecessor formulation as it enables the mod-

eling of spatially non-homogenous geometries without using a low-level branch based

description and preserves the assumed dynamic similary through temporal filtering.

In addition, the overall predicted drag force of the non-planar fractal trees is shown to

agree well with experimental data. In addition to RNS, a methodology for generating

accurate inflow conditions in multiscale turbulence simulations is present. This tech-

nique called concurrent precursor simulation (CPS) allows the synchronous generation

of inflow data from an upstream precursor simulation. This approach conceptually

is the same as the standard precursor simulations (Lund et al. [72] and Ferrante and

Elghobashi [35]) used in the past, however, it eliminates the I/O bottleneck of disk

reads and writes by transferring sampled data directly between domains using MPI.

Furthermore, issues with recycling time scales of the sample inflow library are removed

since the upstream, precursor simulation is performed concurrently with the target

simulation. This methodology is applied to a single fractal tree (modeled using RNS)

iii

ABSTRACT

in turbulent duct flow and to a finite length, developing wind farm. In the second

part of this work, data-intensive computing strategies addressing the large-scale data

problem in direct numerical simulation (DNS) of turbulent flows are presented. DNS

provides the highest fidelity of predicited turbulence data. As a result, these data have

served a vital in role in turbulence research and access to such data is key to continued

development of the field. Classical approaches to the management and dissemination

of these large-scale datasets, however, has proven to be cumbersome and prohibitively

expensive in some instances thus minimizing the usefulness of these data to a broad

community. Therefore, the Johns Hopkins Turbulent Databases (JHTDB) (Perlman

et al. [89] and Li et al. [68]) have been created which expose large-scale turbulence

datasets to the reasearch community worldwide using Web services. The JHTDB

project provides Web service libraries for C, Fortran, and Matlab which allow inter-

action with the DNS data. The design and implementation of the Matlab interface

along with several examples are presented. Also, the first Web service based, publicly

available channel flow DNS database is produced in this work. The implementation

of the channel flow DNS and construction of the subsequent database are presented.

These data are then used to study the structure and organization of channel flow

turbulence. In this study, the Q criterion [50] is employed to measure the vortex sizes

and organization. Results appear to indicate good, qualitative agreement with the-

oretical predictions with respect to the prescence of large-scale near wall structures

and the preponderance of buffer layer vortices.

iv

ABSTRACT

Primary Reader: Professor Charles Meneveau

Secondary Readers: Professors Gregory L. Eyink and Randal Burns

v

Acknowledgments

I first and foremost thank my Lord and Savior Jesus Christ, from whom all bless-

ings proceed, for the opportunity to pursue the endevours in this work. Without

His guidance and care it would not have been possible to complete this journey. An

immense thanks goes to my advisor Professor Charles Meneveau for his patience,

kindness, and encouragement throughout my doctoral studies. His direction and

thoughtful conversations are also greatly acknowledged. I thank Professors G. L.

Eyink and R. Burns for their many kind and insightful suggestions made throughout

this work, for their stimulating lectures, and fruitful collaborations. To Professors A.

Prosperetti, O. Knio, J. Katz, and R. Mittal, I am very greatful for the exceptional

courses which they taught and for their endless pursuit of academic excellence.

A great thanks goes to Dr. Edward Givelberg, Kalin Kanov, and the entire

JHTDB team for exciting and rewarding collaborations. For providing access and

support to the PoongBack code along with fruitful collaborations, a cordial thanks

belongs to Professor Robert Moser, Myoungkyu Lee and Nicholas Malaya of the

University of Texas. To my collegues Claire VerHulst, Adrien Thormann, and Dr.

vi

ACKNOWLEDGMENTS

Kunlun Bai, I am very thankful for fun and thought provoking conversations which

have certainly added to the richness of this work.

For financial support, I am indebted to the JHU IGERT program on “Modeling

Complex Systems” (NSF grant #0801471) and the NSF grant #CMMI-094153 for

supporting this effort.

And finally, to my wife Cindy, for her unwaivering support and endless patience–

without which this work would not have been possible–I am deeply grateful.

vii

Dedication

To my mom, though departed too soon, your inspiration and love live on in this

work. And to Noah, who reminds me each day what it means to live.

viii

Contents

Abstract ii

Acknowledgments vi

List of Tables xiv

List of Figures xv

1 Introduction 1

1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Background and Motivation . . . . . . . . . . . . . . . . . . . . . . . 5

1.2.1 Multiscale Modeling . . . . . . . . . . . . . . . . . . . . . . . 5

1.2.2 Data-intensive Computing . . . . . . . . . . . . . . . . . . . . 12

1.3 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

I Multiscale Modeling 23

2 Renormalized Numerical Simulation 24

ix

CONTENTS

2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

2.2 RNS Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.3 LES Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2.4 Planar tree canopy test case . . . . . . . . . . . . . . . . . . . . . . . 32

2.5 RNS formulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

2.5.1 Model M1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

2.5.2 Model M2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

2.5.3 Model M3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

2.5.4 Model M4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

2.6 Test results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

2.6.1 RNS quantities . . . . . . . . . . . . . . . . . . . . . . . . . . 50

2.6.2 Selected flow statistics . . . . . . . . . . . . . . . . . . . . . . 55

2.6.3 Temporal averaging time-scale . . . . . . . . . . . . . . . . . . 58

2.6.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

2.6.5 Grid and RNS Modeling Sensitivity . . . . . . . . . . . . . . . 60

2.7 Applications to canopy consisting of fractal trees with three non co-

planar branches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

2.7.1 Flow Field and RNS Results . . . . . . . . . . . . . . . . . . . 66

2.7.2 Comparison with Experimentally Determined Drag Coefficient 75

2.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

3 Concurrent Precursor Simulation 80

x

CONTENTS

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

3.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

3.3 Application Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

3.3.1 Single fractal tree in turbulent duct flow . . . . . . . . . . . . 86

3.3.2 Finite length wind farm . . . . . . . . . . . . . . . . . . . . . 89

3.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

II Data-Intensive Computing 94

4 Johns Hopkins Turbulence Databases 95

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

4.2 Design and Construction . . . . . . . . . . . . . . . . . . . . . . . . . 96

4.2.1 Database Cluster . . . . . . . . . . . . . . . . . . . . . . . . . 99

4.2.2 Web Services . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

4.3 Channel Flow Database Interpolation and Differentiation Methods . . 102

4.3.1 Spatial Interpolation . . . . . . . . . . . . . . . . . . . . . . . 103

4.3.2 Spatial Differentiation . . . . . . . . . . . . . . . . . . . . . . 108

4.4 Matlab Client Interface . . . . . . . . . . . . . . . . . . . . . . . . . . 110

4.4.1 Design and Implementation . . . . . . . . . . . . . . . . . . . 111

4.4.2 Code Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 113

4.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

xi

CONTENTS

5 Channel Flow DNS 118

5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

5.2 Governing Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

5.3 Production Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . 121

5.4 Vortex Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

5.4.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 130

5.4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

5.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

6 Concluding Remarks 144

Appendix A LESGO Validation: Flow over wall mounted cubes 148

Appendix B MPI-DB 152

B.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

B.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154

B.3 Channel Flow Simulation . . . . . . . . . . . . . . . . . . . . . . . . . 155

B.4 The MPI-DB software library . . . . . . . . . . . . . . . . . . . . . . 157

B.5 MPI-DB Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . 158

B.6 Fortran Interface Design . . . . . . . . . . . . . . . . . . . . . . . . . 160

B.7 Fortran Interface Example . . . . . . . . . . . . . . . . . . . . . . . . 161

B.8 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

B.9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

xii

CONTENTS

Appendix C B-Spline Collocation Method 172

Appendix D Channel Flow DNS: Pressure Solver 175

D.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

D.2 Pressure Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176

D.2.1 Non-Zero Wavemode Solution . . . . . . . . . . . . . . . . . . 176

D.2.2 Zero Wavemode Solution . . . . . . . . . . . . . . . . . . . . . 177

D.3 Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

D.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183

Vita 200

xiii

List of Tables

2.1 Definitions of the tested RNS models. . . . . . . . . . . . . . . . . . . 392.2 Time averaged drag coefficient, RNS error, and forces for each of

the RNS models when applied to simulation of the “V-tree” canopy.Results shown pertain to the sample tree in the middle of the domain.Along with time, all of the quanties are also averaged across both belements of the tree. ‖ eb ‖ is defined as |eb|/|Fb|; |FR| the totalresolved force on the target tree; |FS| the total subgrid force on thetarget tree; |FT | the force on the target tree. . . . . . . . . . . . . . . 59

2.3 Definitions of the case configurations used in the grid and RNS mod-eling analysis. Listed are the case names, grid resolution, number ofresolved generations (Ng), number of grid points across the diameterof the branches in the last resolved generation (Np), and the canopyaveraged total drag forces. . . . . . . . . . . . . . . . . . . . . . . . . 62

4.1 Table of Web service functions for each JHTDB database. . . . . . . 101

B.1 Summary of grid resolution and Re used for the channel flow simulations.156B.2 Simulation test results for various grid sizes . . . . . . . . . . . . . . 167B.3 Simulation test results for the 512×256×256 grid for variable number

of processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167

xiv

List of Figures

1.1 Example of a fractal tree (“3D-V fractal tree”), in which smaller branchesoccur at increasing elevations as an idealization of multiple-scale veg-etation element interacting with a turbulent boundary layer. . . . . . 7

2.1 Decomposition of fractal geometry into resolved and subgrid compo-nents, as well as between r, β and b elements. Numerically, in thepresent work, the resolved portions are treated using the immersedboundary method (IBM), while sub grid portions are accounted forusing RNS. The distribution of forces within the sub grid portion usesa filtered indicator function χ̃β (see text). . . . . . . . . . . . . . . . . 26

2.2 Fractal tree canopy composed of planar V-trees. Shown are the two“resolved” generations g0 and g1. The solid black box indicates thephysical domain for the simulations. The entirety of the tree as repre-sented in the simulation is shown in Figure 2.1. . . . . . . . . . . . . 33

2.3 Definitions of RNS references regions used in the “V-tree” canopy sim-ulations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

2.4 Contour plot of instantaneous velocity magnitude along a constant yand constant x plane. . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

2.5 Instantaneous velocity magnitude contours on horizontal planes at themid-plane heights of branch generations g2, g3 and g4. . . . . . . . . . 38

2.6 Contour plot of instantaneous local force magnitude along a constant-xplane across the middle tree in the domain. . . . . . . . . . . . . . . . 38

2.7 Time series of hydrodynamic forces in the x-direction acting of thetarget tree for RNS model M2. . . . . . . . . . . . . . . . . . . . . . . 51

2.8 Time series of hydrodynamic forces acting on one of the branches (el-ement b1) of the target tree for model M2. The left axis shows thex-direction forces, while the right axis (lower group of lines) shows they-spanwise direction forces. . . . . . . . . . . . . . . . . . . . . . . . . 52

2.9 Time series of reference velocities measured from element b1 and itsdescendant β elements in the target tree, for model M2. . . . . . . . 52

xv

LIST OF FIGURES

2.10 Time history of drag coefficient obtained from RNS using models fromTable 2.1. For models M1 – M3 the single, global drag coefficientsare presented. In the bottom plot, the time-series for M4 are showncorresponding to each branch of the “V-tree”, denoted as elements b1and b2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

2.11 Mean values (vertical bars) and standard deviations (error bars) of thecomputed RNS drag coefficients. . . . . . . . . . . . . . . . . . . . . . 56

2.12 Horizontally averaged (a) mean velocity and (b) turbulent shear stressprofiles evaluated from RNS of flow over the “V-tree” canopy, for dif-ferent RNS formulations M1-M4. . . . . . . . . . . . . . . . . . . . . 57

2.13 Time history of the RNS drag coefficient obtained for several temporalaveraging time-constants. . . . . . . . . . . . . . . . . . . . . . . . . . 58

2.14 Mean values of the measured drag forces decomposed into the resolved,subgrid, and total contributions. The error bars indicate the estimatedstandard error of the mean total drag force due to statistical convergence. 63

2.15 Fractal tree canopy composed of “3D V-trees”. The solid black boxindicates the computational domain used in the simulations. . . . . . 65

2.16 Reference regions b and β used when applying RNS to the “3D V-tree”canopy simulations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

2.17 Contours of instantaneous velocity magnitude from RNS of flow overa fractal “3D V-tree”. Shown are contours along three vertical planes,one at constant y and two at constant x. . . . . . . . . . . . . . . . . 68

2.18 Contours of instantaneous velocity magnitude on the branch mid-planesof generations g2, g3 and g4 in the region of unresolved branches of thefractal “3D V-tree”. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

2.19 Instantaneous force magnitude contours along the mid-planes of gen-erations g2, g3 and g4 (same as in Figure 2.18). . . . . . . . . . . . . . 69

2.20 (a) Mean streamwise velocity profile, averaged in time and horizontaldirections, for simulation of boundary layer flow over a canopy of “3DV-trees” using RNS. A side view of such a tree is also shown in the pro-file as a reference. (b) Shear stress profiles, including mean turbulenceshear stresses as well asd ispersive and total stress. . . . . . . . . . . 70

2.21 Time series of canopy averaged drag forces. Solid line: force on theresolved branches, dashed line: subgrid-scale force, and small-dashedline: total force. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

2.22 Vertical profiles of (a) the mean velocity, (b) frontal area density (perunit height) and (c) resulting mean drag force. . . . . . . . . . . . . . 72

2.23 Time series of computed forces on the three b elements of the sampletree, during representative time period. . . . . . . . . . . . . . . . . . 73

2.24 Mean values (vertical bars) and plus minus one standard deviation(error bars) of the computed RNS drag coefficients for each of thethree branches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

xvi

LIST OF FIGURES

2.25 Total drag coefficient computed for an entire tree. The solid line (com-puted from RNS) is based on the total force exerted on the trees aver-aged across all trees (canopy averaged). The horizontal (dashed) lineis the mean value obtained in a laboratory experiment [43]. . . . . . . 75

3.1 An example CPS of turbulent boundary layer flow over a wall mountedcube. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

3.2 Schematic showing the domains containing precursor simulation (“red”)and the target simulations (“blue”). The intra-domain decompositionand respective MPI ranks are shown next to the domains. . . . . . . . 83

3.3 Schematic showing the splitting of the default MPI COMM WORLD com-municator into two communicators associated with the red and bluedomains for the CPS implementation. The bridge communicator usedduring the sampling operation is also shown. . . . . . . . . . . . . . . 84

3.4 Domain setup for the CPS of single fractal tree in turbulent duct flow.Note that the third branch for each branch cluster in the second gen-eration are aligned along the line-of-sight direction of the viewing angle. 88

3.5 Instantenous streamwise velocity along a y-plane for the precursor (red)domain in the CPS of a single fractal tree in turbulent duct flow. . . . 89

3.6 Instantenous streamwise velocity along three z-planes for the target(blue) domain in the CPS of a single fractal tree in turbulent duct flow. 90

3.7 Instantaneous streamwise velocity in the upstream and downstreamdomains during the CPS of the developing, finite length wind farm.Figure prepared by Dr. Richard Stevens and reused from Stevens et al.[100] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

3.8 Power output comparsion of field data from Horns Rev wind farm andCPS of two grid resolutions. Figure prepared by Dr. Richard Stevensand reused from Stevens et al. [100] . . . . . . . . . . . . . . . . . . . 92

4.1 Schematic of the JHTDB indicating the logical layout of the remoteclients, Web server, and the database cluster. Source JHTDB [55]. . . 98

4.2 Visualizations of the forced isotropic turbulence database using theMatlab client interface. . . . . . . . . . . . . . . . . . . . . . . . . . . 114

5.1 Friction velocity Reynolds number during the channel flow simulationduring the database time interval . . . . . . . . . . . . . . . . . . . . 125

5.2 Mean velocity profile in viscous units. Standard values of κ = 0.41 andB = 5.2 are used in the log-law (dashed line) for reference. . . . . . . 126

5.3 Profiles of statitical quantities from the channel flow DNS. . . . . . . 1275.4 Streamwise power spectral densities at various y+ locations as function

of kx . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1285.5 Spanwise power spectral densities at various y+ locations as function

of kz . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

xvii

LIST OF FIGURES

5.6 Q isosurfaces in a sub-region of the channel flow domain for threethreshold values: Q=4, Q=16, Q=64. . . . . . . . . . . . . . . . . . . 135

5.7 The joint PDFs of the normalized vortex volume with respect to thelog-layer eddy scale and center of mass location for various Q thresh-olds: a) 0.5, b) 1.0, c) 2.0, d) 4.0, e) 8.0, f) 16.0, g) 32.0, h) 64.0. . . 137

5.8 Marginal PDFs for the normalized vortex volume with respect to thelog-layer eddy scale and center of mass. . . . . . . . . . . . . . . . . . 139

5.9 Marginal PDF for the normalized vortex volume with respect to thelog-layer Kolmogorov scale. . . . . . . . . . . . . . . . . . . . . . . . 140

5.10 Joint PDFs for the surrogate vortex ellipsoid volume to the vortexvolume ratio and center of mass location for various Q thresholds: a)0.5, b) 1.0, c) 2.0, d) 4.0, e) 8.0, f) 16.0, g) 32.0, h) 64.0. . . . . . . . 141

5.11 Marginal PDF of the surrogate vortex ellipsoid volume to the vortexvolume ratio. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

A.1 Domain setup, contours of instantaneous x component velocity, andtime averaged streamlines for the wall mounted cubes test-case (stream-lines originating near x/h=4 have been seeded at that location). . . . 149

A.2 Mean velocity profiles for the wall mounted cubes case for: (a) x com-ponent of velocity at y = 0, (b) x component of velocity at z = 0.5h,and (c) y component of velocity at z = 0.5h. In each figure, the hori-zontal arrow denotes the x component of the measured reference velocity.151

B.1 Instantenous streamwise velocity (vertical contour planes) and vortic-ity fields (iso-surfaces) for case C3 from Table B.1. The iso-surfacesare colored according to the vertical height. . . . . . . . . . . . . . . 157

B.2 Throughput of the data ingestion as a function of the grid size. . . . . 168B.3 Throughput of the data ingestion for grid size 512 × 256 × 256 as a

function of the number of processes used in the simulation. . . . . . . 168

D.1 Comparison between the numerical (lines) and analytical solutions(symbols) of (D.8) with Dirichlet boundary conditions and two val-ues of k: a) k = 1.1180 and b) k = 18.901. . . . . . . . . . . . . . . . 181

D.2 Comparison between the numerical (lines) and analytical solutions(symbols) of (D.8) with Neumann boundary conditions and two valuesof k: a) k = 1.1180 and b) k = 18.901. . . . . . . . . . . . . . . . . . 181

D.3 PDF of the PPE Dirichlet boundary condition residual with the twonormalization types . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183

xviii

Chapter 1

Introduction

1.1 Overview

Accurate simultions of turbulent flows pose many challenges due to turbulence’s

inherent complexity and large number of degrees of freedom. Turbulence paradox-

ically possesses both chaotic and well ordered characteristics and remains a major

unsolved problem in physics even though the governing equations–the Navier-Stokes

(N-S) equations–have been known since the 1800’s. The N-S equations for an incom-

pressible fluid are expressed as

∂u

∂t+∇ · (u⊗ u) = −∇p+ ν∇2u+ f , ∇ · u = 0 (1.1)

1

CHAPTER 1. INTRODUCTION

where u is the velocity vector field, p the kinematic pressure, ν the molecular viscosity,

and f a body force. Except for several idealized flows, currently no known analytical

solutions exist thereby leaving only numerical solutions of the N-S equations. This is

especially true for turbulence.

The numerical solutions to the N-S equations may be divided into two camps:

1) direct numerical simulations (DNS) where the N-S equations are solved directly

and all turbulence scales are resolved or 2) a modeled approach where turbulence

models are applied to capture unresolved turbulence scales. These two approaches

are discussed further below.

DNS is a powerful tool for studying turbulent phenomena. In a DNS, all of

the turbulent scales of motion are accurately resolved and no turbulent models are

employed. As a result, data generated from a DNS are high-quality and provide

researchers access to complete information with regard to the turbulent field [81]. The

ability to compute all of the relavent degrees of freedom of a turbulent field, however,

comes at a cost. DNS is both computationally expensive and typically produces very

large data sets that must be properly managed in order to make practical use of

the resulting data. This inherent expense is due to the requirement of resolving the

dissipative scales (and near wall viscous scales in wall bounded flows) which decrease

as the Re is increased, thus requiring finer computational meshes. Furthermore, the

finer meshes restrict the allowable size of the computational time step in the numerical

integration of the governing equations due to numerical stability and accuracy issues.

2


In isotropic turbulence, for example, the resulting overall complexity of a DNS scales

as Re11/4 (where Re = UL/ν is the Reynolds number, U a characteristic turbulence

velocity scale and L a large-scale length of turbulence) leaving researchers the ability

to only perform DNSs for modest Re. Wall bounded flows are even more expensive

due to viscous wall interactions.

For high Re or complex flows where DNS can not be afforded turbulence modeling

is required. There are numerous approaches to turbulence modeling (see Pope [91]),

however, the two most commonly used methodologies are Reynolds Averaged Navier-

Stokes (RANS) and large eddy simulation (LES). In RANS, the N-S equations are

either time or ensemble averaged. This results in governing equations for the mean

fields where all of the effects of the turbulence on the mean field are modeled. In

LES, the N-S equations are spatially filtered, thereby, separating the turbulence into

resolved and unresolved (subgrid) scales. As a result, governing equations for the

resolved field are produced and the aggregate effects of the subgrid scale turbulence

are captured through models.

The LES filtering operation can be expressed as

f̃(x) =

∫G∆(x− x′)f(x′)dx′ (1.2)

where f is a fully resolved field of interest (such as velocity), f̃ is the filetered version

of this field, and G∆ the filter kernel at scale ∆ (commonly the grid scale). Applying

3


this filter to the N-S equations produces the filtered N-S equations which may be

expressed as

∂ũ

∂t+∇ · (ũ⊗ ũ) = −∇p̃+ ν∇2ũ−∇ · τ + f̃ , ∇ · ũ = 0 . (1.3)

In the filtered N-S equations an additional term τ–the subgrid-scale (SGS) stress

tensor–has been introduced. This quantity arises from filtering the non-linear terms

of the N-S equations and is expressed as τ = ũ⊗ u − ũ ⊗ ũ. The generation of

this additional term is a result of the closure problem [78]. A detailed review of SGS

closure models can be found in Meneveau and Katz [78]. In addition to modeling

subgrid turbulence scales, flows containing unresolved geometric features also reguire

modeling. These subgrid-scale geometric features may be represented in the resolved

field equations as a momentum sink imposed by the body force f̃ [19, 21, 20, 43, 42,

77].

The first of two parts of this thesis is devoted to the modeling of turbulent flow

interacting with multiscale objects which possess both resolved and unresolved ge-

ometric features. The methodology is developed and applied to study turbulence

generated by fractal trees using LES. In addition, the first part also describes a new

methodology for imposing accurate inflow conditions in periodic domains for turbu-

lent flows which are vital for correct numerical solutions. The second part of this

thesis is devoted to the development of data-intensive computing methodologies for

4


the large-scale data problem in DNS. These methodologies along with an application

to channel flow turbulence are presented and discussed. In the following sections the

background and motivation for both of these two parts are presented.

1.2 Background and Motivation

1.2.1 Multiscale Modeling

Fluid flows involving multiple-scale boundaries can be found in nature, such as

wind flow through tree canopies, over rough terrain and natural landscapes, and flow

through porous media. Important transport processes of mass, momentum and energy

can occur at the interface between fluid and such bounding surfaces. Simulation of

fluid flow in such conditions involves challenges due to the typically large ranges of

spatial and temporal scales that must be resolved. As a result, in most applications it

is required to employ subgrid-scale models to represent the small scale features while

resolving the large scale problem on a computational mesh. In the bulk of turbulent

flow, this is the Large Eddy Simulation (LES) approach. Along the boundaries,

additional modeling is required if the boundaries include large ranges of length-scales

with topological features that occur at subgrid-scales, when viewed at the resolution of

the LES. The present work is devoted to modeling turbulent flow over multiple-scale,

tree-like objects.

Fractals provide a useful idealization of multiple-scale objects since they may be

5


described using simple geometric rescaling rules [74, 6]. Basic fractal objects can

serve as surrogates for more random and complex multi-scale objects often found in

nature [85], while remaining tractable for systematic study. Fractals have been used

to model trees, see e.g. de Langre [26], and the fractal dimension of trees has been

found to be mostly between 1.45 and 1.74 [14].

A series of papers studying turbulence in the wake of fractal objects (Staicu et

al. [99], Hurst and Vassilicos [51], and Laizet and Vassilicos [64]) show that there

exists a strong coupling between the geometric (multi-scale) features of the fractal

object and the turbulence properties observed in the wake downstream. Therefore,

in order to accurately simulate momentum transport and turbulent flow dynamics

in the presence of fractal objects, it appears desirable to retain relevant informa-

tion about the multi-scale features of the fractal geometry, while remaining within

computationally feasible and affordable approaches.

Figure 1.2.1 shows an example of a fractal tree to be used in this study. It shares

with real vegetation elements the preponderance of small scales (smaller branches)

at increasing elevations, and thus idealizes a fractal vegetation element interacting

with a turbulent boundary layer. For these reasons this particular tree geometry has

also been studied in a laboratory experiment in which the total drag force on such a

fractal tree placed in a canopy has been measured [43] (a related study has measured

eddy-length scales in the wake of a single individual such tree [4]). We point out that

the simulated fractal object only represents certain features of real trees and, as stated

6


Figure 1.1: Example of a fractal tree (“3D-V fractal tree”), in which smaller branchesoccur at increasing elevations as an idealization of multiple-scale vegetation elementinteracting with a turbulent boundary layer.

already, must be regarded as a particular idealization. Unlike real trees, the simulated

objects are rigid, i.e. they do not ‘sway’ in the wind, they exhibit deterministic scale-

invariance, i.e. each generation is an exact geometric replica, there are no leaves that

would break scale-invariance at the smallest scale, etc. With these limitations in mind,

we proceed to focus on studying interactions between such objects and turbulent flow.

One of the most important aspects of the interactions between flow and such

objects is the associated momentum exchange, the drag force. Classical methods for

characterizing the momentum transport due to canopies have been been based on drag

models using a leaf area index (LAI) description [93, 97, 16, 37], log-law based models

that use a roughness length scale to parameterize the entire vegetation canopy [93,

1], and models that consider the canopy as a porous medium characterized by a

7


prescribed porosity [69]. In the application of these classical methods to the simulation

of fractal canopy flows, two notable deficiencies arise: 1) the characterization of the

multiple-scale geometrical features by a single length scale, or elevation-dependent

length scale (in the case of LAI), and 2) the requirement that model parameters must

be known a priori for specification in parameterizations of the drag forces.

Renormalized numerical simulation (RNS) was first introduced in Chester et al.

[21] as a technique for simulating flows over partially resolved multiple-scale (fractal)

objects in high-Reynolds number flows. The RNS methodology is a downscaling

strategy in which drag forces due to the large, resolved scales are obtained directly

from the explicit representation of the resolved objects in the computational domain

(e.g. an immersed boundary method - IBM) and the resolved-scale information is

renormalized appropriately to predict the drag forces due to the small, unresolved

scales. During the renormalization, geometric and dynamic similarity is invoked and

the resulting small-scale information is “repeatedly fed back into the simulation of

the large-scale problem” [19]. This recursive, iterative procedure is carried out for

the duration of the simulation in order to capture the effects of both the resolved

and unresolved geometry. For applications to high Reynolds number flow, the force

is modeled using a form drag representation for both the resolved and unresolved

forces. These modeled forces depend on a drag coefficient which is assumed to be

scale invariant due to geometric similarity and high Reynolds number flow. The drag

coefficient needs not to be specified a priori, since it is dynamically evaluated in a

8


manner analogous to the dynamic subgrid scale model for LES [39].

In the first applications of RNS (i.e. Chester et al. [21] and Chester and Mene-

veau [20]), a classic drag model was used that assumed that the instantaneous drag

coefficient for each of the small branches is the same as that at the large branches.

Note, however, that in a time-varying turbulent flow, it is more natural to expect only

the time-averaged drag coefficient to be the same at various scales, assuming geomet-

ric similarity of the (fractal) boundary and (complete) dynamical similarity at high

Reynolds numbers. There is little basis for the stronger assumption of instantaneous

similarity. Therefore, it makes sense to develop and test a RNS method that involves

time averaging so it can be based on the weaker assumption of similarity between

scales in an averaged sense. Thus, as the main purpose of this work, we develop and

apply a temporal averaging technique in order to help justify the similarity assump-

tion when using a classical drag model. In addition, we also consider effects of spatial

inhomogeneity where the methodology of Chester et al. [21] can only be applied to

homogenous canopies. In the current work, we introduce a local model that may

be used to treat heterogeneous tree canopies and complex flow structures interacting

with irregular trees. In Chester and Meneveau [20], the authors extended the RNS

methodology of Chester et al. [21] for the application of flow over non-planar fracatal

trees. In their approach, they used the cross-flow principle [65] to preserve kinematic

similary with branches having irregular orientation with respect to the flow direction.

The approach decomposes the drag forces into normal and axial forces with respect to

9


the tree branches. The primary drawback of this approach arises from the complexity

of the implementation which requires a “low-level”, branch based description of the

imposed drag forces. In this work, we present an generalized RNS framework that

describes the imposed drag forces in terms of branch groups. These groups, which

we call RNS elements, may be constructed in such a way that kinematic similary is

implicitly preserved for non-planar geometry. Moreover, this generalized framework

has the additional advantage that it may readily be applied to non-tree fractal geom-

etry such as fractal grids. As part of evaluating the RNS framework presented in this

work, we distinguish between explicit and implicit time formulations of RNS. Though

not explicitly presented, an implicit time formulation was mentioned in Chester et al.

[21] in which the authors observed stability issues with the implicit formulation which

were not found with the explicit formulation. In this work we test whether similar

stability limitations with the implicit formulation are observed in the current RNS

framework.

In addition to correct modeling strategies, inflow conditions are one of the key in-

gredients for producing accurate results from numerical simulations. Since analytical

realizations of turbulent fields are not known a priori, enforcing correct and accurate

inflow conditions for turbulence simulations poses a challenging problem. Techniques

such as generating synthetic fields from Fourier modes [24], reduced order methods

such as proper orthogonal decompostion [32, 56], or the superposition of synthetic

eddies [90] have been proposed. Another approach is the generation of inflow data

10


using precursor simulations [72, 35]. In this approach a precursor simulation is per-

formed in which an inflow data library is stored to disk. These data are then loaded

and imposed as inflow conditions during the target simulation. Depending on the

problem, one approach may prove more attractive than another. Techniques which

modulate inflow conditions or impose modeled turbulent structues will require time

for the input signature of the imposed inflow field to decay in order to minimize the

influences of the inflow technique on the simulation results. In the case of simula-

tions with periodic boundary conditions (e.g. atmosphere boundary layer or channel

flow), the precursor simulation method provides an idealized approach since the pre-

cursor simulation may be performed directly without any inflow modulation and the

turbulent field can evolve until it becomes fully developed.

In this work, a concurrent precursor simulation has been developed which allows

a precursor simulation to be peformed concurrently with the target turbulent simu-

lation. Conceptually, the concurrent precursor simulation (CPS) is equivalent to a

standard precursor simulation (SPS) mentioned previously. In both cases, a full tur-

bulent simulation is performed generating inflow data for a target simulation. These

data are then sampled from the precursor simulation (typically a subregion or plane)

and transfered to the target simulation as an inflow condition. The major differences

between the two arise in practice. The SPS must be peformed before the target sim-

ulation is conducted which extends the overall time of the simulation. Conversely,

the CPS performs the precursor and target simulations concurrently. Also the SPS

11


stores the sampled precursor data to disk. These data are then read from disk during

the target simulation. Since disk I/O bandwidth is significantly smaller than that of

modern network interconnects and random access memory (RAM), simulation time

for both the precursor and target simulations are hindered for the SPS when com-

pared to the CPS approach. For the CPS, sampled inflow data is transfered directly

from RAM of the precursor simulation to the target simulation using memory copies

thereby minimizing the overhead of the sampling operation. Moreover, since in the

CPS the simulations are performed concurrently, there is no need to “recycle” the

precursor data as may be required in the SPS once a target simulation extends beyond

the temporal extend of the stored inflow data library. This eliminates the introduction

of artificial “recycling” time scales into the target simulation.

1.2.2 Data-intensive Computing

The origins of DNS can be traced back to 1972 to the work of Orszag and Patterson

[87] in which a DNS of incompressible flow was performed on a 323 mesh simulating

isotropic, homogeneous turbulence. Though the simulation was performed only on a

small 323 mesh, it laid the foundations for the usage of spectral methods [81] crucial for

highly accurate simulations of turbulence. During the 1970’s and early 1980’s, most

DNS were limited to isotropic, homogenous turbulence or at most flows containing

one inhomogenous direction. Simulations of wall bounded flows were not possible at

the time due to the additional computational costs associated with resolving the near

12


wall viscous scales. It wasn’t until 1987, that the first DNS of plane channel flow

was conducted by Kim et al. [62] for a Reτ = 180 where Reτ is the Reynolds number

based on the friction velocity and half-channel height. In the decades following,

increasing computing power afforded researchers the possibility of simulating ever

increasing Re and domain sizes. One consequence of the increasing simulation sizes

are the growing data volumes produced by the DNS. For instance, in 2002, the worlds

largest DNS of isotropic turbulence was conducted by Yokokawa et al. [111]. The

simulation was performed on a computational mesh of 40963 producing an estimated

2 TB per snapshot of velocity and pressure fields. Recently work has begun by Yeung

and Sreenivasan [109] in which an isotropic DNS on a mesh of 81923 is performed.

Once complete, their simulation will break the long standing record of Yokokawa

et al. [111] thus producing an even large dataset of an estimated 16 TB per output

snapshot. With regard to wall bounded flows, the largest channel flow DNS was

recently completed at Reτ = 5200 producing almost 4 TB per snapshot of velocity

and pressure fields. This simulation is approximately 10 times larger than the previous

largest channel flow DNS of Hoyas and Jiménez [49] which generated 25 TB of raw

data.

As computing power continues to allow larger simulations to be performed, the

data generated from large-scale DNS will only continue to grow. This generation of

ever increasing turbulence datasets will continue to put a higher demand on the data

storage scheme used for DNS. A data storage approach that is not only sufficient in

13


storage capacity but also amenable to post-simulation analytics is important. Due to

the complexity of the turbulent fields, raw velocity and pressure data provide basic

information that must be processed when studying the structure, organization, and

topology of the underlying turbulence. Therefore, secondary analyses are required

where post-simulation analytics can be performed to study the resulting data. As a

result, efficient data layout and high I/O throughput are important qualities of the

data storage scheme. Furthermore, the ability to easily perform data manipulations

within the data server is important in order to remove low-level details of the data

access from the analysis application. Classical non-persistent, flat-file data storage

schemes or array-oriented storage schemes such as NetCDF and HDF5 do not eas-

ily address these needs. One promising approach, however, to this large-scale data

problem is to utilize database technology as the storage medium. An example of this

approach are the Johns Hopkins Turbulence Databases in which the spatio-temporal

data from three large-scale DNS of forced isotropic, magnetohydrodynamic, and chan-

nel flow turbulence are stored in a public database [89, 68]. For a more detailed survey

of data management systems see Appendix B.2.

The Johns Hopkins Turbulent Databases (JHTDB) provide public access to lage-

scale, turbulent DNS data [89, 68]. These data reside within distributed SQL databases

and are made available to any one in the world, using Web services over the internet.

Remote clients may easily access the data using client software which sends and re-

ceives data using the Simple Object Access Protocol (SOAP). In addition to primary

14


field variables (e.g. velocity, pressure, magnetic field, etc.), secondary calculations

may be perfomed in-situ within the database cluster to obtain spatial derivatives of the

primary fields, perform particle tracking, and other operations. The JHTDB project

provides client libraries for C, Fortran, and Matlab including example codes that may

be easily extended and adapted for personal research. In addition to the client library,

any programming language that provides SOAP functionality (e.g. Python) may be

used directly with the Web service interface. The JHTDB currently contains three

DNS turbulent datasets: forced isotropic, forced magnetohydrodynamic, and channel

flow turbulence. The databases contain 27TB, 56TB, and 48TB of data, respectively.

The forced isotropic turbulence database is the first database generated for the

JHTDB [89, 68]. This database has been used extensively throughout the reseach

community including those who are not members of the JHTDB project. Several

examples of these studies include works by Lüthi et al. [73], Gungor and Menon

[47], Holzner et al. [48], Wu and Chang [108], and Cardesa et al. [15]. The second

database added to the JHTDB is the forced MHD turbulence database and has been

used, for example, in the study of breakdown of flux freezing in MHD [33]. The latest

database constructed is the channel flow database which was made publicly available

recently [45]. The generation of these data along with their ingestion in the JHTDB

are discused in Chapter 5 of this work.

Direct numerical simulations of turbulent channel flow have played an important

role in the study of wall bounded turbulence. A brief history of such DNSs along

15


with sample cases where the DNS data are used are given below. As mentioned

previously, the first channel flow DNS was performed by Kim et al. [62]. In their work

they compare a large number of turbulence statistics to experimental data giving the

first glance of the applicability of DNS for wall bounded flows. Moreover, the wall

normal, velocity-vorticity formulation presented in their work has been a foundational

methodology where it has been used extensively in subsequent DNSs [84, 29, 30, 31,

66]. In a follow up paper, Kim [60] utilized the data from Kim et al. [62] and studied

pressure fluctuations in a channel. Following this [61] performed a DNS at Reτ = 395.

In this work the authors showed that velocity and vorticity spectra near the channel

centerline exhibits local isotropy as predicted by Kolmogorov (1941). These data were

then used by Blackburn et al. [10] who studied the topology in turbulent channel flow

using the invariants of the velocity gradient tensor. In a similar effort, Jeong et al. [54]

utilized the Reτ = 180 data from Kim et al. [62] in which near wall coherent structures

were studied using the eduction scheme from Jeong and Hussain [53]. The work of

Moser et al. [84] brought the first generally available datasets for Reτ = 180, 395, and

595. Their work indicated the absence of low-Re effects in the velocity fluctuations

for Reτ > 395. In del Álamo and Jiménez [28], extended domains compared to the

Moser et al. [84] cases were utilized for Reτ = 180 and 550 in order to study large

scale anisotropic turbulent structures. These large scales were not possible with the

smaller domains used in Moser et al. [84]. Following this, del Álamo and Jiménez [29]

studied the spectra of these large scales. Furthermore, statistical datasets were made

16


available for these simulations. Continuing the analysis of large scale structures, del

Álamo et al. [30] performed DNS for Reτ up to 1900. In that work large domains

were used to capture large scale energetic structures while shorter domains with more

refined grids were used to study overlap layer structures. The work of Hoyas and

Jiménez [49] produced a channel flow DNS for Reτ = 2003. In their analysis, velocity

scalings were compared to the data from del Álamo and Jiménez [29] and del Álamo

et al. [30]. Statistical datasets for these simulations were also made publicly available.

Data from Hoyas and Jiménez [49] at Reτ = 934 were compared against experimental

data by Monty and Chong [82] and found excellent agreement with the velocity

statistics and energy spectra. Panton [88] then used data produced by del Álamo

and Jiménez [29], del Álamo et al. [30], and Hoyas and Jiménez [49] to study Re

effects on vorticity fluctuations. This work demonstrated two inner and one outer

scaling for the mean square of vorticity fluctuations. Also in 2009, Klewicki et al.

[63] utilized data sets from Moser et al. [84], Kawamura et al. [59] (at Reτ = 636),

and Hoyas and Jiménez [49] to study the logarithmic behavior of velocity statistics.

These data were used to develop a theory which predicts the von Kármán constant.

A final example of DNS data usage is Gao et al. [38] in which data from del Álamo

et al. [30] and Moser et al. [84] is used to study statistical characteristics of vortex

cores in wall bounded flows.

Datasets mentioned in the previous paragraphs for Reτ = 180, 550, 934, 2003

are publicly available from the UPM Fluid Dynamics Group [105] and ICES [52].

17


In addition to these data, a dataset for Reτ = 5200 is also being generated [66].

Although, these data are publicly available, they are limited to statistical profiles and

a small number of coarsely spaced (in time) velocity fields for Reτ up to 934 [105].

The statistical profiles for instance are very useful for validation or comparing data

from differing sources and are easily downloaded. The fields, however, are challenging

to obtain. The recipient must ship hardware to the location of the data where the

fields may be copied and stored to disk. The hardware is then shipped back to

the receipient [52]. Once the data are received, the data must be transformed from

spectral space to physical space thus requiring machines with a sufficient amount of

memory, especially for the larger Reτ datasets, to perform the global transformations

[52]. The complex procedures involved for processing the data are also prone to user

and equipment error.

In order to allow easy access to turbulent channel flow data, a channel flow DNS of

Reτ = 1000 is performed in this work (Chapter 5) and the resulting velocity and pres-

sure fields are transferred and ingested into the JHTDB (data ingestion is performed

by Kalin Kanov). Using the publicly available Web services and client interfaces,

researchers may easily retrieve and interact with the data. The construction and im-

plementation of the channel flow database for the JHTDB is discussed in Chapter 4.

Also presented in Chapter 4 is the Matlab client interface developed by the author for

the JHTDB Web services. The Matlab client interface provides the ability to interact

with the database from within a Matlab session. This ability allows clients to uti-

18


lize Matlab intrisic functions such as plotting tools, fast Fourier transform functions,

eigenvalue/vector procedures, etc. The details of the implementation and several

examples are also discused in Chapter 4.

In Chapter 5, the channel flow DNS for the JHTDB is presented. Also included

in the chapter is a study of coherent vortical structures in wall bounded flows. In

the study, a vortex identification scheme (Q criterion) is used to identify vortical

structures within the channel flow data. A brief discussion on vortex identification

schemes is given in below.

The goal of vortex identification schemes is to correctly extract vortex regions

that contain sufficiently strong, well-organized rotation about a filament axis. In

general, using the vorticiy field directly is inadequate since in addition to containing

well defined vortical filaments with spiraling patterns, it also may contain dominant

background vorticity as in shear flows or other high vorticity regions such as near

walls in wall bounded flows which do not necessarily guarantee vortical motion. To

properly identify and track vortical structures, an accurate definition of a vortex core

and discrimination method are needed. Though there is currently no singly accepted

mathematical definition for a vortex core, several attributes that a vortex core should

contain are, however, generally sought; these include:

1. Dominant rate of rotation with respect to rate of strain

2. Local pressure minimum

19


3. Compact swirling motion about an axis

Numerous definitions have been presented in the past which attempt to satisfy one

or more of the attributes listed above. Several of these definitions based on the local

kinematics of the velocity gradient tensor are: 1) the Q criterion [50] which defines

regions with relative higher rotation rates than strain rates; 2) λ2 criterion which

Jeong and Hussain [53] states, “corresponds to the pressure minimum in a plane,

when contributions of unsteady irrotational straining and viscous terms in the Navier-

Stokes equations are discarded”; 3) ∆ criterion [22] which is based on finding regions

having intense rotation based on complex eigenvalues of the velocity gradient tensor;

4) λci (swirl strength) criterion [112] which like the ∆ criterion determines regions of

intense rotation, but limits the definition to regions with complex conjugate eigenpair

of the velocity gradient tensor; and 5) enhanced λci criterion [17] which extends the

the λci criterion to also limit regions which possess radially compact swirling patterns.

In the work of Chakraborty et al. [17], all of these methods are compared. It is shown

in their work that, in general, these methods identify similar vortical structures in

most turbulent flows. Although in some instances discrepancies between the methods

are obsevered. For the ∆ criterion, Jeong and Hussain [53] found vortex structures

to be more noisy than the Q or λ2 methods; Chakraborty et al. [17] also observed

noisy vortex boundaries for the ∆ criterion compared to the other methods. In forced

isotropic flow, Chakraborty et al. [17] reports a significantly lower value for vortex

region overlap between the ∆ and λci criteria (73.55% overlap) than for Q and λ2

20


criteria compared with the λci criterion which both indicate a +99% overlap. Also,

Jeong and Hussain [53] observed differences in identified vortex structures between

the Q and λ2 criteria for “a conically symmetric vortex with axial velocity”; this

difference is also noted by the swirling jet analysis of Chakraborty et al. [17] where it

is shown to be differences in the spiraling compactness thresholds implicily enforced

by the two criteria.

From a pragmatic point of view, the eigenvalue/vector calculations of the λ2, λci,

and the enhanced λci criteria methods make them more expensive to compute than

the Q and ∆ criteria methods. These differences in computational costs become

most appearent when performing vortex analyses for large domain and/or for many

ensembles. Between theQ and∆ criteria, it is previously argued that theQ criterion is

superior. As a result, in this work the Q criterion is used as the vortex identification

method for the analysis in §5.4. In this analysis, vortex regions are identified and

assembled using the pointwise vortex definition provided by the Q criterion. The

corresponding size, shape, and location of the vortical structures are studied and

discussed.

1.3 Thesis Outline

In this work, modeling and computing strategies for turbulence simulations are

presented. The first part of this work (Chapters 2-3) is devoted to multiscale mod-

21


eling methodologies for LES. In Chapter 2, RNS, which provides a dynamic, down-

scaling strategy for modeling subgrid scale geometry in LES, is discussed. Following

in Chapter 3 is the presentation of a concurrent precursor methodology (CPS). This

methodology provides a novel method for generating accurate inflow conditions for

multiscale turbulence simulations. The second part of this work (Chapters 4-5) is

focused on data intensive computing strategies for DNS. Discussed in Chapter 4 are

an overview of the JHTDB along with the design and construction of the new channel

flow database. Also included in the chapter is presentation of the Matlab Web ser-

vice client interface. The presentation of the channel flow DNS used to generate the

JHTDB channel flow database is presented in Chapter 5. The implemenation of the

channel flow DNS along with a study of coherent vortical structures are discussed. The

concluding remarks of this work are presented in Chapter 6. Additional discussions

can be found in the appendices. For example in Appendix A, a validation case for the

LES code LESGO used in the RNS work is presented. The next three appendices dis-

cuss topics from the second part of this work. Appendix B presents a data-intensive

computing approach for performing DNS in which a turbulence database (as used in

the JHTDB) may be generated during runtime of the DNS. Following this presenta-

tion is Appendix C in which the B-spline collocation method empolyed in the channel

flow DNS code (PoongBack [66]) used in this work is discussed. In the final appendix

(Appendix D) is a discussion on the pressure solver implemented by the author in

PoongBack; several validation and verification studies are therein presented.

22

Part I

Multiscale Modeling

23

Chapter 2

Renormalized Numerical

Simulation1

2.1 Introduction

In this chapter, the RNS methodology first introduced in Chapter 1 is discussed.

The proposed update to the RNS framework is presented in §2.2. An overview of

the LES implemtation used for the RNS is given in §2.3. Before introducing and

comparing the various formulations, a particularly simple tree geometry is presented

on which the various formulations will be tested. In this tree geometry all the branches

reside on a plane perpendicular to the flow direction. The geometry and simulation

set-up for this test case are presented in §2.4. The various RNS formulations are1Portions reused with permission from J. Graham and C. Meneveau, Phys. Fluids 24, 125105.

Copyright 2012, AIP Publishing LLC.

24

CHAPTER 2. RENORMALIZED NUMERICAL SIMULATION

described in §2.5. In §2.6 we present predicted forces, drag coefficients, and other

aspects characterizing the performance of the various formulations. Based on the

comparisons among them, we provide arguments about which can be considered the

best option. It is then applied to simulations of a fully three-dimensional fractal tree

in §2.7, and the total predicted drag force is compared to an available experimental

measurement. A summary and conclusions are presented in §2.8.

2.2 RNS Framework

It is convenient to represent unresolved or subgrid drag elements by a momentum

sink that extracts a prescribed amount of linear momentum from the flow. In high-Re

flows over blunt objects, the momentum sink or hydrodynamic drag force acting on

the fluid is expressed in terms of a quadratic law based on a representative velocity

Ũ according to,

FD = −1

2ρcdA|Ũ |Ũ . (2.1)

Here, ρ is the fluid density, cd is the drag coefficient, and A a representative surface

area. While the representative area may be defined based on resolved geometric

features and the reference velocity Ũ measured directly, the parameter cd associated

with complex multiple-scale structures is not typically known a-priori. For self-similar

or fractal-like objects, RNS may be used to determine cd in a dynamic fashion during

simulation.

25


Figure 2.1: Decomposition of fractal geometry into resolved and subgrid components,as well as between r, β and b elements. Numerically, in the present work, the resolvedportions are treated using the immersed boundary method (IBM), while sub gridportions are accounted for using RNS. The distribution of forces within the sub gridportion uses a filtered indicator function χ̃β (see text).

The RNS framework is based on decomposing the self-similar object into resolved

and subgrid-scale (unresolved) geometric features. An illustration of this decompo-

sition to a particular fractal tree is shown in Figure 2.1. The large scales at near

the base of the tree are resolved with the immersed boundary method (IBM), while

the remaining scales are modeled using a drag force parameterized with the RNS-

determined drag coefficient. The resolved and subgrid-scale regions are decomposed

into RNS elements, i.e. geometric components such as branches or branch clusters.

The smallest of the explicitly resolved components are grouped into what will be

called “r elements”, while the unresolved components are grouped into “β elements”

and contain the largest unresolved components as well as all of its descendants. Fur-

thermore, “b element” are defined as being composed of the union of any given r

element and all of the β elements that are direct descendants of the given r element.

The core of the RNS methodology is the statement that the total force associated

26


with a b element must be equal to the forces from its constituent r and β elements.

The total fluid force acting on the b elements (Figure 2.1) is given by

Fb,i(t) =

∫

Sb

(−pδij + τ vij)njdS (2.2)

where p is the pressure, τ vij is the viscous stress tensor, Sb is the wetted surface of

the entire b element and ni is the unit normal to this surface. We can decompose the

Sb surface into the surface corresponding to the r and β elements, and use Fr,i(t) =

∫Sr(−pδij + τ vij)njdS and Fβ,i(t) =

∫Sβ(−pδij + τ vij)njdS to write

Fb(t) =∑

r∈bFr(t) +

∑

β∈bFβ(t) . (2.3)

This (trivial) identity provides a self-consistency constraint that is analogous to the

Germano identity [39, 78] relating momentum fluxes filtered at various scales. The

identity becomes useful once certain modeling assumptions are introduced for Fb and

Fβ, while the resolved forces, Fr, can be obtained directly from the forces acting on

the numerically resolved portions of the object. In our case, when using the immersed

boundary method, the forces are provided directly during application of IBM. The

force associated with the unresolved elements, Fβ, is expressed using a generic form

drag model according to

Fβ(t) = −cd,β(t)Γβ(t) (2.4)

27


where cd,β is the β elements’ drag coefficient, and Γβ = ρ|Vβ|VβAβ/2. Here ρ is the

fluid density, and Vβ and Aβ are the reference velocity and representative frontal area,

respectively, of the β element. The reference velocity is computed by averaging the

fluid velocity over a predefined volume enclosing the β element. Ideally the volume

should contain only fluid that interacts with the element, in order to ensure that the

reference velocity correctly characterizes the hydrodynamic loading.

Next, we ‘zoom out’, and consider the b elements. The total force acting on these

elements can also be expressed in terms of a form drag model according to

Fb(t) = −cd,b(t)Γb(t) (2.5)

where cd,b is the drag coefficient appropriate for the b elements, and the vector Γb is

defined according to Γb = ρ|Vb|VbAb/2. Here Vb and Ab are the reference velocity

and representative frontal area, respectively, of the b element. The reference velocity

is computed by averaging the fluid velocity over a predefined volume enclosing the b

element, that is geometrically similar (but larger) than that used for the β elements.

If there exists complete geometric similarity between b and β elements (which ex-

ists in the case of a deterministic fractal shape shown), and if one assumes sufficiently

large Reynolds numbers so that the drag force does not depend on Reynolds number

(or scale), one may assume that both drag coefficients are equal, at least on average,

i.e. cd,β = cd,b for all β ∈ b. In general, however, this equality may not be exactly true

28


and may require additional considerations. We relegate further discussion to §2.5,

which is devoted to several RNS formulations including numerical, spatial, and tem-

poral treatments for the evaluation of cd,β and subsequently cd,b. For the remainder

of this section we assume that cd,β and cd,b are already known (their determination is

described in §2.5).

In a simulation, it is also necessary to prescribe the spatial distribution of this

force. As described in Chester et al. [21], the unresolved hydrodynamic drag force

is represented on a point x of the computational grid pertaining to element β as a

momentum sink (local force per unit mass) given by

fβ(x, t) = −κβ(t)|ũ(x, t)|ũ(x, t)χ̃(x) (2.6)

where κβ is determined so as to enforce that the total force due to element β is equal

to the prescribed force Fβ. ũ is the local resolved velocity vector at that point, and

χ̃ is the filtered indicator function. The filtered indicator function (illustrated in

Figure 2.1) is defined as χ̃ = G ∗ χ where χ is the true indicator function (1 inside

the object and 0 outside), and G is a Gaussian filter kernel of width 2∆. The filtered

indicator function is the representation of the subgrid-scale geometric information

within the computational mesh at resolved scales. The undetermined factor κβ is

chosen such that the integrated distributed force vector Fβ(t) =∫fβ(x, t)d

dx is as

close as possible to the total force computed from its parameterized form given by

29


(2.4) such that

κβ(t)Iβ(t) = cd,β(t)Γβ(t) (2.7)

where Iβ(t) =∫|ũ(x, t)|ũ(x, t)χ̃(x)ddx. Performing a least-squares minimization of

this overdetermined system of equations (since it is a vector equation for a scalar

quantity) results in the contraction each side of (2.7) by Iβ. Solving for κβ from the

resulting scalar equation results in

κβ(t) = cd,β(t)Γβ · Iβ(t)|Iβ(t)|2

. (2.8)

Once κβ is determined for each β element, (2.6) is applied as a forcing term in the

momentum equation.

2.3 LES Implementation

Simulations are performed using a variant of the JHU-LES code [92, 12, 21] called

LESGO. LESGO solves the filtered, incompressible Navier-Stokes equations for a

neutrally buoyant and high-Re flow such that,

∂ũ

∂t+ ũ ·

(∇ũ−∇ũT

)

= −ρ−1∇p̃+∇ · τ + ρ−1f̃ + ρ−1Πi (2.9a)

30


∇ · ũ = 0 (2.9b)

where ũ the filtered velocity, ρ the fluid density, p̃ the filtered (modified) pressure, τ

the deviatoric component of the subgrid scale stress tensor, f̃ the forcing term which

contains the IBM and RNS forces, and Π = −dP/dx the applied mean pressure

gradient forcing where i = (1, 0, 0). For all simulations in this study, the scale-

dependent Lagrangian dynamic subgrid stress model [12] is employed.

The governing equations are discretized using a pseudo-spectral method where

spectral discretization is used in the x and y (horizontal) directions and 2nd order

finite differencing in the z (or vertical) direction. Time integration is performed using

a 2nd order Adams-Bashforth scheme. Periodic boundary conditions are used along

the sides of the domain. A stress-free boundary condition is imposed at the top of the

domain, while a rough-wall (low-law) boundary condition is imposed at the bottom

surface.

Solid objects (resolved scale features) in the domain are represented using the

IBM implementation already described in Chester et al. [21]. This implementation

employs a level set description for the representation of the objects in the domain.

The IBM enforces a zero velocity condition inside of solid objects along with a log-law

wall shear stress that is applied as a tangential boundary condition at the wall. In

Appendix A, a validation case using the IBM is performed for flow over an array of

wall mounted cubes.

31


2.4 Planar tree canopy test case

For the purpose of testing various RNS formulations to be presented in §2.5, a

configuration for flow over a canopy of “planar fractal trees” is used. The branches

of each tree are arranged in a “V” configuration. Shown in Figure 2.2 is a canopy

of regularly spaced trees consisting of only two resolved generations (g0 and g1).

Figure 2.1), shows the tree containing the filtered representation of the additional

subgrid-scale branches. The planar elements provide the multiple-scale features in

the y and z directions while remaining relatively simple.

A constant scale factor r = 1/2 coupled to a doubling of each element at each gen-

eration provides a completely self-similar tree with a fractal similarity dimension[74]

of D = log(NB)/log(r−1) = 1 where NB = 2 is the number of branches of the gener-

ator at g0. The tree geometry is described by an iterated function system[34] (IFS),

with the similarity contraction {wi : X → X | i = 0, . . . , NB − 1} where X is

a closed subset of R3. It follows that, a branch cluster B at generation gn is ob-

tained by performing n contraction mappings of the fractal generator G such that

B = win ◦ · · · ◦wi1(G), where

wi(x) = rRi · x+ si, (2.10)

Ri a rotation matrix, and si a translation vector. To describe the current geometry,

we have Ri = 1, s0 = (0,−1.179h, h) and s1 = (0, 1.179h, h) where h is the height

32


Figure 2.2: Fractal tree canopy composed of planar V-trees. Shown are the two“resolved” generations g0 and g1. The solid black box indicates the physical domainfor the simulations. The entirety of the tree as represented in the simulation is shownin Figure 2.1.

of the generator. The generator branch diameter is d = 0.571h and has a length

of l = h/cos(θ) where θ = 45◦ is the skew angle relative to the vertical axis. Each

branch is also offset a distance of 0.179h from the center of the branch cluster.

The simulated domain is the region indicated by the black box in Figure 2.2 and

is defined as {(x, y, z) : 0 ≤ x ≤ 12h, 0 ≤ y ≤ 6h, 0 ≤ z ≤ 8.2h}. Three trees

were placed in the periodic domain in order to minimize coupling between turbulence

generated by the leading tree from that of the last tree (and vice versa). A uniform

spatial discretization of Nx × Ny × Nz = 300 × 150 × 206 is used such that there

is a minimum of eight grid points across the diameter of the branches in the last

resolved generation. This helps ensure sufficient resolution for the IBM at least as

33


far as predictions of drag forces is concerned [21, 104]. The flow is forced in the +x

direction using a mean pressure gradient forcing of Π. The normalization parameters

used in the simulations are based on ρ, h and Π such that the time, velocity, and

force scales are defined as,

τ =√

ρhΠ−1 (2.11)

up =

√h

ρΠ (2.12)

fp =ρu2ph

= Π (2.13)

respectively.

Boundary conditions along the x − y perimeter are periodic. A stress-free, no-

permeability condition is applied at the top boundary. Along the bottom surface,

the IBM is used to implement a rough wall, log-law stress[21, 20] with a surface

roughness of zo = 10−4h. The IBM treatment of the bottom surface was required

because in the IBM implementation, grid points normal to the solid surfaces at a

distance δ ≤ 1.1∆, where ∆ is the grid spacing, are required for evaluation of the

tangential stresses applied to the fluid. As a result, there exist points xs = (xs, ys, zs)

on the tree surface near z = 0 such that z = zs+δnz < 0, where nz is the z component

of the local surface normal, that consequently lie outside the computational domain.

As a result it was required to place several grid points below the bottom surface of

the simulated domain and use the IBM to represent the bottom surface consistently.

34


As in prior applications[21, 20], the same log-law stress boundary condition is applied

on the resolved tree branch surfaces, also imposing a constant surface roughness of

zo = 10−4h.

Shown in Figure 2.3 are the RNS reference regions (volumes) for both b elements

and one set of β elements for the target tree in the domain. The base of the ref-

erence region is centered about the base of the bottom most branch cluster within

the respective elements and the top of the region extends to the top of the tree at

z = 2h. Proportionality factors are applied to determine the width (y direction)

and depth (x direction) of the reference region such that the width and depth are

2.0 and 1.143, respectively, times the height. The reference area used for the RNS

calculations is taken to be the frontal area (whose normal is in the −x direction) of

the reference regions. The reference velocity for the b and β elements is calculated

from the volumetric mean of the velocity inside the reference region for each element.

During the calculation of the reference velocity, the LES velocity is sampled at evenly

distributed points within the volume with a spacing at or below the grid spacing. A

trilinear interpolation is used to place the LES velocity on these sampling points.

During RNS, the flow field is initialized with a log-law boundary layer profile with

superimposed background perturbations. The simulations are allowed to “spinup”

for fully developed, statistically steady-state canopy turbulence to be established.

In Figure 2.4, the magnitude of the instantaneous velocity from one of the simu-

lations is shown. These results are from the RNS using model M2 (to be described

35


Figure 2.3: Definitions of RNS references regions used in the “V-tree” canopysimulations.

in detail in §2.5). From Figure 2.4 the structure of the subgrid geometric scales is

readily observed within the contour slices along the y− z (constant x) plane through

the low velocity areas highlighted by the black contour regions. Qualitatively, this

clearly hightlights the impact of the subgrid-scale force field on the surrounding fluid

where we observe a similar “footprint” in the upper canopy as would be expected

from a direct (resolved) representation of all the branches. Shown in Figure 2.5 are

contours of the magnitudes of the instantaneous velocity on x−y (constant z) planes.

The planes are along the branch mid-planes of generations g2, g3, and g4, which are all

within the subgrid-scale region. Vortex shedding due to the parameterized subgrid-

scale branch structures is observed thus showing the ability of the modeling technique

to capture physical flow phenomenon. Also noted are the scale reduction and increase

36


Figure 2.4: Contour plot of instantaneous velocity magnitude along a constant y andconstant x plane.

in number of the wakes with increasing generation and height. This is a direct result

of retaining the multiscale, geometric information in the applied subgrid-scale forcing.

The magnitude of the applied force field as given by (2.6) is shown in Figure 2.6 for

the same time step as the previous velocity plots. From this figure the retention of

the multiscale, geometric information by the model is further illustrated. Also from

this figure, the smoothed subgrid-scale structure of the tree is clearly visible. We

note the non-uniformity in the force distribution within the subgrid region, due to

the non-uniformity of the instantaneous velocity distribution highlighting the local

nature of the applied subgrid-scale forcing.

37


Figure 2.5: Instantaneous velocity magnitude contours on horizontal planes at themid-plane heights of branch generations g2, g3 and g4.

Figure 2.6: Contour plot of instantaneous local force magnitude along a constant-xplane across the middle tree in the domain.

38


2.5 RNS formulations

After having presented the numerical simulation technique and some representa-

tive views of the flow, in this section we present various alternative formulations to

evaluate drag coefficients for the unresolved forces. Four versions will be considered,

to be denoted as models M1 through M4, as listed in Table 2.1. There are three

different types of treatments we apply to the evaluation of the RNS drag coefficient.

First, the temporal treatment indicates whether or not time averaging is used in the

evaluation of the drag coefficient. Second, spatial treatment refers to whether a single

drag coefficient (global) is computed for all the objects in the domain (e.g. all trees

in a canopy or all branches on a tree), or if a drag coefficient is computed for each

of the b elements in the domain (local). Last, the numerical treatment indicates if

an explicit or implicit time treatment is used for the drag coefficient calculations.

We present four permutations of the three treatment categories. For each model a

simulation is performed using the fractal tree canopy as presented in §2.4.

Table 2.1: Definitions of the tested RNS models.

Model Temporal Spatial NumericalM1 Instantaneous Global ExplicitM2 Averaged Global ExplicitM3 Averaged Global ImplicitM4 Averaged Local Explicit

39


2.5.1 Model M1

This is the original RNS model presented by Chester et al. [21]. It was based

on an instantaneous approach, computing a single (global) drag coefficient for all

branches, using an explicit dynamic formulation, and a branch-based description. In

this section we recast the original formulation using a more generalized framework,

based on “b” and “β” elements.

The basic relationship between the drag coefficients of each b element and its

descendant β elements is written as

cd,β(t) = cd,b(t) (2.14)

for all β ∈ b. This equality assumes complete dynamic similarity. This assumption

breaks down if the subgrid-scale drag becomes affected by viscous drag, since then

there is an additional parameter, the Reynolds number, which would differ at the

two scales. Here we assume that the Reynolds number is large enough so that even

the full subgrid-scale range is in the inertia-dominated regime. For rough boundary

layers, this would correspond to a subgrid range that is in the “fully-rough” regime.

Similar arguments were developed and tested for LES using the dynamic rough-wall

model [3]. Moreover, Eq. 2.14 also assumes dynamic similarity at any instant t. A

limitation of this approach is that for a turbulent flow, complete similarity between

branches is only satisfied in a time-averaged sense, whereas here we are assuming

40


that it holds at each individual time. In using a global definition for the RNS drag

coefficient, the drag coefficient for all b elements reduces to a common global value

such that we use cd,b = cd for all b. The time explicit treatment implies that model

M1 uses the drag coefficient cd(t −∆t) determined at the previous time-step (∆t is

the time step) in the formulation of the force equality. With these assumptions in

mind, the force balance on each b element – as given by (2.3) – may be expressed as,

Fb(t) =∑

r∈bFr(t)− cd(t−∆t)

∑

β∈bΓβ(t). (2.15)

By virtue of the time-explicit treatment, the right hand side of the above equation

is fully specified at every instance in time. Self-consistency also requires that the

vector expression given by (2.5) be satisfied, with cd(t) chosen to enforce the equality

in some sense. Following Chester et al. [21] and usual practice for overdetermined

systems as in the dynamic model [71], a least-squares error minimization approach is

applied. The error to be minimized is expre

turbulence simulations: multiscale modeling and data-intensive … · abstract in this two part...

Documents