12 march 2004 a high-fidelity, high-performance turbulent dispersion framework damian rouson...
TRANSCRIPT
12 March 200412 March 2004
A High-Fidelity, High-A High-Fidelity, High-Performance Turbulent Performance Turbulent Dispersion Framework Dispersion Framework
Damian RousonDamian RousonCombustion Modeling & ScalingCombustion Modeling & ScalingU.S. Naval Research Laboratory, U.S. Naval Research Laboratory,
Washington, DCWashington, DC
Prof. Joel Koplik & Karla MorrisProf. Joel Koplik & Karla MorrisDepts. of Physics & Mechanical Depts. of Physics & Mechanical
EngineeringEngineeringThe City College of CUNY, New York, NYThe City College of CUNY, New York, NY
Sponsors: NSF, NRLSponsors: NSF, NRL
OverviewOverview Motivation & ObjectivesMotivation & Objectives Previous workPrevious work PhysicsPhysics
– Fire suppressionFire suppression– Quantum turbulenceQuantum turbulence
SoftwareSoftware– Componentization strategyComponentization strategy– O.O. Fortran 90/95 ImplementationO.O. Fortran 90/95 Implementation
Future WorkFuture Work– Study Study
architecturearchitecturecomplexitycomplexityperformanperformancece
– Study fine-grained parallelism (OpenMP)Study fine-grained parallelism (OpenMP)
MotivationMotivation
Motivation Motivation ObjectivesObjectivesPrevious WorkPrevious WorkPhysicsPhysicsSoftwareSoftwareFuture WorkFuture Work
NN ObjectsObjects
00 Point masses: particles, Point masses: particles, dropletsdroplets
11 Quantum vortices, “snails”Quantum vortices, “snails”
22 Interfaces (fluid/fluid & Interfaces (fluid/fluid & fluid/solid)fluid/solid)
33 Two-fluid mixturesTwo-fluid mixtures
A broad class of phenomena involve A broad class of phenomena involve interactions between fluid turbulence & interactions between fluid turbulence & N-dimensional objectsN-dimensional objects
Algorithms & data structures for these Algorithms & data structures for these problems exhibit a high degree of problems exhibit a high degree of commonality.commonality.
Exploiting software design Exploiting software design commonalities can lead to greater code commonalities can lead to greater code reuse and more efficient interoperationreuse and more efficient interoperation
To build an extensible set of software To build an extensible set of software components suitable for the N={0,1} subset components suitable for the N={0,1} subset of turbulent dispersion problems.of turbulent dispersion problems.
To quantify the influence of object-oriented To quantify the influence of object-oriented design & componentization on software design & componentization on software complexity and performance.complexity and performance.
To simulate new physics & improve To simulate new physics & improve engineering models:engineering models:– Fire suppressionFire suppression– Quantum turbulence Quantum turbulence – Boundary-layer combustionBoundary-layer combustion
ObjectivesObjectives
MotivationMotivationObjectivesObjectivesPrevious WorkPrevious WorkPhysicsPhysicsSoftwareSoftwareResultsResultsConclusions & Future WorkConclusions & Future Work
Previous WorkPrevious Work
MotivationMotivationObjectivesObjectivesPrevious WorkPrevious WorkPhysicsPhysicsSoftwareSoftwareFuture WorkFuture Work
Rouson & Eaton (2001) “On the preferential Rouson & Eaton (2001) “On the preferential concentration of solid particles in a fully-developed concentration of solid particles in a fully-developed turbulent channel flow”, turbulent channel flow”, J. Fluid Mech.J. Fluid Mech.
Rouson & Xiong (2004) “Design metrics in quantum Rouson & Xiong (2004) “Design metrics in quantum turbulence simulations: How physics influences turbulence simulations: How physics influences software architecture”, software architecture”, Scientific ProgrammingScientific Programming..
Decyk, Norton & Szymanski (1997-) Object-oriented Decyk, Norton & Szymanski (1997-) Object-oriented programming in Fortran 90, programming in Fortran 90, ACM Fortran ForumACM Fortran Forum, , etc.etc.
Akin (2003), Akin (2003), Object-Oriented Programming via Object-Oriented Programming via Fortran 90/95Fortran 90/95, Cambridge U. Press., Cambridge U. Press.
Armstrong et al. (1999-) “Toward a common Armstrong et al. (1999-) “Toward a common component architecture for high-performance component architecture for high-performance scientific computing”scientific computing”
Particle-Laden FlowParticle-Laden FlowDirect Numerical Simulation (DNS) of Direct Numerical Simulation (DNS) of Navier-Stokes Equations:Navier-Stokes Equations:
Spectral Spatial Representation:
0/
/Re)/1(///
jj
jjiijiji
xu
xxuxpxuutu
fp
DDD
iiiDi
ii
St
C
gStuCdtd
dtdx
/
Re/Re124
/)v)((/v
v/
687.0
x
Lagrangian Particle Equation of Motion:
zyx
zyx
zyxkkk
zkykxkikkk ett
,,
)()(ˆ),( uxu
3rd-Order Runge-Kutta Time Advancement
MotivationMotivationObjectivesObjectivesPrevious WorkPrevious WorkPhysicsPhysicsSoftwareSoftwareFuture WorkFuture Work
Isotropic turbulence:
Particle-Laden Particle-Laden FlowFlow
Particles near Particles near centerplane of a centerplane of a channel flow:channel flow:
Preferential concentration: Particles move independently Particles move independently
from each other and slip from each other and slip relative to flowrelative to flow
Coherent flow structures Coherent flow structures segregate particles segregate particles according to their inertiaaccording to their inertia– e.g. “cyclone separation” by e.g. “cyclone separation” by
vortices vortices
MotivationMotivationObjectivesObjectivesPrevious WorkPrevious WorkPhysicsPhysicsSoftwareSoftwareFuture WorkFuture Work
Below 2.17 K, liquid helium behaves as a two-Below 2.17 K, liquid helium behaves as a two-fluid mixture:fluid mixture: Normal viscous fluidNormal viscous fluid Inviscid superfluidInviscid superfluid
– – characterized by discrete (quantized) characterized by discrete (quantized) vortex motionsvortex motions
– – vortex core ~ 1 A acrossvortex core ~ 1 A across
– – can be modeled by can be modeled by
1. Nonlinear Schrodginger Eq.1. Nonlinear Schrodginger Eq.
2. Euler equation 2. Euler equation
3. Discrete vortex equations3. Discrete vortex equationsSurprisingly, turbulence statisticsSurprisingly, turbulence statistics
match those of classical flows.match those of classical flows.
Quantum TurbulenceQuantum Turbulence
Density ratio of each Density ratio of each fluid:fluid:
MotivationMotivationObjectivesObjectivesPrevious WorkPrevious WorkPhysicsPhysicsSoftwareSoftwareFuture WorkFuture Work
SuperfluidSuperfluid: vortex filament method. : vortex filament method.
Quantum TurbulenceQuantum Turbulence
3
00
////
/)(4
)]([)(
rSSrSv
vvvSSvvvSvvS
d
dt
d
i
isnisnis
),( tS
Arbitrary originArbitrary origin
),(
),(/t
tS
S
Increasing Increasing
Vortex filamentVortex filament
Normal fluid: Normal fluid: DNS of Navier-Stokes DNS of Navier-Stokes equations.equations.
Material properties: Material properties: ’’Quantum of circulation:Quantum of circulation: = ħ/m= ħ/mHeHe= 9.97*10= 9.97*10-4-4 cm cm22/sec/sec
Equations of motion:Equations of motion:
(Biot-Savart Law)(Biot-Savart Law)
MotivationMotivationObjectivesObjectivesPrevious WorkPrevious WorkPhysicsPhysicsSoftwareSoftwareFuture WorkFuture Work
Animation Animation Barenghi et al. (1997)Barenghi et al. (1997) Our resultOur result
)2
cos()2
sin(
)2
cos()2
sin(
)2
cos()2
sin(
xB
yCv
zA
xBv
yC
zAv
z
y
x
A=B=C=0.748cm/s, 0.2cm
As a prelude to turbulence, we studied the As a prelude to turbulence, we studied the steady Arnold-Beltrami-Childress (ABC) steady Arnold-Beltrami-Childress (ABC) flow:flow:
Quantum TurbulenceQuantum Turbulence
MotivationMotivationObjectivesObjectivesPrevious WorkPrevious WorkPhysicsPhysicsSoftwareSoftwareFuture WorkFuture Work
ComponentizationComponentization
FluidFluidProvides: Provides: d_dt(), u(x)d_dt(), u(x)
Uses: Uses:
f(x)f(x)
VectorFielVectorFieldd
GridGrid
CloudCloudProvides: Provides: d_dt(), f(x)d_dt(), f(x)
Uses: Uses:
u(x)u(x)
DropleDroplett
TangleTangleProvides: Provides: d_dt(), f(x)d_dt(), f(x)
Uses:Uses:
u(x)u(x)
VortexPoinVortexPointt
IntegratorIntegratorProvides:Provides: eulereuler RK3RK3
Uses:Uses:
d_dt()d_dt()
MotivationMotivationObjectivesObjectivesPrevious WorkPrevious WorkPhysicsPhysicsSoftwareSoftwareFuture WorkFuture Work
OOP in Fortran 90/95OOP in Fortran 90/95
OOP OOP ConceptConcept
Fortran Fortran ConstructConstruct
ClassClass MODULEMODULE
Abstract data Abstract data typetype
Derived typeDerived type
InheritanceInheritance AggregationAggregation
PolymorphisPolymorphismm
Generic Generic interfacesinterfaces
MotivationMotivationObjectivesObjectivesPrevious WorkPrevious WorkPhysicsPhysicsSoftwareSoftwareFuture WorkFuture Work
Fortran ImplementationFortran ImplementationMODULE Cloud_ClassMODULE Cloud_Class USE Droplet_ClassUSE Droplet_Class PRIVATEPRIVATE PUBLIC :: d_dtPUBLIC :: d_dt(), operator(*), operator(-)(), operator(*), operator(-)
TYPE CloudTYPE Cloud PRIVATE PRIVATE TYPE(Droplet), DIMENSION(:), ALLOCATABLE :: dropTYPE(Droplet), DIMENSION(:), ALLOCATABLE :: drop END TYPE Cloud END TYPE Cloud
INTERFACE d_dt INTERFACE d_dt MODULE PROCEDURE d_dt_CloudMODULE PROCEDURE d_dt_Cloud END INTERFACE END INTERFACE
CONTAINSCONTAINS
FUNCTION d_dt_Cloud(this) RESULT(dState_dt)FUNCTION d_dt_Cloud(this) RESULT(dState_dt) TYPE(Cloud), INTENT(IN) :: this TYPE(Cloud), INTENT(IN) :: this TYPE(Cloud), POINTER :: dState_dtTYPE(Cloud), POINTER :: dState_dt ......
DO i=1,nDO i=1,n dState_dt%drop(i) = d_dt(this%drop(i)) dState_dt%drop(i) = d_dt(this%drop(i)) END DOEND DO
InheritanInheritancece
PolymorphiPolymorphismsm
EncapsulatEncapsulationion
MotivationMotivationObjectivesObjectivesPrevious WorkPrevious WorkPhysicsPhysicsSoftwareSoftwareFuture WorkFuture Work
Fortran ImplementationFortran ImplementationMODULE Droplet_Class MODULE Droplet_Class
PRIVATE PRIVATE PUBLIC :: d_dt(), operator(*), operator(-)PUBLIC :: d_dt(), operator(*), operator(-)
TYPE Droplet TYPE Droplet PRIVATE PRIVATE REAL :: St REAL :: St REAL, DIMENSION(3) :: s REAL, DIMENSION(3) :: s REAL, DIMENSION(3) :: v REAL, DIMENSION(3) :: v END TYPE DropletEND TYPE Droplet
… …
CONTAINS CONTAINS FUNCTION d_dt_Droplet(this,u) RESULT(dState_dt) FUNCTION d_dt_Droplet(this,u) RESULT(dState_dt) TYPE(Droplet),POINTER :: dState_dtTYPE(Droplet),POINTER :: dState_dt TYPE(Droplet),INTENT(IN) :: thisTYPE(Droplet),INTENT(IN) :: this REAL ,INTENT(IN) ,DIMENSION(3) :: uREAL ,INTENT(IN) ,DIMENSION(3) :: u
… …
dState_dt%St = 0.0dState_dt%St = 0.0 dState_dt%s = this%vdState_dt%s = this%v dState_dt%v = (u-this%v)/this%StdState_dt%v = (u-this%v)/this%St
MotivationMotivationObjectivesObjectivesPrevious WorkPrevious WorkPhysicsPhysicsSoftwareSoftwareFuture WorkFuture Work
Fortran ImplementationFortran ImplementationMODULE Integrator_ClassMODULE Integrator_Class
USE Cloud_Class USE Cloud_Class USE Fluid_ClassUSE Fluid_Class
PRIVATE PRIVATE PUBLIC :: euler_IntegratorPUBLIC :: euler_Integrator
TYPE IntegrandTYPE Integrand PRIVATEPRIVATE TYPE(Cloud) , POINTER :: cloud_ptrTYPE(Cloud) , POINTER :: cloud_ptr TYPE(Fluid) , POINTER :: fluid_ptrTYPE(Fluid) , POINTER :: fluid_ptr END TYPE IntegrandEND TYPE Integrand…… CONTAINS CONTAINS SUBROUTINE euler_Integrator(this,dt)SUBROUTINE euler_Integrator(this,dt) TYPE(Integrand), INTENT(INOUT) :: thisTYPE(Integrand), INTENT(INOUT) :: this REAL(NBYTES) :: dtREAL(NBYTES) :: dt IF (ASSOCIATED(this%fluid_ptr) ) THENIF (ASSOCIATED(this%fluid_ptr) ) THEN
this%fluid_ptr = this%fluid_ptr + dt*d_dt(this%fluid_ptr)this%fluid_ptr = this%fluid_ptr + dt*d_dt(this%fluid_ptr)
ELSE IF (ASSOCIATED(this%cloud_ptr)) THENELSE IF (ASSOCIATED(this%cloud_ptr)) THEN……
MotivationMotivationObjectivesObjectivesPrevious WorkPrevious WorkPhysicsPhysicsSoftwareSoftwareFuture WorkFuture Work
Run-time Run-time polymorphismpolymorphismvia dynamic via dynamic dispatchingdispatching
Complete componentization of F90 codeComplete componentization of F90 code Study performance via fine-grained Study performance via fine-grained
parallelizationparallelization– Several recent developments presage a Several recent developments presage a
potentially renewed importance of OpenMP:potentially renewed importance of OpenMP: Multi-core chips (Intel)Multi-core chips (Intel) Hardware support for multithreading (Sun’s Hardware support for multithreading (Sun’s
Niagara)Niagara) Efficient emulation of SMP via NUMA clusters Efficient emulation of SMP via NUMA clusters
(SGI’s Altix)(SGI’s Altix)
Future WorkFuture Work
MotivationMotivationObjectivesObjectivesPrevious WorkPrevious WorkPhysicsPhysicsSoftwareSoftwareFuture WorkFuture Work