organizers: bertrand rollin and jason hackl agenda · :4248-4267. cmt/artvisc.f. subroutine...
TRANSCRIPT
![Page 1: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/1.jpg)
CCMT
Agenda
• CMT-nek: Anatomy of the BeastSpeaker: Jason Hackl | 1:30 pm – 2:15 pm
• Lagrangian Particles in CMT-nekSpeaker: David Zwick | 2:15 pm – 2:45 pm
Break
• Running CMT-nekSpeakers: Goran Marjanovic and Brad Durant | 3:00 pm – 3:45 pm
• Post-Processing and visualization in CMT-nekSpeakers: David Zwick and Brad Durant | 3:45 pm – 4:15 pm
• Lesson Learnt from CCMT’s SimulationsSpeakers: Fred Ouellet and Yash Mehta | 4:15 pm – 4:45 pm
A Boot Camp on CMT-nekNovember 29, 2017 – CCMT’s seminar room
Organizers: Bertrand Rollin and Jason Hackl
![Page 2: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/2.jpg)
CCMT
Participates
![Page 3: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/3.jpg)
CCMT
CCMT
CMT-NekDevelopers’ learning cliff
November 29, 2017
Jason Hackl
![Page 4: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/4.jpg)
CCMT4
CMT-nek developers’ learning cliff• Discretization of conservation laws
– Time marching
– Semi-discretized right-hand-side• Volume Terms: convective and diffusive
• Surface Terms: convective, then diffusive
• Where things happen– nek5000 time loop and cmt-nek execution branch
– compute_rhs_and_dt
• Governing equations– Euler equations of gas dynamics
– Artificial diffusive stress tensor
– Entropy viscosity method
• What gets done: some details
![Page 5: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/5.jpg)
CCMT5
CMT-nek: conservation laws discretizedflux = convective + diffusive
Particle source terms RDavid ZwickEquate coefficients of vT , left multiply by B-1
dU/dt = res1 = volume + surface + source
res1 = Iconv + Isfc + IKU + IGU + IGTU
volume terms done 1 element at a timesurface terms done for all faces at oncenek5000 time loop and cmt-nek execution branchcompute_rhs_and_dt fills res1 (CMTDATA) with RHS terms
BM1
![Page 6: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/6.jpg)
CCMT6
Time loop TVDRK3• core/drive1.f
• nek__multi_advance(kstep,msteps)• do i=1,msteps
call nek_advance• call cmt_nek_advance• call userchk
do stage=1,nstageif (stage.eq.1) call copy(res3(1,1,1,1,1),U(1,1,1,1,1),n)call compute_rhs_and_dtdo e=1,nelt
do eq=1,toteqdo i=1,nxyz1
c multiply u with bm1 as res has been multiplied by bm1 in compute_rhsu(i,1,1,eq,e) = bm1(i,1,1,e)*tcoef(1,stage)*res3(i,1,1,eq,e)
> +bm1(i,1,1,e)*tcoef(2,stage)*u(i,1,1,eq,e)+> - tcoef(3,stage)*res1(i,1,1,e,eq)
cmt/drive_cmt.f• set_tstep_coef sets tcoef
![Page 7: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/7.jpg)
CCMT7
drive1_cmt.f compute_rhs_and_dtvolume surface
convective
diffusive
call compute_primitive_varsif(stage.eq.1) thencall setdtcmtcall set_tstep_coefendif
call entropy_viscositycall compute_transport_props
call fluxes_full_field Iconvdo eq=1,toteq
call surface_integral_full(res1(1,1,1,1,eq),flux(ieq))enddo
call imqqtucall igtu_cmt(flux(iwm),flux(iuj),graduf) IGTU
res1 = Iconv + Isfc + IKU + IGU + IGTU
![Page 8: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/8.jpg)
CCMT8
drive1_cmt.f compute_rhs_and_dtvolume surface
convective
diffusive
do e=1,neltcall cmtusrf(e)call compute_gradients(e)do eq=1,toteq
call convective_cmt(e,eq) Iconvcall viscous_cmt(e,eq) IKUcall compute_forcing(e,eq)
enddoenddo
call igu_cmt(flux(iwp),graduf,flux(iwm)) IGUdo eq=1,toteq
call surface_integral_full(res1(1,1,1,1,eq),flux(ieq))enddo
res1 = Iconv + Isfc + IKU + IGU + IGTU
![Page 9: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/9.jpg)
CCMT9
drive1_cmt.f compute_rhs_and_dtvolume surface
convective
diffusive
ONE ELEMENT AT A TIME
All elements in sequence
![Page 10: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/10.jpg)
CCMT10
Thermodynamic stateTemperaturePressureTotal energy
CMT-nek Governing EquationsVolume fractions of
gas…
…and particles
Convective fluxes of
MassHc=(U2 ,U3 ,U4 )T
Momentum
Energy
Conserved variablesto primitive variables
Upwind numerical flux H*c
• Function of primitive variables (p+,-, u+,-,…)• Riemann solver
(Advection Upstream Splitting (AUSM))
Liou (1996) ”AUSM+” J. Comp. Phys.,129:362-382
Boundary conditions from Dirichlet U+ → AUSM+• Riemann invariants for inflow and outflow (extrapolated supersonic)• Reflect about wall-normal rst lines for walls (geometry already high-order)
No particles here; 𝜙𝜙𝑔𝑔 = 1Zwick L2.00012
![Page 11: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/11.jpg)
CCMT11
Artificial viscosity
Diffusive flux of…Mass
Momentum
Energy
X
1Guermond & Popov (2014) “Viscous regularization of Euler eqn and entropy principles” J. Appl. Math. 74:284-305
Artificial stress tensor1 (NOT Navier-Stokes) with AV that tracks the entropy residualNOT weighted by volume fraction 𝜙𝜙𝑔𝑔
AGRADUadu_evm
![Page 12: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/12.jpg)
CCMT12
Entropy viscosity method
due to entropy viscosity
Rs approx by finite diff between RK3 stages’ s
C0 by average Rs at face points. Smooth via
1Guermond & Popov (2014) “Viscous regularization of Euler eqn and entropy principles” J. Appl. Math. 74:284-305Guermond, Pasquetti & Popov (2011) J. Comp. Phys. 230:4248-4267
cmt/artvisc.fsubroutine entropy_viscositycall compute_entropy(tlag)call entropy_residual(tlag)call wavevisc(t(1,1,1,1,3))call resvisc(res2) ! overwrite res2call evmsmoothcall dsavg(res2) DONE IN USERVP
![Page 13: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/13.jpg)
CCMT13
Deville, Fischer and Mund (2002) Higher-Order Methods for Incomp. Flows CambridgeLottes & Fischer (2005) J. Sci. Comp. 24(1):45-78 http://nek5000.mcs.anl.gov , http://nek5000.github.io/NekDoc/
U- from element U+ from neighbor
U+ from neighbor
Boundary conditions weakly imposed through H*• Dirichlet: prescribe U+
• Neumann: prescribe flux directly
Discontinuous Galerkin SEM
Average
Jump
From the element’s point of view• Restriction operator E gets nodal values on my faces
• Gather-scatter operator QQT
adds them to my neighbor’svalues and gives us both the result
• All interelement communication through QQT
• I get U+ via QQT too
![Page 14: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/14.jpg)
CCMT14
DGSEM: diffusive
Baumann & Oden (1999) Comput. Methods Appl. Math. Engrg. 175:311-341
Quasi-linear flux Jacobian APrimal formNOT weighted by volume fraction 𝜙𝜙𝑔𝑔
Dolejší, V. (2004) Intl. J. Numer. Meth. Fluids 45:1083-1106Hartmann & Houston (2008) J. Comp. Phys. 227:9670-9685
Weighted residual, integrate by parts → primal form → DGSEM ops
flux = convective + diffusive
Hackl, Shringarpure, Koneru, Delchini & Balachandar (2017) in preparation for Computers & Fluids
imqqtu = [I-QQT]U
![Page 15: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/15.jpg)
CCMT15
CMT-nek geography: what lives where?Description Dimensions Var Include,common
Quad weight x J (lx1,ly1,lz1,lelt) BM1 nek/MASS, /MASS/
d/dr matrix (lxd,lxd) D,dt,… /dgradl/
Interpolation matrix (lxd,lx1) Jgl,jgt /dgradl/
Mesh Jacobian (lx1,ly1,lz1,lelt) JACM1,jacmi nek/GEOM, /GISO1/
Grid metric (lxd^3,ldim^2,lelt) rx nek/GEOM, /GISOD/
Flux at grid points (lxd^3,ldim) totalh nek/cmt/CMTDATA
Indices of face pts subroutine facind iface_flux nek/cmt/DG
Face interp matrix (lxd,lx1) jgl,jglt /dgrad/
Face quadrature (lxd,lxd) wghtf nek/cmt/DG, /facewz/
Jacobian on face (lx1,lx1) Jaco_c /SCRNS/
Numerical flux huge lots /CMTSURFLX/
Conserved variable (lx1,ly1,lz1,toteq,lelt)
u nek/cmt/CMTDATAU
J
![Page 16: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/16.jpg)
CCMT16
Spectral elements
Reference element
![Page 17: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/17.jpg)
CCMT
CCMT
Do you have any questions?
![Page 18: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/18.jpg)
CCMT
CCMT
Lagrangian Particles in CMT-nek
D. Zwick
![Page 19: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/19.jpg)
CCMT19
Background• Fluid solved in Eulerian reference frame
• Particles solved in Lagrangian reference frame
• Up-to-date particle code that will always work with particle example cases:– Source code:
• https://github.com/dpzwick/Nek5000/tree/particles_new
– Particle examples:• https://github.com/dpzwick/NekExamples
Source synced with UFCCMT
~ monthly
![Page 20: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/20.jpg)
CCMT20
Capabilities and Options• One-way coupling:
– Interpolation
– Integration
– Force models
– Heat transfer models
– Boundary conditions
– Restarting
– Injection
– Variable particle sizes
• Two-way coupling:– Projection filter type
– Projection filter width
– Super particle loading
– Number of ghost particles
• Four-way coupling:– Normal collision spring
coeffecient
– Normal collision damper
![Page 21: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/21.jpg)
CCMT21
Particle Files and Options
• Source code:– cmtparticles.f (Nek5000/core/cmt/)– CMTPART (Nek5000/core/cmt/)
• User code:– cmtparticles.usrp (case_directory/)– particles.inp (case_directory/)
Development, force models, driver routines, etc.
Common blocks, arrays, parameters, etc.
Body forces, initial particle distribution, pre-simulation, etc.
Input file, options, etc.
![Page 22: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/22.jpg)
CCMT22
Main Datastructures
• rpart(i,j) and ipart(i,j)– Real (r) and integer (i) properties for the jth particle– Index i is for properties (see common block /ptpointers/ )
• ptw(i,j,k,e,l)– Feedback two-way coupling from particles to fluid– (i,j,k)th grid point of eth element for lth feedback field
• rpart(jx,i) is x location of ith particle• ipart(je0,i) is local element number
of ith particle
![Page 23: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/23.jpg)
CCMT23
Lagrangian Phase
Position:
Momentum:
Energy:
• For a single particle:V
x
y
z
X
u,Tf,p,ρf
Hydrodynamic force on particle
Collisional force
Body force
Hydrodynamic heat transfer
Tp
Mp
![Page 24: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/24.jpg)
CCMT24
Note:
Hydrodynamic Models
Quasi-steady:
Added-mass:
Undisturbed:
Quasi-steady:
Forcing
Heat Transfer
![Page 25: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/25.jpg)
CCMT25
Input file: particles.inp
• Demarcated by sections:
• Note that values given in following slides provide a good starting point to run a case (taken from example cases)
- - - - PARTICLE PHYSICAL PROPERTIES - - - - - - - - - - - - - - - -- - - - PARTICLE FORCE MODELS - - - - - - - - - - - - - - - - - - - - - -- - - - USER OPTIONS - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -- - - - PROJECTION OPTIONS - - - - - - - - - - - - - - - - - - - - - - - - -- - - - BOUNDARY TREATMENT - - - - - - - - - - - - - - - - - - - - - - - -- - - - RESTARTING - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -- - - - DEM - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
![Page 26: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/26.jpg)
CCMT26
- - - - PARTICLE PHYSICAL PROPERTIES - - - - - - - - - - - - - - - -
1000 ! total computational particles119E-6 119E-6 ! Dp , [m]293. ! Tp0 , [K]2500. ! rho_p, [kg/m^3]840.0 ! C_p,p, [J/kg K]
Total number of particles
Particle diameter (two numbers)• If same, monodisperse• If different, polydisperse (set either uniform/gaussian in cmtparticles.usrp file)
Initial particle temperature
Particle density
Particle specific heat at constant pressure
![Page 27: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/27.jpg)
CCMT27
- - - - PARTICLE FORCE MODELS - - - - - - - - - - - - - - - - - - - - - -
2 ! quasi-steady force0 ! undisturbed force0 ! added-mass force0 ! quasi-steady heat transfer0 ! undisturbed heat transfer
Quasi-steady force (cmtparticles.f: usr_particles_f_qs)• 0: off• 1: on with Reynolds #, Mach #, and Volume fraction corrections• 2: on with Reynolds # and Volume fraction corrections
Undistrubed force(cmtparticles.f: usr_particles_f_un)• 0: off• 1: on
Added-mass force (cmtparticles.f: usr_particles_f_iu)• 0: off• 1: on with Mach # and Volume fraction corrections
Quasi-steady heat transfer (cmtparticles.f: usr_particles_q_qs)• 0: off• 1: on with Reynolds # and
Prandlt # corrections
Undisturbed heat transfer (cmtparticles.f: usr_particles_q_uu)• Scaffolding, but no model now
![Page 28: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/28.jpg)
CCMT28
- - - - USER OPTIONS - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 ! time integration method4 ! coupling method1 ! interpolation method1 ! particle io method0 ! particle injection rate step0 ! particle delay time step1 ! random number seed
Time integration (cmtparticles.f: rk3_integrate)• 1: RK3
Coupling method (cmtparticles.f)• 1: One-way (particles move but no feedback force or collisions) • 2: Two-way (one-way with feedback force to fluid, spread_props_grid)• 4: Four-way (two-way with collisions between particles, usr_particles_f_col)
Interpolation (cmtparticles.f: interp_props_part_location)• 1: Reduced barycentric Lagrange• 2: Barycentric Lagrange• 3: Tri-linear
![Page 29: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/29.jpg)
CCMT29
- - - - USER OPTIONS - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 ! time integration method4 ! coupling method1 ! interpolation method1 ! particle io method0 ! particle injection rate step0 ! particle delay time step1 ! random number seed
Particle I/O (cmtparticles.f: usr_particles_io)• < 0 : Output timers• abs() = 1: Output Lagrangian values• abs() = 2: Output Planar averages (needs tuning when used)• abs() = 3: Output 2 and 3• Note: two-way field files are output automatically
Inject total particles every this many steps
Delay particles until after this time step
Random number seed (change if different random numbers are needed between consecutive runs)
![Page 30: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/30.jpg)
CCMT30
- - - - PROJECTION OPTIONS - - - - - - - - - - - - - - - - - - - - - - - - -
2 ! projection method0 ! super particle pre-iterations0.1 ! mean initial particle volume fraction0.6 ! filter width/grid spacing0.3 ! smallest element width a particle will see1E-2 ! gaussian decay percent
Projection method• 1: Top-hat by counting number of particles in element• 2: Gaussian filter
Super Particle parameters
Filter width over element spacing• Any number from 0 to 1• If < 0.5, set assume rectangular domain (next slide) to 1• If > 0.5, set assume rectangular domain (next slide) to 2Smallest element width• Used if you want a small filter width• If larger than element sizes, automatically reset to smallest element widthTo what percent is Gaussian clipped at
![Page 31: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/31.jpg)
CCMT31
- - - - PROJECTION OPTIONS - - - - - - - - - - - - - - - - - - - - - - - - -
2 ! assume rectangular domain-1 1 ! xL_rect, xR_rect-1 -1 ! yL_rect, yR_rect-1 -1 ! zL_rect, zR_rect
Assume rectangular domain• 1: At most 7 ghost particles created, filter width/element spacing < 0.5• 2: At most 26 ghost particles created, filter width/element spacing > 0.5
X Boundary Conditions for Particles
Y Boundary Conditions for Particles
Z Boundary Conditions for Particles
• 0 : Periodic (need both left and right)• 1 : Outflow (particles leave domain)• -1: Wall (four-way coupled particles
will collide with wall, < four-way coupled will be treated as outflow)
![Page 32: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/32.jpg)
CCMT32
- - - - RESTARTING - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
5000 ! particle restart output step0 ! particle restart read last step
Every this number of steps, restart files will be output
If different than zero, particles will read this number rpartxyz and rpartdatafile to restart
- - - - DEM - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
10000 ! particle injection rate pre-simulation DEM step1E1 ! ksp0.85 ! coeffecient of restitution
Not really used anymore
Spring coefficient for collisions
Restitution coefficient for collisions
![Page 33: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/33.jpg)
CCMT33
Adding Particles to Fluid Case
• Make sure particle git code is checked out• Edit .rea file
– Param 1: initial fluid density (interpolated after)– Param 2: fluid dynamic viscosity
• Edit SIZE file– Make sure llpart is large enough to fit at most
llpart particles per processor (stay much below this in practice)
![Page 34: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/34.jpg)
CCMT34
Adding Particles to Fluid Case
Edit .usr file• Add forcing to userf()• Add to top of file:
– include "cmtparticles.usrp"
subroutine userf (ix,iy,iz,eg)include 'SIZE'include 'TOTAL'include 'NEKUSE'include 'CMTPART'include 'CMTDATA’integer e,ege = gllel(eg)
! particle forcingif (two_way .ge.2) then
if (istep .gt. time_delay) thenffx = ptw(ix,iy,iz,e,1)ffy = ptw(ix,iy,iz,e,2)ffz = ptw(ix,iy,iz,e,3)qvol= ptw(ix,iy,iz,e,5) + rhs_fluidp(ix,iy,iz,e,4) elseffx = 0.0ffy = 0.0ffz = 0.0endif
elseffx = 0.0ffy = 0.0ffz = 0.0
endif
returnend
![Page 35: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/35.jpg)
CCMT35
Adding Particles to Fluid Case• Move and edit cmtparticles.usrp file
– Set initial placement of particles (place_particles)
• Move and edit particles.inp file into directory
• Make and run as usual
![Page 36: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/36.jpg)
CCMT36
Output Files• Given input settings, output files are created:
– If > one-way coupling, files with ptw prefix are created
• Run visnek on ptwcasename in directory to get ptwcasename.nek5000 file, which is field file that can be opend in VisIt/ ParaView (certain versions)
– If Lagrangian output is set, partdata and partxyzfiles are output
• Run ./partview in directory to convert binary files to npartxyz ASCII files to be read by VisIt/ ParaView
– If restart files output, rpartdata and rpartxyz files created
![Page 37: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/37.jpg)
CCMT37
Visualization Demonstration (VisIt)• Eulerian Data
– Pseudocolor– Volume rendering– Light Sources– Clipping (box)– Three-slice– Bounding box– Annotations
• Lagrangian Data– Points– Sphere geometry– Bounding box
Sync time datahttps://wci.llnl.gov/simulation/computer-codes/visit/manuals
![Page 38: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/38.jpg)
CCMT
CCMT
Do you have any questions?
![Page 39: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/39.jpg)
CCMT
CCMT
Running CMT-nek
Goran Marjanovic and Bradford Durant
11/29/2017
![Page 40: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/40.jpg)
CCMT40
BEFORE WE BEGINYOU NEED, ON YOUR UNIX-LIKE SYSTEM, git make (e.g. GNU buildutils) Fortran 77 and ANSI C compilers (e.g. GCC & gfortran, Intel Fortran & C) MPI and wrappers (e.g. mpich, openmpi)
![Page 41: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/41.jpg)
CCMT
Getting Started
1. cd ~2. git clone https://github.com/UFCCMT/Nek5000.git -b master3. git clone https://github.com/UFCCMT/NekExamples.git -b master4. cd Nek5000/tools5. “./maketools all” or “./maketools genbox” and “./maketools genmap”
1. Change compiler options2. export PATH=“$HOME/Nek5000/bin:$PATH”
6. cd $HOME/Nek5000/short_tests/CMT/inviscid_vortex/7. cp $HOME/Nek5000/bin/makenek .
1. Open makenek, change source_root, PPLIST=“CMTNEK”
8. ./makenek pvort
![Page 42: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/42.jpg)
CCMT42
Running
SESSION.NAMEcase/full/path/to/case/dir/with/trailing/slash/
<case>.rea
<case>.map
SESSION.NAME
nek5000 *0.f0*
> logfile
Solution files (big)
Lots of stdout(redirect it)
Invoking nek5000– “./nek5000 >logfile” for serial runs– “nek5000” is executable arg to mpirun, mpiexec
e.g. “mpirun –np 128 ./nek5000 >logfile”
Every CMT-nek case needs these things1. <case>.rea file mesh, BC, runtime settings2. <case>.map file element connectivity graph3. <case>.usr file Fortran 77 source code4. SIZE file parameters, dimensions5. makenek script build script6. SESSION.NAME makes running easier
![Page 43: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/43.jpg)
CCMT43
CMT-nek geography
An element’s worth ofStorage isnx1 x ny1 x nz1 doubles
CMT-nek computes a d-dimensional gas flow field ina domain covered by a mesh of NEL elements• Each element is hexahedral, determined by 2d vertices
and face curvature specification.• Each element has nx1d grid points inside of iton np MPI tasks
Whole domain in.rea filenek uses the term “global” to refer to to the whole domain
.rea file
CMT-nek is a SIMD Fortran 77 and C code using MPI-1• Procedures act on data
for nelt=NEL/np elements
nelt of these stores 1 MPI task’s worth of data (for a given variable, or grid points, etc.)
SIZE file
![Page 44: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/44.jpg)
CCMT
SIZE fileSIZEldim = 2 or 3 lx1 – controls polynomial order of approximation (N = lx1 – 1)lxd – polynomial order of over-integration/dealiasing for convective terms. In general, lxd = 3*(lx1-1)/2lx2 = lx1 for CMT-neklelg - upper bound on elementslpmax - max number of processorslpmin - min number of processorsldimt - max number of auxillary fields (temperature, scalars, etc)
memory ~ E(N+1)^3 points x 400 (wds/pt) x 8 (bytes/wd)
![Page 45: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/45.jpg)
CCMT
usr fileCASENAME.USR
Fortran subroutines that allow direct access to all runtime variables
1. uservp – variable properties (see Jason’s talk for artificial viscosity)
2. usrdat – called right after geometry is loaded and before GLL points are
distributed. Elements, BCs, can be modified here
3. usrdat2 – called after GLL points are distributed and allows for only
affine transformations of geometry
4. userf – forcing term in momentum equation
5. userq – source term in energy/passive scalar equation
6. userbc, useric – boundary and initial conditions
7. userchk – called once per processor after each time step
![Page 46: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/46.jpg)
CCMT46
Running CMT-nek
*Unless NEL> lelg, the usr file expects different boundary conditions, np>lp,or you are migrating from a 2D case to a 3D case or vice-versa
I changed… Do I need to…
quantity Which lives in
recompile? re-rungenmap?
lx1,ly1,lz1 SIZE Y N
lxd,lyd,lzd SIZE Y N
ldim SIZE, .rea Y Y
np Arg to mpirun N* N*
NSTEPS .rea file N NDT .rea file N NIOSTEP .rea file N NUser file .usr file Y NThe entire mesh
.rea file N* Y
nek5000 has some unusual constraints on the compiled executable; it may not be obvious what changes are needed for different cases
Given NEL< lelt x np, the same executable that ran on 4 tasks can run on 500,000 MPI tasks.NSTEPS=#stepsDT=physical sim time stepIOSTEP=write *0.f0* files # step
Don’t mess with the usr file yet.Oddly, within limits*, you can change the whole rea file without recompiling.
![Page 47: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/47.jpg)
CCMT
Data layout in Nek1. Typical loop structure
2. Nek5000 structure
3. Simple low level loop routine
![Page 48: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/48.jpg)
CCMT
Routines of Interestcore/math.f
![Page 49: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/49.jpg)
CCMT
Functions of Interest
![Page 50: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/50.jpg)
CCMT
Variables of Interest
![Page 51: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/51.jpg)
CCMT
Other Variables of Interest
![Page 52: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/52.jpg)
CCMT
CCMT
Do you have any questions?
![Page 53: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/53.jpg)
CCMT
CCMT Lessons Learnt from CCMT Simulations
Yash Mehta, Frederick Ouellet and Rahul Koneru
CMT-Nek Workshop, Fall 2017
![Page 54: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/54.jpg)
CCMT| 54
Lesson Learnt from CCMT’s Simulations Grid Generation
Currently we use 2-3 tools to generate unstructured tetrahedral grids for fully resolved simulations
We had to write scripts to automate grid generation
Rocflu’s “original” distribution has some scripts to convert grid from one form to the other and generate some simple grids (box)
Pre-Processing
Partitioning and Initialization routines run outside the solver routine
Depending on the size of the simulations these routines can take lot of computational time
At times we had issues that these routines take longer than the walltime on super-computers at the labs
Elements per core
Rocflu can’t run if there are too few elements per core
![Page 55: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/55.jpg)
CCMT| 55
Lesson Learnt from CCMT’s Simulations Post Processing
User can select solution dump frequency in Rocflu
Flow field data is written by each core for each selected time
Post processing routine merges data from all the cores into one file for each time
Higher the number of cores, longer it takes for this routine to run
Safe Restart
Rocflu can be restarted from the time for which solution files are available
There is flag which forces solution dump “x” minutes before the wall-time so that code can be restarted from that time
This flag took 6 months to debug because of issues with LLNL super-computer
Compiling the code
We had issues with compliers on some super-computers
![Page 56: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/56.jpg)
CCMT| 56
Lesson Learnt from CCMT’s Simulations
Non-EoS Lessons:
• Due to extreme initial conditions for blast problems, we must run at least the first thousand iterations at a small, fixed time step
• For reference, this value in Rocflu is 3.0e-12• This may have to be reduced in CMT-nek
• When dealing with a sharp change in particle volume fraction (𝜙𝜙𝑝𝑝 > 25%), it is sometimes necessary to smoothen the curtain edges for stability
• Upgrading to AUSM+up may be needed, it is an all-speed formulation
![Page 57: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/57.jpg)
CCMT| 57
Lesson Learnt from CCMT’s SimulationsEoS Lessons:
• Implementing the pure JWL equations should be simple
• Handling mixtures is non-trivial:
• Another conserved variable/gov. equation must be used to track each additional species
𝐷𝐷(𝜙𝜙𝑔𝑔𝜌𝜌𝑌𝑌𝑖𝑖)𝐷𝐷𝐷𝐷
= 0 (for the Euler equations)
• Then a mixture EoS algorithm is needed
• One equation model is probably the easiest to implement• This is simply a linear fit that decays the JWL eqns. into the ideal gas eqns.
• Iterative/surrogate solvers will need a little more work• There is a good chance that the iterative solver will mess with the scaling of
the code in its current form
• Will also need to artificially clip the mass fractions if they dip below zero slightly
![Page 58: Organizers: Bertrand Rollin and Jason Hackl Agenda · :4248-4267. cmt/artvisc.f. subroutine entropy_viscosity. call compute_entropy(tlag) call entropy_residual(tlag) call wavevisc(t(1,1,1,1,3))](https://reader031.vdocument.in/reader031/viewer/2022011917/5fed9e976bfca44fda3337a4/html5/thumbnails/58.jpg)
CCMT
CCMT
Do you have any questions?