chombo: a framework for structured-grid methods...
TRANSCRIPT
![Page 1: Chombo: A framework for structured-grid methods …flash.uchicago.edu/~dubey/Feb28_2011/Colella_chombo.pdfPhillip Colella Chombo: A framework for structured-grid methods for PDE on](https://reader031.vdocument.in/reader031/viewer/2022022423/5a9f2b6c7f8b9a8e178c6a3a/html5/thumbnails/1.jpg)
Phillip Colella
Chombo: A framework for structured-grid methods for PDE on high-performance
computers
![Page 2: Chombo: A framework for structured-grid methods …flash.uchicago.edu/~dubey/Feb28_2011/Colella_chombo.pdfPhillip Colella Chombo: A framework for structured-grid methods for PDE on](https://reader031.vdocument.in/reader031/viewer/2022022423/5a9f2b6c7f8b9a8e178c6a3a/html5/thumbnails/2.jpg)
Structured-Grid Methods For Partial Differential Equations
The original formulation of this method was for a uniform rectangular mesh.
![Page 3: Chombo: A framework for structured-grid methods …flash.uchicago.edu/~dubey/Feb28_2011/Colella_chombo.pdfPhillip Colella Chombo: A framework for structured-grid methods for PDE on](https://reader031.vdocument.in/reader031/viewer/2022022423/5a9f2b6c7f8b9a8e178c6a3a/html5/thumbnails/3.jpg)
Structured-Grid Methods For Partial Differential Equations
Mapped multiblock grids Embedded boundary methods
![Page 4: Chombo: A framework for structured-grid methods …flash.uchicago.edu/~dubey/Feb28_2011/Colella_chombo.pdfPhillip Colella Chombo: A framework for structured-grid methods for PDE on](https://reader031.vdocument.in/reader031/viewer/2022022423/5a9f2b6c7f8b9a8e178c6a3a/html5/thumbnails/4.jpg)
Block-Structured Adaptive Mesh Refinement (AMR) • Refined regions are organized into rectangular patches.
• Refinement in time as well as in space for time-dependent problems. • Can be applied to both explicit and implicit discretizations of PDE.
![Page 5: Chombo: A framework for structured-grid methods …flash.uchicago.edu/~dubey/Feb28_2011/Colella_chombo.pdfPhillip Colella Chombo: A framework for structured-grid methods for PDE on](https://reader031.vdocument.in/reader031/viewer/2022022423/5a9f2b6c7f8b9a8e178c6a3a/html5/thumbnails/5.jpg)
Synchronization of Multilevel Solution
• Average coarse-grid solution onto fine grid. • Correct coarse cells adjacent to fine grid to maintain conservation.
![Page 6: Chombo: A framework for structured-grid methods …flash.uchicago.edu/~dubey/Feb28_2011/Colella_chombo.pdfPhillip Colella Chombo: A framework for structured-grid methods for PDE on](https://reader031.vdocument.in/reader031/viewer/2022022423/5a9f2b6c7f8b9a8e178c6a3a/html5/thumbnails/6.jpg)
Chombo: A Software Toolkit for Structured-Grid Applications
SciDAC requirement: support a wide variety of applications that use block-structured AMR by means of a common software framework. • Mixed-language model: C++ for higher-level data structures,
Fortran for regular single-grid calculations. • Reusable components: component design based on mapping
of mathematical abstractions to classes. • Parallel performance: scalability to 100000ʼs of processors
with low-level details hidden from the applications developer. • Build on public-domain standards, e.g. HDF5. • Interoperability with other tools, e.g. PETSC. • Not a black box: users can (and do) customize, and user
changes can migrate back into trunk. Related work: FIDIL, Titanium; BoxLib.
![Page 7: Chombo: A framework for structured-grid methods …flash.uchicago.edu/~dubey/Feb28_2011/Colella_chombo.pdfPhillip Colella Chombo: A framework for structured-grid methods for PDE on](https://reader031.vdocument.in/reader031/viewer/2022022423/5a9f2b6c7f8b9a8e178c6a3a/html5/thumbnails/7.jpg)
A Software Framework for Structured-Grid Applications
• Layer 1: Data and operations on unions of rectangles - set calculus, rectangular array library (with interface to Fortran). Data on unions of rectangles, with SPMD parallelism implemented by distributing boxes to processors. Load balancing tools (e.g., SFC).
• Layer 2: Tools for managing interactions between different levels of refinement in an AMR calculation - interpolation, averaging operators, coarse-fine boundary conditions.
• Layer 3: Solver libraries - multigrid solvers on unions of rectangles, AMR hierarchies; hyperbolic solvers; AMR time stepping.
• Layer 4: Complete parallel applications.
• Utility Layer: Support, interoperability libraries.
A Layered approach to the Chombo design mitigates risk to the applications developers, insulating them from changes to implementations due to changes in platforms, optimizations to enhance performance.
![Page 8: Chombo: A framework for structured-grid methods …flash.uchicago.edu/~dubey/Feb28_2011/Colella_chombo.pdfPhillip Colella Chombo: A framework for structured-grid methods for PDE on](https://reader031.vdocument.in/reader031/viewer/2022022423/5a9f2b6c7f8b9a8e178c6a3a/html5/thumbnails/8.jpg)
Mechanisms for Reuse
• Algorithmic reuse. Identify mathematical components that cut across applications. Easy example: solvers. Less easy example: Layer 2.
• Reuse by templating data holders. Easy example: rectangular array library - array values are the template type. Less easy example: data on unions of rectangles - “rectangular array” is a template type.
• Reuse by inheritance. Control structures (Iterative solvers, Berger-Oliger timestepping) are independent of the data, operations on that data. Use inheritance to isolate the control structure from the details of what is being controlled (interface classes).
![Page 9: Chombo: A framework for structured-grid methods …flash.uchicago.edu/~dubey/Feb28_2011/Colella_chombo.pdfPhillip Colella Chombo: A framework for structured-grid methods for PDE on](https://reader031.vdocument.in/reader031/viewer/2022022423/5a9f2b6c7f8b9a8e178c6a3a/html5/thumbnails/9.jpg)
Scaling of AMR in Chombo
![Page 10: Chombo: A framework for structured-grid methods …flash.uchicago.edu/~dubey/Feb28_2011/Colella_chombo.pdfPhillip Colella Chombo: A framework for structured-grid methods for PDE on](https://reader031.vdocument.in/reader031/viewer/2022022423/5a9f2b6c7f8b9a8e178c6a3a/html5/thumbnails/10.jpg)
Optimizations for Performance
• Distributing patches to processors to maximize locality. Sort the patches by Morton ordering, and divide into equal-sized intervals.
• Overlapping local copying and MPI communications in exchanging ghost-cell data (only has an impact above 4K processors).
• Compressing the metadata that describes the distribution of boxes onto processors enabled us to scale above 10K processors.
Morton-ordered load balancing (slice through 3D grids).
Berger-Rigoutsos + recursive bisection.
![Page 11: Chombo: A framework for structured-grid methods …flash.uchicago.edu/~dubey/Feb28_2011/Colella_chombo.pdfPhillip Colella Chombo: A framework for structured-grid methods for PDE on](https://reader031.vdocument.in/reader031/viewer/2022022423/5a9f2b6c7f8b9a8e178c6a3a/html5/thumbnails/11.jpg)
Core AMR capabilities • MHD for tokamaks using allspeed projection methods (R. Samtaney, PPPL / KAUST) • Cosmology: compressible CFD + particles (F. Miniati, ETH) • Space plasmas: compressible CFD + electromagnetic, kinetic effects (G. Zank, UA-Huntsville) • Astrophysical MHD turbulence (C. McKee / R. Klein, UCB; D. Balsara, Notre Dame) • Lattice Boltzmann method for reduction of drag using polymers (B. Alder, LLNL) • HEDP applications (D. Lamb, U. of Chicago; exascale co-design project). Embedded Boundary Methods • SF Bay and Delta Hydrology modelling – shallow water (CA Department of Water Resources) • Plasma-wakefield accelerators – compressible viscous high-temperature flow (W. Leemans, LBNL) • Blood flow in cerebral arteries – fluid / solid coupling (UCB ParLab project) • Complex fluids at the microscale (UNC / UCD / LBNL multiscale math project; LBNL EFRC; SciDAC-e) • Cell modeling (UCHC Virtual Cell project) Mapped Multiblock Methods • Type II Supernovae – compressible flow (S. Woosley, UCSC; LLNL) • 4D gyrokinetic models of tokamak edge plasmas (LLNL) • Land ice model for climate simulation – Stokesean solid (LANL) • Atmospheric models for climate simulation – low-Mach number CFD (C. Jablonowski, U. Mich.; CSSEF proposal)
Current Chombo Applications
![Page 12: Chombo: A framework for structured-grid methods …flash.uchicago.edu/~dubey/Feb28_2011/Colella_chombo.pdfPhillip Colella Chombo: A framework for structured-grid methods for PDE on](https://reader031.vdocument.in/reader031/viewer/2022022423/5a9f2b6c7f8b9a8e178c6a3a/html5/thumbnails/12.jpg)
• High-order accurate discretizations, including mapped and embedded boundary methods. • AMR on mapped multiblock grids. PDE on a sphere. • Two-level parallelism on multicore systems. Test case: AMRPoisson on 1012 cells. • High-performance support for particles. • Continuing work on a variety of applications: GK plasmas, LPWA, climate, multiphase subsurface flow at the pore scale, water resources, cell biology, …
Chombo – Current Development Activities
![Page 13: Chombo: A framework for structured-grid methods …flash.uchicago.edu/~dubey/Feb28_2011/Colella_chombo.pdfPhillip Colella Chombo: A framework for structured-grid methods for PDE on](https://reader031.vdocument.in/reader031/viewer/2022022423/5a9f2b6c7f8b9a8e178c6a3a/html5/thumbnails/13.jpg)
Initial Plan: Use Chombo Layer 1and Layer 2 capabilities for core AMR data structures and operations in Flash (Chris Daley, Univ. of Chicago).
Chombo in Flash
![Page 14: Chombo: A framework for structured-grid methods …flash.uchicago.edu/~dubey/Feb28_2011/Colella_chombo.pdfPhillip Colella Chombo: A framework for structured-grid methods for PDE on](https://reader031.vdocument.in/reader031/viewer/2022022423/5a9f2b6c7f8b9a8e178c6a3a/html5/thumbnails/14.jpg)
Whatʼs Next ?
![Page 15: Chombo: A framework for structured-grid methods …flash.uchicago.edu/~dubey/Feb28_2011/Colella_chombo.pdfPhillip Colella Chombo: A framework for structured-grid methods for PDE on](https://reader031.vdocument.in/reader031/viewer/2022022423/5a9f2b6c7f8b9a8e178c6a3a/html5/thumbnails/15.jpg)
Bytes per flop going down.
• Adaptive meshes • Scalable matrix-free methods for
sparse linear systems. • Changing the discretizations to trade
bytes for flops (higher-order methods)
• Models / problems with local higher compute / memory ratio - for detailed chemistry, ratio scales like (Nspecies)2 (but coefficient varies in space and time).
![Page 16: Chombo: A framework for structured-grid methods …flash.uchicago.edu/~dubey/Feb28_2011/Colella_chombo.pdfPhillip Colella Chombo: A framework for structured-grid methods for PDE on](https://reader031.vdocument.in/reader031/viewer/2022022423/5a9f2b6c7f8b9a8e178c6a3a/html5/thumbnails/16.jpg)
Grid resolution, problem complexity going up.
Applications users want same throughput on finer grids, but CFL condition on time step for explicit methods says that dt ~ (dx)p , p at least 1. More complex multiphysics problems.
Consequences: • More implicit methods (without increased
memory footprint). • Reformulation of problem that more
effectively parallelizes across state space.
• Aggressive use of subcycling in time, locally in time, physical space, state space.
• Load balancing becomes more dynamic.