scidac tops petsc work
DESCRIPTION
SciDAC TOPS PETSc Work. SciDAC TOPS Developers Satish Balay Chris Buschelman Matt Knepley Barry Smith Hong Zhang Other PETSc Team Members Lois Curfman McInnes, Bill Gropp, Dinesh Kaushik. http://www.mcs.anl.gov/petsc [email protected]. PETSc. - PowerPoint PPT PresentationTRANSCRIPT
1
SciDAC TOPS PETSc Work
http://www.mcs.anl.gov/[email protected]
SciDAC TOPS DevelopersSatish Balay
Chris BuschelmanMatt KnepleyBarry SmithHong Zhang
Other PETSc Team MembersLois Curfman McInnes, Bill Gropp, Dinesh Kaushik
2
PETSc
• Portable, Extensible, Software for the scalable solution of nonlinear PDEs
• Focusing on algebraic solvers– Preconditioners, multigrid, Krylov methods,
Newton, FAS, time integrators
3
Activities• Derivative calculations – ADIC/ADIFOR
– Matrix-free preconditioners– FAS (future)
• Accessing other software– HYPRE preconditioner usage– RAMG,SAMG– SuperLU– ESI
• SIDL/Babel/CCA – PETSc 3.0 plans• SIDL/Babel MPI
4
Derivatives
• Automatic tools ADIC/ADIFOR are great but require manual processing (painful when developing code)
• Would like to have a variety of derivatives “for free” (or nearly)– Matrix element based Newton which requires explicit
sparse Jacobian– Matrix-free Newton– SOR preconditioners and FAS require one equation at
at time
5
To Fully Automate
• Code comments to indicate what needs to be processed
• Predefined calling sequences
• Libraries that can use generated code
• Make system automatically generates required derivative code
6
../* Process adiC: FFL */int FFL (DALocalInfo *info,Field **x, Field **f,void *ptr) { AppCtx *user = (AppCtx*)ptr; ..
DMMGSetSNESLocal(dmmg,FFL,0,ad_FFL,admf_FFL);
typedef struct { PassiveReal lidvelocity,prandtl,grashof; PetscTruth draw_contours;} AppCtx;
typedef struct { double u,v,omega,temp;} Field;
7
> make BOPT=g_c++ ex19Running adiC on function FFLExtracting structure FieldExtracting structure AppCtxExtracting function FFL
adiC -a -d gradient -i /sandbox/bsmith/petsc/bmake/adic.init ex19.c.tmpADIC 1.1 beta 4 08/07/00 18:06:31This software was created in the course of academic and/or researchendeavors by Argonne National Laboratory. See the disclaimer in the
g++ -fPIC -o ex19.o -c -Dad_GRAD_MAX=36 -I/sandbox/bsmith/petsc/include/adic -g -I/sandbox/bsmith/petsc ..
8
Matrix-free Multiply
• Differencing (well understood)
• AD
int mf_FFL(DALocalInfo *info,mf_Field **x, mf_Field **f,void *ptr) {
typedef struct { double value;double grad[1];} mfDERIV_TYPE;typedef struct { mfDERIV_TYPE u, v, omega, temp;}
9
Matrix-free Multigrid Preconditioners
• Krylov smoothing with – no preconditioner– (point block) Jacobi preconditioner– (point block) SOR preconditioning
10
Matrix-free Jacobi/SOR
• If function is provided an “(block) equation at a time” matrix-free multiply can be done an equation at a time– can be computed by providing a seed of – For SOR
• Similar computations with differencing
iiJ ie1
:( )i i ii i iu u J b J u
11
FAS – Full Approximation Scheme (Nonlinear Multigrid)
• Smoothing uses Newton to solve a single (block) equation at a time– Need the same “(block) equation at a time” and
its derivative as for matrix-free Newton!
12
We are developing a common software infrastructure and user interface for many Newton methods and FASwhich can use AD or differencing for derivatives.
13
Activities• Derivative calculations – ADIC/ADIFOR
– Matrix-free preconditioners– FAS (future)
• Accessing other software– HYPRE preconditioner usage– RAMG, SAMG– SuperLU– ESI
• SIDL/Babel/CCA – PETSc 3.0 plans• SIDL/Babel MPI
14
Philosophy on Solvers
• Use any solver without changing the application code
• Add to application code to take advantage of special solver features (optimizations)– but additions cannot break other solvers
PCILUSetDropTolerance(pc,1.e-5);PCSPAISetEpsilon(pc,1.e-2);
15
HYPRE-PETSc Matrix Element based preconditioners
• No vector copies
• Currently a matrix copy (plan to remove)
-pc_type hypre -pc_hypre_type <pilut,parasails,boomerAMG,euclid>-pc_hypre_pilut_tol 1.e-4-pc_hypre_parasails_thresh .1-pc_hypre_parasails_filter .2
16
RAMG, SAMG – PETScSequential Preconditioners
• AMG – written by John Ruge, Klaus Stueben, and Rolf Hempel. A classic scalar PDE AMG code.
• SAMG – written by Klaus Stueben. A complete rewrite that supports systems of PDEs (interface coming soon).– PETSc interfaces provided by Domenico
Lahaye
17
SuperLU_Dist• MPI parallel LU factorization and solve written by Sherry Li and Jim Demmel
• Henry Tufo – Paul Fischer XYT fast direct parallel solver for “small problems”, i.e. the coarse grid problem in multigrid
-mat_aij_superlu_dist-mat_aij_superlu_dist_colperm <MMD_AT_PLUS_A,NATURAL,MMD_ATA,COLAMD>
18
ESI Equation Solver Interface
• A C++ “standard” of abstract classes for setting up and solving linear systems
• At least three implementations– Trilinos (Sandia, NM)– ISIS (Sandia, CA)– PETSc
19
PETSc ESI Goal
• Allow PETSc to use matrices and solvers written by others in ESI
• Allow anyone to use PETSc matrices and solvers from ESI– Achieved by using two sets of wrappers
• PETSc objects “look like” ESI objects
• ESI objects “look like” PETSc objects
20
-is_type esi –is_esi_type Petra_ESI_IndexSpace-vec_type esi -vec_esi_type Petra_ESI_Vector -mat_type esi –mat_esi_type Petra_ESI_CSR_Matrix
PETSc using Trilinos objects
ESI using PETSc objects
esi::Vector<double,int> *v = new esi::petsc::Vector<double,int>(petscvector)esi::Operator<double,int> *o = new esi::petsc::Matrix<double,int>(petscmatrix)
21
Activities• Derivative calculations – ADIC/ADIFOR
– Matrix-free preconditioners– FAS (future)
• Accessing other software– HYPRE preconditioner usage– RAMG, SAMG– SuperLU– ESI
• SIDL/Babel/CCA – PETSc 3.0 plans• SIDL/Babel MPI
22
SIDL/Babel (LLNL)
• Scientific Interface Definition Language
• Babel – generates language stubs for– C, C++, Fortran 77, Python, Java, …
• Provides – Object model– Exception (error) model
• Immature, but very exciting development
23
PETSc 3.0
• SIDL/Babel for inter-language, object model and exception model
• Multiple precisions – Single for preconditioners?
• Vector and matrix sizes mutable– Adaptive mesh refinement
• Not a rewrite, just a rewrap!
24
Activities• Derivative calculations – ADIC/ADIFOR
– Matrix-free preconditioners– FAS (future)
• Accessing other software– HYPRE preconditioner usage– RAMG, SAMG– SuperLU– ESI
• SIDL/Babel/CCA – PETSc 3.0 plans• SIDL/Babel MPI
25
Conclusions
• Many new tools and solvers
• Accessible via – PETSc 2.0 interface– ESI– (coming) SIDL/Babel/CCA
• We are happy to support users, especially those interested in nontrivial problems