using the petsc linear solvers lois curfman mcinnes in collaboration with satish balay, bill gropp,...
TRANSCRIPT
![Page 1: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/1.jpg)
Using the PETSc Linear Solvers
Lois Curfman McInnesin collaboration with
Satish Balay, Bill Gropp, and Barry Smith
Mathematics and Computer Science DivisionArgonne National Laboratoryhttp://www.mcs.anl.gov/petsc
Cactus TutorialSeptember 30, 1999
![Page 2: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/2.jpg)
PETSc: Portable, Extensible Toolkit for Scientific Computing
• Focus: data structures and routines for the scalable solution of PDE-based applications
• Freely available and supported research code • Available via
http://www.mcs.anl.gov/petsc• Usable in C, C++, and Fortran77/90 (with minor
limitations in Fortran 77/90 due to their syntax)• Users manual in Postscript and HTML formats• Hyperlinked manual pages for all routines • Many tutorial-style examples• Support via email: petsc-
![Page 3: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/3.jpg)
Computation and Communication KernelsMPI, MPI-IO, BLAS, LAPACK
Profiling Interface
PETSc PDE Application Codes
Object-OrientedMatrices, Vectors, Indices
GridManagement
Linear SolversPreconditioners + Krylov Methods
Nonlinear Solvers,Unconstrained Minimization
ODE Integrators Visualization
Interface
PDE Application Codes
![Page 4: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/4.jpg)
CompressedSparse Row
(AIJ)
Blocked CompressedSparse Row
(BAIJ)
BlockDiagonal(BDIAG)
Dense Other
Indices Block Indices Stride Other
Index SetsVectors
Line Search Trust Region
Newton-based MethodsOther
Nonlinear Solvers
AdditiveSchwartz
BlockJacobi
Jacobi ILU ICCLU
(Sequential only)Others
Preconditioners
EulerBackward
EulerPseudo Time
SteppingOther
Time Steppers
GMRES CG CGS Bi-CG-STAB TFQMR Richardson Chebychev Other
Krylov Subspace Methods
Matrices
PETSc Numerical Components
![Page 5: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/5.jpg)
Linear iterations
0
1000
2000
3000
128 256 384 512 640 768 896 1024
Nonlinear iterations
01020304050
128 256 384 512 640 768 896 1024
Execution time
0
1000
2000
128 256 384 512 640 768 896 1024
Aggregate Gflop/s
0
40
80
128 256 384 512 640 768 896 1024
Mflops/s per processor
020406080
100
128 256 384 512 640 768 896 1024
Efficiency
0
1
128 256 384 512 640 768 896 1024
600 MHz T3E, 2.8M vertices
Sample Scalable Performance
• 3D incompressible Euler• Tetrahedral grid• Up to 11 million unknowns • Based on a legacy NASA code, FUN3d, developed by W. K. Anderson• Fully implicit steady-state
• Newton-Krylov-Schwarz algorithm with pseudo-transient continuation• Results courtesy of Dinesh Kaushik and David Keyes, Old Dominion University
![Page 6: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/6.jpg)
PETSc
ApplicationInitialization
Evaluation of A and bPost-
Processing
SolveAx = b PC KSP
Linear Solvers (SLES)
PETSc codeCactus code
Linear PDE Solution
Cactus driver code
![Page 7: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/7.jpg)
Vectors
• Fundamental objects for storing field solutions, right-hand sides, etc.
• VecCreateMPI(...,Vec *)– MPI_Comm - processors that share the
vector– number of elements local to this processor– total number of elements
• Each process locally owns a subvector of contiguously numbered global indices
proc 3
proc 2
proc 0
proc 4
proc 1
![Page 8: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/8.jpg)
Sparse Matrices
• Fundamental objects for storing linear operators (e.g., Jacobians)
• MatCreateMPIAIJ(…,Mat *)– MPI_Comm - processors that share the
matrix– number of local rows and columns– number of global rows and columns– optional storage pre-allocation
information
• Each process locally owns a submatrix of contiguously numbered global rows.
proc 3proc 2proc 1
proc 4
proc 0
![Page 9: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/9.jpg)
SLES Solver Context Variable
• Key to solver organization• Contains the complete algorithmic state, including
– parameters (e.g., convergence tolerance)– functions that run the algorithm (e.g., convergence monitoring
routine)– information about the current state (e.g., iteration number)
• Creating the SLES solver– C/C++: ierr = SLESCreate(MPI_COMM_WORLD,&sles); – Fortran: call SLESCreate(MPI_COMM_WORLD,sles,ierr)
• Provides an identical user interface for all linear solvers (uniprocessor and parallel, real and complex numbers)
![Page 10: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/10.jpg)
Basic Linear Solver Code (C/C++)
SLES sles; /* linear solver context */Mat A; /* matrix */Vec x, b; /* solution, RHS vectors */int n, its; /* problem dimension, number of iterations */
MatCreate(MPI_COMM_WORLD,n,n,&A); /* assemble matrix */VecCreate(MPI_COMM_WORLD,n,&x); VecDuplicate(x,&b); /* assemble RHS vector */
SLESCreate(MPI_COMM_WORLD,&sles); SLESSetOperators(sles,A,A,DIFFERENT_NONZERO_PATTERN);SLESSetFromOptions(sles);SLESSolve(sles,b,x,&its);
![Page 11: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/11.jpg)
Basic Linear Solver Code (Fortran)SLES sles Mat AVec x, binteger n, its, ierr
call MatCreate(MPI_COMM_WORLD,n,n,A,ierr) call VecCreate(MPI_COMM_WORLD,n,x,ierr)call VecDuplicate(x,b,ierr)
call SLESCreate(MPI_COMM_WORLD,sles,ierr)call SLESSetOperators(sles,A,A,DIFFERENT_NONZERO_PATTERN,ierr)call SLESSetFromOptions(sles,ierr)call SLESSolve(sles,b,x,its,ierr)
C then assemble matrix and right-hand-side vector
![Page 12: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/12.jpg)
Customization Options
• Procedural Interface– Provides a great deal of control on a usage-by-usage
basis; gives full flexibility inside an application• SLESGetKSP(SLES sles,KSP *ksp)• KSPSetType(KSP ksp,KSPType type)• KSPSetTolerances(KSP ksp,double rtol,double
atol,double dtol, int maxits)
• Command Line Interface– Applies same rule to all queries via a database; enables
complete control at runtime, with no extra coding• -ksp_type [cg,gmres,bcgs,tfqmr,…]• -ksp_max_it <max_iters>• -ksp_gmres_restart <restart>
![Page 13: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/13.jpg)
Recursion: Specifying Solvers for Schwarz Preconditioner Blocks
• -sub_pc_type lu-mat_reordering [nd,1wd,rcm,qmd]-mat_lu_fill <fill>
• -sub_pc_type ilu -sub_pc_ilu_levels <levels>
• Can also use inner iterations, e.g.,-sub_ksp_type gmres -sub_ksp_rtol <rtol> -sub_ksp_max_it <maxit>
![Page 14: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/14.jpg)
Linear Solvers: Monitoring Convergence
• -ksp_monitor - Prints preconditioned residual norm
• -ksp_xmonitor - Plots preconditioned residual norm
• -ksp_truemonitor - Prints true residual norm || b-Ax ||
• -ksp_xtruemonitor - Plots true residual norm || b-Ax ||
• User-defined monitors, using callbacks
![Page 15: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/15.jpg)
SLES: Selected Preconditioner Options
Functionality Procedural Interface Runtime Option
Set preconditioner type PCSetType( ) -pc_type [lu,ilu,jacobi, sor,asm,…]
Set level of fill for ILU PCILULevels( ) -pc_ilu_levels <levels>Set SOR iterations PCSORSetIterations( ) -pc_sor_its <its>Set SOR parameter PCSORSetOmega( ) -pc_sor_omega <omega>Set additive Schwarz variant
PCASMSetType( ) -pc_asm_type [basic, restrict,interpolate,none]
Set subdomain solver options
PCGetSubSLES( ) -sub_pc_type <pctype> -sub_ksp_type <ksptype> -sub_ksp_rtol <rtol>
And many more options...
![Page 16: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/16.jpg)
SLES: Selected Krylov Method Options
And many more options...
Functionality Procedural Interface Runtime Option
Set Krylov method KSPSetType( ) -ksp_type [cg,gmres,bcgs, tfqmr,cgs,…]
Set monitoring routine
KSPSetMonitor() -ksp_monitor, –ksp_xmonitor, -ksp_truemonitor, -ksp_xtruemonitor
Set convergence tolerances
KSPSetTolerances( ) -ksp_rtol <rt> -ksp_atol <at> -ksp_max_its <its>
Set GMRES restart parameter
KSPGMRESSetRestart( ) -ksp_gmres_restart <restart>
Set orthogonalization routine for GMRES
KSPGMRESSetOrthogon alization( )
-ksp_unmodifiedgramschmidt -ksp_irorthog
![Page 17: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/17.jpg)
SLES: Runtime Script Example
![Page 18: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/18.jpg)
Viewing SLES Runtime Options
![Page 19: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/19.jpg)
Providing Different Matrices to Define Linear System and
Preconditioner
• Krylov method: Use A for matrix-vector products• Build preconditioner using either
– A - matrix that defines linear system– or P - a different matrix (cheaper to assemble)
• SLESSetOperators(SLES sles, – Mat A, – Mat P, – MatStructure flag)
Precondition via: M A M (M x) = M bRL
-1R-1
L-1
Solve Ax=b
![Page 20: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/20.jpg)
Matrix-Free Solvers
• Use “shell” matrix data structure– MatCreateShell(…, Mat *mfctx)
• Define operations for use by Krylov methods– MatShellSetOperation(Mat mfctx,
• MatOperation MATOP_MULT, • (void *) int (UserMult)(Mat,Vec,Vec))
• Names of matrix operations defined in petsc/include/mat.h
• Some defaults provided for nonlinear solver usage
![Page 21: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/21.jpg)
User-defined Customizations
• Restricting the available solvers– Customize PCRegisterAll( ), KSPRegisterAll( )
• Adding user-defined preconditioners via – PCShell preconditioner type
• Adding preconditioner and Krylov methods in library style– Method registration via PCRegister( ), KSPRegister( )
• Heavily commented example implementations– Jacobi preconditioner:
petsc/src/sles/pc/impls/jacobi.c– Conjugate gradient: petsc/src/sles/ksp/impls/cg/cg.c
![Page 22: Using the PETSc Linear Solvers Lois Curfman McInnes in collaboration with Satish Balay, Bill Gropp, and Barry Smith Mathematics and Computer Science Division](https://reader035.vdocument.in/reader035/viewer/2022081603/56649eb75503460f94bc1077/html5/thumbnails/22.jpg)
SLES: Example Programs
• ex1.c, ex1f.F - basic uniprocessor codes • ex2.c, ex2f.F - basic parallel codes • ex11.c - using complex numbers• ex4.c - using different linear system and
preconditioner matrices• ex9.c - repeatedly solving different linear
systems• ex15.c - setting a user-defined preconditioner
for more information: http://www.mcs.anl.gov/petscAnd many more examples ...
Location: petsc/src/sles/examples/tutorials/