linear solvers for reservoir...
TRANSCRIPT
LINEAR SOLVERS FOR RESERVOIR
SIMULATION
CERFACS
Serge Gratton, Pavel Jiranek, Xavier Vasseur
INRIA-LJLL
Hussam Al Daas, Laura Grigori
TOTAL E&P
Pascal Hénon
SUMMARY
CEMRACS 2016
● Introduction
● Works on AMG for reservoir (collaboration with CERFACS)
● PhD 2015-2017: Enlarge GMRES (INRIA: H. Al Daas, L. Grigori)
● Conclusions and Prospects
2
INTRODUCTION
CEMRACS 2016 3
RESERVOIR SIMULATION : PURPOSES
4
● Estimation of Recovery Factor, Production plateau, EOR, modelling physic
● We have a limited knowledge of underground properties (fault, kr, ..)
● A big part of the work consists in « history matching » : needs many runs
CEMRACS 2016
DYNAMIC SIMULATOR MAIN EQUATIONS TO BE SOLVED
Conclusion:
For each cell and for each component, convergence is evaluated by measuring
the non-linear residual Rfl
5 Advanced Reservoir Simulation - Dec 2014 5
LINEAR EQUATION RESOLUTION
Conclusion: Most of the CPU is spent on linear resolution
6 Advanced Reservoir Simulation - Dec 2014 6
PARALLEL PERFORMANCE
● Scalability of a few models : (upper one =13M cells BO: around 50k
cells/processors)
7
Source : Schlumberger : SPE ACTE, Oct 31th 2011
CEMRACS 2016
SCALABILITY BOTTLENECK IN RESERVOIR SIMULATION
● Distributed memory framework (MPI)
● Load balancing : work per reservoir cell varies during the simulation.
● Linear solver method CPR-AMG is scalable with number of unknowns (weak
scalability) but poorly scalable with number of processors (strong scalability)
8
CEMR
ACS
2016
0
10
20
30
40
50
60
IX
Property calc
Linearizer
Linear solver
Updater
Boundary cells
B Interior cells
Ghost cells
● The Constraint Pressure Residual (CPR) solver : this solver specific to reservoir is a
two stages method (John Wallis and co. SPE 1985) :
- M1 : 1er stage is global : find approximate pressure (near-elliptic problem : AMG)
- M2 : 2nd stage is applied on the ARR system (eg BILU(0)) : block preconditioner
LINEAR SOLVER : CONSTRAINT PRESSURE RESIDUAL
PPPSPW
SPSSSW
WPWSWW
AAA
AAA
AAA
PPPS
SPSS
RR
t
AA
AAPAP ..
RRRW
WRWW
AA
AA
WWA
RWA
WPAWWA
RWA
WPA
1
1
1
1
1
2
1 ).( MMAIMMCPR
9 CEMRACS 2016
IMPROVING AMG FOR RESERVOIR
SIMULATION
CEMRACS 2016 10
CEMRACS 2016
RESERVOIR SIMULATION: ONE ITERATION OF CPR-AMG
1
1
1
1
1
2
1 ).( MMAIMMCPR
Pressure pseudo-
decoupling
11
The critical performance bottleneck in reservoir is the pressure solver part
CPR
AMG update
(different grid
levels)
SCALABILITY IN A RESERVOIR SIMULATION
CEMRACS 2016 12
● Example on a one million grid blocks BO model (SPE10) in TPP
● The figure represents TCPU*number of procs (perfect == same
length)
● The scalability bottleneck is clearly in AMG (setup + solve)
0,00E+00
2,00E+02
4,00E+02
6,00E+02
8,00E+02
1,00E+03
1,20E+03
1,40E+03
1,60E+03
1,80E+03
2,00E+03
1 2 4 8 16 32 64 128
CP
U t
ime (
s)
Number of MPI process (x8 = Nb cores)
AMG Solve
AMG Setup
CPR
Reservoir
● Pavel Jiranek : post-doc CERFACS 2012 – 2014 then 15 month in CSTJF until 11-30-2015
● Hybrid strategy Aggregation/RS is more efficient than standard RS strategy
● PMG package relies on MPI/Thread parallelization
● Ref: « Reducing complexity of algebraic multigrid by aggregation », in Num. Lin. Algebra.
with applications
● AMG based on Aggregation is efficient for the two first levels
PMG: HYBRID AGGREGATION/CLASSIC COARSENING
13 CEMRACS 2016
Classic coarsening Aggregation coarsening
RESULTS: PMG VS BOOMERAMG VS SAMG
CEMRACS 2016 14
● BoomerAMG from Lawrence Livermore (DOE).
● SAMG from Fraunhofer Institute.
● Minimum setup consists in keeping same grid interpolators for
several linear system (we limit to the same time step)
● On small-medium cases (< 1’000’000 cells); no significant
improvement over BoomerAMG for total simulation time.
AMG CPU Time
x1.31
RESULTS: PMG VS BOOMERAMG (MPI+THREAD)
CEMRACS 2016 15
● Viscous fingering study.
● Tuned parameter for BoomerAMG: aggressive coarsening
● 60’000’000 cells model. Test up to 8192 cores (1024 processors of
PANGEA1).
● The speed-up is more important (first layers of AMG represents more
computations).
x5
x2
1024*8 = 8192 cores
ENLARGED KRYLOV METHOD
CEMRACS 2016 16
Slides courtesy of Laura Grigori and Hussam Al Daas
GMRES: SCALABILITY LIMITATION
CEMRACS 2016 17
ENLARGE KRYLOV METHOD (L. GRIGORI ET AL 2014)
CEMRACS 2016 18
ENLARGED GMRES: ALGORITHM
CEMRACS 2016 19
● TSQR = Tall Skinny QR
ENLARGED GMRES: DETAILS
CEMRACS 2016 20
ENLARGED GMRES: EXPERIMENTS (1/2)
CEMRACS 2016 21
ENLARGED GMRES: EXPERIMENTS (2/2)
CEMRACS 2016 22
CONCLUSION & PROSPECTS
CEMRACS 2016 23
CONCLUSION & PROSPECTS
CEMRACS 2016 24
● Works on AMG: some improvements for large models in parallel
have been obtained by an hybrid coarsening strategy.
● Parallel implementation (C++) of EGMRES method and integration in
a reservoir simulator
● Experiment EGMRES for pressure solve (with BJ precond) in a
reservoir simulator