computational framework for subsurface energy and environmental modeling and simulation mary fanett...
TRANSCRIPT
Computational Framework for Computational Framework for Subsurface Energy and Subsurface Energy and
Environmental Modeling and Environmental Modeling and SimulationSimulation
Mary Fanett Wheeler, Sunil ThomasMary Fanett Wheeler, Sunil Thomas
Center for Subsurface ModelingCenter for Subsurface ModelingInstitute for Computation Engineering and SciencesInstitute for Computation Engineering and Sciences
The University of Texas at AustinThe University of Texas at Austin
Acknowledge Collaborators:
• Algorithms: UT-Austin (T. Arbogast, M. Balhoff, M. Delshad, E. Gildin, G. Pencheva, S. Thomas, T. Wildey): Pitt (I. Yotov); ConocoPhillips (H. Klie)
• Parallel Computation: IBM (K. Jordan, A. Zekulin, J. Sexton); Rutgers (M. Parashar)
• Closed Loop Optimization: NI (Igor Alvarado, Darren Schmidt)
Support of Projects: NSF, DOE, and Industrial Affiliates (Aramco, BP, Chevron, ConocoPhillips, ExxonMobil, IBM, KAUST)
Outline Introduction General Parallel Framework for Modeling Flow,
Chemistry, and Mechanics (IPARS)• Solvers• Discretizations• Multiscale and Uncertainty Quantification• Closed Loop Optimization
Formulations (IPARS-C02)
• Compositional and Thermal Computational Results
• Validation and Benchmark Problems Current and Future Work
Resources Recovery • Petroleum and natural gas recovery from
conventional/unconventional reservoirs • In situ mining • Hot dry rock/enhanced geothermal systems • Potable water supply • Mining hydrology
Societal Needs in Relation to Geological Systems
Site Restoration • Aquifer remediation • Acid-rock drainage
Waste Containment/Disposal • Deep waste injection • Nuclear waste disposal • CO2 sequestration • Cryogenic storage/petroleum/gas
Underground Construction • Civil infrastructure • Underground space • Secure structures
Petroleum Engineering
Highly Integrated Highly Integrated Multidisciplinary, Multiscale, MultiprocessMultidisciplinary, Multiscale, Multiprocess
GeologicalEng.
Geology
PhysicsMathematics
Mining Engineering
MechanicalEngineering
Computer Sciences
Geochemistry
Geophysics
Geomechanics
GeoHydrology
Civil Engineering
ExplorationCharacterization
Fluid Flow Waste
isolation
Earth StressesMechanicalRock/Soil Behavior
Mining Design/Stab, Waste,
Land Reclamation
ExplorationCharacterizationDiagnostics
HydrocarbonRecoverySimulationConstruction
Soil Mech.Rock Mech.Struct. Anal.
Drilling & Excav.,Support, Instruments
Code Development, Software Engineering
Waste isolation
Geomechanics
Complex Geosystem
ManagementOptimization and Control
Multiscale Simulation
Characterization & Imaging
Geophysical Interpretation
Uncertainty Assessment
Sensor Placement
Sensor Data Management
Petrophysical Interpretation
Multiphysics Simulation
A Powerful Problem Solving Environment
3D Visualization & Interpretation
Data Management
Long Range Vision: Characterization Long Range Vision: Characterization And Management of And Management of DiverseDiverse Geosystems Geosystems
Framework Components High fidelity algorithms for treating relevant physics:
• Locally Conservative discretizations (e.g. mixed finite element and DG)
• Multiscale (spatial & temporal multiple scales)• Multiphysics (Darcy flow, biogeochemistry, geomechanics)• Complex Nonlinear Systems (coupled near hyperbolic &
parabolic/ elliptic systems with possible discrete models)• Robust Efficient Physics-based Solvers (ESSENTIAL)• A Posteriori Error Estimators
Closed loop optimization and parameter estimation• Parameter Estimoation (history matching) and uncertainty
quantification
Computationally intense: • Distributed computing• Dynamic steering
The Instrumented Oil Field The Instrumented Oil Field
Detect and track changes in data during production.Invert data for reservoir properties.Detect and track reservoir changes.
Assimilate data & reservoir properties into the evolving reservoir model.
Use simulation and optimization to guide future production.
IPARS: Integrated Parallel and IPARS: Integrated Parallel and Accurate Reservoir SimulatorAccurate Reservoir Simulator
PHYSICS BASED SOLVERSPHYSICS BASED SOLVERS
PhysicsPhysics
Heterogeneity
Multiple Physics
K Tensor
Flow Regimes
Fractures
Well Operations
DiscretizationDiscretization
MFE
CVM
DG
MPFA
Mortar
FDM
Numerical representati
on
HPCHPC
Insights
SolversSolvers
AMG
AML
DD
Krylov
LU/ILU
Numerical Solution
RandomizedRandomizedAlgorithms
MultiresolutionMultiresolutionAnalysis
Random GraphRandom GraphTheoryTheory
ReinforcedReinforcedLearning
PhysicsPhysics-based-basedSolversSolvers
Why Multiscale? Subsurface properties vary on the
scale of millimeters Computational grids can be refined
to the scale of meters or kilometers Multiscale methods are designed to
allow fine scale features to impact a coarse scale solution• Variational multiscale finite
elements Hughes et al 1998 Hou, Wu 1997 Efendiev, Hou, Ginting et al
2004• Mixed multiscale finite elements
Arbogast 2002 Aarnes 2004
• Mortar multiscale finite elements Arbogast, Pencheva, Wheeler,
Yotov 2004 Yotov, Ganis 2008
Upscale
Domain Decomposition and MultiscaleDomain Decomposition and Multiscale
Domain Decomposition
For each stochastic realization,time step and linearization
Compute data forinterface problemCompute data forinterface problem
Subdomainsolves
Preconditiondata
Preconditiondata
Solve theinterface problem
Solve theinterface problem
Solve local problemsgiven interface valuesSolve local problems
given interface values
Applyprecond.
Multiplesubdomain
solves
Multipleprecond.
applications
Subdomainsolves
Multiscale Approach
For each stochastic realization,time step and linearization
Compute data forinterface problemCompute data forinterface problem
Subdomainsolves
Compute multiscalebasis for coarse scaleCompute multiscale
basis for coarse scale
Solve theinterface problem
Solve theinterface problem
Solve local problemsgiven interface valuesSolve local problems
given interface values
Multiplesubdomain
solves
Multiple linearcombinations
of basis
Subdomainsolves
Domain Decomposition and MultiscaleDomain Decomposition and Multiscale
For each stochastic realization,time step and linearization
Compute data forinterface problemCompute data forinterface problem
Subdomainsolves
Preconditiondata
Preconditiondata
Solve theinterface problem
Solve theinterface problem
Solve local problemsgiven interface valuesSolve local problems
given interface values
ApplyMultiscaleprecond.
Fixed numberof subdomain solves
Fixed number ofmultiscale precond.
applications
Subdomainsolves
Compute the multiscalebasis for a training operator
Compute the multiscalebasis for a training operator
Multiple subdomain solves
Example: Uncertainty Quantification
360x360 grid 25 subdomains of equal size 129,600 degrees of freedom Continuous quadratic
mortars Karhunen-Loéve expansion
of the permeability truncated at 9 terms
Second order stochastic collocation
512 realizations Training operator based on
mean permeability
Mean Permeability
Mean Pressure
Number of Interface Iterations
Interface Solver Time
Example: IMPES for Two Phase Flow
360x360 grid 25 subdomains of equal
size 129,600 degrees of
freedom Continuous quadratic
mortars 50 implicit pressure
solves 100 explicit saturation
time steps per pressure solve
Training operator based on initial saturation
Absolute Permeability
Initial Saturation
Number of Interface Iterations
Interface Solver Time
Continuous Measurement and Data Analysis for Reservoir Model
Estimation
Source: E. Gildin, CSM, UT-Austin
Continuous Measurement and Data Analysis for Reservoir Model
Estimation
IPARS
Data Acquisition (Sensors + DAQ)
Online Analysis(Data Fusion,
Denoising, Resampling…)
Optimization & Supervisory
Control
Reservoir
Data Assimilation(EnKF)
Field Controller(s)
DynamicI/F
Source: I. Alvarado and D. Schmidt, NI
Key Issues in C02 Storage What is the likelihood and magnitude of CO2 leakage and what are
the environmental impacts?
How effective are different CO2 trapping mechanisms?
What physical, geochemical, and geomechanical processes are important for the next few centuries and how these processes impact the storage efficacy and security?
What are the necessary models and modeling capabilities to assess the fate of injected CO2?
What are the computational needs and capabilities to address these issues?
How these tools can be made useful and accessible to regulators and industry?
groundwater flow
CO2 leakage
deep brine aquifer
drinking-water aquifer
COCO22 Sequestration Modeling Approach Sequestration Modeling Approach
Numerical simulationNumerical simulation Characterization (fault, fractures)Characterization (fault, fractures)
Appropriate griddingAppropriate gridding
Compositional EOSCompositional EOS
Parallel computing capabilityParallel computing capability
Key processesKey processes COCO22/brine mass transfer/brine mass transfer
Multiphase flowMultiphase flow
During injection During injection (pressure driven)(pressure driven)
After injection After injection (gravity driven)(gravity driven)
Geochemical reactionsGeochemical reactions Geomechanical modelingGeomechanical modeling
ParallelParallel
IPARS-COMPIPARS-COMP
ThermalThermal2-P Flash2-P Flash
GeomechanicsGeomechanics
Numerics Numerics
GraphicsGraphics
EOSEOS CompComp.GeochemicGeochemic
alalReactionReaction
GriddingGridding
SolversSolvers
Physical Physical PropProp
IPARS-COMP Flow EquationsIPARS-COMP Flow Equations
Mass Balance EquationMass Balance Equation
Pressure EquationPressure Equation
Solution MethodSolution Method Iteratively coupled until a volume balance convergence Iteratively coupled until a volume balance convergence
criterion is met or a maximum number of iterations criterion is met or a maximum number of iterations exceeded.exceeded.
ii i i i
N. u S D q
t
Thermal & Chemistry EquationsThermal & Chemistry Equations
Energy BalanceSolved using a time-split
scheme (operator splitting)Higher-order Godunov for
advectionFully implicit/explicit in time and
Mixed FEM in space for thermal conduction
ChemistrySystem of (non-linear) ODEsSolved using a higher order
integration schemes such as Runge-Kutta methods
Tp H
T
T s vs v
M T. C u T T q
t
Internal energy : M
M 1 C C S
ValidationValidation
SPE5 -- A quarter of 5 spot benchmark WAG problemSPE5 -- A quarter of 5 spot benchmark WAG problem
3-phase, 6 components C1, C3, C6, C10, C15, C203-phase, 6 components C1, C3, C6, C10, C15, C20
IPARS-CO2 vs CMG-GEMIPARS-CO2 vs CMG-GEM
Cum. oil produced Cum. gasInj
Prod
ValidationValidation
COCO22 pattern flood injection pattern flood injection3-phase, 10 components 3-phase, 10 components CO2, N2, C1, C3, C4, C5, C6, C15, C20CO2, N2, C1, C3, C4, C5, C6, C15, C20
IPARS-CO2 vs CMG-GEMIPARS-CO2 vs CMG-GEM
CO2 conc.Cum. gas
Inj
Prod.
Parallel SimulationsParallel Simulations
Modified SPE5 WAG injectionModified SPE5 WAG injection Permeability from SPE10Permeability from SPE10 160x160x40 (1,024,000 cells)160x160x40 (1,024,000 cells) 32, 64, 128, 256, 512 processors32, 64, 128, 256, 512 processors
Oil pressure and water saturation@ 3 yrs
Gas saturation and propane conc. @ 3 yrs
HardwareHardwareLonestar: Linux Lonestar: Linux cluster systemcluster system
Blue GeneP: CNK Blue GeneP: CNK system, Linux I/Osystem, Linux I/O
1,3001,300 Nodes / Nodes / 5,200 cores5,200 cores
262,144 Nodes / 262,144 Nodes / 1,048,576 cores1,048,576 cores
Processor Arch: Processor Arch: 2.66GHz, Dual 2.66GHz, Dual core, Intel Xeon core, Intel Xeon 5100, Peak: 55 5100, Peak: 55
TFlops/s TFlops/s
Processor Arch: Processor Arch: 850MHz, 850MHz,
IBM CU-08, Peak: IBM CU-08, Peak: ~1 PFlop/s~1 PFlop/s
8 GB/node8 GB/node 2 GB/node2 GB/node
Network: Network: InfiniBand, 1GB/sInfiniBand, 1GB/s
Network: Network:
10Gb Eth,1.7GB/s10Gb Eth,1.7GB/s
SoftwareSoftwareGMRES, BCGS, LSOR, Multigrid.GMRES, BCGS, LSOR, Multigrid.
MPI: MVAPICH2 library for parallel MPI: MVAPICH2 library for parallel communicationcommunication
Texas Advanced Computing Center The University of Texas at Austin
Parallel ScalabilityParallel Scalability
Scalability On Ranger (TACC) & Blue Gene PScalability On Ranger (TACC) & Blue Gene P
Ranger (TACC)Ranger (TACC) Blue Gene PBlue Gene P
GMRES solver with Multigrid Preconditioner3500ft, 3500 ft, 100ft reservoir40x160x160=1,024,000 elementsCPUs: 32, 64, 128, 256, 512, 1024
CO2 Storage Benchmark Problems
A Benchmark-Study on Problems Related to CO2 Storage inGeological formations, Summary and Discussion of the Results H. Class, A. Ebigbo, R. Helming et al., 2008
Benchmark Problem 1.1Benchmark Problem 1.1COCO22 Plume Evolution and Leakage via Plume Evolution and Leakage via
Abandoned WellAbandoned Well
ObjectiveObjectiveQuantification of leakage rate Quantification of leakage rate in deep aquifer @2840-3000 min deep aquifer @2840-3000 m
OutputOutput1- Leakage rate = %CO1- Leakage rate = %CO22 mass mass
flux/injection rateflux/injection rate
2- Max. leakage value 2- Max. leakage value
3- Leakage value at 1000 d3- Leakage value at 1000 d
=0.15
K = 20 md
P = 3.08x104 KPa
Benchmark Problem 1.1Benchmark Problem 1.1Leakage Rate of COLeakage Rate of CO22
CO2 BT: 10 days
Peak Leakage value: 0.23%Final leakage value: 0.11%Agrees with semi-analytic solution (Nordbotten et al.)
Comparison with Published ResultsComparison with Published Resultsat 80 daysat 80 days
Gas Saturation
Pressure
Ebigbo Ebigbo et al.et al., 2007, 2007IPARS-COMP
Frio Brine COFrio Brine CO22 Injection Pilot Injection Pilot
Bureau of Economic GeologyJackson School Of GeosciencesThe University of Texas at AustinFunded by DOE NETL
Frio Brine Pilot SiteFrio Brine Pilot Site
Injection interval: 24-m-thick, Injection interval: 24-m-thick, mineralogically complex fluvial mineralogically complex fluvial sandstone, porosity 24%, sandstone, porosity 24%, Permeability 2.5 DPermeability 2.5 D
Unusually homogeneousUnusually homogeneous Steeply dipping 16 degreesSteeply dipping 16 degrees 7m perforated zone7m perforated zone Seals Seals numerous thick shales, numerous thick shales,
small fault blocksmall fault block Depth 1,500 mDepth 1,500 m Brine-rock, no hydrocarbonsBrine-rock, no hydrocarbons 150 bar, 53 C, supercritical CO150 bar, 53 C, supercritical CO22
Injection interval
Oil production
From Ian Duncan
Frio Modeling EffortFrio Modeling Effort Stair stepped approximation on a 50x100x100 grid (~70,000
active elements) has been generated from the given data. Figure shows porosity in the given and approximated data.
Solution profilesSolution profiles Pressure and close-up of top-view of gas (CO2) saturation at t=33
days. Simulations on bevo2 cluster at CSM, ICES on 24 processors.
CO2 Plume TransportCO2 Plume Transport CO2 saturation as seen below the shale barrier at t=2 and 33 days.
Breakthrough time is observed to be close to 2 days.
Current Research Activities at CSMCurrent Research Activities at CSM
Model COModel CO22 injection either in deep saline aquifers or injection either in deep saline aquifers or depleted oil and gas reservoirs using compositional depleted oil and gas reservoirs using compositional and parallel reservoir simulator (IPARS-COand parallel reservoir simulator (IPARS-CO22))
Large scale parallel computingLarge scale parallel computing Efficiency with different solversEfficiency with different solvers Couple IPARS-COCouple IPARS-CO22 with geochemistry with geochemistry Couple IPARS with geomechanicsCouple IPARS with geomechanics Enhance EOS model and physical property Enhance EOS model and physical property
models (effect of salt, hysteresis, etc)models (effect of salt, hysteresis, etc) Data sources, field sites, practical applicationsData sources, field sites, practical applications
(in collaboration with Duncan from BEG at UT)(in collaboration with Duncan from BEG at UT) Gridding and a posteriori error estimatorsGridding and a posteriori error estimators OptimizationOptimization Risk and uncertainty analysisRisk and uncertainty analysis