David Holmes, John WilliamsCivil and Environmental EngineeringMassachusetts Institute of Technology

Peter TilkeMathematics and Modeling DepartmentSchlumberger-Doll Research

December 9th, 2008

Mitigating the Energy Crisis usingSimulation for Enhanced Oil RecoveryAnalyzing oil fields grain by grain

www.milner-photo.com/industrial/home.html

Enhanced Oil Recovery– Problem Statement– Modeling Challenges

Developing an Advanced Simulation Framework– Simulation Challenges for Multi-Core– Dynamic Execution Management

Testing and Applications Conclusions

Outline

Primary Development 20 – 40% Recovery

Existing EOR Such as– Water Flooding Optimistically an– Gas Injection Additional 10 – 20%– Chemical Injection Recovery– Thermal Stimulation

Enhanced Oil RecoveryProblem Statement

www.llnl.gov/str/November01/Kirkendall.html

pubs.usgs.gov/dds/dds-033/

USGS_3D/ssx_txt/all.htm

Oil Saturated

Pores

The Department of Energy (DOE) estimates that using ‘Next Generation EOR’ the United States could generate an additional 240 billion barrels of recoverable oil resources

This corresponds to approximately 30 years supply at current consumption

Developing new EOR is critical to maintaining our way of life

Enhanced Oil RecoveryProblem Statement

http://photo.net/photodb/photo?photo_id=5219385&size=lg http://www.shmolnick.com/images/traffic_jam_web.jpg

Enhanced Oil RecoveryModeling Challenges

Multi-phase fluid flow at the pore scale

Enhanced Oil RecoveryModeling Challenges

www.co2crc.com.au/aboutgeo/storage.html

Multi-phase fluid flow at the pore scale Interpretation and calibration of down-

hole measurement techniques such as electrical resistivity and acoustic wave propagation

Enhanced Oil RecoveryModeling Challenges

www.thewaterexperts.com/

welldevelopment.htm

Multi-phase fluid flow at the pore scale Interpretation and calibration of down-

hole measurement techniques such as electrical resistivity and acoustic wave propagation

Understanding the hydro-fracturing of rocks

Enhanced Oil RecoveryModeling Challenges

Multi-phase fluid flow at the pore scale Interpretation and calibration of down-

hole measurement techniques such as electrical resistivity and acoustic wave propagation

Understanding the hydro-fracturing of rocks

Understanding the mechanisms of sand production and borehole collapse

Multi-phase fluid flow at the pore scale Interpretation and calibration of down-

hole measurement techniques such as electrical resistivity and acoustic wave propagation

Understanding the hydro-fracturing of rocks

Understanding the mechanisms of sand production and borehole collapse

Carbon sequestration and hydrate mining

Enhanced Oil RecoveryModeling Challenges

http://www.netl.doe.gov/technologies/oil-gas/

FutureSupply/MethaneHydrates/projects/DOEProjects/

MH_43067GasHydSediments.html

Multi-phase fluid flow at the pore scale Interpretation and calibration of down-

hole measurement techniques such as electrical resistivity and acoustic wave propagation

Understanding the hydro-fracturing of rocks

Understanding the mechanisms of sand production and borehole collapse

Carbon sequestration and hydrate mining Architecting integrated multi-physics

software systems that can run on multi-core/multi-machine architectures

Enhanced Oil RecoveryModeling Challenges

Developing an Advanced Simulation FrameworkSimulation Challenges for Multi-Core

Concurrency Packages Conventional parallel simulation implementations use MPI A powerful new library for multi-core is Microsoft’s CCR Further room for generalization for simulation applications

Challenges to Concurrency General challenges

– Synchronization– Thread safety– Load balance

Challenges unique to simulation in parallel– Spatial reasoning and task distribution– Dynamic evolution of numerical tasks

Spatial Reasoning and Task DistributionDomain Decomposition Domain Distribution(Predictive Load Balance) (Events Based Load Balance)

Developing an Advanced Simulation FrameworkSimulation Challenges for Multi-Core

Developing an Advanced Simulation FrameworkSimulation Challenges for Multi-Core

Dynamic Evolution of the Numerical TaskAdaptive Remeshing Removal of Elements Outside Critical Zone

Variable Free Surface on a Grid

Developing an Advanced Simulation FrameworkDynamic Execution Management

Microsoft’s Concurrency and Coordination Runtime (CCR)

AcknowledgementsGeorge Chrysanthakopoulos Henrik Nielsen

Primary Concurrency Tools– Port

– Receiver

Developing an Advanced Simulation FrameworkDynamic Execution Management

The Developed Dispatch Mechanism

Developing an Advanced Simulation FrameworkDynamic Execution Management

Advantages– Perfectly load balanced to within required operations on 1 data point– Accommodates any CPU number– Accommodates variable CPU efficiency/availability and remains load balanced

Programming Challenges– Dispatch must know when all data has been received– Dispatch must recognize when data has been distributed– All processes must complete before finalization

Testing and ApplicationsDEM and Particle Methods Focus on Grain to Macro Scale Analysis

FD Particle Methods DEM

FIXED GRID MOVING PARTICLE METHODS

FEM

Goal is to handle Multi-Physics, Multi-Scale

NanoGrain

Focus on Macro and Grain Scale

MesoMacroPhysicalScale

Molecular Dynamics

Testing and Applications

Speed-Up Efficiency

Testing and ApplicationsSpeed tests carried out on a Dell Server PE2900with 8-core Intel Xeon E5345 CPU

Testing and ApplicationsFalling drop example ~200 000 particles, 10 000 time steps

Testing and ApplicationsRayleigh-Taylor Instability test ~500 000 particles, 25 000 time steps

Testing and ApplicationsSimulation of water flooding – Goal is to optimize recovered oil by “designing” fluids

A

B

Non-wetting water phase (blue) invading wetting oil phase (red)

Wetting water phase (blue) invading non-wetting oil phase (red)

Testing and ApplicationsCalibration of large filed scale models based on a better understanding of the pore scale phenomena

K. Geel, Delft University of Technology

Testing and ApplicationsCarbon sequestration and hydrates mining

Multi-phase modeling of gas/liquid/solid interactions to allow– Reduction of green-house gas

emissions through carbon sequestration– Reduction of the environmental impact

of unstable sub-sea methane deposits– Extraction of methane as an alternate

fuel source

http://openlearn.open.ac.uk/file.php/2292/formats/print.htm

www.llnl.gov/str/November01/Kirkendall.html

Conclusions

Developing an Advanced Simulation FrameworkDynamic Execution Management

Conventional CCR example of a repeated scatter-gather

Scatter

Parallel

Gather