ices report 12-01 petroleum reservoir parameter estimation

ICES REPORT 12-01

January 2012

Petroleum Reservoir Parameter Estimation andUncertainty Assessment with the Parallel Ensemble Based

Framework Coupled with IPARSby

Reza Tavakoli, Gergina Pencheva, Mary F. Wheeler, Benjamin Ganis

The Institute for Computational Engineering and SciencesThe University of Texas at AustinAustin, Texas 78712

Reference: Reza Tavakoli, Gergina Pencheva, Mary F. Wheeler, Benjamin Ganis, Petroleum ReservoirParameter Estimation and Uncertainty Assessment with the Parallel Ensemble Based Framework Coupled withIPARS, ICES REPORT 12-01, The Institute for Computational Engineering and Sciences, The University ofTexas at Austin, January 2012.

Petroleum Reservoir Parameter Estimation and Uncertainty

Assessment with the Parallel Ensemble Based Framework

Coupled with IPARS

Reza Tavakoli

Gergina Pencheva, Mary F. Wheeler, Benjamin Ganis

The Center for Subsurface Modeling

Institute for Computational Engineering and Sciences (ICES)1

The University of Texas at Austin

December 2011

12001 E. 24th Street, ACES 5.324, C0200, Austin, TX, 78712

Abstract

We present a parallel framework for history matching and uncertainty characterizationbased on the Kalman filter update equation for the application of reservoir simulation. Themain advantages of ensemble-based data assimilation methods are that they can handlelarge scale numerical models with a high degree of non-linearity and large amount of data,making them perfectly suited for coupling with a reservoir simulator. However, the sequentialimplementation is computationally expensive as the methods require relatively high numberof reservoir simulation runs. Therefore, the main focus of this work is to develop a paralleldata assimilation framework with minimum changes into the reservoir simulator source code.In this framework, multiple concurrent realizations are computed on several partitions of aparallel machine. These realizations are further sub-divided among different processors andcommunication is performed at data assimilation times. Although this parallel framework isgeneral and can be used for different ensemble techniques, we discuss the methodology andcompare results of two algorithms, the ensemble Kalman filter (EnKF) and the ensemblesmoother (ES). Special attention is given to the solver technique, in order to both optimizeand synchronize the forward simulation runs. Computational results show the absoluteruntime is greatly reduced using a parallel implementation versus a serial one. In particular,a parallel efficiency of about 35 percent is obtained for the EnKF, and an efficiency of morethan 50 percent is obtained for the ES.

Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Forward Model: IPARS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 History Matching Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3.1 Ensemble Kalman Filter . . . . . . . . . . . . . . . . . . . . . . . . . . 53.2 Ensemble Smoother . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

4 Parallel Ensemble Based Framework . . . . . . . . . . . . . . . . . . . . . . . 65 Computational Example and Discussion . . . . . . . . . . . . . . . . . . . . . 9

5.1 Example Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95.2 EnKF and ES Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 115.3 Solver Sensitivity Results . . . . . . . . . . . . . . . . . . . . . . . . . 145.4 EnKF and ES Scaling Results. . . . . . . . . . . . . . . . . . . . . . . 15

6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

i

1 Introduction

Data assimilation is now considered as the state-of-the-art approach within the atmosphericand oceanographic sciences. This technique consists of integrating observed data into dy-namical models to determine the best estimate of poorly known parameters and/or states.The ensemble Kalman filter (EnKF) algorithm was introduced by Evensen [3] as a bet-ter alternative to solving the computationally demanding equation used in the extendedKalman filter. In the reservoir engineering field, EnKF was first used for history match-ing and uncertainty characterization by Nævdal et al [22, 23], and since that time it hasbeen used extensively for parameter estimation and uncertainty quantification as a reservoirmanagement tool [1, 10, 9, 8, 25].

The EnKF is a Monte Carlo formulation of the Kalman filter, in which an ensembleof reservoir models is used to estimate the correlation between predicted data (such asproduction rates and bottom-hole pressures) and state variables (consisting of static modelparameters such as porosity and permeability, as well as dynamic variables such as pressureand saturation for two-phase flow). Every time observation data is present, one applies theKalman update equation to assimilate this data into prior reservoir model parameters andprimary variables. In addition, the ensemble is used to provide an estimate of uncertaintyin future reservoir performance.

It is important to note that there are modeling assumptions necessary for EnKF methodthat do not hold in the reservoir simulation context. In particular, the prior and posteriorprobability distribution functions (pdfs) of state variables are assumed to be Gaussian, whichis usually not the case in practice. Moreover, the relationships between the model parame-ters, primary variables, and observation data are all assumed to be linear, which is certainlynot true for multiphase flow. Nevertheless, the EnKF method usually provides valuableestimates on both history-matched model parameters and future uncertainty. However, onemust be careful that the violation of these assumptions does not lead to inconsistent andunphysical results.

The EnKF uses recursive updates in time, and implicitly assumes independent data atdifferent assimilation steps. At each assimilation time, this requires reading and writingrestart files to disk. For especially large state vectors, this can be costly. Moreover, froma parallel viewpoint, the simulation runs must be synchronized at these times. Both ofthese factors can potentially have a negative impact on total simulation time. These issuesmotivate us to implement the Ensemble Smoother (ES) in our parallel framework, sincethis data assimilation method is more appropriate for a deluge of observed data, as in CO2

sequestration and large-scale reservoir simulation.The ES method was introduced in oceanography by van Leeuwen and Evensen [19] and

recently has been used in the context of the history matching problems by Skjervheim et al[26]. In the Bayesian statistics, the EnKF and ES methods use similar update formulations.In the former, a recursive update is performed at specific measurement times based oncurrent ensemble statistics, while in the latter, a global update is calculated over the entirespace-time domain based on prior ensemble statistics. The assumptions necessary for the ESmethod are less strict, because observed data need not be independent in time. It is knownthat when the prior pdfs are Gaussian and the the relationship between the model parametersand observation data is linear, the ES and EnKF methods produce the same result [7, 4].The implementation of ES is much simpler, because one does not need to update simulatorprimary variables (e.g. pressure and saturation) and only the model parameters are updated.Therefore, the restart files are completely eliminated. After updating the model parameters

1

using all the observation data with ES, we rerun all the ensemble members from time zeroto compute history matching results.

The implementation of the ensemble-based algorithms is more efficient for dynamic dataassimilation than automatic history matching based on optimization algorithms. However,the serial run of ensemble members is time consuming for large scale numerical models.In addition, the memory of a single processor dictates the size of a reservoir model (anensemble member) that can be used in practice. For these reasons, it is advantageous todevelop a parallel framework for assimilation of production data. Successful applications ofparallel EnKF have been published within weather forecasting and oceanographic sciences.Keppenne [13] and Keppenne and Rienecker [14] have developed a parallel EnKF algorithmwhich use two-level parallelization. The inherent parallelism in the EnKF is used by runningeach ensemble member on a different processor of a parallel computer. The EnKF updatestep is parallelized by letting each processor handle the observations from a limited region.The idea of doing the analysis on a regional basis is introduced by Houtekamer and Mitchell[11], who uses a Schur (element-wise) product of a correlation function with the background-error covariances to eliminate or reduce the spurious correlations due to the use of smallensemble sizes.

Despite intense focus on the application of the ensemble Kalman filter to history matchingproblems, very limited work has been reported on parallel EnKF applications in petroleumengineering. Xin [31] developed a high-performance Grid computing environment that per-forms the forecast step of the EnKF algorithm in parallel by simultaneous run of eachsimulation model. Initial testing of a parallel module for automatic history matching hasbeen reported in Liang et al [20], where a weighted EnKF method is used to update per-meability field for a compositional model. The contribution of each ensemble member isassumed to be different in the assimilation step. In their implementation, all the simula-tion jobs were submitted to a high performance computing cluster at a same time. In arecent paper, Tavakoli et al [27] proposed a simple methodology for paralellization of theupdate step of the EnKF and implemented a multi-level paralellization algorithm to a set ofhistory-matching problems in different high performance computing environments.

In this paper, we present an application of parallel ensemble-based framework that usesthe EnKF and the ES algorithms for automatic history matching. Specifically, this frame-work can submit multiple concurrent simulation jobs into a high performance computingenvironment, each of which uses a parallel reservoir simulator to propagate the ensemblemembers to the next assimilation time. We present computational cost and parallel ef-ficiency results of implementing this framework for a history matching problem. In ourimplementation, we use the Integrated Parallel Accurate Reservoir Simulator (IPARS) de-veloped at the Center for Subsurface Modeling in ICES, the University of Texas at Austin[29, 30]. Since the simulation time varies from one ensemble member to another, the slowestsimulation run always controls the whole process. IPARS has a capability of using differentsolvers and preconditioners in the forward run. We have performed extensive experimentsto calibrate the solver parameters in an attempt to make the multiple simulation runs bothsynchronized and efficient.

The contents of this report are as follows. In Section 2, we present a brief descriptionof IPARS simulator and its implemented solvers and preconditioners. In Section 3, wereview EnKF and ES formulations. In Section 4, we present the details and structure ofimplemented parallel framework. In Section 5, we compare the history matching results ofthe EnKF and ES algorithms, followed by parallel efficiency results and some discussions.In Section 6, we summarize and conclude the report.

2

2 Forward Model: IPARS

One of the main tools in reservoir management is the reservoir simulator. A great deal ofeffort has been devoted to construct accurate and efficient reservoir models. The governingequations of multi-phase flow in porous media are given by a set of partial differential equa-tions that represent conservation of mass, momentum and energy together with equations ofstate which describe the fluid property as a function of pressure and saturation. The IPARSframework is a research software developed mainly for the purposes of examining differentphysics, approaches and methods [29]. This framework supports three dimensional transientsubsurface flow of multiple phases containing multiple components. In this paper, we willconsider the two-phase (oil and water), slightly compressible flow model which by means ofDarcy’s law is characterized by the following simplified conservation of mass equations:

C1∇ ·[ kroµoBo

K(∇Po(Ω, t)− γo∇h(Ω))]

=φ

C2

∂

∂t

(So(Ω, t)Bo

)+ qo(Ω, t), (1)

C1∇ ·[ krwµwBw

K(∇Pw(Ω, t)− γw∇h(Ω))]

=φ

C2

∂

∂t

(Sw(Ω, t)

Bw

)+ qw(Ω, t), (2)

where the subscripts o and w stand for oil and water phases, respectively, the constantsC1 and C2 depend on the system of units, ∇· denotes the divergence operator, ∇ denotesthe gradient operator, Bo and Bw are the oil and water formation volume factors, µ is theviscosity, γ is the specific density, h is the depth of the reservoir, q is the sink or sourceterm, S is the saturation, kr is the relative permeability of each phase (which is a functionof S), φ is the porosity and K is the permeability tensor. Note that equations (1) and (2)are defined in space domain Ω and t represents time. To complete the system of equationsgiven by equations (1) and (2), two additional closure equations are obtained by assumingthat the porous medium is completely saturated with oil and water and that the relationshipbetween oil and water pressures is defined by capillary pressure which is a function of watersaturation

So + Sw = 1, (3)

Po − Pw = Pc(Sw). (4)

In IPARS, a fully implicit time stepping scheme is used for the solution of the coupledmultiphase nonlinear flow equations, resulting in nonlinear systems to be solved at each timestep. Each nonlinear system is solved with a two-level iterative scheme, in which the outeriteration is an inexact Newton method, and the inner iteration is a Krylov method to solvethe linear system on each Newton step. The Jacobian matrix is sparse, non-symmetric,and ill conditioned. For the linear system, a preconditioned generalized minimum residual(GMRES) iterative solution method is used. The GMRES preconditioning is based on atwo-stage algorithm. For the first stage, a decoupling preconditioner is used to separate thepressure and saturation (or concentration) equations, and does not depend on the geometryof the grid. For the second stage, the pressure block is preconditioned. Different techniquesfor decoupling and pressure block preconditioning are implemented in the IPARS software,see Table 1. A detailed description of each of these techniques is beyond the scope of thispaper and can be found elsewhere [16, 17, 18].

In addition to choosing a proper preconditioner for iteratively solving the Jacobian equa-tion, it is also necessary to choose adequate tolerances for the linear and nonlinear iterations,

3

Table 1: List of main GMRES preconditioners implemented in IPARS.

Preconditioner ID Description

3 Block Gauss-Siedel (GS)4 Line Successive Over Relaxation (LSOR)5 Truncated Neumann series (NS) with

conjugate gradient (CG) iteration6 Truncated NS with BCGstab iteration7 Truncated NS with GMRES iteration8 Un-accelerated truncated NS11 SEP-PREC with LSOR smoother13 Multilevel incomplete LU (MILU)14 MILU with LSOR smoother15 Algebric Multigrid (AMG)16 AMG with LSOR17 Sparse factorization

so that an efficient solution is obtained for the forward model. An optimal balance betweenthe linear and nonlinear iterations is provided by the forcing term technique [2], which dy-namically adjusts the tolerance in the course of the Newton process. A loose linear toleranceis used at the start of the nonlinear iterations, and it is tightened as the inexact Newtonmethod converges.

The IPARS simulator supports three methods for parallelism: data decomposition usingdistributed matrix and vector operations, domain decomposition using mortar finite ele-ments, and the enhanced velocity method for treating non-matching grids. In this work, thefirst method is used to parallelize forward model realizations. The underlying meshes are 3Drectangular, with grid cells grouped on a processor in vertical columns for load-balancing.When neighboring processor information is needed during computation, updates are per-formed into ghost cells. The linear solver stores a piece of the Krylov vector associated witha portion of the domain on each processor. Interprocess communication is used to calcu-late matrix-vector and dot-products and also when the preconditioners are applied. Furtherinformation regarding the parallel capabilities of IPARS is described in Vassilevski [28].

3 History Matching Methodology

The implementation and development of the ensemble-based data assimilation algorithms forautomatic history matching problems has been steadily increasing over the past few years.In these algorithms, an ensemble of realizations is employed to represent the uncertainty bya low-rank approximation of error covariance for the state vector. The ensemble of modelstates is integrated forward in time to predict the uncertainty. Whenever observation data isavailable, each realization is updated to assimilate the information from new measurements.In this section, we provide detailed formulations and notation used for the EnKF and ESmethods.

4

3.1 Ensemble Kalman Filter

When applying EnKF to nonlinear history matching problems, we consider the vector ofreservoir model parameters (grid block porosities and permeabilities), m, and the vectorof state variables (grid block pressures and saturations), pn, in an augmented state vector[5, 6], yn, i.e.,

yn =

[mpn

]. (5)

Note that here the superscript n denotes the time at which we are assimilating data. Allvectors are column vectors. The dimension of m is given by Nm, and the dimension of pn isgiven by Np. Thus, the dimension of yn, denoted by Ny is given by Ny = Nm +Np.

For simplicity in notation, assume that we assimilate data at every simulation time stepn with Ne ensemble members. Therefore, the EnKF update equation for each ensemblemember j is given by

yn,aj = yn,fj + CfY nDn(CfDnDn + CDn)−1(dnuc,j − dn,fj ) for j = 1, 2, · · ·Ne. (6)

Here, CfY nDn denotes the cross-covariance between the state vector yn,fj and predicted data

dn,fj ; CfDnDn represents auto-covariance for predicted data; dnuc,j is a sample of the normal dis-tribution N(dnobs, CDn); dnobs represents observed data; and CDn denotes the data covariancematrix at time n. The superscript f denotes the forecast or predicted data, the superscript adenotes to the result of the assimilation step, and the subscript uc denotes an unconditionalperturbation of observed data. For example, yn,aj represents the updated state vector at timen for ensemble member j by assimilating data dnuc,j . This EnKF update equation is similarto the traditional Kalman filter update equation [12], with the difference being the way we

propagate the covariance matrices. The predicted data, dn,fj , can be obtained by runningthe reservoir simulator forward, which is represented by the nonlinear equation

dn,fj = gn(mj). (7)

The cross-covariance matrix, CfY nDn , and the auto-covariance matrix, CfDnDn , can beexpressed by

CfY nDn =1

Ne − 1

Ne∑j=1

(yn,fj − yn,f

)(dn,fj − dn,f

)T, (8)

and

CfDnDn =1

Ne − 1

Ne∑j=1

(dn,fj − dn,f

)(dn,fj − dn,f

)T, (9)

where yn,f and dn,f denote the mean of the Ne predicted state vectors and the mean ofthe Ne predicted data at time n, respectively. In the implementation of the EnKF analysisequation (6), the Ny ×Nd matrix CfY nDn as defined by (8) is never explicitly computed orstored. Note Nd denotes the number of data to be assimilated at each assimilation step.

5

3.2 Ensemble Smoother

The ES assimilates all observed data for the state vectors of each ensemble member usinga single update step [19], whereas the EnKF performs recursive update steps at each mea-surement time. Therefore, the ES method only updates model parameters m, instead of thefull state vector yn required for EnKF. The ES update equation for each ensemble memberis given by

maj = mf

j + CfMD(CfDD + CD)−1(Duc,j −Dfj ) for j = 1, 2, · · ·Ne. (10)

where Duc,j is a vector of perturbed observed data which contains all dynamic data forensemble member j, i.e.,

Duc,j = [d1uc,j d2uc,j . . . d

Nuc,j ]

T , (11)

and N denotes the total number measurement times. The vector Dj represents the predicteddata corresponding to Duc,j . Since all the data are being assimilated simultaneously, wesuppress the time superscript n in equation (10). The cross covariance between vector ofmodel parameters mj and the vector of all predicted data, Dj , is defined as

CfMD =1

Ne − 1

Ne∑j=1

(mfj − m

f)(Dfj − D

f)T, (12)

where mf and Df denote the mean of the prior model parameters and predicted data,respectively. In the ES update (10), CD denotes the data covariance matrix correspondingto vector Dobs, containing all measurements. Therefore, there is no need to assume thatindividual data are uncorrelated in time.

Similar to the EnKF, the ES algorithm consists of two main steps, the forecast step andthe analysis step. The forecast step is equivalent to running the forward model (reservoirsimulator) for each of the ensemble members from t = 0 to t = tN , the time of last dataassimilation. In the analysis step, the vector of predicted data Dj corresponding to allmeasured data Dobs is used in equations (10) to update the model parameters for each ofthe ensemble members.

4 Parallel Ensemble Based Framework

As discussed in the previous section, the forecasting step is a computationally intensive pro-cess because it requires the simulation of Ne reservoir models. In a serial implementation,these simulation runs are performed sequentially. The size of the reservoir, the grid resolu-tion, and the complexity of the forward simulation model can all contribute to computationaltime, leading to simulations that take a few hours to several days. As the size of the ensemblegrows, running a series of lengthy simulations in serial quickly becomes unfeasible in orderto perform the data assimilation necessary for reservoir characterization. This motivates theneed to use parallelization in ensemble-based algorithms.

Due to the independent nature of the ensemble members, multiple simulation runs canbe performed in parallel during the forecast step. Each separate realization is assigned to aseparate group of processors (compute node) on a parallel computer. However, individualrealizations are still very expensive in terms of both computation time and available memory.For these reasons, a second level of parallelism is used, wherein several processors on a single

6

compute node work together on a single realization using the parallelism built-in to thereservoir simulator. To summarize, we use IPARS in parallel to propagate each ensemblemember forward in time, and in addition, Ne ensemble members are executed concurrently.

The analysis step of the ensemble-based algorithms such as EnKF and ES require col-lecting a very large state vector from each ensemble member. Hence, the synchronization ofrealizations during the forecast step is desirable to ensure that no available compute nodesare left waiting for others to finish. Since IPARS has a variety of solver options and precon-ditioners, it is important to choose parameters that make forward simulation runs both fastand synchronized. Performing the analysis step on a single processor will pose additionallimitations on the overall size of the history matching problem in terms of both memory andcomputation time. In [27], a strategy for parallelization of the analysis step was proposed,which partitions the matrix of all ensemble members and distributes the computation amongmultiple processors. We mention this possible third level of parallelism for completeness, butdo not use it in this work.

The schematic structure for the parallel ensemble based framework for automatic historymatching is described in Figure 1. As mentioned in the introduction, one of the objectiveof this development is to couple filter algorithms to the IPARS simulator with minimumchanges to the simulator source code. We developed this parallel framework in Fortran 90that uses the Message Passing Interface (MPI) library for interprocess communication. Theframework consists of three main components, which are described in the remainder of thissection.

1. Initialization Step: Once the parallel framework is launched into a high perfor-mance computing (HPC) environment, it will first call the initialization subroutine.This subroutine will read all the input parameters from an input file provided by theuser. The input parameters include the dimension of the reservoir (number of gridblocks), number of ensemble members (Ne), assimilation time intervals and numberof assimilation steps, number of observation data and types of the model parameters(porosity, permeability, ...). The number of concurrent simulation runs during forecaststep (NRun) and number of processors used for each simulation run (NpSim) are alsoasigned in the input file. The type of the filtering algorithm (EnKF or ES) is alsospecified to be used in the analysis step. Note NRun ≤ Ne. The path to the IPARSsimulator executable file is also defined in this step.

After reading all the input parameters, the framework creates Ne sub-folders and copiesall required “general” input files from the main folder and all the “specific” realizationsinput files from the prior model folder into these sub-folders. The “general” input filesinclude simulation related input files as well as the HPC job script, and the “specific”realizations input files contain prior model parameters for each ensemble member. Notehere we assume that the initial prior ensemble members are already generated by somegeostatistical software. The link to the IPARS executable file is also placed in thesesub-folders. Then, the execution goes to the assimilation loop which starts with theinitial forecast step.

2. Forecast Step: In this step, ensemble members are evolved in time with nonlinearsimulation model, starting from the last update step and ending at the next assimilationstep. If this is the first time that the forecast subroutine has been called or in thecase of the first run of ES, the prior model parameters with initial distribution ofpressure and saturation (initial condition) are used in the simulation runs. Otherwise,

7

Figure 1: Schematic flow diagram of the parallel ensemble-based framework.

8

the updated model parameters and updated restart files (for the EnKF) are used topropagate the ensemble members to the next assimilation step. The forecast step isinitialized by submitting NRun jobs (tasks) simultaneously into a HPC environmenteach of which uses NpSim processors. As soon as the first job is done, the next jobis submitted and this continues until we reach to a maximum of Ne submitted job.Note by increasing NRun from one to a maximum of Ne in the input file, the forecaststep is switched from serial mode into fully parallel mode. The maximum number ofconcurrent simulation runs is equal to NRun. Note that at the completion of eachsimulation run, an augmented state vector is generated and written in a separate filefor each ensemble member and placed into a corresponding sub-folder. The statevectors contain model parameters, primary variables (in the case of EnKF by readingthe restart file) and predicted data corresponding to the observation data. Once theslowest simulation job is finished, the program enters the analysis step.

3. Analysis Step: The filter updating step of the algorithms is performed in this stage.This could be done by using a single processors or multiple processors as suggestedin Tavakoli et al [27]. We will focus here on using one processor, called the centralprocessor, to perform the update step. The central processor reads the augmentedstate vectors generated in the forecast step for each ensemble member and generatethe ensemble members matrix with the jth column equal to the augmented statevector of jth ensemble member. Then, the filter updating step is performed by usingequation (6) for the EnKF or equation (10) for the ES algorithm. The updated modelparameters and the updated primary variables (restart files) are placed into a propersub-folder for each ensemble members. If this is the last assimilation step, the programis terminated. Otherwise, we go back to the previous step to perform next forecaststep.

5 Computational Example and Discussion

In this section we provide a synthetic reservoir simulation problem to test the automatichistory matching framework. In Subsection 5.1, we describe the problem setup. In Subsec-tion 5.2, we compare the qualitative and quantitative differences between the EnKF and ESalgorithms. In Subsection 5.3, we perform a comprehensive set of sensitivity experimentsfor forward simulation runs, in order to calibrate IPARS solver parameters. In Subsection5.4, we present the computational cost, parallel scaling and performance of the implementedframework in different modes.

Computational experiments were performed on the bevo2 cluster at the University ofTexas at Austin. This machine has 23 compute nodes, each of which have two quad core2.66 GHz Intel Xeon processors with 16GB of RAM. This gives a total of 184 available coreswith communication handled by a high speed infiniband adapter. Source code was compiledusing the Intel Fortran compiler and Math Kernel Library with mvapich MPI distribution.

5.1 Example Description

We consider a two-phase (oil and water), three-dimensional reservoir model with a uniformareal grid of 80 × 60 blocks of size 100 ft × 100 ft. The reservoir consists of 5 layers withthicknesses ranging from 15 to 25 ft. The areal dimensions of the reservoir are 8, 000 ft ×6, 000 ft, and the total thickness of the reservoir is 105 ft. There are four inverted five-spot

9

Table 2: Geostatistical Parameters.Parameters Layer1 Layer2 Layer3 Layer4 Layer5

[ln(k)]mean 4.00 4.50 3.90 4.30 4.50φmean 0.20 0.22 0.23 0.20 0.22σln(k) 1.75 1.20 2.00 1.60 1.20

σφ 0.035 0.045 0.025 0.04 0.045ρφ,ln(k) 0.85 0.80 0.90 0.60 0.80

α 40.0 42.0 43.0 50.0 42.0l1 (ft) 7200 8700 3000 8600 8700l2 (ft) 1000 1500 1900 1500 1500

patterns for well configurations with nine production wells and four water injection wells.The three-dimensional view of the reservoir with the distribution of water saturation at1000 days is shown in Figure 2. All wells are vertical and fully perforated. The producersare constrained by fixed liquid production rate of 4000 STB/day, and if the bottom holepressure (BHP) falls below 800 psi, the well constraint is then switched to fixed BHP atthat level. The injectors are constrained by constant water injection rate of 8000 STB/dayand maximum injection BHP of 7300 psi. All wells start operating at the beginning of thesimulation time. The total simulation time is 2800 days.

Figure 2: True water saturation distribution at time 1000 days.

The true horizontal log-permeability and porosity fields are generated using an anisotropicspherical covariance function with different geostatistical parameters for each layer of thereservoir. The main geostatistical parameters used to generate the truth are summarizedin Table 2. In this table, [ln(k)]mean and φmean denote the prior mean of horizontal log-permeability and porosity, respectively. The standard deviations of log-permeability andporosity are denoted by σln(k) and σφ. Here, ρφ,ln(k) denotes the correlation coefficient be-tween porosity and log-permeability, α is the angle measured counterclockwise from thex-axis to the principal direction of the covariance function, l1 is the correlation range in theprincipal direction and l2 is the correlation range in the orthogonal direction.

Note that the prior means of horizontal log-permeability and porosity fields are uniformin each reservoir layer. We assume that the ratio of vertical to horizontal permeability is

10

(a) (b) (c) (d)

Figure 3: True horizontal log-permeability and porosity fields with the location of wells. a Ln(k)Layer 1. b Ln(k) Layer 3. c Porosity Layer 2. d Porosity Layer 4.

0.2. The true horizontal log-permeability and true porosity fields for different layers of thereservoir with the location of wells are shown in Figure 3.

The observation data dobs to be matched include oil production rate, qoil, water oilratio (WOR) and flowing bottom hole pressure (BHP) for production wells and BHP forinjection wells. The water production rate, qwat, and water injection rate, qwat,inj are alsoincluded as observed data in order to satisfy the second constraints for the producers andinjectors. We add Gaussian random noise to the true synthetic data generated by IPARSsimulator to obtain the observation data. The standard deviations of the Gaussian noiseare σqoil = 0.035 qoil,true, σWOR = 0.025 WORtrue and σBHP = 0.03 BHPtrue for all wells.The data are recorded in variable time intervals, ranging from 50 to 1550 days, resulting intotal of 8 time steps. Therefore, there are 8 assimilation steps for the EnKF, with Nd = 44observed data at each step. For the ES algorithm, we assimilate all Nd = 352 data at onestep. The assimilation step is followed by a prediction phase till 2800 days.

5.2 EnKF and ES Results

Model parameters include gridblock porosities and horizontal log-permeabilities leading tothe number of model parameters being Nm = 48, 000. All initial ensembles share the samestatistics and are generated using the same covariance function and geostatistical propertiesas the truth (Table 2). In all experiments with the EnKF and ES algorithms we use an en-semble size of 100 realizations. The final estimates of the ensemble mean of log-permeabilityand porosity fields for different layers of the reservoir obtained with EnKF are shown in Fig-ure 4. Note the prior ensemble mean of ln(k) and porosity for each layer of the reservoir isuniform. In comparison to the true fields (Figure 3), it is clear that the EnKF has capturedthe main features of the truth in the final models.

Figure 5 shows the ensemble means of ln(k) and porosity after simultaneous assimilationof all the observed data using the ES algorithm. Although the implementation of the ESalgorithm is much faster than the EnKF method (computational time is presented later),the estimated ensemble means are not as satisfactory as those obtained with the EnKF. Inorder to improve the results, we perform a second ES update in which we use the updatedensemble members of the first run of ES as prior (initial) ensemble members in the secondrun and we assimilate all the observed data again. As can be seen from Figure 6, the finalestimated means of ln(k) and porosity fields after the second assimilation are comparable tothe EnKF results as well as the truth.

The history matching and future prediction results for EnKF and ES (first and secondruns) are shown in Figures 7 and 8, respectively. The gray curves represent prior ensemble

11

(a) (b) (c) (d)

Figure 4: Final ensemble mean of horizontal log-permeability and porosity fields for EnKF with 100ensemble members. a Ln(k) Layer 1. b Ln(k) Layer 3. c Porosity Layer 2. d Porosity Layer 4.

(a) (b) (c) (d)

Figure 5: Final ensemble mean of horizontal log-permeability and porosity fields for first ES runwith 100 ensemble members. a Ln(k) Layer 1. b Ln(k) Layer 3. c Porosity Layer 2. d PorosityLayer 4.

(a) (b) (c) (d)

Figure 6: Final ensemble mean of horizontal log-permeability and porosity fields for second ES runwith 100 ensemble members. a Ln(k) Layer 1. b Ln(k) Layer 3. c Porosity Layer 2. d PorosityLayer 4.

12

(a) (b) (c) (d)

Figure 7: Production forecasts from the final updated ensembles for EnKF with 100 ensemble mem-bers. Open red circles are observed data, red curve is the true model, green curve is the updatedensemble mean, blue curves are updated ensemble predictions, gray curves are prior ensemble predic-tions, and the black curve is the prior ensemble mean. The vertical line marks the end of historicaldata. a BHP of Prod-4. b qoil of Prod-6. c WOR of Prod-3. d WOR of Prod-7.

(a) (b) (c) (d)

(e) (f) (g) (h)

Figure 8: Production forecasts from the final updated ensembles for the first ES run (first row) andthe second ES run (second row) with 100 ensemble members. Open red circles are observed data,red curve is the true model, green curve is the updated ensemble mean, blue curves are updatedensemble predictions, gray curves are prior ensemble predictions, and the black curve is the priorensemble mean. The vertical line marks the end of historical data. a, e BHP of Prod-4. b, f qoil ofProd-6. c, g WOR of Prod-3. d, h WOR of Prod-7.

13

predictions, black curve is the prior ensemble mean, green curve is the updated ensemblemean, blue curves represent updated ensemble predictions, open red circles are observeddata and red curve is from the true model. The updated results shown in these figures wereobtained by rerunning the reservoir simulator with the final updated ensemble. As can beseen from Figure 7, the production profiles from the prior ensemble have large uncertainty(variance), and the prior ensemble mean is far from the truth. The uncertainty is signifi-cantly reduced after assimilating all the observed data with EnKF and the ensemble meanof prediction is consistent with the true performance. While the prior uncertainty has sig-nificantly reduced with the first ES run (first row of Figure 8), the updated ensemble mean(green curve) is still far from the production profile from the true model (red curve). As itis expected from the results of Figure 6, the history matching results are improved signif-icantly by the second ES run (second row of Figure 8). These results suggest that simplyperforming ES twice will improve the results significantly with doubling the computationalcost. Since the ES method is significantly less expensive, running the algorithm twice stillcosts less than the EnKF method in terms of computation time.

5.3 Solver Sensitivity Results

The ensemble-based algorithms use multiple realizations in the forecast step to propagate thestatistical information and model uncertainty. Since all realizations need to finish before onecan proceed with the update step, the slowest simulation will always control the efficiencyof the algorithm. Clearly, to reduce the overall runtime, a synchronization of all simulationruns at assimilation step(s) is desirable. We investigate one way to achieve this goal, namelyby choosing a proper solver/preconditioner options for the forward simulation run. TheGMRES method is one of the most popular choices for solving non-symmetric systems[15, 24] and is the solver that has been extensively tested in IPARS [16, 17, 18]. Out ofthe many possible input parameters for the solvers in IPARS, we restricted our choices intothe type of the preconditioner, as summarized in Table 1, and the value of the forcing termwhich dynamically controls the convergence criteria.

In order to pick a proper preconditioner for the GMRES, we ran 25 prior realizationswith different porosity and horizontal permeability fields for 1550 days. Tables 3 and 4summarize the simulation wall-clock time obtained with different preconditioners for fourdifferent sets of runs as follows: 1) using one processor (NpSim=1) with no forcing option.2) using one processor (NpSim=1) with Forcing=0.1. 3) using four processors (NpSim=4)with no forcing option. and 4) using four processors (NpSim=4) with Forcing=0.1. Notebased on our extensive experiments (not presented here), the optimal choice of forcing factorhas appeared to be Forcing=0.1. In Tables 3 and 4, ‘Fastest’ column refers to the fastestmodel with minimum simulation run-time, ‘Slowest’ column refers to the slowest model withmaximum simulation run-time, ‘Ave.’ column refers to the average of simulation run-timeamong all 25 realizations and ‘Del T’ column refers to the run-time difference between theslowest and the fastest models. According to the results of Tables 3 and 4, the optimaluse of the forcing term technique during the course of simulation significantly reduces thecomputational cost. Therefore, we will always use the forcing term in our simulation runs.Further, it can be seen that the PREC=16 (the two-stage Algebraic MultiGrid (AMG) withLSOR) gave the fastest simulation runs and the minimum value of ‘Del T’ for both choicesof the number of processors. The minimum ‘Del T’ is desirable in the process of paralleldata assimilation because it is equivalent to the close completion of multiple simulation runs.Thus, our optimal choice of preconditioner is PREC=16.

14

Table 3: Summary of simulation wall-clock time (in seconds) using different GMRES pre-conditioners with one processor in IPARS for 25 prior realizations.

PRECNpSim=1

No Forcing Forcing=0.1Fastest Ave. Slowest Del T Fastest Ave. Slowest Del T

3 1693 1936 3532 1839 370 450 664 2944 1119 1335 1958 839 294 345 441 1475 2659 3226 3839 1180 611 770 899 2886 1070 1185 1373 303 332 383 519 1877 1260 1445 1869 609 334 399 652 3188 2000 2361 4944 2944 431 531 776 34511 1017 1425 2422 1405 391 463 620 22913 545 670 889 344 312 340 402 9014 648 728 837 189 249 279 356 10715 398 439 489 91 258 282 334 7616 578 650 733 155 221 243 277 5617 855 908 994 139 727 774 861 134

In general, the parallel performance and scaling characteristics of a parallel simulatordepend on the size of the problem (reservoir dimension) and the duration of simulationamong other factors. To demonstrate the performance of IPARS simulator for this particularproblem, we used different processors to run one of the prior realizations (realization 1) fortwo different simulation end times of 150 days and 2800 days. Our aim is to use propernumber of processors for each simulation run during the forecast steps. Figure 9 presentsthe execution time as a function of number of processors. The results illustrate that theparallel simulator significantly reduces the computational time and that the IPARS gave upto 72% parallel efficiency with NpSim=4 for this problem. The parallel efficiency, Ep, isdefined as

Ep =T1TpNp

, (13)

where T1 and Tp denote the execution time of a task on a single processor and Np processors,respectively. It is worth mentioning that we have seen much better scaling results usingIPARS with far more number of processors for larger problem sizes, e.g. [21]. However,since the size of the problem used here is relatively small (80× 60 areal grid), the overheadof communication hurts parallel efficiency as processors increase.

5.4 EnKF and ES Scaling Results.

In this subsection, we present the computational results of conducted experiments withEnKF and ES using the parallel ensemble-based framework. The main goal is to determineand demonstrate the parallel performance gains of utilization of HPC environments for typ-ical history matching problems. In all experiments, we have used 100 ensemble members.The experiments were performed using different values of NRun and NpSim for both EnKFand ES algorithms. Based on previous results on the number of processors for each simu-lation runs, we chose a maximum value of NpSim=4. Also, because of limited number ofavailable nodes (processors) in the cluster (bevo2), the maximum number of concurrent jobs

15

Table 4: Summary of simulation wall-clock time (in second) using different GMRES precon-ditioners with four processors in IPARS for 25 prior realizations.

PRECNpSim=4

No Forcing Forcing=0.1Fastest Ave. Slowest Del T Fastest Ave. Slowest Del T

3 431 608 1240 809 102 133 209 1074 354 440 657 303 76 105 135 596 288 366 428 140 87 117 155 687 354 449 598 244 95 118 175 808 500 626 1219 719 135 184 289 15411 307 452 1480 1173 108 136 184 7613 269 368 436 167 102 132 155 5314 243 330 385 142 90 115 136 4615 243 356 611 368 106 135 154 4816 214 300 354 140 77 99 113 3617 265 374 447 182 117 154 180 63

Figure 9: Simulation wall-clock time (seconds) vs. number of processors for one prior ensemblemember. The blue curve represents an end-time of 2800 days using the left y-axis. The red curverepresents an end-time of 150 days using the right y-axis.

(simulation runs) is chosen to be NRun=20. Since the focus here is to evaluate the parallelperformance of the framework, we have used the same parameters in all the experiments. Inparticular, the same initial ensemble members and same set of perturbed observation dataare used for all experiments.

The cumulative computational time of completion of an experiment consists of two maincomponents, ‘Forecast Time’ and ‘Analysis Time’, i.e., ‘Cumulative Time’=‘Forecast Time’+ ‘Analysis Time’. The ‘Forecast Time’ is the cumulative time computed from submittingall the jobs after the current assimilation step or initial step to the completion of all jobsbefore the next assimilation (or update) step. Therefore, the waiting times in the queueof cluster and actual simulations run-time are included in the ‘Forecast Time’. For bevo2cluster, the queue waiting time is negligible compared to the actual simulation run-time.The ‘Analysis Time’ is the cumulative time of performing all the update steps in EnKFor a single update step in ES. This time is computed from the completion of the slowestsimulation in the final NRun concurrent submitted jobs to the start of the next forecast step.Therefore, the time of reading and writing files ( restart files, model parameters files, etc.)

16

Table 5: Total forecast and cumulative execution time (in seconds) for EnKF with 100ensemble members as a function of number of concurrent forward runs (NRun) and numberof processors for each simulation run (NpSim).

NpSim Computational TimeNRun

1 2 4 5 10 20

1Forecast Time 62763 31514 15840 12692 6467 3316Cumulative Time 63381 32102 16433 13281 7069 3899




into disk is also included in this time.Table 5 summarizes the computational cost of performing EnKF in terms of ‘Forecast

Time’ and ‘Cumulative Time’ as a function of NRun and NpSim. The presented resultsinclude the cumulative time of performing 8 assimilation steps, with the last step being attime 1550 days. It can be seen from Table 5 that the wall-clock execution time significantlydecreases as the values of NRun and NpSim increase. In a sequential mode, with NRun=1and NpSim=1, it takes a full 17.6 hours to complete the history matching run. Whereas,with NRun=20, the execution time reduces to 1.1 hours using NpSim=1, and is furtherreduced to 0.63 hours with NpSim=4. Ideally the execution time should become half if onedoubles the NRun. However, the results presented in Table 5 show that the performancedeviates from the ideal behavior as we increase NRun. This is mainly due to the processorswaiting for the slowest simulation run to finish. We note that although the optimal choice ofpreconditioner has been used in the forward runs as a way of synchronization of concurrentsimulation jobs, there are still some time differences (‘Del T’) among different runs, hencethe waiting time. Although increasing the NpSim in each experiment reduces the executiontime, this reduction is not proportional to the increase in the number of processors. This islargely due to the additional communication overhead between processors, as each simulationtask requires opening and closing the MPI library through MPI-INIT and MPI-FINALIZE.Moreover, the small size of the problem and short simulation time between assimilation stepsboth cause a deviation from optimal scaling.

According to the results of Table 5, there are small fluctuations in the amount of the‘Analysis Time’ (= ‘Cumulative Time’ - ‘Forecast Time’) for different experiments. Theaverage amount of the ‘Analysis Time’ is 598 seconds for all 24 experiments presented here.This is about 1% of the total execution time for the case with a single processors (NpSim=1)in a sequential mode (NRun=1). Since the presented example is similar to “realistic” historymatching problems, seeing such a small percentage of time for the update steps is encouragingto efficiently parallelize the forecast step which is the most time consuming part of any historymatching task.

Next we present the summary of the parallel performance of the same set of experimentswith the ES algorithm in Table 6. For ES, all of the observed data from 0 to 1550 daysis assimilated in one step after the completion of the all simulation runs. Similar to EnKFexperiments, the execution time of ES is significantly reduced as we move from sequentialmode to parallel mode by increasing both NRun and NpSim. Results show we obtain a

17

Table 6: Total forecast and cumulative execution time (in seconds) for ES with 100 ensem-ble members as a function of number of concurrent forward runs (NRun) and number ofprocessors for each simulation run (NpSim).

NpSim Computational TimeNRun

1 2 4 5 10 20





(a) EnKF (b) ES

Figure 10: Parallel efficiency (Ep) as a function of number of concurrent forward runs (NRun) andnumber of processors for each simulation run (NpSim) with 100 ensemble members. a EnKF. b ES.

better parallel speed-up with ES than EnKF. For example, in the cases with NRun=1 andNpSim=4, the speed-up is about 2.7 for the ES algorithm as opposed to the speed-up of1.95 for the EnKF algorithm. This is mainly due to the fact that the forward simulationruns are longer in ES and there is no frequent stops during the forecast step. Therefore, theutilization of a parallel simulator is even more beneficial in the case of the ES implementation.Although all the observed data are assimilated at the same time in ES by computing a globaltime-space Kalman gain matrix, according to the results of Table 6 the ‘Analysis Time’ isstill a small fraction of the total computation time. The average of ‘Analysis Time’ for allthe experiments is about 128 seconds which is less than 1% of the cumulative time for thecase of sequential run, i.e. NRun=1 and NpSim=1.

Another important observation that can be made from the comparison of the resultspresented in Tables 5 and 6 is that the implementation of ES algorithm significantly reducesthe history matching time compared to the EnKF implementation. In particular for thepresented example, the ratio of the cumulative time of EnKF to ES for the sequential runs isequal to 1.96. In other words, the implementation of ES is almost twice as fast as the EnKFimplementation. Considering the amount of computational gain, the ease of implementationand the presented history matching results, we note that the ES could be considered as an

18

efficient ensemble-based algorithm for history matching problems.Figure 10 shows the parallel efficiency of implementing EnKF and ES algorithms. Note

the results presented in this figure correspond to the same experiments presented in Tables 5and 6. The implementations of ES gave better parallel efficiency than EnKF. The minimumvalue of Ep for ES is about 53% for the case with NRun=20 and NpSim=4, whereas the Epof EnKF for the same case is 35%.

While the results presented in this section are encouraging for the implementation ofparallel framework, they are specific to the characteristics of the cluster (bevo2). We notethat we have performed many other synthetic examples on bevo2 and obtained similar par-allel performances. Based on our previous experiments on different HPCs [27], we believethat one should obtain similar performances with the implementation of this parallel frame-work for other computational environments. Additionally, although the propagation of theensemble members during the forecast step(s) are independent, the parallelization does notyield into optimal scalability. In oder words, the term “embarrassingly parallel” which hasbeen used frequently in the implementation of the ensemble-based algorithms, in particularEnKF, is not equivalent to perfect speed-up for the history matching problems.

6 Conclusions

We presented the application of a parallel ensemble-based framework for history matchingproblems. Two data assimilation algorithms based on the Kalman filter have been imple-mented, the ensemble Kalman filter and the ensemble smoother. In this paper, the inherentparallelism of the forecast step is considered by running multiple concurrent realizations ondifferent partitions of a massively parallel computer. Furthermore, each of the realizationuses a parallel reservoir simulator.

We have used this parallel ensemble-based framework for a synthetic 3D history matchingproblem. Comparing the results of EnKF and ES algorithms, we observed that the EnKFgave better estimation of model parameters and better data match than the ES. However,assimilating the observed data twice with ES improved the history matching results andreduced the uncertainty in prediction of future performance of the reservoir. We are currentlyimplementing the ES algorithm to more realistic parameter estimation problems and we willfurther compare the results with the EnKF applications.

In addition, we performed sensitivity tests for an optimal choice of GMRES precondi-tioner during the forward simulation runs of IPARS. According to the presented results,the algebraic multigrid (AMG) with LSOR smoothing steps (PREC=16) was the best pre-conditioner such that it gave the fastest simulation runs with close simulation run-times(synchronized). Then, the EnKF and ES algorithms have been extensively tested usingthis optimal preconditioner with different experiments to examine the parallel efficiency ofthe parallel ensemble-based framework. The computational results showed that the anal-ysis step(s) takes only about 1% of cumulative time, the ES is twice as fast as the EnKFimplementation and the minimum parallel efficiencies are about 35% and 53% for EnKFand ES, respectively. In our future studies, we will utilize this parallel framework on dif-ferent massively parallel computers and we will perform realistic parameter estimations anduncertainty assessments for data based on CO2 sequestration fields.

19

7 Acknowledgments

A portion of this research was supported by the U.S. Department of Energy, Office of Science,Office of Basic Energy Sciences. The Center for Frontiers of Subsurface Energy Security(CFSES) is a DOE Energy Frontier Research Center, under Contract No. DE-SC0001114.

20

Bibliography

[1] Aanonsen SI, Nævdal G, Oliver DS, Reynolds AC, Valles B (2009) The ensembleKalman filter in reservoir engineering–a Review. SPE Journal 14(3):393–412, DOI10.2118/117274-PA

[2] Eisenstat S, Walker H (1996) Choosing the forcing terms in an inexact newton method.SIAM Journal on Scientific Computing 17(1):16–32

[3] Evensen G (1994) Sequential data assimilation with a nonlinear quasi-geostrophic modelusing Monte Carlo methods to forecast error statistics. Journal of Geophysical Research99(C5):10,143–10,162

[4] Evensen G (2004) Sampling strategies and square root analysis schemes for the EnKF.Ocean Dynamics 54(6):539–560

[5] Evensen G (2005) The combined parameter and state problem. In: Technical Report,Norsk Hydro Research Center

[6] Evensen G (2007) Data Assimilation: The Ensemble Kalman Filter. Springer

[7] Evensen G, van Leeuwen PJ (2000) An ensemble Kalman smoother for nonlinear dy-namics. Monthly Weather Review 128(6):1852–1867

[8] Evensen G, Hove J, Meisingset HC, Reiso E, Seim KS, Espelid O (2007) Using theEnKF for assisted history matching of a North Sea reservoir model (SPE 106184). In:Proceedings of the 2007 SPE Reservoir Simulation Symposium

[9] Gao G, Zafari M, Reynolds AC (2006) Quantifying uncertainty for the PUNQ-S3 prob-lem in a Bayesian setting with RML and EnKF. SPE Journal 11(4):506–515

[10] Gu Y, Oliver DS (2005) History matching of the PUNQ-S3 reservoir model using theensemble Kalman filter. SPE Journal 10(2):51–65

[11] Houtekamer PL, Mitchell HL (2001) A sequential ensemble Kalman filter for atmo-spheric data assimilation. Monthly Weather Review 129(1):123–137

[12] Kalman RE (1960) A new approach to linear filtering and prediction problems. Trans-actions of the ASME, Journal of Basic Engineering 82:35–45

[13] Keppenne CL (2000) Data assimilation into a primitive-equation model with a parallelensemble Kalman filter. Monthly Weather Review 128(6):1971–1981

[14] Keppenne CL, Rienecker MM (2002) Initial testing of a massively parallel ensembleKalman filter with the Poseidon isopycnal ocean general circulation model. MonthlyWeather Review 130(12):2951–2965

21

[15] Kuznestov Y (1969) Some problems in the theory and applications of iterative methods.PhD thesis, Novosibirsk, Russia

[16] Lacroix S, Vassilevski Y, Wheeler M (2000) Iteretive solvers of the implicit parallelaccurate reservoir simulator (ipars), i: Single processor case. TICAM report 00-28, TheUniversity of Texas at Austin, Austin

[17] Lacroix S, Vassilevski Y, Wheeler M (2001) Decoupling preconditioners in the implicitparallel accurate reservoir simulator (ipars). Numerical linear algebra with applications8(8):537–549

[18] Lacroix S, Vassilevski Y, Wheeler J, Wheeler M (2003) Iterative solution methods formodeling multiphase flow in porous media fully implicitly. SIAM Journal of ScientificComputing 25-3:905–926

[19] van Leeuwen PJ, Evensen G (1996) Data assimilation and inverse methods in terms ofa probabilistic formulation. Monthly Weather Review 124:2898–2913

[20] Liang B, Sepehrnoori K, Delshad M (2009) An automatic history matching module withdistributed and parallel computing. Petroleum Science and Technology 27(10):1092–1108

[21] Lu B, Wheeler MF (2009) Iterative coupling reservoir simulation on high performancecomputers. Pet Sci 6:43–50

[22] Nævdal G, Mannseth T, Vefring EH (2002) Near-well reservoir monitoring through en-semble Kalman filter: SPE 75235. In: Proceedings of SPE/DOE Improved Oil RecoverySymposium

[23] Nævdal G, Johnsen LM, Aanonsen SI, Vefring EH (2003) Reservoir monitoring andcontinuous model updating using ensemble Kalman filter. SPE 84372

[24] Saad Y (1996) Iterative Methods for Sparse Linear Systems. PWS Publishing Co.,Boston

[25] Skjervheim JA, Evensen G, Aanonsen SI, Ruud BO, Johansen TA (2007) Incorporat-ing 4D seismic data in reservoir simulation models using ensemble Kalman filter. SPEJournal 12(3):282–292, DOI 10.2118/95789-PA

[26] Skjervheim JA, Evensen G, Hove J, Vabo JG (2011) An ensemble smoother for assistedhistory matching. SPE 141929, presented at the SPE Reservoir Simulation Symposium,February 21-23, 2011

[27] Tavakoli R, Pencheva G, Wheeler MF (2011) Multi-level parallelization of ensemblekalman filter for reservoir history matching. SPE 141657–PP, presented at the SPEReservoir Simulation Symposium, February 21-23, 2011

[28] Vassilevski Y (2000) Iterative solvers for the implicit parallel accurate reservoir simula-tor (ipars), ii: Parallelization issues. In: Numer. Linear Algebra Appl, Citeseer

[29] Wheeler J, et al (2002) IPARS User’s Manual. Center for Subsurface Modeling, ICES,The University of Texas at Austin

22

[30] Wheeler MF (2001) Advanced techniques and algorithms for reservoir simulation, II:the multiblock approach in the integrated parallel accurate reservoir simulator (IPARS).Tech. Rep. 01-25, ICES, The University of Texas at Austin, Austin, TX

[31] Xin L (2008) Continuous reservoir model updating by ensemble kalman filter ongrid computing architectures. Ph.D. thesis, Louisiana State University, Baton Rouge,Louisiana

23

ices report 12-01 petroleum reservoir parameter estimation

Documents