Optimal Well Control Based on Auto-Adaptive Decision Tree—Maximizing Energy Efficiency in High-Nitrogen Underground Gas Storage

Abstract

To move the world toward a more sustainable energy future, it is crucial to use the limited hydrocarbon geological resources efficiently and to develop technologies that facilitate this. More rational management of petroleum reservoirs and underground gas storage can be obtained by optimizing well control. This paper presents a novel approach to optimal well control based on the combination of optimal control theory, innovative artificial intelligence methods, and numerical reservoir simulations. In the developed algorithm, well control is based on an auto-adaptive parameterized decision tree. Its parameters are optimized by state-of-the-art machine learning, which uses previous results to determine favorable parameters. During optimization, a numerical reservoir simulator is applied to compute the objective function. The developed solution enables full automation of the wells for optimal control. An exemplary application of the developed solution to optimize underground storage of gas with high nitrogen content confirmed its effectiveness. The total nitrogen content in the gas decreased by 2.4%, increasing energy efficiency without increasing expense, as only well control was modified.

1. Introduction

The geological resources of hydrocarbons are limited, and therefore, it is necessary to use them efficiently. In industrial practice, well control is most often selected based on production scenario analysis. Testing various production variants is a manual iterative process based on expert knowledge. However, such an approach does not guarantee obtaining the best (optimal) solution in terms of the adopted quality criterion. Therefore, the effectiveness of the production process is low. One of the possibilities to increase the technical and economic efficiency of the production process is to optimize well control, which translates into more rational management of natural resources.

An important aspect of rational management of resources related to short-term (operational) and seasonal (tactical) energy security is the storage of large volumes of natural gas. The gas stored in periods of low consumption (summer) can then be used to cover increased demand for gas in winter. The stored gas can also be used in the event of a break in gas supplies as the result of extraordinary situations or to cover differences in demand during the day. In practice, the storage of large volumes of natural gas is possible through underground gas storage (UGS) in partially depleted gas reservoirs [1]. Optimal well control is increasingly important in this case.

Underground gas storage in partially depleted reservoirs is most often associated with the problem of water coning, which can reduce gas withdrawn effectiveness [1]. The gas cushion in the reservoir is designed to keep water a safe distance from the production wells. However, improper gas injection and withdrawal may lead to undesirable water production. Moreover, during underground gas storage in a partially depleted reservoir, the cushion gas mixes with the active gas [1]. Typically, the stored gas has properties similar to those of the cushion gas. However, in the case of large differences in the properties of the gas from these two parts, the produced gas may have a different quality than the gas that has been injected into the storage. A problem occurs if the stored gas is of a higher quality (e.g., methane) than the cushion gas (e.g., nitrogen). The quality of the withdrawn gas is then significantly influenced by the injection and withdrawal. Hence, it is important to operate the storage in a way that reduces the adverse impact of the factors mentioned above.

To correctly determine the optimal control, it is necessary to take into account the control theory, including the optimal control theory. Control acts on a given system to achieve the desired behavior of this element of the environment [2]. An object under control is called the control process. The control signals are generated by the controller. The control process together with the controller constitute the control system [2]. When the control process is a real industrial process, the problem of control becomes difficult to solve due to the high complexity of such processes. When the control problem aims to obtain the best quality indicator, then it comes down to the optimal control problem. The solution to this problem is even more difficult.

One of the basic tools for solving the problem of optimal control is the theorem known as Pontryagin’s maximum principle. This method allows determination of the optimal process control depending on the state variables and the introduced conjugate variables [2]. Then, after solving a system of state equations and a system of conjugate equations, it is possible to determine the optimal process control as a function of time [2]. Another way to solve the problem of optimal control is Bellman’s optimality principle. This principle introduces the statement that optimal control depends only on the current state of the system, not on the previous states [2]. Therefore, for a given optimal trajectory, each final section is an optimal trajectory for itself. Therefore, with the use of dynamic programming, it is possible to determine optimal control [2]. In some special cases, dynamic programming is a very useful method, but in most optimal control problems, the calculation effort for this method is much greater than that required for the maximum principle [2]. Apart from the mentioned main analytical methods that enable determination of the optimal control, there are many others, such as functional analysis [2]. However, determining the optimal control of real industrial processes using analytical methods is difficult, and sometimes even impossible due to the high complexity and nonlinear nature of the problem. Then, numerical methods are most often used. However, numerical methods also have some limitations, such as considerable computation time. Therefore, artificial intelligence methods are increasingly used to control industrial processes. Artificial intelligence (AI) is a branch of computer science that includes techniques allowing computers to assist or replace humans in solving nontrivial problems perceived as requiring human intelligence [3]. Advantages of AI-based solutions to the optimal control problem include a better understanding of the controlled process by defining rules, using expert knowledge, reducing process costs by automating activities, and improving process efficiency as a result of learning from the experience gained [4]. Moreover, their combination with industrial data enables automatic control of real processes [5].

Currently, there is a growing interest in the application of artificial intelligence to optimize various processes related to hydrocarbon production. This has translated into an increasing number of scientific publications in renowned journals related to this topic [6,7,8]. To develop an optimal production strategy, more and more often nature-inspired optimization algorithms that mimic natural mechanisms increasing the evolutionary adaptability of individuals to the environment are applied. Nature-inspired algorithms encompass evolutionary algorithms; swarm intelligence-based algorithms such as particle swarm optimization inspired by the social behaviors exhibited by swarms of animals; and physics-based algorithms inspired by physical rules in nature, such as simulated annealing that mimics steel annealing [7]. Several nature-inspired optimization algorithms, such as particle swarm optimization and the genetic algorithm, have been evaluated by the authors to optimize polymer injection [7] and gas condensate reservoirs [9]. In [9], the authors confirmed that nature-inspired algorithms outperform gradient optimization. However, the main disadvantage of the solutions most often proposed in the literature, such as genetic algorithms or other evolutionary methods, is the fact that they require thousands of simulations. In solutions where the numerical simulation is used to evaluate the objective function, convergence speed is the crucial parameter. Moreover, most of the research is carried out on synthetic, significantly simplified reservoir models, for which these methods are performed in finite time and give positive results. However, in the case of a real hydrocarbon reservoir model with a high degree of complexity (resulting from the complex geological structure and heterogeneity of the reservoir), the duration of a single simulation can take several hours. For such cases, these methods are no longer applicable due to their duration.

In this study, a novel procedure that automatically optimizes hydrocarbon reservoir and underground gas storage well control is proposed. The idea of the developed solution is to determine well control based on an auto-adaptive parameterized decision tree. The limit values for the particular attributes of the decision tree are replaced with parameters that are optimized with the use of state-of-the-art machine learning. During optimization, the objective function is computed with the numerical reservoir simulator. The auto-adaptive decision tree is linked to the reservoir simulator and the optimization tool by the novel procedure proposed in this paper. In contrast to existing solutions, the developed algorithm ensures high time and computational efficiency, enabling its application to real industrial problems.

This work is divided into six parts. The first part is an introduction to the subject of the paper and an introduction to the theoretical basis of industrial process control. Then, the problem of optimal well control is defined, and the possibility of applying potential solutions to the analyzed problem by using artificial intelligence is analyzed. In part three, the algorithm developed for determining well control based on auto-adaptive decision trees is described. In the next part, an example of the developed solution is presented. The fifth part contains the results and discussion, including potential limitations. The last part contains the conclusions.

2. Optimal Well Control

2.1. General Overview of the Optimal Well Control Problem

The use of classical methods to optimize well control is difficult because, in this case, it is impossible to link the decision variables to the optimized quality indicator in a functional manner [10]. This is due to the properties of hydrocarbon reservoirs, which are characterized by a complex geological structure with complicated geometry and spatial heterogeneity of petrophysical parameters, as well as the multidimensional physics of fluid flow, which complicates the description of the system’s dynamics [11]. Hence, the processes taking place in hydrocarbon reservoirs are modeled by reservoir simulators, which forecast reservoir behavior. Reservoir simulators are based on the following general equation describing the dynamics of a hydrocarbon reservoir [12]:

$$\frac{dx}{dt}=f\left(x,\nabla x,{\nabla }^{2}x,u\right)$$

(1)

where

x 

state vector,

f 

function describing the reservoir condition,

u 

control.

The form of the state vector and the shape of the function describing the state of the reservoir depend on the mathematical model adopted to describe the dynamics of the reservoir.

The purpose of an optimal reservoir or UGS control is to define a production strategy that minimizes or maximizes the quality indicator, taking into account the limitations resulting from the physical nature of the problem presented in the form of differential equations describing the dynamics of the analyzed system [12]. The optimization objective may include: minimizing water production, maximizing the calorific value of the extracted gas, or maximizing the economic value of the project. Therefore, the problem discussed in this paper is the selection of such control that allows the extremization of the adopted objective function.

In the case of hydrocarbon reservoirs, there are two possible well control options. One of them is to maintain a constant top hole pressure (THP), which results in variation in well performance. The second option is to assume a constant flow rate, which causes pressure variability. Reservoir control depends on the functions controlling individual well performance:

$$u\left(t\right)=q\left(t\right)\vee u\left(t\right)=p\left(t\right)$$

(2)

where

u(t) 

well control function,

q(t) 

well performance as a function of time,

p(t) 

THP as a function of time.

During the production of a reservoir/UGS, the flow rates of the wells change over time. For practical reasons, the wells’ performance changes over a period of time, during which it is kept constant. As a result, the reservoir/UGS control can be presented in the form of a vector including controls in subsequent periods:

$$u_r=\left(\begin{array}{cccc}{u}_{r_1}& u_{{\mathit{r}}_2}& \cdots & u_{r_M}\end{array}\right)$$

(3)

where

u_r 

reservoir/UGS control,

M 

number of time steps.

Moreover, the control of the reservoir/UGS depends simultaneously on the operation of all wells located there. Thus, the reservoir/UGS control problem consists in determining the control of all wells in each time step and takes the following form:

$$u_{r_i}=\left(\begin{array}{c}{u}_{1_i}\\ u_{2_i}\\ ⋮\\ u_{W_i}\end{array}\right)$$

(4)

where

u_ri 

control of the reservoir/UGS in a particular time step,

u_i 

well control in a particular time step,

W 

number of wells located in the reservoir/UGS.

Taking into account the above assumptions, optimization of the reservoir/UGS operation is aimed at determining optimal control in the following form:

$${\hat{u}}_r=\left(\begin{array}{cccc}{\hat{u}}_{11}& {\hat{u}}_{12}& \cdots & {\hat{u}}_{1M}\\ {\hat{u}}_{21}& {\hat{u}}_{22}& \cdots & {\hat{u}}_{2M}\\ ⋮& ⋮& \ddots & ⋮\\ {\hat{u}}_{W1}& {\hat{u}}_{W2}& \cdots & {\hat{u}}_{\mathit{WM}}\end{array}\right)$$

(5)

where

${\hat{u}}_r$

optimal control of the reservoir/UGS,

$\hat{u}$

well optimal control,

M 

number of time steps,

W 

number of wells located in the reservoir/UGS.

Moreover, when determining the optimal control of a reservoir/UGS, it is necessary to take into account inviolable constraints in the form of:

• bottom hole pressure (BHP) in each of the wells,
• allowed performance of each well, i.e., the maximum flow rate with which a given well can produce reservoir fluids.

2.2. Potential AI-Based Solutions to the Optimal Well Control Problem

As the use of commonly known methods of determining optimal control to solve the problem of optimal well control is limited, it is justified to consider the possibility of using artificial intelligence methods. To solve the analyzed problem, various decision models based on a wide variety of artificial intelligence techniques can be used. The solutions most suited to the analyzed problem are discussed below.

2.2.1. Support Vector Machine

A possible solution to the problem of well control is to initially select a very large number of features of the reservoir simulation model, and then build a model that imitates the behavior of the reservoir simulation. For this purpose, a support vector machine (SVM), commonly used in regression and classification problems, could be applied. In the case of regression, this algorithm allows determining the hyperplane around which the points from the training set are as close as possible (the least squares method) [13]. This algorithm can be effectively used for multivariate problems by applying a nonlinear transformation (kernel) to the training data [14]. The kernel transforms the nonlinear input data into a certain multidimensional space in which the data is arranged linearly [14]. In this solution to the analyzed problem, linear regression regularization methods such as LASSO or ridge regression could then be used to extract the most relevant features. LASSO minimizes the absolute sum of the coefficients of the regression model (L1 regularization), whereas ridge regression minimizes the sum of squared coefficients (L2 regularization) [15].

In the case of the analyzed problem, the issue with applying the discussed solution would be the selection of initial features because the reservoir simulation model is characterized by a large number of reservoir and production parameters. Selecting the most essential features to be taken into account would be difficult and ambiguous. Moreover, an additional problem would be the limited amount of historical data available, which could lead to overfitting of the surrogate model. As a result, this would translate into low reliability of the final results.

2.2.2. Deep Neural Networks

Another potential solution to the analyzed problem is the use of deep neural networks that effectively solve difficult and complex problems for which traditional mathematical methods are ineffective [16]. Deep neural networks are a key group of deep learning algorithms. Deep learning is a new branch of machine learning that includes algorithms that enable a more abstract representation of high-level features through the use of a large number of layers of nonlinear transformations [16,17]. The successive layers of the deep network represent the features ordered from the least to the most abstract. The representation of higher-level features is created using many hidden layers connected to the low-level features [17]. The discussed solution enables automatic extraction of features that are important for a given problem based on a given dataset [17]. Traditional feature extraction methods are often limited by the prior knowledge of experts, whereas deep neural networks can select the most important features without their participation.

In the case of the analyzed well control problem, it would be possible to learn the deep network of interesting features of the reservoir simulation model based on historical data. If the deep network model were to mimic the simulation, the final layer of the deep neural network could be torn off and the rest of it would constitute the feature model. Then, a layer that returns performance for each well could be added to such a model. In such a case, the network would make control decisions in a given time step based on the most important features of the reservoir simulation model that had been learned previously. However, deep neural networks have several major disadvantages. The biggest one is that the greater the number of hidden layers, the greater the number of parameters need to be learned. Thus, a very large amount of training data is required, which is often difficult to obtain for a real object. When many layers are trained on a small amount of data, the network overtrains very quickly, losing the ability to generalize, which negatively affects the final results.

2.2.3. Convolutional Neural Network

A possible solution is also the use of a convolutional neural network, the idea of which is to filter the input signal to discover the characteristics of the input data. The purpose of the convolution operator used is to capture the neighborhood features of a given element by moving the filter (weight matrix) over the input signal within a smaller window [18]. In the case of convolutional neural networks, not all neurons of subsequent layers are connected, which reduces the number of network parameters [17]. The convolutional network consists of a convolutional layer, which is a set of filter maps, a pooling layer, the task of which is to reduce network size by calculating simple activation statistics, and a full (output) layer, which, based on the discovered features, provides conclusions adequate to the analyzed problem [18].

In the case of the analyzed problem, the output of the convolutional network would represent the features of the simulation model. This convolutional network output would be the input to the next network, which would select the performance of production wells based on these characteristics. In addition, the advantage of the discussed solution is the ability to check the vicinity of the sample, which would allow reacting to the situation near the well. However, despite the advantages of a convolutional network, the interpretation of its parameters is difficult. For the analyzed problem concerning a real object, the physical interpretation of the chosen decision model is essential. In addition, there is still the problem of the finite amount of historical data on which the network would learn, which could lead to overfitting of the network.

2.2.4. Decision Trees

Another potential solution to the analyzed problem is the use of decision trees. In the internal nodes of the decision tree, there are individual attributes whose limit values are assigned to the branches, whereas the possible inference results are contained in the leaves of the tree [19]. The path from the root of a decision tree to a specific leaf is a conjunction of the conditions determining the decision. ID3 is the most famous decision tree learning algorithm, which enables automatic generation of a decision tree based on a labeled dataset [19]. The selection of subsequent attributes is made based on the greatest information gain.

Due to the practical limitations related to the production of petroleum reservoirs/UGS, it is sufficient for the decision model to determine the control of a single well in a particular time step. Then, such a decision model can be used for each well located on a given reservoir/UGS during all time steps. Decision trees constitute a solution that is well suited to the described situation. They represent a structured scheme enabling easy interpretation of the physical basis of the decisions made. At the nodes of the decision tree, there could be attributes such as the reservoir pressure and leaves that could define control.

However, the direct application of the algorithm generating the decision tree would be difficult due to the large number of time-varying attributes that could be analyzed. Moreover, historical control decisions are most often made solely based on the operator’s knowledge. Thus, control based on the decision tree generated using such historical data would also not be optimal.

Therefore, the structure of the decision tree would have to be developed based on engineering experience and good industrial practices that would enable a fairly accurate identification of the attributes influencing the decisions made. The problem with this approach would be how to select the limit values for decision tree attributes. Making decisions about their values based on engineering knowledge and experience does not guarantee the determination of optimal control.

2.2.5. Parameterized Decision Trees

A concept similar to the discussed decision trees, but enabling the elimination of the problem resulting from arbitrary selection of the limit values for decision tree attributes, is the use of parameterized decision trees (Figure 1). Parameterization by replacing the limit values assigned to tree branches with parameters enables the optimization of decision conditions. Artificial intelligence methods can be used to optimize tree parameters, in particular machine learning algorithms, enabling the automatic determination of parameters of a given decision model.

The discussed solution could be successfully used for well control due to its versatility and simplicity of interpretation. In addition, this method allows the enrichment of engineering knowledge and good industrial practices used to develop a decision tree structure using artificial intelligence methods that allow the selection of optimal limit values of attributes, which would improve the quality of decisions. Moreover, the advantage of the discussed solution is the possibility of using the decision tree constructed for a given problem even in conditions different from the base ones, through its reconfiguration consisting of the selection of the optimal limit values of the attributes under changed conditions.

A complication is the fact that reservoir simulations are used to calculate the value of the objective function when optimizing the operation of reservoirs/UGS. Moreover, commonly used optimization methods require a very large number of iterations. Therefore, the possibilities to use artificial intelligence methods that enable the optimization of parameters, such as genetic algorithms, are limited due to the long evaluation time of a single simulation of a real reservoir.

2.2.6. Model-Based Sequential Algorithm Configuration

A method designed to solve “black box” problems with high computational cost is the sequential configuration of algorithms based on search space [20]. The algorithm configuration procedure consists in adapting the existing general algorithm to the given problem, allowing the replacement of the expert’s attempts by automatic determination of algorithm parameters [21]. The use of surrogate models in the optimization process speeds up the calculation, and the use of the sequential strategy improves the quality of the surrogate model during optimization [21,22]. The discussed method is characterized by high calculation efficiency due to the use of adaptive sampling, in which parameter values are selected based on the criterion of expected improvement of the solution [22]. These parameters balance the exploration of the search space to improve the quality of the surrogate model and the exploitation of this model to improve optimization [22].

Based on the discussed method, an optimization tool named Sequential Model-based Algorithm Configuration (SMAC) was developed. SMAC learns from the results collected during optimization which parameter values have a greater chance of improving the quality of the solution. Based on user-defined domains of decision variables (parameters of a given model) and their initial values, this tool repeats the three main steps in each iteration [20,21]:

• builds a model of the search space in the form of a random forest, based on the data collected so far,
• selects such parameter values which, based on the constructed model, enable the best improvement of the quality of the solution, i.e., those that are expected to be good and those that are located in the area that has not yet been tested,
• determines the value of the objective function using the target algorithm, which enables the determination of the exact value of the quality index for a given set of parameters, thus creating a new dataset.

The result of SMAC is a set of parameter values of a given model that minimizes a given quality [20].

Sequential configuration of the algorithms based on search space is a potentially well-suited solution to the well control optimization problem. Due to the use of a surrogate model limiting the number of evaluations of the target algorithm itself, the reservoir simulation would be run in each iteration only for initially optimized parameters, which would reduce computation time. In the case of the oil and gas industry, most model-based solutions consist of a complete replacement of the reservoir simulation with a surrogate model (most often in the form of an artificial neural network). However, such an approach does not guarantee the correctness of the calculations performed. Thus, an additional advantage of the discussed method is the fact that the solutions selected as potentially optimal would be evaluated directly with the use of a reservoir simulation mapping the behavior of the reservoir with high accuracy.

Moreover, the combination of the discussed concept of parameterized decision trees with the SMAC optimization tool would constitute a clear and computationally effective procedure for optimizing reservoir/UGS control [1,23].

3. Proposed Solution to Optimize Well Control

In this study, the combination of a parameterized decision tree with the SMAC optimization tool linked directly to the ECLIPSE reservoir simulator is proposed to solve the reservoir/UGS control problem. Determination of control based on the decision tree enables its unambiguous physical interpretation. SMAC determines optimal values of the parameters assigned to tree branches at a low calculation cost. The use of a highly complex reservoir simulation model to calculate the value of the quality indicator during the optimization process, instead of relying solely on the surrogate model, increases the reliability of the obtained results. The proposed solution is a new concept of auto-adaptive decision trees—parameterized decision trees that automatically adjust to a specific case.

3.1. Automation of Control Determination

As in the proposed solution, the decision tree limit values are changed by parameters; the reservoir/UGS control system is characterized by incomplete initial information. To automatically determine the control, an extreme control system has been developed. It finds the optimal settings of the controller parameters, which in the analyzed case is a decision tree.

The extreme control system enables the automatic determination of the operating point, which is extreme on the characteristic showing dependence of the given quality indicator on the tuned parameters. In such systems, there is no explicit input signal, and the change of the control signal is to keep one of the output signals at the minimum or maximum value [2]. The extreme control system is obtained by supplementing the conventional control system with an optimizing element, to automatically select the optimal values of the controller, parameters may use not only information about the quality indicator value but also about [2]:

• output—y,
• control—u,
• noise—z.

The implementation of this type of control in a closed-loop system does not require the knowledge of noise values, which is a big asset in the case of real industrial processes. The diagram of a closed-loop extreme control system, in which the quality indicator is related to the dynamics of the object, and the goal is to search for the optimal settings of the controller parameters, is presented in Figure 2.

The main part of the presented diagram is an optimizing element (automatic optimizer), which examines the effect of changes in the value of the control signal on the quality indicator J and changes it to achieve the maximum or minimum of the objective function [2]. Such an automatic search method reduces the necessary initial information, as the automatic optimizer collects the necessary information about the search space by examining the object’s response to a trial input signal [2]. However, in the case of real industrial processes, it is often impossible to use the real object to determine optimal working conditions. In such cases, the object model in the form of a system of equations or simulation of the object’s behavior is used, and only the obtained results are transferred to a real industrial object [2].

Due to the presented characteristics of extreme systems, they are a good representation of the developed solution. For the considered reservoir/UGS control problem, consisting of the determination of well control, a closed-loop extreme control system was constructed, as presented in Figure 3. In the proposed solution:

• the real object model is used in the form of a reservoir simulation model (reservoir simulator ECLIPSE),
• the operation of the reservoir/UGS during a single simulation depends on the control u in the form of the flow rate of all operating wells,
• a parameterized decision tree is used as the controller,
• the control of each well depends on the values of the tree attributes a returned by the simulator at each time step,
• the decision tree parameters p are processed by the automatic optimizer,
• the SMAC tool is used as an automatic optimizer,
• the SMAC tool extremizes the quality indicator J,
• the procedure calculating the quality indicator J is implemented directly into the ECLIPSE reservoir simulator so that one of the output values of the object model is the value of the objective function.

Thus, in the proposed solution to the reservoir/UGS control problem, control of each well in the reservoir is determined. At each time step of the reservoir simulation, reservoir/UGS control is determined based on the parameterized decision tree, and it is valid in the next time step. The optimizer (SMAC tool) receives the value of the quality indicator at the end of the entire reservoir simulation. Based on this value, the SMAC tool returns the next set of decision tree parameters. After performing the assumed number of iterations, the decision tree parameter values that maximize the given quality indicator are determined.

3.2. Implementation of the Developed Solution

To automatically determine reservoir/UGS control, the developed extreme control system, which combines a parameterized decision tree, the ECLIPSE reservoir simulator, and the SMAC optimization tool, is implemented in the developed solution using the Python programming language and the internal programming option offered by the ECLIPSE reservoir simulator.

Internal programming in the ECLIPSE reservoir simulator enables a direct declaration of the procedure for calculating the quality indicator. In the developed solution, it is also used for the direct implementation of a parameterized decision tree in the simulator. After declaration of the tree parameters, which are updated in subsequent iterations of the developed procedure, a set of actions—that are triggered when the conjunction of conditions assigned to tree branches is met—is defined.

The element requiring the development of an external IT procedure is the combination of the SMAC optimization tool and the reservoir simulator as well as the connection of the remaining elements of the developed extreme control system. Integration of the tools used in the proposed solution was carried out using the Python programming language. The following actions were programmed in the developed solution:

• reading the parameter values returned in subsequent iterations by the SMAC tool,
• saving read parameter values to a file loaded into the reservoir simulator,
• automatically running the reservoir simulation with updated decision tree limit values,
• loading the output file from the reservoir simulation,
• reading the value of the quality indicator from the loaded output file,
• passing the read value of the objective function to the SMAC optimization tool.

In practice, the developed solution begins by preparing the SMAC tool configuration file, containing, among other things, domains and starting values of the decision variables. The configured SMAC tool is then run. At each iteration, this tool calls a developed computer program written in the Python programming language.

Arguments of the developed program constitute the values of the decision tree parameters. This program reads the values of the parameters provided by the SMAC tool and runs the updated reservoir simulation. The simulation iteration depends on the well control determined at each time step based on the configured decision tree and attribute values calculated by the simulator. At the end of the simulation, the value of the quality indicator for a given control is calculated. The returned value of the quality indicator is transferred by the developed program to the optimization tool and is the basis for determining the next values of the decision tree parameters.

After performing the assumed number of iterations, the developed solution, which automatically improves the quality of the reservoir/UGS control, returns the values of tree parameters at which the wells are controlled in a way that extremizes the given quality indicator. Therefore, the developed procedure enables full automation of decision tree parameter optimization, which translates into automatic optimization of reservoir/UGS control.

4. Case Study—Underground Gas Storage with High Nitrogen Content

To confirm the effectiveness of the developed solution to optimize control, an example of its application was carried out on a simulation model of underground gas storage with unconventional cushion gas with high nitrogen content. In the analyzed case, gas with a high methane content is stored in the depleted nitrogen-rich natural gas reservoir. The presented nitrogen-rich UGS is only an example of a production problem that requires control optimization and can be solved with the proposed approach. It needs to be emphasized that the developed methodology is general and applicable to all kinds of UGS and hydrocarbon reservoirs.

4.1. Object and Model Definition

The reservoir in which the analyzed UGS was built has a complicated geological structure and the shape of an approximately semicircular elevation. The reservoir rock is Zechstein Dolomite with permeability 5–160 mD and porosity 2–22%. It is characterized by a double porosity system with a porous rock matrix and a fracture system. The area of the reservoir is 24 km³ with an effective thickness 31–39 m. The initial reservoir pressure was 164 bar. The primary geological resources of the reservoir were estimated at 11.9 billion m³. The native gas is nitrogen-rich, with an original nitrogen content of 30.7%.

After 23 years, the reservoir was converted into UGS. During this period 7.8 billion m³ of nitrogen-rich gas was recovered, which equals 65.5% of resources. The remaining 4.1 billion m³ is a cushion gas, which maintains pressure in the UGS and controls the movement of reservoir water [24]. The analyzed UGS is used to compensate for seasonal fluctuations in the demand for high-methane gas. Gas injection and withdrawal are performed with the use of 12 regularly distributed wells. Some of them were used during the reservoir production period, and some were drilled after the reservoir was converted into UGS. The analyzed UGS has been operating for 27 years, and initially its working capacity equaled 0.6 mld s·m³. After 10 years of operation, the working capacity was increased to 1.2 mld s·m³. The most important parameters of the storage installation are included in

Table 1. Parameters of the analyzed storage installation.

Parameter	Value
working capacity	1.2 mld s·m3
maximal injection rate	9.6 mln s·m3/day
maximal withdrawal rate	14.4 mln s·m3/day
nominal injection time	150 days/year
nominal withdrawal time	110 days/year

.

The simulation model of the reservoir in which the analyzed UGS is located was made with the use of a compositional simulator ECLIPSE 300 (Schlumberger Limited, Houston, TX, USA). It includes a three-component reservoir fluid containing hydrocarbon components from C1 to C2 and also N2. The reservoir simulation model consists of 7920 blocks defined by the coordinates of their vertices and includes six layers. As the reservoir rock is characterized by double porosity, the simulation model is divided into two identical regions (one reflecting the properties of the rock matrix and one of the fracture system). The reservoir model was created based on data collected from wells located on the reservoir as well as seismic records and then calibrated to historical production data.

4.2. Parameterized Decision Tree Model Definition

The considered UGS is an object that specifically requires optimization, as it stores high-methane gas while the cushion gas is nitrogen-rich. In the injection cycle, nitrogen-rich gas is displaced into further parts of the structure, but during withdrawal it flows to the wells, reducing the quality of the produced gas. As a result of the undesirable mixing of high-methane injection gas with nitrogen-rich native gas, the gas withdrawn from the UGS is of lower quality than the injected gas. The quality of the produced gas is significantly influenced by the UGS control method. The UGS should be controlled in such a way as to produce gas of the best quality, i.e., the highest energy.

Therefore, the problem of optimal control of the analyzed UGS is to determine the control that maximizes the total amount of energy that can be obtained from the gas produced during the withdrawal cycle, subject to the limitations in the form of differential equations describing the dynamics of the reservoir in which the storage is located. The simplified mathematical model of the analyzed optimization problem takes the following form:

$$E_k\left(u\right)=\sum _{j=1}^D\sum _{i=1}^S{C}_i{y}_{i,j,k}{u}_{j,k}$$

(6)

$$J\left(u\right)=\sum _{k=1}^W{E}_k\left(u\right)$$

(7)

$$J\left(\hat{u}\right)=\underset{u}{\max }J\left(u\right)$$

(8)

where

$E_k$

energy contained in gas withdrawn from k-th well,

J 

total energy contained in gas withdrawn from storage, objective function,

$u_{j,k}$

k-th well flow rate on the j-th day (control),

$\hat{u}$

optimal storage control,

$C_i$

heat of combustion of the i-th component of the withdrawn gas,

$y_{i,j,k}$

mole fraction of the i-th component in the withdrawn gas at the k-th well on the j-th day,

W 

number of wells,

S 

number of components in the withdrawn gas,

D 

number of days in the withdrawal cycle.

In the case of the analyzed UGS, the three main components are methane, ethane, and nitrogen. Thus, the higher the nitrogen content, the lower the mole fraction of hydrocarbon components in the withdrawn gas mixture. Moreover, the heat of combustion of methane is 39.831 MJ/m³, of ethane 70.330 MJ/m³, while nitrogen’s is zero. Therefore, increasing the mole fraction of nitrogen reduces the heat of combustion of the withdrawn gas. Thus, the nitrogen content directly affects the amount of energy that can be obtained from the gas produced during withdrawal. To maximize energy, the gas withdrawn from the storage must have the lowest nitrogen content possible.

Based on the conducted analysis of the behavior of the considered UGS during historical withdrawal cycles, a parameterized decision tree was developed, as presented in Figure 4. The presented diagram describes the decision-making process that determines the control of a given production well applicable to each subsequent day of the withdrawal cycle. The decision tree structure was based on analysis of the UGS control strategy used thus far, based on the engineering experience and practice of the operators.

In the proposed decision tree, nitrogen content is the most important factor influencing the total amount of energy that can be obtained from the withdrawn gas. Depending on the mole fraction of nitrogen in the gas mixture produced from the well, the production wells have been divided into three groups:

• low nitrogen content, the mole fraction of nitrogen in the withdrawn gas is greater than zero but lower than the value of p1,
• medium nitrogen content, the mole fraction of nitrogen is greater than or equal to the value of p1 and less than p2,
• high nitrogen content, the mole fraction of nitrogen is between p2 and the total nitrogen content.

In the proposed control scheme, the relationships that determine the control of a well belonging to a given group have been assigned to individual groups of wells.

When the gas withdrawn from a given production well has a high nitrogen content, its performance is reduced by a percentage corresponding to the value of p3:

$$q_{i+1}=\left(1-p3\right)q_i$$

(9)

where

$q_{i+1}$

flow rate of the well in the next time step,

$q_i$

flow rate of the well in a given time step.

If the mole fraction of nitrogen in the produced gas is at a medium level, the performance of such a well in the next time step remains unchanged:

$$q_{i+1}=q_i$$

(10)

where

$q_{i+1}$

flow rate of the well in the next time step,

$q_i$

flow rate of the well in a given time step.

The output of wells with low-nitrogen gas is increased. Due to the fact that the volume of gas withdrawn daily from storage is an imposed value, the increase in the output of these wells depends on the control of other production wells. In this case, three situations can be distinguished, depending on the size of the group of high-nitrogen wells:

• if there are no high-nitrogen wells in a given time step, then the performance of the wells in the analyzed group remains unchanged (Equation (10) is applied).
• if there are high-nitrogen wells and no low-nitrogen wells in a given time step, the flow rate of all wells is increased. The difference between the assumed daily output from the UGS and the volume of gas that could be withdrawn by the remaining wells after decreasing the output of high-nitrogen wells is evenly distributed:

  $$q_{i+1}=q_i+\frac{\Delta Q}{W}$$

  (11)

  where

  $q_{i+1}$

  flow rate of the well in the next time step,

  $q_i$

  flow rate of the well in a given time step,

  $\Delta Q$

  the difference between the assumed daily production from the UGS and the volume of gas withdrawn by the remaining wells (those with medium and high nitrogen content),

  W 

  total number of production wells located in the UGS;

• at each time step in which there are wells with high and low nitrogen content in the produced gas, the difference $\Delta Q$ is distributed evenly between the wells in the analyzed group:

  $$q_{i+1}=q_i+\frac{\Delta Q}{N}$$

  (12)

  where

  $q_{i+1}$

  flow rate of the well in the next time step,

  $q_i$

  flow rate of the well in a given time step,

  $\Delta Q$

  the difference between the assumed daily production from the UGS and the volume of gas withdrawn by the remaining wells (those with medium and high nitrogen content),

  N 

  number of production wells with low nitrogen content in the gas withdrawn from the storage at a given time step.

In the proposed solution, the performance of all production wells located in the UGS is dependent on their share of the total production. The initial shares of all wells are proportional to their productivity indexes. Then, on each day of the withdrawal cycle, well rates are automatically updated according to the proposed decision tree and the presented mathematical relationships. The developed decision scheme determines the withdrawal process while maintaining the assumed total daily production, which is essential from a practical point of view. To optimize gas withdrawal, the value by which the performance of wells with high nitrogen content should be reduced and the limit values separating individual groups of wells have been replaced by parameters determined by the SMAC optimization tool.

5. Results and Discussion

5.1. Reservoir Results

As a result of the application of the developed solution, the time-varying control of the analyzed UGS was determined automatically. The performance of the production wells for each day of the withdrawal cycle was determined based on the optimized decision tree. These performance values guarantee maximization of the energy obtained from the withdrawn gas.

To illustrate the effectiveness of the developed procedure and validate the optimization results, application of the developed methodology was performed on the historical withdrawal cycle performed on the analyzed UGS. This allowed historical data to be compared with the optimization results obtained for the same cycle. The most important parameters related to the analyzed problem were compared, i.e., the total nitrogen content in the gas mixture withdrawn from the UGS and the amount of energy that can be obtained from the gas produced during the analyzed withdrawal cycle. To present the computational efficiency of the developed procedure, the change in the objective function during the optimization process was also analyzed.

The domains of individual parameters of the developed decision model and their initial values, constituting the starting point in the process of UGS control optimization, are presented in

Table 2. The domains and initial values of the parameters of the developed decision model.

Parameter	Domain	Initial Value
p1	0.02–0.05	0.030
p2	0.02–0.05	0.037
p3	0–0.5	0.100

. Since p1 and p2 represent the mole fraction of nitrogen in the produced gas, categorizing individual groups of wells, their values theoretically range from 0 to 1. The p3 parameter, which specifies the percentage by which the performance of wells with high nitrogen content should be reduced, can also take values from 0 to 1. However, in the developed solution, both the exact domains of the parameters and their initial values were determined based on the simulation tests performed and the analysis of historical data.

As a result, the domains of parameters were limited only to the technologically justified scope, which made it possible to reduce the search space and for faster convergence of the developed IT procedure. Additionally, a condition that p2 cannot take a lower value than p1 was defined. This is connected to the fact that these parameters correspond to the values separating groups of wells with increasing nitrogen content in the produced gas. The definition of an additional condition concerning the relationship between parameters also limits the space of possible solutions.

The optimized decision tree parameters were determined after only 100 iterations of the developed procedure. Such a high efficiency of the developed solution was achieved thanks to the limited parameter domains, the additional condition regarding the relationship between the parameters, and the selection of initial values guaranteeing a potentially good quality of the solution. As a result of the optimization of gas withdrawal, the values contained in

Table 3. Optimized values of decision tree parameters.

Parameter	Optimized Value
p1	0.0212
p2	0.0347
p3	0.4902

have been assigned to the defined parameters of the decision tree.

Although the initial values of the parameters determined based on the simulation tests suggested the best solution, only p2 was slightly changed. In the optimization process, the limit value separating the group of wells with low nitrogen content in the produced gas was reduced. Hence, more wells were classified into the group with medium nitrogen content. In addition, high-nitrogen well output was significantly decreased. The difference between the initial and final values of the decision tree parameters confirms the importance of applying the developed optimization procedure.

In addition, the developed procedure rapidly increased the value of the objective function, as illustrated in Figure 5. Beyond 80 iterations, the quality of the solution improved only slightly. Moreover, the quality indicator (energy obtained from the gas produced during the analyzed withdrawal cycle) after 100 iterations is very similar to that obtained after 50 iterations. Thus, even such a small number of iterations could potentially be enough to effectively determine an approximately optimal solution. Comparing this result with the nature-inspired optimization algorithms evaluated by the authors in [7,9], such as particle swarm optimization and the genetic algorithm, the result obtained in this study confirms that the optimization tool used converges very quickly. As in the proposed methodology, the numerical simulator is used to evaluate the objective function (resource/time intensive step); convergence speed is the crucial parameter. For the optimization method used, the greatest increase in the objective function was observed in the initial iterations. Therefore, it outperforms the analyzed nature-inspired optimization algorithms, as, even with the limited number of iterations, the final results were more robust. This confirms the high computational efficiency of the proposed procedure for determining optimal well control.

5.2. Model Performance in Contrast to Expert Knowledge

The influence of the developed solution on the total nitrogen content in the gas produced from the storage during the analyzed withdrawal cycle is presented in Figure 6. As a result of the optimization of the control of the analyzed historical cycle, the total nitrogen content in the produced gas decreased by 2.4%. During the historical withdrawal cycle, the average heat of combustion was 38.72 MJ/m³, while, thanks to the reduction of nitrogen content in the produced gas, it increased to 39.06 MJ/m³. The increase in the heat of combustion of the produced gas directly influences the increase in energy that can be obtained during the gas withdrawal cycle.

Moreover, application of the developed solution made it possible to increase the energy efficiency of the analyzed UGS in relation to historical data. The influence of the developed control on the energy withdrawn from storage during the analyzed cycle is presented in Figure 7. The amount of energy withdrawn from storage during the historic withdrawal cycle increased by 2.4% for the same volume of gas produced. In the case of the analyzed UGS, this value translates into 578 mln MJ of energy. Assuming the energy price at the level of 0.006 EUR/MJ, the use of the developed method to automatically optimize UGS control allowed for an increase in the value of gas produced during a single withdrawal cycle by 3.5 mln EUR. It needs to be taken into account that the analyzed UGS is a small object used to illustrate the idea of the proposed method. In the case of large objects, the increase will be much greater. The analyzed example shows that even with a small object, the proposed solution allows an energy increase, which translates into additional income. Moreover, this additional income does not require any expenses, as the only modification introduced in relation to the historical data is the change of well control. Hence, the application of the proposed solution requires only the implementation of different gas production using the existing infrastructure, machines, and systems.

The comparison of the gas production rate before and after optimization for two exemplary wells is presented in Figure 8. The P1 well is one of the wells that was used during the reservoir production period, and the P2 well was drilled after the reservoir was converted into UGS. This is the reason why the flow rate of the P2 well is much greater than that of the P1 well. It can be seen that the optimized flow rate for the P1 well is greater than it was historically, but for P2 the proposed flow rate is smaller than what was performed by the operator. This shows that, thanks to the artificial intelligence-based method, even non-intuitive solutions can be selected if they guarantee quality indicator optimization; this could be difficult relying only on the decisions made by a qualified operator. It needs to be highlighted that the technical and technological limitations that ensure safe UGS operation were taken into account in the optimization process. Moreover, the proposed solution is developed in such a way that ensures that the assumed flow rate from the entire UGS is produced, and only the share of individual wells in total UGS output changes with time.

An important issue is that in the case of both analyzed parameters (Figure 6 and Figure 7), the difference between the optimized option and the historical data increases with the duration of the gas withdrawal cycle. Therefore, a solution based on artificial intelligence methods is more effective than the control defined by a qualified operator when the situation in the UGS is more complicated due to increased nitrogen content around the production wells. This is connected to the fact that solutions based on artificial intelligence methods constantly optimize decisions, but the operator cannot cope as well with difficult and non-intuitive situations. This confirms the sense of using innovative solutions based on artificial intelligence and control theory to optimize industrial processes.

As historic USG control and quality (energy) data are available for the analyzed withdrawal cycle, the results presented in Figure 6 and Figure 7 can be treated as the verification of the optimal results. Moreover, the optimization was performed on the reservoir simulation model calibrated to the whole history of 23 years of the reservoir production and then more than 20 years of UGS operation. Hence, it can be assumed that the results of the control strategy proposed based on the optimized decision tree are in line with the expected field application.

5.3. Limitations and Recommendations

In the proposed solution, the structure of the decision model for a given problem is developed manually because control based on a decision tree generated by an algorithm based on artificial intelligence and historical data would not be optimal, as historical control decisions are most often made solely based on the operator’s knowledge. On the one hand, the use of engineering experience and good industrial practices allows for quite accurate identification of the attributes influencing the decisions made. However, on the other hand, it may be a limitation of the proposed solution, because this step is not automatic and still depends on human knowledge. It is recommended to use engineering experience and good industrial practice to develop a decision tree structure, and then select the limit values for the decision tree attributes using the solution proposed in this article. Making decisions about their values based on experience does not guarantee the determination of the optimal well control, whereas optimization of the limit values of the parameters allows optimization of the entire control.

Another limitation of the proposed solution is the fact that reservoir simulators are used to calculate the value of the objective function when optimizing the reservoir/UGS control. However, in the case of the reservoir/UGS control optimization problem, it is impossible to link the decision variables to the optimized quality indicator in a functional manner, which makes the use of simulators necessary. Moreover, the use of a highly complex reservoir simulation model to calculate the value of the quality indicator increases the reliability of the obtained results.

The advantage of the developed optimization procedure is that it is widely applicable. In the case of the optimal well control problem, the developed optimization approach is used to optimize the parameters of the decision tree defining the well control. In this way, any production problem requiring well control optimization can be solved, from optimization of primary production, to more advanced production processes such as CCS-EOR, as in [23]. It can be adopted also for any optimization problem related to reservoir management, such as optimization of well design problem covered by [25]. It can be used to enhance the recovery factor of the reservoir or reduce water production by finding the optimal wellbore design, e.g., the shape and location of perforations. An additional advantage of the proposed method is that the optimization tool is directly linked to the reservoir simulator. It is not required in the proposed approach to link the optimization tool with the reservoir simulator, it can be any model of the analyzed problem, from simple mathematical models to complex numerical models. Thanks to that, the proposed optimization procedure can be also adopted for any parameter estimation problem of any model. In the case of adaptation of the proposed optimization procedure to other cases, such as parameter estimation in history-matching of production data, as in [26], it can be directly used to find optimal values for the parameters of the model/system.

6. Conclusions

In this work, a new concept of autoadaptive decision trees was proposed and implemented. It enables automatic optimization of hydrocarbon reservoir and UGS well control. The developed method combines a parameterized decision tree with the SMAC optimization tool linked directly to the reservoir simulator. A parameterized decision tree determines the reservoir/UGS control, a SMAC tool uses sequential configuration of algorithms based on search space to optimize the decision tree parameters, and the reservoir simulator models reservoir dynamics.

Automation of determining the control is ensured by the combination of all elements of the developed algorithm (decision tree, reservoir simulator, and SMAC optimization tool) using the Python programming language. The constructed algorithm and tools used ensure high time and computational efficiency of the procedure, enabling its application for real reservoirs/UGS. The potential limitation of the proposed method is the need for a reservoir model, which is required to optimize well control.

To assess the effectiveness of the developed solution, it was used to determine the control of UGS located in a depleted natural gas reservoir. The obtained results confirm the benefits that can be obtained by using the created algorithm to optimize the control of complex reservoir/UGS production processes. The total nitrogen content in the gas produced in the analyzed withdrawal cycle decreased by 2.4% compared to historical data based only on operator experience. As a result, the average heat of combustion of the produced gas increased from 38.72 MJ/m³ to 39.06 MJ/m³, increasing the energy efficiency of the analyzed UGS in relation to the historical data. The amount of energy withdrawn from storage during the analyzed withdrawal cycle increased by 2.4% with the same volume of gas produced. In the case of the analyzed UGS, this value translates into 578 mln MJ of energy. Hence, the developed methodology allows the energy efficiency of UGS to be increased and, as a result, more-sustainable energy management to be achieved. Moreover, these benefits do not require any expenses, as in the proposed method only well control is modified.