Groundwater Contaminant Transport Solved by Monte Carlo Methods Accelerated by Deep Learning Meta-Model

Abstract

Groundwater contaminant transport modeling is a vitally important topic. Since modeled processes include uncertainties, Monte Carlo methods are adopted to obtain some statistics. However, accurate models have a substantial computational cost. This drawback can be overcome by employing the multilevel Monte Carlo method (MLMC) or approximating the original model using a meta-model. We combined both of these approaches. A stochastic model is substituted with a deep learning meta-model that consists of a graph convolutional neural network and a feed-forward neural network. This meta-model can approximate models solved on unstructured meshes. The meta-model within the standard Monte Carlo method can bring significant computational cost savings. Nevertheless, the meta-model must be highly accurate to obtain similar errors as when using the original model. Proposed MLMC with the new lowest-accurate level of meta-models can reduce total computational costs, and the accuracy of the meta-model does not have to be so high. The size of the computational cost savings depends on the cost distribution across MLMC levels. Our approach is especially efficacious when the dominant computational cost is on the lowest-accuracy MLMC level. Depending on the number of estimated moments, we can reduce computational costs by up to ca. $25\%$ while maintaining the accuracy of estimates.

1. Introduction

Groundwater quality might be affected by many natural and industrial processes. There are many sources of groundwater contamination, for instance, polluted precipitation, runoff from roadways, leaking barrels of waste chemicals, etc. [1] (p. 500). High concentrations of dissolved substances can result in water being unfit for drinking or irrigation. Our research focuses on observing groundwater processes in the vicinity of a deep geological repository (DGR) of radioactive waste. Despite all the protective layers, radioactive material will be exposed to the geosphere in the distant future. As a result, groundwater may become contaminated. Since it is not feasible to measure these processes directly, we model and simulate them numerically. Taking the uncertainties into account, we consider bedrock properties as random variables, specifically, spatially correlated random fields (SRF). Consequently, a quantity of interest (QoI) is also a random variable. We investigate statistical moments of a QoI in the first place and a probability density function (PDF) afterward.

Models can be process-based or data-driven [2] (p. 4). Our research takes advantage of both approaches. First, we run a stochastic process-based contaminant transport model. A physical process is numerically solved by the finite element method on an unstructured mesh with prescribed SRF. To estimate the expectation of QoI, the model run (=simulation) is performed multiple times in accordance with the Monte Carlo method (MC); for some applications, see [3,4,5]. However, MC suffers from high computational costs. Many highly accurate simulations must be performed to obtain statistical estimates with reasonably low variance, see [6] (p. 2). To save the cost, it is beneficial to construct a hierarchical system of less accurate but cheaper models that approximate the same QoI; for details, see [7] (p. 552). In the field of numerical modeling, the accuracy of a model is given by the number of mesh elements. Thus, we consider the hierarchy of models with gradually coarsened computational meshes, see [7] (p. 555). For this purpose, we adopt the multilevel Monte Carlo method (MLMC) [6] described in more detail in Section 4. For some MLMC applications, see [8,9,10,11].

A meta-modeling approach is another option for decreasing the total computational cost of MC. A function of inputs and parameters of a complex and expensive model is approximated by a simpler and faster meta-model, also called a surrogate model. Meta-modeling techniques are widely used across scientific fields (see [12]), and groundwater modeling is no exception [13,14,15,16]. In terms of the classification of meta-models by Robinson [17] and Asher et al. [13], we concentrate our effort on data-driven meta-models. In our case, the input of the model is a 2D SRF, and the output is some scalar QoI. Asher et al. [13] (p. 5958) also provides a list of popular data-driven meta-modeling techniques, such as radial basis functions, Gaussian processes, support vector regression (SVR), and neural networks (NN). For their brief description, see [18,19]. We have focused on meta-models based on neural networks, the applicability of which to hydrological processes was discussed in [20,21]. The latter article also provides the basics of deep learning and briefly introduces some popular DNN architectures. Neural networks and deep learning have already been utilized in groundwater modeling to predict its: level [22], quality [23,24], flow [25,26], or contaminant transport [27]. With regard to the nature of input data of our model, it would be felicitous to use a convolutional neural network (CNN) as a meta-model. CNNs excel in learning spatial relationships within data on a regular grid such as an image. Since we use unstructured meshes, we cannot adopt a CNN directly (see [28]). We briefly discussed some ways to overcome this difficulty in [29]. Representing the structure of SRF as a graph (for graph theory, see [30]) and then using a graph convolutional neural network (GCNN) [31] is the option we selected.

The general objective of the presented article is to reduce the computational cost of Monte Carlo methods by incorporating a meta-model. In this regard, we build on the findings of our initial article [29], which addressed the same objective with a GCNN meta-model of a groundwater flow problem. In this article, we specifically focus on improving the accuracy of the meta-model and its applicability for groundwater contaminant transport problems. To the best of our knowledge, no previous research in groundwater modeling has investigated the computational cost reduction in Monte Carlo methods using a meta-model.

The article is organized as follows. First, the contaminant transport model is described. Next, its deep learning meta-model is presented in Section 3. The multilevel Monte Carlo method is briefly introduced, and the proposed incorporation of the meta-model is delineated in Section 4 and Section 5. Section 6 is devoted to numerical experiments, meta-models are assessed, and their usage is put into the context of MC and MLMC. The article is concluded with a summary and discussion of the important findings.

2. Groundwater Contaminant Transport Model

We consider a conceptual 2D model of the transport of contaminants from DGR, see Figure 1. The Darcy flux of groundwater q [ms${}^{-1}$] is described by the equations:

$$div\left(\mathit{q}\right)=0,\mathit{q}=\kappa \left(\mathit{x}\right)\nabla p,$$

(1)

where p is the pressure head [m], and $\kappa$ is the hydraulic conductivity considered to be smaller at a stripe near the surface. The Darcy flow is driven by the seepage face condition:

$$(\mathit{q}·\mathit{n})p=0,\mathit{q}·\mathit{n}\ge q_0,p\ge 0$$

(2)

on the top boundary, with $q_0$ denoting a precipitation flux. No flow is prescribed on the remaining boundaries. The surface geometry is intentionally overscaled to pronounce its impact on the groundwater flow.

The advection equation is used as the contaminant transport model:

$${\partial }_t\left(\vartheta c\right)+div\left(\mathit{q}c\right)=0,$$

(3)

where $\vartheta$ is the porosity and c is the concentration [kg m${}^{-3}$] of a contaminant in terms of mass in the unit volume of water. The equation is solved for the unknown concentration with the initial condition $c^0=1$ in the repository and 0 elsewhere. Zero concentration is prescribed at the inflow part of the boundary.

The $\vartheta$ is given as a correlated random field. Then, the random field of hydraulic conductivity $\kappa$ is determined from $\vartheta$ using the Kozeny–Carman equation and slight random scaling given by an independent random field. Figure 1 displays the geometry of the model, the random conductivity field (conductivity in the figure), and the concentration field (X_conc in the figure) at the time $t=3$ of the total simulation time $T=20$. The simulation is performed using the Flow123d software [32].

3. Deep Learning Meta-Model

3.1. Graph Convolutional Neural Network Meta-Model

Graph neural networks (GNNs) are deep-learning-based models that operate on graph data [31]. They are adopted in many areas, including natural language processing, computer vision, bioinformatics, and physics. Similar to standard neural networks [33], they are categorized into several groups, such as graph convolutional neural networks (GCNN), graph attention networks, graph recurrent networks, etc. We employ GCNNs, which have a variety of different architectures. They can differ in many ways, such as the representation of a convolution operation, the size of a convolutional filter, the number of graph vertices taken into account, and many others. For a comprehensive survey on GCNNs, see [31,34].

In order to use GCNNs, the dual graph $G=(V,E)$ of the computational mesh is considered with the vertices V corresponding to mesh elements and associated with the values of the input random fields. The edges correspond to the adjacent mesh elements. Figure 2 illustrates the graph representation of the SRF prescribed on an unstructured mesh.

3.2. ChebNet GCNN

After testing several architectures of GCNN, we ascertain that GCNN with the Chebyshev convolutional layer (ChebNet GCNN) [35] provides the best results in our case. It pertains to spectral GCNNs, which represent the graph convolution in terms of polynomials of a graph Laplacian matrix L.

For a brief general description, let ${\ast }_G$ be the graph convolutional operator, $\mathit{x}\in {\mathbb{R}}^{\left|V\right|}$ be a vector of input values associated with graph vertices, and ${\mathit{g}}_{\theta }\in {\mathbb{R}}^{K\times Ch}$ be the convolutional kernel. The variable K stands for the maximal polynomial degree, and $Ch$ represents the number of channels of the kernel [33] (ch. 9). Then, the graph convolution of $\mathit{x}$ and ${\mathit{g}}_{\theta }$ is:

$$\mathit{x}{\ast }_G{\mathit{g}}_{\theta }=p_{\theta }^K\left(L\right)\mathit{x},$$

(4)

where p is a polynomial of eigenvalues of L and has maximum degree K. When using ChebNet GCNN, $p^K\left(L\right)$ is defined via Chebyshev polynomials $T_k$:

$$\mathit{x}{\ast }_G{\mathit{g}}_{\theta }=(\sum _{k=0}^{K-1}{\mathbf{\theta }}_k{T}_k\left(L\right))\mathit{x},$$

(5)

where ${\mathbf{\theta }}_k\in R^{Ch}$ represents the learnable parameters of the kernel that are adjusted during the neural network learning procedure. The kernel is K-localized, i.e., the filtered signal at a particular vertex depends on information from vertices in its K-hop neighborhood. For a comprehensive explanation of GCNN, see [31] (p. 81).

3.3. Architecture of Meta-Model

Our deep learning meta-model consists of ChebNet GCNN followed by a deep feed-forward neural network (FNN) [33] (ch. 6). As depicted in Figure 3, the input graph enters the ChebNet GCNN with $K=4$ and $Ch=8$. The following global summation pool sums up information of all vertices for each channel. This flattened output of ChebNet GCNN forms the input to the deep FNN with two hidden layers of 64 and 32 neurons. To provide complete information about the neural networks employed, it is necessary to specify hyperparameters (generally about hyperparameters, see [33] (p. 120)). Our settings: Adam optimizer with learning rate $0.001$; hidden layer activation function: ReLU; output activation function: identity; loss function: MSE; maximum number of epochs: 2000; and patience: 1000.

3.4. Assessment of Meta-Model

The following procedure was employed to assess different meta-models. Let $\mathcal{D}={\left\{({\mathit{x}}_i,c_i)\right\}}_{i=1}^D$ denote all available i.i.d. samples, ${\mathit{x}}_i\in {\mathbb{R}}^{\left|V\right|}$ be an input vector of the hydraulic conductivity values of an SRF, and $c_i\in \mathbb{R}$ be the corresponding simulated concentration on the surface (see Section 2). Since we conduct supervised learning [33], $\mathcal{D}$ is divided into learning (=training) samples $\mathcal{L}$ and test samples $\mathcal{T}$. Then, $\mathcal{L}$ is evenly divided into ${\mathcal{L}}_s$, $s=1,\cdots ,S$ to repeat the whole meta-model learning procedure S times. For each repetition, data are pre-processed to facilitate learning: $c_{{\mathcal{L}}_s}=(c_{{\mathcal{L}}_s}-{\langle c\rangle }_{{\mathcal{L}}_s})/{\mathrm{std}}^2\left(c_{{\mathcal{L}}_s}\right)$, where $\mathrm{std}$ denotes the sample standard deviation.

Given ${\mathcal{L}}_s$, a meta-model learns a function $f_s\left(\mathbf{x}\right):{\mathbb{R}}^{\left|\mathit{x}\right|}\to \mathbb{R}$. The quality of the approximation is generally evaluated by a loss function, in our case, the mean squared error (MSE): $\lambda (\mathcal{M},f_s)=\frac{1}{\left|\mathcal{M}\right|}{\parallel c_{\mathcal{M}}-f_s\left({\mathit{x}}_{\mathcal{M}}\right)\parallel }_2^2$, where $\mathcal{M}\subseteq \mathcal{D}$. Namely, the training loss $\lambda ({\mathcal{L}}_s,f_s)$ and test loss $\lambda (\mathcal{T},f_s)$ are observed. For the sake of comparing the accuracy of different meta-models, we use the normalized root mean squared error ($NRMSE=\sqrt{MSE}/\mathrm{std}$). The training NRMSE $J_{\mathcal{L}}=\frac{1}{S}{\sum }_{s=1}^S\frac{\sqrt{\lambda ({\mathcal{L}}_s,f_s)}}{\mathrm{std}\left(c_{{\mathcal{L}}_s}\right)}$ and test NRMSE $J_{\mathcal{T}}=\frac{1}{S}{\sum }_{s=1}^S\frac{\sqrt{\lambda (\mathcal{T},f_s)}}{\mathrm{std}(c_{\mathcal{T}})}$ are calculated. This metric allows us to draw a comparison among models on different mesh sizes. The lower the NRMSE, the better. Moreover, if the NRMSE is above 1, we can use a simple random generator instead of a complex meta-model.

4. Multilevel Monte Carlo Method

The multilevel Monte Carlo method [6] (MLMC) is based on the variance reduction principle [6] (p. 3). Many simulation samples are collected at low levels, with less accurate but inexpensive approximations of the model available. While much fewer samples are collected at high levels, there are differences between approximations of the model that are more accurate but computationally expensive. If these differences have a significantly smaller variance, we obtain estimates with the same accuracy but at a fraction of the cost compared to the standard Monte Carlo method.

Let P be a random variable and $P_1$, …, $P_L$ be a sequence of its approximations, where $P_{l-1}\approx P_l$. The approximations are becoming more accurate from $P_1$ to $P_L$. Then, the expected value of $P_L$ satisfies the identity:

$$\mathbb{E}\left[P_L\right]=\mathbb{E}\left[P_1\right]+\sum _{l=2}^L\mathbb{E}[P_l-P_{l-1}].$$

(6)

The MLMC estimates an individual expectation as follows:

$$\widehat{P}={\langle P_1\left({\mathit{x}}_n^1\right)\rangle }_{N_1}+\sum _{l=2}^L{\langle P_l\left({\mathit{x}}_n^l\right)-P_{l-1}\left({\mathit{x}}_n^l\right)\rangle }_{N_l},$$

(7)

where L is the total number of levels, $N_l$ stands for the number of simulation samples on level l, ${\langle \rangle }_{N_l}$ denotes the average over $N_l$ samples. Pairs of random inputs $({\mathit{x}}_i^l,{\mathit{x}}_i^{l-1})$ are both discrete evaluations of a single realization of a random field $x\left({\omega }_{l,i}\right)$, while the random states ${\omega }_{l,i}$ are independent and come from a given probability space $(\mathsf{\Omega },\mathsf{\Sigma },\mathcal{P})$.

When using estimator (7), we are particularly interested in its total computational cost C and estimated variance $\widehat{V}$:

$$C=\sum _{l=1}^L{N}_l{C}_l,$$

(8)

$$\widehat{V}=\sum _{l=1}^L\frac{{\widehat{V}}_l}{N_l},$$

(9)

where $C_1,{\widehat{V}}_1$ denote the cost and the estimated variance of $P_1$. Meanwhile, for $l>1$, $C_l,\widehat{V_l}$ represent the cost and the estimated variance of differences $P_l-P_{l-1}$. For a comprehensive MLMC theory, see [6].

In the presented way, it is feasible to estimate the expectations of quantities derived from the original QoI. In particular, we utilize the so-called generalized statistical moments of QoI. In our study, the moments’ functions $\varphi \left(x\right)$ are Legendre polynomials, as they are suitable for the PDF approximation (see [36]).

4.1. Optimal Number of Samples

As previously mentioned, the principal motivation of our research is to reduce C. However, it is also essential to keep the variance of the estimator sufficiently low. To achieve this, we prescribe the target variance $V_t$ of the estimator. Then, the optimal $N_l$ is found by minimizing function (8) under the constraint

$$V_t=\sum _{l=1}^L\frac{{\widehat{V}}_l}{N_l}.$$

(10)

After some calculus:

$$N_l^r=\sqrt{\frac{{\widehat{V}}_l^r}{C_l}}\frac{{\sum }_{i=1}^L\sqrt{{\widehat{V}}_i^r{C}_i}}{V_t},r=1,\cdots ,R,$$

(11)

where ${\widehat{V}}_l^r$ is an estimated variance of ${\varphi }_l^r\left(x\right)-{\varphi }_{l-1}^r\left(x\right)$ for $\mathrm{r}-\mathrm{th}$ moment function on level l, ${\varphi }_0^r=0$. Finally, ${\displaystyle N_l=\underset{r=1,\cdots ,R}{max}{N}_l^r}$. This procedure is crucial in our study and determines the results presented.

The Python library MLMC [37], developed by the authors of this article, is used to schedule simulation samples and post-process the results. The software also implements the maximal entropy method (MEM) to determine the PDF from moments estimated by MLMC. The description of MEM is beyond the scope of this article. The interested reader is advised to read [36,38] for a comprehensive study or [29] for a brief introduction.

5. Monte Carlo Methods with a Meta-Model

In this section, we propose combining Monte Carlo methods with a meta-model. We investigate two scenarios. The first one, MC-M, uses a meta-model within the standard MC. The latter, denoted as MLMC-M, extends the MLMC by a level of meta-model approximations.

5.1. MC-M

A meta-model fully replaces an original model within the standard Monte Carlo method. Let ${\tilde{P}}_L$ denote a meta-model approximation of $P_L$, which is the most accurate approximation of P used in MC. Thus, the MC-M estimator has this form:

$${\widehat{P}}_{MC-M}={\langle {\tilde{P}}_L({\mathit{x}}_n^L)\rangle }_N.$$

(12)

This approach is strictly dependent on the approximation quality of the meta-model. The calculation of variance (Formula (9)) is unchanged, whereas the computational cost (Formula (8)) takes this form:

$$C_M=C_1^{sim}\left|{\mathcal{L}}_s\right|+C_1{N}_1+C_{ml}(\mathcal{L},\psi ),$$

(13)

where $C_1^{sim}\left|{\mathcal{L}}_s\right|$ represents the cost of $P_L$ runs needed for learning procedure of the meta-model. The cost of a meta-model sample $C_1$ includes the necessary preprocessing and the cost of prediction. In this case, it comprises the costs needed for generating a random field and its adjustment to the form of the input of the GCNN. The learning cost of the meta-model $C_{ml}({\mathcal{L}}_s,\psi )$ depends not only on learning samples in ${\mathcal{L}}_s$ but also on hyperparameters $\psi$, especially the number of epochs, batch size, and the number of learnable parameters.

5.2. MLMC-M

The meta-model approximations form the new lowest-accuracy level of MLMC, which we denote as the meta-level. Let ${\tilde{P}}_1$ be a meta-model approximation of $P_1$, and the difference between $P_1$ and ${\tilde{P}}_1$ forms the subsequent first level. Thus, the MLMC estimator is extended by a single level:

$${\widehat{P}}_{MLMC-M}={\langle {\tilde{P}}_1\left({\mathit{x}}_n^1\right)\rangle }_{N_1}+{\langle P_1\left({\mathit{x}}_n^2\right)-{\tilde{P}}_1\left({\mathit{x}}_n^2\right)\rangle }_{N_2}+\sum _{l=3}^L{\langle P_l\left({\mathit{x}}_n^l\right)-P_{l-1}\left({\mathit{x}}_n^l\right)\rangle }_{N_l}.$$

(14)

Again, the total computational cost (Formula (8)) is affected:

$$C_M=C_1{N}_1+(C_2^{sim}+C_2^{meta})N_2+C_{ml}(\mathcal{L},\psi )+\sum _{l=3}^L{N}_l{C}_l,$$

(15)

where $C_2^{sim}$ is the cost of $P_1$, and $C_2^{meta}$ represents the cost of preprocessing an SRF, which was already generated for the corresponding simulation. It also includes the cost of the meta-model prediction, which, however, is negligible.

The distribution of the computational cost across levels affects the possible savings in C. Considering the MLMC estimator theorem (M. Giles [6] (Theorem 1)), there are bounds of $V_l\le c_1{2}^{-\beta l}$ and $C_l\le c_2{2}^{\gamma l}$, where $\beta ,\gamma ,c_1,c_2$ are positive constants. Our approach is especially effective in the case of $\beta >\gamma$, when the dominant computational cost is at low accuracy levels. At the same time, adding another coarse level brings no savings, which is a case of unsatisfactorily scaling computational cost for models on low levels, often due to the overhead of the numerical solver. If $\beta =\gamma$, then the computational cost is spread evenly across levels, and the amount of savings depends on the total number of levels L. If the dominant computational cost is on high-accuracy levels ($\beta <\gamma$), then we cannot expect significant savings at all.

6. Results

6.1. Analysis of Meta-Models

In order to evaluate the meta-model in accordance with our assessment procedure described in Section 3.4, the contaminant transport model was run on meshes of different sizes, and obtained data are used as $\mathcal{D}$. From the Monte Carlo method’s point of view, we generally set $V_t\le {10}^{-5}$ to obtain sufficiently accurate estimates of moments to decently approximate PDF by MEM. Consequently, we have at least 2000 samples at the lowest Monte Carlo level. Thus, from now on, we undertake all meta-model learnings with $|{\mathcal{L}}_s|=2000$, and 400 samples out of ${\mathcal{L}}_s$ account for validation samples, $\left|\mathcal{T}\right|=$ 50,000, $S=12$.

Table 1. Comparison of accuracy of meta-models for models on different mesh sizes.

Mesh Size	Accuracy of Meta-Model (${\mathit{J}}_{\mathcal{L}}$)
	Deep Meta-Model	Shallow Meta-Model
53	$0.779$	$0.945$
115	$0.799$	$0.949$
474	$0.763$	$0.885$
2714	$0.773$	$0.896$
10,481	$0.799$	$0.898$
18,397	$0.792$	$0.954$

shows the accuracy of meta-model approximations of models on different mesh sizes. Two architectures of meta-models are compared. The Deep meta-model is the one proposed in this article, whereas the Shallow meta-model was propounded in our previous study [29]. The accuracy is characterized by NRMSE (see Section 3.4). Apparently, the Deep meta-model outperforms the Shallow meta-model in all observed cases. Since $J_{\mathcal{L}}$ stays almost steady and $J_{\mathcal{L}}/J_{\mathcal{T}}\approx 1$, our approach can be used regardless of the mesh size, at least up to the ca. 20,000 elements, larger meshes were not tested. Considering $J_{\mathcal{L}}$ is just slightly below $0.8$, we believe there is still some room for improvement in future research.

All meta-models were trained on GeForce GTX 1660 Ti Mobile graphics card with 16 Gb RAM. The time of meta-model learning ranges from around 400 s for a model on 53 mesh elements to about 3000 s for a model on 18,397 mesh elements.

6.2. Comparison of MC-M and MLMC-M

Once we have a promising meta-model, it can be utilized by Monte Carlo methods. Based on the approach proposed in Section 5, we provide MC-M and two-level MLMC-M in comparison with the standard MC. Both are compared in terms of computational cost and the quality of estimations of moments. The number of samples $N_l$ were calculated by formulas presented in Section 4.1 with $V_t={10}^{-5}$, $R=25$. The efficiency of Monte Carlo methods with a meta-model compared to Monte Carlo methods without a meta-model is expressed as computational cost savings $\mathrm{CS}[\%]$ calculated as $\mathrm{CS}=100(1-\frac{C_M}{C})$.

Figure 4 shows the computational cost ($Cost\left(sec\right)$) and moments error ($J(\widehat{\mathbf{\mu }},\tilde{\mathbf{\mu }})$). The cost is measured as the CPU time of simulations, including auxiliary processing and time needed for meta-model learning. The moments error has the form of MSE: $J(\widehat{\mathbf{\mu }},\tilde{\mathbf{\mu }})=\frac{1}{S}{\sum }_{s=1}^S\left|\right|{\widehat{\mathbf{\mu }}}^s-{\tilde{\mathbf{\mu }}}^s{\left|\right|}_2^2,$ where $\widehat{\mathbf{\mu }}$ represents the moments estimated by the standard MC, whereas $\tilde{\mathbf{\mu }}$ are moments calculated by either MC-M or MLMC-M. As expected, the computational cost increases with the precision of a model expressed as a number of mesh elements. As you can see, the cost can be greatly reduced ($\mathrm{CS}\approx 87\%$) by using MC-M instead of MC. However, the meta-model error causes poor estimates of moments: $J(\widehat{\mathbf{\mu }},\tilde{\mathbf{\mu }})\approx {10}^{-3}$. On the contrary, MLMC-M suppresses the effect of the meta-model error, and estimates of moments are obtained with $J(\widehat{\mathbf{\mu }},\tilde{\mathbf{\mu }})\approx {10}^{-5}$. The cost savings are not so substantial but still significant: $\mathrm{CS}\approx 25\%$. Generally, using MC-M can be reasonable if we have an exceptionally accurate meta-model or we intend to obtain just a basic idea of the moments and PDF in a short time. Otherwise, using the MLMC-M is recommended.

6.3. Multilevel Case

In this section, we investigate an MLMC-M with more than two levels. In order to demonstrate some properties and limitations of our approach, we introduce the following three pairs of Monte Carlo methods:

• 1LMC and 1LMC-M: standard MC of models on 18,397 mesh elements and its extension by meta-level;
• 2LMC and 2LMC-M: 2-level MLMC on models with 18,397 and 2714 mesh elements, and its extension by meta-level trained on the model of 2714 mesh elements;
• 3LMC and 3LMC-M: 3 level MLMC with models on 18,397, 2714, and 474 mesh elements and its extension by meta-level trained on the model of 474 mesh elements

Figure 5 shows the computational costs of these Monte Carlo methods for a different number of estimated moments R. We see that 1LMC and 1LMC-M have a much higher cost than methods with more levels. This is the crucial observation that demonstrates the limitations of the standard MC.

Let us focus on the course of the costs depending on R. In all the cases, the lowest cost was obtained for $R=2$. However, as R increases, the behaviors of the methods differ. For 1LMC, 1LMC-M, 2LMC, and 2LMC-M, we observe a similar course that is steady for $R⪆15$. It means we can add more moments without affecting computational cost. On the other hand, for 3LMC and 3LMC-M, we observe a gradual increase in cost up to $R=35$ and $R=75$, respectively.

Regarding computational cost savings, in cases of 1LMC and 2LMC, there is $\mathrm{CS}\approx 25\%$ utilizing 1LMC-M and 2LMC-M, whereas 3LMC provides us with savings, from around $\mathrm{CS}\approx 25\%$ for $R=2$ to just $\mathrm{CS}\approx 2\%$ for $R=100$. As mentioned in Section 5.2, the computational cost distribution across levels can be described using $\beta$ and $\gamma$ variables. Since for 1LMC and 2LMC, $\beta >\gamma$, the dominant computational cost is on the lowest-accuracy levels. Therefore, adding the meta-level results in a much higher $\mathrm{CS}$ (for $R>4$) than for $\beta <\gamma$, which is the case with 3LMC. Thus, the behavior of our experiments corresponds to the theoretical properties of the MLMC.

A closer look at the variances of moments across levels ${\widehat{V}}_l^r$ (Figure 6) provides a rationale for the claims already presented. For clarity, we display only the first five moments ($R=5$), which capture the behavior observed also in cases with more moments. To recall, we employ Legendre polynomials as $\varphi \left(x\right)$; therefore, ${\widehat{V}}_l^1=0$. The total computational cost (see Formula (8)) depends on the number of simulation samples $N_l$ that are determined by ${\displaystyle {\widehat{V}}_l^{MAX}=max{\left\{{\widehat{V}}_l^r\right\}}_{r=1}^R}$ (see Section 4.1) and $C_l$, which is also displayed in the figure. To meet $V_t={10}^{-5}$, the pictured variances are calculated based on the following numbers of samples: $[N_1=54,063,N_2=2550,N_3=1426]$ for 3LMC and [$N_1=53,342,N_2=36,638,N_3=3038,$$N_4=1218$] for 3LMC-M.

We can see that ${\widehat{V}}_l^{MAX}$ increases with increasing r for some l, leading to the observed growth of C and $C_M$ with R to the point where ${\widehat{V}}_l^{MAX}$ is stable. When it comes to different $CS$, we need to focus on the course of ${\widehat{V}}_l^r$ across levels. We observe increasing ${\widehat{V}}_2^{MAX}$ of 3LMC-M, whereas the corresponding ${\widehat{V}}_1^{MAX}$ of 3LMC remains constant. This trend is accentuated for $R>5$ and causes a decrease in $CS$. The distinguishable increase in ${\widehat{V}}_1^{MAX}$ for 3LMC-M has just a slight impact on $C_M$ due to the minor $C_1=0.338$ compared to $C_l$, for $l>1$. A more accurate meta-model could prevent ${\widehat{V}}_2^{MAX}$ from increasing for 3LMC-M.

We faced a similar behavior for the groundwater flow problem investigated in our previous research [29]. It is good to be aware of this behavior, although the distribution of variances of moments across MLMC levels would deserve a much more detailed explanation, which is beyond the scope of this article.

6.4. Approximation of Probability Density Function

Finally, we use moments estimated by the cheapest Monte Carlo methods presented (3LMC and 3LMC-M) to approximate the PDF of the contaminant concentration c[kgm${}^{-3}$] on the surface, $V_t={10}^{-5}$, $R=25$. Let ${\rho }_{3LMC}$, ${\rho }_{3LMC-M}$, ${\rho }_{MC}$ be the PDFs approximated based on the moments estimated by 3LMC, 3LMC-M, and reference MC. We run $N=50,000$ model samples on a mesh with 18,397 elements to obtain the reference MC.

Figure 7 depicts the approximated PDFs, as well as the Kullback–Leibler (KL) divergence (for details, see [39]) used for the accuracy assessment of PDFs: $D_{3LMC}=KL({\rho }_{ref}\left|\right|{\rho }_{3LMC})$ and $D_{3LMC-M}=KL({\rho }_{ref}\left|\right|{\rho }_{3LMC-M})$. Considering the values of $V_t$ and R, we have decent PDF approximations. Both 3LMC and 3LMC-M provide almost the same results in terms of KL divergence. It means $\tilde{\mathbf{\mu }}$ are sufficiently accurate, in particular $J(\widehat{\mathbf{\mu }},\tilde{\mathbf{\mu }})\approx 8.6e-6$. Moreover, $\tilde{\mathbf{\mu }}$ are obtained with $CS\approx 8\%$. If we are interested just in the expected value and the variance, which is very common, we have $CS\approx 25\%$. Importantly, we can decrease $V_t$ to obtain a better PDF approximation; in this case, the computational cost naturally increases, but $CS$ does not change.

7. Discussion and Conclusions

This study presented the deep learning meta-model to reduce the computational costs of Monte Carlo methods. We followed up on our previous research [29] and improved the meta-model, which now consists of a graph convolutional neural network connected to the feed-forward neural network. This meta-model can better approximate models such as the tested groundwater contaminant transport problem. We showed that our meta-model could be trained with comparable accuracy for models solved on unstructured meshes from 53 to 18,397 mesh elements.

We adopted Monte Carlo methods to obtain generalized statistical moments that are utilized to approximate the probability density function of the contaminant concentration on the surface above a future deep geological repository of radioactive waste. In order to reduce the computational cost of MC, two approaches were propounded. MC-M, a meta-model instead of a model within the standard Monte Carlo method, brings substantial cost savings ($\mathrm{CS}\approx 87\%$). However, the accuracy of moments J was severely affected by the meta-model error and was just around $J\approx {10}^{-3}$. Under the described experiment setting, it is tough to obtain minor meta-model errors for such a complex model such as the one we have. Thus, this procedure has limited usability. What is favorable, though, is the second approach. A meta-level is employed as the lowest-accuracy level of MLMC, denoted MLMC-M. Generally, this approach reduces the computational cost and keeps the error of moments low, $J\approx {10}^{-5}$, which is sufficient for PDF approximations. We presented three pairs of MLMCs and MLMC-Ms to demonstrate the impact of a different distribution of computational cost across levels. In accordance with theory, the most significant cost savings ($\mathrm{CS}\approx 25\%$) was achieved when the dominant computational cost was on the lowest-accuracy level. On the contrary, if the prevailing computational cost is on the highest-accuracy level, we have $\mathrm{CS}\approx 2\%$ in the worst case. Importantly, the computational cost and savings are affected by the number of moments R we estimate. In many applications, we are interested in the first few moments, such as the expected value or the variance. For that, we have $\mathrm{CS}\approx 25\%$ in all the cases. Intending to approximate PDF, we need at least around $R=25$, then $\mathrm{CS}\approx 8\%$ in the worst-case scenario.

In our future research, we shall try to improve the accuracy of the current meta-model further and investigate the applicability of standard convolutional neural networks as a meta-model. We shall also rigorously describe the causes of the different course of variances of moments across MLMC levels. We shall deal with other applications in groundwater modeling, especially in connection with fractured porous media.