Dual Domain Decomposition Method for High-Resolution 3D Simulation of Groundwater Flow and Transport

Abstract

The high-resolution 3D groundwater flow and transport simulation problem requires massive discrete linear systems to be solved, leading to significant computational time and memory requirements. The domain decomposition method is a promising technique that facilitates the parallelization of problems with minimal communication overhead by dividing the computation domain into multiple subdomains. However, directly utilizing a domain decomposition scheme to solve massive linear systems becomes impractical due to the bottleneck in algebraic operations required to coordinate the results of subdomains. In this paper, we propose a two-level domain decomposition method, named dual-domain decomposition, to efficiently solve the massive discrete linear systems in high-resolution 3D groundwater simulations. The first level of domain decomposition partitions the linear system problem into independent linear sub-problems across multiple subdomains, enabling parallel solutions with significantly reduced complexity. The second level introduces a domain decomposition preconditioner to solve the linear system, known as the Schur system, used to coordinate results from subdomains across their boundaries. This additional level of decomposition parallelizes the preconditioning of the Schur system, addressing the bottleneck of the Schur system solution while improving its convergence rates. The dual-domain decomposition method facilitates the partition and distribution of the computation to be solved into independent finely grained computational subdomains, substantially reducing both computational and memory complexity. We demonstrate the scalability of our proposed method through its application to a high-resolution 3D simulation of chromium contaminant transport in groundwater. Our results indicate that our method outperforms both the vanilla domain decomposition method and the algebraic multigrid preconditioned method in terms of runtime, achieving up to 8.617× and 5.515× speedups, respectively, in solving massive problems with approximately 108 million degrees of freedom. Therefore, we recommend its effectiveness and reliability for high-resolution 3D simulations of groundwater flow and transport.

1. Introduction

Groundwater, as a vital natural resource, plays a crucial role in sustaining ecosystems and meeting human needs for drinking water, agriculture, and industry. For many years, the numeric modeling of groundwater flow and solute transport has been essential for gaining a deep understanding of complex groundwater systems and making informed decisions in groundwater resource protection [1,2,3,4,5,6,7,8]. Due to the three-dimensional (3D) spatial extent, high level of spatial variability, and high level of heterogeneity of groundwater systems, a groundwater simulation necessitates a large-scale, high-resolution numeric model over 3D space. This numeric model’s discretized solution leads to the solution of massive discretized linear systems that require a lot of computational time and space.

Scalable algorithms are necessary for efficiently solving massive groundwater flow and transport problems [9,10,11,12,13]. Generally, the Krylov space methods, such as the conjugated gradient (CG) and BiCGSTAB [14] methods, just to name a few, are used to solve the large linear system for groundwater flow and transport simulations [15,16]. However, as the linear system increases in scale, the convergence of these methods tends to decelerate [17]. To mitigate this issue with Krylov subspace methods, multigrid methods have been introduced to enhance solution efficiency, either as standalone solvers or as preconditioners in Krylov subspace methods. Multigrid methods [18] offer the advantage of convergence rates that are less dependent on the problem size, thereby enabling the number of iterations to be nearly constant. Hence, by using multigrid methods, the scalability of groundwater flow and transport simulations is greatly improved. Nevertheless, these multigrid methods tend to be algorithmically complex and are often tailored to specific problems, making them less robust compared with Krylov space methods.

Another effective approach to achieving algorithmic scalability in groundwater simulations is through parallelism. If the algorithm can be parallelized, the increased computational burden can be distributed across multiple processing units, resulting in runtimes that are less sensitive to the number of unknowns [19,20,21,22,23,24,25]. Parallelization computation has been developed in several hydrological software programs, such as tough2-MP V2.0 [26], PFLOTRAN V3.0.0 [27], RT3D V2.5 [28], PGREM3D V1.0 [29], and SWMS V1.00 [30], among others. To achieve parallelization for groundwater simulations, the multigrid method offers a solution for the parallelization of the linear system [31,32,33,34]. With the increasing computing power of graphics hardware, GPUs have demonstrated efficiency in the parallelization of groundwater simulations [24,35,36,37,38,39].

One of the most important techniques for parallelizing the computation associated with large linear system problems for groundwater simulations is the domain decomposition method, which was originally proposed by Herman Schwarz [40] and was used to solve elliptic partial differential equations [41]. Given a numeric problem associated with a large-scale spatial domain, the fundamental principle of the domain decomposition method is to employ a “divide and conquer” strategy that partitions the domain into multiple subdomains which are associated with a subsystem of the original numeric problem. Every subsystem is allowed to be solved independently and thus can be processed in parallel. The subsystems associated with subdomains are connected by their boundary nodes and constraints that combine the sublinear systems, which ensures that the solution of the subsystems is equivalent to that of the original problem. Because of the promising nature of the domain decomposition method, it has been exploited for the parallelization of groundwater simulations [42,43,44,45,46,47,48,49,50].

Developing an efficient domain decomposition approach for large-scale groundwater simulations is challenging. Directly applying a domain decomposition strategy for solving a large linear system proves impractical. Partitioning a large domain into a regular number of subdomains may result in a domain size that is prohibitively large for an efficient solution. However, an increased number of subdomains leads to exponentially increased computational overheads for algebraic operations that combine the results obtained from subdomains. Therefore, it is crucial to efficiently combine the solutions of subsystems to ensure the scalability of the domain decomposition algorithm.

In this paper, we present a novel two-level domain decomposition approach aimed at efficiently solving large-scale discrete linear systems for 3D groundwater flow and transport simulations. Our method leverages the principle of divide and conquer that is inherent in domain decomposition. We partition the large domain into appropriately sized subdomains to enhance the efficiency of solving linear systems within each subdomain. To integrate the solutions from each subdomain, we introduce an additional level of domain decomposition over the boundaries of the subdomains. This approach allows us to distribute the high computational burden of combining independent subdomains across multiple threads, thereby addressing the performance bottleneck inherent in the standard domain decomposition scheme. These speedup schemes constitute a novel dual-domain decomposition solver for large groundwater flow and transport problems, which permits the following:

• Finely grained subdomain sizes, enabling a low-cost and parallel solution for linear subsystems decoupled from the large domain;
• An efficient combination of subsystem solutions, enabling the seamless integration of subsystem solvers into a global solver for scalable groundwater simulations.

We demonstrate the efficacy of our proposed method through its application in a high-resolution 3D numerical model of chrome contaminant transport in groundwater. Our results highlight a significant performance boost for large-scale groundwater numerical systems with hundreds of millions of unknowns.

2. Methods

2.1. Governing Equations and Discrete Solution Methods

Groundwater flow through a porous medium, involving the movement of contaminant solutes, can be represented as the transient process of hydraulic heads. In this context, the groundwater flow can be represented by the continuity equation and Darcy’s law. Applying Darcy’s equation alongside the mass continuity equation, a comprehensive 3D model for groundwater flow across a confined aquifer, characterized by the hydraulic head $h$, can be formulated as follows:

$$\nabla \cdot \left(K\nabla h\right)+q_s=S_s\frac{\partial h}{\partial t},$$

(1)

where $K$ is the tensor of hydraulic conductivity of porous medium in 3D space, respectively, $q_s$ is the rate of volume flow of unit volume fluid source/sink, and $S_s$ is the specific storage of the aquifer.

On the other hand, solute transport in transient groundwater flow can be modelled as the Fickian advection–dispersion model with retardation, first introduced by Taylor [51] and Bear [52] among others.

$$\theta R\frac{\partial C}{\partial t}=\nabla \cdot \left(\theta D\nabla C\right)-\nabla \cdot \left(qC\right)+q_s{C}_s,$$

(2)

where $C$ is the solute concentration, $t$ is the time, $R$ is the retardation factor, $\theta$ is the porosity, $D$ is the dispersion coefficient tensor, $q$ is the Darcy velocity, $q_{\mathrm{s}}$ is the rate of volume flow of unit volume fluid source/sink, and $C_{\mathrm{s}}$ is the solute concentration of the fluid sink/source.

Given Equations (1) and (2), we can use the finite difference method or finite element method to spatially discretize them on a mesh [53]. This leads to a large linear system as follows:

$$A_h{u}_h=f_h,$$

(3a)

$$A_c{u}_c=f_c.$$

(3b)

where $u_h$ and $u_c$ are vectors of hydraulic heads and solute concentration. Matrix $A_h$ and $A_c$, encoding the coefficients related to groundwater flow and transport, respectively, have the order of $N$. For finite difference over a uniform mesh, for instance, $N$ is $N_x\times N_y\times N_z$, where $N_x$, $N_y$, and $N_z$ are the number of finite definite grids in $x$, $y$, and $z$ dimensions, respectively. In high-resolution 3D groundwater transport simulations, wherein the detailed representation of both the simulated site’s size and spatial heterogeneities is crucial, the order $N$ can be several million, reaching up to ${10}^8$ in our specific case study. Therefore, the solution of the massive linear system in Equation (3) can be highly computationally expensive and even prohibitive.

In this regard, we devise a domain decomposition method to parallelize the solution of groundwater flow and solute transport equations. Domain decomposition offers an effective approach for parallelism, facilitating accelerated computation while maintaining solution accuracy.

2.2. Domain Decomposition Framework: Substructuring of Groundwater Flow and Transport Systems

In this section, we present our domain decomposition framework that substructures the discrete linear systems for groundwater flow and transport. Our domain decomposition is grounded in the principles of the Schur complement method [54], which is integral to certain domain decomposition approaches, especially for the computational domain, and is divided into non-overlapping subdomains. In domain decomposition, the coupling between subdomains occurs at the interface boundaries. The Schur complement method is used to form a reduced system (Schur complement system) that only involves the degrees of freedom at the subdomain boundaries. This reduced system over subdomain boundaries connects the solutions of each subdomain, ensuring that the combined solutions of all subdomains are equal to the solution of the original problem over the entire computational domain. Due to its effectiveness and efficiency, the Schur complement method has been widely applied to solve numerical problems within the domain decomposition framework [55,56,57,58]. In the following, we derive the domain decomposition framework based on the Schur complement method.

Given a large matrix $A$ in Equation (3) due to massive nodes within the mesh, we first partition the nodes into several regular subdomains. As illustrated in Figure 1, nodes in the mesh are categorized based on their spatial relation to subdomains, specifically into interior nodes and boundary nodes. Interior nodes are located inside a subdomain and do not directly connect to nodes in other subdomains, while boundary nodes reside on the interface between subdomains. Herein, boundary nodes can further be categorized into face nodes and edge nodes. The face nodes are located at the interface between two adjacent subdomains, while the edge nodes, as the remaining boundary nodes, are located at the interfaces connected to more than two subdomains.

Based on this categorization, we reorder the matrix to generate block-wise substructures. By reordering the matrix $A$, we let interior columns of interior nodes be prioritized in the ordered matrix, followed by those of boundary nodes. Therefore, the linear system in Equation (3) can be reformulated as follows:

$$\left(\begin{array}{cc}{A}_{II}& A_{I\Gamma }\\ A_{I\Gamma }^T& A_{\Gamma \Gamma }\end{array}\right)\left(\begin{array}{c}{u}_I\\ u_{\Gamma }\end{array}\right)=\left(\begin{array}{c}{f}_I\\ f_{\Gamma }\end{array}\right),$$

(4)

where $A_{II}$ and $A_{\Gamma \Gamma }$ are the blocks associated with only interior nodes and boundary nodes, respectively, $A_{I\Gamma }$ is the block of coefficients connecting interior and boundary nodes, $u_I$ and $u_{\Gamma }$ are the variables of interior and boundary nodes, respectively, and $f_I$ and $f_{\Gamma }$ are the associated blocks on the right-hand side for these two categories of nodes. By further rearranging the columns of interior nodes based on their respective subdomains, the blocks in Equation (4) can be further refined and formulated as follows:

$$\left(\begin{array}{cc}\begin{array}{ccc}{A}_{II}^{\left(1\right)}& & \\ & \ddots & \\ & & A_{II}^{\left(n\right)}\end{array}& \begin{array}{c}{A}_{I\Gamma }^{\left(1\right)}\\ ⋮\\ A_{I\Gamma }^{\left(n\right)}\end{array}\\ \begin{array}{ccc}{A_{I\Gamma }^{\left(1\right)}}^T& \cdots & {A_{I\Gamma }^{\left(n\right)}}^T\end{array}& A_{\Gamma \Gamma }\end{array}\right)\left(\begin{array}{c}\begin{array}{c}{u}_I^{\left(1\right)}\\ ⋮\\ u_I^{\left(n\right)}\end{array}\\ u_{\Gamma }\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}{f}_I^{\left(1\right)}\\ ⋮\\ f_I^{\left(n\right)}\end{array}\\ f_{\Gamma }\end{array}\right),$$

(5)

where the superscript ${\left(\cdot \right)}^{\left(i\right)}$ indicates the block is associated with $i$-th subdomain. Figure 1 illustrates an instance of $A$ for four subdomains. It is noted that each subdomain is linked to a diagonal block, labeled as $A_{II}^{\left(i\right)}$. Additionally, each boundary corresponds to a boundary block, labeled as $A_{I\Gamma }^{\left(i\right)}$. Solution of interior nodes in Equation (5) yields the following:

$$u_I^{\left(i\right)}={A_{II}^{\left(i\right)}}^{-1}\left(f_I^{\left(i\right)}-{\sum }_{i=1}^N{A}_{I\Gamma }^{\left(i\right)}{u}_{\Gamma }\right).$$

(6)

Equation (6) indicates the variables for each subdomain can be solved independently, without knowing the values of degrees of freedom within other subdomains. Especially, since the order of $A_{II}^{\left(i\right)}$ is substantially reduced compared with that of $A_{II}$, the solution of Equation (6) has much less complexity compared with that of Equation (3). For example, in a sparse solver with $O$($N^3$) time complexity for $N$-ordered linear system, the overall time complexity for the solution of sublinear systems over subdomains is $O(\sum N_i^3)$, where $N_i$ is the order of $i$-th sublinear systems, satisfying $\sum N_i=N$. Thus, the overall complexity of these sublinear systems is much smaller than that of the original problem.

Inserting Equation (6) into Equation (5), we can derive a linear system, referred to as the Schur complement system or Schur system for brevity, as follows:

$$S_{\Gamma }{u}_{\Gamma }={\tilde{f}}_{\Gamma },$$

(7)

where

$$S_{\Gamma }=A_{\Gamma \Gamma }-A_{I\Gamma }^T{A}_{II}{A}_{I\Gamma }=A_{\Gamma \Gamma }-\sum _{i=1}^N{A_{I\Gamma }^{\left(i\right)}}^T{A_{II}^{\left(i\right)}}^{-1}{A}_{I\Gamma }^{\left(i\right)},$$

(8a)

$${\tilde{f}}_{\Gamma }=f_{\Gamma }-A_{I\Gamma }^T{A}_{II}{r}_{\Gamma }=f_{\Gamma }-\sum _{i=1}^N{A_{I\Gamma }^{\left(i\right)}}^T{A_{II}^{\left(i\right)}}^{-1}{f}_I^{\left(i\right)}.$$

(8b)

Equations (7) and (8) are related to the boundary nodes, representing the connection between subdomains. Thus, even though the solutions of the sublinear systems are independent of each other, the Schur system can “merge” the subdomains as a whole. This ensures that the independent solutions of the subsystems in Equation (6) collectively solve the original system in Equation (3).

While this domain decomposition technique significantly reduces complexity and enables parallelism by dividing the original problem into sub-problems across subdomains, it requires high computational overhead associated with the Schur complement system in Equation (7). Firstly, direct estimation of $S_{\Gamma }$ in Equation (8) involves a series of calculations of ${A_{II}^{\left(i\right)}}^{-1}{A}_{I\Gamma }^{\left(i\right)}$. This issue could be avoided by using the Krylov space method, such as CG or BiCGSTAB algorithm, which estimates the form, like $S_{\Gamma }p$ in the algorithm, thus avoiding the explicit computation of $S_{\Gamma }$. Secondly, the size of the Schur complement system depends on the number of boundary nodes. Thus, there exists a trade-off between partition granularity and solution complexity for Equation (7). Although a finely grained partition achieves low level of computation complexity over subdomains, a large linear system would be derived for increased boundary nodes between these subdomains. In the discrete solution of the solute transport equation, the preconditioned BiCGSTAB solver calls the preconditioner three times (see Appendix A). Especially in high-resolution 3D simulations, the approximated $O\left(N^3\right)$ complexity for the solution of Equation (7) might result in unacceptable computational overhead.

2.3. Domain Decomposition Preconditioner: Another Level of Substructuring

To address the efficiency issue for Schur system solution, we observe that while the boundary nodes are massive, their subset, face nodes, and edge nodes are well organized in terms of their geometry and topology, with edge nodes forming the boundary of face nodes. Hence, we could explore an additional level of domain decomposition scheme for solving groundwater flow and solute transport systems. This approach creates a domain decomposition preconditioner in the CG and BiCGSTAB algorithms.

To derive the domain decomposition preconditioner for Equation (7), in which the boundary nodes are partitioned into substructures consisting of individual faces and edges, Equation (5) is rewritten to indicate the relation between each level of substructures as follows:

$$\left(\begin{array}{ccc}\begin{array}{ccc}{A}_{II}^{\left(1\right)}& & \\ & \ddots & \\ & & A_{II}^{\left(n\right)}\end{array}& \begin{array}{ccc}{A}_{IF}^{\left(\mathrm{1,1}\right)}& \cdots & A_{IF}^{\left(1,k\right)}\\ ⋮& \ddots & ⋮\\ A_{IF}^{\left(n,1\right)}& \cdots & A_{IF}^{\left(n,k\right)}\end{array}& \begin{array}{c}{A}_{IE}^{\left(1\right)}\\ ⋮\\ A_{IE}^{\left(n\right)}\end{array}\\ \begin{array}{ccc}{A}_{FI}^{\left(\mathrm{1,1}\right)}& \cdots & A_{FI}^{\left(1,n\right)}\\ ⋮& \ddots & ⋮\\ A_{FI}^{\left(k,1\right)}& \cdots & A_{FI}^{\left(k,n\right)}\end{array}& \begin{array}{ccc}{A}_{FF}^{\left(1\right)}& & \\ & \ddots & \\ & & A_{FF}^{\left(k\right)}\end{array}& \begin{array}{c}{A}_{FE}^{\left(1\right)}\\ ⋮\\ A_{FE}^{\left(k\right)}\end{array}\\ \begin{array}{ccc}{A}_{EI}^{\left(1\right)}& \cdots & A_{EI}^{\left(n\right)}\end{array}& \begin{array}{ccc}{A}_{EF}^{\left(1\right)}& \cdots & A_{EF}^{\left(k\right)}\end{array}& A_{EE}\end{array}\right)\left(\begin{array}{c}\begin{array}{c}{u}_I^{\left(1\right)}\\ ⋮\\ u_I^{\left(n\right)}\end{array}\\ \begin{array}{c}{u}_F^{\left(1\right)}\\ ⋮\\ u_F^{\left(k\right)}\end{array}\\ u_E\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}{f}_I^{\left(1\right)}\\ ⋮\\ f_I^{\left(n\right)}\end{array}\\ \begin{array}{c}{f}_F^{\left(1\right)}\\ ⋮\\ f_F^{\left(k\right)}\end{array}\\ f_E\end{array}\right),$$

(9)

where $A_{IF}^{\left(i,j\right)}$ and its transpose, $A_{IF}^{\left(j,i\right)}$, correspond to coefficients connecting $j$-th face and $i$-th subdomain, and $A_{FF}^{\left(i\right)}$ denotes block associated with the face nodes over $i$-th face. Figure 2 illustrates Equation (9) using the example from Figure 1, generated by substructuring the boundary nodes into face nodes and edge nodes.

With this formulation, the Schur system in Equation (7) can be formulated as follows:

$$\left(\begin{array}{cc}\begin{array}{ccc}{S}_{FF}^{\left(\mathrm{1,1}\right)}& \cdots & S_{FF}^{\left(1,k\right)}\\ ⋮& \ddots & ⋮\\ S_{FF}^{\left(k,1\right)}& \cdots & S_{FF}^{\left(k,k\right)}\end{array}& \begin{array}{c}{S}_{FE}^{\left(1\right)}\\ ⋮\\ S_{FE}^{\left(k\right)}\end{array}\\ \begin{array}{ccc}{S}_{EF}^{\left(1\right)}& \cdots & S_{EF}^{\left(k\right)}\end{array}& s_{EE}\end{array}\right)\left(\begin{array}{c}\begin{array}{c}{u}_F^{\left(1\right)}\\ ⋮\\ u_F^{\left(k\right)}\end{array}\\ u_E\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}{\tilde{f}}_F^{\left(1\right)}\\ ⋮\\ {\tilde{f}}_F^{\left(k\right)}\end{array}\\ {\tilde{f}}_E\end{array}\right),$$

(10)

where

$$\begin{array}{c}{S}_{FF}^{\left(i,i\right)}=A_{FF}^{\left(i\right)}-{\sum }_{l=1}^n{A}_{FI}^{\left(i,l\right)}{A_{II}^{\left(l\right)}}^{-1}{A}_{IF}^{\left(l,i\right)},\hfill \\ \begin{array}{cc}{S}_{FF}^{\left(i,j\right)}=-{\sum }_{l=1}^n{A}_{FI}^{\left(i,l\right)}{A_{II}^{\left(l\right)}}^{-1}{A}_{IF}^{\left(l,j\right)},& i\ne j\end{array},\hfill \\ S_{EF}^{\left(i\right)}=A_{EF}^{\left(i\right)}-{\sum }_{l=1}^n{A}_{EI}^{\left(l\right)}{A_{II}^{\left(l\right)}}^{-1}{A}_{IF}^{\left(l,i\right)},\hfill \\ S_{EE}=A_{EE}-{\sum }_{l=1}^n{A}_{EI}^{\left(l\right)}{A_{II}^{\left(l\right)}}^{-1}{A}_{IE}^{\left(l\right)}.\hfill \end{array}$$

(11)

Note that $A_{FI}^{\left(i,l\right)}$ are non-zero blocks only when $i$-th face is on the boundary of $l$-th subdomain. $S_{FF}^{\left(i,i\right)}$ and $S_{FF}^{\left(i,j\right)}$ can be reformulated as follows:

$$\begin{array}{c}{S}_{FF}^{\left(i,i\right)}=A_{FF}^{\left(i\right)}-A_{FI}^{\left(i,i_F\right)}{A_{II}^{\left(i_F\right)}}^{-1}{A}_{IF}^{\left(i_F,i\right)}-A_{FI}^{\left(i,i_B\right)}{A_{II}^{\left(i_B\right)}}^{-1}{A}_{IF}^{\left(i_B,i\right)},\hfill \\ S_{FF}^{\left(i,j\right)}=\begin{array}{cc}\left\{\begin{array}{c}\begin{array}{cc}-A_{FI}^{\left(i,l\right)}{A_{II}^{\left(l\right)}}^{-1}{A}_{IF}^{\left(l,j\right)},& l\in \left\{i_F,i_B\right\},A_{IF}^{\left(l,j\right)}\ne O\end{array}\\ \begin{array}{cc}O,& \mathrm{e}\mathrm{l}\mathrm{s}\mathrm{e}\end{array}.\end{array},\right.& i\ne j\end{array}\hfill \end{array}$$

(12)

where $i_F$ and $i_B$ are the indices of two subdomains adjacent to $i$-th face.

Given the formulation in Equation (9), at first glance, we can use a preconditioner such as a block Jacobi preconditioner for solving the system.

$$M=\left(\begin{array}{cc}\begin{array}{ccc}{S}_{FF}^{\left(\mathrm{1,1}\right)}& & \\ & \ddots & \\ & & S_{FF}^{\left(k,k\right)}\end{array}& \begin{array}{c} \\ \\ \end{array}\\ \begin{array}{ccc} & & \end{array}& s_{EE}\end{array}\right),$$

(13)

In this preconditioner, however, each face between two subdomains requires the calculation of $A_{FI}^{\left(i,l\right)}{A_{II}^{\left(l\right)}}^{-1}{A}_{IF}^{\left(l,i\right)}$ twice for estimation of $S_{FF}^{\left(i,i\right)}$ or the solution of the Neumann–Neumann preconditioner [59] twice for the approximation of $S_{FF}^{\left(i,i\right)}$ as shown below:

$$\left(\begin{array}{cc}{A}_{II}^{\left(i_N\right)}& A_{IF}^{\left(i_N,i\right)}\\ A_{FI}^{\left(i,i_N\right)}& A_{FF}^{\left(i,i\right)}\end{array}\right)\left(\begin{array}{c}{z}_I^{\left(l\right)}\\ z_F^{\left(i,l\right)}\end{array}\right)=\left(\begin{array}{c}O\\ r_F^{\left(i\right)}\end{array}\right).$$

(14)

For either of the above schemes, the complexity is dependent on the size of subdomains, with a complexity of at least $O\left({\left(N_I+N_F\right)}^3\right)$, where $N_I$ and $N_F$ are the numbers of interior nodes and face nodes, respectively. For $k$ faces within the boundary, the overall computation would be highly expensive.

Observing the form of $S_{FF}^{\left(i,j\right)}$ in Equation (11), we ignore the term involving the calculation of the inverse ${A_{II}^{\left(l\right)}}^{-1}$ in preconditioning Equation (10). Moreover, we observe that the subdomain is box-shaped. When employing a seven-point stencil in the finite element discretization of Equations (1) and (2), we obtain $A_{EI}^{\left(l\right)}=O$, $l=1,\cdots ,k$ [60]. This means $S_{EF}^{\left(i\right)}=A_{EF}^{\left(i\right)}$, $S_{EE}=A_{EE}$. In summary, our preconditioner for Equation (10) is as follows:

$$\left(\begin{array}{cc}\begin{array}{ccc}{A}_{FF}^{\left(1\right)}& & \\ & \ddots & \\ & & A_{FF}^{\left(k\right)}\end{array}& \begin{array}{c}{A}_{FE}^{\left(1\right)}\\ ⋮\\ A_{FE}^{\left(k\right)}\end{array}\\ \begin{array}{ccc}{A}_{EF}^{\left(1\right)}& \cdots & A_{EF}^{\left(k\right)}\end{array}& A_{EE}\end{array}\right)\left(\begin{array}{c}\begin{array}{c}{z}_F^{\left(1\right)}\\ ⋮\\ z_F^{\left(k\right)}\end{array}\\ z_E\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}{r}_F^{\left(1\right)}\\ ⋮\\ r_F^{\left(k\right)}\end{array}\\ r_E\end{array}\right).$$

(15)

Given the upper-left diagonal block in Equation (15), we employ its special pattern for the solution of the preconditioner, which solves $z_F^{\left(k\right)}$ with respect to each face in parallel:

$$z_F^{\left(i\right)}={A_{FF}^{\left(i\right)}}^{-1}\left(z_F^{\left(i\right)}-A_{FE}^{\left(i\right)}{z}_E\right),i=1,\cdots ,k.$$

(16)

After eliminating $z_F^{\left(i\right)}$, Schur system for Equation (15) is obtained as follows:

$$S_{EE}{z}_E={\tilde{r}}_E,$$

(17)

where

$$S_{EE}=A_{EE}-\sum _{i=1}^k{A}_{EF}^{\left(i\right)}{A_{FF}^{\left(i\right)}}^{-1}{A}_{FE}^{\left(i\right)},$$

(18a)

$${\tilde{r}}_E=r_E-\sum _{i=1}^k{A}_{EF}^{\left(i\right)}{A_{FF}^{\left(i\right)}}^{-1}{r}_F^{\left(i\right)}.$$

(18b)

It is worth noting that the order of the linear system in Equation (18) is the number of edge nodes, with only a small proportion of boundary nodes. Therefore, the complexity for solving Equation (17) is significantly lower than that of Equation (7). It can be solved directly using preconditioned CG or BiCGSTAB. To avoid the estimation of the inverse ${A_{FF}^{\left(i\right)}}^{-1}$, $A_{EE}$ is used as a preconditioner for Equation (17), following the approach in Equation (15).

By combining Equations (16)–(18), we establish domain decomposition conditioner for Equation (7). This preconditioner avoids the expensive computation in block Jacobi preconditioner, such as the $2k$ times estimation of $A_{FI}^{\left(i,l\right)}{A_{II}^{\left(l\right)}}^{-1}{A}_{IF}^{\left(l,i\right)}$ or $2k$ times solution of Neumann–Neumann preconditioner, as in Equation (13), in which the complexity is $O\left({\left(N_I+N_F\right)}^3\right) \mathrm{a}\mathrm{n}\mathrm{d} N_I\gg N_F$ for each face. By fully exploiting the disconnected topology of faces over the boundary, the preconditioner can parallelize the solution of sublinear systems associated with faces, with the complexity of $O\left(N_F^3\right)$ for each face. Thereby, the preconditioner reduces complexity substantially, improving the efficiency of the solution of the Schur system as in Equation (7). We will demonstrate in our case study that, due to the speedup in solution of Equation (7), we do not need to be too sensitive to the size of boundary nodes, thereby allowing for finely grained domain decomposition for low levels of complexity and efficiency in the groundwater simulation.

2.4. Solution of Subdomain Linear Systems

When the above domain decomposition method is applied to groundwater simulation, the linear system in Equation (3) will be solved multiple times in the numeric modeling process. In a solution of the linear system, the sublinear system with respect to ${A_{II}^{\left(i\right)}}^{-1}$, ${A_{FF}^{\left(i\right)}}^{-1}$, and ${A_{EE}^{\left(i\right)}}^{-1}$ in each subdomain will be invoked many times. In particular, the order of $A_{II}^{\left(i\right)}$ is generally large. Thus, the selection of an appropriate linear solver for these linear systems is essential for the efficiency of the proposed method.

It is observed that the linear system (Equation (3a)) discretized from the groundwater flow equation is fixed in the time step of numeric modeling if the model parameters, such as hydraulic conductivity and specific storage, are independent of time. In such a case, we can precompute the Cholesky or LU factorizations for these linear systems before the iterations of numeric modeling. The Cholesky f and LU factorizations have time and space complexities of $O\left(N^3\right)$ and $O\left(N^2\right)$, respectively, where $N$ is the number of interior nodes for a subdomain. In the solution of these linear systems, the precomputed matrix decomposition can be reused, and only back-substitute with the time complexity of $O\left(N^2\right)$ is needed, thereby enhancing the performance of the domain decomposition method. This positive attribute is due to the fact that the dual-domain decomposition method allows for smaller subdomain sizes, thereby reducing the time and memory complexity of matrix factorization.

The linear system (Equation (3b)) derived from solute transport equations undergoes changes during iterations due to variable groundwater flow. In this scenario, we propose using preconditioned BiCGSTAB to solve these linear systems within each subdomain. Preconditioners such as incomplete LU decomposition can be utilized. It is noteworthy that BiCGSTAB invokes the preconditioner three times (see Algorithm A1 in Appendix A). Thus, the efficiency of solving the preconditioning system is crucial for optimizing the performance of the Schur system solution. The precomputation of incomplete LU decomposition is performed in the initialization of each iteration step. Although the linear the linear system is different in each iteration, the structures of their nonzero entries can remain fixed if parameters such as dispersion coefficient tensor, porosity, and retardation factor in the solute transport equation are time-independent. Therefore, in computing the incomplete LU decomposition, we optimize efficiency by reusing fill-reducing reordering and symbolic factorization, updating only values of the factorization.

2.5. Algorithm and Implementation

According to Section 2.2, Section 2.3 and Section 2.4, the workflow for the dual-domain decomposition can be summarized. For a time-sequential simulation, initially, the mesh is discretized into non-overlapping subdomains $\mathcal{D}=\left\{{\Omega }_1,\cdots ,{\Omega }_n\right\}$, generating matrix blocks corresponding with the interior, boundary, faces, and edges of subdomains. The matrix factorization for blocks associated with subdomains is precomputed. The dual-domain decomposition algorithm, outlined in Algorithm 1, receives blocks and their matrix factorizations, and solves the linear system discretized from groundwater flow and solute transport equations (Equation (3)) using two levels of domain decomposition. In the first level of domain decomposition, the subsystems for each subdomain are solved in parallel through efficient back-substitution based on the precomputed matrix factorization. The CG or BiCGSTAB method (see BiCGSTAB version in Appendix A) is utilized to solve the Schur system resulting from the domain decomposition. In particular, another domain decomposition preconditioner, which is summarized in the pseudo-code in Algorithm 2, is used for preconditioning the Schur system. Figure 3 illustrates a schematic diagram of the dual-domain decomposition process for the solution of Equation (3).

Algorithm 1. Dual-domain decomposition for the solution of linear system discretized from groundwater flow and solute transport equations

Input: Sets of blocks ${\mathcal{A}}_{II}=\left\{A_{II}^{\left(1\right)},\cdots ,A_{II}^{\left(n\right)}\right\}$, ${\mathcal{A}}_{I\Gamma }=\left\{A_{I\Gamma }^{\left(1\right)},\cdots ,A_{I\Gamma }^{\left(n\right)}\right\}$, $A_{\Gamma \Gamma }$, ${\mathcal{A}}_{FF}=\left\{A_{FF}^{\left(1\right)},\cdots ,A_{FF}^{\left(k\right)}\right\}$, ${\mathcal{A}}_{I\Gamma }=\left\{A_{FE}^{\left(1\right)},\cdots ,A_{FE}^{\left(k\right)}\right\}$, $A_{EE}$, ${\mathcal{f}}_I=\left\{f_I^{\left(1\right)},\cdots ,f_I^{\left(N\right)}\right\}$, $f_{\Gamma }$, ${\mathcal{f}}_F=\left\{f_F^{\left(1\right)},\cdots ,f_F^{\left(k\right)}\right\}$, and $f_E$; Sets of matrix factorization for blocks of interior and face nodes ${\mathcal{L}}_{II}=\left\{L_{II}^{\left(1\right)},\cdots ,L_{II}^{\left(n\right)}\right\}$, ${\mathcal{L}}_{FF}=\left\{L_{FF}^{\left(1\right)},\cdots ,L_{FF}^{\left(k\right)}\right\}$, and $L_{EE}$. Ouput: $u$ // Equation (8b) parallel for $i=1,\cdots ,n$ do        $v^{\left(i\right)}$ ← LinearSolver($\left\{A_{II}^{\left(i\right)},L_{II}^{\left(i\right)}\right\},f_I^{\left(i\right)}$)        $w^{\left(i\right)}$ ← ${A_{I\Gamma }^{\left(i\right)}}^T{v}^{\left(i\right)}$ end parallel for ${\tilde{f}}_{\Gamma }$ ← $f_{\Gamma }-{\sum }_{i=1}^n{w}^{\left(i\right)}$ // Equation (7) Preconditioner ← DomainDecompositionPreconditioner PrecondParam ← $\{{\mathcal{A}}_{FF}$,${\mathcal{L}}_{FF}$,${\mathcal{A}}_{FE}$,$A_{EE}$, $L_{EE}\}$ $u_{\Gamma }$ ← SchurSystemSolver($\left\{{\mathcal{A}}_{II},{\mathcal{L}}_{II}\right\}$,${\mathcal{A}}_{I\Gamma }$,$A_{\Gamma \Gamma }$, ${\tilde{f}}_{\Gamma }$, $n$, Preconditioner, PrecondParam) // Equation (6) parallel for $i=1,\cdots ,n$ do    $t_I^{\left(i\right)}$ ← $f_i^{\left(i\right)}-A_{I\Gamma }^{\left(i\right)}{u}_{\Gamma }$    $u_I^{\left(i\right)}$ ← LinearSolver($\left\{A_{II}^{\left(i\right)},L_{II}^{\left(i\right)}\right\},t_I^{\left(i\right)}$) end parallel for $u$ ← Reorder($x_I=\left\{u_I^{\left(1\right)},\cdots ,u_I^{\left(n\right)}\right\},u_{\Gamma }$)

Algorithm 2. Domain decomposition preconditioner

function DomainDecompositionPreconditioner (PrecondParam ← $\{{\mathcal{A}}_{FF},{\mathcal{L}}_{FF},{\mathcal{A}}_{FE}$,$A_{EE}$, $L_{EE}$}, $r\leftarrow \{r_F$,$r_E$})     // Equation (18b)    parallel for $i=1,\cdots ,k$ do        $v^{\left(i\right)}$ ← LinearSolver($\left\{A_{FF}^{\left(i\right)},L_{FF}^{\left(i\right)}\right\},r_F^{\left(i\right)}$)        $w^{\left(i\right)}$ ← ${A_{FE}^{\left(i\right)}}^T{v}^{\left(i\right)}$    end parallel for    $r_F$ ← $r_{\Gamma }-{\sum }_{i=1}^n{w}^{\left(i\right)}$    // Equation (17)    Preconditioner ← IncompleteCGPreconditioner PrecondParam ← $L_{EE}$    $z_{\Gamma }$ ← SchurSystemSolver($\left\{{\mathcal{A}}_{FF}, {\mathcal{L}}_{FF}\right\}$, ${\mathcal{A}}_{FE}$,$A_{EE}$, $r_F$, $k$, Preconditioner)    // Equation (16)    parallel for $i=1,\cdots ,k$ do        $t^{\left(i\right)}$ ← $z_F^{\left(i\right)}-A_{FE}^{\left(i\right)}{z}_E$        $z_F^{\left(i\right)}$ ← LinearSolver($\left\{A_{FF}^{\left(i\right)},L_{FF}^{\left(i\right)}\right\},$ $t^{\left(i\right)}$)    end parallel for    return $z$ ← Reorder($z_F=\left\{z_F^{\left(1\right)},\cdots ,z_F^{\left(k\right)}\right\},z_{\Gamma }$) end

The proposed algorithm parallelizes major computations over multiple threads across subdomains and boundary faces. There are a total of four parallel blocks in the algorithm. Each thread corresponds to independent computations over subdomains and boundary faces, with no communication between threads, thereby requiring only one synchronization point when the parallel branches are completed. Additionally, each thread’s memory is independent and only related to its associated subdomains. The subdomain-related variables, $A_{II}^{\left(i\right)}$, $A_{FF}^{\left(i\right)}$, $A_{I\Gamma }^{\left(i\right)}$, and $A_{EE}^{\left(i\right)}$, and the vectors, $f_I^{\left(i\right)}$ and $r_F^{\left(i\right)}$, remain fixed during the solution process. In a shared memory parallel system, these data can be organized independently, thus avoiding additional communication costs between threads. In a distributed-memory parallel system, the total memory communication across all threads is equivalent to the size of matrix $A$, which also introduces no additional communication cost. For each thread in a shared-memory system, the required communication cost during the solution process includes only a single instance of communication of vector $u_{\Gamma }$, which depends on the problem size, and multiple instances of communication of ${\tilde{r}}_E$, which are related to the size of boundary nodes and are kept small in practice. Therefore, the proposed algorithm maintains low communication cost and minimal synchronization cost in the parallel solution of massive linear systems.

3. Study Area and Data Preparation

The study area is located in a chromium slag legacy site of a former ferroalloy factory in Hunan Province, China. It is bordered by Baituo village to the west and Alunan village and the Lianshui River to the southeast. We evaluated the performance of the proposed method by modeling the transport of Cr(VI) in confined aquifers. The study area (Figure 4) is a chromium slag-contaminated site located in Hunan Province, China, with the Lianshui River roughly 1.0 km southeast of the site, which is in a humid subtropical climate region. The site was an abandoned ferroalloy plant covering 0.5901 km², which produced chromium metals. Chromium was produced for nearly 50 years at this site until it was closed in 2006. It is estimated that approximately 200,000 tons of chromium-bearing slags and tailings were produced and disposed of on the ground during this period. This extensive production and accumulation of slag at the site are believed to have caused the leakage of chromium into the soil and groundwater, resulting in severe Cr(VI) contamination. Therefore, a high-resolution 3D simulation of Cr(VI) transport in the groundwater is essential for understanding its migration and guiding post-remediation efforts at the site.

Based on the 3D model in Figure 5, the site of the study area is discretized into 3D grids of different sizes.

Table 1. Summary of different settings of finite difference grids and the numbers of degrees of freedom.

Scale	Grid Size(m)			Number of Grids in Each Direction
	X	Y	Z	X	Y	Z
1,080,000	10	10	0.5	120	150	60
2,160,000	5	5	1	240	300	30
5,400,000	4.5	4.5	0.5	300	300	60
6,750,000	4	4	0.5	300	375	60
9,000,000	3.5	3.5	0.5	400	375	60
13,500,000	2	2	1	600	750	30
27,000,000	2	2	0.5	600	750	60
54,000,000	1.5	1.5	0.5	1200	750	60
108,000,000	1	1	0.5	1200	1500	60

summarizes different settings of finite difference grids and the number of degrees of freedom in the simulation. The spatial resolutions of the 3D grids range from 10 m to 1 m, corresponding to a number of nodes with 1.08 million to 108 million degrees of freedom. Thus, our simulations cover high-resolution 3D simulation cases.

Figure 6 illustrates the northwest boundary condition, which specifies the flux along the northwest boundaries, representing the lateral flow recharge northwest of the site. The southeast boundary condition is a specified-head boundary set uniformly to a constant. A uniform recharge rate of $3.0\times {10}^{-7}$ m/s is applied to the northwest boundary, and a water head of 48 m is used for the outflow boundary (southeast boundary) based on the river stage of the Lianshui River. The top boundary condition is defined as a flux-specific boundary representing the rainfall recharge at a rate of 1400 mm/yr based on the region’s annual precipitation. The bottom boundary condition is a no-flow boundary since the silty mudstone layer at the bottom of the aquifer is considered impermeable. In the solute transport model, all boundaries except the southeast (outflow) boundary are set to zero-mass flux. The mass flux for the southeast boundary is set to the outflow rate through the boundary. Besides the above boundary condition, the other model parameters are set according to our previous work [61].

Given the above 3D grid models, boundary conditions, and model parameters, we use the finite difference method to discretize the models of groundwater flow and solute transport (Equations (1) and (2)). This discretization generates linear systems (Equation (3)) for the groundwater flow and transport simulations. The size of these linear systems is massive, corresponding to the number of nodes in the 3D grid models. We employ the dual-domain decomposition method to solve these extensive linear systems.

4. Results and Discussion

Our dual-domain decomposition method was implemented with C++. To validate the performance of our method, we conducted a comparative study on a PC equipped with an Intel i7 12700KF CPU with 12 cores and 20 threads and 32 GB RAM. The PC runs under the Windows 10 operating system. The numeric problems of groundwater flow and transport were tested. Various scenarios were designed with different degrees of freedom, ranging from 13.5 to 108 million. By default, the CG and CG-BISTAB algorithms are used to solve the sublinear systems over subdomains for the groundwater flow equation and the solute transport equation, respectively.

4.1. Performance of the Dual-Domain Decomposition Method

The efficiency of the dual-domain decomposition method is evidently dependent on the grain sizes of subdomains. To assess this dependency, we evaluated the performance of the proposed method with varying numbers of subdomains.

Table 2. Performance statistics of the dual-domain decomposition method for groundwater flow and groundwater flow problems, including the number of degrees of freedom (#.dofs), number of subdomains (#.subdomains), number of iterations (#.iterations), time to obtain the solution of Schur system (Schur), and total solution time (total). The thresholds of the subdomains with the best performance in each scenario are highlighted in bold.

#.Dofs	Groundwater Flow				Groundwater Transport
	#.Subdomains	#.Iterations	Schur (s)	Total (s)	#.Subdomains	#.Iterations	Schur (s)	Total (s)
13,500,000	40	12	0.532795	126.187	20	4	0.0258069	12.769
	80	8	0.742615	86.0236	40	3	0.0320886	11.0436
	180	6	1.44983	46.5398	80	3	0.0412827	9.8365
	360	5	3.55114	29.4948	150	3	0.0687989	10.3101
	500	5	6.00332	49.4242	180	3	0.0768018	11.4264
27,000,000	80	11	1.04149	264.314	20	4	0.0414583	40.6114
	160	8	2.04289	162.173	40	3	0.0477507	33.7732
	360	6	4.84669	77.7762	80	3	0.0587087	27.8985
	400	6	5.59473	40.5947	150	3	0.0984913	28.7126
	450	5	7.28272	85.256	180	3	0.110725	29.075
108,000,000	180	35	3.82804	659.22	40	4	0.0889159	144.339
	240	34	6.25082	517.344	80	4	0.119546	130.989
	300	12	7.57162	211.722	100	3	0.251899	116.139
	600	10	33.4211	194.079	180	3	0.265036	134.533
	800	7	41.15969	432.267	240	3	0.350629	149.863

summarizes the number of iterations and solution times obtained with twenty threads in total employed in the proposed method. The results indicate that, for both groundwater flow and groundwater transport problems, both the timing and number of iterations of the proposed method generally decrease as the number of subdomains increases. This demonstrates that parallelizing the computation across more subdomains enables an improvement in solution performance. The performance continues to improve as the number of subdomains increases, surpassing the available number of threads. This is attributed to the finely grained partitioning of subdomains, resulting in smaller linear systems for each domain. While increasing the number of subdomains may lead to a large Schur system, our observations indicate that the solution remains efficient. This efficiency is maintained through the utilization of second-level domain decomposition methods during the solution process.

It is noteworthy that the performance begins to decline when the number of subdomains exceeds a certain threshold. This deterioration is attributed to the limited number of threads in our experiment, which are insufficient to efficiently process the increased number of subdomains. When the size of the subdomains is refined sufficiently, the solution time for sublinear problems associated with subdomains cannot be significantly reduced. Thus, the computations required to handle the increased number of subdomains and the larger Schur system outweigh the efficiency gains from smaller subdomain sizes. Conducting a grid test of the runtime, as shown in Table 2, to find the optimal number of subdomains is impractical. However, since the runtime for subproblems across different subdomains is generally similar, we can test the runtime of a single subdomain (using one thread) to estimate the overall runtime for different numbers of subdomains, thereby approximating the optimal number of subdomains. Anyway, the dual-domain decomposition method enables a fast solution of the Schur system over subdomain boundaries. This performance boost helps mitigate the impact of the increased size of the Schur system and allows us to focus more on the variation in runtime over subdomains when tuning the number of subdomains.

The number of subdomains in different grid resolutions can influence the convergence rates in solving the Schur system. Table 2 includes the number of iterations required to solve the Schur system. It is observed that the number of iterations progressively decreases as the number of subdomains increases. This indicates that our dual-domain decomposition preconditioner attains higher convergence rates for large Schur systems due to the finely grained partitioning of the domain, which includes more face and edge nodes. This is reasonable because finely grained subdomains result in a greater number of smaller boundary faces. Since we use approximations for the face nodes and no approximation for the boundary nodes in the preconditioning system in Equation (15), a smaller size for the boundary faces and more boundary edges between them facilitate a better preconditioning of the Schur system, thereby improving the convergence rates in solving the Schur system. A larger number of grids tends to generate more efficient solutions, and in such cases, the number of iterations is less sensitive to the grid resolution (Table 2). This characteristic enables the dual-domain decomposition method to scale effectively for high-resolution groundwater simulation problems.

4.2. Comparision with Dual-Domain Decomposition Method

To assess the efficiency gains of the proposed dual-domain decomposition method, specifically focusing on the impact of second-level domain decomposition on the Schur system, we conducted a comparative analysis with the vanilla domain decomposition method. Firstly, we evaluated the performance of the vanilla domain decomposition method across different numbers of subdomains. In the first evaluations, all 20 threads were employed for the solution of the vanilla domain decomposition method. Similar to the dual-domain decomposition method, the efficiency of the vanilla domain decomposition method increases with the number of subdomains (see

Table 3. Performance statistics of the vanilla domain decomposition method for groundwater flow and groundwater flow problems, including the number of degrees of freedom (#.dofs), number of subdomains (#.subdomains), number of iterations (#.iterations), time to obtain the solution of Schur system (Schur), and total solution time (total). The thresholds of the subdomains with the best performance in each scenario are highlighted in bold.

#.Dofs	Groundwater Flow				Groundwater Transport
	#.Subdomains	#.Iterations	Schur (s)	Total (s)	#.Subdomains	#.Iterations	Schur (s)	Total (s)
13,500,000	80	115	1.91617	206.608	20	7	1.18273	35.08
	100	113	2.04219	173.782	40	7	1.21301	34.58
	150	113	2.15086	362.008	60	7	1.41886	35.35
27,000,000	100	126	4.16009	348.216	20	10	1.68527	105.09
	150	126	4.179493	269.078	40	10	1.88841	104.00
	180	123	5.233974	324.146	60	11	2.37461	105.12
108,000,000	150	137	8.358371	1932.62	40	10	5.80962	888.14
	180	137	8.445143	1672.46	60	10	6.67187	876.72
	240	117	9.950672	1984.15	80	10	7.38562	881.48

). However, its efficiency peaks at a much smaller number of subdomains compared with that of the dual-domain decomposition method. Consequently, the efficiency of the vanilla method is significantly lower compared to that of the dual-domain decomposition method, particularly with larger numbers of subdomains. This difference is further magnified by speedup ratios of 8.617× (groundwater flow) and 4.541× (groundwater transport) when handling large linear systems with 108 million degrees of freedom.

We further compared the performance of the vanilla algorithm using the same number of subdomains, as detailed in

Table 4. Solution time, iterations, and acceleration ratio for groundwater flow problem at different scales.

Scales	Vanilla Domain Decomposition		Dual-Domain Decomposition
	Iterations	Solution Time (s)	Iterations	Solution Time (s)	Speedup
2,160,000	24	5.60752	6	4.47508	1.253
5,400,000	36	37.5881	6	16.3776	2.295
9,000,000	87	58.0438	7	18.1569	3.197
13,500,000	118	177.883	7	31.6426	5.622
27,000,000	122	274.263	6	39.9694	6.862
54,000,000	123	891.416	8	100.641	8.857
108,000,000	151	1814.169	10	194.079	9.348

and

Table 5. Solution time, iterations, and acceleration ratio for groundwater transport problem at different scales.

Scales	Vanilla Domain Decomposition		Dual-Domain Decomposition
	Iterations	Solution Time (s)	Iterations	Solution Time (s)	Speedup
2,160,000	13	8.63802	4	3.55728	2.403
5,400,000	15	24.7724	3	9.3891	2.563
9,000,000	15	32.7028	3	10.7667	2.990
13,500,000	14	40.8291	4	13.1502	3.045
27,000,000	17	158.1768	5	34.9801	4.541
54,000,000	17	495.139	5	83.895	5.904
108,000,000	21	979.871	8	144.2816	6.788

and illustrated in Figure 7. All twenty threads were utilized to evaluate the efficiency of the vanilla domain decomposition method. The vanilla decomposition method exhibits a significant performance deterioration in timing compared with that of the dual-domain decomposition approach. This deterioration in performance becomes particularly noticeable with the larger sublinear systems and increased solution times for the Schur system. For linear systems with 108 million degrees of freedom, the dual-domain decomposition method achieves a 9.348× improvement in groundwater flow and a 6.788× improvement in groundwater transport.

When the vanilla domain decomposition method is used to solve the linear systems of the same size over subdomains in parallel, the performance deterioration is mainly due to bottlenecks in the solution of large Schur systems for the boundary nodes. Figure 8 illustrates a comparison of the convergence process in the solution of the Schur systems. Clearly, a fast convergence is observed in the dual decomposition method, highlighting the effectiveness of the second level domain decomposition method as a preconditioner for solving Schur systems.

In groundwater simulations over a time sequence, precomputed matrix factorization techniques as explained in Section 2.4 and Section 2.5 can accelerate the simulation of groundwater flow and transport processes. We compared the performance between the dual-domain decomposition and vanilla domain decomposition methods when employing the precomputation trick in time-sequential simulations. The groundwater flow and transport were simulated over 280 days (Figure 9), with a time differential step of 1 day.

Table 6. Performance comparisons of the dual-domain decomposition (DDD), the vanilla domain decomposition (VDD) with the optimal number of subdomains (Opt.), and the vanilla domain decomposition with same number of subdomains (Same) in time-sequential groundwater simulation.

Simulated Time (day)	1.08 Million Model					6.75 Million Model
	DDD (s)	VDD				DDD (s)	VDD
		Opt. (s)	Speedup	Same (s)	Speedup		Opt. (s)	Speedup	Same (s)	Speedup
40	17.19	24.36	1.42	27.59	1.61	55.81	122.05	2.19	140.94	2.53
80	25.57	40.65	1.59	43.81	1.71	87.01	235.50	2.71	264.34	3.04
120	32.59	56.36	1.73	59.44	1.82	113.00	351.37	3.11	378.29	3.35
160	40.88	76.25	1.87	79.50	1.95	145.93	460.98	3.16	497.82	3.41
200	49.25	96.57	1.96	99.91	2.03	176.96	571.64	3.23	619.87	3.50
240	56.34	113.51	2.02	117.12	2.08	202.86	680.53	3.36	739.50	3.65
280	64.71	133.92	2.07	137.43	2.12	235.01	790.47	3.36	858.12	3.65
Simulated Time (day)	27 Million Model					108 Million Model
	DDD (s)	VDD				DDD (s)	VDD
		Opt. (s)	Speedup	Same (s)	Speedup		Opt. (s)	Speedup	Same (s)	Speedup
40	308.19	893.31	2.90	1210.17	3.93	3644.59	N/A	N/A	21,992.52	6.03
80	397.48	1307.08	3.29	1782.37	4.48	6648.55	N/A	N/A	41,686.35	6.27
120	508.20	1835.59	3.61	2458.92	4.84	9360.13	N/A	N/A	63,472.67	6.78
160	639.35	2519.05	3.94	3370.02	5.27	11,779.57	N/A	N/A	85,143.74	7.23
00	764.28	3202.51	4.19	4251.11	5.56	13,914.12	N/A	N/A	111,016.07	7.98
240	883.11	3886.06	4.40	5032.31	5.70	15,972.98	N/A	N/A	130,371.25	8.16
280	958.93	4339.93	4.53	5634.61	5.88	17618.69	N/A	N/A	149,036.50	8.46

and Figure 10 present the runtime and speedup values achieved in the time-sequential simulation. Our results demonstrate that the dual-domain decomposition method maintains a superior time efficiency compared to that of the vanilla domain decomposition approach, achieving a speedup of 8.46× with 108 million degrees of freedom.

Figure 11 illustrates the memory consumption data. The groundwater flow and the groundwater transport problems are solved concurrently in the time sequence simulation, which requires a rather high memory overhead. Notably, the memory usage of the vanilla method with the optimal number of subdomains is significantly higher than that of the proposed method. Its memory requirements exceed the capacity of our testing PC in the case studied with 108 million degrees of freedom. This limitation prohibits the vanilla method from reaching its optimal timing, as shown in Table 6. The pre-computed matrix factorization requires $O\left(N^2\right)$ of memory complexity for N-ordered linear systems. Therefore, a more finely grained partition of subdomains would improve the scalability of the domain decomposition method. However, this would also increase the size of the Schur system, adding computational burden for solutions. Thanks to the second level of domain decomposition, the proposed method enables a finely grained partition of subdomains while maintaining efficiency in the solution of the Schur system, with only a small compromise in memory overheads.

4.3. Comparision with Algebraic Multigrid Preconditioned Method

Algebraic multigrid (AMG) methods represent a significant advancement in groundwater simulations. The AMG method has been widely recognized for its remarkable efficiency improvements in comparison to other Krylov subspace methods, such as CG and BiCGSTAB [34]. Many industrial groundwater simulation software packages integrate the AMG method as an advanced solver and/or preconditioner [62], often acknowledged as the fastest solver option in their toolkit.

In this context, we conducted a comparative analysis between the proposed method and an efficient parallel AMG preconditioner [63] implemented in PETSC [64]. Firstly, we assessed the runtime performance of the AMG method for solving groundwater flow and transport problems. Following a setup similar to that used for testing the proposed method, we tested problem sizes ranging from 2.16 to 108 million degrees of freedom.

Table 7. Performance comparison between the algebraic multigrid (AMG) preconditioned method and dual-domain decomposition (DDD) method for groundwater flow and transport problems with different numbers of degree of freedom (#.dof).

#.Dof	Groundwater Flow			Groundwater Transport
	AMG (s)	DDD (s)	Speedup	AMG (s)	DDD (s)	Speedup
2,160,000	5.8122	4.47508	1.299	5.8579	3.55728	1.647
5,400,000	28.4848	16.3776	1.739	16.9661	9.3891	1.807
9,000,000	32.7345	18.15694	1.803	24.7465	10.7667	2.298
13,500,000	65.766	31.6426	2.078	32.9329	13.1502	2.504
27,000,000	99.8649	39.9694	2.499	88.3089	34.9801	2.524
54,000,000	414.229	100.641	4.116	385.892	83.895	4.600
108,000,000	901.023	194.079	4.643	795.7181	144.2816	5.515

presents the solution times achieved by the AMG method for the groundwater flow and transport problems. Additionally, Figure 12 provides a comparison of the runtime performance between the AMG method and the proposed method. The results demonstrate that the dual-domain decomposition method consistently outperforms the AMG preconditioned method across both groundwater flow and transport simulations, regardless of the problem size. The speedup ratios achieved by the dual-domain decomposition method are particularly noteworthy, reaching up to approximately 5× for the simulation with 108 million degrees of freedom.

We further evaluated the performance of the AMG preconditioned method in time-sequential simulations, as outlined in

Table 8. Performance comparison between the algebraic multigrid (AMG) preconditioned method and the dual-domain decomposition (DDD) method in time-sequential groundwater simulation.

Time (day)	1.08 Million			6.75 Million			27 Million			54 Million			108 Million
	AMG (s)	DDD (s)	Speedup	AMG (s)	DDD (s)	Speedup	AMG (s)	DDD (s)	Speedup	AMG (s)	DDD (s)	Speedup	AMG (s)	DDD (s)	Speedup
40	25.83	17.19	1.50	127.11	55.81	2.278	1163.53	308.19	3.78	5199.29	1207.56	4.31	N/A	3644.59	N/A
80	41.52	25.57	1.62	241.37	87.01	2.773	1657.53	397.48	4.17	7606.45	1682.26	4.52	N/A	6648.55	N/A
120	56.79	32.59	1.742	349.00	113.00	3.089	2281.97	508.20	4.49	10,595.04	2184.81	4.85	N/A	9360.13	N/A
160	75.65	40.88	1.85	453.52	145.93	3.108	3021.25	639.35	4.73	14,144.13	2573.68	5.50	N/A	11,779.57	N/A
200	94.56	49.25	1.92	568.02	176.96	3.21	3762.34	764.28	4.92	17,650.56	3057.26	5.77	N/A	13,914.12	N/A
240	110.00	56.34	1.95	673.29	202.86	3.319	4506.21	883.11	5.10	21,120.64	3535.43	5.97	N/A	15,972.98	N/A
280	131.81	64.71	2.04	815.13	235.01	3.468	5130.56	958.93	5.35	23,414.58	3796.48	6.17	N/A	17,618.69	N/A

. In this evaluation, groundwater flow and transport over 280 days are simulated. The results presented in Table 8 and Figure 13 illustrate a comparison between the AMG preconditioned method and the proposed method. It is evident that the proposed method significantly outperforms the AMG preconditioned method in time-sequential simulations. Particularly noteworthy is the substantial increase in speedup ratios compared to those reported in Table 8 and Figure 13. This improvement is attributed to the dual-domain decomposition scheme, which enables the reuse of matrix factorization to expedite the solution process.

Please note that the runtime of the AMG preconditioned method for the problem with 108 million degrees of freedom is not available (Table 8). This is due to the high level of memory usage for the AMG preconditioned method. For the problem with 108 million degrees of freedom, the memory usage exceeds the maximum system memory capacity. Therefore, besides its superior time efficiency, the proposed method also outperforms the AMG preconditioned method in terms of memory usage.

While the AMG preconditioned method [63] offers superior convergence rates compared to those of the vanilla Krylov subspace methods, the proposed method presents several advantages over the AMG method. Firstly, the proposed method is inherently parallel and incurs low communication and synchronization costs. Although the MG method can be parallelized, it often requires careful designing to maintain its efficiency, especially considering the high cost of inter-grid transfers. Consequently, the proposed method scales well in parallel systems, as each subdomain can be handled independently, making it efficient for extremely large groundwater simulation problems. Secondly, the nature of independent subdomains results in minimal memory requirements. Even with a second level of domain decomposition, the memory needed is related only to the scale of boundary nodes. In contrast, when addressing extremely large problems, the MG method requires a significant amount of memory due to managing multiple grid levels and ensuring efficient transfers between them. Overall, the proposed method achieves both time and memory efficiency in comparison to the AMG method.

5. Conclusions

In this paper, we propose a novel dual-domain decomposition method designed for groundwater flow and transport simulations. In summary, our method achieves the following contributions and key findings:

(1)

An additional level of domain decomposition is strategically devised to address the efficiency problem associated with solving the Schur system, a common bottleneck in vanilla domain decomposition methods. The tailored domain decomposition preconditioner enables the parallelization of the preconditioning process, ensuring a faster convergence rate compared with that of traditional Neumann–Neumann preconditioners when solving the Schur system;

(2)

The enhanced efficiency afforded by the proposed method allows for the partitioning of large-scale groundwater simulation problems into finely grained subdomains. This significantly reduces the computational overhead required for solving subdomains, resulting in notable speedups and memory savings during the solution process;

(3)

Compared to the vanilla domain decomposition method, our approach achieves efficient performance boosts in solving the Schur system and thus enables finely grained subdomains. For an extremely large groundwater flow and transport problem with 108 million degrees of freedom, these characteristics lead to 8.617× and 4.541× speedups for groundwater flow and transport scenarios, respectively, and enable the use of a precomputed matrix factorization trick with an optimal number of subdomains to enhance its performance in time-sequential simulations;

(4)

Compared to the AMG preconditioned method, our approach offers a well-established parallelization mechanism and lower memory costs. When solving a problem with 108 million degrees of freedom, these characteristics result in 4.643× and 5.515× speedups for groundwater flow and transport scenarios, respectively. Moreover, our method supports efficient time-sequential simulations, which are not feasible with the AMG preconditioned method due to memory overload.

The limitation of the proposed method is the difficulty in specifying the optimal number of subdomains, a challenge inherited from the vanilla domain decomposition method. While we propose a low-cost heuristic approach to determine the number of subdomains, fine-tuning this number still requires grid testing. In the future, we will incorporate the machine learning techniques, such as Bayesian optimization, into the heuristic to infer the optimal number of subdomains. Since the relationship between the number of subdomains and the solution time can be regarded as a convex function, machine learning techniques can be promising in finding a well-tuned number of subdomains in only a few low-cost attempts. This would result in sufficiently finely grained subdomains, thus unlocking the full potential of the domain decomposition scheme for efficiently simulating large-scale, high-resolution, and long-term 3D groundwater simulations.