Greedy Kernel Methods for Approximating Breakthrough Curves for Reactive Flow from 3D Porous Geometry Data

Abstract

We address the challenging application of 3D pore scale reactive flow under varying geometry parameters. The task is to predict time-dependent integral quantities, i.e., breakthrough curves, from the given geometries. As the 3D reactive flow simulation is highly complex and computationally expensive, we are interested in data-based surrogates that can give a rapid prediction of the target quantities of interest. This setting is an example of an application with scarce data, i.e., only having a few available data samples, while the input and output dimensions are high. In this scarce data setting, standard machine learning methods are likely to fail. Therefore, we resort to greedy kernel approximation schemes that have shown to be efficient meshless approximation techniques for multivariate functions. We demonstrate that such methods can efficiently be used in the high-dimensional input/output case under scarce data. Especially, we show that the vectorial kernel orthogonal greedy approximation (VKOGA) procedure with a data-adapted two-layer kernel yields excellent predictors for learning from 3D geometry voxel data via both morphological descriptors or principal component analysis.

1. Introduction

The reactive flow in porous media plays an important role for many industrial, environmental and biomedical applications. Since the reactions occur at the pore scale, 3D pore scale simulations are very important. At the same time, pore scale measurements are difficult or impossible, and usually, some averaged quantities are measured. One such quantity of interest that can be measured is the breakthrough curve, i.e., the time-dependent integral of the species over the outlet. These breakthrough curves can be computed from a reaction–advection–diffusion equation on a porous medium, which is numerically solved. However, this usually leads to very high computational costs, which might be prohibitively high in a multiquery application, e.g., optimization of the geometry of the porous medium. In that case, a surrogate model for the full order simulation model (FOM) is required. A promising way of obtaining an adequate surrogate model is the use of machine learning (ML) techniques. ML techniques are nowadays widely spread and employed for a variety of tasks, including the estimation of damage for buried pipelines [1] or automatic vision-based sewer inspection [2]. ML algorithms are developed and tested for porous media flows with different complexity. The literature in this area is very rich, and we cite some exemplary papers simply to point to the place of our research. Numerous papers discuss predicting permeability from microscale geometry. For example, see [3,4,5] and the references therein. At the next level of complexity, e.g., in [6], the capability of using deep learning techniques has been shown in the case where the velocity field is predicted from the morphology of a porous medium. On the same topic, improvement is achieved by incorporating coarse velocities in the learning process, see [7]. In the last decade, the application of ML techniques for the simulation of passive and reactive transport has rapidly grown. One of the directions in this case is using ML to predict reaction rates when those are very expensive in the case of complex reactions. This can be implemented without taking into account the geometry, see e.g., [8,9,10], or predicting the rate from structure features, see, e.g., [4,11,12], to name just a few. Recently, the potential of using machine learning models as surrogates for predicting breakthrough curves for varying physical parameters, i.e., Damköhler and Peclet numbers, on a fixed porous medium geometry [13] has been reported.

In the current paper, we address the task of predicting the breakthrough curves for varying geometries of the porous medium with fixed Damköhler and Peclet numbers. Our work is in the same direction as that of [4,11], but there are essential differences. In [11], ML addresses the impact of the structural features on the effective reaction rate to overcome the limitation of the well-mixing assumption. Pore scale reactive flow in an inert skeleton is considered there. In [4], pore scale colloid transport is considered as a part of a filtration problem. Steady-state problems are solved. In our case, our pore scale geometry is in fact a two scale media, as the active washcoat particles are nanoporous. We consider the diffusion of the species within the washcoat, where the reaction occurs. Furthermore, while the other papers discuss different neural network algorithms, we consider a kernel-based method, namely, VKOGA. To the best of our knowledge, such studies have not yet been discussed in the literature.

Because of the computationally demanding simulation of the FOM, we are limited to a scarce data regime as we can only afford to compute a few samples. Furthermore, the input dimension (the number of elements of the discretization of the porous medium) and the output dimension (the number of time steps used during the FOM simulation) are very high. This yields a challenging task for machine learning techniques [14]. For this purpose, greedy kernel approximation schemes [15,16] have been shown to be efficient. These meshless approximation techniques can be combined with deep learning techniques to yield two-layered kernels [17,18], which have also already been successfully applied for certified and adaptive surrogate modeling for heat conduction [18], as well as surrogate modeling of chemical kinetics [8].

Our work is organized as follows: In Section 2, we introduce the underlying equations of the 3D porous medium reaction–advection–diffusion model. In Section 3, we give an overview on greedy kernel methods and two-layered kernels. The numerical experiments are provided in Section 4, and we conclude our work in Section 5.

2. Problem Formulation

2.1. Geometry

In the current paper, we consider artificially generated voxel-based geometries on the unit cube $\Omega ={\left[0,1\right]}^3$ as geometries for porous filter fragments. These consist of the solid skeleton, ${\Omega }_s$, free pores, ${\Omega }_f$, and effective porous space (washcoat or unresolved porosity region) ${\Omega }_w$. Accordingly, we assume that the porous geometry sample can be represented as a union of non-overlapping domains $\Omega ={\Omega }_s\cup {\Omega }_f\cup {\Omega }_w$. Each of the domains ${\Omega }_w,{\Omega }_f$ is represented as a union of non-overlapping volume voxels:

$$\begin{array}{c}\hfill V_{i,j,k}=[(i-1)h;ih)\times [(j-1)h;jh)\times [(k-1)h;kh),i,j,k=1,\dots ,N_h,h=1/N_h,\end{array}$$

where $N_h$ is the number of voxels in each dimension. This leads to a uniform grid with $N_V={150}^3$ elements (voxels). The assignment of the voxels to the three subdomains is stored within two boolean arrays (a third one is not necessary as it can be calculated from the other two). That means each porous sample is represented by a vector $\mathbf{z}\in {\{0,1\}}^{2N_V}$.

Depending on the application and the production process, the structure of the porous space in real filters could be strongly irregular and highly anisotropic. Nevertheless, for certain types of filters, the micro structure of porous media could be represented as a combination of regular shaped nanoporous granules and solid binder material [19]. We use this representation to reconstruct the porous domain ${\Omega }_f$ and the washcoat domain ${\Omega }_w$ and later, using existing experimental data on porosity, to generate a series of artificial porous geometries that could be used as models for real filter fragments. The size of each sample in voxels, $N_h=150$, is chosen according to a representative volume element (RVE) study on the one hand and the amount of computational work to generate enough data for training on the other hand. In order to generate each porous sample, we use an analytical spheres piling algorithm for the washcoat region, then voxelize analytical spheres and distribute voxelized solid material (binder) uniformly and randomly, covering the washcoat skeleton surface. The main varying parameter during the generation of the samples was the washcoat volume fraction ${ϵ}_w$; thus, in order to generate each sample, the target value of ${ϵ}_w$ was set, but depending on geometry realization, the resulting value of ${ϵ}_w$ for a generated sample could differ by more than 5%. A typical porous sample generated with this two-step procedure with porosity $ϵ=0.553$ is depicted in Figure 1.

In addition to the typical porous geometry characterization parameters, such as porosity $ϵ$, phase volume fraction (for washcoat) ${ϵ}_w$, and phase specific surface area (for free pores) $A_S$, we also compute the integral Minkowski parameters as morphological features. A complete list of parameters with their definitions for each porous sample can be found in the supplementary Appendix A. In total, 59 artificial porous samples were generated for further processing. The geometry generation time per sample varied greatly depending on the porosity, from the minimum of 6 wall-clock seconds (wcs) to the maximum of 222 wcs, with the average of 111 wcs for a system with 40 cores Intel Xeon CPU E5-2600 v2, 2.8 GHz. On the contrary, the time for evaluation of geometric parameters was almost the same for all samples and did not exceed 2 wcs.

Both the generation and parameter evaluation phases for each geometry were performed with the GeoDict software (Release 2022) [20] and Python scripts.

2.2. Governing Equations for FOM

Different approaches can be used to describe the flow and transport of chemical species through the porous medium at the pore-scale. Many of them are based on solving convection–diffusion–reaction (CDR) equations for the species transport and (Navier–) Stokes equations for the bulk flow. Thus, we consider, as the FOM solution in the current paper, the solution of the CDR equation for a species of interest and additionally assume, based on typical filtration application conditions, that (i) flow Reynolds numbers are sufficiently small $Re\ll 1$ and Stokes equations are valid for any sub-region of the pore space ${\Omega }_f$, and (ii) the concentration of the species of interest, c, is much smaller than the bulk mixture concentration. Although these are rather strict assumptions, they work well for catalytic filtration applications and are in fact the standard in this field. This justifies a one-way coupling approach when the solution of the flow equations can be decoupled from the species transport equation. We also restrict ourselves to scalar CDR equations with a linear source term. This type of source term can be used to describe a first order sorption–desorption process in porous media (Henry isotherm). According to the aforementioned assumptions, the CDR equation for species concentration, $c\ge 0$, can be written in dimensionless form as

$${\partial }_{t}c-\Delta c+{\mathrm{Pe}}_L\nabla ·\left(\mathbf{v}c\right)+{\mathrm{Da}}_{L}c=0,x\in \left({\Omega }_f\cup {\Omega }_w\right),t>0,$$

(1)

where ${\mathrm{Da}}_L,{\mathrm{Pe}}_L$ are the parameters: ${\mathrm{Da}}_L=\frac{k_{R}L}{D}$ is the piece-wise-constant Damköhler number, ${\mathrm{Pe}}_L=\frac{u_{in}L}{D}$ is the Peclet number, $L,D>0$ are the characteristic length and diffusion coefficient, respectively; $u_{in}$ is the inlet velocity, $k_R\left(x\right)\ge 0$ is the reaction rate constant and reaction occurs only in the washcoat region:

$$k_R\left(x\right)=\left\{\begin{array}{c}0,\mathbf{x}\in {\Omega }_f,\hfill \\ k_r>0,\mathbf{x}\in {\Omega }_w.\hfill \end{array}\right.$$

In Equation (1), the characteristic diffusion time was used as a time scale. Moreover, we fix the parameter values, $({\mathrm{Pe}}_L,{\mathrm{Da}}_L)=(5,0.1)$, and thus, consider only a convection-dominated regime for each geometry. For the numerical solution, Equation (1) was complemented with zero initial conditions, Dirichlet conditions for the inlet boundary section and zero Neumann conditions for the outlet boundary section and all other boundaries, see Figure 1. Based on the solution of Equation (1), a quantity of interest, the breakthrough curve, can be evaluated as an integral concentration over the outlet section of the geometry, ${\Gamma }_{\mathrm{outlet}}$:

$$a\left(t\right)=\underset{{\Gamma }_{\mathrm{outlet}}}{\int }c(x,t)\mathrm{d}\sigma .$$

(2)

For the velocity $\mathbf{v}$ within Equation (1), the stationary Stokes equations in ${\Omega }_f$ were considered:

$$\begin{array}{c}\hfill \mu \Delta \mathbf{v}=\nabla p,\\ \hfill \nabla ·\mathbf{v}=0,\end{array}$$

(3)

where $p:{\Omega }_f\to \mathbb{R}$ is the pressure and $\mu \in {\mathbb{R}}_{+}$ is the dynamic viscosity of the gas mixture. Velocity inlet–pressure outlet boundary conditions were used for system Equation (3).

Thus, at the first step of the overall solution procedure, the velocity field $\mathbf{v}$ was determined as a solution of system Equation (3). In the second step, due to (ii), using the predetermined velocity field, the concentration field c and integral Equation (2) were computed. Both steps were performed with the PoreChem software (Release 1.0 beta) [21].

3. Kernel Methods

Kernel methods [22] comprise versatile tools for scattered data approximation, revolving around the notion of a symmetric kernel $k:{\Omega }_d\times {\Omega }_d\to \mathbb{R}$. An important type of kernel is given by strictly positive definite kernels, i.e., kernels such that the so-called kernel matrix $k(X_N,X_N)={\left(k({\mathbf{x}}_i,{\mathbf{x}}_j)\right)}_{i,j=1}^N$ is positive definite for any choice of pairwise distinct points $X_N={\left\{{\mathbf{x}}_i\right\}}_{i=1}^N\subset {\Omega }_d$, $N\in \mathbb{N}$. In the context of machine learning, the set $X_N$ is often referred to as the training set $X_{\mathrm{train}}$, i.e., $X_N=X_{\mathrm{train}}.$ In the following, we focus on the popular class of radial basis function kernels on ${\Omega }_d\subseteq {\mathbb{R}}^d$, which can be expressed as $k(\mathbf{x},{\mathbf{x}}^{\prime })=\varphi (\parallel \mathbf{x}-{\mathbf{x}}^{\prime }\parallel )$ using a radial basis function $\varphi :\mathbb{R}\to \mathbb{R}$. Popular instances of such kernels are given by the following:

$$\begin{array}{cccc}\hfill k(\mathbf{x},{\mathbf{x}}^{\prime })& =e^{-{ϵ}^2\left|\right|\mathbf{x}-{\mathbf{x}}^{\prime }{\left|\right|}^2}\hfill & \hfill & \mathrm{Gaussian}\mathrm{kernel},\hfill \\ \hfill k(\mathbf{x},{\mathbf{x}}^{\prime })& =(1+ϵ||\mathbf{x}-{\mathbf{x}}^{\prime }\left|\right|)e^{-ϵ\left|\right|\mathbf{x}-{\mathbf{x}}^{\prime }\left|\right|}\hfill & \hfill & \mathrm{Matérn}1,\hfill \\ \hfill k(\mathbf{x},{\mathbf{x}}^{\prime })& =(3+3ϵ||\mathbf{x}-{\mathbf{x}}^{\prime }\left|\right|+{ϵ}^2\left|\right|\mathbf{x}-{\mathbf{x}}^{\prime }{\left|\right|}^2)e^{-ϵ\left|\right|\mathbf{x}-{\mathbf{x}}^{\prime }\left|\right|}\hfill & \hfill & \mathrm{Matérn}2,\hfill \end{array}$$

which are also later on used in Section 4. As we do not know about the regularity of the target function, we apply kernels that are infinitely smooth (Gaussian kernels) or finitely smooth (Matérn kernels) in order to cover various possible regularities of the target function. The parameter $ϵ$ is the so-called shape parameter, which allows us to tune the width of these kernels.

Given not only the input data ${\left\{{\mathbf{x}}_i\right\}}_{i=1}^N$ but also the corresponding (possibly vector-valued) target values ${\left\{{\mathbf{f}}_i\right\}}_{i=1}^N\subset {\mathbb{R}}^b$, $b\ge 1$, a well known representer theorem states that the optimal least squares kernel approximant is given by

$$\begin{array}{c}\hfill s_N\left(\mathbf{x}\right)=\sum _{i=1}^N{\alpha }_{i}k(\mathbf{x},{\mathbf{x}}_i),\end{array}$$

(4)

with coefficients ${\left\{{\alpha }_j\right\}}_{j=1}^N\subset {\mathbb{R}}^b$. These coefficients can be computed directly by solving the regularized linear equation system $(k(X_N,X_N)+\lambda I)\alpha =y$, where $\alpha \in {\mathbb{R}}^{N\times b}$ and $y\in {\mathbb{R}}^{N\times b}$ refer to a collection of the coefficients and target values in arrays. The regularization parameter $\lambda \ge 0$ allows us to steer the robustness to outliers/noise versus the approximation accuracy in the training points. The value $\lambda =0$ corresponds to kernel interpolation. For this case $\lambda =0$, the assumption on the strict positive definiteness of the kernel still ensures the solvability of the system.

Greedy kernel approximation: In order to obtain a sparse kernel model, one strives to have a smaller expansion size $n\ll N$ within Equation (4). For this, greedy algorithms can be leveraged, which select a suitable subset of centers $X_n$ from $X_N$. For this, they start with $X_0=\left\{\right\}$ and iteratively add points from $X_N$ to $X_n$ as $X_{n+1}:=X_n\cup \left\{{\mathbf{x}}_{n+1}\right\}$, according to some selection criterion. While there are several selection criteria in use in the literature, we focus on the f-greedy selection criterion, which reads

$$\begin{array}{c}\hfill {\mathbf{x}}_{n+1}:={\mathrm{argmax}}_{{\mathbf{x}}_i\in X_N\setminus X_n}|{\mathbf{f}}_i-s_n\left({\mathbf{x}}_i\right)|.\end{array}$$

This residual-based selection criterion incorporates the data point of the largest error, i.e., directly aims at minimizing the maximal absolute error of the kernel model. An efficient implementation of such greedy kernel algorithms is provided by the VKOGA package [15], and a comprehensive analysis of the convergence of such greedy kernel algorithms was provided in [16]. Because of the greedy strategy, VKOGA is often a superior method compared to other kernel-based strategies. Other methods for obtaining sparse, kernel-based models are sparse Gaussian processes [23,24], support vector regression (SVR) [25] or the reduced support vector set approach [25]. The computational efficiency of VKOGA kernel models is shown, for example, in [26], where a comparison of VKOGA with support vector machines was performed, which revealed that VKOGA results in sparser models and, hence, less computational time for training. Similarly, less computational time for prediction was observed in [13], where Gaussian processes and VKOGA were compared. In [27], a comparison of VKOGA and SVR over other ML techniques was performed in a turbulent flow prediction scenario. The study revealed that those kernel methods gave the best performance regarding the quality of the approximated discrete-time dynamics.

Two-layered kernels: In order to incorporate feature learning into kernel models, we make use of two-layered kernels [17,18]: These make use of a base kernel k, as given above, and combine it with a matrix $\mathbf{A}\in {\mathbb{R}}^{d\times d}$, such that the two-layered kernel is given by

$$\begin{array}{c}\hfill k_{\mathbf{A}}(\mathbf{x},{\mathbf{x}}^{\prime })=k(\mathbf{A}\mathbf{x},\mathbf{A}{\mathbf{x}}^{\prime })=\varphi (\parallel \mathbf{A}(\mathbf{x}-{\mathbf{x}}^{\prime })\parallel ).\end{array}$$

(5)

With this, the two-layered kernel can be optimized to the given data $({\left\{{\mathbf{x}}_i\right\}}_{i=1}^N,{\left\{{\mathbf{f}}_i\right\}}_{i=1}^N)$ by optimizing the first layer matrix $\mathbf{A}\in {\mathbb{R}}^{d\times d}$. Thus, this two-layered structure may considerably improve the performance of the kernel model, especially for medium- to high-dimensional input data, where an equal importance of all features is highly unlikely. The strength of two-layered kernel models and their superior performance over shallow kernel models has been observed, for example, when applied to heat conduction [18] or when used as a surrogate model for chemical kinetics [8]. We investigate whether this behavior can also be observed for our current problem. For the optimization of the matrix $\mathbf{A}$, we employ the fast-gradient-descent-based mini-batch optimization proposed in [17] and extended it to vector valued target values $b>1$ in [18]. The overall idea is to leverage an efficiently computable leave-one-out cross validation (LOOCV) error loss, which, thus, makes the kernel generalize well to the unseen data. In particular, this cross validation error loss makes use of both input and target values, i.e., it is a supervised optimization.

The matrix $\mathbf{A}$ makes the two-layered kernel $k_{\mathbf{A}}$ even more interpretable: The large singular values with corresponding right singular vectors within the singular value decomposition of the matrix $\mathbf{A}$ correspond to the more important features within the data set, while the smaller singular values with the corresponding right singular vectors correspond to the less important features within the data set.

In the following Section 4, we make use of the notion “single layered kernel” to refer to the standard kernel k, while we make use of the notion “two-layered kernel” to refer to (optimized) kernels $k_{\mathbf{A}}$.

4. Numerical Experiments

In this section, we present the numerical experiments. The goal is to approximate breakthrough curves Equation (2) from voxel data, which characterizes the geometry of the porous medium. As explained in Section 2, the voxel data are described by a vector $\mathbf{z}\in {\{0,1\}}^{2N_V}$. In our numerical experiment, we choose $N_V={150}^3$. We discretize a breakthrough curve $a\left(t\right)$ on an equidistant temporal grid $t_i$, $i=1,\dots ,n_t$, as the vector $\mathbf{a}:={\left(a\left(t_i\right)\right)}_{i=1}^{n_t}\in {\mathbb{R}}^{n_t}$ with $n_t=500$ time steps.

We compare two different kernel-based models for learning the breakthrough curves (see Figure 2), in which either morphological features (MF) prescribed by expert knowledge (see Appendix A) are extracted or principal component analysis (PCA) features are learned from the data using a PCA. The kernel function k is chosen to be either a shallow one-layered or a two-layered kernel. We refer to these models in the scope of our paper as MF-1L-kernel and MF-2L-kernel or PCA-1L-kernel and PCA-2L-kernel, depending on the depth of the kernel (one-layered/two-layered) and whether we extract morphological features or PCA features.

Both types of models (MF) and (PCA) are based on a feature map $\Phi :{\mathbb{R}}^{2N_V}\to {\mathbb{R}}^{n_{\mathrm{f}}}$ with $n_{\mathrm{f}}$ features and a kernel function $k:{\mathbb{R}}^d\times {\mathbb{R}}^d\to \mathbb{R}$ with $d=n_{\mathrm{f}}$, such that the resulting kernel models using the single-layer or the two-layered kernels are given as

$$\begin{array}{c}\hfill s_n\left(\mathbf{z}\right)=\sum _{i=1}^n{\alpha }_{i}k(\Phi \left(\mathbf{z}\right),\Phi \left({\mathbf{z}}_i\right)),\mathrm{respectively},s_n\left(\mathbf{z}\right)=\sum _{i=1}^n{\alpha }_{i}k(\mathbf{A}\Phi \left(\mathbf{z}\right),\mathbf{A}\Phi \left({\mathbf{z}}_i\right)),\end{array}$$

(6)

with coefficients ${\alpha }_i\in {\mathbb{R}}^b$, $b=n_t$ and centers ${\mathbf{z}}_i\in {\{0,1\}}^{2N_V}$ for $i=1,\dots ,n$. The expansion size n is fixed to $n=10.$ Choosing smaller values for n would worsen the results presented in the next subsections considerably, while increasing n would only influence the approximation quality on the test set slightly.

All of these models involve hyper-parameters, which are listed in

Table 1. Hyper -parameters of the kernel function k, shape parameter $ϵ$ and regularization parameter $\lambda$ used during LOOCV of the MF-based models and PCA-based models.

Hyper-Parameter	Values
kernel fun.	Matérn 1, Matérn 2, Gaussian kernel
Shape parameters	1, 1/2, 1/4, 1/8, 1/16, 1/32
Regularization parameters	0, ${10}^{-2},{10}^{-3},{10}^{-4},{10}^{-5},{10}^{-6}$

. Suitable values for the hyper-parameters are determined via an LOOCV on the training data set. Note that in the cases where two-layered kernels are applied, no LOOCV for the shape parameter is performed, and instead, the matrix $\mathbf{A}$ is optimized using the procedure described in the previous section. The kernel function and the regularization parameter are determined via LOOCV in both cases.

In the following, we discuss the generation of the training and test data (Section 4.1), the training and results of the (MF) models based on morphological features (Section 4.2) and of the (PCA) models based on PCA features (Section 4.3).

4.1. Training and Test Data

In total, we consider $n_{\mathrm{s}}=59$ samples $X:={\left\{{\mathbf{z}}_i\right\}}_{i=1}^{n_{\mathrm{s}}}$ of voxel data ${\mathbf{z}}_i$ and the corresponding time-discretized breakthrough curves ${\mathbf{a}}_i$ (see Figure 3). The breakthrough curves ${\mathbf{a}}_i$ are obtained by solving Equations (1) and (3) based on the geometry described by the corresponding voxel data ${\mathbf{z}}_i$. The average computational cost of solving Equations (1) and (3) for one sample is 7514 wcs with a standard deviation of 286 wcs on a workstation with two Intel Xeon Gold 6240R (48 cores in total).

We use an approximate 80–$20\%$ training–test split, which results in $n_{\mathrm{s},\mathrm{train}}=N=47$ training samples $X_{\mathrm{train}}=X_N$ and $n_{\mathrm{s},\mathrm{test}}=12$ test samples $X_{\mathrm{test}}$ with $X=X_{\mathrm{train}}\cup X_{\mathrm{test}}$ and $X_{\mathrm{train}}\cap X_{\mathrm{test}}=\varnothing$. To demonstrate robustness with respect to the choice of the training–test split, we consider three different random splits, where we use the same three random splits to measure the performance of each model. As an error measure, we use a relative error on the test set

$$e_{\mathrm{rel}}=\frac{1}{|X_{\mathrm{test}}|}\sqrt{\sum _{{\mathbf{z}}_i\in X_{\mathrm{test}}}\frac{\left|\right|{\mathbf{a}}_i-s_N\left({\mathbf{z}}_i\right){\left|\right|}^2}{\left|\right|{\mathbf{a}}_i{\left|\right|}^2}}.$$

(7)

4.2. Kernel Models on Morphological Features

Because of the high dimensionality of the input space, it is necessary to apply some reduction technique before training the machine learning model. In this section, we extract $n_f=6$ morphological geometry features that describe the porous medium. These are the porosity, the washcoat volume fraction, the volume of free pores, the surface area for the free pores, the integral of mean curvature of the free pores and the integral of total curvature of the free pores. For further information on how they are computed, see supplementary Appendix A. Thus, the machine learning model can be represented by setting $\Phi ={\Phi }_{\mathrm{MF}}$ in Equation (6) ${\Phi }_{\mathrm{MF}}:{\mathbb{R}}^{2N_V}\to {\mathbb{R}}^{n_f}$ the mapping that computes the morphological features from a given geometry.

For the first experiment, we use shallow kernels and present the approximated breakthrough curves in Figure 4. We observe that except for the red outlier curve in the bottom diagram, all curves are well approximated. This corresponds to the relative test error presented in

Table 2. MF-1L-kernel: Relative error (7) and the selected hyper-parameters from Table 1 for three different training–test splits (0–2).

Split	Rel. Err.	Kernel Fun.	Shape Parameter	Reg. Par.
0	1.49 $·{10}^{-3}$	Matérn 1	1.00	1.00$·{10}^{-2}$
1	1.60$·{10}^{-4}$	Matérn 1	1.00	1.00$·{10}^{-2}$
2	4.23$·{10}^{-3}$	Matérn 1	2.50$·{10}^{-1}$	1.00$·{10}^{-4}$
average	1.51$·{10}^{-3}$

, where we see that for the first two data splits, a relative error of approximately $0.01\%$ can be achieved; whereas, for the third split, we only achieve an error of approximately $0.42\%$. We further observe from Table 2 that for the first two data splits, exactly the same hyper-parameters (kernel function, shape parameter and regularization parameter) are chosen. However, they are chosen differently for the third data set, which may be due to the red outlier curve being part of the test set for the third data split.

For the next experiment, we included two-layer optimization of the kernel model and present the approximated breakthrough curves in Figure 5. We observe that a similar approximation quality can be achieved using two-layered kernels. All breakthrough curves except for the red outlier curve in the bottom diagram are well approximated. The relative test error (see

Table 3. MF-2L-kernel: Relative error (7) and the selected hyper-parameters from Table 1 for three different training–test splits (0–2).

Split	Rel. Err.	Kernel Fun.	A	Reg. Par.
0	1.54$·{10}^{-4}$	Matérn 2		1.00$·{10}^{-3}$
1	2.37$·{10}^{-4}$	Matérn 2		1.00$·{10}^{-3}$
2	3.99$·{10}^{-3}$	Matérn 1		1.00$·{10}^{-4}$
average	1.46$·{10}^{-3}$

) is slightly improved for the third data split. However, for the first and second split, it becomes slightly worse. We observe that similar to the experiment with shallow kernel models, for each data split, the Matérn 1 kernel is selected and that for the third split, a smaller regularization parameter is selected than for the first and second one. To compare the shape transformations by the first layer, we present the first layer matrix $\mathbf{A}$ in Table 3. For an easier comparison, the matrix $\mathbf{A}$ is visualized using the color-map at the bottom of the table. We observe that the matrices look very similar for each data split.

4.3. Kernel Models on PCA Features

Next, we use the PCA to define a feature map $\Phi$. In order to be comparable to the previous experiment, we choose the number of features as $n_{\mathrm{f}}=6$. Following the idea of the PCA [28], we choose the PCA feature map ${\Phi }_{\mathrm{PCA}}\left(\mathbf{z}\right)={\mathbf{U}}_{n_{\mathrm{f}}}^{\mathsf{T}}\mathbf{z}$ based on the first $n_{\mathrm{f}}$ left-singular vectors ${\mathbf{U}}_{n_{\mathrm{f}}}\in {\mathbb{R}}^{2N_V\times n_{\mathrm{f}}}$ of the matrix $\mathbf{Z}:={\left(\mathbf{z}\right)}_{\mathbf{z}\in X_{\mathrm{train}}}\in {\mathbb{R}}^{2N_V\times n_{\mathrm{s},\mathrm{train}}}$ via the SVD

$$\begin{array}{cccccc}\hfill & \mathbf{Z}=\mathbf{U}\Sigma {\mathbf{V}}^{\mathsf{T}}\hfill & \hfill & \mathrm{with}\hfill & \hfill & {\mathbf{U}}_{n_{\mathrm{f}}}:=\mathbf{U}(:,:n_{\mathrm{f}})\hfill \end{array}$$

(8)

and set $\Phi ={\Phi }_{\mathrm{PCA}}$ in Equation (6) (technical note: The voxel data are saved as a boolean array $\mathbf{z}\in {\{0,1\}}^{2N_V}$. To compute the SVD, we convert these data to a floating point number, which is why $\mathbf{Z}\in {\mathbb{R}}^{2N_V\times n_{\mathrm{s},\mathrm{train}}}$).

In Figure 6, we present the prediction of the breakthrough curves for the PCA-1L-kernel model on the test set $X_{\mathrm{test}}$ for the three different randomized training–test splits. We observe that many of the breakthrough curves are well approximated and a mean relative error of approximately 0.5–1.3% (see

Table 4. PCA-1L-kernel: Relative error (7) and the selected hyper-parameters from Table 1 for three different training–test splits (0–2).

Split	Rel. Err.	Kernel Fun.	Shape Parameter	Reg. Par.
0	9.62$·{10}^{-3}$	Matérn 1	6.25$·{10}^{-2}$	1.00$·{10}^{-3}$
1	4.77$·{10}^{-3}$	Gaussian kernel	6.25$·{10}^{-2}$	1.00$·{10}^{-4}$
2	1.34$·{10}^{-2}$	Matérn 2	2.5$·{10}^{-1}$	1.00$·{10}^{-2}$
average	9.26$·{10}^{-3}$

) can be achieved. However, some breakthrough curves are badly approximated, especially the two blue curves for the first data split. We observe that three different kernels, three different shape parameters and three different regularization parameters are chosen for each data split. We further observe from Section 4.3 that $n_{\mathrm{f}}=6$ is also a suitable choice in the sense that a small test error is achieved for that setting compared to higher and smaller values of $n_{\mathrm{f}}$.

For the next experiment, we consider the PCA-2L-kernel model. The results for the analogous experiments to the previous paragraph are presented in Figure 7. We chose again $n_{\mathrm{f}}=6$ to be consistent with the previous experiments. However, as we observe in Section 4.3, for $n_{\mathrm{f}}=3$, slightly better results could have been achieved.

We observe that the results from the shallow kernel models are considerably improved by using two-layered kernels. The relative errors from

Table 5. PCA-2L-kernel: Relative error (7) and the selected hyper-parameters from Table 1 for three different training–test splits (0–2).

Split	Rel. Err.	Kernel Fun.	A	Reg. Par.
0	7.94$·{10}^{-4}$	Matérn 1		1.00$·{10}^{-3}$
1	1.10$·{10}^{-3}$	Matérn 2		1.00$·{10}^{-2}$
2	7.05$·{10}^{-3}$	Matérn 1		1.00$·{10}^{-4}$
average	2.98$·{10}^{-3}$

can be reduced to 0.08–0.7%, and we do not observe badly approximated curves for the first data split anymore. Compared to the the MF-2L-kernel experiments from the previous section, we also observe that the approximation qualities are quite similar. This means that the PCA features transformed by the first layer of the two-layered kernel are able to describe the geometry almost as well as the morphological features. This shows the strength of two-layered kernels, as the PCA feature extraction is purely data-based, and no expert knowledge had to be used.

We further observe from Table 5 that, again, similar hyper-parameters are selected for each data split. For example, the same kernel (Matérn 1) was selected for the first and third data split. Moreover, we see similarities in the first layer matrices $\mathbf{A}$ of the kernels. For all three matrices, the entries in the left upper $3\times 3$ block are considerably larger than the other entries, which means that the first three modes are the important ones. Compared to Table 3, there are larger differences between the first layers. This can be explained by the fact that in contrast to the MF-feature map, the data-based PCA-feature map is different for all three data splits. This is due to the linear mappings defined by the PCA modes being different in each split.

Lastly, we compare the mean errors averaged over the three different training–test splits in

Table 6. Mean relative test errors, averaged over the three training–test splits.

	1L	2L
MF	1.51$·{10}^{-3}$	1.46$·{10}^{-3}$
PCA	9.26$·{10}^{-3}$	2.98$·{10}^{-3}$

. We observe that the models based on morphological feature extraction work very well in combination with both one-layered and two-layered kernels. In contrast, combining one-layered kernels with PCA feature extraction yields an average error that is almost one order of magnitude larger. This error is considerably improved by applying two-layered kernels, and almost the same accuracy as with the morphological feature extraction is achieved.

4.4. Runtime Discussion

In this subsection, we compare the runtimes of the different methods in

Table 7. Runtime comparison of the machine learning methods in wcs averaged over the three training–test splits.

Model	Feat. Map	LOOCV Time	Final Model Train Time	Eval. Time
FOM	-	-	-	9.02$·{10}^4$
MF extraction	9.40$·{10}^1$	-	-	2.40$·{10}^1$
MF-1L-kernel	-	5.90$·{10}^0$	9.95$·{10}^{-4}$	2.91$·{10}^{-4}$
MF-2L-kernel	-	2.01$·{10}^2$	2.12$·{10}^{-1}$	2.67$·{10}^{-4}$
PCA feat. map comp.	1.28$·{10}^1$	-	-	-
PCA-1L-kernel	-	2.29$·{10}^1$	1.36$·{10}^{-3}$	1.83$·{10}^{-1}$
PCA-2L-kernel	-	9.18$·{10}^3$	8.63$·{10}^{-1}$	2.13$·{10}^{-1}$

. We compare the time needed for the computation of the feature map added to the feature extraction time on the test set (first column), the time needed for LOOCV, the training time of the final model and the evaluation time on the test set. Note that the full order model solves and the morphological feature extraction are performed on a different machine (workstation with two Intel Xeon Gold 6240R (48 cores in total)) than the PCA feature extraction and kernel model training (performed on a computer with 64 GB RAM and a 13th Gen Intel i7-13700K processor) due to licensing limitations. We observe that the LOOCV for the two-layered models is way more expensive than for the shallow models, which is due to the optimization of the first layer. However, it is still considerably lower (approximately 30 times for the MF-2L-kernel and 7 times for the PCA-2L-kernel) than the generation of a single training sample (7514 wcs). Furthermore, the LOOCV time and final model training of the MF-kernel models is less expensive than the training on the morphological features. However, the computation of the PCA feature map is less expensive than the computation of the morphological features. Moreover, the evaluation time of the PCA-kernel models is larger than the evaluation of the MF kernel models. Since the computation of the morphological features i.e., the evaluation of ${\Phi }_{\mathrm{MF}}$ takes approximately 2 wcs per sample, the MF approaches have a larger overall evaluation time (summing evaluation time of MF extraction and MF-1L-kernel/MF-2L-kernel). Compared to the FOM, this results in a speed-up of almost 6 orders of magnitude for the PCA kernel models and 3 orders of magnitude for the MF kernel models. Nevertheless, in both cases, the evaluation times are considerably lower than the computation of the training samples.

5. Conclusions and Outlook

In this work, we demonstrated how breakthrough curves can be predicted from the geometry of a 3D porous medium. We presented two approaches on how to treat the high dimension of the input data: For the first approach, we computed morphological features of the geometries and learned a mapping from these morphological features to the breakthrough curves. For the second approach, we computed PCA features of the geometry data set and learned a mapping from these PCA features to the breakthrough curve.

We observed that the MF approach worked well in combination with one-layered kernels and that both approaches worked well in combination with two-layered kernels. This is compelling, as the morphological features used in the study are well-known informative and predictive quantities for porous media, and in contrast, the PCA feature extraction approach is purely data-based. Thus, the results indicate the strength of using two-layered kernels, as they automatically adapt to the relevant features. We further observed that the scarce-data situation did not prevent these approaches from easily predicting the high-dimensional outputs.

The future work will focus on how ideas from convolutional neural networks can be combined with the framework of deep multi-layered kernels. Moreover, we will investigate whether data augmentation (rotating/reflecting geometry samples without changing the breakthrough curves) can further improve the PCA-feature approach while avoiding the computation of expensive FOM solutions.