A 3D Geological Modeling Method Using the Transformer Model: A Solution for Sparse Borehole Data

Abstract

Three-dimensional (3D) geological models are essential for geological analysis and mineral resource estimation. Although conventional on-site survey methods, such as boreholes, provide local engineering geological information for 3D geological modeling, accurately predicting strata in areas with sparse borehole data remains a challenge. This study proposes a 3D geological modeling method using the Transformer model under the conditions of sparse borehole data. First, a K-dimensional tree was used to identify boreholes adjacent to the target point, and a borehole context sequence was constructed using stratigraphic information from neighboring boreholes. Subsequently, the relationship between the target point and its adjacent borehole sequence was calculated using the multi-head attention mechanism of the Transformer model. Finally, trained Transformer encoders were used to predict the stratigraphic category of the target point, and the normalized information entropy was used to quantify uncertainty during the modeling process. Experimental results showed that the accuracy of the method was 0.86, outperforming the accuracy and uncertainty of a recurrent neural network. The root mean square error is smaller than the inverse distance weight and Kriging. Compared to other methods, the proposed method can more accurately describe the geometric shape and distribution of geological bodies and reveal the sedimentary laws of the study area.

1. Introduction

In engineering geology, 3D geological models constructed using 3D geological modeling technologies are very helpful for geological analysis and the quantitative estimation of mineral resources [1,2]. Boreholes provide accurate, direct, and detailed information on stratigraphic distribution, which is commonly used in 3D geological modeling [3,4]. However, drilling techniques are typically limited by budget and terrain constraints [5,6]. Although various 3D geological modeling methods have been developed over the past three decades, performing 3D geological modeling with sparse borehole data remains a major challenge [7,8].

The existing 3D geological modeling methods can be divided into deterministic modeling methods and stochastic modeling methods (

Table 1. Comparison of advantages and disadvantages of 3D geological modeling methods.

Modeling Methods		Advantages	Disadvantages
Deterministic methods	Explicit method	It is convenient for geologists to participate directly and maximize the use of geological knowledge, and the results are controllable.	Time-consuming, laborious, and subjective.
	Implicit method	High modeling efficiency.	This method cannot best use available data or contain sufficient geological constraints.
Stochastic methods	Coupled Markov chain, Markov random field	This method can generate multiple possible geological models, reflecting the uncertainty of geology.	Using this method it is difficult to determine the transition probability matrix; the reliability depends on experience.
	Gaussian simulations	Smooth model.	The ability to represent complex geological structures is limited.
	Multi-point statistics	This method can capture complex geological models; the generated model is highly consistent with the training image.	At least one training image is needed to represent geological knowledge; the tuning of MPS parameters is also needed.
	Machine learning and deep learning	It has strong objectivity, a strong modeling ability for complex nonlinear geological structures, and high modeling efficiency.	High computing power requirements.

). On the one hand, deterministic modeling methods can be divided into explicit modeling and implicit modeling. Explicit modeling allows geologists to directly contribute and fully utilize geological knowledge. However, explicit modeling requires extensive interactive modifications and is labor intensive [9,10]. Implicit modeling can automate the modeling process [11]. Implicit modeling may not optimally use available data or include sufficient geological constraints to produce plausible models with consistent geological relevance [12]. For cases with sparse data, both explicit and implicit modeling rely on interpolation techniques as solutions. Common interpolation methods include discrete smooth interpolation [13], inverse distance weighting (IDW) [14], spline interpolation [15], and Kriging [16,17]. The spline and discrete smooth interpolation methods assume data smoothness, are subjective in parameter selection, and share the limitation of being isotropic interpolation techniques. In IDW, a distance attenuation parameter is uniformly applied across the study area without considering the spatial variability of the data [18]. As a typical geostatistical interpolation method, Kriging not only captures the spatial distance relationships between sample and interpolation points but also reflects the spatial variability of the data by creating a variogram. However, this method has high requirements for the distribution of data assumptions and depends on expert-driven parameter selection, which is subjective and limited [19,20]. In practical applications, borehole data are typically densely sampled along the borehole line and do not necessarily follow spherical, cubic, or exponential distribution models [21,22]. On the other hand, the stochastic modeling method can generate multiple numerical geographic domain models representing the inherent uncertainties of these geographic domains, which can better describe the spatial random distribution and uncertainty characteristics of geological blocks and geological region boundaries [23]. Stochastic modeling methods mainly include coupled Markov chain (CMC), stochastic Markov random field (MRF), Gaussian simulation, and multi-point statistics (MPS). With CMC and MRF, it is difficult to determine the probability of the transition matrix, which has a certain subjectivity [24,25,26]. Gaussian simulation has a limited ability to represent complex geological features. MPS uses training images to simulate correlations between multiple points, making it suitable for describing complex geological phenomena. However, MPS faces challenges when it is difficult to obtain complete and reliable 3D training images [27,28,29].

Three-dimensional geological modeling based on deep learning has recently emerged as an area of active research. Studies on deep neural networks [30,31], convolutional neural networks (CNNs) [32,33], recurrent neural networks (RNNs) [19], graph neural networks (GNNs) [34,35], and generative adversarial networks (GANs) [8] have demonstrated the effectiveness of various deep learning algorithms in 3D geological modeling. In neural networks, it is not necessary to calculate the experimental variogram, and nonlinear features can be captured globally by learning the basic interrelationships within the data set. These advantages make deep learning a practical alternative for dealing with sparse data issues [36,37]. Many studies have explored deep learning methods to deal with data sparsity. Guo et al. [31] proposed a semi-supervised learning method based on pseudo-labels to generate pseudo-labels for unlabeled data, thereby supplementing high-confidence predictions for further training. Lyu et al. [8] developed a multi-scale GAN method for developing 3D underground geological models from limited borehole data and 3D training images that represent prior geological knowledge. The Transformer model, developed by Google researchers in 2017 [38], has been recognized as the fourth largest category of deep learning models after multilayer perceptron, CNNs, and RNNs [39]. Initially designed for machine translation, it is the foundation for various models such as BERT and generative pre-trained transformer (GPT), showing excellent performance in image classification and natural language processing [40,41]. The Transformer model uses a unique self-attention mechanism to simulate long-term dependencies and global relationships in the data, dynamically weighing the importance of different parts of the input, which is particularly beneficial for tasks that need to understand the global context [38,39]. Previous studies have demonstrated the effectiveness of the Transformer model in solving sparse data problems. Based on this model, Feng et al. [42] effectively extracted internal correlations between multiple traffic parameters with sparse data, while Tiwari et al. [43] used the Transformer model to quickly estimate diffusion tensor parameters from sparse measurements.

This study proposes a method for 3D geological modeling using the Transformer model under the conditions of sparse borehole data. First, the borehole data were preprocessed, and the K-dimensional tree (KD-tree) was used to identify boreholes adjacent to the target point. The borehole context sequence was then constructed using stratigraphic information from adjacent boreholes. Subsequently, feature relationships between the target point and its adjacent borehole sequence were calculated using the unique multi-head attention mechanism of the Transformer model. Finally, Transformer encoders were used to predict the stratigraphic category at the target point, and normalized information entropy was used to quantitatively evaluate the uncertainty during the modeling process. The remainder of this manuscript is organized as follows: Section 2 describes details of the proposed method; Section 3 introduces the data preparation and experimental results; Section 4 discusses the results; Section 5 presents the conclusions.

2. Materials and Methods

Compared to image data, borehole data exhibit clustering characteristics, with local concentration and overall dispersion. In this study, a method of 3D geological modeling using the Transformer model under conditions of sparse borehole data is proposed. The algorithm flowchart is shown in Figure 1. This method predicts the stratigraphic classification of the target point by performing data preprocessing, building KD-tree, constructing borehole context sequences, and training the Transformer model. Among them, KD-tree is used to efficiently organize borehole data, borehole context sequences are used to provide features of adjacent boreholes, and the Transformer model is used to identify complex relationships between borehole context sequences.

2.1. Data Preprocessing

The borehole data include the borehole identifier (ID), location coordinates (X and Y), stratum category, and the start and end elevations of each stratum. The preprocessing of borehole data was conducted as follows:

(1)

Data cleaning: This step checked for errors in each borehole’s data. For example, if the thickness of a stratum was zero, the stratum was removed. Similarly, if the start elevation of a stratum was less than the end elevation of the stratum, it was considered an error, and the stratum was removed.

(2)

Data normalization: Considering the large difference in the number of bits between the X and Y coordinates and the elevation, the coordinates were normalized to the range [0, 1]. The normalization formula is shown in Equation (1):

$${\mathrm{I}}_0={\displaystyle \frac{{\mathrm{I}}_{\mathrm{i}}-{\mathrm{I}}_{\mathrm{m}\mathrm{i}\mathrm{n}}}{{\mathrm{I}}_{\mathrm{m}\mathrm{a}\mathrm{x}}-{\mathrm{I}}_{\mathrm{m}\mathrm{i}\mathrm{n}}}}$$

(1)

where I₀ is the normalized value, I_i is the original value, and I_max and I_min are the maximum and minimum values of the coordinates, respectively.

(3)

Data encoding: As the stratum name was a character string, a mapping between the string and the integer was created, and the string was encoded as an integer label for processing by the classifier. To prevent data leakage, the preprocessed borehole data were divided into three parts according to the borehole ID, in a specific proportion, to construct the training set, validation set, and test set.

2.2. Construction of KD-Tree

To facilitate the search for boreholes adjacent to the target point, a KD-tree was constructed based on the X and Y coordinates of each borehole. The KD-tree is a data structure widely used in computer science and computational geometry. It divides the data space based on the number of dimensions of each data point, enabling efficient organization and search of points in the multidimensional data space [44,45,46]. The KD-tree constructed in this study had two dimensions, these being the X and Y coordinates of the boreholes. During the construction, each node represented a data point, and the data set was recursively divided into subsets according to the dimensions. Equation (2) represents the dimension selection when the dimension k of the KD-tree is 2.

$$s\mathrm{p}\mathrm{l}\mathrm{i}\mathrm{t}\_\mathrm{d}\mathrm{i}\mathrm{m}(\mathrm{d})=\left\{\begin{array}{c}\mathrm{x}, \mathrm{i}\mathrm{f} \mathrm{d} \mathrm{m}\mathrm{o}\mathrm{d} 2=0\\ \mathrm{y}, \mathrm{i}\mathrm{f} \mathrm{d} \mathrm{m}\mathrm{o}\mathrm{d} 2=1\end{array}\right.$$

(2)

where d is the depth of KD-tree, split_dim(d) represents the dimension of segmentation, and d mod 2 represents the remainder of d divided by 2. When split_dim(d) = x, it represents that the left and right subtrees of the depth are divided according to the X coordinate of the borehole. When split_dim(d) = y, it represents that the left and right subtrees of the depth are divided according to the Y coordinate of the borehole.

The data on dimension split_dim(d) are arranged from small to large to obtain the data set {$A_1$, $A_2$, $A_3$,…, $A_n$}, where $A_1$ < $A_2$ < $A_3$ <…< $A_n$ and n is the number of data points on dimension split_dim(d). According to Equation (3), the median of the dimension split_dim(d) can be calculated.

$$\mathrm{m}=\left\{\begin{array}{c}{\mathrm{A}}_{{\displaystyle \frac{\mathrm{n}+1}{2}}} ,n=2l+1\\ {\displaystyle \frac{{\mathrm{A}}_{\frac{\mathrm{n}}{2}}+{\mathrm{A}}_{\frac{\mathrm{n}}{2}+1}}{2}},\mathrm{n}=2l\end{array}\right.$$

(3)

where m represents the median in the current dimension, n = 2l + 1 represents n being odd, and n = 2l represents n being even.

For each partition, the median of the current dimension was selected as the partition point, dividing the data set into two parts: the left subtree contained all coordinates smaller than the point, while the right subtree contained all coordinates greater than the point. Equation (4) is a formula for dividing subtrees.

$$\left\{\begin{array}{c}{\mathrm{S}}_{\mathrm{l}\mathrm{e}\mathrm{f}\mathrm{t}}=\{\mathrm{s}\mathrm{p}\mathrm{l}\mathrm{i}\mathrm{t}\_\mathrm{d}\mathrm{i}\mathrm{m}(\mathrm{d})<\mathrm{m}\}\\ {\mathrm{S}}_{\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}}=\{\mathrm{s}\mathrm{p}\mathrm{l}\mathrm{i}\mathrm{t}\_\mathrm{d}\mathrm{i}\mathrm{m}(\mathrm{d})>\mathrm{m}\}\end{array}\right.$$

(4)

where $S_{left}$ is a left subtree divided by m and $S_{right}$ is a right subtree divided by m.

For each subset, the next dimension was recursively selected, and the division continued based on the median until each leaf node contained only one data point. Through this recursive space division, the KD-tree could effectively organize data, enabling nearest neighbor queries to be completed in logarithmic time.

2.3. Construction of Borehole Context Sequence

To fully exploit the stratigraphic relationship between adjacent boreholes, this study constructed multiple context sequences for each borehole. A given elevation in a borehole was treated as the target point, and the KD-tree was used to query the nearest neighboring borehole ID to the target point. Each nearest neighbor borehole ID was then traversed to extract the corresponding stratigraphic records, and features (including stratum ID, start and end elevations of the stratum, and X and Y coordinates) were added to the sequence list. The spatial information of the target point was subsequently added to the end of the sequence, and the stratum category of the target point was used as the prediction target for the Transformer model. For example, when the number of adjacent boreholes was equal to three (K = 3), the 1st stratum (Figure 2a), the 2nd stratum (Figure 2b), the 3rd stratum (Figure 2c), and the 4th stratum (Figure 2d) of the training borehole T were treated as the target points to construct the borehole context sequence. The borehole context sequence for borehole T, with the 1st stratum as the target point, is shown in Figure 3. This sequence contains the stratigraphic features of adjacent boreholes A, B, and C, as well as the spatial characteristics of the target point. After constructing all the borehole context sequences, the maximum length of each sequence was calculated to determine the filling length, and each sequence was then filled to the maximum length to ensure uniformity across all sequences, facilitating batch training.

2.4. The Training of the Transformer Model

The original Transformer model [38] design includes both encoders and decoders. In common generation tasks, such as machine translation, the input is a sentence, and the output is a translated sentence. In this case, the decoder is used to gradually generate the output sequence, relying on the context information produced by the encoder. In our study, the stratum classification task was a classification problem. The task was to classify the stratum based on the adjacent borehole context information of the target point, rather than generating a new sequence. Therefore, the model only needed to use the Transformer encoders (as shown in Figure 4) to encode the input data, extract useful features, and output the probability of the target point belonging to each category.

To extract the characteristic relationships between the target point and its adjacent borehole stratigraphic sequence, it was necessary to calculate the self-attention mechanism. The calculation of self-attention can be divided into the following steps. First, each point in the input sequence was mapped to three different representations with dimensions through the input embedding layer, which were called query (Q), key (K), and value (V). Subsequently, the Q vector was matched with all K vectors using dot product multiplication. The resulting matrix was scaled and passed through the softmax function to obtain the attention scores between different points in the sequence. Finally, the attention score was multiplied by the V vector to generate a new representation of the input sequence. Essentially, the self-attention mechanism modified the representation of each point by considering the weighted contribution of all other points in the sequence. This allowed distant points in the sequence to focus on each other’s values and share important information that may otherwise be overlooked. The self-attention mechanism is represented by Equation (5) [38]:

$$\mathrm{Attention}\left(\mathrm{Q},\mathrm{K},\mathrm{V}\right)=\mathrm{soft}\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{\mathrm{Q}{\mathrm{K}}^{\mathrm{T}}}{\sqrt{{\mathrm{d}}_{\mathrm{k}}}}}\right)\mathrm{V}$$

(5)

where Q, K, and V are the three matrices used for calculating self-attention, and d_k is the dimension of the K matrix.

The multi-head attention mechanism allowed the model to focus on the self-attention mechanism of different subspaces. Under the condition that the parameter quantity was generally unchanged, multi-head attention divided the three parameters, Q, K, and V, into multiple groups. Each group was mapped to different subspaces in the high-dimensional space to calculate the self-attention weights, enabling the model to focus on different parts of the input. After performing multiple parallel calculations, the self-attention information in all subspaces was merged. As self-attention was distributed differently across subspaces, multi-head attention was seeking for associations from different perspectives of the input data, making it possible to encode multiple relationships and subtle differences. The multi-head attention formula [38] is shown in Equation (6):

$$\left\{\begin{array}{c}\mathrm{M}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}\mathrm{H}\mathrm{e}\mathrm{a}\mathrm{d}(\mathrm{Q},\mathrm{K},\mathrm{V})=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}\left({ \mathrm{head} }_1,{ \mathrm{head} }_2,\cdots ,{ \mathrm{head} }_{\mathrm{h}}\right){\mathrm{W}}^{\mathrm{o}}\\ { \mathrm{head} }_{\mathrm{i}}=\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left(\mathrm{Q}{\mathrm{W}}_{\mathrm{i}}^{\mathrm{Q}},\mathrm{K}{\mathrm{W}}_{\mathrm{i}}^{\mathrm{K}},\mathrm{V}{\mathrm{W}}_{\mathrm{i}}^{\mathrm{V}}\right)\end{array}\right.$$

(6)

Taking one of the heads in the multi-head self-attention mechanism as an example, the process of calculating self-attention is shown in Figure 5. The Transformer model iterates through each position in the borehole context sequence, computing self-attention from the 1st, 2nd, 3rd… to the N-th position. As there may be long-distance dependencies between different stratigraphic categories, the multi-head self-attention mechanism can help the model identify complex associations across multiple strata.

As the self-attention mechanism is independent of the order in the input sequence, position encoding was added to track the location of different points. Position encoding was typically introduced using sine and cosine functions, as shown in Equation (7) [38]:

$$\left\{\begin{array}{c}\mathrm{P}{\mathrm{E}}_{\left(\mathrm{pos} ,2\mathrm{i}\right)}=\mathrm{s}\mathrm{i}\mathrm{n}\left(\mathrm{p}\mathrm{o}\mathrm{s}/{10000}^{2\mathrm{i}/\mathrm{d}}\right)\\ \mathrm{P}{\mathrm{E}}_{\left(\mathrm{pos} 2\mathrm{i}+1\right)}=\mathrm{c}\mathrm{o}\mathrm{s}\left(\mathrm{p}\mathrm{o}\mathrm{s}/{10000}^{2\mathrm{i}/\mathrm{d}}\right)\end{array}\right.$$

(7)

where pos denotes the position in the sequence, d represents the dimension of the position encoding, 2i denotes the even dimension, and 2i + 1 denotes the odd dimension.

In the actual training process, batch training can be used to improve model efficiency, with each batch processing a sequence of multiple target points. The hyperparameters of the model, such as embedding dimension, the number of heads of multi-head attention, the number of stacked layers of the encoder, and the learning rate, were adjusted by the grid search method. The standard cross-entropy loss function was used as the loss function. The use of the Adam optimizer can well adapt to the training of the Transformer model. After training, model performance was evaluated on the test set using conventional classification evaluation indicators including confusion matrix, receiver operating characteristic (ROC) curve, accuracy, precision, recall, F1 score, and Kappa coefficient.

$$\mathrm{accuracy} ={\displaystyle \frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

(8)

$$\mathrm{precision} ={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}}$$

(9)

$$\mathrm{r}\mathrm{ecall} ={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

(10)

$$\mathrm{F}1 \mathrm{score}=2\times {\displaystyle \frac{\mathrm{p}\mathrm{recision} \times \mathrm{r}\mathrm{ecall} }{\mathrm{p}\mathrm{recision} +\mathrm{r}\mathrm{ecall} }}$$

(11)

$$\mathrm{K}\mathrm{a}\mathrm{p}\mathrm{p}\mathrm{a}={\displaystyle \frac{{\mathrm{P}}_0-{\mathrm{P}}_{\mathrm{c}}}{1-{\mathrm{P}}_{\mathrm{c}}}}$$

(12)

$${\mathrm{p}}_0={\displaystyle \frac{\sum _{\mathrm{i}=1}^{\mathrm{n}} {\mathrm{x}}_{\mathrm{i}\mathrm{i}}}{\mathrm{N}}},{\mathrm{p}}_{\mathrm{c}}={\displaystyle \frac{\sum _{\mathrm{i}=1}^{\mathrm{n}} {\mathrm{x}}_{\mathrm{i}+}{\mathrm{x}}_{+\mathrm{i}}}{{\mathrm{N}}^2}}$$

(13)

where True Positive (TP) represents the number of samples that the model correctly predicts as positive; True Negative (TN) represents the number of samples that the model correctly predicts as negative; False Positive (FP) represents the number of negative samples that the model incorrectly predicts as positive; and False Negative (FN) represents the number of positive samples that the model incorrectly predicts as negative. ${\mathrm{x}}_{\mathrm{i}\mathrm{i}}$ denotes the elements on the diagonal of the confusion matrix, ${\mathrm{x}}_{\mathrm{i}+}$ denotes the sum of all elements in row i, ${\mathrm{x}}_{+\mathrm{i}}$ denotes the sum of all elements in column i, and N denotes the sum of all elements. ${\mathrm{p}}_0$ represents the proportion of observation accuracy or the consistency unit; ${\mathrm{p}}_{\mathrm{c}}$ denotes the proportion of coincident or expected coincident units.

2.5. Model Prediction and Uncertainty Analysis

The prediction process involved dividing the study area into a large number of grids, finding K adjacent boreholes for each grid, and constructing borehole context sequences based on the stratigraphic information of adjacent boreholes. The borehole context sequence for all grids in the study area was then input into the trained Transformer model to predict the stratigraphic category of each grid (as shown in Figure 6).

The softmax layer outputs the probability that each grid unit belongs to a specific stratum category. In this study, the normalized information entropy of each grid unit was calculated by combining the probability distribution to quantify the uncertainty in the prediction modeling process. The calculation of the normalized information entropy is shown in Equation (14):

$$\mathrm{H}\left(\mathrm{X}\right)=-{\displaystyle \frac{\sum _{\mathrm{x}\in \mathcal{S}} \mathrm{p}\left(\mathrm{x}\right)\mathrm{l}\mathrm{n}\left(\mathrm{p}\right(\mathrm{x}\left)\right)}{{\mathrm{S}}_{\mathrm{m}\mathrm{a}\mathrm{x}}}}$$

(14)

where S is the possible stratum category of each target point, S_max is equal to ln (n), and n is the number of possible stratigraphic categories. The information entropy of each data point was obtained by calculating the probability p(x) of each target point across all stratum categories. The magnitude of the information entropy reflected the complexity of a certain position in the geological model. The closer the information entropy was to 0, the higher the certainty that the data point belonged to a specific stratum category. On the contrary, the closer the information entropy was to 1, the higher the uncertainty.

3. Experiments and Results

3.1. Experiments

The test data consist of 37 boreholes located in Huangpu District, Shanghai, with a depth range of 85.3–134.6 m, an average depth of 103.4 m, a minimum thickness of 0.5 m, and a maximum thickness of 45.8 m. These boreholes are distributed over an area of 5000 × 6200 m (as shown in Figure 7). The engineering geological layer refers to the stratum divided according to geological age, geological origin, material composition, and physical and mechanical properties of rock and soil. The study area was divided into 15 engineering geological layers, with the geological age, sequence number, and lithology of each stratum shown in

Table 2. Engineering geological stratigraphy of Huangpu District, Shanghai (modified after [48,49]).

Geological Era		Engineering Geological Layer
		No.	Name
Holocene	$\mathrm{Late} \mathrm{phase} Q{h}_3$	①	Fill
		②	Brown–yellow silty clay
	$\mathrm{Middle} \mathrm{phase} Q{h}_2$	③	Gray muddy silty clay
		④	Gray muddy clay
	$\mathrm{Early} \mathrm{phase} Q{h}_1$	⑤1	Gray silty clay
		⑤2	Gray sandy clay
		⑤3	Gray silty clay
		⑤4	Gray–green silty clay
Upper Pleistocene	$\mathrm{Late} \mathrm{phase} Q{p}_3^2$	⑥	Dark green–brown–yellow silty clay
		⑦1	Grass yellow–gray sandy silt
		⑦2	Gray yellow–gray powder sand
		⑧	Gray silty clay
	$\mathrm{Early} \mathrm{phase} Q{p}_3^1$	⑨1	Cyan–gray silty sand
		⑨2	Gray gravelly medium sand
Middle Pleistocene	$Q{p}_2$	⑩	Blue–gray silty clay

. The strata in this area are mainly composed of Quaternary loose sediments, which are located in human activity sites. The stratum is characterized by the frequent formation of lenticles and pinch-outs and is not significantly affected by faults, joints, and other fault structures [3,4]. The borehole data are public data from the website https://data.sigs.cn/sigs-service-platform/#/home (accessed on 5 January 2024).

The experimental environment for this study included PyTorch 2.1.0 and Python 3.10. The device used is an Intel Xeon Platinum 8358P 2.60 GHz CPU and an NVIDIA-A800 GPU. The grid search method was used for parameter optimization, and the optimal values for each parameter are shown in

Table 3. Transformer model parameter values.

Parameters	Value
Training set: validation set: test set	6:2:2
Embed dims	256
Num heads	8
Num layers	6
Learning rate	$1 \text{×} {10}^{-5}$
Number of training epochs	500
Loss function	Cross-entropy loss
Optimizer	Adam
Number of neighbor boreholes (k)	3

.

3.2. Results

Table 4. Accuracy, precision, recall, F1 score, and Kappa Coefficient values for the test data set.

Metric	Value
Accuracy	0.86
Precision	0.88
Recall	0.86
F1 Score	0.86
Kappa Coefficient	0.85

shows that all accuracy indexes for the test set exceeded 0.85. The confusion matrix for the test set classification is shown in Figure 8. The confusion matrix displays the predicted and actual values for each stratum category, reflecting the reliability of the model’s classification results. As seen in Figure 8, the classification accuracy for most layers was very high, with only a few layers being misclassified as adjacent layers. The ROC curve shows the performance of the classifier under different thresholds. The closer the curve is to the upper left corner, the better the prediction performance of the model. The area under the ROC curve (AUC) serves as a comprehensive measure of all potential classification thresholds. AUC values greater than 90% are considered excellent, in the range of 75%–90% they are considered good, in the range of 50%–75% they are poor, and values under 50% represent unacceptable performance [50]. The experimental results (Figure 9) show that the classification performance on the test set is generally above good, with most of the results categorized as excellent.

In this study, the grid size used for 3D geological modeling is 100 m × 100 m × 0.5 m, and the study area is divided into 50 × 62 × 271 grids. Figure 10a shows the results of the 3D geological model, illustrating the geometry and distribution of geological bodies. Figure 10b shows the fence diagrams derived from the model, which presents the stratigraphic coverage relationship inside the model. Figure 11 shows the distribution of some strata on the plane. It can be seen that some strata such as brown–yellow silty clay and grass yellow–gray sandy silt, gray sandy clay are only distributed in some areas, which is consistent with the sedimentary law of the delta area. In addition, Figure 12 shows the proportion of each stratum in the model. Figure 13 shows the comparison between the test set boreholes and the predicted boreholes. The predicted results align with the original boreholes, including the thickness of each stratum and the sedimentary sequence in the original boreholes.

4. Discussion

4.1. Comparison of the Transformer Model and Other Methods

To verify the reliability of the proposed method, this study used IDW, Kriging, and RNN to construct a 3D geological model. Firstly, by comparing the profile (Figure 14c) obtained by the Transformer model with the profiles (Figure 14a,b) obtained by traditional modeling methods (IDW and Kriging), it can be seen that the overall distribution of strata in these profiles is consistent. In detail, the transformer profile is closer to adjacent boreholes in some strata such as gray silty clay and grass yellow–gray sandy silt. In addition, according to

Table 5. Comparison of RMSE of the Transformer model, IDW, Kriging.

Borehole ID	RMSE of IDW	RMSE of Kriging	RMSE of Transformer Model
3	4.135	3.435	2.965
17	4.515	3.947	3.293
36	4.093	3.376	2.936
Average	4.248	3.586	3.065

, the root mean square error (RMSE) of the Transformer is smaller than the RMSE of the IDW and Kriging methods. Then, by comparing the profile (Figure 14c) obtained by the Transformer model with the profile (Figure 14d) generated by the RNN, it can be seen that the profile obtained by the Transformer model is consistent with the stratigraphic information from the adjacent boreholes, accurately revealing the stratigraphic coverage relationship and the pinch-out phenomenon. In contrast, the model obtained by the RNN fails to capture the pinch-out phenomenon observed in the nearby boreholes. In addition, the Transformer model outperforms the RNN in terms of accuracy, precision, recall, F1 score, and Kappa coefficient (Figure 15). A possible explanation for this is that during the calculation of the RNN, information with very long interval distances will be lost, preventing the establishment of long-term dependencies in the context [51,52].

4.2. Analysis of Model Uncertainty

For the 3D geological model, the stratum boundary information provided by the borehole data is accurate, while the stratum boundary outside the borehole data area is predicted [31]. In this study, normalized information entropy was introduced to quantify the uncertainty during the modeling process. The visualized 3D normalized information entropy model (Figure 16) shows the uncertainty at different positions within the model and profile. Figure 16 reveals that the uncertainty of the region at the boundaries of the strata is high, as the classifier is susceptible to interference in these areas. In addition, the RNN also shows high uncertainty in the interior of some strata, especially in the strata above −60 m, indicating that the RNN model performs less effectively than the Transformer model. The possible reason is that the thickness of the strata above −60 m is thinner than that below −60 m, and RNN does not capture the complex stratigraphic relationship.

4.3. Advantages and Disadvantages of the Proposed Method

The formation and evolution of geological structures and phenomena are challenging to describe because they are accompanied by a high degree of nonlinearity. Deep learning methods can extract high-order features from complex structures [53,54]. Although borehole data provide accurate stratigraphic distribution information, budget and terrain constraints often limit the availability of borehole data, making accurate stratigraphic prediction in sparse data areas a challenge. Traditional interpolation methods fail to account for the anisotropy between borehole data points. The self-attention mechanism in the Transformer model completely replaces the cyclic layer, enabling the analysis of longer input sequences [55]. While the Transformer model has achieved significant success in natural language processing and other fields, its application to 3D geological modeling is rare, particularly under the conditions of sparse borehole data. By using the Transformer model to extract relationships between different boreholes, the proposed method effectively utilizes limited borehole data. It improves modeling accuracy in sparse data areas and provides a more reliable basis for geological analysis and mineral resource estimation. In many fields, only sparse measurement data can be obtained due to the limitation of measurement equipment, cost considerations, or the difficulty of data acquisition. The idea of the proposed method can be extended to solve the problem of modeling with sparse measurement data; for example, groundwater modeling, medical image modeling, traffic flow prediction, etc.

Although the proposed method can learn complex geological laws from borehole data, effectively integrating the prior knowledge and experience of geologists into the model is still a problem to be solved. A possible solution is to introduce multi-source data such as geological profile data, geophysical data, and geological structure data, and convert these data into virtual boreholes and geological constraints [56,57]. The next research will explore the solution further.

5. Conclusions

In this study, a method for 3D geological modeling using the Transformer model under conditions of sparse borehole data is proposed. The borehole context sequence is constructed by using the stratigraphic information from adjacent boreholes and is input into the Transformer encoders. The model calculates the multi-head self-attention for each part of the borehole context sequence to quantify the relationship between the target point and its adjacent borehole strata. The experimental results show an accuracy of 0.86, with better classification accuracy and lower uncertainty compared to an RNN. In addition, the RMSE of this method is smaller than IDW and Kriging. A comparison with the IDW, Kriging, and RNN results demonstrates that this method more accurately reveals the geometric shape and distribution of the geological body, which aligns with the sedimentary laws of the region. Compared with the deterministic modeling method, this method allows for uncertainty assessment. This method holds significant practical value in geological engineering, mineral resource exploration, and other fields.