A Novel Transformer-CNN Approach for Predicting Soil Properties from LUCAS Vis-NIR Spectral Data

Abstract

Soil, a non-renewable resource, requires continuous monitoring to prevent degradation and support sustainable agriculture. Visible-near-infrared (Vis-NIR) spectroscopy is a rapid and cost-effective method for predicting soil properties. While traditional machine learning methods are commonly used for modeling Vis-NIR spectral data, large datasets may benefit more from advanced deep learning techniques. In this study, based on the large soil spectral library LUCAS, we aimed to enhance regression model performance in soil property estimation by combining Transformer and convolutional neural network (CNN) techniques to predict 11 soil properties (clay, silt, pH in CaCl₂, pH in H₂O, CEC, OC, CaCO₃, N, P, and K). The Transformer-CNN model accurately predicted most soil properties, outperforming other methods (partial least squares regression (PLSR), random forest regression (RFR), support vector machine regression (SVR), Long Short-Term Memory (LSTM), ResNet18) with a 10–24 percentage point improvement in the coefficient of determination (R²). The Transformer-CNN model excelled in predicting pH in CaCl₂, pH in H₂O, OC, CaCO₃, and N (R² = 0.94–0.96, RPD > 3) and performed well for clay, sand, CEC, P, and K (R² = 0.77–0.85, 2 < RPD < 3). This study demonstrates the potential of Transformer-CNN in enhancing soil property prediction, although future work should aim to optimize computational efficiency and explore a wider range of applications to ensure its utility in different agricultural settings.

1. Introduction

The European Union (EU) aims to ensure that 75% of soils are healthy by 2030 to secure healthy food, people, nature, and climate. As of 2020, approximately 60–70% of EU soils remain unhealthy [1]. Healthy soils are the foundation of agricultural production and a crucial component of the ecosystem. Effective soil management and conservation practices, such as crop rotation, cover cropping, and reduced fertilizer usage, sustain soil fertility, enhance biodiversity, and facilitate the colonization of beneficial microorganisms [2]. Soil with appropriate elemental content provides essential nutrients for plants, promoting healthy growth. It enhances soil moisture retention and increases water-holding capacity [3]. Soil testing and fertilizer field trials help ascertain crop fertilizer demand patterns and soil nutrient-supplying capacity, thereby enabling the scientific formulation of effective fertilizer application programs [4]. Therefore, predicting soil elemental content is crucial for guiding agricultural production, promoting plant growth, and reducing costs. Spectral remote sensing acquires soil spectral information without direct contact with the ground, making it an indispensable tool for soil investigation and analysis [5].

The key stages of spectral soil property prediction are data preprocessing and model building. Data preprocessing primarily aims to enhance model accuracy and robustness. Common mathematical transformations, Savitzky–Golay (SG) smoothing, and differential preprocessing methods are effective in improving prediction models [6,7]. SG smoothing effectively suppresses spectral noise and reduces random noise impact [8]. Differential processing mitigates instrumental background and drift influence on the signal, amplifying the spectral response [9]. Recently, more researchers have employed machine learning and deep learning techniques to explore the relationship between hyperspectral data and soil elemental content. PLSR [10], RFR [11], and SVR remain robust prediction models in this field. With advancing computational power, deep learning is gradually being applied across various domains. CNNs [12] are widely used in computer vision and geographic information systems (GISs) due to their exceptional performance and versatility [13,14,15]. Integrating mathematical concepts into differential convolutional neural networks facilitates faster error minimization and convergence [16]. Combining CNN component construction with hyperparameter tuning can significantly advance CNN application in soil spectroscopy modeling [17]. The Transformer model, proposed by Google researchers in 2017, is a deep learning method based on an attention mechanism without convolutional or recurrent units. The Transformer model is capable of time series prediction [18], air quality prediction [19], risk prediction [20], and more. Its architecture can be scaled to the training dataset size and model, enabling efficient parallel training and long-range dependency capture [19]. These advantages motivated us to consider using such models for spectral soil content prediction.

The LUCAS 2009 Topsoil Survey dataset [21] is one of the most comprehensive and consistent soil datasets on a continental scale. Numerous studies have utilized Vis-NIR spectra from the LUCAS dataset to predict various soil properties. Evangelos Simpris [22] proposed using stacked autoencoders to predict eight key soil properties simultaneously, improving prediction accuracy. Zhong et al. [23] evaluated data preprocessing and multitasking Deep Convolutional Neural Network (DCNN) models, finding they outperformed shallow CNN and other traditional machine learning methods. Singh et al. [24] used principal component analysis and locally preserved projections to reduce hyperspectral data dimensionality and proposed a framework combining hybrid features and Recurrent Neural Network (RNN) variants (LSTM and Gated Recurrent Unit (GRU)), demonstrating enhanced predictive capabilities. Stanislaw Grushchensky et al. [25] compared the effectiveness of different machine-learning models for large spectral libraries. Hamid Tavakoli et al. [26] investigated stacked models’ role in improving the prediction accuracy of traditional individual models. Midi Wan et al. [27] proposed a near-infrared spectral masking autoencoder that learns highly robust and generalized spectral features from a large public near-infrared spectral dataset. Alex Wangic et al. [28] compared the performance of laser-induced breakdown spectroscopy (LIBS) and visible near-infrared spectroscopy (Vis-NIRS) in predicting soil organic carbon (SOC), texture, extractable phosphorus, clay, and organic carbon ratio. Existing studies have demonstrated the ability to predict a broad spectrum of soil properties with high accuracy (R² > 0.8). Although these studies have been extensively discussed and have yielded reasonably accurate predictions, many have excluded samples from certain portions of the LUCAS dataset and selectively focused on specific soil properties. While this approach may yield favorable results, it does not fully capture the comprehensive nature of the dataset. Building upon prior research, this study aims to comprehensively explore the complete LUCAS dataset and leverage deep learning techniques to enhance prediction accuracy, potentially opening new avenues for soil property prediction.

Many studies have demonstrated that integrating different types of neural networks significantly improves model performance. Yuhao Wang et al. [29] used a deep learning LSTM model combined with an attention mechanism and peak NDVI images to generate more accurate and timely predictions. Shujun Liu et al. [30] utilized a tensor connection module to combine temporal convolutional networks (TCN) and LSTM neural networks for wind power prediction, achieving excellent results. The powerful sequence processing capabilities of Transformers, combined with the superior feature extraction capabilities of CNN, have improved performance in both predictive modeling and image processing tasks. This fusion approach leverages the strengths of different models to provide richer feature representations, significantly advancing various application scenarios [31,32,33]. These successes support our use of a similar parallel approach in this study. This study introduces an innovative approach combining Transformer and CNN models to enhance prediction accuracy and model stability. Transformer models excel in capturing long-range dependencies, while CNN is effective at extracting localized features. In contrast to several recent studies [34,35,36,37] that primarily focus on improving accuracy under specific soil properties or controlled conditions, this study seeks to broaden the model’s applicability across a wider range of soil properties and environmental contexts. By doing so, it not only advances the current state of the art in soil property prediction but also addresses practical challenges that hinder the deployment of deep learning models in soil science.

By combining these techniques, we can leverage their strengths for more comprehensive and accurate soil property predictions. Traditional machine learning methods frequently struggle to capture the full complexity and variability of soil spectral data, particularly when confronted with noise and high-dimensional features. These limitations impede both the accuracy and scalability of soil property predictions. Our goal is to overcome traditional model limitations and provide a more robust and efficient tool for soil property prediction through this fusion approach. We will objectively evaluate the strengths and weaknesses of this approach by comparing its performance against traditional machine learning models and existing literature methods. We will explore its applicability and superiority in predicting different soil properties. The evaluation will be multi-dimensional, including prediction accuracy, computational efficiency, applicability, and limitations.

2. Materials and Methods

2.1. LUCAS Database of Topsoil

The LUCAS 2009 Topsoil Survey dataset is one of the most comprehensive and harmonized soil datasets on a continental scale [38]. The dataset comprises around 20,000 samples from 25 member countries. Sampling sites were selected based on a standardized procedure, collecting samples at designated locations through a composite sampling process. At each sampling point, five soil samples were collected: one at the center, coinciding with the LUCAS point, and the other four in a cross shape, 2 m from the center. Approximately 0.5 kg of topsoil samples (0–20 cm) were collected at each point, representing the area of the LUCAS point. The soil sampling points in this database are primarily biased towards agricultural land (especially cropland) and to a lesser extent grassland and woodland. The LUCAS soil points reflect the land use and topography of the countries to varying degrees, depending on land use and topography heterogeneity. The LUCAS measurements exclude areas above 1000 m above sea level. All samples were registered and visually inspected; mineral soils were air-dried and repackaged. Samples were sent to laboratories for analysis, including measurements of coarse fragments, particle size distribution (% clay, silt, and sand content), pH (in CaCl₂ and H₂O), organic carbon (g/kg), carbonate content (g/kg), phosphorus content (mg/kg), total nitrogen content (g/kg), extractable potassium content (mg/kg), cation exchange capacity (cmol(+)/kg), and multispectral properties.

2.1.1. Spectral Measurements and Pre-Processing

The visible-near infrared (Vis-NIR) absorbance spectra of the samples were measured using a FOSS XDS Rapid Content Analyzer (NIRSystems, INC., Hilleroed, Denmark) within the wavelength range of 400–2400.5 nm, with a spectral resolution of 0.5 nm. Each sample was scanned twice, and the average values were used for subsequent analysis. Although spectral pre-processing was not the primary focus of this experiment, a standard approach was adopted based on similar studies. Referring to the study by Tavakoli et al. [26], the Savitzky–Golay (SG) smoothing algorithm [39] was used to filter the raw data with a first-order polynomial fit and an 11-point window size, followed by first-order differentiation of the smoothed data. Data in the wavelength range of 400–499.5 nm were excluded due to noise, and the data were downsampled by retaining one value every 20 nm, resulting in 200 data points after pre-processing (Figure 1).

2.1.2. Splitting of Soil Samples

The LUCAS database for this experiment contains 19,036 soil samples, released after outliers were detected and removed; no additional outlier analysis was performed in this paper. We used all available soil samples in the dataset, including mineral and organic soil samples, without considering additional information such as geographic origin or soil class. Missing values in the samples were filled using the mean, a common preprocessing method [40]. First, we referred to other research data segmentation methods [22], and used the Conditional Latin Hypercube Sampling algorithm [41] to segment the data into a training set (66.$\overline{6}$%, 12,690 samples) and an independent test set (33.$\overline{3}$%, 6346 samples) (

Table 1. Descriptive statistics of 11 soil properties in the LUCAS dataset.

Soil Properties	Set	Minimum	Maximum	Mean	Standard Deviation	Coefficient of Variation
clay (%)	Complete	0	79.00	18.89	12.62	0.67
	Training	0	79.00	18.95	12.59	0.66
	Test	0	77.00	18.78	12.68	0.68
silt (%)	Complete	0	92.00	38.21	17.76	0.46
	Training	0	92.00	38.28	17.80	0.46
	Test	1.00	88.00	38.08	17.69	0.46
sand (%)	Complete	1.00	99.00	42.89	25.34	0.59
	Training	1.00	99.00	42.77	25.35	0.59
	Test	1.00	99.00	43.13	25.33	0.59
pH in CaCl2	Complete	2.57	9.25	5.59	1.43	0.25
	Training	2.57	9.25	5.59	1.42	0.25
	Test	2.62	8.64	5.60	1.43	0.26
pH in H2O	Complete	3.21	10.08	6.20	1.35	0.22
	Training	3.21	10.08	6.20	1.35	0.22
	Test	3.40	9.65	6.20	1.35	0.22
CEC (cmol(+)/kg)	Complete	0	234.00	15.76	14.48	0.92
	Training	0	234.00	15.68	14.04	0.90
	Test	0	227.70	15.90	15.33	0.96
OC (g/kg)	Complete	0	586.80	50.0	91.30	1.83
	Training	0	586.40	50.1	91.07	1.82
	Test	0	586.80	49.9	91.77	1.84
CaCO3 (g/kg)	Complete	0	944.00	51.60	125.31	2.43
	Training	0	944.00	50.68	123.88	2.44
	Test	0	865.00	53.43	128.12	2.40
N (mg/kg)	Complete	0	38.60	2.92	3.75	1.28
	Training	0	36.20	2.93	3.74	1.28
	Test	0	38.60	2.91	3.78	1.30
P (mg/kg)	Complete	0	1366.40	30.08	32.85	1.09
	Training	0	1366.40	29.91	33.53	1.12
	Test	0	575.10	30.42	31.44	1.03
K (mg/kg)	Complete	0	7342.00	197.04	229.29	1.16
	Training	0	7342.00	197.10	237.57	1.21
	Test	0	4902.40	196.92	211.77	1.08

). The training set was then divided into five folds for quintuple cross-validation using the fuzzy C-means algorithm [42]. Each fold was used in turn as a validation set, with the remaining four folds combined into a training set, repeating this process five times. The training set was used to learn the model parameters, while the validation set was used to prevent overfitting and optimize hyperparameter combinations. In the final testing phase, the test set obtained from the data partition was used for model evaluation. This test set was independent of the cross-validation process and was not used for model training or validation.

2.2. Modeling Methodology

We propose a model that combines Transformer and CNN in parallel for the processing and analysis of hyperspectral data. The Transformer captures global dependencies within sequences, while the CNN extracts local features. Initially, both components process the hyperspectral data in parallel, and their outputs are then fused to enhance prediction accuracy and robustness. This architecture effectively overcomes the limitations of traditional models in handling complex, high-dimensional spectral data, improving both performance and generalization through comprehensive optimization.

2.2.1. Transformer

So far, the Transformer and its variants have become the preferred choice for natural language processing tasks. This pioneering work marked a significant departure from traditional RNN and CNN architectures by introducing the self-attention mechanism, which has been successfully applied to machine translation. Beyond the advances in machine translation, the Transformer has shown substantial advantages over traditional RNN and CNN in various natural language processing tasks. Since its inception, numerous models have been developed to address a wide range of tasks across different disciplines, simplifying the interpretation and processing of sequences. The multi-head attention mechanism enables the Transformer model to learn contextual information sequentially without long-range dependency issues [43]. The Transformer model achieves efficient sequence learning through a highly parallelizable self-attention mechanism, easily establishing relationships between different features at various locations [44]. The self-attention mechanism is enhanced with multiple attention heads, each learning distinct attention weights to better capture different types of relationships. Multi-head attention enables the model to process different information subspaces in parallel. Deep learning models are often computationally expensive and require significant training time. However, Transformer models mitigate some of the challenges inherent in traditional deep-learning sequence modeling approaches. Their high parallelizability and simple components, such as the attention module and fully connected layer, make them computationally efficient and appealing [45]. In contrast, traditional methods, like linear regression, are constrained by their reliance on linear relationships between inputs and outputs, which hinders their ability to handle complex nonlinear spectral data. Additionally, these methods are prone to overfitting and lack generalization capabilities when dealing with high-dimensional data [46].

Recent research has demonstrated the feasibility of applying the Transformer architecture to forecasting tasks. Wu et al. [47], for instance, used a decoder design similar to the original Transformer architecture in a case study on influenza-like illness (ILI) forecasting to predict time series data. Qu et al. [48] combined features for wind power forecasting by removing the embedding and SoftMax layers of the Transformer, enabling the model to effectively capture key information from wind data. Nascimento et al. [49] integrated wavelet transform into a Transformer-based architecture, a technique that increases the number of features in the data by extracting more relevant information from the existing data. This Transformer-based approach serves as a general framework for modeling various nonlinear dynamical systems, supporting its application in nonlinear spectral data analysis. In this paper, we develop a Transformer model to predict multiple hyperspectral soil properties, implementing the architecture from “Attention is All You Need” [50] using Tensorflow. The model includes a Transformer module with an encoder and a fully connected layer; the encoder consists of a multi-head attention layer and a feedforward layer (Figure 2). The Transformer model uses the following parameters: an embedding dimension of 64, projecting input data into a high-dimensional space, and 4 attention heads. The multi-head attention mechanism allows the model to focus on multiple information subspaces simultaneously; a drop probability (drop_prob) of 0.1 is used. Each Transformer module contains a Multihead Attention Layer, Feedforward Layer, Layer Normalization, and Dropout Layer. The Feedforward Layer consists of two fully connected layers: the first with 4 times the number of neurons in the embedding dimension and using Leaky ReLU activation, and the second with 2 times the number of neurons in the embedding dimension.

Hyperspectral imaging captures soil spectral information at multiple wavelengths, critical for determining soil’s physical and chemical properties. Traditional regression models often struggle to capture complex relationships between wavelengths in high-dimensional spectral data. In contrast, Transformers can efficiently process and analyze high-dimensional data through their self-attention mechanism. Data in each spectral band are treated as part of a sequence, allowing the model to establish global dependencies between bands. The multi-attention mechanism enables the model to focus on multiple wavelength interactions simultaneously, improving prediction accuracy and robustness. Consequently, the Transformer model extracts soil information from preprocessed spectral features and inputs these into the encoder, with the output passing through the fully connected layer.

2.2.2. CNN

A typical CNN comprises an input layer, convolutional layer, pooling layer, fully connected layer, and output layer [51]. The input layer receives data as vectors and matrices, preserving internal structure integrity. The convolutional layer scans localized regions of the input data using convolutional kernels with shared weights, partially connecting to previous layer neurons. More convolutional kernels enable the extraction of more abstract features. The pooling layer reduces feature size by nonlinear downsampling, shrinking network size and improving computational efficiency. The fully connected layer’s neurons connect to all previous layer elements, converting features into one-dimensional vectors. The output layer, typically a fully connected layer, generates the model’s final output. The CNN also incorporates dropout and activation functions. Dropout randomly discards network units during training to prevent overfitting. The activation function fits the nonlinear mapping of features extracted from the convolutional layer, enhancing CNN’s ability to handle nonlinear data [52].

This paper designs and implements a CNN for processing spectral data and making predictions. The model includes three convolutional layers, each followed by a batch normalization layer and an average pooling layer. The model uses 64, 128, and 256 filters with a convolutional kernel size of 5 and a step size of 2 for each layer. Finally, the prediction results are output through the fully connected layer. CNN offers significant advantages in hyperspectral soil property prediction. Hyperspectral imaging captures soil spectral information at multiple wavelengths, crucial for determining its physical and chemical properties. CNN excels in regression tasks by efficiently capturing local features and gradually aggregating them into global information when processing high-dimensional spectral data. Data in each spectral band are considered part of the image, and the model extracts local features between different wavelengths through convolutional operations.

2.2.3. Model Architecture

In soil property prediction, uncertainty may be introduced by noise effects in the data, choice of model parameters, and data preprocessing methods. Specifically, noise in spectral data may lead to input uncertainty, while the choice of different preprocessing methods (e.g., smoothing, mathematical transformations) may also have an impact on the final prediction of the model. Uncertainty quantification (UQ) methods play a pivotal role in reducing Uncertainty quantification (UQ) methods play a pivotal role in reducing the impact of uncertainties during both optimization and decision-making processes. Dropout can be used to quantify the model’s prediction uncertainty by randomly dropping neurons during the testing phase to simulate multiple runs of the model [53]. When using CNN and Transformer models in parallel, the input data are processed by both models. The CNN model extracts local features using convolutional and pooling layers, progressively capturing the spatial features of the input data and generating corresponding feature representations. Meanwhile, the Transformer model captures global information and sequential relationships, modeling long-range dependencies between input data through multi-head attention. The outputs of both models are fused through a joint operation to form a combined feature representation. This fused feature representation is then fed into a fully connected layer for final prediction or classification. During training, the parameters of the CNN and Transformer models can be optimized simultaneously in an end-to-end manner (

Table 2. Model framework parameterization.

Hyper-Parameters	Setting
Learning rate	0.001
Optimiser	Adam
Dense Layer parameters	128/256/1
Batch size	1024
Loss Function	MSE
Epoch	2000
Patience	200
Dropout	0.1
Cov1D filters	64/128/256
Cov1D strides	2
Activation	leaky_relu

), allowing them to work together to improve performance and generalization ability. This parallel structure leverages the respective strengths of CNN and Transformer, which we call Transformer-CNN, to enhance the model’s capability in modeling complex and high-dimensional data. This approach is suitable for tasks that require both local and global information and plays a significant role in the regression analysis of hyperspectral soil data.

2.2.4. Model Assessment

In this investigation, the coefficient of determination (R²), root mean square error (RMSE), and relative prediction deviation (RPD) are taken as performance evaluation metrics. R² is utilized to assess the goodness-of-fit between the model’s predicted values and the actual observations, with values approaching 1 indicating greater accuracy of the model. RMSE represents the average deviation between predicted and actual values, with lower values signifying better performance. RPD calculated as the ratio of the standard deviation to the RMSE, is employed to evaluate the model’s accuracy, where a value greater than 2 suggests enhanced predictive capability. Their calculation formulas are shown in Equations (1)–(3), respectively:

$${\mathrm{R}}^2=1-{\displaystyle \frac{{{\displaystyle \sum }}_{i=1}^n{\left(y_i-{\widehat{y}}_i\right)}^2}{{{\displaystyle \sum }}_{i=1}^n{\left(y_i-\overline{y}\right)}^2}}$$

(1)

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{n}}{{\displaystyle \sum }}_{i=1}^n{\left(y_i-{\widehat{y}}_i\right)}^2}$$

(2)

$$\mathrm{RPD}={\displaystyle \frac{SD}{RMSE}}=\sqrt{{\displaystyle \frac{1}{\mathrm{n}}}{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{y}}_{\mathrm{i}}-\overline{\mathrm{y}}\right)}^2}/\sqrt{{\displaystyle \frac{1}{\mathrm{n}}}{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{y}}_{\mathrm{i}}-{\hat{\mathrm{y}}}_{\mathrm{i}}\right)}^2}$$

(3)

In predictive model evaluation, RPD is an important metric for assessing the accuracy of a model. The higher the RPD value, the better the predictive ability of the model (

Table 3. Classification tiers of the performance-to-deviation ratio (RPD).

RPD	Meaning	Level
RPD > 3	Excellent Model	A
2.5 ≤ RPD ≤ 3.0	Good Model	B
2.0 ≤ RPD ≤ 2.5	Approximate Model	C
RPD < 2.0	Unsatisfactory Model	D

). Specifically, an RPD greater than 3 indicates that the model is an excellent model (rank A); an RPD between 2.5 and 3.0 indicates that the model is a good model (rank B); an RPD between 2.0 and 2.5 indicates that the model is an approximate model (rank C); whereas an RPD of less than 2.0 indicates that the model is unsatisfactory (rank D) [27].

2.2.5. Experimental Set-Up

Computations were performed using a 13th generation Intel^® Core™ i9-13900KF CPU and an NVIDIA^® GeForce RTX 4090 GPU (24GB graphics memory). The operating system is Windows 11, the programming language is Python 3.7.16, and the deep learning framework is TensorFlow-gpu 2.6.0.

3. Results

3.1. Evaluation Results of Transformer-CNN

This section presents regression predictions for eleven soil properties (Clay, Silt, Sand, pH in CaCl₂, pH in H₂O, CEC, OC, CaCO₃, N, P, and K). Ablation experiments were conducted with the proposed model to evaluate its performance using metrics such as R², RMSE, and RPD (

Table 4. The effect of ablation experiments.

Soil	Transformer			CNN			Transformer-CNN
Properties	R2	RMSE	RPD	R2	RMSE	RPD	R2	RMSE	RPD
Clay	0.85	4.91	2.58	0.78	5.89	2.15	0.85	4.86	2.61
Silt	0.71	9.57	1.85	0.60	11.22	1.58	0.72	9.40	1.88
Sand	0.77	12.25	2.07	0.66	14.69	1.72	0.77	12.08	2.10
pH in CaCl2	0.91	0.42	3.40	0.92	0.40	3.57	0.94	0.36	3.99
pH in H2O	0.90	0.42	3.22	0.91	0.39	3.42	0.93	0.36	3.77
CEC	0.82	6.43	2.39	0.77	7.33	2.09	0.83	6.36	2.41
OC	0.95	20.08	4.57	0.94	22.80	4.02	0.96	19.47	4.71
CaCO3	0.96	25.99	4.93	0.94	31.64	4.05	0.96	24.85	5.16
N	0.93	0.97	3.89	0.92	1.09	3.46	0.94	0.93	4.07
P	0.40	24.32	1.29	0.20	28.04	1.12	0.41	24.19	1.30
K	0.58	137.61	1.54	0.43	160.13	1.32	0.60	133.93	1.58

). The Transformer model showed superior predictive accuracy compared to the CNN model. Despite their similarities, the Transformer-CNN hybrid model outperformed the Transformer alone in prediction performance. Performance metrics indicate that most soil properties were predicted with satisfactory accuracy. Specifically, the Transformer-CNN model exhibited excellent predictive performance for pH in CaCl₂, pH in H₂O, OC, CaCO₃, and N, with R² values ranging from 0.94 to 0.96 and RPD values exceeding 3. It achieved good performance for Clay, Sand, CEC, P, and K, with R² values between 0.77 and 0.85 and RPD values ranging from 2 to 3. In contrast, Silt, P, and K demonstrated average predictive performance, with R² values ranging from 0.41 to 0.72 and RPD values below 2.

The ablation experiments reveal that incorporating the CNN model substantially enhances the prediction performance for Clay, Silt, Sand, P, and K. Notably, the inclusion of CNN leads to significant improvements in Phosphorus (R² increased by 21 percentage points) and Potassium (R² increased by 17 percentage points). This advancement highlights the method’s progress in integrating global and local information, optimizing feature extraction, and refining model fusion. According to the tabular data, the proposed Transformer-CNN method shows superior performance in predicting several key attributes, including N with an R² of 0.94, OC with an R² of 0.96, and CaCO₃ with an R² of 0.96. These results underscore the distinctive advantages of the Transformer-CNN method in synthesizing diverse information sources, optimizing multi-scale feature extraction, and enhancing prediction accuracy. Figure 3 illustrates scatter plots comparing the measured and predicted values for the eleven soil attributes using the Transformer-CNN model.

3.2. Performance Comparison with Other Models

To further evaluate the prediction ability of the Transformer-CNN model for LUCAS soil properties, we first used traditional machine learning models (PLSR, RFR, SVR) and deep learning models (LSTM, ResNet18). Traditional regression methods usually rely on linear assumptions, making it difficult to capture the complex interactions among soil properties [54]. RFR can handle a large number of input variables and reduce the risk of overfitting by integrating multiple decision trees, thereby improving the robustness and accuracy of the predictions. SVR optimizes predictions by defining the optimal hyperplane in a high-dimensional feature space, making it suitable for complex nonlinear problems. Deep learning models can handle nonlinear relationships and high-dimensional data. Compared to traditional regression methods, deep models show significant advantages in practical applications. They can effectively utilize large-scale datasets for training, learning, and adapting to variations in soil properties and complex relationships [55].

RFR is capable of handling large-scale data.

Table 5. Compare and contrast the effects of the experiment.

Soil Properties	PLSR			RFR			SVR			LSTM			ResNet18
	R2	RMSE	RPD	R2	RMSE	RPD	R2	RMSE	RPD	R2	RMSE	RPD	R2	RMSE	RPD
Clay	0.70	6.99	1.81	0.72	6.75	1.88	0.71	6.78	1.86	0.70	7.29	1.74	0.77	6.02	2.11
Silt	0.41	13.62	1.30	0.51	12.19	1.45	0.45	13.18	1.35	0.43	12.37	1.32	0.62	10.91	1.62
Sand	0.52	17.40	1.46	0.61	15.91	1.59	0.55	16.90	1.50	0.53	17.34	1.46	0.66	14.67	1.73
pH in CaCl2	0.84	0.58	2.47	0.84	0.53	2.52	0.87	0.52	2.76	0.87	0.51	2.83	0.89	0.45	2.98
pH in H2O	0.83	0.56	2.43	0.85	0.57	2.57	0.86	0.50	2.71	0.81	0.58	2.32	0.87	0.48	2.79
CEC	0.69	8.58	1.79	0.72	8.04	1.91	0.62	9.51	1.48	0.65	9.06	1.69	0.76	7.48	2.05
OC	0.89	30.25	3.03	0.93	23.51	3.90	0.89	30.72	2.96	0.91	27.57	3.33	0.94	22.19	4.14
CaCO3	0.90	40.05	3.20	0.92	36.77	3.48	0.90	40.00	3.10	0.89	42.42	3.02	0.94	30.31	4.23
N	0.83	1.55	2.44	0.88	1.33	2.83	0.81	1.62	2.31	0.87	1.37	2.76	0.89	1.24	3.05
P	0.22	27.75	1.13	0.23	27.56	1.14	0.25	27.08	1.24	0.09	29.98	1.05	0.27	26.86	1.17
K	0.36	169.92	1.25	0.31	175.77	1.20	0.36	169.15	1.25	0.21	188.30	1.12	0.48	152.02	1.39

shows that RFR in traditional machine learning outperforms PLSR and SVR in most properties, comparing the performance of different machine learning models in predicting various soil properties. The results show that the ResNet18 model significantly outperforms traditional machine learning models in predicting most soil properties, with an average R² of 0.77. In comparison, the PLSR model has an average R² of 0.67, and ResNet18 outperforms PLSR by about 14.93 percentage points. The RFR model has an average R² of 0.74, and ResNet18 outperforms RFR by about 4.05 percentage points. The average R² of the SVR model is 0.68, and ResNet18 is about 13.24 percentage points higher than SVR. The average R² of the LSTM model is 0.64, and ResNet18 is about 19.88 percentage points higher than LSTM. Overall, ResNet18 shows excellent performance in predicting clay, silt, sand, pH, CEC, organic carbon, calcium carbonate, and nitrogen content, significantly outperforming the other models and demonstrating the strong potential and advantages of deep learning in soil property prediction. Figure 4 compares the R² and RMSE (The closer to the edge, the closer the value of RMSE is to 0.) of the Transformer-CNN model with other methods in a radar chart fashion. It can be seen that the Transformer-CNN model outperforms other machine learning and deep learning models.

3.3. Effects of Data Pre-Processing

In data preprocessing, SG smoothing and differentiation techniques significantly affect data processing and analysis. SG smoothing is a commonly used technique that reduces noise and fluctuations by averaging over a sliding window, resulting in smoother and more stable data. However, SG smoothing may over-smooth the data, blurring or losing detailed information, and may yield unsatisfactory results, especially with complex patterns or rapid changes. In contrast, differentiation techniques, especially first-order differentiation, more accurately capture trend and slope information. First-order differentiation helps identify local extremes and change points, aiding in the accurate analysis of dynamic characteristics and trends. Compared to SG smoothing, first-order differentiation retains more original data features, reduces additional errors, and performs better in many data analysis tasks.

This experiment used filtering (SG1-11) and first-order differentiation with a first-order polynomial fit and 11 window sizes. We compared pre- and post-treatments, and the predictive effectiveness of all soil properties improved by an average of 3 percentage points. Although the SG method effectively reduces spectral signal noise, it may not eliminate all noise forms. First-order discretization methods are commonly used to eliminate noise caused by background inconsistencies or drift [56]. The importance of SG smoothing is demonstrated by numerous studies using LUCAS datasets to predict various soil properties [23,57,58]. When choosing an appropriate preprocessing technique, consider the data characteristics, analysis purpose, and required precision and accuracy. SG smoothing suits scenarios requiring noise reduction and curve smoothing, while first-order differentiation is better for preserving data details and dynamic changes. Correct selection and application of preprocessing techniques can significantly improve data analysis effectiveness and result reliability.

4. Discussion

To further evaluate the predictive performance of the proposed modeling approach for soil attribute content, we will compare it with findings from other related studies. First, our study demonstrated significant advantages in predicting different soil attributes. For example, in predicting N, P, and K contents, our model achieved R² values of 0.94, 0.41, and 0.60, respectively, outperforming other studies. This indicates that our model has higher accuracy and reliability in capturing and predicting soil nutrient contents. For OC, our study also achieved impressive results. Our results in most soil prediction accuracies were higher than those in other studies. This demonstrates that our model performs better in predicting soil properties and helps analyze soil trends more precisely.

Our experiments were designed to be reproducible, but the magnitude of the experimental RMSE compared to other literature could not be effectively compared due to errors caused by the sampling method used to split the dataset. This may limit the ability to reproduce the exact experimental results in this study.

Table 6. Effectiveness in comparison with other literature.

Soil Properties	Transformer-CNN			(Singh et al., 2019) [59]		(Zhong et al., 2021) [23]		(Gruszczyński et al., 2022) [25]		(Tavakoli et al., 2023) [26]
	R2	RMSE	RPD	R2	RMSE	R2	RMSE	R2	RMSE	R2	RMSE
Clay	0.85	4.86	2.61	0.80	5.95	NA	NA	0.84	5.1	NA	NA
Silt	0.72	9.40	1.88	NA	NA	NA	NA	NA	NA	NA	NA
Sand	0.77	12.08	2.10	0.71	14.59	NA	NA	NA	NA	0.73	13.41
pH in CaCl2	0.94	0.36	3.99	NA	NA	NA	NA	0.92	0.74	0.94	0.35
pH in H2O	0.93	0.36	3.77	0.90	0.42	0.94	0.33	NA	NA	NA	NA
CEC	0.83	6.36	2.41	0.77	6.75	0.82	6.14	0.83	4.0	0.80	6.89
OC	0.96	19.47	4.71	0.94	23.25	0.95	19.84	0.79	8.9	0.95	21.34
CaCO3	0.96	24.85	5.16	NA	NA	0.96	24.91	0.96	27.4	0.96	25.71
N	0.94	0.93	4.07	0.91	1.15	0.93	0.96	0.79	0.6	0.92	1.11
P	0.41	24.19	1.30	NA	NA	0.37	24.65	NA	NA	NA	NA
K	0.60	133.93	1.58	NA	NA	0.59	131.07	NA	NA	NA	NA

shows that Zong et al. [23] achieved a superior R² for pH in H₂O compared to our model, but we obtained good results for other soil properties. According to Figure 5, the matrix heat map clearly shows the comparison of our model with other research models’ R² values. It demonstrates higher prediction accuracy and reliability compared to other studies. This indicates that our method has significant advantages in predicting the content of key organic and inorganic substances in soil. Our study demonstrated significant advantages in the prediction and analysis of multiple soil attributes, not only in terms of better prediction accuracy but also in capturing and interpreting the complexity of soil data more comprehensively.

A study [35] employed a CNN model to predict six soil properties (OC, CEC, Clay, Sand, pH, and N) without using additional spectral preprocessing methods. Although this approach showed improvements over traditional methods, a single model could not comprehensively extract spectral features. Another study [36] used the combined SG smoothing filter (with a second-order polynomial fit and a window size of 11 data points) and a first-order derivative transform to predict seven soil properties (Clay, Sand, Silt, pH, OC, CaCO₃, and N), resulting in improved predictive performance compared to previous methods. In this experiment, we employed a one-dimensional convolutional neural network (1D-CNN) for modeling. While two-dimensional convolutional neural networks (2D-CNNs) for spectral image processing have proven effective [34], our study remained consistent by using both organic and mineral soils from the dataset. However, their study did not incorporate spectral preprocessing steps, which could help reduce experimental uncertainty. Our method improved the R² for six soil properties (OC, N, CEC, pH, Sand, and Clay) in the test set. Although our results might be enhanced by spectral preprocessing, recent research [37] investigating seven soil properties (OC, CaCO₃, N, CEC, pH, Clay, and Sand) employed seven preprocessing methods. While this approach may improve prediction accuracy from a preprocessing perspective, model performance could be heavily dependent on the choice of preprocessing methods. Such dependence may lead to inconsistent results and raise concerns about model stability. Variability among different preprocessing methods could cause significant differences in model performance under various conditions, increasing result variability and uncertainty. Compared to previous studies using the LUCAS dataset, we utilized the complete dataset and common preprocessing methods to predict all 11 soil properties. This approach reduces the uncertainty associated with excessive reliance on preprocessing, enhances model stability, and improves the reliability of predictive outcomes, providing a more comprehensive and robust predictive framework for readers.

In this experiment, silt, sand, K, and P were poorly predicted. However, Osayande Pascal Omondiagbe et al. [17] used Adapted-PBT to optimize CNN for predicting soil texture (three soil properties) with significant results for clay (R² = 0.90, RMSE = 4.2), silt (R² = 0.78, RMSE = 11.4), and sand (R² = 0.81, RMSE = 0.3). Additionally, Wan et al. [27] utilized LUCAS to enhance feature extraction for small datasets, achieving good results: available nitrogen (R² = 0.941, RMSE = 3.873), available phosphorus (R² = 0.926, RMSE = 3.684), and available potassium (R² = 0.903, RMSE = 3.422). These studies demonstrated good application results using the LUCAS dataset. Our model requires a large amount of data, and its performance is not satisfactory with small datasets. Our next goal is to utilize the LUCAS dataset to enhance the prediction of soil properties with a small amount of data. Training the Transformer-CNN model using an NVIDIA GeForce GTX 4090 requires 8 ms/epoch, the Transformer model requires 5 ms/epoch, CNN requires 6 ms/epoch, LSTM requires 16 ms/epoch, and ResNet18 requires 33 ms/epoch. Although this may be considered slow compared to traditional preprocessing methods, it is tolerable. The current studies around the LUCAS dataset are monolithic because they are all based on spectral information collected indoors, which is different in future large-scale outdoor studies. Future research should be performed to evaluate the validity of the proposed methods on soil spectra collected in the field with portable spectrometers or drones, which are more challenging than laboratory data. With a focus on reducing computational complexity and enhancing model robustness on the experimental side, there is a need to further investigate the performance of other deep learning algorithms such as semi-supervised learning and self-supervised learning. The effect of the two-dimensional transformation of spectra on the experimental results is also explored, and as soil spectral databases continue to emerge in different parts of the world, we will continue to investigate the effectiveness of our model on other spectral databases.

5. Conclusions

In this paper, we first summarize the previous research on the LUCAS dataset and analyze the challenges of predicting soil properties using the LUCAS spectral database. A deep learning method combining Transformer and CNN is proposed for the simultaneous prediction of soil properties from Vis-NIR spectral data. Experimental validation on the LUCAS dataset shows that by combining the advantages of Transformer and CNN, the method captures long-range dependencies and local features in the spectral data, significantly improving the prediction accuracy and stability of soil properties. Savitzky–Golay smoothing and first-order differential preprocessing methods play an important role in reducing spectral noise and improving model prediction performance. We predicted all 11 soil properties in the LUCAS dataset, and the parallel model outperformed a single model and made better predictions than traditional machine learning. Although this study demonstrated the great potential of the fusion model, there is still room for improvement in computational efficiency, model complexity, and application breadth. Unlike laboratory data, VNIR spectra collected in the field are affected by a variety of environmental factors such as weather, light intensity, and humidity. These factors may introduce higher variability in the data, thus complicating the prediction of soil properties. Based on the favorable results obtained in this study, we will use the more challenging field-collected soil spectra to evaluate our model in future studies.

Future research should continue to optimize the model structure, explore more application scenarios, and promote its use in real agricultural production. Enhancing the prediction capability of soil properties can improve soil management practices, promote sustainable agricultural development, and achieve significant ecological and economic benefits. The Transformer and CNN fusion model proposed in this paper provides an effective and robust new approach for hyperspectral soil property prediction. Through further research and optimization, this method is expected to play a greater role in soil monitoring and agricultural management, helping achieve the goals of healthy soil and sustainable agriculture.