An Independent Validation of SoilGrids Accuracy for Soil Texture Components in Croatia

Abstract

While SoilGrids is an important source of soil property data for a wide range of environmental studies worldwide, there is currently an extreme lack of studies evaluating its accuracy against independent ground truth soil sampling data. This study aimed to provide a comprehensive insight into the accuracy of SoilGrids layers for three physical soil properties representing soil texture components (clay, silt, and sand soil contents) using ground truth data in the heterogeneous landscape of Croatia. These ground truth data consisted of 686 soil samples collected within the national project at a 0–30 cm soil depth, representing the most recent official national data available. The main specificity of this study was that SoilGrids was created based on zero soil samples in the study area, according to the ISRIC WoSIS Soil Profile Database, which is very sparse for the wider surroundings of the study area. The accuracy assessment metrics indicated an overall low accuracy of the SoilGrids data compared with the ground truth data in Croatia, with the average coefficient of determination (R²) ranging from 0.039 for silt and sand to 0.267 for clay, while the normalized root-mean-square error (NRMSE) ranged from 0.362 to 2.553. Despite the great value of SoilGrids in a vast range of environmental studies, this study proved that the accuracy of its products is highly dependent on the presence of ground truth data in the study area.

1. Introduction

Digital soil mapping refers to the collection and analysis of data to produce detailed maps of soil properties such as soil type, texture, and organic matter content [1]. These maps are essential for understanding the physical, chemical, and biological characteristics of soils in a particular area, which is critical for making informed decisions about land use and management [2]. For this reason, digital soil mapping at the country level provides a comprehensive overview of the soil conditions across a nation and allows the development of effective policies to manage this vital resource. Supporting agricultural production is one of the primary reasons for soil mapping’s importance at the country level [3]. In most countries, agriculture is a key source of income and employment, and soil conditions are important for optimizing crop yields and minimizing environmental impacts [4,5]. Soil maps can also indicate areas that are vulnerable to erosion, nutrient depletion, and other factors that can negatively affect crop yield [6]. In addition to supporting agricultural production, soil mapping can also help to mitigate the effects of climate change [7]. Soils are the major component of the global carbon cycle, and soil mapping provides valuable information on soil organic matter content, which is a key indicator of carbon sequestration potential [8]. Similarly, soil mapping is important for managing natural resources such as forests, wetlands, and grasslands. These ecosystems rely on healthy soils to support their functioning and provide valuable ecosystem services such as water regulation and biodiversity conservation [9,10,11]. Soil maps provide information on soil conditions in these areas, which can be used to develop effective management strategies that maintain soil health and support ecosystem functioning. Another important reason for soil mapping is to support land use planning and infrastructure development at the country level [12]. Understanding soil conditions can help to identify areas that are suitable for different land uses, such as agriculture, urban development, or conservation. This information can be used in the planning process and to ensure sustainable development [13], in addition to being able to improve soil health.

Despite the importance of soil mapping at the country level, many countries still lack comprehensive soil maps [14,15]. This is often due to a lack of resources or expertise, as soil mapping can be a complex and time-consuming process [16]. However, technological advances and the availability of global soil databases, such as SoilGrids [17,18,19], have made soil mapping more accessible and cost-effective than ever before. SoilGrids is a global soil mapping platform that uses machine learning algorithms and environmental covariate data to produce high-resolution maps of soil properties, produced by the International Soil Reference and Information Center (ISRIC) [17]. SoilGrids provides a range of data products that provide information on soil properties, including soil texture components (clay, silt, and sand soil contents). This information is essential for the development of sustainable land management practices and can be used by farmers, land managers, and policymakers to make informed decisions about land use [20]. One of the key advantages of SoilGrids is its global coverage. The platform provides soil information for every country in the world, and the data are available at resolutions of up to 250 m [18]. Another advantage of SoilGrids is its accessibility, as all SoilGrids data are available online free of charge. The platform is also user-friendly, with a number of tools and resources available to help users navigate and interpret the data, including its availability on Google Earth Engine [21].

However, there are also some potential issues with using SoilGrids. SoilGrids relies on a range of data sources, including soil profile data, remote sensing data, and climate data, which may be inaccurate or outdated [22]. The accuracy of SoilGrids has been validated via numerous studies comparing its predicted soil properties with ground truth data as a part of its development, with an R² ranging from 0.540 to 0.834 [18]. Additionally, de Sousa et al. [23] determined the root-mean-square error of SoilGrids in the range of 4.6–5.1% for soil organic matter. Poggio et al. [24] produced median model efficiency coefficients within the 0.22–0.73 range based on 9 soil parameters. Besides studies by the SoilGrids developers, as well as many instances in which SoilGrids rasters were used as covariates in machine learning prediction [25,26], there has been no independent comprehensive validation of these products according to independent ground truth soil sampling data. Another potential limitation of SoilGrids is its reliance on environmental covariate data, which may not always accurately capture soil properties. SoilGrids has been developed using global data, and while this approach is useful for broad-scale studies, its application in local conditions may not always be accurate [27]. In addition, SoilGrids is also limited by the quality and quantity of the soil data used to train the machine learning algorithms. Soil data are often limited, particularly in developing countries where access to soil data is scarce, meaning that the accuracy of SoilGrids may be limited due to the lack of adequate training data [28].

The aim of this study is to evaluate to provide an accurate representation of physical soil parameters based on the most recent ground truth soil sampling data in Croatia, according to independent ground truth soil sampling data. Since there are currently no comprehensive studies on SoilGrids’s accuracy using an independent soil sampling dataset, these results are expected to aid in developing a comprehensive and independent outlook on the ability of SoilGrids to provide an accurate representation of physical soil parameters.

2. Materials and Methods

The study workflow for the SoilGrids accuracy analysis in Croatia was divided into three steps: (1) acquisition and preprocessing of SoilGrids data, (2) acquisition of the ground truth soil sampling data, and (3) accuracy assessment of SoilGrids based on the ground truth data (Figure 1). A total of three physical soil properties were evaluated, including clay, silt, and sand soil contents. Despite the availability of several chemical soil properties in the ground truth data, soil texture components were selected for the accuracy assessment of SoilGrids as very stable soil properties [29] with a tendency to remain stable over several years [30,31]. Therefore, the potential effect of temporal discrepancy in the soil sampling was minimized.

2.1. Study Area and Ground Truth Soil Sampling Data

The study area covered the Republic of Croatia, a 56,594 km² area that includes three distinct biogeoregions: the Continental, Alpine, and Mediterranean regions (Figure 2). According to Corine 2018 Land Cover data, the terrestrial land cover classes are dominantly represented by forest and semi-natural areas (55.7%) and agricultural areas (40.1%), while artificial surfaces and wetlands cover 3.8% and 0.4%, respectively. Previous studies in Croatia noted the high heterogeneity of soil properties on a national level, including the components of soil texture [32]. Other soil properties, such as soil organic carbon, total nitrogen [33], and soil organic matter [34] yielded similar observations. Climate conditions are also very heterogeneous in the study area, ranging from a temperate climate with dry and hot summers to a cold climate with no dry season and cold summers, as per the Köppen–Geiger classification by Beck et al. [35] (

Table 1. The coverage of Köppen–Geiger climate classes in the study area according to Beck et al. [35].

Köppen–Geiger Climate Class	Description	Coverage of Study Area (%)
Csa	Temperate, dry summers, and hot summers	12.5
Csb	Temperate, dry summers, and warm summers	1.0
Cfa	Temperate, no dry season, and hot summers	28.4
Cfb	Temperate, no dry season, and warm summers	7.6
Dsb	Cold, dry summers, and warm summers	0.1
Dfa	Cold, no dry season, and hot summers	0.1
Dfb	Cold, no dry season, and warm summers	50.1
Dfc	Cold, no dry season, and cold summers	0.2

, Figure 3).

The ground truth soil sampling data were acquired from the openly available web feature service (WFS) provided by the Ministry of Economy and Sustainable Development of the Republic of Croatia [36]. The field soil sampling was performed between April 2015 and October 2016 as a part of the national project “Change in soil carbon stocks and calculation of total nitrogen and soil organic carbon trends and C:N”. The soil sampling methodology was performed according to the “Soil Sampling Protocol to Certify the Changes of Organic Carbon Stock in Mineral Soils Of European Union” by the Joint Research Centre of the European Commission [37]. According to this protocol, soil samples were georeferenced using the global positioning system (GPS) with a positioning accuracy of a few meters and were distributed as point vector data. Each collected soil sample was collected as a composite of 25 soil sampling points within the sampling grid, representing the aggregated value of clay, silt, and sand soil contents in the specified proximity of each point displayed in Figure 2. A total of 686 soil samples at a 0–30 cm soil depth were filtered from the original 725 samples according to land cover classes which included agricultural areas, forests and seminatural areas, and wetlands as per the CORINE classification [38]. Soil samples collected from artificial surfaces were removed from the analysis as SoilGrids did not include soil data in these areas.

2.2. Acquiring and Preprocessing of SoilGrids Data

SoilGrids data were acquired from the Google Earth Engine SoilGrids 250m v2.0 Application Programming Interface (API) [39]. Clay, silt, and sand soil contents were acquired in the native 250 m spatial resolution and reprojected to WGS 84/Pseudo-Mercator projection (EPSG: 3857) and clipped to the study area. Each soil property was downloaded in 3 layers to match the soil depth of the ground truth data, with 0–5 cm, 5–15 cm, and 15–30 cm data. While SoilGrids provides more soil depth information, these layers were selected to match the 0–30 cm soil information from the ground truth data. The units of SoilGrids data were converted to the units of the ground truth data, as presented in

Table 2. Unit conversion coefficients of the acquired SoilGrids data according to their native units [24].

Soil Parameter	Native Units	Conversion Coefficient	Converted Units
Clay	g/kg	10	g/100 g (%)
Silt	g/kg	10	g/100 g (%)
Sand	g/kg	10	g/100 g (%)

. The harmonized preprocessed SoilGrids layers are displayed in Figure 3.

Figure 4 displays a comparison of the locations of the soil samples from the ground truth data with those used for SoilGrids creation according to the ISRIC World Soil Information Service (WoSIS) Soil Profile Database. It represented a data source of soil samples for the latest SoilGrids products evaluated in this study [24] and was downloaded from the webpage specified by the authors for clay [40], silt [41], and sand [42]. The WoSIS Soil Profile Database included zero soil samples in the study area for SoilGrids, so the ground truth data used in this study represent a fully independent dataset for its accuracy assessment. The temporal range of the field soil sampling from the WoSIS Soil Profile Database samples displayed in Figure 4 ranges from May 1963 to April 2009, but over 80% of them lacked temporal field sampling metadata. Therefore, it was assumed that the ground truth data also represented more recent soil sampling results in the study area.

2.3. Accuracy Assessment of SoilGrids Based on the Ground Truth Soil Sampling Data

To minimize the effect of the unevenness of soil depths from the ground truth soil sampling data and SoilGrids layers, the accuracy assessment was performed according to 5 evaluated layers for each soil property (Figure 5): (1) 0–5 cm layer; (2) 5–15 cm layer; (3) 15–30 cm layer; (4) an average of 0–5 cm, 5–15 cm, and 15–30 cm data; and (5) a weighted average according to the coverage of the ground truth 0–30 cm soil depth, in which 0–5 cm data had a weight of 1, 5–15 cm data had a weight of 2, and 15–30 cm data had a weight of 3.

The three statistical metrics selected for the accuracy assessment were the coefficient of determination (R²), root-mean-square error (RMSE), and normalized RMSE (NRMSE), which are complementary metrics for the accuracy assessment of soil properties as they provide different aspects of accuracy [43,44]. The R² provided an indication of the goodness of fit of the evaluated soil data, the RMSE provided information on the absolute magnitude of the errors, and the NRMSE provided a relative measure of the accuracy of the evaluated SoilGrids data. RMSE is sensitive to outliers, and it emphasizes large errors more than small errors, while NRMSE ensures that the RMSE metric is comparable across different soil properties [45]. Higher R² values and lower RMSE and NRMSE values indicated a proportionally higher accuracy of the SoilGrids data according to the ground truth soil sampling data. The R², RMSE, and NRMSE values for the accuracy assessment of the SoilGrids data according to the ground truth soil sampling data were calculated according to Equations (1)–(3):

$${\mathrm{R}}^2=1-\frac{{\sum }_1^{\mathrm{n}}{\left({\mathrm{y}}_{\mathrm{i}}-{\widehat{\mathrm{y}}}_{\mathrm{i}}\right)}^2}{{\sum }_1^{\mathrm{n}}{\left({\mathrm{y}}_{\mathrm{i}}-{\stackrel{\mathrm{-}}{\mathrm{y}}}_{\mathrm{i}}\right)}^2},$$

(1)

$$\mathrm{RMSE}=\sqrt{\frac{{\sum }_1^{\mathrm{n}}{\left({\mathrm{y}}_{\mathrm{i}}-{\widehat{\mathrm{y}}}_{\mathrm{i}}\right)}^2}{\mathrm{n}},}$$

(2)

$$\mathrm{NRMSE}=\frac{\mathrm{RMSE}}{{\stackrel{\mathrm{-}}{\mathrm{y}}}_{\mathrm{i}}},$$

(3)

where ${\mathrm{y}}_{\mathrm{i}}$ is the ground truth soil sampling data, ${\widehat{\mathrm{y}}}_{\mathrm{i}}$ is the evaluated SoilGrids data, ${\stackrel{\mathrm{-}}{\mathrm{y}}}_{\mathrm{i}}$ is the average of ground truth soil sampling data per soil parameter, and n is the number of ground truth soil samples.

3. Results

The medians and value ranges of the evaluated SoilGrids layers resulted in high variability in comparison with the ground truth soil sampling data. These are presented in violin plots (Figure 6) containing the density information about the ground truth data and evaluated SoilGrids rasters, as well as boxplots, which indicate the median, minimum, maximum, and first and third quartile values [46]. The median values based on the average of 3 SoilGrids layers per soil parameter differed from the medians of the ground truth data by 19.4%, 64.1%, and 84.5% for clay, silt, and sand, respectively. Moreover, the SoilGrids layers had much lower value ranges for the evaluated soil texture components, with a maximum average clay value of 48.04% (78.51% for the ground truth data) and a maximum average silt value of 46.62% (87.89% for the ground truth data). The values of the SoilGrids layers per soil texture component had similar value ranges regardless of the soil depth.

Based on the 686 ground truth soil samples, the SoilGrids layers had a low average accuracy represented as R², which varied from 0.039 for silt and sand to 0.267 for clay (

Table 3. The accuracy of SoilGrids layers according to the ground truth soil sampling data.

Soil Property	Accuracy Metric	Evaluated SoilGrids Layers
		0–5 cm	5–15 cm	15–30 cm	Average	Weighted Average
Clay (%)	R2	0.264	0.258	0.260	0.267	0.266
	RMSE	13.173	12.866	12.016	12.594	12.398
	NRMSE	0.378	0.370	0.345	0.362	0.356
Silt (%)	R2	0.034	0.041	0.038	0.039	0.040
	RMSE	24.311	23.790	25.698	24.570	24.799
	NRMSE	0.451	0.441	0.477	0.456	0.460
Sand (%)	R2	0.037	0.035	0.043	0.039	0.040
	RMSE	29.324	28.279	28.744	28.742	28.642
	NRMSE	2.605	2.512	2.553	2.553	2.544

The statistical metrics indicating the highest accuracy per evaluated soil property are bolded.

). The average NRMSE ranged from 0.362 for clay to 2.553 for sand soil content. Overall, the SoilGrids did not show a tendency for any individual native soil depth to be superior in terms of accuracy, while all soil parameters resulted in slightly higher accuracy for either the 5–15 cm or 15–30 cm layer. The average and weighted average only performed better for clay compared with the most accurate individual layers of evaluated soil parameters. The accuracy of the weighted average values was particularly less affected by the lower accuracy of the 0–5 cm data for silt in comparison with the 5–15 cm and 15–30 cm soil layers. The absolute accuracy represented by the RMSE and NRMSE values and the relative accuracy represented by R² generally matched in terms of ranking the evaluated SoilGrids layers per soil parameter. A slight exception occurred for clay and sand, which had similar R², RMSE, and NRMSE values for individual soil layers, but their mutual rankings in terms of absolute and relative accuracy were in disagreement. The accuracy assessment of the native SoilGrids layers according to the Köppen–Geiger climate classes and major CORINE Land Cover 2018 classes is presented in Appendix A. Overall, the Csb climate class resulted in the highest accuracy for all 3 evaluated soil parameters, but its coverage of 1.0% of the study area possibly skewed the results. Among other climate classes, consistently low accuracy was achieved for the temperate Csa class with hot and dry summers. The land cover classes had a minor effect on the accuracy per soil property, while no consistent effect was observed for their respective soil depth layers.

Figure 7 displays the value distributions of the square residuals according to the ground truth soil sampling data. All evaluated soil properties showed a very similar distribution of values across the soil depths, while the medians and value ranges per soil property only slightly differed. The spatial distributions of the square residuals according to the ground truth soil samples from which they were calculated are presented in Figure 8. The largest squared residuals for clay were observed at distinct locations in the entire study area, which was not related to the biogeoregions. The northern part of Croatia, dominantly present in the Continental biogeoregion, contained the largest residuals for silt and sand.

4. Discussion

The accuracy assessment of SoilGrids, as well as future potential soil parameter prediction products, can generally be conducted using cross-validation and independent validation. Out of the two, cross-validation’s accuracy is well documented as part of the scientific papers on SoilGrids 1km [17], SoilGrids 250m [18], and SoilGrids 250m v2.0 [24], of which the latter was evaluated in this study. This approach involved dividing the soil sampling data into training and test sets, using the training set to develop the SoilGrids model, and testing its accuracy using the test set. These soil sampling data sets were created by harmonizing the data from several national and international agencies [17], as comparable to the source of the independent ground truth soil sampling dataset used in this study. The first official SoilGrids product resulted in an amount of explained variation values ranging from 22.9% to 50.5%, based on the 5-fold cross-validation [17]. While this metric was based on the RMSE values, it included additional parameters for its calculation [47], which does not allow objective comparison with the accuracy assessment results from this study. However, the comparable metrics were calculated in SoilGrids 250m [18], which is also represented in the same spatial resolution as the products analyzed in this study. The results of repeated 10-fold cross-validation ranged from 0.635 to 0.834, represented by R² for the soil properties evaluated in this study, which is significantly higher than the results from the independent evaluation in this study. The RMSE values for the physical soil parameters were in the range of 9.5–10.9, which is comparable to the accuracy results for clay in this study, while silt and sand had more than double the RMSE values of these. The 10-fold cross-validation accuracy assessment results for SoilGrids 250m v2.0 [24], which was evaluated in this study, had higher RMSE values for clay, silt, and sand in comparison with the previous version of SoilGrids. However, these results ranging from 13.0–18.0 are comparable to the results from this study, while silt and sand still performed with lower accuracy than the cross-validation results. Their results across the soil depths were mutually similar with slightly more accurate predictions in the top 0–5 cm soil layer, which is contrary to the observations in this study.

The independent validation involved the accuracy assessment of SoilGrids according to soil sampling data, which were not used for the prediction of soil properties as a part of SoilGrids’ creation. This approach provided an unbiased estimate of the accuracy of SoilGrids, and it is essential to ensure that the model is valid for use in the study area [48]. One of the critical factors in accuracy assessment is the selection of appropriate ground truth data [49]. The ground truth soil sampling data used for the validation of SoilGrids data in this study met the criteria of representativeness of the soil variability within the study area. They were also collected using a robust sampling design that accounts for the spatial variability of soil properties [36]. However, the main specificity of this study was that SoilGrids was created based on zero soil samples in the study area, according to the ISRIC WoSIS Soil Profile Database. Additionally, the minor limitation of the used approach was that the soil sampling depths of the ground truth data did not directly match the soil depths of the SoilGrids layers. The high heterogeneity of the landscape and climate classes in the study area, as well as the presence of three distinct biogeoregions, also potentially affected the accuracy assessment. The 250 m spatial resolution of SoilGrids likely disabled the representation of local variations in soil properties that were otherwise contained in the ground truth data. The cause of such challenges in the independent accuracy assessment of SoilGrids is the lack of a comprehensive global soil sampling program. Global soil sampling programs, such as the Global Soil Partnership [50], the Harmonized World Soil Database [51], and the SoilGrids program, provide valuable data for validation. The spatial coverage and density of soil sampling data are often limited, which can affect the accuracy of SoilGrids predictions, particularly in areas with high soil variability. Despite the great value of SoilGrids in a vast range of environmental studies, this study proves that the accuracy of its products is highly dependent on the presence of ground truth data in the study area.

The major challenge of achieving the uniform accuracy of SoilGrids products remains, as recent and independent soil sampling data are scarce, which is the reason this study is among the very few of its kind. SoilGrids has often been used in previous studies as an important soil covariate for total carbon stock estimates [52], soil mapping [25], and cropland suitability prediction [26], but its reliability has not been considered. Soil properties can vary significantly at different spatial scales, from small-scale variations within fields to large-scale variations across continents [53,54,55]. Therefore, the accuracy of SoilGrids predictions should be assessed at multiple spatial scales to ensure that it is suitable for various applications. In heterogeneous landscapes, such as Croatia, where soil properties can vary widely over short distances [56], the reliability of SoilGrids predictions may be a concern. Additionally, the temporal offset between the ground truth soil sampling data between 2015 and 2016 from the soil sampling of the training data used for SoilGrids’ creation could provide further discrepancy in future studies in evaluating chemical soil properties [57]. The potential upgrades of this study in terms of accuracy assessment according to soil types, biogeoregions, climate classes, and land cover classes could provide a more thorough insight into SoilGrids and potential similar future programs. Future studies will also aim to perform digital soil mapping of physical soil parameters using ground truth soil sampling data on a national level. This will likely provide a supplement to SoilGrids for areas with existing soil samples not included in the training of SoilGrids, as was the case in Croatia.

5. Conclusions

The results of the SoilGrids accuracy evaluation in Croatia based on independent ground truth soil sample data show a low level of accuracy for all three evaluated components of soil texture. Despite low resulting R² values, the RMSE values were similar to those of the cross-validation as part of the SoilGrids creation process. The main feature of this validation was that SoilGrids used zero soil samples in the study area, based on the ISRIC WoSIS Soil Profile Database. This implies that a similar accuracy for soil texture components in the 0–30 cm soil layer from SoilGrids can be expected in other study areas that were sparsely represented in soil samples during the training process and have a high heterogeneity in terms of landscape and climate properties. Moreover, future global mapping initiatives based on environmental data might encounter similar unevenness in terms of mapping accuracy in areas sparsely represented in the ground truth data. Additionally, the time frames of the field sampling of the ground truth data and data used to create SoilGrids did not match, as the ground truth data represented significantly more recent field data. This study minimized this potential problem by using the most recent official ground truth soil sampling data in Croatia, as well as the most recent SoilGrids data. The minor limitation of this study was a minor discrepancy in the soil sampling depths of the ground truth data (0–30 cm) and the native soil depth of the SoilGrids layers, which were segmented into 0–5 cm, 5–15 cm, and 15–30 cm soil depths. Finally, due to the high heterogeneity of the landscape, climate, and biogeoregions in the study area, the variability in soil texture components in Croatia likely could not be contained in the globally trained SoilGrids data at a 250 m spatial resolution.

Despite a few limitations, this study represents an important contribution to the very few studies that have evaluated the accuracy of SoilGrids using independent soil sampling data. The proposed approach of evaluating SoilGrids in various study areas globally, as well as for similar global mapping projects, can be systematically performed with the sole requirement of independent and reliable ground truth data. SoilGrids currently represents the most comprehensive global open-source soil data, and its reliability affects scientific studies in several environmental disciplines. With respect to the SoilGrids accuracy assessment results from this study and their potential limitations, further studies are needed to provide a reliable and independent conclusion on SoilGrids’s accuracy in more locations, globally. Besides an independent validation of SoilGrids in study areas with varying densities of soil samples from the ISRIC WoSIS Soil Profile Database, the inclusion of soil properties from deeper soil layers will likely aid in developing a more complete knowledge of the accuracy of SoilGrids and similar future initiatives.