Estimating Rootzone Soil Moisture by Fusing Multiple Remote Sensing Products with Machine Learning

Abstract

This study explores machine learning for estimating soil moisture at multiple depths (0–5 cm, 0–10 cm, 0–20 cm, 0–50 cm, and 0–100 cm) across the coterminous United States. A framework is developed that integrates soil moisture from Soil Moisture Active Passive (SMAP), precipitation from the Global Precipitation Measurement (GPM), evapotranspiration from the Ecosystem Spaceborne Thermal Radiometer Experiment on Space Station (ECOSTRESS), vegetation data from the Moderate Resolution Imaging Spectroradiometer (MODIS), soil properties from gridded National Soil Survey Geographic (gNATSGO), and land cover information from the National Land Cover Database (NLCD). Five machine learning algorithms are evaluated including the feed-forward artificial neural network, random forest, extreme gradient boosting (XGBoost), Categorical Boosting, and Light Gradient Boosting Machine. The methods are tested by comparing to in situ soil moisture observations from several national and regional networks. XGBoost exhibits the best performance for estimating soil moisture, achieving higher correlation coefficients (ranging from 0.76 at 0–5 cm depth to 0.86 at 0–100 cm depth), lower root mean squared errors (from 0.024 cm³/cm³ at 0–100 cm depth to 0.039 cm³/cm³ at 0–5 cm depth), higher Nash–Sutcliffe Efficiencies (from 0.551 at 0–5 cm depth to 0.694 at 0–100 cm depth), and higher Kling–Gupta Efficiencies (0.511 at 0–5 cm depth to 0.696 at 0–100 cm depth). Additionally, XGBoost outperforms the SMAP Level 4 product in representing the time series of soil moisture for the networks. Key factors influencing the soil moisture estimation are elevation, clay content, aridity index, and antecedent soil moisture derived from SMAP.

1. Introduction

Accurate knowledge of soil moisture is crucial for numerous applications, including agricultural water management [1,2], water resources sustainability [3], weather forecasting [4,5], climate modeling [6,7], wildfire prediction [8,9], and monitoring of floods and droughts [10,11]. Rootzone soil moisture is particularly important because it significantly influences plant growth [12], water availability [13], and ecological processes [14,15]. However, obtaining reliable rootzone soil moisture data at fine spatial resolutions (10–100 m grid cells) across large regions (10–100 km extents) remains challenging.

Several microwave satellite missions provide soil moisture nearly globally, including Soil Moisture Active Passive (SMAP) [16], the Advanced Scatterometer [17], Soil Moisture and Ocean Salinity (SMOS) [18], and the Advanced Microwave Scanning Radiometer for the Earth Observing System—Eos (AMSR-E) [19]. However, these datasets have limitations, including coarse spatial resolutions (often ranging from 9 to 60 km) and shallow depths of measurement (around 5 cm) [20]. SMAP also provides rootzone soil moisture estimates by merging the remote sensing information with modeling, but the spatial resolution remains coarse. Downscaling techniques can improve the spatial resolution of microwave soil moisture products. For example, Tagesson et al. [21] used the land surface temperature and vegetation dryness index to disaggregate SMOS soil moisture from ~40 km resolution to ~5 km resolution in West Africa. Das et al. [22] used the Sentinel-1A and Sentinel-1B synthetic aperture radar data to disaggregate SMAP L-band radiometer measurements from ~40 km to 3 km and 1 km. Wei et al. [23] utilized the Moderate Resolution Imaging Spectroradiometer (MODIS) and a digital elevation model (DEM) with a gradient boosting decision tree to downscale SMAP soil moisture estimates from a 36 km to 1 km resolution across the Tibetan Plateau. Nuñez et al. [24] used MODIS products, sand fraction, and elevation to downscale AMSR2 soil moisture estimates from 25 km to 1 km in Puerto Rico. Vergopolan et al. [25] used a hyper-resolution land surface model, a radiative transfer model, and a Bayesian scheme to merge and downscale SMAP 36 km soil moisture to 30 m, and evaluated the results using four watersheds in the United States. Fischer [26] used topographic attributes and vegetation indices to downscale to 30 m, 10 m, and 3 m resolutions at Maxwell Ranch in Colorado. Fewer soil moisture downscaling methods have considered soil moisture beyond the top 5 cm. Dumedah et al. [27] used Disaggregation based on Physical and Theoretical scale Change (DisPATCh) to downscale satellite soil moisture data and estimate rootzone soil moisture at a spatial resolution of 1 km and depths of 0–30 cm, 30–60 cm, and 60–90 cm.

Soil moisture can also be estimated using optical and thermal data from satellites such as Landsat and the MODIS. These methods typically characterize the relationship between soil moisture and water-stressed vegetation using the visible and near-infrared bands, and they use the thermal infrared band to derive the relationship between rootzone soil moisture and soil thermal properties [28]. The methods provide soil moisture estimates at spatial resolutions (30 m to 1 km) over large spatial extents (185 km to 2330 km). The methods include the triangle and trapezoid methods [29,30,31,32,33], drought index method [34,35,36,37,38], thermal inertia method [39], single optical methods [40,41], energy balance methods [42,43,44,45,46], and synergistic optical/thermal and microwave methods [47,48,49]. Although acceptable accuracies have been reported, optical and thermal remote sensing methods have limitations. The triangle method, for instance, requires a flat surface, a large number of pixels, and a wide range of vegetation and moisture conditions, making it less effective in non-flat terrain and somewhat subjective in determining the warm edge and vegetation limits [29]. The drought index method calculates soil moisture retroactively and neglects temperature and rainfall effects on vegetation [28]. The thermal inertia method assumes consistent soil properties in both horizontal and vertical directions [28]. Optical methods can be precise for soil samples but are influenced by various factors like vegetation, atmospheric conditions, and topography, and they rely on empirical relationships [28].

An alternative soil moisture estimation approach fuses microwave remote sensing, optical and thermal remote sensing products, and ancillary datasets [50,51,52,53,54,55,56]. The ancillary datasets include variables that can impact soil moisture including antecedent soil moisture [57,58], landcover [59,60], soil properties [61,62], meteorological variables such as precipitation and land surface temperature [63,64], and topographic indices [65,66,67,68]. Many data fusion methods use machine learning algorithms [69,70,71,72,73,74], which are data-driven approaches that learn patterns and relationships from data without making assumptions about the processes that govern soil water dynamics [75,76]. Machine learning can merge large volumes of data from various sources, including in situ measurements, meteorological variables, and remote sensing datasets [69,77]. Machine learning algorithms can also perform feature selection, automatically identifying the inputs that are most relevant for estimating soil moisture [78]. Machine learning models have shown strong correlations between in situ soil moisture observations and the predicted soil moisture values [76,79]. For example, Abowarda et al. [70] employed a random forest (RF) model to produce surface soil moisture at a 30 m resolution for the Haihe Basin in northern China. SMAP Level 4 surface soil moisture was incorporated as the background field, and Landsat and MODIS data were used to determine the Normalized Difference Vegetation Index (NDVI), surface albedo, and land surface temperature. Precipitation and soil texture were also used as model inputs. The study reported root mean squared error (RMSE) values ranging from 0.031 to 0.050 cm³/cm³. Singh and Gaurav [80] used a feed-forward artificial neural network (ANN) with nine input variables derived from the Sentinel-1 and Sentinel-2 satellites as well as topographic characteristics from a DEM to estimate surface soil moisture at a spatial resolution of 60 m over the Kosi Fan in the Himalayan Foreland in the north Bihar plain, India. The study reported an RMSE value of 0.04 cm³/cm³. They also compared the ANN results to ten other machine learning methods and found that the ANN was most accurate. Zhao et al. [74] downscaled SMAP passive surface soil moisture (SSM) (0–5 cm depth) from 36 km to 1 km using a random forest (RF) method, reporting an R above 0.95 and an RMSE of 0.022 cm³/cm³. Fathololoumi et al. [73] also employed an RF method to downscale the Advanced Scatterometer (ASCAT) Soil Water Index (SWI) for the top 5 cm depth from a 10 km to 30 m resolution across three diverse field sites in the USA, France, and Iran, achieving RMSE values ranging from 0.072 to 0.172 cm³/cm³. Fewer studies have explored machine learning methods for rootzone soil moisture estimation. Fuentes et al. [81] used a deep learning approach to fuse SMAP, Sentinel-1, MODIS products (surface reflectance, land surface temperature, and land cover), and gridded soil properties to estimate soil moisture at a 90 m resolution for multiple depths. They used a multilayer perceptron model for the surface (0–10 cm) soil moisture and recurrent neural network model for 0–30 cm and 30–60 cm soil moisture at the Ozflux and Oznet networks across Australia. The study reported RMSE values of 0.073 cm³/cm³ for 0–10 cm and 0.070 cm³/cm³ for 0–30 cm and 30–60 cm. Karthikeyan and Mishra [82] used extreme gradient boosting (XGBoost) to estimate soil moisture at 5, 10, 20, 50, and 100 cm depths at a 1 km resolution and reported an unbiased root mean squared error (ubRMSE) of less than 0.040 cm³/cm³ for most locations.

Despite these recent advancements in using data fusion to estimate soil moisture, research gaps remain. Many studies have focused on surface soil moisture rather than rootzone soil moisture, and most studies have considered relatively small spatial extents. Prior studies have also used relatively few features as inputs and employed a single machine learning algorithm [70,80,81,82] without comparing its performance to other machine learning algorithms.

The primary objective of this study is to estimate soil moisture at five depths (0–5 cm, 0–10 cm, 0–20 cm, 0–50 cm, and 0–100 cm) using five machine learning methods (feed-forward ANN, random forest, XGBoost, Catboost, and LightGBM). The methods fuse microwave soil moisture data from SMAP, optical and thermal evapotranspiration products from the ECOSTRESS, vegetation data from the MODIS, precipitation data from the GPM, soil properties from gNATSGO, and land cover information from NLCD. The methods aim to estimate soil moisture for unobserved locations across the contiguous U.S. (CONUS). The machine learning methods are evaluated across eight in situ soil moisture networks that span arid to humid regions. This research also assesses the importance of individual predictor variables on the estimation of soil moisture.

2. Materials and Methods

2.1. Datasets

2.1.1. In Situ Soil Moisture Data

The in situ soil moisture data for training and evaluating the machine learning estimates were obtained from the freely available International Soil Moisture Network (ISMN). Only soil moisture observations for 0–102 cm depth were utilized as very few stations have deeper observations (approximately 0.1%). All datasets are available at the 1 h time step. To combine the datasets, soil moisture was estimated for five consistent depth ranges (0–5 cm, 0–10 cm, 0–20 cm, 0–50 cm, and 0–100 cm) using the weighted average method described by Gao et al. [41] and Liu et al. [83]. Non-uniform depth increments were used to align with the available SMAP products (0–5 cm and 0–100 cm) and to emphasize near surface conditions where more roots and variability occur.

The study period (January 2019 to December 2022) was selected based on the combined availability of all data used. Among the 1430 stations in CONUS, 801 stations are available for this period and have soil moisture observations from 0 to 102 cm. All soil moisture values flagged by the ISMN [84], including those under frozen conditions and those outside the expected soil moisture range (<0.0 cm³/cm³ or >0.6 cm³/cm³), were excluded from the analysis. The final in situ dataset includes soil moisture measurements from 731 stations. These stations belong to eight different operational networks (

Table 1. Number of stations utilized from each soil moisture network.

Network	Stations	Reference
Atmospheric Radiation Measurement Climate Research Facility (ARM)	17	Cook [85]
Cosmic-Ray Soil Moisture Observing System (COSMOS)	29	Zreda et al. [86]
AMERIFLUX	3	Baldocchi et al. [87]
Roaring Fork Observation Network (iRON)	8	Osenga et al. [88]
Soil Climate Analysis Network (SCAN)	168	Schaefer et al. [89]
Snow Telemetry (SNOTEL)	359	Fleming et al. [90]
Texas Soil Observation Network (TxSON)	40	Caldwell et al. [91]
U.S. Climate Reference Network (USCRN)	107	Bell et al. [92]

). A higher density of gages occurs in the western U.S. due to the abundance of SNOTEL sites in that region (Figure 1).

2.1.2. Satellite Soil Moisture Data

The seventh version of the SMAP Level-4 (SPL4SMGP.007) [93] surface soil moisture (SSM) (0–5 cm), rootzone soil moisture (RZSM) (0–100 cm), and profile soil moisture (PSM) (0-bedrock depth) products were used as inputs to the machine learning algorithms (

Table 2. Datasets used in this study and their spatial and temporal resolutions. Most of the satellite data were obtained and processed using the Land Processes Distributed Active Archive Center (LPDAAC) (https://lpdaac.usgs.gov/, accessed on 1 June 2023) and Google Earth Engine (GEE) [95,96].

Data Type	Variables	Product/Access	Spatial Resolution	Temporal Resolution	Reference
In Situ Soil Moisture	0–5 cm 0–10 cm 0–20 cm 0–50 cm 0–100 cm	Soil moisture/ ISMN	Point or field measurements	Hourly	Dorigo et al. [97]
Satellite Soil Moisture	SSM (0–5 cm) RZSM (0–100 cm) PSM (0–bedrock)	SMAP SPL4SMGP.007/ GEE	9 km	3 h	Reichle et al. [93]
Landcover and Vegetation	NLCD	NLCD 2019/GEE	30 m	Static	Dewitz [98,99]
	NDVI EVI	MOD13Q1.061/ LPDAAC	250 m	16 Days	Didan [100]
	LAI fPAR	MCD15A3H.061/ LPDAAC	500 m	4 Days	Myneni et al. [101]
Soil	Sand Silt Clay Organic Matter Bulk Density Electrical Conductivity pH Depth to Restrictive Layer	Soil layers/ USDA-NRCS (gNATSGO)	30 m	Static	Soil Survey Staff [102]
Weather and Climate	Precipitation Measurement	GPM_3IMERGHH/GEE	11 km	Half-hourly	Huffman et al. [103]
	Instantaneous LST	ECO_L2_LSTE v002/LPDAAC	70 m	Varies	Hook and Hulley [104]
	Instantaneous ET	ECO3ETPTJPLv001/LPDAAC	70 m	Varies	Hook and Fisher [105]
	Instantaneous ESI, Instantaneous PET	ECO4ESIPTJPLv001/LPDAAC	70 m	Varies	Hook and Fisher [105]
	Aridity Index AI	Climate Database v3/CGIAR-CSI	1 km	Static	Zomer et al. [106]
Topography	Elevation Slope Aspect Hillshade	SRTMGL1 v003/ GEE	30 m	Static	NASA-JPL [107]
	mTPI	Global SRTM mTPI/GEE	270 m	Static	Theobald et al. [108]

). These products are generated using a land data assimilation system that combines satellite-based L-band brightness temperature measurements, precipitation observations, and land surface modeling [94]. The SMAP Level-4 products were used because they provide complete spatial and temporal coverage and include soil moisture values throughout the rootzone.

Higher antecedent soil moisture leads to slower initial infiltration rates and more rapid declines in infiltration rates through time [109]. Thus, antecedent moisture may be predictive of current moisture. The SMAP surface, root zone, and profile soil moisture data were used to calculate antecedent soil moisture values for each hour using a moving average with windows of 1 day, 3 days, 7 days, and 14 days.

2.1.3. Land Cover and Vegetation Data

The 2019 National Land Cover Database (NLCD) was used to characterize the land cover type. The 2019 dataset was used because it aligns with the start of the study period and contained the most recent data available when the analysis was performed. NLCD contains 16 landcover classes at a spatial resolution of 30 m (https://www.mrlc.gov/data/nlcd-2019-land-cover-conus, accessed on 1 June 2023) [99]. To characterize the density and photosynthetic activity (greenness) of the vegetation cover, the following indices were used: Normalized Difference Vegetation Index (NDVI), Enhanced Vegetation Index (EVI), Leaf Area Index (LAI), and Fraction of Photosynthetically Active Radiation (fPAR). The NDVI and EVI were obtained from MODIS Vegetation Indices (MOD13Q1) Version 6.1 (Table 2). Higher NDVI and EVI values indicate thicker and/or greener vegetation. The NDVI uses the red and near infrared bands, while the EVI also uses the blue band. The EVI is less sensitive to atmospheric conditions than the NDVI and is therefore preferred if aerosol content is high or soil/background influences are significant [110]. The LAI and fPAR were obtained from MODIS Vegetation Indices (MCD15A3H) Version 6.1 (Table 2). The LAI is the total one-sided green leaf surface area per unit ground area, while the fPAR is the fraction of photosynthetically active radiation that is absorbed by vegetation [111]. The fPAR is an indicator of the water, energy, and carbon balance that plants require for photosynthesis [112]. The LAI typically ranges from 0 for no vegetation to >5 for dense forests. The fPAR ranges between 0 for no vegetation and 1 for dense, healthy vegetation [111].

2.1.4. Soil Data

The Gridded National Soil Survey Geographic (gNATSGO) dataset was used to obtain percent sand, silt, and clay, organic matter, bulk density, electrical conductivity (EC), pH, and depth to restrictive layer (Table 2). EC is an indicator of soil salinity, which can hinder water uptake by plants, and soil pH can affect nutrient availability and thus ET [113,114]. All soil properties except the depth to restrictive layer were calculated for the five depth ranges: 0–5 cm, 0–10 cm, 0–20 cm, 0–50 cm, and 0–100 cm. gNATSGO is a composite of the Soil Survey Geographic (SSURGO) dataset (mostly 1:24,000 scale), State Soil Geographic 2 (STATSGO2) dataset (1:250,000 scale), and the detailed Raster Soil Surveys (RSS) dataset [115]. The gNATSGO dataset was obtained from the Natural Resources Conservation Service (https://www.nrcs.usda.gov/resources/data-and-reports/gridded-national-soil-survey-geographic-database-gnatsgo, accessed on 1 June 2023).

2.1.5. Weather and Climate Data

The weather and climate were characterized using precipitation, land surface temperature (LST), evapotranspiration (ET), the evaporative stress index (ESI), potential evapotranspiration (PET), and the aridity index (AI). Precipitation data were obtained from Integrated Multi-Satellite Retrievals for GPM (IMERG) Final Precipitation Level 3 Half Hourly (GPM_3IMERGHH) (Table 2). The GPM was chosen because it has been applied in other soil moisture studies [116,117,118] and has high temporal resolution [103], which helps quantify the short-term precipitation effects on soil moisture. The GPM data were used in moving averages to calculate antecedent precipitation for 6 h, 1 day, 3 days, 7 days, and 14 days (denoted as P6H, P1DAY, P3DAY, P7DAY, and P14DAY, respectively).

The LST, ET, ESI, and PET were obtained from the ECOSTRESS (Table 2). The LST product is derived from five thermal infrared bands. The actual ET product estimates instantaneous ET using the Priestly–Taylor Jet Propulsion Laboratory algorithm [105], which uses a series of eco-physiological scaling functions to reduce the potential ET to the actual ET [119]. The ESI is the ratio of ET and PET, which is an indicator of plant water stress [119].

The AI was obtained from the Global AI and PET Database v3 (Global-AI_PET_v3), which was developed by Trabucco and Zomer [120] (Table 2). The AI is defined as the ratio of the mean annual precipitation to mean annual PET. The PET is based on the FAO Penman–Monteith reference crop evapotranspiration equation [121]. The data were obtained from CGIAR-Consortium for Spatial Information (CGIAR-CSI, https://csidotinfo.wordpress.com, accessed on 1 June 2023).

2.1.6. Topographic Data

The elevation, slope, aspect, and hillshade values were determined from the Shuttle Radar Topography Mission Global 1-arc second (SRTMGL1) DEM in Google Earth Engine (GEE). Higher elevations tend to have lower temperatures, which reduce ET and increase soil moisture [68]. Lower slopes can reduce lateral hydraulic gradients, promoting higher rootzone soil moisture [122]. Aspect affects insolation and thus ET and soil moisture [66,67,123]. Hillshade describes the relative shading of a location and depends on the variations in elevation across the landscape as well as the sun’s azimuth and altitude angles. Higher hillshade values indicate more shading [124], which can affect soil moisture [125,126,127]. For simplicity and consistency, 270° (due south) and 45° were used for the azimuth and altitude angles, respectively, for all locations. The Shuttle Radar Topography Mission multi-scale Topographic Position Index (SRTM_mTPI) was used to measure the elevation of a location relative to its surrounding area (Table 2). mTPI is calculated by subtracting the mean elevation of a 3 × 3 pixel neighborhood from the elevation of the central point (the point of interest). It distinguishes peaks, ridges, plains, and valleys [108]. Other topographic attributes such as the drainage area and topographic wetness index have been shown to influence soil moisture [26,127] but were not included because they were not available in GEE.

2.1.7. Data Preprocessing

All remote sensing datasets were projected to the NAD_1983_2011 CONUS Albers projection and resampled to 70 m to match the ECOSTRESS resolution. For example, 30 m topographic attributes were resampled to 70 m using inverse distance weighting. The 9 km SMAP data were resampled to 70 m using the nearest neighbor method. The nearest neighbor approach retains the same soil moisture value across each subdivided 9 km grid cell. Then, the values from the resampled products were obtained at each in situ soil moisture location. The temporal resolution was also based on the ECOSTRESS. The ECOSTRESS had the most missing data, so only dates with available ECOSTRESS data were considered in the study. The missing values in the other datasets were estimated by linearly interpolating in time. For example, the LAI is available every four days, so it was linearly interpolated to create hourly values and these hourly values were resampled to match the ECOSTRESS timestamps. The final dataset has 42 columns. The columns are associated with the 41 predictor (input) variables including a categorical depth column that shows the depth range associated with the in situ soil moisture, and the in situ soil moisture at that depth (the dependent variable). This structure allows the depth range to have its own set of input variables in the machine learning models. The dataset has 72,233 rows, where each row represents a station and time.

The Python 3.11 library sklearn.preprocessing was used to preprocess the data [128]. The categorical depth columns (0–5 cm, 0–10 cm, 0–20 cm, 0–50 cm, and 0–100 cm) and NLCD (landcover type) were encoded using LabelEncoder and OneHotEncoder, respectively [129]. LabelEncoder assigns a unique numerical value to categorical data, and OneHotEncoder creates a new binary feature for each possible value of the categorical feature.

2.2. Machine Learning Algorithms

Five machine learning methods were used: feed-forward ANN, RF, XGBoost, Catboost, and LightGBM. The methods were utilized as regression tools (they can also be used as classification tools). These machine learning methods were selected because they are well suited for modeling complex nonlinear relationships, handling high-dimensional data, processing large datasets efficiently, and capturing variable interactions [130,131,132]. All five methods have been used previously for estimating soil moisture in some manner [82,133,134,135,136,137]. The RF, XGBoost, Catboost, and LightGBM methods are used to determine the importance of each predictor variable to the prediction [128,129,138,139].

2.2.1. Feed-Forward Artificial Neural Network (ANN)

ANNs [137] are inspired by the structure and functioning of the human brain. An ANN consists of interconnected nodes or neurons that are organized into layers. An input layer receives preprocessed data, one or more hidden layers process the data through weighted connections, and an output layer generates the predictions. In the feed-forward ANN, the information flows from the input layer to the output layer without any feedback loops. Activation functions, which are applied to each neuron’s output, introduce nonlinearity to the network, enabling it to capture more complex relationships in the data. L2 regularization is used to prevent overfitting in the machine learning model [138]. During the training process, the weights of the connections between neurons are adjusted iteratively through a backpropagation process to minimize the discrepancy between the predicted outputs and the actual target values [139]. The optimization is achieved using a gradient descent algorithm, which updates the weights to reduce the prediction error as quantified by the loss function.

2.2.2. Random Forest (RF)

RF uses decision trees to make predictions [131]. The algorithm first selects a random sample of the training data with replacement. This means that some observations may be included multiple times in the sample, while others may not be included at all. A decision tree is then grown on each sample. When growing a decision tree, the algorithm randomly selects a subset of the predictor variables to consider at each node. This helps to ensure that the trees are diverse and not too correlated with each other. The predictions of the individual trees are then averaged to produce the final prediction. The averaging process improves the robustness and generalization of the model as it reduces the impact of individual tree outliers or overfitting [131]. An importance score (indicating the relative importance of a given predictor variable) is determined by averaging the reduction in impurity or the decrease in accuracy when the feature values are permuted across all trees [131].

2.2.3. Extreme Gradient Boosting (XGBoost)

Like RF, XGBoost is an ensemble learning algorithm that utilizes decision trees. However, XGBoost employs a gradient boosting framework where the decision trees are sequentially trained with each tree aiming to correct the errors made by the preceding ones. This sequential training process, combined with regularization techniques, focuses on minimizing the residual errors and enhancing predictive performance. Thus, unlike RF, XGBoost trees are not constructed independently, and strong interplay occurs between the trees [132]. The importance score for each predictor variable is calculated based on the number of times a feature is used to split the data and the associated gain in model accuracy [132].

2.2.4. Categorical Boosting (CatBoost)

CatBoost [140] is a gradient boosting decision tree algorithm like XGBoost, but it differs from XGBoost in its approach to training weak learners. CatBoost uses a greedy algorithm (a greedy algorithm iteratively makes the most optimal local decision at each step with the objective of eventually converging to the global optimum) to effectively combine categorical features and their interactions, and it utilizes a prior value to reduce noise from infrequent categories [140,141]. This allows CatBoost to learn more complex relationships between categorical inputs and the target variable [140]. CatBoost also employs ordered boosting, which trains a model for each sample in the training dataset to estimate the gradient of the loss function [140]. These gradient estimates are then aggregated to construct the final model. Ordered boosting improves gradient estimation precision and reduces the risk of overfitting [141]. The importance score is determined using the change in the loss function when features are included or permuted [140].

2.2.5. Light Gradient Boosting Machine (LightGBM)

LightGBM [142] is a gradient boosting decision tree algorithm that is designed to be fast and memory-efficient. It is similar to XGBoost but incorporates techniques to enhance performance. One of the key features of LightGBM is gradient-based one-side sampling, which selectively excludes data instances with small gradients to reduce the sample size and computational demands [142]. LightGBM also integrates exclusive feature bundling, which groups highly correlated features together, reducing the number of input variables that need to be considered during decision tree growth. This further enhances computational efficiency [142]. The importance score is determined by calculating the overall improvement in accuracy from all splits that involve each feature across all trees in the model [142].

2.3. Model Training and Evaluation

All five machine learning methods are provided the same 41 input (predictor) variables. The predictor variables include the satellite soil moisture, landcover/vegetation, soil, weather/climate, and topographic variables in Table 2. The outputs for each method are the predicted soil moisture (${\theta }_{pred}$) for the five depths: 0–5 cm, 0–10 cm, 0–20 cm, 0–50 cm, and 0–100 cm. The models are trained and evaluated using the in situ soil moisture networks.

The in situ dataset was divided into 70% for training/validation and 30% for testing. Of the 70% used for training/validation, a further split was made into training (~80%) and validation (~20%) datasets. Additionally, a 5-fold cross-validation technique was employed to ensure robust model evaluation and prevent overfitting. The training dataset is used to train the machine learning models, while the validation dataset is used to fine-tune the machine learning algorithm’s hyperparameters (i.e., parameters that control the development of the machine learning models). The testing dataset is not used for model development, so it is used to assess the performance of the machine learning algorithms when they are applied to unobserved conditions. The divisions were determined based on stratified splitting [143], which ensures that all in situ networks are represented in each division and that the models are trained on representative samples. Divisions were based only on location (not time), so the entire record of a given in situ soil moisture station occurs in a single division of the dataset. Thus, the testing dataset evaluates the ability of the machine learning algorithms to estimate the soil moisture at unobserved locations.

RMSE was used as the evaluation metric (loss function) for all the machine learning methods. Random search was used for hyperparameter optimization, and the range of hyperparameters was defined based on each method’s documentation and the literature reviews of similar applications.

Table 3. Optimal hyperparameters for five machine learning models used in this study.

Model	Hyperparameter	Optimal Value	Default
ANN	Number of hidden layers	3	1
	Hidden layer sizes	100	100
	Activation function	Relu	Relu
	Training algorithm	Adam	Adam
	Regularization term	0.01	0.0001
	Learning rate	0.001	constant
	Maximum iterations	100	200
RF	Number of trees	200	100
	Maximum depth	10	None
	Min. samples for split	5	2
	Min. samples for leaf	2	1
	Max. features at split	sqrt	1
	Split criterion	Squared error	Squared error
XGBoost	Learning rate	0.3	0.3
	Maximum depth	6	6
	Number of trees	500	100
	Subsample for tree	1	1
	Depth sample fraction	1	1
	Min. child weight	0.8	1
CatBoost	Number of trees	1000	1000
	Learning rate	0.05	0.03
	Depth of tree	10	6
	Subsample for iteration	1	1
	Level feature proportion	1	1
	Regularization	3	3
LightGBM	Number of boosting iterations	1000	100
	Learning rate	0.05	0.01
	Number of leaves	31	31
	Maximum depth	10	−1 (unlimited)
	Min. data in leaf	20	20
	Regularization	0.1	0.0

summarizes the optimized hyperparameter values for each machine learning algorithm, and Appendix A describes the roles of the main hyperparameters.

The Pearson correlation coefficient ($R$), mean bias error (MBE), RMSE, ubRMSE, Nash–Sutcliffe Efficiency (NSE), and KGE were used to evaluate the accuracy of the soil moisture predictions (${\theta }_{pred}$) in reproducing the in situ observations (${\theta }_{obs}$). These metrics are calculated as follows:

$$R={\displaystyle \frac{{\displaystyle {\sum }_{i=1}^N\left({\theta }_{obs,i}-\overline{{\theta }_{obs}}\right)}\left({\theta }_{pred,i}-\overline{{\theta }_{pred}}\right)}{\sqrt{{{\displaystyle {\sum }_{i=1}^N\left({\theta }_{obs,i}-\overline{{\theta }_{obs}}\right)}}^2{{\displaystyle {\sum }_{i=1}^N\left({\theta }_{pred,i}-\overline{{\theta }_{pred}}\right)}}^2}}}$$

(1)

$$\mathrm{MBE}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N\left({\theta }_{pred,i}-{\theta }_{obs,i}\right)}$$

(2)

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{N}}{{\displaystyle {\sum }_{i=1}^N\left({\theta }_{pred,i}-{\theta }_{obs,i}\right)}}^2}$$

(3)

$$\mathrm{ubRMSE}=\sqrt{{\displaystyle \frac{1}{N}}{{\displaystyle {\sum }_{i=1}^N\left({\theta }_{pred,i}-{\theta }_{obs,i}\right)}}^2}-\mathrm{MBE}$$

(4)

$$\mathrm{NSE}=1-\left[{\displaystyle \frac{{\displaystyle {\sum }_{i=1}^N{\left({\theta }_{obs,i}-{\theta }_{pred,i}\right)}^2}}{{\displaystyle {\sum }_{i=1}^N{\left({\theta }_{obs,i}-\overline{{\theta }_{obs}}\right)}^2}}}\right]$$

(5)

$$\mathrm{KGE}=1-\sqrt{{\left(R-1\right)}^2+{\left(\alpha -1\right)}^2+{\left(\beta -1\right)}^2}$$

(6)

The metric R describes the linear correlation between the predicted (i.e., model) and observed values where $\overline{{\theta }_{pred}}$ and $\overline{{\theta }_{obs}}$ are the predicted and observed soil moisture means [144]. MBE describes whether a model typically overestimates or underestimates the observed values (positive values indicate overestimations) [145]. ubRMSE considers the error that remains if the bias is removed from the model estimates [146]. NSE compares the squared error to the variance of the observations [147,148]. NSE ranges from −∞ to 1 with a value of 1 indicating a perfect agreement between the model and observations and a value of 0 occurring if the mean of the observations is used as the model. KGE combines three measures of model performance including R, $\beta ={\mu }_{pred}/{\mu }_{obs}$, and $\alpha ={\sigma }_{pred}/{\sigma }_{obs}$, where ${\mu }_{pred}$ and ${\mu }_{obs}$ are the predicted and observed means and ${\sigma }_{pred}$ and ${\sigma }_{obs}$ are the predicted and observed standard deviations [149]. KGE ranges from −∞ to 1, with values closer to 1 indicating more accurate model estimates.

3. Results

3.1. Performance of Machine Learning Algorithms

Figure 2 shows the performance metrics for the machine learning methods’ soil moisture predictions of the testing dataset across all networks and depth ranges (combined). The results indicate that the RF, XGBoost, and CatBoost models exhibit better testing performance than SMAP as well as the ANN and LightGBM models. In particular, the RF, XGBoost, and CatBoost models typically have lower RMSE and ubRMSE values and higher R, NSE, and KGE values. These same three models also tend to have smaller biases than the SMAP, ANN, and LightGBM models. Overall, the XGBoost method exhibits the best testing accuracy among the methods. In contrast, SMAP shows the lowest accuracy among the methods, primarily due to higher bias. However, this evaluation uses SMAP as a direct estimate of soil moisture at each station. SMAP is expected to have better performance as an estimate of the spatial average soil moisture across its 9 km grid cells.

Figure 3a examines the RMSE of the machine learning methods when the testing data are divided according to the in situ soil moisture networks (ARM, COSMOS, AMERIFLUX, SCAN, SNOTEL, TxSON, USCRN, and iRON). The machine learning methods exhibit similar performance across most of the networks. However, the RMSE values are higher for the COSMOS network than the other networks. The poorer performance at the COSMOS stations likely occurs because the spatial support for the cosmic ray neutron measurements (~700 m diameter) [86] is much larger than the support for the in situ probes used in the other networks (centimeters at most). The machine learning methods are trained on all networks simultaneously, so the COSMOS data are inconsistent with the other datasets. No predictor variable allows the machine learning methods to identify whether an in situ measurement is from the COSMOS dataset or the other networks. Furthermore, all predictor variables are represented at a 70 m resolution, so the machine learning methods lack information to characterize much of the spatial support for the COSMOS data.

Figure 3b compares the performance of the machine learning methods when the testing dataset is divided according to depth. XGBoost consistently has the lowest RMSE values across all depths, followed by RF, CatBoost, LightGBM, ANN, and SMAP. As the depth increases, the accuracy of the methods usually improves. The largest improvement in performance occurs between a 0–50 cm and 0–100 cm depth. The improved performance for deeper layers may occur due to the greater uniformity of soil moisture at greater depths, which facilitates model learning and prediction. Maps of both the surface and rootzone soil moisture using the machine learning methods are provided in the Supplementary Materials (Figures S1 and S2).

Figure 4 compares the XGBoost soil moisture estimates to the in situ soil moisture observations when the testing dataset is divided by climate classification. The climate classification for each location was determined using the UNEP [150] system, which is based on the AI. An arid region has an AI below 0.20, a semiarid region has an AI from 0.20 to 0.50, a sub-humid region has an AI from 0.50 to 0.65, and a humid region has an AI above 0.65. In the dataset, 59 stations are arid, 401 stations are semiarid, 86 stations are sub-humid, and 185 stations are humid. The soil moisture estimates from XGBoost are relatively accurate across all climatic regions with R exceeding 0.8 and RMSE below 0.045 cm³/cm³ for all four climates. The weakest performance occurs in the semiarid region, which is the only region where the NSE and KGE values are below 0.70. Lower RMSE values might occur for the semiarid climate because that climate’s dataset is more diverse than the others. A wider variety of topography, soil types, vegetation types, and other factors that influence soil moisture occurs within this climatic region. Consequently, the machine learning model might have more difficulty capturing the underlying patterns of soil moisture.

Figure 5 and Figure 6 examine whether XGBoost captures the temporal dynamics of soil moisture for an arid location (USCRN LasCruces20N) and a humid location (USCRN Versailles3NNW), respectively. Both stations are members of the testing dataset and typical for their climatic region. In each figure, the upper part considers the surface soil moisture (0–5 cm), and the lower part considers the rootzone soil moisture (0–100 cm). For the arid location (Figure 5), XGBoost closely follows the in situ soil moisture variations at both depths including responses to individual precipitation events. XGBoost has a small wet bias, but the magnitude of the bias is smaller than the dry bias seen when using the SMAP L4 product at this station. XGBoost provides a more accurate representation of soil moisture dynamics at the arid location than directly using SMAP. XGBoost has correlations of 0.79 for surface and 0.75 for rootzone while SMAP has correlations of 0.64 for surface and 0.70 for rootzone. XGBoost has RMSE values of 0.016 cm³/cm³ for surface and 0.017 cm³/cm³ for rootzone while SMAP has RMSE values of 0.043 cm³/cm³ for surface and 0.054 cm³/cm³ for rootzone.

At the humid location (Figure 6), XGBoost also tracks the temporal variations of in situ soil moisture, maintaining high moisture values except in prolonged periods of low precipitation. Again, XGBoost provides a better representation of the time series than directly using the SMAP estimates. XGBoost has small wet biases at this station (MBE of 0.007 cm³/cm³ for the surface and 0.013 cm³/cm³ for the rootzone), whereas SMAP exhibits substantial wet biases (MBE of 0.107 cm³/cm³ for the surface and 0.124 cm³/cm³ for the rootzone). XGBoost has correlations of 0.80 for the surface and 0.76 for the rootzone while SMAP has correlations of 0.62 for the surface and 0.69 for the rootzone. XGBoost has RMSE values of 0.043 cm³/cm³ for the surface and 0.036 cm³/cm³ for the rootzone while SMAP has an RMSE value of 0.124 cm³/cm³ for both the surface and the rootzone.

3.2. Importance of Predictor Variables

Figure 7 presents the correlations between all the predictor variables and the in situ soil moisture at different depths for the complete dataset. The correlations indicate that the in situ soil moisture at a given depth is most related to the soil moisture in nearby layers. It also shows that the in situ soil moisture values for 0–100 cm are most different from the in situ soil moisture at the other depths.

Soil moisture from SMAP is highly correlated with in situ soil moisture values (Figure 7). Unexpectedly, the SMAP surface soil moisture exhibits higher correlations than the SMAP rootzone and profile soil moisture products with the deep in situ soil moisture. This behavior likely occurs because the surface soil moisture is inferred more directly from the satellite sensors, while the rootzone and profile soil moisture products rely in part on models. Both the SMAP soil moisture values from the same date and the antecedent SMAP soil moisture values exhibit similar correlations to the in situ soil moisture.

Vegetation indices such as the NDVI and EVI exhibit moderate correlations to the in situ soil moisture, with the EVI displaying the highest correlations. The EVI is less sensitive to variations in canopy structure and background factors such as soil conditions and atmospheric influences, and it is more sensitive to changes in canopy chlorophyll content, which makes it a better indicator of plant health and moisture stress than the NDVI [151,152,153]. Among the vegetation indices, the LAI usually exhibits the weakest correlations with soil moisture, perhaps because it is less indicative of the degree of soil surface shading than the NDVI or EVI.

In situ soil moisture exhibits positive correlations with clay and silt content, and negative correlations with sand content. Higher sand content (and lower clay and silt content) is expected to increase drainage and reduce soil moisture. Wang et al. [154] found that the spatial pattern of soil moisture is greatly influenced by soil factors, such as the sand and clay fractions. Bulk density, organic matter, and EC exhibit low correlations with soil moisture. Higher bulk density and lower organic matter are expected to reduce porosity, which would reduce the allowable range for soil moisture. However, errors in the estimated bulk density and organic matter values may cause the low correlations seen in Figure 7. pH exhibits moderate negative correlations with in situ soil moisture. Soil pH plays a role in determining microbial activity, nutrient availability, and soil structure, which can indirectly influence the soil moisture [155].

Positive correlations occur between in situ soil moisture and precipitation. The correlations are strongest for 14-day antecedent precipitation and weaken as the time period shortens to 6 h. This result highlights soil moisture’s memory (i.e., its ability to retain information) about past precipitation events [156,157,158]. The ESI and ET are positively correlated with soil moisture, meaning that higher soil moisture allows higher ET rates. The LST exhibits a weak negative correlation with soil moisture. Lower soil moisture reduces the latent heat flux and warms the land surface. Moreover, the AI is positively correlated with soil moisture, reflecting the greater availability of water in more humid climates.

The topographic variables exhibit a range of correlations with in situ soil moisture. Elevation displays a strong negative correlation with soil moisture. Rong et al. [159] reported that soil moisture at low elevation is often supplemented by surface runoff and subsurface flow from higher elevation points. This leads to a negative correlation between soil moisture and elevation. However, within smaller spatial extents (Reynolds Creek watershed in southern Idaho), Cowley et al. [68] found that soil moisture has a positive correlation with elevation due to increased precipitation at higher elevations. Slope exhibits a negative correlation, likely because it promotes lateral outflow of moisture [60,160]. Aspect, hillside, and mTPI exhibit only weak correlations in this dataset.

Figure 8 presents the predictor variable importance scores for RF, XGBoost, CatBoost, and LightGBM. These scores differ substantially between the methods, which likely occurs because some of the input variables contain similar information. Thus, different variables can be selected depending on the methods’ learning architectures. Among topographic variables, elevation is most important to the model predictions. However, the remaining topographic factors all typically have substantial importance (all are in the top half of the table in terms of average importance). Elevation above sea level does not directly influence soil moisture but serves as a proxy for other environmental factors. As the elevation increases, temperatures decrease, reducing ET and potentially increasing soil moisture. Precipitation patterns also shift, with higher elevations typically receiving more precipitation and more snow. Among soil texture variables, percent clay is most important for three of the four models (LightGBM relies more heavily on silt). The reliance on clay is interesting because percent sand is more correlated with soil moisture (Figure 7). The reliance on clay suggests either that clay has a nonlinear relationship with soil moisture or that its role is more independent from other predictor variables than that of sand. Soil depth, organic matter, and bulk density are also somewhat important. Vegetation plays a smaller role than topography and soil. Among vegetation variables, the EVI is most important, followed by the NDVI and LAI. This result aligns with Figure 7, where the EVI shows the strongest relationships with soil moisture among the vegetation variables. Land cover classification provides little value in predicting soil moisture (Figure 8). The AI is the most important climate/weather variable for the machine learning methods. Among temporally varying weather variables, 14-day antecedent precipitation is the most important. Although recent rainfall directly impacts soil moisture, a 14-day window provides a more complete indication of moisture additions. The other meteorological variables (shorter antecedent precipitation as well as PET, the ESI, and ET) have only moderate importance. Considering the SMAP products, the most important predictor variables are all related to surface soil moisture. All rootzone and profile soil moisture variables have low importance. The most important SMAP variables are usually the 3-day and 14-day antecedent soil moisture. However, the current SMAP surface soil moisture is used heavily by the RF model.

4. Discussion

Overall, this study suggests that machine learning algorithms can provide accurate estimates of rootzone soil moisture at unobserved locations within CONUS. It also suggests that XGBoost provides the most accurate estimates among the algorithms tested (XGBoost produced an overall RMSE of 0.042 cm³/cm³). The results are generally consistent with the findings of similar studies conducted in other regions. For example, Kornelsen et al. [133] obtained an RMSE for soil moisture of around 0.07 cm³/cm³ using an ANN. Senyurek et al. [161] reported an RMSE of approximately 0.052 cm³/cm³ for surface soil moisture using an RF. Liu et al. [133] reported an RMSE of 0.048 cm³/cm³ using a GBM. However, the present study considered a larger region than prior studies, such as those by Singh et al. [80] and Abowarda et al. [70]. The larger spatial extent likely includes more heterogeneity, which may impact the algorithms’ performance. In the present study, the performance was weakest in the semiarid region, which is consistent with Ren et al. [162] and Jamie et al. [135], who reported challenges in semiarid regions due to diverse topographic, soil, and vegetation characteristics. Future studies could consider more advanced machine learning techniques, such as convolutional neural networks and recurrent neural networks. Ensemble models could also be developed to combine the predictions from multiple machine learning algorithms.

The performance of machine learning models usually improved with depth, which supports their suitability for estimating rootzone soil moisture. Smaller errors likely occurred at greater depths because the soil moisture is steadier with less dependence on individual precipitation events or recent evaporation.

The present study considered more predictor variables (including vegetation indices, soil characteristics, weather and climate variables, and topographic features) than prior studies. For example, Du et al. [163] and Park et al. [164] primarily used vegetation indices and climate data. Using more predictor variables likely improves performance, but it also increases the effort required for data preparation and the time needed to train and apply the machine learning algorithms. Future studies could consider more refined feature selection. Landcover classifications and ECOSTRESS products typically have low importance in the trained models, so excluding these variables may simplify the models while having little effect on the results. However, this study only considered soil moisture at locations with long-term monitoring where landcover has been stable through time. If predictions are made at other locations where landcover has changed, landcover (and landcover changes) may play a more important role. The limited temporal data coverage of ECOSTRESS data (due to clouds) is a significant limitation. It allowed for only a small fraction (1%) of the available hourly soil moisture data to be utilized for the study. Other remote sensing data with similar spectral bands, such as Sentinel-1, Sentinel-2, and Landsat, could potentially be fused to enhance temporal coverage. The selection of topographic indices in this study was based on their availability in GEE and their relevance in previous soil moisture studies. Including other relevant indices such as the drainage area and topographic wetness index could enhance the soil moisture predictions, especially in regions with complex terrain.

The predictor variables used in this study were represented at a 70 m resolution, which produces soil moisture estimates with the same nominal spatial resolution. This resolution is finer than most prior studies. Jamei et al. [135] considered a 9 km resolution for rootzone soil moisture, Senyurek et al. [161] considered a 3 km resolution for surface soil moisture, and Huang et al. [165] and Yang et al. [166] considered a 1 km resolution covering various depths. Sun et al. [136] and Singh and Gaurav [80] considered 500 m and 60 m resolutions, respectively. Greifeneder et al. [77] and Abowarda et al. [70] considered 50 m and 30 m for surface soil moisture, respectively, and Nguyen et al. [167] and Meyer et al. [78] considered 10 m for surface soil moisture. In the present study, the models were trained by comparing to point measurements of soil moisture (aside from the COMOS data), which implicitly assumes that the point measurements are representative of the average soil moisture over the 70 m grid cell. Because the in situ soil moisture data are widely spaced, the soil moisture estimates from the machine learning methods are not expected to fully capture fine-scale spatial variations in soil moisture. Future studies could consider the accuracy of these estimates when compared to more closely spaced in situ soil moisture observations from sub-regions.

5. Conclusions

This study evaluated the accuracy of five machine learning algorithms (feed-forward ANN, RF, XGBoost, CatBoost, and LightGBM) for estimating soil moisture at multiple depths (0–5 cm, 0–10 cm, 0–20 cm, 0–50 cm, and 0–100 cm) at unobserved locations across CONUS. The machine learning methods operated at a 70 m spatial resolution and were trained and tested by comparing to several in situ soil moisture networks.

• For the dataset considered, XGBoost provides more accurate soil moisture estimates than the other machine learning models considered. XGBoost also provides better accuracy than directly using SMAP as an estimate of point soil moisture. XGBoost achieves the lowest mean RMSE of 0.042 cm³/cm³ compared to random forest (0.048 cm³/cm³), CatBoost (0.050 cm³/cm³), ANN (0.067 cm³/cm³), LightGBM (0.066 cm³/cm³), and SMAP (0.101 cm³/cm³) for the testing locations. XGBoost produces the best accuracy when considering the entire testing dataset, and it produces the best accuracy when separately considering each in situ soil moisture network and each depth.
• All the machine learning algorithms perform more poorly when comparing to data from the COSMOS network than to the other in situ data networks. The COSMOS measurements have a larger footprint (~700 m diameter) than the point measurements in the other networks. This inconsistency as well as the inconsistency between the COSMOS footprint and the 70 m resolution used to represent the predictor variables likely contributes to the poorer performance at the COSMOS sites.
• The machine learning algorithms typically provide more accurate estimates as the depth of the estimate increases. For example, XGBoost produces a median RMSE of around 0.041 cm³/cm³ for a 0–5 cm depth and 0.029 cm³/cm³ for a 0–100 cm depth. The accuracy may improve with depth because deeper soil moisture varies more gradually and predictably than surface soil moisture.
• Although XGBoost exhibits similar accuracy for arid, semiarid, sub-humid, and humid regions, the accuracy is lowest for the semiarid region (semiarid is the only region with NSE and KGE values below 0.70). The accuracy might be lower in semiarid regions due to more complex topographic, soil, and vegetation characteristics. XGBoost can reproduce the typical dynamics of soil moisture in arid regions, where soil moisture remains low except during responses to precipitation events. It can also reproduce the typical behavior in humid regions, where soil moisture remains high except for prolonged periods with low precipitation.
• Feature importance analysis identified elevation as the most important topographic variable when the machine learning models are applied to CONUS, and percent clay is typically the most important soil characteristic. Vegetation plays a lesser role in the models, with EVI being the most important vegetation variable. Land cover classification provides little value to the machine learning algorithms. Among SMAP soil moisture products, surface soil moisture is the most important, with rootzone and profile products having lower importance. Furthermore, 3-day and 14-day antecedent soil moisture variables are more important to the algorithms than the current soil moisture.