Ensemble Machine Learning on the Fusion of Sentinel Time Series Imagery with High-Resolution Orthoimagery for Improved Land Use/Land Cover Mapping

Abstract

In the United States, several land use and land cover (LULC) data sets are available based on satellite data, but these data sets often fail to accurately represent features on the ground. Alternatively, detailed mapping of heterogeneous landscapes for informed decision-making is possible using high spatial resolution orthoimagery from the National Agricultural Imagery Program (NAIP). However, large-area mapping at this resolution remains challenging due to radiometric differences among scenes, landscape heterogeneity, and computational limitations. Various machine learning (ML) techniques have shown promise in improving LULC maps. The primary purposes of this study were to evaluate bagging (Random Forest, RF), boosting (Gradient Boosting Machines [GBM] and extreme gradient boosting [XGB]), and stacking ensemble ML models. We used these techniques on a time series of Sentinel 2A data and NAIP orthoimagery to create a LULC map of a portion of Irion and Tom Green counties in Texas (USA). We created several spectral indices, structural variables, and geometry-based variables, reducing the dimensionality of features generated on Sentinel and NAIP data. We then compared accuracy based on random cross-validation without accounting for spatial autocorrelation and target-oriented cross-validation accounting for spatial structures of the training data set. Comparison of random and target-oriented cross-validation results showed that autocorrelation in the training data offered overestimation ranging from 2% to 3.5%. The XGB-boosted stacking ensemble on-base learners (RF, XGB, and GBM) improved model performance over individual base learners. We show that meta-learners are just as sensitive to overfitting as base models, as these algorithms are not designed to account for spatial information. Finally, we show that the fusion of Sentinel 2A data with NAIP data improves land use/land cover classification using geographic object-based image analysis.

1. Introduction

Accurate extraction of information from remote sensing data is of utmost importance for environmental monitoring, land-use planning, and the management of natural resources. Unlike field-based data, remote sensing offers cost-effective, time-efficient, and frequent data delivery over a broad geographical extent. In the recent past, high-resolution land use/land cover (LULC) map production was hindered by the geographical coverage of an area by satellite sensors and the cost of commercial data [1,2]. In the last three decades, however, these limitations have largely been resolved, with several mapping efforts accomplished in the United States using Landsat data and, more recently, Sentinel [3,4]. Satellite images are still the dominant data used in LULC mapping; however, these maps are regional with a relatively moderate spatial resolution (10–60 m) that often do not capture small-scale resources present on the ground. In contrast to the relatively coarse scale of such data, the National Agricultural Imagery Program (NAIP) in the conterminous United States captures high spatial resolution (0.6 m) data every 2–3 years. Although NAIP data are acquired chiefly for agricultural purposes, they have been rarely used for general mapping of land cover in relatively large (county to state)-scale LULC classification efforts. Such large-area mapping at a fine resolution remains challenging due to radiometric differences among scenes, landscape heterogeneity, and computational limitations. Various classification approaches have been developed to address these challenges.

With the launch of Landsat in 1972, pixel-based classification soon became the dominant paradigm [5]. However, with the advent of high-resolution satellite sensing, another paradigm challenged the pixel-based approach, focusing on grouping high-resolution pixels into one based on several criteria, a method called geographic object-based image analysis (GEOBIA). Several studies have employed pixel-based [6,7,8] or object-based [9] approaches to NAIP data. Comparative studies have shown that GEOBIA yields higher accuracy than traditional pixel-based classification [10]. Additionally, in recent decades, remote sensing data analysis has benefited from exploiting the power of machine learning (ML) algorithms in LULC classification, such as support vector machines (SVM) [11,12], Decision Trees (DT) [13], Random Forest (RF) [9,14,15], Nearest Neighbor [16,17], and Deep Learning (DL) [18].

Because each approach has its own biases and advantages, typically, a variety of ensemble models are subjectively selected to decrease variance (bagging), reduce bias (boosting), and improve the predictive power of the classifier (stacking) [19,20]. Specifically, ensemble ML methods have gained much recent attention in various fields, especially in applied remote sensing [20,21,22,23,24], because they can handle high-resolution images to capture fine structural information. After building such an ensemble model, current state-of-the-art practices involve model evaluation using held-out data (a random subset of training data, i.e., random cross-validation). The data collection process is not usually random; therefore, a random selection of observations to create the testing data set does not guarantee independence from training observations when a spatial dependency is inherent in the data. Spatial dependency in the sampling data arises from spatial sampling biases, as observers tend to collect data from a clustered hotspot of their activity. This phenomenon of spatial dependency [25] is called spatial autocorrelation (SAC) [26]. When spatial data are autocorrelated, the considered model fails to account for a critical determinant that is spatially structured and thus causes spatial structure in the response. Another SAC issue in the input data set can result in unreliable results through validation approaches because an observation point cannot act as spatially independent validation of proximity data points [27,28]. Using such cross-validation approaches without accounting for spatial structures of input data provides overoptimistic results, making accuracy assessment unreliable [9]. We compared accuracy based on random cross-validation without accounting for spatial autocorrelation and on target-oriented cross-validation accounting for spatial structures of the training data set.

This study took a GEOBIA approach to Sentinel 2A and NAIP data to evaluate the effect of the stacking ensemble on classification accuracy. We compared random cross-validation to target-oriented cross-validation strategies (i.e., leave-location-out cross-validation) in reducing the effects of spatial bias in training data in LULC classification. We demonstrated a reduction in overinflation of overall accuracy and documented the effect of accounting for spatial autocorrelation.

2. Materials and Methods

2.1. Study Area

The 5381.73 km² study area was a heterogeneous landscape in central Texas in Irion and Tom Green counties (Figure 1). The mean annual precipitation is 508 mm, and the minimum and maximum temperatures range from 0 °C to 35 °C. Most of the study area is situated on the Semiarid Edwards Plateau with scattered woody vegetation [29]. The study area contains cropland [29] as well as forests of live oak (Quercus fusiformis), Texas oak (Q. buckleyi), Ashe juniper (Juniperus ashei), and mixed grass prairie with scatted honey mesquite (Prosopis glandulosa) [29].

2.2. Imagery Data Set

2.2.1. NAIP Orthoimagery

The National Agriculture Imagery Program acquires digital orthophoto quarter quads (DOQQ) every 2–3 years. The NAIP DOQQs used in this study were acquired in 2020 between August 23 and October 18 through ten flight missions. Each DOQQ contains four working bands, including red (465–676 nm), green (604–664 nm), blue (533–587 nm), and near-infrared (NIR) (420–492 nm). The spatial resolution of the entire data set is 0.6 m, and the radiometric resolution is 8-bit (digital number values range from 0 to 255).

2.2.2. Sentinel Multispectral Imagery

The Sentinel-2 multispectral instrument is managed by the European Space Agency, which consists of a constellation of two polar-orbiting satellites in the same Sun-synchronous orbit covering areas from 56° to 84° latitude [30,31]. Sentinel-2 has 13 spectral bands between visible and short-wave infrared, with three bands at 60 m, six at 20 m, and four at 10 m spatial resolution. The radiometric resolution of 13 bands is 12-bit (digital number values range from 0 to 4095). The temporal resolution of Sentinel 2B is five days, whereas Sentinel 2A has a slightly coarser temporal resolution of ten days.

This study used 11 Sentinel 2A images for the year 2020, with one image per month except for February. The criterion for image selection was that the images must have little or no contamination from clouds or haze. All Sentinel 2A scenes were downloaded in level-1C processing format from the Copernicus Open Access Hub (https://scihub.copernicus.eu/) accessed on 7 June 2021. This study only used four bands (red, green, blue, and near-infrared) available at 10 m spatial resolution.

2.3. Layer Generation and Dicing

In addition to original visible and NIR bands from NAIP and Sentinel, we generated byproducts and indices following methods by Subedi et al. [32,33]. We first calculated the soil-adjusted vegetation index (SAVI) on each NAIP and Sentinel image. Then, for Sentinel imagery, we extracted all near-infrared bands and produced principal component axes (PCA) [9,34,35]. In addition, we also ran PCA on the SAVI layers. Finally, we calculated textures on the first PCA axis on SAVI layers [32,33].

To harness the high spatial resolution of NAIP imagery and the radiometric and temporal resolution of Sentinel imagery, we applied multi-resolution image segmentation on NAIP imagery using eCognition software (Version 10.5) to derive quality objects [36]. Due to the large extent of the study area, all necessary images were first generated and then diced (clipped) into manageable sizes. Next, we generated several features based on NAIP and Sentinel images on each object/polygon in eCognition. We used multi-resolution segmentation in NAIP, NIR and PCA axis (bands) at a scale value of 50 pixels with shape and compactness weights of 0.04 and 0.98, respectively. Multi-resolution segmentation was followed by spectral difference segmentation and region growth at a spatial scale of 75 pixels. The detailed segmentation workflow for each diced tile is described in our previous work on land use/land cover mapping using the GEOBIA approach [9]. Finally, all objects with generated features were exported to merge into the final data set.

2.4. Selection of Features

Combining geometry, spectral bands, spectral indices, principal component axes on NAIP data, spectral indices, and PCAs on near-infrared and spectral indices resulted in several feature variables. However, since these features were produced on the original NAIP and Sentinel bands, the information held by these features became redundant. Therefore, we performed recursive feature elimination (RFE) in the scikit-learn package in Python [37] to retain the most critical features. A RFE relies on variable importance scores calculated using training data [38].

2.5. Machine Learning Algorithms

2.5.1. Random Forest

RF is an ensemble learning technique proposed by Breiman [39]. A RF uses several classification and regression trees to make a prediction. Initial data are split into in-bag (about 2/3rds of the data) and out-of-bag (OOB) (the remaining 3rd) samples. Trees are created using in-bag samples, whereas OOB samples are used to estimate the OOB error [39]. The central idea of RF is that several bootstrap aggregated classifiers perform better than a single classifier [39]. Each tree is created by bootstrapping in-bag samples. The resulting classification is based on the majority votes of all trees for the most frequent input feature $\left(\mathbb{x}\right),{\widehat{\mathrm{C}}}_{\mathrm{r}\mathrm{f}}^{\mathrm{B}}=\mathrm{m}\mathrm{a}\mathrm{j}\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{y} \mathrm{v}\mathrm{o}\mathrm{t}\mathrm{e}{\left\{{\widehat{\mathrm{C}}}_{\mathrm{B}}\left(\mathbb{x}\right)\right\}}_1^{\mathrm{B}}$, where ${\widehat{\mathrm{C}}}_{\mathrm{B}}\left(\mathbb{x}\right)$ is the class predictin of the bth Random Forest tree. Each decision tree is independently produced, and each node is split using a randomly selected defined number of features (mtry). RF has been widely applied in LULC classification [9,40,41].

2.5.2. Gradient Boosting

Gradient Boosting Machines (GBM) [42] build the boosting procedure as an optimization task over a differential loss function in an iterative fashion. A classical GBM trains a series of base learners that minimizes some loss function iteratively in a sequence [43]. In classification, GBM uses multi-class logistic likelihood algorithms for fitting criteria. Extreme gradient boosting (XGB) implements a traditional GBM that builds additive models while optimizing the loss function [44].

2.6. Stacking Ensemble Machine Learning

Before stacking models, all base learning algorithms (level 0) were first developed. Stacking, also called stack generalization (level 1), uses meta-learning to integrate different algorithms. Stacking ensemble machine learning algorithms (in this case, classification trees and boosted trees) estimate base learner generalization biases and minimize these biases while stacking [45]. The essence of stacking is to use the level 1 model to learn from the predictions of the base models (level 0). Generally, a stacked generalization framework improves prediction performance compared to the best level 0 model. We tested RF, GBM, XGB, and logistic regression and selected the model that produced better results than the base learner. Figure 2 shows the basic workflow of the stack generalization used in this study.

2.7. Accuracy Assessment

The performances of the base and stack models were measured using a 20% holdout of the total data in random cross-validation. In the target-oriented cross-validation (leave-location-out (LLO-CV)), the holdout data varied at each iteration (Appendix A). Five performance criteria derived from the confusion matrix were evaluated to evaluate the model: (a) overall accuracy (OA), (b) Precision (User’s Accuracy), (c) Recall (Producer’s Accuracy), (d) Kappa [46], and (e) Mathews correlation coefficient (MCC) [47]. In general, the higher the values of these performance metrics, the better the performance of the classifiers in discriminating land use/land cover classes.

3. Results

3.1. Level 0 and Level 1 Classification Comparison

OA, Kappa, and MCC of the three base learners were relatively similar (

Table 1. Performance metrics produced by base learners and meta-learners using random cross-validation. These statistics were calculated using a confusion matrix in the holdout data set (Figure 3).

Model	Accuracy Statistic	Land Use/Land Cover Class
		Cropland	Grassland	Shrubland	Built-Up	Water	Shadow
Random Forest	Precision	0.9804	0.8977	0.9669	0.9015	0.9737	1
	Recall	0.9259	0.965	0.9733	0.9632	0.74	0.75
	F1-Statistic	0.9524	0.9301	0.9701	0.9313	0.8409	0.8571
		Overall accuracy [0.9326], Kappa [0.9141], MCC [0.9149]
GBM	Precision	0.9813	0.9363	0.9605	0.9086	0.9762	0.9
	Recall	0.9722	0.955	0.9733	0.9421	0.82	0.8182
	F1-Statistic	0.9767	0.9455	0.9669	0.9251	0.8913	0.8571
		Overall accuracy [0.9407], Kappa [0.9248], MCC [0.925]
XGB	Precision	0.9811	0.9234	0.973	0.9278	0.9545	0.878
	Recall	0.963	0.965	0.96	0.9474	0.84	0.8182
	F1-Statistic	0.972	0.9438	0.9664	0.9375	0.8936	0.8471
		Overall accuracy [0.942], Kappa [0.9265], MCC [0.9267]
Stacking	Precision	0.972	0.9363	0.9613	0.9424	0.9545	0.9024
	Recall	0.963	0.955	0.9933	0.9474	0.84	0.8409
	F1-Statistic	0.9674	0.9455	0.977	0.9449	0.8936	0.8706
		Overall accuracy [0.9474], Kappa [ 0.9334], MCC [0.9335]

) when random cross-validation was used on holdout data. All metrics were derived based on the confusion matrix (Appendix B). In classification, XGB produced the highest OA (0.942), followed by GBM and Random Forest. Kappa and MCC followed the same pattern: XGB produced the highest Kappa and MCC statistics. GBM, followed by XGB, yielded the highest producer’s accuracy on cropland.

The stacking model (level 1) produced the highest OA, Kappa, and MCC statistics. At the land use/land cover class level, however, the stacking model did not produce the best results, although it was consistent with other competitive models (Figure 3).

3.2. Identifying and Reducing Overfitting

Both cross-validation approaches produced competitive overall accuracy statistics (Figure 4). Selected base learners and stacked models produced higher accuracy statistics when repeated with random cross-validation (10 folds, 5 repetitions). However, the overall accuracy values were reduced when predicted at unknown locations, hinting at spatial overfitting of the models.

A random validation data set that was not part of the training or testing data set showed slightly different results about the performance metrics (

Table 2. Accuracy metrics produced in the independent validation data set. These metrics were produced on the confusion matrix.

Cross-Validation	Classifier	Accuracy Metrics (%)
		Overall Accuracy	Kappa	MCC
Random-CV	RF	89.64	86.94	87.45
	GBM	90.6	88.17	88.54
	XGB	90.75	88.35	88.75
Target Oriented (leave location out CV (LLO-CV))	STACK	91.08	88.79	89.15
	RF	89.93	87.31	87.77
	GBM	92.92	91.09	91.26
	XGB	92.96	91.15	91.33
	STACK	93.98	92.43	92.57

). In general, leave-location-out cross-validation (LLO-CV) resulted in similar statistics and three accuracy metrics that were lower than those produced on holdout data. This indicates that random cross-validation could predict better when the testing data set was closer to the training data set. On the other hand, LLO-CV showed similar results to the training data set. The Mathews correlation coefficient suggested that all the base models and stack models were consistent and produced better spatial prediction abilities (random cross-validation < LLO-CV). The class-level prediction ability of these models varied significantly in both cross-validation strategies (Appendix B).

3.3. Contribution of Features

The 20 highest contributing features across the base learners (RF, XGB, and GBM) and the meta-learner indicated that SAVI minimum (SMSMI), SAVI standard deviation (SSSMI), the density of an object (GDENS), NAIP’s near-infrared band standard deviation (NSNIR), and area of the object (AIPXL) were among the essential variables in model performance (Figure 5). Permutation-based variable importance demonstrated similar features as important features of boosting (GBM and XGB) save for a few exceptions in the top ten features. For example, PCA1 on SAVI was indicated as one of the top ten important features in RF but not in XGB. The stacking algorithm (meta-learner) found the area of objects (AIPXL) to be the most important feature, followed by SMSMI, NSNIR, and SMSSD.

In general, the area of objects, density mean, and standard deviation of SAVI in the Sentinel data and the standard deviation on the near-infrared band of NAIP data were among the most significant. In contrast, the standard deviation from the PCA bands extracted in time-series SAVI layers of the Sentinel images was not so crucial in the LLO cross-validations (Figure 5). Random cross-validation demonstrated patterns somewhat similar to those of LLO-CV. The XGB and stacking algorithms precisely identified the same variables as the most critical features in random cross-validation (Appendix C). The final LULC map based on stacking and LLO-CV is presented in Figure 6. Several subsets in the study area using level 0 and level 1 are presented in Appendix D. Selected variables used in classification are defined in Appendix E.

4. Discussion

Machine learning on remotely sensed data has become widespread and promises to improve LULC classification results over traditional supervised methods [48]. Our study evaluated geographic object-based implementation of bagging (Random Forest), boosting (GBM and XGB), and stacking ensemble machine learning in land use/land cover mapping and found that machine learning enhanced the discrimination ability of base learners in multi-class classification problems. The improvement was apparent in the target-oriented cross-validation approach with consistency and reduced overfitting problems, in contrast to base learners and meta-learners in random cross-validation. A stacking ensemble does not necessarily guarantee improvement over its base models. Results from random cross-validation often inflate accuracy estimates, which may not hold when validated on independent data sets or when there is a spatial structure in the data [9]. Here, we evaluated the fusion of Sentinel data with NAIP and compared image accuracy using random cross-validation and target-oriented validation. Although the accuracy on the holdout data set from random cross-validation appeared similar, the stacking ensemble performed best under the target-oriented LLO-CV (Figure 4). The independent data set suggested (Table 2) that the stacking ensemble is the best approach because the overall accuracy obtained from random cross-validation was overoptimistic. Our results suggest that meta-learners outperform base learners by training weak-performing learners into super-learners. Although the meta-learner showed improved accuracy over base learners, our validation results do not warrant indicating that meta-learner ML models consistently improve discrimination ability among classes. The LLO-CV demonstrated that the meta-learner was weak in discriminating among grassland, shrubland, and built-up LULC classes. In our case, we used two boosting models based on our initial classification results with different hyperparameters. We did not find significant performance enhancement in one of the widely used ensembles of machine learning models, such as SVM. In stacking ensemble models, each base learner should be carefully selected as a candidate base learner.

Our evaluation showed that the random cross-validation method introduces spatial overfitting when the spatial structure of the data is not considered. All models performed better, producing better evaluation metrics but resulting in smaller accuracy metrics when used with target-oriented cross-validation (LLO-CV). This highlights that the discrimination ability of the models increased when the training and testing samples were in spatial proximity. However, when tested with the independent validation data set, the accuracy metrics performed poorly compared to the usual training/testing data splitting approach. Unlike random cross-validation, the consistent performance of learners in target-oriented cross-validation showed that avoidance of spatial structure in the data could overcome spatial overfitting issues [28,49]. Despite advocation to account for spatial structure in remote sensing data [50] and the implications of failing to do so, the debate is ongoing on whether spatial validation approaches should be used to improve model performance [28,51,52].

Our findings indicate that spatial structures in the data can inflate overall model performances if the training data are spatially structured. Mannel et al. [53] reported that the random cross-validation of training data resulted in 5–8% inflation of overall accuracy when training data were autocorrelated. In our case, the overall accuracy overestimation based on random cross-validation ranged from 2% to 3.4%. Moreover, our approach showed that the spatial structure in the data still exists even after feature selection. A possible limitation of variable selection is based on the fact that different algorithms calculate the importance of each variable differently. For example, RFE depends on the importance of initial model training. RFE removes the least important features until the best subset is determined [38]; however, the algorithm ranks the features without considering their importance and prediction ability in spatial space [49].

Our assessment of random and target-oriented cross-validation in Sentinel 2A time series and NAIP data showed that spatial resolution of NAIP, variation in the near-infrared band of NAIP, variations of SAVI, and sizes of objects were among the most important factors for the assessment of land use/land cover in a heterogeneous landscape of Texas. Using time-series Sentinel data helped discriminate vegetation from other LULC classes. NAIP DOQQS were captured in autumn, which hindered the generation of meaningful comparable high-resolution information from NAIP data. Moreover, the noise in the data may have cumulative effects due to the use of multi-temporal Sentinel data by increasing dimensionality and redundancy. As described above, RFE failed to remove such variables effectively. Feature selection combined with a target-oriented cross-validation approach reduced the overestimation of accuracy.

5. Conclusions

High-resolution LULC maps are essential for land-use planning, yield estimation, examining the effects of habitat fragmentation, and many other applications. Therefore, it is axiomatic that more accurate and reliable high-resolution maps can provide a critical decision base for land managers, researchers, and wildlife biologists. However, accurate high-resolution LULC map generation is prohibited by the high cost associated with high-resolution remotely sensed data. The effective use of high-resolution NAIP data is marred by heterogeneity in sensors, multiple acquisitions, higher volume, and low radiometric resolution. Sentinel spectral bands are slightly coarser but have high temporal, spectral, and radiometric resolutions, so they can provide more temporal depth and the capability of capturing differences in moisture and vegetation phenology. The approach we used can be extended to other geographic locations; however, the specific band combinations and initial pre-processing steps may vary depending on the characteristics and objectives of the project.

This study demonstrated the efficacy of fusing Sentinel and NAIP data and the effect of a target-oriented cross-validation approach in reducing spatial overfitting. In multi-class LULC mapping, we demonstrated a 2–3.5% accuracy inflation using random cross-validation. Furthermore, independent validation data sets suggested that target-oriented cross-validation produced consistent accuracy metrics, avoiding the overinflation of accuracy.

This study applied widely used bagging algorithms (Random Forest) and boosting (GBM and XGB) in remote sensing. In addition, we stacked these models to improve the final classification. The main contribution of this research is that meta-learners are just as sensitive to overfitting as base models that are not designed to consider spatial autocorrelation. We found that classification performance could be improved by harvesting spatial resolution from NAIP and time-series information from Sentinel data. Furthermore, the boosting algorithms were more proficient in discriminating LULC classes compared to bagging. The LULC map generated from stacking diverse machine learning models coupled with feature selection and target-oriented cross-validation presents a practical method for improved land use management. This approach can assist decision-makers in providing cost-effective ways to accurately quantify resources on the grounds. Testing different base learners with target-oriented cross-validation in different landscapes on a relatively large scale would offer broader insight into improving overall land use/land cover mapping.