Deep Learning in Forest Tree Species Classification Using Sentinel-2 on Google Earth Engine: A Case Study of Qingyuan County

Abstract

Forest tree species information plays an important role in ecology and forest management, and deep learning has been used widely for remote sensing image classification in recent years. However, forest tree species classification using remote sensing images is still a difficult task. Since there is no benchmark dataset for forest tree species, a forest tree species dataset (FTSD) was built in this paper to fill the gap based on the Sentinel-2 images. The FTSD contained nine kinds of forest tree species in Qingyuan County with 8,815 images, each with a resolution of 64 × 64 pixels. The images were produced by combining forest management inventory data and Sentinel-2 images, which were acquired with less than 20% clouds from 1 April to 31 October, including the years 2017, 2018, 2019, 2020, and 2021. Then, the images were preprocessed and downloaded from Google Earth Engine (GEE). Four different band combinations were compared in the paper. Moreover, a Principal Component Analysis (PCA) and Normalized Difference Vegetation Index (NDVI) were also calculated using the GEE. Deep learning algorithms including DenseNet, EfficientNet, MobileNet, ResNet, and ShuffleNet were trained and validated in the FTSD. RGB images with red, green, and blue (PC1, PC2, and NDVI) obtained the highest validation accuracy in four band combinations. ResNet obtained the highest validation accuracy in all algorithms after 500 epochs were trained in the FTSD, which reached 84.91%. As a famous and widely used remote sensing classification satellite imagery dataset, NWPU RESISC-45 was also trained and validated in the paper. ResNet achieved a high validation accuracy of 87.90% after training 100 epochs in NWPU RESISC-45. The paper shows in forest tree species classification based on remote sensing images and deep learning that (1) PCA and NDVI can be combined to improve the accuracy of classification; (2) ResNet is more suitable than other deep learning algorithms including DenseNet, EfficientNet, MobileNet, and ShuffleNet in remote sensing classification; and (3) being too shallow or deep in ResNet does not perform better in the FTSD, that is, 50 layers are better than 34 and 101 layers.

1. Introduction

Forest tree species play a virtual role in forest management and forest resource surveys [1,2,3], which have been adopted as an indicator for evaluating forest carbon. Accurate classification of forest tree species is conducive to mastering the overall situation and ecology quality of forest resources in a country or region.

Ground survey of forest resources is time consuming and inefficient. Furthermore, some places cannot be covered. However, remote sensing images provide more survey area and higher frequency than human field surveys, which have been used for forest tree species classification. High-resolution remote sensing images such as IKONOS or WorldView are used for forest tree species classification [4]. Immitzer et al. [5] adopted the high spatial resolution of 8-band WorldView-2 satellite data to classify and identify 10 tree species in Austrian temperate forests. Fang et al. [6] used multi-temporal WorldView-3 imagery to classify tree species in different taxonomic levels in Washington D.C., USA. Multispectral remote sensing images have been combined with other sensors, which include airborne laser scanning, light detection and ranging [7], and drone-based image point clouds [8] in individual tree species identification. However, the above studies on forest tree species classification based on remote sensing images have only focused on a small area, and cannot be applied to a large area.

Medium-resolution remote sensing images enable the monitoring of larger areas than high-resolution images. The repetition cycle of the Sentinel-2 satellites (2–5 days) is higher than Landsat (8–16 days). Furthermore, Sentinel-2 provides higher spatial resolution (10–20 m) than Landsat (30 m), so more detailed information can be captured from Sentinel-2 [9]. Axelsson et al. [10] used Sentinel-2 images and Bayesian inference for tree species classification and selected images from April to October. Immitzer et al. [11] classified tree species using Sentinel-2 and a random forest classification, and they found that images from April to August obtained the highest accuracy. Persson et al. [12] classified five tree species using four images (April, May, July, and October) of the same year in a random forest classification. Puletti et al. [13] selected Sentinel-2 images of spring, summer, and autumn and used a random forest classification to discriminate between coniferous, broad-leaved, and mixed forests, achieving a maximum overall accuracy of 86.2% using a separate validation dataset. From the above, it can be seen that random forest classification is usually used in tree species classification, but it easily leads to overfitting, so deep learning was used in this paper. The images were selected from April to October in this paper.

Principal Component Analysis (PCA) is a useful and important tool for dimension reduction [14]. Due to remote sensing images having more bands than RGB channels, PCA was used in our paper to retain most information and combine it with RGB images. Zhang et al. [15] used PCA to detect clouds using remote sensing. Li et al. [16] adopted PCA and other methods to calculate the ecology index of the Beijing-Hangzhou Grand Canal. The Normalized Difference Vegetation Index (NDVI) has also been widely applied to classify vegetation units [17]. Therefore, PCA and NDVI were used in the paper for band combination.

As the amount of available data has increased, a convolutional neural network (CNN) has been developed in tree species classification [18]. Li et al. (2016) used a CNN to detect and count oil palm trees in remote sensing images [19]. Mubin et al. [20] developed a CNN to detect juvenile and mature oil palms. Freudenberg et al. [21] applied a UNET neural network to detect large-scale palm tree images. From the above research, it can be seen that CNN has not been used for multi-type forest tree species classification.

This study aims to answer the questions of which band combination is the best combination for RGB visual identification, and which deep learning algorithm is the best estimator in the above band combinations.

The other object of the study was to build a forest tree species dataset as a benchmark of classification, utilizing other applications with forest management inventories, and verify which deep learning was well accomplished in forest tree species identification using Sentinel-2 images. A forest tree species dataset was built in this paper named FTSD, which includes nine kinds of forest tree species with 8815 images. Due to utilizing various band information of Sentinel-2, four different band combinations were compared in the paper, and PCA and NDVI were also used. Eight kinds of different deep learning algorithms were trained and validated, and ResNet50 reached the best validation accuracy in the FTSD and NWPU RESISC-45.

2. Material and Methods

2.1. Overall Workflow

The study consisted of three primary steps (Figure 1): first, the Sentinel-2 images of the study area were processed and downloaded from the GEE platform; secondly, GGC was calculated by *.shp files of the domain tree species in forest management inventories, and sample images were clipped through GGC in ArcGIS 10.4 software; finally, the above images were trained and validated using deep learning algorithms with Pytorch.

2.2. Study Area

The study area of Qingyuan County is located in Zhejiang Province, which is in southeastern China. The geographical location is 118°50′–119°30′ E and 27°25′–27°51′ N, as shown in Figure 2. The county has a total area of 1898 km², which spans 49 km from north to south and 67 km from east to west.

Qingyuan County has a subtropical monsoon climate with four distinct seasons throughout the year, which is warm in winter and cool in summer. The average temperature is 17.4 °C, the precipitation is 1760 mm, and the frost-free period is 245 days annually. Qingyuan County is rich in natural resources, and its forest land is 1.6233 × 10⁵ hectares in 1.8976 × 10⁵ hectares of total land, with the forest land accounting for 85.5% of the county.

2.3. Sentinel-2

The Sentinel-2 images were downloaded from GEE (https://earthengine.google.com, accessed on 15 May 2022), which are released in the Level-1C format. Radiometric, atmospheric, and cloud corrections but no other processing is required in Level-1C format product [22]. The bands of Sentinel-2 are shown in

Table 1. The Parameters of Sentinel-2.

Sentinel-2 Bands	Central Wavelength (nm)	Resolution (m)
Band 1: coastal aerosol	0.443	60
Band 2: blue	0.490	10
Band 3: green	0.560	10
Band 4: red	0.665	10
Band 5: vegetation red edge	0.705	20
Band 6: vegetation red edge	0.740	20
Band 7: vegetation red edge	0.783	20
Band 8: NIR	0.842	10
Band 8A: vegetation red edge	0.865	20
Band 9: water vapor	0.945	60
Band 10: SWIR-cirrus	1.375	60
Band 11: SWIR	1.610	20
Band 12: SWIR	2.190	20

.

Level-1C products utilize a Digital Elevation Model (DEM) to project the images into cartographic geometry. In addition to per-pixel radiometric measurements in Top of Atmosphere (TOA) reflectance, the products also include parameters for converting the measurements into radiances. Level-1C products are resampled with constant Ground Sampling Distances (GSDs) of 10, 20, and 60 m depending on the native resolution of the different spectral bands. To generate images with an equal spatial resolution in all bands, images were re-sampled to 10 m [23] in GEE before being downloaded in the paper. Attributed to the Sentinel-2 images being sensitive to weather conditions, all images were acquired with less than 20% clouds from 1 April to 31 October, including the years 2017, 2018, 2019, 2020, and 2021.

To compare classification accuracy, four different band combinations were used in our paper, which are shown in

Table 2. Different band combinations.

Combination	Red	Green	Blue
432	Band 4	Band 3	Band 2
843	Band 8	Band 4	Band 3
PCA	PC1	PC2	PC3
PCANDVI	PC1	PC2	NDVI

.

As a standard remote sensing product for discriminating and interpreting mapped vegetation units, NDVI is often used in classification [17]. NDVI is calculated by the following equation:

$$NDVI=\frac{NIR-Red}{NIR+Red}$$

(1)

where NIR is the near-infrared band and Red is the red band, which is Band 8 and Band 4 in Sentinel-2 images. PC1, PC2, and PC3 are the first, second, and third most important principal components after PCA manipulation.

2.4. Forest Management Inventories

Forest management inventories produce stand-level estimates to guide management decisions using data obtained with sampling intensities, which include the most detailed and up-to-date information on trees. The inventories of the study area were updated in 2020, and geographical information on dominant tree species was used in this paper. GGC was calculated, which was stored as latitude and longitude together with attribute information.

2.5. Dataset Description

2.5.1. Forest Tree Species Dataset (FTSD)

The forest tree species dataset (FTSD) was used in our research to classify forest tree species using Sentinel-2 data. To ensure clipped Sentinel-2 images in the dominant tree species zone, a minimum area of 8 hectares was acquired and a sample balance of forest tree species was considered. Therefore, 9 kinds of forest tree species with different areas were selected in the inventory data as

Table 3. Tree species selected in inventory.

Dominant Tree Species	Area (Hectares)	Samples	Images
Bambusoideae	>13	223	935
Cunninghamia lanceolata	>13	225	1000
Mixed broadleaf	>12	205	945
Mixed broadleaf-conifer	>16	237	1080
Mixed conifer	>12	227	1035
Other hardwood	>13	216	985
Pinus massoniana Lamb.	>8	208	880
Pinus taiwanensis	>10	217	930
Quercus spp.	>16	212	1025

. The dataset contains 9 dominant tree species and 8815 RGB images with 64 × 64 pixels. The spatial resolution of images is 10 m per pixel.

2.5.2. NWPU RESISC-45

RESISC-45 (https://www.tensorflow.org/datasets/catalog/resisc45, accessed on 15 May 2022) is a famous satellite imagery database and is widely used in remote sensing classification [24]. It includes 45 scene classes and 31,500 RGB images. Each class includes 700 images with a size of 256 × 256 pixels. The spatial resolution of the dataset ranges from 20 cm to more than 30 m, and it covers a wide range of countries [25]. A forest class is in the dataset, but there is no subdivision category of forest in the dataset. Therefore, the dataset cannot be provided for forest tree species classification.

2.6. Methods

The main steps of the study are as follows: first, the geographical geometric center (GGC) was calculated for the nine kinds of forest tree species, based on forest management inventories and acquiring an area larger than 8 hectares to ensure the clipping image in the same tree species; similarly, after some pre-processing, Sentinel-2 images were downloaded from the GEE platform in the study area; then, the downloaded images were clipped with GGCs of different kinds of forest tree species; and finally, the clipped images with different deep learning algorithms were trained, validated, and compared to find the best validation accuracy algorithm.

3. Methodology

3.1. Data Preprocess

After downloading Sentinel-2 images from GEE, the center mass of the geographical location was calculated in ArcGIS 10.4. The main process is described as follows. Firstly, the .shp files of tree species were selected, which are described in Table 1, then the corresponding features were changed to points as the center of geographical location and images were clipped using a square with 64 × 64 pixels. An example of the above steps is described in Figure 3.

Visual inspection was taken for removing some low-quality images (Figure 4) from the dataset, and the quantity of tree species images was shown in Table 1 in the images column.

3.2. Transfer Learning

Due to few samples, the transfer learning method has been widely used in image classification and other fields [26]. Because higher layers of representation are crucial to classification [27], transfer learning was used in this study for obtaining deeper layers. After being trained in a big dataset, the parameters of the deep learning model are preserved, which will improve the accuracy and efficiency of forest tree species identification.

ImageNet is usually used for training models as transfer learning [28]. By dropping the final layer parameters of the trained model and fixing other features, then training and learning complex features can be achieved from the Sentinel-2 images. A fully connected layer, a batch normal layer, and a dropout layer were sequentially combined, then connected to a dense net as the final layer to classify forest tree species.

Eight deep learning models were used to train in both datasets using transfer learning to compare them with each other. Five kinds of popular deep learning models of CNN, which include DenseNet, EfficientNet, MobileNet, ResNet, and ShuffleNet, were used for forest tree species classification by transfer learning.

3.2.1. DenseNet

DenseNet was introduced with a densely connected convolutional network architecture [29]. For retaining more information between layers in the feed-forward network, all layers are connected directly with each other. Feature maps of all preceding layers are used as inputs in all layers. The problem of vanishing gradient has been alleviated in DenseNet, which has a small number of parameters. DenseNet121 was used in our study, which contains 121 layers. Pre-trained weights of the model were downloaded from PyTorch.

3.2.2. EfficientNet

The EfficientNet model consists of a group of 8 CNN models, and the Swish activation function was used while the Rectifier Linear Unit (ReLU) activation function was adopted in other CNN models [30]. EfficientNet also aims to achieve better accuracy with fewer parameters. EfficientNet v2 trains up to 11x faster while being up to 6.8× smaller compared to EfficientNet and other CNN models, which is a smaller and faster CNN model in image classification [31]. Therefore, EfficientNet v2 was used for forest tree species classification.

3.2.3. MobileNet

MobileNet v2 was first proposed by Sandler [32] in 2018, which was based on MobileNet v1. The model reduces parameters by using deep separable convolution, obtains deeper layers by proposing a reverse residual structure, and reduces the loss of low-dimensional features by using a linear bottleneck structure. Therefore, MobileNet v2 was used for forest tree species classification.

3.2.4. ResNet

ResNet was first introduced by He et al. [33] in 2015, and the model won 1st place with an error rate of 3.57% in the ImageNet Large Scale Visual Recognition Challenge. A network-in-network architecture is adopted in ResNet, which relies on many stacked residual units. These residual units are used to construct the network. The residual units are composed of convolution and average pooling layers [34]. With a further updating of the residual module to use identity mappings to update ResNet, a higher accuracy will be obtained [33]. Pre-trained weights with 50,101,152 layers of ResNet were downloaded from Pytorch and used in our paper.

3.2.5. ShuffleNet

ShuffleNet v2 is an upgraded version of ShuffleNet v1 [35]. To minimize the memory access cost in the convolution operation, the model keeps the same channels of input and output layers in the convolution and uses group convolution [36]. To enhance parallel operation ability, the model minimizes branch structure to improve the network operation speed by reducing element-level operations. Due to the above advantages, ShuffleNet v2 was also used for forest tree species classification.

In comparison with the above-mentioned CNNs, ResNet aims to solve the problem of vanishing gradients that occur in very deep networks by introducing residual connections, which allows the network to learn identity mappings as well as more complex mappings. This allows for much deeper networks, with over 100 layers, to be trained effectively.

4. Results

4.1. Training

To calculate the best parameters, Stochastic Gradient Descent (SGD) was used for training models, which makes it run faster and converge easily when training models [33]. SGD is an optimization algorithm used for finding the minimum of a function. SGD is particularly useful when the function to be minimized is very large or has many local minima, because it is computationally efficient and only requires one training example at a time to update the parameters. SGD is widely used in machine learning and deep learning, particularly for training neural networks. The batch normalization technique and ReLU activation function [37] were adopted in all models. The batch size was set as 16, which was limited to GPU memory. A learning rate of 0.001 was set up in all networks. For a common evaluation framework, a standard training and validation split was completed using the 80% and 20% ratios, respectively.

Data augmentations, which include up and down flip and random crop of 64 × 64 pixel images, were performed to all networks to relieve the lack of quantity of remote sensing images.

4.2. Results of the Experiments

Deep learning algorithms including DenseNet121, EfficientNetv2_s, MobileNet_v3_large, ResNet34, ResNet50 v2, ResNet101 v2, and ShuffleNetv2_x1_0 were trained and validated in the FTSD. In our paper, 80% of the images were for training, and the other 20% images were for validation. All models were trained and validated for 100 epochs in different band combinations.

The accuracy metric is usually adopted for the evaluation of the models and used to find the appropriateness of state-of-the-art deep learning [26]. The accuracy formula is as follows:

$$accuracy=\frac{TP+TN}{TP+TN+FP+FN}$$

(2)

where TP is True Positive, TN is True Negative, FP is False Positive, and FN is False Negative.

The best validation accuracies of different bands after 100 epochs of training are shown in

Table 4. The best validation accuracy of different models in different band combinations after 100 epochs of training.

Models	Best Validation Accuracy (%)
	432	843	PCA	PCANDVI
DenseNet121	25.87	26.55	26.09	33.18
EfficientNetv2_s	25.75	25.35	25.30	41.69
MobileNet_v3_large	21.50	21.95	21.16	36.07
RegNety_400mf	23.03	21.15	21.72	23.65
ResNet34	49.29	64.04	68.17	71.64
ResNet50	40.50	47.99	58.25	64.89
ResNet101	35.28	43.28	51.05	53.26
ShuffleNet_v2_x1_0	40.44	38.23	51.28	58.88

.

From Table 4, PC1, PC2, and NDVI grouped as red, green, and blue channels obtained the best accuracy in all deep learning algorithms.

The accuracy of the eight different deep learning algorithms with PCANDVI in the FTSD is shown in Figure 5.

Figure 5 shows that the performance of ResNet is better than other deep learning algorithms in the FTSD, and ResNet50 is better than ResNet34 and ResNet101. This is consistent with the results of Maxim Neumann’s paper [24]. Eight different deep learning algorithms were also trained and validated after 100 epochs of training in RESISC-45, and accuracy is shown in Figure 6. ResNet50 obtained the highest validation accuracy with 87.90%.

The confusion matrix is used to visualize the performance of a classifier and is usually used to assess remote sensing classification [38]. The confusion matrix depicts the proper and mistaken classification for each category [39]. The confusion matrix of ResNet50 in the FTSD is shown in Figure 7.

Classification mistakes were focused on the following aspects which were concluded from Figure 7: (1) the same genus of tree species led to wrong classification, such as that nine Pinus massoniana Lamb. were misclassified as Pinus taiwanensis; (2) intersection and overlap between species in the dataset took place, especially in mixed classes, as nine of the mixed conifers were mistaken as mixed broadleaf-conifers.

The precision formula is as follows:

$$\mathrm{precision}=\frac{TP}{TP+FP}$$

(3)

The recall was calculated as the following equation.

$$accuracy=\frac{TP}{TP+FN}$$

(4)

Precision and recall of classification in the FTSD are shown as follows in

Table 5. Precision and recall of ResNet50 in FTSD.

Dominant Tree Species	Precision	Recall
Bambusoideae	0.958	0.968
Cunninghamia lanceolata	0.929	0.845
Mixed broadleaf	0.789	0.852
Mixed broadleaf-conifer	0.803	0.847
Mixed conifer	0.867	0.787
Other hardwood	0.859	0.832
Pinus massoniana Lamb.	0.900	0.818
Pinus taiwanensis	0.756	0.866
Quercus spp.	0.822	0.834

. The precisions of the mixed broadleaf and Pinus taiwanensis were 0.789 and 0.756, which shows that the algorithm mistakenly classified other forest tree species as these species. The recall of mixed conifer was the lowest accuracy at 0.787, which depicts that mixed conifer species cannot be identified as a conifer. Bambusoideae was identified well in the FTSD, and the precision and recall of Bambusoideae were 0.958 and 0.968, respectively.

5. Discussion

More data will help to improve the accuracy of deep learning, and Sentinel-2 has 13 spectral bands. However, almost all deep learning methods in computer vision use RGB images. Therefore, our study transformed images from all bands to RGB images by PCA. PCA is usually used to find the best features [40] and retains most information from all bands. Table 4 shows that the band combinations of PCA (PC1, PC2, and PC3) obtained higher validation accuracy than 843 (Band 8, Band 4, and Band 3) and 432 (Band 4, Band 3, and Band 2) when using ResNet or ShuffleNet, but did not perform well in other algorithms. Therefore, detaining most information from all bands is just one of the important factors but is not enough to identify forest tree species. As one of the most widely used indices to assess vegetation from remote sensing imagery [41], NDVI was added to our study for improving accuracy. Table 4 shows that the combination of PCANDVI (PC1, PC2, and NDVI) obtained higher accuracy than other combinations (PCA, 843, and 432) in all deep learning classifications.

Many models showed a trend toward improved classification accuracy as epochs increased [42]. In deep learning and other machine learning methods, excessive training led to no help in improving validation classification accuracy because the testing data are overfitted to the training data. Overfitting was found after 200 epochs of training in Figure 5 and 50 epochs depicted in Figure 6. ResNet is a network-in-network architecture that relies on many stacked residual units [33]. Because of the usage of global average pooling rather than fully connected layers, it can obtain more accuracy by updating the residual module to use identity mappings. ResNet usually shows better results than other CNNs, which is also demonstrated in Figure 5 and Figure 6. The deeper the layers of the network, the higher accuracy the model achieves in the same dataset [43]. However, this was not the same as the result of our study. Figure 5 and Figure 6 show that ResNet34 and ResNet50 performed better than ResNet101, which has deeper layers of the network.

Forest tree species and their distribution are key factors in a regional ecology [44]. To broaden the use of forest management inventory, dominant forest tree species information was used in the study. The dominant tree species were labeled as the only class in each stand. However, different tree species are on the same stand, which becomes a big challenge for forest tree species classification. With more heterogeneity, the more difficult the task of classification is. Therefore, intersection and overlap between species in the dataset, especially in mixed classes, were easily misclassified in this study, which is shown in Figure 7 and nine mixed conifers were mistaken as mixed broadleaf-conifers. Another challenge of forest tree species classification is that some tree species belong to the same genus. Owing to the many similarities between them, it is hard to classify them. For example, nine Pinus massoniana Lamb. were misclassified as Pinus taiwanensis, which is depicted in Figure 7.

6. Conclusions and Future Work

Forest management inventories not only produce stand-level estimates to guide management decisions but also can be used for forest tree species classification. Primary tree species information in forest management inventories was used to clip Sentinel-2 images in our paper. Therefore, another application of forest management inventories was found. In our study, PCA and NDVI were extracted from images, and RGB image classification with PC1, PC2, and NDVI reached the best performance band combinations. This paper shows that the combination of PC1, PC2, and NDVI is suitable for remote sensing classification, and it promotes classification accuracy.

Deep learning classification methods, such as DenseNet, EfficientNet, MobileNet, ResNet, and ShuffleNet, were used to train and validate. The result showed that the combination with red, green, and blue (PC1, PC2, and NDVI) adopted with ResNet50 obtained the highest accuracy rate of 84.91% after 500 epochs of training in the FTSD; ResNet50 also reached the highest accuracy rate of 87.90% after 100 epochs of training in RESISC-45.

In this study, the Sentinel-2 images were combined with a deep learning method and used to classify forest tree species in the research area. More remote sensing images of forest tree species will be collected in future work as a standard recognized benchmark dataset and evaluation frameworks, and higher spatial resolution remote sensing images will be used for comparison.