Application of a Novel Multiscale Global Graph Convolutional Neural Network to Improve the Accuracy of Forest Type Classification Using Aerial Photographs

Abstract

The accurate classification of forest types is critical for sustainable forest management. In this study, a novel multiscale global graph convolutional neural network (MSG-GCN) was compared with random forest (RF), U-Net, and U-Net++ models in terms of the classification of natural mixed forest (NMX), natural broadleaved forest (NBL), and conifer plantation (CP) using very high-resolution aerial photographs from the University of Tokyo Chiba Forest in central Japan. Our MSG-GCN architecture is novel in the following respects: The convolutional kernel scale of the encoder is unlike those of other models; local attention replaces the conventional U-Net++ skip connection; a multiscale graph convolutional neural block is embedded into the end layer of the encoder module; and various decoding layers are spliced to preserve high- and low-level feature information and to improve the decision capacity for boundary cells. The MSG-GCN achieved higher classification accuracy than other state-of-the-art (SOTA) methods. The classification accuracy in terms of NMX was lower compared with NBL and CP. The RF method produced severe salt-and-pepper noise. The U-Net and U-Net++ methods frequently produced error patches and the edges between different forest types were rough and blurred. In contrast, the MSG-GCN method had fewer misclassification patches and showed clear edges between different forest types. Most areas misclassified by MSG-GCN were on edges, while misclassification patches were randomly distributed in internal areas for U-Net and U-Net++. We made full use of artificial intelligence and very high-resolution remote sensing data to create accurate maps to aid forest management and facilitate efficient and accurate forest resource inventory taking in Japan.

1. Introduction

Forest type classification is of fundamental importance for sustainable forest management, i.e., biodiversity modelling [1], management of disturbances [2,3] and fires [4], harvesting [5], biomass evaluation [6], and carbon stock calculations [7]. Unlike urban area classification, forest type classification is challenging given the limitations in training images and ground truth. Expert knowledge is required to prepare training images for convolutional neural networks (CNNs), but there are few open-source datasets for forests [8]. Mountainous terrain is more susceptible to spectral reflectance distortion and canopy shadowing than flat land [9] and has high biodiversity [10]. Moreover, the vegetation is heterogeneous [11,12,13] because of the effects of climatic and regional parameters [14,15]. In the 1930s, before the advent of aerial photography, forest managers collected information using ground inventories [16]. The classification of medium-resolution satellite images has been performed since the 1970s [17]. Natural RGB photographs have been used to identify forest tree species [18] and to classify forest types [19]. Although extensive calibration and pre-processing are not required [20], traditional forest mapping and monitoring requires expensive, time-consuming, and potentially unreliable field measurements, along with professional expertise for manual interpretation [19]. Forest labels are assigned to aerial photographs by humans who evaluate textures, shapes, colours, sizes, patterns and associations [21], vegetation composition, forest structural properties (i.e., stem density, tree size, and vertical structure), and surface morphology [22].

Automated approaches for forest type mapping with remote sensing data can be categorised into three types: traditional thresholding methods, classical machine learning methods, and CNN methods. Traditional thresholding methods are limited to colour features (hue, chroma, and lightness) [23,24] and the discrimination of under-story (background) and over-story (tree canopy) signals in forests, given the subtle changes in complicated illumination environments. Classical machine learning methods, i.e., support vector machine (SVM) [25] and random forest (RF) [26], handle classification tasks [27,28,29,30,31,32] using a small training dataset with Gabor, Canny, Prewitt, Robert, and Gaussian filters applied to aid classification [33,34]. Recently developed CNN methods are effective for representing spatial patterns, such as edges, corners, textures, and abstract shapes, but CNN-based forest type classification with remote sensing data is still a very new field [35]. Most deep learning models are based on classical CNN models (e.g., U-Net [36] and DeepLabv3+ [37]) for mapping non-forest [38,39], forest [40] or tree species [18,37].

U-Net [41] can obtain underlying spatial features using contraction and expansion blocks throughout the encoder and decoder, thereby effectively expanding the horizon of the sensing fields to assess the global contextual environment and to derive detailed information. However, U-Net extracts high-level semantic information using a filter of a certain size and continuously feeds the features to convolutional layers, while ignoring the large amount of low-level information and ignore the correlations between adjacent pixels [42,43,44,45,46,47,48]. Redundant information is used many times during the processing flow, which reduces the capacity to represent key features. The skip connection of U-Net++ [49] helps prevent the potential loss of information caused by sampling [36]. However, the number of operation parameters is very large given the dense skip connections [50], which reduces model efficiency and increases the computational load. Also, during feature fusion, upper-layer semantic information is ignored and the network cannot yield effective fine-grained features at the decoding stage, resulting in serious loss of edge and positional information [51]. Principle component analysis (PCA) [52] and vision transformer (ViT) [53] were proposed for extracting minimal features. Research has also focused on using critical local image patches and discarding useless information [47,48,54].

Graph convolutional neural networks (GCNs) have recently been used successfully to analyse irregular data. Graphical spectral, spatial, and geometric data are rich in node and edge information [55,56]. In a previous study, a GCN [57] performed very well when extracting features from irregular graphical data and edge connections to aggregate node information and generate new representations of the nodes. A GCN is limited to shallow layers because of the vanishing gradient. To extract more features during semantic segmentation, several studies [58,59,60] combined CNNs with GCNs to strengthen the spatial and spectral information and to reduce pixel-level noise by identifying graphical nodes and the spatial relationships between them, as represented by graphical edges. Liu et al. [61] collaborate the Euclidean data-oriented CNN with non-Euclidean data-oriented GCN in a single network to generate complementary spectral-spatial features from the pixel and superpixel levels respectively. Ding et al. [62] and Wang et al. [63] fused two global feature vectors produced by individual CNNs and GCNs; the fused features were optimised. However, robustness may have been low. Peng et al. [64] fused features using MopNet and the parameters of a CNN and GCN; all features were continuously updated via backpropagation guided by joint loss.

Japan has a typical marine climate with abundant rainfall that facilitates forest growth. Nearly 70% of Japan is covered by forests, of which 40% are plantations [65]. Natural mixed forest (NMX) is defined as forest in which broadleaved species account for 25–75% of the coniferous canopies [66]. Plantation and natural forests differ at the leaf scale (i.e., in leaf inclination, morphology, and clumping) [9] and canopy structure scale (i.e., in crown morphology and canopy cover) [67], as well as in spectral characteristics [68,69,70,71]. They also differ at the forest stand scale (i.e., in forest composition and diversity and tree distributions and interactions within the forest). It is challenging to segment NMX, natural broadleaved forest (NBL), and conifer plantation (CP) because a mixed forest is more heterogeneous than other forests; it contains both conifers and broadleaved trees. In terms of vertical structure, unlike an NBL with a single-story stand, an NMX has multistorey classes with varying vertical structures within each stand type [72]. Furthermore, the natural forests of Japan are mostly located on steep mountainsides, which makes forest inventory taking and monitoring difficult [19]. Challenges to achieving highly accurate forest type classification include the complexity of natural forest canopy [73], which exhibits multiscale differences [9]. We can easily identify vegetation properties of interest, even with little reference data, in remote sensing data due to the distinct canopy structure or contrasting flowers, while subtle differences [18] and complex relationships require complex algorithms and more ground truth samples to identify specific features [74]. Second, the ambiguity of the boundaries of different forest types [9], which usually have a high chance of land-cover mixing [11,26,30], affect the edge pixels. Finally, redundant content and noise in very high spatial-resolution remote sensing datasets [25] (such as crown shadow effects [75]) and continuous CNN filters [54] lead to intraclass variation and high interclass similarity.

Our main aims were to develop a novel U-shaped deep learning method using a multiscale global graph convolutional neural network (MSG-GCN) to improve forest type classification accuracy. This network has several notable features. First, convolution kernels of different scales are used to capture image features and data from different receptive fields are fused to capture features that reduce the computational complexity when combining semantic information [76,77] from a previous level. Second, a multiscale graph convolution network (MSGCN) module serves as a transitional module between the encoder and decoder. The MSGCN combines the features of the encoding module with its own features to better represent edge and multiscale features [78]. Third, the skip connection is replaced by local attention (LA) [79] that focuses on salient features, thereby reducing the use of redundant information. Fourth, fusion of the decoding layers ensures consideration of both high- and low-level feature information, and improves the attribute decision-making capacity for boundary cells [80]. The main goals of this study are to compare the novel MSG-GCN with other state-of-the-art (SOTA) methods, investigate the specific areas and digital number (DN) of correctly and misclassified forest types, compare U-shaped deep learning models (such as U-Net and U-Net++) and the classical RF machine learning method, and map and classify entire forest areas, as well as determine the spatial distribution of misclassified forest types.

2. Materials and Methods

2.1. Study Area

The study area was the University of Tokyo Chiba Forest (UTCBF) in central Japan (longitude = 140°05′33″ to 10′10″E, latitude = 35°08′25′′ to 12′51′′N) (Figure 1). The area is in a warm temperate zone with a mean annual temperature of 14.1 °C and mean annual precipitation of 2474 mm [81]. The altitude ranges from 50 to 370 m above sea level. The area exhibits the unique forest landform of the Boso Hills, which merit academic study. The terrain and slopes are generally complex and very steep. The main soil type is brown forest soil and the geological structure comprises marine deposits from the Neogene Period that are partly covered by nonmarine deposits from the Quaternary Period [81].

The total forest area is approximately 2160 ha and can be divided into three forest types: 267 ha of NMX, 1117 ha of NBL, and 715 ha of CP (Figure 1). NMX areas have complicated horizontal and vertical structures, mixed stands, and various age classes. The main tree species include Abies firma, Tsuga sieboldii, evergreen Quercus spp., and Castanopsis siedoldii [81]. Natural forest is composed of many vegetation communities that exhibit natural succession over long periods of time and thus have multivertical layers and a rough texture (Figure 1, NMX). NBL has a complex and diverse crown structure, but the vertical does not exhibit extensive variance. NBL is dominated mainly by C. siedoldii, evergreen Quercus spp., Zelkova serrata, and Acer spp. (Figure 1, NBL) [81]. Although NBL areas often have a smoother texture than other areas, NMX areas can be easily misidentified as NBL areas. CP areas are generally monocultures with simple structures characterised by coniferous trees (Cryptomeria japonica and Chamaecyparis obtusa) that are cone-like in shape with a characteristic crown structure. It is easier to detect this forest type than others. CP established for timber production is manually planted and managed and exhibits well-organised arrangements in columns and rows (Figure 1, CP).

2.2. Data Sources and Preprocessing

We used digitally georeferenced orthorectified aerial photographs taken by the Geospatial Information Authority of Japan (GSI) that were purchased from the Japan Map Center (https://www.jmc.or.jp/, accessed on 18 June 2019). Orthorectification was applied only to the RGB bands; data in the form of 30′′ × 30′′ tiles were available. The orthophoto tiles use the JGD2000 or JDG2011 datum geographical coordinate system. The entire study area is covered by three blocks of GSI aerial photographs collected in different years. Four image tiles were acquired on 25 November 2012 over the Katsuura Tateyama district at a height of 2000 m. The size of each photograph was 3750 × 4650 pixels and the ground sample distance (GSD) was 0.2 m. One hundred image tiles were acquired on 27 October 2017 over the Katsuura Otaki district at a height of 4000 m. The size of each photograph was 1875 × 2325 pixels and the GSD was 0.4 m. Finally, 144 image tiles were acquired on 15 July 2017 over the Minami Boso district at a height of 2350 m. The size of each photograph was 1875 × 2325 pixels and the GSD was 0.2 m. We resampled all photographs to a resolution of 0.2 m using the nearest neighbour algorithm [82], mosaicked them into a single tile, and ensured compatibility of the data with JGD_2011_Japan_Zone_9 projection using ArcGIS software version 10.8 (Esri Inc., Redlands, CA, USA). We used TNTmips for linear raster contrast enhancement of the mosaicked image. Since 1970, the NMXs have not been logged, except to remove obstructive trees [83]. Until the early 1960s, the NBLs were used as fuelwood forests, but logging has been minimal since 1980 [83]. Although the aerial photographs were taken in 2012 and 2017, we assumed that the boundaries between forest types did not change greatly over this period.

The fully annotated ground truth classification map was used as the forest type map of the entire UTCBF (Figure 2). This was produced by UTCBF staff in vector format based on a forest map compilation, forest register, inventory data, and aerial photograph interpretations. The original map had a scale of 1:10,000 and was converted into a raster format tiff file that covered 35,152 × 41,152 pixels using the ‘polygon to raster’ method in ArcGIS. We reclassified the original classes into three types: NMX, NBL, and CP; other categories, such as the nursery and forest museum [81] used for education and research, covered very small areas and were not considered in this study. The images of these areas were removed, set as non-forest background (BG), and assigned a value of 255. The ground truth map was processed using the raster extraction boundary of UTCBF and then overlapped with the mosaic aerial photograph using the ‘extract by mask’ routine of ArcGIS to create an aerial photographic image of the same scale and spatial resolution as the ground truth map.

The aerial photograph and ground truth map, both of which are 35,152 × 41,152 in size, were cropped into 512 × 512 tiles without any repetition or overlap to ensure that each image was completely unique. Tiles lacking pixel values were removed. We randomly separated the remaining image and ground truth maps into three parts into training, evaluation, and test datasets at a ratio of 8:1:1 (1981, 248, and 248 image tiles, respectively).

2.3. Model Architecture

2.3.1. Basic Overview

U-Net uses encoders and decoders to model the space of the segmentation target. U-Net is an effective deep learning model for segmentation [41]. However, during modelling, subtle changes in the target may be ignored and sensitivity to fine details can be low. The continuous use of single-scale convolutional filters and pooling operations during encoding makes feature extraction insufficient. Therefore, given the complex content and high-frequency spectrum of a remote sensing image, we developed a locally aggregated MSG-GCN for accurate image segmentation.

Figure 3 shows the overall structure of the segmentation framework. There are three basic components: an encoder with three encoding blocks; a transitional MSGCN module between the contraction and expansion blocks; a decoder with three decoding blocks. The LA [79] integrates encoding and decoding features through an attention mechanism. LA effectively filters out irrelevant information, enhances the capacity to represent key information, and completes the segmentation task. For each encoding block and corresponding decoding block, we use 1 × 1 convolution for compression and 1 × 1 and 3 × 3 convolution layers for excitation. Thus, each encoding and decoding block extracts local and global semantic features by splicing features at different graph scales to capture multiscale information from the segmentation target. Simultaneously, pooling operations at various scales are run using the final decoder, which aids description of the global and contextual semantics [84]. The MSGCN is used by both the encoder and decoder as a transitional layer (strictly speaking, the MSGCN is an encoding block). This accurately identifies multiscale local and global information of the segmentation target. The MSGCN determines interactions between pixels by merging and transferring node information, reduces the intraclass ambiguity caused by different characteristics, and thus aids the modelling of long-term dependence [85].

2.3.2. Encoding Module

Convolutional operations at various scales replace the continuous convolution layer of the U-Net model. Specifically, a simple squeeze excitation multiscale [77] is used to model the multiscale information of the target. The number of network parameters and computational complexity are thus reduced and multiscale local semantics are captured more accurately. Each encoding block has an extruded layer, two excitation layers of different scales, and a splicing layer. Then, the coding information is input into the maximum pooling layer for fusion and dilution, which improves multiscale representation. The splicing and pooling process is given by the equation below:

$$\{\begin{array}{l}{O}_{cat}^k=Concat\left(\left[Con{v}_{1\times 1\times 2^{\kappa }w}(f_{ex}(x));Con{v}_{1\times 1\times 2^{\kappa }w}(Con{v}_{3\times 3\times 2^{\kappa }w}(f_{ex}(x)))\right]\right)\hfill \\ f_{ex}(x)=Con{v}_{1\times 1\times 2^{\kappa -1}w}(x)\hfill \\ O_{mp}^k=MP(O_{cat}^k),k=1,2,3\hfill \end{array}$$

(1)

where O_cat is the output of the splicing layer, $w$ is the number of filters, Conv1 × 1 × 2^kω(·) and Conv3 × 3 × 2^kω(·) are convolutions using 1 × 1 and 3 × 3 filters, 2^kω is the number of filters, ƒ_ex(x) are ‘squeeze incentive features’, k is the kth encoding module, and O^k_mp is the maximum pooling layer of the kth encoding module.

2.3.3. Multiscale Graph Convolution Network (MSGCN) Module

The CNN described above effectively captures local and global information. To create long-term dependencies and to identify the spatial relationships between different semantic details, the encoded information is sent to a large optimised topological GCN that extracts detailed semantic information and constructs spatial relationships by merging and transmitting node information [59]. A topological graph can be expressed as G = (V,E), where V is the set of nodes and E is the set of edges between nodes. The steps used to construct the spatial relationship are as follows.

Step 1. To improve feature representation, we first send the encoded features to a 1 × 1 convolutional layer for information conversion. In other words, we embed the data into a unified low-dimension space to compress the features and improve the “expressive” nature of the network; the reconstructed features provide a basis for the subsequent graphical representation. The reconstruction process is described as follows:

$$\{\begin{array}{c}{O}_N=reshape(Con{v}_{1\times 1\times 2^{\kappa +1}w}(O_{mp}))\\ O_M=reshape(Con{v}_{1\times 1\times 2^{\kappa +1}w}(O_{mp}))\\ O_N\in R^{C\times (H\times W)},O_M\in R^{(H\times W)\times C}\end{array}$$

(2)

where O_M and O_N are the reconstruction features, $O_{mp}$ is the maximum pooling layer of the convolutional encoding layer, and reshape(·) is the information conversion function.

Step 2. The reconstructed features are used to construct a multiscale graph with dimensions of C_𝑁 × C_𝑀, where C_𝑁 is a channel feature with N nodes and C_𝑀 is a channel feature with M nodes. The Euclidean distance is used to determine the spatial relationships between nodes. The construction of adjacency matrix 𝐴 is shown in Figure 3 and described by Equation (3). The reconstructed feature map is used to construct the topology map G. If there is a relationship between adjacent feature points, a value of 1 is assigned; if not, the value assigned is 0. Next, graph convolution is used for node learning and optimisation, and node transfer is then employed to gather detailed information on spatial relationships. The multilayer learning graph convolution is given by the following equation:

$$f_G^{(l)}=\sigma ({\tilde{D}}^{-\frac{1}{2}}\tilde{A}{\tilde{D}}^{-\frac{1}{2}}{f}_G^{(l-1)}{W}_G^{(l-1)})\in R^{C\times C},\tilde{A}=A+I$$

(3)

where $f_G^{(l)}$ is the output of the lth graph convolution layer, $\sigma (·)$ is the LeakyRelu activation function, $\tilde{A}$ is the adjacency matrix normalised by the Laplace method, $\tilde{D}$ is the degree matrix, 𝐼 is the identity matrix, A is the adjacent scale, and $W_G^{\left(l-1\right)}$ is the weight matrix of the (l − 1)th graph convolution layer.

Step 3. The MSGCM creates effective long-term dependencies between nodes using the node transfer function and efficiently captures the global spatial semantics of target objects using the aggregation function. Given that the initial encoding features contribute to the representation of local semantics, we can reconstruct the features from the third encoding module, fuse them with the multiscale graph convolution features, and input these into the 3 × 3 convolution layer to further strengthen the representations of local and global features and establish interactions between them.

2.3.4. Decoding Module

In the decoding stage, the decoder module restores the feature maps to the original input size. In U-Net, the expansive path involves upsampling of the feature map followed by a 2 × 2 convolution that halves the number of feature channels, a concatenation with the corresponding cropped feature map from the contracting path, and two 3 × 3 convolutions, each followed by a ReLU. A 1 × 1 convolution at the final layer maps the feature vector to the desired number of classes. Our redesigned expansive path is roughly symmetric to the contracting path with a U-shaped architecture. The parallel 1 × 1 convolution and 3 × 3 convolution replaced by two consecutive 3 × 3 convolutions were used to extract multiscale features. Extrusion and excitation feature learning can better represent target objects through this process. Furthermore, concatenation of high-dimensional upsampled features can easily lead to confusion regarding the features of objects at different scales, especially when the contour boundaries are blurred and irregular. Therefore, to explicitly learn the position and boundary information from the corresponding encoder blocks and fine-grained feature maps from the previous decoding convolution layer, we designed an LA embedding module to enhance the recognition ability, increase differences among target objects at various scales, and refine the features of different categories of target objects. The LA is calculated using Equations (4) and (5).

$$LA=AttE(O_{mp},f_D)$$

(4)

where LA is the local attention module, $AttE(·)$ is the embedding operation for LA, and $O_{mp},f_D$ are the maximum pooling output feature of the encoding module and the corresponding position decoding output feature, respectively. LA improves feature representation, eliminates redundant information, and establishes a complementary relationship between high- and low-level features. Thus, when high-level features are insufficiently represented, low-level features are strengthened. To ensure that low-level features do not interfere with high-level ones, a residual module is used to emphasise (or de-emphasise) the importance of low-level features. The LA embedding operation is as follows:

$$LA=\frac{O_{mp}+{\alpha }_{mp}{O}_{mp}+f_D+{\alpha }_D{f}_D}{2}$$

(5)

where ${\alpha }_{mp},{\alpha }_D$ are the attention matrices of the encoding and decoding features, respectively. Low-level detailed feature maps capture rich spatial and concrete information that highlights a target’s boundaries and morphology; high-level feature maps capture abstract information, which results in the loss of detailed information. Spatial and size differences make it difficult to fuse the high- and low-level features directly and effectively. As a result, we upscaled each decoding output feature map to the original input size of 512 × 512 using bilinear interpolation. These three feature maps are added directly to the unchanged channels, which increases the amount of information and reduces the model calculations. Adding the decoder stage output can be computed as:

$$\{\begin{array}{l}{O}_{out}=U{P}_1(f_{D3})+U{P}_2(f_{D2})+U{P}_3(f_{D1})\hfill \\ Y=SoftMax(Con{v}_{1\times 1}(O_{out}))\hfill \end{array}$$

(6)

The output feature map of the different decoding modules ƒ_D3, ƒ_D2, and ƒ_D1 and bilinear interpolation upscaling operations of the various filter kernels are represented by UP₁(), UP₂(), and UP₃(), respectively; the kernel sizes are 8 × 8, 4 × 4, and 2 × 2. O_out is the total feature plot of the decoded path output.

2.3.5. Loss Function

To ensure the robustness of the feature representations used for learning, we employed Dice loss function [86] to optimise the model and to facilitate attribute decisions for the boundary cells [87]. This effectively suppresses the class imbalance problem [87].

$$L_{dice}(\theta )=1-\frac{2\left|X\cap Y\right|}{\left|X\right|+\left|Y\right|}$$

(7)

where Y is the prediction label and X is the truth label.

2.3.6. Comparison with SOTA Models and Experimental Settings

We compared the proposed MSG-GCN with two machine learning methods (RF [88] and SVM [25]) and five SOTA methods (U-Net [41], U-Net++ [49], fully convolutional networks (FCN) [89], vision transformer (ViT) [90], and GCN [78]). The methods were implemented using their original codes and trained on the datasets used herein. The MSG-GCN module had two graph convolutional layers. ResNet101 [91] was used to create the initial feature matrices. AdamW [92] served as the optimiser; the learning rate was 1 × 10⁻⁴, the weight decay was 1 × 10⁻³, and the learning rate was adjusted by cosine annealing warm restarts. All experiments involved 100 training epochs. The batch size was 33. All models were implemented using the PyTorch 1.7.1 framework and the experiments were conducted on the Wisteria/BDEC-01 Supercomputer System (FUJITSU Server PRIMERGY GX2570 M6 (FUJITSU, Tokyo, Japan)) at the Information Technology Center (The University of Tokyo, Tokyo, Japan), which is equipped with NVIDIA A100 Tensor Core (NVIDIA, Santa Clara, CA, USA) graphical processing unit (GPU) (40 GB SXM). For the RF and SVM models, the OpenCV 3.4.2 [93], scikit-learn 0.24.2, and scikit-image 0.18.1 [94] Python 3.7.0 libraries were used to implement machine learning. Using the methods of Canny, Robert, Sobel, Scharr, and Prewitt, edge-based features were extracted as grayscale images that identified and highlighted edge information [95]. Gaussian, median, and variance filters were applied to extract noise and to reduce blurring [96].

2.4. Data Analysis

2.4.1. Accuracy and Complexity Evaluation

To identify the best model, we calculated the overall accuracy (OA), mean intersection over union (mIoU), kappa, F1-score, precision (Pre), and recall (Rec) metrics and evaluated classification accuracy using a confusion matrix. In this matrix, true positives (TPs) are correctly classified pixels, false positives (FPs) are incorrectly classified pixels, true negatives (TNs) are correctly predicted failures, and false negatives (FNs) are incorrectly predicted failures. Pre is the ratio of TPs to the sum of the TPs and FPs and indicates the accuracy within the class. Rec is the ratio of TPs to the sum of TPs and FNs; this is a measure of class confusion. The inclusion of Rec in the F1-score mitigates the imbalance between different types. The OA and kappa value were calculated as percentages of correctly classified pixels [97]. Kappa is more accurate and objective when there is an imbalance in dataset types. Kappa denotes the correlation between human-based validation and that of the machine learning classifier. The kappa value [98] ranges from –1 to 1, where values below 0 indicate no agreement, 0–0.2 indicate slight agreement, 0.2–0.4 indicate fair agreement, 0.4–0.6 indicate agreement, 0.6–0.8 indicate substantial agreement, and 0.8–1 indicate near-perfect or perfect agreement. IoU is the ratio of the overlapping area of the sum of the ground truth and predicted area to the total area and is widely used to assess semantic segmentation [2]. We compared the number of floating-point operations per second (FLOPs) and number of parameters [99] between the MSG-GCN and other SOTA models.

2.4.2. Classification Difference Analysis

The MSG-GCN architecture was refined and optimised based on U-shape encoder and decoder models (U-Net and U-Net++). The MSG-GCN, U-Net, U-Net++, and RF classical machine learning methods were used to map the spatial distributions of correctly classified and misclassified areas. A confusion matrix was applied to the test dataset using the raster calculator of ArcGIS version 10.8 (Esri Inc., Redlands, CA, USA). The results are shown as I–J values, where I is the ground truth and J is the predicted forest type. This provides insight not only into the number of errors but also the type [100]. The percentage is the sum of the correctly classified and misclassified areas divided by the total area of each forest type and makes comparisons within and between forest types more intuitive. A DN is associated with the value of a given pixel for each forest type in the different spectral bands. We used ArcGIS software to extract the DNs for each forest type, which were then summed and divided by the total number of pixels for each classification type. To evaluate differences in mean DNs between classification types, we used Fisher’s least significant difference (LSD) test [101] and the ‘boxplert’ function of R software v 4.2.0 (R Development Core Team) to analyse significant differences between pairs of spectral DNs for correctly classified and misclassified forest types.

The spatial positions of correctly classified and misclassified forests over the entire research site were visualised. The forest type tiles predicted by each model were merged into a 35,152 × 41,152 map and overlaid with the ground truth using the ArcGIS raster calculator. The misclassification rates of multiple buffer zones (10–170 m) around the research site were extracted by ArcGIS [102] to analyse the spatial distribution of the misclassified forests. Given the limitations in data availability, all aerial photographs and composites of the three series were collected. Thus, the effects of seasons, solar radiation and weather conditions were included in the dataset. The areas of misclassified forests from the three districts were extracted to determine how the dataset affected classification.

3. Results

3.1. Classification Accuracy Indices of Different Models

Table 1. Performance comparison of the eight methods. Accuracy, FLOPs, and parameters are listed as a function of model size on the aerial photo. The best results are in bold.

Models	OA	Kappa	IoU_NMX	IoU_NBL	IoU_CP	F1_NMX	F1_NBL	F1_CP	Pre_NMX	Pre_NBL	Pre_CP	Rec_NMX	Rec_NBL	Rec_CP	FLOPs (GMac)	Params(M)
RF	0.5739	0.3373	0.0596	0.4605	0.1584	0.1124	0.6306	0.2734	0.1017	0.6964	0.2307	0.1257	0.5762	0.3357	-	-
SVM	0.6344	0.0028	0.1245	0.2324	0.1497	0.2214	0.3772	0.2605	0.3416	0.2938	0.3659	0.1637	0.5264	0.2022	-	-
U-Net	0.8098	0.6706	0.344	0.6679	0.6776	0.5119	0.8009	0.8078	0.5151	0.7758	0.8504	0.5088	0.8277	0.7693	218.94	31.04
U-Net++	0.8143	0.7263	0.3367	0.6741	0.6992	0.5037	0.8054	0.8229	0.5221	0.7919	0.8358	0.4867	0.8192	0.8105	153.00	47.18
FCN	0.6895	0.5209	0.0007	0.5531	0.4367	0.0014	0.7123	0.6079	0.3857	0.6149	0.6561	0.0007	0.8462	0.5664	102.19	3.93
ViT	0.7914	0.6942	0.3499	0.6454	0.6530	0.5184	0.7845	0.7900	0.5913	0.7475	0.8189	0.4615	0.8254	0.7632	22.66	23.28
GCN	0.8240	0.7473	0.5463	0.6651	0.6811	0.7066	0.7989	0.8103	0.7232	0.8154	0.7781	0.6907	0.7830	0.8453	57.66	9.18
MSG-GCN	0.8523	0.7808	0.4374	0.7341	0.7451	0.6086	0.8467	0.8539	0.6103	0.8475	0.8510	0.6069	0.8459	0.8569	104.99	88.10

shows the quantitative evaluation metrics (OA, kappa, IoU, F1-score, Pre, and Rec) for all classification types, which were calculated using the test dataset to allow for quantitative comparisons. The accuracy metric showed that the MSG-GCN model performed best (kappa = 0.7808, OA = 0.8523), while U-Net++ (kappa = 0.7263, OA = 0.8143) and GCN (kappa = 0.7473, OA = 0.8240) performed well; FCN (kappa = 0.5209, OA = 0.6895), U-Net (kappa = 0.6706, OA = 0.8098), and ViT (kappa = 0.6942, OA = 0.7914) performed moderately. RF (kappa = 0.3373, OA = 0.5739) and SVM (kappa = 0.0028, OA = 0.6344) did not perform as well as the other models. In terms of the classification quantitative metrics for the three forest types, the MSG-GCN model performance was competitive relative to the other methods. Of the three forest types, NMX was the least accurate for all models because of dataset imbalance; few NMX areas were correctly classified. Although GCN had the best accuracy for NMX, the accuracies of NBL and CP were lower than that of MSG-GCN. Of the three forest types, CP was most accurately detected by U-Net, U-Net++, ViT, GCN, and MSG-GCN. RF and SVM detected NBL most accurately, followed by CP. Considering the parameter numbers and FLOPs required to process the images, our MSG-GCN architecture combining GCNs with CNNs has the largest model size (88.10 M parameters) and required more operations for processing and slightly fewer FLOPs (104.99 GMac). The FCN has the fewest parameters (3.93 M) and U-Net is optimal in terms of FLOPs (218.94 GMac).

3.2. Area and Digital Number for Each Forest Type Predicted by the Various Models

To further evaluate classification, the confusion matrix was applied to the test dataset. All evaluation metrics (Table 1) showed the same tendencies in terms of the percentages of correctly classified forest areas of each type (

Table 2. Areas and percentages of forest types correctly classified and misclassified by the RF, U-Net, U-Net++, and MSG-GCN models.

Classification	Ground Truth		MSG-GCN		U-Net++		U-Net		RF
	Number of Pixels	Percentage (%)	Number of Pixels	Percentage (%)	Number of Pixels	Percentage (%)	Number of Pixels	Percentage (%)	Number of Pixels	Percentage (%)
BG	11,654,816	100	11,624,871	99.74	11,651,030	99.97	11,650,029	99.96	11,645,984	99.92
BG-NMX			1138	0.01	343	0	55	0	665	0.01
BG-NBL			18,005	0.16	1597	0.01	2461	0.02	5039	0.04
BG-CP			10,802	0.09	1846	0.02	2271	0.02	3128	0.03
NMX	6,282,528	100	3,833,885	61.02	3,279,871	52.21	3,235,934	51.51	638,718	10.17
NMX-BG			3691	0.06	2741	0.04	5495	0.09	4416	0.07
NMX-NBL			2,342,218	37.28	2,742,520	43.65	2,633,321	41.91	4,320,615	68.77
NMX-CP			102,734	1.64	257,396	4.10	407,778	6.49	1,318,779	20.99
NBL	30,431,072	100	25,789,099	84.75	24,098,604	79.19	23,609,032	77.58	21,192,944	69.64
NBL-BG			37,705	0.12	30,617	0.10	51,540	0.17	42,704	0.14
NBL-NMX			2,351,894	7.73	3,308,442	10.87	2,935,850	9.65	2,920,111	9.60
NBL-CP			2,252,374	7.40	2,993,409	9.84	3,834,650	12.60	6,275,313	20.62
CP	16,643,296	100	14,163,402	85.1	13,911,171	83.58	14,153,704	85.04	3,838,865	23.07
CP-BG			13,638	0.08	9119	0.05	22,016	0.13	17,659	0.10
CP-NMX			129,378	0.78	150,747	0.91	188,197	1.13	1,521,940	9.15
CP-NBL			2,336,878	14.04	2,572,259	15.46	2,279,379	13.70	11,264,832	67.68

BG indicates background areas lacking spectral information; BG-NMX indicates BG areas misclassified as NMX forests; BG-NBL indicates BG areas misclassified as NBL forests; BG-CP indicates BG areas misclassified as CP forests; NMX-BG indicates NMX forests misclassified as BG areas; NMX-NBL indicates NMX forests misclassified as NBL forests; NMX-CP indicates NMX forests misclassified as CP forests; NBL-BG indicates NBL forests misclassified as BG areas; NBL-NMX indicates NBL forests misclassified as NMX forests; NBL-CP indicates NBL forests misclassified as CP forests; CP-BG indicates CP forests misclassified as BG areas; CP-NMX indicates CP forests misclassified as NMX forests; CP-NBL indicates CP forests misclassified as NBL forests.

). NMX and CP forests tended to be more susceptible to be classified as NBL forest, whereas the misclassification rates between NMX and CP were very low. Taking MSG-GCN as an example, NMX was more likely to be classified as NBL than CP, while CP was more likely to be classified as NBL than NMX. U-Net and U-Net++ showed the same tendencies as MSG-GCN. There were very few misclassifications of BG points as other forest types; such errors were confined to the edges because of the inevitable errors in raster pixel values. The RF model could not detect NMX forest and tended to misclassify NMX as CP rather than NBL.

Figure 4 shows that NMX, NBL, and CP forests correctly classified by the four models had very similar mean DN values for the three RGB bands. LSD analysis revealed that the mean DN of NBL was higher than that for CP (Figure 4 and Figure 5) and that the DNs for correctly classified and misclassified forest types were very similar. The distributions of the correctly classified and misclassified forest types overlapped greatly in terms of the DNs, as shown in the plot figures (Figure 4).

The differences between the DNs of forest types correctly classified and misclassified by the four models were analysed using the LSD test. In terms of the DNs of correctly classified forest types, NMX(c) and NBL(a) differed significantly. In terms of the DNs of misclassified forest types, taking the red band of the MSG-GCN model as an example (Figure 5), a significant difference between NBL(a) and NBL-NMX(bc) was shown, while there was no significant difference between NBL-NMX(bc) and NMX(c). There was no significant difference between CP(ab) and CP-NBL(bc), but there was a difference between CP-NBL(bc) and NBL(a). There was no significant difference between NMX(c) and NMX-NBL(c), but there was a difference between NMX-NBL(c) and NBL(a).

In terms of the prediction maps (Figure 6) for the test data set, severe salt-and-pepper noise was apparent in the RF map, but this was less severe in the U-Net and U-Net++ maps. However, there were patches where NBL and NMX areas were misclassified as CP or confused with each other. Moreover, the forest type edges were unclear, especially for NMX in U-Net and U-Net++, whereas the MSG-GCN map showed fewer misclassified patches and the edge areas between different forest types were sharper (Figure 6).

3.3. Mapping and Spatial Distributions of Forest Types by the Different Models

For MSG-GCN, misclassified (Figure 7 and Figure 8) areas were mostly distributed in boundary areas. For U-Net and U-Net++, the misclassified areas were not confined to the boundaries; instead, they were randomly distributed in internal regions. Buffer zone analysis (Figure 8 and Figure 9) showed that MSG-GCN errors were mostly in the external 30-m boundary buffer areas. The error rate decreased rapidly from the internal 40-m buffer zones and plateaued at 3.26% for the 50-m buffer zones. For U-Net and U-Net++, the errors were concentrated in the external 30-m boundary buffer zones; the internal buffer zones had lower error rates. However, because of the randomly misclassified fragments, the internal error rates were larger than for MSG-GCN.

The misclassification rates of the three district images showed that year and season did not greatly influence classification accuracy (Figure 9). The Katsuura Tateyama and Minami Boso areas showed slightly higher error rates than the Katsuura Otaki area with U-Net, U-Net++, and RF models; for the MSG-GCN model, the error rate for Katsuura Tateyama was higher than those for Katsuura Otaki and Minami Boso. The RF, U-Net, and U-Net++ made more errors than MSG-GCN (Figure 9 and Figure 10).

4. Discussion

4.1. Classification Accuracy

The results (Table 1 and Table 2) showed that MSG-GCN outperformed the other models. MSG-GCN better extracted the features of minority classes. The IoU accuracy for NMX was 0.4374, which was higher than those of U-Net++ (0.3367) and U-Net (0.3440), and shows the utility of augmentation techniques, including the cropping, rotating, and flipping (of input images), as well as the Dice loss function [103]. The results are in line with previous studies; the classification accuracy for NMX was lower than those for CP or NBL [10]. Cheng et al. [32] provided a relatively reliable planted forest (OA = 84.93 %, F1 = 0.85) classification at a national scale using multisource remote sensing data, while the poor data quality and complex terrain and vegetation conditions brought errors in planted and natural forest classification. In CP areas, seedlings are usually planted in lines, such that high classification accuracy is possible. However, if the lines are not evident because some trees have died or been harvested or broadleaved trees have naturally regenerated, there is a tendency toward error [2]. The borders of NMX areas are not sharp and the manual and automatic classification results varied. Although visual classification can be considered accurate, errors may exist due to inaccurate manual delineation and objective variance [3]. NMX areas with heterogeneous and irregular boundaries, diverse species, and multicanopy layers were the most difficult to map. The edges were not smooth and the canopy textural features were heterogeneous. This severely imbalanced the forest types in terms of the relative proportions of the NMX, NBL, and CP areas, which complicated classification because the dataset was biased toward the majority class; this was associated with errors among the minority classes [104]. For the SOTA experiments, FCN [89] can effectively capture the local feature information of the target, while it cannot obtain the global information due to the small receptive field. A shallow GCN [78] with a single structure cannot effectively spread label information in a large area. During the transmission of node information, it is easy to cause the node feature vector to be oversmoothed and result in similar characteristics of each node, which is not conducive to the final segmentation performance. ViT [90] regards each patch as a task by extracting and refining relational information through a multihead self-attention mechanism, while the model requires for heavy computation capacity and high-quality training data. In addition, the oversmoothing problem limited the number of stacking layers and prevented the model from encoding the position information, which ultimately degraded the segmentation performance. The computation complexity performance of MSG-GCN is not the best among the SOTA experiments, our main aim is to improve classification accuracy while ensuring moderate FLOPs. In the future, we will try to build concise efficient graph semantic models to improve the accuracy of forest type classification.

4.2. Area and Digital Number for Each Forest Type Predicted by the Different Models

In terms of forest type misclassification, our model (Table 2) made fewer errors than the U-Net, U-Net++, and RF models. A drawback of U-Net and U-Net++ is the use of identical kernel weights, which can cause loss of semantic information during information transfer between layers [105]. Unlike previous studies that used fixed graphs, we imported a multiscale graph, which can perform flexible convolution on any irregular image area and describe the target object in the image at different scales, increase the receptive field, and improve the feature representation ability. The MSG-GCN can exploit multiscale information, remove noise, and preserve edges [85]. We found that MSG-GCN segmented clear edges. The GCN uses edge information to aggregate node information and then generate a new node representation that automatically learns both node features and associations between nodes [106]. This allows the model to learn the characteristic information of boundary pixels better, establishes spatial correlations, effectively resolves differences within or between classes, and improves the classification accuracy of objects in aerial forest images. Although the segment edges of MSG-GCN were clearer than those of U-Net and U-Net++, some misclassification (Figure 8) still occurred in edge areas, in line with previous research [107,108]. In an NMX area, broadleaved species covered 25–75% of the land occupied by coniferous canopies [66]. Some tree species are found in more than one forest type. In UTCBF areas, the main tree species are evergreen Quercus spp., but C. siedoldii, which is found in NBL areas, can also be seen in the NMX forest type; this may explain the misclassification of NBL as NMX. Spectral similarities, crown overlap, and similar noise among different forest types compromise classification accuracy [109], leading to misclassification. The models find it difficult to delineate forest types based on the surface, colour, and patterns. Thus, we analysed the spectral values of correctly classified and misclassified areas to understand why misclassification occurred. The DNs (Figure 4) of different forest types varied, but the DNs of misclassified forest types were similar to those of correctly classified areas [107]. The DN of NBL was relatively high compared with that of CP, in line with previous results [28,110]; the forward-scattering direction of a coniferous forest shows lower reflectance than that of a deciduous forest because of the distinct optimal angles and leaf directions. An example LSD analysis of the digital red band of MSG-GCN showed that, within the NMX group, there was no significant difference between NMX(c) and NMX-NBL(c). Although a significant group difference between NMX-NBL(c) and NBL(a) was apparent, misclassification remained an issue. There was no significant difference between CP(ab) and CP-NBL(bc) within the CP group, but MSG-GCN showed a significant difference between CP-NBL(bc) and NBL(a). However, some CP-NBL misclassification still occurred, perhaps because the spectral information was not adequate to allow the model to classify the forest types. Other factors may also trigger misclassification. There were only three information bands and the spectral reflectance varied among the bands for different forest types. Within-species variation in reflectance may be caused by site conditions, species composition, the vertical structure, and shadowing effects [10,111].

4.3. Mapping and Spatial Distributions of Forest Types for the Different Models

As shown in the visualisation map (Figure 7), RF showed severe salt-and-pepper noise. Object-based classification by U-Net and U-Net++ reduced this noise, but many patches remained where NBF was misclassified as CP and the edges were not sharp and clear (unlike MSG-GCN) (Figure 7 and Figure 10). With MSG-GCN, misclassified forests clustered mainly along the transition zones of the forest boundary areas (Figure 8), similar to previous findings [30,73], perhaps because of homogeneity in the spectral responses and shade due to the highly enclosed overlapping crowns [2], as well as the greater sensitivity of edges compared with internal areas. This is a longstanding problem in semantic segmentation. Unlike the regular morphology of well-defined urban land, the morphology and texture of forest canopies are heterogeneous and complex and vary according to species composition [66]. Thus, ground truths may not be recorded accurately given the inevitable subjective boundary errors and variation in remote sensing images caused by weather or the sensor type used. Weight or edge masks could be assigned to boundary area pixels [112] to effectively mine edge and neighbourhood information. Apart from edge misclassification, U-Net and U-Net++ misclassified internal forest areas (i.e., NBF as CP; Figure 8), perhaps because of the similar spectral values and crown sizes of certain species. However, MSG-GCN rarely made such misclassifications. Node transfer and aggregation yielded multiscale graphical information that highlighted both intra- and interclass differences, thereby improving the recognition of different categories [106].

5. Conclusions

In this paper, we developed a novel MSG-GCN model that uses a combination of multiscale convolutional kernels, a MSGCN module, LA, and output features from different decoding blocks to extract both high- and low-level features. To our knowledge, this is the first application of multiscale graph convolutions to forest type classification with aerial photos. Our results show that MSG-GCN is useful for the segmentation task. The main contributions of this study are as follows: First, multiscale convolutional kernels were used to learn features from different receptive field scales for forest type classification using aerial photos. Unlike the traditional fixed square area convolution, this method successfully learned the correlations between adjacent pixels in an irregular area with a multiscale graph convolutional kernel filter. Second, LA was used to refine the features and to highlight the representation ability of salient features. In the stage of high-level feature representation, excessive interference from low-level features with the representation of high-level features (and vice versa) is avoided. Finally, the MSGCN module should resolve the incompatibility between convolution and graph convolution in the data structure, which has an excessive influence on encoding and decoding features. Moreover, CNN (encoding and decoding modules) and GCN (multiscale graph convolution) are used to perform feature learning on small-scale regular areas and large-scale irregular areas, respectively, which aids decision making regarding boundary pixels.

We also found that NMXs and CPs were more susceptible to misclassification than NBLs. Classification of forest types using only the similar and overlapping spectral DNs is not sufficient. The visualisation map of the entire area revealed that edge pixels were more likely to be misclassified as neighbourhood pixels by all networks and that the CNN approaches were associated with random misclassification patches in internal zones. NMF was more challenging to classify than other forest types, given the imbalances in datasets, the heterogeneous canopy texture, and the fact that edge regions are evident only in very high-resolution aerial RGB images.

In future research, we will aim to combine a multisensor remote sensing dataset and a multimodal machine learning model [113,114,115,116] to enhance the multidisciplinary nature of remote sensing images and deep-learning technology, to overcome the remaining problems with the MSG-GCNs method (such as how to make full use of multimodal [117] data to aid segmentation of edge pixels). Simultaneously, multisensor data will be applied to build concise efficient graph semantic models to improve the accuracy of forest type classification.