Multi-Source Remote Sensing Images Semantic Segmentation Based on Differential Feature Attention Fusion

Abstract

Multi-source remote sensing image semantic segmentation can provide more detailed feature attribute information, making it an important research field for remote sensing intelligent interpretation. However, due to the complexity of remote sensing scenes and the feature redundancy caused by multi-source fusion, multi-source remote sensing semantic segmentation still faces some challenges. In this paper, we propose a multi-source remote sensing semantic segmentation method based on differential feature attention fusion (DFAFNet) to alleviate the problems of difficult multi-source discriminant feature extraction and the poor quality of decoder feature reconstruction. Specifically, we achieve effective fusion of multi-source remote sensing features through a differential feature fusion module and unsupervised adversarial loss. Additionally, we improve decoded feature reconstruction without introducing additional parameters by employing an attention-guided upsampling strategy. Experimental results show that our method achieved 2.8% and 2.0% mean intersection over union (mIoU) score improvements compared with the competitive baseline algorithm on the available US3D and ISPRS Potsdam datasets, respectively.

1. Introduction

As an important means of earth observation, the rapid development of remote sensing has provided humans with a new way to efficiently perceive the land and ocean on which they depend for survival [1]. Semantic segmentation of remote sensing images is one of the important applications of the intelligent interpretation of remote sensing images [2]. The core goal is to assign category labels to each pixel in the image based on sample features, and pixels with the same category label have the same visual characteristics [3]. The full-scene and high-precision ground object recognition results make remote sensing image semantic segmentation widely used in ecological monitoring [4], urban planning [5], disaster assessment [6], change detection [7], and other fields.

In recent years, driven by massive data, models, and computing power, artificial intelligence technologies represented by deep convolutional neural networks have greatly promoted the development of remote sensing semantic segmentation [8]. Due to the complexity of remote sensing images, it is impossible to effectively represent different types of objects using a single remote sensing image. For example, low vegetation and trees, gray buildings, and the ground may be misidentified. In recent years, the rapid development of remote sensing sensors and platforms has greatly increased the quantity and quality of remote sensing data [9]. Multi-source remote sensing data can provide more detailed information about land cover attributes [10]. Infrared images can reflect the radiation characteristics of land cover, visible light images can reflect visual features such as the texture and geometry of land cover, and multi-spectral images can reflect the spectral characteristics of land cover. These multi-source remote sensing images with different characteristics can effectively compensate for the limitations of single data sources, greatly improving the application scope and resolution accuracy of remote sensing images [11].

The single-source remote sensing image semantic segmentation method mainly considers the combination of low-level detail information and high-level semantic information, while the deep multi-source model further integrates intermediate or hierarchical features to improve performance by mining complementary information. A large number of improved segmentation models have been proposed to address the characteristics of multi-source remote sensing images and the limitations of existing methods. Zhou et al. [12] proposed a multi-source semantic segmentation network based on modal memory fusion and morphological multi-scale assistance to promote cross-modal feature fusion by considering the connections between different samples. In view of the fact that existing methods ignore the unique characteristics of each modality at different encoding layers, Liang et al. [13] proposed a multi-branch bidirectional fusion segmentation network, which guides the distinguishable fusion of features at different layers through a detail enhancement module and an RGB-guided semantic enhancement module. Ma et al. [14] proposed a new adjacent two-layer fusion semantic segmentation network to address the problems of small targets and complex scenes in remote sensing images. It fully explores multi-scale complementary features through a DSM enhancement module and an adjacent context exploration module. The above methods are based on the perspective of complementary feature mining and multi-stage fusion strategy design for network construction. However, while multi-source data enrich the information of ground object categories, they also cause feature redundancy and noise information. Therefore, it is a challenge to effectively mine the differential information of object categories, maximize the advantages of different types of remote sensing images, reduce the impact of noise information on segmentation performance, and achieve collaborative representation of multi-source semantic information in complex scenes.

In order to solve the above problems, we proposed a multi-source remote sensing image semantic segmentation method based on differential feature attention fusion by studying the multi-source fusion mechanism, enriching contextual information, and maintaining the diversity of fusion features. As mentioned above, the model mainly focuses on two basic issues: one is how to efficiently achieve the full fusion of multi-source remote sensing image features and improve the model’s ability to mine discriminative features, and the other is how to overcome the loss of shallow image information caused by downsampling operations in the encoding stage. For the first problem, we used differential feature fusion modules and unsupervised adversarial losses to mine the discriminative features and complementary features of multi-source data. For the second problem, we designed an upsampling module based on the attention mechanism to model the correlation between pixels. Different from existing methods, DFAFNet uses the attention mechanism to mine differential features and guide the upsampling process, which can significantly improve the performance of remote sensing image semantic segmentation tasks.

To summarize, the main contributions of this work are in four-fold:

• We propose an end-to-end multi-source semantic segmentation network based on differential feature attention fusion. The network enhances the feature expression ability of the model through the multi-source differential feature fusion mechanism and enriches the context information in the decoding stage.
• We develop a differential feature fusion module based on spatial attention. By aligning the weight distribution of multi-source feature maps and giving higher attention to the differential part, the model’s discriminative feature mining ability is improved.
• We design a shallow attention-guided upsampling method based on the self-attention mechanism to better achieve image reconstruction without introducing additional parameters.
• We use an unsupervised loss function to perform deep supervision on the feature extraction and fusion modules so that the fused features can better retain the diversity of multi-source data features.

The rest of this paper is organized as follows. In Section 2, we briefly review the current situation of the remote sensing semantic segmentation task. Section 3 introduces the details of the architecture. Section 4 gives experiments and comparisons with other methods. Finally, this paper is concluded in Section 5.

2. Related Work

In this section, relevant works on remote sensing image semantic segmentation and multi-source semantic segmentation are briefly reviewed.

2.1. Remote Sensing Image Semantic Segmentation

Remote sensing image semantic segmentation is a multi-disciplinary research field that continuously develops and improves by integrating theoretical knowledge from remote sensing image processing and computer science [15]. According to the differences in image feature extraction methods, remote sensing image segmentation methods are mainly divided into traditional image segmentation methods [16] and semantic segmentation methods based on deep learning [17,18]. Traditional image segmentation methods rely on manually designed shallow features for processing and analysis. However, due to the complexity of remote sensing image segmentation scenarios and the diversity of segmentation requirements, traditional image segmentation methods are less able to effectively meet the needs of remote sensing image semantic segmentation tasks in terms of segmentation accuracy and generalization performance [19].

With the breakthrough progress of convolutional neural networks in computer vision tasks, more and more researchers are paying attention to deep learning-based semantic segmentation methods for remote sensing images [20]. Compared to traditional image segmentation methods, deep neural networks can automatically learn abstract nonlinear high-level semantic features, greatly improving the performance of semantic segmentation tasks [21]. In 2015, the introduction of the fully convolutional network laid a solid foundation for using deep neural networks to solve the semantic segmentation problem of remote sensing images in an end-to-end manner [22]. However, it still has two limitations: First, after the image is continuously pooled, the spatial position information of the pixels is lost. Second, the segmentation process does not consider the context information and cannot fully utilize the local and global features of the image. A large number of improved semantic segmentation models for remote sensing images have been proposed, taking into account the characteristics of remote sensing images and the limitations of existing methods. They have been found to improve the model’s ability to represent features and segmentation performance through encoder–decoder structures [23], feature fusion mechanisms [24,25], attention mechanisms [21,26], Transformer techniques [27,28], and other methods.

These methods have greatly promoted the development of remote sensing image semantic segmentation technology by enhancing the representation ability of feature maps. However, due to the complex background of remote sensing images, the large difference in feature distribution within a class, and the small difference in feature distribution between classes, remote sensing image semantic segmentation is still a difficult task to accomplish.

2.2. Multi-Source Fusion Semantic Segmentation

With the rapid development of remote sensing sensors, the number and application level of remote sensing images with different imaging mechanisms are constantly increasing. It has become a development trend to use the complementarity of multi-source data to enhance the segmentation accuracy and robustness of remote sensing image semantic segmentation tasks [29]. Multi-source data can reflect the various characteristics of ground object categories.

How to fully consider the correlation of multi-source data, effectively improve the ability to extract discriminative features, and reduce the impact of redundant features on segmentation accuracy is an important research direction. Multi-source remote sensing images with different characteristics can effectively make up for the limitations of single data sources and greatly improve the application scope and analysis accuracy of remote sensing images [30]. In response to the limitations of single-modal data, Guo et al. [31] proposed a multi-modal semantic segmentation network, PIF-Net. This network extracts deep multi-modal features through two independent branches, incorporating Res-Pooling and point attention blocks. Currently, most multi-source fusion methods merely combine the features of two modalities without considering their differences and complementarity. Fan et al. [32] proposed a novel network called the progressive adjacent layer coordinated symmetric cascade network (PACSCNet). This network employs a two-stage fusion symmetric cascade encoder to utilize the similarity and differences between adjacent features for cross-layer fusion, thereby preserving spatial details. In addition, Ma et al. [33] proposed a multi-level multi-modal fusion scheme called FTransUNet, which integrates CNN and Vit into a unified fusion framework to provide a robust and effective multi-modal fusion backbone for semantic segmentation.

Multi-source data play an important role in improving the performance of remote sensing semantic segmentation. However, due to the interference of noise information carried by multi-source data and complex scenes of remote sensing images, these methods are usually difficult to adapt to the semantic segmentation task of multi-source remote sensing images. Therefore, we have conducted some explorations on the difficulty of mining complementary features and discriminative features of multi-source data based on the existing methods.

3. Method

In this section, we will introduce the model details of DFAFNet in detail. They consist of three parts: a spatial attention-based differential feature fusion module, a shallow pixel-guided upsampling operation, and an unsupervised adversarial loss function. On the one hand, the DFAFNet module uses the differences in multi-source data to extract discriminant information for feature fusion. On the other hand, it introduces the relationship between shallow pixels into the feature map reconstruction process to enhance the context information of the decoder, thereby improving the accuracy and robustness of the model.

The DFAFNet model architecture is shown in Figure 1. It can be seen from the figure that DFAFNet is a multi-branch encoding–decoding structure, which consists of two feature extraction encoding branches, a feature fusion branch, and a decoding branch.

In the encoder part of the model, each feature extraction branch contains 5 CBR modules (In this section, the convolution layer, BN layer, and ReLU function are abbreviated as CBR). For each convolution block, the size of the convolution kernel, stride, and padding are set to 3, 1, and 1 respectively. Then, the output feature map of each CBR module is downsampled using the maximum pooling operation, and the pooling kernel size and pooling stride are set to 2. Taking the multi-source input images RGB image and DSM image as examples, the RGB feature map, depth feature map, and fusion feature map can be finally obtained through the encoder feature extraction branch. The multi-source feature maps are interactively fused through a difference feature fusion module (DFF).

In the decoder part of the model, its structure is consistent with the decoder structure of the original UNet, except for the upsampling operation. This chapter replaces the traditional upsampling operation with the proposed self-attention-guided upsampling operation (AGU). Similarly, DFAFNet adds skip connections between the encoder and decoder to enrich the contextual information of the decoder. In addition, considering the significant role of residual blocks in the ResNet network, this chapter uses ResNet34 as the backbone network of the model.

3.1. Difference Feature Attention Fusion Module

Accurate feature representation is the core issue of multi-source remote sensing semantic segmentation. In order to achieve optimal multi-source feature learning, a large number of studies have been conducted to improve the performance of segmentation models by fusing the most discriminative features of multi-source remote sensing images. However, existing methods often have limitations such as information redundancy and incomplete discriminative features. In contrast, a better strategy is to extract differential features that are focused on in the depth image feature map and ignored in the visible light image feature map and then fuse them with the original visible light image feature map. The obtained fused feature map not only enhances the discriminative features, but it also effectively reduces information redundancy. Based on this idea, we designed a differential feature fusion module, which consists of two stages: one is feature alignment, and the other is differential feature mining. The structure of the fusion module is shown in Figure 2.

Feature alignment: It aims to match the activation regions of common interest between RGB image feature maps and depth image feature maps. Regions with high activation values indicate that the model pays more attention to the corresponding regions. In other words, the region has a higher impact on model performance, while regions with low activation values indicate a lower impact on model performance. For example, in the RGB image feature map, the activation response values of the forest and grassland categories are not significantly different, while in the depth feature map, the activation response value of the forest category is higher than that of the grassland category. Therefore, we used the feature activation response value to represent the importance of the features. After obtaining regions of common interest from multiple feature maps, we selected regions with different response values from the two feature maps, laying the foundation for subsequent differential feature mining.

First, we calculated the Hadamard product of the RGB image feature map $F_{rgb}$ and the depth feature map $F_{dep}$ to mine the feature areas of common interest. In order to increase the diversity of features, we also used maximum pooling and average pooling operations to process the feature map and cascade the pooled results. Given the feature map F, the calculation method of this operation can be formulated as follows:

$$\mathrm{Pool}\left(F\right)=\mathrm{MaxPool}\left(F\right)\oplus \mathrm{AvgPool}\left(F\right).$$

(1)

Secondly, we used 1D convolution to compress the number of channels in the cascaded feature maps that had undergone max pooling and average pooling to 1, thereby obtaining activation maps for different regions. Then, the sigmoid function was used to map the score of each region to between 0 and 1. The feature map extracted by the l-$th$ CBR block of the RGB image feature extraction branch is $F_{rgb}^{\left(l\right)}$, and similarly, the feature map extracted by the deep feature extraction branch is $F_{dep}^{\left(l\right)}$. In addition, in the l-$th$ difference feature fusion block, the weights of the common attention region extraction branch and the deep feature extraction branch are defined as $A_{had}^{\left(l\right)}$ and $A_{dep}^{\left(l\right)}$. Then, the calculation method of the two activation maps is as follows:

$$A_{had}^{\left(l\right)}=\mathrm{Sigmoid}(W_{had}^{\left(l\right)}⊛\mathrm{Pool}(F_{rgb}^{\left(l\right)}\otimes F_{dep}^{\left(l\right)})),$$

(2)

$$A_{dep}^{\left(l\right)}=\mathrm{Sigmoid}(W_{dep}^{\left(l\right)}⊛\mathrm{Pool}\left(F_{dep}^{\left(l\right)}\right)),$$

(3)

where ⊛ means 1D convolution, and ⊗ means Hadamard product. We can obtain activation maps $A_{had}^{\left(l\right)}$ and $A_{dep}^{\left(l\right)}$ through the above steps, which reflect the areas that the model focuses on. The former represents the important areas common to the RGB image and the depth image, while the latter represents the important areas of the depth image.

Feature complementary: The activation map reflects the importance of the model’s attention area. The higher the score, the greater the weight assigned by the model, and the more valuable the corresponding feature. To obtain the difference information between multi-source feature maps, a simple approach is to negate the above activation map. So, we need matching the $A_{had}^{\left(l\right)}$ and $A_{dep}^{\left(l\right)}$ to select the complementary regions from different feature maps. We can simplify this problem through the following logical propositions:

$${\Delta A}^{\left(l\right)}=\neg A_{had}^{\left(l\right)}\wedge A_{dep}^{\left(l\right)},$$

(4)

where $\neg A_{had}^{\left(l\right)}$ refers to areas where RGB and depth focus on inconsistencies. So, ${\Delta A}^{\left(l\right)}$ represents the regions where depth features are interesting, whereas the RGB features ignore them. We call ${\Delta A}^{\left(l\right)}$ as the complementary regions’ scoring map. In practice, we use specific mathematical operations in place of the logical operations described above:

$${\Delta A}^{\left(l\right)}=\left(1-A_{had}^{\left(l\right)}\right)\otimes A_{dep}^{\left(l\right)}.$$

(5)

Since the score of each region in the feature map is between [0, 1] by the $sigmoid$ function, we can use the $\left(1-A_{had}^{\left(l\right)}\right)$ value to replace the $\neg A_{had}^{\left(l\right)}$ value for performing the inversion operation. Under the guidance of ${\Delta A}^{\left(l\right)}$, we can select features unique to the depth feature map to strengthen the RGB features. Firstly, we purify the RGB features by utilizing $A_{had}^{\left(l\right)}$. $A_{had}^{\left(l\right)}$ only focuses on the regions that RGB images and depth images both concern, so it can significantly filter out the background noise of RGB feature maps. Then, we extract the features unique to the depth images by ${\Delta A}^{\left(l\right)}$. Finally, we obtain the fusion feature maps $F_{fuse}^{\left(l\right)}$. The operations above can be formulated as follows:

$$F_{fuse}^{\left(l\right)}=A_{had}^{\left(l\right)}\otimes F_{rgb}^{\left(l\right)}+{\Delta A}^{\left(l\right)}\otimes F_{dep}^{\left(l\right)}.$$

(6)

3.2. Attention-Guided Upsampling Module

In semantic segmentation networks based on encoder–decoder structures, the quality of feature upsampling has a non-negligible impact on segmentation performance. However, traditional upsampling operations are mainly bilinear interpolation and transposed convolution, which have limitations to varying degrees because different regions share the same sampling weights. Therefore, we proposed a novel attention-guided upsampling module to improve segmentation performance. The structure of the attention-guided upsampling module is shown in Figure 3.

Given the image feature map F, we denote the feature vector set of feature map F as ${\mathbf{X}}_F$ and the feature vector for the pixels in column j of row i as ${\mathbf{x}}_{ij}\in {\mathbf{R}}^d$, where d means the embedding dimension of feature vectors. So, any feature vector ${\mathbf{x}}_{ij}$ which belongs to F is an element of ${\mathbf{X}}_F$. Assuming that the elements of the set ${\mathbf{X}}_F$ are distributed in a low-dimensional manifold space, downsampling on the feature map ${\mathbf{X}}_F$ should meet the following conditions: (a) The number of feature map’s feature vectors should decrease after downsampling. (b) In the process of downsampling, the information loss should be as little as possible. Thus, the downsampling can be described as the process to select the ${\mathbf{X}}_F$ value’s real subset ${\mathbf{X}}_s$, and ${\mathbf{X}}_s$ should contain a set of bases. That means any feature vector ${\mathbf{x}}_i$ in the set ${\mathbf{X}}_F$ can be represented linearly by the elements of ${\mathbf{X}}_s$.

In practice, considering that the number n of elements of $\mathbf{X}$ is much larger than the embedding dimension d ($n\gg d$), and there is a lot of redundant information in the image features, the above conditions in the definition are very easy to be satisfied for 2x downsampling according to the $DrawerPrinciple$. So, if we obtain a proper ${\mathbf{X}}_s$ after downsampling, any feature vector ${\mathbf{x}}_i$ in the set ${\mathbf{X}}_F$ can be described as follows:

$${\mathbf{x}}_i=\sum _{k=1}^{n_s}{\alpha }_k{\mathbf{x}}_k,{\mathbf{x}}_k\in {\mathbf{X}}_s,$$

(7)

where ${\alpha }_k$ defines the weights for different vectors ${\mathbf{x}}_k$. We can organize the elements of ${\mathbf{X}}_F$ and ${\mathbf{X}}_s$ as matrix $X_F$ and matrix $X_s$ separately, and we can then combine corresponding weights ${\alpha }_k$ as weight matrix W. Thus, we convert Equation (7) to the equation in the form of matrices as follows:

$${\mathbf{X}}_F={\mathbf{W}}^{\top }{\mathbf{X}}_s,$$

(8)

where $\mathbf{W}$ means the relationships between the elements of ${\mathbf{X}}_F$ and the elements of ${\mathbf{X}}_s$. Consider ${\mathbf{X}}_s\subset {\mathbf{X}}_F$; thus, matrix W refers to the 2nd-order information of images. According to the Equation (8), the quality of upsampling is decided by two factors: (a) the quality of $X_s$ and (b) the 2rd-order information W. To illustate this clearly, we can focus on the ith decoder block, which would process the feature maps from (i − 1)th decoder block. Between the ith decoder block and (i − 1)th decoder block, we should upsample the feature map from (i − 1)th decoder block first, where the feature map can be seen as the source of matrix ${\mathbf{X}}_s$. To improve the quality of ${\mathbf{X}}_s$, it is a good idea to directly utilize the encoded feature from the encoder block in the same level as the ith decoder block. That is why U-net adds the skip connections between the encoder and decoder. For the 2rd-order matrix W, which is often ignored in related research, it is obvious that we can compute it in the encoded feature from the same level encoder block and transit it by the skip connection.

According to the analysis above, we designed the algorithm to improve the upsampling results by introducing the 2nd-order information of images. In this way, the relationship between the pixel to be reconstructed and all global pixels can be established, and this relationship has an adaptive characteristic, that is, different weights can be obtained by inputting different feature maps into the module.

Given the i-$th$ layer encoding feature map $F_i^{enc}$ and the i-$th$ layer decoding feature map $F_i^{dec}$, the i + 1-$th$ layer encoding feature map $F_{i+1}^{enc}$ is obtained by downsampling the feature map $F_i^{enc}$. Since feature maps $F_i^{dec}$ and $F_i^{enc}$ are at the same level, they have the same image size. Therefore, the encoding feature map $F_i^{enc}$ and the decoding feature map $F_i^{dec}$ can be cascaded to obtain a fused feature map $F_i$ to enhance the context representation of the feature map. The calculation method is as follows:

$$F_i=F_i^{dec}\oplus F_i^{enc}.$$

(9)

During the process of downsampling $F_i^{enc}$ to generate $F_{i+1}^{enc}$, we can record the position information between pixels to guide the upsampling process of the decoded feature map $F_{i+1}^{dec}$. In this paper, this position information is defined as indices $\mathbf{I}$. Next, we start to calculate the 2nd-order relationship matrix $\mathbf{W}$. According to indices $\mathbf{I}$, we can divide the feature map $F_i^{enc}$ into two parts: $F_i^a$ and $F_i^b$. $F_i^a$ contains all feature vectors indexed by $\mathbf{I}$, and the remaining vectors constitute $F_i^b$. We convert $F_i^a$ and $F_i^b$ separately to $({\mathbf{X}}_a,{\mathbf{I}}_a)$, and $({\mathbf{X}}_b,{\mathbf{I}}_b)$, where ${\mathbf{X}}_a$ and ${\mathbf{X}}_b$ are all matrices, while ${\mathbf{I}}_a$ and ${\mathbf{I}}_b$ are all indices indicating each feature vector’s position in the original feature map of the corresponding set. For convenience of calculation, we borrowed the self-attention mechanism to simulate the matrix $\mathbf{W}$:

$$W_{ij}={\displaystyle \frac{exp\left({\mathbf{x}}_i^{\top }{\mathbf{y}}_j/\sqrt{d}\right)}{{\sum }_{k=1}^{m}exp\left({\mathbf{x}}_k^{\top }{\mathbf{y}}_j/\sqrt{d}\right)}},{\mathbf{x}}_i\in {\mathbf{X}}_a,{\mathbf{y}}_j\in {\mathbf{X}}_b.$$

(10)

So far, we can obtain the 2nd-order information $\mathbf{W}$ of the image through the above steps, which can guide the upsampling process of the decoder. Then, the final feature map after upsampling can be generated according to the position indexes ${\mathbf{I}}_a$ and ${\mathbf{I}}_b$.

3.3. Loss Function

The loss function used in this paper consists of two parts: one is the unsupervised adversarial loss function, and the other is the semantic segmentation loss.

In order to better extract fusion features from multi-source data and maintain the diversity of fusion features, we used an unsupervised adversarial loss function for deep fusion supervision of feature encoders. Abandoning the segmentation results, an effective multi-source fusion feature should be as close to a certain data source as possible while not being too biased towards a certain data source, otherwise the multi-source fusion feature extraction will degenerate into a single-source feature extraction. Therefore, we introduced the adversarial idea when designing the loss function to supervise the fusion process of multi-source data. The specific calculation method is as follows:

At the end of encoder, we have three feature maps: the fusion feature map $F_{fuse}$, the RGB feature map $F_{rgb}$, and the depth feature map $F_{dep}$. There is a constraint between the three feature maps, that is, the final fusion feature map $F_{fuse}$ can neither be too similar to the RGB feature map $F_{rgb}$ nor too similar to the depth feature map $F_{dep}$, otherwise the use of multi-source data will lose its meaning. So, we introduced the idea of adversariality when designing the loss function to supervise the fusion process:

$$L_{adv}=L(F_{fuse},F_{rgb})+L(F_{fuse},F_{dep}),$$

(11)

where $L(.)$ defines the PIKD function [34], which can be described as follows:

$$L(F_1,F_2)=\sqrt{{(\mathrm{AvgPool}\left(F_1\right)-\mathrm{AvgPool}\left(F_2\right))}^2+{\epsilon }^2}.$$

(12)

where $\epsilon$ is a balancing term, which prevents the model from overfitting and improves the generalization ability of the model. In this paper, the value of $\epsilon$ used is 1 × 10⁻⁶. The PIKD function can measure the distance between two feature distributions. Its loss reaches the global minimum only when the fused feature distribution lies between the multiple source feature distributions. Minimizing the distance among these three feature distributions inherently forms an adversarial relationship.

Next, we introduced the Lovász Softmax loss function [35], which is calculated as follows:

$$L_{seg}={\displaystyle \frac{1}{\left|C\right|}}\sum _{c\in C}\Delta \overline{J_c}\left(m_i\left(c\right)\right),$$

(13)

where $\left|C\right|$ represents the total number of categories, $J_c$ is the Jaccard similarity coefficient, $m_i\left(c\right)$ is the error vector, and $\Delta J_c(·)$ represents the convex closure of the Jaccard loss.

In summary, the loss function of the difference feature attention fusion model can be expressed as follows:

$$\mathrm{Loss}=L_{adv}+L_{seg}.$$

(14)

4. Experiments

In order to fully verify the effectiveness of the DFAFNet method, we conducted a series of experiments on two public datasets. First, we introduced the datasets, experimental details, evaluation indicators, and comparison methods. Secondly, we conducted an ablation experiment on the proposed method to verify the effectiveness of the module. Finally, we demonstrated the performance of the DFAFNet method by comparing it with other advanced methods in terms of indicator results and visual effects.

4.1. Experimental Conditions

Datasets. Two public and commonly used multi-modal remote sensing data were used in our experiments. One of them is Urban Semantic 3D (https://ieee-dataport.org/open-access/data-fusion-contest-2019-dfc2019/, accessed on 8 October 2024) (US3D); the other is ISPRS Potsdam Semantic Labeling data set (https://www.isprs.org/education/benchmarks/UrbanSemLab/2d-sem-label-potsdam.aspx, accessed on 8 October 2024)) (ISPRS Postdam).

Table 1. The detailed information of US3D and ISPRS datasets for DFAFNet.

Dataset	Attributes	Categories
	Name	Ground	Vegetation	Building	Water	Road	Background
	Color
	Training	63.05%	15.8%	15.34%	4.46%	1.34%	0.01%
	Testing	73.65%	14.46%	9.08%	1.93%	0.86%	0.02%
	Name	Low vegetation	Tree	Car	Building	Surface	Background
	Color
	Training	24.28%	16.11%	1.86%	26.91%	30.83%	0.01%
	Testing	22.28%	15.95%	1.89%	28.26%	31.61%	0.01%

shows examples and legends of the two datasets. The table shows the categories included in the dataset and the corresponding proportions and color markings. In the US3D dataset, the image size is 512 × 512, and the ground sampling distance is 1.3 m. We obtained a total of 10,848 images by cropping. Among them, 8692 images were used for training, and 2156 images were used for testing. There are five labels in this dataset, and the background class is ignored. In ISPRS Potsdam, the image size is 512 × 512, the ground sampling distance is 5 cm, and a total of 4598 images were obtained by cropping. Among them, 3678 images were used for training, and 920 images were used for testing. The first category in the original dataset was changed to the background class.

Implementation Details. In this paper, the deep learning framework we used is PyTorch, the model optimizer was Adam, the initial learning rate was 0.0001, and the momentum and weight decay coefficients were set to 0.9 and 0.0005, respectively. When conducting experiments on the US3D dataset, the model was iteratively trained for 150 times, the learning rate was reduced by half every 100 iterations, and the batch size was set to 15. When conducting experiments on the ISPRS Postdam dataset, the model was iteratively trained for 100 times, the learning rate was reduced by half every 50 iterations, and the batch size was set to 6. An NVIDIA Tesla P100 GPU (12G) was used to train and test the network.

Evaluation Metrics. Five different evaluation metrics were adopted to measure the performance quantitatively between the DFAFNet and the comparison methods. In this paper, we used mIoU as our main metrics. The specific calculation method of the evaluation indicators is as follows:

(1) Pixel Accuracy (PA): It measures the ratio of correctly classified pixels to the total number of pixels, as calculated by Equation (15).

$$PA={\displaystyle \frac{{\sum }_{i=0}^k{p}_{ii}}{{\sum }_{i=0}^k{\sum }_{j=0}^k{p}_{ij}}},$$

(15)

where k represents the number of classes, $p_{ii}$ denotes the number of pixels correctly classified, and $p_{ij}$ denotes the number of pixels where the true class i is predicted as class j.

(2) Mean Pixel Accuracy (MPA): It measures the average accuracy of all class pixels, as calculated by Equation (16).

$$MPA={\displaystyle \frac{1}{k+1}}\sum _{i=0}^k{\displaystyle \frac{p_{ii}}{{\sum }_{j=0}^k{p}_{ij}}}.$$

(16)

(3) Mean Intersection Over Union (mIoU): It measures the ratio of the intersection to the union of the predicted pixel set and the true pixel set, as calculated by Equation (17).

$$mIoU={\displaystyle \frac{1}{k+1}}\sum _{i=0}^k{\displaystyle \frac{p_{ii}}{{\sum }_{j=0}^k{p}_{ij}+{\sum }_{j=0}^k{p}_{ji}-p_{ii}}},$$

(17)

where $p_{ji}$ denotes the number of pixels, where class j is predicted as the true class.

(4) Frequency-Weighted Intersection Over Union (FWIoU): It is an improvement to the mIoU metric, where the weight for each class is set based on its frequency of occurrence, as calculated by Equation (18).

$$FWIoU={\displaystyle \frac{1}{{\sum }_{i=0}^k{\sum }_{j=0}^k{p}_{ij}}}\sum _{i=0}^k{\displaystyle \frac{p_{ii}{\sum }_{j=0}^k{p}_{ij}}{{\sum }_{j=0}^k{p}_{ij}+{\sum }_{j=0}^k{p}_{ji}-p_{ii}}}.$$

(18)

(5) Kappa: It measures the consistency between the model’s prediction and the true results. The higher the Kappa coefficient, the more accurate the semantic segmentation result of the remote sensing image. The calculation method is shown in Equation (19).

$$Kappa={\displaystyle \frac{N·tr\left(\mathbf{M}\right)-\nu {\mathbf{M}}^2{\nu }^{\mathrm{T}}}{N-\nu {\mathbf{M}}^2{\nu }^{\mathrm{T}}}},$$

(19)

where N represents the number of pixels in the image, $tr\left(\mathbf{M}\right)$ denotes the trace of the confusion matrix $\mathbf{M}$, and v is a row vector of length 2 with all elements equal to 1.

Comparative Methods. In the experiments, we compared the following seven semantic segmentation methods in the past four years to verify the effectiveness of the proposed model: the Unified Perceptual Parsing Network (UperNet) [36], Multi-Path Residual Network (MP-ResNet) [37], Dual-Domain Optimized Class-Aware Network (DOCNet) [38], Attention-Based Network (ACNet) [39], Efficient Scene Analysis Network (ESANet) [40], Fusion Height Network (FHNet) [41], and Multi-Source Segmentation Fusion Network (SegFusion) [42].

4.2. Ablation Studies

We conducted a series of ablation experiments on the US3D and ISPRS Potsdam datasets to verify the effectiveness of the key modules in the DFAFNet. The ablation points of the DFAFNet method include the two-stream structure, the difference feature fusion module, the attention-guided upsampling module, and the unsupervised adversarial loss. We used the dual-stream structure as the baseline model for comparison, which is an improved UNet network based on dual encoders. The ablation experiment results are shown in

Table 2. The results of ablation studies on DFAFNet ¹. “DS” denotes the dual-stream structure, “DFF” is the difference feature fusion module, “AGU” means the attention-guided upsampling operation, “UAL” denotes the unsupervised adversarial loss function. The values of the evaluation metrics in bold indicate the best results.

Datasets	DS	DFF	AGU	UAL	Metrics
					PA	mIoU	FWIoU	Kappa
US3D	✓	-	-	-	93.15	79.53	87.74	84.34
	✓	✓	-	-	93.57	80.83	88.33	85.04
	✓	✓	-	✓	93.69	81.22	88.50	85.26
	✓	✓	✓	✓	93.73	82.33	88.62	85.42
ISPRS	✓	-	-	-	89.59	80.73	81.49	86.02
	✓	✓	-	-	90.23	81.42	82.60	86.89
	✓	✓	-	✓	90.43	81.80	82.96	87.16
	✓	✓	✓	✓	91.07	82.73	83.95	88.01

¹ “✓” indicates that the corresponding module is retained. “-” indicates that the corresponding module is removed.

.

First, we analyzed the index results of the DFAFNet components on the US3D dataset. From the upper part of Table 2, it can be seen that the difference feature fusion module increased the PA value, mIoU value, and Kappa value of the baseline model by 0.42%, 1.3%, and 0.7%, respectively. Using unsupervised adversarial loss on the basis of the difference feature fusion module increased the mIoU value and Kappa value of the baseline model by 1.69% and 0.92%, respectively. The combination of the three modules increased the mIoU and Kappa values of the baseline model by 2.8% and 1.08%, respectively.

Second, we analyzed the index results of the DFAFNet components on the ISPRS dataset. From the lower part of Table 2, it can be seen that the difference feature fusion module increased the PA, mIoU, and Kappa values of the baseline model by 0.64%, 0.69%, and 0.87%, respectively. Using unsupervised adversarial loss on the basis of the difference feature fusion module increased the mIoU and Kappa values of the baseline model by 1.07% and 1.14%, respectively. The combination of the three modules improved the mIoU and Kappa values of the baseline model by 2.0% and 1.99%, respectively.

In the ablation experiments on the two datasets, we found that the complete architecture achieved the highest performance. If any of the three modules—the difference feature fusion module, the attention upsampling module, and the unsupervised adversarial loss—is removed, the performance of the indicators will decrease. Therefore, it can be determined that the improvement in DFAFNet’s performance is due to the effective design of these modules.

In addition, to further demonstrate the performance of the designed modules, we also counted the average pixel accuracy MPA of different module combinations in each category, and the results are shown in

Table 3. The performance of different modules combinations on each category. The values of the evaluation metrics in bold indicate the best results.

Datasets	DS	DFF	AGU	UAL	Categories
					Ground	Vegetation	Building	Water	Road
US3D	✓	-	-	-	84.68	83.52	91.14	93.74	90.62
	✓	✓	-	-	85.35	84.22	91.61	94.67	91.66
	✓	✓	-	✓	86.60	83.39	91.63	95.18	92.37
	✓	✓	✓	✓	86.93	85.28	91.41	95.71	93.05

“✓” indicates that the corresponding module is retained. “-” indicates that the corresponding module is removed.

. It can be seen from the table that DFAFNet achieved the best MPA in the four categories of “ground”, “vegetation”, “water”, and “road” on the US3D dataset. Although the accuracy of the “building” category did not improve, it was not much different from the best result. The reason may be that the addition of multi-source elevation information greatly improved the segmentation accuracy of the building category. Therefore, only the baseline model was needed to increase the segmentation accuracy of the building category to more than 91%.

Compared to the baseline model, the proposed method has obvious advantages in each category of the US3D dataset. As the number of modules continued to increase on the baseline model, the average segmentation accuracy of the three categories of “ground”, “water”, and “road” showed a gradual improvement trend. However, after adding the unsupervised adversarial loss, the segmentation effect of the “vegetation” category did not increase but decreased. The reason for this phenomenon may be that the distinction between this category and the surrounding categories is not high, so the addition of the unsupervised adversarial loss caused the model to overfit. Overall, the three proposed modules are all conducive to the segmentation of remote sensing images.

4.3. Comparative Experiments and Discussions

4.3.1. Comparative Experiments on US3D Data Set

The comparative experimental results of the DFAFNet method and other advanced methods on the US3D dataset are shown in

Table 4. Comparison results of DFAFNet and other methods on the US3D dataset. The values of the evaluation metrics in bold indicate the best results.

Methods	Metrics
	PA	MPA	mIoU	FWIoU	Kappa
UperNet [36]	87.96	75.82	67.79	79.11	70.79
MP-ResNet [37]	89.14	80.73	72.54	80.88	73.57
DOCNet [38]	88.71	79.39	70.37	81.39	75.08
ACNet [39]	90.78	75.87	60.97	84.38	79.05
ESANet [40]	92.92	80.05	72.41	87.26	83.43
FHNet [41]	91.92	87.79	79.39	85.58	82.34
SegFusion [42]	93.18	89.36	81.54	87.63	84.71
DFAFNet	93.73	90.46	82.33	88.62	85.42

.

From the table, we can see that DFAFNet outperformed the other seven comparison methods in all five evaluation indicators. Except for ACNet, the methods with multi-source data as input were better than the methods with single-source data as input in all five indicators. It fully proves the research motivation of this paper. Among the three methods with single-source data as input, MP-ResNet achieved good performance, and its results are comparable to those of the network designed specifically for multi-source data. Compared with the suboptimal model SegFusion, DFAFNet achieved a certain degree of improvement in all five indicators, and the PA increased by 0.55%, MPA increased by 1.1%, mIoU increased by 0.79%, FWIoU increased by 0.99%, and Kappa increased by 0.71%.

In order to further visualize the segmentation performance of DFAFNet and other comparison methods on the US3D dataset, we also compared the segmentation visualization results, as shown in Figure 4.

As can be seen from the figure, all methods showed good segmentation performance for the recognition of “water” and “vegetation”. The reason may be that the appearance of these two categories is not easily confused with other categories, and the number of samples is large. In the four rows (a), (b), (d), and (f), most methods had low recognition accuracy for the “road” category and the “building” category. This is because the colors of these two categories are very close in the US3D dataset. However, this situation did not occur in the DFAFNet proposed in this paper. On the one hand, the fusion strategy adopted by DFAFNet fully exploited the discriminative features and complementary features. On the other hand, the attention-guided upsampling module of this method enhanced the contextual information of the decoder.

4.3.2. Comparative Experiments on ISPRS Data Set

In order to further verify the generalization performance of DFAFNet, we also selected the ISPRS dataset and the same comparison method and evaluation index for comparative analysis. The results are shown in

Table 5. Comparison results of DFAFNet and other methods on the ISPRS dataset. The values of the evaluation metrics in bold indicate the best results.

Methods	Metrics
	PA	MPA	mIoU	FWIoU	Kappa
UperNet [36]	88.99	88.43	79.52	80.56	85.22
MP-ResNet [37]	89.76	88.96	80.39	81.80	86.27
DOCNet [38]	89.24	89.03	80.78	81.57	86.53
ACNet [39]	86.99	85.57	75.31	77.49	82.57
ESANet [40]	85.03	83.67	70.76	74.65	79.91
FHNet [41]	88.79	85.35	79.83	80.30	85.84
SegFusion [42]	90.23	89.41	81.36	81.72	87.13
DFAFNet	91.07	90.12	82.73	83.95	88.01

.

From the table, we can see that DFAFNet also achieved the best performance compared with the other seven comparison algorithms on the ISPRS dataset. The mIoU value reached 82.73%. Unlike the methods with a single source as input on the US3D dataset, which are generally worse than the methods with multiple sources as input, on the ISPRS dataset, the three methods with a single source as input, UperNet, MP-ResNet, and DOCNet, performed well. It may be due to the lower segmentation difficulty of the ISPRS dataset compared to the US3D dataset. For example, the MPA of the comparison method on the US3D dataset was about 80%, while on the ISPRS dataset, this indicator was about 86%. Compared to the suboptimal model SegFusion, DFAFNet improved the mIoU and Kappa values by 1.37% and 0.88%, respectively, on the ISPRS dataset.

In order to vividly demonstrate the segmentation performance of DFAFNet and other methods on the ISPRS dataset, the segmentation visualization comparison results are shown in Figure 5. It can be seen from the figure that the segmentation results of DFAFNet are visually superior to the other seven comparison methods. In particular, the DFAFNet prediction results are almost consistent with the label map in the three rows (a), (e), and (f). Although other methods also yielded high segmentation accuracy, DFAFNet’s accuracy was significantly higher than other methods for the recognition of the “building” category. The reason may be that the "building" category occupies a larger area in the image, which improved the model’s fitting ability for this category.

In addition, in order to clearly show the segmentation performance of various methods in the ISPRS dataset, we show the confusion matrix of eight methods, and the results are shown in Figure 6. In the confusion matrix, the values on the diagonal represent the percentage of correct classifications, while the rest of the positions represent the percentage of incorrect classifications. From the figure, it can be seen that DFAFNet is slightly lower than DOCNet on the diagonal except for the “tree” category, but it is higher than the other seven comparison methods in other categories. The segmentation accuracy of all methods for the “tree” and “vegetation” categories is lower than that of the other three categories, which is due to the similar appearance of these two categories. Among all the comparison methods, ESANet has the lowest category segmentation accuracy compared with other methods. The reason may be that its fusion strategy cannot effectively process redundant information.

5. Conclusions

In this paper, we focused on multi-source discriminative feature mining and enhancing the quality of decoder feature reconstruction. We proposed a multi-source remote sensing semantic segmentation method based on differential feature attention fusion. It consists of three parts: differential feature fusion module, attention-guided upsampling mechanism, and unsupervised adversarial loss. On the one hand, the method proposed in this paper can achieve sufficient fusion of multi-source remote sensing image features and enhance the mining ability of the model’s discriminative features. On the other hand, it can overcome the loss of shallow information in the image caused by downsampling operations during the encoding stage, thereby improving the quality of feature reconstruction. In addition, it can also preserve the diversity of multi-source data fusion features. In future work, we will optimize the structure of multi-source fusion networks to improve the segmentation accuracy of edge features and small targets.