Transform Dual-Branch Attention Net: Efficient Semantic Segmentation of Ultra-High-Resolution Remote Sensing Images

Abstract

With the advancement of remote sensing technology, the acquisition of ultra-high-resolution remote sensing imagery has become a reality, opening up new possibilities for detailed research and applications of Earth’s surface. These ultra-high-resolution images, with spatial resolutions at the meter or sub-meter level and pixel counts exceeding 4 million, contain rich geometric and attribute details of surface objects. Their use significantly improves the accuracy of surface feature analysis. However, this also increases the computational resource demands of deep learning-driven semantic segmentation tasks. Therefore, we propose the Transform Dual-Branch Attention Net (TDBAN), which effectively integrates global and local information through a dual-branch design, enhancing image segmentation performance and reducing memory consumption. TDBAN leverages a cross-collaborative module (CCM) based on the Transform mechanism and a data-related learnable fusion module (DRLF) to achieve adaptive content processing. Experimental results show that TDBAN achieves mean intersection over union (mIoU) of 73.6% and 72.7% on DeepGlobe and Inria Aerial datasets, respectively, and surpasses existing models in memory efficiency, highlighting its superiority in handling ultra-high-resolution remote sensing images. This study not only advances the development of ultra-high-resolution remote sensing image segmentation technology, but also lays a solid foundation for further research in this field.

1. Introduction

Under the impetus of globalization and digitization, remote sensing technology has emerged as a cornerstone of geographic information science, offering a fresh perspective on Earth exploration [1]. Ultra-high-resolution remote sensing images, with pixel counts exceeding 4 million, reveal finer texture details and object characteristics compared to lower resolutions [2,3,4]. In particular, in fields such as automated road detection and multi-angle urban classification analysis, these images demonstrate significant advantages, greatly improving decision-making precision [5,6]. However, handling these complex image data poses significant challenges in memory management and computational efficiency, increasing the need for manual intervention and potentially causing delays in critical applications such as emergency response [2]. Despite advancements in semantic segmentation technology offering new opportunities for understanding and automating the annotations of these ultra-high-resolution images, developing more efficient and accurate segmentation algorithms remains a critical issue in remote sensing image processing.

In recent years, deep learning technologies have made significant advances in the field of remote sensing image semantic segmentation, leading to innovations in image analysis and interpretation techniques. Classical methods like U-Net and its variants [7,8,9], multi-scale context aggregation networks [10,11,12], and multi-level feature fusion networks [13,14,15] have demonstrated strong segmentation performance. Furthermore, due to their advantages in enhancing key feature identification and boosting model efficiency, attention mechanisms (i.e., focusing on relevant information and ignoring irrelevant details) have been widely used in several models [16,17,18]. Meanwhile, Transformer-based and generative adversarial network (GAN)-based methods have also gained increasing attention in recent years [19,20,21]. For example, integrating Transformer models with U-Net and the novel guided focal attention mechanism has demonstrated outstanding segmentation accuracy, further enhancing their feature representation ability [22]. These advancements highlight a clear trend of combining Transformer-based models with attention mechanisms to address the challenges posed by high-resolution remote sensing imagery. Nevertheless, a persistent challenge remains in balancing segmentation quality with computational efficiency and memory usage, particularly as the demand for processing ultra-high-resolution images continues to rise. Therefore, there is increasing focus on models that can balance fine-grained segmentation needs with memory and computational efficiency, which is a critical area in current research.

For example, Chen et al. [2] and Shan et al. [3] found that, when segmenting a 6-million-pixel DeepGlobe ultra-high-resolution remote sensing image, high-precision models such as FCN-8s [23] and DANet [24] require 5 GB to 10 GB of GPU memory for inference. Although these models offer high accuracy, they also come with high memory consumption. In contrast, fast segmentation methods such as ICNet [25] and BiSeNet [26] significantly reduce memory usage but at the cost of accuracy drops ranging from 9.6% to 30%. These data indicate the difficulty that traditional model designs face in striking a balance between accuracy and memory efficiency when handling ultra-high-resolution remote sensing imagery.

To address this challenge, various methods specifically designed for ultra-high-resolution image segmentation have emerged [2,27,28]. These methods are based on networks that jointly model features using dual-branch structures, significantly reducing the dependence on high-performance hardware while maintaining segmentation precision. For example, GLNet2 [2] uses a global and local dual-branch structure, with the global branch extracting contextual information and the local branch capturing image details, both of which together improve segmentation accuracy and optimize memory usage. Although it offers advantages in accuracy and memory usage compared to traditional methods, the image block processing strategy increases computational burden, resulting in a slower inference speed. To further optimize the segmentation performance and memory usage, ISDNet [27] proposed a new method that captures semantic information at different levels using a dual-branch structure and integrates this information via a relation-aware feature fusion module. Experiments conducted on several ultra-high-resolution datasets demonstrate the model’s strong generalizability. However, due to the complexity of the relation-aware feature fusion module, it is challenging to understand, and concerns remain regarding the reliability of its actual performance. To resolve this, RSDNet [28] introduced a simplified Res-CBAM module to replace the relation-aware feature fusion module in ISDNet, boosting feature extraction effectiveness and improving the segmentation accuracy. However, while the model shows improvements in memory efficiency and computational load balancing, its inference speed is still slower than ISDNet, though it excels in large-scale crop scenarios, indicating that there is still potential to further optimize input processing and the model.

Based on these challenges, we propose TDBAN, a dual-branch feature enhancement framework for ultra-high-resolution remote sensing image segmentation. TDBAN optimizes the conventional dual-branch structure using a lightweight Transformer architecture and incorporates a cross-sharing module to efficiently fuse global and local information in one step. Additionally, TDBAN incorporates a data-related learnable fusion module (DRLF) that adaptively adjusts the weights of global and local features, significantly reducing computational costs while maintaining high segmentation accuracy. In general, our work significantly contributes to the following aspects.

(1)

Optimized Dual-Branch Network Architecture: Drawing inspiration from the GLNet framework, optimizations are made to process global and local information in ultra-high-resolution remote sensing images to achieve more effective information processing and feature recognition.

(2)

Innovative Sharing Strategy: An innovative cross-collaboration module is introduced, achieving perfect feature fusion in one step through the Transform mechanism and significantly enhancing model efficiency through lightweight design.

(3)

Dynamic weight adjustment: The newly developed DRLF module automatically adjusts weights for global and local information based on image content characteristics, optimizing information fusion to improve segmentation accuracy and processing speed. Additionally, the module adopts knowledge distillation techniques, effectively reducing the computational complexity of the model.

(4)

Significant performance improvement: The above techniques demonstrated significant performance enhancements on the DeepGlobe and Inria Aerial datasets, validating the network’s efficiency.

2. Related Work

2.1. Transformer in Semantic Segmentation

Transformer was originally applied to Natural Language Processing, utilizing the self-attention mechanism to effectively capture global contextual information. In recent years, Transformer has successfully expanded into the field of Computer Vision, especially showcasing strong modeling capabilities in semantic segmentation tasks. For example, SETR [29] pioneered the use of a pure Transformer structure to replace traditional CNN encoders, enabling comprehensive global feature modeling. However, the computational complexity of the Transformer grows quadratically with input resolution when processing image tasks, significantly restricting its use in ultra-high-resolution image segmentation. To address this issue, Swin Transformer [30] effectively reduced the computational complexity through sliding windows and hierarchical design, while maintaining the ability to model the global context. Enhanced versions of the Swin Transformer have also found extensive applications in remote sensing image segmentation [31,32,33].

In ultra-high-resolution remote sensing image segmentation tasks, computational efficiency and memory usage are critical challenges. To optimize the performance of traditional Transformer models, some studies have proposed lightweight designs. For example, SegFormer [34] introduces a hierarchical Transformer encoder that avoids the use of positional encoding, reducing performance degradation caused by varying test resolutions, while simplifying the model structure with a lightweight MLP decoder, significantly improving computational efficiency. UniFormer [35] combines 3D convolution with Transformer, proposing an efficient spatiotemporal representation learning framework that leverages the fusion of local convolution and global self-attention, effectively reducing computational complexity while maintaining a strong ability to capture spatiotemporal features. Another study, Masked-attention Mask Transformer [36], introduces a masked attention mechanism in Transformer to reduce computations in irrelevant background areas, thus improving efficiency in image segmentation tasks. These studies provide important references for high-resolution image segmentation tasks, but existing methods still exhibit significant limitations in fusing global and local features [37].

2.2. Dual-Branch Architectures for Feature Fusion

The dual-branch network structure is a widely adopted design for the collaborative processing of global and local features in semantic segmentation tasks. For instance, GLNet [2] utilizes a global–local dual-branch structure: the global branch extracts contextual information while the local branch captures image details, together improving segmentation accuracy and memory efficiency. However, while this approach offers advantages in accuracy and memory efficiency, the incorporation of image patch processing significantly increases computational overhead and reduces inference speed. Furthermore, GLNet [2] imposes mandatory information exchange between global and local streams, neglecting the varying importance of branches or the resolution requirements of complex regions, thereby constraining its multi-branch architecture’s effectiveness. Consequently, the efficient integration of global and local features for high-resolution images remains an area with considerable research potential.

2.3. Attention Mechanisms in Feature Interaction

The application of attention mechanisms in feature interaction has been proven to be a key method for improving the segmentation performance [38,39,40]. Among them, the self-attention mechanism, as the core component of the Transformer, effectively improves the modeling of contextual information by capturing long-range dependencies within the input sequence [41,42,43]. For example, Focal-UNET utilizes the focal self-attention mechanism to integrate local and global information within a single feature sequence, demonstrating excellent segmentation performance in complex scenes with multi-scale objects [44]. Channel2dTransformer directly applies self-attention to 2D feature maps, reducing the computational complexity of traditional methods while preserving spatial structures [42]. However, self-attention mechanisms generally focus on intra-sequence relationships within a single feature sequence, which makes modeling interactions between features from different sources difficult, especially in scenarios where global context and local details must collaborate. Therefore, this study introduces a cross-attention mechanism that supports cross-modal interaction and simplifies the Transformer architecture, enhancing global and local feature interactions while effectively reducing computational complexity.

Additionally, channel attention (CA) and spatial attention (SA) are widely used for feature enhancement. CA emphasizes important features by assigning weights to different channels, while SA focuses on enhancing information in key regions. Research indicates that the integration of CA and SA can markedly enhance segmentation performance [45,46]. For example, the Zoom Module in MBNet [3] utilizes a multi-scale attention mechanism to enhance focus on features at different resolutions, but its high computational complexity makes it unsuitable for resource-constrained scenarios.

To address the aforementioned issues, Knowledge Distillation (KD) technology has been widely applied in lightweight multi-resolution feature learning in recent years [47,48,49,50]. KD achieves a compromise between lightweight design and high performance by utilizing a high-performing Teacher Network to train the Student Network. For example, stonet-S* [48] utilizes KD to enhance lightweight multi-resolution feature learning, achieving outstanding segmentation accuracy with just 7.19 million parameters, thereby improving its efficacy in challenging remote sensing settings. Inspired by this, we propose a data-driven learnable fusion (DRLF) module, which simplifies the complex feature modeling at a lower computational cost by distilling the attention output of the Zoom Module, while preserving its performance. This module integrates channel and spatial attention and achieves the efficient fusion of global and local features through dynamic weight adjustment, offering a lightweight and efficient solution for high-resolution remote sensing image segmentation tasks.

3. Method

3.1. Overview

The entire network architecture is divided into two main pathways (Figure 1): the global and local branches. The architecture begins with processing a dataset D, containing N pairs of high-resolution images and their corresponding segmentation maps, each image and segmentation map being matrices of size $H\times W$. The global branch G processes downsampled low-resolution image and label pairs, denoted as

$$\mathit{G}={\left\{\left({\mathit{I}}_i^{\mathrm{lr}},{\mathit{L}}_i^{\mathrm{lr}}\right)\right\}}_{j=1}^N,$$

(1)

where ${\mathit{I}}_i^{\mathrm{lr}}$ and ${\mathit{L}}_i^{\mathrm{lr}}$ are images and labels of size $h_1\times w_1$, significantly smaller than the original image’s $H\times W$ size.

The local branch L focuses on processing high-resolution image patches extracted from the original dataset D, denoted as

$$\mathit{L}={\left\{{\left\{\left({\mathit{I}}_{ij}^{\mathrm{hr}},{\mathit{L}}_{ij}^{\mathrm{hr}}\right)\right\}}_{j=1}^{n_i}\right\}}_{i=1}^N,$$

(2)

where ${\mathit{I}}_{ij}^{\mathrm{hr}},{\mathit{L}}_{ij}^{\mathrm{hr}}\in {\mathbb{R}}^{h_2\times w_2}$, with $h_2\times w_2$ being noticeably smaller than the original image size. Each image ${\mathit{I}}_i$ and label ${\mathit{L}}_i$ is cropped from top to bottom and left to right into overlapping patches, divided into $n_i$ image blocks, preserving the characteristics of ultra-high resolution. Both branches use the same backbone network, and images cropped and downsampled sequentially pass through ResNet, FPN, CCM, DRLF, and then through the branch aggregation layer $f_{\mathrm{agg}}$ to aggregate two sets of high-level feature maps, generating the final segmentation mask. To ensure stability and improve the training of both branches, weakly coupled regularization is also applied to the training of the local branch. The CCM and DRLF are our innovations, introduced in Section 3.2 and Section 3.3, respectively.

3.2. Cross-Collaborative Module

To fully harness the potential of the dual-branch structure, this study introduces an efficient and flexible cross-sharing module aiming to achieve deep collaboration and information exchange between global and local branch features. The module is based on the Transformer architecture and incorporates a cross-attention mechanism, which effectively handles two distinct dimensions of embedded sequences—global and local branch features. By asymmetrically fusing these features and assigning them adaptive weights, the module can thoroughly explore their interrelation and complementarity.

Unlike the cubic iterative fusion method with four layers per iteration employed by GLNet2, TDBAN introduces an innovative cross-sharing module that leverages the Transformer mechanism to achieve efficient feature fusion in a single step. We have the lightweight Transformer by adopting a single-layer Channel Attention (CA) mechanism and simplifying the query (Q) and value (V) acquisition process, directly taking in global (G) and local (L) features as input. Furthermore, we significantly enhance both computational efficiency and model performance through the two following lightweight design strategies:

(1)

Dimensionality Reduction and Load Mitigation: We incorporate dimensionality reduction operations within the fully connected layers to alleviate computational burdens.

(2)

Local Attention Optimization: A local attention strategy is employed to refine the cross-attention mechanism, thereby effectively capturing fine-grained feature relationships.

Additionally, the application of Feed-Forward Neural Networks (FFNs) is omitted, and the desired fusion effect is achieved through a single Cross-attention calculation, resulting in a computationally concise and efficient lightweight Transformer model. This design not only enhances the collaborative efficiency between global and local branch features but also significantly improves the overall performance of the model.

The computation of cross-fusion can be expressed by the following formula:

$$softmax\left(\left(W_Q{S}_2\right){\left(W_K{S}_1\right)}^{\top }\right)W_V{S}_1$$

(3)

where $S_1$ and $S_2$ represent the features of the global and local branches, respectively, Q, K, and V denote the linear transformation matrices of queries, keys, and values, respectively, and K represents the dimension of the keys. The softmax function represents the normalization function.

Figure 2 presents the architecture of the Cross Collaborative Module, which integrates global and local features. This diagram illustrates how the cross-attention mechanism efficiently integrates diverse feature layers to enhance the model’s discriminative power and accuracy. The input features originate from the final four layers of ResNet. The Local Feature Array is represented by the input array M, capturing the features from a global perspective to provide extensive contextual information. The Global Feature Array, represented by latent array N, focuses on capturing rich local details, emphasizing detailed performance in specific regions.

The Refined Query Array undergoes initial encoding and processing to further refine and extract features during the decoding stage. The Cross-Attention Mechanism is the core of the model, dynamically adjusting the weights and importance of features by computing interactions among the Query (Q), Key (K), and Value (V) arrays. This mechanism optimizes the feature integration based on the correlation between global and local information. Q represents the query vector dynamically generated from the local feature array, guiding how relevant information is extracted from global features. K and V vectors are generated from the global feature array and weighted with the query vector to enhance the local features. Attention Scores demonstrate the strength of interactions between different features, determining how these features are combined to form refined outputs.

The Final Output Features represent the comprehensive understanding and processing results of the input data after undergoing cross-attention processing and are suitable for various subsequent applications. Specifically, the cross-collaborative module first inputs the features of the local branch as K and V and the global branch as Q. It calculates the dot product attention weights between Q and K and then applies these to V to obtain the output. The output features have the same dimension as the features of the global branch, retaining the overview of global information while integrating the details of local information. Then, the output features of the cross-attention are inputted as K and V, and the features of the global branch are inputted as Q. This repeated process strengthens the interaction between global and local features, thereby enhancing the model’s feature expression capabilities and recognition accuracy.

3.3. Data-Related Learnable Fusion Module

To fully leverage the potential of multibranch networks, we designed a data-adaptive learnable fusion module (DRLF) that adjusts weights adaptively to differentiate focus on global and local information flows. This module emphasizes local features in edge areas and prioritizes global information in ambiguous regions, greatly improving the segmentation accuracy through dynamic weight allocation.

Module Architecture:

The data-related learnable fusion module (Figure 3) consists of two parallel learning submodules: the spatial attention generation module and the channel attention generation module. These submodules replicate the outputs of the zoom module (Shan et al’s) [3] spatial and channel attention, obtaining accurate attention maps without extensive computational operations.

Knowledge distillation is a commonly used method for model compression. Unlike pruning and quantization in model compression, knowledge distillation involves constructing a lightweight, small model and utilizing the supervision from a larger, better-performing model to train this smaller model, aiming for improved performance and accuracy. It was first proposed and applied to classification tasks by Hinton in 2015 [47]. The large model is called the “teacher” model, while the small model is called the “student” model. The supervision information from the teacher model’s outputs is referred to as “knowledge”, and the student learning process from the teacher’s supervision is termed “distillation”.

The DRLF adopts the strategy of knowledge distillation, using the Spatial and Channel Attention Masks from the zoom module of MBNet [3] as supervision from a more sophisticated, complex model. The learning process of the sub-module is guided by the supervision signal, with the generated spatial and channel attention masks serving as training targets. By optimizing these sub-modules to generate attention patterns that match those of the zoom module in MBNet [3], the DRLF approximates the functionality of the original module with lower computational complexity.

Knowledge distillation typically involves several steps:

• Soft label distillation loss:
  Using the softmax function with a high-temperature parameter T to compute the output of the teacher network, generating soft labels to provide behavioral guidance to the student network. The formula is as follows:

  $$L_{\mathrm{soft}}=-\sum _{i=1}^N{p}_i^{T}log\left(q_i^T\right)$$

  (4)

  where $p_i^{\mathrm{T}}$ represents the smooth probability distribution of the teacher network’s output, and $q_i^{\mathrm{T}}$ denotes the corresponding probability distribution of the student network.
• Hard target student loss:
  Simultaneously, the student network optimizes its performance by learning to predict correct hard targets (actual labels). The formula is:

  $$L_{\mathrm{hard}}=-\sum _{i=1}^N{c}_{i}log\left(q_i\right)$$

  (5)

  where $c_i$ represents the one-hot encoded representation of the true labels, and $q_i$ signifies the student network’s softmax probability output.
• Combined distillation and student loss:
  The final loss function is a linear combination of distillation loss and hard target loss, enabling the student network to learn from soft and hard labels. The formula is:

  $$L=\alpha ·L_{\mathrm{soft}}+\beta ·L_{\mathrm{hard}}$$

  (6)

  where $\alpha$ and $\beta$ are hyperparameters used to balance the contributions of the two types of losses.

Computational benefits:

Our design ensures that the submodule can significantly reduce computational costs while learning the complex attention maps in the Zoom Module of MBNet [3]. This inherent computational efficiency makes this module well suited for resource-constrained environments while maintaining the model’s focus on essential features at different resolutions. In this way, the DRLF facilitates rapid and efficient information exchange between features of different resolutions, thereby enhancing the overall performance of the model.

3.4. Loss Functions and Training Process

To effectively address feature conflicts and convergence instability in the dual-branch network, we have designed a comprehensive loss function and introduced a specific weighting scheme to stabilize the training process.

Dual-Branch Feature Aggregation and Loss Function Design

In this study, the global and local branches generate high-level feature maps at their respective L layers, denoted as ${\widehat{X}}_{Loc}^L$ and ${\widehat{X}}_{Glb}^L$. These feature maps are then passed through an aggregation layer $f_{agg}$, which concatenates them along the channel dimension using a $3\times 3$ convolution filter, resulting in the aggregated feature map ${\widehat{S}}_{Agg}$, which serves as the final segmentation output.

To maintain consistency between the outputs of the two branches, we introduce an auxiliary loss function that constrains the outputs of the local and global branches to approach their respective target feature maps. This auxiliary loss function helps stabilize the training process, avoiding conflicts between branches and assisting the network in balancing local and global features.

Weak Coupling Regularization

To further alleviate feature conflicts in the dual-branch network, especially when the local branch overfits local features, we introduce weak coupling regularization. This regularization adds a constraint term $\lambda \parallel {\widehat{X}}_{Loc}^L-{\widehat{X}}_{Glb}^L{\parallel }^2$ to limit the difference between the local and global feature maps. The regularization term effectively controls the learning rate of the local branch, preventing it from dominating the entire training process.

In the experiments, we adjusted the regularization coefficient $\lambda$ (from 0.15 to 0.25, 0.35, 0.45, and 1) to explore the impact of different regularization strengths on the training dynamics. The results show (

Table 1. The impact of different regularization coefficients on the network segmentation performance metric (mIoU).

$\lambda$	0.15	0.25	0.35	0.45	1
mIoU(%)	73.6	73.1	68.2	67.4	62.0

) that a reasonable regularization coefficient ($\lambda =0.15$) helps maintain the synchronized updates of the local and global branches and significantly improves the stability of the training process.

4. Experiment

4.1. Experimental Details

We employ the feature pyramid network (FPN) [51] as the backbone network based on ResNet50 [52]. During the bottom–up phase of FPN, we utilize a deep feature map-sharing strategy to process the feature maps of ResNet50’s conv2 to conv5 blocks and continue to apply this strategy in the top–down and smoothing phases of FPN. In the final lateral connection phase of FPN, we adopt feature map regularization and summarize this phase to achieve final segmentation. To simplify processing, the downsampled global images and cropped local patches are set to 500 × 500 pixels with 50 pixels of overlap between adjacent patches. We adopt focal loss ($\gamma =6$) [53] as the primary loss function, supplemented by two auxiliary losses, each weighted equally at 1.0. Additionally, we set the regularization coefficient $\lambda$ for feature maps to 0.15. To measure GPU memory usage during model training, the command-line tool “gpustat” is utilized to avoid computing gradients. Training and inference are performed using a single GPU card, and the experiments are conducted using the PyTorch framework [54]. We train with the Adam optimizer [55] (${\beta }_1=0.9$, ${\beta }_2=0.999$), setting the learning rate for the global branch to $1\times {10}^{-4}$ and the local branch to $2\times {10}^{-5}$, with a batch size of 6 for all training. All experiments are conducted on a workstation with NVIDIA 1080Ti GPU cards (NVIDIA Corporation, Santa Clara, CA, USA).

4.2. Datasets

This study utilizes two ultra-high-resolution remote sensing image datasets: the DeepGlobe Land Cover Classification Dataset [56] and the Inria Aerial Image Dataset [57]. The selection of these datasets aims to demonstrate the robustness and applicability of our method across diverse scenarios.

The DeepGlobe Land Cover Classification Dataset [56] is the first publicly available benchmark that provides high-resolution sub-meter satellite images focused on rural areas. It includes pixel-level ground truth masks for seven land cover classes: urban, agriculture, pasture, forest, water, barren, and unknown. The dataset consists of 1146 annotated satellite images, each with a resolution of 2448 × 2448 pixels, significantly higher than that of conventional land cover classification datasets. For our study, the DeepGlobe images are randomly partitioned into training, validation, and test sets, containing 455, 142, and 206 images, respectively.

In contrast, the Inria Aerial Image Dataset [57] covers a diverse array of urban landscapes, including dense metropolitan areas and mountainous resort regions. It provides 180 images, each with a size of 5000 × 5000 pixels (from five cities), with each image annotated using binary masks distinguishing between building and non-building areas. Unlike DeepGlobe, the Inria dataset is split by city into training, validation, and test sets, consisting of 126, 27, and 27 images, respectively. Each image in the Inria dataset contains up to 25 million pixels—approximately four times more than DeepGlobe—and its foreground regions are more detailed, presenting a greater challenge for segmentation methods. By testing on these two datasets, we are able to thoroughly evaluate the performance of our proposed method across different resolutions and scenarios.

4.3. Evaluation Metrics

The performance of the proposed model is evaluated using three widely adopted metrics: inference time (time), memory usage (memory, in MB), and mean intersection over union (mIoU) on the joint. Inference time is a crucial metric for assessing the model’s response speed, while memory usage serves as an important indicator for evaluating the practical feasibility of model deployment. mIoU serves as a comprehensive evaluation metric that not only quantifies the accuracy of segmentation results but also considers the balance between different classes. It effectively assesses the overall performance of models in segmenting various classes in ultra-high-resolution remote sensing images, with higher values indicating better model performance. The expression is defined as:

$$\mathrm{mIoU}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathrm{IoU}}_i$$

(7)

where N denotes the total number of classes, and ${\mathrm{IoU}}_i$ represents the intersection of union for each class.

To provide a more comprehensive evaluation of the model’s performance, we also introduce two key indicators of model complexity: the number of parameters and computational cost. The number of parameters represents the total count of trainable parameters in the model, which is typically proportional to the model’s storage requirements and computational resource consumption. A lower number of parameters indicates a more lightweight model that is easier to deploy and incurs lower storage and transmission overheads. Computational cost (FLOPs) refers to the number of floating-point operations required during each inference process. A lower FLOP count indicates higher computational efficiency during inference, helping to reduce latency and enhance the response speed. In this study, we analyze the model’s parameter count and FLOPs to provide deeper insights into its complexity and computational overhead, offering valuable guidance for its deployment in practical applications.

5. Results and Analysis

5.1. Performance Metric Comparison

To validate the superiority of the model, we conducted a comprehensive evaluation on the DeepGlobe and Inria datasets, and quantitatively compared the proposed TDBAN with various advanced segmentation methods. The methods compared include traditional general methods (such as ICNet [25], BiSeNet [26], SegNet [58], DANet [24], FCN-8s [23]) as well as models incorporating the latest attention mechanisms (such as MBNet [3], SegFormer [34], Uniformer [35], Mask2Former [36]). Additionally, we compared our optimized baseline model GLNet [2] for ultra-high-resolution remote sensing imagery. The experimental results (

Table 2. Comparison of the overall performance of networks on DeepGlobe.

Model	mIoU (%)	Memory (MB)	Time (s)
ICNet [25]	40.2	2557	23.8
BiSeNet [26]	47.0	1801	10.2
SegNet [58]	61.2	10339	17.8
DANet [24]	63.4	6914	60.9
FCN-8s [23]	70.1	5227	45.6
MBNet [3]	72.6	1549	162.9
SegFormer [34]	70.4	6880	404.4
Uniformer [35]	58.5	7311	596.0
Mask2Former [36]	71.2	4821	327.3
GLNet [2] (baseline)	71.6	1865	113.2
TDBAN	73.6	1771	68.9

Note: The bold values indicate the best performance in each respective column.

,

Table 3. Comparison of the overall performance of networks on Inria Aerial.

Model	mIoU (%)	Memory (MB)	Time (s)
ICNet [25]	30.1	2379	50.9
DeepLabv3+ [59]	55.9	4323	98.4
FCN-8s [23]	61.6	8253	43.2
MBNet [3]	71.4	2778	169.4
FovNet [60]	70.3	4469	274.6
LDNet [61]	71.5	5401	225.1
GLNet [2] (baseline)	71.2	2663	167.3
TDBAN	72.7	2501	79.4

Note: The bold values indicate the best performance in each respective column.

and

Table 4. Performance comparison of models with different patch sizes on DeepGlobe.

Net	Patch Sizes
	500	400	256	128
GLNet [2]	71.6	70.6	66.4	64.6
MBNet [3]	72.6	72.2	71.1	69.1
TDBAN	73.6	73.3	72.6	70.2

) show that TDBAN not only achieves the best performance in mIoU but also strikes a better balance between accuracy, speed, and memory consumption.

The key observations are as follows: (1). Compared to traditional general networks, networks using the Transformer attention mechanism (Mask2Former [36], SegFormer [34]) significantly improved segmentation accuracy but also resulted in higher computational costs, including increased memory consumption and inference time. (2). GLNet, MBNet, and the proposed TDBAN all adopt multi-branch network architectures, which effectively reduce memory consumption while improving mIoU. (3). TDBAN, through the optimized dual-branch network framework and innovative attention modules, achieved an mIoU of 73.6% on the DeepGlobe dataset, a 2% improvement over the baseline model GLNet; on the Inria dataset, the mIoU reached 72.7%, a 1.5% improvement over the baseline. Additionally, the GPU memory usage decreased, with significant speed improvements of nearly 40% and 52%, respectively. (4). On the DeepGlobe dataset, we experimentally verified the impact of different patch sizes on the model segmentation accuracy. The results show that TDBAN exhibits excellent stability under different patch settings, with the mIoU consistently above 70%, ensuring high accuracy.

Through detailed analysis, we further clarify the performance disparities between TDBAN and other approaches. The innovative Transformer-based lightweight dual-branch architecture of TDBAN, together with the DRLF module, enhances feature extraction and fusion efficiency, improving segmentation accuracy while preserving low memory usage. Specifically, TDBAN’s dual-branch architecture effectively balances the processing of both global and local information. The global branch captures broad contextual information through downsampling, providing background support for the entire image; the local branch, on the other hand, concentrates on detailed recognition and refines local regions, enhancing the accuracy and precision of image segmentation. Furthermore, CCM introduced by TDBAN optimizes the fusion of global and local features through a single-step transformation mechanism, reducing redundancy and enhancing the coherence between global and local information. The DRLF module adapts to remote sensing images, preserving fine local features while effectively leveraging global background information, thus avoiding the accuracy loss caused by an excess or lack of local information.

Overall, TDBAN achieves an efficient balance of global and local information through the optimized dual-branch network architecture and innovative modules (such as CCM and DRLF), providing outstanding performance for semantic segmentation tasks in remote sensing images while significantly reducing computational overhead.

5.2. Segmentation Result Comparison

Figure 4 demonstrates the performance of our method in comparison to GLNet [2] and MBNet [3] through the visualizations of segmentation results, feature maps, and binary feature maps. In Figure 4a, we visualize the segmentation outcomes, where our method (top row) produces more accurate segmentation boundaries compared to GLNet [2] and MBNet [3] (middle and bottom rows, respectively). Figure 4b illustrates the feature maps, highlighting how our approach better captures relevant features, showing clear differentiation between objects in the scene. Finally, in Figure 4c, the binary feature map emphasizes the effectiveness of our method in isolating key structures with higher precision than the other two models. These results confirm the superiority of our method in both segmentation accuracy and feature representation.

Figure 5 illustrates the results of the improvements, particularly through the comparison of the two panels. When considering the results of the innovative sharing strategy alone (Figure 5c) or the enhancement of the model’s ability to integrate features of different resolutions (Figure 5b,d), undesirable grid-like artifacts and inaccurate boundaries are still evident. However, a gradual improvement can be observed from the bidirectional deep feature map sharing in GLNet to the effective integration of the TDBAN dual-module, with a significant reduction in misclassifications and inaccurate boundaries.

6. Ablation Study

In this ablation study, we analyzed the contributions of the CCM and DRLF to the overall performance of the model. All models included the Agg and Fmreg, which play essential roles in the model’s aggregation capability and feature normalization.

6.1. CCM

In this experiment, the CCM module introduces a lightweight Transformer mechanism and an effective dimensionality reduction design, successfully maintaining low computational costs while improving model performance (

Table 5. Ablation study of the CCM: performance on DeepGlobe.

Architecture	mIoU (%)	Memory (MB)	Time (s)	FLOPs (G)	Number of Parameters (M)
GLNet [2]	71.6	1865	113.2	—	—
MBNet [3]	72.6	1549	162.9	141	90.5
Cross-attention only G to L	69.1	1500	90.0	78	24
Cross-attention only L to G	70.0	1504	90.4	78	24
Transformer	71.1	4701	356.9	156	48
CCM Only	73.0	1994	94.8	39	12

). Specifically, the CCM module achieves 73.0% mIoU on the DeepGlobe dataset, significantly outperforming other ablation configurations, while maintaining low memory usage (1994 MB) and computational overhead (94.8 s). With 12 M parameters and only 39 G FLOPs, it significantly reduces the computational complexity compared to the traditional Transformer (48 M parameters and 156 G FLOPs) and MBNet (90.5 M parameters and 141 G FLOPs). A comparison shows that using the cross-attention mechanism alone (whether GtoL or LtoG) results in a limited performance improvement (mIoU of 69.1% and 70.0%, respectively), while the Transformer, although offering better performance (71.1% mIoU), leads to a substantial rise in computational cost. The CCM module retains the benefits of the cross-attention mechanism while effectively controlling computational costs through a streamlined design, showcasing higher computational efficiency and model deployability.

In summary, the CCM module optimizes the Transformer design and effectively merges cross-domain features, not only improving segmentation performance but also achieving a good balance between model complexity and computational efficiency, enabling efficient operation in resource-constrained environments and offering high practical value.

6.2. DRLF

Table 6. Ablation study of the DRLF: performance on DeepGlobe.

Attention	mIoU (%)	Memory (MB)	Time (s)
Dual attention [24]	63.4	6914	60.9
Zoom module [3]	72.6	1549	162.9
Only channel attention	71.9	1404	60.0
Only spatial attention	50.4	1494	65.0
DRLF only	72.6	1543	71.0

displays the ablation experiment results of the DRLF module on the DeepGlobe dataset, analyzing the effectiveness of the knowledge distillation strategy by comparing different configurations. The DRLF module effectively reduces computational complexity through knowledge distillation, significantly lowering the computational costs while maintaining high performance. DRLF and the Zoom module have the same mIoU (72.6%), but DRLF consumes less memory and computation time (1543 MB and 71 s, respectively), demonstrating that DRLF, with distillation, can significantly enhance computational efficiency while preserving segmentation accuracy. Using only channel attention results in a performance of 71.9%, while using only spatial attention leads to a lower mIoU (50.4%). This indicates that channel attention plays a more significant role in feature modeling and image segmentation, with the DRLF module combining the strengths of both, optimizing the learning process of submodules, and enhancing performance. This ablation study confirms the crucial role of knowledge distillation in enhancing the balance between model efficiency and performance. The DRLF module not only increases segmentation accuracy but also offers a more efficient solution in environments with limited computational resources.

6.3. Comprehensive Analysis

In our study, we introduce an innovative improvement to the GLNet model’s traditional bidirectional deep feature map sharing strategy (which involves 4-layer feature fusion performed three times) by employing a Transformer-based fast fusion mechanism—CCM. This mechanism enables perfect feature fusion in a single step, greatly enhancing fusion efficiency. At the same time, our Transformer fusion utilizes a lightweight design, including dimensionality reduction in the fully connected layer and the use of a local attention strategy in the cross-attention mechanism, thereby further reducing computational complexity. While maintaining the model’s parameter count at 12 M and FLOPs at 39 G, the model’s mIoU increased to 73%, and the processing time was significantly reduced by at least 140 s, with the final processing time being 94.8 s (

Table 7. Comparison of the improved comprehensive performance of the TDBAN on DeepGlobe.

Model	CCM	DRLF	mIoU (%)	Memory (MB)	Time (s)
TDBAN	✓	–	73.0	1994	94.8
	–	✓	72.6	1543	71.0
	✓	✓	73.6	1771	68.9

Note: The modules marked with “✓” are used in building the network.

).

Subsequently, we introduced the DRLF module with distillation technology, which further optimized the model’s performance. Through the DRLF module, the model’s mIoU was further improved to 73.6%, and the processing time was further reduced to 68.9 s (Table 7) while the memory usage (1771MB) remained almost unchanged. This indicates that the DRLF module has a significant effect in simplifying the computation process and improving computational speed.

Although this paper mainly focuses on ultra-high-resolution remote sensing image segmentation tasks, the design of the CCM and DRLF modules is highly adaptable and can be applied to a wide range of other computer vision tasks. For example, the CCM module improves small object detection accuracy in object detection by fusing multi-scale features, while in image classification tasks, it enhances the combination of global and local information through the cross-attention mechanism, improving classification performance. The DRLF module optimizes computational efficiency via knowledge distillation, enabling fast and accurate feature fusion in tasks such as semantic segmentation, image super-resolution, and medical image analysis, and is especially well suited for resource-limited environments. Nonetheless, in cross-task applications, the module designs must still be adjusted according to the specific requirements of each task. Future studies may explore ways to further optimize the computational efficiency of these modules and enhance their performance in multi-task learning.

7. Conclusions

This study proposes the TDBAN model, which significantly improves the semantic segmentation accuracy of ultra-high-resolution remote sensing images through an innovative dual-branch architecture and module design, while also demonstrating excellent performance in memory efficiency and processing speed. Experiments on the DeepGlobe and Inria Aerial datasets show that TDBAN achieves mIoU scores of 73.6% and 72.7%, respectively, with only 12 million parameters and a computational complexity (FLOPs) of 39 G, exhibiting low memory consumption and a short processing time.

The success of TDBAN can be attributed to its innovative dual-branch architecture, which cleverly balances the processing of global and local information. The global branch captures broad contextual information through downsampling, providing background support for the image, while the local branch focuses on fine details, improving the segmentation accuracy in finer regions. CCM further optimizes the fusion of global and local features, reducing redundant information and enhancing the coordination of information, thus improving overall performance. The introduction of the DRLF module effectively utilizes global background information while preserving local details, further enhancing model accuracy and optimizing computational overhead.

Additionally, TDBAN significantly improves feature fusion efficiency by adopting a lightweight Transformer-based design and a fast feature fusion mechanism. The lightweight fully connected layers and local attention mechanism not only reduce computational complexity but also enhance the processing speed while maintaining performance. Notably, the introduction of knowledge distillation technology further improves segmentation accuracy while reducing computational time in the DRLF module.

It is worth emphasizing that our method is a general improvement, not an optimization for specific scenes or environments. Its effectiveness has been proven on datasets such as DeepGlobe, and due to its innovative generalizability, it is expected to perform well in various remote sensing application scenarios. The diversity of the DeepGlobe dataset further validates the effectiveness of the method.

Overall, TDBAN achieves good balance between accuracy and efficiency, making it particularly suitable for resource-constrained environments. TDBAN’s performance demonstrates that it is a feasible solution that balances accuracy and resource utilization, with broad application prospects in fields such as 3D object detection.