GCF-Net: Stage-Aligned Optical–Elevation Fusion for Aerial Remote Sensing Semantic Segmentation

Highlights

What are the main findings?

• GCF-Net assigns pre-fusion structural correction, joint frequency–spatial representation construction, and decoder-stage elevation-guided refinement to different representation states.
• GCF-Net achieves mIoU scores of 72.37%, 75.38%, and 52.23% on ISPRS Vaihingen, ISPRS Potsdam, and MMHunan, respectively, outperforming the evaluated unimodal, multimodal, and SAM-based methods under the unified protocol.

What are the implications of the main findings?

• Organizing cross-modal processing according to representation state provides an alternative to fusion strategies driven mainly by feature resolution or network depth.
• Frequency-aware conditioning complements attention-based correspondence modeling, while the complete stage-aligned design demonstrates tolerance to moderate elevation perturbations and a balanced accuracy–efficiency trade-off.

Abstract

Optical–elevation data fusion is widely used in aerial remote sensing semantic segmentation, as optical imagery provides rich spectral and textural information, while DSM or DEM data offer complementary elevation-related structural cues. However, effective fusion remains challenging because optical and elevation representations may exhibit cross-modal structural inconsistency, frequency–spatial response imbalance, and decoder-stage structural attenuation. To address these challenges, we propose GCF-Net, a stage-aligned optical–elevation fusion network that matches different cross-modal processing objectives to the evolving representation states of the encoder–decoder pipeline. A Structure-Guided Cross-Modal Correction Module first performs structure-conditioned correction of modality-specific features before fusion. A Frequency–Spatial Cross-Modal Fusion Module then constructs joint representations through bounded cross-modal magnitude conditioning, frequency-to-spatial reconstruction, and spatial recalibration. During decoding, a Geometry-Aware Cross-Scale Refinement Module reintroduces elevation-derived structural guidance into multiscale fused features. Experiments on ISPRS Vaihingen, ISPRS Potsdam, and MMHunan yield mIoU scores of 72.37%, 75.38%, and 52.23%, respectively, achieving the highest mIoU among the evaluated unimodal, multimodal, and SAM-based methods under the unified protocol. Ablation and replacement experiments verify the complementary roles of the three stage-specific components, while sensitivity, elevation perturbation, and complexity analyses indicate architectural flexibility, tolerance to moderate elevation degradation, and a balanced accuracy–efficiency trade-off.

1. Introduction

Remote sensing semantic segmentation assigns semantic labels to image pixels and supports applications such as urban mapping, land-cover analysis, environmental monitoring, and disaster assessment [1,2,3,4]. With the growing availability of heterogeneous Earth observation data, multimodal learning has become an important means of extending scene understanding beyond appearance-dominated representations [5,6,7]. In particular, optical imagery provides rich spectral and textural information, whereas digital surface models (DSMs) and digital elevation models (DEMs) provide complementary cues related to height variation, terrain structure, spatial layout, and object boundaries [8,9,10]. Their combination is therefore well suited to high-resolution aerial segmentation in complex urban and land-cover scenes [11,12].

Effective optical–elevation fusion nevertheless remains difficult. Beyond standard benchmarks such as ISPRS Vaihingen and Potsdam, practical optical–elevation datasets may vary in spatial resolution, terrain complexity, scene composition, and registration quality [13,14,15,16,17,18]. These variations weaken the correspondence between optical appearance and elevation-related structure, making cross-modal interaction less reliable.

Three challenges arise as optical and elevation representations evolve through the network. First, cross-modal structural inconsistency concerns whether the two modalities produce coherent spatial responses around corresponding objects and boundaries. Optical transitions may be induced by texture, illumination, or material changes, whereas elevation discontinuities more directly reflect height variation and surface structure [19,20,21]. Unconstrained interaction may therefore transfer modality-biased structural responses across branches. Second, frequency–spatial response imbalance concerns unequal modality contributions during fusion rather than spatial correspondence itself. Even for well-registered inputs, optical and elevation features may exhibit different frequency-domain energy distributions and spatial response patterns [22,23]. Without appropriate regulation, stronger appearance-driven optical responses may dominate the fused representation, limiting the effective use of complementary elevation-derived structural information. Third, decoder-stage structural attenuation concerns the progressive weakening of structural cues during semantic reconstruction. Although informative elevation cues are encoded at earlier stages, their structural contribution may gradually diminish during hierarchical decoding [24,25,26]. This attenuation may blur object boundaries and fragment compact or elongated targets, reducing the structural coherence of the final predictions. Because these challenges emerge at different representation states, they motivate stage-specific cross-modal processing rather than the uniform application of a single interaction mechanism.

Recent multimodal segmentation studies have developed diverse strategies, including gated propagation, feature correction, cross-attention, cross-scale aggregation, progressive decoding, and foundation-model adaptation [7,11,12,13,17,21,27,28]. Despite these advances, existing fusion pipelines are commonly organized around feature resolution, network depth, cross-layer information flow, or model-adaptation strategy. In many cases, similar interaction mechanisms are applied across multiple feature levels, while the changing objectives of cross-modal processing before fusion, during joint representation construction, and during decoder-stage structural refinement are not explicitly distinguished. Consequently, modality correction, multimodal integration, and structural recovery may remain partly entangled within a common fusion pipeline.

To address this gap, we propose GCF-Net, a stage-aligned optical–elevation fusion network. Here, “stage-aligned” does not merely mean applying fusion at multiple resolutions. Instead, it denotes matching the objective of multimodal interaction to the current representation state of the encoder–decoder pipeline. Specifically, the Structure-Guided Cross-Modal Correction Module (SGCM) performs pre-fusion correction by using modality-specific structural priors to guide channel selection and cross-modal correspondence. The Frequency–Spatial Cross-Modal Fusion Module (FSCM) constructs a joint representation through frequency-aware cross-modal conditioning, spatial reconstruction, and feature recalibration. The Geometry-Aware Cross-Scale Refinement Module (GACS) subsequently reintroduces elevation-derived structural guidance when aggregating multiscale fused features during decoding. Within GACS, “geometry-aware” refers to elevation-conditioned structural guidance rather than an explicit geometric constraint.

The main contributions of this work are summarized as follows:

• We formulate optical–elevation segmentation from a stage-aligned perspective, in which cross-modal processing objectives are aligned with the evolution from modality-specific encoding to joint representation learning and decoder-stage structural reconstruction.
• We develop GCF-Net to instantiate this formulation through three complementary components. SGCM performs structure-prior-conditioned correction before fusion, FSCM constructs frequency–spatial joint representations, and GACS provides elevation-guided refinement during multiscale decoding.
• We conduct extensive experiments on ISPRS Vaihingen, ISPRS Potsdam, and MMHunan. Comparisons with representative unimodal, multimodal, and SAM-based methods demonstrate competitive and consistent overall performance. Ablation studies verify the complementary roles of the stage-specific components, while robustness and efficiency analyses further characterize the practical behavior of the proposed design.

2. Related Work

2.1. Unimodal Remote Sensing Semantic Segmentation

Unimodal remote sensing semantic segmentation has progressed from convolutional encoder–decoder and high-resolution architectures to attention- and Transformer-based contextual modeling. Representative CNN-based architectures, including U-Net, SegNet, DeepLab, and HRNet, improve dense prediction through skip connections, atrous convolution, multiscale feature extraction, and high-resolution representation learning [29,30,31,32,33,34]. Transformer-based and CNN–Transformer hybrid networks, such as SSNet, CMLFormer, and CMTFNet, further strengthen long-range dependency modeling and multiscale contextual reasoning [35,36,37,38,39].

Recent studies have also explored frequency-aware and boundary-aware representations. MsanlfNet introduces frequency-domain modeling to complement spatial feature extraction [23], whereas HBGNet incorporates boundary guidance to improve local delineation [40]. Related studies further investigate multiscale frequency modeling and explicit boundary representations [22,41,42]. Foundation-model-assisted optical segmentation methods further exploit SAM-derived masks or structural priors to improve object and boundary consistency [43,44].

Despite these advances, the structural evidence used by unimodal methods is still mainly inferred from optical appearance and may become ambiguous under weak textures, occlusions, similar spectral responses, or complex object layouts [19,20]. This limitation motivates the use of complementary elevation data, such as DSM or DEM, for structure-aware multimodal segmentation [11,21].

2.2. Multimodal Remote Sensing Semantic Segmentation

Multimodal remote sensing semantic segmentation exploits complementary information from heterogeneous sensors to improve scene understanding [45,46]. Optical imagery and elevation data are particularly complementary: the former provides spectral and textural information, whereas the latter supplies height- and terrain-related structural cues [47]. Early approaches mainly relied on fixed operations such as concatenation, summation, or cross-layer aggregation [48], while recent methods increasingly employ gated propagation, feature correction, cross-attention, and Transformer-based interaction.

Representative methods differ in how they organize multimodal interaction. SA-Gate performs cross-modal feature filtering, spatially gated aggregation, and bidirectional propagation across encoder stages [27]. CMX and CFFormer combine cross-modal correction or rectification with attention-based interaction at multiple encoder levels, whereas FTransDeepLab introduces Transformer-based feature interaction across hierarchical representations [13,21,28]. FTransUNet instead differentiates shallow local fusion from deep contextual interaction according to feature depth [7]. These designs organize multimodal interaction mainly according to encoder level, feature resolution, or shallow–deep semantic hierarchy.

Cross-scale and decoder-involved fusion have also been explored. CMFNet jointly models cross-modal and multiscale dependencies and injects the resulting representations into corresponding decoder layers through residual skip fusion [12]. PACSCNet coordinates adjacent-layer features before progressive cross-modal fusion, while its decoder further integrates cross-layer, modality-specific, and fused representations [11]. These methods demonstrate that multimodal information can participate in both hierarchical feature fusion and decoder reconstruction.

Recent studies have further broadened multimodal segmentation through specialized interaction mechanisms and model architectures. MFNet adapts SAM to optical–DSM segmentation through multimodal adapters or LoRA-based fine-tuning and performs multiscale integration before decoding [17]. Other representative directions include asymmetric interaction, modality-specific modeling, state-space architectures, and knowledge distillation [10,49,50,51]. These studies expand the architectural and optimization choices for multimodal segmentation.

Overall, existing methods organize multimodal interaction mainly according to feature scale, semantic depth, propagation path, or model architecture. However, the changing objectives of cross-modal processing across modality-specific encoding, joint representation construction, and decoder-stage refinement are not always explicitly distinguished. This observation motivates a stage-aligned formulation in which the objective of multimodal interaction is matched to the current representation state rather than being determined by feature resolution or network depth alone.

3. Methodology

3.1. Overall Framework

Figure 1 illustrates the overall architecture of GCF-Net. The default implementation uses two parallel PVTv2-B2 encoders [52] with identical architectures but independent parameters to extract hierarchical modality-specific features. Each encoder produces four stages of hierarchical features, denoted as ${\mathbf{F}}_{\mathrm{opt}}^k$ and ${\mathbf{F}}_{\mathrm{dsm}}^k$, where $k=1,2,3,4$. For notation simplicity, the elevation branch is denoted by the subscript “dsm”, although its input may be a DSM, nDSM, or DEM, depending on the dataset.

GCF-Net follows a stage-aligned multimodal processing pipeline in which different representation states are assigned distinct processing objectives. At each encoder level, SGCM first corrects the separate optical and elevation features before fusion to reduce unreliable cross-modal structural responses. FSCM subsequently operates on the corrected features to construct a joint frequency–spatial representation. The resulting multiscale fused features are progressively aggregated and refined by GACS under elevation-derived structural guidance during decoding. The target issue and distinct function of each component are summarized in

Table 1. Stage-specific roles of the main components in GCF-Net.

Module	Stage	Target Issue	Distinct Function
SGCM	Pre-fusion	Cross-modal structural inconsistency	Structure-conditioned correction of modality-specific features.
FSCM	Fusion	Frequency–spatial response imbalance	Bounded cross-modal magnitude conditioning and spatial recalibration.
GACS	Decoder	Decoder-stage structural attenuation	Elevation-conditioned refinement of fused decoder features.

. Finally, the highest-resolution decoder feature is projected into the semantic label space and upsampled to the original image resolution to generate the pixel-wise prediction.

3.2. Structure-Guided Cross-Modal Correction

Cross-modal structural inconsistency may cause direct optical–elevation interaction to propagate modality-biased responses. SGCM therefore performs pre-fusion correction at each encoder stage. Given optical and elevation features ${\mathbf{F}}_{\mathrm{opt}}^k$ and ${\mathbf{F}}_{\mathrm{dsm}}^k$, it uses modality-specific structural priors to condition channel selection and cross-modal correspondence, while affine modulation regulates bidirectional feature transfer, as illustrated in Figure 2.

• Structural Prior Extraction. For a stage-k feature map ${\mathbf{F}}^k\in {\mathbb{R}}^{B\times C_k\times H_k\times W_k}$, fixed depth-wise Sobel operators extract horizontal and vertical gradients:

  $${\mathbf{G}}_x^k={\mathbf{F}}^k\ast {\mathbf{K}}_x,{\mathbf{G}}_y^k={\mathbf{F}}^k\ast {\mathbf{K}}_y,$$

  (1)

  where ∗ denotes depth-wise convolution, and ${\mathbf{K}}_x$ and ${\mathbf{K}}_y$ are fixed Sobel kernels. The channel-averaged gradient magnitude defines the structural prior:

  $${\mathbf{P}}^k={\displaystyle \frac{1}{C_k}}{\displaystyle \sum _{c=1}^{C_k}}\sqrt{{\left({\mathbf{G}}_{x,c}^k\right)}^2+{\left({\mathbf{G}}_{y,c}^k\right)}^2+ϵ}\in {\mathbb{R}}^{B\times 1\times H_k\times W_k},$$

  (2)

  where $ϵ$ ensures numerical stability.

Although the same operator is applied to both modalities, the optical prior ${\mathbf{P}}_{\mathrm{opt}}^k$ mainly reflects appearance and textural transitions, whereas the elevation prior ${\mathbf{P}}_{\mathrm{dsm}}^k$ emphasizes elevation-related discontinuities. For each branch, the global summary of the complementary prior conditions channel selection, while its unpooled spatial map guides cross-modal attention.

• Structure-Guided Channel Selection (SGCS). To reduce the influence of low-importance channels before cross-modal correspondence, SGCS estimates channel importance $\mathbf{s}$ from each modality-specific feature ${\mathbf{F}}^k$ through global average pooling and a $1\times 1$ convolution:

  $$\mathbf{s}={\mathrm{Conv}}_{1\times 1}\left(\mathrm{GAP}\left({\mathbf{F}}^k\right)\right)\in {\mathbb{R}}^{B\times C_k}.$$

  (3)

  A soft Top-K selection mask is then constructed using the K-th largest channel score as the threshold:

  $$\tau =\mathrm{TopK}{(\mathbf{s},K)}_K,\mathbf{m}=\sigma \left({\displaystyle \frac{\mathbf{s}-\tau }{t}}\right)\in {\mathbb{R}}^{B\times C_k},$$

  (4)

  where $\tau$ is the K-th largest element of $\mathbf{s}$, $K=max(1,\lfloor r{C}_k\rfloor )$ with $r=0.7$, and $t=0.1$ controls the smoothness of the soft selection mask.

Since the feature-derived channel scores may be dominated by modality-specific activations and local response bias, the global summary of the complementary structural prior is incorporated to provide cross-modal structural conditioning. Specifically, the elevation prior ${\mathbf{P}}_{\mathrm{dsm}}^k$ conditions the optical branch, whereas the optical prior ${\mathbf{P}}_{\mathrm{opt}}^k$ conditions the elevation branch:

$${\mathbf{m}}_p=\sigma \left({\mathrm{Conv}}_{1\times 1}\left(\mathrm{GAP}({\mathbf{P}}^k)\right)\right)\in {\mathbb{R}}^{B\times C_k}.$$

(5)

Accordingly, ${\mathbf{m}}_p$ provides global structure-conditioned channel modulation rather than region-wise spatial selection. The modulated channel weights and the resulting reweighted feature are computed as

$$\widehat{\mathbf{m}}=\mathbf{m}\odot (1+{\mathbf{m}}_p)\in {\mathbb{R}}^{B\times C_k},{\mathbf{F}}_{\mathrm{sel}}^k={\mathrm{Conv}}_{3\times 3}\left({\mathbf{F}}^k\odot \widehat{\mathbf{m}}\right).$$

(6)

• Prior-Guided Cross-Attention (PGCA). After channel reweighting, PGCA establishes cross-modal correspondence using the original target-modality feature as the query and the reweighted complementary feature as the key and value:

  $$\mathbf{Q}=\mathcal{T}\left({\mathbf{F}}_q^k\right),\mathbf{K}=\mathcal{T}\left({\mathbf{F}}_{kv,\mathrm{sel}}^k\right),\mathbf{V}=\mathbf{K},$$

  (7)

  where $\mathcal{T}(·)$ denotes spatial tokenization with downsampling, and the subscripts q and $kv$ denote the target and complementary branches, respectively. For optical-branch correction, $(q,kv)=(\mathrm{opt},\mathrm{dsm})$; for elevation-branch correction, $(q,kv)=(\mathrm{dsm},\mathrm{opt})$.

The unpooled structural prior ${\mathbf{P}}_{kv}^k$ from the same complementary branch is then tokenized and projected into the key space to condition the cross-modal attention:

$$\tilde{\mathbf{K}}=\mathbf{K}+\mathrm{Proj}\left(\mathcal{T}\left({\mathbf{P}}_{kv}^k\right)\right),{\mathbf{F}}_{\mathrm{cross}}^k=\mathcal{U}\left(\mathrm{MHA}\left(\mathbf{Q},\tilde{\mathbf{K}},\mathbf{V}\right)\right),$$

(8)

where $\mathcal{U}(·)$ restores the original spatial resolution. This formulation transfers structurally conditioned information from the complementary branch while retaining the target modality as the query reference.

• Spatially Guided Affine Modulation (SGAM). Although PGCA transfers complementary information across modalities, the resulting cross-modal response may contain local spatial perturbations. SGAM therefore conditions the transferred response on the original target feature through spatially varying affine modulation. It first predicts an affine-parameter embedding from the target feature and splits it along the channel dimension:

  $$\mathbf{A}={\mathrm{Conv}}_{3\times 3}\left({\mathbf{F}}^k\right)\in {\mathbb{R}}^{B\times 2C_k\times H_k\times W_k},[\mathit{\gamma },\mathit{\beta }]=\mathrm{Chunk}\left(\mathbf{A}\right),$$

  (9)

  where $\mathit{\gamma },\mathit{\beta }\in {\mathbb{R}}^{B\times C_k\times H_k\times W_k}$ denote the spatially varying multiplicative and additive parameters, respectively. They are then applied to the cross-modal response as follows:

  $${\mathbf{F}}_{\mathrm{corr}}^k={\mathbf{F}}_{\mathrm{cross}}^k\odot \left(1+\tanh \left(\gamma \right)\right)+\beta ,$$

  (10)

The $\tanh (·)$ function bounds the multiplicative factor, while $\mathit{\beta }$ enables local additive adjustment. Because both parameters are predicted from the original target feature, the correction remains conditioned on its spatial structure. The same operation is applied symmetrically to the optical and elevation branches. For compact notation, this conditional affine operation is denoted by $\mathrm{SGAM}({\mathbf{F}}_{\mathrm{mod}},{\mathbf{F}}_{\mathrm{cond}})$, where ${\mathbf{F}}_{\mathrm{cond}}$ generates the spatially varying affine parameters used to modulate ${\mathbf{F}}_{\mathrm{mod}}$.

The corrected features are residually combined with the corresponding encoder features:

$${\overline{\mathbf{F}}}_{\mathrm{opt}}^k={\mathbf{F}}_{\mathrm{opt}}^k+{\mathbf{F}}_{\mathrm{opt},\mathrm{corr}}^k,{\overline{\mathbf{F}}}_{\mathrm{dsm}}^k={\mathbf{F}}_{\mathrm{dsm}}^k+{\mathbf{F}}_{\mathrm{dsm},\mathrm{corr}}^k.$$

(11)

The resulting representations are subsequently passed to FSCM for multimodal integration.

3.3. Frequency–Spatial Cross-Modal Fusion

After pre-fusion correction, optical and elevation features may still exhibit frequency–spatial response imbalance. To explicitly regulate frequency-domain response strength while retaining modality-specific information during spatial reconstruction, FSCM introduces Fourier decomposition as a complementary inductive bias. It integrates ${\overline{\mathbf{F}}}_{\mathrm{opt}}^k$ and ${\overline{\mathbf{F}}}_{\mathrm{dsm}}^k$ through bounded cross-modal magnitude modulation, frequency-to-spatial reconstruction, and global–local recalibration, as illustrated in Figure 3.

• Cross-Spectral Mask Injection (CSMI). The input features are first projected by $1\times 1$ convolutions and transformed into magnitude and phase components:

  $$({\mathbf{M}}_{\mathrm{opt}},{\mathbf{\Phi }}_{\mathrm{opt}})=\mathcal{D}\left(\mathcal{F}\left({\mathrm{Conv}}_{1\times 1}\left({\overline{\mathbf{F}}}_{\mathrm{opt}}^k\right)\right)\right),$$

  (12)

  $$({\mathbf{M}}_{\mathrm{dsm}},{\mathbf{\Phi }}_{\mathrm{dsm}})=\mathcal{D}\left(\mathcal{F}\left({\mathrm{Conv}}_{1\times 1}\left({\overline{\mathbf{F}}}_{\mathrm{dsm}}^k\right)\right)\right),$$

  (13)

  where $\mathcal{F}(·)$ and ${\mathcal{F}}^{-1}(·)$ denote orthogonally normalized 2D real Fourier and inverse Fourier transforms, implemented using rfft2 and irfft2, respectively. $\mathcal{D}(·)$ extracts the magnitude and phase of a complex spectrum $\mathbf{Z}$ as $\left|\mathbf{Z}\right|$ and $\arg \left(\mathbf{Z}\right)$.

Although frequency decomposition exposes the response distributions of the two modalities, direct or unconstrained mixing of their magnitude spectra may retain modality-dominant responses or cause unstable spectral amplification. CSMI therefore uses the magnitude spectrum of the complementary modality to predict a bounded enhancement mask for each branch:

$${\tilde{\mathbf{M}}}_{\mathrm{opt}}={\mathbf{M}}_{\mathrm{opt}}\odot \left(1+\tanh \left(\mathcal{G}\left({\mathbf{M}}_{\mathrm{dsm}}\right)\right)\right),$$

(14)

$${\tilde{\mathbf{M}}}_{\mathrm{dsm}}={\mathbf{M}}_{\mathrm{dsm}}\odot \left(1+\tanh \left(\mathcal{G}\left({\mathbf{M}}_{\mathrm{opt}}\right)\right)\right).$$

(15)

Here, $\mathcal{G}(·)$ consists of two stacked $1\times 1$ convolutions followed by a sigmoid activation. Therefore, $0<\mathcal{G}(·)<1$, and the modulation factor satisfies $1<1+\tanh \left(\mathcal{G}\right(·\left)\right)<1+\tanh \left(1\right)$. CSMI thus performs bounded, enhancement-only cross-modal spectral conditioning rather than explicit normalization or suppression of modality responses. Each branch is conditioned by complementary spectral evidence while retaining its own magnitude response.

To construct a joint spectral representation from the two conditioned branches, their modulated magnitude spectra are concatenated and projected as

$${\mathbf{M}}_{\mathrm{mix}}={\mathrm{Conv}}_{1\times 1}\left(\left[{\tilde{\mathbf{M}}}_{\mathrm{opt}},{\tilde{\mathbf{M}}}_{\mathrm{dsm}}\right]\right),$$

(16)

where $[·,·]$ denotes channel-wise concatenation.

• Frequency-to-Spatial Reconstruction. The mixed magnitude representation contains jointly conditioned spectral responses, whereas the modality-specific phase components are retained during spatial reconstruction. FSCM therefore recombines the shared magnitude with each modality-specific phase and transforms the resulting spectra back to the spatial domain:

  $${\tilde{\mathbf{F}}}_{\mathrm{opt}}^k={\mathrm{Conv}}_{1\times 1}\left({\mathcal{F}}^{-1}\left(\mathcal{R}\left({\mathbf{M}}_{\mathrm{mix}},{\mathbf{\Phi }}_{\mathrm{opt}}\right)\right)\right),$$

  (17)

  $${\tilde{\mathbf{F}}}_{\mathrm{dsm}}^k={\mathrm{Conv}}_{1\times 1}\left({\mathcal{F}}^{-1}\left(\mathcal{R}\left({\mathbf{M}}_{\mathrm{mix}},{\mathbf{\Phi }}_{\mathrm{dsm}}\right)\right)\right),$$

  (18)

  where $\mathcal{R}(·,·)$ denotes magnitude–phase recomposition.

Because global spectral modulation may introduce local inconsistencies during inverse reconstruction, a spatial refinement operator $\mathcal{S}(·)$, implemented using two stacked $3\times 3$ convolutions with ReLU activation, is applied to both reconstructed features. Meanwhile, a frequency-independent spatial baseline is retained to preserve a direct representation of the residual-updated inputs:

$${\mathbf{F}}_{\mathrm{opt}}^{k,\mathrm{ref}}=\mathcal{S}\left({\tilde{\mathbf{F}}}_{\mathrm{opt}}^k\right),{\mathbf{F}}_{\mathrm{dsm}}^{k,\mathrm{ref}}=\mathcal{S}\left({\tilde{\mathbf{F}}}_{\mathrm{dsm}}^k\right),{\mathbf{F}}_{\mathrm{avg}}^k={\displaystyle \frac{1}{2}}\left({\overline{\mathbf{F}}}_{\mathrm{opt}}^k+{\overline{\mathbf{F}}}_{\mathrm{dsm}}^k\right).$$

(19)

The two reconstructed features provide spectrally conditioned modality-specific responses, whereas ${\mathbf{F}}_{\mathrm{avg}}^k$ supplies a stable spatial reference that reduces over-reliance on the frequency branch. These three representations are subsequently aggregated as

$${\mathbf{F}}_{\mathrm{agg}}^k={\mathrm{Conv}}_{3\times 3}\left(\left[{\mathbf{F}}_{\mathrm{opt}}^{k,\mathrm{ref}},{\mathbf{F}}_{\mathrm{dsm}}^{k,\mathrm{ref}},{\mathbf{F}}_{\mathrm{avg}}^k\right]\right).$$

(20)

• Lightweight Cross-Coupled Global–Local Gating (LCCG). Although the aggregation operation combines the reconstructed features and the spatial baseline, a single convolutional aggregation does not explicitly distinguish globally relevant channel responses from locally informative spatial responses. LCCG therefore extracts complementary spatial and channel contexts from ${\mathbf{F}}_{\mathrm{agg}}^k\in {\mathbb{R}}^{B\times C_k\times H_k\times W_k}$:

  $$\mathbf{S}={\mathrm{Mean}}_c\left({\mathbf{F}}_{\mathrm{agg}}^k\right)\in {\mathbb{R}}^{B\times 1\times H_k\times W_k},$$

  (21)

  $$\mathbf{C}=\mathrm{Expand}\left(\mathrm{GAP}\left({\mathbf{F}}_{\mathrm{agg}}^k\right)\right)\in {\mathbb{R}}^{B\times C_k\times H_k\times W_k}.$$

  (22)

  Here, $\mathbf{S}$ summarizes spatial responses across channels, whereas $\mathbf{C}$ represents the global channel context replicated over the spatial dimensions. LCCG combines each context with the aggregated feature to predict two cross-coupled gates:

  $${\mathbf{G}}_c=\sigma \left({\mathrm{Conv}}_{1\times 1}\left(\left[{\mathbf{F}}_{\mathrm{agg}}^k,\mathbf{S}\right]\right)\right)\in {\mathbb{R}}^{B\times C_k\times H_k\times W_k},$$

  (23)

  $${\mathbf{G}}_s=\sigma \left({\mathrm{Conv}}_{1\times 1}\left(\left[{\mathbf{F}}_{\mathrm{agg}}^k,\mathbf{C}\right]\right)\right)\in {\mathbb{R}}^{B\times 1\times H_k\times W_k}.$$

  (24)

  Accordingly, ${\mathbf{G}}_c$ is a spatial-context-guided channel gate, whereas ${\mathbf{G}}_s$ is a channel-context-guided spatial gate. This cross-coupled design allows local spatial evidence to regulate channel responses and the global channel context to regulate spatial responses.

The two gates are then applied through bounded multiplicative residual modulation:

$${\mathbf{F}}_{\mathrm{gate}}^k={\mathbf{F}}_{\mathrm{agg}}^k\odot \left(1+\alpha {\mathbf{G}}_c\right)\odot \left(1+\alpha {\mathbf{G}}_s\right),$$

(25)

where $\alpha =0.5$ controls the modulation strength. Since $0<{\mathbf{G}}_c,{\mathbf{G}}_s<1$, the corresponding modulation factors lie within $(1,1+\alpha )$, providing controlled channel-wise and spatial enhancement. A residual depthwise convolution is finally introduced for local spatial refinement:

$${\mathbf{F}}_{\mathrm{fuse}}^k={\mathbf{F}}_{\mathrm{gate}}^k+{\mathrm{Conv}}_{3\times 3}^{\mathrm{dw}}\left({\mathbf{F}}_{\mathrm{gate}}^k\right).$$

(26)

The resulting feature ${\mathbf{F}}_{\mathrm{fuse}}^k$ forms the unified multimodal representation used by the subsequent decoder.

3.4. Geometry-Aware Cross-Scale Refinement

Decoder-stage structural attenuation may weaken boundary and compact-object details during hierarchical reconstruction. Although skip fusion reintroduces multiscale encoder features, it cannot fully preserve structural cues throughout decoding. Therefore, GACS progressively reintroduces elevation-derived structural information to refine the fused decoder representation. The decoding process is initialized as ${\mathbf{D}}^4={\mathbf{F}}_{\mathrm{fuse}}^4$. For $k=3,2,1$, GACS combines the coarser decoded feature ${\mathbf{D}}^{k+1}$ with the corresponding fused encoder feature ${\mathbf{F}}_{\mathrm{fuse}}^k$ to produce ${\mathbf{D}}^k$, as illustrated in Figure 4.

• Cross-Scale Feature Fusion. The coarser decoded feature contains high-level semantic context, whereas the fused encoder feature retains finer spatial and multimodal information. At decoding stage k, ${\mathbf{D}}^{k+1}$ is first upsampled and projected to match the channel dimension of ${\mathbf{F}}_{\mathrm{fuse}}^k$. The aligned features are then concatenated and fused:

  $${\mathbf{X}}^k=\varphi \left(\left[{\mathrm{Conv}}_{1\times 1}\left(\mathrm{Up}\left({\mathbf{D}}^{k+1}\right)\right),{\mathrm{Conv}}_{1\times 1}\left({\mathbf{F}}_{\mathrm{fuse}}^k\right)\right]\right),$$

  (27)

  where $\varphi (·)$ is a compact fusion block composed of $1\times 1$ and $3\times 3$ convolutions.
• LCCG-Based Recalibration. Although cross-scale fusion combines semantic and spatial information, convolutional aggregation alone treats the resulting channel and spatial responses uniformly. To adaptively recalibrate the fused representation, the LCCG operation defined in Section 3.3 is applied:

  $${\mathbf{X}}_{\mathrm{gate}}^k=\mathrm{LCCG}\left({\mathbf{X}}^k\right).$$

  (28)

  This operation introduces bounded channel- and spatial-context modulation, providing a more selective cross-scale representation for subsequent structural refinement.
• Elevation-Guided Affine Refinement. The recalibrated cross-scale feature contains a multilevel context but does not explicitly reintroduce an elevation-derived structural prior. GACS therefore extracts the elevation structural prior ${\widehat{\mathbf{P}}}_{\mathrm{dsm}}^k$ from the residual-updated elevation feature ${\overline{\mathbf{F}}}_{\mathrm{dsm}}^k$ using Equation (2) and projects it into the decoder feature space. The affine formulation of SGAM is then reused, with ${\mathbf{X}}_{\mathrm{gate}}^k$ as the feature to be refined and the projected elevation prior as the structural conditioning signal:

  $${\widehat{\mathbf{X}}}^k=\mathrm{SGAM}\left({\mathbf{X}}_{\mathrm{gate}}^k,{\mathrm{Conv}}_{1\times 1}\left({\widehat{\mathbf{P}}}_{\mathrm{dsm}}^k\right)\right),$$

  (29)

  Unlike encoder-stage SGAM, which stabilizes bidirectional cross-modal transfer, the affine operation here introduces elevation-guided structural conditioning into an already-fused decoder representation.

To further improve local consistency and preserve a direct path from the fused encoder feature, local convolutional refinement and residual injection are applied as follows:

$${\mathbf{D}}^k={\mathrm{Conv}}_{3\times 3}\left({\widehat{\mathbf{X}}}^k\right)+{\mathrm{Conv}}_{1\times 1}\left({\mathbf{F}}_{\mathrm{fuse}}^k\right).$$

(30)

The local branch refines the elevation-conditioned response, whereas the residual branch preserves encoder-stage spatial information and stabilizes the decoded representation.

4. Experiments and Results

4.1. Datasets and Evaluation Metrics

4.1.1. ISPRS Datasets

The ISPRS Vaihingen and Potsdam datasets are widely used benchmarks for multimodal urban semantic segmentation. Vaihingen comprises 33 aerial tiles with varying spatial sizes and provides IRRG imagery and DSM data at a ground sampling distance of approximately 9 cm. Potsdam comprises 38 tiles of $6000\times 6000$ pixels and provides three optical image products, namely IRRG, RGB, and four-band RGBIR, together with DSM and nDSM data at approximately 5 cm. Both datasets contain six semantic classes: impervious surfaces, buildings, low vegetation, trees, cars, and clutter/background.

4.1.2. MMHunan Dataset

MMHunan is a multimodal land-cover dataset covering mountainous regions in Hunan, China [16]. It provides multispectral optical imagery and corresponding digital elevation model (DEM) data. The dataset contains 500 patches of $256\times 256$ pixels at a spatial resolution of approximately 10 m, with annotations for seven classes: cropland, forest, grassland, wetland, water, unused land, and built-up area. Compared with the ISPRS datasets, MMHunan differs substantially in sensor platform, spatial resolution, terrain complexity, scene composition, and class taxonomy. It therefore provides a complementary evaluation setting beyond fine-resolution urban aerial imagery.

Figure 5 presents representative optical images, elevation data, and ground-truth annotations from the three datasets. The displayed Vaihingen, Potsdam, and MMHunan patches throughout the manuscript cover approximately $23.0\times 23.0$ m, $25.6\times 25.6$ m, and $2.56\times 2.56$ km, respectively. The official six-class palette is used consistently for all ISPRS annotation and prediction maps, whereas a fixed seven-class palette is used for MMHunan because of its different class taxonomy. DSM, nDSM, and DEM data are displayed in grayscale.

4.1.3. Evaluation Metrics

We report class-wise Intersection over Union (IoU) and F1-score, their mean values (mIoU and mF₁), Overall Accuracy (OA), and Cohen’s Kappa coefficient [53]. For class c, IoU and F1-score are defined as

$${\mathrm{IoU}}_c={\displaystyle \frac{T{P}_c}{T{P}_c+F{P}_c+F{N}_c}},{\mathrm{F}}_{1,c}={\displaystyle \frac{2T{P}_c}{2T{P}_c+F{P}_c+F{N}_c}},$$

(31)

where $T{P}_c$, $F{P}_c$, and $F{N}_c$ denote the corresponding class-wise pixel counts. The mean metrics are obtained by averaging the class-wise values over the specified class set.

For the ISPRS Vaihingen and Potsdam datasets, mIoU₆ and mF₁ are calculated over all six classes. To assess the influence of the heterogeneous clutter/background category, we additionally report mIoU₅, which averages the IoU values of the five foreground classes and excludes clutter/background. For MMHunan, the mean metrics are calculated over all seven land-cover categories.

OA and Cohen’s Kappa are computed over the complete class set as

$$\mathrm{OA}={\displaystyle \frac{{\sum }_{i=1}^C{n}_{ii}}{{\sum }_{i=1}^C{\sum }_{j=1}^C{n}_{ij}}},\kappa ={\displaystyle \frac{p_o-p_e}{1-p_e}},$$

(32)

where $n_{ij}$ denotes the number of pixels belonging to class i but predicted as class j, while $p_o$ and $p_e$ denote the observed and chance-expected agreement, respectively.

4.2. Implementation Details

4.2.1. Dataset Composition and Split Strategy

The ISPRS benchmarks do not prescribe a fixed local training/validation/test partition among the publicly labeled tiles. To support reproducible local comparison, we construct fixed tile-level partitions for Vaihingen and Potsdam before patch extraction, augmentation, and model training, thereby preventing patches from the same source tile from appearing in different subsets.

For each tile, we calculate the pixel count and proportion of every semantic class. In Vaihingen, the four major classes each account for 21.25–27.81% of the labeled pixels, whereas cars and clutter/background account for only 1.25% and 0.78%, respectively. In Potsdam, the four major classes account for 15.51–29.57%, while cars and clutter/background account for 1.79% and 4.83%. The tiles are therefore assigned according to their class statistics so that all six classes are represented and their distributions remain broadly comparable across the training, validation, and test subsets, with particular attention to the less frequent categories.

The resulting Vaihingen split contains 22 training tiles, 6 validation tiles, and 5 test tiles, accounting for 64.6%, 17.9%, and 17.5% of the labeled pixels, respectively. The Potsdam split contains 26 training tiles, 6 validation tiles, and 6 test tiles, corresponding to 68.4%, 15.8%, and 15.8% of the labeled pixels. MMHunan follows its released partition of 400 training patches, 50 validation patches, and 50 test patches.

As shown in Figure 6, the fixed ISPRS partitions retain broadly comparable class distributions across the three subsets. The complete tile identifiers and the exact class-wise pixel counts and proportions for the full datasets and individual subsets are provided in the Supplementary Materials. These partitions are intended for controlled and reproducible local comparison rather than for reproducing official benchmark-server scores.

4.2.2. Dataset Preparation and Augmentation

After tile-level partitioning, Vaihingen tiles are cropped into $256\times 256$ patches with a stride of 192, while Potsdam tiles are cropped into $512\times 512$ patches with a stride of 384, corresponding to a 25% overlap. Mirror padding is applied at the outer boundaries of each tile. MMHunan is directly used in its released $256\times 256$ patch form without additional patch extraction.

IRRG imagery is used as the optical input in the main experiments on all three datasets. Vaihingen uses the nDSM generated by Gerke [54], Potsdam uses the officially released nDSM, and MMHunan uses Z-score-normalized DEM data. For the extended spectral analysis on Potsdam, RGB and RGBIR inputs are additionally evaluated against the main IRRG configuration.

For the ISPRS datasets, offline photometric augmentation includes Gaussian noise injection and brightness adjustment, both applied only to the optical imagery in the training set. During training, online geometric augmentation is applied to all three datasets: random scaling in the range of 0.8–1.2 and rotations by multiples of 90° are synchronously applied to the optical image, elevation map, and segmentation label. No offline photometric augmentation is applied to MMHunan.

Table 2. Patch extraction settings and final sample counts.

Dataset	Patch Size	Overlap	Train	Val	Test
Vaihingen	256 × 256	25%	5756	789	780
Potsdam	512 × 512	25%	11,700	1350	1350
MMHunan	256 × 256	0%	400	50	50

summarizes the patch extraction settings and final sample counts; the ISPRS training counts include the offline augmented samples.

4.2.3. Experimental Settings

We compare GCF-Net with seven task-specific multimodal baselines: CFFormer [13], FTransDeepLab [21], FTransUNet [7], PACSCNet [11], CMX [28], CMFNet [12], and SA-Gate [27]. MFNet [17], with its MMAdapter-based ViT-B configuration, is additionally included as a representative SAM-based multimodal baseline. Six unimodal baselines are also evaluated: HBGNet [40], SSNet [35], CMLFormer [36], BEDSN [42], CMTFNet [37], and MsanlfNet [23]. The comparison therefore covers conventional unimodal, boundary-aware, task-specific multimodal, and foundation-model-based approaches.

All methods use the same data partitions, preprocessing, augmentation, checkpoint-selection criterion, and evaluation code. All multimodal methods receive the same optical–elevation input pairs, whereas unimodal methods use the corresponding optical input only. For GCF-Net and the task-specific baselines, a common optimizer, learning-rate schedule, and training budget are used to support a controlled architectural comparison, while architecture-specific backbones, pretrained initializations, loss functions, and auxiliary supervision follow the corresponding papers or public implementations. This matched protocol is not intended to reproduce the separately optimized best-case configuration of every baseline. MFNet retains its pretrained SAM initialization, MMAdapter configuration, parameter-freezing strategy, and recommended fine-tuning settings, while all remaining evaluation settings follow the unified protocol described above.

All experiments are conducted on a single NVIDIA RTX 5070 Ti GPU. Except for MFNet, models are trained for 100 epochs using AdamW with an initial learning rate of $1\times {10}^{-3}$ and a weight decay of $0.01$. A linear warm-up is applied during the first five epochs, followed by cosine annealing. The batch size is set to 4 for Potsdam and 16 for Vaihingen and MMHunan. GCF-Net is optimized using the standard pixel-wise cross-entropy loss, whereas the original loss formulations and architecture-specific auxiliary supervision of the compared methods are retained.

For every method, the checkpoint with the highest validation mIoU over the complete class set is selected for testing. The additional mIoU₅ metric is used only for reporting and does not affect model selection.

4.3. Comparative Results

4.3.1. Results on Vaihingen

Table 3. Quantitative comparison on ISPRS Vaihingen (%). Best and second-best results are bold and underlined; mIoU₅ excludes clutter, whereas mIoU₆ includes it.

Method	IoU						mIoU5	mIoU6	mF1	OA	Kappa
	Imp. Surf.	Build.	LowVeg	Tree	Car	Clutter
Multimodal methods
GCF-Net (ours)	82.73	89.08	70.26	77.20	66.51	48.43	77.16	72.37	83.26	88.40	84.69
MFNet [17]	82.00	88.15	69.36	76.03	61.45	42.75	75.40	69.96	81.35	88.37	84.64
CFFormer [13]	79.22	83.71	68.62	76.96	50.54	29.58	71.81	64.77	76.78	86.57	82.24
FTransDeepLab [21]	80.02	85.64	68.15	76.36	50.11	28.12	72.06	64.73	76.58	86.78	82.53
FTransUNet [7]	78.98	84.01	69.28	77.22	54.50	35.17	72.80	66.53	78.53	86.79	82.53
PACSCNet [11]	80.51	87.17	69.81	77.01	64.02	46.58	75.70	70.85	82.20	87.67	83.72
CMX [28]	79.84	85.14	68.92	76.58	62.15	25.32	74.53	66.32	77.69	86.93	82.73
CMFNet [12]	80.94	87.59	68.96	76.68	60.59	29.27	74.95	67.34	78.67	87.41	83.38
SA-Gate [27]	79.14	85.58	67.69	76.82	52.89	30.74	72.42	65.48	77.40	86.60	82.32
Unimodal methods
HBGNet [40]	81.92	87.83	69.18	77.04	59.58	37.47	75.11	68.84	80.26	87.86	83.96
SSNet [35]	77.21	79.15	63.10	73.45	47.23	40.06	68.03	63.37	76.49	84.12	79.01
CMLFormer [36]	78.91	84.08	64.88	74.40	52.15	35.87	70.88	65.05	77.49	85.61	80.97
BEDSN [42]	80.91	86.53	67.31	75.49	58.99	30.93	73.85	66.69	78.36	86.90	82.69
CMTFNet [37]	80.73	86.77	68.52	75.18	58.56	40.27	73.95	68.34	80.11	86.99	82.82
MsanlfNet [23]	78.30	83.12	64.47	74.60	47.50	40.55	69.60	64.75	77.43	85.30	80.56

reports the quantitative results on the ISPRS Vaihingen dataset. GCF-Net obtains 77.16% mIoU₅ and 72.37% mIoU₆, together with an mF₁ of 83.26%, an OA of 88.40%, and a Kappa coefficient of 84.69%. Relative to PACSCNet, which provides the second-highest overlap-based mean scores, the margins are 1.46% for mIoU₅ and 1.52% for mIoU₆. These results indicate a consistent, although moderate, overall improvement under the controlled protocol.

At the class level, GCF-Net achieves the highest IoU on five of the six categories and the second-highest result on trees, where it differs from FTransUNet by only 0.02%. The largest foreground-class margin is observed for cars, for which GCF-Net exceeds PACSCNet by 2.49%. The clutter result is retained for completeness but is interpreted as a supplementary observation because of the limited and heterogeneous samples in this category. MFNet provides competitive OA and Kappa values, indicating the potential of foundation-model adaptation under this setting, while its overlap-based mean scores remain below those of GCF-Net and PACSCNet.

Figure 7 shows that GCF-Net produces relatively coherent car and building regions and reduces fragmented predictions around vegetation boundaries and visually similar roof surfaces. Compared with several baselines, compact vehicles are more frequently preserved as continuous regions, while local confusion among buildings, impervious surfaces, and clutter is reduced. These observations are consistent with the quantitative results on compact urban objects and structure-related categories.

4.3.2. Results on Potsdam

Table 4. Quantitative comparison on ISPRS Potsdam (%). Best and second-best results are bold and underlined; mIoU₅ excludes clutter, whereas mIoU₆ includes it.

Method	IoU						mIoU5	mIoU6	mF1	OA	Kappa
	Imp. Surf.	Build.	LowVeg	Tree	Car	Clutter
Multimodal methods
GCF-Net (ours)	83.09	92.01	73.41	76.15	83.39	44.22	81.61	75.38	85.00	88.58	85.21
MFNet [17]	80.56	91.47	71.47	75.06	82.65	34.66	80.24	72.65	82.65	87.46	83.68
CFFormer [13]	80.45	91.38	71.64	73.96	81.72	35.44	79.83	72.43	82.57	87.37	83.56
FTransDeepLab [21]	79.02	90.74	68.72	70.08	75.69	31.55	76.85	69.30	80.24	85.98	81.74
FTransUNet [7]	79.27	92.19	71.81	73.51	78.76	31.99	79.11	71.26	81.55	84.12	83.16
PACSCNet [11]	81.34	91.26	72.40	75.63	81.08	35.34	80.34	72.84	82.84	87.92	84.23
CMX [28]	80.75	91.18	71.03	73.36	82.84	34.67	79.83	72.31	82.42	87.23	83.38
CMFNet [12]	80.76	90.02	70.68	73.33	82.01	33.04	79.36	71.64	81.88	86.86	82.91
SA-Gate [27]	79.98	90.50	69.47	72.54	79.19	34.02	78.34	70.95	81.51	86.59	82.53
Unimodal methods
HBGNet [40]	82.62	90.17	72.38	75.31	83.11	36.76	80.72	73.39	83.29	87.87	84.22
SSNet [35]	75.18	83.56	66.34	65.66	75.99	35.91	73.35	67.11	79.18	83.25	78.21
CMLFormer [36]	79.78	88.70	69.89	71.08	79.87	36.51	77.86	70.97	81.74	86.33	82.19
BEDSN [42]	80.79	90.49	71.91	74.23	83.21	38.06	80.13	73.12	83.20	87.58	83.81
CMTFNet [37]	78.24	86.68	70.90	74.01	78.78	37.68	77.72	71.05	81.93	86.10	81.94
MsanlfNet [23]	80.09	89.12	69.82	70.74	80.43	35.32	78.04	70.92	81.61	86.35	82.21

summarizes the quantitative results on the ISPRS Potsdam dataset. GCF-Net obtains 81.61% mIoU₅ and 75.38% mIoU₆, together with an mF₁ of 85.00%, an OA of 88.58%, and a Kappa coefficient of 85.21%. Relative to the second-highest results, its margins are 0.89% for mIoU₅ and 1.99% for mIoU₆. Because the larger six-class margin is partly affected by the clutter/background category, mIoU₅ provides the more direct comparison of the five principal foreground classes.

GCF-Net achieves the highest IoU on five categories and the second-highest result on buildings. For cars, GCF-Net exceeds the second-ranked method by 0.18%, whereas its building IoU is 0.18% below the best-performing method. The results therefore support consistent performance across urban categories rather than a large advantage on every individual class. MFNet also provides competitive results, but its mIoU₅ and mIoU₆ remain 1.37 and 2.73% below those of GCF-Net, respectively.

Figure 8 shows that GCF-Net produces relatively complete urban regions and preserves the continuity of compact or elongated vehicles. Several competing methods exhibit more fragmented car predictions or local confusion among buildings, impervious surfaces, and vegetation. The qualitative differences are consistent with the moderate quantitative gains reported in Table 4.

4.3.3. Results on MMHunan

MMHunan provides a substantially different evaluation setting from the two ISPRS datasets, using approximately 10 m multispectral satellite imagery and elevation data over mountainous land-cover scenes. As shown in

Table 5. Quantitative comparison on MMHunan (%). Best and second-best results are bold and underlined, respectively.

Method	IoU							mIoU	mF1	OA	Kappa
	Crop	Forest	Grass	Wet.	Water	Unused	Built.
Multimodal methods
GCF-Net (ours)	68.92	80.27	21.85	23.32	77.24	29.49	64.52	52.23	65.07	82.42	76.07
MFNet [17]	63.21	73.13	22.96	17.68	76.93	28.25	62.81	49.28	62.50	76.89	68.96
CFFormer [13]	60.20	76.13	19.85	12.52	74.71	25.91	58.59	46.84	59.65	77.43	69.86
FTransDeepLab [21]	61.43	78.99	12.63	6.15	71.73	20.43	58.35	44.24	55.65	78.80	71.02
FTransUNet [7]	66.34	80.76	15.57	12.46	74.58	23.43	63.80	48.14	59.93	81.14	73.34
PACSCNet [11]	66.69	76.59	27.30	18.38	75.17	22.06	63.48	49.96	62.91	80.10	73.36
CMX [28]	63.83	78.93	22.59	5.96	71.92	21.13	63.07	46.78	58.59	79.65	72.46
CMFNet [12]	63.62	76.63	22.44	16.23	75.89	21.41	63.47	48.53	61.12	73.40	72.38
SA-Gate [27]	60.97	79.49	17.10	6.24	74.47	17.98	60.99	45.32	56.70	80.03	72.82
Unimodal methods
HBGNet [40]	65.45	78.06	27.03	11.70	75.63	21.72	60.93	48.65	61.12	80.62	73.77
SSNet [35]	60.10	74.28	7.89	15.26	76.59	14.41	54.70	43.32	54.87	77.83	69.59
CMLFormer [36]	59.43	77.07	8.61	11.42	77.29	22.71	57.94	44.92	56.50	78.99	71.14
BEDSN [42]	63.37	77.29	15.02	9.03	70.48	20.32	63.16	45.52	57.33	78.39	70.88
CMTFNet [37]	65.16	77.51	16.11	15.51	65.54	29.59	57.94	46.77	59.87	77.91	70.14
MsanlfNet [23]	55.13	76.14	9.78	1.65	75.17	12.09	51.72	40.24	50.60	77.08	68.45

, GCF-Net obtains an mIoU of 52.23%, an mF₁ of 65.07%, an OA of 82.42%, and a Kappa coefficient of 76.07%. Compared with PACSCNet, which achieves the second-highest mIoU and mF₁, the corresponding margins are 2.27 and 2.16%, respectively.

At the class level, GCF-Net achieves the highest IoU values on cropland, wetland, and built-up area, and the second-highest results on forest, water, and unused land. Grassland remains challenging because of its fragmented distribution and confusion with other vegetation categories. Overall, GCF-Net ranks among the top-two methods on six of the seven classes, indicating relatively consistent performance across heterogeneous land-cover types. This setting provides sensor- and resolution-diverse in-domain evaluation, but is not interpreted as direct cross-dataset transfer.

Figure 9 shows that GCF-Net produces relatively coherent cropland, wetland, forest, and built-up regions in terrain-complex scenes. Compared with several baselines, it reduces fragmented predictions and local confusion along rivers and heterogeneous land-cover boundaries. These observations are consistent with the aggregate results in Table 5.

4.4. Ablation Study and Component-Wise Analysis

4.4.1. Overall Ablation of Major Components

We first evaluate the individual and cumulative contributions of SGCM, FSCM, and GACS on ISPRS Vaihingen. All variants use the same dual-branch encoder–decoder backbone and follow the training settings in Section 4.2.3. The plain baseline V0 uses direct optical–elevation feature concatenation and standard skip-based decoding. V1, V2, and V3 individually introduce SGCM, FSCM, and GACS, respectively, while the full model combines all three components.

As shown in

Table 6. Overall component ablation on ISPRS Vaihingen. $\Delta {\mathrm{mIoU}}_6$ is relative to V0; best and second-best results are bold and underlined, respectively. A check mark indicates that the corresponding module is enabled.

Variant	Modules			Performance (%)
	SGCM	FSCM	GACS	mIoU6	ΔmIoU6	mF1	OA	Kappa
V0				66.51	–	78.65	86.51	82.18
V1	✓			70.08	+3.57	81.41	87.61	83.65
V2		✓		68.08	+1.57	79.41	87.82	83.89
V3			✓	67.18	+0.67	78.70	87.07	82.92
Full	✓	✓	✓	72.37	+5.86	83.26	88.40	84.69

, SGCM provides the largest individual contribution, improving mIoU₆ from 66.51% to 70.08%, corresponding to a gain of 3.57%. This result supports the role of structure-conditioned correction before multimodal fusion. FSCM improves mIoU₆ by 1.57% through bounded cross-modal magnitude modulation, spatial reconstruction, and global–local recalibration. GACS produces a smaller gain of 0.67%, indicating that elevation-conditioned decoder refinement is beneficial but less effective when used without the preceding correction and fusion stages. The complete model improves mIoU₆ by 5.86% and also obtains the highest mF₁, OA, and Kappa values. The cumulative result supports the complementary roles of the three components at different representation stages.

Figure 10 shows that the proposed components progressively reduce fragmented predictions and improve the completeness of compact objects, particularly cars. The full model produces more coherent object regions than the individual variants, consistent with the quantitative trends in Table 6.

As SGCM yields the largest individual gain in the ablation study, Figure 11 visualizes its Stage-2 correction process in the optical and nDSM branches. The before- and after-correction maps show the channel-wise RMS feature energy, $\mathbf{E}\left(\mathbf{F}\right)=\sqrt{{\displaystyle \frac{1}{C_k}}{\sum }_{c=1}^{C_k}{\mathbf{F}}_c^2}$, whereas the structural prior maps are shown directly for reference. The signed energy change is computed as $\Delta \mathbf{E}=\mathbf{E}\left({\mathbf{F}}_{\mathrm{after}}\right)-\mathbf{E}\left({\mathbf{F}}_{\mathrm{before}}\right)$. The before- and after-correction maps share the same normalization range within each modality and sample, while $\Delta \mathbf{E}$ is normalized symmetrically around zero. The localized positive and negative changes suggest that SGCM selectively redistributes feature responses rather than uniformly amplifying them, which is consistent with its role as a pre-fusion correction module.

4.4.2. Stage-Wise Effect of Structural Priors

We further examine how structural priors contribute at the encoder and decoder stages. The encoder configuration controls the modality-specific priors used by SGCM, whereas the decoder configuration controls the elevation-derived prior used by GACS. Four variants are evaluated: without structural priors (w/o Priors), encoder-only guidance (Enc.-only), decoder-only guidance (Dec.-only), and joint encoder–decoder guidance (Enc.+Dec.). The w/o Priors configuration retains the SGCM, FSCM, and GACS computational pathways but removes their structural-prior inputs. All other operations are kept unchanged to isolate the contribution of prior conditioning.

To assess whether the relatively modest decoder-stage contribution is stable across repeated runs, each configuration is independently trained using three fixed random seeds under identical settings. The results are reported as mean ± standard deviation to evaluate the stability of the observed trends.

Table 7. Stage-wise structural-prior ablation on ISPRS Vaihingen. Results are mean ± standard deviation over three seeds, and $\Delta {\mathrm{mIoU}}_6$ is relative to w/o Priors. Best and second-best results are bold and underlined, respectively. A check mark indicates that the corresponding structural prior is enabled.

Variant	Priors		Performance (%)
	Enc.	Dec.	mIoU6	ΔmIoU6	mF1	OA	Kappa
w/o Priors			$70.07\pm 0.22$	–	$81.40\pm 0.18$	$87.95\pm 0.08$	$84.10\pm 0.11$
Enc.-only	✓		$\underline{71.78\pm 0.18}$	+1.71	$\underline{82.91\pm 0.15}$	$88.05\pm 0.07$	$84.23\pm 0.09$
Dec.-only		✓	$70.79\pm 0.20$	+0.72	$81.95\pm 0.17$	$\underline{88.09\pm 0.08}$	$\underline{84.26\pm 0.10}$
Enc.+Dec.	✓	✓	$\mathbf{72.38}\pm \mathbf{0.13}$	+2.31	$\mathbf{83.27}\pm \mathbf{0.11}$	$\mathbf{88.41}\pm \mathbf{0.05}$	$\mathbf{84.70}\pm \mathbf{0.07}$

shows that encoder-stage structural guidance improves the mean mIoU₆ from 70.07% to 71.78%, corresponding to a gain of 1.71%. Decoder-only guidance yields a smaller gain of 0.72%. Combining encoder and decoder priors produces $72.38\pm 0.13$% mIoU₆, exceeding Enc.-only by 0.60% and w/o Priors by 2.31%. The relatively small standard deviations and consistent ordering across the three runs provide repeated-run evidence that these trends are stable.

The results indicate that encoder-stage structural correction is the primary contributor, whereas decoder-stage elevation guidance provides a modest but complementary benefit. Accordingly, “geometry-aware” in GACS refers to elevation-derived structural conditioning during decoder refinement rather than an explicit geometric constraint or a dominant independent source of performance improvement.

4.4.3. Comparison of Fusion Operators

To compare FSCM with a standard attention-based alternative, we evaluate direct concatenation, cross-attention, and FSCM using the same dual-branch baseline. SGCM and GACS are disabled in all configurations, and only the fusion-stage operator is changed.

As shown in

Table 8. Comparison of fusion operators on ISPRS Vaihingen. $\Delta {\mathrm{mIoU}}_6$ is relative to direct concatenation; best and second-best results are bold and underlined, respectively.

Fusion Operator	mIoU6	ΔmIoU6	mF1	OA	Kappa
Direct concatenation	66.51	–	78.65	86.51	82.18
Cross-attention	67.82	+1.31	79.40	87.61	83.63
FSCM	68.08	+1.57	79.41	87.82	83.89

, replacing direct concatenation with cross-attention improves mIoU₆ from 66.51% to 67.82%, indicating that explicit cross-modal interaction is beneficial under this setting. FSCM further increases mIoU₆ to 68.08% and obtains slightly higher mF₁, OA, and Kappa values than cross-attention. The margin over cross-attention is modest, with a 0.26% improvement in mIoU₆, and should not be interpreted as a large advantage. Nevertheless, the consistent gains across the reported metrics support the use of complementary frequency-domain magnitude conditioning and spatial recalibration rather than relying solely on attention-based feature interaction.

4.4.4. Qualitative Analysis of FSCM Magnitude Modulation

To illustrate the frequency-domain behavior of FSCM, Figure 12 presents representative Stage-2 magnitude spectra from the optical and nDSM branches, together with the fused magnitude representation obtained after cross-modal mask injection and magnitude mixing. Each column corresponds to one Vaihingen sample, while the three rows show the optical, nDSM, and fused representations, respectively.

As shown in Figure 12, the optical and nDSM branches exhibit visibly different magnitude-response distributions. In the displayed examples, the optical spectra are generally more dispersed, whereas the nDSM spectra contain comparatively concentrated response patterns. After cross-modal conditioning and magnitude mixing, the fused spectra exhibit response distributions that differ from either input branch alone, illustrating that FSCM constructs a jointly conditioned spectral representation rather than directly retaining one modality.

This visualization is intended to provide qualitative insight into the magnitude-modulation process rather than to establish the superiority of a particular frequency pattern. Together with the fusion-operator comparison in Table 8, it supports the interpretation of FSCM as a complementary frequency-aware conditioning pathway that provides modest but consistent improvements over direct concatenation and cross-attention. Because CSMI performs bounded enhancement-only modulation, these results should not be interpreted as explicit spectral normalization, balancing, or suppression of the two modalities.

4.5. Sensitivity to Backbone and Optical Spectral Configuration

4.5.1. Backbone Sensitivity

To examine the sensitivity of GCF-Net to the encoder architecture, we evaluate four representative backbones on ISPRS Vaihingen: PVTv2-B2 [52], Swin-Tiny [55], ConvNeXt-Tiny [56], and ResNet-50 [57]. They represent pyramid transformers, window-based transformers, modern convolutional networks, and conventional residual networks, respectively, with backbone parameter counts ranging from 23.51 M to 28.29 M. All variants use the same data partition, preprocessing, augmentation, training budget, checkpoint-selection criterion, and evaluation code, while retaining the corresponding backbone-specific pretrained initialization.

As shown in

Table 9. Backbone sensitivity on ISPRS Vaihingen. mIoU₆ is calculated over all six classes. Best and second-best results are bold and underlined, respectively.

Backbone	Backbone Params (M)	Performance (%)
		mIoU6	mF1	OA	Kappa
PVTv2-B2	24.85	72.37	83.26	88.40	84.69
Swin-Tiny	28.29	72.31	83.41	87.99	84.14
ConvNeXt-Tiny	27.82	70.97	82.17	88.12	84.31
ResNet-50	23.51	69.90	81.15	88.24	84.46

, PVTv2-B2 obtains the highest mIoU₆, OA, and Kappa values, whereas Swin-Tiny achieves the highest mF₁. The mIoU₆ difference between the two Transformer-based backbones is only 0.06%, while ConvNeXt-Tiny and ResNet-50 yield lower overlap-based mean scores, although their OA and Kappa values remain relatively close. Across the four backbones, mIoU₆ ranges from 69.90% to 72.37%, indicating that GCF-Net can be instantiated with different encoder families, while its final performance remains affected by backbone selection. The results therefore support architectural portability rather than complete insensitivity to the encoder architecture.

4.5.2. Optical Spectral Configuration Sensitivity

We further examine the influence of the optical spectral input using IRRG, RGB, and RGBIR configurations on ISPRS Potsdam. The same nDSM input, data partition, preprocessing, training settings, and evaluation protocol are retained in all variants, and only the optical-band configuration is changed. The input projection of the optical branch is adapted to the corresponding number of channels, while the remaining network architecture is unchanged.

As shown in

Table 10. Sensitivity to optical spectral configuration on ISPRS Potsdam. mIoU₆ is calculated over all six classes. Best and second-best results are bold and underlined, respectively.

Optical Input	Performance (%)
	mIoU6	mF1	OA	Kappa
IRRG	75.38	85.00	88.58	85.21
RGB	74.11	84.19	87.80	84.18
RGBIR	74.35	84.22	88.28	84.76

, IRRG obtains the highest scores across all four metrics, with mIoU₆ exceeding those of RGBIR and RGB by 1.03% and 1.27%, respectively. The three configurations nevertheless remain within a relatively narrow performance range, indicating that GCF-Net can operate with different optical-band combinations. The fact that RGBIR does not outperform IRRG also suggests that increased spectral dimensionality alone does not necessarily improve optical–elevation fusion. These results support the use of IRRG as the main configuration in this study without implying that it is universally optimal across sensors or datasets.

4.6. Robustness to Elevation Noise and Misalignment

We evaluate sensitivity to elevation noise and registration errors on ISPRS Vaihingen by perturbing only the test-set nDSM while keeping the optical images, annotations, trained parameters, and evaluation protocol unchanged. GCF-Net, PACSCNet, MFNet, CMFNet, and FTransUNet are evaluated on the same perturbed samples without retraining or adaptation.

Zero-mean Gaussian noise is added as

$$\tilde{E}=E+\sigma s_Eϵ,ϵ\sim \mathcal{N}(0,1),$$

(33)

where $s_E$ is the global standard deviation of the training-set nDSM values and $\sigma \in \{0.05,0.10,0.20\}$. A fixed base noise realization is scaled across the three levels. For spatial misalignment, the nDSM is translated by 1, 3, and 5 pixels, corresponding to approximately 0.09, 0.27, and 0.45 m. The four cardinal directions are evenly assigned across the test samples, with reflection padding at image boundaries.

As shown in

Table 11. Robustness to elevation perturbations on ISPRS Vaihingen. Parentheses indicate the mIoU₆ change from the clean-input result; best and second-best scores are bold and underlined.

Perturbation	Level	mIoU6 (%)
		GCF-Net	PACSCNet	MFNet	CMFNet	FTransUNet
Clean	–	72.37	70.85	69.96	67.34	66.53
Gaussian noise	$\sigma =0.05$	72.01 ($-0.36$)	70.37 ($-0.48$)	69.97 ($+0.01$)	67.29 ($-0.05$)	65.43 ($-1.10$)
	$\sigma =0.10$	71.55 ($-0.82$)	69.11 ($-1.74$)	70.00 ($+0.04$)	67.03 ($-0.31$)	63.55 ($-2.98$)
	$\sigma =0.20$	71.39 ($-0.98$)	65.48 ($-5.37$)	70.06 ($+0.10$)	60.85 ($-6.49$)	60.80 ($-5.73$)
Misalignment	1 pixel	72.38 ($+0.01$)	70.80 ($-0.05$)	69.97 ($+0.01$)	67.29 ($-0.05$)	66.57 ($+0.04$)
	3 pixels	72.34 ($-0.03$)	70.79 ($-0.06$)	69.94 ($-0.02$)	67.15 ($-0.19$)	66.65 ($+0.12$)
	5 pixels	72.25 ($-0.12$)	70.73 ($-0.12$)	69.94 ($-0.02$)	66.98 ($-0.36$)	66.34 ($-0.19$)

, GCF-Net maintains the highest absolute mIoU₆ under all evaluated perturbation settings, with a maximum decrease of 0.98% under Gaussian noise and 0.12% under spatial translation. In contrast, PACSCNet, CMFNet, and FTransUNet exhibit larger degradations under the strongest noise level, indicating greater sensitivity to corrupted elevation inputs. MFNet also remains highly stable under both noise and translation, with variations within 0.10%. This limited sensitivity may be associated with the strong optical representation inherited from its pretrained SAM backbone, although the present experiment does not isolate the relative contributions of the optical and elevation modalities. Overall, GCF-Net combines high absolute accuracy with limited degradation under the tested elevation perturbations, but these results do not imply invariance to severe corruption, non-rigid misregistration, or substantially larger displacements.

4.7. Computational Complexity and Inference Efficiency

We evaluate computational complexity, total parameter count, and inference throughput using FLOPs, parameters, and FPS, respectively. FLOPs and parameters are estimated using ptflops. FPS is measured in FP32 with a batch size of 1 on a single NVIDIA RTX 5070 Ti GPU, using 10 warm-up iterations followed by 100 timed forward passes with CUDA synchronization. Data loading and post-processing are excluded. Multimodal methods use their complete dual-input forward process, whereas unimodal methods retain their original single-input setting.

As shown in

Table 12. Computational complexity and inference efficiency. Best and second-best values within each method type are bold and underlined, respectively; lower FLOPs and total parameter counts and higher FPS are preferred.

Type	Method	Input Size	FLOPs (G)	Total Params (M)	FPS
Multimodal	GCF-Net (ours)	$256\times 256$	18.53	81.27	30.50
	MFNet [17]	$256\times 256$	71.81	106.03	29.00
	CFFormer [13]	$256\times 256$	22.42	77.24	19.91
Multimodal	FTransDeepLab [21]	$256\times 256$	14.98	64.56	43.52
	FTransUNet [7]	$256\times 256$	59.73	203.40	34.74
	PACSCNet [11]	$256\times 256$	37.86	94.05	48.80
	CMX [28]	$256\times 256$	41.48	182.63	16.43
	CMFNet [12]	$256\times 256$	79.76	104.07	28.90
	SA-Gate [27]	$256\times 256$	41.33	110.85	59.74
Unimodal	HBGNet [40]	$256\times 256$	59.01	30.45	68.89
	SSNet [35]	$256\times 256$	10.35	47.23	90.91
	CMLFormer [36]	$256\times 256$	6.54	21.53	89.22
	BEDSN [42]	$256\times 256$	85.86	22.28	90.15
	CMTFNet [37]	$256\times 256$	8.56	30.07	116.53
	MsanlfNet [23]	$256\times 256$	6.95	34.96	57.29

, GCF-Net requires 18.53 G FLOPs and 81.27 M total parameters and achieves 30.50 FPS. Among the multimodal methods, it has the second-lowest FLOPs and the third-lowest parameter count, while its inference throughput is moderate rather than the highest. The unimodal methods generally exhibit lower computational costs or higher throughput because they process only a single optical input; in particular, CMLFormer has the lowest FLOPs and parameter count, while CMTFNet achieves the highest FPS within this group. These unimodal results provide a computational reference rather than a directly equivalent comparison with dual-input multimodal models. Combined with its quantitative performance on the three datasets, GCF-Net provides a balanced accuracy–efficiency trade-off among the evaluated multimodal methods, achieving the highest overall segmentation scores with lower theoretical computation than most multimodal baselines.

5. Discussion

5.1. Interpretation of Stage-Aligned Fusion

Stage-aligned fusion organizes multimodal processing according to the representation state rather than repeatedly applying a common interaction mechanism across feature levels. SGCM operates on separate modality-specific features before fusion, FSCM constructs a joint representation from the corrected features, and GACS refines already-fused representations during decoding. Their different inputs, conditioning signals, and processing objectives make these three modules complementary rather than interchangeable. This interpretation is consistent with the component-wise ablation results, in which the modules provide distinct and cumulative contributions. Accordingly, the contribution of GCF-Net does not lie in introducing a universally superior fusion operator, but in explicitly separating modality correction, multimodal integration, and decoder-stage structural refinement according to the evolution of representations. Within this formulation, “geometry-aware” denotes elevation-conditioned structural guidance rather than an explicit geometric constraint.

The replacement experiment further clarifies the relationship between FSCM and cross-attention-based interaction. Cross-attention, as used in SGCM, models cross-modal correspondence through query–key interaction, whereas FSCM performs bounded frequency-aware conditioning of modality-specific magnitude responses before spatial reconstruction. The modality-specific phase components, together with the direct spatial baseline, preserve branch-dependent reconstruction cues and reduce over-reliance on frequency-domain processing. The modest performance difference observed in the replacement experiment does not establish frequency-domain fusion as universally superior to cross-attention. Instead, it suggests that frequency-aware conditioning and attention-based correspondence modeling are complementary when they are assigned to different representation states.

5.2. Robustness, Efficiency, and Practical Implications

The sensitivity and perturbation experiments jointly indicate that the stage-aligned formulation is not restricted to a single encoder or optical-band configuration and remains effective under moderate degradation of the elevation input. The relatively stable performance across different backbones and spectral settings supports a degree of architectural and input flexibility, although these results should not be interpreted as evidence of complete insensitivity or cross-domain generalization. Under elevation perturbations, GCF-Net maintains high absolute segmentation accuracy with limited performance degradation. MFNet shows smaller relative variations in some settings, possibly because its pretrained SAM backbone provides a strong optical representation; however, the present experiments do not isolate the individual contributions of these two modalities. These results suggest that robustness should be assessed jointly in terms of absolute accuracy and sensitivity to modality degradation.

GCF-Net provides an accuracy–efficiency trade-off rather than a lightweight or runtime-optimal solution. Its dual-branch architecture and stage-specific operations introduce additional computational cost, but the reported complexity remains lower than that of most evaluated multimodal baselines while maintaining strong segmentation performance. This trade-off is relevant to applications such as urban land-cover mapping, building and compact-object delineation, and terrain-aware scene interpretation, particularly when reliable DSM, nDSM, or DEM data are available. The consistent gains across datasets and structure-sensitive categories suggest that stage-aligned fusion is potentially useful when semantic discrimination and structural continuity are both important.

5.3. Limitations and Future Directions

Several limitations remain. First, the evaluation includes two high-resolution aerial urban benchmarks and one lower-resolution satellite land-cover dataset, but all experiments follow in-domain training and testing protocols. The results therefore do not establish cross-dataset, cross-region, or cross-sensor generalization. Moreover, GCF-Net assumes that elevation data are available and reasonably aligned with the optical imagery. The current perturbation analysis considers synthetic Gaussian noise and global rigid translations of up to 5 pixels, but does not cover local non-rigid misregistration, missing elevation regions, systematic height bias, or severe sensor artifacts.

The dual-branch encoder and stage-specific processing also introduce non-negligible parameter and runtime costs. In addition, the structural priors and frequency-aware operations are empirically motivated and are not supported by a formal guarantee of geometric consistency. Future work will therefore investigate cross-region and cross-sensor evaluation, explicit optical–elevation alignment, elevation reliability and uncertainty modeling, missing-modality adaptation, and lighter shared or distilled architectures. Integrating these directions with remote sensing foundation models may further improve transferability while retaining the task-specific benefits of optical–elevation fusion.

6. Conclusions

This paper presented GCF-Net, a stage-aligned optical–elevation fusion network for remote sensing semantic segmentation. Rather than organizing multimodal interaction according to feature resolution alone, GCF-Net assigns distinct processing objectives to different representation states. SGCM performs structure-conditioned correction before fusion, FSCM constructs frequency–spatial joint representations, and GACS reintroduces elevation-derived structural guidance during decoding.

Experiments on ISPRS Vaihingen, ISPRS Potsdam, and MMHunan demonstrate competitive and consistent performance compared with representative unimodal, multimodal, and SAM-based methods. Component-wise and replacement experiments support the distinct and complementary roles of the stage-specific modules, while sensitivity, elevation-perturbation, and complexity analyses indicate architectural flexibility, tolerance to moderate elevation degradation, and a balanced accuracy–efficiency trade-off. Overall, the results support stage-aligned fusion as an effective framework for coordinating optical appearance and elevation-derived structural information throughout the encoder–decoder process.