DBCS-T: A Dual-Branch Cross-Attention Synergistic Transformer for Multimodal Image Fusion and Semantic Segmentation

Highlights

What are the main findings?

• The proposed DBCS-T-based fusion method can effectively extract complementary polarization, spectral and intensity features. The fused multimodal image has higher information entropy and lower distribution divergence compared with original inputs, while maintaining high consistency with source data.
• The generated fusion image achieves accurate semantic segmentation of real vegetation, artificial foliage and same-color metallic objects under natural illumination, verifying the effectiveness of the designed network and fusion strategy.

What are the implications of the main findings?

• This method fully exploits the complementary advantages of spectro-polarimetric data, providing a feasible solution to the problem of insufficient utilization of multimodal information in spectro-polarimetric imaging.
• It shows great application potential in remote sensing image interpretation, and can serve as a powerful foundation for various high-level computer vision and remote sensing analysis tasks.

Abstract

Spectro-polarimetric imaging systems can simultaneously acquire spatial, spectral, and polarimetric information during remote sensing, yet the multimodal fusion data are often constrained in practical applications by insufficient exploitation of complementary information across different modalities. To address this issue, we propose a multimodal image fusion method based on a Dual-Branch Cross-Attention Synergistic Transformer (DBCS-T). In our method, three complementary feature components are independently extracted, i.e., a Characteristic Polarization Image (CPI), a Characteristic Spectral Image (CSI) and a Characteristic Intensity Image (CII). For CPI, it is derived from Angle of Linear Polarization (AoLP) and Degree of Linear Polarization (DoLP) inputs via DBCS-T, which integrates a Cross-Channel Transposed Attention (CCTA) module for cross-modal interaction and a Multi-Scale Polarization Feature Adaptive Modulation (MPAM) module for local feature enhancement. CSI is obtained by leveraging maximum-divergence spectral band differences guided by prior spectral radiance curves. CII is computed from the Stokes parameter $S_0$. These three components are then fused via Principal Component Analysis (PCA) into a Multimodal Fusion Image (MFI). To demonstrate the effectiveness of our method, an experiment was conducted on a scene that contains real vegetation, artificial foliage, and same-color metallic objects. The experimental results show that the proposed method achieves effective semantic segmentation of all three target categories. Furthermore, quantitative evaluation demonstrates that the MFI attains the lowest Kullback–Leibler (KL) and Jensen–Shannon (JS) Divergence values among all evaluated modalities, with image entropy exceeding that of individual source inputs. These results validate the complementarity of the extracted multimodal features, significantly enhance the interpretation performance for complex scenes, and demonstrate the broad application potential of the proposed fusion framework in remote sensing and multidimensional imaging.

1. Introduction

Remote sensing monitoring relies on diverse optical detection means to acquire ground object information over large areas, and has become an indispensable technical tool for Earth observation. As a fundamental carrier of multidimensional information, the light field enables the characterization of various physical properties of matter, including spatial distribution, spectral composition and polarization state. The acquisition and analysis of multidimensional information have long been a central focus of optical remote sensing imaging technology [1,2,3,4]. Spectro-polarimetric imaging (SPI), which aims to capture spatial, spectral and polarimetric data simultaneously, has gained increasing importance in optical imaging in recent years [5] and has found wide applications in various fields such as agriculture [6], environmental monitoring [7] and object detection [8]. However, during the analysis of multimodal images of the same scene, current SPI systems do not fully exploit the complementary information available across different modalities. Specifically, rich information in other dimensions is often sacrificed or oversimplified, which in turn leads to information loss in scene analysis. Multimodal image fusion can address this limitation by integrating multidimensional information from different modal images to produce a unified representation. As a result, scene interpretation and visual perception can be enhanced, and thereby downstream computer vision tasks such as semantic segmentation [9,10], image classification [11] and object detection [12,13,14] can be facilitated. Such technical advantages make multimodal fusion highly applicable to large-scale ground observation, vegetation monitoring, object identification and other mainstream remote sensing monitoring scenarios.

The input data for multimodal image fusion algorithms are typically acquired from various types of sensors, including panchromatic, multispectral and polarization cameras. The fused results preserve both complementary and redundant information, thereby overcoming the limitations of individual modalities in representing physical properties and preserving informative content [15]. As a result, multidimensional feature information can be utilized more comprehensively and effectively [16]. To date, fusion algorithms targeting spectral-domain imagery have been extensively studied, with most research focusing on the fusion of visible and near-infrared images [17]. Mo et al. proposed an attribute-guided fusion method for infrared and visible images using edge-preserving filters and Laplacian pyramid, which effectively preserves infrared targets while retaining spatial details from the visible domain [18]. Chen et al. introduced a Transformer-based infrared and visible image fusion network featuring dual-branch feature extractors equipped with contrast residual and texture enhancement modules. By learning cross-feature correlations, their method achieves strong performance both visually and quantitatively [19]. Kang et al. proposed GrFormer, a Grassmann manifold-based attention mechanism that projects features into multi-level subspaces, facilitating complementary interaction between infrared and visible modalities and yielding superior fusion results [20]. Although the aforementioned studies have made notable progress in the field of infrared and visible image fusion, they predominantly focus on information fusion within a single spectral dimension. However, these methods predominantly rely on only two spectral modalities—infrared and visible—without adequately considering the complementarity of other dimensional information, such as polarization and intensity. Moreover, they lack a synergistic mechanism for effectively exploiting information derived from different physical properties. The in-depth exploration of the collaborative fusion involving multidimensional information remains insufficient. In response to this limitation, the fusion of spectral information—such as RGB, infrared, or multispectral data—with polarization information has emerged as a prominent research direction in recent years [21,22]. Shen et al. proposed an attention-based spectro-polarimetric fusion network for target segmentation. By integrating dual-modality features and employing multi-scale enhancement, it achieves high-accuracy real-time segmentation on a self-constructed dataset [23]. Xiao et al. constructed a multimodal dataset comprising visible, near-infrared, long-wave infrared, and linear polarization images using a multi-camera system, thereby supporting research on fusion, super-resolution, and image restoration [24]. Liu et al. proposed DT-F, a dual transposed attention network for polarimetric and panchromatic image fusion. The network incorporates multi-scale feature refinement and a gradient median enhancement loss, significantly improving the quality of polarization fusion [25]. Tong et al. proposed a Transformer-based spectro-polarimetric fusion network that leverages multi-head self-attention for enhanced feature extraction. An information-theoretic layer is introduced to adaptively integrate spectral and polarization data, demonstrating strong robustness in infrared-visible tasks [26]. Although the aforementioned multimodal image fusion methods have achieved notable progress in their respective domains, most of them fail to fully exploit the characteristic spectra and polarization information of different objects, instead focusing primarily on maximizing the information content presented in the fused image. Consequently, while these fused images contain abundant information, they suffer from a severe lack of discriminability among objects with different physical attributes, making them less effective in providing robust data support for subsequent remote sensing monitoring tasks.

To effectively fuse multimodal images while preserving the physical attribute differences for downstream remote sensing monitoring tasks, we propose a multimodal image fusion method based on the Dual-Branch Cross-Attention Synergistic Transformer (DBCS-T). The Characteristic Polarization Image (CPI) extracted by DBCS-T, together with the Characteristic Spectral Image (CSI) derived from a spectral prior-guided band difference and the Characteristic Intensity Image (CII) obtained from the Stokes parameter $S_0$, are fused via Principal Component Analysis (PCA) to produce a pseudo-color Multimodal Fusion Image (MFI). The MFI significantly enhances the discriminability among real vegetation, artificial foliage and same-color metallic objects in the corresponding experimental scenes, demonstrating great potential for remote sensing scene interpretation, including agricultural monitoring and object detection.

The main contributions of this study are as follows:

• We propose a dual-branch Transformer-CNN hybrid network (DBCS-T), which integrates a Cross-Channel Transposed Attention (CCTA) module for cross-modal interaction between Angle of Linear Polarization (AoLP) and Degree of Linear Polarization (DoLP) features and a Multi-Scale Polarization Feature Adaptive Modulation (MPAM) module for local detail enhancement, thereby generating a high-quality CPI.
• We construct a spectral prior-guided differential feature extraction strategy that effectively leverages the physical knowledge of target spectral radiance curves to directly derive the CSI, significantly improving the discrimination capability for metameric targets.
• We establish a PCA-based multimodal feature fusion framework that achieves synergistic utilization and redundancy elimination among polarization, spectral, and intensity information. The PCA-generated pseudo-color Multimodal Fusion Image retains essential information while effectively removing linear redundancy, and its practicality and effectiveness are validated in the corresponding experimental scenarios.

2. Methodology

Spectral information, polarization information, and intensity information represent three distinct dimensions of the light field during propagation, each playing a unique role in computer vision tasks such as target segmentation and object detection. As the foundation of optical imaging systems, intensity information was the earliest to be utilized and provides essential spatial location information of targets. Spectral information, on the other hand, enables the identification of material properties, such as chemical composition, by detecting characteristic reflectance or absorption peaks. However, advances in materials science have led to the development of camouflage coatings inversely engineered based on spectral characteristics, making it increasingly difficult to rely solely on spectral information for effective target detection. Consequently, complementary polarization information is needed to supply detailed surface texture cues to support subsequent discrimination. The effective fusion of spectral information, which characterizes material properties, and polarization information, which characterizes textural details, enables efficient representation of most features of ground objects. By further superimposing spatial information and comprehensively presenting the above information in remote sensing images, different ground objects can be effectively distinguished across different dimensions of information during the multimodal information fusion process.

This paper proposes a multimodal image fusion method aimed at effectively integrating multidimensional information. As illustrated in Figure 1, the proposed method consists of three main components: CPI extraction based on the DBCS-T network, CSI extraction guided by spectral radiance curves, and CII extraction derived from the Stokes vector. These components are subsequently combined via PCA [27] for color space mapping, ultimately generating a MFI.

The detailed steps are shown in Algorithm 1.

Algorithm 1: Multimodal Fusion Method Process

Input:  Polarization_Img $\{I_0,I_{45},I_{90},I_{135}\}$ Spectrum_Img $\{B_{472},G_{532},R_{650},R{E}_{730}\}$. Output:  Fusion_Img $\left\{MFI\right\}$. Step1  Stokes Vector Calculation. $S_0=I_0+I_{45}+I_{90}+I_{135}$, $S_1=I_0-I_{90}$, $S_2=I_{45}-I_{135}$, $\mathrm{DoLP}=\sqrt{{\left(S_1\right)}^2+{\left(S_2\right)}^2}/S_0$, $\mathrm{AoLP}=0.5\times \arctan 2(S_2,S_1)$. Step2  DBCS-T Train and Test. input1: AoLP, DoLP. output1: $\mathrm{CPI}=\mathrm{DBCS}\text{-}\mathrm{T}(\mathrm{AoLP},\mathrm{DoLP})$. Step3  Spectral Prior-guided Difference Strategy. input2: $B_{472},G_{532},R_{650},R{E}_{730}$, Spectral Radiance Curves. output2: $\mathrm{CSI}=\mathrm{MaxNormalize}(R{E}_{730}-G_{532})$. Step4  Replace the Intensity Information with $S_0$. $\mathrm{CII}=S_0$. Step5  Multimodal Fusion Based on PCA. input3: CPI, CSI, CII. output3: $\mathrm{MFI}=\mathrm{PCA}\text{\_}\mathrm{Fusion}(\mathrm{CPI},\mathrm{CSI},\mathrm{CII})$. Step6  Return MFI Step7  Indicator Evaluation. $\mathrm{Index}=\mathrm{Evaluation}(\mathrm{MFI},\mathrm{CPI},\mathrm{CSI})$.

2.1. CPI Extraction Based on the DBCS-T Network

The DBCS-T network adopts an encoder-decoder architecture grounded in the Transformer framework. It incorporates a dual-branch design within the Shallow Feature Extraction Module (SFEM) to process AoLP and DoLP polarization information independently. At the feature level, cross-channel polarimetric interaction is enabled through a cross-attention synergistic fusion mechanism, while end-to-end learning is achieved via multi-stage modulation of polarization features across multiple scales.

The overall architecture of the proposed network is illustrated in Figure 2, which consists of three principal modules: the SFEM, the Transformer module, and the reconstruction module. Specifically, the SFEM employs a dual-branch structure that performs independent convolution operations on the input AoLP and DoLP images, extracting their respective shallow features, denoted as $\overline{A}$ and $\overline{D}$. These two images are then concatenated and passed through convolutional layers and activation functions to generate a shallow fusion feature, denoted as $\overline{P}$, which is subsequently fed into the Transformer module for further enhancement. The Transformer module is composed of multiple cascaded Transformer blocks, each integrating two core components: the CCTA and the MPAM modules. The features $\overline{A}$, $\overline{D}$ and $\overline{P}$ are fed into the Transformer module, where an attention-guided collaborative fusion mechanism is employed to facilitate explicit cross-modal interaction and fusion of polarization features at a deep level. Finally, the reconstruction module is utilized to complete the extraction and generation of the CPI.

The CCTA module is designed to capture cross-channel global dependencies and extract complementary information across channels. Taking the output of the SFEM as input, it employs a dual-branch global attention mechanism to compute the covariance matrix of AoLP and DoLP features over all spatial positions, thereby establishing global spatial correlations. Two transposed attention maps—denoted as $A\text{-}D$ and $D\text{-}A$—are generated to aggregate polarization information from complementary perspectives, enabling thorough exploitation of the complementary characteristics between AoLP and DoLP. This process is formulated as:

$$\begin{array}{cc}\hfill \mathrm{Att}A\text{-}D(\overline{A},\overline{D},\overline{V_1})& =\overline{V_1}·\mathrm{Softmax}(\overline{A}·\overline{D}/\gamma )\hfill \\ \hfill \mathrm{Att}D\text{-}A(\overline{D},\overline{A},\overline{V_2})& =\overline{V_2}·\mathrm{Softmax}(\overline{D}·\overline{A}/\gamma )\hfill \end{array}$$

(1)

where $\overline{A}$ and $\overline{D}$ denote the normalized input image features, $\overline{V_1}$ and $\overline{V_2}$ represent the normalized fused features, and $\gamma$ is a learnable scaling parameter. The two attention maps are applied to query the fused features, yielding distinct feature representations. By adaptively adjusting the attention weights via the learnable parameter $\gamma$, the module achieves effective cross-channel global feature fusion.

The MPAM module enables multi-scale extraction and modulation of polarization features across different spatial scales by emulating the multi-scale pyramid attention mechanism of the human visual system. It employs four parallel branches operating at distinct scales to independently capture hierarchical features. Specifically, 3 × 3 convolutional kernels are used to obtain standard local receptive fields that focus exclusively on neighboring pixels, thereby reinforcing local feature representations. Spatial position responses are dynamically adjusted in an adaptive manner according to varying feature characteristics, with per-pixel independent weights generated to enhance salient features while suppressing redundant information.

DBCS-T employs a dual-branch input architecture to process AoLP and DoLP images separately. The shallow features extracted by the SFEM are passed through multiple cascaded Transformer blocks, enabling stage-wise global information interaction and adaptive modulation of local features. This design enhances the network’s capacity to represent polarization characteristics. Through iterative global-local synergistic fusion, the DBCS-T ultimately generates a CPI that exhibits both global consistency and local fine-grained details.

2.2. CSI Extraction via Spectral Prior-Guided Difference Strategy

With respect to spectral information, conventional multispectral cameras face inherent limitations in distinguishing metameric targets—objects that appear identical in color but exhibit distinct spectral signatures—based solely on characteristic absorption or reflectance peaks. This challenge necessitates the use of hyperspectral or ultraspectral data with higher spectral resolution to provide a more discriminative basis. On the other hand, as an intrinsic property of light field propagation, polarization conveys rich information regarding polarimetric state, intensity, and spatial distribution. These two modalities offer complementary information about the target scene; therefore, their fusion can yield images that provide richer and more accurate representations, enhancing subsequent analysis and interpretation.

As illustrated in Figure 3, the proposed multimodal image fusion method leverages target spectral radiance curves as prior knowledge to guide a difference operation on four-band spectral images—namely the Red (R), Green (G), Blue (B), and Red Edge (RE) channels. This operation effectively separates the background from foreground targets, generating the CSI, which provides spectral information to the final MFI.

2.3. CII Extraction Derived from Stokes Vector

Concurrently, polarization, as an intrinsic property of light field propagation, carries a wealth of information regarding polarimetric state, intensity, and spatial distribution. In the polarimetric domain, based on mutually orthogonal linear polarization measurements, the majority of polarimetric parameters (including the AoLP and DoLP) can be derived through Stokes vector computation. The total intensity parameter $S_0$, acquired from the Stokes vector, provides target image information that closely aligns with the intensity and grayscale distribution of a panchromatic image. Under identical spectral bands and scene conditions, the two exhibit strong visual similarity, rooted in the same physical origin yet not entirely equivalent. In the proposed method, the $S_0$ component is obtained by acquiring linear polarization images at four orientations (0°, 45°, 90°, and 135°) and serves as CII, supplying grayscale distribution, texture features, and contour information of the target scene to the MFI.

Subsequently, the CPI, CSI, and CII are fed as inputs into the PCA algorithm. PCA is an unsupervised orthogonal linear transformation method that maps linearly correlated input features into an orthogonal principal component space. By retaining the highest-variance information in the input features, PCA eliminates linear redundancy between features while minimizing information loss. Since the CPI is generated through feature learning from both AoLP and DoLP polarimetric representations, it exhibits a strong linear correlation with the total intensity $S_0$ and consequently with the CII. Through PCA, redundant information among the three feature inputs is effectively reduced while essential information is preserved. The resulting mutually orthogonal feature projections are then mapped into the color space to generate a pseudo-color MFI.

3. Experiment

3.1. Dataset Preparation

The PIF-dataset, constructed and publicly released by the authors of [25], was captured across diverse indoor and outdoor scenes encompassing various target objects, as shown in Figure 4. It comprises multiple groups of co-registered linear polarization images acquired at four orientations—0°, 45°, 90°, and 135°—with a resolution of 1024 × 1224 pixels. The dataset was preprocessed to compute the AoLP and DoLP images, which were subsequently used as inputs for self-supervised training of the DBCS-T network.

3.2. Loss Function Design

During the training of the DBCS-T, a composite loss function ${\mathcal{L}}_{\mathrm{total}}$ is employed, consisting of three terms: the base loss ${\mathcal{L}}_{\mathrm{base}}$, the reconstruction loss ${\mathcal{L}}_{\mathrm{set}}$, and the gradient loss ${\mathcal{L}}_{\mathrm{grad}}$:

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{base}}+{\mathcal{L}}_{\mathrm{set}}+{\mathcal{L}}_{\mathrm{grad}}$$

(2)

The base loss ensures that the output image maintains both pixel fidelity and structural integrity relative to the source images, preventing significant structural deviation from the original polarization inputs. It is formulated by computing the Structural Similarity Index (SSIM) and Mean Squared Error (MSE) between the output image and each source image, weighted by corresponding coefficients:

$${\mathcal{L}}_{\mathrm{base}}={\mathcal{L}}_{\mathrm{base}1}(P,A)+{\mathcal{L}}_{\mathrm{base}2}(P,D)$$

(3)

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{base}1}(P,A)& ={\alpha }_1·(1-\mathrm{SSIM}(P,A))+{\beta }_1·\mathrm{MSE}(P,A)\hfill \\ \hfill {\mathcal{L}}_{\mathrm{base}2}(P,D)& ={\alpha }_2·(1-\mathrm{SSIM}(P,D))+{\beta }_2·\mathrm{MSE}(P,D)\hfill \end{array}$$

(4)

where ${\mathcal{L}}_{\mathrm{base}1}$ and ${\mathcal{L}}_{\mathrm{base}2}$ denote the base losses computed between the CPI and the AoLP and DoLP source images, respectively, and ${\alpha }_1$, ${\alpha }_2$, ${\beta }_1$, ${\beta }_2$ are weighting parameters that balance the contributions of the SSIM and MSE terms. In this study, greater emphasis is placed on the structural and contour features captured by SSIM; thus, the parameters are empirically set to ${\alpha }_1$ = ${\alpha }_2$ = 0.7 and ${\beta }_1$ = ${\beta }_2$ = 0.3, with further adjustments made based on actual fusion performance.

The reconstruction loss ${\mathcal{L}}_{\mathrm{set}}$ generates gradient masks by computing the gradients of the source images, identifying regions with high gradient magnitude as “gradient-dominant regions”. During training, ${\mathcal{L}}_{\mathrm{set}}$ adaptively preserves these regions, encouraging the output image to approximate the high-gradient details of the source images across different local patches, thereby enriching the local texture detail of the fused result. The gradient loss ${\mathcal{L}}_{\mathrm{grad}}$ constrains the output image to match or surpass the source images in terms of gradient characteristics, enhancing global edge sharpness and textural hierarchy while mitigating detail blurring and degradation that may arise from feature fusion. Together, these three loss terms act synergistically to impose multi-faceted constraints on the output image from the perspectives of fundamental pixel-level structure, local gradient dominance, and global gradient enhancement. This combination preserves the core characteristics of the source images while improving overall detail fidelity. The design of loss function ensures that the fused image retains critical structural information from the source inputs while suppressing artifacts and noise. By explicitly modeling gradient-dominant regions, the reconstruction loss encourages the network to focus on perceptually significant details rather than treating all pixels equally. In practice, these loss terms enable the DBCS-T to produce fusion results that not only exhibit high fidelity to the original polarimetric measurements but also enhance visual interpretability for downstream tasks such as semantic segmentation and object recognition.

3.3. Network Training

The hardware configuration and software environment used for network training are summarized in

Table 1. Network training configuration details.

Category	Details	Configuration
Hardware	CPU	11th Gen Intel Core i7-11700K
	GPU	NVIDIA GeForce RTX 3060 12 GB
Software Environment	Programming Language	Python 3.8
	Network Architecture	Transformer
Training Parameters	Epochs	30
	Learning Rate	0.00005
	Batch size	1

.

Following network training, multiple representative image quality assessment metrics were employed to quantitatively evaluate the polarization information fusion performance of the DBCS-T. Image entropy is used to reflect the richness of information content and the capability of detail preservation in the CPI generated by the DBCS-T. In multimodal fusion, higher entropy indicates successful integration of complementary polarimetric patterns (e.g., edges, textures) from AoLP and DoLP, rather than simple averaging. Peak Signal-to-Noise Ratio (PSNR) is utilized to assess the level of noise introduction and pixel-level distortion in the CPI relative to the input images. A moderate PSNR is acceptable if accompanied by significant gains in entropy, because fusion often trades off noise suppression for detail enhancement; excessively high PSNR may imply over-smoothing and loss of polarimetric nuances. Kullback–Leibler (KL) Divergence [28] and Jensen–Shannon (JS) Divergence are applied to characterize the similarity between the grayscale distributions of the CPI and the input images, thereby reflecting the degree to which statistical properties are inherited. Lower divergence values indicate that the CPI preserves the statistical properties of the inputs without introducing artificial biases, ensuring authentic transfer of relative contrast between different polarization states. Conditional Entropy [29] is employed to quantify the amount of novel information introduced in the CPI with respect to the input images, as well as the extent of their mutual association. Conditional entropy close to the self-entropy of the sources implies that the fused image does not hallucinate new features but efficiently integrates existing complementary information. Multi-Scale Structural Similarity (MS-SSIM) [30] is used to evaluate the preservation of structural and textural information from the input images across multiple scales in the CPI. If significantly different MS-SSIM values are observed for different inputs, it reveals the structural bias of the fusion method. In summary, these metrics characterize fusion performance from complementary perspectives, and their combined interpretation provides a comprehensive understanding of the method’s ability to integrate complementary polarimetric information while maintaining physical consistency.

3.4. Analysis of Metrics for Comparison and Ablation Experiments

As shown in Figure 5, taking a representative image pair from the dataset as an example, the aforementioned fusion image quality assessment metrics were computed for the CPI image fused by DBCS-T. To demonstrate the effectiveness of our method, we adopted the DT-F network as a comparative baseline and conducted ablation studies on each module of the DBCS-T. Among these metrics, image entropy is a no-reference metric, while the remaining metrics are full-reference metrics. For the full-reference metrics, the values corresponding to Source 1 represent the metrics of each fused image relative to the AoLP image, Source 2 represents those relative to the DoLP image, and Average Source represents those relative to the mean of the two source images. Methods 1–3 correspond to three ablation configurations: removing only the CCTA module, removing only the MPAM module, and removing both the CCTA and MPAM modules. DT-F denotes the comparative experiment using AoLP and DoLP as source image inputs to the DT-F network. The quantitative evaluation results of polarization information fusion quality for both the comparative and ablation experiments are presented in

Table 2. Fusion quality evaluation metrics for comparative and ablation experiments.

		Method 1	Method 2	Method 3	DT-F	DBCS-T
Entropy (bit)	Source 1	5.9004	5.9004	5.9004	5.9004	5.9004
	Source 2	5.7813	5.7813	5.7813	5.7813	5.7813
	Average source	6.8293	6.7692	6.8339	6.8369	6.8526
PSNR (dB)	Source 1	24.8928	21.9852	22.8205	20.7187	19.0155
	Source 2	5.3350	5.6551	4.8954	6.0593	7.0368
	Average source	15.1139	13.8201	13.8580	13.3890	13.0262
KL Divergence	Source 1	1.2561	1.4176	1.5213	1.0351	1.9482
	Source 2	2.1707	3.9125	2.8492	2.0901	2.3000
	Average source	1.2067	2.1584	1.6786	1.0559	1.6174
JS Divergence	Source 1	0.2083	0.2054	0.2711	0.1736	0.3255
	Source 2	0.4509	0.5175	0.4787	0.4678	0.4510
	Average source	0.2204	0.2585	0.2772	0.2105	0.2984
Conditional Entropy (bit)	Source 1	4.3672	4.4634	4.4314	4.8442	4.5359
	Source 2	5.5861	5.5701	5.5508	5.5586	5.6028
MS-SSIM	Source 1	0.9531	0.9360	0.9510	0.8878	0.9258
	Source 2	0.3644	0.4013	0.3800	0.4012	0.4302
	Average source	0.5272	0.5618	0.5430	0.5526	0.5874

.

As shown in Table 2, the fused image produced by the DBCS-T network achieves a higher entropy than both the AoLP and DoLP inputs, indicating a measurable enhancement in polarization information richness. This suggests that the DBCS-T effectively fuses the complementary structural and textural attributes of the source images, enriching the hierarchical detail of the fused output. The DBCS-T fused image achieves the lowest KL and JS divergences relative to the input source images. This statistical result confirms that the global distribution and local texture details of the fused result are highly consistent with those of the source images, demonstrating effective information preservation in the “structure-texture” domain. Conditional entropy analysis shows that the values for the DBCS-T fused image relative to each input are comparable to the self-entropy of the inputs themselves. This confirms that the fused output maintains a strong correlation with both inputs, with a marginally higher affinity toward AoLP, while still preserving the distinct information from DoLP. Regarding PSNR, the notable discrepancy between values of the DBCS-T fused image computed against the two distinct inputs suggests a degree of distortion in the fusion process, with the noise and blur characteristics of the result more closely resembling those of the AoLP image. Finally, the MS-SSIM, results reveal an exceptionally high structural similarity between the DBCS-T fused image and AoLP, corroborating the subjective visual assessment. The moderate reduction in MS-SSIM when evaluated against the mean input indicates that while the DBCS-T fused image inherits the primary structural framework from AoLP, the incorporation of fine-grained texture details from both inputs introduces a slight trade-off between global structure fidelity and local detail richness. Overall, despite a slight trade off between global structural fidelity and local detail richness, the quantitative results collectively validate the effectiveness and reliability of the proposed DBCS-T fusion method.

According to the ablation experimental results, Method 1, which retains only the MPAM module, achieves a slight improvement in preserving AoLP features, yet its contribution to enhancing the weak texture information from DoLP remains limited. Method 2, equipped with only the CCTA module, can strengthen multi-scale detail extraction from AoLP and improve image entropy, but still remains insufficient to compensate for the degradation of DoLP features. The baseline backbone without both modules (Method 3) achieves intermediate performance across all metrics, yet exhibits a notable deficiency in preserving the weak polarization features of DoLP, indicating that the CCTA and MPAM modules impose certain mutual constraints during the fusion of different polarization features. The complete DBCS T fusion outperforms all ablation groups in entropy, DoLP-side PSNR, and MS-SSIM, and the removal of either CCTA or MPAM consistently leads to a simultaneous degradation of fusion metrics. These results demonstrate that the CCTA module builds global cross-channel dependencies between AoLP and DoLP via dual transposed attention, fully exploiting their complementary polarization features, while the MPAM module performs hierarchical polarization feature extraction and adaptive weight modulation through a multi-scale pyramid attention mechanism. The two modules are complementary and indispensable, jointly supporting the thorough fusion of polarization characteristics.

In a comparative evaluation against DT-F, in terms of information representation capability, the fusion entropy of DT-F is lower than that of DBCS-T, indicating insufficient ability to exploit complementary features between modalities. For the DoLP polarization degree, which exhibits smooth textures and low signal-to-noise ratio, the PSNR and MS-SSIM of DT-F are significantly inferior to those of DBCS-T, suffering from severe loss of weak features. Meanwhile, DT-F performs weaker than DBCS-T in preserving the structural information of the AoLP polarization angle, and tends to cause suppression of single-modal information. In contrast, our DBCS-T achieves substantial improvement in DoLP features with only a slight compromise in the individual fidelity metric of AoLP, thereby realizing balanced fusion of the two types of polarization information. Consequently, DBCS-T outperforms DT-F in terms of information mining, weak polarization feature preservation, and dual-modal synergistic representation.

In summary, the DBCS-T effectively integrates the complementary polarization information from both AoLP and DoLP input images, demonstrating its capability to generate a high-quality polarimetric component for the subsequent multimodal fusion process.

4. Results and Analysis

As illustrated in Figure 6, using the trained DBCS-T network and the proposed multimodal image fusion method, we acquired and fused multimodal imagery of a scene under natural illumination. The scene comprised three categories of targets: real vegetation, artificial foliage, and same-color metallic objects. The fusion results enabled effective semantic segmentation of the distinct target classes.

4.1. Multimodal Image Acquisition

A HIKROBOT (MV-CS050-10UM, HangZhou, China) industrial camera equipped with a lens (TECHSPEC C VIS-NIR Series # 67-716, Edmund Optics, Barrington, NJ, USA) was employed for target scene acquisition. This camera is based on the Sony (IMX264, Sony Semiconductor Solutions, Tokyo, Japan) monochrome image sensor with a native resolution of 2448 × 2048 pixels. Multispectral and multi-angle polarization images were obtained by replacing narrowband filters or rotating a linear polarizer mounted in front of the lens.

Four narrowband filters with center wavelengths of 472 nm, 532 nm, 650 nm, and 730 nm were selected to cover the R, G, B, and RE spectral bands, with full width at half maximum (FWHM) of 20 nm, 23 nm, 20 nm, and 15 nm, respectively. A linear polarizer (extinction ratio of 500:1) was used for polarization image acquisition, together with a visible-light absorptive neutral density filter with 1% transmittance to attenuate incident energy. This configuration ensured consistent irradiance levels across different imaging modalities, preventing overexposure or underexposure.

Following the acquisition of four-band spectral images and four-angle linear polarization images, the AoLP, DoLP, and $S_0$ images were derived by solving the Stokes vector. Owing to the inherent noise amplification mechanism of the Stokes vector formulation [31,32] and angular positioning errors introduced during polarizer rotation [33,34], the directly computed polarization images exhibited severe salt-and-pepper noise, which substantially degraded subsequent multimodal fusion quality. To address this issue, the Block-Matching and 3D Filtering (BM3D) denoising algorithm [35] was applied as a preprocessing step to the polarization images. This method leverages nonlocal self-similarity and collaborative filtering in a 3D transform domain, achieving an effective balance between noise suppression and detail preservation for polarization imagery. Furthermore, since the multimodal images were not acquired simultaneously, unavoidable pixel displacement occurred during the capture. Feature point matching and image cropping were therefore performed to achieve inter-modal registration. The final preprocessed image dimensions are 2048 × 1920 pixels.

4.2. Multimodal Image Fusion

Based on the trained DBCS-T network described in Section 3.3, the denoised AoLP and DoLP polarization images were fed as inputs to the network, from which the CPI of the target scene was derived, completing the polarization information extraction stage. This polarization information extraction strategy ensures that the fine-grained polarimetric characteristics of different ground objects are faithfully preserved.

Under identical natural illumination conditions, spectral radiance curves were acquired using a fiber optic spectrometer (FX2000, Ideaoptics, Shanghai, China), with a standard whiteboard serving as reference background. The spectral band exhibiting the maximum difference in relative spectral radiance intensity was identified as the spectral-domain prior knowledge for the three target categories—real vegetation, artificial foliage, and same-color metallic objects. The maximum contrast was then obtained by computing the difference between the images corresponding to the characteristic spectral bands. For this target scene, the difference between the RE-band image at a nominal center wavelength of 730 nm and the G-band image at 532 nm was computed, yielding the CSI and completing the spectral information extraction stage. Such a spectral prior-guided difference operation effectively highlights the discriminative spectral signatures of the three target categories, enabling the CSI to capture subtle material differences that are not discernible in conventional multispectral images.

The $S_0$ image, which encodes both intensity and spatial information, was adopted as the CII for the target scene. The CPI, CSI, and CII were then supplied as inputs to the PCA algorithm. Following principal component analysis based on grayscale statistical feature transformation of the three images, the resulting principal components were mapped to the R, G, and B channels of the color space, respectively, producing the final MFI. Semantic segmentation of the distinct target categories was subsequently achieved through pseudo-color rendering with category-specific color assignments. The PCA-based fusion not only eliminates linear redundancy among the three feature components but also assigns optimal weights to each modality in the color space, resulting in an MFI with enhanced visual interpretability. The subsequent pseudo-color rendering with category-specific colors enables intuitive and rapid identification of different target types, which is particularly advantageous for real-time remote sensing monitoring tasks.

4.3. Fusion Result Analysis

4.3.1. Subjective Visual Evaluation

As illustrated in Figure 7, the semantic segmentation performance of different target categories is assessed through chromatic contrast between target types. The same-color metallic character regions serve as indicators of contour structural fidelity, while the vein regions of artificial foliage indicate texture detail preservation. Although the Characteristic Intensity Image (CII) provides indispensable spatial and contour information that is crucial for many computer vision tasks, the proposed multimodal image fusion method primarily aims to integrate complementary spectral and polarization information. Since the CII mainly contributes partial spatial and edge details rather than spectro-polarimetric features, it is not considered as a core component in the subjective and objective analyses of this paper, which focus specifically on evaluating the effectiveness of spectral-polarization fusion. Subjective visual evaluation is conducted across three dimensions—target semantic segmentation, contour structure, and texture detail—for the following images: pre- and post-denoising AoLP and DoLP images, CPI, CSI, and the final MFI.

Regarding target semantic segmentation performance, both the pre- and post-denoising AoLP images are capable of clearly distinguishing same-color metallic objects through chromatic contrast, whereas artificial foliage and real vegetation cannot be effectively differentiated—a performance consistent with the intrinsic characteristics of polarization images. The pre- and post-denoising DoLP images, by contrast, provide limited contrast information for semantic discrimination. The CPI largely inherits the chromatic contrast characteristics of the AoLP image, retaining the ability to effectively distinguish same-color metallic objects. The CSI, on the other hand, effectively separates the background dominated by real vegetation but fails to discriminate between artificial foliage and same-color metallic objects. The final MFI achieves visually compelling discrimination among all three target categories through pseudo-color rendering, demonstrating markedly superior semantic segmentation performance.

With respect to contour structure, the pre-denoising AoLP image exhibits blurred character boundaries accompanied by pronounced surrounding noise, following BM3D denoising, the noise is effectively suppressed and character contours are moderately enhanced. The pre-denoising DoLP image presents relatively well-defined character boundaries; however, post-denoising introduces a degree of boundary blurring. This indicates that while the BM3D algorithm achieves favorable noise suppression for the salt-and-pepper noise characteristic of AoLP images, it inadvertently smooths the character contours in DoLP images. The contour quality of the CPI lies intermediate between that of the AoLP and DoLP images, whereas the CSI preserves character boundary contours with exceptional fidelity. The MFI effectively inherits and even surpasses the contour structural quality of the CSI, exhibiting sharper character boundaries and more effectively suppressed internal noise, confirming that contour structural information is well preserved throughout the fusion process. This superior contour preservation demonstrates that the proposed multimodal fusion method effectively integrates the complementary strengths of polarimetric and spectral modalities, successfully inheriting the crisp edge information from the CSI while suppressing internal noise more thoroughly than any single input. The ability to maintain structural fidelity under natural illumination conditions highlights the method’s robustness and its advantage over conventional fusion approaches that often sacrifice edge sharpness for noise reduction.

In terms of texture detail, the pre-denoising AoLP image renders the venation of artificial foliage with considerable clarity, though the real vegetation overlying it is entirely indiscernible. After denoising, the venation is smoothed and fine texture detail is lost. Neither the pre- nor post-denoising DoLP images convey meaningful texture detail information. The texture detail of the CPI is comparable to that of the post-denoising AoLP image. The CSI, however, faithfully preserves the venation texture of the artificial foliage while also achieving partial discrimination of the overlying real vegetation through enhanced contrast. The MFI exhibits clear and well-defined texture detail, with venation patterns rendered prominently and the overlying real vegetation rendered distinguishable. Such enhancement in texture detail confirms that the proposed fusion framework not only preserves but actively amplifies discriminative textural features from the source images, enabling clear separation of visually similar targets (e.g., artificial foliage versus real vegetation) that remain indistinguishable in any single modality. This advantage originates from the synergistic combination of the spectral prior-guided CSI and the CPI, which together provide richer feature representations than the sum of individual contributions, thereby validating the effectiveness of the multimodal fusion strategy for complex scene interpretation.

In summary, from a subjective visual standpoint, the final MFI not only achieves effective semantic segmentation of all three target categories but also inherits—and in certain respects surpasses—the respective advantages of the source images (CPI and CSI) in terms of contour structural fidelity and texture detail preservation. This demonstrates effective integration and complementary utilization of multimodal information, validating both the effectiveness and practical utility of the proposed multimodal image fusion method.

4.3.2. Objective Quantitative Analysis

Beyond subjective visual evaluation, we conducted an objective quantitative analysis on the pre- and post-denoising AoLP and DoLP images, the CPI, the CSI, and the final MFI using the image fusion quality metrics described in Section 3.3, as summarized in

Table 3. Quantitative metrics for the multimodal fusion image.

		Entropy (bit)	PSNR (dB)	KL Divergence	JS Divergence	Conditional Entropy (bit)	MS-SSIM
DBCS-T Fusion	AoLP	6.8550	10.9263	2.8476	0.3646	6.4445 (5.99%)	0.5123
	DoLP	4.7243	10.7838	5.4926	0.5715	4.3795 (7.30%)	0.6106
	CPI	6.7389	\	3.6110	0.4136	\	0.5571
Multimodal Fusion	CPI	6.7389	13.4823	0.4069	0.1109	6.6194 (1.77%)	0.3367
	CSI	6.7686	13.8214	1.1559	0.2667	6.2682 (7.39%)	0.5006
	MFI	6.7816	\	0.3714	0.0936	\	0.4026

.

As indicated by the metrics in Table 3, the entropy of the MFI exceeds that of both source images (CPI and CSI), confirming that following PCA-based fusion, the fused image not only integrates the overall structural and texture detail characteristics of the source images but also further elevates information density along the polarimetric and spectral dimensions. Such consistency is observed with the subjective visual performance of the MFI. With respect to PSNR, the values computed between the MFI and each source image (CPI and CSI) are higher than those between the CPI and its respective source images (AoLP and DoLP). This suggests that the multimodal fusion algorithm further improves the fidelity of texture details in the fused output relative to the source images, although some structural distortion remains. Regarding KL and JS Divergence, the MFI achieves the lowest values relative to the mean source image among all compared results, demonstrating strong information consistency between the MFI and the source images at both the global structural and local texture levels, with particularly prominent integration of the overall structures of CPI and CSI. With respect to conditional entropy—where the values in parentheses in Table 3 represent the relative deviation from the self-entropy of the corresponding input or source image—the conditional entropy of the CPI relative to the AoLP and DoLP images, as well as that of the MFI relative to the CPI and CSI, are all close in magnitude to the self-entropy of their respective input or source images. This confirms that both the polarization fusion by DBCS-T and the subsequent multimodal fusion process preserve strong correlations between the fused image and its input, with the information from each modality effectively retained. Furthermore, the relative deviation values indicate that the MFI exhibits a closer affinity toward the AoLP image among the polarization inputs, a finding that aligns with subjective visual observation. Regarding MS-SSIM, the MFI exhibits closer structural similarity to the CSI than to the CPI. Nevertheless, the overall structural similarity values are not high, implying that while the multimodal fusion method preserves the core structural features of the CSI, a degree of visual discrepancy with the source images remains. This observation aligns with the findings of the subjective visual evaluation. Collectively, these quantitative results collectively confirm that the proposed fusion framework consistently outperforms single-modal inputs and intermediate products, demonstrating a balanced and robust integration of complementary information. The effectiveness of the cross-attention mechanism and the spectral prior-guided strategy is well reflected in the multidimensional superiority of the fusion output, proving that the method is particularly suitable for complex scenes where conventional approaches struggle to maintain both global coherence and local fidelity.

Taking both the subjective visual evaluation and the quantitative fusion quality metrics into account, the proposed multimodal image fusion method based on the DBCS-T effectively achieves semantic segmentation of three target categories—real vegetation, artificial foliage, and same-color metallic objects—through pseudo-color rendering, enabling clear visual discrimination among different target classes based on the MFI. Moreover, the quantitative analysis of the fusion quality metrics further corroborates that the MFI efficiently integrates the overall structural and texture detail information of the source images CPI and CSI. The strong alignment between subjective visual evaluation and quantitative metrics further validates the practical utility of the proposed method for real-world applications. In the context of remote sensing image monitoring, this fusion framework enables reliable discrimination of spectrally and polarimetrically similar ground objects under varying illumination conditions, providing high-quality input for downstream tasks such as land cover classification, change detection, and target recognition. Its capability to enhance semantic separability while preserving structural details offers a promising technical pathway for intelligent interpretation of multisource remote sensing data.

5. Conclusions

In this paper, we proposed a multimodal image fusion method based on a Dual-Branch Cross-Attention Synergistic Transformer (DBCS-T) to address the underutilization of complementary spectral and polarization information in spectro-polarimetric imaging. The proposed framework extracts three distinct feature components—CPI (polarization), CSI (spectrum), and CII (intensity)—and fuses them via PCA into a pseudo-color MFI. Specifically, the DBCS-T processes AoLP and DoLP images through a dual-branch Transformer-CNN hybrid architecture integrating CCTA and MPAM modules. Quantitative evaluation demonstrates that the resulting CPI achieves higher entropy than the individual AoLP and DoLP inputs, while low KL and JS Divergences confirm strong consistency with the source images. The spectrometer-guided CSI captures discriminative spectral information through band-difference operations, and the $S_0$-based CII provides spatial intensity and contour details. The final MFI exhibits the lowest distributional divergence relative to the source inputs and enables clear semantic segmentation of real vegetation, artificial foliage, and same-color metallic objects under natural illumination. Both subjective visual evaluation and objective quantitative analysis validate that the proposed fusion method effectively integrates multimodal image information, which can provide strong support for subsequent high-level computer vision and remote sensing image interpretation tasks.

Despite these contributions, this work still has certain limitations, including limited scene diversity and insufficient real-time performance. In future work, we will construct a more comprehensive co-registered multispectral and multi-angle polarization image dataset with greater diversity in both target categories and acquisition conditions. Meanwhile, we intend to perform manual annotations on the dataset, so that the performance of the proposed multimodal fusion method in downstream segmentation or detection tasks can be further characterized through precise segmentation metrics. Finally, we will also promote the practical engineering deployment of multimodal fusion algorithms through lightweight network design and integration with edge computing devices.