Cross-Modal Adaptive Interaction Network for RGB-D Saliency Detection

Abstract

The salient object detection (SOD) task aims to automatically detect the most prominent areas observed by the human eye in an image. Since RGB images and depth images contain different information, how to effectively integrate cross-modal features in the RGB-D SOD task remains a major challenge. Therefore, this paper proposes a cross-modal adaptive interaction network (CMANet) for the RGB-D salient object detection task, which consists of a cross-modal feature integration module (CMF) and an adaptive feature fusion module (AFFM). These modules are designed to integrate and enhance multi-scale features from both modalities, improve the effect of integrating cross-modal complementary information of RGB and depth images, enhance feature information, and generate richer and more representative feature maps. Extensive experiments were conducted on four RGB-D datasets to verify the effectiveness of CMANet. Compared with 17 RGB-D SOD methods, our model accurately detects salient regions in images and achieves state-of-the-art performance across four evaluation metrics.

1. Introduction

Salient object detection (SOD) is a fundamental but challenging task in the field of computer vision. It is primarily used to automatically detect the most distinctive areas in images and has been widely applied in various fields such as video segmentation [1], image retrieval [2], video detection [3], visual tracking [4], and others. Saliency detection tasks can be roughly divided into two basic types: one is RGB salient object detection, which uses RGB images as inputs; the other is RGB-D salient object detection, which inputs both images into the network to obtain the results.

In the field of CV, with the development of deep learning and neural networks, RGB salient object detection has made significant progress, and many high-quality models have been proposed [5]. However, due to challenges such as the similarity of image textures and complex backgrounds, the detection performance using only RGB images remains unsatisfactory. Generally speaking, humans can see not only colors, texture, and other information in the scene but also capture depth information through the binocular visual system. This depth information is not affected by complex backgrounds and lighting intensity, and depth images are a form of expressing this depth information. With the rise of depth cameras, we can obtain corresponding depth images at a lower cost. Introducing them to participate in salient object detection tasks can simulate the visual perception abilities of the human eye and enhance the computer’s recognition of targets. This has aroused researchers’ interest in developing the RGB-D SOD method. Many experiments have verified that using RGB-D images for salient object detection tasks can achieve better detection results in many difficult environments [6]. However, how to fully explore the correlation properties between the two modalities and fully utilize their respective advantages to integrate RGB and depth information is still an issue that needs attention [7,8].

As a prediction task at the pixel level, RGB-D salient object detection based on deep learning mostly adopts an encoder–decoder network architecture. The encoder part primarily uses a pre-trained convolutional neural network model to map the input image to the hidden layer to extract multi-level feature representations; then, the decoder is used to restore the image resolution and finally generate the saliency map detection results. Currently, researchers can divide the fusion methods into three types according to the different positions of RGB images and depth images in the network: early fusion, mid-term fusion, and late fusion [9]. As shown in Figure 1, specifically, early fusion occurs at the input stage; for the RGB image and the depth image, the two are only connected at the channel angle, forming a four-channel image as input, followed by subsequent encoder and decoder operations [10]. However, this fusion method does not consider the inherent differences between RGB images and depth images, which may lead to inaccurate detection results. Late fusion refers to using two independent network models to detect the saliency maps of RGB and depth images, respectively, and then simply multiplying the two saliency maps or obtaining their sum as the final result [11]. This method loses a lot of deep information in the image and cannot exploit the intrinsic relationships between RGB images and depth images, often resulting in blurred edges or insufficiently predicted regions in the resulting image [12]. To address these issues, researchers have explored the mid-term fusion strategy. Mid-term fusion mainly investigates how to fuse multi-level features to extract complementary information, thereby achieving a balance between accuracy and efficiency.

Ji et al. [13] proposed ICNet to mine the complementary relationship between two modalities and the coherence between layers, fusing advanced RGB and depth features in an interactive and adaptive manner, distinguishing the information from two different modalities, and using multiple levels of deep features to express the RGB features in more detail. Liu et al. [14] proposed a selective self-mutual attention module (S2MA) to fuse the features learned in the two modalities. Additionally, several methods assist in the task of enhancing RGB features for saliency detection through features extracted from depth images. Zhu et al. [15] processed depth images by using an auxiliary sub-network, encoding the resulting depth image features, concatenating them with the RGB image in the channel direction, and finally obtaining the final saliency map through operations such as upsampling. Chen et al. [16] aimed to create a high-accuracy detection network, so they designed a deep feature extraction network to separately extract depth image information and proposed an alternating correction strategy in the decoding stage, in which depth features are gradually added to the decoding process for prediction. Although the above models have made undeniable progress and improved the accuracy of RGB-D saliency detection tasks, there are still many obvious problems that need to be solved. First, the above methods generally use depth information as an auxiliary cue to refine the RGB features of the SOD task while ignoring the attribute dependence between the two modalities. Furthermore, the above methods mainly extract shared features by fusing the single-modal feature representations of RGB images and depth images, and finally, the final predicted saliency map is obtained through the decoder network. This approach focuses excessively on the fusion of cross-modal information, neglecting the complementary properties between RGB images and depth images at different stages. Moreover, if the RGB-D cross-modal information is simply concatenated, spliced, or added during the fusion process, the truly effective information may be submerged in the redundancy of a large amount of data.

To address the above problems, we propose a cross-modal adaptive interaction network (CMANet). This network follows an encoder–decoder architecture. In the encoder part, we first use the Res2Net-50 [17] network with the PSA polarization attention mechanism [18] to extract multi-scale features from RGB images and depth images. This combination not only retains the multi-scale feature extraction capability of Res2Net-50 [17] but also utilizes PSA to enhance the global contextual information of the features. We propose a cross-modal feature integration module (CMF) to fully integrate the multi-scale features of RGB and depth images. In the decoder part, we employ the U-Net network structure [19], which can directly transfer the high-resolution features from the encoder to the corresponding layers of the decoder, aiding in the restoration of spatial information and details. To fuse feature maps of different scales into comprehensive full-scale feature maps and improve the model’s multi-scale perception ability, we propose an adaptive feature fusion module (AFFM) to integrate these features within the decoder. Based on the above structure, the model can capture global information while retaining the details and boundary information of the image, thereby generating high-quality output results. As shown in Figure 2, this is a comparison chart of our model with other models.

This paper mainly proposes the following contributions:

• A cross-modal adaptive interaction network (CMANet) is proposed to address the RGB-D SOD problem. This network integrates and enhances multi-scale features from both modalities, propagating them to subsequent layers while adaptively preserving crucial information at each level. Simultaneously capturing the global context, it preserves image details and spatial information, thereby yielding more accurate output results;
• We propose a cross-modal feature integration module (CMF), which mainly fuses information from different modalities (such as RGB and depth) to capture the complementary characteristics between the two modalities. Through this module, the model can simultaneously utilize the visual features such as color and texture of RGB images, as well as the spatial structure information of depth images, to generate more discernible and representative feature maps;
• We propose an adaptive feature fusion module (AFFM) that utilizes the intrinsic features of both modalities and adaptively integrates them with the outcomes from the previously shared network layer. This module merges features of different scales into the full-scale feature map, enhancing the model’s ability to perceive multi-scale information;
• Extensive experiments show that our cross-modal adaptive interaction network (CMANet) model outperforms 17 previous state-of-the-art methods on four public RGB-D SOD datasets.

The organization of this paper is as follows: In Section 2, we review two related works, namely RGB salient object detection and RGB-D salient object detection. Subsequently, in Section 3, we provide a detailed description of the proposed cross-modal adaptive interaction network (CMANet). In Section 4, extensive experiments are conducted to verify the effectiveness of CMANet, along with detailed experimental setups and results. Finally, Section 5 concludes this paper and discusses future research directions.

2. Related Work

In this section, two types of related work are reviewed, namely salient object detection based on RGB images and salient object detection based on RGB-D images.

2.1. RGB Salient Object Detection

In the early stages of salient object detection, researchers mainly relied on hand-designed features and heuristics. Itti et al. [20] proposed a biologically inspired saliency detection model in 1998, which uses features such as color, intensity, and orientation of images to calculate saliency maps through multi-scale feature fusion. This method is simple and intuitive and simulates the attention perception of the human eye, but it often performs poorly when processing complex scenes.

As a result, researchers developed image contrast and saliency map generation methods. Achanta et al. [21] proposed a saliency detection method based on global contrast in 2009, which generates saliency maps by calculating the global contrast of each pixel in the image. Cheng et al. [22] proposed the regional contrast method in 2014, which generates saliency maps by segmenting the image into superpixel regions and calculating the contrast between regions. These methods improve the performance of saliency detection to a certain extent but still rely on hand-designed features and heuristic rules.

After entering the era of deep learning, most people choose convolutional neural networks as the main salient target detection method. Deep learning methods use an end-to-end framework to automatically extract high-level features from massive amounts of information, thereby greatly improving the accuracy of saliency detection. Li et al. [23] proposed the DeepSaliency model in 2016, which uses a multi-task deep neural network to learn saliency detection and related tasks simultaneously, thereby improving detection performance. In recent years, researchers have continued to explore more efficient deep learning models and network architectures to further enable salient object detection with higher accuracy. RGB salient object detection continues to make significant progress in model architecture and training strategies. Zhuge et al. [24] proposed an Integrity Consensus Network (ICON) in 2023, aiming to improve integrity learning for salient object detection, which effectively improved the accuracy and robustness of salient object detection at both micro and macro levels. Li et al. [25] proposed a cross-modal coordinate attention network (CCAFusion) in 2024 for the fusion of infrared and visible light images. Through feature awareness fusion and feature enhancement fusion modules, combined with methods based on multi-scale skip connections, effective information fusion is achieved. Xia et al. [26] proposed a hierarchical attention-related context-driven network (RCNet) for salient object detection in 2024. Through multi-receptive field dilated convolutions and multi-source mixed channel attention mechanisms in a diamond hierarchical structure, RCNet effectively improves the accuracy and robustness of feature representation. Although RGB-based salient object detection methods perform well in many applications, they still present challenges when dealing with complex backgrounds, occlusions, and low-contrast targets. Relying only on the information of RGB images, it is difficult to fully distinguish the foreground and background in some cases, thus affecting detection performance.

2.2. RGB-D Salient Object Detection

In order to overcome the limitations of RGB saliency object detection, RGB-D saliency object detection emerged. RGB-D saliency object detection not only utilizes the color and texture information of RGB images but also combines the geometric and structural information provided by depth images, enabling effective detection of salient objects in more complex environments. Depth information can provide additional clues about the three-dimensional structure of the scene, helping to more accurately locate salient objects in the presence of complex backgrounds and occlusions.

Early RGB-D saliency object detection methods mainly focused on how to effectively fuse RGB and depth information. The approach of feeding RGB images and depth images into two separate networks for feature extraction and fusion and then passing the high-level features to a decoder to obtain the final saliency map [27,28] does not fully exploit the unique properties of depth information. In response, Zhao et al. [29] proposed the CPFP model, which introduces contrast enhancement for depth information within the CNN architecture and then integrates the enhanced depth with RGB features to generate the saliency map. However, this approach does not fully utilize the advantages of the dual-stream network and may result in blurred edges in the generated saliency map. To address this, Wang et al. [30] proposed an adaptive fusion scheme that implements efficient detection through the design of a dual-stream convolutional neural network and an adaptive fusion module. Zhang et al. [31] designed the SSF model, which allows the RGB and depth information to complement each other in detail, refining the features and sharpening the boundaries by adding an edge loss function. However, these methods generally perform a simple fusion of the two modalities without fully leveraging the unique properties of modality-specific features. To tackle this issue, researchers began exploring the connections between modality-specific features. Wang et al. [32] proposed a novel deep neural network that learns discriminative cross-modal features, establishing a connection between the modalities of RGB and depth images. Liao et al. [33] introduced the MMNet method, which consists of a multi-stage fusion module and a multi-scale decoder, enhancing RGB information with depth information to improve performance, but it neglects the important spatial features inherent in depth information. Piao et al. [34] combined depth information with multi-scale contextual information and added an attention module to enhance model robustness. Additionally, several models have emerged that deeply explore the connections between modality-specific features, allowing the features of the two modalities to refine each other and enhance the model’s generalization ability [9,35]. However, this approach often results in significant data redundancy, making it crucial to reduce model complexity and make the model more lightweight. Consequently, Ling et al. [36] designed a lightweight model suitable for mobile devices that relies solely on RGB images without utilizing depth information.

Recently, Wei et al. [37] proposed a novel network named EGA-Net aimed at improving edge quality and emphasizing the main features in RGB-D salient object detection. This network innovatively addresses the issues of predicting structurally complete but blurred edges and effectively separating salient objects in complex backgrounds. Chen et al. [38] were the first to propose the use of 3D convolutional neural networks for RGB-D salient object detection, which performs early fusion in the encoder stage and deep fusion in the decoder stage to effectively promote the comprehensive integration of RGB and depth streams. Additionally, Minhyeok et al. [39] proposed a superpixel prototype sampling network (SPSN), which, through the design of a prototype sampling network and a dependency selection module, addresses the challenges in RGB-D salient object detection caused by the significant domain gap between RGB and depth images and the low quality of depth maps.

3. Methodology

In this chapter, we first introduce the overall framework of CMANet in Section 3.1. Section 3.2 and Section 3.3, respectively, discuss the modality-specific learning network and the modality integration network. Finally, Section 3.4 covers the overall loss function.

3.1. Overview

Figure 3 illustrates the overall structure of the cross-modal adaptive interaction network (CMANet). In the encoder part, the RGB and depth images are preprocessed, and then two sub-networks are used to perform feature extraction on the two modalities to obtain their primary features at multiple scales. Within the integrated encoder subnetwork, the CMF module is utilized to learn integrated features. Moving to the decoder section, the U-Net’s [19] skip connection mechanism integrates corresponding multi-level features from the encoder to generate saliency maps. In the integrated decoder subnet, the AFFM module adaptively fuses features from the modality-specific subnetwork and integrated features from the upper layer, enhancing integrated features and enriching cross-modal information. The DeFR module combines feature concatenation and feature enhancement operations from the RFB [40] module. Finally, the saliency maps generated by the integrated decoder subnet serve as the final output.

3.2. Single-Modality Dedicated Network

As depicted in Figure 3, we employ the Res2Net-50 [17] network pre-trained on the ImageNet [41] dataset as the backbone network, enhanced with the PSA polarization attention [18] as the modality-specific encoder network. This network extracts multi-scale features from RGB images and depth images separately. Each image generates five multi-level features denoted as $f_r^i$ and $f_d^i$, where i ∈ {1, 2, 3, 4, 5}. After obtaining the high-level features $f_r^5$ and $f_d^5$ from the RGB and depth inputs, respectively, through the modality-specific encoder network, these features are passed to the corresponding RGB branch and depth branch in the decoder part. Subsequently, we construct the decoder part using the U-Net network [19] structure, where the multi-level features produced by the modality-specific encoder are jump-connected to the corresponding decoder levels. This step aims to further enhance the features at each level. Furthermore, the input features $f_r^5$ and $f_d^5$, along with those combined via skip connections, pass through the DeFR module. This module extracts information across different scales using multi-scale convolution and dilated convolution, integrating this information to enhance the representation of feature maps. This enhancement aids in improving the model’s capability to detect features across various scales and complex patterns. Finally, these features are input into the modal integration network to enhance the accuracy of saliency detection.

3.3. Modal Integration Network

In the modal integration network, the cross-modal features of RGB images and depth images are fused to learn their integrated representations. Using skip connections, the hierarchical features of the encoder and decoder are fused to enhance the integrated features and enrich cross-modal information.

3.3.1. Cross-Modal Feature Integration Module

Because RGB data focus on the expression of color and depth data focus on showing the spatial information of objects, they cannot be simply integrated. Therefore, we propose the CMF module to integrate and enhance cross-modal features. As shown in Figure 4, 1 × 1 convolutions are applied to the two inputs, respectively, reducing the number of channels to half of the original to speed up the model. To reduce the redundancy of single-modal features and learn the complementarity of the two modalities, we first use a cross-refinement enhancement strategy, employing attention mechanisms to automatically select and cross-refine important image features. Specifically, combining the unique characteristics of the two modalities, we apply channel attention and spatial attention to them, respectively, to highlight the characteristic responses of salient regions. This process can be described as

$$f_{out}^r=f_{in}^r⨂\sigma \left(conv1\left(GAP\left(f_{in}^r\right)\right)+conv1\left(GMP\left(f_{in}^r\right)\right)\right)$$

(1)

$$f_{out}^d=f_{in}^d⨂\sigma \left(conv2\left(concat\left(GAP\left(f_{in}^d\right),GAP\left(f_{in}^d\right),dim=channel\right)\right)\right)$$

(2)

Among them, $f_{in}^r$ and $f_{in}^d$ represent the input features of the RGB branch and the depth branch after 1 × 1 convolution, respectively. Global Average Pooling is represented by $\mathrm{G}\mathrm{A}\mathrm{P}$, and Global Maximum Pooling is represented by $\mathrm{G}\mathrm{M}\mathrm{P}$. $convi$ (i = 1, 2) represents the convolution layers, $⨂$ represents element-wise multiplication, $\sigma$ represents the ReLU activation function, $concat$ represents the concatenation operation in the channel dimension, and $f_{out}^r$ and $f_{out}^d$ represent the RGB and depth features after attention selection (i.e., $f_r^i$ and $f_d^i$), respectively.

Inspired by the success of the self-attention mechanism [42], as shown in Figure 5, we designed two submodules, DA and RA, to cross-enhance and refine modal features from a cross-modal perspective. Taking DA as an example, since the depth image can provide spatial and positional information for the RGB image, the features of the depth image are used to generate a weight for the RGB image and multiply it to achieve the purpose of enhancing the features of the RGB image. Specifically, first, use 1 × 1 convolution to map $f_d^i$ to $W_q\in R^{C_1\times \left(H\times W\right)}$ and $W_k\in R^{C_1\times \left(H\times W\right)}$ and map $f_r^i$ to $W_v\in R^{C\times \left(H\times W\right)}$. The number of channels, height, and width of the input feature map are represented by C, H, and W. The value of C1 is set to 1/6 of C to speed up the model operation. The results of these mappings are used to calculate the enhanced features. The attention weight matrix $W_a$ is obtained by multiplying the inputs $W_k$ and $W_k$ and then applying the $softmax$ function. The size of this matrix is (HW) × (HW), which corresponds to the attention weights of each position in the input feature map. Next, the attention weight matrix $W_a$ is multiplied by the projected $W_v$ matrix to obtain the enhanced feature vector $f_{da}$. Finally, the enhanced feature vector $f_{da}$ is transformed back to the same shape as the original input feature map, that is, C × H × W. The process can be described as

$$W_a=softmax\left(W_q^T⨂W_k\right)$$

(3)

$$f_{da}=W_v⨂W_a$$

(4)

The other modules, RA and DA, are complementary and are designed to use RGB image features to generate spatial weights that provide rich texture and color information for depth image features. Additionally, to retain the information of the original modality images, we use the residual connection method to further combine the enhanced features with the original features. The residual connection representation process of the two modalities is as follows:

$$\left\{\begin{array}{c}{f}_R^i=f_r^i+f_r^i⨂sigmoid\left(f_{da}\right)\\ f_D^i=f_d^i+f_d^i⨂sigmoid\left(f_{ra}\right)\end{array}\right.$$

(5)

After obtaining the enhanced features, the most important task is to fuse them together. Specifically, first, $f_R^i$ and $f_D^i$ are convolved with a 3 × 3 convolution to obtain a smooth representation of the features. Then, the two features are fused by element-wise multiplication. Finally, the result is fused with the output $f_S^{i-1}$ from the previous CMF and then subjected to a second 3 × 3 convolution to enhance the network’s representation capability. The output result is the $f_S^i$ of the i-th CMF. The specific process is as follows:

$$f_S^i=Sconv\left(concat\left(Sconv\left(f_R^i\right)⨂Sconv\left(f_D^i\right),f_S^{i-1}\right)\right)$$

(6)

In summary, our CMF module initially employs a cross-refinement enhancement strategy to capture feature dependencies between the two modalities, reducing redundancy in single-modal features while learning their complementarity. Subsequently, it effectively integrates these features and passes them to the next layer to acquire multi-level information.

3.3.2. Adaptive Feature Fusion Module

As illustrated in Figure 6, we introduce an adaptive feature fusion module to dynamically compute weights for different modalities and facilitate effective fusion. Specifically, leveraging the skip connection structure of the U-Net network, we derive modality-specific features $g_r^i$ and $g_d^i$ (i = 1, 2, 3, 4, 5) after integrating skip connections. Notably, when i = 1, $g_r^1$ and $g_d^1$ correspond to the high-level features $f_r^5$ and $f_d^5$ generated by the modality-specific encoder. Similarly, $g_s^1$ represents the high-level feature $f_s^5$ generated by the modality integration encoder. Initially, these three features undergo a 3 × 3 convolutional layer to reduce channel dimensions and smooth the features. Subsequently, we devise an attention weight layer, GSA, which integrates Global Average Pooling (GAP), a 1 × 1 convolutional layer, and sigmoid function to compute weights for the RGB and depth branches, thereby enhancing their respective features. Finally, leveraging features generated by the preceding modality integration network, we further amplify and fuse the features of both modalities. The detailed procedure is outlined as follows:

$$\left\{\begin{array}{c}{g}_{r_1}^i=conv3\left(g_s^{i-1}\right)⨂\left(conv3\left(g_r^i\right)⨂GSA\left(conv3\left(g_r^i\right)\right)\right)\\ g_{d_1}^i=conv3\left(g_s^{i-1}\right)⨂\left(conv3\left(g_d^i\right)⨂GSA\left(conv3\left(g_d^i\right)\right)\right)\end{array}\right.$$

(7)

$$g_s^i=conv3\left(g_s^{i-1}\right)⨁conv3\left(concat\left(g_{r_1}^i,g_{d_1}^i\right)\right)$$

(8)

Among them, $⨂$ represents element-wise multiplication, and $\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}$ represents concatenating two features. After obtaining the fused result $g_{s_1}^i$, a 3 × 3 convolutional layer smooths the feature representation. Subsequently, it is connected with $g_s^{i-1}$ using addition to generate an enhanced feature with rich cross-modal information. This enhances the model’s ability to perceive multi-scale features. It is particularly important to note that the input to all AFFM modules is preprocessed through the RFB module [40] to enhance the model’s perception and detection capabilities of features across different scales and complexities.

3.4. Loss Function

Our loss function consists of three components: SSIMLoss (Structural Similarity Index) [43], BCELoss (Binary Cross-Entropy) [44], and IOULoss (Intersection Over Union) [45]. These components are weighted as 0.8, 1, and 1, respectively, and combined to form $SL$. $F_r$ and $F_d$ denote the saliency maps of the RGB branch and depth branch; $F_s$ and $GT$ represent the outputs generated by the modality integration network and the ground truth saliency maps, respectively. Thus, the loss function is expressed in detail as

$$Loss=SL\left(F_r,GT\right)+SL\left(F_d,GT\right)+SL\left(F_s,GT\right)$$

(9)

4. Experimental Results and Analysis

Section 4.1 explains the specific details of the experiment, and Section 4.2 shows the comparison of the performance of this experiment with other models, including qualitative and quantitative evaluations, as well as the expressiveness of the model when using different backbone networks. Finally, Section 4.3 conducts an ablation study to investigate the importance and effectiveness of each module.

4.1. Experimental Settings

4.1.1. Datasets

We evaluate the experimental results on four datasets, namely NJU2K [46], NLPR [47], SIP [8], and STERE [48]. To obtain universal results, we use 2185 samples as the training set, of which 1485 images are from NJU2K [46] and 700 images are from NLPR [47]. Other images from the NJU2K and NLPR datasets and all samples from SIP and STERE are used as test sets.

4.1.2. Evaluation Metrics

We use four commonly used evaluation metrics to assess the effectiveness of our network, namely F-measure ($F_{\beta }$) [21], Mean Absolute Error ($MAE$) [49], S-measure ($S_{\lambda }$) [50], and E-measure ($E_{\gamma }$) [51]. The detailed definitions of these metrics are as follows:

F-measure ($F_{\beta }$) is a widely used comprehensive evaluation metric that considers both precision and recall. It is defined as follows:

$$F_{\beta }={\displaystyle \frac{{\left(1+\beta \right)}^2\cdot Precision\times Recall}{{\beta }^2\times Precision+Recall}}$$

(10)

Among them, $Precision$ and $Recall$ represent the precision score and recall score, respectively, and ${\beta }^2$ is set to 0.3 to emphasize precision.

Mean Absolute Error ($MAE$) represents the average difference between the saliency map $S$ and the ground truth $Y$, and the detailed formula is as follows:

$$MAE={\displaystyle \frac{1}{W\times H}}\sum _{x=1}^W\sum _{y=1}^H\left|S\left(x,y\right)-Y\left(x,y\right)\right|$$

(11)

where $W$ and $H$ represent the width and height of the saliency map, respectively.

S-measure ($S_{\lambda }$) evaluates the spatial structural similarity between the saliency map $S_r$ and the saliency label map $S_o$, taking into account the saliency of regions and boundaries. It is also known as the structured measure and is defined as follows:

$$S_{\lambda }=\alpha \ast S_o+\left(1-\alpha \right)\ast S_r$$

(12)

where $\alpha$ ∈ [0, 1] is the balance parameter, which is set to 0.5 by default [50].

E-measure ($E_{\gamma }$) is an evaluation metric based on the enhanced alignment mapping between the prediction result map and the salient label map. It is defined as follows:

$$E_{\gamma }={\displaystyle \frac{1}{W\times H}}\sum _{x=1}^W\sum _{y=1}^H\varphi FM\left(x,y\right)$$

(13)

The width and height of the saliency map are denoted by $W$ and $H$, respectively; $\varphi FM\left(\ast \right)$ represents the enhanced alignment matrix [51].

4.1.3. Experimental Details

The model was implemented in the PyTorch library. An L20 graphics card with 100GB memory was used. The backbone network uses the Res2Net-50 network [17] pre-trained on the ImageNet dataset [41] and incorporates the PSA (Polarization Self-Attention) mechanism [18]. We use the Adam algorithm as the optimizer to optimize our network. The initial learning rate is set to 1 × 10⁻⁴, and it decreases to one-tenth of its original value every 60 epochs. During the training phase, we employ random flipping, cropping, and rotation for data augmentation to enhance the robustness of the model. The resolution of the input images is 352 × 352, and the model is trained for 200 epochs. In the testing phase, we first resize the RGB and depth images to a resolution of 352 × 352, then input them into the network to obtain saliency predictions while generating a saliency map of the same size as the input image. The model has 145 MB of parameters, 7 h of training time, and 31 ms of inference time. It should be emphasized that we use the output of the branch from the modality integration network as our final predicted saliency map.

4.2. Performance Comparison

4.2.1. Comparison with RGB-D SOD Models

To verify the effectiveness of the model, we compared CMANet with 17 RGB-D saliency object detection models, namely CTMF₁₈ [27], PCF₁₈ [28], AFNet₁₉ [30], CPFP₁₉ [29], SSF₂₀ [31], CoNet₂₀ [13], D³Net₂₁ [8], DCMF₂₂ [32], DMRA₂₂ [34], MMNet₂₂ [33], CMINet₂₂ [52], DSU₂₂ [53], CFIDNet₂₂ [35], SPNet₂₃ [9], DAL₂₃ [36], EGA-Net₂₃ [37], and RD3D₂₄ [38].

4.2.2. Quantitative Evaluation

Our model is compared with 17 state-of-the-art RGB-D SOD methods. For both NJU2K and NLPR datasets, the four evaluation metrics achieve the best results, as shown in

Table 1. Using the above four metrics, CMANet is compared with 17 models on four RGB-D public datasets. ($S_{\lambda }\uparrow$, $F_{\beta }\uparrow$, $E_{\gamma }\uparrow$, $MAE$). “↑” and “↓” indicate that higher values are better and lower values are better, respectively. The creation year of each model is indicated at the lower right corner of the model name, and the best results are highlighted in bold.

Model	NJU2K				NLPR				SIP				STERE
	${\mathit{S}}_{\mathit{\lambda }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	${\mathit{E}}_{\mathit{\gamma }}\uparrow$	$\mathit{M}\downarrow$	${\mathit{S}}_{\mathit{\lambda }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	${\mathit{E}}_{\mathit{\gamma }}\uparrow$	$\mathit{M}\downarrow$	${\mathit{S}}_{\mathit{\lambda }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	${\mathit{E}}_{\mathit{\gamma }}\uparrow$	$\mathit{M}\downarrow$	${\mathit{S}}_{\mathit{\lambda }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	${\mathit{E}}_{\mathit{\gamma }}\uparrow$	$\mathit{M}\downarrow$
CTMF18 [27]	0.849	0.845	0.913	0.085	0.860	0.825	0.929	0.056	0.716	0.694	0.829	0.139	0.848	0.831	0.912	0.086
PCF18 [28]	0.877	0.872	0.924	0.059	0.874	0.841	0.925	0.044	0.842	0.838	0.901	0.071	0.875	0.860	0.925	0.064
AFNet19 [30]	0.772	0.775	0.853	0.100	0.799	0.771	0.879	0.058	0.720	0.712	0.819	0.118	0.825	0.823	0.887	0.075
CPFP19 [29]	0.878	0.877	0.923	0.053	0.888	0.867	0.932	0.036	0.850	0.851	0.903	0.064	0.879	0.874	0.925	0.051
SSF20 [31]	0.899	0.896	0.935	0.043	0.914	0.896	0.953	0.026	0.876	0.882	0.922	0.052	0.893	0.890	0.936	0.044
CoNet20 [13]	0.895	0.893	0.937	0.046	0.908	0.887	0.945	0.031	0.858	0.867	0.913	0.063	0.908	0.905	0.949	0.040
D3Net21 [8]	0.900	0.900	0.950	0.041	0.912	0.897	0.953	0.030	0.860	0.861	0.909	0.063	0.899	0.891	0.938	0.046
DCMF22 [32]	0.913	0.922	0.925	0.043	0.922	0.914	0.937	0.029	0.875	0.774	0.844	0.115	0.910	0.906	0.914	0.043
DMRA22 [34]	0.886	0.897	0.921	0.051	0.899	0.888	0.942	0.031	0.806	0.852	0.863	0.085	0.886	0.895	0.934	0.047
MMNet22 [33]	0.910	0.919	0.922	0.038	0.925	0.918	0.955	0.023	0.824	0.860	0.871	0.080	0.901	0.910	0.941	0.040
CMINet22 [52]	0.916	0.924	0.945	0.034	0.922	0.919	0.955	0.024	0.892	0.916	0.930	0.043	0.905	0.912	0.946	0.038
DSU22 [53]	0.900	0.909	0.926	0.036	0.919	0.907	0.957	0.022	0.870	0.895	0.918	0.050	0.886	0.895	0.927	0.037
CFIDNet22 [35]	0.914	0.923	0.913	0.038	0.922	0.915	0.950	0.026	0.864	0.891	0.905	0.060	0.901	0.908	0.924	0.043
SPNet23 [9]	0.924	0.934	0.953	0.028	0.927	0.925	0.959	0.021	0.893	0.916	0.929	0.043	0.906	0.914	0.943	0.037
DAL23 [36]	0.904	0.909	0.940	0.044	0.930	0.918	0.966	0.023	0.898	0.898	0.929	0.047	0.902	0.904	0.941	0.040
EGA-Net23 [37]	-	0.883	0.922	0.033	-	0.912	0.966	0.020	-	0.877	0.922	0.048	-	0.870	0.924	0.042
RD3D24 [38]	0.916	0.914	0.947	0.036	0.930	0.920	0.965	0.022	0.885	0.889	0.924	0.048	0.911	0.906	0.947	0.037
CMANet (Ours)	0.925	0.936	0.954	0.028	0.933	0.932	0.966	0.019	0.893	0.920	0.930	0.043	0.907	0.917	0.943	0.036

. Higher values of $S_{\lambda }$, $F_{\beta }$, and $E_{\gamma }$ indicate better performance, while lower $MAE$ values indicate better performance. Notably, our model achieves superior results compared to most RGB-D saliency detection models on the SIP and STERE datasets. For the SIP dataset, the results are similar to those of the RD3D model, and for the STERE dataset, the results are similar to those of the CMINet model. Overall, our CMANet model demonstrates excellent performance in RGB-D saliency object detection tasks, particularly in various indoor and outdoor scenes. Furthermore, we compared the performance of the model using different backbone networks. As shown in

Table 2. This table provides a quantitative analysis of the results obtained when the model uses different backbone networks.

Model	NJU2K				NLPR				SIP				STERE
	${\mathit{S}}_{\mathit{\lambda }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	${\mathit{E}}_{\mathit{\gamma }}\uparrow$	$\mathit{M}\downarrow$	${\mathit{S}}_{\mathit{\lambda }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	${\mathit{E}}_{\mathit{\gamma }}\uparrow$	$\mathit{M}\downarrow$	${\mathit{S}}_{\mathit{\lambda }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	${\mathit{E}}_{\mathit{\gamma }}\uparrow$	$\mathit{M}\downarrow$	${\mathit{S}}_{\mathit{\lambda }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	${\mathit{E}}_{\mathit{\gamma }}\uparrow$	$\mathit{M}\downarrow$
Ours*-Res2Net-101	0.924	0.933	0.951	0.029	0.930	0.929	0.961	0.020	0.896	0.917	0.932	0.041	0.910	0.918	0.944	0.035
Ours-Res2Net-50	0.925	0.936	0.954	0.028	0.933	0.932	0.966	0.019	0.892	0.920	0.930	0.043	0.907	0.917	0.943	0.036

, when using Res2Net-101 as the backbone network, the performance remains outstanding. But, in general, the model has a better detection effect when using Res2Net-50 as the backbone network.

4.2.3. Qualitative Evaluation

We conducted a qualitative evaluation of the proposed model by comparing the generated saliency maps with those of six representative models, covering scenarios such as low scene brightness, cluttered objects, low contrast, high interference, and numerous objects. As shown in Figure 7, the proposed CMANet model can accurately detect salient objects under various challenging conditions. In contrast, other models tend to lose accuracy in such scenarios.

4.2.4. Model Size and Inference Time

We tested the inference time of our model on a NVIDIA GeForce RTX 4090 GPU with 24GB of memory, comparing the model size and inference time against state-of-the-art models (DMRA [34], DSU [53], CFIDNet [35], DAL [36], SPNet [9], EGA-Net [37]). As shown in

Table 3. Comparison of model size and inference time of different methods.

Model	Ours	DMRA22	DSU22	CFIDNet22	DAL23	SPNet23	EGANet23
Model Size (MB)	145	187	96	55	6	174	170
Inference Time (ms)	31	50	48	46	38	84	33

, the difference from lightweight networks like DAL is that our model has a larger size and requires more inference time due to the use of two modality-specific networks and one modality integration network, which separately learn unimodal features and integrated modality features. However, among these models, our model achieved the best inference time while also delivering superior performance across four evaluation metrics. In the future, we aim to improve our model by making it more lightweight, thereby reducing complexity, saving time, and achieving higher efficiency.

4.3. Ablation Studies

We performed ablation studies on different modules of our proposed model to verify their effectiveness.

4.3.1. Effectiveness of the PSA Polarized Attention Mechanism

The PSA polarized attention mechanism enhances the quality of feature representation by modeling spatial and channel attention on feature maps. To evaluate its effectiveness, we removed the PSA attention mechanism and used Res2Net-50 directly, represented as “A” in

Table 4. This table presents the quantitative evaluation of the ablation studies.

Model	NJU2K		NLPR		SIP		STERE
	${\mathit{S}}_{\mathit{\lambda }}\uparrow$	$\mathit{M}\downarrow$	${\mathit{S}}_{\mathit{\lambda }}\uparrow$	$\mathit{M}\downarrow$	${\mathit{S}}_{\mathit{\lambda }}\uparrow$	$\mathit{M}\downarrow$	${\mathit{S}}_{\mathit{\lambda }}\uparrow$	$\mathit{M}\downarrow$
Ours	0.925	0.028	0.933	0.019	0.892	0.043	0.907	0.036
A	0.924	0.028	0.929	0.020	0.883	0.049	0.902	0.040
B1	0.924	0.030	0.933	0.020	0.890	0.043	0.906	0.036
B2	0.924	0.029	0.930	0.022	0.892	0.044	0.905	0.039
B3	0.920	0.029	0.926	0.021	0.889	0.045	0.902	0.040
C1	0.925	0.029	0.931	0.020	0.896	0.042	0.908	0.036
C2	0.923	0.030	0.931	0.020	0.891	0.045	0.908	0.037
C3	0.922	0.030	0.930	0.021	0.953	0.030	0.906	0.037

. Comparing the results, it is evident that the PSA polarized attention mechanism improves the model’s performance. Combining Res2Net-50 with PSA further enhances performance in pixel-level regression tasks.

4.3.2. Effectiveness of the CMF Module

To verify the effectiveness of the CMF module, we first removed it, allowing the modality-specific network results to be directly connected and passed to the next layer, resulting in “B1”. Notably, since the DA and RA sub-modules in the CMF module are responsible for cross-enhancing and refining module features, we deleted these sub-modules while keeping the CMF module intact, resulting in “B2”. Additionally, because the CMF module effectively aggregates the complementary properties of the two modes and transfers this complementary information to subsequent layers, we deleted the connections between CMF modules so the output of one CMF would not pass to the next CMF, resulting in “B3”. The results of “B1”, “B2”, and “B3” are shown in Table 4. The comparison indicates that saliency detection performance declines in these cases, confirming the effectiveness of the DA and RA sub-modules and the importance of connections between CMF modules.

4.3.3. Effectiveness of the AFFM Module

The AFFM module integrates features of different scales and modes, enhancing the model’s ability to understand different modes and scales. Unlike the CMF module, which enhances feature representation and understanding across modalities, the AFFM module uses integrated features to derive enhanced features rich in cross-modal information. To verify the effectiveness of the AFFM module, we deleted it, resulting in “C1”. Additionally, we deleted the attention weight layer GSA within the module to verify its effectiveness, resulting in “C2”. Similarly, we removed the information transfer between AFFM modules to test its necessity, resulting in “C3”. The results of “C1”, “C2”, and “C3” are presented in Table 4. Comparing “C1” shows that the AFFM module improves overall model accuracy. Comparing “C2” and “C3” with the CMANet model demonstrates that the GSA sub-module and the information transfer between AFFM modules are effective and enhance model accuracy.

5. Conclusions

This paper proposes a cross-modal adaptive interaction network (CMANet) to address the RGB-D salient object detection task. Specifically, we introduce a cross-modal feature integration module (CMF) and an adaptive feature fusion module (AFFM), which effectively aggregate the complementary information between the RGB and depth modalities by integrating RGB and depth features. Subsequently, features at different scales are integrated into a full-scale feature map to enhance the model’s multi-scale perception capability. Experiments on four widely used datasets demonstrate that the proposed CMANet achieves state-of-the-art performance compared to 17 other methods. However, there are some limitations, such as the high cost of acquiring and annotating RGB-D data, which restricts the scale and diversity of available datasets. Future work could focus on developing new datasets with increased diversity and annotation quality. Additionally, exploring the xAI-based detection method [54,55] and multi-scale convolution techniques [56] could further optimize the algorithm to improve performance.

We would like to emphasize that the research presented in this paper is conducted with the aim of advancing the field of computer vision and improving technology for positive societal applications. We explicitly disclaim any intent to use or contribute to the use of this technology for harmful purposes, such as automated drones or other forms of autonomous weapons. Our work is guided by ethical principles, and we encourage its use only for beneficial, constructive purposes.