Multiscale Balanced-Attention Interactive Network for Salient Object Detection

Abstract

The purpose of saliency detection is to detect significant regions in the image. Great progress on salient object detection has been made using from deep-learning frameworks. How to effectively extract and integrate multiscale information with different depths is an open problem for salient object detection. In this paper, we propose a processing mechanism based on a balanced attention module and interactive residual module. The mechanism addressed the acquisition of the multiscale features by capturing shallow and deep context information. For effective information fusion, a modified bi-directional propagation strategy was adopted. Finally, we used the fused multiscale information to predict saliency features, which were combined to generate the final saliency maps. The experimental results on five benchmark datasets show that the method is on a par with the state of the art for image saliency datasets, especially on the PASCAL-S datasets, where the MAE reaches 0.092, and on the DUT-OMROM datasets, where the F-measure reaches 0.763.

1. Introduction and Background

Salient object detection (SOD) aims to localize the most visually obvious regions in an image. SOD has been used in many computer-vision tasks, such as image retrieval [1,2], visual tracking [3], scene segmentation [4], object recognition [5], image contrast enhancement [6], assisted medicine [7,8,9], etc. Meanwhile, the specific scenarios of salient object detection in mobile communications applications, such as image background filtering and background atomization in mobile applications, all rely on the high accuracy of foreground target extraction, as shown in Figure 1. Scholars have also proposed many models [10,11,12], but the accurate extraction of salient objects in complex and changeable scenes is still a problem to be solved.

Traditional salient detection methods [13,14,15] used bottom-up computational models and low-level hand-crafted features to predict saliency, such as image contrast [15,16], center prior [17], background prior [18], and texture [19]. Although the artificially designed features can locate some salient areas, it is difficult to improve the accuracy of salient object detection due to the lack of high-level semantic information. In recent years, SOD has made with notable progress with the striking development of deep-learning techniques. Convolutional neural networks (CNNs), which are trained end-to-end, provide more discrimination and distinguishable features, thereby enhancing the performance of the salient object detection algorithm. Even so, the algorithm of SOD based on the deep-learning framework still has two issues of concern. First, the spatial correlation becomes worse due to continuous downsampling for the deeper layer. Second, the potential correlation was ignored due to the self-attention mechanism needed to extract deep-feature information.

The salient object detection algorithm based on the deep-learning framework actually extracts the features by layer [20,21]. The features of the deeper layer are a larger receptive field and stronger representation ability in semantic information. However, the deeper layer, obtained by continuous downsampling, results in the reduction of the resolution of the feature maps, which means that the spatial geometric features lack the ability to represent the detail of the object. On the contrary, the receptive field of the feature of the shallow network is relatively small, but the resolution of the extracted feature map is higher, which makes the representation ability of geometric detail information strong. In fact, the disadvantage of the shallow network is obvious, that is, the representation ability for semantic information is weak. Based on the above analysis, the multiscale features can be obtained from different layers, and aggregating multilevel features is undoubtedly one of the effective methods to solve the first problem mentioned above. Zhao et al. [22] extracted local and global context information from two different scales of super-pixel blocks, and then they used it in the multilevel perception of foreground and background classification. Wang et al. [23] used two CNNs to analyze the local super-pixel block and the global proposal for the salient object detection. Some literatures [24,25,26,27] adopted feature fusion from shallow layers to deep layers. Due to the continuous accumulation of depth features, some detailed features from the shallow layers were lost. Zhang et al. [28] proposed an Amulet network, which integrated multilevel feature maps into multiple resolutions and learned to combine feature maps. Although the positioning accuracy of salient targets has improved, external noise was also mixed with characteristic parameters, which makes it impossible to refine the edge contour of salient objects. Zhang et al. [29] proposed a saliency model that used a bi-directional structure to pass messages between multilevel features. This model implemented information fusion using a gate function to control the message passing rate. However, due to a large amount of noise in the shallow layer, the final predictions were blurred at the edges. In order to better integrate the features of different scales, inspired by the literature [30], we adopted an improved fusion scheme based on a bi-directional propagation model (BPM). Step-by-step fusion in two directions, from shallow to deep and from deep to shallow, can complement the IRM model, which can refine the feature edges and improve the accuracy of salient positioning. Not only are the detail parameters preserved in the shallow features, but also the external noise is suppressed. At the same time, the weight of high-level semantic information is implicitly increased to improve the accuracy of salient object detection.

The attention mechanism can be regarded as a mechanism for redistributing resources based on the importance of activation [31,32]. It plays a vital role in the human visual system, enabling us to find salient objects quickly and accurately from complex scenes. Therefore, the effect of salient object detection based on the attention mechanism is similar to that of the human visual system, which extracts the most salient parts from complex and changeable scenes. In deep learning, the attention method can capture rich semantic information by choosing to weight and aggregate the features of each target region through the contextual information of the feature map. Since the self-attention mechanism [33] can update the feature at each position within a single sequence to capture long-range dependency, some models [4,11,34] use a self-attention mechanism to detect semantic information in depth. However, self-attention has a quadratic complexity of location numbers in the sample and ignores the potential correlations between different samples. Guo et al. [35] proposed an external attention mechanism, based on two external, small, learnable, and shared memories, which uses two cascaded linear layers and two normalized layers to compute the feature maps by calculating the similarity between the query vector and the external learnable key memory. The feature map is multiplied by another external learnable value memory to refine the feature map. External attention has linear complexity and implicitly considers the correlations between all samples. Combining the advantages of the self-attention mechanism and the beyond-attention mechanism, this paper proposes a balanced attention mechanism (BAM). Firstly, the BAM inherits from self-attention mode and calculates the affinity between features at each position within a single sample. Then, with the help of the external attention mechanism, the whole data set is shared through linear and normalization operations, which improves the potential correlation between different samples and reduces the computational complexity.

Pang et al. [12] proposed a self-interactive module (SIM) to extract features from the shallow and middle layers so that the multiscale information can be adaptively extracted from the data. The multiscale information was used to deal with the scale changes of salient objects, which effectively meets the multiscale requirements of SOD tasks. Inspired by reference [12], the interactive residual model (IRM) was designed in this paper. IRM models learned the multiscale features of a single convolution block in two different resolutions, interactively. To extract richer features and reduce redundant information, IRM removed the original feedback mechanism of SIM, optimized the internal sampling structure, and added a dropout block to prevent overfitting. For the deep semantic information, we used the BAM to extract the features in space and channels. The spatial attention module computed the affinity between any two feature parameters in the spatial position. For the channel information, balanced attention was also used to calculate the affinities between any two-channel maps. Finally, two attention modules were fused together to further enhance the feature representation. The complete flowchart is shown in Figure 2.

A multiscale balanced-aware interactive network (MBINet) is proposed for salient object detection. Our contributions are summarized as three items:

• An interactive residual model (IRM) was designed to capture the semantic information of multiscale features. The IRM can extract multiscale information adaptively from the samples and can deal with the scale changes better.
• We proposed a balanced-attention model (BAM), which not only captures the dependence between different features of a single sample, but considers the potential correlation between different samples, which improves the generalization ability of attention mechanism.
• To effectively fuse the output of IRMs and BAM cascade structure, an improved bi-directional propagation strategy was adopted, which can fully capture contextual in-formation of different scales, thereby improving detection performance.

The rest of this paper is organized as follows. Section 2 discusses the improved MBINet algorithm. Section 3 shows the simulation experimental results, and Section 4 concludes the paper.

2. Proposed Method

In this section, we first introduce the overall framework of the MBINet we have proposed. Figure 3 shows the network architecture. Next, we introduce the principle of the interactive residual model and formula derivation in detail in Section 2.2. In Section 2.3, we describe how to derive the balanced attention model step-by-step and the implementation details. Finally, we effectively merge all the features together by introducing the BPM to reflect the multiscale feature fusion further.

2.1. Network Architecture

In our model, the VGG-16 [36] network is used as the pre-training backbone network. Similarly to other SOD methods, we removed all fully connected layers and the last pooling layer, and marked the side outputs of different scales as {Conv1, Conv2, Conv3, Conv4, Conv5}. Since the receptive field of Conv1 was too small and there was too much noise, we decided to only use the side output of Conv2–5 for feature extraction. First, we used the dilated convolution (DC) with the dilation rate of {1, 2, 4} to extract the features of the side output, denoted as C = {C1, C2, C3, C4}, and send the output result to the interactive residual networks (IRMs) model. The characteristic of DC is to expand the receptive field with a fixed-size convolution kernel. The larger the receptive field, the richer the semantic information captured. The purpose of introducing DC is to extract features of different scales from the same feature map. However, IRMs use fixed-size convolution kernels to extract feature maps of different scales. Through complementary learning of DC and IRMs, more effective multiscale features can be obtained. In addition, we added a residual structure (RES) after Conv5, as shown in Figure 4, in order to reduce channel numbers of in-depth features and to pass the output features to a BAM based on both spatial and channel directions. The in-depth information output by BAM was fused together, and the output of IRMs was used as input to the bidirectional propagation network [4]. A prediction graph was generated at each fusion node. Next, we will introduce these models in detail.

2.2. Interactive Residual Model

The side output of different depths contains different features of information. We use DC to extract the features from the side output of the encoder, and the output result is expressed as $F_d$. The function of DC is to expand the receptive field, which can improve the representation ability of small features and ensure the integrity of feature extraction. However, while improving the integrity of the features, the noise is also preserved. In order to further refine the features and complement DC, we introduce an interactive residual model (IRM), as shown in Figure 5. IRM divides the input features into two in a pooling manner, denoted as $f_{l1}$ and $f_{h1}$, where $f_{h1}=F_d$. To extract the features of two resolutions in parallel, and fuse the two output results with their respective sizes, denoted as $f_{l2}$ and $f_{h2}$, the expressions are as follows:

$$\left\{\begin{array}{l}{f}_{h2}^i=DownC(Up(f_{l1}^i)+F_d^i)\\ f_{l1}^i=DownC(Down(F_d^i))\\ f_{l2}^i=Up(DownC(DOwn(F_d^i)+f_{l1}^i))\end{array}\right.,$$

(1)

where $i\in \{1,2,3,4\}$ represents the level of the side output depth, $Up(\cdot )$ and $Down(\cdot )$ represent the de-convolution and pooling operations on the feature, and $DownC(\cdot )$ represents channel numbers for reducing the feature. Channel numbers of $f_{l2}$ is 1/4 of the number of input channels, and the channel numbers of $f_{h2}$ are the same as the number of input channels. To keep the input and output consistent and have the same size as the input, we upsampled $f_{l2}$, downchanneled $f_{h2}$, and merged the results of the two branches, denoted as $F_I$. In order to facilitate optimization, each feature size change underwent normalization and nonlinear processing, and the input feature $F_d$ was processed again, and the $F_d$ was changed directly to the output port of IRMs after changing the channel. The whole process can be expressed by the following formula:

$$F_I^i=Up(f_{l2}^i)+DownC(f_{h2}^i)+DownC(F_d^i),$$

(2)

2.3. Balanced-Attention Model

The attention mechanism is widely used in computer vision, and it is also popular in salient detection. Some attention models [35,37,38,39,40] have also been proposed in recent years. Self-attention is used to compute similarity between local features in order to obtain large-scale dependence. Specifically, the input features are linearly projected into a query matrix $Q\in R^{N\times d^{\prime }}$, a key matrix $K\in R^{N\times d^{\prime }}$, and a value matrix $V\in R^{N\times d}$ [33]. The self-attention model can be formulated as:

$$F_O=V\cdot \mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(Q\cdot K^T),$$

(3)

$$\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(a_{i,j})=\frac{\exp (a_{i,j})}{{\displaystyle {\sum }_{i=1}^N\exp (a_{i,j})}},$$

(4)

where $N$ represents the number of pixels, $a_{i,j}$ is the corresponding pixel value, $d$ and $d^{\prime }$ represent the number of feature dimensions, and $F_O$ represents the output. Obviously, the high computational complexity of self-attention is an obvious disadvantage. Based on this, the beyond-attention model uses two storage $M_k$ units and $M_v$ to substitute the three self-attention matrices $Q$, $K$, and $V$, for calculating the correlation between feature pixels and external storage unit $M$, and using double-normalization to complete the normalization of features. Since $M$ is a learnable parameter, affected by the whole datasets, it acts as a medium for the association of all features and is the memory of the whole training datasets. Double-normalization is essentially the nesting of softmax, at the expense of computational cost. Although features can be further refined and the impact of noise can be reduced, the effect is not obvious for deep features that have been processed multiple times. To this end, combining the advantages of the two attention models above, we proposed a model named the balanced-attention model (BAM). The details of the BAM are shown in Figure 6.

First of all, we extracted the deep features in the spatial position, and transformed the input feature $F\in R^{C\times H\times W}$ projection into a query matrix $Q\in R^{N\times C/8}$ and a key matrix $K\in R^{N\times C/8}$, where $N=H\times W$ is the number of pixels, then multiplied the transpose matrix of $Q$ and $K$, and calculated the resulting matrix using the soft max function. The result is the spatial correlation between different pixels. Then a linear layer was used as the memory $M_v$ to share the whole datasets, and the input was fused into the above operation results to get the final output $F_P\in R^{C\times H\times W}$. The whole process can be formulated as:

$$F_P=M_v\cdot \mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(Q\cdot K^T)+F,$$

(5)

where $F_P$ represents the output feature in the spatial direction.

While paying attention to the affinity relationship between different location features, we also noticed that the interdependence between channel mappings would affect the feature representation of specific semantics. Therefore, we also used the attention mechanism in the channel direction to improve the interdependence between channels. First of all, the input feature $F$ was transformed into $\tilde{F}\in R^{C\times N}$, and then the multiplication operation was performed between $\tilde{F}$ and its own transpose matrix, and the resulting square matrix was sent to the softmax classification function to calculate the pixel correlation degree in the channel direction. Then a linear layer was used as the memory to transform the result to $R^{C\times H\times W}$, and the final result is represented by $F_C$. The whole process can be expressed by the following formula:

$$F_C=\tilde{M}\cdot \mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(\tilde{F}\cdot \widetilde{F^T})+F,$$

(6)

2.4. Model Interaction and Integration

In order to strengthen the interdependence between different features, we fused the output results in the space and channel direction together and denoted it as $F_A$. In addition, the BAM was regarded as an independent part, and the detected semantic information and the output characteristics of the IRMs were fused in a propagation mode. First, we changed the number of output feature channels for BAM and IRMs to 21 and produced two results of the same dimension simultaneously. Then, one of the results of each module was compiled into a group for cascade fusion in the direction from shallow to deep, and the other group was cascaded and fused in the direction from deep to shallow, and the output obtained after each level was fused as the predicted value. The above process is referred to as the bidirectional propagation model (BPM), as shown in BPM in Figure 3. Take $F_I$ as the IRMs output layer and $F_A$ as the BAM output layer. The features are gradually superimposed in two directions from shallow to deep and from deep to shallow, and the output of each side is used as the final prediction result. The whole process can be expressed by the following formula:

$$\left\{\begin{array}{l}{F}_{O(s2d)}^i=fuse(F_I^i,F_I^{i+2},F_A^{s2d}),\hspace{1em}i=1,3,5,7\\ F_{O(d2s)}^i=fuse(F_I^{i-2},F_I^i,F_A^{d2s}),\hspace{1em}i=4,6,8,10\\ F_A=fuse(F_P,F_C,F)\\ F_O=Cat.(F_{O(s2d)}^i,F_{O(d2s)}^i,F_A)\end{array}\right.,$$

(7)

Among them, $fues(\cdot )$ is the feature fusion operation, $Cat.(\cdot )$ is the feature stitching operation according to the channel direction, $F_A^{s2d}$ and $F_A^{d2s}$ are the results of $F_A$ after the reduced channel, $F_{O(s2d)}$ is the output of each level from shallow to deep, and $F_{O(d2s)}$ is similar. Finally, we used the standard binary cross-entropy to train the predicted value; the expression is as follows:

$$L=-{\displaystyle \sum _{x,y}[G_{x,y}\log (P_{x,y})+(1-G_{x,y})\log (1-P_{x,y})]},$$

(8)

where $G_{x,y}\in \{0,1\}$ represents the true value of pixel $(x,y)$, and $P_{x,y}$ represents the predicted value of pixel $(x,y)$ training. The calculation process is as follows:

$$P=Sigmoid(Conv(F_O)),$$

(9)

3. Experiment

3.1. Experimental Setup

Datasets: We evaluated the proposed model on six benchmark datasets: DUT-OMRON [19], DUTS [41], ECSSD [42], PASCAL-S [43], HKU-IS [44], and SOD [45]. DUTS contains 10,553 training images and 5019 testing images, of which the images used for training are called DUTS-TR and the images used for testing are called DUTS-TE, and both contain complex scenes. ECSSD contains 1000 semantic images with various complex scenes. PASCAL-S contains 850 images with cluttered backgrounds and complex salient regions, which are from the verification set of the PASCAL VOC 2010 segmentation datasets. HKU-IS contains 4447 images with high-quality annotations, and has multiple unconnected salient objects in many images. DUT-OMRON contains 5168 challenging images. Most of the images in these datasets contain one or more salient objects with complex backgrounds. The SOD datasets have 300 very challenging images, most of which contain multiple objects with low contrast or connection with image boundaries.

Evaluation criteria: We used three metrics to evaluate the performance of the proposed model MBINet and other state-of-the-art SOD algorithms. The three metrics were $F_{\beta }$-measure [46], mean absolute error (MAE) [15], and S-measure [47]. The $F_{\beta }$-measure was computed by the weighted harmonic mean of precision value and recall value, and its expression is:

$$F_{\beta }=\frac{(1+{\beta }^2)Precision\times Recall}{{\beta }^2\times Precision+Recall},$$

(10)

where ${\beta }^2$ is generally set to 0.3 for weight precision. Since the precision value and recall value are calculated on the basis of a binary image, we first needed to threshold the prediction map to a binary image. Threshold calculation is to combine multiple thresholds to calculate the appropriate value adaptively. Different thresholds corresponded to different $F_{\beta }$ scores. Here we reported the maximum $F_{\beta }$ score of all thresholds. MAE is calculated to measure the average difference between the predicted map $P$ and the ground-truth map $G\in \{0,1\}$. The expression of MAE is:

$$MAE=\frac{1}{W\times H}{\displaystyle \sum _{x=1}^W{\displaystyle \sum _{y=1}^H|P(x,y)-G(x,y)|}},$$

(11)

where W and H are the width and height of the image respectively. S-measure calculates the structural similarity of object perception $S_o$ and region perception $S_r$ between the predicted map and the ground-truth map, and the expression is:

$$S=\gamma \cdot S_o+(1-\gamma )\cdot S_r,$$

(12)

where $\gamma$ is set to 0.5 [47].

Implementation details: We implemented our proposed model based on the Pytorch framework. To facilitate comparison with other works, we chose VGG-16 [36] as the backbone network. Following most existing methods, we used the DUTS-TR datasets as a training set, the SOD datasets as a validation set to update best weights, the DUTS-TE, ECSSD, PASCAL-S, HKU-IS, and DUT-OMRON as testing sets, and the DUTS-TE datasets as the benchmark for ablation experiments. Our network was based on the NVIDIA GTX 1080 Ti GPU, and the operating system was trained on Ubuntu 16.04. To ensure that the model converges, we used the Adam [48] optimizer to train our model. The initial learning rate was set to 1e-4, the batch size was set to 8, and the resolution was adjusted to 320 × 320. The training process of our model took about 18 h and converged after 22 epochs.

3.2. Ablation Studies

The proposed model is composed of three parts: IRM, BAM, and BPM. In this section, we conducted ablation studies to verify the effectiveness of each module combination. The experimental setup follows

Table 1. Ablation analysis on the DUTS-TE datasets. BAMp is the BAM submodule at the position, BAMc is the BAM submodule at the channel, and BAM means BAMp+BAMc.

No.	Model	F-Measure↑	MAE↓	S-Measure↑
a	VGG	0.760	0.074	0.790
b	VGG + IRM	0.782	0.065	0.803
c	VGG + BAMp	0.775	0.066	0.804
d	VGG + BAMc	0.763	0.069	0.801
e	VGG + BAMp + BAMc	0.781	0.066	0.806
f	VGG + BPM	0.777	0.068	0.801
g	VGG + IRM + BAM	0.800	0.061	0.820
h	VGG + IRM + BPM	0.789	0.063	0.809
i	VGG + BAM + BPM	0.785	0.060	0.822
j	VGG + IRM + BAM + BPM	0.809	0.058	0.824

, and we mainly report the results using the DUTS-TE datasets.

Effectiveness of the BAM: BAM combines the advantages of the self-attention and the beyond-attention models. These attention models are embedded into our model, and compared on the DUTS-TE datasets, and the results are recorded in

Table 2. The different models embedded and measured on the DUTS-TE datasets.

No.	Model	F-Measure↑	MAE↓	S-Measure↑
a	+Self-attention	0.804	0.060	0.824
b	+Beyond-attention	0.801	0.059	0.818
c	+BAM	0.809	0.058	0.824

.

We found that the performance of BAM was better than that of the self-attention and the external attention models alone. Especially in terms of MAE, which increased by 1.72% and 3.45% compared with the beyond-attention and self-attention models respectively, and the visual effects are shown in Figure 7. We can see that BAM can capture richer contextual information.

Effectiveness of the IRM and BPM: IRM mainly processes the features of the shallow layer and the middle layer in order to further refine the features extracted by DC. BPM is a feature fusion mechanism between IRM and BAM. In Table 1, different combinations of the three models are shown. From Table 1, it is obviously that the integration of these independent modules can complement each other.

For a more comprehensive analysis of IRM, we further studied the impact of the number of dilate convolutions and the dilation rate. First of all, we tested the number of convolutions $k$ when the dilation rate is {1, 2, 4}. We conducted five different $k$ values experiments, and recorded the test results in

Table 3. Evaluate convolutions on DUTS-TE. k represents the number of dilate convolutions.

$\mathit{k}$	F-Measure↑	MAE↓	S-Measure↑
1	0.7534	0.0742	0.7860
2	0.7517	0.0756	0.7859
3	0.7634	0.0711	0.7927
4	0.7578	0.0741	0.7899
5	0.7588	0.0715	0.7881

. It can be found from Table 3 that $k$ = 3 has the best performance. Next, in order to test the influence of the dilation rate, we chose five different sets of dilation rates: {1,1,1}, {1,2,3}, {1,3,5}, {1,2,4} and {1,4,7}, and recorded the test results in

Table 4. Evaluation of different dilation rates with the DUTS-TE datasets.

No.	F-Measure↑	MAE↓	S-Measure↑
{1,1,1}	0.7634	0.0711	0.7927
{1,2,3}	0.7658	0.0710	0.7988
{1,3,5}	0.7683	0.0694	0.7971
{1,2,4}	0.7768	0.0678	0.8014
{1,4,7}	0.7513	0.0760	0.7848

. We found that the performance is the best when the dilation rate is {1,2,4}. Therefore, we finally determined the dilation rate of {1, 2, 4}.

3.3. Comparison with State-of-the-Art

In this section, we compare our proposed module with the state-of-the-art salient detection methods, including RFCN [10], MCDL [22], LEGS [23], Amulet [28], CANet [34], MDF [44], RSD [49], DSS [50], NLDF [51], DCL [52], BSCA [53], ELD [54], and UCF [55]. For fair comparison, the salient maps of the above methods are provided by the author or calculated by running the source code.

Quantitative comparison: We evaluated the proposed method from the three aspects of $F_{\beta }$-measure, MAE, and S-measure, and also compared it with other SOD methods, as shown in

Table 5. Quantitative evaluation. Comparison of different salient detection methods (the best three results are marked as red, green, and blue respectively).

Model	DUTS-TE			ECSSD			PASCAL-S			HKU-IS			DUT-OMROM
	F↑	M↓	S↑	F↑	M↓	S↑	F↑	M↓	S↑	F↑	M↓	S↑	F↑	M↓	S↑
Ours	0.809	0.058	0.824	0.909	0.058	0.886	0.821	0.092	0.806	0.901	0.042	0.880	0.763	0.069	0.781
CANet [34]	0.796	0.056	0.840	0.907	0.049	0.898	0.832	0.120	0.790	0.897	0.040	0.895	0.719	0.071	0.795
NLDF [51]	0.813	0.065	0.816	0.905	0.063	0.875	0.822	0.098	0.805	0.902	0.048	0.878	0.753	0.080	0.771
Amulet [28]	0.773	0.075	0.796	0.911	0.062	0.849	0.862	0.092	0.820	0.889	0.052	0.886	0.737	0.083	0.771
DCL [52]	0.782	0.088	0.795	0.891	0.088	0.863	0.804	0.124	0.791	0.885	0.072	0.861	0.739	0.097	0.764
UCF [55]	0.771	0.116	0.777	0.908	0.080	0.884	0.820	0.127	0.806	0.888	0.073	0.874	0.735	0.131	0.748
DSS [50]	0.813	0.065	0.812	0.906	0.064	0.882	0.821	0.101	0.796	0.900	0.050	0.878	0.760	0.074	0.765
ELD [54]	0.747	0.092	0.749	0.865	0.082	0.839	0.772	0.122	0.757	0.843	0.072	0.823	0.738	0.093	0.743
RFCN [10]	0.784	0.091	0.791	0.898	0.097	0.860	0.827	0.118	0.793	0.895	0.079	0.859	0.747	0.095	0.774
BSCA [53]	0.597	0.197	0.630	0.758	0.183	0.725	0.666	0.224	0.633	0.723	0.174	0.700	0.616	0.191	0.652
MDF [44]	0.729	0.093	0.732	0.832	0.105	0.776	0.763	0.143	0.694	0.860	.0129	0.810	0.694	0.092	0.720
RSD [49]	0.757	0.161	0.724	0.845	0.173	0.788	0.864	0.155	0.805	0.843	0.156	0.787	0.633	0.178	0.644
LEGS [23]	0.655	0.138	-	0.827	0.118	0.787	0.756	0.157	0.682	0.770	0.118	-	0.669	0.133	-
MCDL [22]	0.461	0.276	0.545	0.837	0.101	0.803	0.741	0.143	0.721	0.808	0.092	0.786	0.701	0.089	0.752

. It can be seen from the results that our method is significantly better than other salient detection methods. In particular, in terms of MAE, the best scores are obtained on all five datasets, and the performance is improved on average compared to the second-best method.

Qualitative evaluation: Figure 8 shows some representative examples. These examples reflect different scenes, including small objects, complex scenes, and images with low contrast between foreground and background. It can be seen from the figure that the proposed method can predict the foreground area more accurately and completely.

4. Conclusions

In this paper, we proposed a novel multiscale balanced-attention interactive perception network for salient object detection. First, we used dilated convolutions to extract multiscale features on the side output of the encoder. Next, interactive residual modules (IRMs) were designed to further refine the edge information of the multiscale features. Thus, the features, extracted by the interactive residual network and the dilated convolutions module, were complementary to each other, and the noise was suppressed. In addition, we proposed a balanced-attention model (BAM), which captured the deep context information of the objects in both spatial and channel directions respectively. The ablation experiments showed that the BAM’s and IRM’s cascade structures could extract richer semantic information for different scales features. Finally, in order to better describe and accurately locate the predicted object, we adopted the improved bi-directional propagation module (BPM) to improve the interdependence between different features. Whether IRM and BPM modules were cascaded for testing, or BAM and BPM modules were cascaded for testing, the results showed that the bi-directional propagation module could more effectively integrate multiscale features. In conclusion, the experimental evaluation on five datasets demonstrated that the designed method could predict the saliency map more accurately than the existing partial saliency detection methods under different evaluation metrics. In further research, we will plan to maintain the existing strengths of our method while considering the challenging problem of model lightweighting. In the future, we will try to further optimize our solution to achieve better predictive performance.