Self-Improved Learning for Salient Object Detection

Abstract

Salient Object Detection (SOD) aims at identifying the most visually distinctive objects in a scene. However, learning a mapping directly from a raw image to its corresponding saliency map is still challenging. First, the binary annotations of SOD impede the model from learning the mapping smoothly. Second, the annotator’s preference introduces noisy labeling in the SOD datasets. Motivated by these, we propose a novel learning framework which consists of the Self-Improvement Training (SIT) strategy and the Augmentation-based Consistent Learning (ACL) scheme. SIT aims at reducing the learning difficulty, which provides smooth labels and improves the SOD model in a momentum-updating manner. Meanwhile, ACL focuses on improving the robustness of models by regularizing the consistency between raw images and their corresponding augmented images. Extensive experiments on five challenging benchmark datasets demonstrate that the proposed framework can play a plug-and-play role in various existing state-of-the-art SOD methods and improve their performances on multiple benchmarks without any architecture modification.

1. Introduction

Salient Object Detection (SOD) refers to the problem of identifying the most visually distinctive objects. SOD methods typically learn a mapping from an image to a saliency mask, which benefits various downstream tasks such as object tracking [1], image captioning [2], video analysis [3], person re-identification [4], image retrieval [5], semantic segmentation [6], and object detection [7].

Despite the efforts long devoted to it, SOD remains a challenging task. One difficulty lies in the fact that the annotations are binary, i.e., each pixel is labeled either salient or non-salient, which does not fully capture the nuanced nature of saliency. For example, the four giraffes in Figure 1a are labeled as equally salient regardless of their different sizes, locations, and depths in the image. Furthermore, the binary annotations pose challenges in learning the mapping from images to saliency masks. The abrupt transition between salient and non-salient regions can make the learning process less smooth and hinder the ability of SOD models to generalize well to unseen data.

In this regard, we propose a novel Self-Improvement Training (SIT) strategy to reduce the learning difficulty incurred by the binary annotations. We design a progressively updated module to generate smooth labels, providing flexible degrees of saliency for each object to represent the difference of saliency between objects. The smooth labels provide more nuanced information about the saliency levels within an image, allowing the model to better understand the varying degrees of importance assigned to different objects. The use of smooth labels together with the binary annotations smoothens the learning process. However, it is worth noting that introducing smooth labels can also introduce noise into the training process. To this end, we mitigate the noise brought by the smooth labels by adapting the weights of smooth labels in the optimization phase.

Another difficulty is that the annotations are subjective, i.e., the saliency of an object is influenced by the annotator’s personal preference. As shown in Figure 1b, the hollow part of a bicycle is labeled salient in the left but not salient in the right. Moreover, both images with butterflies in Figure 1b share similar layouts (butterfly, flowers, and green background) but the flowers are marked as salient in the left image and not salient in the right. Therefore, SOD requires the model to be of high robustness to noisy labeling.

To improve the robustness, we propose a novel Augmentation-based Consistent Learning (ACL) scheme. By simulating a cross-dataset validation scenario, ACL introduces salient-related data augmentation to images, aiming to enhance the consistency of predictions between raw and augmented images. In this way, ACL makes the prediction of a raw image be consistent to that of the augmented image. As a result, the SOD model is capable of accurately locating the most salient objects while disregarding irrelevant variations caused by augmentation, thus improving the robustness of the model.

Our main contributions are summarized as follows.

• We propose a novel Self-Improved Training strategy for SOD, which reduces the learning difficulty and effectively improves the performance.
• We present an Augmentation-based Consistent Learning scheme that regularizes the consistency between raw images and their corresponding augmented images at training time and improves the robustness of the models.
• The proposed method is model independent, which can be applied to existing prevalent methods without modifying the architecture to gain considerable improvements.

2. Related Work

2.1. Network Designs for SOD

The network designs for Salient Object Detection can be summarized into three categories: architecture-based, aggregation-based, and contour-based methods.

First, architecture-based methods explore various feature extractors for SOD. Early methods such as [11,12,13,14,15,16,17] obtain handcrafted or deep features from image subunits and use MLP classifiers to predict saliency scores based on the features. For example, SuperCNN [16] and Multi-Context [13] use super-pixels or patches as image subunits. These methods fail to model the global context and cannot be trained end to end. Inspired by FCNs [18], CNN-based methods [19,20,21,22,23,24,25] formulate the salient object detection as a pixel-level prediction task and generate the saliency map in an end-to-end manner. Recently, Transformer has also been applied to SOD [26,27,28,29,30]. They combine both Transformer and CNN to process multi-level features. For instance, VST [30] considers the SOD task from the sequence-to-sequence perspective and designs a pure transformer model to handle the tokenized input. In addition, Transformer is also utilized to model the cross-modal information from the RGB and depth data [26,27].

Second, aggregation-based methods focus on feature aggregation or feature fusion to combine features from different layers. These methods capture both spatial details and global semantics for an accurate saliency map [13,24,31,32,33,34,35,36]. A simple yet effective way is to skip connect the features from the encoder to the features with the same spatial size from the decoder. This U-shape-based structure has been employed in many works [13,35,36]. Furthermore, GateNet [34] introduces gated mechanisms to control the information fusion flow. Also, channel-wise attention and spatial-wise attention have been applied to weigh the importance during the feature aggregation process [31,32,33]. MENet [37] designs a multi-scale feature enhancement module to gradually aggregate and refine global or detailed features by changing the size order of the input feature sequence.

To achieve high-quality saliency prediction with fine edge details, contour-based methods attempt to explicitly regularize the contours of salient objects. This line of work involves two aspects: network design and loss function design. Contour-aware networks attempt to model contours explicitly [32,33,38,39]. Specifically, EGNet [38] proposes to make the model predict saliency and contours simultaneously. Based on this, C2SNet [39] adopts an alternating converting method between contour maps and saliency maps. Unlike existing two-branch methods [33,38] that use one branch for the boundary and the other one for the entire object, MENet [37] uses two parallel branches that learn internal regional features and global features, respectively, so as to reduce the interference of inaccurate boundary information with global features. Some other works design contour-aware loss functions. The structural similarity index (SSIM) [40] and weighted BCE loss [41] are introduced for more edge details in the saliency maps.

Our method aims to smoothen the learning process and improve the robustness of SOD. We do not design a specific architecture or model contours for SOD explicitly. Instead, our learning strategy is capable of working with all these methods without modifying their networks.

2.2. Non-Fully Supervised Learning for SOD

To alleviate the burden of human efforts on pixel-level annotations, semi-/weakly-supervised methods for SOD have been proposed. These methods require image-level [42,43,44,45,46], scribble-level [47,48], or point-level annotations [49,50]. For example, FIN [42] leverages the category labels and is jointly optimized with an FCN to capture potentially salient regions. WS3A [47] uses scribbles annotation to do local supervision while utilizing auxiliary edges to provide details. PSOD [49] uses point annotation to provide the locations of salient objects to obtain pseudo labels to gain the first round of supervision, and then again uses point annotation to suppress non-salient objects to obtain optimized pseudo labels for the second round.

Recent years also witness the adoption of self-supervised learning for SOD, which requires no human annotations. A2S [51] and A2Sv2 [52] employ large-scale unsupervised pre-trained networks. They utilize a two-step approach where the first stage involves constructing reliable pseudo labels and the second stage trains the final segmenter after intermediate label processing. UNSS [53] introduces a top–down context guidance strategy that extracts detailed signals for both global and local segmentation learning.

In this paper, the training of our method consists of two parts: the Self-Improvement Training strategy (SIT) and Augmentation-based Consistent Learning. They are motivated by self-supervised learning, but both of them utilize pixel-level annotations. Specifically, SIT utilizes the SOD model, which uses pixel-level annotations, to generate smooth labels to alleviate the training difficulty of itself. ACL adopts the CutOut augmentation to remove part of the image and forces the model to make the results of the augmented image consistent with the results of the original image. The removed part is determined using the ground truth, and thus, ACL cannot be treated as unsupervised. In essence, the aim of ACL is to make the model robust instead of reducing human efforts on annotations.

3. Method

Salient object detection learns a function from an image to saliency predictions. Generally, an SOD network ${\mathcal{F}}_{\mathrm{SOD}}$ consists of an encoder E and a decoder D. Given an input RGB image $x\in {\mathbb{R}}^{H\times W\times 3}$ and its ground truth $y\in {\{0,1\}}^{H\times W}$ with the height H and width W, encoder E extracts features and decoder D generates a saliency prediction $p\in {\mathbb{R}}^{H\times W}$ by $p={\mathcal{F}}_{\mathrm{SOD}}\left(x\right)=D\left(E\left(x\right)\right)$.

Next, we present our framework as illustrated in Figure 2a. We design the proposed Self-Improvement Training strategy (SIT) in Section 3.1 and Augmentation-based Consistent Learning scheme (ACL) in Section 3.2. Finally, we summarize the whole training procedure in Section 3.3.

3.1. Self-Improvement Training Strategy

In this work, we propose a Self-Improvement Training (SIT) strategy to reduce the learning difficulty by relaxing the binary annotations with smooth labels. We decompose SIT into two steps. First, we generate smooth labels with a proposed Progressively Updated Module (PUM). Second, we design a Sample Adaptive Module (SAM) to weigh the balance between binary and smooth labels.

3.1.1. PUM: Progressively Updated Module

Formally, we design a progressively updated module (PUM), denoted by ${\mathcal{F}}_{\mathrm{PUM}}$, to generate smooth labels. ${\mathcal{F}}_{\mathrm{PUM}}$ shares the same network structure with the SOD network ${\mathcal{F}}_{\mathrm{SOD}}$. Specifically, let the parameters of ${\mathcal{F}}_{\mathrm{SOD}}$ be ${\theta }_{\mathrm{SOD}}$, and the parameters of ${\mathcal{F}}_{\mathrm{PUM}}$ be ${\theta }_{\mathrm{PUM}}$. First, we randomly initialize ${\theta }_{\mathrm{SOD}}$. Then, we initialize ${\theta }_{\mathrm{PUM}}$ by

$$t=0:{\theta }_{\mathrm{PUM}}^{\left(0\right)}={\theta }_{\mathrm{SOD}}^{\left(0\right)},$$

(1)

where t is the iteration of the training (Iteration t refers to a mini-batch training iteration instead of an epoch). At this point, $F_{\mathrm{PUM}}$ is identical to $F_{\mathrm{SOD}}$. They can only generate a random salient output before training. To progressively improve the quality of the smooth labels, we update ${\theta }_{\mathrm{PUM}}$ at a momentum coefficient $\eta$ from ${\theta }_{\mathrm{SOD}}$ as follows

$$\begin{array}{cc}\hfill t=1:{\theta }_{\mathrm{PUM}}^{\left(1\right)}& =\eta ·{\theta }_{\mathrm{PUM}}^{\left(0\right)}+(1-\eta )·{\theta }_{\mathrm{SOD}}^{\left(1\right)},\hfill \\ \hfill t=2:{\theta }_{\mathrm{PUM}}^{\left(2\right)}& =\eta ·{\theta }_{\mathrm{PUM}}^{\left(1\right)}+(1-\eta )·{\theta }_{\mathrm{SOD}}^{\left(2\right)},\hfill \\ \hfill ⋮& ⋮\hfill \\ \hfill t=T:{\theta }_{\mathrm{PUM}}^{\left(T\right)}& =\eta ·{\theta }_{\mathrm{PUM}}^{(T-1)}+(1-\eta )·{\theta }_{\mathrm{SOD}}^{\left(T\right)}.\hfill \end{array}$$

(2)

It is worth noting that only parameters ${\theta }_{\mathrm{SOD}}$ are updated via backpropagation, whereas parameters ${\theta }_{\mathrm{PUM}}$ are updated progressively by ${\theta }_{\mathrm{SOD}}$ at the training time of ${\mathcal{F}}_{\mathrm{SOD}}$. The momentum update in Equation (2) makes ${\theta }_{\mathrm{PUM}}$ evolve more smoothly than ${\theta }_{\mathrm{SOD}}$. As a result, the generated smooth labels are improved smoothly with the training of ${\mathcal{F}}_{\mathrm{SOD}}$. In this way, the PUM progressively integrates the parameter from the SOD network and generates more informative supervisions for the output of the SOD network during training, as illustrated in Figure 2b.

With PUM, we can obtain smooth labels z by feeding an input image x into ${\mathcal{F}}_{\mathrm{PUM}}$ as

$$\begin{array}{cc}\hfill z^{\left(t\right)}& ={\mathcal{F}}_{\mathrm{PUM}}(x;{\theta }_{\mathrm{PUM}}^{\left(t\right)}),\hfill \end{array}$$

(3)

where predictions z are of the same size as ground truth y. While y is $\in {\{0,1\}}^{H\times W}$, z is $\in {(0,1]}^{H\times W}$

With smooth labels z, we regularize ${\mathcal{F}}_{\mathrm{SOD}}$ by

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{PUM}}& =\sum _{i=1}^H\sum _{j=1}^W{(p_{i,j}^{\left(t\right)}-z_{i,j}^{\left(t\right)})}^2,\hfill \end{array}$$

(4)

where $p\in {(0,1]}^{H\times W}$ are the predictions of ${\mathcal{F}}_{\mathrm{SOD}}$, and $z\in {(0,1]}^{H\times W}$ are the obtained smooth labels by ${\mathcal{F}}_{\mathrm{PUM}}$.

3.1.2. SAM: Sample Adaptive Module

Although smooth labels z aim at providing more informative supervisions for the output of the SOD network, they are noisy and unreliable at the beginning of the training. Figure 3 shows that the quality of the smooth labels is getting better and better as the training goes. To alleviate the noise incurred by smooth labels z, we introduce an adaptive weight $\lambda$ defined by

$$\lambda =exp(-\zeta ·(y·log{z}^{\left(t\right)}+(1-y)·log(1-z^{\left(t\right)}))),$$

(5)

where y is the ground truth, $z^{\left(t\right)}$ are the smooth labels at training iteration t, and $\zeta$ is a hyperparameter. The definition of $\lambda$ implies that $\lambda$ is always positive, and that when z is significantly different from y, which means the quality of the smooth labels is bad, $\lambda$ will be close to zero. In this way, instead of directly enforcing ${\mathcal{F}}_{\mathrm{SOD}}$ to mimic the output from ${\mathcal{F}}_{\mathrm{PUM}}$, our Sample Adaptive Module (SAM) measures the quality of generated smooth labels and strikes a balance between ground truth and the smooth labels. As a result, the overall loss function of SIT is

$${\mathcal{L}}_{\mathrm{SIT}}={\mathcal{L}}_{\mathrm{SOD}}+\lambda ·{\mathcal{L}}_{\mathrm{PUM}},$$

(6)

where ${\mathcal{L}}_{\mathrm{PUM}}$ is defined in Equation (4), and ${\mathcal{L}}_{\mathrm{SOD}}$ is the commonly used SOD loss function defined by

$${\mathcal{L}}_{\mathrm{SOD}}=\sum _{i=1}^H\sum _{j=1}^W(y_{i,j}·log{p}_{i,j}+(1-y_{i,j})·log(1-p_{i,j})).$$

(7)

At the beginning of training, z is noisy, which leads to a very small $\lambda$ to punish the impact of ${\mathcal{L}}_{\mathrm{PUM}}$. As the training goes, the quality of z will be better and better and $\lambda$ will be larger, which makes the impact of ${\mathcal{L}}_{\mathrm{PUM}}$ more significant. In other words, as the quality of smooth labels becomes better, $F_{\mathrm{SOD}}$ will learn more from $F_{\mathrm{PUM}}$.

3.2. Augmentation-Based Consistent Learning

We design an augmentation-based consistent learning (ACL) scheme to keep the model’s robustness for the salient-related data augmentation. By regularizing the model at both the prediction level and feature level, ACL helps the SOD model to capture more discriminative contextual information and improve the robustness.

3.2.1. Regularizing with Prediction Consistency

We first augment the raw image x with task-specific augmentation methods (detailed in Section 4.1.3) and obtain its augmentations $\widehat{x}$ and its corresponding saliency prediction $\widehat{p}$. Next, we regularize the prediction consistency between the raw image and its augmentations by

$${\mathcal{L}}_{\mathrm{logit}}=\sum _{i=1}^H\sum _{j=1}^W{(p_{i,j}^{\left(t\right)}-{\widehat{p}}_{i,j}^{\left(t\right)})}^2,$$

(8)

where p is the prediction of x and $\widehat{p}$ is the prediction of $\widehat{x}$.

3.2.2. Regularizing with Feature Consistency

Moreover, we introduce a multiscale feature consistency loss ${\mathcal{L}}_{\mathrm{feature}}$ to regularize the feature consistency for each layer of encoder $E_i$ between the raw image x and augmented image $\widehat{x}$ by

$${\mathcal{L}}_{\mathrm{feature}}=\sum _{i=1}^N{∥E_i(x;{\theta }_{\mathrm{SOD}}^{\left(t\right)})-E_i(\widehat{x};{\theta }_{\mathrm{SOD}}^{\left(t\right)})∥}_F^2,$$

(9)

where N is the number of feature levels in encoder E, $E_i\left(x\right)$ are the features of x at the i-th layer of E, $E_i\left(\widehat{x}\right)$ are the features of $\widehat{x}$ at the i-th layer, and ${\parallel ·\parallel }_F$ is the Frobenius norm of the matrix.

3.3. The Training Procedure

Finally, we put all things together and train ${\mathcal{F}}_{\mathrm{SOD}}$ in an end-to-end manner. The total loss ${\mathcal{L}}_{\mathrm{total}}$ is defined as

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{SIT}}+{\mathcal{L}}_{\mathrm{logit}}+{\mathcal{L}}_{\mathrm{feature}}.$$

(10)

The optimization procedure of our proposed method is summarized in Algorithm 1. It is worth noting that our method is orthogonal to the network architectures; thus, it can work as a plug-and-play module to boost the existing methods.

Algorithm 1 SIT and ACL training strategy at the t iteration

Input: x, an input images; y, its ground-truth

Networks: ${\mathcal{F}}_{\mathrm{SOD}}(·;{\theta }_{\mathrm{SOD}})$, an SOD network with its parameters; ${\mathcal{F}}_{\mathrm{PUM}}(·;{\theta }_{\mathrm{PUM}})$, a PUM network with its parameters

Output: ${\theta }_{\mathrm{SOD}}$, ${\theta }_{\mathrm{PUM}}$.

1: Initialization: ${\theta }_{\mathrm{PUM}}^{\left(0\right)}\leftarrow {\theta }_{\mathrm{SOD}}^{\left(0\right)}$   2: for $t=1$ to  T  do     // SIT Self-Improvement Training Strategy   3:     $p={\mathcal{F}}_{\mathrm{SOD}}(x;{\theta }_{\mathrm{SOD}}^{(t-1)})$ // Obtain saliency prediction   4:     $z={\mathcal{F}}_{\mathrm{PUM}}(x;{\theta }_{\mathrm{PUM}}^{(t-1)})$ // Obtain smooth label   5:     Compute the SIT loss ${\mathcal{L}}_{\mathrm{SIT}}$ defined in Equation (6)       $//$ ACL Augmentation-based Consistent Learning   6:     $\widehat{x}=\mathrm{Augment}\left(x\right)$ $//$ Get augmentation   7:     $\widehat{p}={\mathcal{F}}_{\mathrm{SOD}}(\widehat{x};{\theta }_{\mathrm{SOD}}^{(t-1)})$ // Obtain saliency prediction   8:     Compute ${\mathcal{L}}_{\mathrm{logit}}$ and ${\mathcal{L}}_{\mathrm{feature}}$ // defined in Equations (8) and (9)       $//$ Network Optimization   9:     Compute the total loss ${\mathcal{L}}_{\mathrm{total}}$ defined in Equation (10)  10:     ${\theta }_{\mathrm{SOD}}^{\left(t\right)}\leftarrow \mathrm{Adam}({\mathcal{L}}_{\mathrm{total}},{\theta }_{\mathrm{SOD}}^{(t-1)})$ // Optimize SOD network  11:     ${\theta }_{\mathrm{PUM}}^{\left(t\right)}\leftarrow \eta ·{\theta }_{\mathrm{PUM}}^{(t-1)}+(1-\eta )·{\theta }_{\mathrm{SOD}}^{\left(t\right)}$ // Update PUM  12: end for  13: return ${\theta }_{\mathrm{SOD}}$, ${\theta }_{\mathrm{PUM}}$

4. Experiment

In this section, we evaluate our method. First, we describe the experiment setup in Section 4.1. Next, we conduct a series of ablation studies to demonstrate the impact of each component in our proposed framework and compare our method with others in Section 4.2. Furthermore, we carry out a detailed analysis of different design choices of our proposed SIT and ACL in Section 4.3. Finally, we present a visual comparison in Section 4.4.

4.1. Experiment Setup

4.1.1. Datasets

We evaluate our method on five popular datasets as shown in

Table 1. The datasets we use. Only part of DUTS is used at training time and the rest are used at testing.

Datasets	Training Images	Testing Images
DUTS [42]	10,533	5019
ECSSD [10]	–	1000
DUT-OMRON [9]	–	5168
HKU-IS [11]	–	4447
PASCAL-S [8]	–	850

, including ECSSD [10] with 1000 images, PASCAL-S [8] with 850 images, DUT-OMRON [9] with 5168 images, HKU-IS [11] with 4447 images, and DUTS [42] with 15,572 images. All datasets are human-labeled with pixel-wise ground truth for quantitative evaluations. DUTS is currently the largest SOD dataset, which is divided into 10,553 training images (DUTS-TR) and 5019 testing images (DUTS-TE). We follow [40,54] to use DUTS-TR as the training dataset and the others as the testing datasets.

4.1.2. Evaluation Metrics

For all the experiments, we use four evaluation metrics to measure the performance, including the Mean Absolute Error (MAE), Mean F-measure ($mF$), structural similarity measure ($S_{\alpha },\alpha =0.5$) [55], and E-measure ($E_{\xi }$) [56].

First, MAE is a metric that computes the average absolute difference between the predicted saliency map p and its corresponding ground-truth map y pixel-by-pixel as follows

$$MAE=\frac{1}{H\times W}\sum _{i=1}^H\sum _{j=1}^W|p_{i,j}-y_{i,j}|,$$

(11)

where H and W are the height and width of p correspondingly. Second, the F-measure, termed $mF$, is a metric that evaluates both precision and recall comprehensively, and we provide the mean F-measure using varying fixed (0–255) thresholds. Next, the S-measure, termed $S_{\alpha }$, is a metric that consists of the region-aware ($S_r$) and object-aware ($S_o$) structural similarity:

$$S_{\alpha }=\alpha ·S_o+(1-\alpha )·S_r,$$

(12)

where $\alpha$ is set to 0.5 to weigh the balance. Finally, the E-measure, termed $E_{\xi }$, is a metric that takes both the image-level mean value and local pixel matching information into consideration to evaluate the similarity between the prediction and the ground truth.

4.1.3. Implementation Details

Our framework is implemented via PyTorch. As our goal is to design a general model independent framework, we choose three state-of-the-art SOD algorithms: F${}^3$Net [57], MINet [58], and GateNet [34] as baselines and integrate our methods into these different baselines. ResNet-50 [59], pre-trained on ImageNet, is used as the backbone network for all the three baselines. For training the hyperparameters’ setting, we follow different baseline’s implementation details according to their paper. For the data augmentation method in ACL, we use CutOut [60]. Specifically, we will conduct Cutout on the image area where there are no salient objects.

4.2. Ablation Study

First, we carry out ablation experiments to validate the effectiveness of our proposed SIT strategy and ACL scheme. Then, we integrate the proposed framework into different SOD baseline models. Since our method is orthogonal to the network architectures, we conduct all ablation studies on the basis of F${}^3$Net due to its flexibility and effectiveness without a loss of generality.

4.2.1. Effectiveness of SIT

The integration of SIT (the second row in

Table 2. Ablation study for SIT and ACL on the F${}^3$Net [57] baseline on the DUTS-TE dataset. SIT: Self-Improvement Training. ACL: Augmentation-based Consistent Learning. ↑ and ↓ indicate that the larger and smaller scores are better, respectively. The best results are highlighted in bold.

SIT	ACL	MAE ↓	mF ↑	${\mathit{S}}_{\mathit{\alpha }}$ ↑	${\mathit{E}}_{\mathit{\xi }}$ ↑
—	—	0.035	0.840	0.888	0.902
✓	—	0.035	0.847	0.891	0.904
—	✓	0.035	0.845	0.888	0.905
✓	✓	0.034	0.846	0.891	0.904

) gives performance gains of 0.7%, 0.3%, and 0.2%, in terms of $mF$, $S_{\alpha }$, and $E_{\xi }$ on the DUTS-TE dataset over the F${}^3$Net baseline, respectively. Such results validate that the SIT strategy serves as a useful way to reduce the learning difficulty and improve the performance.

4.2.2. Effectiveness of ACL

As shown in the third row of Table 2, simply embedding the ACL scheme into the F${}^3$Net baseline also helps improve the performance on both $mF$ and $E_{\xi }$ on the DUTS-TE dataset by 0.5% and 0.3%, respectively. It reveals that the robustness of the model is improved by enforcing consistency regularization for augmented images in ACL. The visual effects of ACL are illustrated in Figure 4. We can see that ACL helps effectively suppress the distracting background and accurately locate the salient objects because the richer contextual information can be captured via the consistency learning process.

4.2.3. The Whole Framework

To demonstrate the effectiveness of the proposed methods, we compare it against 13 state-of-the-art SOD algorithms, including AFNet [33], BASNet [40], CPD-R [54], BMPM [24], R${}^3$Net [61], PiCA-R [23], DGRL [22], TDBU [62], PoolNet [63], PAGE [32], RAS [64], C2SNet [39], and F${}^3$Net [57]. As shown in

Table 3. Performance comparison with 12 state-of-the-art methods over five datasets. MAE (smaller is better), mean F-measure ($mF$, larger is better), S-measure ($S_{\alpha }$, larger is better), and E-measure ($E_{\xi }$, larger is better) are utilized to evaluate the model’s performance. ↑ and ↓ indicate that the larger and smaller scores are better, respectively. The best results are highlighted in red. Our model ranks first on most datasets and metrics.

Algorithm	ECSSD				PASCAL-S				DUTS-TE				HKU-IS				DUT-OMRON
	MAE ↓	$\mathit{mF}\uparrow$	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{E}}_{\mathit{\xi }}\uparrow$	MAE ↓	$\mathit{mF}$ ↑	${\mathit{S}}_{\mathit{\alpha }}$ ↑	${\mathit{E}}_{\mathit{\xi }}$ ↑	MAE ↓	$\mathit{mF}$ ↑	${\mathit{S}}_{\mathit{\alpha }}$ ↑	${\mathit{E}}_{\mathit{\xi }}$ ↑	MAE ↓	$\mathit{mF}$ ↑	${\mathit{S}}_{\mathit{\alpha }}$ ↑	${\mathit{E}}_{\mathit{\xi }}$ ↑	MAE ↓	$\mathit{mF}$ ↑	${\mathit{S}}_{\mathit{\alpha }}$ ↑	${\mathit{E}}_{\mathit{\xi }}$ ↑
C2SNet	0.059	0.853	0.882	0.906	0.086	0.761	0.822	0.835	0.066	0.710	0.817	0.841	0.051	0.839	0.873	0.919	0.079	0.664	0.780	0.817
RAS	0.055	0.890	0.894	0.916	0.102	0.782	0.792	0.832	0.060	0.750	0.838	0.861	0.045	0.874	0.888	0.931	0.063	0.711	0.812	0.843
R${}^3$Net	0.051	0.883	0.910	0.914	0.101	0.775	0.809	0.824	0.067	0.716	0.837	0.827	0.047	0.853	0.894	0.921	0.073	0.690	0.819	0.814
PiCA-R	0.046	0.886	0.917	0.913	0.075	0.798	0.849	0.833	0.051	0.759	0.869	0.862	0.043	0.870	0.904	0.936	0.065	0.717	0.832	0.841
BMPM	0.044	0.894	0.911	0.914	0.073	0.803	0.840	0.838	0.049	0.762	0.861	0.859	0.039	0.875	0.906	0.937	0.063	0.698	0.809	0.839
DGRL	0.043	0.903	0.906	0.917	0.074	0.807	0.834	0.836	0.051	0.764	0.846	0.863	0.037	0.881	0.896	0.941	0.063	0.709	0.810	0.843
PAGE	0.042	0.906	0.912	0.920	0.077	0.810	0.835	0.841	0.052	0.777	0.854	0.869	0.037	0.882	0.903	0.940	0.062	0.736	0.824	0.853
AFNet	0.042	0.908	0.913	0.918	0.070	0.821	0.844	0.846	0.046	0.792	0.867	0.879	0.036	0.888	0.905	0.942	0.057	0.738	0.826	0.853
TDBU	0.041	0.880	0.918	0.922	0.071	0.779	0.844	0.852	0.048	0.767	0.865	0.879	0.038	0.878	0.907	0.942	0.061	0.739	0.837	0.854
PoolNet	0.039	0.915	0.921	0.924	0.074	0.822	0.845	0.850	0.040	0.809	0.883	0.889	0.032	0.899	0.916	0.949	0.055	0.747	0.835	0.863
BASNet	0.037	0.880	0.916	0.921	0.076	0.775	0.832	0.847	0.048	0.791	0.866	0.884	0.032	0.895	0.909	0.946	0.056	0.756	0.836	0.869
CPD-R	0.037	0.917	0.918	0.925	0.072	0.824	0.842	0.849	0.043	0.805	0.869	0.886	0.034	0.891	0.905	0.944	0.056	0.747	0.825	0.866
PoolNet	0.039	0.915	0.921	0.924	0.075	0.810	0.836	0.847	0.040	0.809	0.883	0.889	0.033	0.893	0.913	0.946	0.056	0.747	0.836	0.863
PoolNet+ours	0.039	0.918	0.922	0.924	0.074	0.815	0.841	0.848	0.039	0.814	0.884	0.892	0.031	0.899	0.915	0.950	0.052	0.752	0.838	0.870
F${}^3$Net	0.033	0.925	0.924	0.927	0.062	0.840	0.855	0.859	0.035	0.840	0.888	0.902	0.028	0.910	0.917	0.953	0.053	0.766	0.838	0.870
F${}^3$Net+ours	0.032	0.930	0.927	0.929	0.061	0.837	0.855	0.861	0.034	0.846	0.891	0.904	0.027	0.916	0.921	0.955	0.052	0.769	0.842	0.870
MINet	0.033	0.924	0.925	0.927	0.063	0.829	0.850	0.851	0.037	0.828	0.884	0.898	0.029	0.909	0.919	0.953	0.055	0.755	0.833	0.865
MINet+ours	0.033	0.926	0.925	0.927	0.059	0.835	0.857	0.859	0.036	0.834	0.887	0.901	0.029	0.910	0.920	0.954	0.054	0.756	0.834	0.869
GateNet	0.040	0.916	0.920	0.924	0.067	0.819	0.851	0.851	0.040	0.807	0.885	0.889	0.033	0.899	0.915	0.949	0.055	0.746	0.838	0.861
GateNet+ours	0.038	0.915	0.924	0.924	0.065	0.820	0.856	0.857	0.039	0.808	0.888	0.891	0.031	0.901	0.921	0.951	0.055	0.746	0.839	0.862

, the performances of these baselines are boosted considerably by being integrated with our proposed framework.

Compared with the baselines, the $mF$ results of our method on the DUTS-TE dataset are improved by 0.6%, 0.6%, and 0.2% based on three different baselines, respectively. In addition, Figure 5 shows the standard PR curves and the F-measure curves of the aforementioned baselines on the DUTS dataset, which can evaluate the performance of the models comprehensively. From these curves, we can observe that the models trained with our framework consistently outperform all corresponding baselines under different thresholds, which means that our framework has an excellent capability to detect salient regions and generate accurate saliency maps. All of these reveal that the proposed framework is effective on different SOTA methods and achieves a clearly better-averaged performance gain without modifying any architecture.

4.3. Analysis on Design Choices

We analyze the design choices of our method from three aspects. First, we explore the updating strategy of PUM. Second, we conduct an ablation study of the selection of momentum coefficient. Finally, we show the effectiveness of the proposed SAM.

4.3.1. Updating Strategy of PUM

To obtained smooth labels in SIT, we need to update the PUM module from the SOD network. A straightforward updating strategy is to update the PUM module only once per epoch, which is similar to the $\Pi$ model [65]. In this paper, we adopt a momentum-updating manner to integrate models of different steps to the PUM module progressively. From

Table 4. The effect of different updating strategies of PUM in our proposed SIT on the DUTS-TE dataset. ‘Epoch-based’ denotes updating ${\theta }_{\mathrm{PUM}}$ only once per epoch. ‘Iteration-based’ denotes updating ${\theta }_{\mathrm{PUM}}$ by Equation (2). $\eta$ denotes the momentum coefficient. ↑ and ↓ indicate that the larger and smaller scores are better, respectively. The best results are highlighted in bold.

Updating Strategy	$\mathit{\eta }$	MAE ↓	$\mathit{mF}$ ↑	${\mathit{S}}_{\mathit{\alpha }}$ ↑	${\mathit{E}}_{\mathit{\xi }}$ ↑
Epoch Based	—	0.038	0.833	0.882	0.895
Iteration Based	0.9	0.035	0.843	0.891	0.901
Iteration Based	0.99	0.034	0.846	0.891	0.904
Iteration Based	0.999	0.035	0.846	0.889	0.903
Iteration Based	0.9999	0.035	0.841	0.888	0.902

, we can see that all the momentum-updating strategy settings outperform the epoch-based updating strategy setting. With the momentum coefficient $\eta$ in the range of 0.9 to 0.9999, the momentum-updating strategy could achieve a stable performance improvement, exceeding the epoch-based updating strategy on average by 1.1% on $mF$ and 0.8% in $S_{\alpha }$.

4.3.2. The Selection of Momentum Coefficient $\eta$

As described in Section 3.1, the momentum coefficient $\eta$ determines the update speed of ${\theta }_{\mathrm{PUM}}$. Our model’s sensitivity to $\eta$ is shown in Table 4. When $\eta$ is set to 0.99, the model can achieve the best performance on all metrics. We have revised this part to present our results more clearly.

4.3.3. Effect of Sample Adaptive Module

In this Sample Adaptive Module (SAM), we introduce an adaptive weight to dynamically tune the loss contribution between ground truth and smooth labels. To demonstrate the effectiveness of our SAM, we compared our dynamic adaptive weight strategy with a fixed weight strategy. We choose three different fixed weights, which are 0.3, 0.5, and 0.7, respectively. From

Table 5. The effect of the fixed $\lambda$ compared with the adaptive $\lambda$ in SAM on the DUTS-TE dataset. ↑ and ↓ indicate that the larger and smaller scores are better, respectively. The best results are highlighted in bold.

$\mathit{\lambda }$	MAE ↓	$\mathit{mF}$ ↑	${\mathit{S}}_{\mathit{\alpha }}$ ↑	${\mathit{E}}_{\mathit{\xi }}$ ↑
0.3	0.035	0.841	0.889	0.901
0.5	0.036	0.838	0.885	0.899
0.7	0.040	0.829	0.883	0.891
SAM	0.034	0.846	0.891	0.904

, we can observe that our SAM method exceeds all the fixed weight settings. Compared with the best fixed weight setting ($\lambda$ = 0.3), the $mF$ of our SAM method exceeds it by 0.5%, which illustrates the effectiveness of SAM.

The hyperparameter $\zeta$ in Equation (5) is used to modulate the loss contribution of the PUM module. Thus, we conduct a series of experiments to evaluate the effect of hyperparameter $\zeta$. As shown in

Table 6. The effect of the scale factor $\zeta$ in the SAM module on the DUTS-TE dataset. ↑ and ↓ indicate that the larger and smaller scores are better, respectively. The best results are highlighted in bold.

$\mathit{\zeta }$	MAE ↓	$\mathit{mF}$ ↑	${\mathit{S}}_{\mathit{\alpha }}$ ↑	${\mathit{E}}_{\mathit{\xi }}$ ↑
30	0.035	0.843	0.889	0.901
70	0.034	0.846	0.891	0.904
110	0.037	0.839	0.885	0.898

, setting $\zeta$ to 70 makes the model learn well from the PUM module and obtain the best performance with 0.846 on $mF$.

4.4. Visualization Analysis

4.4.1. Visualization of Smooth Labels

In this part, we show the smooth labels generated via the PUM at different training iterations. As shown in Figure 3, we can observe that the PUM gradually improves the quality of the smooth labels. As the learning process progresses, the smooth labels not only obtain a sharper boundary but also suppress the distractors in the background.

4.4.2. Visualization of Feature Attention Maps

To further demonstrate the effectiveness of our framework, we visualize some attention maps in Figure 6. It can be observed that even if the squirrel has very low contrast with the ground (see the second column), by using our framework, the high contrast between the object region and the background is always maintained, thereby making the salient objects be effectively distinguished. In addition, our framework can effectively suppress distraction objects (see the fifth column). The people in the corner of the image are suppressed well in the feature attention map.

4.4.3. Visual Comparison with Baseline Methods

To evaluate the robustness and effectiveness of our framework, we visualize some saliency maps and exhibit some typical images from the public test dataset of saliency detection in Figure 7.

In the first, second, and fifth rows, the complex backgrounds. such as seats and windows in the first row, stacked boxes in the second row, and books in the fifth row, are suppressed in ours but highlighted in all other baselines. Especially for the first and second rows, the backgrounds are suppressed well. Presumably, it is because the color of these backgrounds’ objects are similar. Thus, our method can accurately locate and suppress all of them. In the third row, the baseline methods are severely affected by the gray box. Our framework can not only better pick out the salient objects accurately, but it also well suppresses these distractors. In the fourth and sixth row, the baseline methods fail to capture complete salient objects, such as the person on the left in the fourth row and the elephant on the right in the sixth row. However, our framework can accurately locate these salient objects.

In a word, our proposed framework can possess good robustness and effectiveness in detecting various salient objects.

4.5. Discussion

In this part, we will discuss the strengths and weaknesses of our method. Let us review the visual comparison in Figure 6. The third column shows a sailing boat. Our method misses its hull. The fourth column is a squirrel. Our method misses its paw. The fifth column is a dog. Our method misses its forelegs. These errors have a common point: the method misses their spatial details.

That being said, our method can predict results with spatial details as shown in the first row of Figure 7. In this case, all of our models (F${}^3$Net+ours, MINet+ours, and GateNet+ours) can detect the contour of the person’s feet. However, by looking at the pictures more carefully, one can find that the detection of the foot’s contour should be attributed to the baseline models (F${}^3$Net, MINet, and GateNet). F${}^3$Net+ours has a similar foot’s contour as F${}^3$Net. Similar results can be found in MINet vs. MINet+ours and GateNet vs. GateNet+ours.

Our method aims to alleviate the learning difficulty and improve the robustness of the SOD methods. We do not design specific contour-based techniques to improve the performance, but these techniques are orthogonal and complementary to our work.

5. Conclusions

In this paper, we present a learning framework to reduce the learning difficulty for the SOD task. A novel self-improvement training (SIT) strategy is designed to generate smooth labels, which alleviates the learning difficulty. Moreover, by regularizing the prediction consistency and multi-level feature consistency in augmentation-based consistent learning (ACL), the robustness of the model can be further improved. Comprehensive benchmarks on several popular datasets illustrate the advantage of the proposed framework. A further ablation study shows the effectiveness of each method of our framework. Briefly, our framework can play a plug-and-play role to be easily embedded in the existing SOD networks to achieve a promising performance gain, without any modification of the network architecture.