High Edge-Quality Light-Field Salient Object Detection Using Convolutional Neural Network

Abstract

The detection result of current light-field salient object detection methods suffers from loss of edge details, which significantly limits the performance of subsequent computer vision tasks. To solve this problem, we propose a novel convolutional neural network to accurately detect salient objects, by digging effective edge information from light-field data. In particular, our method is divided into four steps. Firstly, the network extracts multi-level saliency features from light-field data. Secondly, edge features are extracted from low-level saliency features and optimized by ground-truth guidance. Then, to sufficiently leverage high-level saliency features and edge features, the network hierarchically fuses them in a complementary manner. Finally, spatial correlations between different levels of fused features are considered to detect salient objects. Our method can accurately locate salient objects with exquisite edge details, by extracting clear edge information and accurate saliency information and fully fusing them. We conduct extensive evaluations on three widely used benchmark datasets. The experimental results demonstrate the effectiveness of our method, and it is superior to eight state-of-the-art methods.

1. Introduction

Salient object detection (SOD) aims to detect the salient attention-grabbing objects in a scene and segment the whole objects [1]. It plays a key role in many computer vision tasks, such as image segmentation [2,3], object tracking [4,5,6], image retrieval [7,8], and content-aware image editing [9,10]. With the development of light-field imaging technology, light-field SOD has also received increasing attention [11]. Accurate edge detection is a part of light-field SOD, which is crucial for numerous subsequent light-field-based vision applications. For example, in 3D display [12], salient objects with fine edges can guide precise depth correction and enhance the 3D display of major content; in light-field image compression [13], salient objects with fine edges can help to preserve the details of important objects and improve the compression quality of light-field images.

Existing light-field SOD methods can be roughly divided into two categories: traditional methods [14,15,16] and deep learning-based methods [17,18,19,20]. Traditional methods [14,15,16] employ hand-crafted features such as color contrast, texture contrast, and background prior to detect salient objects, which lack consideration of global contextual information. Consequently, the detection performance is limited in complex scenes such as low light and clutter background. Deep learning-based methods [17,18,19,20] leverage the powerful feature learning capability of CNN to detect salient objects, and make up for the shortcomings of traditional methods. Zhang and Ji et al. [17] refined the complementary information in light-field features, and reduced the interference of background to the salient objects. Wang [18] proposed a feature fusion module, which can adaptively fuse across-modal visual features to enhance the network robustness. Zhang and Li et al. [19] detected complete salient objects, by introducing both attention mechanism and ConvLSTM to fuse light-field features hierarchically. Piao and Rong et al. [20] optimized the variability between light-field features to accurately locate salient objects. These methods improved the detection accuracy in complex scenes and showed excellent performance in previous light-field SOD methods [20]. However, they focus more on the accurate saliency information in deep network and lack effective utilization of the rich edge information in shallow network, which leads to poor quality of detected edge details.

Therefore, we propose a novel convolutional neural network to exploit the edge information in shallow network. Firstly, the network extracts multi-level saliency features from all-in-focus images and focus stacks. Secondly, edge features are extracted from low-level saliency features and the invalid edge information in them is sieved out. Then, high-level saliency features and edge features are hierarchically integrated to take full advantage of the complementary strengths between them. Finally, the network learns the spatial correlations between different levels of fused features to detect salient objects. To verify the effectiveness of our proposed method, we compare it with eight state-of-the-art light-field SOD methods [14,15,16,17,19,20,21,22] on three widely-used datasets [14,19,23]. The results show that our method achieves the highest detection accuracy.

In summary, the main contributions of this paper are concluded as follows:

• We propose a novel light-field salient object detection network, which hierarchically extracts saliency features from light-field data. The high-level semantic information and low-level edge information in the saliency features are fully exploited to detect accurate salient objects.
• We propose an edge feature extraction strategy to extract clear edge information from light-field data and optimize it by ground-truth guidance.
• Extensive experiments conducted on three benchmark datasets show that our method achieves state-of-the-art performance.

The remainder of this paper is organized as follows. Section 2 presents the related work of light-field SOD to summarize existing methods and suggest ideas for improvement. In Section 3, we describe our proposed method in terms of network structure and loss function. Following this, to validate the effectiveness of our proposed method, we present the experimental setup, experimental comparisons, and experimental results in Section 4. Then, we discuss in Section 5 to provide possible ideas for the development of light-field SOD technology. Finally, concluded remarks are made in Section 6.

2. Related Works

In this section, we briefly summarize the related work of light-field SOD. In recent years, research in light-field images, such as light-field subjective evaluation [24], light-field quality model [25] and light-field stitching [26] has effectively improved the light-field images quality and promoted the development of light-field SOD. Light-field SOD methods can be roughly divided into two categories: traditional methods and deep learning-based methods.

Traditional methods use hand-crafted features such as color contrast, texture contrast, and background prior to detect salient objects. Li and Ye et al. [14] proposed the first light-field SOD method, which calculates saliency candidates by background prior and color contrast, respectively, and then objectness cues were used to weight the candidates. This motivated Li and Sun et al. [15] to propose a weighted sparse coding framework to construct saliency dictionary and non-saliency dictionary from light-field data, and then detect the salient objects by iterative optimization. Zhang and Wang et al. [16] computed depth-induced contrast, color contrast, and background prior from light-field data to enhance light-field SOD. Then, they also proposed a random-search-based weighting strategy to fuse multiple saliency cues extracted from light-field data [23]. Wang and Li et al. [27] proposed a two-stage Bayesian framework to fuse saliency features extracted from the light-field data, in order to improve the detection performance by exploiting the complementarity between the saliency features. In addition, Wang and Yan et al. [28] performed a coarse detection using the depth information in light-field data and then calculated the final saliency map based on the color contrast and texture contrast. Piao and Li et al. [29] took account of interactions and complementarities of color, depth, focusness and location cues to detect salient objects and improved detection accuracy by enforcing spatial consistency. However, these traditional methods rely on hand-crafted features and omit the semantic information of salient objects, leading to object errors and partial misses in complex scenes such as low light, low contrast, and clutter background.

The deep learning-based methods leverage the powerful feature learning capability of CNN to detect salient objects. Wang and Piao et al. [30] first proposed a deep learning-based light-field SOD network, in which a uniquely designed recurrent attention mechanism can adaptively fuse focal slice image features according to importance. However, they ignored the complementarity between the light-field features of various modalities. Affected by them, Zhang and Ji et al. [17] proposed a light-field refinement module to refine light-field features and then sufficiently fused them by introducing recurrent attention mechanism. The main advantage of this method is that it strengthens the connection between light-field features of different modalities, but it fails to exploit the features at each level. Therefore, Zhang and Li et al. [19] extracted features from light-field data hierarchically to exploit the expressiveness of features at different levels, and then fused them using recurrent attention mechanism to detect complete salient objects. However, it also increases the computational complexity. In order to improve the efficiency of light-field SOD, Piao and Rong et al. [20] designed a teacher network to exploit focus stack images and transfer comprehensive focusness knowledge to student network. However, the method cannot guarantee both accuracy and efficiency. Piao and Jiang et al. [21] explored light-field data in a regionwise manner under the guidance of saliency, boundary and location and improved the detection accuracy of small objects. In addition, Zhang and Liu et al. [22] designed an angle change module to learn multi-view information in light-field data and used dilated convolution to extract multi-scale spatial features, which improves the robustness of light-field SOD. Wang [18] proposed a feature fusion module to leverage multi-modal features, which can effectively reduce the impact of light-field data quality. The limitation of the method is that three types of light-field data are required as input. Jing and Zhang et al. [31] extracted accurate scene depth and occlusion information from epipolar plane images to detect complete saliency objects. However, the method lacks the exploitation of visual features in shallow network. Liang and Qin et al. [32] presented a dual guidance enhanced network to improve the detection accuracy by progressively global and boundary guidance of light-field features. Unfortunately, the two guidance strategies will limit each other and the edge information in light-field features cannot be fully utilized. Benefiting from the powerful feature learning capability of CNN [33,34], these deep learning-based methods are able to accurately locate salient objects in complex scenes, but their detection results generally suffer from loss of edge details.

3. Methodology

In this section, we introduce our high edge-quality light-field SOD network, which consists of four parts: saliency feature extraction, edge feature extraction module (EFEM), multi-feature fusion module (MFM), and multi-feature joint detection module (MJDM). The network structure is illustrated in Figure 1. Firstly, the two sets of convolutional blocks in Section 3.1 extract multi-level saliency features from all-in-focus images and focus stacks. Secondly, the EFEM in Section 3.2 is proposed to extract rich edge features from low-level saliency features. Then, the MFM in Section 3.3 hierarchically fuses edge features and high-level saliency features. Finally, the MJDM in Section 3.4 detects salient objects from the fused features. In addition, the saliency ground-truth and the edge ground-truth calculated from the saliency ground-truth are used to guide the generation of accurate saliency information and clear edge information.

3.1. Saliency Feature Extraction

Different levels of convolutional features have different representations for the image [35], thus we adopt two sets of convolutional blocks to extract multi-level saliency features from light-field data. As shown in Figure 1, we drop the last pooling layer and fully connected layers of the ResNet-50 network [36], and build 2D convolutional blocks and 3D convolutional blocks [37] with remaining convolutional layers to extract features from all-in-focus images and focus stacks. As in Piao and Jiang [21] et al. each focus stack consists of 12 focus slices, and we retain 3 focus slices in high-level 3D convolutional blocks after multiple pooling operations. In addition, the resolution size of all input image is 372 × 372 pixels. The saliency feature extracted by Conv-1 and Conv-2 is kept at 88 × 88 pixels, while the spatial resolution of the saliency features extracted by each of the other convolutional blocks is reduced by 1/2 in turn. We denote the outputs of the all-in-focus images and focus stacks in the convolutional block as ${\left\{f_{\mathrm{A}}^i\right\}}_{i=1}^5$ and ${\left\{f_{\mathrm{F}}^i\right\}}_{i=1}^5$ separately, where $i$ represents the convolution block level.

3.2. Edge Feature Extraction Module (EFEM)

We propose an edge feature extraction module to make effective use of the edge information in light-field data. On the one hand, the low-level features extracted by Conv-1 and Conv-2 contain rich edge texture information [35]. On the other hand, all-in-focus images have fewer data compared to focus stacks, which facilitates the extraction of high-quality edge features. Hence, EFEM extracts edge features from the all-in-focus image features $f_{\mathrm{A}}^1$ and $f_{\mathrm{A}}^2$ in Conv-1 and Conv-2. As shown in Figure 1, $f_{\mathrm{A}}^1$ and $f_{\mathrm{A}}^2$ are first concatenated and then repeatedly fed into a convolutional layer, a batch normalization layer, and an $ReLU$ activation function three times. This procedure can be defined as:

$$f_{\mathrm{E}}={(ReLU(BN(W_{\mathrm{E}}\ast Cat(f_{\mathrm{A}}^1,f_{\mathrm{A}}^2)+b_{\mathrm{E}})))}_{\times 3}$$

(1)

where $\ast$, $W_{\mathrm{E}}$, and $b_{\mathrm{E}}$ represent convolutional operator and convolutional parameters. $f_{\mathrm{E}}$ means extracted edge features. $BN(\cdot )$, $ReLU(\cdot )$, and $Cat(\cdot )$ denote batch normalization, activation function, and concatenation operation. ${(\cdot )}_{\times 3}$ means three-times repeated operation.

In addition, to ensure EFEM is able to learn valid edge information, we optimize the extracted edge features by supervision. First, the edge feature $f_{\mathrm{E}}$ is adjusted to single channel by a convolutional layer, which corresponds to the probability that each pixel belongs to the salient edge. Then, we supervise the single-channel edge feature using edge ground-truth map to sieve out the invalid background edge information. Specifically, we convert the 8-bit saliency ground-truth into binarized image by the adaptive threshold, followed by computing convolution with Sobel operator to obtain the edge ground-truth. Given saliency ground-truth $G_{\mathrm{S}}$, the operation can be defined as follows:

$$G_{\mathrm{E}}=\sqrt{{(W_{sobel-x}\ast G_{\mathrm{S}})}^2+{(W_{sobel-y}\ast G_{\mathrm{S}})}^2}$$

(2)

where $\ast$ represents convolutional operator. $W_{sobel-x}$ and $W_{sobel-y}$ represent Sobel operator in the x and y directions, which can be defined as:

$$W_{sobel-x}=\left[\begin{array}{ccc}-1& 0& 1\\ -2& 0& 2\\ -1& 0& 1\end{array}\right]W_{sobel-y}=\left[\begin{array}{ccc}1& 2& 1\\ 0& 0& 0\\ -1& -2& -1\end{array}\right]$$

We employ cross-entropy loss to supervise the extracted edge features. Given size $M\times N$ of edge ground-truth $G_{\mathrm{S}}$, the loss function can be defined as:

$$L_{\mathrm{E}}=-{\displaystyle \sum _{m=1}^M{\displaystyle \sum _{n=1}^N(G_{\mathrm{E}}^{mn}\log f_{\mathrm{E}}^{mn}+(1-G_{\mathrm{E}}^{mn})\log (1-f_{\mathrm{E}}^{mn}))}}$$

(3)

3.3. Multi-Feature Fusion Module (MFM)

We adopt multi-feature fusion module (MFM) to fuse high-level saliency features and edge features hierarchically. The high-level saliency features and edge features contain accurate location information and rich edge detail information, respectively. To take full advantage of them, we introduce MFM shown in Figure 2 to fuse different levels of all-in-focus image features, focus stack features, and edge features.

The design of MFM is inspired by the memory-oriented spatial fusion module (Mo-SFM) [19], whose parameters are adjusted according to the number of input features. MFM consists of three parts: attention mechanism, ConvLSTM [38], and global perception module (GPM). The attention mechanism assigns corresponding weights to each input feature based on contribution, in order to enhance useful features and suppress unnecessary features. This procedure can be defined as:

$$f_{att}^i(t)=F^i(t)\odot \delta (W_{att}\ast AvgPooling(F^i)+b_{att}))$$

(4)

where $F^i$ denotes the cascaded input features, including 1 edge feature, 1 all-in-focus image feature, and 3 focal slice features. $\ast$, $W_{att}$, and $b_{att}$ represent convolutional operator and convolutional parameters. $AvgPooling(\cdot )$ means global pooling operation and $\odot$ means feature-wise multiplication. $\delta (\cdot )$ denotes softmax function. $F^i(t)$ and $f_{att}^i(t)$ denote separate feature in $F^i$ and corresponding output. Then, ConvLSTM fuses saliency features and edge features in a complementary manner by learning the connection between them. Compared to Mo-SFM, the ConvLSTM in MFM contains 5 time steps corresponding to 5 feature inputs. Furthermore, by enlarging the receptive field, the global perception module (GPM) captures global contextual information in the output feature $f_s^i$ of ConvLSTM. This operation is defined as follows:

$$f_r^i=Con{v}_{1\times 1}(Cat(f_s^i,{\coprod }_d(Con{v}_{rate=d}(f_s^i))),d=1,3,5,7$$

(5)

where $Con{v}_{rate=d}(\cdot )$ denotes convolution with dilation rate $d$. ${\coprod }_d(\cdot )$ means perform multiple operations using different values of $d$. $Cat(\cdot )$ denotes concatenation operation.

3.4. Multi-Feature Joint Detection Module (MJDM)

We utilize multi-feature joint detection module (MJDM) to detect salient objects by learning spatial correlations between multi-level fused features. MJDM is derived from the memory-oriented feature integration module [19] and the architecture is shown in Figure 3.

Firstly, MJDM emphasizes the important channels in the current fusion feature $f_r^i$ by learning the channel attention of the higher-level memory feature $f_c^{i+1}$ and the current fusion feature $f_r^i$. Channel attention can be formulated as follows:

$$f_{ch}^i=f_r^i\otimes \delta (AvgPooling((W_{t-1}\ast f_c^{i+1}+b_{t-1})\oplus (W_r\ast f_r^i+b_r)))$$

(6)

where $\oplus$ and $\otimes$ denote pixel-wise addition and pixel-wise multiplication, respectively. Then, ConvLSTM generates new memory features by learning the spatial correlation between the high-level memory features $f_c^{i+1}$ and the optimized current fusion features $f_{ch}^i$. The calculation process is similar to Equation (4) by replacing the inputs. After four iterations, MJDM outputs the memory features $f_c^3$, followed by a transition convolutional layer and an upsample operation to obtain the final saliency map. To ensure ConvLSTM can explicitly learn the most important information, we perform intermediate supervision on each memory feature. Specifically, we adopt a cross-entropy loss $L_{\mathrm{S}}$ to supervise the generation of memory features, $L_{\mathrm{M}}$ is defined as:

$$L_{\mathrm{S}}=-{\displaystyle \sum _{x=1}^X{\displaystyle \sum _{y=1}^Y(G_{\mathrm{S}}^{xy}\log {(f_c^i)}^{xy}+(1-G_{\mathrm{S}}^{xy})\log (1-{(f_c^i)}^{xy}))}}$$

(7)

where $X$ and $Y$ represent the size of saliency ground-truth $G_{\mathrm{S}}$. Therefore, the total loss function can be expressed as:

$$L=L_{\mathrm{E}}+L_{\mathrm{S}}$$

(8)

The specific Algorithm 1 for the whole procedure is as follows.

Algorithm 1. High Edge-Quality Light-Field Salient Object Detection

Input: All-in-focus images, focus stacks, Saliency ground-truth $G_{\mathrm{S}}$, Iteration number T, Sobel operator $W_{sobel-x}$ and $W_{sobel-y}$ Output: Parameters of network model $\theta$ 1. Initialize the network parameters by pre-trained ResNet-50, t = 1 2. for t = 1: T do 3.      Generate multi-level saliency features ${\left\{f_{\mathrm{A}}^i\right\}}_{i=1}^5$ and ${\left\{f_{\mathrm{F}}^i\right\}}_{i=1}^5$ 4.      Compute the $f_{\mathrm{E}}$ from $(f_{\mathrm{A}}^1,f_{\mathrm{A}}^2)$, as in Equation (1) 5.      Compute the $G_{\mathrm{E}}$ from $G_{\mathrm{S}}$, as in Equation (2) 6.      for i = 3: 5 do 7.            Generate $F^i$ by concatenating $f_{\mathrm{E}}$, $f_{\mathrm{A}}^i$ and $f_{\mathrm{F}}^i$ 8.      end for 9.      Compute the $f_c^i$ from $F^i$ 10.    Calculate the $L_{\mathrm{E}}$ and $L_{\mathrm{S}}$ from $(f_{\mathrm{E}},G_{\mathrm{E}})$ and $(f_c^i,G_{\mathrm{S}})$ 11.    $L=L_{\mathrm{E}}+L_{\mathrm{S}}$ 12.    Update $\theta$ using Adam algorithm 13. end for

4. Experiments

In this section, we conduct experiments to evaluate the performance of our proposed method. First, the experimental setup is outlined, in terms of datasets, evaluation metrics, and parameter settings. Next, our proposed method is compared with 8 state-of-the-art light-field SOD methods, with respect to quantitative evaluation and qualitative evaluation. At last, different edge feature extraction approaches are compared to verify the effectiveness of the EFEM.

4.1. Experimental Setup

4.1.1. Datasets

To evaluate our method, we conduct experiments on three public benchmark datasets: LFSD [14], HFUT [23], and DUT-LF [19]. LFSD is the first light-field saliency dataset consisting of 100 light-fields captured by a Lytro camera. The light-fields in this dataset typically contain only one single large-sized object with simple background. HFUT consists of 255 light-fields captured by a Lytro camera, most of which contain multiple objects or complex backgrounds. DUTLF-FS consists of 1462 light-fields acquired by a Lytro Illum camera. Compared to LFSD and HFUT, DUT-LF includes more complex scenes such as low light, low contrast, and clutter background with higher light-field quality. Similar to recent work [20,21], we chose 1000 samples from DUT-LF and 100 samples from HFUT to train our proposed model. The remaining samples and LFSD are used for testing.

4.1.2. Evaluation Metrics

We adopt four widely-used metrics to comprehensively evaluate the performance of our proposed method, including F-measure ($F_{\beta }$) [39], Mean Absolute Error ($\mathrm{MAE}$) [40], S-measure ($S_{\alpha }$) [41] and E-measure ($E_{\varphi }$) [42]. They are widely accepted in light-field SOD methods [19,20,21].

4.1.3. Parameter Settings

Our network is implemented on Pytorch framework and optimized by Adam algorithm. The two sets of convolutional blocks used for saliency feature extraction are initialized with corresponding pre-trained ResNet-50 network. The parameters of the newly added convolution layer are initialized with Gaussian kernels, and the biases are initialized to 0. In the training phase, batch size is set to 6, learning rate is initialized with $5\times {10}^{-6}$ and then reduced by 1/10 after 60 epochs. It takes 160 epochs for the whole training procedure. We do not use the validation dataset during training, and it takes about 7 h to train our network on an RTX 3090 GPU.

4.2. Comparison with the State-Of-The-Art Methods

We compare our proposed method with 8 previous state-of-the-art methods, including 5 deep learning-based methods: PANet [21], ERNet [20], MoLF [19], LFNet [17], MAC [22], and 3 traditional methods: WSC [15], DILF [16], LFS [14]. To ensure fairness, all saliency results from competing methods are produced by running the source codes or pre-computed by the authors.

4.2.1. Quantitative Evaluation

As shown in

Table 1. Quantitative comparisons on three light-field datasets. The best results are marked in bold. ↑ and ↓ denote larger and smaller is better respectively.

Methods	DUT-LF [19]				HFUT [23]				LFSD [14]
	${\mathit{F}}_{\mathit{\beta }}\uparrow$	$\mathbf{MAE}\downarrow$	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{E}}_{\mathit{\varphi }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	$\mathbf{MAE}\downarrow$	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{E}}_{\mathit{\varphi }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	$\mathbf{MAE}\downarrow$	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{E}}_{\mathit{\varphi }}\uparrow$
Ours	0.911	0.034	0.917	0.947	0.733	0.081	0.805	0.833	0.825	0.078	0.847	0.876
PANet [21]	0.892	0.039	0.908	0.932	0.727	0.074	0.795	0.831	0.820	0.081	0.835	0.871
ERNet [20]	0.891	0.040	0.899	0.943	0.709	0.082	0.778	0.832	0.829	0.083	0.830	0.879
MoLF [19]	0.855	0.052	0.887	0.921	0.639	0.095	0.742	0.790	0.800	0.092	0.825	0.864
LFNet [17]	0.843	0.054	0.878	0.912	0.628	0.093	0.736	0.777	0.794	0.092	0.821	0.867
MAC [22]	0.746	0.103	0.804	0.806	0.620	0.108	0.732	0.733	0.753	0.118	0.789	0.790
WSC [15]	0.505	0.181	0.589	0.705	0.487	0.155	0.609	0.677	0.716	0.152	0.699	0.748
DILF [14]	0.450	0.181	0.624	0.606	0.513	0.152	0.673	0.656	0.685	0.153	0.787	0.738
LFS [14]	0.333	0.261	0.556	0.515	0.306	0.229	0.560	0.515	0.497	0.209	0.671	0.560

, we evaluate and compare our proposed method with other light-field SOD methods in terms of F-measure, MAE, S-measure, and E-measure. It can be seen that our proposed method achieves the highest scores on DUT-LF, HFUT, and LFSD datasets almost across four evaluation metrics. In particular, our proposed method far outperforms the state-of-the-art methods on DUT-LF, which is the largest and most challenging dataset (2.1% and 12.8% improvements in F-Measure and MAE). Our method performs worse than other algorithms in some measures. This is because some light-field samples in the HFUT and LFSD datasets provide only few focal slices, and our method has poor generalization ability for these light-field samples. In addition to numerical comparison, we also plot the precision-recall curves of all methods over three datasets as shown in Figure 4. It can be seen that our proposed method which is denoted by solid red line performs the best results on three benchmark datasets.

4.2.2. Qualitative Evaluation

Figure 5 shows some representative visualization results by comparing our proposed method with other state-of-the-art light-field SOD methods. It can be noticed that our proposed method can locate and segment salient objects more accurately in various complex scenes. For example, low contrast (rows 1, 2), clutter background (rows 3, 4), multiple objects (rows 5, 6), small objects (row 7), and transparent objects (row 8). Notably, thanks to the effective utilization of edge information, salient objects detected by our proposed method retain detailed edge details.

4.3. Comparison of Edge Feature Extraction Approaches

We compare different edge feature extraction methods in terms of both feature extractors and extraction sources to verify the effectiveness of EFEM. First, we remove EFEM from the network shown in Figure 1 to build the baseline network. Then, we implement experiments on different edge feature extraction approaches.

4.3.1. Comparison of Edge Feature Extractors

We compare the effect of a convolutional layer and our proposed EFEM on the network. The quantitative results are shown in

Table 2. Quantitative results of different edge feature extraction methods. (a) denotes the baseline network with no edge features extracted. (b) represents using a convolutional layer to extract edge features from all-in-focus (AiF) images. (c) means using EFEM to extract edge features from focus stacks (FS). (d) denotes using EFEM to extract edge features from AiF images.

Indexes	Methods	DUT-LF [19]				HFUT [23]				LFSD [14]
		${\mathit{F}}_{\mathit{\beta }}\uparrow$	$\mathbf{MAE}\downarrow$	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{E}}_{\mathit{\varphi }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	$\mathbf{MAE}\downarrow$	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{E}}_{\mathit{\varphi }}\uparrow$	${\mathit{F}}_{\mathit{\beta }}\uparrow$	$\mathbf{MAE}\downarrow$	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{E}}_{\mathit{\varphi }}\uparrow$
(a)	Baseline	0.896	0.042	0.902	0.932	0.716	0.091	0.788	0.819	0.807	0.091	0.826	0.853
(b)	(a) + Conv (AiF)	0.902	0.039	0.914	0.937	0.721	0.087	0.796	0.826	0.811	0.083	0.835	0.867
(c)	(a) + EFEM (FS)	0.905	0.037	0.916	0.941	0.725	0.084	0.799	0.826	0.823	0.081	0.843	0.872
(d)	(a) + EFEM (AiF)	0.911	0.034	0.917	0.947	0.733	0.081	0.805	0.833	0.825	0.078	0.847	0.876

(b) and (d), it can be seen that EFEM presents better performance on all datasets (12.8%, 7.1%, and 6.0% improvement in MAE). The qualitative results are shown in Figure 6b,d, which demonstrate that EFEM is more favorable to improve the edge details of salient objects.

4.3.2. Comparison of Edge Feature Extraction Sources

We compare the impact of extracting edge features from all-in-focus images (AiF) and focus stacks (FS). The quantitative results are shown in Table 2 (c) and (d), it is consistent with our assertion that all-in-focus images are more beneficial for high-quality edge feature extraction. Significant improvement between them can be visualized in Figure 6c,d. These results suggest that more effective edge features can be extracted from all-in-focus images compared to the focus stacks.

It is worth noting that different edge feature extraction methods all improve the network performance compared to no edge feature extraction. This indicates that edge information is extremely important for light-field SOD.

5. Discussion

Although our proposed method significantly improves the detection accuracy, it still has shortcomings. In

Table 3. Comparison of model complexity.

Methods	Size (M) ↓	FPS (frames/s) ↑	DUT-LF [19]	HFUT [23]	LFSD [14]
			${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{S}}_{\mathit{\alpha }}\uparrow$	${\mathit{S}}_{\mathit{\alpha }}\uparrow$
Ours	298.7	26.4	0.917	0.805	0.847
PANet [21]	60.4	11.4	0.908	0.795	0.835
ERNet [20]	88.2	26.1	0.899	0.778	0.830
MoLF [19]	177.8	23.8	0.887	0.742	0.825

, we compare the model complexity with several representative methods. It can be seen that our proposed method bears little improvement in inference speed (only 0.3 improvement over ERNet in FPS) and the size of the model is much larger than other methods. This limits further applications. We argue that adopting some lightweight models can alleviate this problem, such as MobileNet [43] and ShuffleNet [44], which can effectively reduce the computational complexity by optimizing the network structure.

Existing light-field SOD methods are mainly used in 3D imaging, 3D display and light-field image compression. Our proposed method can accurately locate the key contents in scenes for subsequent visual processing. However, there are some other applications that deserve to be explored, such as transparent object segmentation and defect detection. The rich scene information in light-field data can provide useful clues.

6. Conclusions

In this paper, we propose a novel light-field SOD method to solve the loss of edge details in existing methods. It hierarchically leverages light-field features to reduce information loss. Specifically, we employ two sets of convolutional blocks based on the ResNet-50 network to extract multi-level saliency features. For low-level saliency features, we propose an edge feature extraction module to extract high-quality edge features from them. For high-level saliency features, we employ the MFM to fuse them and edge features to take full advantage of both saliency information and edge information. Finally, we use the MJDM to detect salient objects from fused features. Different from other methods, we adopt a dual supervision strategy to guide the generation of edge information and saliency information. Experiments demonstrate that our method can detect salient objects with fine edge details even in complex scenes such as clutter background, low light, and low contrast. Comprehensive experiments on three benchmark datasets show that our method outperforms eight state-of-the-art methods quantitatively and qualitatively.