Image Segmentation from Sparse Decomposition with a Pretrained Object-Detection Network

Abstract

Annotations for image segmentation are expensive and time-consuming. In contrast to image segmentation, the task of object detection is in general easier in terms of the acquisition of labeled training data and the design of training models. In this paper, we combine the idea of unsupervised learning and a pretrained object-detection network to perform image segmentation, without using expensive segmentation labels. Specially, we designed a pretext task based on the sparse decomposition of object instances in videos to obtain the segmentation mask of the objects, which benefits from the sparsity of image instances and the inter-frame structure of videos. To improve the accuracy of identifying the ’right’ object, we used a pretrained object-detection network to provide the location information of the object instances, and propose an Object Location Segmentation (OLSeg) model of three branches with bounding box prior. The model is trained from videos and is able to capture the foreground, background and segmentation mask in a single image. The performance gain benefits from the sparsity of object instances (the foreground and background in our experiments) and the provided location information (bounding box prior), which work together to produce a comprehensive and robust visual representation for the input. The experimental results demonstrate that the proposed model boosts the performance effectively on various image segmentation benchmarks.

1. Introduction

Supervised learning has been successfully applied to many complex computer vision tasks [1,2]. However, training deep neural networks requires a tremendous amount of manual labeling and prior knowledge from experts. Especially in image segmentation tasks, the cost of pixel-level annotations acquisition is significantly larger than that of region-level and image-level labels, respectively [3]. This motivates the development of weakly supervised image segmentation methods [4]. Different forms of supervision are explored to promote the performance of image segmentation [5,6].

Previous works [7,8] study the forms of supervision to solve the image segmentation problems. Co-segmentation regards a set of images with the same category as the supervision information and segments the common objects. The initial works are mainly based on effective computational models [9,10,11] to process an image pair. More recent co-segmentation methods are formulated as modeling optimization [12], object saliency [13], or data clustering [14,15,16] problems in more than two images. However, the existing methods still suffer from certain limitations including hand-crafted features and unscalable prior [17]. Moreover, these methods rely on the collections of images to segment the objects and fail to segment in a single image. Recent works [18,19] have demonstrated that the pretrained network could provide powerful semantic representation for a given image. Therefore, it is necessary to explore the potential of exploiting the pretrained networks: networks trained for one task, but that could be reused in scenarios that are different from their original purposes [20].

Self-supervised learning aims at designing various pretext tasks to assist the main task in exploring the properties of unlabeled data [21,22,23,24,25]. The methods [26,27] use the generative models to reconstruct inputs. However, most methods [21,22,23,26,27] are mainly based on static images to train the models. Video sequences give rise to characteristic patterns of visual change including geometric shape, motion tendencies and many other properties. Natural videos could serve as a more powerful signal for both static and dynamic visual tasks. Self-supervised methods based on videos [24,25] utilize optical flow estimation to find correspondences between pixels across video frames. All moving objects can result in optical flow; therefore, these methods cannot perceive the objects of interest in the scene.

To address these limitations, we aimed to design a pretext task based on the sparse decomposition of object instances in videos, and use the encoder–decoder structures to reconstruct the input to promote the image segmentation tasks. Our motivation is based on two observations. (1) The sparsity of image instances: An image typically consists of multiple objects instances (such as the foreground and background). The objects can be sparsely represented by a deep neural network to learn the features. (2) The inter-frame structure of videos. Compared with static images, the objects instances between continuous frames have high correlation and can be sparsely represented. Video data is more common and conducive to visual learning. We explore the two ideas and utilize the sparsity of image instances and the inter-frame structure of videos.

In this paper, we propose an Object Location Segmentation (OLSeg) model of three branches with bounding box prior. The model is trained from videos and is able to capture the foreground, background and segmentation mask in a single image. We consider a relatively simple scenario where each image consists of a foreground and a background, and construct the model based on the following three aspects. (1) The first of these is the foreground and background branches. On the one hand, we use an autoencoder [28] in the foreground and background branches to construct the foreground and background respectively. The encoder in the foreground branch outputs more channels than the encoder in the background branch to express more complex motion information of the foreground. On the other hand, we apply a gradient loss to smooth the background, which avoids the appearance of the foreground object. (2) The second is the mask branch. We use a U-Net [29] to generate the mask, and adopt an object loss to focus on the information in the bounding box of the foreground object. Our motivation is that the segmentation mask can be calculated if the object location is given. The location information is obtained by a pretrained object-detection network [30]. In addition, we consider a closed loss to ensure that the mask shows smooth contours without holes, and a binary loss to generate a binary mask. (3) The final aspect is image reconstruction. The original image is reconstructed by combining the foreground and background with the binary mask.

We conclude our contributions as follows:

• We designed a pretext task based on the sparse decomposition of object instances in videos for image segmentation. The task benefits from the sparsity of image instances and the inter-frame structure of videos;
• We propose an OLSeg model of three branches with bounding box prior. The location information of the object is obtained by a pretrained object-detection network. The model trained from videos is able to capture the foreground, background and segmentation mask in a single image;
• The proposed UnsupRL model is demonstrated to boost the performance effectively on various image segmentation benchmarks. The ablation study shows the gains of different components in OLSeg.

2. Related Work

2.1. Image Segmentation from Unlabeled Data

Image segmentation from unlabeled data is challenging due to the fact that it does not have the pixel-level annotations rather than a given unlabeled image. Recent methods [31,32] explore different forms of supervision to promote the segmentation performance. Stretcu et al. [33] matched multiple video frames for image segmentation. Papazoglou et al. [34] relied on the optical flow to identify moving objects. Koh et al. [35] iteratively generated the object proposals. Wang et al. [36] focused on the relations between pixels across different images. Zhou et al. [37] used the optical flow and attention mechanism to segment the video objects; image co-segmentation requires a set of images containing objects from the same category as a weak form of supervision. Rother et al. [38] minimised an energy function to segment image pairs. Kim et al. [12] explored an optimization problem of saliency to find the common object in multiple images. Joulin et al. [14] formulated the co-segmentation problem in terms of a discriminative clustering task. Joulin et al. [15] used spectral and discriminative clustering for fully unsupervised segmentation. Rubinstein et al. [13] combined a visual saliency and dense correspondences to capture the sparsity and visual variability of the common objects in a group of images. Quan et al. [16] used a pretrained network to obtain the semantic features, and proposed a manifold ranking method to discover the common objects. Zhao et al. [17] constrained the proportion of foreground object in the image. These methods segmented images based on hand-crafted features and unscalable prior. Our model does not depend on any assumptions about the existence of the common objects, and is able to segment in a single image.

2.2. Pretrained Networks

Current deep learning methods have achieved great success on a variety of visual tasks [39,40]. However, these deep frameworks still heavily rely on a large amount of training data and a time-consuming training process. Therefore, some researchers have recently turned to a new direction: network reuse [20]. Obviously it would be appealing if a pretrained network can be reused in a new domain that is different from its initial training purpose, and without needing to fine-tune it. Some recent works attempt to train deep models on a large image dataset collecting from web images [41], which can further leverage the generalization of pretrained networks. Fortunately, benefiting from the large-scale dataset COCO [3], which consists of over 330000 images in over 80 categories, the pretrained networks have revealed powerful object detection ability. Inspired by [42], our proposed OLSeg extracts the location of the foreground object from a pretrained object-detection network without any training or fine-tuning process. This operation can be easily implemented and is conducive to object segmentation.

2.3. Self-Supervised Learning

Self-supervised learning methods usually construct a pretext task to learn features from raw data, and improve the performance of the main task. Some methods use generation-based models [43,44,45] to obtain the latent feature representations of the input. Various pretext tasks have been explored, e.g., predicting relative patch locations within an image [21], recovering part of the data [46], solving jigsaw puzzles [22], colorizing grayscale images [23], counting visual primitives [47] and predicting image rotations [48]. For videos, self-supervised signals come from motion consistency [24,25] and temporal continuity [49,50]. However, the pretext tasks based on videos estimate the optical flow as the clue of the moving object. The object of interest may be stationary and cannot be perceived through optical flow. We design a pretext task based on the sparse decomposition of object instances in videos, and use the encoder–decoder structures to reconstruct the original input.

3. Object Location Segmentation (OLSeg)

We propose a three-branch OLSeg model with bounding box prior, as shown in Figure 1. The model consists of a foreground branch, a background branch and a mask branch. For an input image from continuous video frames, the foreground and background branches construct the foreground and background information with an autoencoder [28], respectively. The output channels of encoder in the two autoencoders are different. The foreground branch contains more encoder channels than the background branch to represent more complex foreground information. The gradient loss in the background branch smoothes the background and eliminates the influence of foreground. The mask branch with bounding box prior uses an U-Net [29] to generate the segmentation mask. The bounding box of the object is obtained from a pretrained object-detection network Yolov4 [30] acting on the input. The location information is used in an object loss to make the U-Net [29] focus more on the area where the foreground object appears. The closed loss and binary loss ensure that the generated mask is binary without concave holes. The mask is combined with the foreground and background to reconstruct the input via a reconstruction loss. The contributions of the three branches are complementary to each other for final decomposition of the object instances. For clarity, the pseudocode for OLSeg is shown in Algorithm 1.

3.1. The Foreground and Background Branches

The foreground and background branches aim to decompose the object instances into the foreground and background. The foreground branch consists of an autoencoder [28] $f_{a_1}$, and the background branch consists of an autoencoder [28] $f_{a_2}$. The two autoencoders [28] are trained separately. Let $U=(u_k;k\in (1,\dots ,K))$ be continuous video frames. Given an input image $u_k$, the ouputs of $f_{a_1}$ and $f_{a_2}$ are obtained using Equations (1) and (2) respectively:

$$F_k=f_{a_1}\left(u_k\right),$$

(1)

$$B_k=f_{a_2}\left(u_k\right),$$

(2)

where $F_k$ and $B_k$ are the foreground and background of the input image $u_k$ respectively.

Algorithm 1: Training OLSeg.
Input:  An autoencoder $f_{a_1}$, an autoencoder $f_{a_2}$, an U-Net $f_u$ and a pretrained Yolov4 $f_p$. Batch of continuous video frames $U=(u_k;k\in (1,\cdots ,K))$, coordinates $(i,j)$, numbers of coordinates $N_{mk}$ and $N_{ak}$ in $M_k$ and $A_k$. The encoder channels $C_f, C_b$, and the loss balancing hyperparameters $\alpha , {\beta }_o, {\beta }_c, {\beta }_b$.
1: for $k=1$ to K do
2:   $F_k=f_{a_1}\left(u_k\right)$	▹ Foreground $F_k$ with the encoder channels $C_f$
3:   $B_k=f_{a_2}\left(u_k\right)$	▹ Background $B_k$ with the encoder channels $C_b$
4:   $M_k=f_u\left(u_k\right)$	▹ Segmentation mask $M_k$
5:   $A_k=f_p\left(u_k\right)$	▹ Area of the foreground object $A_k$
6:   $R_k=M_k^b\times F_k+(1-M_k^b)\times B_k$	▹ Reconstruction image $R_k$ and binary mask $M_k^b$
7: end for
8: $L_g=\frac{1}{\left|U\right|}{\sum }_{k=1}^K\nabla B_k$	▹ Gradient loss
9: $L_o=\frac{1}{\left|U\right|}{\sum }_{k=1}^K{\sum }_{(i,j)\notin A_k}^{N_{mk}-N_{rk}}{|M_{k,(i,j)}-0|}^2$	▹ Object loss
10: $$\begin{gathered} L_c=\frac{1}{4\left|U\right|}{\sum }_{k=1}^K{\sum }_{(i,j)\in M_k}^{N_{mk}}\left[\right|M_{k,(i,j)}-M_{k,(i-1,j)}{|}^2+{|M_{k,(i,j)}-M_{k,(i+1,j)}|}^2 \\ +|M_{k,(i,j)}-M_{k,(i,j-1)}{|}^2+{|M_{k,(i,j)}-M_{k,(i,j+1)}|}^2] \end{gathered}$$	▹ Closed loss
11: $L_b=-\frac{1}{\left|U\right|}{\sum }_{k=1}^K{\sum }_{(i,j)\in M_k}^{N_{mk}}{|M_{k,(i,j)}-0.5|}^2$	▹ Binary loss
12: $L_m={\beta }_o{L}_o+{\beta }_c{L}_c+{\beta }_b{L}_b$	▹ Loss of the mask branch
13: $L_r=\frac{1}{\left|U\right|}{\sum }_{k=1}^K{|u_k-R_k|}^2$	▹ Reconstruction loss
14: $L=L_r+\alpha L_g+L_m$	▹ Overall loss of OLSeg
Output: Segmentation network $f_u$.

The difference between $f_{a_1}$ and $f_{a_2}$ is that they have different output channels of encoders. Generally, the foreground information is more complex and variable than the background information. As shown in Figure 1, the horse is moving and the background is relatively simple in continuous video frames. Let $C_f$ and $C_b$ be the output channels of encoders in $f_{a_1}$ and $f_{a_2}$, respectively. We use more output channels $C_f$ to express the foreground information.

The key to realizing the foreground and background separation is to ensure that the background does not contain the foreground information. The powerful generation ability of autoencoders [28] leads to the appearance of foreground in the background. In order to ensure that the background is smooth and clean, we add a gradient loss to the background branch as Equation (3):

$$L_g=\frac{1}{\left|U\right|}\sum _{k=1}^K\nabla B_k,$$

(3)

where ∇ is the operation of minimizing the gradient.

Although the foreground and background extracted from $f_{a_1}$ and $f_{a_2}$ are blurred in Figure 1, the outline of the horse is visible in the foreground, and the sky and grass are clean in the background. This proves the decomposition ability of the foreground and background branches to the object instances.

3.2. The Mask Branch

The mask branch consists of an U-Net [29] $f_u$ to generate the segmentation mask. Multi-scale connections between the encoding and decoding paths of U-Net [29] ensure efficient integration of information in image segmentation. The output of U-Net [29] for the input $u_k$ is represented as Equation (4):

$$M_k=f_u\left(u_k\right),$$

(4)

where $M_k$ is the segmentation mask and the values range from 0 to 1.

It is challenging for the mask branch to distinguish the foreground and background of the input. The previous methods [24,25] use optical flow to treat the moving object as the foreground. However, the foreground may be relatively static in continuous video frames. We consider the location information of the foreground object to assist the mask branch to better discover the object. Our motivation is that the segmentation mask could be obtained if the object location is known. The operation of extracting object location information is shown in Figure 2. For the given input $u_k$, we select a Yolov4 [30] trained on the COCO dataset [3] to obtain the bounding box of the object area. The comparison of different pretrained object-detection networks is not considered because we only need to obtain the approximate location of the object. Then the corresponding location of the bounding box in the input is mapped to the segmentation mask. The object area in the bounding box for the segmentation mask is represented as $A_k$ ($A_k$ is their union if there are multiple object areas). Let the value of coordinates $(i,j)$ in the segmentation mask $M_k$ be $M_{k,(i,j)}$, and we introduce an object loss as Equation (5):

$$L_o=\frac{1}{\left|U\right|}\sum _{k=1}^K\sum _{(i,j)\notin A_k}^{N_{mk}-N_{rk}}{|M_{k,(i,j)}-0|}^2,$$

(5)

where $N_{mk}$ and $N_{rk}$ represent the numbers of coordinates in $M_k$ and $A_k$ respectively. For the segmentation mask $M_k$, we set the values outside $A_k$ to 0. The mask branch focuses on the object area inside the bounding box to more accurately obtain the mask.

The segmentation mask sometimes shows unclosed concave holes. Small holes inside the object have an adverse impact on segmentation. We adopt a closed loss to produce a smooth mask without concave holes. The idea is that the value of the current coordinates $(i,j)$ in the mask $M_k$ is consistent with the values of the surrounding adjacent coordinates. The closed loss $L_c$ is calculated as Equation (6):

$$\begin{array}{cc}\hfill L_c=\frac{1}{4\left|U\right|}\sum _{k=1}^K\sum _{(i,j)\in M_k}^{N_{mk}}& \left[\right|M_{k,(i,j)}-M_{k,(i-1,j)}{|}^2\hfill \\ \hfill & +|M_{k,(i,j)}-M_{k,(i+1,j)}{|}^2\hfill \\ \hfill & +|M_{k,(i,j)}-M_{k,(i,j-1)}{|}^2\hfill \\ \hfill & +|M_{k,(i,j)}-M_{k,(i,j+1)}{|}^2]\hfill \end{array}$$

(6)

The reconstruction image is obtained by the combination of the foreground, background and mask. The values of the segmentation mask $M_k$ range from 0 to 1. The low-entropy prediction of the mask branch can ensure that either the foreground or the background appears in the reconstruction image, rather than both of them appearing simultaneously. We minimize a binary loss to achieve entropy minimization as Equation (7):

$$L_b=-\frac{1}{\left|U\right|}\sum _{k=1}^K\sum _{(i,j)\in M_k}^{N_{mk}}{|M_{k,(i,j)}-0.5|}^2,$$

(7)

where $M_{k,(i,j)}$ is constrained to be close to 0 or 1.

The loss of the mask branch is thus defined as Equation (8):

$$L_m={\beta }_o{L}_o+{\beta }_c{L}_c+{\beta }_b{L}_b,$$

(8)

where ${\beta }_o$, ${\beta }_c$ and ${\beta }_b$ are hyperparameters to control the scales of $L_o$, $L_c$ and $L_b$ respectively.

3.3. The Overall Loss

We reconstruct the input image by combining the outputs of the foreground, background and mask branches. The reconstruction image $R_k$ of $u_k$ is represented as Equation (9):

$$R_k=M_k^b\times F_k+(1-M_k^b)\times B_k,$$

(9)

where $M_k^b$ is the binary mask obtained from the mask branch, and the values are close to 0 or 1.

We minimize the reconstruction loss using Equation (10):

$$L_r=\frac{1}{\left|U\right|}\sum _{k=1}^K{|u_k-R_k|}^2$$

(10)

The overall loss L of OLSeg is described as Equation (11):

$$L=L_r+\alpha L_g+L_m,$$

(11)

where $\alpha$ is the hyperparameter to balance the loss.

4. Experiments

In this section, we conduct extensive experiments to verify the performance of the proposed OLSeg model. We first introduce the implementation details in Section 4.1. Subsequently, we study the effect of parameter selection in Section 4.2. Next, we evaluate OLSeg for the image segmentation tasks in Section 4.3 and Section 4.4 respectively. Finally, we show the ablation study in Section 4.5.

4.1. Implementation Details

4.1.1. Datasets

• The YouTube Objects dataset [51] is a large-scale dataset that includes 10 types of objects (e.g., airplane, bird, boat, car, cat, cow, dog, horse, motorbike, and train) downloaded from the YouTube website. It contains 5484 videos and a total of 571,089 video frames. The video consists of the object entering and leaving the line of sight, the object being occluded and the significant changes in the object scale and visual angle. The dataset provides ground-truth bounding boxes on the object of interest in one frame for each of 1407 video shots as the validation set.
• The Internet dataset [13] is a commonly used object segmentation dataset. There are 15,000 images downloaded from the internet. The dataset contains 4542 images of airplanes, 4347 images of cars and 6381 images of horses with the high-quality annotation masks.
• The Microsoft Research Cambridge (MSRC) dataset [52] contains 14 object classes and about 420 images with accurate pixel-wise labels. Each object in the dataset has different background, illumination and pose. This is a real-world dataset, which is often used to evaluate the image segmentation tasks.

4.1.2. Training Details

The PyTorch framework is adopted on a single GPU machine with NVIDIA TITAN V in all experiments. We train our proposed model on the YouTube Objects dataset [51], and select the appropriate parameters on the validation set. The inputs of the training phase are continuous video frames to consider the motion clue. The foreground and background branches have the same network structure. We use ResNet18 [53] as the encoder, and 4 deconvolution and convolution operations followed by 3 convolution layers as the decoder [54]. For the mask branch, we use a 5-layer U-Net [29], and each layer contains 2 convolution operations.

Table 1. The hyperparameter settings on the YouTube Objects dataset.

Parameters	Values
Size of input	128 × 128
Size of output	128 × 128
Optimizer	Adam
Learning rate	0.00001
Epochs	10
Batch size	16
Output channels $C_f$ of encoder in the foreground branch	128
Output channels $C_b$ of encoder in the background branch	32
Hyperparameter $\alpha$	1.5
Hyperparameter ${\beta }_o$	0.15
Hyperparameter ${\beta }_c$	0.05
Hyperparameter ${\beta }_b$	1

shows the the hyperparameter settings on the YouTube Objects dataset [51]. The encoders output different channels in the foreground and background branches. The trainable parameters in the foreground, background and mask networks are about $1.8\times {10}^7,1.7\times {10}^7$ and $1.3\times {10}^7$, respectively. The floating point operations (FLOPs) of the proposed model are about $1.8\times {10}^{10}$. The total training time takes about 19 h.

4.1.3. Evaluation Metrics

For the image segmentation task, we use the P and J metrics as in [13]. The P refers to the ratio of the correctly labeled pixels. The J is the Jaccard similarity, which represents the intersection over union of the prediction and the ground truth. Higher values of P and J indicate better model performance. We also adopt the correct localization (CorLoc) metric following previous image segmentation works [33,34], which measures the percentage of images that are correctly localized according to the PASCAL criterion: the intersection over union (IoU) overlap ratio of the predicted box and the ground-truth box is greater than 0.5. These metrics are commonly used for image segmentation evaluation.

4.2. Parameter Selection

We conduct detailed parameter selection experiments on the validation set of the YouTube Objects dataset [51] to study the influence of hyperparameters.

4.2.1. Encoder Output Channels in the Foreground and Background Branches

We design different output channels for the encoders in the foreground and background branches based on the complexity of the foreground and background information. The foreground branch needs more encoder output channels to deal with the changeable foreground. The experimental results of different channel combinations in the foreground and background branches are shown in Figure 3. The recovered background still contains some foreground pixels when the output channels of both encoders are 64. We further increase the channels in the foreground branch, and reduce the channels in the background branch. The appropriate channel combination of $C_f=128, C_b=32$ obtains a clear and clean foreground and background.

4.2.2. Hyperparameter $\alpha$ for the Gradient Loss

The separation of the foreground and background is important for reconstructing the input. The model is able to learn powerful feature representation only when the foreground and background are completely separated. We assign a hyperparameter $\alpha$ to balance the gradient loss in the background branch. Figure 4 shows the experimental results of different $\alpha$ for the gradient loss. The foreground information appears in the background when $\alpha$ is 0.5. We get a cleaner background with the increase of $\alpha$, which is also conducive to improving the quality of segmentation mask. The hyperparameter $\alpha =1.5$ is used for subsequent experiments.

4.2.3. Hyperparameter ${\beta }_o$ for the Object Loss

The object loss uses a priori knowledge of object location in the mask branch. The hyperparameter ${\beta }_o$ controls the scale of the object loss, and affects the quality of the segmentation mask. The experimental results of different ${\beta }_o$ for the object loss are shown in Figure 5. For the mask, the object area in the red box is mapped from the original input. The information outside the red boxes cannot be completely removed when ${\beta }_o$ is 0.05. However, a larger ${\beta }_o$ may reduce the effect of other losses and lead to the unclear boundary of the object. We select ${\beta }_o$ as 0.15 to obtain an accurate mask.

4.2.4. Hyperparameter ${\beta }_c$ for the Closed Loss

The closed loss aims to promote the aggregation of the segmentation mask, and form a smooth object shape without concave holes. The hyperparameter ${\beta }_c$ controls the proportion of the closed loss. The experimental results of different ${\beta }_c$ for the closed loss are listed in Figure 6. The segmentation mask only retains the contour of the object and cannot form a closed area under a small value of ${\beta }_c$. The hyperparameter ${\beta }_c=0.1$ ensures that the mask is closed and has a clear boundary.

4.2.5. Hyperparameter ${\beta }_b$ for the Binary Loss

The binary loss makes the segmentation mask binary to great extent. The pixels at each location of the foreground and background will contribute to the reconstruction image if the mask is not binary, which is contrary to the purpose of separation. Figure 7 shows the experimental results of different ${\beta }_b$ for the binary loss. The experiment with ${\beta }_b=1$ achieved better results.

4.2.6. Results and Visualization

We evaluated OLSeg on the validation set of the YouTube Object dataset [51]. The dataset provides ground-truth bounding boxes on the object of interest. We used the metric CorLoc as in [33,34] for quantitative analysis. For the purpose of this evaluation, we automatically fitted a bounding box to the largest connected component in the pixel-level segmentation output by our model. We compared the proposed model with the object segmentation methods [33,34,35] in

Table 2. Comparisions of CorLoc (%) on the YouTube Object dataset. Time (s) denotes the processing speed of each image.

Methods	Airplane	Bird	Boat	Car	Cat	Cow	Dog	Horse	Mbike	Train	Avg	Time (s)
Stretcu et al. [33]	38.3	62.5	51.1	54.9	64.3	52.9	44.3	43.8	41.9	45.8	49.9	6.9
Papazoglou et al. [34]	65.4	67.3	38.9	65.2	46.3	40.2	65.3	48.4	39.0	25.0	50.1	4
Koh et al. [35]	64.3	63.2	73.3	68.9	44.4	62.5	71.4	52.3	78.6	23.1	60.2	N/A
OLSeg	70.9	60.8	74.7	61.3	65.5	64.1	66.3	56.7	45.1	48.3	61.4	0.03

. OLSeg outperformed the others in 6 out of 10 classes and achieved the best overall performance. However, the results for birds showed a poor performance due to the lack of constraints to account for background complexity. The foreground we defined may also contain multiple instances. For motorbikes, masks of all instances in the images were obtained, such as humans and motorbikes. In addition, we show the test complexity of these methods. The proposed model processed each image in 0.03 s, which is faster than other methods. The reason is that our model only needs to forward the U-Net [29] to obtain the segmentation mask, without relying on multiple images or complex operations. Our model takes much longer during its training phase and requires a large amount of training data.

Figure 8 shows the visualization results on the validation set of the YouTube Objects dataset [51]. OLSeg extracted the segmentation mask, foreground and background from 10 classes. The generated background is smooth and contains little foreground information. We acquired the clear segmentation boundary of the mask. Multiple objects could also be accurately segmented, such as the cows and horses. However, as shown in the third row on the right of Figure 8, it is difficult to extract the mask when the color of the horse is similar to the background color. The proposed model produces high-quality segmentation results in multiple classes.

4.3. Evaluation on the Internet Dataset

We ran the trained OLSeg to evaluate the performance of image segmentation on the Internet dataset [13]. We used two metrics, P (precision) and J (Jaccard similarity), and compared our results with five previously proposed co-segmentation methods [12,13,14,15,16]. The results of different methods are shown in

Table 3. Comparisons of P and J (%) on the Internet dataset. The P and J denote the precision and Jaccard similarity respectively.

Methods	Airplane		Car		Horse
	$\mathit{P}$	$\mathit{J}$	$\mathit{P}$	$\mathit{J}$	$\mathit{P}$	$\mathit{J}$
Joulin et al. [14]	49.25	15.36	58.70	37.15	63.84	30.16
Joulin et al. [15]	47.48	11.72	59.20	35.15	64.22	29.53
Kim et al. [12]	80.20	7.90	68.85	0.04	75.12	6.43
Rubinstein et al. [13]	88.04	55.81	85.38	64.42	82.81	51.65
Quan et al. [16]	91.00	56.30	88.50	66.80	89.30	58.10
OLSeg	91.87	61.50	90.23	68.45	89.56	58.72

. The performance on airplanes was slightly better than cars and horses. For the P of airplanes, cars and horses, OLSeg is 0.87%, 1.73% and 0.26% higher than the method in [16], respectively. The J obtained by OLSeg is also greatly improved. The comparison results show that our proposed model outperformed other methods on all three object classes.

We compared the proposed model with the methods in [12,13,14,15] to visualize the segmentation performance. The qualitative results of different methods on the Internet dataset [13] are shown in Figure 9. Compared with other methods, OLSeg achieves a clearer segmentation boundary. When the segmentation mask cannot be obtained by Kim et al. [12], the proposed model also acquires a more realistic mask. The proposed model has a powerful ability to adapt to the size of the foreground object. The airplane image with a complex background could also be well segmented. The visualization results show the effect of OLSeg.

4.4. Evaluation on the MSRC Dataset

We evaluated the proposed model on the MSRC dataset [52]. We show the metrics $\overline{P}$ (average precision) and $\overline{J}$ (average Jaccard similarity) of our model as well as five related segmentation methods [12,13,14,15,16] in

Table 4. Comparisons of $\overline{P}$ and $\overline{J}$ (%) on the MSRC dataset. The $\overline{P}$ and $\overline{J}$ denote the average precision and average Jaccard similarity, respectively.

MSRC	Joulin et al. [14]	Joulin et al. [15]	Kim et al. [12]	Rubinstein et al. [13]	Quan et al. [16]	OLSeg
$\overline{P}$	71.53	77.01	61.34	78.31	86.21	87.56
$\overline{J}$	45.27	50.97	34.48	56.69	63.32	65.85

. OLSeg achieves a $\overline{P}$ of 87.56% and a $\overline{J}$ of 65.85%, which are 1.35% and 2.53% better than the method in [16]. Our overall $\overline{P}$ and $\overline{J}$ have higher values than the compared methods. The comparisons from Table 4 demonstrate that the proposed model produces good segmentation results on multiple object classes.

We further report the qualitative results of different methods on the MSRC dataset [52] in Figure 10. The proposed model is able to accurately segment the foreground object despite the large variation in style, color, texture, scale and position. For the complex objects such as bikes, OLSeg can also segment their finer contours. The tree is not very distinctive from the background in terms of color, but it is still successfully segmented. The text inside the sign is well removed due to the closed loss. There is a clear visual improvement for OLSeg over other compared methods. The experimental results have demonstrated the generalization of the proposed model.

4.5. Ablation Study

We performed an ablation study of the proposed OLSeg on the Internet and MSRC datasets, and the results of $\overline{P}$ and $\overline{J}$ are shown in

Table 5. Ablation study of OLSeg on the Internet and MSRC datasets.

Methods	Internet		MSRC
	$\overline{\mathit{P}}$	$\overline{\mathit{J}}$	$\overline{\mathit{P}}$	$\overline{\mathit{J}}$
OLSeg	90.55	62.89	87.56	65.85
Without the gradient loss	83.12	54.67	78.73	56.29
Without the closed loss	84.62	56.02	80.84	58.23
Without the binary loss	86.45	59.13	84.01	62.72

. Specifically, the contributions of different components in OLSeg were investigated. We did not analyze the contribution of the object loss because it is decisive in mask generation. Without the object loss, the segmentation mask will be cluttered and show a poor performance to distinguish the foreground from background. We removed the gradient loss, the closed loss and the binary loss, respectively, and observed the effect on the final results. We can see that the gradient loss contributes most to the performance. The results are reasonable since a clean background cannot be recovered without the gradient loss, which affects the quality of the mask. The closed loss plays an important role in preventing the generation of the mask with concave holes and noise. The binary loss also contributes to the final results. The ablation study from Table 5 demonstrates the effectiveness of the proposed OLSeg model.

5. Conclusions

In this paper, we designed a pretext task of decomposing object instances in video for image segmentation, and propose an OLSeg model of three branches with bounding box prior. The pretext task benefits from the sparsity of image instances and the inter-frame structure of videos. The proposed model is trained from video and is able to capture the foreground, background and segmentation mask in a single image. The constraints in the foreground and background branches ensure the generation of the foreground and background. The mask branch uses the bounding box prior, and consists of multiple losses to produce an accurate segmentation mask for the object of interest. This is consistent with the assumption that segmentation mask could be obtained if the object location is known. The experimental results show our model achieves better segmentation performance than compared methods on various image segmentation datasets.

The parameters in OLSeg are roughly estimated by grid search in this work. In future work, how to use a more adaptive method for parameter optimization is worthy of further study. In addition, we will also explore more efficient networks and other constraints to obtain more robust mask for the object of interest in unlabeled data.