Unsupervised Object Segmentation Based on Bi-Partitioning Image Model Integrated with Classification

Abstract

The development of convolutional neural networks for deep learning has significantly contributed to image classification and segmentation areas. For high performance in supervised image segmentation, we need many ground-truth data. However, high costs are required to make these data, so unsupervised manners are actively being studied. The Mumford–Shah and Chan–Vese models are well-known unsupervised image segmentation models. However, the Mumford–Shah model and the Chan–Vese model cannot separate the foreground and background of the image because they are based on pixel intensities. In this paper, we propose a weakly supervised model for image segmentation based on the segmentation models (Mumford–Shah model and Chan–Vese model) and classification. The segmentation model (i.e., Mumford–Shah model or Chan–Vese model) is to find a base image mask for classification, and the classification network uses the mask from the segmentation models. With the classifcation network, the output mask of the segmentation model changes in the direction of increasing the performance of the classification network. In addition, the mask can distinguish the foreground and background of images naturally. Our experiment shows that our segmentation model, integrated with a classifier, can segment the input image to the foreground and the background only with the image’s class label, which is the image-level label.

1. Introduction

Presently, automatic image segmentation tasks are required to obtain accurate information about each region of an image because the number of images continues to surge. With the development of convolutional neural network (CNN) [1] in deep learning, there are many works for image segmentation. However, to achieve high performance of segmentation result in an supervised manner [2], the convolutional neural network (CNN) requires many ground truth data that show the area of the objects at the pixel level. Creating each of these data is cumbersome and requires much time and other resources. Some works solve image segmentation problems in unsupervised manners [3]. Mumford–Shah functional [4] and Chan–Vese algorithm [5,6,7] are well-known models for classical unsupervised image segmentation problems.

However, since the Mumford–Shah functional and Chan–Vese algorithm rely on pixel intensities and the objective functions are non-convex, these models cannot distinguish between the foreground and background of images, and the foreground and background results are changed depending on the initialized value of the networks’ weights. To solve this problem, we propose a segmentation model that is integrated with a classifier for image segmentation. In our work, the segmentation model, which solves the Chan–Vese algorithm or Mumford–Shah functional, is used for the region proposals. Since these two algorithms minimize their objective functions with a curve in a level-set method [8], which evolves the initial surface defined as a function by minimizing the energy function, these algorithms can detect the edges of the images. With this segmentation model, we can get more precise boundaries. However, as we mentioned above, the results of the Chan–Vese algorithm and Mumford–Shah functional cannot distinguish the foreground and the background of the image alone. Therefore, the regions divided by these segmentation model alone have no meaning. To solve this problem, we integrate the segmentation model with the classifier. With the classifier, the detected meaningless regions by the output mask of the segmentation model become foreground regions. Furthermore, we propose a simple loss that can distinguish the background more precisely. With this simple loss, we can obtain more meaningful foreground regions. One thing to note is that with our classification loss, the classifier can extract the foreground regions alone. However, the regions’ boundaries are not precise. Therefore, the segmentation model and the classifier work to complement each other.

For our experiment, we used the dog-and-cat dataset from kaggle [9] and PASCAL VOC 2012 [10] as a dataset. To this end, the contributions of this work are as follows:

• We integrate the segmentation model with a classifier to solve the image segmentation problem.
• We use the Mumford–Shah functional and the Chan–Vese algorithm for the segmentation model. Furthermore, with the segmentation model, we can achieve more accurate boundaries.
• For the classifier, we propose a loss function that can meaningfully distinguish between the background and the foreground.

In the remainder of this paper, we briefly present related works on image segmentation in Section 2 and then explain our network’s structure with loss in Section 3. Next, we show our segmentation results in Section 4. Finally, we summarize and conclude our paper in Section 5.

2. Related Works

2.1. Image Segmentation

Image segmentation refers to labeling each pixel of an image to a particular class, and it aims to separate a given image into several meaningful regions for more manageable analyses. It has been approached in classical manners such as those of Osher et al. [8] and Lloyd et al. [11]. However, with the development of convolutional neural networks (CNNs) [1] in deep learning, many studies solve the image segmentation problem through CNN. In this paper, we detect the foreground and the background of the input images by generating the binarized segmentation masks.

2.1.1. Classical Image Segmentation

The classical image segmentation methods are mostly based on mathematical or statistical methods. Tobias et al. [12] and Arifin et al. [13] use the characteristic histogram, Ma et al. [14] approach the segmentation problem with edge and boundary detection. Furthermore, classical variational methods solve the problem with clustering. These classical variational methods minimize their objective functions such the as Mumford–Shah functional [4].

2.1.2. Supervised Image Segmentation

To achieve high image segmentation performance in a supervised manner [2] with CNN-based models, UNet [15] uses skip connections between contracting and expansive parts. Furthermore, FCN [16] changes the fully connected layer to fully convoluntional network to preserve the location information. However, even though these methods have led to high performance, supervised learning has a considerable limitation: it must have ground truth for all training data.

2.1.3. Unsupervised Image Segmentation

To overcome the limitations of the supervised methods, some works solve the image segmentation problem with unsupervised methods [3] with CNN. These unsupervised learning methods segment images without any ground truth data. Usually, these unsupervised manners use the objective functions of the classical variational approach [4,5,6,7]. Similar to our work, Kim et al. [17] also minimizes the pixel-wise constant Mumford–Shah functional. However, the main difference is [17] uses pixel-level labels, but we use image-level labels for image segmentation for a weakly supervised method.

2.1.4. Weakly Supervised Image Segmentation

Although deep learning methods based on CNN can optimize the objective functions of the classical variational method, the results are not accurate because most of the objective functions are non-convex functions. Thus, many weakly supervised methods solve the image segmentation problem with much more simple objective functions and use given ground-truth class labels for segmentation. Zhou et al. [18] use global average pooling layers instead of fully connected layers with a classifier to visualize the most prominent part of the input image. Selvaraju et al. [19] uses the gradient to overcome the limitation of [18], such that [18] cannot be applied to the models without global average pooling layers. Lee et al. [20] extract the class activation map from a randomly selected feature map of classifier for their seed loss and combine it with the boundary loss, which consists of a conditional random field. More simply, Huang et al. [21] use only one class activation map for their seed loss. Wang et al. [22] construct a pixel correlation module to solve the problem that the class activation map is not consistent when the input resolutions change. References [20,21,22] use the class activation map of [18] for segmentation, but our work does not use class activation map. Instead, we directly extract the foreground regions by generating a binary mask from the input image, not from the feature maps of the classifier, which can enhance the classification confidence of the classifier. Similarly, Zolna et al. [23] generates a binary mask for input image, using classifiers to find all parts of the image that any classifier could use. Furthermore, Araslanov et al. [24] uses the classification loss to generate a mask and uses a local mask refinement module called PAMR to refine the mask. Unlike the method of elaborating the boundaries of segmentation areas detected by these works, we used the Chan–Vese model or the Mumford–Shah model.

2.2. Mumford–Shah Functional and Chan–Vese Algorithm

The Mumford–Shah functional [4] and Chan–Vese algorithm [5,6,7] are well-known approaches that solve the image segmentation problem in the classical method. These methods find optimal segmentation results, which minimize specific energy functions.

2.2.1. Chan–Vese Algorithm

The edge gradient-based energy function that [25,26,27] proposed cannot segment the images well when the edges are too smooth or the noises are too big. The Chan–Vese model [5] defines a region-based energy function based on Mumford–Shah functional [4]. The original Mumford–Shah functional is (1).

$$\begin{array}{cc}\hfill & F^{MS}(\mu ,C)\hfill \\ \hfill & =\mu ·Length\left(C\right)\hfill \\ \hfill & +{\lambda }_1{\int }_{\Omega }^{}{\left|u_0(x,y)-u(x,y)\right|}^{2}dxdy\hfill \\ \hfill & +{\int }_{\Omega /C}^{}{\left|\nabla u(x,y)\right|}^{2}dxdy\hfill \end{array}$$

(1)

The energy function that Chan–Vese model [5] proposes is (2)

$$\begin{array}{cc}\hfill & F(c_1,c_2,C)\hfill \\ \hfill & =\mu ·Length\left(C\right)+\nu ·Area\left(inside\right(C\left)\right)\hfill \\ \hfill & +{\lambda }_1{\int }_{inside\left(C\right)}^{}{\left|u_0(x,y)-c_1\right|}^{2}dxdy\hfill \\ \hfill & +{\lambda }_2{\int }_{outside\left(C\right)}^{}{\left|u_0(x,y)-c_2\right|}^{2}dxdy\hfill \end{array}$$

(2)

where C is the curve at the Level-Set Method [8], and $\mu \ge 0,v\ge 0,{\lambda }_1,{\lambda }_2>0$ are fixed parameters. $Length\left(C\right)$ is the length of the curve C. $Area\left(inside\right(C\left)\right)$ is the area of the region inside C that is added to the original Mumford–Shah [4]. The equation of the original Mumford–Shah functional for segmentation is (1). $c_1$ and $c_2$ are constants depending on C, since $c_1$ is the average value inside C and $c_2$ is the average value outside C. Furthermore, $u_0$ is a given input image. In this energy function, the first term controls the length of C. The second term controls the area of C to control its size. The third and fourth terms are used to control the difference between the piece-wise constant model’s result and the input image $u_0$. To solve (2) with level-set method [8], Reference [5] uses Heavyside function H, and the one-dimensional Dirac measure ${\delta }_0$. These are defined in (3).

$$H\left(z\right)=\left\{\begin{array}{l}1,ifz\ge 0\\ 0,ifz<0,\end{array}\right.{\delta }_0=\frac{d}{dz}H\left(z\right)$$

(3)

With this H and ${\delta }_0$, the energy function (2) can be re-formulated to (4).

$$\begin{array}{cc}\hfill & F(c_1,c_2,\varphi )\hfill \\ \hfill & =\mu {\int }_{\Omega }^{}\delta \left(\varphi (x,y)\right){\left|\nabla \varphi (x,y)\right|}^{2}dxdy\hfill \\ \hfill & +\nu {\int }_{\Omega }^{}H\left(\varphi (x,y)\right)dxdy\hfill \\ \hfill & +{\lambda }_1{\int }_{\Omega }^{}{\left|u_0(x,y)-c_1\right|}^{2}H\left(\varphi (x,y)\right)dxdy\hfill \\ \hfill & +{\lambda }_2{\int }_{\Omega }^{}{\left|u_0(x,y)-c_2\right|}^2(1-H\left(\varphi (x,y)\right))dxdy\hfill \end{array}$$

(4)

where $\Omega$ is a bounded open subset of ${\mathbb{R}}^2$ and $\varphi :\Omega \to \mathbb{R}$ is a zero level set of a Lipschitz function that represent the curve C. Furthermore, $c_1$ and $c_2$ are calculated with (5).

$$\begin{array}{cc}\hfill & c_1=\frac{{\int }_{\Omega }{u}_0(x,y)H\left(\varphi (x,y)\right)dxdy}{{\int }_{\Omega }H\left(\varphi (x,y)\right)dxdy}\hfill \\ \hfill & c_2=\frac{{\int }_{\Omega }{u}_0(x,y)(1-H\left(\varphi (x,y)\right))dxdy}{{\int }_{\Omega }(1-H\left(\varphi (x,y)\right))dxdy}\hfill \end{array}$$

(5)

2.2.2. Mumford–Shah Functional

Unlike the Chan–Vese algorithm [5], piecewise-smooth Mumford–Shah functional [4] assumes $c_1$ and $c_2$ as the functions rather than constant values. In the piecewise-smooth Mumford–Shah functional, the $c_1$ and the $c_2$ in (4) and (5) are rewritten as $R_1(x,y)$ and $R_2(x,y)$ when we let $R_1$,$R_2$ : $\Omega$→$\mathbb{R}$.

2.3. Region of Interest Detection

Detecting the regions of the input image that stimulate the deep learning network is called Region of Interest detection. Li et al. [28] makes the soft masks for input images with classification loss. Fong et al. [29] makes spatial perturbation masks that maximally affect a model’s output. More simply, Singh et al. [30] use randomly generated hidden image patches for each image at every iteration. Fong et al. [31] optimize the objective function with binarized masks that resize the area of input images. Furthermore, Jaderberg et al. [32] transform the input image with affine transformation matrix via classification loss.

3. Method

In this section, we describe two network structures and two corresponding loss functions for each network that segments the input image to the foreground and the background. Each network consists of a segmentation stage and a classification stage that classifies the segmentation results of the segmentation stage. As we mentioned, segmented results by the Mumford–Shah model and Chan–Vese model can detect the edges of the objects very precisely, but the results do not mean the foreground and the background. Since these two models segment the input images with pixel-intensity, the segmented regions consist of pixels with similar intensity. In our work, we use a classifier like [33,34,35,36], to make each segmented region, which is the result of the segmentation stage, meaning the foreground or the background. As shown in [18,19], some areas of the image, not the entire area, have a significant impact on the classifier. Therefore, when we train the classifier with multiplication of the outputs of the segmentation stage and the input image, the meaningless regions become the foreground and the background. Notice that the outputs of the segmentation stage are binarized with 0 and 1 to detect the foreground area of the input image. When we set the mask as $\varphi$ and the input image as I, $\varphi \odot I$ means the foreground and $(1-\varphi )\odot I$.⊙ means the element-wise multiplication. However, we use $(1-\varphi )\odot I$ as the foreground regions and $\varphi \odot I$ as the background regions, because it shows better results experimentally. In our experiment, the classifier is trained to classify the $(1-\varphi )\odot I$ as the foreground (i.e., image-level label) and $\varphi \odot I$ as the background. For the foreground, we use the cross-entropy loss between the ground-truth image class label and the output of the classifier. Furthermore, for the background, we construct a loss. We explain this background loss in (13).

3.1. Network Structure

For our experiment, we exploit the network structure of [23]. We do not need to make additional encoders with this structure because all the decoders directly use classifiers’ feature maps. Furthermore, for the classifier, we use ResNet-18 [37] structure. We experiment with two network structures. The first network, Figure 1, is composed of the Chan–Vese energy function, and the second network, shown in Figure 2, is composed of piecewise-smooth Mumford–Shah energy function. In Figure 1 and Figure 2, $\widehat{o}$, $\widehat{f}$, and $\widehat{b}$ are the classification score of the original input image, foreground image, and background image, respectively. Notice that because we use a same classifier for the original image, foreground image, and background image like Siamese networks [38], which share the weights, the classifier’s weights are updated by the $\widehat{o}$, $\widehat{f}$, and $\widehat{b}$. We indicate the classifier as $classifierA$ in each of Figure 1 and Figure 2. $\varphi$ is the generated binary mask and $1-\varphi$ is the reversed generated mask. Furthermore, in Figure 2, F is the foreground matrix and B is the background matrix for the calculation of the piecewise smooth Mumford–Shah functional loss (10).

3.1.1. Network with Chan–Vese Energy Function

The mask $\varphi$ is a one-channel image and has the same height and width as the input image. Furthermore, since we use the sigmoid function instead of the HeavySide function (3), the mask has only 0.1 values when the regularization term $\left|1-\varphi \right|$ is used together. The upsampled feature maps from the $classifierA$ used for the original input image are concatenated and pass convolutional layers. The output of the last convolutioinal layers is a mask that shows the background of the image. To get the foreground of the image, we reverse the mask by subtracting the mask from 1. The foreground image and background image pass the classifier, which we use for the original image. The $classifierA$ classifies the original images, the foreground, and the background of the input image. $\varphi$ and $1-\varphi$ are used for Chan–Vese loss (7) and (8). $\widehat{b}$ and $\widehat{f}$ are used for classification loss (14).

3.1.2. Network with Piecewise-Smooth Mumford–Shah Energy Function

As in Figure 1, we use ResNet-18 for the $classifierA$, and the mask $\varphi$ is a one-channel image and has same the height and length as the input image. The only difference is that there are three decoders. The first one is for a mask $\varphi$, the second one is for a foreground matrix F, and the last one is for a background matrix B. The foreground matrix F and the background matrix B are three-channel images and have the same height and length as the input image. $\varphi$, $1-\varphi$, F, and B are used for Mumford–Shah loss (10). $\widehat{b}$ and $\widehat{f}$ are used for classification loss (14).

3.2. Loss Function

Our total loss (6) can be divided into two parts. The first part is a segmentation loss for the segmentation stage, and the second part is a classification loss for the classification stage.

$${\mathcal{L}}_{totalloss}=\alpha {\mathcal{L}}_{segmentation}+{\mathcal{L}}_{classification}$$

(6)

${\mathcal{L}}_{segmentation}$ uses either the energy function of the Chan–Vese algorithm (4) or the energy function of the Mumford–Shah, neither of which calculates the mean values of the each region, for our deep learning network, reflecting some modification. ${\mathcal{L}}_{classification}$ is composed of classification loss about the foreground region and the background region.

3.2.1. ${\mathcal{L}}_{segmentation}$

• Chan–Vese Energy Function for Segmentation: To apply the Chan–Vese algorithm to our network, we modify (4) to (7). Where I is the input image, $\varphi$ is the mask, and $\nabla \varphi$ is the derivative of the mask with forward difference.

  $$\begin{array}{cc}\hfill & {\mathcal{L}}_{segmentation\_chanVese}\hfill \\ \hfill & ={\int }_{\Omega }^{}(1-\varphi (x,y)){(I(x,y)-c_1)}^{2}dxdy\hfill \\ \hfill & +{\int }_{\Omega }^{}\varphi (x,y){(I(x,y)-c_2)}^{2}dxdy\hfill \\ \hfill & +{\lambda }_1{\int }_{\Omega }^{}\left|1-\varphi (x,y)\right|dxdy\hfill \\ \hfill & +{\lambda }_2{\int }_{\Omega }^{}\left|\nabla \varphi (x,y)\right|dxdy\hfill \end{array}$$

  (7)
  Furthermore, $c_1$ and the $c_2$ are calculated with (8). The $c_1$ is the mean value of the first region, and the $c_2$ is the mean value of the second region.

  $$\begin{array}{cc}\hfill & c_1=\frac{{\int }_{\Omega }I(x,y)(1-\varphi (x,y))dxdy}{{\int }_{\Omega }(1-\varphi (x,y))dxdy}\hfill \\ \\ \hfill & c_2=\frac{{\int }_{\Omega }I(x,y)\varphi (x,y)dxdy}{{\int }_{\Omega }\varphi (x,y)dxdy}\hfill \end{array}$$

  (8)
  In (7), the first term and the second term divide the input image into two regions with similar pixel intensity. With the $1-\varphi (x,y)$, which only has values 0 and 1, and the constant mean value $c_1$, the first term only considers the first regions regardless of the second regions, and the first regions are grouped into the pixels of similar value. Similarly, with the $\varphi (x,y)$ and the constant mean value $c_2$, the second term only considers the second regions regardless of the other regions and the pixel values in the second regions have similar values. The third term controls the regions’ size of the mask, and the fourth term controls the noise of the mask. We call each loss term foreground fidelity, background fidelity, mask region regularization, and mask smooth regularization. For a given mini-batch of training set {($I_1$, $y_1$)$,...,$($I_N$, $y_N$)} when the $y_i$ is the ground-truth image level labels (i.e., images class labels) of mini-batch of input images $y_i$, we set (7) as (9). $E(I,\varphi ,c_1,c_2)$ is the sum of the foreground fidelity term and the background fidelity term. $R\left(\varphi \right)$ is the regularization of $\varphi$, and M is the size of mini-batch.

  $$\frac{1}{M}E(I_i,{\varphi }_i,{c_1}_i,{c_2}_i)+\frac{1}{M}R\left({\varphi }_i\right)$$

  (9)
• Piecewise-smooth Mumford–Shah Energy Function for Segmentation: As with the relationship between (4) and traditional piecewise-smooth Mumford–Shah Energy Function, the differences between each loss function of our two networks are that constant values $c_1$ and $c_2$ of (7) are changed to $F(x,y)$ (i.e., foreground matrix) and $B(x,y)$ (i.e., background matrix). The loss function is (10).

  $$\begin{array}{cc}\hfill & {\mathcal{L}}_{segmentation\_mumfordShah}\hfill \\ \hfill & ={\int }_{\Omega }^{}(1-\varphi (x,y)){(I(x,y)-F(x,y))}^{2}dxdy\hfill \\ \hfill & +{\int }_{\Omega }^{}\varphi (x,y){(I(x,y)-B(x,y))}^{2}dxdy\hfill \\ \hfill & +{\lambda }_1{\int }_{\Omega }^{}\left|1-\varphi (x,y)\right|dxdy\hfill \\ \hfill & +{\lambda }_2{\int }_{\Omega }^{}\left|\nabla \varphi (x,y)\right|dxdy\hfill \\ \hfill & +{\lambda }_3{\int }_{\Omega }^{}\left|\nabla F(x,y)\right|dxdy\hfill \\ \hfill & +{\lambda }_4{\int }_{\Omega }^{}\left|\nabla B(x,y)\right|dxdy\hfill \end{array}$$

  (10)
  The terms in $L_{segmentation\_mumfordShah}$ are foreground fidelity, background fidelity, mask region regularization, mask smooth regularization, foreground smooth regularization, and background smooth regularization in order. With the $1-\varphi (x,y)$ and the foreground matrix $F(x,y)$, the first term only considers the remained regions by the $1-\varphi (x,y)$ regardless of the other regions, and the remained regions resemble the same regions of the input image. Same as the first term, the remaining regions of the input image by the second term depend on the $\varphi (x,y)$. The foreground smooth regularization and the background smooth regularization work with the foreground fidelity and the background fidelity, respectively, to control the smooth of the foreground matrix and the background matrix. These terms work closely with mask smooth regularization and adjust the smooth of the mask. The effect of the mask region regularization and the mask smooth regularization are the same as (7). Furthermore, we set (10) as (11), where $E(I,\varphi ,c_1,c_2)$ is the sum of the foreground fidelity term and the background fidelity term. Furthermore, $R(\varphi ,F,B)$ is the summation of the regularizations of $\varphi$, F, and B.

$$\frac{1}{M}E(I_i,{\varphi }_i,F_i,B_i)+\frac{1}{M}R({\varphi }_i,F_i,B_i)$$

(11)

Notice that the divided regions of these two models’ outputs are not the foreground and the background of the input images. The Chan–Vese algorithm and the Mumford–Shah functional cannot distinguish the foreground and the background, because these energy functions are based on pixel intensities, like K-means clustering [11]. Because these two models divide the images into two regions regardless of the meaning of each region, we use a classifier to make each region be the foreground and the background of the input images. We use this ${\mathcal{L}}_{segmentation}$ as a regional proposal to the next stage (i.e., classifier).

3.2.2. ${\mathcal{L}}_{classification}$

Our classification loss (12) consists of the foreground classification loss and the background classification loss.

$${\mathcal{L}}_{classification}={\mathcal{L}}_{foreground}+{\mathcal{L}}_{background}$$

(12)

The ${\mathcal{L}}_{foreground}$ is a classification loss for the foreground image (i.e., $(1-\varphi )\odot I$), and the ${\mathcal{L}}_{background}$ is a classification loss for the background image (i.e., $\varphi \odot I$). Operator ⊙ means element-wise multiplication. $\widehat{o}$, $\widehat{f}$, and $\widehat{b}$ are the vectors and the size is $minibatchsize\times numberofclasses$. The values of the vectors are the probabilities corresponding to each class. For the foregrounds, we use cross-entropy loss. However, it is impossible to find common features of the backgrounds with a classifier because the components of the backgrounds are too diverse to assign them to the same label. Therefore, we formulate a loss (13) that can distinguish the background from the foreground.

$${\mathcal{L}}_{background}=-log4\widehat{b}(1-\widehat{b})$$

(13)

To minimize this (13), all the values of $\widehat{b}$, meaning probability values belonging to each class, must be 0.5. This means that the backgrounds do not belong to any given class. In (13), the constant inside the log is set to 4 to make the minimum zero. With the foreground loss and our background loss, we set (12) as (14).

$$\frac{1}{M}S(f_i,b_i,y_i)=-\frac{1}{M}{y}_{i}log{f}_i-\frac{1}{M}log4b_i(1-b_i)$$

(14)

With this (12), the proposed regions from ${\mathcal{L}}_{segmentation}$ gradually change into the regions that have the meaning of foreground.

It is remarkable that, like (7) and (10), when (12) works with two regularization terms in (15), (12) can generate the mask of the input image I that extracts the regions as foreground. We compare the result in Section 4.

$${\lambda }_1{\int }_{\Omega }^{}\left|1-\varphi (x,y)\right|dxdy+{\lambda }_2{\int }_{\Omega }^{}\left|\nabla \varphi (x,y)\right|dxdy$$

(15)

3.3. Algorithm

We solve the two-stage segmentation problem with the Stochastic Gradient Descent (SGD) [39] method. For convenience, we rearrange the terms as follows; the classifier C, the decoder D, the mask $\varphi$, the foreground matrix F, and the background matrix B. The only difference between using (7) and using (10) for segmentation is the number of D.

3.3.1. Chan–Vese Algorithm

First, to extract the feature maps l, which are the input of the decoder D, we use the original input Image I. Using the original image for the classifier is more helpful for $\varphi$ to detect the whole part of the foreground of I.

$${\widehat{o}}^{\left(t\right)},l^{\left(t\right)}\leftarrow C^{\left(t\right)}\left(I\right)$$

(16)

With the feature maps l from the classifier C and the original image I, the D generates the $\varphi$.

$${\varphi }^{\left(t\right)}\leftarrow D^{\left(t\right)}\left(l^{\left(t\right)}\right)$$

(17)

With the $\varphi$, we compute the $c_1$ and $c_2$ with (8) and get the probability of foreground $\widehat{f}$ and background $\widehat{b}$. The probability refers to the degree to which foreground and background belong to a particular class.

$$\begin{array}{cc}\hfill & {\widehat{f}}^{\left(t\right)},\_\leftarrow C^{\left(t\right)}((1-{\varphi }^{\left(t\right)})\odot I)\hfill \\ \hfill & {\widehat{b}}^{\left(t\right)},\_\leftarrow C^{\left(t\right)}({\varphi }^{\left(t\right)}\odot I)\hfill \end{array}$$

(18)

The loss function for the Chan–Vese algorithm for the mini-batch i at step t is

$$\begin{array}{cc}\hfill {\mathcal{L}}_i^{\left(t\right)}=& \frac{1}{M}E(I_i,{\varphi }_i^{\left(t\right)},{c_1}_i^{\left(t\right)},{c_2}_i^{\left(t\right)})+\frac{1}{M}R\left({\varphi }_i^{\left(t\right)}\right)\hfill \\ \hfill & +\frac{1}{M}S({\widehat{f}}_i^{\left(t\right)},{\widehat{b}}_i^{\left(t\right)},y_i)\hfill \end{array}$$

(19)

The parameters of each network’s component are updated by gradient descent algorithm. Where $\theta$ is the parameters of the networks, ${\eta }^{\left(t\right)}$ is the learning rate at step t, and $CE$ is the cross entropy loss, $CE({\widehat{o}}_i^{\left(t\right)},y_i)=-\frac{1}{M}{y}_{i}log{\widehat{o}}_i$.

$$\begin{array}{cc}\hfill & {\theta }_{C^{(t+1)}}\leftarrow {\theta }_{C^{\left(t\right)}}-{\eta }^{\left(t\right)}{\nabla }_{C^{\left(t\right)}}({\mathcal{L}}_i^{\left(t\right)}+CE({\widehat{o}}_i^{\left(t\right)},y_i))\hfill \\ \hfill & {\theta }_{D^{(t+1)}}\leftarrow {\theta }_{D^{\left(t\right)}}-{\eta }^{\left(t\right)}{\nabla }_{D^{\left(t\right)}}{\mathcal{L}}_i^{\left(t\right)}\hfill \end{array}$$

(20)

We show this process with Algorithm 1.

Algorithm 1 Chan–Vese Segmentation Algorithm.

Require:  Mini-batch size M, Classifier C, Decoder D Ensure:  Mask ${\varphi }^{\left(T\right)}$ 1: fort← 1 $to$ T do 2:     Sample mini-batch of data {($I_1$, $y_1$)$,...,$($I_N$, $y_N$)} 3:     for i← 1 $to$ N do 4:         ${\widehat{o}}_i^{\left(t\right)},l_i^{\left(t\right)}\leftarrow C^{\left(t\right)}\left(I_i\right)$ 5:         ${\varphi }_i^{\left(t\right)}\leftarrow D^{\left(t\right)}\left(l_i^{\left(t\right)}\right)$ 6:         Compute $c_1$ and $c_2$ 7:         ${\widehat{f}}_i^{\left(t\right)},\_\leftarrow C^{\left(t\right)}((1-{\varphi }_i^{\left(t\right)})\odot I)$ 8:         ${\widehat{b}}_i^{\left(t\right)},\_\leftarrow C^{\left(t\right)}({\varphi }_i^{\left(t\right)}\odot I)$ 9:         ${\theta }_{C^{(t+1)}}\leftarrow {\theta }_{C^{\left(t\right)}}-{\eta }^{\left(t\right)}{\nabla }_{C^{\left(t\right)}}({\mathcal{L}}_i^{\left(t\right)}+CE({\widehat{o}}_i^{\left(t\right)},y_i))$ 10:         ${\theta }_{D^{(t+1)}}\leftarrow {\theta }_{D^{\left(t\right)}}-{\eta }^{\left(t\right)}{\nabla }_{D^{\left(t\right)}}{\mathcal{L}}_i^{\left(t\right)}$ 11:     end for 12: end for

3.3.2. Piecewise-Smooth Mumford–Shah Algorithm

The differences with the Chan–Vese algorithm are shown in (21).

$$\begin{array}{c}\hfill {\varphi }^{\left(t\right)}\leftarrow D_{\varphi }^{\left(t\right)}\left(l^{\left(t\right)}\right)\\ \hfill F^{\left(t\right)}\leftarrow D_F^{\left(t\right)}\left(l^{\left(t\right)}\right)\\ \hfill B^{\left(t\right)}\leftarrow D_B^{\left(t\right)}\left(l^{\left(t\right)}\right)\end{array}$$

(21)

Therefore, the loss function for the Mumford–Shah Algorithm about the mini-batch i at step t is

$$\begin{array}{cc}\hfill {\mathcal{L}}_i^{\left(t\right)}=& \frac{1}{M}E(I_i,{\varphi }_i^{\left(t\right)},{c_1}_i^{\left(t\right)},{c_2}_i^{\left(t\right)})+\frac{1}{M}R({\varphi }_i^{\left(t\right)},F_i^{\left(t\right)},B_i^{\left(t\right)})\hfill \\ \hfill & +\frac{1}{M}S({\widehat{f}}_i^{\left(t\right)},{\widehat{b}}_i^{\left(t\right)},y_i)\hfill \end{array}$$

(22)

Furthermore, the parameters of the networks are updated:

$$\begin{array}{cc}\hfill & {\theta }_{C^{(t+1)}}\leftarrow {\theta }_{C^{\left(t\right)}}-{\eta }^{\left(t\right)}{\nabla }_{C^{\left(t\right)}}({\mathcal{L}}_i^{\left(t\right)}+CE({\widehat{o}}_i^{\left(t\right)},y_i))\hfill \\ \hfill & {\theta }_{D_{\varphi }^{(t+1)}}\leftarrow {\theta }_{D_{\varphi }^{\left(t\right)}}-{\eta }^{\left(t\right)}{\nabla }_{D_{\varphi }^{\left(t\right)}}{\mathcal{L}}_i^{\left(t\right)}\hfill \\ \hfill & {\theta }_{D_F^{(t+1)}}\leftarrow {\theta }_{D_F^{\left(t\right)}}-{\eta }^{\left(t\right)}{\nabla }_{D_F^{\left(t\right)}}{\mathcal{L}}_i^{\left(t\right)}\hfill \\ \hfill & {\theta }_{D_B^{(t+1)}}\leftarrow {\theta }_{D_B^{\left(t\right)}}-{\eta }^{\left(t\right)}{\nabla }_{D_B^{\left(t\right)}}{\mathcal{L}}_i^{\left(t\right)}\hfill \end{array}$$

(23)

This process is shown in Algorithm 2.

Algorithm 2 Mumford–Shah Segmentation Algorithm.

Require:  Mini-batch size M, Classifier C Require:  Decoder $D_{\varphi }$, Decoder $D_F$, Decoder $D_B$ Ensure:  Mask ${\varphi }^{\left(T\right)}$ 1: fort← 1 $to$ T do 2:     Sample mini-batch of data {($I_1$, $y_1$)$,...,$($I_N$, $y_N$)} 3:     for i← 1 $to$ N do 4:         ${\widehat{o}}_i^{\left(t\right)},l_i^{\left(t\right)}\leftarrow C^{\left(t\right)}\left(I_i\right)$ 5:         ${\varphi }_i^{\left(t\right)}\leftarrow D_{\varphi }^{\left(t\right)}\left(l_i\right)$ 6:         $F_i^{\left(t\right)}\leftarrow D_F^{\left(t\right)}\left(l_i\right)$ 7:         $B_i^{\left(t\right)}\leftarrow D_B^{\left(t\right)}\left(l_i\right)$ 8:         ${\widehat{f}}_i^{\left(t\right)},\_\leftarrow C^{\left(t\right)}((1-{\varphi }_i^{\left(t\right)})\odot I)$ 9:         ${\widehat{b}}_i^{\left(t\right)},\_\leftarrow C^{\left(t\right)}({\varphi }_i^{\left(t\right)}\odot I)$ 10:         ${\theta }_{C^{(t+1)}}\leftarrow {\theta }_{C^{\left(t\right)}}-{\eta }^{\left(t\right)}{\nabla }_{C^{\left(t\right)}}({\mathcal{L}}_i^{\left(t\right)}+CE({\widehat{o}}_i^{\left(t\right)},y_i))$ 11:         ${\theta }_{D_{\varphi }^{(t+1)}}\leftarrow {\theta }_{D_{\varphi }^{\left(t\right)}}-{\eta }^{\left(t\right)}{\nabla }_{D_{\varphi }^{\left(t\right)}}{\mathcal{L}}_i^{\left(t\right)}$ 12:         ${\theta }_{D_F^{(t+1)}}\leftarrow {\theta }_{D_F^{\left(t\right)}}-{\eta }^{\left(t\right)}{\nabla }_{D_F^{\left(t\right)}}{\mathcal{L}}_i^{\left(t\right)}$ 13:         ${\theta }_{D_B^{(t+1)}}\leftarrow {\theta }_{D_B^{\left(t\right)}}-{\eta }^{\left(t\right)}{\nabla }_{D_B^{\left(t\right)}}{\mathcal{L}}_i^{\left(t\right)}$ 14:     end for 15: end for

4. Experimental Results

4.1. Training and Testing Details

For training, we use ResNet-18 [37] as a classifier. Since we use the classifier’s feature maps for the decoder’s input, we do not construct a encoder. This is more efficient because we do not need additional parameters [23]. Furthermore, for the decoder that generates the masks, we upsample the each feature map of the classifier to $56\times 56$ and apply a $1\times 1$ convolution layer with the ReLU activation function to make each input’s channel be 64 since the last feature map of the classifier has 64 channels and the first feature map’s size is $56\times 56$. Furthermore, after concatenating the all upsampled feature maps, we apply a convolution layer (kernel with a size of 3, stride 1, and padding 1), upsample to $224\times 224$, and apply the sigmoid function to make the mask within 0 and 1. For the pre-processing, we resize the input images to the $224\times 224$ and normalize the input images value from 0 to 1 by $\frac{x_i-min\left(I\right)}{max\left(I\right)-min\left(I\right)}$, when I is the input image and $x_i$ is the value of $i-th$ pixel. Furthermore, for each dataset [9,10], we train and evaluate the network separately.

Furthermore, for the test to generate the masks, we use the same pre-processing for the input images. The network for the test is shown in Figure 3. The $1-\varphi$ is our segmentation result for an input image, which segments the input image to the foreground and the background.

4.2. Qualitative Comparisons

With the [9] dataset, we conducted three additional experiments to show the effectiveness of each loss term (classification loss (12) and segmentation loss (7),(10)): Generate mask only with classification loss, Chan–Vese network without any classification loss, and Mumford_Shah network without any classification loss. The experiment “Generate mask only with classification loss” is conducted through the structure of Figure 1, except for the Chan–Vese loss. In this experiment, the entire loss consists only of classification loss (12), and the generated masks are learned only by classification loss, similarly to [23]. For the two experiments “Chan–Vese network without any classification loss” and “Mumford_Shah network without any classification loss”, we make an auto-encoder network. For the encoder, we use the Vgg-16 [40], and the decoder is constructed symmetrically with the encoder using the transposed convolution. There are no skip connections between the encoder and the decoder. In these two experiments, only the segmentation loss (Chan–Vese loss (7) and Mumford–Shah loss (10)) affects the masks.

Figure 4 shows the results of these three experiments and our main two experiments’ (Figure 1 and Figure 2) results. For the baseline, we use the results of CASM [23] that segments the image to the foreground and the background, because we use the same network structure as CASM. For each set of result images, the first row is the input image, the second row is the generated mask, the third row is the foreground image (i.e., $(1-\varphi )\odot I)$), and the last row is the background image (i.e., $\varphi \odot I$). When we observe Figure 4b, it is remarkable that when we use our classification loss (12) and a regularization term $\left|1-\varphi \right|$ for segmentation (i.e., using Figure 1 except the Chan–Vese loss to classify the $(1-\varphi )\odot I)$ as the foreground and classify the $\varphi \odot I$ as the background), we can get the area of the foreground. However, we cannot detect the whole area of the object. With Figure 4c,d, applying the Chan–Vese loss (7) and the Mumford–Shah loss (10) without classification loss (12) to the our simple auto-encoder can achieve a precise boundary of the object. However, the inside of the mask is not homogeneous. This is because (7) and (10) are minimized when pixels of similar intensity of the image are grouped together. Furthermore, the mask cannot detect the object as the foreground, since (7) and (10) are the functions of the intensity of the pixels. To make each region detected by the mask have the meaning of the foreground or the background, we concatenate the Chan–Vese loss (7) and the Mumford–Shah loss (10) with the classification loss (12) to make our total loss (6). The results of our total loss are in Figure 4e,f. The result of total loss combining Chan–Vese loss and classification loss is Figure 4e, and the result of total loss combining Mumford-Shan loss and classification loss is Figure 4e. Comparing the results of Figure 4c,d with those of Figure 4e,f shows that the area that was detected as black by the mask changed to the white area. This means that the mask finally detects the object as the foreground. Furthermore, when we compare with the baseline Figure 4a, we can get a more precise boundary since the Chan–Vese loss or the Mumford–Shah loss is used in addition to the classification loss as the segmentation loss. Figure 5 and Figure 6 show more segmentation results when using our total loss (6). Figure 5 is the result with the [9] dataset and uses our total loss (6), where the ${\mathcal{L}}_{segmentation}$ is composed of the Mumford–Shah functional. In Figure 5, the first column is the input images, the second column is the generated masks (i.e., $1-\varphi$) by our total loss (6), the third column is the foreground images (i.e., $(1-\varphi )\odot I$), and the last column is the background images (i.e., $\varphi \odot I$). We can see that the masks detect the dogs and cats very accurately, and the boundaries resemble the shape of the dogs and cats. Furthermore, there are no parts of the dogs and cats in the background images. In Figure 6, we additionally add the ground-truth masks in the second column. When we compare the ground-truth mask (second column) and the generated mask by our total loss (6), we could get masks of very similar shape to the ground-truth masks even if the objects classified as foreground are small. However, when we use the PASCAL VOC dataset, we assume that the one image has only one image-level label, although there are multiple objects since our mask segments the image as the foreground and the background.

4.3. Quantitative Comparison

Since we assume that the one image has only one image-level class label, we calculate the mean IoU (intersection of union) for each image-level class label, not for the entire dataset.

Table 1. Mean Intersection of Union of each class label.

Class	Dog	Cat	Horse	Cow	Car	Person
Baseline CASM	0.79	0.80	0.81	0.79	0.66	0.63
only classification loss	0.56	0.54	0.42	0.36	0.21	0.62
Chan–Vese with classification loss	0.86	0.82	0.83	0.79	0.65	0.65
Mumford–Shah with classification loss	0.88	0.82	0.80	0.76	0.75	0.64

shows the mean IoU calculated only for images belonging to each class. We show the results for four classes (dog, cat, horse, and cow) with a high IoU and for two classes (car, person) with a low IoU. When we compare the results of Baseline and our results (fourth row and fifth row of Table 1), our total loss (6) gets a higher mean IoU.

5. Discussions and Conclusions

Unlike other weakly supervised image segmentation, we assume that the one image only has one image-level label. Therefore, with our network and loss, we only can get the foreground and the background of the image. However, the segmentation energy functions of the Chan–Vese model or the Mumford–Shah model, which are revised to be applied in the deep learning framework, are efficient to get the precise boundaries of the objects. Furthermore, we do not optimize these energy functions with conventional level-set method. Instead, we optimize them with the stochastic gradient descent method, since these energy functions’ elements are the output of the deep learning network. Because we use deep learning, the initial curve of the level-set method is not important, and the number of regions is fixed as two (the foreground and the background). Furthermore, with the classifier, the meaningless regions are changed to the foreground and the background, since the classifier needs the foreground regions to classify the image. Moreover, through our background loss, which acts similarly to assigning a background image to a new class label, the classification loss (12) with two regularizers (15) can also get the mask. Therefore, with our work, we can get the foreground and the background, although the image only has an image level class label and not other information. However, our networks have one significant limitation. Since we use the classifier’s feature maps as the input of the decoder, the results change significantly with the classifier’s performance. Therefore, we construct a new network that does not depend on the classifier for the following study.