Weakly Supervised Conditional Random Fields Model for Semantic Segmentation with Image Patches

Abstract

Image semantic segmentation (ISS) is used to segment an image into regions with differently labeled semantic category. Most of the existing ISS methods are based on fully supervised learning, which requires pixel-level labeling for training the model. As a result, it is often very time-consuming and labor-intensive, yet still subject to manual errors and subjective inconsistency. To tackle such difficulties, a weakly supervised ISS approach is proposed, in which the challenging problem of label inference from image-level to pixel-level will be particularly addressed, using image patches and conditional random fields (CRF). An improved simple linear iterative cluster (SLIC) algorithm is employed to extract superpixels. for image segmentation. Specifically, it generates various numbers of superpixels according to different images, which can be used to guide the process of image patch extraction based on the image-level labeled information. Based on the extracted image patches, the CRF model is constructed for inferring semantic class labels, which uses the potential energy function to map from the image-level to pixel-level image labels. Finally, patch based CRF (PBCRF) model is used to accomplish the weakly supervised ISS. Experiments conducted on two publicly available benchmark datasets, MSRC and PASCAL VOC 2012, have demonstrated that our proposed algorithm can yield very promising results compared to quite a few state-of-the-art ISS methods, including some deep learning-based models.

1. Introduction

Different from conventional image segmentation, by combining image segmentation and object recognition, image semantic segmentation (ISS) divides an image into many image blocks to identify the semantic category of each block [1]. It has been widely applied in semantic information extraction from images for scene understanding and object recognition [2,3].

In general, ISS approaches can be mainly divided into two categories, i.e., fully supervised and weakly supervised [4]. Fully supervised ISS requires pixel based labeling of the whole image, which is often achieved manually. To complete the labeling of a picture, skilled annotators on average need nearly 10 min, which is quite time consuming and labor intensive [5]. Considering the difficulty of obtaining pixel-level labeling in fully supervised learning, weakly supervised ISS is more desirable as it does not require pixel based labeling of the whole images thus the associated labor cost and time consumption can be reduced significantly. As a result, weakly supervised ISS has become a research hotspot in recent years.

Although deep learning-based ISS has been recently proposed, such as the Fully Convolutional Networks (FCN) [6], in which an end-to-end pixel-level training is adopted. The FCN uses a 1 × 1 convolution layer instead of a full connection layer, along with unsampling, for improving the resolution of the feature map. Although FCN has made great success of the segmentation results, it still has some problems. For example, the feature graph is greatly shrunk during the pooling operation, resulting in inaccurate boundary of segmentation [7].

Following the analysis above, in this paper, a novel image patch and conditional random field (CRF) based weakly supervised semantic segmentation (IPCRFWSS) algorithm is proposed. First, we improved the SLIC algorithm to extract the image patches. Second, we construct a second-order CRF with image patches as the node, and the unary and pairwise potential energy functions of CRF are calculated. In addition, the inference of semantic label is transformed into the problem of minimizing potential energy function, and the image patches with semantic labels are obtained. Also, each image patch has been assigned with appropriate semantic labels. Comprehensive experiments on the MSRC datasets have been conducted for quantitative performance evaluation when benchmarking with several state-of-the-art approaches.

The major contributions of our work can be highlighted as follows:

• We propose an image patch and CRF based weakly supervised ISS algorithm (IPCRFWSS), which can successfully achieve semantic label inference and prediction;
• We propose an algorithm for automatic estimation of the recommended number of superpixels for different images, which has significantly improved the efficiency and accuracy of image segmentation as it can be used to generate image patches for image-level labels;
• A PBCRF model is introduced for semantic class inference from image-level to pixel-level labels. With trained patch based CRF, class correlation and similarity functions are added into pairwise potential function to improve the accuracy and robustness of semantic label inference;
• Experimental results on two publicly available datasets have fully validated the efficacy and efficiency of the proposed approach, which has outperformed quite a few state-of-the-art, including some deep learning models.

2. Related Work

2.1. Conventional Image Segmentation

As a fundamental task for semantic image processing and image understanding, Image segmentation divides an image into different non-overlapped regions according to its color, texture and other visual properties. At present, image segmentation methods can be roughly divided into three categories [8], i.e., region-based [9], edge-based [10], and cluster-based methods [11]. Among them, the region-based segmentation is popularly used.

According to the consistency within the region and the inconsistency between regions, region-based segmentation methods can be further divided into three groups, including thresholding [12], region growing [13], and splitting and merging [14] based techniques. The advantage of thresholding is that it is easy for implementation and the computational complexity is low. However, the spatial position information of the image is ignored, which has led to the difficulty in balancing the segmentation effect in the global and local areas. The region growing method determines a suitable region by using a point as the seed point along with a growing criterion, which is often measured by the similarity of the formed region and the pixel under processing. This method improves the performance of image segmentation, but it is sensitive to noise and can easily lead to over- segmentation [13]. The splitting and merging method divide the image into many small regions by local similarity, where the neighboring small regions can be further merged iteratively if they are sufficiently similar to each other [15].

2.2. Superpixel Based Image Segmentation

As a newly proposed splitting and merging method, superpixel based image segmentation has achieved great progress in recent years. As first proposed by Ren et al. [16], superpixel is defined as a sub-region composed of adjacent pixels with similar texture, color and other visual characteristics. Superpixel based image segmentation is the process of clustering pixels into superpixels, and relevant algorithms can be roughly divided into graph-based and gradient descent-based methods. Graph-based methods mainly include: normalized cut (NC) [17], superpixel lattices (SL) [18], and Felzenszwalb and Huttenlocher (FH) algorithm [19]. Typical gradient descent-based methods are watershed [20], Meanshift [21] and the Simple Linear Iterative Cluster (SLIC) algorithm [22] et al.

Simple linear iterative clustering was first proposed in [22]. Hsu et.al [23] proposed an image segmentation algorithm based on SLIC superpixel, and region merging based on 5-D spectral clustering and boundary-focused region clustering. Ning et.al [24] proposed a novel image segmentation method based on interactive region merging, but users should roughly mark the location and region of the target and background. Gu et.al [25] proposed an algorithm to add the color covariance matrix to the features of superpixel to improve the accuracy of image segmentation.

Compared with pixels, the advantages of superpixels are reflected in two aspects: the calculation is simple, which is helpful to reduce the size of processing objects and the computational complexity of subsequent processing. The number of superpixels can be controlled by adjusting the parameter $K$, however $K$ needs to be set manually. If $K$ is too large, the advantage of superpixel segmentation will be lost and the unnecessary computational complexity of image segmentation will be increased. If $K$ is too small, the accuracy of image segmentation results will be reduced. Therefore, it is more challenging to manually set the appropriate number of superpixels.

Image patches are merged into larger image regions based on weakly supervised information. Each image patch has only one semantic category, but a target region can be composed of multiple image patches. Compared with superpixel, the proposed method has more advantages in using image patches as the basic unit of weakly supervised ISS. There are two main advantages: the number of image patches is much less than the number of superpixels, which can greatly reduce the complexity of the algorithm. Image patches have more neat object boundaries, which can improve the accuracy of semantic label inference.

2.3. Image Semantic Segmentation

In the past years, ISS has attracted much attention and become one of the hotspots of computer vision. Among the existing ISS methods, there are two main categories: fully supervised and weakly supervised semantic segmentation algorithms. The difference between them is that full supervision requires pixel-level label learning, while weak supervision only needs image-level label learning, which greatly reduces the cost of human and material resources caused by manual annotation [26]. It has a good application prospect, although there are many issues to be addressed in weakly supervised semantic segmentation. These weakly supervised semantics segmentation methods can be roughly divided into two categories: traditional methods and deep learning methods.

Traditional semantic segmentation needs a process of feature extraction followed by several different classifiers to complete the segmentation. Duygulu et al. [27] first proposed the concept of Blob-World, and used image-level label training classifier to conduct image semantic segmentation. Zhang et al. [28] proposed an effective support vector machine classifier based on spatial sparse reconstruction method. The classifier is trained with noisy data and denoised by subspace reconstruction method. The optimal parameters are obtained by iterative optimization. Vezhnevets et al. [29] proposed multi-image model (MIM), using conditional random field model, the one-dimensional potential energy function is established with single superpixel pairs and the two-dimensional potential energy function is established with superpixel pairs. The semantic segmentation result is obtained by CRF parameter approximation solution. Liu et al. [30] proposed weakly-supervised dual clustering for image segmentation and label correspondence. Zhang et al. [31] proposed a graph model for recovering the pixel based on the appearance similarity of training image superpixels. This model is different from the traditional classifier and has achieved good results when learning multi-class kernel matrices. Wang et al. [32] proposed a probabilistic graph model called TCPR for weakly supervised labeling. This method adds neighborhood context constraint to the MRF model and can use automatic inference mechanism to automatically infer category labels.

Deep learning semantic segmentation method generally consists of a general network framework and a segmentation network. The performance improvement of network structure also brings great improvement to the precision of image processing. To solve the problem of feature loss caused by network framework pooling and down-sampling operation, Noh et al. [33] proposed a deconvolution neural network, which combined the prediction method of network and full convolution network to achieve semantic segmentation task. Farabet et al. [34] proposed multi-scale convolution neural network based deep learning for semantic segmentation, in which pixel-level features were extracted including texture, shape and context information. Qi et al. [35] proposed a framework to reduce the error in weakly supervised learning with image-level supervision. In this way, semantic segmentation and object localization are unified to improve segmentation performance. Wei et al. [36] proposed a framework for generating localization maps by hypotheses-aware classification and cross image contextual refinement. Chen et al. proposed to refine the pixel-wise prediction from the last DCNN layer with a fully connected CRF and achieved better segmentation results [3]. Papandreou et al. [37] develop expectation maximization (EM) methods for semantic image segmentation model training under these weakly supervised and semi-supervised settings. Wei et al. [38] proposed a simple to complex (STC) framework, which used simple image-level labels to enhance the Initial-DCNNs network, and then used the Enhanced-DCNNs network to complete more complex ISS tasks. Although these DCNN-based methods improved the performance of weakly supervised ISS, they rely on the precision of pre-trained classification networks.

3. The Proposed Method

We propose a novel framework for weakly supervised ISS based on image patches and CRF. As shown in Figure 1, the flowchart of the proposed framework contains three main parts, i.e., superpixel generation and image segmentation, CRF model construction, and CRF based semantic inference of image patches for ISS. First, the improved SLIC algorithm is used to segment the training images into superpixels, which are merged into image patches based on the weakly supervised information. Second, the class correlation function and similarity function are introduced into the CRF model to construct a CRF model for inferring semantic class labels. Finally, the trained PBCRF model is applied for weakly supervised semantic segmentation of images. Relevant details of these three parts within the proposed framework are presented as follows.

3.1. Superpixel Generation

3.1.1. SLIC Superpixel Generation

First, the Simple Linear Iterative Cluster (SLIC) algorithm is used to segment the training images to generate superpixels. Second, superpixels are merged into image patches based on image-level label. The termination condition is that the number of pieces equals three times the number of image-level labels. The better results of superpixel generation and merging are of great help to the construction of CRF model.

Therefore, we have improved the SLIC algorithm so that $K$ can be adaptively determined according to different input images, based on the color information of the images. To better reflect human visual perception, the color space of the image is converted from red, green and blue (RGB) to hue, saturation and value (HSV). In order to simplify the calculation, the H, S, and V components in the HSV space is quantified into 16, 5, and 5 levels respectively, which are further combined to yield a one-dimensional eigenvector $Z$ as follows.

$$Z=H{Q}_S{Q}_V+S{Q}_V+V=25H+5S+V$$

(1)

where $Q_S$ and $Q_V$ are the quantization grades of $S$ and $V$ respectively ($Q_S=5$, $Q_V=5$); the three-dimensional vectors of $H$, $S$ and $V$ are transformed into one-dimensional vectors ($H\in \left[0,360\right]$, $S\in \left[0,1\right]$, $V\in \left[0,1\right]$). The median value of all elements in $Z$ is determined as $m$ which is used as the initial value for $K^{\prime }=\left[m\right]$. We put $m$ in brackets, as it indicates a rounding up function to ensure an integer value for $K^{\prime }$.

3.1.2. SLIC Merging Based on Image-level Labels

In the process, regional feature similarity is taken as the criterion of superpixel merging. Color feature, texture feature and scale invariant feature transform (SIFT) feature are extracted to describe each superpixel. The similarity of superpixels is determined by using the extracted feature vector, where the adjacent superpixels are merged according to the similarity between different superpixels to obtain the image patches.

Considering the spatial information of two superpixels, $i$ and $j$, denote $\mathrm{N}\left(i\right)$ as the neighboring superpixels of $i$, the adjacency matrix $B\left(i,j\right)$ can be defined as:

$$B\left(i,j\right)=\{\begin{array}{l}1\hspace{1em}\mathrm{if} \mathrm{j}\in \mathrm{N}\left(i\right)\\ 0\hspace{1em}\mathrm{if} \mathrm{j}\notin \mathrm{N}\left(i\right)\end{array}$$

(2)

In this way, the similarity function can be defined as:

$${\Psi }_{i,j}=\{\begin{array}{cc}{\lambda }_1{S}_{ij}^c+{\lambda }_2{S}_{ij}^t+{\lambda }_3{S}_{ij}^s;({\displaystyle {\sum }_i{\lambda }_i=}1)& \hspace{1em}\mathrm{if} \mathrm{B}\left(i,j\right)=1\hfill \\ 0& \hspace{1em}\mathrm{if} \mathrm{B}\left(i,j\right)=0 \hfill \end{array}$$

(3)

among them, $S_{ij}^c$, $S_{ij}^t$ and $S_{ij}^s$, which are measured Euclidean distances of the color, texture and SIFT feature extracted from two superpixels, $i$ and $j$, respectively; ${\Psi }_{i,j}$ denotes the overall distance between the two superpixels, ${\lambda }_i$ are the adjusting weights for the three featured.

If ${\Psi }_{i,j}$ is less than a threshold $T$, the two superpixels will be merged. The termination condition is set as $P=3L$, where $P$ is the number of target image patches in the image, and $L$ is the number of labelled categories within the image. The flow chart of superpixel merging algorithm are shown in Figure 2.

3.2. The Patch Based CRF Model Construct

Semantic inference has always been crucial to weakly supervised ISS as it directly affects the segmentation results. For the determined image patches, they are used as nodes of CRF to construct an undirected graph $G\left(V,E\right)$, where $V$ and $E$ denote respectively the node and the edge connecting the nodes. Each image patch needs to be assigned with a unique category of labelled class, which is determined via the CRF model.

On this basis, given the observed field $x$ formed by the extracted patches, the conditional probability distribution of the CRF model for $y$ is defined as:

$$P\left(y|x\right)= \propto \exp \left(-E\left(x,y\right)\right)$$

(4)

where $P\left(y|x\right)$ is a conditional probability, $E\left(x,y\right)$ is an energy function, the final category label assignment result is $\tilde{y}$ which satisfies the maximum posteriori probability.

$$\tilde{y}=\arg {\max }_{y}P\left(y|x\right)$$

(5)

The energy function of patch based CRF model can be defined as:

$$E\left(x,y\right)=w_1{\displaystyle \sum _{i\in v}{\varphi }_i\left(y_i,x_i\right)+w_2{\displaystyle \sum _{\left(i,j\right)\in E}{\varphi }_{ij}\left(y_i,y_j,x_i,x_j\right)}}$$

(6)

where $w_1$ and $w_2$ are weights, ${\varphi }_i\left(y_i,x_i\right)$ is a unary potential energy function, which measures the probability that a node $i$ is labeled as $y_i$ for a given $x$. ${\varphi }_{ij}\left(y_i,y_j,x_i,x_j\right)$ is a pairwise potential energy function between adjacent nodes $i$ and $j$.

Obviously, the solution of $y$ in Equation (5) is the minimum value in Equation (6), thus Equation (5) is equivalent to Equation (7) below.

$$\tilde{y}=\arg {\min }_{y}E\left(x,y\right)$$

(7)

Let $X=\left[x_1,\cdots ,x_P\right]$ be an image containing $P$ image patches, and $x_i$ is the $i\text{-}th$ image patch. Its corresponding semantic category is labelled as $y=\left[y_1,\cdots y_p\right]$ where $y\in \left[1,\cdots L\right]$, and $L$ denotes the total number of categories. However, we can be encoded the label information at the image level, $l\left(x_i\right)={\left[l_1,l_2,\cdots l_L\right]}^T$, where $l_i\in \left[0,1\right]$, and $l_i=1$ means that the category appears in this image, $l_i=0$ means not appearing.

The PBCRF model is used to assign similar image patches with the same semantic categories and less similar patches to different semantic categories. In the process of assigning each image patch to an appropriate semantic label, the unary potential energy function of the CRF is formulated as (8)

$${\varphi }_i\left(y_i,x_i\right)=\frac{1}{Z\left(x\right)}\exp \left({\displaystyle \sum _{x_i\in N\left(i\right)}\left(1-l_i\left(x_i{}^{\ast },x_i\right)\right)}\right)$$

(8)

where $Z\left(x\right)$ is the normalization factor, $N(i)$ refers to the set of image patches adjacent to $x_i$. $l$ is the value of the image label, and $l_i(·)$ equals to only 0 or 1. $l\left(x_i{}^{\ast }\right)\in R^L$ is the true label, $x_i$ is the element of $l_i$.

Furthermore, in order to assign an appropriate semantic label to each superpixel, category correlation and similarity function are added to the pairwise potential function. Pairwise potential energy functions are defined by

$${\varphi }_{ij}\left(y_i,y_j,x_i,x_j\right)=t\left(y_i,y_j\right)\exp \left(-\frac{{\Vert {\Psi }_{x_i}-{\Psi }_{x_j}\Vert }_{}^2}{\delta }\right)+\left(1-\mu \left(y_i,y_j\right)\right)\exp \left(-\frac{{\Vert D_{x_i}-D_{x_j}\Vert }_{}^2}{\delta }\right) \left(y_i\ne y_j\right)$$

(9)

where δ is used to adjust the width of the Gauss nucleus, which is set to $\delta =1$ in the experiment. ${\Psi }_{x_i}$ is the feature descriptor of $x_i$, $D_{x_i}$ is the distance feature of $x_i$.

It is very important to categorize association information for semantic label inference. The category correlation function can be defined by

$$t\left(y_i,y_j\right)=P(l\left(x_i\right)|l\left(x_j\right))\frac{P(l\left(x_i\right)l\left(x_j\right))}{P\left(l\left(x_j\right)\right)}$$

(10)

Let $l={\left[l_x{}_{{}_1},l_x{}_{{}_2},\cdots l_x{}_{{}_P}\right]}^T\in R^{P\times L}$ be the category label of the image and $L$ the total number of label categories In Equation (10), $P(l\left(x_i\right)l\left(xj\right))$ is the probability of both class labels $l\left(x_i\right)$ and $l\left(x_j\right)$, and $P\left(l\left(x_j\right)\right)$ is the probability of the class labels $l\left(x_j\right)$.

At the same time, cosine similarity function is used to test the similarity between semantic categories:

$$\mu \left(y_i,y_j\right)=\frac{{\displaystyle \sum _{i=1}\left(l\left(x_i\right) \mathbf{·} l\left(x_j\right)\right)}}{\sqrt{{\displaystyle \sum _{i=1}l{\left(x_i\right)}^2}}\times \sqrt{{\displaystyle \sum _{i=1}l{\left(x_j\right)}^2}}}$$

(11)

In this way, the semantic label inference is transformed into the energy function of minimizing conditional random fields, and the semantic category of each image patch is the result of minimizing the energy function.

3.3. CRF Based Semantic Inference of Image Patches for ISS

After each image patch is assigned the appropriate semantic label, the image patches belonging to the same class are put together. According to Equation (6), the PBCRF model is constructed, and the mapping issue between category labels and image patches is transformed into a problem of minimizing the energy function. The main steps of CRF based semantic label inference are shown in Algorithm 1:

Algorithm 1: Semantic label inference based on CRF

Input:  Image patches $P$ of training images, image-level semantic label and parameters Output: Semantic Segmentation Results $\hat{y}$ Step 1: Random arrangement of training images Step 2: Constructing undirected graph $G\left(V,E\right)$ with superpixels as nodes Step 3: Calculating unary potential energy function ${\varphi }_i\left(y_i,x_i\right)$ according to Equation (8) Step 4: The class correlation function $t\left(y_i,y_j\right)$ and cosine similarity function $\mu \left(y_i,y_j\right)$ are calculated by Equation (10) and Equation (11), and the pairwise potential energy function ${\varphi }_{ij}\left(y_i,y_j,x_i,x_j\right)$ is calculated by Equation (9). Step 5: Constructing potential energy function of patch based on CRF by Equation (6) Step 6: The semantic segmentation result $\hat{y}$ can be obtained by minimizing the potential energy function

In Section 3.2, we add the category correlation and similarity information to the pairwise potential energy function to for semantic label inference.

$$\hat{y}=\arg {\min }_{y}E\left(x,y\right)$$

(12)

4. Experiments and Discussion

In this section, comprehensive experiments on two publicly available datasets, MSRC and PASCAL VOC 2012, are used to evaluate the performance of our proposed IPCRFWSS method for ISS. Relevant details including the description of the datasets, parameter settings and benchmarking with several state-of-the-art approaches are presented as follows.

4.1. Dataset Description

Comparative experiments were conducted on two standard datasets, MSRC and PASCAL VOC 2012, both are multi-class data sets including many common natural scenes as detailed below.

MSRC: A multi-class dataset which contains 591 pictures in 21 categories, of which ~80% of the pictures have multiple categories. We divide the dataset into training and test sets according following the same way in [39]. As shown in Figure 3, the 21 categories include: aeroplane, building, bike, bird, book, body, boat, cow, car, chair, cat, dog, face, flower, grass, road, sheep, sky, sign, tree, and water.

PASCAL VOC 2012: Serving as the segmentation benchmark for weakly supervised ISS for years [40], this dataset contains 20 object categories and one background category. It contains three parts: training (1464 images), validation (1449 images) and testing (1459 images). In this multi-class dataset, almost each image has 2 to 4 categories, where most of the images have a complicated background. Figure 4 shows for example some typical images and the corresponding ground-truth. These pictures can be subdivided into four main categories [41] i.e., Vehicles: aeroplane, bicycle, bus, car, motorbike, and train; Animals: bird, cat, cow, dog, horse, and sheep; Household: bottle, chair, dining table, potted plant, sofa, and TV/monitor; and Person, including adults and children though these are not explicitly labelled.

4.2. Parameter Settings and Evaluation

To show more intuitively the effect of initial superpixel K on superpixel segmentation, the three most widely used evaluation indicators are adopted. Boundary recall (BR) measures is the proportion of the target boundaries recovered by the superpixel boundaries. Achievable segmentation accuracy (ASA) is a performance upperbound measure. Under-segmentation error (UE) is an error generated by the algorithm when the image is segmented compared with ground truth. The definition of evaluation index used in [42] is adopted here. We compared the number of initial superpixels manually set on MSRC datasets, which visualizes the effect of the number of initial superpixel K on the result of superpixel segmentation. The effect of the initial number of superpixels K is shown in Figure 5.

As shown in Figure 5, from the three evaluation indicators, the performance of image segmentation also improves with the increase of the initial number of superpixels $K$ in a specific range. After that, it will be saturated later. In this state, although the value of $K$ continues to increase, the segmentation performance remains basically unchanged. If the value of $K$ is too large for the image segmentation, it will increase redundant information and the complexity of subsequent calculation.

The accuracy of superpixel segmentation directly affects the results of subsequent ISS, and the value of $K$ determines the size and number of superpixels. If the value of $K$ is too small to achieve good segmentation results, while too large $K$ will bring redundant information.

Therefore, the appropriate $K$ helps to achieve good segmentation results. For visual assessment, we conduct a comparison experiment on the number of superpixels $K$ in SLIC superpixel segmentation. As shown in Figure 6 and

Table 1. The effect of different $K$ values on the precision of segmentation.

Test Image and Size	$\mathbf{Manually} \mathbf{Set} \mathit{K}$	F1	$\mathbf{Determined} {\mathit{K}}^{\prime }$	F1
Image 1 (320 × 213)	150	0.829	275	0.832
	350	0.816
Image 2 (320 × 213)	250	0.834	320	0.857
	450	0.828
Image 3 (350 × 375)	250	0.823	323	0.845
	450	0.853
Image 4 (500 × 375)	150	0.852	280	0.841
	450	0.834

, the proposed algorithm is feasible in generating satisfactory results on different cases.

Figure 6b,e show the results of image segmentation from our algorithm with the parameter K being set as 310 and 332, respectively. To better compare the effectiveness of the algorithm, the experiment compares the K obtained by our proposed algorithm with the segmentation result of K around 100 (K is set manually with traditional SLIC). Figure 6a is the result of K = 250, the head and tail of the plane are not well segmented. Figure 6d shows the Figure 6 the segmentation result of K =250. The outline of the plane and some details are not very good. Figure 6b,e have better segmentation effects.

In Table 1, although the number of superpixels is increasing, the F₁-score has not increased significantly. The proposed method can determine the number of superpixels according to different images without multiple attempts to determine appropriate superpixels, which thus saves the running time cost and improves the efficiency.

4.3. Comparing with the State-of-the-Art

Here, we perform a group of experiment to evaluate the performance of weakly supervised ISS method. For reference, we compared the state-of-the-art methods, such as PLSA [43], WSDC [30], Textonboost [39], MIM [29].

Table 2. Comparison with the state-of-the-art algorithms in term of IOU (%) on the MSRC dataset.

Methods	Building	Grass	Tree	Cow	Sheep	Sky	Aeroplane	Water	Face	Car	Bicycle	Flower	Sign	Bird	Book	Chair	Road	Cat	Dog	Body	Boat	Miou	Stdev.
PLSA [43]	45	64	71	75	74	86	81	47	1	73	55	88	6	6	63	18	80	27	26	55	8	50.0	28.8
WSDC [30]	49	-	58	43	66	-	36	-	46	52	40	85	60	48	-	54	-	52	51	-	-	52.9	11.7
Textonboost [39]	62	98	86	58	50	83	60	53	74	63	75	63	35	19	92	15	86	54	19	62	7	57.8	25.5
MIM [29]	12	83	70	81	93	84	91	55	97	87	92	82	69	51	61	59	66	53	44	9	58	66.5	23.8
Ours	35	82	76	80	73	78	77	56	76	78	75	89	72	60	77	57	65	58	64	57	56	68.6	12.3

shows the segmentation performance on MSRC dataset. The proposed algorithm is compared with CCNN [44], EM-Adapt [37], MIL-ILP-seg [45], SN-B [36] and H&M [46]. The experiments of these methods are carried out on PASCAL VOC 2012 dataset, and the performance of image segmentation is shown in

Table 3. Comparison with the state-of-the-art algorithms in term of IOU (%) on the PASCAL VOC 2012 dataset.

Methods	Background	Aeroplane	Bicycle	Bird	Boat	Bottle	Bus	Car	Cat	Chair	Cow	Dining table	Dog	Horse	Motorbike	Person	Potted plant	Sheep	Sofa	Train	Tv	Miou	Stdev.
CCNN [44]	69	26	18	25	20	36	47	47	48	16	38	21	44	35	46	41	30	36	22	39	37	35.3	12.6
EM-ADAPT [37]	76	37	22	42	26	38	51	45	49	17	41	29	47	46	55	28	30	44	29	34	46	39.6	12.9
MIL-ILP-SEG [45]	79	48	21	31	28	35	51	55	53	8	56	20	54	50	40	39	28	52	25	33	46	40.6	15.9
SN-B [36]	82	54	12	54	30	42	47	46	50	17	49	17	61	52	62	36	25	58	19	49	46	43.2	17.4
H&M [46]	78	51	18	47	32	35	59	51	50	16	40	27	45	48	55	32	30	44	29	34	46	41.9	14.2
Ours	78	35	23	48	33	39	53	55	52	18	49	24	51	53	44	52	32	49	24	32	43	42.2	13.8

. The performance is measured in terms of pixel intersection-over-union (IoU) and mean intersection-over-union (mIoU) across 21 classes. In Table 2 and Table 3, each column represents the accuracy of each semantic class, and the last column is the average accuracy of all classes. For the values in the table, the values in bold represent the best segmentation performance of this category.

As seen in Table 2, our method provides competitive results when compared with the state-of-the-art methods on the MSRC dataset. Although the accuracy is not as high as others in some categories, yet the overall mIoU is the best among the group. Actually, our approach has produced the best results in six categories, whilst the second and the third overall best approaches, MIM and Textonboost, are dominant in eight and six categories, respectively. This shows our approach can balance in between different classes for good overall performance.

For the results on the PASCAL VOC 2012 dataset in Table 3, our results are the second best in terms of mIoU, which is quite comparable to the best one produced by SN-B, a deep learning-based model. However, our approach significantly outperforms another deep learning model, CCNN, and two other approaches, MIL-ILP-seg and EM-Adapt. Actually, SN-B produces the best results in nine categories of objects, whilst our approach generates the best in seven categories, although the overall mIoU is only 1% lower. Again, the proposed approach seems to be more robust over different categories.

To show the performance of our proposed algorithm more intuitively, extensive experiments were performed on the PASCAL VOC 2012 dataset. As shown in Figure 7, the experimental results are compared with the ground truth. It can be seen from the comparison of segmentation results in Figure 7. that better segmentation results can be achieved when the image object contains only one dominant (merged) superpixel or if the background is relatively simple. On the contrary, when the background of the image is more complex, the accuracy of ISS will also be reduced. In addition, for example, there are many objects in the image, so that occlusion or small shadows between these objects will affect the result of ISS.

5. Conclusions

In this paper, a novel PBCRF model is proposed for ISS with image-level labels, which provides an effective solution to the weakly supervised ISS problems. It has three advantages over existing approaches. First, based on the improved SLIC algorithm, optimal numbers of superpixels are automatically estimated for different images for improving the accuracy of image segmentation rather than using a fixed parameter. Second, by taking an image patch as the basic processing unit of ISS, this has significantly improved the performance and reduced the computational costs. Last but not the least, by combining category correlation and similarity information of each semantic category in training the PBCRF model, the inference of semantic label is transformed into the problem of minimizing a potential energy function. Extensive experimental results conducted on the MSRC and PASCAL VOC 2012 datasets segmentation benchmark have demonstrated that the proposed IPCRFWSS algorithm can produce improved or comparable results in comparison to a few state-of-the-art, even some deep learning methods. An improved or much higher mIoU along with a lower variance has also indicated the proposed approach is more robust to different semantic categories. To further improve the results in semantic image segmentation, we will focus on three topics in the future. The first is fusion of color, edge and other information for refined segmentation [47,48], and the second is saliency based extraction of objects from images [49,50]. The third direction is deep learning based image segmentation and object detection, where convolutional neural networks and other models will be explored [51,52], even in combination with the first two topics such as multiscale segmentation and extreme learning machines [53,54].