A Fuzzy Plug-and-Play Neural Network-Based Convex Shape Image Segmentation Method

Abstract

The task of partitioning convex shape objects from images is a hot research topic, since this kind of object can be widely found in natural images. The difficulties in achieving this task lie in the fact that these objects are usually partly interrupted by undesired background scenes. To estimate the whole boundaries of these objects, different neural networks are designed to ensure the convexity of corresponding image segmentation results. To make use of well-trained neural networks to promote the performances of convex shape image segmentation tasks, in this paper a new image segmentation model is proposed in the variational framework. In this model, a fuzzy membership function, instead of a classical binary label function, is employed to indicate image regions. To ensure fuzzy membership functions can approximate to binary label functions well, an edge-preserving smoothness regularizer is constructed from an off-the-shelf plug-and-play network denoiser, since an image denoising process can also be seen as an edge-preserving smoothing process. From the numerical results, our proposed method could generate better segmentation results on real images, and our image segmentation results were less affected by the initialization of our method than the results obtained from classical methods.

1. Introduction

Convex shape image segmentation, which refers to partitioning convex object regions from images, is a challenging, but important, problem in image processing, since convex objects, such as books, buildings and some human organs, are widely found in many application scenarios [1,2,3,4,5,6]. Mathematically, a convex set means that a line segment linked by any two points in the set also lies in this set. Based on this point, Gorelick in [7] proposed a discrete graph cut-based convex shape image segmentation model, in which all 1-0-1 configurations in the inner region of convex objects in images were penalized. However, as the energy in discrete graph cut-based models is not submodular [8], some zigzag edges can be found in the image segmentation results. To overcome this drawback, continuous convex shape image segmentation methods are proposed. Bae in [9] found that a convex region could be guaranteed its convexity if the curvature of the continuous boundary curves of the region were kept non-negative. Then, by using a signed distance function (SDF), which is also a special level set function, to implicitly represent an estimated object region, Yan, in [10], proposed a level set function-based convex shape image segmentation model, in which a simple linear constraint whereby the Laplacian of the SDF is held non-negative could ensure that this estimated region was convex. However, it is time-consuming to strictly keep the estimated level set function an SDF during the iteration-solving process.

In order to improve the efficiency of convex shape image segmentation, Luo, in [11], proposed employing a classical binary label function (BLF) instead of a level set function, such as the SDF used in [10], to indicate the estimated target region and to construct a quadratic convex shape constraint on the BLF to guarantee the convexity of the estimated region. However, as the non-continuity problem of the BLF is quite similar to the problem of the above graph cut-based image segmentation models, some zigzag edges can also be found in the image segmentation results obtained from the model in [11]. To overcome the zigzag edge problem, the BLF needed to be smoothed. Here, a smoothed BLF is usually called a fuzzy membership function (FMF) [12]. In order to ensure the above quadratic convex shape constraint held for the FMF as well, the FMF needed to approximate to the BLF well. Classically, the convex total variation (TV) regularizer in [13] and some nonconvex regularizers in [14,15,16,17,18,19,20,21] are used to measure the smoothness of the FMF, while some important geometrical structural details of the FMF are preserved. Here, different regularizers mean different smoothness priors are assumed on the FMF. As image smoothness priors are usually learned from some deep convolutional neural networks (DCNNs) in [22,23,24,25] in the field of image denoising, in this paper we tried to integrate a well-trained DCNN, named DRUNet in [26], into our convex shape image segmentation task to obtain better smoothing performance on the estimated FMF. In fact, an FMF itself can be seen as an image, and an image denoising process is equivalent to an image smoothing process, while some important image details need to be preserved during the smoothing process. The integration wherein the DRUNet is embedded into convex shape image segmentation tasks can be seen as a plug-and-play (P&P) approach [27,28,29], since the DRUNet to be employed in this paper is pre-trained for image denoising tasks, rather than being directly trained for image segmentation tasks.

The main idea of a deep P&P approach is that, with the aid of the alternating direction method of multipliers (ADMM) [30,31,32] or half-quadratic splitting (HQS) [33,34,35], an image processing problem can be separated into several subproblems, one of which corresponds to a denoising problem that can be solved via well-trained denoisers of DCNNs. Classically, the deep P&P approach is only used to deal with image restoration problems, such as image deblurring [36,37], image inpainting [38,39] and single image super resolution [40,41], etc. In this paper, we propose introducing the P&P approach to deal with the convex shape image segmentation problem.

In this paper we used FMFs to represent the foreground or the background and to develop the convex shape prior. A P&P network denoiser intended to ensure the FMFs approximated typical BLFs well, since an image denoising process can be regarded as an edge-preserving smoothing process. The main contributions involve three aspects. Firstly, we employ continuous FMFs to represent image regions and propose the convex shape prior. Secondly, a denoising neural network is incorporated into our image segmentation method to control the smoothing process of FMFs. Lastly, a corresponding algorithm is constructed to obtain our final solutions. The P&P–ADMM method and two convex set operators constitute the main body of our algorithm.

The rest of this paper is organized as follows. In Section 2, two related works are reviewed, including a BLF-based convex shape image segmentation model and a typical P&P–ADMM-based image segmentation framework with a DCNN denoiser. Our proposed P&P network-based convex shape image segmentation model and details of our corresponding algorithm of the proposed model are given in Section 3. Numerical results, as well as their analysis, are provided in Section 4. Section 5 concludes the whole paper.

2. Related Work

2.1. The BLF-Based Convex Shape Image Segmentation Model

Let $I:\mathsf{\Omega }\subseteq {\mathbb{R}}^2\to {\mathbb{R}}^d$ be an image defined on the image domain $\mathsf{\Omega }$, where $d=1$ or $d=3$ mean that I represents a grayscale image or a color image, respectively. A general image segmentation model to partition $\mathsf{\Omega }$ into the foreground region ${\mathsf{\Omega }}_0$ and the background region $\mathsf{\Omega }\backslash {\mathsf{\Omega }}_0$ to solve the minimization problem with respect to ${\mathsf{\Omega }}_0$ is as follows:    

$$\underset{{\mathsf{\Omega }}_0}{min}\left\{{\int }_{{\mathsf{\Omega }}_0}{\rho }_0{f}_{I,0}\left(\mathbf{x}\right)d\mathbf{x}+{\int }_{\mathsf{\Omega }\backslash {\mathsf{\Omega }}_0}{\rho }_1{f}_{I,1}\left(\mathbf{x}\right)d\mathbf{x}+\lambda \left|\partial {\mathsf{\Omega }}_0\right|\right\},$$

(1)

where the function $f_{I,0}\left(\mathbf{x}\right)$ (resp. $f_{I,1}\left(\mathbf{x}\right)$) penalizes the dissimilarity between the pixel $\mathbf{x}$ and other pixels in this region ${\mathsf{\Omega }}_0$ (resp. $\mathsf{\Omega }\backslash {\mathsf{\Omega }}_0$), $\partial {\mathsf{\Omega }}_0$ indicates the enclosed boundary curve of ${\mathsf{\Omega }}_0$, $\left|\partial {\mathsf{\Omega }}_0\right|$ represents the length of $\partial {\mathsf{\Omega }}_0$, and ${\rho }_0$, ${\rho }_1$ and $\lambda$ are positive tuning parameters. Usually, $f_{I,i}\left(\mathbf{x}\right)=-log{P}_{I,i}\left(\mathbf{x}\right)$, $i=0,1$, where $P_{I,i}\left(\mathbf{x}\right)$ can be modeled by a Gaussian mixture model defined as follows:

$$P_{I,i}\left(\mathbf{x}\right)=\frac{{\gamma }_i{P}_i\left(I\left(\mathbf{x}\right)\right)}{{\gamma }_0{P}_0\left(I\left(\mathbf{x}\right)\right)+{\gamma }_1{P}_1\left(I\left(\mathbf{x}\right)\right)},$$

(2)

where ${\gamma }_i$$(i=0,1)$ are positive weights satisfied such that ${\gamma }_0+{\gamma }_1=1$, and

$$P_i\left(I\left(\mathbf{x}\right)\right)=\frac{1}{{\left(2\pi \right)}^{K/2}det{\left({\mathsf{\Sigma }}_i\right)}^{1/2}}exp\left(-\frac{1}{2}{\left|\mathbf{F}\left(I\left(\mathbf{x}\right)\right)-{\mu }_i\right|}_{{\mathsf{\Sigma }}_i^{-1}}^2\right),$$

(3)

where the operator $\mathbf{F}(·)$ maps $I\left(\mathbf{x}\right)$ to be a K dimensional feature image for each pixel $\mathbf{x}$, the expectation ${\mu }_i\in {\mathbb{R}}^K$ and the covariance matrix ${\mathsf{\Sigma }}_i\in {\mathbb{R}}^{K\times K}$ are two key features of a general K dimensional Gaussian distribution. Different features about each pixel $\mathbf{x}$ are proposed to deal with partitioning different kinds of images. such as images with inhomogeneous lightness and textures.

As the minimization problem in model (1) is hard to directly minimize with respect to a geometrical variable ${\mathsf{\Omega }}_0$, variable ${\mathsf{\Omega }}_0$ usually needs to be equivalently transferred into a BLF $v\left(\mathbf{x}\right)$, which is defined by:

$$v\left(\mathbf{x}\right)=\left\{\begin{array}{cc}0\hfill & ,\mathbf{x}\in {\mathsf{\Omega }}_0,\hfill \\ 1\hfill & ,\mathbf{x}\in \mathsf{\Omega }\backslash {\mathsf{\Omega }}_0.\hfill \end{array}\right.$$

(4)

Then, model (1) can be converted into the following form,

$$\underset{v\left(\mathbf{x}\right)\in \{0,1\}}{min}\left\{{\int }_{\mathsf{\Omega }}{f}_I\left(\mathbf{x}\right)v\left(\mathbf{x}\right)d\mathbf{x}+\lambda {\int }_{\mathsf{\Omega }}\phi \left(\left|\nabla v\left(\mathbf{x}\right)\right|\right)d\mathbf{x}\right\},$$

(5)

where $f_I\left(\mathbf{x}\right)={\rho }_1{f}_{I,1}\left(\mathbf{x}\right)-{\rho }_0{f}_{I,0}\left(\mathbf{x}\right)$, and when $\phi \left(s\right)=s$, the second term in model (5) equals to $\left|\partial {\mathsf{\Omega }}_0\right|$ in model (1). The first and the second terms of model (5) are called the data fidelity term and the regularization term, respectively. The regularization term is usually specified by some explicit and hand-crafted smoothness priors on $v\left(\mathbf{x}\right)$. Different choices of function $\phi (·)$ mean that different smooth measurements are given on $v\left(\mathbf{x}\right)$, and smoothing levels of $v\left(\mathbf{x}\right)$ on pixels $\mathbf{x}\in \partial \mathsf{\Omega }$ should be higher than pixels $\mathbf{x}\in \mathsf{\Omega }\backslash \partial \mathsf{\Omega }$. Obviously, the performances of model (5) depend largely on different choices of function $\phi (·)$, since most classical regularization terms are usually hand-crafted.

To make the boundary curves of $v\left(\mathbf{x}\right)$ estimated from model (5) have convex shapes, Luo, in [11], introduced a convex shape inequality constraint on the BLF $v\left(\mathbf{x}\right)$ into model (5), namely,

$${\mathsf{\Psi }}_r\left(v\left(\mathbf{x}\right)\right)=v\left(\mathbf{x}\right)b_r\left(\mathbf{x}\right)\ast v\left(\mathbf{x}\right)-\frac{1}{2}v\left(\mathbf{x}\right)\ge 0,\forall r>0,\mathbf{x}\in \mathsf{\Omega },$$

(6)

where ∗ is the classical convolution operator and the neighborhood function $b_r\left(\mathbf{x}\right)$ satisfies

$$b_r\left(\mathbf{x}\right)=\left\{\begin{array}{cc}1/\pi r^2\hfill & ,\left|x\right|\le r,\hfill \\ 0\hfill & ,\mathrm{otherwise}.\hfill \end{array}\right.$$

(7)

For computational simplicity, variable r in constraint (6) is usually quantized to $r_i=4+5(i-1)$$(i=1,2,\cdots )$.

2.2. A P&P–ADMM Image Segmentation Framework with a DCNN Denoiser

For convenience of discussion, model (5) without the constraint $v\left(\mathbf{x}\right)\in \{0,1\}$ can be rewritten into the following more general form,

$$\underset{v\left(\mathbf{x}\right)}{min}\left\{{\int }_{\mathsf{\Omega }}{f}_I\left(\mathbf{x}\right)v\left(\mathbf{x}\right)d\mathbf{x}+\lambda R\left(v\left(\mathbf{x}\right)\right)\right\}.$$

(8)

This model can be solved by using an ADMM approach which contains the following three main steps.

Firstly, by introducing an auxiliary function $w\left(\mathbf{x}\right)$, which equals to $v\left(\mathbf{x}\right)$, model (8) can be converted into the following equivalent constrained form:

$$\underset{v\left(\mathbf{x}\right),w\left(\mathbf{x}\right)}{min}\left\{{\int }_{\mathsf{\Omega }}{f}_I\left(\mathbf{x}\right)v\left(\mathbf{x}\right)d\mathbf{x}+\lambda R\left(w\left(\mathbf{x}\right)\right)\right\},\mathrm{s}.\mathrm{t}.v\left(\mathbf{x}\right)=w\left(\mathbf{x}\right),\forall \mathbf{x}\in \mathsf{\Omega }.$$

(9)

This step is also called a variable splitting step.

Secondly, a so-called augmented Lagrange function $L(v,w,\xi )$ and a proximal operator ${prox}_R:\mathbb{R}\to \mathbb{R}$ need to be, respectively, defined as follows:

$$\begin{array}{cc}\hfill L\left(v\left(\mathbf{x}\right),w\left(\mathbf{x}\right),\xi \left(\mathbf{x}\right)\right)=& {\int }_{\mathsf{\Omega }}{f}_I\left(\mathbf{x}\right)v\left(\mathbf{x}\right)d\mathbf{x}+{\int }_{\mathsf{\Omega }}\lambda R\left(w\left(\mathbf{x}\right)\right)d\mathbf{x}+\hfill \\ \hfill & \frac{\rho }{2}{\int }_{\mathsf{\Omega }}{\left(v\left(\mathbf{x}\right)-w\left(\mathbf{x}\right)+\frac{\xi \left(\mathbf{x}\right)}{\rho }\right)}^{2}d\mathbf{x}-\frac{1}{2\rho }{\int }_{\mathsf{\Omega }}{\xi }^2\left(\mathbf{x}\right)d\mathbf{x}\hfill \end{array}$$

(10)

and

$${prox}_{\lambda R}\left(v\left(\mathbf{x}\right)\right):=arg\underset{w\left(\mathbf{x}\right)}{min}\left\{\frac{1}{2}{\int }_{\mathsf{\Omega }}{\left(w\left(\mathbf{x}\right)-v\left(\mathbf{x}\right)\right)}^{2}d\mathbf{x}+\lambda R\left(w\left(\mathbf{x}\right)\right)\right\},$$

(11)

where $\xi :\mathsf{\Omega }\to \mathbb{R}$ is the Lagrange multiplier and $\rho$ denotes a step size to update $\xi$. In the following, variables $\mathbf{x}$ in formulae are dropped for conciseness. Then, for a given initialization $v^0$ and $w^0$, the ADMM approach obtains a sequence ${\{v^k,w^k,{\xi }^k\}}_{k=1}^{\infty }$ from the following iteration system:

$$\left\{\begin{array}{cc}{w}^{k+1}\hfill & ={prox}_{\lambda R/\rho }(v^k+{\xi }^k/\rho ),\hfill \\ v^{k+1}\hfill & =w^{k+1}-(f+{\xi }^k)/\rho ,\hfill \\ {\xi }^{k+1}\hfill & ={\xi }^k+\rho ·(v^{k+1}-w^{k+1}),\hfill \\ k\hfill & =k+1.\hfill \end{array}\right.$$

(12)

Thirdly, as the minimization problem shown in Equation (11) can also be seen as a denoising problem with a noise level parameter $\lambda$, the proximal operator ${prox}_{\lambda R/\rho }(·)$ can be correspondingly substituted by a more effective pre-trained DCNN denoising operator $Denoiser(·,\lambda /\rho )$, where different $\lambda /\rho$ indicate different noise levels. More precisely, the first sub-problem shown in the iteration system (12) is changed into the following form:

$$w^{k+1}=Denoiser\left(v^k+\frac{{\xi }^k}{\rho },\sqrt{\frac{\lambda }{\rho }}\right).$$

(13)

Note that the above substitution procedure is the so-called P&P approach.

3. Our Proposed Convex Shape Image Segmentation Method

3.1. Our Proposed Model

To overcome the difficulties of minimizing model (5) because of the non-continuity of the BLF $v\left(\mathbf{x}\right)\in \{0,1\}$, we introduced an FMF $u\left(\mathbf{x}\right)\in [0,1]$ into our model as a continuous approximation of the classical BLF $v\left(\mathbf{x}\right)$. Moreover, to ensure the convexity of the FMF $u\left(\mathbf{x}\right)$ in our model, we followed Equation (6) to use the constraint ${\mathsf{\Psi }}_r\left(u\left(\mathbf{x}\right)\right)\ge 0$. However, in order to make $u\left(\mathbf{x}\right)\approx v\left(\mathbf{x}\right)$ and ${\mathsf{\Psi }}_r\left(u\left(\mathbf{x}\right)\right)\approx {\mathsf{\Psi }}_r\left(v\left(\mathbf{x}\right)\right)$ as far as possible, we followed model (8) to introduce the DRUNet [26] as a P&P DCNN into our model. Then, based on model (8), our proposed image segmentation model solves a constrained minimization problem as follows:

$$\begin{array}{cc}\hfill \underset{u\left(\mathbf{x}\right)}{min}\left\{{\int }_{\mathsf{\Omega }}{f}_I\left(\mathbf{x}\right)u\left(\mathbf{x}\right)d\mathbf{x}+\lambda R\left(u\left(\mathbf{x}\right)\right)\right\},\mathrm{s}.\mathrm{t}.& \left(\mathrm{I}\right)0\le u\left(\mathbf{x}\right)\le 1,\hfill \\ \hfill & \left(\mathrm{II}\right){\mathsf{\Psi }}_r\left(u\left(\mathbf{x}\right)\right)\ge 0.\hfill \end{array}$$

(14)

Here, the FMF $u\left(\mathbf{x}\right)$ satisfying constraints (I) and (II) can be seen as a loose version of the classical BLF $v\left(\mathbf{x}\right)$ in model (8) together with constraints $v\left(\mathbf{x}\right)\in \{0,1\}$ as well as ${\mathsf{\Psi }}_r\left(v\left(\mathbf{x}\right)\right)\ge 0$. The regularization term $R\left(u\left(\mathbf{x}\right)\right)$ in our model (14) makes the FMF $u\left(\mathbf{x}\right)$ approximate to the BLF $v\left(\mathbf{x}\right)$ well. More precisely, $R\left(u\left(\mathbf{x}\right)\right)$ ensures that some important details of $u\left(\mathbf{x}\right)$, which can differentiate image foreground and background regions, are preserved well. This is quite similar to the requirements of image denoising on $R\left(w\left(\mathbf{x}\right)\right)$ in model (11), namely, noises on $w\left(\mathbf{x}\right)$ need to be removed/smoothed, while some important image details of $w\left(\mathbf{x}\right)$ need to be preserved. Then, it makes sense that the well-trained $Denoiser(·,·)$, mentioned in Section 2.2, can be integrated into our image segmentation task in a P&P way. However, different from the classical unconstrained model (8), our proposed model (14) is a constrained model. Then, some projecting operators needed to be integrated into the original P&P–ADMM algorithm, discussed in Section 2.2, to solve our constrained model (14).

3.2. Algorithm to Our Model

Constraints (I) and (II) on the FMF $u\left(\mathbf{x}\right)$ in model (14) can be respectively achieved by two simple projection operators, as follows:

$${\mathbf{P}}_1\left(u\left(\mathbf{x}\right)\right)=max\left\{0,min\{1,u(\mathbf{x}\left)\right\}\right\}$$

(15)

and

$${\mathbf{P}}_2\left(u\left(\mathbf{x}\right)\right)=\left\{\begin{array}{cc}u\left(\mathbf{x}\right)\hfill & ,{\mathsf{\Psi }}_r\left(u\left(\mathbf{x}\right)\right)\ge 0,\hfill \\ 0\hfill & ,\mathrm{otherwise}.\hfill \end{array}\right.$$

(16)

Obviously, $0\le {\mathbf{P}}_1\left(u\left(\mathbf{x}\right)\right)\le 1$ and ${\mathsf{\Psi }}_r\left({\mathbf{P}}_2\left(u\left(\mathbf{x}\right)\right)\right)\ge 0$ hold for all $u\left(\mathbf{x}\right)$.

Below, we focus on solving the minimization problem in model (14) without considering constraints (I) and (II):

$$\underset{u\left(\mathbf{x}\right)}{min}\left\{{\int }_{\mathsf{\Omega }}{f}_I\left(\mathbf{x}\right)u\left(\mathbf{x}\right)d\mathbf{x}+\lambda R\left(u\left(\mathbf{x}\right)\right)\right\}.$$

(17)

As the minimization problem (17) is the same as the one shown in model (8), we followed the iteration system (12) to solve the minimization problem (17) and incorporated $Denoiser\left(·,·\right)$ into the iteration system (12). More precisely, for a given initialization $u^0$ and $w^0$, a sequence ${\{u^k,w^k,{\xi }^k\}}_{k=1}^{\infty }$ (k is an iteration index) can be obtained by the following iteration system:

$$\left\{\begin{array}{cc}{w}^{k+1}\hfill & ={Denoiser}_{\lambda R/\rho }\left(u^k+{\xi }^k/\rho ,\sqrt{\lambda /\rho }\right),\hfill \\ u^{k+1}\hfill & =w^{k+1}-(f+{\xi }^k)/\rho ,\hfill \\ {\xi }^{k+1}\hfill & ={\xi }^k+\rho ·(u^{k+1}-w^{k+1}),\hfill \\ k\hfill & =k+1.\hfill \end{array}\right.$$

(18)

Here, $Denoiser\left(·,\sqrt{\lambda /\rho }\right)$ is a well-trained denoising operator with the noise level $\sqrt{\lambda /\rho }$. The solution of the unconstrained minimization problem (17), ${\overline{u}}^{*}$, can be obtained by setting ${\overline{u}}^{*}=u^k$ when $k\to +\infty$ in the iteration system (18). Then, the final solution of our constrained model (14), $u^{*}$, can be generated by:

$$u^{*}={\mathbf{P}}_2\left({\mathbf{P}}_1\left({\overline{u}}^{*}\right)\right).$$

(19)

Our complete algorithm is shown in Algorithm 1. For simplicity, we call our plug-and-play neural network-based convex shape image segmentation algorithm the PPA-CS algorithm. In the algorithm, the initial image segmentation result, $u^0$, is manually obtained. More concretely, a foreground region is roughly marked by several curves, and $u^0\left(\mathbf{x}\right)=0$ for each pixel $\mathbf{x}$ in the region of the convex envelope of the curves, and otherwise, $u^0\left(\mathbf{x}\right)=1$. In addition, we initialized $w^0=u^0$ and ${\xi }^0=0$. The stopping criterion of our PPA–CS algorithm is to reach the maximum iteration number ($k=200$ in our experiments), or to satisfy the relative error $\mathrm{RE}\le Tol$ ($Tol={10}^{-6}$ in our experiments), where:

$$\mathrm{RE}=\frac{{∥u^{k+1}-u^k∥}_2}{{∥u^k∥}_2},$$

(20)

and ${∥·∥}_2$ represents the classical $l_2$ norm.

The figure of methodology about our method is shown in Figure 1.

Algorithm 1 Our proposed plug-and-play neural network-based convex shape image segmentation algorithm (the PPA–CS algorithm).

Initialization: initialize $u^0$ manually, $w^0=u^0$ and ${\xi }^0=0$, give some parameters, including ${\gamma }_0$, ${\gamma }_1$, ${\rho }_0$, ${\rho }_1$, $\rho$ and $\lambda$, and set the iteration index $k=0$.

Repeat:

update $w^{k+1}$ in (18);

update $u^{k+1}$ in (18);

update ${\xi }^{k+1}$ in (18);

project $u^{k+1}={\mathbf{P}}_2\left({\mathbf{P}}_1\left(u^{k+1}\right)\right)$;

$k=k+1$;

Until a stopping criterion is satisfied

Output: our final estimated FMF result $u^{*}=u^k$.

4. Numerical Results and Discussion

In this section, several representative convex shape image segmentation results are listed to validate the performance of our proposed PPA–CS method. Here, the images to be tested were chosen from two widely used image datasets, called the Berkeley segmentation dataset and benchmark 500 (BSDS500) and the Weizmann segmentation evaluation database (WSED). During the evaluation process, our results were also compared with those obtained from the BLF and TV regularizer-based convex shape (BLF-TV-CS) image segmentation method in [11] and the FMF and TV regularizer-based convex shape (FMF–TV–CS) image segmentation method. Note that the only difference between the BLF–TV–CS method and the FMF–TV–CS method lies in the fact that the BLF in the former method is changed to become the FMF in the latter method. The FMF–TV–CS method is also a new method constructed in this paper, and this method can be seen as an intermediate method between the classical BLF–TV–CS method and our finial PPA–CS method. The reasons why we chose these methods as comparisons are that all of these methods are discussed in the variational framework. It is convenient to verify the effectiveness of our designed FMF-based convex shape constraint, as well as the superiority of our introduced P&P network regularizer.

4.1. Environmental Setting

The BLF–TV–CS method, the FMF–TV–CS method and our proposed PPA–CS method were all implemented in the environment of Python 3.10 on an Intel (R) Xeon (R) E-2176M CPU and NVIDIA GeForce GTX 1650Ti PC. Some necessary model parameters, like ${\gamma }_0$, ${\gamma }_1$, ${\rho }_0$, ${\rho }_1$, $\rho$ and $\lambda$, were necessary to start up our PPA–CS method. We simply set ${\gamma }_0={\gamma }_1=0.5$ (resp. ${\rho }_0={\rho }_1=1$), which meant that $P_{I,0}$ and $P_{I,1}$ (resp. $f_{I,0}\left(\mathbf{x}\right)$ and $f_{I,1}\left(\mathbf{x}\right)$) in Equation (2) were assigned the same weights in our model. According to the expressions shown in the iteration system (18), parameter $\rho$ was not only used as a step size for updating the Lagrange multiplier $\xi$, but also played a role in the smoothing process of $Denoiser\left(·,\sqrt{\lambda /\rho }\right)$. By means of testing, we found that we could simply fix $\rho =1$. In addition, we set $\lambda =25$ as the default value and searched for the best $\lambda$ of each image manually. If the target region was rich in detail or the whole foreground was not convex, the $\lambda$ could choose a larger value. We tuned $\sqrt{\lambda }$ from a 1-step in application since the denoiser in our method used $\sqrt{\lambda }$ as the argument when $\rho =1$. To even things up, all of the parameters in the BLF–TV–CS method and the FMF–TV–CS method were optimized. The parameter optimizing process was similar to that of tuning the parameter $\lambda$ in our PPA–CS method.

4.2. Segmentation Results on BSDS500 and WSED

In Figure 2 (resp. Figure 3), we compared our image segmentation results with those obtained from the BLF–TV–CS method by randomly choosing ten test images from BSDS500 (resp. WSED). In both Figure 2 and Figure 3, (a) shows the original test images on which blue and red markers were used to differentiate the foreground region and the background region, respectively. For convenience of discussion, the images shown in Figure 2a (resp. Figure 3a) are numbered Figure 2a $(i,j)$ (resp. Figure 3a $(i,j)$) from left to right and from top to bottom, where $i=1:5$ and $j=1:2$. Similar notations can also be defined, such as Figure 2b $(i,j)$ and Figure 3c $(i,j)$. We also used the notations ${\mathsf{\Omega }}_{\mathrm{blue}}$, ${\mathsf{\Omega }}_{\mathrm{blue}}^{\mathrm{envelope}}$ and ${\mathsf{\Omega }}_{\mathrm{red}}$ to represent the set of blue marker points, the convex envelope region of the blue markers and the set of red marker points, respectively. Then, for pixel $\mathbf{x}\in {\mathsf{\Omega }}_{\mathrm{blue}}^{\mathrm{envelope}}$, the initialization of our image segmentation $u^{\left(0\right)}\left(\mathbf{x}\right)=0$, and for pixels $\mathbf{x}\in \mathsf{\Omega }\backslash {\mathsf{\Omega }}_{\mathrm{blue}}^{\mathrm{envelope}}$, $u^{\left(0\right)}\left(\mathbf{x}\right)=1$, and for the k-th iteration of our PPA–CS method, $u^{\left(k\right)}\left(\mathbf{x}\right)\equiv 0$ for $\mathbf{x}\in {\mathsf{\Omega }}_{\mathrm{blue}}$, and $u^{\left(k\right)}\left(\mathbf{x}\right)\equiv 1$ for $\mathbf{x}\in {\mathsf{\Omega }}_{\mathrm{red}}$.

In Figure 2 and Figure 3, (b) and (c) show image segmentation results obtained from the BLF–TV–CS method and our proposed PPA–CS method, respectively. Here, green curves show the estimated boundaries between foreground regions and background regions. In order to make more detailed comparisons on image segmentation results between the BLF–TV–CS method and our proposed PPA–CS method,

Table 1. Comparison of different image segmentation methods as to whether the estimated shape of the foreground region of each image shown in Figure 2a was convex from vision, where the notation $(i,j)$ corresponds to the image shown in Figure 2a $(i,j)$$(i=1:5,j=1:2)$ and the notation “T” means the corresponding estimated shape was convex. Otherwise, the notation “F” was given.

	(1, 1)	(1, 2)	(1, 3)	(1, 4)	(1, 5)	(2, 1)	(2, 2)	(2, 3)	(2, 4)	(2, 5)
BLF-TV-CS	F	T	F	F	T	F	F	F	T	T
PPA-CS	T	T	T	T	T	T	T	T	T	T

(resp.

Table 2. Comparison of different image segmentation methods as to whether the estimated shape of the foreground region of each image shown in Figure 3a was convex from vision, where the notation $(i,j)$ corresponds to the image shown in Figure 3a $(i,j)$$(i=1:5,j=1:2)$ and the notation “T” means the corresponding estimated shape was convex. Otherwise, the notation “F” was given.

	(1, 1)	(1, 2)	(1, 3)	(1, 4)	(1, 5)	(2, 1)	(2, 2)	(2, 3)	(2, 4)	(2, 5)
BLF-TV-CS	F	T	F	T	F	F	F	T	T	F
PPA-CS	T	T	T	T	T	T	T	T	T	T

) lists the judgments on whether the BLF–TV–CS method and our proposed PPA–CS method could generate a correct convex shape image segmentation result for each test image shown in (a) of Figure 2 (resp. Figure 3).

Table 3. The value of $\lambda$ in the proposed PPA–CS method when it obtained our segmentation results in Figure 2 and Figure 3, where the notation $(i,j)$ corresponds to the image shown in Figure 2 (resp. Figure 3) (a) $(i,j)$$(i=1:5,j=1:2)$.

	(1, 1)	(1, 2)	(1, 3)	(1, 4)	(1, 5)	(2, 1)	(2, 2)	(2, 3)	(2, 4)	(2, 5)
Figure 2	$4^2$	$3^2$	${12}^2$	$7^2$	$5^2$	$5^2$	$5^2$	${11}^2$	$4^2$	$4^2$
Figure 3	${10}^2$	$6^2$	$6^2$	$5^2$	$5^2$	$6^2$	${16}^2$	$4^2$	$1^2$	${14}^2$

presents the optimal value of $\lambda$ for each image in Figure 2 and Figure 3. More precisely, for a given method, if the estimated foreground region from an image was not convex in shape from vision, we marked it with an “F” symbol. If not, we marked it with a “T” symbol.

According to the results shown in Figure 2 and Figure 3, as well as Table 1 and Table 2, we simply found that our proposed PPA–CS method performed better than the BLF–TV–CS method from vision. In fact, for the twenty images shown in Figure 2a and Figure 3a, our method and the BLF–TV–CS method held 100% and 40% accuracy in extracting the convex shape foreground regions of the images, respectively. Furthermore, considering the undesired image segmentation results marked by “F” in Table 1 and Table 2, we found that the estimated foreground regions from the BLF–TV–CS method tended to be strongly hampered by their corresponding similar background regions. For example, according to the undesired image segmentation results of the BLF–TV–CS method, for the original image shown in Figure 2a $(2,2)$, we found that the branch colors in the background region of this image were very close to the feather colors in the foreground eagle region. This proved that the introduction of the DRUNet as a regularizer into our method benefitted our image segmentation performance better than the classical TV regularizer used in the BLF–TV–CS method, since our model inherited the powerful image recognition capabilities of neural networks.

The numbering iterations and CPU time for obtaining the desired image segmentation results of the BLF–TV–CS method and our proposed PPA–CS method on the original images shown in Figure 2a are compared in

Table 4. Iterations and CPU time (in seconds) of the BLF–TV–CS method and our proposed PPA-=CS method for the original images shown in Figure 2a $(1,2)$, $(1,5)$, $(2,4)$ and $(2,5)$.

Methods	$(1,2)$		$(1,5)$		$(2,4)$		$(2,5)$
	Iters	Time/s	Iters	Time/s	Iters	Time/s	Iters	Time/s
BLF-TV-CS	94	7.74	65	5.39	19	1.96	97	8.37
PPA-CS	18	7.95	28	12.87	18	7.95	64	31.22

. Here, the desired image segmentation results are those shown in the subfigures $(1,2)$, $(1,5)$, $(2,4)$ and $(2,5)$ of both Figure 2b and Figure 2c. From the results shown in Table 4, we found that, compared with the BLF–TV–CS method, which took about 5.87 s, in our proposed PPA–CS method it took, on average, about 15.00 s for each image to obtain the desired image segmentation result. However, integrating the results listed in both Table 1 and Table 4, we believe that, compared with the effectiveness of our proposed image segmentation method, the CPU time was a secondary consideration.

4.3. Model Sensitivity to Initial Contours

To validate the robustness of our proposed image segmentation method, we compared some image segmentation results generated by different methods with different initial contours in Figure 4 and Figure 5. In these figures, (a) and (b) show two different groups of image segmentation results with different initial given contours. From left to right, the first columns shown in (a) and (b) are the original images on which foreground and background regions are, respectively, marked by blue and red curves, the second columns show the initial contours of foreground regions, and the last three columns are the image segmentation results obtained from the BLF–TV–CS method, the FMF–TV–CS method and our proposed PPA–CS method, respectively.

Comparing the results shown in the last three columns of Figure 4a, we found that all of the three methods generated the desired image segmentation results when the initial contours were given as the forms shown in the first column of Figure 4a. However, when the initial contours changed to the forms shown in the first column of Figure 4b, the results of the BLF–TV–CS method and the FMF–TV–CS method varied significantly, while our results experienced no noticeable change. In order to further validate the robustness of our proposed method to initial contours, we designed a special test image segmentation experiment in Figure 5. Here, the background region of the original image contained similar content to the foreground region. From the results shown in Figure 5a,b, we found that the desired image segmentation results were obtained from our proposed method, given the two different initial contours shown in the first column of (a) and (b), while the other two methods could not generate effective results in the experiment. Based on the results shown in Figure 4 and Figure 5, we drew a conclusion that our results were more robust to initial contours than the other two methods.

In order to further demonstrate the robustness of our proposed method on different initial contours, Figure 6, Figure 7 and Figure 8 show some image segmentation results generated by our proposed method with two different initial contours for each original image. Here, the original images in Figure 6 and Figure 7 of the results correspond to the ones shown in Figure 2 and Figure 3 and were not properly partitioned by the BLF–TV–CS method, while the original images in Figure 8 are images which could be partitioned by the BLF–TV–CS method in Figure 3. From these results, we found that, for a given original image, our two different image segmentation results had no obvious distinctions from vision. This demonstrated that, even for challenging cases, our proposed method was robust to initial contours. Here, challenging cases meant that proper image segmentation results were hard to obtain from the BLF–TV–CS method in these cases.

5. Conclusions

In this paper, we proposed a novel convex shape image segmentation method called the PPA–CS method. This method is based on a well-trained DRUNet regularizer on a fuzzy membership function. Here, the fuzzy membership function was used to indicate different image regions. The DRUNet regularizer was used to preserve the geometrical structure details of the fuzzy membership function and to make the fuzzy membership function approximate the classical binary label function well. By integrating a P&P–ADMM approach and a projection method, we designed an iteration algorithm to solve our model. Compared with the BLF–TV–CS method and the FMF–TV–CS method, our proposed method not only generated the desired convex shape image segmentation results better by testing the images of the BSDS500 and the WSED, but also showed more robust performance on the choices of initial segmentation contours.