On a Variational and Convex Model of the Blake–Zisserman Type for Segmentation of Low-Contrast and Piecewise Smooth Images

Abstract

This paper proposes a new variational model for segmentation of low-contrast and piecewise smooth images. The model is motivated by the two-stage image segmentation work of Cai–Chan–Zeng (2013) for the Mumford–Shah model. To deal with low-contrast images more effectively, especially in treating higher-order discontinuities, we follow the idea of the Blake–Zisserman model instead of the Mumford–Shah. Two practical ideas are introduced here: first, a convex relaxation idea is used to derive an implementable formulation, and second, a game reformulation is proposed to reduce the strong dependence of coupling parameters. The proposed model is then analysed for existence and further solved by an ADMM solver. Numerical experiments can show that the new model outperforms the current state-of-the-art models for some challenging and low-contrast images.

1. Introduction

Image segmentation is a widely studied yet still challenging subject, especially for new and emerging imaging modalities where Mumford–Shah and extremely strong noise may be present. Of course, extremely simple images with clear contrast, without noise and without blur may be segmented by the simple methods, such as thresholding the image intensity values.

Real-life images inevitably have noise and low contrast which poses a challenge for the simple algorithms. Variational segmentation models generally provide more robust solutions for complex images and can usually be categorised loosely into two categories: edge-based or region-based models. Well-known edge-based methods include Kass et al. [1] and Caselles et al. [2]. Region-based models are generally referred to the pioneering work of Mumford–Shah (MS) [3], with some simplified variants such as Chan–Vese [4,5] that are most widely used.

In the last few years, when mentioning segmentation of challenging images, we would automatically recommend machine-learning-based approaches such as the UNet [6] and Resnet [7]. However, such works are data-dependent, and often, networks are tailored to a specific task. Firstly, they require training data which may not be available (or reliably available) at all. Secondly, we cannot yet conduct automatic transfer learning from a subject area to another to overcome the lack of sufficient training data, e.g., aircraft identification network cannot be adapted to identification of livers in medical imaging. A reliable way of overcoming the lack of sufficient training data is by weakly or semi-supervised learning which uses a small set of training data (in a supervised way) and a larger set of data without annotations (in an unsupervised way) [8,9]. Here, ‘unsupervised’ means that a suitable segmentation model is required; developing such a model is our aim.

This paper addresses the fundamental problem of how to segment low-contrast images where image features of interest have piecewise smooth intensities. In fact, the difficulties of the two problems, namely low-contrast and piecewise smooth features, are well-known challenges. Low contrast implies that edge information by way of image gradients alone is not sufficient enough to detect weak jumps. Moreover, many well-known models such as [4] or its variants assume an input image has approximately piecewise constant intensities; piecewise smooth features imply these models cannot segment such features (or a feature would be split into sub-regions (or multiple phases) according to the intensity distribution, which means that the segmentation is already incorrect). Many approximation models based on the MS [3] can deal with segmentation of piecewise smooth features but not necessarily images displaying low contrast.

Therefore, this paper considers the Blake–Zisserman model [10] which can improve on the MS model [3]. The model [10] cannot be implemented directly and exactly, just as with the MS [3], which was never solved directly.

The rest of the paper is organised as follows. Section 2 briefly reviews related segmentation models. Section 3 introduces our new model and a game theory reformulation to facilitate subsequent solutions. Proof of the solution existence of the game formulation is given. Section 4 presents our numerical algorithm for the game formulation, and Section 5 shows numerical experiments. Brief conclusions are drawn in Section 6.

2. Related Works

The above-mentioned Mumford–Shah model [3] minimises the following:

$$\begin{array}{c}\hfill F_{MS}(g,\mathsf{\Gamma })=\frac{\lambda }{2}{\int }_{\mathsf{\Omega }}{(f-g)}^{2}d\mathbf{x}+{\int }_{\mathsf{\Omega }\backslash \mathsf{\Gamma }}{|\nabla g|}^{2}d\mathbf{x}+\nu {\mathcal{H}}^1\left(\mathsf{\Gamma }\right),\end{array}$$

(1)

given the input (possibly noisy) image $f:\mathsf{\Omega }\to {\mathbb{R}}^2$, where, most importantly, the segmentation is defined by the unknown boundary $\mathsf{\Gamma }$, $g:\mathsf{\Omega }\to {\mathbb{R}}^2$ is a piecewise smooth approximation of f, and ${\mathcal{H}}^1\left(\mathsf{\Gamma }\right)$ denotes the Hausdorff measure (i.e., length of the boundary). In the literature, there are many follow-up works of this paper, proposed to make revised models implementable numerically. Successful results have been obtained. See [11,12,13], among others.

However, for images that have weak edges possibly buried in noise and blur, the Mumford–Shah type models may fail to capture the ‘discontinuities of second kind’ or gradient discontinuity, which may be called the staircasing effect for gradients. The Blake–Zisserman (BZ) type model [10], though less well-known and published earlier than [3], can be very useful for a class of challenging images where MS is less effective; e.g., see [14,15]. The functional of a BZ model takes the form

$$\begin{array}{c}\hfill F_{BZ}(g,\mathsf{\Gamma },{\mathsf{\Gamma }}_{\nabla })=\frac{\lambda }{2}{\int }_{\mathsf{\Omega }}{(f-g)}^{2}d\mathbf{x}+{\int }_{\mathsf{\Omega }\backslash (\mathsf{\Gamma }\bigcup {\mathsf{\Gamma }}_{\nabla })}{\left|{\nabla }^{2}g\right|}^{2}d\mathbf{x}+{\nu }_1{\mathcal{H}}^1\left(\mathsf{\Gamma }\right)++{\nu }_2{\mathcal{H}}^1({\mathsf{\Gamma }}_{\nabla }\backslash \mathsf{\Gamma }),\end{array}$$

(2)

where $g,\nabla g\in BV\left(\mathsf{\Omega }\right)$. Here, ${\mathsf{\Gamma }}_{\nabla }$ is the discontinuity of $\nabla g$. As with the original formulation (1), the BZ model (2) is theoretical, not in a readily solvable form. This paper will propose an approximate and solvable model.

Our work is motivated by Cai–Chan–Zeng [12], who derived a solvable and convex model for (1). We now review this model briefly. As a first step of reformulation of (1), Cai–Chan–Zeng [12] rewrites (1) in an equivalent form

$$\begin{array}{cc}\hfill E(\mathsf{\Sigma },g_1,g_2)=& \frac{\lambda }{2}{\int }_{\mathsf{\Omega }\backslash \mathsf{\Sigma }}{\left(f-g_2\right)}^{2}d\mathbf{x}+\frac{\mu }{2}{\int }_{\mathsf{\Omega }\backslash \mathsf{\Sigma }}{\left|\nabla g_2\right|}^{2}d\mathbf{x}+\hfill \\ \hfill & \frac{\lambda }{2}{\int }_{\mathsf{\Sigma }\backslash \mathsf{\Gamma }}{\left(f-g_1\right)}^{2}d\mathbf{x}+\frac{\mu }{2}{\int }_{\mathsf{\Sigma }\backslash \mathsf{\Gamma }}{\left|\nabla g_1\right|}^{2}d\mathbf{x}+{\mathcal{H}}^1\left(\mathsf{\Gamma }\right),\hfill \end{array}$$

(3)

where $\mathsf{\Gamma }$ is assumed to be a Jordan curve as the boundary $\partial \mathsf{\Sigma }$ for the closed domain $\mathsf{\Sigma }=\overline{Inside\left(\mathsf{\Gamma }\right)}$. Hence, $g_1,g_2$ are defined in the inside and outside of $\mathsf{\Gamma }$, respectively. Of course, both $g_1,g_2$ can be smoothly extended to the entire domain $\mathsf{\Omega }$. A key observation in [12], motivated by [5], is that the term ${\mathcal{H}}^1\left(\mathsf{\Gamma }\right)$, which is the length of $\mathsf{\Gamma }$, may be approximated by ${\int }_{\mathsf{\Omega }}|\nabla g_1|d\mathbf{x}$. Then, viewing the smooth functions $g_1,g_2$ as a single function, the model by [12] is the following:

$$\begin{array}{c}\hfill \underset{g}{min}\frac{\lambda }{2}{\int }_{\mathsf{\Omega }}{\left(f-g\right)}^{2}d\mathbf{x}+\frac{\mu }{2}{\int }_{\mathsf{\Omega }}{\left|\nabla g\right|}^{2}d\mathbf{x}+{\int }_{\mathsf{\Omega }}\left|\nabla g\right|d\mathbf{x}.\end{array}$$

(4)

We now propose a solvable model based on the Blake–Zisserman model (2). Assume the given image is f, and our approximation is $g\in W^{1,2}\left(\mathsf{\Omega }\right)$, with $\nabla g\in {\left(W^{1,2}\left(\mathsf{\Omega }\right)\right)}^2$.

Motivated by the work of [12], we shall respectively approximate the key quantities ${\mathcal{H}}^1\left(\mathsf{\Gamma }\right),{\mathcal{H}}^1({\mathsf{\Gamma }}_{\nabla }\backslash \mathsf{\Gamma })$ by ${\int }_{\mathsf{\Omega }}|\nabla g|d\mathbf{x},{\int }_{\mathsf{\Omega }}|\nabla (\nabla g)|d\mathbf{x}$. Therefore, our initial minimisation model takes the form

$$\begin{array}{cc}\hfill E_{BZ}\left(g\right)=& \frac{\lambda }{2}{\int }_{\mathsf{\Omega }}{(g-f)}^{2}d\mathbf{x}+\frac{{\nu }_1}{2}{\int }_{\mathsf{\Omega }}|{\nabla }^2{g|}^{2}d\mathbf{x}+\frac{{\nu }_2}{2}{\int }_{\mathsf{\Omega }}{|\nabla g|}^{2}d\mathbf{x}\hfill \\ \hfill & +{\int }_{\mathsf{\Omega }}|{\nabla }^{2}g|d\mathbf{x}+{\int }_{\mathsf{\Omega }}|\nabla g|d\mathbf{x}.\hfill \end{array}$$

(5)

While (5) is well-defined in terms of solvability, to facilitate the choice of coupling parameters, we now consider a game formulation. A game formulation encourages independent players to complete with each. Here, each player is a sub-problem in an optimisation formulation; see [16]. Here, independence means that parameters of sub-problems do not have to rely on each other.

3. The New Model and Reformulation as a Nash Game

In this work, we are interested in a particular case of a two-player game formulation. Instead of optimising the single energy (5), we consider a game reformulation, where two individuals, or ‘players’, are involved. The first player is the variable g, and the second one will be introduced by using the idea of operator splitting [17] to reduce the high-order derivatives in (5) as first-order terms and to simplify subsequent solution. The solution to this game is the Nash equilibrium, whose existence must be established. For important techniques and results in game theory and its connections to partial differential equations (PDEs) for other problems, the reader is directed to [18,19,20,21].

More precisely, let $\mathbf{G}$ be an approximation for vector $\nabla g$. Then, we propose our new model, approximating (5), as

$$\begin{array}{cc}\hfill \underset{g}{min}{E}_1(g,\mathbf{G})& =\left\{\frac{{\lambda }_1}{2}\left|\right|f-{g\left|\right|}_2^2+\frac{{\mu }_1}{2}{\left|\right|\nabla g\left|\right|}_2^2+{\left|\right|\nabla g\left|\right|}_1+\frac{{\xi }_1}{2}\left|\right|\mathbf{G}-{\nabla g\left|\right|}_2^2\right\},\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill \underset{\mathbf{G}}{min}{E}_2(\mathbf{G},g)& =\left\{\frac{{\lambda }_2}{2}\left|\right|\nabla f-{\mathbf{G}\left|\right|}_2^2+\frac{{\mu }_2}{2}{\left|\right|\nabla \mathbf{G}\left|\right|}_2^2+{\left|\right|\nabla \mathbf{G}\left|\right|}_1+\frac{{\xi }_2}{2}\left|\right|\mathbf{G}-{\nabla g\left|\right|}_2^2\right\},\hfill \end{array}$$

(7)

where $g\in W^{1,2}\left(\mathsf{\Omega }\right)$ and $\mathbf{G}\in {\left(W^{1,2}\left(\mathsf{\Omega }\right)\right)}^2$.

Definition 1.

A pair $(g^{*},{\mathbf{G}}^{*})$ in the space $W=W^{1,2}\left(\mathsf{\Omega }\right)\times {\left(W^{1,2}\left(\mathsf{\Omega }\right)\right)}^2$ is called a Nash equilibrium for the game involving the two energies $E_1(·)$ and $E_2(·)$, defined on W, if

$$\left\{\begin{array}{cc}& E_1(g^{*},{\mathbf{G}}^{*})\le E_1(g,{\mathbf{G}}^{*}),\forall g\in W^{1,2}\left(\mathsf{\Omega }\right),\hfill \\ & E_2(g^{*},{\mathbf{G}}^{*})\le E_2(g^{*},\mathbf{G}),\forall \mathbf{G}\in {\left(W^{1,2}\left(\mathsf{\Omega }\right)\right)}^2.\hfill \end{array}\right.$$

One could consider only the single energy $E_1+E_2$ to be optimised; however, for the theoretical analysis, the ellipticity of the sum energy is not guaranteed because of the coupling term between g and $\mathbf{G}$. Hence, the existence of minimisers is not straightforward. However, we emphasise that in the game formulation, the energies $E_1(·,\mathbf{G})$ and $E_2(g,·)$ are partially elliptic, i.e., with respect to the variables g and $\mathbf{G}$, respectively. This is a very important property which eases the proof of the existence of Nash equilibrium.

Proposition 1.

There exists a unique Nash equilibrium $(g^{*},{\mathbf{G}}^{*})\in W^{1,2}\left(\mathsf{\Omega }\right)\times {\left(W^{1,2}\left(\mathsf{\Omega }\right)\right)}^2$ for the two-player game involving the costs functional $E_1(·,·)$ and $E_2(·,·)$ in (6) and (7).

Proof of Proposition 1.

Since

• $E_1(·,\mathbf{G})$ is partially strict convex, partially elliptic and weakly lower semi-continuous with respect to variable g,
• $E_2(g,·)$ is partially strict convex, partially elliptic and weakly lower semi-continuous with respect to variable $\mathbf{G}$,

the proof is a straightforward and direct application of the the Nash theorem [22]. □

4. Numerical Algorithms and Implementation

In this section, we detail the numerical algorithm to solve our game model and show how we utilise the outputs to obtain a segmentation result.

4.1. Stage One: Solution of the Main Model Using ADMM

The discretised version of our two-player game model (6) and (7) is given as follows:

$$\begin{array}{cc}\hfill & \underset{g}{min}\left\{\frac{{\lambda }_1}{2}\left|\right|f-{g\left|\right|}_2^2+\frac{{\mu }_1}{2}{\left|\right|\nabla g\left|\right|}_2^2+{\left|\right|\nabla g\left|\right|}_1+\frac{{\xi }_1}{2}\left|\right|\mathbf{G}-{\nabla g\left|\right|}_2^2\right\},\hfill \\ \hfill & \underset{\mathbf{G}}{min}\left\{\frac{{\lambda }_2}{2}\left|\right|\nabla f-{\mathbf{G}\left|\right|}_2^2+\frac{{\mu }_2}{2}{\left|\right|\nabla \mathbf{G}\left|\right|}_2^2+{\left|\right|\nabla \mathbf{G}\left|\right|}_1+\frac{{\xi }_2}{2}\left|\right|\mathbf{G}-{\nabla g\left|\right|}_2^2\right\},\hfill \end{array}$$

where ${\left|\right|\nabla g\left|\right|}_1={\sum }_{i\in \mathsf{\Omega }}\sqrt{{\left({\nabla }_{x}g\right)}_i^2+{\left({\nabla }_{y}g\right)}_i^2}$ and ${\left|\right|\nabla \mathbf{G}\left|\right|}_1={\sum }_{i\in \mathsf{\Omega }}|{\left({\nabla }_x{G}_x\right)}_i+{\left({\nabla }_y{G}_y\right)}_i|$. The gradient operator $\nabla =({\nabla }_x,{\nabla }_y)$ is discretised using backwards differences with zero Neumann boundary conditions.

We aim to solve the coupled problem using the split-Bregman variant of the alternating direction method of multipliers (ADMM) [23], which is commonly used for problems containing $L_1$ regularisation. In order to do this, we introduce a new variable into each sub-problem:

$$\begin{array}{cc}\hfill & \underset{g,\mathbf{v}}{min}\left\{\frac{{\lambda }_1}{2}\left|\right|f-{g\left|\right|}_2^2+\frac{{\mu }_1}{2}{\left|\right|\nabla g\left|\right|}_2^2+{\left|\right|\mathbf{v}\left|\right|}_1+\frac{{\xi }_1}{2}\left|\right|\mathbf{G}-{\nabla g\left|\right|}_2^2\right\},\mathrm{such}\mathrm{that}\mathbf{v}=\nabla g,\hfill \\ \hfill & \underset{\mathbf{G},w}{min}\left\{\frac{{\lambda }_2}{2}\left|\right|\nabla f-{\mathbf{G}\left|\right|}_2^2+\frac{{\mu }_2}{2}{\left|\right|\nabla \mathbf{G}\left|\right|}_2^2+{\left|\right|w\left|\right|}_1+\frac{{\xi }_2}{2}\left|\right|\mathbf{G}-{\nabla g\left|\right|}_2^2\right\},\mathrm{such}\mathrm{that}w=\nabla \mathbf{G}.\hfill \end{array}$$

Applying split-Bregman to enforce the constraints gives us:

$$\begin{array}{cc}\hfill & \underset{g,\mathbf{v}}{min}\left\{\frac{{\lambda }_1}{2}\left|\right|f-{g\left|\right|}_2^2+\frac{{\mu }_1}{2}{\left|\right|\nabla g\left|\right|}_2^2+{\left|\right|\mathbf{v}\left|\right|}_1+\frac{{\xi }_1}{2}\left|\right|\mathbf{G}-{\nabla g\left|\right|}_2^2+\frac{{\rho }_1}{2}\left|\right|\mathbf{v}-\nabla g-{\mathbf{b}}_1{\left|\right|}_2^2\right\},\hfill \\ \hfill & \underset{\mathbf{G},w}{min}\left\{\frac{{\lambda }_2}{2}\left|\right|\nabla f-{\mathbf{G}\left|\right|}_2^2+\frac{{\mu }_2}{2}{\left|\right|\nabla \mathbf{G}\left|\right|}_2^2+{\left|\right|w\left|\right|}_1+\frac{{\xi }_2}{2}\left|\right|\mathbf{G}-{\nabla g\left|\right|}_2^2+\frac{{\rho }_2}{2}\left|\right|w-\nabla \mathbf{G}-b_2{\left|\right|}_2^2\right\}.\hfill \end{array}$$

We detail briefly how to solve each of the sub-problems:

g sub-problem: We aim to solve the minimisation problem for fixed ${\mathbf{v}}^{\left(k\right)},{\mathbf{G}}^{\left(k\right)},{\mathbf{b}}_1^{\left(k\right)}$:

$$g^{(k+1)}=arg\underset{g}{min}\left\{\frac{{\lambda }_1}{2}\left|\right|f-{g\left|\right|}_2^2+\frac{{\mu }_1}{2}{\left|\right|\nabla g\left|\right|}_2^2+\frac{{\xi }_1}{2}\left|\right|{\mathbf{G}}^{\left(k\right)}-{\nabla g\left|\right|}_2^2+\frac{{\rho }_1}{2}\left|\right|{\widehat{\mathbf{v}}}^{\left(k\right)}-\nabla g{\left|\right|}_2^2\right\},$$

where ${\widehat{\mathbf{v}}}^{\left(k\right)}={\mathbf{v}}^{\left(k\right)}-{\mathbf{b}}_1^{\left(k\right)}$, which amounts to solving the following equation for g:

$$\left({\lambda }_1+({\mu }_1+{\xi }_1+{\rho }_1){\nabla }^2\right)g={\lambda }_{1}f+{\xi }_1{\nabla }^T{\mathbf{G}}^{\left(k\right)}+{\rho }_1{\nabla }^T({\mathbf{v}}^{\left(k\right)}-{\mathbf{b}}_1^{\left(k\right)}).$$

This can be solved by using discrete Fourier transforms $\mathcal{F}$:

$$g^{(k+1)}={\mathcal{F}}^{-1}\left(\frac{\mathcal{F}\left(L\right)}{M}\right)$$

(8)

where

$$L={\lambda }_{1}f+{\xi }_1{\nabla }^T{\mathbf{G}}^{\left(k\right)}+{\rho }_1{\nabla }^T({\mathbf{v}}^{\left(k\right)}-{\mathbf{b}}_1^{\left(k\right)}),M={\lambda }_1+({\mu }_1+{\xi }_1+{\rho }_1)\mathcal{F}\left({\nabla }^2\right).$$

$\mathbf{v}$ sub-problem: We aim to solve this minimisation problem for fixed $g^{(k+1)},{\mathbf{b}}_1^{\left(k\right)}$:

$${\mathbf{v}}^{(k+1)}=arg\underset{\mathbf{v}}{min}\left\{{\left|\right|\mathbf{v}\left|\right|}_1+\frac{{\rho }_1}{2}\left|\right|\mathbf{v}-\nabla g^{(k+1)}-{\mathbf{b}}_1^{\left(k\right)}{\left|\right|}_2^2\right\},$$

which is solved analytically by a generalised shrinkage formula:

$$v_x^{(k+1)}=max\left(s^{\left(k\right)}-\frac{1}{{\rho }_1},0\right)\frac{s_x^{\left(k\right)}}{s^{\left(k\right)}},v_y^{(k+1)}=max\left(s^{\left(k\right)}-\frac{1}{{\rho }_1},0\right)\frac{s_y^{\left(k\right)}}{s^{\left(k\right)}},$$

(9)

where $s_1^{\left(k\right)}={\nabla }_x{g}^{(k+1)}+{\left({\mathbf{b}}_1^{\left(k\right)}\right)}_x$, $s_2^{\left(k\right)}={\nabla }_y{g}^{(k+1)}+{\left({\mathbf{b}}_1^{\left(k\right)}\right)}_y$ and

$s^{\left(k\right)}=\sqrt{{\left(s_1^{\left(k\right)}\right)}^2+{\left(s_2^{\left(k\right)}\right)}^2}.$

The associated Bregman update is:

$${\left({\mathbf{b}}_1^{(k+1)}\right)}_x={\left({\mathbf{b}}_1^{\left(k\right)}\right)}_x+{\nabla }_x{g}^{\left(k\right)}-v_x^{(k+1)},{\left({\mathbf{b}}_1^{(k+1)}\right)}_y={\left({\mathbf{b}}_1^{\left(k\right)}\right)}_y+{\nabla }_y{g}^{\left(k\right)}-v_y^{(k+1)}.$$

(10)

$\mathbf{G}$ sub-problem: We aim to solve the minimisation problem for fixed $g^{(k+1)},w^{\left(k\right)},b_2^{\left(k\right)}$:

$${\mathbf{G}}^{(k+1)}=arg\underset{\mathbf{G}}{min}\{\frac{{\lambda }_2}{2}\left|\right|\nabla f-{\mathbf{G}\left|\right|}_2^2+\frac{{\mu }_2}{2}{\left|\right|\nabla \mathbf{G}\left|\right|}_2^2+$$

$$\frac{{\xi }_2}{2}\left|\right|\mathbf{G}-\nabla g^{(k+1)}{\left|\right|}_2^2+\frac{{\rho }_2}{2}\left|\right|w^{\left(k\right)}-\nabla \mathbf{G}-b_2^{\left(k\right)}{\left|\right|}_2^2\},$$

whose solution is defined by the following:

$$({\lambda }_2+{\xi }_2+({\mu }_2+{\rho }_2){\nabla }^2)\mathbf{G}={\lambda }_2\nabla f+{\xi }_2\nabla g^{(k+1)}+{\rho }_2{\nabla }^T(w^{\left(k\right)}-b_2^{\left(k\right)}).$$

To find the solution $\mathbf{G}$, we apply discrete Fourier transforms $\mathcal{F}$:

$$G_x^{(k+1)}=\mathcal{F}\left(\frac{\mathcal{F}\left(N_1\right)}{P}\right),G_y^{(k+1)}=\mathcal{F}\left(\frac{\mathcal{F}\left(N_2\right)}{P}\right),$$

(11)

where $P={\lambda }_2+{\xi }_2+({\mu }_2+{\rho }_2)\mathcal{F}\left({\nabla }^2\right)$, $N_1={\nabla }_{x}f+{\xi }_2{\nabla }_x{g}^{(k+1)}+{\rho }_2{\nabla }_x^T(w^{\left(k\right)}-b_2^{\left(k\right)})$, and $N_2={\nabla }_{y}f+{\xi }_2{\nabla }_y{g}^{(k+1)}+{\rho }_2{\nabla }_y^T(w^{\left(k\right)}-b_2^{\left(k\right)})$.

w sub-problem: We aim to solve the minimisation problem for fixed ${\mathbf{G}}^{(k+1)},b_2^{\left(k\right)}:$

$$w^{(k+1)}=arg\underset{w}{min}\left\{{\left|\right|w\left|\right|}_1+\frac{{\rho }_2}{2}\left|\right|w-\nabla {\mathbf{G}}^{(k+1)}-b_2^{\left(k\right)}{\left|\right|}_2^2\right\},$$

which, similar to (9), is solved by using a shrinkage formula:

$$w^{(k+1)}=max\left(|r^{\left(k\right)}|-\frac{1}{{\rho }_2},0\right)\frac{r^{\left(k\right)}}{|r^{\left(k\right)}|},$$

(12)

where $r^{\left(k\right)}=\nabla {\mathbf{G}}^{(k+1)}+b_2^{\left(k\right)}$.

4.2. Stage Two: Segmentation of f by Thresholding g

In order to acquire a segmentation result for f, we take the minimiser g from stage one and threshold it according to some suitably defined threshold parameter(s). As in [12], the advantage of this method is that changing the threshold value(s) does not require the re-computation of the optimisation done in stage one.

There are two strategies that can be employed to define the threshold(s). The first is to use the k-means algorithm, which is an automatic method that partitions a given input into K clusters, for $K\ge 2$. The second is to define the threshold value(s) manually, which generally provides better results. As the threshold values are applied after optimisation, a wide range of values can easily be tried and the best selected. In our experiments, we use manual threshold values for two-phase segmentation, whereas for multiphase segmentation with multiple threshold values, we use k-means to simplify the process.

5. Numerical Results

In this section, we display some examples of the performance of our model and compare it with a number of models, namely:

• CRCV: Convex relaxed Chan–Vese model [5];
• CCZ: The two-stage convex variant of the Mumford–Shah model by Cai et al. [12] given in (4);
• CNC: The convex non-convex segmentation by Chan et al. [24];
• T-ROF: The T-ROF model by Cai et al. [25];

and also a deep learning model.

We first show some visual comparisons, where noise is added to the original image, and then later do a quantitative analysis on a dataset. Note that all the models above (and ours) except for the CRCV model is capable of multiphase segmentation, whereas the CRCV model (in the Chan–Vese framework) is only capable of two-phase segmentation. For this reason, in the experiments, we only include the CRCV model in two-phase examples.

5.1. Qualitative Results

In Figure 1, we show an image from an ultrasound. We add additive Gaussian noise with mean 0 and standard deviation 10. We display the outputs of all the competing models, the segmentation result overlaid on the original image, and for all but the CRCV show the segmentation result after thresholding (as the segmentation result after thresholding is the binary output shown first). We see that the segmentation result from our model is better at segmenting the object in the image, noticing that our segmentation effectively segments the “tail” part at the top of the object, whereas the CCZ model fails to segment it well. The CRCV and CNC models segment the tail but fail to remove the noise. We note that the T-ROF model is the best competing model but does not quite segment all the tail.

Similarly, in Figure 2, we show another two-phase segmentation example, where we have the clean image but add Gaussian noise with mean 0 and standard deviation 25. It is clear that none of the competing models are as good as ours. Our result manages to preserve more detail in general, notably at the strand at the top, and the curved structure at the bottom of the image, without being susceptible to the noise.

In Figure 3, Figure 4 and Figure 5, we show some examples of multiphase segmentation on MRI images of the brain. In all cases, we add Gaussian noise with mean 0 and standard deviation 17 and run the noisy image as input to both for all models but the CRCV model (as this is a two-phase model only). The output is then given as input to the k-means algorithm with $K=4$. We show the clustering output in the final column of the relevant figures. We see that the segmentation result of our model is better at finding some of the finer edges; for example, the white matter segmentation from our model is in general more detailed than the segmentation from the competing ones.

5.2. Quantitative Analysis

To assess our method quantitatively, we run our model on 20 images in the Digital Retinal Images for Vessel Extraction (DRIVE) dataset (https://drive.grand-challenge.org/ accessed 25 October 2021). We use the manual segmentation image as the clean image and add additive Gaussian noise with mean 0 and standard deviation 100 to use as the input image, as shown in Figure 6, Figure 7, Figure 8 and Figure 9a,b respectively. We display the output of the competing models and our model here as well as a deep learning model (abbreviated as DL). We trained a U-Net [6] network on 15 of the images (and used the other five as validation set), where the noisy image served as input and we trained with binary cross-entropy loss function to match with the clean image. The results are good; however, we lack a large dataset to provide the impressive result that deep learning approaches usually provide.

Figure 6, Figure 7, Figure 8 and Figure 9 are four examples on the given dataset; however, we run on the 20 images available to provide some quantitative analysis. We use the DICE coefficient and the JACCARD similarity coefficient as quantitative measures to evaluate the performance of segmentation results. Given a binary segmentation result $\mathsf{\Sigma }$ from a model and ground truth segmentation $GT$, the DICE coefficient is given as:

$$DICE(\mathsf{\Sigma },GT)=\frac{2|\mathsf{\Sigma }\cap GT|}{\left|\mathsf{\Sigma }\right|+\left|GT\right|}.$$

The JACCARD similarity coefficient is given as:

$$JACCARD(\mathsf{\Sigma },GT)=\frac{|\mathsf{\Sigma }\cup GT|}{\left|\mathsf{\Sigma }\right|+\left|GT\right|-|\mathsf{\Sigma }\cap GT|}.$$

In

Table 1. Quantitative results from images in the DRIVE dataset. Here, we show the two methods evaluated on 20 images and display the mean and standard deviations of both the DICE coefficient and JACCARD score. Note that the DL method was trained on 15 of these 20 images.

	DICE		JACCARD
	$\mathbf{\mu }$	$\mathbf{\sigma }$	$\mathbf{\mu }$	$\mathbf{\sigma }$
CRCV	0.727	0.0291	0.573	0.0358
CCZ	0.914	0.0139	0.843	0.0236
CNC	0.939	0.0131	0.884	0.0233
T-ROF	0.932	0.0083	0.872	0.0145
DL	0.946	0.0091	0.898	0.0163
Ours	0.950	0.0073	0.905	0.0133

, we show the mean and standard deviation values of the DICE and JACCARD scores on the dataset. We see clearly that our model is more effective than the Cai model from these results. We note that the numerical values provided for the DL method are run on all 20 images in the dataset; however, the DL was trained on 15 of these images. This is somewhat of an unfair comparison; however, we see that the numerical values for our approach are still larger than the values for the DL approach despite this. Figure 10 shows the boxplots of quantitative results on the data, for further visualisation.

6. Conclusions

In this paper, we have developed a convex relaxed game formulation of the less well-known Blake–Zisserman model in order to segment images with low contrast and strong noise. The advantages of the game formulation are that the existence of Nash equilibrium can be proved and there is less dependence on parameters for each sub-problem, i.e., parameters of each sub-problem do not rely on each other, and so can be tuned appropriately and separately. The game model was implemented using a fast split-Bregman algorithm, and numerical experiments show improvements in segmentation results over competing models, especially over the well-known Mumford–Shah type methods for low-contrast images.