A PDE-Free Variational Method for Multi-Phase Image Segmentation Based on Multiscale Sparse Representations ^†

Abstract

We introduce a variational model for multi-phase image segmentation that uses a multiscale sparse representation frame (wavelets or other) in a modified diffuse interface context. The segmentation model we present differs from other state-of-the-art models in several ways. The diffusive nature of the method originates from the sparse representations and thus propagates information in a different manner comparing to any existing PDE models, allowing one to combine the advantages of non-local information processing with sharp edges in the output. The regularizing part of the model is based on the wavelet Ginzburg–Landau (WGL) functional, and the fidelity part consists of two terms: one ensures the mean square proximity of the output to the original image; the other takes care of preserving the main edge set. Multiple numerical experiments show that the model is robust to noise yet can preserve the edge information. This method outperforms the algorithms from other classes in cases of images with significant presence of noise or highly uneven illumination.

1. Introduction

Image segmentation is a technique of partitioning an image domain into multiple regions such that each region is homogeneous with respect to some characteristic such as intensity, texture and/or color. It is often used to locate objects or to find the respective boundary. It is an active research topic with numerous practical applications. Examples include medical imaging, machine vision and video surveillance. Image segmentation can be very challenging for images with noise, low contrast and multi-scale, multi-directional details. Conducting multiphase segmentation, rather than binary “object vs. background” segmentation in these cases adds complexity to the problem.

Image segmentation has been studied for many years. Methods in the literature include simple intensity thresholding, edge detection-based and clustering-based segmentation, watershed approaches, variational and graph-based techniques. A variety of methods was proven to be efficient for a variety of images (datasets) with different properties. For instance, thresholding [1,2,3] is the simplest and fastest method that works well for images with clearly-defined boundaries between target regions and without significant noise present. The watershed method [4,5] tends to be more successful when the objects of interests are closely grouped together and, possibly, touch and share their boundary (e.g., microscopy imagery of cells). It however requires correct seeds for each object to segment [6,7,8]. Image segmentation can be very difficult for images with noise, low contrast and/or multi-scale, multi-directional details. Variational models usually involve solving an appropriate minimization problem, and thus are, on the one hand, more time consuming, yet, on the other hand, are more robust and yield more accurate results due to the regularity and fidelity terms providing clear and numerically-implementable requirements on the output. Many variational segmentation models originate from the Potts model [9] that minimizes the sum of the boundary length and data fidelity. The Potts model is NP hard in multiphase cases. Graph cut-based models were proposed to approximate global minimizers in computationally-efficient ways [10,11,12,13,14,15]. The Mumford–Shah model [16] can be viewed as a discrete case of the Potts model that assumes piecewise smoothness of the image intensity.

The geodesic active contour model [17], Chan–Vese active contour model [18,19] and other level set methods represent another important class of techniques that were proposed to improve segmentation efficiency. In particular, the multiphase Vese–Chan model [19] assumes piecewise constant image intensity and is easier to implement than Mumford–Shah. A method involving $\mathrm{\Gamma }$ approximations of the total variation by functionals with equispaced multiple well potentials in the context of multiphase segmentation is presented in [20]. However, due to the nonconvexity of the model, the segmentation results highly rely on the selection of the initial approximation, thus requiring careful practical implementation. Recently, convex relaxation-based approaches have been studied to improve the robustness of segmentation, as well as the computational efficiency [21,22,23,24,25,26,27]. There is also work focused on multiphase fuzzy segmentation and applied on medical images [28]. Diffuse-interface models adapted to the graph context were shown to be efficient in binary segmentation for a variety of datasets [29,30]. Other not directly related, but effective segmentation methods listed on the Berkeley segmentation dataset and benchmark website include [31,32,33,34,35]. These techniques are contour detection oriented and are directly associated with the object vs. background classification.

About our method. As was remarked in [19], the multi-phase image segmentation problem is different from any binary segmentation task. Our method, just as the above mentioned level set segmentation, by construction avoids the issues of ‘vacuum’ and ‘overlap’; moreover, m binary functions are enough to encode $2^m$-valued piecewise constant approximation. The case when $m=2$ is considered especially interesting, since, formally, the partition of any piecewise smooth image near any boundary junction can be encoded by only two binary functions, per the four color theorem. Therefore, in our problem setup, we focus on the four-phase image segmentation as a typical case of a problem of approximating a given image by a piecewise constant one, attaining at most four intensity values, with connected components of the output satisfying certain regularity conditions. Those conditions are imposed by the properties of the energy functional we introduce in Section 2.1.

Our model uses such features of diffuse interface behavior as coarsening and phase separation to merge relevant image elements (coarsening) and separate others into distinct classes (phase separation). To balance these two tendencies, one can adjust the diffuse interface parameter $ϵ$, just as in the classical diffuse interface models that arise in material science. However, in the new spatial derivative-free setup, the interface width is no longer proportional to $ϵ$ (due to the well-localized elements in the chosen sparse representation systems and, thus, a completely different diffusive nature of the model), allowing combining the advantages of non-local information processing with sharp edges in the output. Numerical experiments confirm the effectiveness of the proposed method.

This technique is especially suitable for images where the segmented output is expected to have piecewise smooth edges and relatively large connected components (that is determined by the diffuse interface value $ϵ$ and the weights of the fidelity terms; see Section 2.2) within each segmented class (and nearly no isolated pixels).

Novelty and contributions. Preliminary findings about this approach appeared in the SPIE Conference Proceedings [36]. In this paper, we do an in-depth investigation of this segmentation model. We provide a detailed description of the numerical scheme and conditions that make it gradient stable; we include the formal proof of the minimizer existence, a detailed description of the fidelity term choice and parameter choice; we also describe generalizations and adaptations of this method for particular applications: using it with post-processing for specific needs of medical imaging (blood vessel detection) and generalization of the method to process color (vector-valued) images, both illustrated with examples of numerical simulations. Moreover, following up on the requests of this journal’s reviewers, we include the quantitative area-segmentation comparisons to other methods (using benchmarks described in [31] and other works associated with the Berkeley segmentation database(s); please, see Section 4.1 and Appendix B.1).

In Section 1.1 and Section 1.2, we provide a more detailed explanation of our model design, the reasons for using wavelets or similar multiscale systems in the variational context and briefly describe the properties of the wavelet Ginzburg–Landau energy. Section 2 and Section 3 introduce the variational model and its discrete implementation, respectively. Section 4 contains the results of numerical simulations and a discussion of the effectiveness of our method.

1.1. PDE-Free Variational Models Based on Sparse Representations: Motivation

Similarities between image topology features and phase transitions behavior in material science and fluid mechanics motivated the integration of diffuse interface models into image processing [37,38]. These techniques are sometimes based on the phase-field models, such as the one of Modica and Mortola [39], and allow for topology transition between two steady states of a certain physical or artificial system. Such a construction seems naturally consonant with image processing applications that consider the image intensity bitwise and treat (recover) respective binary values as two equilibria of a model system [40,41].

In general, non-linear diffusion models allow the inclusion of a priori knowledge to ensure regularity while preserving important features including edges [42]. Anisotropic diffusion provides the advantage of combining image regularization with the possibility of edge enhancement [43,44].

However, in signal and especially image processing, we deal with discrete signals and spatially localized features, which do not directly fit into the context of differentiation, even in the weak sense, and the Fourier basis, containing the eigen-functions of certain differential operators, does not provide spatial localization. Moreover, Fourier coefficients of a typical image are not sparse.

A logical and elegant resolution of these difficulties lies in constructing a class of operators based on multiscale sparse representation systems, such as multiscale frames (wavelet or other *-lets). These operators can be designed to act on the chosen frame in the same scale proportional manner as the differential operators act on the Fourier basis. It turns out that with a suitable choice of the multiscale sparse representation system, the new operators retain the desired properties of their differential prototypes, while bringing in new advantages, such as being inherently multiscale, well-localized in space and frequency and, for some multiscale sparse representation systems, allowing some control of the directional features. However, even using the most basic systems like 2D separable wavelets yields non-trivial improvement in the diffuse interface imaging techniques.

1.2. Wavelets in Variational Models for Image Analysis and Recovery, Wavelet Ginzburg–Landau Energy

Wavelets are a well-known powerful tool of image processing, primarily for their ability to capture important signal features within a few terms of the wavelet decomposition; significant attention has been devoted to investigating the properties of the sparse representation systems as effective tools of harmonic analysis within new models that better reflect the modern signal processing needs. In this paper, we exploit the intrinsic connections between the classical differential operators and pseudo-differential ones based on sparse representations (already studied in [45,46]) in the new context of the segmentation problem where two functions and possibly some additional parameters need to be recovered (via solving coupled systems of evolutional differential equations).

Wavelets were used for image enhancement and recovery in a multitude of variational settings; see [47,48] and more. In particular, [47] discusses the total variation (TV) minimization in the wavelet domain, that successfully reproduces lost coefficients. The relationship between wavelet-based image processing algorithms and variational problems in general is analyzed in [49,50].

The wavelet Ginzburg–Landau energy was constructed from the idea of designing new types of pseudo-differential energy functionals that inherit important properties of the ones involving derivatives, introduce the advantages of non-local analysis of the image features and leave out the computational drawbacks associated with the discrete differentiation.

The key idea in the design of wavelet Ginzburg–Landau (WGL) [45,46] combines the basic geometric framework of diffuse interface methods with advantages of the well-localized and inherently multiscale wavelet operators. It originally appeared in a variational model that was adapted to a wavelet setting with the goal of using it for image inpainting. However, it is an efficient regularizing term, and it was shown to be successfully used in multiple image analysis and recover settings with properly chosen forcing terms.

The analytical properties of the wavelet Ginzburg–Landau (GL) energy and its imaging applications are described in detail in [45,46]. We present a very brief summary below.

Here and further in the text, whenever the domain of integration is omitted, we assume integrating over the entire domain of the respective functions.

The total variation (TV) seminorm was proven to be a natural and efficient measure of image regularity [51,52]. However, in many imaging applications, one would like to retain its advantages while making the processing non-local and reduce the computational load related to the curvature calculations. The problem can be re-formulated using the phase-field method providing approximation to the TV functional (in the $\mathrm{\Gamma }$ sense). The Ginzburg–Landau (GL) functional,

$$GL(u)=\frac{ϵ}{2}\int {\left|\nabla u(x)\right|}^{2}dx+\frac{1}{4ϵ}\int W(u)dx,W(u)={(u^2-1)}^2,$$

(1)

is a diffuse interface approximation to the total variational functional $\int |\nabla u(x)|dx$ in the case of binary images [39,53].

GL energy is used in modeling of a vast variety of phenomena including the second-order phase transitions. However, if used in signal processing applications, diffuse interface models tend to produce results that are oversmoothed comparing to the optimal output. In the new model, the $H^1$ seminorm ${\int \left|\nabla u(x)\right|}^{2}dx$ is replaced with a Besov (1-2-2) seminorm. This allows one to construct a method with properties similar to those of the PDE-based methods, but without as much diffuse interface scale blur.

The “wavelet Laplace operator” was defined by having the wavelet basis functions as eigenfunctions and acting on those in the same “scale - proportional” manner as the Laplace operator acts on the Fourier basis. Given an orthonormal wavelet $\psi$, the “wavelet Laplacian” of any $u\in L^2(R)$ is formally defined as:

$${\Delta }_{w}u=-\sum _{j=0}^{+\infty }{2}^{2j}\int \langle f,{\psi }_{j,\kappa }\rangle {\psi }_{j,\kappa }d\kappa ,{\psi }_{j,\kappa }=2^{j/2}\psi (2^{j}x-\kappa ).$$

(2)

Then, the “wavelet Allen–Cahn” equation $u_t=ϵ{\Delta }_{w}u-\frac{1}{ϵ}W(u)$ describes the gradient descent in the problem of minimizing the wavelet Ginzburg–Landau (WGL) energy:

$$WGL(u):=\frac{ϵ}{2}{\left|u\right|}_B^2+\frac{1}{4ϵ}\int W(u)dx,\mathrm{where}|u{|}_B^2=\sum _{j=0}^{+\infty }{2}^{2j}\int {\left|\langle f,{\psi }_{j,\kappa }\rangle \right|}^{2}d\kappa$$

(3)

is the square of the Besov 1-2-2 (translation-invariant) semi-norm if the wavelet $\psi$ is r-regular, $r\ge 2$. The same exact energy construction (3) works just as well in the 2D case (in the case of the separable 2D wavelet basis, one needs to include summation over all types of coefficients: ‘horizontal’, ‘vertical’, ‘diagonal’, in the above formulas).

WGL functionals are inherently multiscale and take advantage of simultaneous space and frequency localization (due to the wavelet properties), thus allowing much sharper minimizer transitions for the same values of the interface parameters compared to the classical GL energy.

WGL energy does not have any local maxima, but a local minimum can be found via the gradient descent method. The latter problem is equivalent to solving:

$$\left\{\begin{array}{c}{u}_t=ϵ{\Delta }_{w}u-\frac{1}{ϵ}{u}^3+\frac{1}{ϵ}u\hfill \\ u(x,0)=u_0(x)\in H^1({[0,1]}^2).\hfill \end{array}\right.$$

(4)

The above problem is well-posed: it has a unique solution that exists globally in time and converges to a steady state as $t\to \infty$. The steady state solution is infinitely smooth provided that wavelet $\psi$ used in the construction of the energy has sufficient regularity.

The WGL energy approximates a weighted TV functional in the variational ($\mathrm{\Gamma }$-convergence) sense.

The WGL was initially designed for the recovery of binary signals; however, it is easily adaptable to a wide class of imaging applications via bit-wise processing ([40] and more).

Variational techniques with WGL as the regularizer were developed for such problems as inpainting, denoising, superresolution, segmentation and more [45]. In all of those cases, the minimized energy was respectively modified to include one or more fidelity terms suitable for each type of problem. When used for the purposes of (binary) segmentation, the input of the method was assumed to include “partial classification” of the image into two classes, and the variational model “filled in the gaps”: the diffusive nature of the model allowed propagating the information, and the edge-preserving term enforced the boundaries of the objects.

The model we introduce in this paper is different from other WGL-based ones in several ways. In the new model, we recover a more complicated output: the binary functions that define the separation of image pixels into classes and the values of constants of the piecewise constant approximation within one unified procedure. It is achieved by solving the system of ordinary differential equations associated with the gradient descent minimization of the proposed energy functional.

2. The Proposed Model for the Four-Phase Image Segmentation

2.1. The Problem Setup

Let us denote the original image f and assume it is a function from $H^1({[0,1]}^2)$. We are looking for the four-phase segmented version of the given image, i.e., its nearly piecewise constant approximation in the following form:

$$u(x)=c_1{\varphi }_1(x){\varphi }_2(x)+c_2{\varphi }_1(1-{\varphi }_2(x))+c_3(1-{\varphi }_1(x)){\varphi }_2(x)+c_4(1-{\varphi }_1(x))(1-{\varphi }_2(x))$$

(5)

Function u is, by design, a piecewise constant function attaining four possible values $c_1$, $c_2$, $c_3$, $c_4$, and functions ${\varphi }_1$ and ${\varphi }_2$ are binary (take on values of zero and one). For simplicity, let us assume that $c_1\le c_2\le c_3\le c_4$. Since our method is a diffuse interface, the recovered functions ${\varphi }_i$ will be nearly binary, such that the width of the transition between black and white is comparable to the size of one pixel.

We assume that the desired function u can be obtained by minimizing the following energy:

$$E({\varphi }_1,{\varphi }_2,\overrightarrow{c})=WGL({\varphi }_1)+WGL({\varphi }_2){+\mu \parallel u-f\parallel }_{L^2}^2+\lambda {\left|P{r}_{\mathrm{\Lambda }}(u-f)\right|}_B^2$$

(6)

with respect to functions ${\varphi }_1$ and ${\varphi }_2$, as well as $\overrightarrow{c}=(c_1,c_2,c_3,c_4)$. The minimizing functions ${\varphi }_1$, ${\varphi }_2$ are expected to be nearly-binary functions from $H^1({[0,1]}^2)$. The fidelity part of the energy consists of two terms. The term ${\mu \parallel u-f\parallel }_{L^2}^2$ ensures the mean-square proximity of the resulting piecewise constant output u to the original function f. The edge-preserving term $\lambda |P{r}_{\mathrm{\Lambda }}{(u-f)|}_B^2$ takes care of retaining the most important edge information: here $\mathrm{\Lambda }$ is the subset of the wavelet modes in the wavelet decomposition that are significant enough to be preserved in the segmented version of the image; one of the possible ways of choosing $\mathrm{\Lambda }$ is described in Section 4. The values $\mu$ and $\lambda$ are weighting parameters that are chosen depending on the image quality, detail resolution and amount of reliable edge information that needs to be preserved. The seminorm ${|·|}_B$ denotes the Besov 1-2-2 seminorm (just as in (3)).

The existence of minimizers for E can be proven using a standard compactness argument (see Appendix A.1). The minimizers are not unique (the easiest counterexample is a constant input function). The output of our numerical simulations depends on the initial guess, which allows us to steer the solution in the desired direction. We provide details about the initialization procedure in Section 3.2.

2.2. Motivation for the Energy Design

The main objective of any image processing method is to successfully incorporate all of the known data properties and requirements on the output into a tunable mathematical model. In variational models, the terms of the minimized energy often are categorized as either regularizing or fidelity (forcing) terms. Let us explain the features of the above energy in relation to our segmentation goal.

In our case, the pixels in the output image will be inadvertently grouped into colored connected components. Each of such components should have a reasonably regular geometric structure (due to the influence of the regularizing WGL terms) unless otherwise imposed by the edges of the original image (due to the influence of the edge-preserving and/or spatial fidelity terms). The values of the functions ${\varphi }_1$ and ${\varphi }_2$ are also expected to be nearly binary.

The terms $WGL({\varphi }_1)$ and $WGL({\varphi }_2)$ represent the regularizing part of the energy. They bring in the properties of the diffusive model inherited from the classical second order phase transition ‘coarsening’ and ‘phase separation’, which facilitate the needed evolution within the image: merging relevant image elements (coarsening) and separating others into distinct classes (phase separation). To balance these two tendencies, one can adjust the diffuse interface parameter $ϵ$, just as in the classical diffuse interface models that arise in material science. However, in the new spatial derivative-free setup, the interface width is no longer proportional to $ϵ$ (due to the well-localized elements in the chosen sparse representation systems and, thus, a completely different diffusive nature of the model), allowing combining the advantages of non-local information processing with sharp edges in the output.

For each of the functions ${\varphi }_i$, we aim to minimize WGL, thus making their supports somewhat regularly-shaped (due to the coarsening tendency) and their values nearly binary (due to phase-separation).

The fidelity components of the energy are ${\mu \parallel u-f\parallel }_{L^2}^2+\lambda {\left|P{r}_{\mathrm{\Lambda }}(u-f)\right|}_B^2$. These terms impose consistency with the original image f in the $L^2$ sense and preserve significant edges within the original image.

If the piecewise constant approximation values are known in advance, the simplified version of this algorithm can recover the corresponding regions by finding ${\varphi }_1$ and ${\varphi }_2$. However, in general, the coefficients $c_i$ are not defined in advance and cannot be easily recovered from the histogram analysis of the image values, as shown in Appendix A.2. In the rest of the paper, we assume that $\overrightarrow{c}$ is unknown, and in all of our numerical experiments included in this paper, we recovered ${\varphi }_1$, ${\varphi }_2$ and $\overrightarrow{c}$ simultaneously.

In the rest of the paper, we will show that our method provides a powerful tool of using diffusive (in the alternative sense of gradually suppressing the high scale detail in the context of a multiscale multidirectional representation system such as wavelets) propagation of information without the associated blurring. The trade-off between preserving the fine-scale features and connecting all relevant components is resolved by using the forcing terms, including the edge-preserving term, obtained by the specialized scale-dependent thresholding.

2.3. Solving the Minimization Problem

We look for ${\varphi }_1$, ${\varphi }_2$, the minimizers (local) of the energy (4), as well as for the the coefficients $c_i$, if those are not fixed in advance.

Let $c_i=p_i^2$ for convenience in writing the gradient descent equations. We can do so because we are assuming that the range of values of the image is non-negative; thus, the optimal approximation values (even in the case of noisy images) are non-negative.

Consider a variation of this energy at a point $({\varphi }_1,{\varphi }_2,\overrightarrow{p})$ in the direction $({\psi }_1,{\psi }_2,\overrightarrow{s})$:

$$\begin{array}{c}{\left.\frac{d}{dt}E({\varphi }_1+t{\psi }_1,{\varphi }_2+t{\psi }_2,\overrightarrow{p}+t\overrightarrow{s})\right|}_{t=0}=\hfill \\ {\int }_{{[0,1]}^2}[-{\Delta }_w{\varphi }_1+W^{\prime }({\varphi }_1)+2\frac{\partial u}{\partial {\varphi }_1}(\mu (u-f)-\lambda {\Delta }_{w}P{r}_{\mathrm{\Lambda }}(u-f))]{\psi }_{1}d\overrightarrow{x}+\hfill \\ {\int }_{{[0,1]}^2}[-{\Delta }_w{\varphi }_2+W^{\prime }({\varphi }_2)+2\frac{\partial u}{\partial {\varphi }_2}(\mu (u-f)-\lambda {\Delta }_{w}P{r}_{\mathrm{\Lambda }}(u-f))]{\psi }_{2}d\overrightarrow{x}+\hfill \\ {\int }_{{[0,1]}^2}4(\mu (u-f)+\lambda {\Delta }_{w}P{r}_{\mathrm{\Lambda }}(u-f)]\Phi (x)\overrightarrow{p}d\overrightarrow{x}·\overrightarrow{s}\hfill \end{array}$$

(7)

Here, $\frac{\partial u}{\partial {\varphi }_1}=(c_1-c_2-c_3+c_4){\varphi }_2+(c_2-c_4)$, $\frac{\partial u}{\partial {\varphi }_2}=(c_1-c_2-c_3+c_4){\varphi }_1+(c_3-c_4)$, and:

$$\mathrm{\Phi }=\left[\begin{array}{cccc}{\varphi }_1{\varphi }_2\hfill & 0\hfill & 0\hfill & 0\hfill \\ 0\hfill & {\varphi }_1(1-{\varphi }_2)\hfill & 0\hfill & 0\hfill \\ 0\hfill & 0\hfill & (1-{\varphi }_1){\varphi }_2\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0\hfill & (1-{\varphi }_1)({\varphi }_2)\hfill \end{array}\right]$$

(8)

We look for the minimizers of the chosen energy functional as the steady state solutions (in the weak sense) to the gradient descent system:

$$\left\{\begin{array}{c}{\partial }_t{\varphi }_1={\Delta }_w{\varphi }_1-W^{\prime }({\varphi }_1)-2\frac{\partial u}{\partial {\varphi }_1}(\mu (u-f)+\lambda {\Delta }_{w}P{r}_{\mathrm{\Lambda }}(u-f))\hfill \\ {\partial }_t{\varphi }_2={\Delta }_w{\varphi }_2-W^{\prime }({\varphi }_2)-2\frac{\partial u}{\partial {\varphi }_2}(\mu (u-f)+\lambda {\Delta }_{w}P{r}_{\mathrm{\Lambda }}(u-f))\hfill \\ {\partial }_t\overrightarrow{p}=-{\int }_{{[0,1]}^2}4(\mu (u-f)+\lambda {\Delta }_{w}P{r}_{\mathrm{\Lambda }}(u-f)]\mathrm{\Phi }(x)\overrightarrow{p}d\overrightarrow{x}.\hfill \end{array}\right.$$

(9)

Here, ${\varphi }_1(x,t),{\varphi }_2(x,t)\in H^1({[0,1]}^2\times [0,T])$ and $\overrightarrow{p}(t)\in H^1([0,T])$ for any $T>0$.

Our method has a non-local nature as the equations defining the minimizers (${\varphi }_i$) contain a non-local operator (${\Delta }_w$), thus allowing creating better connections between components of the segmented image. Interaction of the diffusive term that facilitates merging of similar-colored regions and the term minimizing the double-well potential, i.e., responsible for keeping functions ${\varphi }_i$ nearly binary, closely resembles the “coarsening” and “phase separation” attributes of the classical Allen–Cahn equation. However, due to the wavelet-based operator we use instead of the Laplacian, we can achieve better phase separation and, thus, narrower black to white transitions even while requiring larger connected components in the output. In other words, larger values of parameter $ϵ$ indeed facilitate the diffusion process that merges more pixels into one connected component, yet the the loss of contrast (i.e., increasing the width of the black to white transition) is most certainly less noticeable than in diffuse-interface methods based on the classical models involving diffusive differential operators. These features were investigated in detail in [45].

The edge-enforcing component enhances the recovery of more precise boundaries of the segmented regions. The spatial consistency term naturally enforces the $L^2$ proximity between the input image and its segmented version.

Since our model is variational and any minimization process involves an initial guess, we describe one of the possible ways to choose the initial values in Section 3.2.

3. Numerical Implementation

3.1. Numerical Scheme

To compute the minimizers of the proposed variational problem, we implement the gradient descent method described in Section 2.3 via a semi-implicit numerical scheme. To guarantee gradient-stable convergence to a steady state (for functions ${\varphi }_1,{\varphi }_2$), we use the convexity splitting scheme from [54]: $E({\varphi }_1,{\varphi }_2,\overrightarrow{c})=(E_{11}-E_{12})+(E_{21}-E_{22})$, where:

$$E_{11}=\frac{ϵ}{2}|{\varphi }_1{|}_B^2+\frac{ϵ}{2}|{\varphi }_2{|}_B^2+A_1(\parallel {\varphi }_1{\parallel }_{L^2}^2+\parallel {\varphi }_2{\parallel }_{L_2}^2)$$

(10)

$$E_{12}=-\frac{1}{4ϵ}\int W({\varphi }_1)-\frac{1}{4ϵ}\int W({\varphi }_2)+A_1(\parallel {\varphi }_1{\parallel }_{L^2}^2+\parallel {\varphi }_2{\parallel }_{L_2}^2)$$

(11)

$$E_{21}=A_2(\parallel {\varphi }_1{\parallel }_{L^2}^2+\parallel {\varphi }_2{\parallel }_{L_2}^2),E_{22}=A_2(\parallel {\varphi }_1{\parallel }_{L^2}^2+\parallel {\varphi }_2{\parallel }_{L_2}^2{)-\mu \parallel u-f\parallel }_{L^2}^2-\lambda {\left|P{r}_{\mathrm{\Lambda }}(u-f)\right|}_B^2$$

(12)

Here, the constants $A_1,A_2>0$ need to be chosen in such a way that all four energies are convex, making E a difference of strictly convex energy functionals. If $\lambda =0$, it can be achieved by choosing $A_1\gtrsim \frac{1}{ϵ}$ and $A_2\gtrsim \mu$. If we consider only the “continuous, non-discretized” form of the energy, the needed $A_2$, generally speaking, does not exist. However, in the discretized form, all functions involved in the gradient descent are represented by matrices; therefore, only finitely many levels of wavelet decomposition are present in the Besov seminorm in the fidelity term; thus, we may choose $A_2\gtrsim \mu +\lambda 2^{2J_{max}}$, where $J_{max}$ is the maximum level of wavelet decomposition possible for matrices of the chosen size.

Let us describe the discretized version of the gradient descent system. The gradient descent is performed with respect to an artificial time parameter t. Let $\delta t$ denote the time step size, then $t=n\delta t$ after n steps of the gradient descent algorithm. Let the spatial domain $[0,1]\times [0,1]$ be represented by the points on the uniform grid of size $N\times N$ with step h: $(x_j,y_k)$, $x_j=jh$, $y_k=kh$, $0\le j,k\le N$, $h=\frac{1}{N-1}$. Due to the use of the discrete wavelet transform, it is convenient to pick $N=2^l$, $l\in \mathbb{N}$. Let ${\varphi }_i^n$ denote a matrix $N\times N$ that approximates the values of the ${\varphi }_i$ at time $n\delta t$, and $u^n$ is defined accordingly, so $u_{j,k}^n\approx u(jh,kh,\delta t)$. Naturally, we assume that the given image f is a matrix $N\times N$ with entries $f_{j,k}=f(hj,hk)$.

If the intensity values of the output are not fixed and, thus, we update $\overrightarrow{c}$ at each step as well, we let $\overrightarrow{c^n}$ denote the value of the intensity vector after n steps of the gradient descent algorithm.

For the semi-implicit numerical scheme implementing the equations of the gradient descent system, we formulated the above results in the following equations:

$$\begin{array}{c}{u}^n=c_1^n{\varphi }_1^n{\varphi }_2^n+c_2^n{\varphi }_1^n(1-{\varphi }_2^n)+c_3^n(1-{\varphi }_1^n){\varphi }_2^n+c_4^n(1-{\varphi }_1^n)(1-{\varphi }_2^n),\hfill \\ {\varphi }_1^{n+1}-\delta t{\Delta }_w{\varphi }_1^{n+1}+\delta t(A_1+A_2){\varphi }_1^{n+1}={\varphi }_1^n+\delta t(-W^{\prime }({\varphi }_1^n)-2((c_1^n-c_2^n-c_3^n+c_4^n){\varphi }_2^n+\hfill \\ +(c_2^n-c_4^n))(\mu (u^n-f)+\lambda {\Delta }_{w}P{r}_{\mathrm{\Lambda }}(u^n-f)(A_1+A_2){\varphi }_1^n),\hfill \\ {\varphi }_2^{n+1}-\delta t{\Delta }_w{\varphi }_2^{n+1}+\delta t(A_1+A_2){\varphi }_2^{n+1}={\varphi }_2^n+\delta t(-W^{\prime }({\varphi }_2^n)-2((c_1^n-c_2^n-c_3^n+c_4^n){\varphi }_1^n+\hfill \\ +(c_3^n-c_4^n))(\mu (u^n-f)+\lambda {\Delta }_{w}P{r}_{\mathrm{\Lambda }}(u^n-f)+(A_1+A_2){\varphi }_2^n).\hfill \end{array}$$

(13)

If the expected constant intensity values of the output image are not known in advance, they can be computed simultaneously with the segmented region defining functions ${\varphi }_1,{\varphi }_2$:

$$\begin{array}{c}{c}_i^n={(p_i^n)}^2,\hfill \\ p_1^{n+1}=p_1^n(1-\delta t)h^2\sum _{j,k}4(\mu (u_{j,k}^n-f_{j,k})+\lambda {\Delta }_{w}P{r}_{\mathrm{\Lambda }}(u_{j,k}^n-f_{j,k})){({\varphi }_1^n)}_{j,k}{({\varphi }_2^n)}_{j,k}\hfill \\ p_2^{n+1}=p_2^n(1-\delta t)h^2\sum _{j,k}4(\mu (u_{j,k}^n-f_{j,k})+\lambda {\Delta }_{w}P{r}_{\mathrm{\Lambda }}(u_{j,k}^n-f_{j,k})){({\varphi }_1^n)}_{j,k}(1-{({\varphi }_2^n)}_{j,k})\hfill \\ p_3^{n+1}=p_3^n(1-\delta t)h^2\sum _{j,k}4(\mu (u_{j,k}^n-f_{j,k})+\lambda {\Delta }_{w}P{r}_{\mathrm{\Lambda }}(u_{j,k}^n-f_{j,k}))(1-{({\varphi }_1^n)}_{j,k}){({\varphi }_2^n)}_{j,k}\hfill \\ p_4^{n+1}=p_4^n(1-\delta t)h^2\sum _{j,k}4(\mu (u_{j,k}^n-f_{j,k})+\lambda {\Delta }_{w}P{r}_{\mathrm{\Lambda }}(u_{j,k}^n-f_{j,k}))(1-{({\varphi }_1^n)}_{j,k})(1-{({\varphi }_2^n)}_{j,k}).\hfill \end{array}$$

(14)

To update functions ${\varphi }_1$ and ${\varphi }_2$, the operators on the left-hand side of the respective equations are inverted in the wavelet domain. In the actual numerical simulations, we use the stationary wavelet transform (redundant, undecimated wavelet transform) due to its translation invariant properties that are pivotal in image processing (especially when dealing with natural images having no intrinsic dyadic structure).

3.2. Initialization

We rescale the range of the intensity values for each processed image to the interval $[0,1]$ for convenience. In particular, it simplifies the description of feasible ranges for the fidelity weights: $\lambda$ should be of the same order as ${ϵ}^{-1}$, and $\mu$ should be comparable to ${ϵ}^{-1}{2}^{-2j_{max}}$, where $j_{max}$ is the maximum level of wavelet decomposition used in the edge-preserving fidelity term (see more details in the next subsection). In cases when the input image is known to have a significant level of noise present, we rescaled the values and discard the furthest outliers at the same time, namely cut off the lowest and highest $0.05\%$ of the value range.

The initial guess for the gradient descent system is provided by a simple procedure involving preliminary intensity value distribution analysis of the processed image. A faster convergence can be obtained if the initial guess is provided by the k-means algorithm, which, however, is more computationally expensive, so it is not the best choice of the initial guess. Nevertheless, it is worth mentioning that our algorithm, among other things, can be efficiently used as a post-processing routine following k-means segmentation with the purpose of reducing the number of connected components in the output and regularizing their boundary. We discuss this in the context of color image classification in Section 4.3. Appendix A.2 provides more comments on the choice of the initial guess for $\overrightarrow{c}$.

Here and further in the text, we will use the percentile function:

$$pctl(k)=\mathrm{such}\mathrm{value}v\mathrm{that}k\%\mathrm{of}\mathrm{the}\mathrm{image}\mathrm{pixels}\mathrm{have}\mathrm{intensity}\mathrm{values}\mathrm{less}\mathrm{than}v$$

(15)

The initial values of the coefficients $c_i$ can be assigned somewhat uniformly from the range of the given image intensity values; we typically took: $c_1=pctl(5)$, $c_2=pctl(35)$, $c_3=pctl(65)$, $c_4=pctl(95)$ (just as earlier, we assume that f is the given image represented by a matrix $N\times N$ with entries $f_{i,j}$). Then, the preliminary segmented domains can be determined by their indicator functions (matrices) as follows: for $m\in \{1,2,3\}$ and any pixel position $(i,j)$:

$$I_m(i,j)=\left\{\begin{array}{cc}1,\hfill & \mathrm{whenever}argmi{n}_{k\in \{1,2,3,4\}}|f_{i,j}-c_k|=m\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(16)

then:

$${\varphi }_1=max\{I_1,I_2\},{\varphi }_2=max\{I_1,I_3\}$$

(17)

The initial value of the approximating piece-wise constant function u then is $u=c_1{\varphi }_1{\varphi }_2+c_2{\varphi }_1(1-{\varphi }_2)+c_3(1-{\varphi }_1){\varphi }_2+c_4(1-{\varphi }_1)(1-{\varphi }_2)$. An example of such initialization for the image in Figure 1b is shown in Figure 2b,c.

3.3. Choosing the Edge-Forcing Term

The fidelity term $\lambda |P{r}_{\mathrm{\Lambda }}{(u-f)|}_B^2$ is responsible for preserving the main set of edges associated with a subset of wavelet modes that we denoted $\mathrm{\Lambda }$. The wavelet modes that are “significant” in the segmentation of a given image are found in the same adaptive thresholding algorithm that was previously described in [45]. Even though the thresholding approach was originally designed for binary images, it works well in detecting the edges that are ‘locally binary’, i.e., rapid jumps from one nearly constant intensity region to another. Here is a summary of the thresholding procedure we use to determine the edges that correspond to significant intensity jumps and, therefore, to define $\mathrm{\Lambda }$.

The redundant (i.e., stationary or undecimated [55]) wavelet transform of an image f ($N\times N$) produces four matrices of coefficients: approximation, horizontal, vertical and diagonal at each level of decomposition, each matrix of the same size as the original image. Let $C_{j,\overrightarrow{k}}^{\tau }$ formally denote coefficients of the wavelet decomposition of f, indexed by the scale (i.e., dilation parameter) j, translation $\overrightarrow{k}$ and the ‘type’ of the coefficient $\tau$. For any index $(j,\overrightarrow{k},\tau )$, $j=0,\cdots ,J_{max}-1$, $\overrightarrow{k}=(k_1,k_2)$ with $k_i=0,\cdots ,N-1$ and $\tau$ is ‘a’, ‘h’, ‘v’ or ‘d’, and $J_{max}$ is the maximum depth of decomposition (number of scales in the decomposition).

We define the subset $\mathrm{\Lambda }$ of significant modes as follows. For any image, depending on its size and quality, we may decide to preserve only significant coefficients of scales within a certain range: $j_{min}\le j\le j_{max}$ (notice that in our notation, the scales increase from coarsest to finest). If the image contains significant amount of noise, it makes sense to choose $j_{max}\le J_{max}-2$. It does not make sense to preserve the approximation coefficients of the coarsest level; sometimes, it is convenient to disregard one or two coarsest scales of decomposition completely. Within the chosen range of decomposition scales, we pick the significant modes as follows:

$$(j,\overrightarrow{k},\tau )\in \mathrm{\Lambda }\mathrm{if}\mathrm{and}\mathrm{only}\mathrm{if}|C_{j,\overrightarrow{k}}^{\tau }|\ge \gamma 2^{j}SD(C_{j,·}^{\tau }),$$

where $SD$ denotes the standard deviation computed over the set of all coefficients of scale j, within the ‘type’ $\tau$ subset, if $\tau =$‘h’, ‘v’, ‘d’.

If $\tau =a$,

$$(j,\overrightarrow{k},a)\in \mathrm{\Lambda }\mathrm{if}\mathrm{and}\mathrm{only}\mathrm{if}|C_{j,\overrightarrow{k}}^a|\ge \gamma 2^{j-1}SD(C_{j,·}^a).$$

The constant coefficient $\gamma$ used in this scale- and data-dependent thresholding technique helps produce the best results when chosen individually for each image, but most typically has order $O(1)$.

We illustrate the performance of the method as well as technical nuances leading to the best results in the next section. For example, in the segmentation of the image in Figure 1b we used the edge-forcing term defined by the edges shown in Figure 2a; the set of respective coefficients was obtained by the procedure described above. The images representing the edge information used in our model in the processing of other images can be found in Appendix B.2.

The design of the model and numerical experiments indicate that one should not use the edge fidelity term at all or at least not as a dominating fidelity term, in the cases of smooth images with no visible edges or very noisy images where noise can interfere with the edge detection. Whenever the edges are present, while the noise is relatively mild, it makes sense to use the edge fidelity term, as well as the spatial term, with equal or comparable weighting coefficients.

4. Results

4.1. Four-Phase Segmentation of Grayscale Images

Our numerical experiments involve images from several commonly-considered classes: natural, artificial (cartoon-like), medical (MRI) and images from those categories corrupted by noise and blur. We show that our variational multi-phase segmentation approach can efficiently segment various images, including the tough noisy and blurry images with edges of various scales and directions and even harder ones with low contrast.

Highlighting the previously-mentioned feature of the model, a more relaxed dependence on $ϵ$ as the diffuse interface parameter comparing to the classical PDE-based models, let us remark that in our numerical examples, a typical value of $ϵ$ is of order $N^{-1/2}$ and is always greater than $\frac{4}{N}$ (assuming $N\times N$ is the size of the segmented image). It facilitates the wavelet-based diffusion that forms components of reasonable scale with a piecewise smooth boundary, yet the actual black to white transitions are far from blurry and have a width that is small enough to leave no artifacts after thresholding (if necessary).

To further illustrate the method’s effectiveness, we include the outputs of three other related segmentation techniques: the Vese–Chan multi-phase method [19], a graph cut method [14] and a fuzzy segmentation method [28] for comparison.

Figure 1, Figure 2 and Figure 3 illustrate segmentation of a noisy MRI image (Figure 1b) that contains many fine details. Specifically, Figure 2 shows additional details concerning the initialization of the gradient descent minimization and the choice of the edge fidelity term. The MRI image has a size of 256 by 256; the parameters used in simulations are $ϵ=1/32$, $\mu =2.7$, $\lambda =0.75ϵ$.

Figure 4 shows the output of some well-established segmentation methods applied to the same noisy MRI image. Visual examination of the segmented images demonstrates that the fuzzy segmentation result still has noise in its output, and the Vese–Chan method mostly captures the big scale regions. Only the proposed method and the graph cut methods can provide accurate separation of white matter (shown in white), gray matter (shown in gray) and the rest, which contains either cerebrospinal fluid or the background (shown in darker shades). The results of the graph cut and the proposed method are very similar. However, a zoom-in shows that the proposed method recovers more details in the white matter region. It is especially noticeable in the lower parts of the image.

Figure 5 illustrates the fact that the proposed method was the only one that was able to segment the veins of a leaf compared to others. The graph cut method was only able to segment the primary vein while leaving smaller ones out. The four-phase Vese–Chan method was stuck at a local minimum and failed to produce a reasonable segmentation. Somewhat similarly to the graph-cut approach, the fuzzy segmentation segmentation algorithm did well in segmenting the big scale veins, but failed to detect the smaller ones. The ‘Leaf’ image has a size of 128 by 128; the parameters used in simulations are $ϵ=1/22$, $\mu =2$, $\lambda =0.5ϵ$.

The noisy ‘Peppers’ image in Figure 6 is a tricky one to segment due to inhomogeneous illumination. The two large peppers in the middle have very bright spots. The segmentation result of the proposed method is more consistent with human perception. The graph cut and the fuzzy segmentation results have leftover artificial noise, while the Vese–Chan method fails to segment the image into four classes. The parameters of simulation were similar to the ones used for the MRI image except for the edge fidelity that had to be decreased due to the noise present.

Our next numerical segmentation example is segmenting a four-color grayscale image after adding various levels of noise to the original image ‘Sectors’ is shown in Figure 7. Without any noise present, the segmentation precision is more than 99.9% (of pixels are classified correctly), and the vector of constants $\overrightarrow{c}$ is recovered within <0.05% of the actual values.

The robustness of our segmentation method to noise is illustrated in Figure 8: the three images in each row show the noisy input image, direct output of the minimization routine and the result of rounding it to the nearest of the four recovered values $c_i$. The rows correspond to the increasing levels of Gaussian noise added to the original image: 15 dB, 10 dB and 5 dB. The figure caption also contains information about the percentage of correctly-classified pixels.

As the noise level increased, it was necessary to decrease the weight of the edge fidelity term and increase the spatial weighting term. For the ‘Sectors’ image with added noise of SNR = 10 dB, the simulation parameters were $ϵ=\frac{1}{30}$, $\mu =5$, $\lambda =0.02ϵ$. As was mentioned in Section 1.2, the WGL energy is not anisotropic. However, for practical purposes, this anisotropy is not affecting the minimization results in any significant way. It is clearly not an issue when we process natural images, where one cannot visually discern straightforward directional patterns. The performance of the method on test images, such as the ‘Sectors’ example, which contains radial edges at a variety of angles, shows the absence of the impact of the WGL anisotropy on the recovery quality. Since the ground truth segmentation for this image was available, our method’s performance was also evaluated with respect to such benchmarks as the Rand index, variation of information and segmentation cover (described in Section 2.3 in [31] and characterized as appropriate methods for evaluating region-oriented, rather than contour-oriented benchmarks); see

Table 1. Evaluation of the performance for various levels of SNR (comparison to ground truth).

SNR/Benchmark	Rand Index	Variation of Information	Segmentation Covering
15 db	0.9881	0.1903	0.9763
10 db	0.9846	0.2382	0.9716
7 db	0.9807	0.2991	0.9618
5 db	0.8901	0.5099	0.9533
3 db	0.9377	0.7165	0.8759

.

Figure 9 shows the comparison of our method with others when applied to the ‘Sectors’ image with a noise level of 10 dB. We see that the graph cut method could not fight the relatively high level of noise, and the result is still polluted with it. The k-means output is also extremely fragmented. The four-phase Vese–Chan method ended up at a local minimum far from the desired output. The fuzzy segmentation method output looks fine except for the noise still noticeable all over the output image. The same holds for the graph-cut technique.

The quantitative comparison is provided in

Table 2. Segmentation evaluation for several methods applied to the ‘Sectors’ image with 10 dB noise.

Method	Rand Index	Variation of Information	Segmentation Covering
Proposed method	0.9846	0.2382	0.9716
Fuzzy	0.7253	2.3699	0.4622
Vese–Chan	0.8901	1.1218	0.774
Graph Cuts	0.7523	1.1897	0.2711
k-means	0.7081	2.6804	0.4137

below.

We include the segmentation results of some images from the Berkeley segmentation database ([33]); see Figure 10. Additional examples are discussed in Appendix B.1 and are shown in Figure A2 and Figure A3. However, it must be noted that our method is not designed to address the problem of object vs. background detection. It is aiming at classifying the pixels of an image into classes while balancing out the requirements on the output such as the mean-square color similarity within one class, preserving the chosen edges from the original image and the “driving force” of the method: the phase separation and coarsening that lead to grouping pixels into major connected components with a piecewise smooth boundary.

We performed tests on images of sizes $2^n\times 2^n$ (for the convenience of applying the wavelet transform), for n = 7, 8, 9. Depending on the image and the choice of parameters, the simulation (which aims at achieving an energy minimizing steady state, i.e., finding a local minimizer) might require between 100 and several hundreds of iterations. The simulations were performed using a PC (Win 7) with Intel i5-3570K CPU 3.40 GHz, 8 Gb RAM, operating system Windows 7, in MATLAB with no parallel computing optimization in our code. It takes about 30 s per 100 iterations for a 256 by 256 image with the maximum level of wavelet decomposition (five scales) w.r.t. the Daubechies-four wavelet (‘db4’). The processing can be significantly sped up by preprocessing, such as preliminary denoising, but we are not addressing this issue here; all simulations were initialized automatically as described in Section 3.2.

4.2. Blood Vessel Detection in Medical Images

The detection of blood vessels has been an important problem for a while. Most of the successful algorithms are based on machine learning and require a training set of images of comparable quality for successful processing ([56], and many other publications).

The output of our segmentation algorithm applied to an image with clear detalization (see Figure 5a,b) shows the advantage of the multi-phase segmentation applicable to the blood vessel detection, since the vessels can actually belong to two classes out of four classes due to highly variable intensity over the entire image. In a way, the detection becomes about the contrast of the blood vessels versus the background, rather than relying purely on the image intensity values. Here, the edge preserving fidelity term plays an important role. However, the images of leaves with clearly pronounced vessels are different in quality from the bio-medical images that typically contain blood vessels. The detalization in bio-medical images is often much worse due to the difficulties of non-invasive acquisition.

A good example is provided by retinal images, where the detection of blood vessels is sometimes needed to exclude the respective pixels from consideration during automatic or semi-automatic processing (for instance, the ratio of intensities at the green and yellow frequencies provides a reasonable estimation for the density of macular pigment, useful in the diagnostics of age-related macular degeneration [57]).

We consider two examples of our segmentation algorithm applied to retinal images.

The first image we process is a cut-out from a relatively high resolution image, and the capillary structures captured there are barely visible, just because the actual size of those vessels is small relative to the image resolution. The second image contains blood vessels of more various scales, and thus, the corresponding pixels vary more in intensity.

To ‘even out’ the background, we preprocessed the images by removing the coarsest component of the wavelet decomposition:

$$u_{new}=u-\int \langle u,{\varphi }_{0,\kappa }\rangle {\varphi }_{0,\kappa }d\kappa =\sum _{j=0}^{+\infty }\int \langle u,{\psi }_{j,\kappa }\rangle {\psi }_{j,\kappa }d\kappa .$$

In terms of practical implementation, we ‘zeroed out’ the coarsest approximation coefficients in the stationary wavelet decomposition of each image. Now, we can expect the blood vessels to be contained within the set of pixels of the two darkest shades of the segmented image, i.e., those with intensities $c_1$ and $c_2$.

Due to the fact that some components of the segmented outputs are actually wider than the blood vessels, thus merging capillaries with shadows or parts of uneven background, we use the “skeleton” feature of the MATLAB morphological function for binary processing, thus extracting the central line of each component of the set with the characteristic function we detected from the gradient descent output: $\chi =max\{sign({\varphi }_1),sign({\varphi }_2)\}$. To emphasize the center lines corresponding to the darker segments, we widened them, as shown in Figure 11d.

4.3. Color Image Segmentation

Color image segmentation falls into a more general category of vector-valued image segmentation, which has been studied for a while. As was mentioned in the Introduction, each segmentation task has its consistency requirements and the expected output properties. We extend our technique to the color images with the purpose of obtaining an output with a certain lower bound on the size of connected components of segmented regions and the piecewise regularity of the boundary of those connected components.

In the case of color image segmentation, the initial guess becomes even more important (due to the general non-convexity of the minimized energy and the increased complexity of the domain of $\overrightarrow{c}$). Furthermore, it becomes impossible to set the initial condition by elementary percentile-based thresholding; therefore, we propose to use our method as a refining ‘post-processing tool’, after obtaining the initial guess by some relatively simple method like k-means. The “crude” method provides a guess that is somewhat correlated with the desired output $I^0$, ${\overrightarrow{c}}^0$, and our variational method refines its output by running the gradient descent algorithm minimizing the energy E we introduced earlier, thus producing a refined output. We chose to use k-means for obtaining the initial guess because of its straightforward relation to our energy: applying k-means is equivalent to minimizing the spatial fidelity term ${\mu \parallel u-f\parallel }_{L^2}^2$.

For simplicity, we assume that the input color images are square and are provided in the RGB format; thus, we are to process a 3D array I of a size $N\times N\times 3$ of real values $I(x,y,clr)$, where $(x,y)$ is the spatial pixel coordinate, $clr$ is an integer index between one and three corresponding to ‘red’, ‘green’ or ‘blue’, respectively. We interpret the color image I as the set of values of a function f: ${[0,1]}^2\to {\mathbb{R}}^3$. Just as in the 2D case, we look for the segmented version of the given image $I(x,y,clr)$ in the form:

$$u=\overrightarrow{c_1}{\varphi }_1{\varphi }_2+\overrightarrow{c_2}{\varphi }_1(1-{\varphi }_2)+\overrightarrow{c_3}(1-{\varphi }_1){\varphi }_2+\overrightarrow{c_4}(1-{\varphi }_1)(1-{\varphi }_2),\mathrm{where}\overrightarrow{c_1},\cdots ,\overrightarrow{c_4}\in {\mathbb{R}}^3.$$

(18)

Unknown functions ${\varphi }_1$ and ${\varphi }_2$ and the vectors $\overrightarrow{c_1},\cdots ,\overrightarrow{c_4}$ are determined, just as in the 2D case, as those minimizing the energy function:

$$E({\varphi }_1,{\varphi }_2,\overrightarrow{c_1},\overrightarrow{c_2},\overrightarrow{c_3},\overrightarrow{c_4})=WGL({\varphi }_1)+WGL({\varphi }_2){+\mu \parallel u-f\parallel }_{L^2}^2+\lambda {\left|P{r}_{\mathrm{\Lambda }}(u-f)\right|}_B^2.$$

(19)

Here, we use the norm and the seminorm for multi-valued functions, and the set of ‘significant’ wavelet modes $\mathrm{\Lambda }$ can be determined from the grayscale version of the image or for each color channel individually.

Let us emphasize what parts of the algorithm are essentially 3D and discuss how the two characteristic functions of the specific 2D sets come into the four-phase color segmentation scenario. Below is a brief intuitive explanation of our segmentation algorithm adapted to a vector-valued case.

The main relation between the three color channels is provided by the functions ${\varphi }_1$ and ${\varphi }_2$, functions of two (spatial) arguments, with no direct dependence on the third, color-related coordinate. The actual color information of the segmented version of the segmented image is stored in the matrix $C(j,clr)$, $j\in \{1,\cdots ,4\}$ (four constant colors in the output image), $clr\in \{1,2,3\}$ (RGB component indexing). As we noticed before, due to the diffuse interface nature of the method, ${\varphi }_1$ and ${\varphi }_2$ are not exactly binary, but rather nearly-binary: for the purpose of our intuitive explanation, let us assume ${\varphi }_1(x,y)\approx {\chi }_{E_1}(x,y)$, ${\varphi }_2(x,y)\approx {\chi }_{E_2}(x,y)$; then (again, in some approximate sense) the segmented image takes on the value $C(1,:)$ on $E_1\cap E_2$, value $C(2,:)$ on $E_1\setminus E_2$, value $C(3,:)$ on $E_2\setminus E_1$ and the value $C(4,:)$ on ${[0,1]}^2\setminus (E_1\cup E_2)$.

After the initial values of the color constants stored in C and the corresponding sets $E_1$ and $E_2$ are assigned, we run the gradient descent algorithm, updating all of the above at every step, until an equilibrium is attained.

Figure 12 shows an example of four-phase color image segmentation, where the initial guess was obtained by the k-means method (with Euclidean distance).

5. Conclusions

We proposed a variational multi-phase segmentation approach based on the wavelet Ginzburg–Landau energy as the regularizer and thus free of partial differential equations. Our model is based on sparse representation and explicitly involves an edge-enhancing term to better segment components. Compared to PDE-based models that are local, the proposed approach involves diffuse interface behaviors, such as coarsening and phase separation that are all nonlocal and create better connections between image segments. The proposed model works effectively on both scalar and vector-valued images that have low contrast, noise and details of multi-direction and multi-scale.

The method can be further generalized by using a system (frame) other than the classical separable wavelet frame, such as composite wavelets. Details of the design for such WGL-like regularization part of the energy can be found [58]. Even though we only discussed the specifics of the four-phase segmentation, we can extend this approach to segmentation of an image into $2^n$ phases using n functions $\varphi$ and further this approach for multi-dimensional data processing and classification.