Graph-Regularized, Sparsity-Constrained Non-Negative Matrix Factorization with Earth Mover’s Distance Metric

Abstract

Non-negative matrix factorization (NMF) is widely used as a powerful matrix factorization tool in data representation. However, the traditional NMF, measured by Euclidean distance or Kullback–Leibler distance, does not take into account the internal implied geometric information of the dataset and cannot measure the distance between samples as well as possible. To remedy the defects, in this paper, we propose the NMF method with Earth mover’s distance as a metric, for short GSNMF-EMD. It combines graph regularization and $L_{1/2}$ smooth constraints. The GSNMF-EMD method takes into account the intrinsic implied geometric information of the dataset and can produce more sparse and stable local solutions. Experiments on two specific image datasets showed that the proposed method outperforms related state-of-the-art methods.

1. Introduction

The non-negative matrix factorization (NMF) method has been widely used in various fields of feature learning and has become one of the most popular methods, among which are text clustering [1,2,3], digital image processing [4,5,6,7,8,9], face recognition [10], and signal analytics [11,12]. Due to the wide range of practical properties of NMF, numerous scholars have proposed some improvements to the original NMF: Imposing different constraints on the two factor matrices. For example, P. O. Hoyer [13] proposed NMF with sparseness constraints, which improve the accuracy of the parts-based representation compared to the basic NMF. Cai et al. [14] proposed a manifold-based NMF (GNMF) method that respects the geometric structure information hidden inside the dataset. He et al. [15] proposed a robust NMF method with sparse constraints in order to deal with both sparse and Gaussian noise. Kong et al. [16] added the $L_{21}$-norm constraint and proposed a robust non-negative matrix decomposition algorithm. Huang et al. [17] proposed a new unsupervised learning model that takes into account the overall and local structure of the data space, called NMF with robustness. Pan et al. [18] introduced an orthogonal non-negative matrix factorization. Luo et al. [19] applied NMF to the collaborative filtering problem and proposed a regularized single-element-based model (RSNMF) that achieves computational complexity reduction and accuracy improvement for large industrial datasets. Sun et al. [20] analyzed the generalization performance of the NMF algorithm from the perspective of algorithm stability and gave bounds on the generalization error.

The vast majority of the various NMF-based variants mentioned above use the Euclidean norm or the K-L divergence [21] to measure the minimization distance between the product of two factor matrices and the original matrix. A shortcoming of the above NMF-based approaches is that either the intrinsic structure of the dataset or the distance between samples is ignored. To remedy the shortcoming, many researchers have tried to adopt new metrics. Recently, correntropy has proven to be a very effective approximation measure by virtue of its stability against outliers or noise [22,23,24,25,26]. For example, Yu et al. [26] proposed the correntropy-based, hypergraph-regularized NMF (CHNMF) method using correlation entropy. The method is used for clustering and feature extraction of multi-cancer integrated data with better robustness. Another measure, called the Earth mover’s distance (EMD), has also attracted great interest. EMD not only provides a better measure of the distance between samples but also has good robustness [27,28]. In [28], the authors showed that EMD has good performance in tasks involving large distortions, such as geometric deformations, illumination changes, or heavy intensity noise.

Rubner et al. [29] constructed an image comparison framework that better accounts for perceptual similarity than other previously proposed methods by combining EMD with a vector quantization-based distribution representation scheme. Sandler et al. [30,31] proposed an EMDNMF algorithm that minimizes the EMD error between the data and the product of factor matrices. The main advantage of this method is enhanced robustness. In order to improve the efficiency and accuracy of approximate EMD calculations, Atasu et al. provided new theoretical and practical results in [32]. Qu et al. [33] proposed a novel EMD method to detect false data injection attacks in smart grids to improve power-system security.

In this paper, the NMF algorithm is improved based on the EMD distance and the use of graph-regularization terms and sparse constraints, which can discover the geometric structure information inherent in the dataset and produce a smooth and more accurate solution. We call it graph-regularized, $L_{1/2}$ sparsity-constrained NMF with EMD metric (GSNMF-EMD). In view of numerous literature showing that $L_{1/2}$-NMF provides sparser and more accurate results than those delivered using the $L_1$ norm [34,35,36,37,38], we also chose the $L_{1/2}$ norm as the sparse constraint. The graph-regularization constraint uncovers the implied semantics while focusing on preserving information about the intrinsic geometric structure of the dataset. Furthermore, update rules and convergence proofs of the GSNMF-EMD algorithm are presented. Experimental results on real datasets demonstrate the validity and accuracy of our proposed multiplicative update algorithm.

The rest of the paper is organized as follows: In Section 2, we present some of the related work. In Section 3, we propose a multiplicative update rule for the GSNMF-EMD model and prove its convergence. We report some experimental results on two image datasets in Section 4. The parameter selection is provided in Section 5. Finally, we briefly summarize in Section 6.

2. Related Work

In this section, we review the EMD metric and the NMF with the Earth Mover’s distance metric (EMDNMF in short). We use bold capital letters to denote matrices and lowercase letters to denote vectors. For example, $\mathbf{A}$ is an $m\times n$ matrix, ${\mathbf{a}}_j$ is the j-$th$ column vector, and $a_{ij}$ is the $(i,j)$-th entry of matrix $\mathbf{A}$. Let x, y$\in {\mathbb{R}}^m$, $\widetilde{KL}(\mathbf{x}\parallel \mathbf{y})={\mathbf{x}}^{\top }log(\mathbf{x}\oslash \mathbf{y})-{\mathbf{1}}^{\top }\mathbf{x}+{\mathbf{1}}^{\top }\mathbf{y}$ be the generalized K-L divergence between $\mathbf{x}$ and $\mathbf{y}$, where $\mathbf{x}\oslash \mathbf{y}$ represents element-wise division and ${(·)}^{\top }$ denotes the transpose operation.

2.1. EMD

EMD was first proposed for certain vision problems by Peleg et al. [39]. EMD is an efficient metric based on an optimal transportation problem solution. To be precise, this is the minimum cost that must be paid to convert one distribution into another. Since it can manipulate the variable-length representation of the distribution, it effectively avoids the quantization and other typical histogram binning problems, causing it to be more robust than histogram matching techniques.

Definition 1 

([29]). Let $\mathbf{x},\mathbf{y}\in {\mathbb{R}}^m$ be two normalized histograms with ${\mathbf{x}}^{\top }{\mathbf{1}}_{\mathbf{m}}=1,{\mathbf{y}}^{\top }{\mathbf{1}}_{\mathbf{m}}=1$. Let $\mathbf{M}$ be an $m\times m$ distance metric matrix. $\mathbf{T}=(t_{pq})\in {\mathbb{R}}^{m\times m}$ is a transport matrix. We call

$$\begin{array}{cc}\hfill {\mathfrak{D}}_{\mathbf{M}}(\mathbf{y},\mathbf{x})& \hfill =\underset{t_{pq}\ge 0}{min}\sum _{p,q=1}^m{m}_{pq}{t}_{pq}s.t.\mathbf{T}{\mathbf{1}}_{\mathbf{m}}=\mathbf{y},{\mathbf{T}}^{\top }{\mathbf{1}}_{\mathbf{m}}=\mathbf{x},\hfill \end{array}$$

(1)

an Earth mover’s distance (EMD) between $\mathbf{x}$ and $\mathbf{y}$.

Despite EMD having many good properties, its calculations based on the well-known solution to the transportation problem are very expensive. Recent years, some scholars have done fruitful work to improve the operation efficiency. Based on Equation (1), Marco Cuturi [40] proposed a maximum entropy perspective. He smoothed the solution to the EMD problem by adding an entropy-regularization term. Based on the excellent results of Cuturi, Frogner [41] replaced the equality constraint with a soft penalty in terms of K-L divergence. He obtained an unconstrained approximate transmission problem. The specific form is:

$$\begin{array}{cc}\hfill {\mathfrak{D}}_{\mathbf{M}}^{\lambda ,\gamma }(\mathbf{y},\mathbf{x})& \hfill =\underset{t_{pq}\ge 0}{min}\{\sum _{p,q=1}^m{m}_{pq}{t}_{pq}+\frac{1}{\lambda }\tilde{H}(\mathbf{T})\}+\gamma (\widetilde{KL}(\mathbf{T}\mathbf{1}\parallel \mathbf{y})+\widetilde{KL}({\mathbf{T}}^{\mathbf{T}}\mathbf{1}\parallel \mathbf{x})),\hfill \end{array}$$

(2)

where $\tilde{H}(\mathbf{T})$=${\displaystyle \sum _{p,q}^m}{t}_{pq}log{t}_{pq}$ is the entropy of $\mathbf{T}$, and $\lambda$ and $\gamma$ are the regularization parameters. When $\mathbf{x},\mathbf{y}$ are normalized vectors, Equation (2) can approximate Cuturi’s algorithm closely for large enough $\gamma$. Moreover, Frogner obtains the optimal solution of Equation (2) by borrowing Cuturi’s results; that is, the optimal solution is a diagonal scaling of a matrix $\mathbf{K}=e^{-\lambda \mathbf{M}-1}$. The specific form is:

$$\begin{array}{cc}\hfill {\mathbf{T}}^{*}& \hfill =diag(\mathbf{u})\mathbf{K}diag(\mathbf{v}),\hfill \end{array}$$

(3)

$$\begin{array}{c}\hfill \mathbf{u}={(\mathbf{y})}^{\frac{\gamma \lambda }{\gamma \lambda +1}}\odot {(\mathbf{K}\mathbf{v})}^{-\frac{\gamma \lambda }{\gamma \lambda +1}},\mathbf{v}={(\mathbf{x})}^{\frac{\gamma \lambda }{\gamma \lambda +1}}\odot {({\mathbf{K}}^T\mathbf{u})}^{-\frac{\gamma \lambda }{\gamma \lambda +1}},\end{array}$$

(4)

where ⨀ denotes element-wise multiplication. The gradient of Equation (2) is given by

$$\begin{array}{cc}\hfill {▽}_{\mathbf{y}}{\mathfrak{D}}_{\mathbf{M}}^{\lambda ,\gamma }(\mathbf{y},\mathbf{x})=& \hfill \gamma (\mathbf{1}-{\mathbf{T}}^{*}\mathbf{1}\oslash \mathbf{y}).\hfill \end{array}$$

(5)

2.2. EMDNMF

Considering n non-negative histograms with m bins, the histogram is expressed in matrix form as $\mathbf{X}=\left[x_{ij}\right]=[{\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n]\in {\mathbb{R}}^{m\times n}$, where the j-$th$ histogram is the j-$th$ column of the matrix. The matrix $\mathbf{X}$ may be decomposed into a product of $\mathbf{U}=\left[u_{ij}\right]=[{\mathbf{u}}_1,{\mathbf{u}}_2,\cdots ,{\mathbf{u}}_r]\in {\mathbb{R}}^{m\times r}$ and $\mathbf{V}=\left[v_{ij}\right]=[{\mathbf{v}}_1,{\mathbf{v}}_2,\cdots ,{\mathbf{v}}_r]\in {\mathbb{R}}^{n\times r}$$(\mathbf{X}\approx {\mathbf{UV}}^{\top })$, where $\mathbf{U}$ and $\mathbf{V}$ are defined as the basis and coefficient matrices, respectively. In many cases, the low-dimensional approximation is more valuable than the exact decomposition. Let $\mathbf{Y}={\mathbf{UV}}^{\top }=[{\mathbf{y}}_1,{\mathbf{y}}_2,\cdots ,{\mathbf{y}}_n]\in {\mathbb{R}}^{m\times n}$, ${\mathbf{y}}_j={\displaystyle \sum _k}{\mathbf{u}}_k{v}_{jk}$. Optimizing the exact EMD throughout the iterative solution process is expensive and makes it difficult to guarantee that ${\displaystyle \sum _i}{x}_{ij}={\displaystyle \sum _i}{y}_{ij}$. Equation (2) describes a regularized approximation which can be calculated quickly and efficiently, even with unnormalized data. Fortunately, $\mathbf{X}\approx \mathbf{Y}$ implies that ${\displaystyle \sum _{j=1}^n}{\mathfrak{D}}_{\mathbf{M}}^{\lambda ,\gamma }({\mathbf{x}}_j,{\mathbf{y}}_j)$ is the sum of distances between the feature histograms. Based on Equation (2), the objective function of EMDNMF can be expressed as:

$$\begin{array}{c}\hfill {ϝ}_1=\underset{\mathbf{U},\mathbf{V}}{min}\sum _{j=1}^n{\mathfrak{D}}_{\mathbf{M}}^{\lambda ,\gamma }({\mathbf{y}}_j,{\mathbf{x}}_j).\end{array}$$

(6)

3. GSNMF-EMD

3.1. The Objective Function

To explicitly explain how to discover information about the geometric structure implied by the dataset, the low-dimensional representation of the original data ${\mathbf{x}}_i$ relative to the basis matrix $\mathbf{U}$ is ${\mathbf{v}}_i$. A reasonable assumption is that if two data points ${\mathbf{x}}_i$ and ${\mathbf{x}}_j$ are close to each other in the intrinsic geometric structure of the dataset, then these two points ${\mathbf{v}}_i$ and ${\mathbf{v}}_j$ are also close to each other in the new basis matrix [42]. For each data point ${\mathbf{x}}_i$, we can use a 0–1 weighting scheme to generate a k-nearest neighbor graph, which in turn generates a weight matrix $\mathbf{W}$ according to the graph theory [43,44]. Given a graph with n vertices, where each vertex corresponds to a data point, the edge weight matrix $\mathbf{W}$ is defined as follows:

$$w_{ij}=\left\{\begin{array}{c}1,if{\mathbf{x}}_i\in {\tilde{N}}_k({\mathbf{x}}_j)or{\mathbf{x}}_j\in {\tilde{N}}_k({\mathbf{x}}_i),\hfill \\ 0,otherwise,\hfill \end{array}\right.$$

where ${\tilde{N}}_k({\mathbf{x}}_j)$ denotes the set of k nearest neighbors of the data point ${\mathbf{x}}_j$. Thus, the formula used to measure the smoothed representation of the lower dimension is

$$\begin{array}{cc}\hfill \underset{\mathbf{V}}{min}\Re & \hfill =\frac{1}{2}\sum _{j,s=1}^n{w}_{js}{\parallel {\mathbf{v}}_j-{\mathbf{v}}_s\parallel }^2.\hfill \end{array}$$

(7)

Combining Equations (6) and (7), we get another objective function:

$$\begin{array}{c}\hfill {ϝ}_2=\underset{\mathbf{U},\mathbf{V}}{min}\sum _{j=1}^n{\mathfrak{D}}_{\mathbf{M}}^{\lambda ,\gamma }({\mathbf{y}}_j,{\mathbf{x}}_j)+\frac{1}{2}\alpha \Re ,\end{array}$$

(8)

where $\alpha \ge 0$ is the regularization parameter.

To obtain a sparser and more accurate solution, the first issue to be addressed is: Should a sparse constraint be imposed on the basis matrix $\mathbf{U}$ or the coefficient matrix $\mathbf{V}$? In this paper, we choose the $L_{1/2}$ sparsity constraint on the coefficients $\mathbf{V}$. Numerous results have shown that the proposed $L_{1/2}$ regularity is both easier to solve than the $L_0$ regularity and more sparse and stable than the $L_1$ regularity [35,36,37], and it is also computationally efficient. Based on the above graph-regularization term and $L_{1/2}$ sparsity constraint, the final objective function of graph-regularized, sparsity-constrained NMF with the Earth mover’s distance metric (GSNMF-EMD) is given as:

$$\begin{array}{c}\hfill ϝ=\underset{\mathbf{U},\mathbf{V}}{min}\sum _{j=1}^n{\mathfrak{D}}_{\mathbf{M}}^{\lambda ,\gamma }({\mathbf{y}}_j,{\mathbf{x}}_j)+\frac{1}{4}\alpha \sum _{j,s=1}^n{w}_{js}{\parallel {\mathbf{v}}_j-{\mathbf{v}}_s\parallel }^2+\beta {\parallel \mathbf{V}\parallel }_{1/2},\end{array}$$

(9)

where ${\parallel \mathbf{V}\parallel }_{1/2}={\displaystyle \sum _{j=1}^n}{\displaystyle \sum _{s=1}^r}{\left|v_{js}\right|}^{1/2}$ is the $L_{1/2}$ regularization and $\beta$ is the regularization parameter.

3.2. Multiplicative Update Rules

The objective function Equation (9) is not convex in both $\mathbf{U}$ and $\mathbf{V}$. Therefore, to find the global minimum is a difficult task, and fortunately we can find an iterative update algorithm to obtain a locally optimal solution. Similar to [14,45], we obtained the two-step multiplicative update rule that can maintain non-negativity and find a local minimum.

Theorem 1. 

The objective function ϝ in Equation (9) has the following update rules:

$$\begin{array}{cc}\hfill u_{ik}& \hfill ⟵u_{ik}\frac{{\sum }_s{v}_{sk}\frac{{\sum }_t{\mathbf{T}}_s^{*it}}{y_{is}}}{{\sum }_s{v}_{sk}},\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill v_{jk}& \hfill ⟵v_{jk}\frac{\gamma {\sum }_s{u}_{sk}\frac{{\sum }_t{\mathbf{T}}_j^{*st}}{y_{sj}}+\alpha {\sum }_{s\ne j}{w}_{js}{v}_{js}}{\gamma {\sum }_s{u}_{sk}+\alpha v_{jk}{\sum }_{s\ne j}{w}_{js}+\frac{\beta }{2}{\sum }_{j,k}{v}_{jk}^{-\frac{1}{2}}},\hfill \end{array}$$

(11)

where ${\mathbf{T}}_s^{*it}$ is the $(i,t)$-entry of the optimal transportation matrix between ${\mathit{x}}_s$ and ${\mathit{y}}_s$ defined in Equation (3). Furthermore, the objective function ϝ is nonincreasing and permanent under these update rules if and only if $\mathbf{U}$ and $\mathbf{V}$ are at a stationary point.

Now the specific procedure for finding the local optimal $\mathbf{U}$ and $\mathbf{V}$ of GSNMF-EMD is summarized in Algorithm 1.

Algorithm 1 GSNMF-EMD algorithm.

Input:  Data matrix $\mathbf{X}\in {\mathbb{R}}^{m\times n}$, metric matrix $\mathbf{M}\in {\mathbb{R}}^{m\times m}$, weight matrix $\mathbf{W}\in {\mathbb{R}}^{n\times n}$. The parameters: $\lambda ,\gamma ,\alpha ,\beta$. Output:  $\mathbf{U}\in {\mathbb{R}}^{m\times r}$, $\mathbf{V}\in {\mathbb{R}}^{n\times r}$. 1: Initialization: ${\mathbf{U}}_0,{\mathbf{V}}_0,{\mathbf{Y}}_{\mathbf{0}}={\mathbf{U}}_{\mathbf{0}}{\mathbf{V}}_{\mathbf{0}}^{\top }$, $k=0$; 2: repeat 3:     Calculate the optimal transport matrix ${\mathbf{T}}^{*}$; 4:     Update $\mathbf{U}$ according to $$\begin{array}{cc}\hfill u_{ik}& \hfill ⟵u_{ik}\frac{{\sum }_s{v}_{sk}\frac{{\sum }_t{\mathbf{T}}_s^{*it}}{y_{is}}}{{\sum }_s{v}_{sk}};\hfill \end{array}$$ 5:     Update $\mathbf{V}$ according to $$\begin{array}{cc}\hfill v_{jk}& \hfill ⟵v_{jk}\frac{\gamma {\sum }_s{u}_{sk}\frac{{\sum }_t{\mathbf{T}}_j^{*st}}{y_{sj}}+\alpha {\sum }_{s\ne j}{w}_{js}{v}_{js}}{\gamma {\sum }_s{u}_{sk}+\alpha v_{jk}{\sum }_{s\ne j}{w}_{js}+\frac{\beta }{2}{\sum }_{j,k}{v}_{jk}^{-\frac{1}{2}}};\hfill \end{array}$$ 6:     $k\leftarrow k+1$ 7: until convergence or maximum iteration is reached.

3.3. Convergence Analysis

In this section, we will draw on the ideas of [14,45] and cite the relevant theory as Definition 2 and Lemma 1.

Definition 2 

([45]). $\mathsf{\Psi }(\mathbf{h},{\mathbf{h}}^{\prime })$ is an auxiliary function for $\widetilde{ϝ}(\mathbf{h})$, if the conditions $\mathsf{\Psi }(\mathbf{h},{\mathbf{h}}^{\prime })\ge \widetilde{ϝ}(\mathbf{h})$ and $\mathsf{\Psi }(\mathbf{h},\mathbf{h})=\widetilde{ϝ}(\mathbf{h})$ are satisfied.

Lemma 1 

([45]). If Ψ is an auxiliary function of $\widetilde{ϝ}$, then $\widetilde{ϝ}$ is nonincreasing under the update

$$\begin{array}{cc}\hfill {\mathbf{h}}^{t+1}& \hfill =arg\underset{\mathbf{h}}{min}\mathsf{\Psi }(\mathbf{h},{\mathbf{h}}^t).\hfill \end{array}$$

By fixing the matrix $\mathbf{U}$, we transform the objective function $ϝ$ as

$$\begin{array}{c}\hfill ϝ(\mathbf{V})=\sum _{j=1}^n{\mathfrak{D}}_{\mathbf{M}}^{\lambda ,\gamma }(\sum _k{v}_{jk}{\mathbf{u}}_k,{\mathbf{x}}_j)+\frac{1}{4}\alpha \sum _{j,s,k}^n{w}_{js}{\parallel {\mathbf{v}}_j-{\mathbf{v}}_s\parallel }^2+\beta {\parallel \mathbf{V}\parallel }_{1/2}.\end{array}$$

(12)

With fixed $\mathbf{V}$, $ϝ(\mathbf{U})$ can similarly be rewritten under the update rules for $\mathbf{U}$. We only prove that $ϝ(\mathbf{V})$ is nonincreasing under the update rule of Equation (10), and Equation (11) can be proved in similar way.

Lemma 2. 

Let ${\phi }_{i{k}_i}=\frac{u_{ik}{v}_{jk}^{(q)}}{{\displaystyle \sum _k}{u}_{ik}{v}_{jk}^{(q)}}$, and $\phi ={[{\phi }_{1k_1},{\phi }_{2k_2},\cdots ,{\phi }_{m{k}_m}]}^{\top }$. Then, function

$$\begin{array}{cc}\hfill \mathsf{\Psi }(\mathbf{V},{\mathbf{V}}^{(q)})& \hfill =\sum _{j,k_1\cdots ,k_m}\prod _i{\phi }_{i{k}_i}{\mathfrak{D}}_{\mathbf{M}}^{\lambda ,\gamma }((\sum _k{v}_{jk}{\mathbf{u}}_k)\oslash \phi ,{\mathbf{x}}_j)\hfill \\ \hfill & \hfill +\frac{1}{4}\alpha \sum _{j,s,k}^n{w}_{js}{\parallel {\mathbf{v}}_j-v_s\parallel }^2+\beta {\parallel \mathbf{V}\parallel }_{1/2}\hfill \end{array}$$

(13)

is an auxiliary function for $ϝ(\mathbf{V})$.

Proof. 

When $\mathbf{V}={\mathbf{V}}^{(q)}$ and ${\phi }_{i{k}_i}=\frac{u_{ik}{v}_{jk}}{{\displaystyle \sum _k}{u}_{ik}{v}_{jk}}$, we know

$$\begin{array}{c}\hfill \sum _{j,k_1\cdots ,k_m}\prod _i{\phi }_{i{k}_i}{\mathfrak{D}}_{\mathbf{M}}^{\lambda ,\gamma }((\sum _k{v}_{jk}{\mathbf{u}}_k)\oslash \phi ,{\mathbf{x}}_j)=\sum _{j=1}^n{\mathfrak{D}}_{\mathbf{M}}^{\lambda ,\gamma }(\sum _k{v}_{jk}{\mathbf{u}}_k,{\mathbf{x}}_j),\end{array}$$

Compare Equation (12) with Equation (13): $\mathsf{\Psi }(\mathbf{V},\mathbf{V})=ϝ(\mathbf{V})$.

It can be seen from reference [41], $d_{\mathbf{M}}^{\lambda ,\gamma }$ is convex. We use the convexity for $i=1,\cdots ,m$ one by one and obtain

$$\begin{array}{c}\hfill \sum _{j,k_1\cdots ,k_m}\prod _i{\phi }_{i{k}_i}{\mathfrak{D}}_{\mathbf{M}}^{\lambda ,\gamma }((\sum _k{v}_{jk}{\mathbf{u}}_k)\oslash \phi ,{\mathbf{x}}_j)\ge \sum _{j=1}^n{\mathfrak{D}}_{\mathbf{M}}^{\lambda ,\gamma }(\sum _k{v}_{jk}{\mathbf{u}}_k,{\mathbf{x}}_j),\end{array}$$

which implies $\mathsf{\Psi }(\mathbf{V},{\mathbf{V}}^{(q)})\ge ϝ(\mathbf{V})$. Therefore, Lemma 2 is proved. □

Proof. 

(Proof of Theorem 1.) The gradient ${▽}_y{\mathfrak{D}}_{\mathbf{M}}^{\lambda ,\gamma }(\mathbf{y},\mathbf{x})$ satisfies Equation (6). Let the gradient of $\mathsf{\Psi }(\mathbf{V},{\mathbf{V}}^{(q)})$ with respect to $\mathbf{V}$ be zero. We can obtain ${\mathbf{V}}^{(q+1)}$. The equations are such that $\frac{\partial \mathsf{\Psi }(\mathbf{V},{\mathbf{V}}^{(q)})}{v_{jk}}=0$. We have

$$\begin{array}{cc}\hfill v_{jk}& \hfill ⟵v_{jk}\frac{\gamma {\sum }_s{u}_{sk}\frac{{\sum }_t{\mathbf{T}}_j^{*st}}{y_{sj}}+\alpha {\sum }_{s\ne j}{w}_{js}{v}_{js}}{\gamma {\sum }_s{u}_{sk}+\alpha v_{jk}{\sum }_{s\ne j}{w}_{js}+\frac{\beta }{2}{\sum }_{j,k}{v}_{jk}^{-\frac{1}{2}}}.\hfill \end{array}$$

Similarly, we can construct $\mathsf{\Psi }(\mathbf{U},{\mathbf{U}}^{(q)})$. Use the same update method as V. We get

$$\begin{array}{cc}\hfill u_{ik}& \hfill ⟵u_{ik}\frac{{\sum }_s{v}_{sk}\frac{{\sum }_t{\mathbf{T}}_s^{*it}}{y_{is}}}{{\sum }_s{v}_{sk}},\hfill \end{array}$$

Combine Lemma 1 and Lemma 2. Theorem 1 is obtained. □

4. Experiments

In this section, we compare the data-clustering results of two popular datasets with the related most well-known methods, such as K-means, NMF [45], GNMF [14], and EMDNMF [30], to evaluate the performance of the proposed GSNMF-EMD algorithm.

4.1. Datasets

We evaluated the clustering performance on two widely used datasets, COIL20 and MNIST. The basic information of these two datasets is presented in

Table 1. Statistics of the datasets used in our experiment.

Data Sets	Size	Features	Classes
COIL20	1440	1024	20
MNIST	70,000	784	10

.

• COIL20 (https://www.cs.columbia.edu/CAVE/software/softlib/coil-20.php, accessed on: 26 June 2022 ): The COIL20 dataset contains 1440 gray images of 20 objects—that is, 72 images of each object acquired from different angles. The images we use here are limited to a size of $32\times 32$.
• MNIST (http://yann.lecun.com/exdb/mnist/, accessed on: 26 June 2022): The MNIST handwritten digit dataset comes from Yann LeCun’s web page and contains a training set of $60,000$ examples and a test set of $10,000$ examples. The size of the images we used here was $28\times 28$.

4.2. Evaluation Metric

We used two of the most popular evaluation metrics to evaluate the clustering performance of the GSNMF-NMF algorithm. One metric was ACC (clustering accuracy), and the other was $MI(C,C^{\prime })$ (mutual information). ACC measures the proportion of correctly classified data points in a clustering task, and NMI measures the mutual information between the true labels of the data points and the cluster assignments. These two metrics are commonly used in clustering tasks because they provide an easy-to-understand measure of the quality of the clustering results. In the context of NMF, ACC and NMI are used to evaluate the quality of the clusters that are produced by the algorithm. ACC is based on the principle of evaluating the clustering performance by comparing the clustering labels of each sample with the labels provided by the dataset, which is defined as [5,9,11,42], etc., as follows:

$$\begin{array}{cc}\hfill ACC& \hfill =\frac{{\sum }_{i=1}^n\delta (s_i,map(r_i))}{n},\hfill \end{array}$$

where $\delta (x,y)$ is the delta function that equals one if $x=y$ and equals zero otherwise, n is the total number of samples, and map$(r_i)$ is the permutation mapping function that maps each cluster label $r_i$ to the equivalent label from the data corpus.

The $MI(C,C^{\prime })$ between the two clusters C and $C^{\prime }$, which is defined as in [5,10,42], etc., it is defined as follows:

$$\begin{array}{cc}\hfill MI(C,C^{\prime })& \hfill =\sum _{c_i\in C,c_j^{\prime }\in C^{\prime }}p(c_i,c_j^{\prime }){log}_2\frac{p(c_i,c_j^{\prime })}{p(c_i)p(c_j^{\prime })},\hfill \end{array}$$

where C is a set of true labels; $C^{\prime }$ is a set of clustering labels obtained from a specific clustering algorithm; $p(c_i,c_j^{\prime })$ denotes the joint probability that this arbitrarily selected document belongs to the clusters $c_i$ and $c_j^{\prime }$ at the same time, as opposed to $p(c_i)$ and $p(c_j^{\prime })$, which denote the probabilities of a document belonging to the clusters $c_i$ and $c_j^{\prime }$, respectively. The other metric is the NMI (normalized mutual information) [5,9,11,42]. It is expressed as follows

$$\begin{array}{cc}\hfill NMI(C,C^{\prime })& \hfill =\frac{MI(C,C^{\prime })}{max(\tilde{H}(C),\tilde{H}(C^{\prime }))},\hfill \end{array}$$

$\tilde{H}(C)$ and $\tilde{H}(C^{\prime })$ are the entropies of C and $C^{\prime }$, respectively.

4.3. Performance Evaluations and Comparisons

To demonstrate the performance of our proposed method in improving image clustering, we compared the GSNMF-EMD algorithm with the following four classical clustering algorithms.

• K-means: Canonical K-means clustering method (K-means in short).
• NMF: [45]: The original NMF is considered to be the baseline algorithm and only imposes non-negative constraints on the two factor matrices.
• GNMF: [42]: Graph-regularized NMF (GNMF in short) with Euclidean distance. It adds graph-regularization constraints to NMF, taking into account information about the geometric structure of the data space. The regularization parameter $\alpha$ is set to 10.
• EMDNMF: [30]: NMF with EMD (EMDNMF in short). We set the $\lambda$ and $\gamma$ to 100 and 0.1, respectively; meanwhile, we used the 2D distance of pixels’ location of the image as the ground metric.
• GSNMF-EMD: Graph-regularized $L_{1/2}$ sparsity-constrained NMF with the Earth mover’s distance metric (GSNMF-EMD in short). We set the $\lambda$, $\gamma$, $\alpha$, and $\beta$ to 100, 0.1, 20, and 1.8, respectively. Meanwhile, we use the 2D distances among pixels’ locations in the image as the ground metric.

In order to demonstrate the clustering performance, we compare our method with K-means, NMF, GNMF, and EMDNMF methods under the same iterations and with the same well-known datasets. In all experiments, we used the 0-1 weighting scheme for constructing the k-nearest neighbor graph with $k=5$.

Table 2. Clustering performance on the COIL20 dataset.

k	Accuracy (%)					Normalized Mutual Information (%)
	K-Means	NMF	GNMF	EMD-NMF	GSNMF-EMD	K-Means	NMF	GNMF	EMD-NMF	GSNMF-EMD
4	53.819	52.083	67.361	54.514	96.528	33.341	32.487	61.494	33.366	92.384
5	60.556	61.944	63.056	58.611	73.333	49.56	51.166	66.436	49.325	67.782
6	90.972	93.519	97.917	85.185	97.685	85.280	89.561	95.585	78.465	95.452
7	69.643	63.294	76.786	60.317	88.095	63.951	63.951	79.354	58.235	84.485
8	73.438	65.104	88.021	52.951	87.847	72.469	73.52	91.678	58.092	91.672
9	65.741	63.889	90.432	61.111	89.969	66.802	63.611	87.646	58.933	87.821
10	67.222	73.611	93.611	75.972	83.472	75.336	75.768	93.049	75.691	87.653
11	57.828	66.793	77.652	62.247	77.778	62.304	67.408	83.464	64.313	87.510
12	67.708	64.583	79.282	62.847	79.630	74.297	68.273	86.313	67.503	88.222
13	72.543	70.192	80.021	66.026	80.449	75.162	74.933	87.055	71.762	86.964
14	69.048	66.468	78.968	55.456	85.218	75.613	73.247	87.700	64.102	89.273
15	61.667	65.556	78.889	58.889	77.500	71.048	73.327	85.515	65.449	84.420
16	63.542	63.368	79.861	59.983	79.688	71.752	72.15	85.899	68.569	84.599
17	63.235	61.601	73.611	59.150	72.876	72.187	71.148	86.209	69.008	86.298
18	67.130	55.247	75.926	53.781	78.086	75.907	68.560	87.027	66.846	86.954
19	59.137	62.792	77.485	54.605	79.313	73.307	72.659	87.899	67.241	87.061
20	67.083	63.403	75.903	55.764	73.958	76.674	72.049	87.108	66.723	86.653
Avg.	66.489	65.497	79.693	61.024	82.437	69.117	68.460	84.672	63.743	86.777

and

Table 3. Clustering performance on the MNIST dataset.

k	Accuracy (%)					Normalized Mutual Information (%)
	K-Means	NMF	GNMF	EMD-NMF	GSNMF-EMD	K-Means	NMF	GNMF	EMD-NMF	GSNMF-EMD
2	20.082	20.902	19.945	19.809	20.219	10.099	10.051	14.094	10.878	14.992
3	30.601	24.772	27.140	25.319	30.328	25.187	13.428	20.438	12.115	25.352
4	39.481	27.254	41.120	24.658	41.120	35.416	19.965	42.834	17.543	39.852
5	39.071	30.710	35.191	26.721	43.770	32.256	26.791	40.075	19.289	37.957
6	35.246	33.197	54.508	36.566	52.505	35.468	31.005	50.292	26.079	47.111
7	44.223	41.491	48.673	44.145	59.133	41.302	38.712	53.542	33.950	57.559
8	50.615	46.619	50.854	38.251	48.770	46.927	44.749	54.334	32.038	48.106
9	54.614	50.213	64.511	48.482	64.359	51.401	44.901	65.380	40.234	59.547
Avg.	39.242	34.395	42.743	32.994	45.026	34.757	28.700	42.623	24.016	41.310

show the ACC and NMI results on the COIL20 and MNIST datasets, respectively, including those of GSNMF-EMD and the four comparison algorithms. For each given cluster number k, 20 test and 100 iterations runs were conducted on different randomly chosen clusters. The average performance is reported in the Table 2 and Table 3. Figure 1 and Figure 2 show the clustering results for the COIL20 and MNIST datasets in graphical form, and we can see from these experiments that our algorithm has high performance.

The GSNMF-EMD method and GNMF method are better than the NMF and EMDNMF methods. This is because graph regularization can respect the geometric information implied by the dataset. Meanwhile, the $L_{1/2}$-smoothing-constraint method in GSNMF-EMD can produce more accurate and smoother solutions. In addition, the GSNMF-EMD method has good perceptual similarity and robustness, thereby having better performance than GNMF and the other three methods.

5. Parameter Selection

GSNMF-EMD has four essential regularization parameters: $\lambda ,\gamma ,\alpha ,$ and $\beta$, when both values of $\alpha$ and $\beta$ are equal to zero. The new model degrades to the EMDNMF method [30]. In this experiment, based on the algorithm of graph theory, we set the number of nearest neighbors on all datasets to five. There are four regularization parameters, as in [14,17,42]. It is very interesting and meaningful to explore the influences of their changes on the clustering results. Of course, there are many combined results of these four parameters, which requires a large number of numerical experiments. In this section, we mainly investigate the sensitivity of $\lambda ,\gamma ,\alpha$, and $\beta$, with the iterations set to 100 and the number of clusters set to eight. Figure 3, Figure 4, Figure 5, Figure 6, Figure 7 and Figure 8 demonstrate the ACC and NMI of GSNMF-EMD with different $\lambda$ and $\gamma$ or $\alpha$ and $\beta$ on COIL20.

As shown in Figure 3, Figure 4, Figure 5, Figure 6, Figure 7 and Figure 8, the performance of the algorithm is influenced by the parameters. It can be seen that the performance of GSNMF-EMD is relatively stable with respect to the parameter $\beta$. In general, $\beta$ can be selected from the range [0.001, 10]. The effect of $\gamma$ on GSNMF-EMD is small. When the value of $\alpha$ is within [0.1, 1], GSNMF-EMD achieves consistently good performances. However, when $\alpha$ is greater than 10, the performance of our algorithm degrades. When the value of $\lambda$ is within [1, 10,000], the performance of the GSNMF-EMD improves as $\lambda$ increases. In particular, when $\lambda$ is greater than 10, the performance of GSNMF-EMD stabilizes due to a slower increase, so the range of $\lambda$ can be set to [10, 10,000]. Overall, the performance of the GSNMF-EMD algorithm can remain relatively stable as the parameters are transformed, and in most cases, it is more accurate than any other algorithm we tested. Based on the experimental results, we conclude that all the regularization parameters in the algorithm can control the clustering performance. Therefore, if certain parameters are not set to reasonable values, the clustering performance may be relatively low.

Figure 9a,b show reconstructions of face images from the COIL20 database and handwritten digits from the MNIST database, respectively. From Figure 9, we can see that our algorithm has good reconstruction performance.

6. Conclusions

We proposed a new algorithm called graph-regularized, $L_{1/2}$ sparsity-constrained NMF with the Earth mover’s distance metric (GSNMF-EMD). GSNMF-EMD builds the geometric structures of the data distribution and sparsity constraints and incorporates them into EMDNMF as the additional regularization terms. The experimental results showed that it can have more clustering power than lots of NMF-based algorithms. Meanwhile, the results also showed that GSNMF-EMD is more stable and works more accurately and rapidly than the original EMD.

In the end, there are two issues that may lead to more interesting work in the future. Firstly, the normalization assumption of the histograms is not essential for GSNMF-EMD. This is because the algorithm proposed by EMD is not limited to the normalized case. Therefore, GSNMF-EMD has the potential ability to deal with occlusions, which is an important problem in local descriptors. Secondly, using our algorithm for hyperspectral analysis is also a good research direction, and finally, as indicated in [29], GSNMF-EMD can also be modeled as a network flow problem, and that would be worthy work, full of research significance.