The Generalization of Non-Negative Matrix Factorization Based on Algorithmic Stability

Abstract

The Non-negative Matrix Factorization (NMF) is a popular technique for intelligent systems, which can be widely used to decompose a nonnegative matrix into two factor matrices: a basis matrix and a coefficient one, respectively. The main objective of NMF is to ensure that the operation results of the two matrices are as close to the original matrix as possible. Meanwhile, the stability and generalization ability of the algorithm should be ensured. Therefore, the generalization performance of NMF algorithms is analyzed from the perspective of algorithm stability and the generalization error bounds are given, which is named AS-NMF. Firstly, a general NMF prediction algorithm is proposed, which can predict the labels for new samples, and then the corresponding loss function is defined further. Secondly, the stability of the NMF algorithm is defined according to the loss function, and two generalization error bounds can be obtained by employing uniform stability in the case where U is fixed and it is not fixed under the multiplicative update rule. The bounds numerically show that its stability parameter depends on the upper bound on the module length of the input data, dimension of hidden matrix and Frobenius norm of the basis matrix. Finally, a general and stable framework is established, which can analyze and measure generalization error bounds for the NMF algorithm. The experimental results demonstrate the advantages of new methods on three widely used benchmark datasets, which indicate that our AS-NMF can not only achieve efficient performance, but also outperform the state-of-the-art of recommending tasks in terms of model stability.

1. Introduction

Currently, Non-negative Matrix Factorization (NMF) [1,2,3,4] is gaining more attention in the machine learning method community, and it has become one of the most effective feature selection methods implemented in various fields, such as documents of clustering and classification [5,6], face recognition [7,8], source separation [9,10], digital image processing [11,12] and video semantic recognition [13]. Unfortunately, due to the non-negative constraints on the two factor matrices (basic matrix and conefficient matrix), only additive operations are allowed, resulting in only parts-based representation for NMF.

How to minimize the reconstruction error of NMF on the training and test datasets is a hot issue in NMF research. Most recent studies aim to use different regularization terms and loss functions, which are used to optimize the generalization performance of the NMF algorithm. Furthermore, in order to decompose non-negative matrices robustly, many loss functions have been proposed and used to improve their robustness, such as the $l_1$ loss [14,15] and the $l_2$ loss [16]. Sandler and Lindenbaum [12] proposed NMF algorithms that minimize error between the data and matrix using earth mover’s distance (EMD). Lee and Seung [17] provided a valid multiplication update rule (MUR) for NMF. In addition, better generalization performance can be achieved by using different regularization, which is accomplished by strengthening the weight on many attributes in the NMF. Incorporating sparseness [18] and the orthogonality penalty in NMF [19,20] can achieve sparse parts-based representation. Ye Zhang et al. [21] explored the blind separation problem for uncorrelated signals and proposed NMF methods with the least correlated component constraints for uncorrelated source signals. Nevertheless, for the electroencephalography (EEG) signals, the greater challenge is the nature of noise and capturing the inconspicuous relations between EEG signals and specific brain activities; Chen et al. [22] introduced a novel spatio-temporal, preserving representations of raw EEG streams to precisely identify human intentions. Zhang et al. [23] proposed an incremental NMF based on correlation and graph regularizer, which considers the geometric structure in the data and correlation among the data. Soon after, Liu et al. [24] proposed a large-cone NMF (LCNMF) algorithm for the problem of obtaining an attractive local solution for NMF, which can obtain attractive local solutions from a large-cone penalty framework for NMF, and they prove that LCNMF can obtain a smaller generalization upper bound under the Vapnik Chervonenkis theory. Therefore, it is interesting to characterize the NMF generalization error upper bound through some theoretical framework for the perspective of NMF generalization performance.

The capacity of an intelligent system is commonly evaluated by its generalization ability; the stronger the generalization ability, the more accurate a system’s prediction of new events. Intelligent systems are an essential approximation of the real-model of a problem, which is called a hypothesis. The risk is defined as an error between the true solution and hypothesis or the accumulation of errors. However, if the real model is unknown, the risk cannot be calculated. Meanwhile, empirical risk is used to approximate the true solution. However, the difference between the predicted results from the hypothesis overfit on the sample data and the true solution of the samples can lead to inaccuracies.

Recently, intelligent systems have increasingly adopted the strategy of empirical risk minimization to find the optimal solution. However, it has been shown that this approach can lead to overfitting of the classification function, resulting in poor generalization performance. The reason for this is that the empirical risk, which is used to approximate the true risk, is not a reliable indicator of the algorithm’s performance due to the limited number of training samples compared to the real-world data to be classified.

To address this issue, the concept of generalization error bounds has been established in statistics, which provides a measure of the bias and convergence rate between the empirical risk and expected risk of a learning algorithm. In order to accurately assess the performance of a learning algorithm, both the empirical risk and confidence risk must be taken into account. The empirical risk measures the error of the classifier on a given sample, while the confidence risk reflects the degree of trust we can place in the classifier’s predictions on new, unknown data.

Hence, the key point is to improve algorithm efficiency and performance, especially for real-world problems [25].

Obviously, there is no way to calculate the confidence risk accurately; therefore, only an estimated interval can be given, which also makes it possible to calculate only an upper bound on the overall error with the accurate value. For this reason it is called the generalization error bound. There is a number of methods [26,27,28,29] that have been proposed to solve this problem. One of the different approaches, based on sensitivity analysis, was introduced and its purpose is to determine how many changes in the input data affect the output of the algorithm. The motivation for such analysis is to design robust systems that will not be affected by noise-corrupted inputs. The main motivation of this analysis is to design a robust system in which the inputs of this model are not affected by noise-corrupted pollution. Bousquet et al. [26] defined the stability concept of a learning algorithm based on sensitivity analysis, which showed how to derive generalized error boundaries based on empirical and omission errors.

The generalization error bound typically has the following properties: it is a function of increasing sample size, where the generalization error bound tends to zero; it is a function of hypothesis space capacity, where the larger the capacity of the hypothesis space, the harder the model is to learn, and the larger the generalization error bound.

Comparing the upper limits of generalization error is a common way to evaluate the generalization ability of learning algorithms. In this work, we analyze the generalization error bound of NMF algorithms with graph regularization using the stability properties of the algorithms.

Analyzing the generalization and stability of machine learning algorithms is a fundamental research topic in intelligent systems and data mining tasks. Our main contribution in this work is to study the generalization performance of the NMF algorithm through stability analysis. We use stability theory to prove the generalization bound of NMF and provide a qualitative description of its generalization performance.

The structure of this paper is as follows: Section 2 introduces necessary notions and definitions relevant to algorithmic stability; In Section 3, we characterize the stability of NMF algorithms through uniform stability and give the generalization of the upper bound of NMF algorithms when U is fixed, and when U is not fixed under the multiplicative rule; In Section 4, we describe the stability of non-negative matrix decomposition algorithms; In Section 5, we perform numerical experiments to demonstrate the conclusion proved in previous sections. In Section 6, we conclude the paper with a summary and outlook for further research.

2. Related Work

In traditional intelligent systems theory, methods for the bounding of generalization error can be classified into two categories: The first one is to constrain the generalization error bound by controlling the complexity of hypothesis spaces, as in the Vapnik-Chervonenkis (VC) theory [30] and Rademacher complexity [31]; The second category is based on algorithmic stability [26] and robustness-based analysis [32]. Based on the stability of the learning algorithm, this paper analyzes the stability and generalization of the NMF algorithm, qualitatively characterizes the generalization performance of NMF, and proves the generalization bound of NMF by using the stability theory.

The robustness of the machine learning method is considered by most real-world applications, especially human activity recognition (HAR) [33], and the stability of an algorithm is also an important theory to analyze the generalization in the machine learning method. The pioneering work can be tracked to [34], who introduced the idea that the expected sensitivity of the k-Nearest Neighbors algorithm to changes of individual examples could be used to obtain a variance bound of a leave-one-out estimator. Besides, the authors obtained a generalization bound for the k-nearest neighbor algorithm from this result. Kutin and Niyogi [35] introduced many weaker variants for stability, which could get a good and stable algorithm generalization boundary. Further, other notions of stability have been proposed, such as uniform argument stability [36], hypothesis stability [26], and hypothesis set stability [37]. Shalev-Shwartz et al. [38] proved that the stability of the algorithm also implies learnability for certain tasks. Hardt et al. [39] argued that models trained by the stochastic gradient method (SGM) with few iterations have fading away generalization error, which is algorithmically stable in the sense of Bousquet and Elisseeff in convex and non-convex optimization.

Recently, a generalization bound based on Bayesian Reinforcement Learning was obtained through the concept of algorithmic stability [40]. Additionally, Wu at al. [41] proposed an innovative unsupervised feature selection algorithm and obtained its generalization error bound by analyzing the stability of the algorithm. Our stability-based approach to NMF is based on the seminal work of Bousquet and Elisseeff [26]. To our knowledge, we are the first to provide a stability result for NMF.

3. Preliminaries

In this section, we begin with introduction of the notation and definitions associated with algorithmic stability analysis. Given a training set $S=\{(x_1,y_1),(x_2,y_2),\cdots ,(x_n,y_n)\}$ drawn i.i.d. from a joint probability distribution $\mathcal{D}(\mathcal{X},\mathcal{Y})$, we denote a labeled sample $(x,y)$ by z, then $S=\{z_1,z_2,\cdots ,z_n\}$, $\mathcal{X}\in R^n$, and $\mathcal{Y}\in \{-1,+1\}$. Hypothesis space is given by the finite family of functions $\mathcal{H}=\{h_1,h_2,\cdots ,h_d\}$, d is a function and h is called a hypothesis. Let h be the hypothesis chosen from $\mathcal{H}$. Our loss function L is defined as the following:

$$L:\mathcal{Y}\times \mathcal{Y}\to R_{+},$$

(1)

then the loss $L_z\left(h\right)$ of hypothesis h at point z can be defined as:

$$L_z\left(h\right)=L(h\left(x\right),y).$$

(2)

The empirical risk (or loss) $\widehat{R}\left(h\right)$ of hypothesis h on the training set $S=\{z_1,z_2,\cdots ,z_n\}$ is

$$\widehat{R}\left(h\right)=\sum _{i=1}^n{L}_{z_i}\left(h\right),$$

(3)

and its generalization error $R\left(h\right)$ are defined by

$$R\left(h\right)=\underset{\begin{array}{c}z\sim \mathcal{D}\end{array}}{E}{L}_z\left(h\right).$$

(4)

Given an algorithm $\mathcal{A}$, let $h_S\in \mathcal{H}$ be the hypothesis returned by training algorithm $\mathcal{A}$ on the training set S, L is the loss function defined on S, if $L_z\left(h\right)\le M(M\ge 0)$ for any $h\in \mathcal{H}$ and $z\in X\times Y$, the loss function L is said to be bounded and $L\le M$. This paper assumes that all hypotheses $h_S$ returned by algorithm $\mathcal{A}$ on the training set S are bounded, i.e., for any $h_S\in \mathcal{H}$, $L_z\left(h_S\right)\le M.$

A stable learning algorithm has the ability to remove an element of its train datasets without changing its results. If the differences between empirical and generalization error is considered as a variable, then it should have a small variance. And the smaller its expectation, the closer an empirical error for a stable algorithm is to its generalization error. In the following, we give the definition of consistent stability as followed.

Definition 1 ([26]). 

Let S and $S^{{}^{\prime }}$ be any two group of training samples that are divided into two parts by one single point, $h_S$ and $h_{S^{\prime }}$ are two hypotheses returned by algorithm $\mathcal{A}$ on the training sets S and $S^{{}^{\prime }}$, respectively, if for any $z\in S$ and $z\in S^{{}^{\prime }}$, we have

$$|L_z\left(h_S\right)-L_z\left(h_{S^{{}^{\prime }}}\right)|\le \beta ,$$

(5)

where $S_i^{{}^{\prime }}=\{(x_1,y_1),(x_2,y_2),\cdots ,(x_i^{{}^{\prime }},y_i^{{}^{\prime }}),\cdots ,(x_n,y_n)\},$ then the learning algorithm $\mathcal{A}$ is said to be uniformly β-stable, and the smallest β, which satisfies the inequality, which is called the stability coefficient of $\mathcal{A}.$ The definition of uniform stability of the algorithm also states that if the training sets S and $S^{{}^{\prime }}$ are similar, the loss incurred by the corresponding hypotheses $h_S$ and $h_{S^{{}^{\prime }}}$ is returned by $\mathcal{A}$, which cannot split by more than β. A uniformly β-stable learning algorithm is frequently referred to as β-stable, or, if no specific is given, the algorithm is considered to be stable. In general, the coefficient β is determined by the size of samples n.

Theorem 1 ([26]). 

Let the loss function $L\le M(M>0)$, the learning algorithm $\mathcal{A}$ be β-stable and the training set $S=\{(x_1,y_1),(x_2,y_2),\cdots ,(x_n,y_n)\}$ drawn i.i.d. from a distribution $\mathcal{D}$. Then it is at least possible that this inequality is true $1-\delta (0<\delta <1)$ for the hypothesis $h_S$ obtained by learning the algorithm $\mathcal{A}$ on the training set S:

$$R\left(h_S\right)\le \widehat{R}\left(h_S\right)+\beta +(2n\beta +M)\sqrt{\frac{log\frac{1}{\delta }}{2n}}.$$

(6)

The generalization bound given in Theorem 1 converges to $n\beta /\sqrt{n}=o\left(1\right)$, i.e., $\beta =o(1/\sqrt{n})$. For instance, when the coefficient $\beta =O(1/n)$, Theorem 1 guarantees that $R\left(h_S\right)-\widehat{R}\left(h_S\right)=O(1/n)$ with high probability.

4. Characterizing the Stability of Non-Negative Matrix Decomposition Algorithms

Numerous studies on NMF have concentrated on improving the generalization ability of NMF models by incorporating regularization terms or other methods. Additionally, generalization analysis has resulted in tighter generalization error bounds. However, no research has explored the generalization error bound for NMF algorithms using algorithmic stability analysis. Generally, analyzing the stability and generalization of learning algorithms is a fundamental and critical research task in intelligent systems. As such, this paper focuses on analyzing the generalization error bound of NMF algorithms through stability-based analysis.

The overview of our algorithm AS-NMF is described in Figure 1. There are three main parts: Algorithmic Stability, Definition of NMF based Algorithmic Stability and Generalization bound of NMF Algorithmic Stability. It defines the concept of algorithmic stability, which can analyze and measure generalization error bounds for NMF algorithm.

4.1. Stability of NMF

The above definition of algorithmic stability for binary classification problems can also be extended to the NMF algorithm. In this section, the NMF algorithm is primarily used to address the collaborative filtering problem, which involves item ratings as the dataset. These item ratings can be considered equivalent to labels in a binary classification problem. However, the datasets encountered an extended binary classification problem and a multi-categorical problem, i.e., $\{-1,+1\}$ to the item rating $\{1,2,\cdots ,R\}$, where $R>0$ is the highest rating given to the item by a user. Let us assume that m is the number of users and n is the labels; each user’s rating for each item can be considered as an input and its prediction as an output; thus, the output of the NMF algorithm in the collaborative filtering algorithm is $Y={\{1,2,\cdots ,R\}}^{m\times n}$. In the collaborative filtering problem, each item is rated by m users, so each input datum $\mathbf{x}\in R^m$ on the training datasets $S=\{({\mathbf{x}}_1,y_1),({\mathbf{x}}_2,y_2),\cdots ,({\mathbf{x}}_n,y_n)\}\in \mathcal{X}\times \mathcal{Y}$, let $X=\{{\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n\}$, then $X\in R^{m\times n}$.

In fact, the scoring matrix X is a sparse matrix, i.e., there are some missing values in the matrix. Let $\Omega =\{(i_1,j_1),(i_2,j_2),\cdots ,(i_m,j_m)\}$ denotes the index of a known value in X, the transpose of $\Omega$ is denoted by ${\Omega }^T=\{(j_1,i_1),(j_2,i_2),\cdots ,(j_m,i_m)\}$. Thus, the training set can be denoted as $X=\{X_{i,j}:(i,j)\in \Omega \}$, and ${\left({\mathcal{P}}_{\Omega }\left(X\right)\right)}_{ij}=\left\{\begin{array}{c}{X}_{ij},(i,j)\in \Omega ,\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.$

The matrix $X_{m\times n}$ processed by NMF is non-negative, and the matrix X is decomposed with

$$X\approx {\mathcal{P}}_{\Omega }\left(U{V}^T\right),U,V\ge 0$$

(7)

where $U=\{{\mathbf{u}}_1,{\mathbf{u}}_2,{\mathbf{u}}_3,\cdots ,{\mathbf{u}}_m\}\in R_{+}^{m\times r}$ is the basis matrix, $V=\{{\mathbf{v}}_1,{\mathbf{v}}_2,{\mathbf{v}}_3,\cdots ,{\mathbf{v}}_n\}\in R_{+}^{n\times r}$ is the coefficient matrix, $r\ll min\{m,n\}$.

The result of NMF is only an approximate solution and does not require X to be strictly equal to the inner product of U and V. It is necessary to establish a loss function to evaluate the degree of approximation before and after decomposition. Thus, the objective of the NMF problem becomes to find two non-negative matrices U and V that can minimize our loss function $L(X,U{V}^T)$. There are many loss functions available; here, we have chosen the Euclidean distance:

$$L_X\left(U{V}^T\right)={\parallel {\mathcal{P}}_{\Omega }(X-U{V}^T)\parallel }_F^2,$$

(8)

The most important goal of NMF is to minimize the error of the loss function $L_X\left(U{V}^T\right)$ by optimizing $\parallel {\mathcal{P}}_{\Omega }(X-U{V}^T){\parallel }_F^2$. In the presence of both U and V, $L_X\left(U{V}^T\right)$ can only be obtained as a local minimum since $L_X\left(U{V}^T\right)$ is non-convex. However, if the basis matrix U is fixed, L is a convex function, thus, V is chosen to minimize $L_X\left(U{V}^T\right)$. The family of hypothesis functions ${\mathcal{H}}_U$, induced by U on X for NMF is defined as follows:

$${\mathcal{H}}_U\left(X\right)=\left\{h_U\left(\mathbf{x}\right)=\underset{\mathbf{v}\in R_{+}^r}{min}{\parallel {\mathcal{P}}_{\Omega }(\mathbf{x}-U{\mathbf{v}}^T)\parallel }^2:\mathbf{x}\in R_{+}^m,U\in R_{+}^{m\times r}\right\}.$$

(9)

According to the definition of the loss function based on NMF, let $h_U$ be the hypothesis chosen from ${\mathcal{H}}_U$; then the loss $L_{\mathbf{x}}\left(h_U\right)$ of hypothesis $h_U$ at point $\mathbf{x}$ is defined as:

$$L_{\mathbf{x}}\left(h_U\right)=h_U\left(\mathbf{x}\right)=\underset{\mathbf{v}\in R_{+}^r}{min}{\parallel {\mathcal{P}}_{\Omega }(\mathbf{x}-U{\mathbf{v}}^T)\parallel }^2.$$

(10)

The loss or empirical error $\widehat{R}\left(h\right)$ of $h_U$ on the samples $\{{\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n\}$ are defined, respectively, by

$${\widehat{R}}_n\left(h_U\right)=\frac{1}{n}\sum _{i=1}^n{L}_{{\mathbf{x}}_i}\left(h_U\right),$$

(11)

and their generalization error $R\left(h_U\right)$ can be defined as

$$R\left(h_U\right)=\underset{\begin{array}{c}\mathbf{x}\sim \mathcal{D}\end{array}}{E}{L}_{\mathbf{x}}\left(h_U\right).$$

(12)

• ${\mathcal{H}}_U\left(X\right)$ for two different training sets X and $X^{{}^{\prime }}$ with U fixed, only the error of the last training sample differ, for any $\mathbf{x}\in X\backslash \left\{{\mathbf{x}}_n\right\}$ and ${\mathbf{x}}^{{}^{\prime }}\in X^{{}^{\prime }}\backslash \left\{{\mathbf{x}}_n^{{}^{\prime }}\right\}$, we have $h_U\left(\mathbf{x}\right)-h_U\left({\mathbf{x}}^{{}^{\prime }}\right)=0.$ Thus, $|h_U\left(\mathbf{x}\right)-h_{U^{{}^{\prime }}}\left({\mathbf{x}}^{{}^{\prime }}\right)|=|h_U\left({\mathbf{x}}_n\right)-h_U\left({\mathbf{x}}_n^{{}^{\prime }}\right)|$. When U is fixed, we only need to define $|h_U\left({\mathbf{x}}_n\right)-h_U\left({\mathbf{x}}_n^{{}^{\prime }}\right)|\le \beta$;
• If U is not fixed, different training sets X and $X^{{}^{\prime }}$ will update different basis matrices U and $U^{{}^{\prime }}$; so, for any $\mathbf{x}\in X,\mathbf{z}\in X\backslash \left\{{\mathbf{x}}_n\right\}$ and ${\mathbf{x}}^{{}^{\prime }}\in X^{{}^{\prime }}$, $\beta$ needs to be defined as follows:

  $$|h_U\left(\mathbf{x}\right)-h_{U^{{}^{\prime }}}\left({\mathbf{x}}^{{}^{\prime }}\right)|=max\{|h_U\left(\mathbf{z}\right)-h_{U^{{}^{\prime }}}\left(\mathbf{z}\right)|,|h_U\left({\mathbf{x}}_n\right)-h_{U^{{}^{\prime }}}\left({\mathbf{x}}_n^{{}^{\prime }}\right)\left|\right\}\le \beta .$$

  (13)

Uncertainty U is easy to obtain, by the nature of the absolute inequality,

$$\begin{array}{cc}\hfill |h_U\left({\mathbf{x}}_n\right)-h_{U^{{}^{\prime }}}\left({\mathbf{x}}_n^{{}^{\prime }}\right)|& =|h_U\left({\mathbf{x}}_n\right)-h_U\left({\mathbf{x}}_n^{{}^{\prime }}\right)+h_U\left({\mathbf{x}}_n^{{}^{\prime }}\right)-h_{U^{{}^{\prime }}}\left({\mathbf{x}}_n^{{}^{\prime }}\right)|\hfill \\ \hfill & \le |h_U\left({\mathbf{x}}_n\right)-h_U\left({\mathbf{x}}_n^{{}^{\prime }}\right)|+|h_U\left({\mathbf{x}}_n^{{}^{\prime }}\right)-h_{U^{{}^{\prime }}}\left({\mathbf{x}}_n^{{}^{\prime }}\right)|.\hfill \end{array}$$

(14)

Therefore, when U is fixed, the stability bound is smaller than the parameter bound when U is not fixed.

Based on the above two analyses, the following two classes of definitions of uniform stability of NMF are provided.

4.2. The Generalization upper Bound of NMF

Definition 2. 

Let $S=(X,Y)$ and $S^{{}^{\prime }}=(X^{{}^{\prime }},Y)$, where $X=\{{\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n\}$ and $X^{{}^{\prime }}=\{{\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n^{{}^{\prime }}\}$ differ only by the last sample, $h_U\left(X\right)=\left\{h_U\left(\mathbf{x}\right)=\underset{\mathbf{v}\in R_{+}^r}{min}{\parallel {\mathcal{P}}_{\Omega }(\mathbf{x}-U{\mathbf{v}}^T)\parallel }^2:\mathbf{x}\in R_{+}^m,U\in R_{+}^{m\times r}\right\}$ is a hypothesis returned by NMF to the training set X, U is the given basis matrix. For any $\mathbf{x}\in X$ and ${\mathbf{x}}^{{}^{\prime }}\in X^{{}^{\prime }}$, we have

$$|L_{\mathbf{x}}\left(h_U\right)-L_{{\mathbf{x}}^{{}^{\prime }}}\left(h_U\right)|\le \beta ,$$

(15)

then the NMF is uniformly β-stable when U is not fixed, and minimum β satisfies the inequality that can be called the stability coefficient of the NMF.

Definition 2 characterizes the definition of stability of NMF when U is fixed. The definition of stability of NMF in the case of multiplicative iterations (when the basis matrix U is uncertain in advance) is presented below.

Definition 3. 

Let $S=(X,Y)$ and $S^{{}^{\prime }}=(X^{{}^{\prime }},Y)$, where $X=\{{\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n\}$ and $X^{{}^{\prime }}=\{{\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n^{{}^{\prime }}\}$ differ only by the last sample, $h_U\left(X\right)=\left\{h_U\left(\mathbf{x}\right)=\underset{\mathbf{v}\in R_{+}^r}{min}{\parallel {\mathcal{P}}_{\Omega }(\mathbf{x}-U{\mathbf{v}}^T)\parallel }^2:\mathbf{x}\in R_{+}^m,U\in R_{+}^{m\times r}\right\}$ is the assumption that it is returned by the NMF to the training sample data X, U and $U^{{}^{\prime }}$ are the basis matrices generated by NMF on the training sets X and $X^{{}^{\prime }}$. For any $\mathbf{x}\in X$ and ${\mathbf{x}}^{{}^{\prime }}\in X^{{}^{\prime }}$, we have

$$|L_{\mathbf{x}}\left(h_U\right)-L_{{\mathbf{x}}^{{}^{\prime }}}\left(h_{U^{{}^{\prime }}}\right)|\le \beta ,$$

(16)

then NMF is said to be uniformly β-stable when U is not fixed, and the minimum β, which satisfies this inequality, is called the stability coefficient of the NMF.

Theorem 2. 

Let training set $S=(X,Y),X=\{{\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n\}$, and $X\approx U{V}^T(X\in R_{+}^{m\times n},U\in R_{+}^{m\times r},V\in R_{+}^{n\times r}),$ if for any $\mathbf{x}\in X(\mathbf{x}\in R_{+}^m)$, there is $\parallel \mathbf{x}\parallel \le R(R>0)$, then every row of V has an upper bound R, i.e., $\parallel \mathbf{v}\parallel \le R(\mathbf{v}\in V).$

In Theorem 2, we give some basic theory of NMF, where U is the basis matrix and V is the coefficient matrix, and every row of V has an upper bound R.

Theorem 3. 

Let $S=(X,Y)$ and $S^{{}^{\prime }}=(X^{{}^{\prime }},Y)$, where $X=\{{\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n\}$ and $X^{{}^{\prime }}=\{{\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n^{{}^{\prime }}\}$ differ only by the last sample, $h_U\left(X\right)=\left\{h_U\left(\mathbf{x}\right)=\underset{\mathbf{v}\in R_{+}^r}{min}{\parallel {\mathcal{P}}_{\Omega }(\mathbf{x}-U{\mathbf{v}}^T)\parallel }^2:\mathbf{x}\in R_{+}^m,U\in R_{+}^{m\times r}\right\}$ is a hypothesis returned by NMF to the training set X, U is the given basis matrix, and U is bounded, for example, there exists a constant $C\ge 0,$$\parallel U\parallel \le C$. Then the NMF is uniformly β-stable, representing the following β when the basis matrix U is fixed,

$$|L_{\mathbf{x}}\left(h_U\right)-L_{{\mathbf{x}}^{{}^{\prime }}}\left(h_U\right)|\le \beta ,$$

(17)

where $\beta =R(\sqrt{mr}+r)\parallel U\parallel .$

Proof. 

For any $\mathbf{x}=\{x_1,x_2,\cdots ,x_m\}\in X$ and ${\mathbf{x}}^{{}^{\prime }}=\{x_1^{{}^{\prime }},x_2^{{}^{\prime }},\cdots ,x_m^{{}^{\prime }}\}\in X^{{}^{\prime }}$,

$$\begin{array}{cc}\hfill & |L_{\mathbf{x}}\left(h_U\right)-L_{{\mathbf{x}}^{{}^{\prime }}}\left(h_U\right)|=|L_{{\mathbf{x}}_{\mathbf{n}}}\left(h_U\right)-L_{{\mathbf{x}}_n^{{}^{\prime }}}\left(h_U\right)|\hfill \\ \hfill =& |h_U\left({\mathbf{x}}_n\right)-h_U\left({\mathbf{x}}_n^{{}^{\prime }}\right)|\hfill \\ \hfill =& |\underset{\mathbf{v}\in R_{+}^r}{min}\parallel {\mathcal{P}}_{\Omega }({\mathbf{x}}_n-U{\mathbf{v}}^T){\parallel }^2-\underset{\mathbf{v}\in R_{+}^r}{min}{\parallel {\mathcal{P}}_{\Omega }({\mathbf{x}}_n^{{}^{\prime }}-U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T)\parallel }^2|\hfill \\ \hfill =& |\underset{\mathbf{v}\in R_{+}^r}{min}\parallel {\mathcal{P}}_{\Omega }({\mathbf{x}}_n-U{\mathbf{v}}^T){\parallel }^2+\underset{{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}(-\parallel {\mathcal{P}}_{\Omega }({\mathbf{x}}_n^{{}^{\prime }}-U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T){\parallel }^2)|\hfill \\ \hfill \le & \left|\underset{\mathbf{v},{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}(\parallel {\mathcal{P}}_{\Omega }({\mathbf{x}}_n-U{\mathbf{v}}^T){\parallel }^2-\parallel {\mathcal{P}}_{\Omega }({\mathbf{x}}_n^{{}^{\prime }}-U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T){\parallel }^2)\right|\hfill \\ \hfill \le & \left|\underset{\mathbf{v},{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\left(2{\mathcal{P}}_{{\Omega }^T}{\left({\mathbf{x}}_n^{{}^{\prime }}\right)}^T{\mathcal{P}}_{\Omega }\left(U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)-2{\mathcal{P}}_{{\Omega }^T}\left({\mathbf{x}}_n^T\right){\mathcal{P}}_{\Omega }\left(U{\mathbf{v}}^T\right)\right)\right|+\hfill \\ \hfill & \left|\underset{\mathbf{v},{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\left(\parallel {\mathcal{P}}_{\Omega }\left(U{\mathbf{v}}^T\right){\parallel }^2-{\parallel {\mathcal{P}}_{\Omega }\left(U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)\parallel }^2\right)\right|\hfill \\ \hfill =& |\underset{\mathbf{v},{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\sum _{i=1}^m{x}_i^{{}^{\prime }}\langle {\mathbf{e}}_i,{\mathcal{P}}_{\Omega }\left(U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)\rangle -\sum _{i=1}^m{x}_i\langle {\mathbf{e}}_i,{\mathcal{P}}_{\Omega }\left(U{\mathbf{v}}^T\right)\rangle |+\hfill \\ \hfill & \left|\underset{\mathbf{v},{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\langle {\mathcal{P}}_{\Omega }\left(U{\mathbf{v}}^T\right)+{\mathcal{P}}_{\Omega }\left(U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right),{\mathcal{P}}_{\Omega }\left(U{\mathbf{v}}^T\right)-{\mathcal{P}}_{\Omega }\left(U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)\rangle \right|\hfill \\ \hfill \le & R|\underset{\mathbf{v},{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\sum _{i=1}^m\langle {\mathbf{e}}_i,{\mathcal{P}}_{\Omega }\left(U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)\rangle -\sum _{i=1}^m\langle {\mathbf{e}}_i,{\mathcal{P}}_{\Omega }\left(U{\mathbf{v}}^T\right)\rangle |+\hfill \\ \hfill & \left|\underset{\mathbf{v},{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\langle {\mathcal{P}}_{\Omega }\left(U{(\mathbf{v}+{\mathbf{v}}^{{}^{\prime }})}^T\right),{\mathcal{P}}_{\Omega }\left(U{(\mathbf{v}-{\mathbf{v}}^{{}^{\prime }})}^T\right)\rangle \right|\hfill \\ \hfill =& R\left|\underset{\mathbf{v},{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\sum _{i=1}^m\langle {\mathbf{e}}_i,{\mathcal{P}}_{\Omega }\left(U{({\mathbf{v}}^{{}^{\prime }}-\mathbf{v})}^T\right)\rangle \right|+\hfill \\ \hfill & \left|\underset{\mathbf{v},{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\langle {\mathcal{P}}_{\Omega }\left(U{(\mathbf{v}+{\mathbf{v}}^{{}^{\prime }})}^T\right),{\mathcal{P}}_{\Omega }\left(U{(\mathbf{v}-{\mathbf{v}}^{{}^{\prime }})}^T\right)\rangle \right|\hfill \\ \hfill =& R\left|\underset{\mathbf{v},{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\sum _{i=1}^m\langle {\mathbf{e}}_i,\sum _{j=1}^r(v_j-v_j^{{}^{\prime }}){\mathcal{P}}_{\Omega }\left(U{\mathbf{e}}_j\right)\rangle \right|+\hfill \\ \hfill & \left|\underset{\mathbf{v},{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\langle \sum _{j=1}^r(v_j+v_j^{{}^{\prime }}){\mathcal{P}}_{\Omega }\left(U{\mathbf{e}}_j\right),\sum _{k=1}^r(v_k-v_k^{{}^{\prime }}){\mathcal{P}}_{\Omega }\left(U{\mathbf{e}}_k\right)\rangle \right|\hfill \\ \hfill =& R\left|\underset{\mathbf{v},{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\sum _{j=1}^r(v_j-v_j^{{}^{\prime }})\sum _{i=1}^m\langle {\mathbf{e}}_i,{\mathcal{P}}_{\Omega }\left(U{\mathbf{e}}_j\right)\rangle \right|+\hfill \\ \hfill & \left|\underset{\mathbf{v},{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\sum _{j,k=1}^r(v_j+v_j^{{}^{\prime }})(v_k-v_k^{{}^{\prime }})〈{\mathcal{P}}_{\Omega }\left(U{\mathbf{e}}_j\right),{\mathcal{P}}_{\Omega }\left(U{\mathbf{e}}_k\right)〉\right|\hfill \\ \hfill \le & R\sqrt{m}\sum _{j=1}^r(v_j-v_j^{{}^{\prime }})\parallel {\mathcal{P}}_{\Omega }\left(U{\mathbf{e}}_j\right)\parallel +\sum _{j,k=1}^r(v_j+v_j^{{}^{\prime }})(v_k-v_k^{{}^{\prime }})\parallel {\mathcal{P}}_{\Omega }\left(U{\mathbf{e}}_j\right)\parallel \parallel {\mathcal{P}}_{\Omega }\left(U{\mathbf{e}}_k\right)\parallel \hfill \\ \hfill \le & R\sqrt{mr}\parallel U\parallel +Rr\parallel U\parallel =\beta .\square \hfill \end{array}$$

The third inequality holds based on Holder inequality, and the fourth and fifth inequalities hold based on the Cauchy-Schwarz inequality.

In Theorem 3 we give the stability parameter $\beta$ of the NMF algorithm, and show that when the basis matrix U is fixed, the stability coefficient of NMF depends on the upper bound of the module length R for input data, the dimension r of the hidden matrix, the number of dimensions m is feature space and the Frobenius norm $\parallel U\parallel$ of the basis matrix.

Theorem 4. 

Let $S=(X,Y)$, where $X=\{{\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n\}$, and $\mathbf{x}\in X$, $\parallel \mathbf{x}\parallel \le R$. ${\mathcal{H}}_U\left(X\right)=\left\{h_U\left(\mathbf{x}\right)=\underset{\mathbf{v}\in R_{+}^r}{min}{\parallel {\mathcal{P}}_{\Omega }(\mathbf{x}-U{\mathbf{v}}^T)\parallel }^2:\mathbf{x}\in R_{+}^m,U\in R_{+}^{m\times r}\right\}$ is a hypothesis returned by NMF to the training set X, U are the basis matrices generated by NMF on the training sets X, where U is bounded, i.e., there exists a constant $C\ge 0$, $\parallel U\parallel \le C$. When U is fixed, the following inequalities can be established at least $1-\delta$:

$$R\left(h_U\right)\le \widehat{R}\left(h_U\right)+R\sqrt{mr}\parallel U\parallel +Rr\parallel U\parallel +(2nR\sqrt{mr}\parallel U\parallel +2nRr\parallel U\parallel +M)\sqrt{\frac{log\frac{1}{\delta }}{2n}}.$$

(18)

In Theorem 3 we have obtained that NMF is $\beta$-stable, and by plugging the stability coefficients $\beta$ obtained in Theorem 3 into the bounds of Theory 1, we obtain an upper bound of this generalization for the NMF algorithm when U is fixed.

The results of Theorem 4 present the upper bound of the generalization for the NMF algorithm, which are controlled by the upper bound of the module length R for the input datasets, the dimension r and m is the hidden matrix and the feature space, the Frobenius norm $\parallel U\parallel$ of basis matrix as well as the input data n.

4.3. The Generalization upper Bound of NMF under Multiplicative Rule

Theorem 5. 

Let $S=(X,Y)$, $S^{{}^{\prime }}=(X^{{}^{\prime }},Y)$, where $X=\{{\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n\}$, $X^{{}^{\prime }}=\{{\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n^{{}^{\prime }}\}$, for any $\mathbf{x}\in X$ and ${\mathbf{x}}^{{}^{\prime }}\in X^{{}^{\prime }}$, $\parallel \mathbf{x}\parallel \le R$, $\parallel {\mathbf{x}}^{{}^{\prime }}\parallel \le R(R>0)$. ${\mathcal{H}}_U\left(X\right)=\left\{h_U\left(\mathbf{x}\right)=\underset{\mathbf{v}\in R_{+}^r}{min}{\parallel {\mathcal{P}}_{\Omega }(\mathbf{x}-U{\mathbf{v}}^T)\parallel }^2:\mathbf{x}\in R_{+}^m,U\in R_{+}^{m\times r}\right\}$ is a hypothesis returned by NMF to the training set X, U and $U^{{}^{\prime }}$ are the basis matrices generated by NMF on the training sets X and $X^{{}^{\prime }}$, where U and $U^{{}^{\prime }}$ is the bound, for example, the existing constant $C\ge 0,$$\parallel U\parallel \le C,\parallel U^{{}^{\prime }}\parallel \le C$. Then NMF is consistently β-stable with respect to the following β,

$$|L_{\mathbf{x}}\left(h_U\right)-L_{{\mathbf{x}}^{{}^{\prime }}}\left(h_{U^{{}^{\prime }}}\right)|\le \beta ,$$

(19)

where $\beta =R(\sqrt{mr}+r+R\sqrt{m}+r\sqrt{m})\parallel U\parallel .$

Proof. 

For any $\mathbf{x}=\{x_1,x_2,\cdots ,x_m\}\in X$, ${\mathbf{x}}^{{}^{\prime }}=\{x_1^{{}^{\prime }},x_2^{{}^{\prime }},\cdots ,x_m^{{}^{\prime }}\}\in X^{{}^{\prime }}$ and $\mathbf{z}=\{x_1,x_2,\cdots ,x_m\}\in X\backslash \left\{{\mathbf{x}}_n\right\}$, when U is not fixed in advance, to different training datasets X and $X^{{}^{\prime }}$, the NMF will update different basis matrices U and $U^{{}^{\prime }}$, based on the above analysis, in this case,

$$\begin{array}{cc}\hfill L_{\mathbf{x}}\left(h_U\right)-L_{{\mathbf{x}}^{{}^{\prime }}}\left(h_{U^{{}^{\prime }}}\right)& =|h_U\left(\mathbf{x}\right)-h_{U^{{}^{\prime }}}\left({\mathbf{x}}^{{}^{\prime }}\right)|\hfill \\ \hfill & =max\left\{\right|h_U\left(\mathbf{z}\right)-h_{U^{{}^{\prime }}}\left(\mathbf{z}\right)|,|h_U\left({\mathbf{x}}_n\right)-h_{U^{{}^{\prime }}}\left({\mathbf{x}}_n^{{}^{\prime }}\right)\left|\right\}.\square \hfill \end{array}$$

(20)

Meanwhile, according to the nature of the absolute inequality, we know

$$\begin{array}{cc}\hfill |h_U\left({\mathbf{x}}_n\right)-h_{U^{{}^{\prime }}}\left({\mathbf{x}}_n^{{}^{\prime }}\right)|& =|h_U\left({\mathbf{x}}_n\right)-h_U\left({\mathbf{x}}_n^{{}^{\prime }}\right)+h_U\left({\mathbf{x}}_n^{{}^{\prime }}\right)-h_{U^{{}^{\prime }}}\left({\mathbf{x}}_n^{{}^{\prime }}\right)|\hfill \\ \hfill & \le |h_U\left({\mathbf{x}}_n\right)-h_U\left({\mathbf{x}}_n^{{}^{\prime }}\right)|+|h_U\left({\mathbf{x}}_n^{{}^{\prime }}\right)-h_{U^{{}^{\prime }}}\left({\mathbf{x}}_n^{{}^{\prime }}\right)|.\hfill \end{array}$$

(21)

Theorem 3 has proved that $|h_U\left({\mathbf{x}}_n\right)-h_U\left({\mathbf{x}}_n^{{}^{\prime }}\right)|\le R\sqrt{mr}\parallel U\parallel +Rr\parallel U\parallel .$ Therefore, only the upper bound of $|h_U\left({\mathbf{x}}^{{}^{\prime }}\right)-h_{U^{{}^{\prime }}}\left({\mathbf{x}}^{{}^{\prime }}\right)|$ is proved below.

$$\begin{array}{cc}\hfill & |L_{{\mathbf{x}}^{{}^{\prime }}}\left(h_U\right)-L_{{\mathbf{x}}^{{}^{\prime }}}\left(h_{U^{{}^{\prime }}}\right)|\hfill \\ \hfill =& |\underset{\mathbf{v}\in R_{+}^r}{min}\parallel {\mathcal{P}}_{\Omega }({\mathbf{x}}_n^{{}^{\prime }}-U{\mathbf{v}}^T){\parallel }^2-\underset{{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{min}{\parallel {\mathcal{P}}_{\Omega }({\mathbf{x}}_n^{{}^{\prime }}-U^{{}^{\prime }}{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T)\parallel }^2|\hfill \\ \hfill \le & |\parallel {\mathcal{P}}_{\Omega }({\mathbf{x}}_n^{{}^{\prime }}-U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T){\parallel }^2+\underset{{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}(-\parallel {\mathcal{P}}_{\Omega }({\mathbf{x}}_n^{{}^{\prime }}-U^{{}^{\prime }}{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T){\parallel }^2)|\hfill \\ \hfill \le & \left|\underset{{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}(\parallel {\mathcal{P}}_{\Omega }({\mathbf{x}}_n^{{}^{\prime }}-U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T){\parallel }^2-\parallel {\mathcal{P}}_{\Omega }({\mathbf{x}}_n^{{}^{\prime }}-U^{{}^{\prime }}{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T){\parallel }^2)\right|\hfill \\ \hfill \le & \left|\underset{{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\left(2{\mathcal{P}}_{{\Omega }^T}{\left({\mathbf{x}}_n^{{}^{\prime }}\right)}^T{\mathcal{P}}_{\Omega }\left(U^{{}^{\prime }}{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)-2{\mathcal{P}}_{{\Omega }^T}{\left({\mathbf{x}}_n^{{}^{\prime }}\right)}^T{\mathcal{P}}_{\Omega }\left(U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)\right)\right|+\hfill \\ \hfill & \left|\underset{{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\left(\parallel {\mathcal{P}}_{\Omega }\left(U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right){\parallel }^2-{\parallel {\mathcal{P}}_{\Omega }\left(U^{{}^{\prime }}{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)\parallel }^2\right)\right|\hfill \\ \hfill =& |\underset{{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\sum _{i=1}^m{x}_i^{{}^{\prime }}\langle {\mathbf{e}}_i,{\mathcal{P}}_{\Omega }\left(U^{{}^{\prime }}{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)\rangle -\sum _{i=1}^m{x}_i^{{}^{\prime }}\langle {\mathbf{e}}_i,{\mathcal{P}}_{\Omega }\left(U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)\rangle |+\hfill \\ \hfill & \left|\underset{{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\langle {\mathcal{P}}_{\Omega }\left(U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)+{\mathcal{P}}_{\Omega }\left(U^{{}^{\prime }}{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right),{\mathcal{P}}_{\Omega }\left(U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)-{\mathcal{P}}_{\Omega }\left(U^{{}^{\prime }}{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)\rangle \right|\hfill \\ \hfill \le & R|\underset{{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\sum _{i=1}^m\langle {\mathbf{e}}_i,{\mathcal{P}}_{\Omega }\left(U^{{}^{\prime }}{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)\rangle -\sum _{i=1}^m\langle {\mathbf{e}}_i,{\mathcal{P}}_{\Omega }\left(U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)\rangle |+\hfill \\ \hfill & \left|\underset{{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\langle {\mathcal{P}}_{\Omega }\left(U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)+{\mathcal{P}}_{\Omega }\left(U^{{}^{\prime }}{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right),{\mathcal{P}}_{\Omega }\left(U{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)-{\mathcal{P}}_{\Omega }\left(U^{{}^{\prime }}{\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)\rangle \right|\hfill \\ \hfill =& R\left|\underset{{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\sum _{i=1}^m〈{\mathbf{e}}_i,{\mathcal{P}}_{\Omega }\left((U^{{}^{\prime }}-U){\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)〉\right|+\hfill \\ \hfill & \left|\underset{{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\langle {\mathcal{P}}_{\Omega }\left((U+U^{{}^{\prime }}){\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right),{\mathcal{P}}_{\Omega }\left((U-U^{{}^{\prime }}){\left({\mathbf{v}}^{{}^{\prime }}\right)}^T\right)\rangle \right|\hfill \\ \hfill =& R\left|\underset{{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\sum _{i=1}^m\sum _{j=1}^r{v}_j^{{}^{\prime }}\langle {\mathbf{e}}_i,{\mathcal{P}}_{\Omega }\left((U^{{}^{\prime }}-U){\mathbf{e}}_j\right)\rangle \right|+\hfill \\ \hfill & \left|\underset{{\mathbf{v}}^{{}^{\prime }}\in R_{+}^r}{max}\sum _{j,k=1}^r{v}_j^{{}^{\prime }}{v}_k^{{}^{\prime }}\langle {\mathcal{P}}_{\Omega }\left((U+U^{{}^{\prime }}){\mathbf{e}}_j\right),{\mathcal{P}}_{\Omega }\left((U-U^{{}^{\prime }}){\mathbf{e}}_k\right)\rangle \right|\hfill \\ \hfill \le & R\sqrt{m}\sum _{j=1}^r{v}_j^{{}^{\prime }}\parallel {\mathcal{P}}_{\Omega }\left((U-U^{{}^{\prime }}){\mathbf{e}}_j\right)\parallel +\sum _{j,k=1}^r{v}_j^{{}^{\prime }}{v}_k^{{}^{\prime }}\parallel {\mathcal{P}}_{\Omega }\left((U+U^{{}^{\prime }}){\mathbf{e}}_j\right)\parallel \parallel {\mathcal{P}}_{\Omega }\left((U-U^{{}^{\prime }}){\mathbf{e}}_k\right)\parallel \hfill \\ \hfill \le & R^2\sqrt{m}\parallel U\parallel +Rr\sqrt{m}\parallel U\parallel \hfill \\ \hfill \le & R\sqrt{m}(R+r)\parallel U\parallel ,\hfill \end{array}$$

The fifth inequality holds based on the Cauchy-Schwarz inequality. Therefore,

$$\begin{array}{cc}\hfill & |L_{{\mathbf{x}}_n}\left(h_U\right)-L_{{\mathbf{x}}_n^{{}^{\prime }}}\left(h_U\right)|+|L_{{\mathbf{x}}^{{}^{\prime }}}\left(h_U\right)-L_{{\mathbf{x}}^{{}^{\prime }}}\left(h_{U^{{}^{\prime }}}\right)|\hfill \\ \hfill \le & R\sqrt{mr}\parallel U\parallel +Rr\parallel U\parallel +R\sqrt{m}(R+r)\parallel U\parallel \hfill \\ \hfill \le & R(\sqrt{mr}+r+R\sqrt{m}+r\sqrt{m})\parallel U\parallel ,\hfill \end{array}$$

(22)

thus,

$$|L_{\mathbf{x}}\left(h_U\right)-L_{{\mathbf{x}}^{{}^{\prime }}}\left(h_{U^{{}^{\prime }}}\right)|\le R(\sqrt{mr}+r+R\sqrt{m}+r\sqrt{m})\parallel U\parallel =\beta .$$

(23)

In Theorem 4, we give the stability parameter $\beta$ of the NMF algorithm when the basis matrix U is not fixed. And the conclusion of NMF shows that its stability parameter depends on the upper bound for the module length R of input data, the dimension r of the hidden matrix and Frobenius norm $\parallel U\parallel$ of the basic matrix.

Theorem 6. 

Let $S=(X,Y)$, where $X=\{{\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_n\}$, and $\mathbf{x}\in X$, $\parallel \mathbf{x}\parallel \le R$. ${\mathcal{H}}_U\left(X\right)=\left\{h_U\left(\mathbf{x}\right)=\underset{\mathbf{v}\in R_{+}^r}{min}{\parallel {\mathcal{P}}_{\Omega }(\mathbf{x}-U{\mathbf{v}}^T)\parallel }^2:\mathbf{x}\in R_{+}^m,U\in R_{+}^{m\times r}\right\}$ is a hypothesis returned by NMF to the training set X, U are the basis matrices generated by NMF on the training sets X, where U is the bound, for example, there exists a constant $C\ge 0$, $\parallel U\parallel \le C$. When U is not fixed, the following inequalities hold with a probability of at least $1-\delta$:

$$R\left(h_U\right)\le \widehat{R}\left(h_U\right)+R(\sqrt{mr}+r+R\sqrt{m}+r\sqrt{m})\parallel U\parallel +\left(2n\right(R(\sqrt{mr}+r+R\sqrt{m}+r\sqrt{m})\parallel U\parallel )+M)\sqrt{\frac{log\frac{1}{\delta }}{2n}}.$$

(24)

In Theorem 5, we have obtained that NMF is $\beta$-stable, and, by plugging the stability coefficients $\beta$ obtained in Theorem 5 into the bounds of Theory 1, we obtain an upper bound for the generalization of the NMF algorithm when U is not fixed.

The results of Theorem 6 present that the generalization for the upper bound of the NMF is controlled via the upper bound of the module length R for the input datasets, the dimension r of the hidden matrix and the Frobenius norm $\parallel U\parallel$ for the basic matrix as well as the input data n.

5. Experiments

5.1. Experimental Datasets

Our experiments are performed on three common and real-world datasets; these datasets are widely used to evaluate the recommendation systems: Movielens-100k [42], Movielens-1m [43] and Yelp2018 [44]. We are primarily verifying the performance of recommendation tasks and in terms of algorithmic stability analysis. We use some existing preprocessing methods to convert these datasets into implicit feedback. Specifically, a user’s rating is converted into a binary representation to indicate whether the user has commented on an item. The statistics of these three datasets are described in the

Table 1. Statistical detail of the evaluation datasets.

Datasets	#Users	#Items	#Ratings	Density
Movielens-100k	943	1628	100,000	6.30%
Movielens-1m	6040	3706	1,000,209	4.47%
Yelps2018	31,668	38,048	1,561,406	0.13%

.

5.2. Compared Models

We compare our method with some state-of-the-art machine learning model on different recommendation tasks. The compared models include:

• BPR [45]: BPR is a commonly used recommendation model that optimizes by matrix factorization with the Bayesian, which can get the personalized ranking via objective function.
• WMF [46]: WMF is a non-sampling recommendation model that can deal all missing interactions with negative instances and weight them uniformly; it is trained via the efficient non-sampling method.
• SISA [47]: SISA is realized by a machine unlearning method that can randomly split the training samples into parts, and then aggregates the results from all models through averaging in order to make the prediction.

5.3. Evaluation Methodology and Settings

In our experiments, we select the 60%, 70%, 80%, 90% samples to construct the training datasets and take the remaining data as test sets. Then, the 10% samples are randomly selected from the training datasets as the validation set. NDCG@K are used to evaluate the recommendation performance. They are widely used methods for measuring the ground truth which is ranked among the top K items, and they also offer a position-ranking metric that is used to measure whether to assign higher scores for higher positions. We use the grid search to adjust our AS-NMF and achieve the best hype-parameters of the optimal performances.

5.4. Performance and Algorithmic Stability Efficiency

In this section, the proposed AS-NMF method is evaluated with different methods of recommendation results, and we set the split rate N = 60, 70, 80 and 90 in our experiments. The experimental results are shown in

Table 2. Comparison of different methods for recommendation accuracy. The proposed AS-NMF method is tested with a different training set rate in this Table. The best results and improvement are highlighted in bold.

Datasets	Methods	Accuracy (Training Set Rate)
		60%	70%	80%	90%
Yelps2018	BPR	0.6135	0.6215	0.6367	0.6881
	WMF	0.6233	0.6381	0.6529	0.7190
	SISA	0.6201	0.7227	0.7371	0.7537
	AS-NMF	0.6289	0.7329	0.7419	0.7719
	Increase (%)	0.90	1.41	0.65	2.4
Movielens-100k	BPR	0.5881	0.6081	0.6638	0.6397
	WMF	0.5988	0.6228	0.6913	0.6981
	SISA	0.5976	0.7059	0.7286	0.7310
	AS-NMF	0.6415	0.7415	0.7306	0.7742
	Increase (%)	7.13	5.04	0.27	5.91
Movielens-1m	BPR	0.5203	0.6466	0.6071	0.6849
	WMF	0.5975	0.6095	0.6689	0.6736
	SISA	0.6148	0.7232	0.7092	0.7625
	AS-NMF	0.6384	0.7384	0.7521	0.7735
	Increase (%)	3.84	2.10	6.05	1.44

.

Our proposed AS-NMF algorithm outperformed state-of-the-art baseline methods on three datasets, namely Yelp2018, Movielens-100k, and Movielens-1m. Specifically, we observed maximum improvements of 2.41%, 7.13%, and 6.60% on Yelp2018, Movielens-100k, and Movielens-1m datasets, respectively, compared to the base models BPR, WMF, and SISA.

In comparison to the other three methods, our AS-NMF achieved accuracy improvements of 7.13%, 5.04%, 0.27%, and 5.91% on Movielens-100k, respectively. Since learning user preferences from sparse user-item interactions is challenging, the sparsity parameter of AS-NMF was carefully set to balance efficiency and performance.

Despite the relative density of Movielens-100k and Movielens-1m datasets, our AS-NMF algorithm produced remarkable results. Our proposed method effectively preserves collaborative information with different partition rates, which is reflected in the enhanced performance observed in the results.

Otherwise, we observe the performances of different methods on the top-ranking 10, 20, 30 and 50 with NDCG@K in order to evaluate performance on each dataset, respectively. The NDCG@K results of different methods are shown in

Table 3. Comparison of NDCG@K for different recommendation methods. The proposed AS-NMF method is tested with a different data partition in this Table. The best results and improvements are highlighted in bold.

Datasets	Methods	NDCG@K
		NDCG@10	NDCG@20	NDCG@30	NDCG@50
Yelps2018	BPR	0.4168	0.4936	0.5001	0.5021
	WMF	0.4229	0.5122	0.5128	0.5220
	SISA	0.4182	0.5294	0.5295	0.5384
	AS-NMF	0.4238	0.5391	0.5365	0.5473
	Increase (%)	0.21	1.83	1.32	1.65
Movielens-100k	BPR	0.4534	0.4614	0.4908	0.4939
	WMF	0.4295	0.4435	0.4924	0.4997
	SISA	0.4019	0.5138	0.4986	0.5026
	AS-NMF	0.4839	0.5146	0.5104	0.5169
	Increase(%)	6.73	0.16	2.37	2.85
Movielens-1m	BPR	0.3079	0.4137	0.4516	0.4633
	WMF	0.3198	0.4301	0.4543	0.4443
	SISA	0.3218	0.4409	0.4611	0.4593
	AS-NMF	0.3373	0.4670	0.4681	0.4636
	Increase (%)	4.82	5.92	1.52	0.06

on datasets of Yelp2018, Movielens-100k and Movielens-1m. The best performance results are improved by 1.83%, 6.73% and 5.92% compared with the other top-ranking algorithms. For example, the performances of AS-NMF on Yelps2018 and Movielens-100k are better than Movielens-1m on NDCG@50. Our proposed stability analysis method of NMF can further improve the performance of generalization. Experimental results show that our AS-NMF algorithm can effectively optimize the effectiveness of recommendation.

Furthermore, Figure 2 shows the performance and stability of models based on different numbers of samples, which is achieved by randomly selected training and the top-N item ranks. Figure 2a displays the accuracy and stability performance of four methods on Movielens-100k, with different training rates. The bar-plot depicts the accuracy for various training sample rates, while the line-plot shows the upper bound for stability performance. The x-axis denotes the percent of training set, and the y-axis represents the accuracy and stability performance of stable analysis.

Figure 2b shows the NDCG@K performance and stability of four models on Yelp2018, with varying top-N item ranks. The bar-plot describes NDCG@K for different top-N item ranks, and the line-plot shows the upper bound for stability performance. The x-axis denotes the percent of training set, while the y-axis represents the NDCG score and stability performance of our stable analysis of NMF.

Combining the bar and line plots, we observe that the accuracy and performance improve as the training samples increase. However, the stability of BPR, WMF, and SISA changes in the opposite direction, indicating that the stable performance of our AS-NMF is superior to those three methods.

The numerical results demonstrate that the stability parameter of AS-NMF depends on the upper bound of the module length of the input data, dimension of the hidden matrix, and Frobenius norm of the basis matrix. The bounds provide additional evidence of the superior stability of our proposed method.

6. Conclusions

In this paper, we propose a general NMF-based recommendation algorithm to predict and recommend the labels of new samples. The model uses NMF and enhanced stability to recommend items to users. Experimental results show that our proposed stability analysis method of NMF can further improve the performance of generalization and can effectively optimize the effectiveness of the recommendation. Then, based on the stability analysis of the algorithm, we studied the general generalization error bounds corresponding to the NMF algorithm, and obtained two general generalization bounds.

In the case of fixed U, the results show that the NMF algorithm is consistently $\beta$-stable, where $\beta =R(\sqrt{mr}+r)\parallel U\parallel$, and it can be obtained that the generalization upper bound of the NMF prediction algorithm depends on the upper bound on the module length of the input data, the dimension of the hidden matrix, the dimension of the eigenspace and the Frobenius norm of the basis matrix, as well as the input data. In the case where U is not fixed under the multiplicative update rule, the results show that the NMF algorithm is consistently $\beta$-stable, where $\beta =R(\sqrt{mr}+r+R\sqrt{m}+r\sqrt{m})\parallel U\parallel$, and it can be obtained that the generalization upper bound of the NMF prediction algorithm depends on its upper bound of the module length for the input data, the dimension of hidden matrix, the dimensions of feature space, the Frobenius norm of the basic matrix, as well as the input data. In the experimental comparison on Yelp2018 and Movielens-1m datasets, our method shows the stability of the recommendation and performs better than others.

In the future, we will try to approach the upper bounds of the real error more closely, because the smaller the upper bound for an algorithm, the better the performance of the model. One can define different loss functions according to actual needs by this method alone. Each of the considered applications is just a straightforward demonstration of the advantages of enhanced stability. In future research, we will apply it to face recognition and other areas.