Fast Global and Local Semi-Supervised Learning via Matrix Factorization

Abstract

Matrix factorization has demonstrated outstanding performance in machine learning. Recently, graph-based matrix factorization has gained widespread attention. However, graph-based methods are only suitable for handling small amounts of data. This paper proposes a fast semi-supervised learning method using only matrix factorization, which considers both global and local information. By introducing bipartite graphs into symmetric matrix factorization, the technique can handle large datasets effectively. It is worth noting that by utilizing tag information, the proposed symmetric matrix factorization becomes convex and unconstrained, i.e., the non-convex problem ${min}_x{(1-x^2)}^2$ is transformed into a convex problem. This allows it to be optimized quickly using state-of-the-art unconstrained optimization algorithms. The computational complexity of the proposed method is $O\left(nmd\right)$, which is much lower than that of the original symmetric matrix factorization, which is $O\left(n^{2}d\right)$, and even lower than that of other anchor-based methods, which is $O(nmd+m^{2}n+m^3)$, where n represents the number of samples, d represents the number of features, and $m\ll n$ represents the number of anchors. The experimental results on multiple public datasets indicate that the proposed method achieves higher performance in less time.

1. Introduction

As technology progresses, particularly with the pervasive adoption of smartphones, the volume of data surrounding us is increasing at an exponential rate. Consequently, the effective processing and analysis of this vast dataset have become critically important. For instance, rapidly distinguishing images within a mobile device’s photo gallery presents significant challenges to both memory capacity and computational power. Therefore, it is imperative to develop algorithms that minimize memory usage while ensuring rapid solution speeds and achieving satisfactory learning outcomes.

Matrix factorization (MF) plays an important role in fields such as machine learning and data analysis. It can learn a low-dimensional representation of data from high-dimensional data. Many classical matrix factorization methods have been proposed, including non-negative matrix factorization [1,2], singular value decomposition (SVD) [3], principal component analysis (PCA) [4], and concept factorization [5].

Non-negative matrix factorization (NMF) has become popular in recent years. It factorizes data into a set of non-negative bases and their representations under these non-negative bases. Despite its impressive performance and interpretability, NMF can only handle non-negative data due to non-negativity constraints. Since in the case of the same low rank the error of NMF is usually larger than that obtained by SVD, Ref. [6] proposed a low-rank matrix factorization with orthogonal constraints.

Matrix factorization considers global information while ignoring local manifolds. Many graph-based matrix factorization methods have been proposed to address this problem. Cai et al. proposed GNMF [7] and LCCF [8], which consider the local manifold. Yi et al. proposed NMF-LCAG [9], which considers both reconstruction and local information. Peng et al. proposed GCCF [10], which uses the maximum correntropy criterion (MCC) against outliers. Wang et al. proposed CHNMF [11], which uses a hypergraph to explore high-order geometric information.

The above methods are all unsupervised and cannot use known information. Using a small amount of labeled data leads to a significant improvement in performance. Liu et al. proposed CNMF [12] and CCF [13], which constrain samples with the same classes into the same representation. Zhang et al. proposed NMFCC [14], which uses the labeled information to construct a must-link and cannot-link matrix. In fact, an unsupervised graph-based MF method can be transformed into a semi-supervised one by incorporating a must-link and cannot-link matrix. Peng et al. proposed CSNMF [15] and CSCF [16], which assign adaptive neighbors to construct an appropriate adjacency matrix. Zhou et al. proposed CLMF [17], which uses sparsity-induced similarity (SIS) to adaptively learn the adjacency matrix.

A particular special type of matrix factorization, called symmetric matrix factorization (SMF), directly factorizes the graph matrix into identical low-rank matrices. Since the graph contains only local relations between data, its result contains only local information. Zhang et al. proposed SNMFCC [14], which incorporates the must-link and cannot-link matrix into the adjacency matrix. Wu et al. proposed PCPSNMF [18], which adaptively and simultaneously teaches the adjacency matrix and propagates the initial pairwise constraints among the data. Chavoshinejad et al. proposed S⁴NMF, which introduces the self-supervised information into semi-supervised symmetric NMF. Yin et al. proposed HSSNMF [19], which uses the hypergraph-based pairwise constraint propagation algorithm to capture the high-order information. It is important to note that the essence of SMF is ${min}_x{(1-x^2)}^2$, which is non-convex.

Graph-based and symmetric matrix factorization methods are unsuitable for addressing excessive data because they require the storage and computation of an n-order matrix. To address this issue, bipartite graph-based methods have been proposed. Liu et al. [20] proposed the local anchor embedding (LAE) algorithm. Wang et al. proposed EAGR [21], which uses FLAE, an improved AGR algorithm, by modifying the estimation of local weight and construction of an adjacency matrix. Nie et al. proposed the BGSSL [22], which applies a bipartite graph to graph-based semi-supervised learning (GSSL). It should be noted that it is difficult to apply an $n\times m$-order bipartite graph to SMF, as it requires an n-order fully adjacent matrix.

In response to the substantial memory demands and high computational complexity in prior methodologies, this paper proposes a novel fast global local matrix factorization (FGLMF) model aimed at enhancing performance while addressing these critical considerations. The method has the following benefits.

• A novel method using only matrix factorization is proposed. It simultaneously learns the global and local information between data, and it is easily interpretable.
• A bipartite graph is introduced to the symmetric matrix factorization. The computational complexity of the proposed method needs $O\left(nmd\right)$, which is smaller than other bipartite graph-based methods’ $O(nmd+m^{2}d+m^3)$.
• The proposed SMF is convex due to using the label information. Therefore, every optimal solution is the global optimal solution, and it can be solved very fast.

2. Preliminaries

2.1. Bipartite Graph

Let n represent the number of samples, m represent the number of anchor points, and d represent the data dimension. A graph is utilized to represent the structural relationships among samples. Therefore, a graph necessitates the computation and storage of an n-dimensional matrix. It is well known that the computational complexity for multiplying an $n\times p$ matrix with a $p\times q$ matrix is $O\left(npq\right)$. Consequently, if there exists an n-dimensional matrix such as a full graph within the model, subsequent computations will inevitably incur a complexity greater than $O\left(n^2\right)$. A bipartite graph is a particular type of graph representing the structural relationship between samples and anchors, it necessitates the computation and storage of an $n\times m$ matrix, where $m\ll n$. Consequently, leveraging a bipartite graph frequently enhances the model’s operational efficiency. A bipartite graph can be represented as $G=[\mathbf{X},\mathbf{S},\mathbf{U}]$, where $\mathbf{X}$ stands for samples, $\mathbf{U}$ stands for anchor points, and $\mathbf{S}$ stands for the adjacency relationship between samples and anchor points. The full adjacency matrix of a bipartite graph can be represented as

$$\mathbf{B}=\left[\begin{array}{cc}\mathbf{0}& \mathbf{S}\\ {\mathbf{S}}^{\top }& \mathbf{0}\end{array}\right].$$

(1)

Given a graph $\mathbf{B}$, its Laplacian matrix can be represented as

$$\mathbf{L}=\mathbf{D}-\mathbf{B},$$

(2)

where $\mathbf{D}$ is the degree matrix of $\mathbf{B}$ with ${\mathbf{D}}_{ii}={\sum }_j{\mathbf{B}}_{ij}$. And, the normalized Laplacian matrix is $\mathbf{L}=\mathbf{I}-{\mathbf{D}}^{-{\displaystyle \frac{1}{2}}}\mathbf{B}{\mathbf{D}}^{-{\displaystyle \frac{1}{2}}}$.

In [23], Huang et al. proposed a method for constructing the adjacency matrix $\mathbf{S}$:

$${\displaystyle \frac{d_{i,k+1}-d_{i,j}}{k{d}_{i,k+1}-{\sum }_{i=1}^k{d}_{i,j}}},$$

(3)

where $d_{i,j}$ represents the distance between the i-th sample and its j nearest neighbors, and k represents the number of nearest neighbors.

2.2. Matrix Factorization

Matrix factorization involves dividing data into two or more matrices, typically represented in mathematical terms as

$$\underset{\mathbf{W},\mathbf{H}}{min}{\parallel \mathbf{X}-\mathbf{W}\mathbf{H}\parallel }_F^2,$$

(4)

where $\mathbf{W}$ and $\mathbf{H}$ are two low-dimensional representations, $\mathbf{W}$ is the base matrix, and $\mathbf{H}$ can be regarded as a label matrix. Since matrix factorization can be viewed as solving the problem of ${min}_{x,y}{(1-xy)}^2$, it has infinitely many solutions. Therefore, it is necessary to impose some constraints on the model to limit the solutions. The most popular matrix factorization algorithm is non-negative matrix factorization, which imposes non-negativity constraints on $\mathbf{W}$ and $\mathbf{H}$.

Symmetric matrix factorization is a specialized form of matrix factorization that factorizes the adjacency matrix and can be expressed as follows:

$$\underset{\mathbf{H}}{min}{\parallel \mathbf{B}-{\mathbf{H}}^{\top }\mathbf{H}\parallel }_F^2,$$

(5)

where $\mathbf{B}$ is the full adjacency matrix, and $\mathbf{H}$ can be regarded as a label matrix. Due to the factorization of the adjacency matrix, the resulting low-dimensional representation of SMF contains local information. Clearly, if ${\mathbf{H}}^{*}$ is the optimal solution for SMF, then $-{\mathbf{H}}^{*}$ is an optimal solution for SMF. Therefore, it is common for SMF to impose constraints to limit the solutions. The most popular SMF is symmetric non-negative MF (SNMF), which imposes non-negativity constraints on $\mathbf{H}$.

3. Proposed Method

In this section, a new fast global local MF model is proposed. First, a novel method is introduced to construct the natural normalized full adjacency matrix using the bipartite graph. Thanks to this, the $n\times n$-order matrix, which is not explicitly represented, can effectively reduce the computational and spatial complexity. For local information, a bipartite graph-based semi-supervised symmetric matrix factorization framework is proposed, which is convex and unconstrained, thus enabling a fast solution while ensuring a global optimal solution. For global information, low-rank matrix factorization is utilized for rapid solution in one step, avoiding non-negative constraints that can handle non-negative data effectively.

3.1. Methodology

Due to the requirement of a full adjacency matrix in the symmetric matrix, it is not practical to use a bipartite graph directly. A concept of constructing the full adjacency matrix using anchor points is proposed, as shown in Figure 1. By connecting each sample with two anchor points, the adjacency relationship between samples is obtained. Therefore, the new full adjacency matrix is as follows:

$$\mathbf{B}=\mathbf{S}{\mathbf{\Lambda }}^{-1}{\mathbf{S}}^{\top },$$

(6)

where $\mathbf{\Lambda }$ is a diagonal matrix with ${\mathbf{\Lambda }}_{ii}={\sum }_j{\mathbf{S}}_{ji}$. According to [24], Equation (6) can be viewed as a non-weighted hypergraph; accordingly, Figure 1 can be considered as connecting samples through hyperedges, each of which connects not just two vertices but multiple vertices. In addition, due to the row sum of the adjacency matrix of a bipartite graph being one, the fully connected matrix generated by Equation (6) is self-normalized:

Theorem 1. 

For a matrix $\mathbf{S}$ with ${\sum }_j{\mathbf{S}}_{ij}=1$, the degree matrix of $\mathbf{S}{\mathbf{\Lambda }}^{-1}{\mathbf{S}}^{\top }$ is $\mathbf{I}$.

The proof of Theorem 1 is in Appendix A.

Therefore, the full adjacency matrix generated by Equation (6) can capture high-order relationships between data, does not require additional normalization, and significantly reduces computational complexity by storing and computing an $n\times m$-order matrix. Therefore, the bipartite graph-based symmetric matrix factorization can be expressed in the following form:

$$\parallel \mathbf{S}{\mathbf{\Lambda }}^{-1}{\mathbf{S}}^{\top }-{\mathbf{H}}^{\top }{\mathbf{H}\parallel }_F^2.$$

(7)

Traditional symmetric matrix factorization requires incorporating label information into the full adjacency matrix, which involves additional computation and necessitates the explicit appearance of $\mathbf{S}{\mathbf{\Lambda }}^{-1}{\mathbf{S}}^{\top }$. Therefore, this paper incorporates the label information into the low-dimensional representation matrix as one-hot constraints. Specifically, let $\mathbf{H}=\mathbf{ZA}$, where $\mathbf{Z}$ and $\mathbf{A}$ are auxiliary matrices. $\mathbf{Z}=\left[\begin{array}{cc}\mathbf{I}& \tilde{\mathbf{Z}}\end{array}\right]\in {\mathbb{R}}^{c\times (n-l+c)}$, $\tilde{\mathbf{Z}}\in {\mathbf{R}}^{c\times (n-l)}$ is the low-dimensional representation of an unlabeled sample, and $\mathbf{I}$ is the identity matrix. $\mathbf{A}=\left[\begin{array}{c}{\mathbf{A}}_1\\ {\mathbf{A}}_2\end{array}\right]\in {\mathbb{R}}^{(n-l+c)\times n}$ records the label information, where ${\mathbf{A}}_1\in {\mathbb{R}}^{c\times n}$ and ${\mathbf{A}}_2\in {\mathbb{R}}^{(n-l)\times n}$. If the j-th sample is labeled as the i-th class, ${\left({\mathbf{A}}_1\right)}_{ij}=1$; otherwise, ${\left({\mathbf{A}}_1\right)}_{ij}=0$. If the j-th column of ${\mathbf{A}}_1$ is a zero vector and the j-th sample is the i-th unlabeled one, ${\left({\mathbf{A}}_2\right)}_{ij}=1$; otherwise, ${\left({\mathbf{A}}_2\right)}_{ij}=0$. For example, there are eight samples with three classes, and ${\mathbf{x}}_3$ is labeled with class I, ${\mathbf{x}}_1$ and ${\mathbf{x}}_6$ are labeled with class II, and ${\mathbf{x}}_7$ is labeled with class III, then $\mathbf{A}$ becomes

$${\mathbf{A}}_1=\left[\begin{array}{cccccccc}0& 0& 1& 0& 0& 0& 0& 0\\ 1& 0& 0& 0& 0& 1& 0& 0\\ 0& 0& 0& 0& 0& 0& 1& 0\end{array}\right],$$

(8a)

$${\mathbf{A}}_2=\left[\begin{array}{cccccccc}0& 1& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 1& 0& 0& 0& 0\\ 0& 0& 0& 0& 1& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 1\end{array}\right].$$

(8b)

Imposing a one-hot constraint on labeled samples is justifiable, as these samples do not necessitate further distinction. The reformed SMF is

$$\underset{\tilde{\mathbf{Z}}}{min}{\parallel \mathbf{S}{\mathbf{\Lambda }}^{-1}{\mathbf{S}}^{\top }-{\mathbf{A}}^{\top }{\mathbf{Z}}^{\top }\mathbf{ZA}\parallel }_F^2.$$

(9)

This reduces computation and, more importantly, transforms the model from non-convex to convex. Therefore, there is no need to add constraints to $\tilde{\mathbf{Z}}$, and any local minimizer is a global minimizer [25]. An explanation of the model’s convexity is provided in Theorem 2.

Theorem 2. 

The objective function of Equation (9) is convex.

Equation (9) only considers the local information between samples; it is also crucial to consider the global information. MF can effectively capture global information. Thus, the following MF framework is proposed,

$$\begin{array}{cc}\hfill & \underset{\mathbf{W},\tilde{\mathbf{Z}}}{min}{\parallel \mathbf{X}-\mathbf{W}\mathbf{ZA}\parallel }_F^2+\lambda {\parallel \mathbf{S}{\mathbf{\Lambda }}^{-1}{\mathbf{S}}^{\top }-{\mathbf{A}}^{\top }{\mathbf{Z}}^{\top }\mathbf{ZA}\parallel }_F^2.\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.{\mathbf{W}}^{\top }\mathbf{W}=\mathbf{I}\hfill \end{array}$$

(10)

The proof of Theorem 2 is in Appendix B.

3.2. Optimization

Considering $\mathbf{A}$, there is

$${\mathbf{A}}_1{\mathbf{A}}_1^{\top }=\mathbf{Y},{\mathbf{A}}_1{\mathbf{A}}_2^{\top }=\mathbf{0},{\mathbf{A}}_2{\mathbf{A}}_2^{\top }=\mathbf{I},$$

(11)

where $\mathbf{Y}$ is a diagonal matrix with ${\mathbf{Y}}_{ii}=l_i$, and $l_i$ is the number of the i-th labeled class.

Firstly, expanding the MF term, there is

$$\begin{array}{cc}\hfill & {\parallel \mathbf{X}\parallel }_F^2-2Tr\left(\mathbf{WZA}{\mathbf{X}}^{\top }\right)+{\parallel \mathbf{WZA}\parallel }_F^2\hfill \\ \hfill =& {\parallel \mathbf{X}\parallel }_F^2-2Tr\left[\mathbf{W}\left({\mathbf{A}}_1+\tilde{\mathbf{Z}}{\mathbf{A}}_2\right){\mathbf{X}}^{\top }\right]+{\parallel \mathbf{W}{\mathbf{A}}_1\parallel }_F^2\hfill \\ \hfill & +Tr\left(\mathbf{W}\tilde{\mathbf{Z}}{\mathbf{A}}_2{\mathbf{A}}_1^{\top }\right)+{\parallel \mathbf{W}\tilde{\mathbf{Z}}{\mathbf{A}}_2\parallel }_F^2\hfill \\ \hfill =& {\parallel \mathbf{X}\parallel }_F^2-2〈\left({\mathbf{A}}_1+\tilde{\mathbf{Z}}{\mathbf{A}}_2\right),{\mathbf{W}}^{\top }\mathbf{X}〉\hfill \\ \hfill & +l+{∥\tilde{\mathbf{Z}}∥}_F^2,\hfill \end{array}$$

(12)

where $<·,·>$ is the inner product of the matrix, representing the sum of element-wise products. Discarding all terms unrelated to $\mathbf{W}$ and $\tilde{\mathbf{Z}}$, we obtain

$${∥\tilde{\mathbf{Z}}∥}_F^2-2〈\left({\mathbf{A}}_1+\tilde{\mathbf{Z}}{\mathbf{A}}_2\right),{\mathbf{W}}^{\top }\mathbf{X}〉.$$

(13)

Secondly, expanding the SMF term, there is

$$\begin{array}{cc}\hfill & \parallel \tilde{\mathbf{S}}{\tilde{\mathbf{S}}}^{\top }-{\mathbf{A}}^{\top }{\mathbf{Z}}^{\top }{\mathbf{Z}\mathbf{A}\parallel }_F^2\hfill \\ \hfill =& \parallel \tilde{\mathbf{S}}{\tilde{\mathbf{S}}}^{\top }{\parallel }_F^2-2∥\mathbf{Z}\mathbf{A}\tilde{\mathbf{S}}∥+{\parallel {\mathbf{A}}^{\top }{\mathbf{Z}}^{\top }\mathbf{Z}\mathbf{A}\parallel }_F^2\hfill \\ \hfill =& \parallel \tilde{\mathbf{S}}{\tilde{\mathbf{S}}}^{\top }{\parallel }_F^2-2{\parallel {\mathbf{A}}_1\tilde{\mathbf{S}}+\tilde{\mathbf{Z}}{\mathbf{A}}_2\tilde{\mathbf{S}}\parallel }_F^2\hfill \\ \hfill & +\parallel \mathbf{Y}+{\tilde{\mathbf{Z}}}^{\top }\tilde{\mathbf{Z}}{\parallel }_F^2.\hfill \end{array}$$

(14)

Discarding all terms unrelated to $\tilde{\mathbf{Z}}$, we obtain

$$\parallel \mathbf{Y}+{\tilde{\mathbf{Z}}}^{\top }\tilde{\mathbf{Z}}{\parallel }_F^2-2{\parallel {\mathbf{A}}_1\tilde{\mathbf{S}}+\tilde{\mathbf{Z}}{\mathbf{A}}_2\tilde{\mathbf{S}}\parallel }_F^2.$$

(15)

3.2.1. Fix $\mathbf{W}$, Optimize $\tilde{\mathbf{Z}}$

According to Equation (13) and Equation (15), the gradient of $\tilde{\mathbf{Z}}$ is

$$\begin{array}{cc}\hfill & 2\tilde{\mathbf{Z}}-2\mathbf{W}{\mathbf{X}}^{\top }{\mathbf{A}}_2^{\top }\hfill \\ \hfill +& 4\lambda \left[\left(\mathbf{Y}+\tilde{\mathbf{Z}}{\tilde{\mathbf{Z}}}^{\top }\right)\tilde{\mathbf{Z}}-\left({\mathbf{A}}_1\tilde{\mathbf{S}}+\tilde{\mathbf{Z}}{\mathbf{A}}_2\tilde{\mathbf{S}}\right){\tilde{\mathbf{S}}}^{\top }{\mathbf{A}}_2^{\top }\right].\hfill \end{array}$$

(16)

Due to the absence of constraints on $\tilde{\mathbf{Z}}$, it can be optimized using state-of-the-art unconstrained optimization algorithms. Here, CG_DESCENT 6.8 (https://people.clas.ufl.edu/hager/software/, accessed on 1 September 2024) [26,27,28] is employed. It is important to note that traditional matrix factorization methods employ multiplicative updating rules (MURs) for optimization. While these methods can ensure a decrease in the objective function, they do not guarantee convergence. Furthermore, as the objective function of $\tilde{\mathbf{Z}}$ is convex, it can be solved rapidly.

3.2.2. Fix $\tilde{\mathbf{Z}}$, Optimize $\mathbf{W}$

According to Equation (13), and using SVD as $\left({\mathbf{A}}_1+\tilde{\mathbf{Z}}{\mathbf{A}}_2\right){\mathbf{X}}^{\top }=\mathbf{GD}{\mathbf{V}}^{\top }$, we have

$$\begin{array}{cc}\hfill Tr\left(\mathbf{W}\left({\mathbf{A}}_1+\tilde{\mathbf{Z}}{\mathbf{A}}_2\right){\mathbf{X}}^{\top }\right)& =Tr\left(\mathbf{W}\mathbf{G}\mathbf{D}{\mathbf{V}}^{\top }\right)\hfill \\ \hfill & =Tr\left({\mathbf{V}}^{\top }\mathbf{W}\mathbf{G}\mathbf{D}\right)\hfill \\ \hfill & =\sum _{i=1}^k{\left({\mathbf{V}}^{\top }\mathbf{W}\mathbf{G}\right)}_{ii}{\mathbf{D}}_{ii}.\hfill \end{array}$$

(17)

Because of the orthogonality of ${\mathbf{V}}^{\top }\mathbf{WG}$, its diagonal entries are not smaller than $-1$. When all the diagonal elements of ${\mathbf{V}}^{\top }\mathbf{WG}$ equal $-1$, Equation (17) has been minimized. Therefore, we have ${\mathbf{V}}^{\top }\mathbf{WG}=-\mathbf{I}$, i.e., $\mathbf{W}=-\mathbf{V}{\mathbf{G}}^{\top }$.

The pseudocode for FGLMF is presented in Algorithm 1.

Algorithm 1 FGLMF

1: Input: Data matrix $\mathbf{X}\in {\mathbb{R}}^{d\times n}$, constraint matrix $\mathbf{A}$, parameters $\lambda$. 2: Output: Clustering indicator matrix (representation matrix) $\mathbf{Z}$ 3: Initialize the anchor point set $\mathbf{U}$ by k-means, and generate bipartite graph by Equation (3) 4: while Not convergent do 5:    Update $\tilde{\mathbf{Z}}$ by CG_DESCENT 6.8. 6:    Compute SVD of ${\mathbf{ZAX}}^{\top }={\mathbf{GDV}}^{\top }$. 7:    Update $\mathbf{W}=-\mathbf{V}{\mathbf{G}}^{\top }$. 8:    Let $t\leftarrow t+1$. 9: end while

3.3. Computational Complexity

The computational complexity of FGLMF is mainly divided into six parts.

• It takes $O\left(mnd\right)$ to compute the anchors.
• It takes $O\left(mnd\right)$ to compute the distance between samples and anchors.
• It takes $O\left(kmn\right)$ to compute the bipartite graph, where k is the number of nearest neighbors.
• It takes $O(cnd+c^{2}n+cmn)$ to compute the objective function of $\tilde{\mathbf{Z}}$.
• It takes $O(cnd+c^{2}n+cmn)$ to compute the gradient of $\tilde{\mathbf{Z}}$.
• It takes $O(cnd+c^{2}d)$ to compute the SVD of $\mathbf{ZA}{\mathbf{X}}^{\top }$.

Due to $c\ll min(m,n,d)$ and $k\ll min(m,n,d)$, the total computational complexity is $O\left(mnd\right)$. It should be noted that other bipartite graph-based methods, such as BGSSL and EAGR, need $O(ndm+n{m}^2+m^3)$.

4. Experiments

4.1. Compared Method

To demonstrate the efficiency and effectiveness of the proposed FGLMF, we choose the following methods for comparison:

• TSVD: Truncated singular value decomposition [29], it can give the best approximation with a preset rank.
• SemiGNMF: Semi-supervised graph regularized non-negative matrix factorization [7].
• CSNMF: Correntropy-based semi-supervised NMF [15].
• CSCF: Correntropy-based semi-supervised concept factorization [16].
• CLMF: Correntropy-based low-rank matrix factorization [17].
• PCPSNMF: pairwise constraint propagation-induced SNMF [18].
• S⁴NMF: Self-supervised semi-supervised non-negative matrix factorization [30].
• HSSNMF: Hypergraph-based semi-supervised symmetric non-negative matrix factorization [19].
• EAGR: Efficient anchor graph regularization [21].
• BGSSL: Bipartite graph semi-supervised learning [22].

Specifically, TSVD is an unsupervised MF method used as the baseline, and the rest are semi-supervised methods. Methods (1)–(5) are matrix factorization methods that factorize the samples, (6)–(8) are matrix factorization methods that factorize the full adjacency matrix, and (9)–(10) are bipartite graph-based methods.

4.2. Datasets

To evaluate the performance of the proposed FGLMF, six real-world datasets are used, and the relevant statistics are shown in

Table 1. Datasets.

Dataset	No. of Instances (n)	No. of Features (d)	No. of Classes (C)
COIL20	1440	1024	20
YaleB	2414	1024	38
COIL100	7200	1024	100
USPS	9298	256	10
MNIST	70000	784	10
Letters	145600	784	26

. COIL20 ¹ (http://www.cad.zju.edu.cn/home/dengcai/Data/MLData.html, accessed on 1 September 2024) and COIL100 ¹ are object datasets; YaleB ² (http://www.cad.zju.edu.cn/home/dengcai/Data/FaceData.html, accessed on 1 September 2024) is a face dataset; USPS ¹ and MNIST ³ (https://www.nist.gov/itl/products-and-services/emnist-dataset#:~:text=The%20EMNIST%20dataset%20is%20a%20set, accessed on 1 September 2024) are handwritten digit datasets; and Letters ³ is a handwritten English letters dataset.

4.3. Experiment Settings

To ensure the fairness of the experiments, all experiments in this paper were performed on a personal computer with an Intel i7-6800k CPU and 16 GB RAM. The parameters of all the compared methods were set according to the papers in which they are described. The number of nearest neighbors was set to five for all graph-based methods. For FGLMF, we fixed the parameter $\lambda$ at 100. For the COIL20 and YaleB datasets, the number of anchor points was set to 500 and 1500, respectively, and for the remaining datasets, the number of anchor points was set to 2000. According to [22], we set the dimensions of the low-dimensional representation of BGSSL to the number of ground truths plus 1, and for the remaining methods, the low-dimensional representation was set to the number of ground truth classes of each dataset. For TSVD, GNMF, CSNMF, and CSCF, we obtained the results using k-means on the low-dimensional representation, while for the remaining methods, we used ${argmax}_i{\mathbf{H}}_{ij}$ to obtain the results. The stop rule of CG_DESCENT was set to ${\parallel ▽f\parallel }_{\infty }<{10}^{-6}$. We randomly selected 30% of the samples as labeled samples. Four commonly used metrics were adopted, including accuracy (ACC), normalized mutual information (NMI), adjusted Rand index (ARI), and F-score. Specifically, accuracy calculates the percentage of correctly assigned labels by comparing real label information with learned label information. NMI measures the amount of shared information between two statistical distributions. ARI assesses the adjusted Rand index to evaluate the agreement between obtained labels and true labels, while the F-score determines clustering accuracy through a weighted average of precision and recall. The values for these evaluation metrics range from 0 to 1, where higher values indicate superior clustering quality. Since these four metrics emphasize different aspects within clustering applications, we present experimental results across these diverse criteria to conduct a comprehensive evaluation of clustering performance for various methods. We conducted ten random trials for all methods and reported the average results for each dataset.

4.4. Experiment Performance

Table 2. Average accuracy (%) ± standard deviation (%) of using different approaches on six datasets (“OM” means “out-of-memory error”). The bold text signifies the best, while the underlined text indicates the second best.

	TSVD	SemiGNMF	CSNMF	CSCF	CLMF	PCPSNMF	S4NMF	HSSNMF	EAGR	BGSSL	FGLMF
COIL20	60.67 ± 03.33	89.23 ± 02.19	86.15 ± 03.46	86.02 ± 03.23	91.44 ± 01.12	87.27 ± 03.22	91.41 ± 02.40	91.42 ± 02.75	95.42 ± 00.97	95.63 ± 00.96	97.79 ± 00.84
YaleB	08.54 ± 00.26	55.58 ± 02.68	58.42 ± 04.02	45.80 ± 02.19	77.46 ± 00.96	66.11 ± 02.77	69.97 ± 02.89	72.82 ± 02.01	58.06 ± 01.55	54.78 ± 01.56	72.24 ± 01.39
COIL100	48.63 ± 01.39	82.62 ± 00.74	72.92 ± 02.25	73.99 ± 02.49	75.21 ± 00.50	83.00 ± 01.22	65.01 ± 01.80	88.75 ± 01.02	86.98 ± 00.33	85.71 ± 00.91	90.44 ± 00.42
USPS	64.04 ± 01.87	-	-	93.92 ± 03.72	90.91 ± 00.26	60.67 ± 07.27	83.22 ± 05.11	83.34 ± 03.70	96.34 ± 00.14	96.03 ± 00.15	96.45 ± 00.18
MNIST	54.21 ± 02.27	OM	OM	OM	OM	OM	OM	OM	96.49 ± 00.03	95.38 ± 00.10	96.62 ± 00.05
Letters	37.31 ± 00.77	OM	OM	OM	OM	OM	OM	OM	82.32 ± 00.16	75.63 ± 00.16	82.50 ± 00.15

,

Table 3. Average NMI (%) ± standard deviation (%) of using different approaches on six datasets (“OM” means “out-of-memory error”). The bold text signifies the best, while the underlined text indicates the second best.

	TSVD	SemiGNMF	CSNMF	CSCF	CLMF	PCPSNMF	S4NMF	HSSNMF	EAGR	BGSSL	FGLMF
COIL20	74.75 ± 01.10	95.27 ± 00.41	93.71 ± 01.39	93.05 ± 01.56	89.82 ± 00.92	92.29 ± 01.44	94.88 ± 01.13	95.63 ± 01.08	96.13 ± 00.81	95.92 ± 00.78	97.54 ± 00.96
YaleB	12.89 ± 00.46	71.54 ± 00.81	70.90 ± 01.62	62.81 ± 01.46	74.58 ± 01.18	68.16 ± 01.94	73.39 ± 01.91	73.00 ± 01.22	60.91 ± 00.89	59.20 ± 01.03	71.38 ± 01.10
COIL100	75.52 ± 00.34	93.56 ± 00.33	88.98 ± 00.87	90.03 ± 00.94	80.46 ± 00.40	90.87 ± 00.43	79.07 ± 01.17	94.17 ± 00.35	93.20 ± 00.20	92.64 ± 00.31	94.41 ± 00.20
USPS	59.10 ± 00.75	-	-	91.73 ± 02.05	80.74 ± 00.45	74.91 ± 03.33	86.14 ± 04.19	85.70 ± 00.98	91.18 ± 00.31	90.59 ± 00.24	91.34 ± 00.31
MNIST	50.91 ± 01.12	OM	OM	OM	OM	OM	OM	OM	91.14 ± 00.07	89.41 ± 00.13	91.40 ± 00.10
Letters	39.48 ± 00.49	OM	OM	OM	OM	OM	OM	OM	75.36 ± 00.17	70.23 ± 00.09	75.3 ± 00.16

,

Table 4. Average ARI (%) ± standard deviation (%) of using different approaches on six datasets (“OM” means “out-of-memory error”). The bold text signifies the best, while the underlined text indicates the second best.

	TSVD	SemiGNMF	CSNMF	CSCF	CLMF	PCPSNMF	S4NMF	HSSNMF	EAGR	BGSSL	FGLMF
COIL20	53.38 ± 02.67	85.40 ± 02.72	86.54 ± 02.75	82.08 ± 03.99	84.16 ± 01.69	83.09 ± 03.27	88.46 ± 02.89	88.92 ± 02.87	91.64 ± 01.37	92.50 ± 01.45	95.60 ± 01.66
YaleB	00.29 ± 00.12	38.76 ± 02.51	47.09 ± 02.97	35.21 ± 02.53	58.79 ± 01.69	47.34 ± 03.02	55.76 ± 03.61	55.28 ± 02.19	37.06 ± 01.77	34.56 ± 01.81	52.96 ± 01.88
COIL100	43.31 ± 00.96	76.97 ± 00.89	63.84 ± 04.11	68.28 ± 04.10	59.45 ± 00.85	76.20 ± 01.31	53.23 ± 02.06	83.78 ± 01.23	81.09 ± 00.61	79.89 ± 01.02	85.75 ± 00.56
USPS	50.64 ± 01.49	-	-	92.35 ± 03.12	81.78 ± 00.53	58.55 ± 07.33	83.63 ± 04.37	80.72 ± 02.27	93.02 ± 00.25	92.41 ± 00.24	93.24 ± 00.30
MNIST	38.07 ± 01.51	OM	OM	OM	OM	OM	OM	OM	92.48 ± 00.06	90.19 ± 00.21	92.75 ± 00.10
Letters	21.66 ± 00.44	OM	OM	OM	OM	OM	OM	OM	68.10 ± 00.26	59.32 ± 00.15	68.08 ± 00.24

and

Table 5. Average F-score (%) ± standard deviation (%) of using different approaches on six datasets (“OM” means “out-of-memory error”). The bold text signifies the best, while the underlined text indicates the second best.

	TSVD	SemiGNMF	CSNMF	CSCF	CLMF	PCPSNMF	S4NMF	HSSNMF	EAGR	BGSSL	FGLMF
COIL20	58.30 ± 03.61	88.04 ± 02.24	83.73 ± 04.51	85.26 ± 03.49	91.65 ± 01.09	86.43 ± 03.50	89.86 ± 03.31	89.58 ± 02.93	95.46 ± 01.00	95.53 ± 00.96	97.80 ± 00.81
YaleB	08.62 ± 00.42	55.63 ± 03.10	57.44 ± 04.64	46.03 ± 02.21	78.84 ± 00.75	66.10 ± 02.74	68.04 ± 03.01	71.84 ± 02.19	58.39 ± 01.41	54.39 ± 01.37	72.34 ± 01.37
COIL100	46.71 ± 01.57	81.35 ± 00.85	71.94 ± 02.24	72.97 ± 02.43	76.76 ± 00.51	82.44 ± 01.28	63.22 ± 01.86	88.14 ± 00.97	86.80 ± 00.29	85.38 ± 00.92	90.42 ± 00.43
USPS	61.56 ± 02.45	-	-	91.76 ± 05.61	90.66 ± 00.27	56.10 ± 08.06	78.75 ± 05.91	78.14 ± 06.16	95.98 ± 00.17	95.64 ± 00.18	96.06 ± 00.21
MNIST	53.76 ± 02.38	OM	OM	OM	OM	OM	OM	OM	96.46 ± 00.03	95.33 ± 00.11	96.59 ± 00.05
Letters	37.29 ± 00.93	OM	OM	OM	OM	OM	OM	OM	82.25 ± 00.17	74.73 ± 00.20	82.73 ± 00.15

shows the clustering results of the various methods with the best results highlighted. Due to negative values in the USPS data, both GNMF and CSNMF are unsuitable for this dataset. It can be observed that the proposed FGLMF can achieve the best effect in most cases. Additionally, on the YaleB data, CLMF demonstrates the best performance and, compared with other bipartite graph-based methods, the performance of FGLMF is significantly enhanced. It should be noted that similar to FGLMF, CLMF is a method that considers global and local information. This indicates that considering global and local information on facial data can yield better results. It should be noted that CLMF can only handle a small amount of data due to the need for a large number of computational resources. FGLMF performs better than other MF methods in most cases, although it is an anchor-based method. This might be because of the non-negative constraints of other MF methods, which use MUR for optimization, meaning they can only guarantee a decrease in the objective function but not convergence. Since a convex, unconstrained SMF is proposed in this paper, convergence to the global optimal solution can be guaranteed for any algorithms that guarantee local optimal solutions. Furthermore, the most advanced unconstrained optimization algorithms are often much faster than MUR.

We also compared the average performance of SemiGNMF, PCPSNMF, HSSNMF, EAGR, BGSSL, and FGLMF under different labeling ratios in Figure 2. As the labeled ratio increases, the clustering performance improves in most cases. Under the USPS dataset, for the remaining MF methods, there exists a situation where the accuracy varies to different degrees, and the performance decreases as the labeling ratio increases. On the one hand, it is observed that the improvement of all methods is not significant, indicating that the labeling information has reached a bottleneck. On the other hand, other MF methods cannot guarantee convergence, and therefore, more iterations may be required, but this will increase the already high computational cost. In most cases, the proposed FGLMF shows good performance, demonstrating its effectiveness.

4.5. Time Consumption

The average time taken for each method is presented in

Table 6. Average time costs (seconds) of different methods.

	Other Learning									Anchor-Based Learning
	TSVD	SemiGNMF	CSNMF	CSCF	CLMF	PCPSNMF	S4NMF	HSSNMF		k-means	EAGR	BGSSL	FGLMF
COIL20	0.34	1.15	38.20	79.87	24.36	4.31	64.44	6.48		0.55	+0.12	+0.06	+0.05
YaleB	0.69	2.67	105.26	273.92	53.71	13.23	234.31	19.82		2.13	+0.23	+0.17	+0.15
COIL100	3.75	12.52	1509.16	4287.62	421.90	150.15	2787.81	176.65		9.84	+0.89	+0.85	+0.74
USPS	0.23	-	-	5522.62	328.54	225.21	3224.49	321.27		4.11	+0.62	+0.78	+0.34
MNIST	3.69	OM	OM	OM	OM	OM	OM	OM		61.5	+4.96	+5.42	+2.91
Letters	8.18	OM	OM	OM	OM	OM	OM	OM		129.77	+14.49	+13.43	+7.04

. The time associated with anchor-based methods can be categorized into two components: the first component involves the duration spent on the k-means algorithm, while the second depends on the models. It is noteworthy that numerous efficient k-means algorithms are currently available, such as [31] and others. However, since this article does not primarily focus on the exploration of k-means algorithms, we opted for the built-in k-means algorithm provided by the scikit-learn [32] library in Python to ensure fairness. FGLMF is the fastest among the anchor-based methods, especially for large-scale datasets, where this improvement is particularly evident. It should be noted that both FGLMF and TSVD require SVD. However, FGLMF performs SVD on a $c\times n$-order matrix, while TSVD needs to perform SVD on a $d\times n$-order matrix (c is much smaller than d), which explains why FGLMF is faster than TSVD in the solution process. Unfortunately, FGLMF requires k-means to find anchors, which makes its total elapsed time slower than TSVD. Nevertheless, FGLMF is still the second fastest among all the methods.

4.6. Sensitivity to $\lambda$ Parameter

FGLMF has a parameter $\lambda$, and the selection of this parameter is examined in this section. Figure 3 presents the average clustering accuracy of FGLMF under different lambda values. It can be observed that either an excessively high or an excessively low $\lambda$ leads to the degradation of the performance of FGLMF. A $\lambda$ that is too low neglects the local structure and only considers global information, reducing clustering performance. An overly high $\lambda$ ignores the global information, reducing the clustering performance. When $\lambda$ takes values of 10 and 100, it exhibits the best performance across all datasets.

4.7. Effect of Anchors

To verify the sensitivity of the proposed method to anchor points, we conducted tests on each dataset using only different anchor points, presented in Figure 4. With the increase in anchor points, the clustering performance improved. Among them, the YaleB dataset is the most sensitive to the number of anchor points. In contrast, the USPS and MNIST datasets are not sensitive to the number of anchor points. This might be attributed to the complexity of the data. Both USPS and MNIST are handwritten digit datasets, and a small number of anchor points can represent the entire dataset. However, YaleB is a dataset of human faces with different expressions and shadows, and a relatively large number of anchor points is necessary to represent the entire dataset. In addition, the solution time tends to increase as the number of anchor points increases. It should be noted that this increase is not significant. This is because FGLMF has a linear relationship with the number of anchor points, rather than $m^3$ as in other methods. This indicates that FGLMF is suitable for situations with a large number of anchor points. As mentioned above, for more complex datasets, a smaller number of anchor points may only represent part of the dataset. Sometimes, it is necessary to increase the number of anchor points to improve performance.

4.8. Bases

To verify the degree of global information extraction by the proposed method, we present the first ten bases extracted by FGLMF for each dataset in Figure 5. It can be observed that each base represents its respective category relatively clearly. It should be noted that each category of the Letters dataset contains both uppercase and lowercase letters. FGLMF does an excellent job of extracting the facial features of each person in the YaleB dataset, which also explains why the proposed method performs much better than other bipartite graph-based methods on this dataset.

4.9. Generated Graph

To validate the situation of the graph generated by Equation (6), we compared the graph generated by Equation (6) with the one generated using Gaussian kernel on the COIL20 dataset, as shown in Figure 6. Figure 6a, which displays the adjacency matrix of the bipartite graph generated using 500 anchors and five nearest neighbors. It can be observed that points are scattered randomly in the graph, and no special structure of the adjacency matrix is apparent. Figure 6b presents the full adjacency matrix generated from the adjacency matrix in Figure 6a, showing a clear block structure, i.e., connections between the same classes. Figure 6c illustrates the graph generated using Gaussian kernel, which exhibits a similar structure to that in Figure 6b, validating our approach’s feasibility. Furthermore, Figure 6c emphasizes connections with the nearest samples, while Figure 6b captures high-order relationships between the data, i.e., connections within classes.

4.10. Convergence Study

To verify the convergence of FGLMF, the convergence curves are presented in Figure 7. The figures show the convergence curves of five iterations. The colored broken lines represent the inner loop of CG_DESCENT, and the blank spaces between the dots of different colors represent the process of solving $\mathbf{W}$ using SVD. If there is only one dot of a specific color, it indicates that the solution at this time has reached the target accuracy, and CG_DESCENT no longer works. At this point, neither $\mathbf{W}$ nor $\tilde{\mathbf{Z}}$ will continue to be optimized. It should be noted that the algorithm should be terminated when the number of iterations of CG_DESCENT is 0. We provide the results of five iterations here to make the pictures easier to understand. A complete iteration can be considered as starting from the dot of one color and ending at the dot of another color.

The figure shows that FGLMF reached the best result after three iterations on USPS, while only two iterations were required to reach the best result on other datasets. Additionally, in an internal loop, only a few iterations are needed to reach the optimal solution. This is attributed to the convexity of the proposed SMF, which eliminates the non-negative constraints and enables the algorithm to solve the problem rapidly. Furthermore, such a small number of iterations further explains why FGLMF requires so little time. Since the first iteration was close to the optimal value, CG_DESCENT requires only a few iterations after the first iteration to reach the optimal value.

To better demonstrate the convergence of FGLMF, we have also magnified and presented the situations before and after the first iteration in the subgraphs. It can be observed from the figure that the objective function continues to decline. After the update of $\mathbf{W}$, the objective function undergoes a significant drop. Additionally, based on the decline in the objective function after updating $\mathbf{W}$, it can also be seen that $\tilde{\mathbf{Z}}$ is already very close to the optimal solution.

5. Conclusions

Traditional matrix decomposition methods face challenges related to high computational complexity and substantial memory requirements. This paper proposes a rapid global–local matrix decomposition model that employs LMF (local matrix factorization) to account for global information while utilizing SMF (sparse matrix factorization) to capture local information. Compared to conventional approaches, the computational complexity is reduced from $O\left(n^{2}d\right)$ to $O\left(ndm\right)$, significantly alleviating both the computational burden and memory demands. Furthermore, we introduce a convex, unconstrained symmetric matrix decomposition method for bipartite graphs and provide an analysis and proof of its convexity. Thanks to the proposed symmetric matrix decomposition technique, our method demonstrates considerable advantages in performance.

Limitations and future work: Despite the reduction in computational and spatial complexity, the proposed method struggles to handle extremely large-scale datasets due to its requirement to store an $n\times m$ matrix. When n becomes excessively large, there may be instances of insufficient memory. In future work, it would be beneficial to explore optimization techniques such as mini-batch stochastic gradient descent to alleviate memory constraints on the model.