Semi-Supervised Learning with Close-Form Label Propagation Using a Bipartite Graph

Abstract

In this paper, we introduce an efficient and effective algorithm for Graph-based Semi-Supervised Learning (GSSL). Unlike other GSSL methods, our proposed algorithm achieves efficiency by constructing a bipartite graph, which connects a small number of representative points to a large volume of raw data by capturing their underlying manifold structures. This bipartite graph, with a sparse and anti-diagonal affinity matrix which is symmetrical, serves as a low-rank approximation of the original graph. Consequently, our algorithm accelerates both the graph construction and label propagation steps. In particular, on the one hand, our algorithm computes the label propagation in closed-form, reducing its computational complexity from cubic to approximately linear with respect to the number of data points; on the other hand, our algorithm calculates the soft label matrix for unlabeled data using a closed-form solution, thereby gaining additional acceleration. Comprehensive experiments performed on six real-world datasets demonstrate the efficiency and effectiveness of our algorithm in comparison to five state-of-the-art algorithms.

1. Introduction

In recent decades, the volume of user-generated content has experienced explosive growth. As an illustration, millions of new photos are added to Flickr daily, in addition to its already substantial collection of several billion images [1]. This massive influx of multimedia data presents novel challenges and vast opportunities for the fields of machine learning and data mining. Given the infeasibility of manually annotating such extensive datasets, Semi-supervised Learning (SSL) emerges as a pivotal approach, utilizing abundant unlabeled data in conjunction with a relatively small amount of labeled data. This paper focus on one principal category of SSL techniques: Graph-based SSL (GSSL) [2,3].

GSSL is an appealing approach due to its superior performance, despite its cubic computational complexity. In GSSL, we assume that similar labels are shared by nearby data points. Based on a small number of given labels, GSSL constructs an affinity graph and utilizes it to perform close-form label propagation, resulting in high accuracy. Representative GSSL methods include Harmonic Energy Minimization (HEM) [2], Local and Global Consistency (LGC) [4], and General Graph Semi-Supervised Learning (GGSSL) [5]. However, their computational complexity of label propagation is cubic (i.e., $\mathcal{O}\left(n^3\right)$) for n data points, as it requires computing the inverse of the graph Laplacian. In addition, $O\left(n^2\right)$ operations are needed to construct their dense affinity graphs, which are impractical for large-scale datasets. Therefore, fast GSSL algorithms are highly desirable.

Several studies have been proposed to seek efficient GSSL algorithm. In [6], label propagation is first carried out on a randomly selected subset, and then on a truncated graph Laplacian which is constructed by connecting the selected subset with remaining data points. In [7], the Nyström method is used to approximate the eigenfunction with that of a randomly selected subset. However, both of these methods experience performance degradation, as they do not take into account the topology structure of the dataset during approximation. In [8], a fast SSL algorithm is proposed by sparsifying manifold regularization. In [9], a method combining the parametric mixture model with harmonic property is proposed to handle raw data and label propagation. Based on spectral graph theory, in [10], an efficient method is proposed to approximate the eigenvectors of the normalized graph Laplacian. However, its assumptions, such as dimension independence and 1-D uniform distributions, are often difficult to satisfy in many real-world datasets. In [11,12], Anchor Graph Regularization (AGR) is introduced, which considers manifold regularization by employing a low-rank approximation of the original graph. Nevertheless, it necessitates a linear transformation between raw data and anchor points. Evidently, the efficiency of these algorithms comes at the expense of accuracy. In [13], Optimal Bipartite Graph Semi-Supervised Learning (OBGSSL) is introduced, which learns a new bipartite graph iteratively instead of fixing the input data graph. In [14], semi-supervised learning via Bipartite graph Construction and Adaptive Neighbors (BCAN) considers both bipartite graph construction and label propagation simultaneously. Since the bipartite graph [15] must be updated iteratively until convergence in both OBGSSL and BCAN, their accuracy and speed are either inferior or marginally superior to AGR, as evidenced by their own experimental results in [13,14]. These inconsistent performances compared to AGR motivate us to develop our algorithm, which ultimately demonstrates a consistent outperformance over AGR, as confirmed by the experimental results presented at the end of this paper.

In this paper, we introduce an efficient and effective algorithm termed Semi-Supervised Learning with Close-Form label propagation using Bipartite Graph (SSLCFBG). Here, closed-form label propagation implies that the soft label matrix $F_u$ for unlabeled data can be computed analytically through an equation, rather than employing an iterative algorithm. This algorithm is based on a bipartite graph, which uses a small set of representative points to represent a large set of raw data. Label propagation is subsequently performed on the graph Laplacian of this structure. Owing to the unique structure of its affinity matrix, the proposed SSLCFBG exhibits efficiency in both graph construction and label propagation steps. Firstly, constructing its sparse graph incurs a cost of $O\left(nm\right)$, as opposed to the $O\left(n^2\right)$ cost associated with traditional GSSL algorithms, where m represents the number of representative points and satisfies $m\ll n$. Secondly, the block structure of the graph Laplacian facilitates accelerated close-form label propagation through an efficient matrix inversion, with a computational complexity that scales nearly linearly with the number of data points n. The advantages of the proposed SSLCFBG in terms of speed and accuracy are validated by experiments conducted on six real-world datasets, outperforming five state-of-the-art and baseline algorithms.

Next, we will first elucidate our notations and then briefly delve into the graph construction of GSSL and the label propagation process in related algorithms.

1.1. Notations

Let $\mathcal{X}=\{x_1,\cdots ,x_l,x_{l+1},\cdots ,x_n\}$ be the dataset, where $x_i\in {\mathbb{R}}^d$ and $i=1,\cdots ,n$. Label set is $\mathcal{L}=\{1,\cdots ,c\}$. Without loss of generality, we assume the first l data points $x_i$$(i\le l)$ are labeled as $y_i\in \mathcal{L}$ and the remaining data points $x_u$$(l+1\le u\le n)$ are unlabeled. Similar to other SSL problems, we assume $l\ll u$. $\mathcal{Q}=\{q_1,\cdots ,q_m\}\subset {\mathbb{R}}^d$ denotes the set of representative points (or anchor points for AGR) with $m\ll n$. A soft label matrix is defined as $F\in \mathcal{F}$, where $\mathcal{F}$ denotes the set of $(n+m)\times c$ matrices with nonnegative entries. Label indicator matrix is defined as $Y\in {\mathbb{R}}^{(n+m)\times c}$ with $Y_{ij}=1$ if $x_i$ is labeled as $y_i=j$, and $Y_{ij}=0$ otherwise. For convenience, we express F and Y in sub-matrices: $F={[F_l^T,F_u^T,F_q^T]}^T$, $Y={[Y_l^T,Y_u^T,Y_q^T]}^T$, where $F_l$, $F_u$ and $F_q$ correspond to the rows for labeled, unlabeled and the representative points in F, respectively. Similarly, we define $Y_l$, $Y_u$ and $Y_q$ for Y.

1.2. Graph Construction in GSSL

The first step of GSSL is to construct an undirected graph $G(\mathcal{V},\mathcal{E},W)$, where $\mathcal{V}=\mathcal{X}\cup \mathcal{Q}$ is a set including raw data and representative points. A set $\mathcal{E}\subseteq \mathcal{V}\times \mathcal{V}$ consists of edges between vertices. Associated with each edge $e_{ij}\in \mathcal{E}$, its weight is defined as $w_{ij}\ge 0$, which forms an affinity matrix $W\in {\mathbb{R}}^{(n+m)\times (n+m)}$, where $w_{ij}=0$ means no direct connection between $v_i$ and $v_j$.

There are different ways to construct the affinity matrix. For example, we use k-nearest neighbor (kNN) to connects any pair of points if one is among the k-th nearest neighbors of the other [16]. The computational complexity of kNN in full graph construction is $\mathcal{O}\left({(n+m)}^2\right)$, making it unsuitable for real-world datasets with large n. In this paper, we propose SSLCFBG with a fast graph construction. Meanwhile, graph Laplician is defined as $L=D-W$, where $L\in {\mathbb{R}}^{l\times l}$ and D is a diagonal matrix with diagonal elements as $d_{ii}={\sum }_j{w}_{ij}$.

1.3. Related Algorithms

After constructing the graph in the above subsection, we are going to discuss label propagation of three closely related algorithms: HEM, LGC and AGR.

Considering the given labels as additional constrains, HEM in [2] aims to minimize

$$\begin{array}{c}\hfill J_{\mathrm{HEM}}\left(F\right):=\mathrm{tr}\left(F^{T}LF\right),\mathrm{s}.\mathrm{t}.F_i=Y_i,\end{array}$$

(1)

for $i=1,\cdots ,l$, where $\mathrm{tr}(·)$ is matrix trace. Here, label consistence of given label is guaranteed by the explicit constraints of $F_i=Y_i$. The closed-form label propagation of HEM is

$$\begin{array}{c}\hfill F_u^{*}={(D_{uu}-W_{uu})}^{-1}{W}_{ul}{Y}_l,\end{array}$$

(2)

where $D=diag(D_{ll},D_{uu})$ is the degree matrix in graph Laplician, while $D_{ll}$ and $D_{uu}$ correspond to labeled and unlabeled data. $W_{uu}\in {\mathbb{R}}^{u\times u}$ is the weight for unlabeled data, and $W_{lu}\in {\mathbb{R}}^{l\times u}$ is the weight between labeled and unlabeled data. For every data $x_k$, its label $y_k$ can be estimated through

$$\begin{array}{c}\hfill y_k=\underset{j\le c}{arg\; max}{F}_{kj},\end{array}$$

(3)

where $k=l+1,\cdots ,n$.

Since HEM is unable to handle noised data, LGC in [4] generalizes HEM with a finite weight of $\gamma >0$, which has a following loss function:

$$\begin{array}{c}\hfill J_{\mathrm{LGC}}\left(F\right):=\mathrm{tr}\left(F^{T}LF\right)+\gamma \sum _{i=1}^n{\parallel F_i-Y_i\parallel }_F^2,\end{array}$$

(4)

where $\mathrm{tr}(·)$ is matrix trace, and ${\parallel ·\parallel }_F$ represents the Frobenius norm. The closed-form solution for LGC is

$$\begin{array}{c}\hfill F^{*}={\left(D-{\displaystyle \frac{1}{1+\gamma }}W\right)}^{-1}Y.\end{array}$$

(5)

To reduce the high computational complexities of HEM and LGC, AGR in [11,12] minimizes

$$\begin{array}{c}\hfill \underset{F}{min}{\displaystyle \frac{1}{2}}{\parallel Z_{lq}{F}_q-Y_l\parallel }_F^2+{\displaystyle \frac{\gamma }{2}}\mathrm{tr}\left(F_q^T{Z}_q^{T}L{Z}_q{F}_q\right),\end{array}$$

(6)

where $Z_{lq}\in {\mathbb{R}}^{l\times q}$ and $Z_q\in {\mathbb{R}}^{n\times q}$ are matrices of sample-adaptive weights. AGR is much faster than HEM and LGC, because it employs a small number of anchor points. However, it requires a linear assumption of

$$\begin{array}{c}\hfill F_{l+u}=Z_q{F}_q,\end{array}$$

(7)

where $F_{l+u}\in {\mathbb{R}}^{n\times c}$. This assumption implies that the labels of the raw data can be linearly represented by the labels of a limited number of anchor points. Given that the ratio of $n/m$ is typically very high, it becomes challenging to accurately fit all the raw data using a linear projection.

In the subsequent section, we introduce SSLCFBG, which aims to preserve the precision of HEM and LGC while maintaining the efficiency of AGR.

2. Materials and Methods

This section presents an efficient and effective algorithm, SSLCFBG, designed to address a general GSSL problem using a bipartite graph. On the one hand, it converges significantly faster than HEM and LGC. On the other hand, the proposed algorithm’s label propagation is faster and more precise, as it does not rely on the linear assumption inherent in AGR.

2.1. General Graph-Based Model

The GSSL problem can be formulated in different ways, such as those in [2,4,5]. In this paper, we consider a general model (loss function) as follows:

$$\begin{array}{c}\hfill J\left(F\right)=\sum _{i,j=1}^{n+m}{w}_{ij}\parallel F_i-F_j{\parallel }_F^2+\sum _{i=1}^{n+m}{\lambda }_i{\parallel F_i-Y_i\parallel }_F^2,\end{array}$$

(8)

where parameter ${\lambda }_i>0$ trades off label smoothness and consistence. Particularly, the loss functions of HEM and LGC can be interpreted as special cases of Equation (8), if

• We set $m=0$, ${\lambda }_i=+\infty$ for $i=1,\cdots ,l$, and ${\lambda }_i=0$ otherwise. Then, Equation (8) is the loss function of HEM.
• We let ${\lambda }_1=\cdots ={\lambda }_n>0$, and $m=0$. Then, Equation (8) becomes the loss function of LGC.

For convenience, Equation (8) can be rewritten in matrix-form,

$$\begin{array}{c}\hfill J\left(F\right)=\mathrm{tr}\left(F^{T}LF\right)+\mathrm{tr}\left({(F-Y)}^T\Lambda (F-Y)\right),\end{array}$$

(9)

where $\Lambda =diag({\lambda }_1,\cdots ,{\lambda }_{n+m})$ is a diagonal matrix.

2.2. Naive Label Propagation

Due to its convexity with respect to F, the minimum of Equation (9) attains when ${\displaystyle \frac{\partial J\left(F\right)}{\partial F}}=2LF+2\Lambda (F-Y)=0$, which leads to the closed-form solution of

$$\begin{array}{c}\hfill F^{*}={(L+\Lambda )}^{-1}\Lambda Y.\end{array}$$

(10)

Then, we apply Equation (3) to assign labels.

However, when it comes to computational complexity, efficiently minimizing Equation (9) presents a significant challenge. The crux lies in Equation (10), where computing the inverse of an $(n+m)\times (n+m)$ matrix requires $\mathcal{O}\left({(n+m)}^3\right)$ operations. Likewise, the closed-form solutions for HEM and LGC, as presented in Equation (2) and Equation (5), respectively, also entail cubic computational complexities.

Consequently, it is highly desirable to reduce the computational cost associated with label propagation.

2.3. Large-Scale Graph Construction

In this subsection, we employ a bipartite graph to expedite the graph construction phase. An illustrative example of constructing such a bipartite graph is provided in Figure 1. Figure 1a shows the original fully connected graph, comprising 8 data points and 28 edges. In Figure 1b, we introduce three representative points to represent the 8 data points; for instance, ‘A’is designated to represent data points ‘1’and ‘2’. This bipartite graph requires only 8 edges. Figure 1c displays just the 3 representative points connected by 3 edges, as opposed to the 8 data points and 28 edges in Figure 1a.

The affinity matrix of GSSL problem based on bipartite graph is sparse and block anti-diagonal, i.e.,

$$\begin{array}{c}\hfill W=\left[\begin{array}{cc}0& Z\\ Z^T& 0\end{array}\right]\in {\mathbb{R}}^{(n+m)\times (n+m)},\end{array}$$

(11)

where $Z={[W_{lq}^T,W_{uq}^T]}^T\in {\mathbb{R}}^{n\times m}$, $W_{lq}\in {\mathbb{R}}^{l\times m}$ and $W_{uq}\in {\mathbb{R}}^{u\times m}$. Meanwhile, we assume that Z is element-wise nonnegative and row-wise normalized, i.e., $Z_{ij}\ge 0$ and ${\sum }_{j=1}^m{Z}_{ij}=1$. Since $m\ll n$, a large portion of the elements in W of Equation (11) are zeros, which reduce the computational complexity to construct the graph. To make it more efficient, it is reasonable to assume that Z is sparse, since the manifold assumption of GSSL implies that it is good enough to have $Z_{ij}=0$ for data $x_i$ if it is far from a representative point $q_j$.

The matrix Z can be determined by Nadaraya–Watson kernel estimation [11,17]:

$$\begin{array}{c}\hfill Z_{ij}={\displaystyle \frac{K(x_i,q_j,\sigma )}{{\sum }_{k\in A_i}K(x_i,q_k,\sigma )}},\end{array}$$

(12)

where set $A_i$ consists of the indices of the s nearest representative points of $x_i$. Parameter s is given by user to define the sparsity number of each row in Z. $K(x_i,q_j,\sigma )$ represents kernel function; for instance, we can use following Gaussian kernel:

$$\begin{array}{c}\hfill K(x_i,q_j,\sigma )=exp\left({\displaystyle \frac{-\parallel x_i-q_j{\parallel }^2}{2{\sigma }^2}}\right).\end{array}$$

(13)

Representative points can be chosen using various methods, including random sampling, k-means clustering, or Density-Based Spatial Clustering of Applications with Noise (DBSCAN) [18]. Owing to space constraints, we opt for k-means in this paper for its simplicity and efficacy. Nevertheless, the proposed SSLCFBG framework can be adapted to incorporate other methods for selecting representative points.

2.4. Label Propagation and SSLCFBG

This subsection further enhances the efficiency of another step in label propagation by leveraging a bipartite graph, then we will summarize the proposed SSLCFBG algorithm. First, of all, the closed-form solution for label propagation in our SSLCFBG algorithm is established through the following theorem.

Theorem 1.

If the GSSL problem has

1. 

Undirected weighted graph as $G(\mathcal{V},\mathcal{E},W)$;

2. 

Label indicator matrix $Y={[Y_l^T,Y_u^T,Y_q^T]}^T$ where $Y_u=0$ and $Y_q=0$;

3. 

Affinity matrix W is defined as Equation (11).

Then, we have following closed-form solution for label propagation:

$$\begin{array}{c}\hfill F_u^{*}=I_{\alpha u}{W}_{uq}{\widehat{\Theta }}^{-1}{I}_{\alpha m}{W}_{lq}^T{I}_{\beta l}{Y}_l\end{array}$$

(14)

where $I_{\alpha }=diag(I_{\alpha l},I_{\alpha u},I_{\alpha m})\in {\mathbb{R}}^{(n+m)\times (n+m)}$ is a diagonal matrix, whose nonzero element satisfies ${\alpha }_{ii}={\displaystyle \frac{1}{1+{\lambda }_i}}$. $I_{\beta }=I-I_{\alpha }=diag(I_{\beta l},I_{\beta u},I_{\beta m})$, and

$$\begin{array}{c}\hfill \widehat{\Theta }=I-I_{\alpha m}{W}_{lq}^T{I}_{\alpha l}{W}_{lq}-I_{\alpha m}{W}_{uq}^T{I}_{\alpha u}{W}_{uq}.\end{array}$$

(15)

Proof. 

According to Equation (10), we have

$$\begin{array}{c}\hfill F^{*}={(L+\Lambda )}^{-1}\Lambda Y={(I-I_{\alpha }W)}^{-1}{I}_{\beta }Y=\left[\begin{array}{cc}I+B_{\alpha l}{\Theta }^{-1}{B}_{\alpha m}& -B_{\alpha l}{\Theta }^{-1}\\ -{\Theta }^{-1}{B}_{\alpha m}& {\Theta }^{-1}\end{array}\right]I_{\beta }Y\end{array}$$

(16)

where $B_{\alpha l}=[0,-I_{\alpha l}{W}_{lq}]$, $B_{\alpha m}=\left[\begin{array}{c}0\\ -I_{\alpha m}{W}_{lq}^T\end{array}\right]$ and

$$\begin{array}{c}\hfill {\Theta }^{-1}={\left[\begin{array}{cc}I& -I_{\alpha u}{W}_{uq}\\ -I_{\alpha m}{W}_{uq}^T& I-I_{\alpha m}{W}_{lq}^T{I}_{\alpha l}{W}_{lq}\end{array}\right]}^{-1}=\left[\begin{array}{cc}I+I_{\alpha u}{W}_{uq}{\widehat{\Theta }}^{-1}{I}_{\alpha m}{W}_{uq}^T& I_{\alpha u}{W}_{uq}{\widehat{\Theta }}^{-1}\\ {\widehat{\Theta }}^{-1}{I}_{\alpha m}{W}_{uq}^T& {\widehat{\Theta }}^{-1}\end{array}\right].\end{array}$$

(17)

Then, we can expand Equation (16) to have the following equation:

$$\begin{array}{c}\hfill F_u^{*}=I_{\alpha u}{W}_{uq}{\widehat{\Theta }}^{-1}{I}_{\alpha m}{W}_{lq}^T{I}_{\beta l}{Y}_l+(I+I_{\alpha u}{W}_{uq}{\widehat{\Theta }}^{-1}{I}_{\alpha m}{W}_{uq}^T)I_{\beta u}{Y}_u+I_{\alpha u}{W}_{uq}{\widehat{\Theta }}^{-1}{I}_{\beta m}{Y}_q,\end{array}$$

(18)

Utilizing $Y_u=0$ and $Y_q=0$ results in Equation (14).    □

Moreover, based on Equation (14), a specific solution can be derived after taking into account additional information. For example, we can utilize Harmonic property in HEM of [2], if set $\widehat{\Theta }=I-W_{uq}^T{W}_{uq}$, ${\alpha }_i$ and ${\beta }_i$ are as follows:

$$\begin{array}{cc}\hfill {\alpha }_i& ={\displaystyle \frac{1}{1+{\mu }_i}}=\left\{\begin{array}{cc}0,\hfill & i\in {\mathcal{I}}_l\\ {\displaystyle \frac{1}{1+0}}=1,\hfill & i\in {\mathcal{I}}_{u+m}\end{array}\right.,\hfill \end{array}$$

(19)

$$\begin{array}{c}\hfill {\beta }_i=\left\{\begin{array}{cc}1,\hfill & i\in {\mathcal{I}}_l\\ 0,\hfill & i\in {\mathcal{I}}_{u+m}\end{array}\right.,\end{array}$$

(20)

where $\mathcal{I}=\{1,\cdots ,n+m\}={\mathcal{I}}_l\cap {\mathcal{I}}_{u+m}$ denotes the index set of $\mathcal{V}=\mathcal{X}\cap \mathcal{Q}$. ${\mathcal{I}}_l$ is the index set for labeled data. ${\mathcal{I}}_{u+m}$ means the index set for unlabeled data and representative points. Then, the closed-form solution for HEM with bipartite graph is $F_l^{*}=Y_l$ and

$$\begin{array}{c}\hfill F_u^{*}=W_{uq}{(I-W_{uq}^T{W}_{uq})}^{-1}{W}_{lq}^T{Y}_l,\end{array}$$

(21)

whose computational complexity, $\mathcal{O}\left(m^3\right)$, is considerably lower than the $\mathcal{O}\left(u^3\right)$ complexity of HEM, under the assumption that $u\gg m$.

Lastly, the proposed SSLCFBG algorithm is encapsulated in Algorithm 1, with its computational complexity to be discussed in the subsequent subsection.

Algorithm 1: Semi-Supervised Learning with Close-Form label propagation using Bipartite Graph (SSLCFBG)

Input: m, s, $Y_l$, ${\lambda }_{1,\cdots ,n+m}$

Output: $F_u^{*}$

1 Selection of m representative points by k-means.

2 Construct bipartite graph then compute the corresponding $Z={[W_{lq}^T,W_{uq}^T]}^T$. Here, we use Equation (12) to decide the elements of Z.

3 Compute $I_{\alpha u}$, $I_{\alpha m}$ and $I_{\beta l}$.

4 Compute $\widehat{\Theta }$ using Equation (15).

5 Compute the soft label matrix for unlabeled data using Equation (14)

6 Determine the label for every data by Equation (3).

2.5. Complexity Analysis

The computational complexity of Algorithm 1 is governed by four main steps, which have a cost of $\mathcal{O}\left(n\right)$, with the exception of the third and final steps. Specifically: (1) the first step incurs a cost of $\mathcal{O}\left(dmnT\right)$, where T is typically small and denotes the number of iterations for k-means, thus simplifying the complexity of this step to $\mathcal{O}\left(dmn\right)$. (2) The second step has a cost of $\mathcal{O}\left(smn\right)$, with s representing the number of nearest representative points to be connected to. (3) The fourth step costs $\mathcal{O}\left(m^{2}n\right)$, given that in SSL problems, we have $u>l$ and $u\approx n$. And (4) The fifth step requires $\mathcal{O}\left(nmc\right)$ operations.

Consequently, the overall computational complexity of SSLCFBG is $\mathcal{O}(max(m,d)·mn)$, assuming $m\ge c$ and $m\ge s$. Moreover, since in SSL problems we typically have $n\gg d$ and $n\gg m$, the complexity of the proposed SSLCFBG is nearly linear with respect to the number of data points, making it more efficient than other GSSL algorithms. We will conduct experiments in the following section to elucidate the advantages of the proposed algorithm.

3. Results

In this section, we conduct extensive experiments on six real-world datasets to compare our algorithm with five state-of-the-art and baseline approaches.

3.1. Setup and Datasets

In our experimental setup, labeled data are randomly selected from a dataset, with the remaining instances treated as unlabeled. The reported results are the average of 20 independent trials. Unless specified otherwise, we construct a set of 1000 representative points. For the sake of simplicity, we employ Equation (21) during the label propagation phase.

All experiments are conducted on a desktop computer equipped with a 3.4 GHz Intel CPU and 16 GB of RAM, using Matlab for implementation. The running time is calculated in Matlab by aggregating the costs associated with graph construction and label propagation.

We carry out experiments on six real-world datasets, each described as follows:

• Handwritten (HW), named as “Multiple Features Data Set” in the UCI machine learning repository [19]). It consists of 2000 data points, each of which is a handwritten digit ranging from 0 to 9 with 10 classes. We use 64 dimensional Karhunen–Loève coefficients [20] as features.
• Caltech-101 (CT-101) [21]: Consists of 9144 images of objects across 101 categories. Each image has a size of about $300\times 200$ pixels. Similar to [22], we also consider two additional sub-datasets: Caltech-7 (CT-7) and Caltech-20 (CT-20). Caltech-7 consists of 1474 images among seven classes. Caltech-20 consists of 2386 images among 20 classes. We use a Histogram of Oriented Gradients [23] as feature with dimension of 1984.
• USPS [24]. The training part of the USPS dataset is used in our experiments. It consists of 10 classes of digits ranging from 0 to 9, with 7291 samples in total. We use all $16\times 16$ pixels in each sample as a feature vector with dimension of 256.
• NUS-WIDE-OBJECT (NUS) [25]. It consists of 31 object categories with totally 30,000 images, in which 17,927 are training images and the other 12,073 are testing data. We first combine all these 30,000 images, then randomly choose our labeled and unlabeled data from it. We use color correlogram in HSV color space [26] as the feature, with dimension of 144.
• MNIST [27]. It consists of 70,000 handwritten images in grayscale with size of $28\times 28$. Its 10 classes ranges from 0 to 9. We randomly choose labeled data, while the remaining images are used as unlabeled data. All of $28\times 28$ pixels in each sample are considered as a feature vector with dimension of 784.
• Animal with attributes (AWA) [28]. This dataset consists of 30,475 images among 50 animal classes, each of which contains at least 92 data points. With semantic attributes assigned to its class, animals can be uniquely characterized by their attribute vector. We employ rgSIFT [29] as the feature, with dimension of 2000.

3.2. Experimental Results

In this subsection, we report our experimental results into two groups: (1) with baseline algorithms: 1NN and SVM [30]. And (2) With the state-of-the-art algorithms: HEM [2], GLC [4], GGSSL [5] and AGR [12].

Meanwhile, six real-world datasets are categorized into two subgroups: (1) small-size datasets: Handwritten, Caltech-101 (with three subsets: Caltech-7, Caltech-20, and Caltech-101) and USPS. And (2) large-size datasets: NUS-WIDE-OBJECT, MNIST, and AwA.

3.2.1. Comparison to Baseline Methods

We first compare the proposed SSLCFBG with two baseline algorithms.

On small datasets, as detailed in

Table 1. Accuracy of the proposed method and baseline methods on HW dataset.

Methods	Label # 20 †	Label # 1000
1NN	0.7337 ± 0.0420	0.9663 ± 0.0049
SVM	0.7565 ± 0.0396	0.9607 ± 0.0060
Proposed	0.9385 ± 0.0237	0.9725 ± 0.0046

^† “# of Label” denotes “the number of labeled data”.

and

Table 2. Running time of the proposed method and baseline methods on HW dataset.

Methods	Label # 20 †	Label # 1000
1NN	0.0012 ± 0.0006	0.0162 ± 0.0014
SVM	0.0056 ± 0.0015	0.0524 ± 0.0019
Proposed	0.1341 ± 0.0023	0.1389 ± 0.0002

^† “# of Label” denotes “the number of labeled data”.

, the accuracy of the proposed algorithm surpasses that of both 1NN and SVM, particularly in scenarios with a limited number of labeled data. This is due to the fact that 1NN requires a larger number of labeled neighbors to accurately determine labels for unlabeled data, while SVM struggles with low accuracy because it attempts to separate data using hyperplanes, which can easily deviate from the correct position when labeled data are scarce. The proposed LSLGG inherits the high accuracies characteristic of graph-based methods. In terms of speed, SSLCFBG operates more slowly than 1NN and SVM, because we employ a relatively large number (i.e., 1000) of representative points to maintain consistency with other experiments in this paper. Evidently, the speed of SSLCFBG can be increased by reducing the number of representative points, as will be demonstrated in subsequent experiments.

On the larger MNIST dataset, SSLCFBG achieves significantly higher accuracy than 1NN and SVM when fewer labeled data are available, as shown in the left sub-figure of Figure 2. In the right sub-figure, the running times of SVM and 1NN increase markedly as the number of available labeled data grows, limiting their applicability in scenarios with a large volume of labeled data. Intriguingly, SSLCFBG maintains a nearly constant running time, independent of the number of labeled data, due to the use of a fixed number of representative points.

Based on these findings, our algorithm demonstrates a favorable balance between efficiency and effectiveness when compared to baseline methods.

3.2.2. Comparisons to State-of-the-Art Methods

As highlighted in our theoretical analysis, the high computational cost poses a significant barrier for many GSSL algorithms. In this subsection, we aim to demonstrate the effectiveness and efficiency of our algorithm in comparison to other state-of-the-art GSSL algorithms, excluding baseline methods that are either computationally intensive or lacking in accuracy.

The accuracy and running times for small and large datasets are presented in

Table 3. Accuracy of the proposed algorithm and other graph-based algorithms.

		Middle Size Datasets					Large Size Datasets
No.	Methods‡	HW	CT-7	CT-20	CT-101	USPS	NUS	MNIST	AwA
1	HEM	0.8970	0.8587	0.7565	0.3256	0.1719	0.1629	0.1844	0.0600
2	GLC	0.8991	0.7519	0.6551	0.2804	0.1513	0.1607	0.1817	0.0347
3	GGSSL	0.9064	0.8587	0.7520	0.2895	0.1513	0.1587	0.1795	0.0231
4	AGR	0.9214	0.8923	0.8072	0.3882	0.8515	0.1828	0.8494	0.0660
5	Proposed	0.9385	0.9017	0.8202	0.3990	0.8613	0.1923	0.8591	0.0700

^‡ The number of labeled data are HW $=20$, CT-7 $=20$, CT-20 $=50$, CT-101 $=200$, USPS $=20$, NUS $=1000$, MNIST $=20$, AwA $=1000$.

and

Table 4. Running time of the proposed algorithm and other graph-based algorithm.

		Middle Size Datasets					Large Size Datasets
No.	Methods	HW	CT-7	CT-20	CT-101	USPS	NUS	MNIST	AwA
1	HEM	0.1411	13.9983	42.3065	955.9894	17.2055	262.4838	13343.1989	5795.7914
2	GLC	0.1341	13.9624	42.2709	964.4532	17.2987	177.7972	12964.3791	6322.7574
3	GGSSL	0.1344	13.9637	42.2850	961.0679	17.2555	180.9809	13167.3352	6133.7992
4	AGR	0.1532	0.6570	1.3209	6.7802	1.4397	3.9459	24.2268	295.9069
5	Proposed	0.1341	0.6349	1.3022	6.7387	1.4175	3.9206	24.1960	295.8366

, respectively. We observe that the proposed SSLCFBG yields the best outcomes in terms of both effectiveness and efficiency across different datasets when compared to all other algorithms.

Regarding accuracy, as shown in Table 3 the proposed algorithm outperforms HEM, GLC, and GGSSL. This is because it not only leverages the advantages of GSSL algorithms, but also employs representative points to capture the underlying manifold structure of the raw data. These points are generated using k-means, which serves as a filtering mechanism to eliminate noisy data. Additionally, SSLCFBG demonstrates greater accuracy than AGR, as it does not impose any linear assumptions on the representative points or raw data, as discussed in our theoretical analysis.

In terms of efficiency, as indicated in Table 4, the proposed algorithm is considerably faster than HEM, GLC, and GGSSL, aligning with our theoretical analysis. This is due to two reasons: (1) label propagation in HEM, GLC, and GGSSL involves computing the inverses of $n\times n$ matrices, whereas SSLCFBG only requires the inverse of an $m\times m$ matrix, with $m\ll n$; and (2) the bipartite graph used in graph construction is highly sparse. Furthermore, our algorithm is slightly faster than AGR, as it does not require additional time to accommodate the linear assumption.

To illustrate the limitations of the linear assumption in AGR, we compare its accuracy to that of our algorithm on the USPS dataset, as depicted in Figure 3. The number of representative points ranges from 100 to 1000, where the number of labeled data are set at 20. As evident from Figure 3, the performance of the proposed SSLCFBG consistently outperforms AGR. Therefore, the proposed algorithm is more effective than AGR, as it is not constrained by linear assumptions.

These experiments collectively highlight the advantages of the proposed SSLCFBG in managing large-scale data with respect to both efficiency and accuracy. By “big data”, we refer to the following scenarios: (1) an increased number of data points, (2) data with higher dimensions, (3) a greater number of labeled data, or (4) any combination of these factors. The benefits of SSLCFBG will be even more pronounced in larger datasets, making it well-suited for big data applications.

4. Conclusions

In this paper, we introduce a novel large-scale semi-supervised learning framework that utilizes a bipartite graph. The label propagation within this framework exhibits nearly linear computational complexity with respect to the number of data points. We attribute its success to the construction of a sparse, anti-diagonal affinity matrix. In this bipartite graph, a limited number of representative points are connected to a larger set of raw data points, resulting in a sparse and anti-diagonal affinity matrix that accelerates both graph construction and label propagation. The abundance of zero entries in the affinity matrix facilitates rapid graph construction. Moreover, the significantly smaller size of the matrix inverse required in our algorithm accelerates the process compared to traditional GSSL algorithms, as the affinity matrix simplifies the closed-form solution of our approach. Compared to existing fast algorithms like AGR, our proposed algorithm is consistently more efficient and accurate, as it does not require any linear assumptions. Extensive experiments conducted on six real-world datasets showcase the advantages of the proposed SSLCFBG over five state-of-the-art and baseline algorithms, both in terms of efficiency and effectiveness.