ReliaMatch: Semi-Supervised Classification with Reliable Match

Abstract

Deep learning has been widely used in various tasks such as computer vision, natural language processing, predictive analysis, and recommendation systems in the past decade. However, practical scenarios often lack labeled data, posing challenges for traditional supervised methods. Semi-supervised classification methods address this by leveraging both labeled and unlabeled data to enhance model performance, but they face challenges in effectively utilizing unlabeled data and distinguishing reliable information from unreliable sources. This paper introduced ReliaMatch, a semi-supervised classification method that addresses these challenges by using a confidence threshold. It incorporates a curriculum learning stage, feature filtering, and pseudo-label filtering to improve classification accuracy and reliability. The feature filtering module eliminates ambiguous semantic features by comparing labeled and unlabeled data in the feature space. The pseudo-label filtering module removes unreliable pseudo-labels with low confidence, enhancing algorithm reliability. ReliaMatch employs a curriculum learning training mode, gradually increasing training dataset difficulty by combining selected samples and pseudo-labels with labeled data. This supervised approach enhances classification performance. Experimental results show that ReliaMatch effectively overcomes challenges associated with the underutilization of unlabeled data and the introduction of error information, outperforming the pseudo-label strategy in semi-supervised classification.

1. Introduction

In the past decade, deep learning has dominated the machine learning landscape in data classification [1,2,3], predictive analysis [4], recommendation systems [5], anomaly detection [6,7] and so on. Within deep learning, supervised classification methods have significantly improved the performance of deep learning in various classification tasks. However, it is still very difficult to obtain labels provided by professionals in many big data application scenarios. In contrast, unsupervised classification methods have obvious advantages in dealing with unlabeled samples. Nevertheless, they greatly sacrificed the accuracy of the model because they cannot directly evaluate and optimize the performance of deep learning models by using the label information. Therefore, semi-supervised learning [8] caught the attention of researchers, and significantly improved the model performance of unsupervised learning by leveraging datasets with a small amount of label data. A more practical value lies in semi-supervised learning methods being able to reduce the cost and time of manually marking data. Recently, many semi-supervised learning methods based on deep learning have been proposed [9,10,11], which can achieve quite a good performance by leveraging the small fraction of labeled samples in the dataset.

However, semi-supervised learning faces two major challenges: (i) how to transfer information obtained from limited labeled data to unlabeled data, and ($ii$) how to learn as accurate information as possible directly from a large amount of unlabeled data. To address these issues, semi-supervised learning uses three key loss terms (i.e., entropy minimization, generalization regularization, and consistency regularization) to better motivate the model to learn the corresponding downstream tasks. Among them, entropy minimization can encourage the model to confidently predict the output of unlabeled data, thereby improving the accuracy and robustness of the model. Generalization regularization constrains the model’s parameters to avoid overfitting to the training data during the training process, thereby improving the model’s generalization performance. According to the consistency assumption, data points that are close to each other often have consistency in the same label and structure [12]. Therefore, consistency regularization can improve the accuracy and generalization ability of the model by making the data consistent on the manifold.

In addition, many studies have explored methods for solving the information propagation problem between different data [13], providing ideas for addressing the first major issue in semi-supervised learning. Among them, pseudo-labeling and consistency regularization methods provide ideas for solving the second major issue in semi-supervised learning. The pseudo-labeling method [14] uses the predictions of a classification model or a clustering algorithm as artificial labels to retrain the model. The consistency regularization method [10,11,15] forces the model to make the same prediction for the same sample under different transformations, learning from unlabeled data. However, these semi-supervised learning methods do not consider the possibility of introducing different levels of erroneous information during the training process, which can lead to low classification accuracy.

In the feature extraction stage, as plotted in Figure 1 (left), the model may have difficulty accurately differentiating between semantic differences at the classification boundary due to the ambiguity of feature representation boundaries. Without setting anchors for each class, the model may learn incorrect semantic information. However, by providing each class with an anchor, confidence thresholds can be set using the similarity between each sample and the anchor, allowing low-confidence features to be filtered out as shown in Figure 1 (right). When assigning pseudo-labels to unlabeled data, as plotted in Figure 2, assigning labels to samples with low predicted confidence may lead to confirmation bias [16], where the model overfits to incorrect labels and reduces its performance. Using a fixed threshold method cannot adapt to the dynamic changes of the dataset. On the contrary, a global dynamic threshold method dynamically adjusts the threshold based on the confidence distribution of unlabeled samples in the current iteration, thus avoiding this problem.

To address these issues, we proposed ReliaMatch, a semi-supervised classification method that filters unreliable information based on a confidence threshold. ReliaMatch adopts a confidence threshold filtering strategy, which matches the similarity of labeled data and unlabeled data in feature space by setting anchor points, thus filtering out outliers and demarcation points with ambiguous semantics. Dynamic threshold is used to select reliable pseudo-labels, so as to eliminate the confirmation deviation of the model to pseudo-labels and improve classification performance. Additionally, ReliaMatch adopts the training mode of Curriculum Learning [18], which combines the screened samples and their pseudo-labels with labeled data, gradually increasing the difficulty of training datasets and participating in model training in a supervised way, thus further improving the classification performance. In summary, we make the following three main contributions:

(1)

We proposed a semi-supervised classification method (Reliable Match), which addresses the issue of confirmation bias that arises from unlabeled data having different semantics and low prediction confidence near the classification boundary.

(2)

ReliaMatch employs a confidence threshold filtering strategy that matches the similarity of labeled and unlabeled data in feature space by setting anchor points, which filter out outliers and demarcation points with ambiguous semantics. To eliminate confirmation deviation of the model to pseudo labels and improve classification performance, ReliaMatch uses a dynamic threshold to select reliable pseudo-labels.

(3)

ReliaMatch employs the Curriculum Learning training mode, which combines the screened samples and their pseudo-labels with labeled data and gradually increases the difficulty of the training dataset, thereby participating in model training in a supervised manner and further improving classification performance.

2. Related Work

2.1. Semi-Supervised Classification

Semi-supervised learning (SSL) has been extensively studied in various fields, including image classification [19], object detection [20], and semantic segmentation [21]. SSL methods in image classification aim to reduce reliance on labeled data by leveraging unlabeled data. In SSL, labeled results are typically obtained through consistent regularization [17,22,23], pseudo-labeling [14], and entropy minimization [24]. Consistent regularization ensures that the model produces consistent predictions for different transformations of the same image. Pseudo-labeling employs model confidence to assign labels and guide the training process, while entropy minimization encourages the model to produce highly confident predictions. These labeling strategies have been widely adopted in many SSL approaches.

2.2. Consistency Regularization

Consistency regularization plays a crucial role in modern semi-supervised learning (SSL) algorithms. The core idea behind consistency regularization is that the same input sample should produce consistent outputs under different perturbations. Early works, such as [10,25,26], proposed this concept, which was further developed in [9,15,17]. The fundamental form of consistency regularization in SSL is often achieved through a loss term. The equation below represents this basic form:

$$\left|\right|p_m\left(y\right|A\left(x\right);\theta )-p_m\left(y\right|A\left(x\right);\theta ){\left|\right|}_2^2,$$

(1)

where A refers to stochastic functions, resulting in different values for $A\left(x\right)$, while $p_m$ represents the model’s output probability. In [26], random data augmentation, dropout, and random maximum pooling were employed as A to ensure similarity among the predictions of neural networks. On the other hand, Ref. [11] adopts adversarial transformations for augmentation. Another related approach, presented in [10], extends the perturbations to different time periods, requiring the current prediction of a sample to be similar to the prediction set of the same sample in the past. These perturbations mainly arise from different network states and data augmentations.

In SSL, consistency regularization techniques aim to ensure that the same input sample produces consistent output predictions under different perturbations. Different approaches have been proposed to achieve this goal. For example, in [26], random data augmentation, dropout, and random maximum pooling were used to promote similarity among the predictions of neural networks. Adversarial transformations were employed for augmentation in [11]. Additionally, Ref. [10] extends the perturbations to different time periods, enforcing similarity between current and past predictions for the same sample. In [15], two networks with the same structure were utilized, and the consistency constraint was enforced by comparing the predicted distributions using KL divergence or cross-entropy functions. This approach was further developed in [27], where uncertainty weighting was applied to unlabeled samples, focusing on samples with lower uncertainty. Virtual adversarial training, proposed by Miyato et al. [11], introduces adversarial noise as interference into data samples, followed by unified regularization of the resulting predictions. Another recent idea by Luo et al. [28] suggests using a comparison loss as the regularization term, ensuring that predictions from the same (or different) categories are similar (or different). This extends the scope of consistency regularization to cover consistency between different samples and can be combined with other methods like [15] or [11] for improved performance. To address model memorization and sensitivity to adversarial data, Mixup, proposed by Zhang et al. [29], pairs examples and labels by training a convex combination of neural networks. Verma et al. [30] built on Mixup with interpolation consistency training, which encourages consistency between unlabeled samples and the interpolation prediction of a single sample. Moreover, in [17], consistent regularization was achieved through estimating low-entropy labels, generating data-augmented unlabeled samples, and utilizing Mixup to combine labeled and unlabeled samples.

2.3. Pseudo-Labeling

Pseudo labels are artificial labels generated by the model itself, which are used to further train the model. Through the pseudo-labeling method, we can use both labeled samples and pseudo labeled samples as new training data to update the model, thus greatly improving the utilization rate of unlabeled samples.

Lee et al. [14] chose the class with the highest prediction probability of the model as the pseudo label; however, pseudo labels are only used in the fine-tuning stage, and the network needs to be pre-trained. Mandal et al. [31] proposed a new deep semi-supervised framework, which can seamlessly process marked and unlabeled data. The framework is trained by two parts in turn: firstly, the label prediction component is used to predict the label of the unlabeled part of the training data, and then the common representation of two patterns is learned for cross-modal retrieval. Caron et al. [32] proposed a deep clustering algorithm combining a K-means clustering algorithm and a convolutional neural network, which used the clustering results of K-meaas with unlabeled data as false labels to assist CNN in classification. Based on the extreme value theory, Cascante-Bonilla et al. [33] put forward the Curriculum Labeling (CL), which uses careful curriculum selection as a pacing standard to strengthen the pseudo labeling. Hu et al. [34] designed a new end-to-end Iterative Feature Clustering Graph Convolution Network (IFC-GCN) to enhance the standard GCN through the iterative feature clustering module, and designed an EM-like framework to improve the network performance by alternately correcting false labels and the node characteristics.

3. Method

The core idea of ReliaMatch is to match the correlation between labeled and unlabeled data, by filtering reliable unlabeled data and generating pseudo-labels, which are then used as new training data in the supervised learning process of the model. The detailed process of ReliaMatch is shown in Figure 3.

This method uses a confidence threshold to filter unreliable information, that is, feature vectors on the boundary of unlabeled data classification or outlier feature vectors and artificially marked false labels that may be incorrect. ReliaMatch adopts a self-training framework, that is, by iteratively learning the information in unlabeled datasets and labeled datasets, the performance of the deep learning model is improved. In the training process, ReliaMatch uses the trained model to predict the unlabeled data and adds the reliable data points in the prediction results and their pseudo-labels to the pseudo-label dataset. Then, the pseudo-labeled dataset and the labeled dataset are merged to train the next round model. This process is repeated until a preset number of iterations or performance convergence is achieved.

For labeled samples, ReliaMatch trains them using the feature extraction model and the classification model. After training, feature extraction and classification prediction are performed on the labeled samples. At the same time, the average feature vector of each category of the labeled samples is calculated and used as a feature anchor for filtering reliable features of the unlabeled samples.

For unlabeled samples, there are two main modules: the feature filtering module and the pseudo-label filtering module, which correspond to the feature and label noise filtering, respectively. In conjunction with Figure 3, the feature filtering module includes filtering 1, and the pseudo-label filtering module (i.e., filtering of label noise) contain filtering 2 and filtering 3. ReliaMatch first performs data processing by using data augmentation techniques to expand the unlabeled samples. Next, the enhanced unlabeled samples are input into the model for feature extraction. Then, the feature filter module calculates the similarity between the extracted features of the unlabeled samples and the feature anchor, and sets a feature similarity threshold to filter out the unlabeled samples with low similarity. This process is called filtering 1. The unlabeled samples selected by filtering 1 are labeled with the class label of the nearest feature anchor, and are used as pseudo-labels (hard labels). Next, ReliaMatch inputs these feature-filtered samples into classifier for classification prediction. For each unlabeled sample, it compares whether the class label of its maximum predicted probability (soft label) is consistent with the class label of the pseudo-label. If the classes are inconsistent, the unlabeled sample is filtered out. This process is called filtering 2. Finally, a dynamic threshold is set for the predicted probability to filter out unlabeled samples with maximum predicted probability below the threshold. This process is called filtering 3.

After three rounds of filtering, the remaining unlabeled samples are considered to be high-confidence reliable samples and are combined with their pseudo-labels to form a pseudo-labeled dataset. These pseudo-labeled samples are merged with labeled samples to form a new labeled dataset, which is used to train a new model. This process is iterated continuously to gradually increase the size of the labeled dataset and improve the performance of semi-supervised learning.

3.1. Problem Description

To describe the design process of the ReliaMatch model more accurately, we assumed that, in the tth round of iteration, the training dataset $X_N^t$ is used, which contains $N^t$ samples including image data from different categories $X_N={\left\{x_i\right\}}_{i=1}^{N^t}$. The training dataset is divided into a labeled dataset $X_L^t={\left\{x_{i,l}^t\right\}}_{i=1}^{N_L^t}$ and an unlabeled dataset $X_U^t={\left\{x_{i,u}^t\right\}}_{i=1}^{N_U^t}$. Assuming that $f_{\theta }^t$ is the convolutional neural network used for feature extraction in the tth round of iteration. $Z_N^t=f_{\theta }^t\left(x_i^t\right)$ represents the feature vector set obtained from $X_N^t$ after being processed by the convolutional neural network $f_{\theta }^t$. $Z_N^t$ consists of two parts, $Z_L^t$ and $Z_U^t$, where $Z_L^t=f_{\theta }^t(x_{i,l}^t,y_{i,l}^t)={(z_{i,l}^t,y_{i,l}^t)}_{i=1}^{N_L^t}$ is the labeled feature vector set in the tth round of iteration, and $Z_U^t=f_{\theta }^t\left(x_{i,u}^t\right)={\left\{z_{i,u}^t\right\}}_{i=1}^{N_U^t}$ is the unlabeled feature vector set in the tth round of iteration. Let $g_{\varpi }^t$ be the fully connected neural network (classifier) used by the model for classification prediction in the tth round of iteration, with its output being the predicted probability of the sample in each category. Let $p_{i,j}^t$ denote the probability that the model predicts sample $x_i^t$ as category j, and K be the number of categories. Then, $P\left(x_i^t\right)=g_{\varpi }^t\left(f_{\theta }^t\left(x_i^t\right)\right)={\left\{p_{i,j}^t\right\}}_{j=1}^K$.

3.2. Feature Anchoring

Our first contribution is feature anchoring, which uses the features of the labeled data to calculate the average of the features of each category as an anchor. We think that nearby points are likely to have the same labels, and we also agree that points on the same structure (usually called clusters or manifolds) may have the same label. Therefore, our method pays attention to the representation study of images, and we used the similarity of feature level to fliter unreliable samples and assigns pseudo labels to each reliable sample. A schematic of filtering poor features out of samples can be seen in Figure 4.

We first calculated the average feature vectors of each class j in the labeled feature dataset $Z_L^t$ at the tth iteration to generate K feature anchor points $A^t={\left\{a_j^t\right\}}_{j=1}^K$. To do this, we used the following equation:

$$a_j^t=\frac{1}{\left|Z_{j,L}^t\right|}\sum _{\left(z_{i,l}^t,y_{i,l}^t\right)\in Z_{j,L}^t}{z}_{i,l}^t,$$

(2)

where $Z_{j,L}^t$ represents a subset of feature samples in the feature vector set $Z_L^t$, where the label $y_{i,l}^t=j$ in the tth round of iteration. $|Z_{j,L}^t|$ denotes the number of samples in the subset $Z_{j,L}^t$.

Next, we used cosine similarity to calculate the similarity between the extracted enhanced unlabeled samples’ features and feature anchor points:

$$\begin{array}{cc}\hfill f_{SimFea}^t& =s(z_{i,u}^t,a_j^t)=\frac{z_{i,u}^t·a_j^t}{|z_{i,u}^t|·|a_j^t|}\hfill \\ \hfill & =\frac{{\sum }_{d=1}^D{z}_{i,u,d}^t{a}_{j,d}^t}{\sqrt{{\sum }_{d=1}^D{\left(z_{i,u,d}^t\right)}^2}·\sqrt{{\sum }_{d=1}^D{\left(a_{j,d}^t\right)}^2}},\hfill \end{array}$$

(3)

where $z_{i,u}^t$ represents the feature vector of the unlabeled data, $a_j^t$ represents the anchor point, · represents the dot product of the vector, $|·|$ represents the norm of the vector, D is the dimension of the vector. A cosine similarity close to 1 indicates that the two vectors are very close in space, while a cosine similarity close to −1 indicates that the two vectors are almost opposite in space. A cosine similarity close to 0 indicates that there is no obvious correlation between the two vectors in space.

Next, the minimum similarity between labeled sample features and anchor points is used as the threshold for feature similarity, and a hyperparameter is used to dynamically adjust the threshold size of feature similarity:

$$\begin{array}{c}\hfill {\tau }_{Fea}^t={\alpha }_{Fea}^{t}min\left(s(z_{i,l}^t,a_j^t)\right)={\alpha }_{Fea}^{t}min\left(\frac{z_{i,l}^t·a_j^t}{|z_{i,l}^t|·|a_j^t|}\right),\end{array}$$

(4)

where ${\alpha }_{Fea}^t\in [0,1]$ is a coefficient used to dynamically adjust the threshold of feature similarity, $z_{i,l}^t$ represents the feature vector of the labeled sample, $a_j^t$ represents the feature anchor point, · denotes the dot product of vectors, and $|·|$ denotes the norm of vectors. The feature similarity threshold ${\tau }_{Fea}^t\in [-1,1]$.

If the similarity $f_{SimFea}^t$ between the feature of an unlabeled sample and the feature anchor point is greater than the feature similarity threshold ${\tau }_{Fea}^t$, that is, $f_{SimFea}^t>{\tau }_{Fea}^t$, then the pseudo-label of the current sample $x_{i,u}^t$ is considered to be the label j of the feature anchor point that is most similar to its feature vector.

In the feature filtering module, the labeled dataset remains unchanged. In the t-th iteration, pseudo-labels of unlabeled samples can be obtained after feature filtering. The feature-filtering module dataset consisting of the filtered unlabeled samples and their pseudo-labels is denoted as $X_{U1}^t={\left\{(x_{i,u1}^t,y_{i,u1}^t)\right\}}_{i=1}^{N_{U1}^t}$, where $y_{i,u1}^t$ is the pseudo-label of $x_{i,u1}^t$, and $N_{U1}^t$ is the number of samples in the dataset.

3.3. Dynamic Allocation Pseudo Lables

In the pseudo-labeling strategy, threshold selection is an important issue. Traditional semi-supervised classification methods usually use a fixed threshold to predict the class of unlabeled samples and assign pseudo-labels to high-confidence samples above this threshold for training. However, this method may not adapt well to changes in the data distribution, and may either filter out useful samples excessively or add unreliable pseudo-labels, leading to classification errors due to samples of different classification difficulties.

To address these issues, the pseudo-label filtering module in ReliaMatch adopts a global dynamic threshold to filter reliable samples. Specifically, in each iteration, the module sets the confidence threshold for predicting probability based on the average confidence of unlabeled samples for the entire dataset. If the average confidence of unlabeled samples is high, it indicates that the algorithm has good classification performance for unlabeled samples, and the threshold can be increased. If the change in labels between two iterations is significant, it indicates that the algorithm has not yet converged and the threshold should be appropriately lowered to ensure the accuracy of the model.

For each unlabeled sample $x_{i,u1}^t$, its predicted probabilities $P\left(x_{i,u1}^t\right)={\left\{p_{i,u1,j}^t\right\}}_{j=1}^K$ can be converted into a hard label by creating a vector of length K, denoted as $O{i}^t={\left\{o_{i,j}^t\right\}}_{j=1}^K$, as follows:

$$o_{i,j}^t=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}j=argmax\left(p_{i,u1,j}^t\right)\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(5)

where $o_{i,j}^t$ indicates the label assigned to sample $x_{i,u1}^t$ for the jth class. Specifically, the class with the highest predicted probability is assigned a label of 1 and the others are assigned a label of 0.

Next, the average confidence score $p_u^t$ can be obtained by computing the average of the maximum predicted probability values for all unlabeled samples, i.e.,

$$\begin{array}{c}\hfill p_u^t=\frac{1}{|X_{U1}^t|}\sum _{x_{i,u1}\in X_{U1}^t}max\left(P\left(x_{i,u1}^t\right)\right),\end{array}$$

(6)

where $X_{U1}$ denotes the unlabeled dataset after feature filtering, and $P\left(x_{i,u1}^t\right)$ represents the predicted probabilities of sample $x_{i,u1}$ in the tthtraining round.

ReliaMatch uses the value of $p_u^t$ as the confidence level of unlabeled data to adjust the credibility threshold of pseudo labels to ensure the quality of pseudo labels. Therefore, the global dynamic threshold ${\tau }_{Pre}^t$ in round t can be represented as:

$${\tau }_{Pre}^t=\left\{\begin{array}{c}max\left(\frac{1}{1+e^{-p_u^t}},{\tau }_{Pre}^0\right),t=1\hfill \\ max\left(\frac{1}{1+e^{-p_u^t}},\frac{p_u^t·{∥y_{i,u1}^t-y_{i,u1}^{t-1}∥}_F}{{∥p_u^t\left(x_{i,u1}^t\right)-p_u^{t-1}\left(x_{i,u1}^{t-1}\right)∥}_F+ϵ}\right)\hfill \end{array},t>1,\right.$$

(7)

where ${\tau }_{Pre}^0$ is the initial threshold. $ϵ$ is a very small constant to avoid division by zero. $\left|\left|·\right|\right|F$ denotes matrix norm, and $p_u^t$ represents the average confidence level in round t. ${\left|\left|y_{i,u1}^t-y_{i,u1}^{t-1}\right|\right|}_F$ represents the size of the difference between the pseudo labels of the unlabeled sample i in round t and those in round $t-1$. ${\left|\left|p_u^t\left(x_{i,u1}^t\right)-p_u^{t-1}\left(x_{i,u1}^{t-1}\right)\right|\right|}_F$ represents the difference between the confidence level $p_u^t$ of the unlabeled sample i in round t and that in round $t-1$.

To select the most reliable pseudo labels, ReliaMatch further filters the unlabeled data in $X_{U1}^t$. The filtering criteria are as follows:

• Only select samples whose maximum predicted probability is greater than the predicted probability threshold ${\tau }_{Pre}$:

  $$max\left(P\left(x_{i,u1}^t\right)\right)>{\tau }_{Pre}.$$

  (8)
• Only select samples whose predicted category is consistent with the pseudo label in the pseudo label filtering module:

  $$O_i^t=y_{i,u1}^t.$$

  (9)

After screening, only samples that meet the above requirements can be added to the labeled dataset and used for supervised network training. Therefore, the resulting pseudo-labeled dataset is denoted as $X_{U2}^t={(x_{i,u2}^t,y_{i,u2}^t)}_{i=1}^{N_{U2}^t}$, where $y_{i,u2}^t$ represents the pseudo-label of $x_{i,u2}^t$, and $N_{U2}^t$ represents the number of samples in this dataset. The formula is as follows:

$$X_{U2}^t={\left\{(x_{i,u2},y_{i,u2})\right\}}_{i=1}^{N_{U2}^t}|max\left(P\left(x_{i,u1}^t\right)\right)>{\tau }_{Pre}\cap O_i^t=y_{i,u1}^t.$$

(10)

3.4. Loss

In ReliaMatch, the unlabeled dataset $X_{U2}^t$, after undergoing feature filtering and pseudo-label filtering, is transferred to the labeled dataset for supervised training, forming a new labeled dataset $X_L^{t+1}$ and an updated unlabeled dataset $X_U^{t+1}$, given by the following formulas:

$$X_L^{t+1}={\{x_{i,l}^{t+1},y_{i,l}^{t+1}\}}_{i=1}^{N_L^t+N_{U2}^t}:=X_L^t+X_{U2}^t,$$

(11)

$$X_U^{t+1}={\{x_{i,u}^{t+1},y_{i,u}^{t+1}\}}_{i=1}^{N_U^t-N_{U2}^t}:=X_U^t-X_{U2}^t.$$

(12)

Based on this, the model parameters are updated using the supervised loss ${\mathcal{L}}_s^t$. Specifically, for a sample $x_{i,l}^t$ in the labeled dataset with its label $y_{i,l}^t$, we have:

$${\mathcal{L}}_s^t={\mathcal{L}}_{CE}(g_{\varpi }^t\left(f_{\theta }^t\left(x_{i,l}^t\right)\right),y_{i,l}^t),$$

(13)

where $x_{i,l}$ is a sample from the labeled dataset in the tth iteration, and $y_{i,l}$ is the label of this sample. ${\mathcal{L}}_{CE}$ represents the cross-entropy loss function.

4. Experiments

4.1. Datasets

CIFAR-10: A dataset containing 60K images, with the shape of 32 × 32, is evenly distributed in ten categories. There are 50 K images in the training set and 10 K images in the test set. Our validation set size is 5000 for CIFAR-10 [35].

SVHN: A dataset containing 99,289 images, with the shape of 32 × 32, is evenly distributed in ten categories. SVHN consists of ten categories. The training contents contains 73,257 images, and the test set contains 26,032 images. Our validation set size is 5000 for SVHN [36].

CIFAR-100: A dataset containing 60 K images, with the shape of 32 × 32, is evenly distributed in one hundred categories. There are 50 K images in the training set and 10 K images in the test set. Our validation set size is 5000 for CIFAR-100 [37].

4.2. Model Details

ReliaMatch uses CNN-13 [38] and WideResNet-28 [39] for classification on CIFAR-10 and SVHN datasets. In order to make fair comparisons with other methods, the same parameters as in [33] were used in this paper. The network optimization is performed using stochastic gradient descent and Nesterov momentum algorithm, combined with weight decay regularization to reduce overfitting. The momentum factor is set to 0.9 and the initial learning rate is 0.1. To further improve optimization, cosine annealing [40] is employed to update the model parameters. By default, the initial hyperparameters ${\alpha }_{Fea}^t$ is set to 1 and ${\tau }_{Pre}^0$ is set to 0.95.

4.3. Experimental Results and Analysis

In this section, we compared ReliaMatch with other common semi-supervised learning methods on CIFAR10 and SVHN datasets, including Pseudo-Label [14], LP-MT [13], PL-CB [16], Curriculum Labeling [33], FlexMatch [41] based on pseudo-labeling, $\pi$ Model [10], Temporal Ensembling [10], Mean Teacher [15], VAT [11], Ladder Net [9], ICT [42] based on consistency regularization, and MixMatch [17] based on strong mixup. We also compared ReliaMatch with Meta-Semi [43], a meta-learning based SSL algorithm.

Additionally, the reliability of ReliaMatch can be assessed by the Test Error Rate(%) and the Error Rate corresponding to different numbers of labeled samples: the lower Test Error Rates on different networks and datasets can prove the effectiveness of ReliaMatch; reducing the number of labeled samples without a significant increase in Error Rate can demonstrate the robustness of ReliaMatch.

From the results shown in

Table 1. Comparison of Test Error Rate (%) of WideResNet-28 with Different Semi-supervised Methods.

Method	CIFAR-10 (${\mathit{N}}_{\mathit{L}}$ = 4000)	SVHN (${\mathit{N}}_{\mathit{L}}$ = 1000)
PL †	17.78 ± 0.57	7.62 ± 0.29
Curriculum Labeling †	8.92 ± 0.03	5.65 ± 0.11
PL-CB †	6.28 ±0.30	-
$\pi$ Model †	16.37 ± 0.63	7.19 ± 0.27
Mean Teacher	10.36 ± 0.28	5.65 ± 0.47
VAT †	13.86 ± 0.27	5.63 ± 0.20
VAT+EntMin †	13.13 ± 0.39	5.35 ± 0.19
ICT †	7.66 ± 0.17	3.53 ± 0.07
MixMatch †	6.24 ± 0.06	3.27 ± 0.31
FlexMatch ${}^{‡}$	4.19 ± 0.01	6.72 ± 0.30
Meta-Semi ${}^{★}$	6.10 ± 0.10	-
ReliaMatch *	5.86 ± 0.12	4.04 ± 0.08

$N_L$ represents the number of labeled samples in the training set, † indicates that the result has been reported in the literature [33], ${}^{‡}$ indicates result has been reported in [44], ${}^{★}$ indicates the result has been reported in [45] and * represents the average of 5 runs of the method proposed in this paper.

and

Table 2. Comparison of Test Error Rate (%) of Different Semi-supervised Methods Using CNN-13.

Method	CIFAR-10 (${\mathit{N}}_{\mathit{L}}=4000$)	SVHN (${\mathit{N}}_{\mathit{L}}=1000$)
LP-MT †	10.16 ± 0.28	-
Curriculum Labeling †	9.81 ± 0.22	4.75 ± 0.28
Ladder Net †	12.16 ± 0.31	-
Temporal Ensembling †	12.16 ± 0.24	4.42 ± 0.16
ReliaMatch *	7.42 ± 0.05	7.13 ± 0.28

$N_L$ represents the number of labeled samples in the training set, † indicates that the result has been reported in the literature [33], and * represents the average of five runs of the proposed method in this paper.

, it can be seen that, compared to other semi-supervised learning methods, ReliaMatch considers issues such as feature and label noise, filters out semantically ambiguous features and unreliable pseudo-labels, and thus performs better. Specifically, the ReliaMatch method uses the same WideResNet-28 network as advanced semi-supervised classification methods proposed by previous researchers. On the CIFAR-10 dataset, this method used 4000 labeled data for testing, and achieved a test error rate of only 5.86%. In comparison, the test error rate of the Pseudo-Label method was 17.78%, Curriculum Labeling method was 8.92%, and PL-CB method was 6.28%. In addition, FlexMatch was slightly better than ReliaMatch, which had a test error rate of only 4.19% on CIFAR-10. On the SVHN dataset, the ReliaMatch method used only 1000 labeled data for testing, achieving a test error rate of 4.04%, significantly better than the test error rates of Pseudo-Label (7.62%), Curriculum Labeling (5.65%), and FlexMatch method (6.72%).

In addition, the ReliaMatch method uses the same CNN-13 network as advanced semi-supervised classification methods proposed by previous researchers. On the CIFAR-10 dataset, using 4000 labeled data for testing, the test error rate of ReliaMatch method was 7.42%, which is lower than the test error rates of Pseudo-Label methods (LP-MT, Curriculum Labeling) and consistency regularization methods (Ladder Net, Temporal Ensembling).

We also performed an evaluation on the CIFAR-100 [37] dataset. ReliaMatch achieved 38.67% and 37.28% Test Error Rate on CIFAR-100 for WidResNet-28 and CNN-13. It is worth noting that it was a large difference from the effect on the other two datasets, which may be due to CIFAR-100 being more complex, with more categories, and the maximum predictive probability of each category may have been significantly different, making the global dynamic threshold not adapted.

One common way to evaluate semi-supervised classification algorithms is by varying the size of the labeled dataset. By reducing the number of available labeled samples, it is possible to better simulate real-world scenarios. As shown in Figure 5, on the CIFAR-10 and SVHN datasets, we tested the error rates of the ReliaMatch method using the WideResNet-28 network under different numbers of labeled samples. We used datasets with 500, 1000, 2000, and 4000 labeled samples, respectively, and only changed the number of labeled samples during each training while keeping other hyperparameters the same as when using 4000 labeled samples.

The experimental results show that the classification performance of the ReliaMatch method does not significantly degrade under different numbers of labeled samples. This indicates that the ReliaMatch method has good robustness and can provide stable performance even with very limited labeled data, which is crucial for practical applications since, in many cases, only very limited labeled data can be obtained. Therefore, these results suggest that the ReliaMatch method is an effective semi-supervised classification algorithm that can be useful in practical applications.

In addition, we also investigated the effectiveness of the feature filtering module and pseudo-label filtering module in the ReliaMatch method. As shown in

Table 3. Exploring the influence of important modules of ReliaMatch on classification results.

Method	Test Error Rate (%)
w/o Feature Filtering	6.74
w/o Pseudo-label Filtering	9.12
w/o Feature Filtering and Pseudo-label Filtering	16.9
ReliaMatch (benchmark)	5.86

, we separately or collectively removed these two modules and evaluated their impact on the method performance by applying data augmentation during training. The experiments were conducted on the CIFAR-10 dataset with 4000 labeled samples, and the WideResNet-28 was used as the backbone network.

After analyzing Table 3, we have reached the following conclusions: the pseudo label filtering module has a significant impact on the performance of the ReliaMatch method, as the removal of this module leads to an increase in model error rate from 5.86 to 9.12. This indicates that the pseudo label filtering module can remove low-confidence pseudo labels, thereby reducing noise and improving model performance. In contrast, the effect of the feature filtering module is not as significant as that of the pseudo label filtering module, but it still contributes to improving model performance. Its removal results in a model error rate increase from 5.86 to 6.74, indicating that the feature filtering module can select high-quality features and thus improve the model’s performance. Additionally, removing both feature and pseudo label filtering modules causes a sharp increase in model error rate, from 5.86 to 16.9. This demonstrates that both feature and pseudo label filtering modules make important contributions to the performance of the ReliaMatch method, and both are necessary.

Therefore, both feature and pseudo label filtering modules play important roles in the ReliaMatch method. The feature filtering module can select high-quality features, while the pseudo label filtering module can remove low-confidence pseudo labels. Their combined effect can improve model performance.

We demonstrated the training process of ReliaMatch using the CNN-13 network on the CIFAR-10 and SVHN datasets. ReliaMatch adopts a curriculum learning training approach. The method combines the selected samples and their reliable pseudo-labels with the labeled data, gradually increasing the difficulty of the training data to participate in model training in a supervised manner. As shown in Figure 6 and Figure 7, during the training process, ReliaMatch reinitializes the model training after each label transfer to mitigate the problem of confirmation bias caused by pseudo-labels.

Discussion. The semi-supervised classification method ReliaMatch solves the problems of underutilizing relationship between labeled and unlabeled data as well as introducing unreliable information into the model, and ReliaMatch also achieves desirable classification results in different networks and dataset application scenarios. Although ReliaMatch performs well in image semi-supervised classification issues, there are still some potential limitations: (1) The algorithm’s performance depends on the confidence threshold. The confidence threshold needs to be well-designed and controlled in both feature filtering module and the pseudo-label filtering module to get the best results. (2) Additionally, ReliaMatch may also be affected by the selection of model’s feature extraction and model types. Therefore, in future work, it can be considered to adaptively select appropriate confidence thresholds and hyper-parameters to improve and optimize ReliaMatch, so as to further enhance the existing methods and provide insights for future semi-supervised learning tasks.

5. Conclusions

In this paper, we proposed a reliable semi-supervised deep learning classification algorithm, i.e., ReliaMatch. The algorithm integrates a course label, a feature filter module, and a pseudo-label filter module, aiming at improving classification accuracy and algorithm reliability, focusing on key features, making better use of unlabeled data and avoiding the confirmation deviation of pseudo-labels. The course label improves the classification accuracy and the reliability of the algorithm. The feature filtering module eliminates unnecessary features, which makes the algorithm pay more attention to key features. The false label filtering module eliminates the confirmation deviation of false labels and filters out unreliable features and labels with low confidence, thus making the algorithm more stable and reliable. The experimental results show that ReliaMatch achieves the most advanced classification results on multiple datasets under the control of the confidence threshold.