DYMatch: Semi-Supervised Learning with Dynamic Pseudo Labeling and Feature Consistency

Abstract

A considerable number of approaches based on consistency regularization and pseudo-labeling have been proposed in Semi-supervised Learning (SSL) so far. These approaches significantly enhance the SSL methods by effectively utilizing large amount of unlabeled data to improve the model’s performance. However, existing methods may fail to utilize the unlabeled data more efficiently, mainly due to the challenges faced in pseudo-label estimation. In this paper, we begin by analyzing the impact of the pseudo-label estimation method on training the SSL model. Furthermore, we emphasize that an effective pseudo-label estimation method should reflect the difference of recognition performance among samples from different categories, and also ensure the maintenance of both high-quantity and high-quality pseudo-labels during each training iteration. Based on above analysis, we propose DYMatch, an innovative SSL method employing a dynamic estimation process. Firstly, a dynamic pseudo-label estimation method based on Gaussian mixture model is proposed to dynamically estimate the confidence threshold on different categories. Secondly, a feature-correlation consistency regularization method is introduced to further enhance the learning on unlabeled data. The experimental results show that DYMatch is a simple and effective SSL method, especially when the labeled data is limited.

1. Introduction

In the field of computer vision, deep learning methods have consistently been at the forefront, and their superior performance heavily relies on the availability of enough accurately annotated labeled data [1,2]. However, the acquisition of enough labeled data is a labor-intensive and expensive task. Now, there are various methods available to address such issues, such as weakly supervised learning [3,4], zero-shot learning [5], and semi-supervised learning [6]. Among them, semi-supervised learning offers an effective approach by employing a few labeled data and a large amount of unlabeled data for training. It shows great potential in practical applications and can significantly reduce the dependence of training on laborious data collection and data annotation [7,8,9,10,11,12].

The main challenge in SSL lies in effectively utilizing unlabeled data to improve the model’s generalization performance [6]. And consistency regularization [13,14,15] and pseudo-labeling [16,17] represent two most widely utilized paradigms designed for SSL, with their combined approaches also demonstrating good performance [9,10,18,19]. The key idea underlying these methods is grounded in the assumption that the training process of the SSL model should follow the low-entropy assumption, that is, the model should maintain the consistency in predictions when different perturbations are applied to the same unlabeled data [6]. However, a potential limitation in employing the pseudo-labeling methods to address this problem is the requirement to set a fixed confidence threshold [7,8,9,18] or implement a stage-wise threshold adjustment method [20] for the selection of suitable pseudo labels.

In pseudo-labeling methods designed with a fixed confidence threshold, the model selects unlabeled data with prediction scores surpassing the threshold for training and directly discard those falling below the threshold. For example, UDA [18] and FixMatch [9] employ a fixed high-confidence threshold to maintain the quality of the chosen pseudo labels. However, during the early training stages, a fixed high-confidence threshold (e.g., 0.99) may exclude too much unlabeled data from training, resulting in a large amount of unlabeled data being left unused. This approach overlooks the fact that, in the early training stages, the model has a limited ability to distinguish unlabeled samples, making it challenging to generate high-confidence predictions. An enhanced approach for the fixed threshold is to employ a stage-wise threshold adjustment strategy. For example, Dash [20] and AdaMatch [21] opt to gradually increase the threshold as the training progresses. The utilization of unlabeled data in such methods demonstrates an improvement compared to methods using a fixed confidence threshold. However, the performance of the stage-wise threshold adjustment strategy is directly influenced by the pre-defined hyper-parameters, and the setting of hyper-parameters is largely decoupled from the model’s training process in most cases. This is especially notable in disregarding the varying difficulties associated with learning data from different categories. Furthermore, optimal hyper-parameters have to be researched on different datasets. FlexMatch [10] attempts to employ different local (class-based) thresholds for samples from different categories. Although the setting of local threshold takes into account the different learning status of the model for different categories, it is still derived from a pre-defined fixed global (dataset-based) threshold. In essence, these methods fundamentally ignore the crucial effect of the model’s training process on the pseudo-label estimation, whether for global or local thresholds.

Essentially, the core requirement for a pseudo-label estimation method is to maintain a balance between the quantity and quality of pseudo labels during the training process. The methods utilizing a high fixed threshold explicitly abandon the requirement for the quantity, concentrating solely on generating high-quality pseudo labels. However, when the labeled data are extremely rare, the challenge with the high fixed threshold lies in its difficulty in ensuring the quality of pseudo labels. Conversely, the stage-wise threshold adjustment method is equivalent to compromising the quality of pseudo labels in favor of increasing their quantity. Such a strategy is susceptible to introducing incorrect pseudo labels, leading to the cognitive bias during the training process. Therefore, a novel dynamic pseudo-labeling estimation method based on Gaussian mixture model is proposed in this paper. We assume that the prediction scores for a specific category in a training iteration are sampled from a Gaussian mixture distribution comprising two distributions, namely the positive and negative distributions. This assumption can be extended to the whole unlabeled data. By solving for this Gaussian mixture model, the model can assign each category a local threshold that aligns with the current training status. Specifically, for a given category, we consider the prediction scores of this category as the weighted sum of the positive and negative distribution, and employ the maximum likelihood estimation to predict the parameters of each Gaussian distribution. Consequently, the dynamic pseudo-label estimation method can adaptively generate the optimal local threshold at different training iterations so as to stabilize the quantity and quality of pseudo labels. Intuitively, during the early training stages, the dynamic pseudo-label estimation method employs a relatively low threshold to encourage the model to train on more unlabeled data and accelerate the convergence. As the classification performance improves and the prediction confidence increases, a higher threshold is generated to filter out incorrect pseudo labels, alleviating the cognitive bias of the model.

Inspired by previous researches [22,23], we recognize the significance of constraining the training process of feature representation in SSL tasks. Therefore, in DYMatch, we also propose a feature-correlation consistency regularization method, and its integration with the dynamic pseudo-label estimation method can effectively enhance the model’s classification performance. Unlike other feature-constraint methods, the feature-correlation consistency regularization method is selectively applied to the unlabeled data with prediction scores exceeding the confidence threshold generated by the dynamic pseudo-label estimation method. Moreover, we implement feature-correlation consistency regularization and dynamic pseudo-label estimation at different layers in the model, thereby effectively avoiding the influence of the strong coupling between these two methods on the model’s performance.

The paper is structured as follows. Section 2 reviews the common pseudo-labelling and consistency regularization methods in SSL, and also introduces the relevant background on Gaussian mixture model and its applications in SSL. Section 3 details the motivation and implementation of the DYMatch. In Section 4, we present the results of the comparative experiments and ablation studies. The paper is ultimately summarized in Section 5.

2. Related Works

2.1. Consistency Regularization and Pseudo-Labeling Methods in SSL

The SSL task addressed in this paper can be defined as a classification task with $C$ categories. We define $\mathcal{X}=\{\left({\mathrm{X}}_b,p_b\right):b\in (1,\cdots ,B\left)\right\}$ as the batches of labeled training samples, where ${\mathbf{X}}_b$ and $p_b$ represent the $b$-th batch of labeled samples and the labels corresponding to these samples, respectively. $B$ represents the number of these batches, and $b$ denotes the index of an unlabeled sample in a batch. In addition, let $\mathcal{U}=\{{\mathrm{U}}_b:b\in (1,\cdots ,B\left)\right\}$ be the batches of unlabeled data. We use $N_L$ and $N_U$ to denote the number of labeled and unlabeled samples in each batch of training data, typically with $N_U{=\sigma N}_L$. $P_{model}(y|z_i;\theta )$ is denoted as the prediction score by the model $p$ for input $z_i$ with parameters $\theta$, where $z_i$ represents the $i$-th sample in a training batch.

In SSL, the motivation of the consistency regularization often follows the smoothness assumption or the low-density assumption [6], which means that the model should generate similar predictions for a specific unlabeled sample when subjected to various perturbations. Therefore, the consistency regularization method can be viewed to find a smooth manifold on which the whole dataset lies by utilizing the unlabeled data [24]. A common strategy for introducing perturbations is employing data augmentation methods [25,26], which apply image transformations while preserving the semantic information of the data. Specifically, consistency regularization demands that an unlabeled sample $u_i$ should exhibit similarity to its corresponding augmented sample, i.e., $Augment\left(u_i\right)$. Therefore, most SSL models employ the consistency regularization loss term in Equation (1) to a batch of unlabeled data:

$$\frac{1}{N_U}\sum _{\mathrm{i}=1}^{N_U}||P_{model}\left(y|Augmen{t}_1\left(u_i\right);\theta \right)-P_{model}(y|Augmen{t}_2\left(u_i\right);\theta ){||}_2^2$$

(1)

where $Augmen{t}_1\left(·\right)$ and $Augmen{t}_2\left(·\right)$ represent two different data augmentation operations, $u_i$ denotes the $i$-th unlabeled sample in a batch. Mean Teacher approach [27] contains a teacher model and a student model. The student model closely resembles the regular mode of Π-Model [13], and the teacher model has the same network structure as the student model but employs an exponentially moving average of the student’s weights for parameter update. In Mean Teacher, the predictions of the student model and the teacher model are used to calculate the consistency regularization loss. Specifically, $Augmen{t}_2\left(·\right)$ in Equation (1) is defined as the predictions of the model updated with the Exponential Moving Average (EMA) method. This operation yields a relatively stable feature representation, significantly improving the classification performance. Dual Student [28] is an enhanced teacher-student model that overcomes the effect of the tightly coupled teacher-student models by introducing another student model to replace the teacher model. These two student models have different initial states and are trained with distinct optimization strategies during the training process. Furthermore, Dual Student also introduces two innovative concepts, that is, stable sample and stabilization constraint, to prevent performance bottlenecks associated with a coupled EMA teacher-student model. Virtual Adversarial Training (VAT) [29] proposed the concept of adversarial training based on consistency regularization, which aims to generate adversarial transformations of unlabeled data to perturb the predictions of the model. Essentially, it is a training strategy where $Augmen{t}_2\left(·\right)$ in Equation (1) represents an adversarial transformation. Consistency regularization method is performed between the predictions of the original unlabeled data and the perturbed one. UDA [18] can be viewed as an unsupervised data augmentation method that analyzes the contribution of noise perturbations methods on consistent regularization and replaces simple perturbations techniques with advanced data augmentation methods, such as RandAugment [25] and AutoAugment [26]. Within the consistency regularization framework, UDA extends the data augmentation method from full-supervised learning to SSL. In this paper, DYMatch also utilizes a strongly-weakly data augmentation method, similar to FixMatch [9], to introduce perturbations to the unlabeled data. However, compared to the feature-correlation consistency regularization applied by DYMatch in the feature space, the existing consistency regularization method tends to impose consistency regularization on model’s predictions. The strong coupling with the pseudo-labeling in these approaches may restrict the model’s classification performance.

The difference between pseudo-labeling and consistency regularization methods is that consistency regularization methods typically rely on constraints imposed by various data transformations. In contrast, the pseudo-labeling method relies more on obtaining good pseudo labels with a high confidence, and adding them to the training set as labeled data. By defining $q_i=P_{model}(y|u_i)$, the loss function for unlabeled data can be derived as follows:

$$\frac{1}{N_U}\sum _{i=1}^{N_U}\mathbb{l}(\mathrm{m}\mathrm{a}\mathrm{x}(q_i)\ge \eta )H({\widehat{q}}_i,q_i)$$

(2)

where ${\widehat{q}}_i=argmax\left(q_i\right)$, $\eta$ is a hyper-parameter known as the confidence threshold, $H(·)$ is the standard cross-entropy loss function, and $\mathbb{l}$ is a mask function based on the $\eta$. Additionally, the $arg max$ operation is used to generate a “one-hot” distributions based on the model’s predictions. In Pseudo-Label [16], the model is firstly trained in the full-supervised approach on labeled data. Subsequently, the identical model is employed to predict the unlabeled data, and the predictions with the maximum confidence are viewed as the pseudo labels. In FlexMatch [10] the Course Pseudo-Labeling method (CPL), a curriculum learning method, is employed to learn the unlabeled data based on the model’s training status. Specifically, the significant contribution of CPL lies in its flexibly adjustment of the thresholds for each category during the training process, allowing more unlabeled data used for training. The dynamic pseudo-label estimation method in DYMatch is based on the model’s training status. It dynamically selects local thresholds for different categories, and simultaneously estimating global thresholds. This strategy effectively ensures that the model can achieve the optimal combination of pseudo labels in terms of both quantity and quality at any training iteration.

2.2. Gaussian Mixture Model and Its Application in SSL

The Gaussian Mixture Models (GMM) are commonly employed in machine learning problems to address clustering problems. This method belongs to mixture models, which assumes that the observed samples with known labels are sampled from a specific multivariate gaussian distribution. Specifically, the GMM assumes that the observed samples in the dataset belong to $K$ different categories. However, due to the lack of label information of the samples, it is necessary to employ the prior probability $p_k$ to denote the likelihood that the sample belonging to a category and the joint distribution corresponding to this category is modeled as a multivariate normal distribution $N({\mu }_k,{\sum }_k)$. Once obtaining the parameter estimates of the model, the posterior distribution of an observed sample can be calculated, ultimately determining its category. In theory, the GMM is capable of fitting dataset with arbitrary distributions, and it shows better performance when handling dataset modeled as the multivariate Gaussian distribution clusters [30]. In this paper, the model’s prediction scores for the samples are continuous variables. Therefore, the GMM model is selected to model and analyze them.

Within the SSL tasks, there is a crucial task known as noise label learning, also referred to as robust training [31]. In this task, GMM is an important and effective method for estimating noise rates [32,33]. The estimated noise rate is widely used to reweight samples in robust classifiers [34,35] or determine the quantity of samples considered as clean training samples [36,37,38].

A novel SSL framework, DivideMix [39], has been proposed to deal with the challenge of training with noisy labels. Firstly, DivideMix proposes the concept of co-division, a process of training two networks simultaneously. The GMM is employed to fit the loss distribution of the training data, which aims to partition the training data into labeled and unlabeled sets. The labeled dataset can be considered as the samples with correct labels, while the unlabeled data can be regarded as the samples with noisy labels. The separated datasets are then used to train an SSL model. By iteratively repeating the previous process, the model gradually overcome the influence of noisy labels. Inspired by DivideMix, DYMatch integrates this dynamic estimation method into semi-supervised classification learning. Specifically, in DYMatch, the dynamic estimation method is employed to yield the local threshold for a specific category, which effectively improves utilization of unlabeled data.

3. DYMatch

The structure diagram of unlabeled data in DYMatch is shown in Figure 1. In this section, we provide a detailed description of how the DYMatch model deals with semi-supervised classification task. Additionally, we analyze DYMatch’s two core components, namely, the dynamic pseudo-label estimation method based on the Gaussian Mixture Model and the feature-correlation consistency regularization method.

3.1. Motivation of Dynamic Pseudo-Label Estimation Method

The core idea of the pseudo-labeling method is to integrate more unlabeled data into the training set by employing appropriate confidence threshold, thereby enriching the distribution of the training set and effectively improving the classification and generalization performance of the model. In this section, inspired by Wang et al. [12], we consider the model’s prediction scores for a specific category as a binary classification problem. The analysis of this binary classification problem provides motivation for the design of dynamic pseudo-label estimation method.

Assuming a binary classification problem, where the true distribution is a uniform mixture of two Gaussian distributions, that is, the sample set $Xs$ contains positive samples ($+1$) and negative samples ($-1$). The input data $X$ follows the following conditional distribution:

$$\begin{gathered} X|\mathrm{Y}=-1~N({\mu }_1,{\sigma }_1^2) \\ X|\mathrm{Y}=+1~N({\mu }_2,{\sigma }_2^2) \end{gathered}$$

(3)

Furthermore, we assume that ${\mu }_2>{\mu }_1$. Given that binary classification models often employ a $Sigmoid(·)$ function as the output activation function, the prediction score of the model can be defined as:

$$s(x)=\frac{1}{1+{\mathrm{e}\mathrm{x}\mathrm{p}}^{-\beta (x-\frac{{\mu }_1+{\mu }_2}{2})}}$$

(4)

where $\beta$ is used to denote the current training status of the model. Typically, during the training process, the model should gradually become more confident for its classification performance, so $\beta$ is a gradually increasing positive parameter. $\left({\mu }_1+{\mu }_2\right)/2$ is the Bayes optimal linear decision boundary. In the early training stages, $\beta$ is relatively small. The decision boundary is around the input data $x$, and $s\left(x\right)$ may be closer to 0.5, indicating that the model fails to generate a confident prediction for the input data $x$. As the model becomes more confident, $\beta$ is expected to gradually grow during training, and the decision boundary gradually moves away from the input data $x$. When $x-\left({\mu }_1+{\mu }_2\right)/2$ is positive and large enough, $s\left(x\right)$ tends towards 1. In this case, the input data $x$ is viewed as a positive sample. And when $x-\left({\mu }_1+{\mu }_2\right)/2$ is negative and its absolute value is large enough, $s\left(x\right)$ will approach 0. At this time, the model assigns the input data x to the negative class.

Based on the above analysis, we can employ a fixed threshold $\eta \in \left(\mathrm{0.5,1}\right)$ to partition the prediction scores. The input data $x$ will be assigned as pseudo label 1, when $x$ satisfies the following conditions:

$$s\left(x\right)=\frac{1}{1+{\mathrm{e}\mathrm{x}\mathrm{p}}^{-\beta (x-\frac{{\mu }_1+{\mu }_2}{2})}}>\eta$$

(5)

And the Equation (5) can be rewritten as:

$$x>\frac{{\mu }_1+{\mu }_2}{2}+\frac{1}{\beta }log(\frac{\eta }{1-\eta })$$

(6)

Likewise, $x$ will be assigned as pseudo label −1 if:

$$s\left(x\right)=\frac{1}{1+{\mathrm{e}\mathrm{x}\mathrm{p}}^{-\beta (x-\frac{{\mu }_1+{\mu }_2}{2})}}<1-\eta$$

(7)

and the Equation (7) is equivalent to:

$$x>\frac{{\mu }_1+{\mu }_2}{2}-\frac{1}{\beta }log(\frac{\eta }{1-\eta })$$

(8)

Finally, when $1-\eta \le s\left(x\right)\le \eta$, the model cannot generate a high-confidence prediction for the input $x$. Generally, the utilization of unlabeled data is directly affected by the confidence threshold $\eta$. Specifically, as the confidence threshold $\eta$ increases, the utilization of unlabeled data gradually gets lower. Moreover, in the early training stages, when $\beta$ is small, indicating that the model cannot generate confident predictions, using a higher threshold may result in a lower utilization rate of the unlabeled data and a slower convergence.

If we integrate over $x$, the following conditional probability can be obtained:

$$P(Y_p=1 | Y=1)=\mathsf{\Phi }(\frac{\frac{{\mu }_2-{\mu }_1}{2}-\frac{1}{\beta }log(\frac{\eta }{1-\eta })}{{\sigma }_2})$$

(9)

$$P(Y_p=1 | Y=-1)=\mathsf{\Phi }(\frac{\frac{{\mu }_1-{\mu }_2}{2}-\frac{1}{\beta }log(\frac{\eta }{1-\eta })}{{\sigma }_1})$$

(10)

$$P(Y_p=-1 | Y=-1)=\mathsf{\Phi }(\frac{\frac{{\mu }_2-{\mu }_1}{2}-\frac{1}{\beta }log(\frac{\eta }{1-\eta })}{{\sigma }_1})$$

(11)

$$P(Y_p=-1 | Y=1)=\mathsf{\Phi }(\frac{\frac{{\mu }_1-{\mu }_2}{2}-\frac{1}{\beta }log(\frac{\eta }{1-\eta })}{{\sigma }_2})$$

(12)

where $\mathsf{\Phi }$ is the cumulative distribution function of a standard normal distribution.

If $P(Y=1)=P(Y=-1)=0.5$, we can obtain the following formula:

$$P(Y_p=1)=\frac{1}{2}\mathsf{\Phi }(\frac{\frac{{\mu }_2-{\mu }_1}{2}-\frac{1}{\beta }log(\frac{\eta }{1-\eta })}{{\sigma }_2})+\frac{1}{2}\mathsf{\Phi }(\frac{\frac{{\mu }_1-{\mu }_2}{2}-\frac{1}{\beta }log(\frac{\eta }{1-\eta })}{{\sigma }_1})$$

(13)

$$P(Y_p=-1)=\frac{1}{2}\mathsf{\Phi }(\frac{\frac{{\mu }_2-{\mu }_1}{2}-\frac{1}{\beta }log(\frac{\eta }{1-\eta })}{{\sigma }_1})+\frac{1}{2}\mathsf{\Phi }(\frac{\frac{{\mu }_1-{\mu }_2}{2}-\frac{1}{\beta }log(\frac{\eta }{1-\eta })}{{\sigma }_2})$$

(14)

When ${\sigma }_1\ne {\sigma }_2$, we can obtain that $P(Y_p=1)\ne P(Y_p=-1)$. Specifically, when the standard deviations (${\sigma }_1$ and ${\sigma }_2$) of the two normal distributions are unequal, it indicates that the samples of these two categories have different variability. In a binary classification task, this gap may affect the model’s decision boundary. The model may pay more attention to the category with the smaller standard deviation, as this means that the samples in that category are relatively more concentrated. As a result, $P(Y_p=1)$ and $P(Y_p=-1)$ are not equal. The model adjusts the decision boundaries according to the distribution properties of the different classes, making it easier to make more confident predictions for the category with the smaller standard deviation while being more cautious for the category with the larger standard deviation. This leads to unequal prediction performance of the model for these two classes. In fact, when using a larger confidence threshold $\eta$, the imbalance in the pseudo-label estimation method becomes more obvious. Imbalanced pseudo-label estimation method may distort decision boundaries and lead to cognitive bias in pseudo-labeling. An easy resolution for such situation is to use different local thresholds for different categories to estimation pseudo labels.

According to Equations (13) and (14), we can define that $a_1=1/2{\sigma }_1$, $a_2=1/2{\sigma }_2$, $b_1=log(\eta /(1-\eta \left)\right)/\beta {\sigma }_1$, $b_2=log(\eta /(1-\eta )/\beta {\sigma }_2$, $z={\mu }_2-{\mu }_1$, and $P(Y_p=1)+P(Y_p=-1)$ can be written as:

$$\begin{array}{ll}\hfill P(Y_p& =1)+P(Y_p=-1)\\ & =\frac{1}{2}\mathsf{\Phi }(a_{1}z-b_1)+\frac{1}{2}\mathsf{\Phi }({-a}_{1}z-b_1)+\frac{1}{2}\mathsf{\Phi }(a_{2}z-b_2)+\frac{1}{2}\mathsf{\Phi }(-a_{2}z-b_2)\end{array}$$

(15)

According to Equation (15), we can obtain that:

$$f\left(z\right)=\frac{1}{2}\mathsf{\Phi }(a_{1}z-b_1)+\frac{1}{2}\mathsf{\Phi }({-a}_{1}z-b_1)$$

(16)

By taking the derivative of $z$ in Equation (16), we can obtain:

$$f^{\prime }\left(z\right)=\frac{1}{2}{a}_1\left[\varphi (a_{1}z-b_1)-\varphi ({-a}_{1}z-b_1)\right]$$

(17)

where $\varphi$ is the probability density function of the standard normal distribution. According to its symmetry, Equation (17) can be rewritten as:

$$f^{\prime }\left(z\right)=\frac{1}{2}{a}_1\left[\varphi (a_{1}z-b_1)-\varphi (a_{1}z+b_1)\right]$$

(18)

Since $|a_{1}z-b_1|<|{-a}_{1}z-b_1|$, that is, $|a_{1}z-b_1|<|a_{1}z+b_1|$, the following conclusion can be derived: $\varphi \left(a_{1}z-b_1\right)>\varphi \left(a_{1}z+b_1\right)$. According to the above derivation, we can conclude that $f^{\mathrm{\text{'}}}\left(z\right)>0$, that is, $f\left(z\right)$ is monotonically increasing in the interval $(0,\mathrm{\infty })$. Extending this conclusion to Equation (15), we can obtain that $P(Y_p=1)+P(Y_p=-1)$ is also monotonically increasing. Therefore, the utilization rate of pseudo labels, $P(Y_p=1)+P(Y_p=-1)$, decreases as ${\mu }_2-{\mu }_1$ becomes smaller. Specifically, when the distributions of two categories are more similar, the model may face challenges in accurately distinguishing the samples in these two categories. As the differentiation of two categories diminishes, more samples may be confused in the feature space, and the model cannot generate confident predictions about these samples. Therefore, a suitable pseudo-label threshold is needed to balance the utilization rate between these categories. Otherwise, there may not be enough samples for training the model to distinguish categories that are already challenging to differentiate.

In summary, an effective pseudo-labeling method should take into account the change in the model’s classification performance between different categories during the training process. Specifically, the confidence threshold $\eta$ should gradually increase as the model’s performance parameter $\beta$ during the training process. This ensures enough unlabeled data for the model to train in the early training iterations, while, in the later iterations, the threshold $\eta$ can filter out incorrect predictions to mitigate the cognitive bias of the model. Moreover, given the model’s different classification performance in different categories, with some being easier to classify than others, the pseudo-labeling method need to adjust the threshold $\eta$ for each class to encourage equitable setting of threshold to different classes. The main contribution of this paper lies in dynamically adjusting class- based local thresholds and dataset-based global thresholds based on the model’s training status. Additionally, the combination of feature-correlated consistency regularization and dynamic pseudo-label estimation further enhances the performance of SSL model. And then, we will provide a detailed description of these two methods used in DYMatch.

3.2. Dynamic Pseudo-Label Estimation Method Based on Gaussian Mixture Model

During the training process, it is crucial to dynamically assign corresponding confidence thresholds to each class, and the model’s prediction scores for the training data can accurately reflect the current training status for the corresponding category. Therefore, in DYMatch, we propose a dynamic pseudo-label estimation method based on GMM. Firstly, utilizing the model prediction scores during training allows us to obtain class-based local thresholds, dynamically partitioning unlabeled data into positive and negative samples. The final dynamic local threshold is then updated using the Exponential Moving Average (EMA) method based on the local thresholds at each training iteration. Additionally, the global dynamic threshold is estimated by using the EMA of the prediction scores from the model, and the dynamic local threshold is also used to adjust the global dynamic threshold.

Therefore, at the beginning of training, the confidence threshold may be relatively smaller, enabling utilization of more unlabeled data with potentially correct predictions for training. As the training of model, the confidence thresholds are dynamically adjusted and generally tend to increase. This adjustment filters out more unlabeled data with potentially incorrect model predictions, reducing the training bias of the model.

3.2.1. Dynamic Local Confidence Threshold Estimation

Dynamic local confidence threshold estimation aims to estimate class-specific local thresholds to account for the inter-class diversity. Furthermore, the dynamic local confidence threshold is steadily increased during training to ensure the discarding of unlabeled data with incorrect prediction.

Therefore, the Gaussian Mixture model is employed to distinguish the positive from negative samples in model’s prediction scores for specific categories. Here, positive samples correspond to unlabeled data that the model predicts correctly, while negative samples correspond to unlabeled data with incorrect prediction. The model’s prediction scores used to distinguish positive and negative samples can be regarded as the local thresholds for the corresponding categories. Specifically, we assume that the prediction scores $s^c$ for category $c$ are sampled from a Gaussian mixture distribution $P\left(s^c\right)$ with two distributions, positive and negative. Local thresholds are dynamically generated by fitting a Gaussian Mixture Model to the prediction scores.

$$P\left(s^c\right)=w_n^{c}N(s^c|{\mu }_n^c,{\left({\sigma }_n^c\right)}^2)+w_p^{c}N(s^c|{\mu }_p^c,{\left({\sigma }_p^c\right)}^2)$$

(19)

where $N(\mu ,{\sigma }^2)$ denotes a Gaussian distribution. The parameters $w_n^c$, ${\mu }_n^c$, ${\left({\sigma }_n^c\right)}^2$ and $w_p^c$, ${\mu }_p^c$, ${\left({\sigma }_p^c\right)}^2$ represent the weight, mean and variance of the negative and positive sample distributions, respectively. Then, the EM algorithm is employed to infer the posterior probability $P(pos|s^c,{\mu }_p^c,{\left({\sigma }_p^c\right)}^2)$. This posterior probability can be directly used to generate the pseudo label, and the dynamic local threshold ${\tilde{\eta }}^l\left(c\right)$ corresponding to category $c$ can be defined as:

$${\tilde{\eta }}^l(c)=\underset{s^c}{\mathrm{argmax}}P(pos|s^c,{\mu }_p^c,{\left({\sigma }_p^c\right)}^2)$$

(20)

By using the EMA method, we obtain the final dynamic local threshold, which reflects the learning status of the model for a specific category:

$${\eta }_t^l(c)=\left\{\begin{array}{c}1/C , if t=0\\ \lambda {\tilde{\eta }}_{t-1}^l\left(c\right)+\left(1-\lambda \right){\tilde{\eta }}_t^l\left(c\right), otherwise\end{array}\right.$$

(21)

where $\lambda \in \left(\mathrm{0,1}\right)$ denotes the momentum decay in EMA, $t$ represents the $t$-th iteration during the training process. Meanwhile, ${\tilde{\eta }}_t^l\left(c\right)=DLTE\left({{DA(q}_i\left(c\right)}_s\right))$, where $DLTE(·)$ represents the dynamic local threshold estimation method based on GMM, and ${q_i\left(c\right)}_s$ denote the prediction scores of unlabeled data with class $c$. $DA(·)$ represents the distribution alignment strategy from ReMixMatch, which balance the distributions of prediction scores. The local thresholds are initialized to $1/C$, where $C$ represents the number of categories. Finally, ${\eta }_t^l=\left[{\eta }_t^l\right(1),{\eta }_t^l(2),\cdots ,{\eta }_t^l(c\left)\right]$ contains confidence thresholds for all categories.

In the implementation of the dynamic local threshold estimation method, we maintain a queue $m\left(c\right)\in {\mathbb{R}}^Q$ of prediction scores with dimension $Q$ for each category. The queues of all categories form a prediction memory bank $M=\left\{m\left(1\right),m\left(2\right),\cdots ,m\left(c\right)\right\},M\in {\mathbb{R}}^{C\times Q}$, where $C$ represents the number of categories. Specifically, the queue $m\left(c\right)$ stores the prediction scores $s^c$ of the unlabeled data predicted by the model as category $c$, and employs these prediction scores to fit a Gaussian mixture model. Subsequently, the dynamic threshold estimation method can dynamically adjust the threshold ${\eta }^l\left(c\right)$ corresponding to category $c$ based on the fitted Gaussian mixture model. As a result, ${\eta }^l\left(c\right)$ can align with the model’s classification ability at different training stages. Moreover, the EM algorithm has a negligible effect on training time and does not impose an additional burden to the training.

3.2.2. Dynamic Global Confidence Threshold Estimation

The global threshold estimation shares similar characteristics as for the local one, that is, they both reflect the training status of the model and maintain a steady increase during training. However, unlike the local threshold estimation, the global threshold estimation should capture the model’s training status for the entire dataset. Therefore, we denote the global threshold as the model’s average prediction score for the unlabeled data. Specifically, the global threshold is estimated as the EMA of the prediction scores in each training iteration. We initialize the global threshold to $1/C$, where $C$ represents the number of categories. The global threshold ${\eta }_t^g$ is defined as:

$${\eta }_t^g=\left\{\begin{array}{c}1/C , if t=0\\ \lambda {\eta }_{t-1}^g+(1-\lambda ) \frac{1}{N_U} {\sum }_{i=1}^{N_U}max\left(DA\right(q_i\left)\right) , otherwise\end{array}\right.$$

(22)

3.2.3. Dynamic Pseudo-Label Estimation

Having obtained the class-specific dynamic local thresholds and dataset-specific dynamic global thresholds, we employ dynamic local thresholds to adjust the dynamic global threshold, and obtain the final dynamic confidence threshold:

$$\begin{gathered} {\eta }_t\left(c\right)={\eta }_t^g\left(c\right)·Norm\left({\eta }_t^l\right(c\left)\right) \\ Norm({\eta }_t^l(c))=\frac{{\eta }_t^l\left(c\right)}{max\left\{{\eta }_t^l\right(c),c\in [C\left]\right\}} \end{gathered}$$

(23)

where $Norm\left(·\right)$ denotes the maximum normalization function. Finally, the loss function for dynamic pseudo-label estimation of unlabeled data at the $t$-th iteration can be defined as:

$${\mathcal{L}}_{pl}=\frac{1}{N_u}\sum _{i=1}^{N_u}\mathbb{I}\left(max\right(DA\left(q_i\right))>{\eta }_t(arg max\left({DA(q}_i\right)\left)\right))·H({DA(\widehat{q}}_i),{DA(\mathcal{Q}}_i))$$

(24)

where $argmax\left(q_i\right)$ represent the model’s predictions for an unlabeled sample, and ${\eta }_t\left(argmax\right(q_i\left)\right)$ is the confidence threshold corresponding to each category. $\mathbb{I}(·)$ is a mask function based on the confidence threshold. In addition, we employ ${\widehat{q}}_i$ and ${\mathcal{Q}}_i$ to represent $P_{model}\left(y|\mathcal{a}\left(u_i\right);\theta \right)$ and $P_{model}\left(y|\mathcal{A}\left(u_i\right);\theta \right)$, where $\mathcal{a}\left(·\right)$ and $\mathcal{A}\left(·\right)$ represent the weak augmentation and the strong augmentation method, respectively.

3.3. Consistency Regularization Method Based on Feature-Correlation

In methods like FixMatch, MixMatch and ReMixMatch, consistency regularization is typically performed on the prediction scores. The strong coupling between consistency regularization and the pseudo-label methods, which also rely on model’s prediction scores, may constrain the performance of the model. Therefore, in this paper, we perform the consistency regularization method on the feature maps fed to the classification module $g(·)$. And this strategy focuses on unlabeled data with prediction scores exceeding the confidence threshold. Specifically, for a selected unlabeled sample, we employ the negative cosine similarity to measure the correlation between the two different augmented versions of this unlabeled sample. The different data augmentations correspond to strong and weak data augmentations.

However, it should be emphasized that there exist differences between the strongly augmented and the weakly augmented versions of the same unlabeled data. Consequently, they cannot be considered to be identical for the feature extraction. Therefore, a learnable linear projection module, denoted as $h(·)$, is performed on the strongly augmented feature representation of the unlabeled data to weakening the constraint in the consistency regularization method. The module $h(·)$ consists of two consecutive linear layers, with a ReLU layer following the first linear layer:

$$h\left(x_i\right)=Linear\left(ReLU\right(Linear\left(x_i\right)\left)\right)$$

(25)

Finally, the loss function of the consistency regularization method based on feature-correlation can be defined as:

$${\mathcal{L}}_{cr}=\frac{1}{N_{mask}}\sum _{i=1}^{N_{mask}}(1-COS(h\left(f\right(\mathcal{A}\left(u_i\right)\left)\right),f\left(\mathcal{a}\left(u_i\right)\right)\left)\right)·Mask$$

(26)

where $COS(·)$ denotes the cosine similarity function, the $f\left(\mathcal{a}\left(u_i\right)\right)\in {\mathbb{R}}^{d\times 1}$ represents the feature representation of the weakly augmented unlabeled data before the classification module $g(·)$, and $f\left(\mathcal{A}\left(u_i\right)\right)\in {\mathbb{R}}^{d\times 1}$ represents the strongly augmented representation. In our consistency regularization method, $f\left(\mathcal{A}\left(u_i\right)\right)$ is fed to a learnable linear projection module $h\left(·\right)$, where $h:{\mathbb{R}}^{d\times 1}\to {\mathbb{R}}^{d\times 1}$. $Mask$ represents a mask vector where all unlabeled data with prediction scores exceeding the dynamic threshold are marked as 1, while others are marked as 0. $N_{mask}$ is the number of unlabeled data marked as 1 in the $Mask$.

3.4. Loss Function of DYMatch

In DYMatch, the final loss function for training the network is:

$$\mathcal{L}={\mathcal{L}}_{sup}+{\lambda }_{pl}{\mathcal{L}}_{pl}+{\lambda }_{cr}{\mathcal{L}}_{cr}$$

(27)

where ${\mathcal{L}}_{sup}$ represents the standard cross-entropy loss used for training with labeled data, ${\mathcal{L}}_{pl}$ and ${\mathcal{L}}_{cr}$ are the loss for dynamic pseudo-label estimation and feature correlation consistency regularization on unlabeled data, respectively. ${\lambda }_{pl}$ and ${\lambda }_{cr}$ are a fixed scalar hyper-parameter, representing the relative weight of ${\mathcal{L}}_{pl}$ and ${\mathcal{L}}_{cr}$, respectively. ${\mathcal{L}}_{sup}$ is defined as:

$${\mathcal{L}}_{sup}=\frac{1}{N_L}\sum _{i=1}^{N_L}H(l_i,P_{model}(y|{\mathcal{a}(x}_i)\left)\right).$$

(28)

In the loss function for unlabeled data, we use the loss term ${\mathcal{L}}_{cr}$ to obtain more discriminative feature representations, minimizing the difference between the feature representations of the weakly augmented and strongly augmented unlabeled data. This method ensures that the feature representations of the two different augmented samples remain consistency during the training process, thereby improving the model’s classification performance. Meanwhile, ${\mathcal{L}}_{pl}$ ensures a balance in the quantity and quality of pseudo labels.

4. Experiments

4.1. Datasets

The experiments were conducted on various object recognition datasets, such as CIFAR-10 [40,41], CIFAR-100 [40], SVHN [42], and STL10 [43], along with some domain adaptation datasets, in order to validate the performance of DYMatch.

The number of the training set and testing set in CIFAR-10 and CIFAR-100 datasets are 50,000 and 10,000, respectively. However, they consist of 10 and 100 categories, respectively. SVHN dataset includes 10 categories, with 73,257 samples used for training and 26,032 samples used for testing. STL-10 dataset is designed for evaluating SSL methods. It includes 5000 labeled samples and 100,000 unlabeled samples. Compared to other standard SSL datasets, STL-10 only has fewer labeled data, and the dataset compensates by offering a considerable amount of unlabeled data. Additionally, each category contains different proportions of labeled data and unlabeled data. These settings make the STL-10 dataset more challenging for SSL training and closer to real-world tasks.

Three domain adaptation datasets are also employed for the performance evaluation, including Office31 [44], Office-Home [41] and DomainNet [45]. The Office31 dataset comprises 4110 images distributed across three domains, which are Amazon, Webcam and Dslr. And these three domains share 31 object categories commonly found in offices. The Office-Home dataset consists of 64 object categories sampled from four domains, which are denoted as Artistic, Clip Art, Product, and Real-World, respectively. And Office-Home comprises around 15,500 images. The DomainNet dataset, containing 345 categories of common objects from six domains, is the largest domain adaptation dataset.

According to the standard settings used in SSL datasets, we randomly select a specific number of labeled samples to constitute the small part of the training set. The remaining data, discarding the labels, forms another part of the training set. In DYMatch, both weak augmentation and strong augmentation methods are employed to the unlabeled data. The weak augmentation involves flip-and-shift transformations, while the strong augmentation combines the RandAugment [25] method with the Cutout [46] method.

4.2. Experimental Settings

The baseline models employed for the comprehensive performance comparison encompassed Mean Teacher [27], Pseudo-Label [16], MixMatch [7], ReMixMatch [8], UDA [18], FixMatch [9], FlexMatch [10], DoubleMatch [23], FeatMatch [22], Meta pseudo-labelling [19], SimMatch [47], Semi-Clustering [48], SoftMatch [11] and FreeMatch [12]. The Mean Teacher and Pseudo-Label, respectively, employed the consistency regularization method and the pseudo-labeling method, which have become widely used in the SSL approach. MixMatch and UDA both explore the impact of different data augmentation methods on SSL. Based on MixMatch, ReMixMatch suggests that achieving a balance in the distribution of prediction scores can significantly enhance the model’s classification performance. FixMatch stands out as an attempt to combine consistency regularization and pseudo-labeling methods, resulting in good classification performance. DoubleMatch and FeatMatch innovatively refined the consistency regularization method, taking inspiration from FixMatch. Meanwhile, FlexMatch also enhanced the pseudo-labeling method upon the foundation laid by FixMatch. SoftMatch and FreeMatch take into account the impact of the model’s training status for the confidence threshold. Meta pseudo-labeling introduced an innovative training diagram. SimMatch and Semi-Clustering strive to integrate methods from other fields, including self-supervised and deep clustering strategies, into FixMatch. The hyper-parameters of all baseline methods are consistent with those in their corresponding papers.

To ensure a fair comparison, we followed the guidelines recommended in [49] and shared the same training pipelines for the same dataset, which include optimizer, learning rate decay schedule and the backbone module. In our experiments on all benchmark datasets, the model was trained employing the SGD optimizer. The momentum and weight decay in SGD were set to 0.9 and 1 × 10⁻³, respectively. The Nesterov method was not utilized in SGD. The learning rate was initialized at 0.02 and gradually decreased employing the cosine annealing scheduler. The Wider ResNet network [50] was employed as the backbone module. In detail, we employed Wider ResNet-28-2 for CIFAR-10, SVHN and the domain adaptation datasets. For the CIFAR-100 and STL-10 datasets, the Wider ResNet-28-8 and the Wider ResNet-37-2 networks are employed as the backbone, respectively. Moreover, we show the average and the standard deviation of the results for each model by trained three times on each number of labeled data on each dataset.

DYMatch involves five hyper-parameters: the dimension of the queue in the dynamic local threshold estimation ($Q$), the momentum decay in the EMA method ($\lambda$), the dimension of the output feature dimension in linear projection module $h(·)$ ($d$) and the relative weight hyper-parameters (${\mathcal{L}}_{pl}$ and ${\mathcal{L}}_{cr}$). In the process of training, $\lambda$ and $Q$ were set as 0.999 and 100, respectively. In most cases, ${\lambda }_{pl}$ and ${\lambda }_{cr}$ were set to 2.0. However, when only a minimal amount of labeled data is available (i.e., 40 labeled data for CIFAR-10, SVHN and STL-10 datasets, 400 labeled data for CIFAR-100), ${\lambda }_{pl}$ and ${\lambda }_{cr}$ were adjusted to $3.0$. This means that in these situations with limited labeled data, we should pay more attention to the constraints for the training on the unlabeled data. Moreover, the different backbones were employed for the different datasets. Thus, for CIFAR-10 and SVHN datasets, the $d$ in $h(·)$ was set to 128. In CIFAR-100, the parameter $d$ was set to 512, and for STL-10, the parameter $d$ was set to 256. The labeled batch size $N_L$ is set to 64. We set the $\sigma$ as 7, which means that the unlabeled batch size $N_U$ is set to 7 times of $N_L$ for all benchmark datasets.

4.3. Results

4.3.1. Experimental Results for Standard SSL Datasets

In this subsection, we presented the classification performance of DYMatch with other baseline methods on the standard SSL datasets from

Table 1. The Error Rates for CIFAR-10 with Five Different Numbers of Labelled Images. The best result is in bold.

	CIFAR-10
Model	40 Labels	250 Labels	500 Labels	2000 Labels	4000 Labels
Mean Teacher [27]	70.09 ± 0.26	47.33 ± 4.71	42.01 ± 5.86	12.17 ± 0.22	9.19 ± 0.19
Pseudo-Label [16]	74.61 ± 1.60	49.98 ± 1.17	40.55 ± 1.70	21.96 ± 0.42	16.21 ± 0.11
MixMatch [7]	36.19 ± 6.48	11.08 ± 0.87	9.65 ± 0.94	7.03 ± 0.15	6.24 ± 0.06
ReMixMatch [8]	9.88 ± 1.03	6.27 ± 0.34	6.04 ± 0.04	5.53 ± 0.18	5.14 ± 0.04
UDA [18]	10.62 ± 3.75	5.43 ± 0.96	4.80 ± 0.09	4.73 ± 0.14	4.32 ± 0.08
FixMatch [9]	7.47 ± 0.28	5.07 ± 0.65	4.85 ± 0.06	4.54 ± 0.12	4.26 ± 0.05
FlexMatch [10]	4.97 ± 0.06	4.98 ± 0.49	4.47 ± 0.34	4.18 ± 0.12	4.19 ± 0.01
Semi-Clustering [48]	7.39 ± 0.61	5.51 ± 0.25	5.40 ± 0.23	4.71 ± 0.12	4.62 ± 0.09
DoubleMatch [23]	14.02 ± 5.71	5.56 ± 0.42	5.12$\pm$0.43	4.66 ± 0.23	4.65 ± 0.17
FeatMatch [22]	7.88 ± 0.52	7.50 ± 0.64	6.21 ± 0.11	4.95 ± 0.20	4.91 ± 0.18
MPL [19]	6.62 ± 0.91	5.76 ± 0.26	5.89 ± 0.23	4.40 ± 0.04	3.89 ± 0.07
SimMatch [47]	5.18 ± 0.15	4.99 ± 0.13	4.62 ± 0.21	4.22 ± 0.14	4.31 ± 0.01
SoftMatch [11]	5.06 ± 0.02	4.84 ± 0.10	4.66 ± 0.12	4.33 ± 0.10	4.27 ± 0.02
FreeMatch [12]	4.97 ± 0.09	4.85 ± 0.10	4.57 ± 0.10	4.21 ± 0.05	4.14 ± 0.02
DYMatch (ours)	4.89 ± 0.11	4.80 ± 0.07	4.53 ± 0.08	4.15 ± 0.11	3.96 ± 0.02

,

Table 2. The Error Rates for SVHN with Five Different Numbers of Labelled Images. The best result is in bold.

			SVHN
Model	40 Labels	250 Labels	500 Labels	1000 Labels	4000 Labels
Mean Teacher [27]	36.09 ± 3.98	6.45 ± 2.43	3.82 ± 0.17	3.75 ± 0.10	3.39 ± 0.11
Pseudo-Label [16]	64.61 ± 5.60	21.16 ± 0.88	14.35 ± 0.37	10.19 ± 0.41	5.71 ± 0.07
MixMatch [7]	30.60 ± 8.39	3.78 ± 0.26	3.64 ± 0.46	3.27 ± 0.31	2.89 ± 0.06
ReMixMatch [8]	24.04 ± 9.13	3.10 ± 0.50	3.02 ± 0.33	2.83 ± 0.30	2.42 ± 0.09
UDA [18]	5.12 ± 4.27	2.74 ± 2.76	2.55 ± 0.09	2.35 ± 0.07	2.28 ± 0.10
FixMatch [9]	3.81 ± 1.18	2.48 ± 0.38	2.35 ± 0.05	2.28 ± 0.11	2.20 ± 0.09
FlexMatch [10]	8.19 ± 3.20	6.59 ± 2.29	6.79 ± 1.20	6.72 ± 0.30	4.45 ± 0.12
Semi-Clustering [48]	3.09 ± 0.54	2.30 ± 0.03	2.29 ± 0.03	2.26 ± 0.12	2.15 ± 0.12
DoubleMatch [23]	16.50 ± 13.73	2.41 ± 0.53	2.50 ± 0.23	2.25 ± 0.09	2.15 ± 0.09
FeatMatch [22]	4.01 ± 0.13	3.34 ± 0.19	3.08 ± 0.12	3.10 ± 0.06	2.62 ± 0.08
MPL [19]	9.33 ± 8.02	2.29 ± 0.03	2.30 ± 0.12	2.28 ± 0.07	2.18 ± 0.12
SimMatch [47]	3.19 ± 0.74	2.26 ± 0.21	2.14 ± 0.20	2.08 ± 0.04	2.97 ± 0.07
SoftMatch [11]	2.31 ± 0.00	2.15 ± 0.05	2.16 ± 0.10	2.08 ± 0.04	1.93 ± 0.00
FreeMatch [12]	3.79 ± 0.03	4.09 ± 0.66	4.38 ± 0.07	4.31 ± 0.00	2.67 ± 0.10
DYMatch (ours)	2.26 ± 0.13	2.18 ± 0.11	2.11 ± 0.02	2.04 ± 0.03	1.90 ± 0.01

,

Table 3. The Error Rates for CIFAR-100 with Four Different Numbers of Labelled Images. The best result is in bold.

	CIFAR-100
Model	400 Labels	1000 Labels	2500 Labels	10,000 Labels
Mean Teacher [27]	-	60.99 ± 0.21	53.91 ± 0.57	36.21 ± 0.19
Pseudo-Label [16]	-	62.78 ± 0.33	57.38 ± 0.46	36.21 ± 0.19
MixMatch [7]	67.61 ± 1.32	54.73 ± 0.24	39.94 ± 0.37	28.31 ± 0.33
ReMixMatch [8]	44.28 ± 2.06	37.86 ± 0.55	27.43 ± 0.31	23.03 ± 0.56
UDA [18]	59.28 ± 0.88	45.20 ± 0.71	33.13 ± 0.22	24.50 ± 0.25
FixMatch [9]	48.85 ± 1.75	40.55 ± 0.09	28.29 ± 0.11	23.03 ± 0.56
FlexMatch [10]	39.94 ± 1.62	38.44 ± 1.20	26.49 ± 0.20	21.90 ± 0.15
Semi-Clustering [48]	66.57 ± 0.23	54.01 ± 0.12	44.35 ± 0.03	35.99 ± 0.23
DoubleMatch [23]	42.61 ± 1.15	42.05 ± 0.34	27.47 ± 0.19	21.69 ± 0.26
FeatMatch [22]	48.70 ± 0.42	41.85$\pm 0.23$	30.43 ± 0.04	26.83 ± 0.04
MPL [19]	44.49 ± 0.99	38.13 ± 0.23	27.43 ± 0.22	22.79 ± 0.18
SimMatch [47]	48.82 ± 1.51	37.12 ± 0.52	32.54 ± 0.27	26.42 ± 0.18
SoftMatch [11]	49.64 ± 1.46	36.42 ± 0.34	33.05 ± 0.05	27.26 ± 0.03
FreeMatch [12]	49.24 ± 2.16	37.15 ± 0.44	32.79 ± 0.21	27.17 ± 0.11
DYMatch (ours)	44.03 ± 0.62	35.42 ± 0.25	31.13 ± 0.10	20.34 ± 0.08

and

Table 4. The Error Rates for STL10 on Two Different Numbers of Labelled Images. The best result is in bold.

	STL10
Model	40 Labels	1000 Labels
Mean Teacher [27]	-	21.43 ± 2.39
Pseudo-Label [16]	74.76 ± 0.99	27.99 ± 0.80
MixMatch [7]	54.93 ± 0.96	10.41 ± 0.61
ReMixMatch [8]	32.12 ± 6.24	5.23 ± 0.45
UDA [18]	37.42 ± 8.44	7.66 ± 0.56
FixMatch [9]	35.97 ± 4.14	5.17 ± 0.63
FlexMatch [10]	29.15 ± 4.16	5.77 ± 0.23
Semi-Clustering [48]	40.90 ± 0.23	4.78 ± 0.29
DoubleMatch [23]	34.34 ± 0.34	4.46 ± 0.20
FeatMatch [22]	51.26 ± 0.33	6.94 ± 0.32
MPL [19]	35.76 ± 3.29	6.45 ± 0.29
SimMatch [47]	37.57 ± 0.23	5.63 ± 0.35
SoftMatch [11]	21.42 ± 3.48	5.73 ± 0.24
FreeMatch [12]	15.56 ± 0.55	5.63 ± 0.15
DYMatch (ours)	17.58 ± 0.67	5.35 ± 0.12

. Results for Mean Teacher and Pseudo-Label with 40 labels per class are not included due to their poor performance on CIFAR-100. MPL, SoftMatch and FreeMatch obtained the best classification performance with few specific settings of the number of labeled data. However, DYMatch demonstrates good classification performance, outperforming other baseline methods on most labeled data settings on each standard SSL dataset.

As shown from Table 1, Table 2 and Table 3, with the results on the CIFAR-10, CIFAR-100, and SVHN datasets, and DYMatch consistently obtains the good classification performance on most different settings of the numbers of labeled data. In experiments with 40 labeled data per class on CIFAR-10, the error rate of DYMatch was slightly higher than MPL by 0.07%, but it was lower than other baseline methods. DYMatch also performed worse than MPL, when employing 2500 labeled samples on CIFAR-100. Furthermore, as the quantity of labeled data decreases, the DYMatch gradually demonstrates its robust classification performance. When utilizing a very small number of labeled samples, for example, only 4 labels per class, DYMatch exhibited the lowest error rates, which were 1.73% and 0.46% lower than MPL on CIFAR-10 and CIFAR-100 datasets, respectively. The classification accuracy of DYMatch was marginally lower by 0.03% than that of SoftMatch when 250 labeled samples were used on SVHN. However, with utilizing other settings of the number of the labeled data, DYMatch can achieve better performance compared to SoftMatch. In general, most models obtained better classification performance on the SVHN dataset, resulting in a relatively small performance gap among some outstanding models.

As shown in Table 4, for the STL-10 dataset, DYMatch still achieved very competitive classification performance. DYMatch can obtain the lowest error rate when using 1000 labeled data. And when only 40 labeled data are available, DYMatch’s error rate is only 2.02% higher than FreeMatch.

4.3.2. Experimental Results for Domain Adaptation Datasets

Domain adaptation datasets pose a challenge for training due to the presence of multiple domains within each category and significant difference existing among these domains. This intrinsic gap complicates the process of obtaining an optimal performance even with fully-supervised training. Typically, the domain adaptation datasets tend to be more complex and challenging in the field of SSL.

Initially, we create the training and testing sets from each of these datasets following the strategy employed in [51]. The proposed DYMatch was evaluated, along with FlexMatch, ReMixMatch, FixMatch, DoubleMatch, SimMatch and SoftMatch on these datasets and compared the classification performance. For a fair comparison, the training pipelines applied to these domain adaptation datasets was the same as that employed in SSL benchmark datasets. We set the weight of loss term for the unlabeled data followed the settings used in the corresponding original paper. According to

Table 5. The Error Rates for Other Object Recognition Datasets with Different Numbers of Labelled Images. (The numbers of data for each category contained in each domain adaptation dataset are given). The best result is in bold.

	Office31 (31)	Office-Home (65)		DomainNet (345)
Model	10 Labels per Class	10 Labels per Class	50 Labels per Class	50 Labels per Class	500 Labels per Class
ReMixMatch [8]	30.12 ± 0.33	68.69 ± 0.07	46.42 ± 0.36	82.54 ± 0.11	76.23 ± 0.23
FixMatch [9]	28.36 ± 0.20	55.73 ± 0.23	46.27 ± 0.23	79.12 ± 0.03	73.79 ± 0.15
FlexMatch [10]	25.13 ± 0.33	54.56 ± 0.30	43.99 ± 0.03	66.12 ± 0.15	53.45 ± 0.23
DoubleMatch [23]	26.30 ± 0.30	56.45 ± 0.13	45.25 ± 0.13	70.32 ± 0.32	55.43 ± 0.12
SimMatch [47]	26.09 ± 0.12	53.65 ± 0.09	45.01 ± 0.04	65.95 ± 0.20	51.33 ± 0.04
SoftMatch [11]	22.50 ± 0.32	52.66 ± 0.14	40.91 ± 0.12	64.24 ± 0.18	50.13 ± 0.05
DYMatch (ours)	20.31 ± 0.26	52.29 ± 0.26	40.42 ± 0.10	63.81 ± 0.10	50.93 ± 0.07
Fully-Supervised	13.54	32.48		43.37

, an accuracy of 56.63% can be obtained through the fully-supervised training on the most complex domain adaptation dataset, DomainNet. All semi-supervised learning methods included in the comparison struggle to achieve comparable classification performance, showing the distinct performance gaps. When facing with only 500 labeled data for each category on DomainNet dataset, DYMatch exhibits an accuracy slightly lower than SoftMatch by 0.8%. However, in all other settings, DYMatch consistently demonstrates the best classification performance.

4.4. Ablation Study

Since DYMatch combines two effective methods, i.e., the dynamic pseudo-label estimation method based on the Gaussian mixture model and the consistency regularization method based on feature correlation, the ablation studies were conducted to provide a deeper understanding of the factors contributing to DYMatch’s good performance. In this subsection, the results are presented only for the experiments involving 40 and 4000 labeled data on CIFAR-10.

As shown in

Table 6. The Results of the Ablation Study on the Influence of the dynamic pseudo-label estimation method and the feature-correlation-based consistency regularization method on DYMatch (PL represents the dynamic pseudo-label estimation method based on Gaussian mixture model, and CR represents the consistency regularization method based on feature correlation). The best result is in bold.

	Methods		CIFAR-10
Model	CR	PL	40 Labels	4000 Labels
DYMatch_PL	✓	✗	8.16 ± 0.06	5.26 ± 0.05
DYMatch_CR	✗	✓	10.58 ± 0.16	8.41 ± 0.24
DYMatch	✓	✓	4.93 ± 0.11	3.96 ± 0.02

, the error rates of DYMatch_PL (i.e., DYMatch with only dynamic pseudo-label estimation method based on the Gaussian mixture model) were lower than those of DYMatch_CR (i.e., DYMatch with only consistency regularization method based on feature correlation), but the combination of these two methods led to the better performance of DYMatch than DYMatch_PL and DYMatch_CR.

The comparison between DYMatch_CR and DYMatch_PL highlights their differences in utilizing the unlabeled data. In DYMatch_CR, the feature maps were used after the backbone $f\left(·\right)$ for feature-correlation consistency regularization. However, the lack of selection on unlabeled data, a large amount of data that cannot be accurately identified by the model may greatly hinder the convergence of the model. On the contrary, DYMatch_PL employed the dynamic pseudo-label estimation method, which effectively filter the unlabeled data during the training process. Consequently, compared to DYMatch_CR, DYMatch_PL proved to be more effective in utilizing the unlabeled data. Moreover, the superior performance of DYMatch indicated that the combination of these two methods resulted in a more efficient utilization of unlabeled data.

We also compared the performance of DYMatch when applying feature-correlation consistency regularization method at different layers, i.e., after the backbone $f\left(·\right)$ or after the classification module $g\left(·\right)$. DYMatch_F denotes DYMatch using the feature representation after the backbone $f\left(·\right)$, and DYMatch_G denotes DYMatch employing the feature representation after the classification module $g\left(·\right)$.

Table 7. The Results of the Ablation Study for Features from Different module. The best result is in bold.

Model	Method	CIFAR-10
		40 Labels	4000 Labels
DYMatch_F	After backbone $f\left(·\right)$	4.93 ± 0.11	3.96 ± 0.02
DYMatch_G	After classification module $g\left(·\right)$	6.31 ± 0.08	5.06 ± 0.07

obviously shows that the error rates of DYMatch_G is significantly higher than that of DYMatch_F. This result also demonstrated that performing consistency regularization after the classification module $g\left(·\right)$ limited the performance of the model due to the coupling with the dynamic pseudo-label estimation method.

According to the results shown in Table 7, the dynamic pseudo-label estimation method based on Gaussian mixture model in DYMatch effectively utilizes unlabeled data. Simultaneously, the feature-correlation consistency regularization method focuses effectively on the better pseudo labels, which be selected by the dynamic pseudo-label estimation method. Therefore, the consistency regularization method based on feature correlation can be used as an enhancement to the dynamic pseudo-label estimation method. The effective combination of these two methods can enable the model to obtain excellent classification performance.

Moreover, we also compared DYMatch with other SSL methods in different metrics within the pseudo-labeling method, such as the variations in confidence threshold, the quantity ang accuracy of the selected pseudo-labels. From Figure 2a,b, the confidence threshold of DYMatch gradually increases during the training process. It allows for the utilization of more unlabeled data in the early stages of training, while becoming more cautious in the later stages of training. This situation aligns with the analysis in Section 4.1. Correspondingly, As shown in Figure 2c, DYMatch can obtain the better accuracy of pseudo labels and classification performance compared to FixMatch, AdaMatch and FlexMatch.

4.5. Results for Different Optimizers and Learning Rate Decay Methods

The classification performance of the model was influenced by the choice of optimizers and their hyper-parameters.

Table 8. The Results of the Ablation Study on Different Optimizers with Different Learning Rates (lr) and Momenta (mom). The best result is in bold.

Optimizer	Hyper-Parameters	CIFAR-10
		40 Labels	4000 Labels
SGD	$lr$$=0.02, mom$ = 0.9	4.93 ± 0.11	3.96 ± 0.02
SGD	$lr$$=0.02, mom$ = 0.9, Nesterov	5.05 ± 0.10	4.10 ± 0.12
SGD	$lr=$$0.01, mom$ = 0.9	4.96 ± 0.09	4.06 ± 0.06
SGD	$lr=$$0.01, mom$ = 0.9, Nesterov	5.01 ± 0.13	4.14 ± 0.03
SGD	$lr=$$0.03, mom$ = 0.9	4.92 ± 0.03	3.99 ± 0.10
SGD	$lr=$$0.03, mom$ = 0.9, Nesterov	5.11 ± 0.02	4.04 ± 0.09
Adam	$lr$$=0.01, wd$ = 1 × 10−3	13.13 ± 0.13	10.34 ± 0.10

reveals that SGD with a momentum of 0.9 can produced the best results, and the Adam optimizer yield the worst results. Additionally, the Nesterov method did not lead to a noticeable enhancement in performance. Furthermore, despite experimenting with different initial learning rates, we were not able to obtain a significant improvement in classification performance from this setting.

In DYMatch, we employ the learning rate decay method designed based on the cosine annealing strategy. To verify the performance of the model, we compared this method with two other learning rate decay methods, namely the Exp-warmup method and the method without learning rate decay. The results in

Table 9. The Results for Different Learning Rate Decay Schedulers. The best result is in bold.

Learning Rate Decay Scheduler	CIFAR-10
	40 Labels	4000 Labels
Cosine (min_lr = 1 × 10−3)	4.93 ± 0.11	3.96 ± 0.02
Exp-Warmup (min_lr = 1 × 10−3)	5.13 ± 0.08	4.10 ± 0.04
No Decay	8.02 ± 0.23	6.23 ± 0.11

proved that the cosine annealing scheduler can achievedthe best classification performance.

5. Conclusions and Future Work

Although semi-supervised learning methods have made rapid progress in recent years, most of the more advanced methods are designed based on increasingly complicated learning algorithms. These methods often introduced the complex data perturbation methods or integrated other complex algorithms into the framework of SSL methods. In this paper, we proposed DYMatch, a new SSL algorithm that obtained better performance on a variety of standard SSL benchmark datasets. The main contribution of this paper lies in the introduction of a dynamic pseudo-label estimation method based on Gaussian mixture models. This strategy effectively improves the utilization of unlabeled data by dynamically estimating class-based confidence thresholds, thereby enhancing the model’s generalization performance. Moreover, the combination of this strategy with a feature-correlation consistency regularization method also significantly enhances the efficiency of utilizing unlabeled data. The good experimental results obtained illustrate the effectiveness of DYMatch on SSL classification tasks. We hope that this direction can serve as inspiration for further research, such as how to design better feature regularization methods that can more effectively utilize multi-domain dataset. If you are interested in our work and want to apply our methods to other fields, please feel free to contact the author.