A Framework Using Contrastive Learning for Classification with Noisy Labels

Abstract

We propose a framework using contrastive learning as a pre-training task to perform image classification in the presence of noisy labels. Recent strategies, such as pseudo-labeling, sample selection with Gaussian Mixture models, and weighted supervised contrastive learning have, been combined into a fine-tuning phase following the pre-training. In this paper, we provide an extensive empirical study showing that a preliminary contrastive learning step brings a significant gain in performance when using different loss functions: non robust, robust, and early-learning regularized. Our experiments performed on standard benchmarks and real-world datasets demonstrate that: (i) the contrastive pre-training increases the robustness of any loss function to noisy labels and (ii) the additional fine-tuning phase can further improve accuracy, but at the cost of additional complexity.

1. Introduction

Collecting large and well-annotated datasets for image classification tasks represents a challenge as human quality annotations are expensive and time-consuming. Alternative methods exist, such as web crawlers [1]. Nevertheless, these methods generate noisy labels decreasing the performance of deep neural networks. They tend to overfit to noisy labels due to their high capacity [2]. That is why developing efficient noisy-label learning (NLL) techniques is of great importance.

Various strategies have been proposed to deal with NLL: (i) Noise transition matrix [3,4,5] estimates the noise probability and corrects the loss function, (ii) a small and clean subset can help to avoid overfitting [6], (iii) sample selection identifies true-labeled samples [7,8,9], and (iv) robust loss functions solve the classification problem only by adapting the loss function to be less sensitive to noisy labels [10,11,12]. Methods also combine other strategies (e.g., ELR+ [9], DivideMix [13]): two networks, semi-supervised learning, label correction, or mixup. They show the most promising results but lead to a large number of hyperparameters. That is why we explore improvement strategies for robust loss functions. They are simpler to integrate and faster to train, but as illustrated in Figure 1, they tend to overfit and have lower performance for high noise ratios.

Meanwhile, new self-supervised contrastive learning algorithms for image representations have been recently developed [14,15]. Such algorithms extract representation (or features) in unsupervised settings by comparing among the input samples. These representations can then be used for downstream tasks such as classification. For this task, methods based on contrastive learning and fine-tuned only on a small fraction of all available labels compete with fully supervised learning. Therefore, using contrastive learning to extract features independently of the amount of noise for NLL appears as promising.

This work is a wide experimental exercise analyzing the current state-of-the-art in noisy label image classification and evaluating if recent (unsupervised) contrastive learning techniques can also be used to provide robustness to classifiers trained in a noisy setting with non-robust and robust loss functions. The key contributions of our work are:

• A framework increasing robustness of any loss function to noisy labels by adding a contrastive pre-training task.
• The adaptation of the supervised contrastive loss to use sample weight values, representing the probability of correctness for each sample in the training set
• An extensive empirical study identifying and benchmarking additional state-of-the-art strategies to boost the performance of pre-trained models: pseudo-labeling, sample selection with GMM, weighted supervised contrastive learning, and mixup with bootstrapping.

2. Related Works

Existing approaches dealing with NLL and contrastive learning in computer vision are briefly reviewed. Extra details can be found in Song et al. [16], Le-Khac et al. [17].

2.1. Noise Tolerant Classification

Sample Selection: This method identifies noisy and clean samples within the training data. Several strategies leverage the interactions between multiple networks to identify the probably correct labels [7,8,9]. Recent works [18,19] exploit the small loss trick to identify clean and noisy samples by considering a certain number of small-loss training samples as true-labeled samples. This approach can be justified by the memorization effect: deep neural networks first fit the training data with clean labels during a so-called early learning phase, before overfitting the noisy samples during the memorization phase [13,20].

Robust Loss Function: Commonly used loss functions, such as Cross Entropy (CE) or Focal Loss, are not robust to noisy labels. Therefore, new loss functions have been designed. Such robust loss functions can be easily incorporated into existing pipelines to improve performance regarding noisy labels. The symmetric Cross Entropy [11] has been proposed by adding a reverse CE loss to the initial CE. This combination improves the accuracy of the model compared to classical loss functions. Ma et al. [12] show theoretically that normalization can convert classical loss functions into loss functions robust to noise labels. The combination of two robust loss functions can also improve robustness. However, the performance of normalized loss functions remains quite low for high noise rates, as illustrated in Figure 1.

Semi-supervised: Semi-supervised approaches deal with both labeled and unlabeled data. Recent works [9,21,22] combine sample selection with semi-supervised methods: the possibly noisy samples are treated as unlabeled and the possibly clean samples are treated as labeled. Such approaches leverage information contained in noisy data, for instance by using MixMatch [23]. Semi-supervised approaches show competitive results. However, they use several hyperparameters that can be sensitive to changes in data or noise type [16,24].

Contrastive learning: recent developments in self-supervised and contrastive learning [24,25,26] inspire new approaches in NLL. Li et al. [26] employed features learned by contrastive learning to detect out-of-distribution samples.

2.2. Contrastive Learning for Vision Data

Contrastive learning extracts features by comparing each data sample with different samples. The central idea is to bring different instances of the same input image closer and spread instances from different images apart. The inputs are usually divided into positive (similar inputs) or negative pairs (dissimilar inputs). Data augmentation is typically used to create positive pairs. Frameworks have been recently developed, such as CPCv2 [27], SimCLR [14], Moco [15]. Once the self-supervised model is trained, the extracted representations can be used for downstream tasks. In this work, the representations are used for noisy label classification.

Chen et al. [14] demonstrate that large sets of negatives (and large batches) are crucial in learning good representations. However, large batches are limited by GPU memory. Maintaining a memory bank accumulating a large number of negative representations is an elegant solution decoupling the batch size from the number of negatives [28]. Nevertheless, the representations get outdated in a few iterations. The Momentum Encoder [15] addresses the issues by generating a dynamic memory queue of representations. Other strategies aim at getting more meaningful negative samples to reduce the memory/batch size [29].

Recent methods in contrastive learning have been developed to avoid computing pairwise comparisons with negative samples. The Bootstrap Your Own Latent (BYOL) algorithm [30] does not use any negative samples and compares only positive pairs with a momentum encoder (similar to Moco) and a stop-gradient mechanism. This mechanism is also used in the SimSiam method [31]. Caron et al. [32] introduce a cluster algorithm and enforce consistency between cluster assignments for different augmentations instead of a direct pairwise comparison.

3. Preliminaries

Let $D={\left\{({\mathit{x}}_{\mathit{i}},\overline{y_i})\right\}}_{i=1..n},{\mathit{x}}_{\mathit{i}}\in {\mathbb{R}}^{d_1\times d_2},\overline{y_i}\in \{1,\cdots ,K\}$ denote a noisy input dataset with an unknown number of samples incorrectly labeled. The associated true and unobservable labels are written as $y_i$. The images ${\mathit{x}}_{\mathit{i}}$ are of size $d_1\times d_2$ and the classification problem has K classes. The goal is to train a deep neural network (DNN) f. Using a robust loss function for training consists of minimizing the empirical risk defined by robust loss functions in order to find the set of optimal parameters $\theta$. The one-hot encoding of the label is denoted by the distribution $q\left(k\right|\mathit{x})$ for a sample $\mathit{x}$ and a class k, such as $q\left(y_i\right|{\mathit{x}}_{\mathit{i}})=1$ and $q(k\ne y_i|{\mathit{x}}_{\mathit{i}})=0,\forall i\in \{1,\cdots ,n\}$. The probability vector of f is defined by the softmax function $p\left(k\right|\mathit{x})=\frac{e^{z_k}}{{\sum }_{j=1}^K{e}^{z_j}}$ where $z_k$ denotes the logits output with respect to class k.

3.1. Classification with Robust Loss Functions

Our method employs noise-robust losses to train the classifier in the presence of noisy labels. Such losses improve the classification accuracy compared to the commonly used Cross Entropy (CE), as illustrated in Figure 1. In this section, the general empirical risk for a given mini-batch is defined by $L={\sum }_{i=1}^N\mathcal{L}(f\left(x_i\right),\overline{y_i})={\sum }_{i=1}^N{l}_i$. The term $l_i$ is modified by each loss function.

The classical CE is used as a baseline loss function not robust to noisy labels [33] and is defined as:

$$l_{ce}=-\sum _{k=1}^{K}q\left(k\right|{\mathit{x}}_{\mathit{i}})log\left(p\left(k\right|{\mathit{x}}_{\mathit{i}})\right).$$

(1)

As presented in Section 2, Ma et al. [12] introduce robust loss functions called Active Passive Losses that do not suffer from underfitting. We investigate the combination between the Normalized Focal Loss (NFL) and the Reversed Cross Entropy (RCE) called NFL+RCE. It shows promising results on various benchmarks. The NFL is defined as:

$$l_{nfl}=\frac{-{\displaystyle \sum _{k=1}^K}q\left(k\right|{\mathit{x}}_{\mathit{i}}){(1-p\left(k\right|{\mathit{x}}_{\mathit{i}}))}^{\gamma }log\left(p\left(k\right|{\mathit{x}}_{\mathit{i}})\right)}{-{\displaystyle \sum _{j=1}^K}{\displaystyle \sum _{k=1}^K}q(y=j|{\mathit{x}}_{\mathit{i}}){(1-p\left(k\right|{\mathit{x}}_{\mathit{i}}))}^{\gamma }log\left(p\left(k\right|{\mathit{x}}_{\mathit{i}})\right)},$$

(2)

where $\gamma \ge 0$ is a hyperparameter. The RCE loss is:

$$l_{rce}=-\sum _{k=1}^{K}p\left(k\right|{\mathit{x}}_{\mathit{i}})log\left(q\left(k\right|{\mathit{x}}_{\mathit{i}})\right).$$

(3)

The final combination following the framework simply gives a different $\alpha$ and $\beta$ to each loss:

$$l_{nfl+rce}=\alpha .l_{nfl}+\beta .l_{rce}.$$

(4)

The two hyperparameters $\alpha$ and $\beta$ control the balancing between more active learning and less passive learning. For simplicity, $\alpha$ and $\beta$ are set to 1.0 without any tuning.

Liu et al. [13] propose another framework to deal with noisy annotations based on the “early learning” phase. The loss, called Early Learning Regularization (ELR), adds a regularization term to capitalize on early learning. ELR is not, strictly speaking, a robust loss but belongs to robust penalization and label correction methods. The penalization term corrects the CE based on estimated soft labels identified with semi-supervised learning techniques. It prevents memorization of false labels by steering the model towards these targets. The regularization term maximizes the inner product between model outputs and targets:

$$l_{elr}=l_{ce}+\frac{{\lambda }_{elr}}{N}log\left(1-\sum _{k=1}^{K}p\left(k\right|{\mathit{x}}_{\mathit{i}})t\left(k\right|{\mathit{x}}_{\mathit{i}})\right).$$

(5)

The target is not set equal to the model output but is estimated with a temporal ensembling from semi-supervised methods. Let $t{\left(k\right|{\mathit{x}}_{\mathit{i}})}^{\left(l\right)}$ denote the target for example ${\mathit{x}}_{\mathit{i}}$ at iteration l of training with a momentum $\beta$:

$$t{\left(k\right|{\mathit{x}}_{\mathit{i}})}^{\left(l\right)}=\beta t{\left(k\right|{\mathit{x}}_{\mathit{i}})}^{(l-1)}+(1-\beta )p{\left(k\right|{\mathit{x}}_{\mathit{i}})}^{\left(l\right)}.$$

(6)

A more complex method improving accuracy, called ELR+, has also been developed and combines weight averaging, two parallel networks, and a mixup data augmentation.

The importance in the choice of the loss function is also discussed for other applications, such as image segmentation [34] or survival data [35].

3.2. Contrastive Learning

Contrastive learning methods learn representations by contrasting positive and negative examples. A typical framework is composed of several blocks [36]:

• Data augmentation: Data augmentation is used to decouple the pretext tasks from the network architecture. Chen et al. [14] study broadly the impact of data augmentation. We follow their suggestion combining random crop (and flip), color distortion, Gaussian blur, and gray-scaling.
• Encoding: The encoder extracts features (or representation) from augmented data samples. A classical choice for the encoder is the ResNet model [37] for image data. The final goal of the contrastive approach is to find correct weights for the encoder.
• Loss function: The loss function usually combines positive and negative pairs. The Noise Contrastive Estimation (NCE) and its variants are popular choices. The general formulation for such a loss function is defined for the ith pair as [38]:

  $$L_i=-log\frac{exp({\mathit{z}}_{\mathit{i}}^T{\mathit{z}}_{\mathit{j}\left(\mathit{i}\right)}/\tau )}{{\sum }_{a\in A\left(i\right)}exp({\mathit{z}}_{\mathit{i}}^T{\mathit{z}}_{\mathit{a}}/\tau )},\mathrm{with}i\in I,$$

  (7)

  where $\mathit{z}$ is a feature vector, I is the set of indexes in the mini-batch, i is the index of the anchor, $j\left(i\right)$ is the index of an augmented version of the anchor source image, $A\left(i\right)=I\backslash \left\{i\right\}$, and $\tau$ is a temperature controlling the dot product. The denominator includes one positive and K negative pairs. The temperature has two competing effects on the loss function: low temperatures help the model to learn from hard negatives while high temperatures allow to use larger learning rate making the optimization easier, and classes more separated [14,39].
• Projection head: That step is not used in all frameworks. The projection head maps the representation to a lower-dimensional space and acts as an intermediate layer between the representation and the embedding pairs. Chen et al. [14,31] show that the projection head helps to improve the representation quality.

4. A Framework Coupling Contrastive Learning and Noisy Labels

As illustrated in Figure 2, our method classifies noisy samples in a two-phased process. First, a classifier pre-trained with contrastive learning produces train-set pseudo-labels (pre-training phase, in panel a), used during the training of a subsequent fine-tuning phase (panel b). The underlying intuition is that the predicted pseudo-labels are more accurate than the original noisy labels. The contrastive learning step performed in the first phase (panel a1) is supposed to reduce the label noise sensitivity of the classifier (panel a2) thanks to the computed representations; the resulting model can be also used in a standalone way with a reduced number of hyperparameters, without the underlying fine-tuning phase.

The second phase leverages the pseudo-labels predicted by the pre-training in all underlying steps (b1–b3). To mitigate the effect of potentially incorrectly predicted pseudo-labels, a Gaussian Mixture Model (GMM, panel b1) with 2 components follows the small loss-trick to predict for each sample the probability of correctness. This value is used as a weight in a supervised contrastive step (panel b2), performed to improve the learned representations by taking advantage of the label information. A classification head is added to the contrastive model in order to produce the final predictions (panel b3). The fine-tuning phase can be seen as an adaptation of the pre-training phase to handle pseudo-labels.

To maximize the impact of the contrastive learning on the underlying classification, the supervised training is performed in 2 steps: a warm-up step, updating only the classifier layer (while keeping the encoder frozen) is followed by the full model training. We compared three different loss functions for the supervised classification: the classical CE, the robust NFL+RCE, and the ELR loss.

4.1. Sample Selection and Correction with Pseudo-Labels

Pseudo labels represent one hot encoded model’s predictions on the training set. Pseudo-labels were initially used in semi-supervised learning to produce annotations for unlabeled data; in the noisy label setting, various techniques (e.g., DivideMix) identify a subset with a high likelihood of correctness and treat the remaining samples as the unlabeled counterpart in semi-supervised learning. In this work, we elaborate on the observation that the training set labels, predicted after training the model with a noise-robust loss function (i.e., the pseudo labels), are more accurate than the ground truth. This observation is supported by the results in Figure 3, depicting the accuracy of pseudo labels predicted on CIFAR100, contaminated with various levels of asymmetric (panel a) and symmetric (panel b) noise. The pseudo labels are more accurate than the corrupted ground truth in both settings and bring a higher gain in performance as the noise ratio increases.

As proposed by other approaches [18], the loss value on train samples can be used to discriminate between clean and mislabeled samples. The sample correctness probability is computed by fitting a 2-components GMM on the distribution of losses [9]. The underlying probability is used as a sample weight:

$$w_i=p(k=0|l_i),$$

(8)

where $l_i$ is the loss for sample i and $k=0$ is the GMM component associated to the clean samples (lowest loss). Figure 4 depicts the evolution of the clean training set identified by GMM on an example: its accuracy grows from 0.6 to 0.93 while the size stabilizes at 60% of the training set.

4.2. Weighted Supervised Contrastive Learning

A modification to the contrastive loss defined in Equation (7) has been proposed to leverage label information [39]:

$$L_i=-log\frac{1}{\left|P\right(i\left)\right|}\sum _{p\in P\left(i\right)}\frac{exp({\mathit{z}}_{\mathit{i}}^T{\mathit{z}}_{\mathit{p}}/\tau )}{{\sum }_{a\in A\left(i\right)}exp({\mathit{z}}_{\mathit{i}}^T{\mathit{z}}_{\mathit{a}}/\tau )},$$

(9)

where $P\left(i\right)=\{j\in I\backslash \left\{i\right\},y_j=\widetilde{y_i}\}$ with $\widetilde{y_i}$ the prediction of the model for input ${\mathit{x}}_{\mathit{i}}$.

As explained in the previous section, the loss value for the training set sample is used to fit a GMM with 2 components, corresponding to correctly and incorrectly labeled samples. We adapted the supervised representation loss to employ w, a weighting factor representing the sample probability of membership to the correctly labeled component. Thus, likely mislabeled samples having large loss values would contribute only marginally to the supervised representations:

$$L_i=-log\frac{1}{\left|P\right(i\left)\right|}\sum _{p\in P\left(i\right)}\frac{\widetilde{w_{p,i}}exp({\mathit{z}}_{\mathit{i}}^T{\mathit{z}}_{\mathit{p}}/\tau )}{{\sum }_{a\in A\left(i\right)}exp({\mathit{z}}_{\mathit{i}}^T{\mathit{z}}_{\mathit{p}}/\tau )},$$

(10)

where $\widetilde{w_{p,i}}$ is a modified version of $w_p$ such as $\widetilde{w_{p,i}}=1$ if $p=j\left(i\right)$ else $\widetilde{w_{p,i}}=w_i$. If all samples are considered as noisy, Equation (10) is simplified into the classical unsupervised contrastive loss in Equation (7).

5. Experiments

The framework is assessed on three benchmarks and the contribution of each block identified in Figure 2 is analyzed.

5.1. Datasets

CIFAR10 and CIFAR100 [40]. These experiments assess the accuracy of the method against synthetic label noise. The two datasets are contaminated with simulated symmetric or asymmetric label noise reproducing the heuristic in Ma et al. [12]. The symmetric noise consists of corrupting an equal arbitrary ratio of labels for each class. The noise level varies from $0.2$ to $0.8$. For asymmetric noise [3,13], sample labels have been flipped within a specific set of classes, thus providing confusion between predetermined pairs of labels. For CIFAR100, 20 groups of super-classes have been created, each consisting of 5 sub-classes. The label flipping is performed only within each super-class circularly. The asymmetric noise ratio is explored between $0.2$ and $0.4$.

Webvision [41]. This is a real-world dataset with noisy labels. It contains 2.4 million images crawled from the web (Google and Flickr) that share the same 1000 classes from the ImageNet dataset. The noise ratio varies from 0.5% to 88%, depending on the class. In order to speed-up the training time, we used mini Webvision [7], consisting of only the top 50 classes in the Google subset (66,000 images).

Clothing1M [42]. Clothing1M is a large real-world dataset consisting of 1 million images on 14 classes of clothing articles. Being gathered from e-commerce websites, Clothing1M embeds an unknown ratio of label noise. Additional validation and test sets, consisting of 14 k and 10 k clean labeled samples, have been made available. In order to speed-up the training time, we selected a subset of 56,000 images keeping the initial class distribution.

Both Webvision and Clothing1M images were resized to $128\times 128$. Therefore, the reported results may differ from other papers cropping the images to a $224\times 224$ resolution. Numerical details about the different datasets can be found in Appendix A.

5.2. Settings

We use the contrastive SimCLR framework [14] (https://github.com/HobbitLong/SupContrast (accessed on 14 December 2020)) with a ResNet18 [37] (without ImageNet pre-training) as the encoder. A projection head was added after the encoder for the contrastive learning (of dimension 128 for CIFAR and dimension 512 for Webvision and Clothing1M) with the following architecture: a multi-layer perceptron with one hidden layer and a ReLU non-linearity. The classifier following the contrastive learning step has a simple multi-layer architecture: a single hidden layer with batch normalization and a ReLU activation function. A comparison with a linear classifier is provided in Appendix C.3.

For all supervised classification, we use the SGD optimizer with momentum 0.9 and cosine learning rate annealing. The NFL hyperparameter $\gamma$ is set to $0.5$. Unlike the original paper, the ELR hyperparameters do no depend on the noise type: the regularization coefficient ${\lambda }_{elr}$ and the momentum $\beta$ are set to $3.0$ and $0.7$. Details on the experiment setting can be found in Appendix B.

All codes are implemented in the PyTorch framework [43]. The experiments for CIFAR are performed with a single Nvidia TITAN V-12GB and the experiments for Webvision and Clothing1M are performed with a single Nvidia Tesla V100-32GB, demonstrating the accessibility of the method. Our implementation has been made available along with the Supplementary Materials.

6. Results

All experiments presented in this section evaluate our method’s performance with the top-1 accuracy score.

6.1. Impact of Contrastive Pre-Training

To evaluate the impact of the contrastive pre-training on the classification model, the proposed method (pre-training phase) is compared with a baseline classifier, trained for 200 epochs without contrastive learning. For each simulated dataset, we compare robust losses (e.g., NLF+RCE and ELR) and Cross Entropy. Results for CIFAR10 and CIFAR100 are depicted in

Table 1. Results on both CIFAR10 and CIFAR100 using symmetric noise (0.2–0.8) and asymmetric noise (0.2–0.4). We compare training from scratch or from pre-trained representation. Best scores are in bold for each noise scenario and each loss.

			CIFAR10		CIFAR100
Type	$\mathit{\eta }$	Loss	Base	Pre-t.	Base	Pre-t.
Sym	0.2	ce	77.2	87.7	55.6	56.5
		elr	90.3	93.0	64.1	67.4
		nfl+rce	91.0	92.7	66.6	68.8
	0.4	ce	58.2	78.0	39.9	41.9
		elr	82.3	92.0	56.9	62.0
		nfl+rce	87.0	91.4	60.2	66.3
	0.6	ce	35.2	59.2	21.8	26.8
		elr	64.2	90.4	40.6	55.7
		nfl+rce	80.2	88.1	47.0	61.8
	0.8	ce	17.0	27.3	7.80	12.4
		elr	18.3	84.8	16.2	45.3
		nfl+rce	42.8	59.9	20.1	50.2
Asym	0.2	ce	84.0	87.9	59.0	57.8
		elr	91.8	92.4	70.3	70.2
		nfl+rce	90.2	91.5	63.9	68.4
	0.3	ce	79.2	83.9	50.6	50.4
		elr	89.6	91.7	69.8	69.3
		nfl+rce	86.7	89.9	53.5	63.5
	0.4	ce	75.3	77.8	41.8	42.4
		elr	72.3	89.5	67.6	67.6
		nfl+rce	80.0	82.4	40.6	47.8

for different levels of symmetric and asymmetric noise. The pre-training improves the accuracy of the three different baselines for both datasets with different types and ratios of label noise. The largest differences are observed for the noisiest case with $80\%$ noise. The pre-training outperforms the baselines by large margins between 10 and 75 for CIFAR10 and between 5 and 30 for CIFAR100.

In addition to the comparisons with ELR and NFL+RCE, performed using our implementations (column Base in Table 1), we present the results reported by other recent competing methods. As shown in the Introduction, numerous contributions have been made to the field in the last years. Six recent representative methods are selected for comparison: Taks [44], Co-teaching+ [45], ELR [13], DivideMix [9], SELF [21], and JoCoR [46]. The results are presented in

Table 2. Accuracy scores compared with 6 methods (Taks, Co-teaching+, ELR, DivideMix, SELF, and JoCoR) on CIFAR10 (C10) and CIFAR100 (C100). The cases most affected by dropout are presented, with symmetric (S) and asymmetric (A) noise. Top-2 scores are in bold.

	C10 80% S	C10 40% A	C100 80% S	C100 40% A
Ours (ELR)	84.8	89.5	45.3	67.6
ELR [13]	73.9	91.1	29.7	73.2
Taks [44]	40.2	73.4	16.0	35.2
Co-teach+ [45]	23.5	68.5	14.0	34.3
DivideMix [9]	92.9	93.4	59.6	72.1
SELF [21]	69.9	89.1	42.1	53.8
JoCoR [46]	25.5	76.1	12.9	32.3

. The difference between the scores reported by ELR and those obtained with our run (using the same implementation, but slightly different hyperparameters and a ResNet18 instead of a ResNet34) suggests that the method is less stable on data contaminated with asymmetric noise and sensitive to small changes hyperparameters. Moreover, ELR proposes hyperparameters having different values depending on the type of dataset (i.e., CIFAR10/CIFAR100) and underlying noise (i.e., symmetric/asymmetric), identified after a hyperparameter search exercise. The best scores are reported by DivideMix and they surpass all other techniques. One can note DivideMix uses a PreAct ResNet18 while we use a classical ResNet18. Moreover, a recent study [24] attempted to replicate these values and reported significantly lower results on CIFAR100 (i.e., $49.5\%$ instead of $59.6\%$ on symmetric data and $50.9\%$ instead of $72.1\%$ on asymmetric data). Our framework compares favorably with the other competing methods, both on symmetric and asymmetric noise.

Webvision and Clothing1M results are presented in

Table 3. Top-1 accuracy for mini-Webvision and Clothing1M. Best scores are in bold for each dataset and each loss. Pre-t represents the pre-training phase while Fine-tune refers to the results after the fine-tuning step.

	Webvision			Clothing1M
Loss	Base.	Pre-t.	Fine-Tune	Base.	Pre-t.	Fine-Tune
ce	51.8	57.1	58.4	54.8	59.1	61.5
elr	53.0	58.1	59.0	57.4	60.8	60.4
nfl+rce	49.9	54.8	58.2	57.4	59.4	60.1

. The contrastive framework outperforms the respective baselines for the three loss functions. As a result that the images have a reduced size, and for Clothing1M, we use a smaller training set, the direct comparison with competing methods is less relevant. However, the observed gap in performance is significant and promising for training images with higher resolution. Moreover, a ResNet50 model has been trained with our framework on the Webvision dataset with a higher resolution ($224\times 224$). The accuracy reaches $75.7\%$ and $76.2\%$ for CE and ELR, respectively. These results are very close to the values reported with DivideMix ($77.3\%$) and ELR+ ($77.8\%$) using a larger model, Inception-ResNet-v2 (the difference is more than $4\%$ on the ImageNet benchmark [47]).

Supported by this first set of experiments, the preliminary pre-training with contrastive learning shows great performances. The accuracy of both traditional and robust-loss classification models is significantly improved.

6.2. Sensitivity to the Hyperparameters

Estimating the best hyperparameters is complex for datasets with noisy labels as clean validation sets are not available. For instance, Ortego et al. [24] show that two efficient methods (e.g., ELR and DivideMix) could be sensitive to specific hyperparameters. Therefore, a hyperparameter sensitivity study has been carried out to estimate the stability of the framework for the learning rate. Figure 5 depicts the sensitivity on CIFAR100 with $80\%$ noise. CE and NFL+RCE seem to have opposite behaviors. The CE reaches competitive results with small learning rates but tends to overfitting for higher learning rates. The NFL+RCE loss tends to underfitting for the lowest learning rates but is quite robust for higher values. The ELR loss has the smallest sensitivity to the learning for the investigated range but does not reach the best values obtained with CE or NFL+RCE. We can assume that the regularization term coupled with pre-training is very efficient. It prevents memorization of the false labels as observed with CE. Results for other noise ratios are documented in Appendix C.2.

This sensitivity analysis is limited to the learning rate. Investigating the impact of other hyperparameters, such as the momentum $\beta$ or the regularization factor ${\lambda }_{elr}$, could be interesting. In their original papers, ELR and NFL+RCE reach respectively $25.2\%$ and $30.3\%$ with other hyperparameters. These values are still far from the improvements brought by the contrastive pre-training but it suggests that the results could be improved with different hyperparameters.

Our empirical results indicate that the analyzed methods may be sensitive to hyperparameters. Despite the promised robustness to label noise, the analyzed robust losses are also affected by overfitting or underfitting. Our experiments have been built upon the parameters recommended in each issuing paper (e.g., ELR, SIMCLR) but, since the individual building blocks can be affected by small variations in input parameters, the performance of our method may also be impacted. Finding a relevant method to estimate proper hyperparameters in NLL remains a challenge.

As overfitting remains an important and recurrent issue affecting the training of noisy label models, we explored three strategies to identify when overfitting starts, without using a clean validation set. This constraint was imposed to simulate the real-world scenario, when only access to a potentially noise-corrupted dataset is provided. First, the behavior of a corrupted validation set was analyzed on symmetric and asymmetric datasets, but we failed to establish a consistent relation between the start of overfitting and the trend in the accuracy of the validation set. The accuracy of the validation set increases for the first few training epochs and then stabilizes over a plateau phase. Secondly, we explored a recent contribution, attempting to identify the Training Stop Point (TSP) [48] under similar conditions, but our experimental tests have shown that proposed heuristic does not consistently apply to all studied types of noise. Thirdly, we studied the Centered Kernel Alignment (CKA) [49] characterizing the representations created in the final network layers, but we could not identify a change in stability associated with the start of overfitting. Thus, we conclude that in the absence of a clean validation set, identifying when overfitting starts remains an open challenge for noisy labels. The three summarized experiments are detailed in Appendix G.

6.3. Impact of the Fine-Tuning Phase

Experimental results on synthetic label noise, depicted in Figure 6, show that continuing the presented pre-training block (Figure 2) with the fine-tuning phase increases the accuracy in over 65% of cases on CIFAR10 and over 80% of cases on CIFAR100. For both datasets, asymmetric noise data benefit more from this approach than symmetric noise. All experiments only use the input parameters proposed in the loss-issuing papers.

The sample selection has also got a positive impact on the two real-world datasets, as shown in Table 3 by the “Fine-tune” columns. The average accuracy improvement is about $1.8\%$. Only the ELR loss function slightly decreases the performance on Clothing1M.

Enriching pre-trained models with sample weighting and selection, pseudo labels instead of corrupted targets, and supervised contrastive pre-training can improve the classification accuracy. However, such an approach raises the question of a trade-off between complexity, accuracy improvement, and computation time.

7. Discussion and Limits of the Framework

In addition to the presented fine-tuning phase, we evaluated the performance of other promising techniques, such as the dynamic bootstrapping with mixup [18]. This strategy has been developed to help convergence under extreme label noise conditions. Details can be found in Appendix D. The improvement that dynamic bootstraping can bring when used after pre-training is depicted in Figure 7. In most of the cases, this technique improves the accuracy, as indicated by the positive accuracy gain scores, measuring the difference between the accuracy after dynamic bootstraping and the accuracy of the pre-training phase. ELR and CE benefit most from this addition for CIFAR100. The impact of the dynamic boostrapping should also be analyzed for the fine-tuning phase and for larger datasets, such as Webvision or Clothing1M.

One of the major drawbacks of our method is the extra computational time needed to learn representations with contrastive learning. A detailed study, comparing the execution time of our framework with 6 other competing methods has been provided in Appendix F. The pre-training phase doubles the execution time of a reference baseline, consisting of performing only a single classification step, while the entire framework increases the execution time 3 to 4 times the baseline value. However, the contrastive learning does not increase the need for GPU memory if the batch size is limited for the contrastive learning [15,50]. The computational time could be reduced by initializing the contrastive step with the pre-trained weights from ImageNet.

Most state-of-the-art approaches also leverage computationally expensive settings, consisting of larger models (e.g., ResNet50), dual model training, or data augmentation such as mixup. In this work, we explored the limits of a restricted computational setting, consisting of a single GPU and 8GB RAM. All experiments use a ResNet18 model, batch sizes of 256, and for real-world datasets, the images have been rescaled (e.g., $128\times 128$ instead of $224\times 224$). We also foresee that the contrastive learning step could be improved by images with higher resolutions as smaller details could be identified in the representation embedding.

There remain multiple open problems for future research, such as: (i) identifying the start of the memorization phase in the absence of a clean dataset, (ii) studying the impact of contrastive learning on other models for noisy labels such as DivideMix, (iii) comparing SimCLR approach in the context of noisy labels with other contrastive frameworks (the impact of Moco is studied in Appendix C.1) and other self-supervised approaches, and (iv) having a better theoretical understanding of the interaction between the initial state pre-computed with contrastive learning and the classifier in presence of noisy labels. Moreover, the analysis carried out in this work should be validated on larger settings, in particular on Clothing1M with a ResNet50, higher resolutions, and the full dataset.

8. Conclusions

In this work, we presented a contrastive learning framework optimized with several adaptations for noisy label classification. Supported by an extensive range of experiments, we conclude that a preliminary representation pre-training improves the performance of both traditional and robust-loss classification models. Additionally, multiple techniques can be used to fine-tune and further optimize these results; however, no approach provides a significant improvement systematically on all types of datasets and label noise. The Cross Entropy penalized by Early Learning Regularization (ELR) shows the best overall results for synthetic noise but also real-world datasets.

However, the training phases remain sensitive to input configuration. Overfitting is the common weakness of all studied models. When trained with tuned parameters, even traditional (Cross Entropy) models provide competitive results, while robust-losses are less sensitive. The typical noisy label adaptations, such as sample selection or weighting, the usage of pseudo labels, or supervised contrastive losses, improve the performance to a lesser extent but increase the framework’s complexity. We hope that this work will promote the use of contrastive learning to improve the robustness of the classification process with noisy labels.